Article Preview
TopIntroduction
Research project evaluation is an important task in government funding agencies and research institutions (Wang et al., 2013). Key performance factors used in evaluating research projects include: 1) scientific merits on the novelty and impact of the research project; 2) relevance of the research to the mission and priorities of the funding scheme; and 3) output of the research project, that is, academic publications, awards, invention patents, and collaborations with other researchers, institutions, and industries.
Peer review methods are widely used in evaluating research projects, where domain experts are invited to evaluate the scientific merit, relevance, and research outputs of completed research projects (Bence & Oppenheim, 2004). Scientific review panels may be invited to make final decisions on the undecided evaluation results. Hence, reviewer assignment plays a significant role in controlling the quality of project evaluation (Thushari et al., 2014), and many approaches have been proposed to support reviewer assignment (Liu et al., 2016), such as heuristic algorithm (Cook et al., 2005), hybrid knowledge, and model approach (Sun et al., 2008). As a main type of research project evaluation method, metrics-based approaches are also used to evaluate research projects based on specific metrics, such as the number and quality of publications, awards, and patents generated by the research project.
Research information systems, a decision support system for government funding agencies to support research management, are developed to track the progress of research projects, monitor research outputs, and evaluate the impact of research projects. These systems are used to streamline the evaluation process, ensure transparency, and facilitate knowledge sharing among researchers in universities and industries. With the increased number of funded research projects, it is becoming difficult to find relevant peer reviewers in the subject disciplines to evaluate the research projects, as a result, the evaluation results may be inconsistent. Current research information systems mainly focus on quantitative analysis that evaluates research projects based on statistical information of their research outputs (Donovan, 2007). The existing research evaluation approaches ignore the qualitative assessment on the novelty and impact of research projects and their relevance to the funding scheme.
This paper aims to propose an integrated approach that leverages machine learning techniques and large language models for the evaluation of research projects. Machine learning methods are employed to calculate the weights and scores of key performance indicators (KPIs) for quantitative analysis. Large language models are utilised to assist in processing text contents and then summarizing the research contributions or findings of the projects for qualitative assessment.
Experiments have been conducted in practical settings, and the results have shown that the proposed decision support framework can assist decision-makers in achieving more consistent evaluation results for funded research projects. It significantly improves the efficiency of research management work.