Mean Reciprocal Rank (MRR) assigns the inverse value of the position in which the correct answer is found (1 if the first, ½ if the second, ¼ if the fourth, and so on), or zero if there is no correct response. This measure considers only the first correct response shown on the list of results offered by the system, and the final value is the average of the values found for each question. MRR assigns a high value to the responses that were in the highest classification positions.
Published in Chapter:
Analysis and Assessment of Cross-Language Question Answering Systems
Copyright: © 2018
|Pages: 9
DOI: 10.4018/978-1-5225-2255-3.ch388
Abstract
Within the sphere of the Web, the overload of information is more notable than in other contexts. Question answering systems (QAS) are presented as an alternative to the traditional Information Retrieval (IR) systems, seeking to offer precise and understandable answers to factual questions instead of showing the user a list of documents related to a given search . Given that the QAS is presented as a substantial advance in the improvement of IR, it becomes necessary to determine its effectiveness for the final user. With this aim, 7 studies were undertaken to evaluate: a) in the first two, the linguistic resources and tools used in these systems for multilingual retrieval (Research 1; Research 2); and b) the performance and quality of the answers of the main monolingual and multilingual QA of general domain and specialized domain in the Web in response to different types of questions and subjects, so that different evaluation means can be applied (Research 3, Research 4, Research 5, Research 6, Research 7).