Evaluation of chatbots can be approached from various perspectives, including user satisfaction, task completion rate, conversational quality, and system performance (Cai et al., 2022). Several studies have investigated the evaluation of chatbots from these perspectives. For instance, in a survey by Mokmin and Ibrahim (2021), the authors evaluated the performance of a chatbot designed to provide education on health support to college students. The study found that the chatbot helped 73.3% of responders grasp health concerns and have pleasant conversations. Similarly, in a study, the authors evaluated the effectiveness of a chatbot designed to help tourists. The study found that user satisfaction depends on the chatbot's informativeness, empathy, and interactivity (Orden-Mejía & Huertas, 2022).
Other studies have focused on the conversational quality of chatbots. For example, Barletta et al. (2023) evaluated a chatbot's ability to converse with users informally. Similarly, in another study, the authors assessed the conversational quality of chatbots using a human evaluation metric. The study found that the chatbot could generate responses that were rated similar to those generated by humans, indicating that the chatbot could maintain natural and engaging conversations with users.
Despite these promising results, the evaluation of chatbots is not without its challenges. One of the significant challenges is the lack of standardized evaluation metrics. Currently, there is no universally accepted set of metrics for evaluating chatbots, which makes it difficult to compare the performance of different chatbots. Liang and Li (2021) provided the solution through standard criteria and definitions for chatbot evaluation. Another challenge is the lack of diverse data sets for training and testing chatbots. Most existing data sets focus on specific domains, such as customer service or restaurant booking, which limits the generalizability of chatbots to other domains (Narducci et al., 2020).
In conclusion, evaluating chatbots is a complex and challenging task requiring a multifaceted approach. While several studies have demonstrated the effectiveness of chatbots in various domains, standardized evaluation metrics and more diverse data sets still need to be standardized. Computers that performed well on the measures used to evaluate the results of natural language processing AI do not match user expectations. This points to a gap in the comprehensiveness of the existing evaluation metrics. Nonetheless, few research studies have examined how various factors affect user experience on chatbot platforms. Table 1 presents a list of such studies. This research aims to quantify the factors associated with chatbot utility that are most important to the users.
Table 1. Existing Literature on Chatbot's Utility Criteria
Reference | Method of evaluation | Domain | Objective |
---|
Orden-Mejía & Huertas, 2022
| Exploratory factor analysis; Hierarchical regression
| Tourism
| Investigation of factors contributing to user satisfaction with chatbots.
|
Barletta et al., 2023
| Multicriteria decision-making (MCDM) (AHP)
| Healthcare
| Assess the quality of the medical chatbot and compare a chatbot's two different iterations.
|
Cai et al., 2022
| Task-oriented user studies
| Music
| Investigate the effectiveness of Dialogue-based conversational recommender systems.
|
Mokmin and Ibrahim, 2021
| Mixed method study
| Education
| Analysis of the efficacy, performance, and technological adoption of a chatbot created to educate users and provide health literacy.
|
Narducci et al., 2020
| Experimental evaluation
| Music
| Analysis of how various interaction styles affect recommendation precision and user input costs
|
Sugisaki and Bleiker, 2020
| Research-based method
| Language interaction
| Synthesis of the linguistic concepts necessary for a discussion in a natural language
|
Huang et al., 2018
| Survey
| General knowledge, reasoning, memory, and personality.
| Assess the capabilities of Tarie, a conversational AI.
|
AbuShawar and Atwell, 2016
| Quantitative and Qualitative evaluation
| Language interaction
| The paper discusses black box, glass box, comparative, quantitative, and qualitative natural language conversation system assessment methods.
|
Liang and Li, 2021 | Review | Criteria and definitions | Provide standard criteria and definitions for chatbot evaluation |
The assessment of platform utility in the context of chatbots has not yet been carried out, as shown in Table 1. Previous research has looked at how chatbots influence users but not at how various aspects of a platform's experience stack up against one another. Experimentation and empirical analysis using survey approaches dominate the bulk of the investigations. Figure 1 shows that there is still a need for research on how to evaluate platforms based on the utility of chatbots.
Multicriteria decision-making (MCDM) is a robust method for evaluating chatbots that fills this void by enabling decision-makers to consider multiple criteria simultaneously (Yalcin et al., 2022). This is especially crucial when judging chatbots, which are typically built to accommodate various users and accomplish numerous objectives.