A Systematic Review of Research on ChatGPT: The User Perspective

Introduction

As a transformer-based neural network by OpenAI, ChatGPT represents state-of-the-art generative artificial intelligence (AI) that is capable of generating human-like text responses in natural language processing (NLP) tasks. The term generative AI refers to algorithms (such as ChatGPT) that can be used to generate new content such as audio, code, images, text, simulations, and videos (McKinsey & Company, 2023). It is estimated that ChatGPT reached 100 million monthly active users just months after its launch in January, making it the fastest-growing consumer application in history and setting a record for the fastest-growing user base (K. Hu, 2023).

ChatGPT has taken the world by surprise. Thousands of studies have been carried out on ChatGPT to assess its performance, capabilities, and limitations across its applications, such as content creation, virtual assistants, and conversational agents in a variety of settings. However, there has not been a systematic literature review to synthesize the key findings from the existing ChatGPT literature with a particular focus on user disposition and responses.

In this chapter, we investigate previous research themes and trending topics on ChatGPT through a comprehensive analysis of the literature. We use an automated technique (web-scraping) to retrieve and compile journal papers, conference proceedings, and book chapters from major publisher databases in the related fields, such as IEEE, Association for Computing Machinery (ACM), Springer, IGI, and Wiley. The search keywords used were "ChatGPT", "transformer-based language models", and "generative language models". Considering the rapidly evolving landscape, we have carefully curated papers from repositories of electronic preprints such as arXiv and SSRN. Industry/trade publications, policy briefs, and government white papers were excluded. The inclusion criteria were as follows: (1) the study should be related to ChatGPT, (2) the study should evaluate the performance of ChatGPT in NLP tasks from a user perspective and/or discuss the user responses to such tools, and (3) the study should be published in a peer-reviewed journal, book or conference or repository. We selected over 228 relevant studies that met our inclusion criteria, which were published between 2020 and 2022.

The abstracts of selected articles were quantitatively analysed using a probabilistic topic modeling procedure - the latent Dirichlet allocation (LDA) approach (Blei, 2012a; Blei et al., 2003a). This technique can reveal the hidden (latent) structure of the articles determining which articles address similar topics. LDA enables us to determine three components of the hidden structure: (1) a relatively small number of topics as research themes; (2) each article can be considered as a compilation of the topics discovered by the model, with the exact mix determined by how heavily each abstract is weighted toward each topic; (3) Specific words from each featured topic are assigned to the article by the model. This strategy is rooted in the notion that each article is made up of a variety of different topics, each with its own collection of words. Topic coherence (C_v), a summary measure that captures the tendency of a topic's high probability words to co-occur in the same document, or simply put, the degree of semantic similarity between top keywords in a topic (Mimno et al., 2011), is used to determine the optimal number of topics for topic extraction and conceptual evaluation.

Based on the topics identified by the LDA model utilizing their most representative terms - terms that have a substantially higher chance of occurring in articles concerning that topic than their average chance of appearing across the corpus, we derive five research themes and corresponding keywords that have emerged in the results. The overall findings indicate that research efforts in this field have primarily focused on performance, user disposition, application practices, and, ethical and privacy concerns.

One of the most important aspects of any AI language model is its performance. In the case of ChatGPT, various studies have shown its capability to perform high cognitive level tasks. In a study by (Gilson et al., 2023), ChatGPT was found to outperform other language models on a range of natural language processing tasks, including question-answering and summarization. These findings suggest that ChatGPT is an effective tool for a range of language-related tasks.