Article Preview
TopIntroduction
Conversational assistants, also called chatbots, offer a flexible medium of communication to access information using natural language. By providing an automated answer to common requests, they contribute to increased rapidity and availability of customer care services. They also guarantee a uniform and efficient treatment of simple requests. Therefore, the use of chatbot in industry has gained momentum over the last few years, for basic question answering, automation of customer requests or access to complex databases (Goasduff, 2019; Costello, 2019).
The growing importance of chatbots has been supported by a rapid development of Natural Language Processing algorithms over the last decade, enabling efficient classification of user requests (using Natural Language Classification) or retrieval of relevant information from sentences (using Named Entities Recognition). As the frameworks for classifying or extracting information grow stronger, the development of chatbots in industry has consensually focused on defining a dialog behaviour with the use of symbolic reasoning, in which user requests will be treated according to the definition of all request categories (coined intents) and named entities (e.g. mails, numbers, products, etc…) (Hoyt et al., 2016; Bocklisch et al., 2017; Alexa Internet, 2018). For instance, the request “Can you play some jazz?” can be modelled with one intent (play music) and one named entity (jazz). In practice, the set of intents is finite and relates to a specific business area (travel booking, banking assistance, etc.). This approach allows fast technical implementation and a good level of control over responses, which could explain its popularity.
Intent detection is generally implemented using supervised classification techniques, where classes represent intents (Adamopoulou & Moussiades, 2020). A dialog management system (e.g. a decision tree) then exploits the intents and named entities extracted from user requests to define the behaviour of the chatbot. Thus, at every step during design, a dataset labelled with intents and named entities is required to generate and maintain the classification model forming the core of the chatbot. Updating and annotating a dataset is usually considered an iterative process involving steps described by the acronym MATTER (Model, Annotate, Train, Test, Evaluate, and Revise) (see Stubbs (2013)). Following this approach, phases of dataset design and annotation are most often the result of manual work, and this methodology has several limitations:
- •
Prior to starting annotation, a model grouping data in relevant intents has to be defined: when performed manually, this task relies exclusively on the knowledge of the types of requests detained by one or several business experts;
- •
Once the modelling of intents is achieved, the task of labelling data requires thorough understanding of this model to avoid errors: when the knowledge of the intents model is not well integrated by one of the annotators, or uniformly shared between them, there is a risk of introducing intra-individual or inter-individual inconsistencies in the dataset;
- •
During initialization or maintenance of the model, the scope of the chatbot may change: in this case, any change to the intents model implies an additional cost of relabelling the dataset accordingly.
The approach hitherto described is therefore time consuming, expensive, dependent of human judgment and has low robustness to changes.
To provide assistance during this annotation task, one possible solution is to introduce computer initiatives. This can be implemented using unsupervised classification (clustering) with automatic partitioning of data based on their intrinsic similarities. However, the similarities exploited by unsupervised text classification algorithms are usually either lexical or syntactical, and do not guarantee that the data belong to a similar business domain. Consequently, the produced results are often qualified as irrelevant by business experts.