Article Preview
TopIntroduction
There has been research interest towards the improvements in textual information processing in the past decade (Ali et al., 2017). Accompanying the evolution of computer technology, the volume of text data available online has had strong growth in recent years. As an important branch in the field of natural language processing (NLP), document classification aims to determine the sentiment polarity of a document, which plays a pivot role in a variety of tasks, including spam email filtering (Liu & Wang, 2010), topic extraction (Sarioglu, 2014), sentiment analysis (Hu et al., 2015), social public opinion mining (Guan et al., 2009) and more. In most cases, the discussed documents are ranked with different scores or stars representing the corresponding sentiment, while a higher score generally indicates more positive sentiment. With an accurate comprehension of the sentiment results and a deep understanding of the given document, the performance of document classification can be improved accordingly.
Advances in deep learning algorithms give rise to new opportunities to promote the efficacy of NLP tasks significantly. State-of-the-art document classification approaches are typically dominated by two distinguishing neural networks: the convolutional neural network (CNN), and the recurrent neural network (RNN). Recent publications report the superiority of the RNN in dealing with sequential inputs of various lengths. That is, the RNN models are capable of not only modeling the long-term dependencies (Habimana et al., 2020), but can also capture the semantics within contextual information (Du et al., 2019). More specifically, the two most well-known RNNs, namely long short-term memory (LSTM) and gated recurrent unit (GRU), are employed as a key module for tackling such issues in miscellaneous NLP methods (Li et al., 2019). On the other hand, the CNN is more effective in extracting the sentiment-related features from word sequences in comparison with the RNN (Du et al., 2019). The main reason is that the CNN can make full use of the textual data to collect the feature vectors with minimum parameters. In such a manner, the local importance from salient parts is thus captured (Zhao et al., 2021).
In order to improve document classification accuracy, the absence of sentiment information within long-distance texts needs to be considered comprehensively. To address this issue, the attention mechanism is employed, which supplements and enhances the long-distance sentiment information delivering in either the CNN or the RNN models. To be more specific, the attention mechanisms identify the significance in exploiting the hidden states and computing the class distributions, based on how to determine the attentive weights of different words (Liang et al., 2017). In this way, the models that integrate the attention mechanism into the RNN/CNN are able to model the textual data regardless of the distances. While restricted to the architecture of deep-learning algorithms, the attention mechanism still shows its distinctiveness by precisely capturing the sentiment of a specific part from the document. In fact, an attention model proposed by Google, namely Transformer, is implemented based solely on the attention mechanisms, without using the neural network structure (Vaswani et al., 2017). This work allows for multi-attention layers running in parallel, and outperforms all previously reported ensembles, which sets the foundation of the multi-head attention network.