Building Lexical Resources for Dialectical Arabic

Introduction

Arabic is the official language of 22 countries, one of the six official languages of the United Nations and has more than 420 million speakers (Kamusella, 2017). Arabic has three varieties: classical Arabic (CA) used in Quran and old literature, modern standard Arabic (MSA) used in formal communication nowadays and dialectal Arabic (DA) which is the spoken variety (Habash, 2010). The last decade has experienced a growing interest in the natural language processing of dialectical Arabic. This growth is attributed to the wide usage of this language variety in social media (Guellil et al., 2019) and to the role of these media in the Arabic revolutions (Aouragh, 2016).

The topics treated by computational linguists for Arabic dialects can be assigned to one of four main categories (Shoufan & Alameri 2015):

• 1.

  Basic language analysis such as morphological and syntactical analysis (Maamouri et al. 2006; Habash et al., 2012; Masmoudi et al., 2015; Khalifa, 2017; Zalmout, 2018).

• 2.

  Building language resources such as lexica, corpora, treebanks, and wordnets (Graff et al., 2006; Cotterell and Callison-Burch, 2014; El-Beltagy, 2016; Qwaider et al., 2018).

• 3.

  Semantic-level analysis and synthesis, such as machine translation and sentiment analysis (Sawaf 2010; Duwairi et al., 2014; Medhaffar et al., 2017).

• 4.

  Identifying Arabic dialects, e.g., using speech recognition techniques (Kirchhoff and Vergyri 2005; El Haj et al., 2018; Ali, 2018).

Besides well-known challenges in NLP for Modern Standard Arabic (MSA) such as the lack of short vowel letters and the non-existence of capitalization, (Farghaly & Shaalan, 2009) processing Arabic varieties is challenged by the lack of language resources such as lexicons and annotated corpora. The few proposed solutions for building lexicons in the literature show two major restrictions. First, the lexicons are specific to one dialect only. This restriction is especially problematic when social media data should be processed because social networks cross the geographic borderlines and allow addressing topics that are of interest to most Arabic people regardless of their regions or countries. Thus, a relevant part of social media data can be described as multi-dialectical. Processing this data using a single-dialect language resource is not expected to provide desired performance. Second, the proposed lexicons are restricted in terms of annotation width, and some of them are even application-specific, e.g., for PoS tagging or sentiment analysis. The authors believe that the development of general-purpose lexicons is essential to serve different NLP applications. This does not only facilitate interfacing the lexicon to different tools. It also allows the community to concentrate its efforts towards building a sophisticated and comprehensive lexicon that can be used as a reference in many applications.