Multi-Input CNN-LSTM for End-to-End Indian Sign Language Recognition: A Use Case With Wearable Sensors

Introduction

Sign language is primarily used by deaf and mute people for communication. A deaf person often finds it difficult to communicate with other people in form of written communication, since that would require knowledge of verbal languages such as English and Hindi, which are difficult to learn without the sense of hearing. On the other hand, sign language is a combination of hand gestures, facial expressions and body language in a well-defined semantics and lexicon that is easy to learn and utilize for a deaf person (Kudrinko, 2018; Gupta, & Kumar, 2021). However, sign language is not common among the hearing community making it difficult for a deaf person to communicate with them in signing. An electronic translator for converting sign language to spoken language could greatly enhance the communication between a sign language user and a non-signer. With this motivation, several researchers have reported about development of sign language recognition. Hundred and seventeen papers on recognition systems reported for 25 different sign languages over the past decade have been reviewed in (Wadhawan, 2021). Sign language recognition has been performed using non-wearable devices by capturing images, videos, depth or color related information, as well as wearable sensors such as flex sensors, motion sensors, electromyograms and even wi-fi signals. The reported research either explores the use of only the dominant hand during signing or both the hands. Moreover, the considered signs are either just static postures or information about the dynamic motion of hands is also captured. The review concludes that over 54% of work is dedicated towards reliably recognizing the posture of static hand(s) during an isolated sign (Wadhawan, 2021). Around 75% of the research was found to be focused on processing information from only the dominant hand. Use of vision-based approaches were found to be most popular, with 44% using camera and 23% using Kinect or leap motion sensor for data acquisition. The recognition may be carried out using machine learning and deep learning techniques.

A survey of literature on vision-based sign language recognition using deep learning techniques is presented in (Rastgoo, 2020). The authors report that isolated hand gestures without motion, such as those used for signing numerals and most alphabets may be classified using images. However, for identifying dynamic signs at word or sentence level, videos are required for data acquisition. Convolutional neural networks (CNN) and recurrent neural networks (RNN) have been found to be most prominently used for classification of hand and body posture and for hand tracking in dynamic signing. For instance, Wadhawan et al. evaluated 50 CNN models for determining hand posture from RGB images of 100 different signs from the Indian sign language (ISL) to achieve classification accuracy as high as 99.9% (Wadhawan, 2020). Hand pose has been determined using depth sensors for American sign language (ASL) in (Kolivand, 2021). First, the hand and forearm are segmented from the depth image of the hand sign. Thereafter, denoising and extraction of geometrical features is carried out to achieve classification accuracy of up to 96.78% with artificial neural network (ANN). Videos have also been used for recognition of dynamic signs. In (Masood, 2018), the authors used a combination of CNN and RNN to extract spatial and temporal features from video inputs to classify 46 gestures from Argentinean Sign Language with 95.2% accuracy. Despite their successful use for sign language recognition, some of the major challenges encountered with vision-based approaches are partial occlusion, poor illumination and background clutter (Rastgoo, 2020).