Computer Science · Human-Computer Interaction
Gesture Recognition in Indonesian Sign Language Using Hybrid Deep Learning Models
Muhammad Yusuf Daffa Izzalhaqqi, Wahyono Wahyono
See on the mapAbstract
Sign language comprises unique hand gestures used to communicate between the hearing and hearing-impaired communities, with variations based on regions or countries. Due to the increasing global prevalence of hearing loss, there is a growing need for efficient sign language recognition systems. This paper presents the development of a hybrid CNN-LSTM model specifically designed to recognize static alphabetical gestures in Indonesian Sign Language (BISINDO). Primary data was collected to address the issue of publicly available dataset. These self-collected images underwent augmentation and preprocessing stages prior to model training. The architecture and hyperparameter configuration of the hybrid model were fine-tuned using the Randomized Search CV method. Performance analysis was conducted on both the hybrid model, its constituent models, and state-of-the-art model. The experimental results revealed remarkable performance by the hybrid model with excellent accuracies of 99.60%, 84.87%, and 98.00% on the training, validation, and testing sets respectively, along with a macro average score of 0.98, indicating high precision, recall, and F1-score across all the classes. However, it was observed that the VGG-16 and CNN exhibited slightly superior performance compared to the hybrid model, while the LSTM demonstrated the least favorable performance among the models.