Open AccessMulti‐modality‐based Arabic sign language recognition
Author(s) -
Elpeltagy Marwa,
Abdelwahab Moataz,
Hussein Mohamed E.,
Shoukry Amin,
Shoala Asmaa,
Galal Moustafa
Publication year - 2018
Publication title -
iet computer vision
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.38
H-Index - 37
eISSN - 1751-9640
pISSN - 1751-9632
DOI - 10.1049/iet-cvi.2017.0598
Subject(s) - sign language , artificial intelligence , computer science , pattern recognition (psychology) , similarity (geometry) , segmentation , histogram , data set , set (abstract data type) , american sign language , face (sociological concept) , sequence (biology) , computer vision , image (mathematics) , social science , philosophy , linguistics , sociology , biology , genetics , programming language
With the increase in the number of deaf‐mute people in the Arab world and the lack of Arabic sign language (ArSL) recognition benchmark data sets, there is a pressing need for publishing a large‐volume and realistic ArSL data set. This study presents such a data set, which consists of 150 isolated ArSL signs. The data set is challenging due to the great similarity among hand shapes and motions in the collected signs. Along with the data set, a sign language recognition algorithm is presented. The authors’ proposed method consists of three major stages: hand segmentation, hand shape sequence and body motion description, and sign classification. The hand shape segmentation is based on the depth and position of the hand joints. Histograms of oriented gradients and principal component analysis are applied on the segmented hand shapes to obtain the hand shape sequence descriptor. The covariance of the three‐dimensional joints of the upper half of the skeleton in addition to the hand states and face properties are adopted for motion sequence description. The canonical correlation analysis and random forest classifiers are used for classification. The achieved accuracy is 55.57% over 150 ArSL signs, which is considered promising.