Open AccessConvolutional and recurrent neural network for human activity recognition: Application on American sign language
Author(s) -
Vincent Hernandez,
Tomoya Suzuki,
Gentiane Venture
Publication year - 2020
Publication title -
plos one
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.99
H-Index - 332
ISSN - 1932-6203
DOI - 10.1371/journal.pone.0228869
Subject(s) - convolutional neural network , computer science , deep learning , artificial intelligence , recurrent neural network , american sign language , sign language , pattern recognition (psychology) , perceptron , speech recognition , generalization , artificial neural network , dropout (neural networks) , machine learning , mathematical analysis , linguistics , philosophy , mathematics
Human activity recognition is an important and difficult topic to study because of the important variability between tasks repeated several times by a subject and between subjects. This work is motivated by providing time-series signal classification and a robust validation and test approaches. This study proposes to classify 60 signs from the American Sign Language based on data provided by the LeapMotion sensor by using different conventional machine learning and deep learning models including a model called DeepConvLSTM that integrates convolutional and recurrent layers with Long-Short Term Memory cells. A kinematic model of the right and left forearm/hand/fingers/thumb is proposed as well as the use of a simple data augmentation technique to improve the generalization of neural networks. DeepConvLSTM and convolutional neural network demonstrated the highest accuracy compared to other models with 91.1 (3.8) and 89.3 (4.0) % respectively compared to the recurrent neural network or multi-layer perceptron. Integrating convolutional layers in a deep learning model seems to be an appropriate solution for sign language recognition with depth sensors data.