Premium
Hybrid features for skeleton‐based action recognition based on network fusion
Author(s) -
Chen Zhangmeng,
Pan Junjun,
Yang Xiaosong,
Qin Hong
Publication year - 2020
Publication title -
computer animation and virtual worlds
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.225
H-Index - 49
eISSN - 1546-427X
pISSN - 1546-4261
DOI - 10.1002/cav.1952
Subject(s) - computer science , artificial intelligence , convolutional neural network , representation (politics) , animation , action recognition , action (physics) , convolution (computer science) , fusion mechanism , term (time) , skeleton (computer programming) , pattern recognition (psychology) , artificial neural network , machine learning , fusion , class (philosophy) , linguistics , philosophy , physics , programming language , quantum mechanics , computer graphics (images) , lipid bilayer fusion , politics , political science , law
In recent years, the topic of skeleton‐based human action recognition has attracted significant attention from researchers and practitioners in graphics, vision, animation, and virtual environments. The most fundamental issue is how to learn an effective and accurate representation from spatiotemporal action sequences towards improved performance, and this article aims to address the aforementioned challenge. In particular, we design a novel method of hybrid features' extraction based on the construction of multistream networks and their organic fusion. First, we train a convolution neural networks (CNN) model to learn CNN‐based features with the raw skeleton coordinates and their temporal differences serving as input signals. The attention mechanism is injected into the CNN model to weigh more effective and important information. Then, we employ long short‐term memory (LSTM) to obtain long‐term temporal features from action sequences. Finally, we generate the hybrid features by fusing the CNN and LSTM networks, and we classify action types with the hybrid features. The extensive experiments are performed on several large‐scale publically available databases, and promising results demonstrate the efficacy and effectiveness of our proposed framework.