Premium
TSNN : Three‐Stream Combining 2D and 3D Convolutional Neural Network for Micro‐Expression Recognition
Author(s) -
Wu Chao,
Guo Fan
Publication year - 2021
Publication title -
ieej transactions on electrical and electronic engineering
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.254
H-Index - 30
eISSN - 1931-4981
pISSN - 1931-4973
DOI - 10.1002/tee.23272
Subject(s) - convolutional neural network , benchmark (surveying) , computer science , pattern recognition (psychology) , domain (mathematical analysis) , artificial intelligence , expression (computer science) , facial expression recognition , facial recognition system , mathematics , geography , mathematical analysis , geodesy , programming language
Facial micro‐expression recognition is a natural mechanism of facial behavior with subtle muscle movements and short duration, which is widely considered to be hard to recognize. In this paper, we propose the temporal sampling deformation (TSD) to normalize the temporal lengths and conserve time domain information for micro‐expression sequences. A three‐stream combining 2D and 3D convolutional neural network (TSNN) is also proposed to capture the features of micro‐expressions and classify the expressions as well. The proposed network has two variants TSNN‐IF and TSNN‐LF, which can automatically learn spatial and temporal features at the same time. Single domain experiments and cross‐domain experiments are also performed in the three benchmark datasets (chinese academy of sciences micro‐expression II (CASME II), spontaneous micro‐expression database (SMIC), and spontaneous micro‐facial movement dataset (SAMM)) to verify the effectiveness and validity of the proposed framework. Comprehensive results and ablation studies show that the proposed method can achieve comparable or even better results compared with other state‐of‐the‐art methods for micro‐expression recognition. © 2020 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.