Open AccessMultimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network
Author(s) -
Jinxiang Liao,
Qinghua Zhong,
Yan Zhu,
Dongli Cai
Publication year - 2020
Publication title -
iop conference series. materials science and engineering
Language(s) - English
Resource type - Journals
eISSN - 1757-899X
pISSN - 1757-8981
DOI - 10.1088/1757-899x/782/3/032005
Subject(s) - convolutional neural network , computer science , emotion recognition , pattern recognition (psychology) , artificial intelligence , speech recognition , feature (linguistics) , emotion classification , feature extraction , signal (programming language) , arousal , modal , psychology , neuroscience , philosophy , linguistics , programming language , chemistry , polymer chemistry
In order to solve the problem that the emotion recognition rate of single-mode physiological signals is not high in the physiological signals based emotion recognition, in this paper, we propose a convolutional recurrent neural network based method for multi-modal physiological signal emotion recognition task. The method used convolutional neural network to learn the spatial representations of multi-channel EEG signals and the Long Short-term Memory network to learn the temporal representations of peripheral physiological signals (EOG, EMG, GSR, RSP, BVP, and TMP). The two representations are combined for emotion recognition and classification. In the two emotion dimensions of Arousal and Valence, our experiments conducted on the open source dataset DEAP shows that, this method achieve 89.68% and 89.19% average accuracy in the EEG emotion classification, 63.06% and 62.41% average accuracy in the peripheral physiological signal emotion classification, 93.06% and 91.95% average accuracy in the combined feature emotion classification. The experimental results show that the convolutional recurrent neural network based method that we proposed efficiently extract multi-modal physiological signal feature to improve the emotion recognition performance.