English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Multimodal emotion recognition techniques are increasingly essential forassessing mental states. Image-based methods, however, tend to focuspredominantly on overt visual cues and often overlook subtler mental statechanges. Psychophysiological research has demonstrated that HR and skintemperature are effective in detecting ANS activities, thereby revealing thesesubtle changes. However, traditional HR tools are generally more costly andless portable, while skin temperature analysis usually necessitates extensivemanual processing. Advances in remote-PPG and automatic thermal ROI detectionalgorithms have been developed to address these issues, yet their accuracy inpractical applications remains limited. This study aims to bridge this gap byintegrating r-PPG with thermal imaging to enhance prediction performance.Ninety participants completed a 20-minute questionnaire to induce cognitivestress, followed by watching a film aimed at eliciting moral elevation. Theresults demonstrate that the combination of r-PPG and thermal imagingeffectively detects emotional shifts. Using r-PPG alone, the predictionaccuracy was 77% for cognitive stress and 61% for moral elevation, asdetermined by SVM. Thermal imaging alone achieved 79% accuracy for cognitivestress and 78% for moral elevation, utilizing a RF algorithm. An early fusionstrategy of these modalities significantly improved accuracies, achieving 87%for cognitive stress and 83% for moral elevation using RF. Further analysis,which utilized statistical metrics and explainable machine learning methodsincluding SHAP, highlighted key features and clarified the relationship betweencardiac responses and facial temperature variations. Notably, it was observedthat cardiovascular features derived from r-PPG models had a more pronouncedinfluence in data fusion, despite thermal imaging's higher predictive accuracyin unimodal analysis.

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging