English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Human state recognition is a critical topic with pervasive and importantapplications in human-machine systems. Multi-modal fusion, the combination ofmetrics from multiple data sources, has been shown as a sound method forimproving the recognition performance. However, while promising results havebeen reported by recent multi-modal-based models, they generally fail toleverage the sophisticated fusion strategies that would model sufficientcross-modal interactions when producing the fusion representation; instead,current methods rely on lengthy and inconsistent data preprocessing and featurecrafting. To address this limitation, we propose an end-to-end multi-modaltransformer framework for multi-modal human state recognition called Husformer.Specifically, we propose to use cross-modal transformers, which inspire onemodality to reinforce itself through directly attending to latent relevancerevealed in other modalities, to fuse different modalities while ensuringsufficient awareness of the cross-modal interactions introduced. Subsequently,we utilize a self-attention transformer to further prioritize contextualinformation in the fusion representation. Using two such attention mechanismsenables effective and adaptive adjustments to noise and interruptions inmulti-modal signals during the fusion process and in relation to high-levelfeatures. Extensive experiments on two human emotion corpora (DEAP and WESAD)and two cognitive workload datasets (MOCAS and CogLoad) demonstrate that in therecognition of human state, our Husformer outperforms both state-of-the-artmulti-modal baselines and the use of a single modality by a large margin,especially when dealing with raw multi-modal signals. We also conducted anablation study to show the benefits of each component in Husformer.

Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition