English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Multimodal intent recognition aims to leverage diverse modalities such asexpressions, body movements and tone of speech to comprehend user's intent,constituting a critical task for understanding human language and behavior inreal-world multimodal scenarios. Nevertheless, the majority of existing methodsignore potential correlations among different modalities and own limitations ineffectively learning semantic features from nonverbal modalities. In thispaper, we introduce a token-level contrastive learning method withmodality-aware prompting (TCL-MAP) to address the above challenges. Toestablish an optimal multimodal semantic environment for text modality, wedevelop a modality-aware prompting module (MAP), which effectively aligns andfuses features from text, video and audio modalities with similarity-basedmodality alignment and cross-modality attention mechanism. Based on themodality-aware prompt and ground truth labels, the proposed token-levelcontrastive learning framework (TCL) constructs augmented samples and employsNT-Xent loss on the label token. Specifically, TCL capitalizes on the optimaltextual semantic insights derived from intent labels to guide the learningprocesses of other modalities in return. Extensive experiments show that ourmethod achieves remarkable improvements compared to state-of-the-art methods.Additionally, ablation analyses demonstrate the superiority of themodality-aware prompt over the handcrafted prompt, which holds substantialsignificance for multimodal prompt learning. The codes are released athttps://github.com/thuiar/TCL-MAP.

Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition