English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The potential of multimodal generative artificial intelligence (mAI) toreplicate human grounded language understanding, including the pragmatic,context-rich aspects of communication, remains to be clarified. Humans areknown to use salient multimodal features, such as visual cues, to facilitatethe processing of upcoming words. Correspondingly, multimodal computationalmodels can integrate visual and linguistic data using a visual attentionmechanism to assign next-word probabilities. To test whether these processesalign, we tasked both human participants (N = 200) as well as severalstate-of-the-art computational models with evaluating the predictability offorthcoming words after viewing short audio-only or audio-visual clips withspeech. During the task, the model's attention weights were recorded and humanattention was indexed via eye tracking. Results show that predictabilityestimates from humans aligned more closely with scores generated frommultimodal models vs. their unimodal counterparts. Furthermore, including anattention mechanism doubled alignment with human judgments when visual andlinguistic context facilitated predictions. In these cases, the model'sattention patches and human eye tracking significantly overlapped. Our resultsindicate that improved modeling of naturalistic language processing in mAI doesnot merely depend on training diet but can be driven by multimodality incombination with attention-based architectures. Humans and computational modelsalike can leverage the predictive constraints of multimodal information byattending to relevant features in the input.

Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models