English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

When speakers describe an image, they tend to look at objects beforementioning them. In this paper, we investigate such sequential cross-modalalignment by modelling the image description generation processcomputationally. We take as our starting point a state-of-the-art imagecaptioning system and develop several model variants that exploit informationfrom human gaze patterns recorded during language production. In particular, wepropose the first approach to image description generation where visualprocessing is modelled $\textit{sequentially}$. Our experiments and analysesconfirm that better descriptions can be obtained by exploiting gaze-drivenattention and shed light on human cognitive processes by comparing differentways of aligning the gaze modality with language production. We find thatprocessing gaze data sequentially leads to descriptions that are better alignedto those produced by speakers, more diverse, and more natural${-}$particularlywhen gaze is encoded with a dedicated recurrent component.

Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze