English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We introduce a unified framework to jointly model images, text, and humanattention traces. Our work is built on top of the recent Localized Narrativesannotation framework [30], where each word of a given caption is paired with amouse trace segment. We propose two novel tasks: (1) predict a trace given animage and caption (i.e., visual grounding), and (2) predict a caption and atrace given only an image. Learning the grounding of each word is challenging,due to noise in the human-provided traces and the presence of words that cannotbe meaningfully visually grounded. We present a novel model architecture thatis jointly trained on dual tasks (controlled trace generation and controlledcaption generation). To evaluate the quality of the generated traces, wepropose a local bipartite matching (LBM) distance metric which allows thecomparison of two traces of different lengths. Extensive experiments show ourmodel is robust to the imperfect training data and outperforms the baselines bya clear margin. Moreover, we demonstrate that our model pre-trained on theproposed tasks can be also beneficial to the downstream task of COCO's guidedimage captioning. Our code and project page are publicly available.

Connecting What to Say With Where to Look by Modeling Human Attention Traces