English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Multi-modal language models (LM) have recently shown promising performance inhigh-level reasoning tasks on videos. However, existing methods still fallshort in tasks like causal or compositional spatiotemporal reasoning overactions, in which model predictions need to be grounded in fine-grainedlow-level details, such as object motions and object interactions. In thiswork, we propose training an LM end-to-end on low-level surrogate tasks,including object detection, re-identification, and tracking, to endow the modelwith the required low-level visual capabilities. We show that a two-streamvideo encoder with spatiotemporal attention is effective at capturing therequired static and motion-based cues in the video. By leveraging the LM'sability to perform the low-level surrogate tasks, we can cast reasoning invideos as the three-step process of Look, Remember, Reason wherein visualinformation is extracted using low-level visual skills step-by-step and thenintegrated to arrive at a final answer. We demonstrate the effectiveness of ourframework on diverse visual reasoning tasks from the ACRE, CATER,Something-Else and STAR datasets. Our approach is trainable end-to-end andsurpasses state-of-the-art task-specific methods across these tasks by a largemargin.

Look, Remember and Reason: Grounded reasoning in videos with language models