English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study object interaction anticipation in egocentric videos. This taskrequires an understanding of the spatiotemporal context formed by past actionson objects, coined action context. We propose TransFusion, a multimodaltransformer-based architecture. It exploits the representational power oflanguage by summarising the action context. TransFusion leverages pre-trainedimage captioning and vision-language models to extract the action context frompast video frames. This action context together with the next video frame isprocessed by the multimodal fusion module to forecast the next objectinteraction. Our model enables more efficient end-to-end learning. The largepre-trained language models add common sense and a generalisation capability.Experiments on Ego4D and EPIC-KITCHENS-100 show the effectiveness of ourmultimodal fusion model. They also highlight the benefits of usinglanguage-based context summaries in a task where vision seems to suffice. Ourmethod outperforms state-of-the-art approaches by 40.4% in relative terms inoverall mAP on the Ego4D test set. We validate the effectiveness of TransFusionvia experiments on EPIC-KITCHENS-100. Video and code are available athttps://eth-ait.github.io/transfusion-proj/.

Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction