English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large-scale image-language pretrained models, e.g., CLIP, have demonstratedremarkable proficiency in acquiring general multi-modal knowledge throughweb-scale image-text data. Despite the impressive performance of image-languagemodels on various image tasks, how to effectively expand them on general videounderstanding remains an area of ongoing exploration. In this paper, weinvestigate the image-to-video transferring from the perspective of the modeland the data, unveiling two key obstacles impeding the adaptation ofimage-language models: non-generalizable temporal modeling and partiallymisaligned video-text data. To address these challenges, we proposeSpatial-Temporal Auxiliary Network with Mutual-guided alignment module(Mug-STAN), a simple yet effective framework extending image-text model todiverse video tasks and video-text data.Specifically, STAN adopts a branchstructure with decomposed spatial-temporal modules to enable generalizabletemporal modeling, while Mug suppresses misalignment by introducing token-wisefeature aggregation of either modality from the other. Extensive experimentalresults verify Mug-STAN significantly improves adaptation of language-imagepretrained models such as CLIP and CoCa at both video-text post-pretraining andfinetuning stages. With our solution, state-of-the-art zero-shot and finetuningresults on various downstream datasets, including MSR-VTT, DiDeMo, LSMDC,Kinetics-400, Something-Something-2, HMDB-51, UCF- 101, and AVA, are achieved.Moreover, by integrating pretrained Mug-STAN with the emerging multimodaldialogue model, we can realize zero-shot video chatting. Codes are available athttps://github.com/farewellthree/STAN

Mug-STAN: Adapting Image-Language Pretrained Models for General Video Understanding