English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A great challenge in video-language (VidL) modeling lies in the disconnectionbetween fixed video representations extracted from image/video understandingmodels and downstream VidL data. Recent studies try to mitigate thisdisconnection via end-to-end training. To make it computationally feasible,prior works tend to "imagify" video inputs, i.e., a handful of sparsely sampledframes are fed into a 2D CNN, followed by a simple mean-pooling orconcatenation to obtain the overall video representations. Although achievingpromising results, such simple approaches may lose temporal information that isessential for performing downstream VidL tasks. In this work, we presentVIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a videotransformer to explicitly model the temporal dynamics of video inputs. Further,unlike previous studies that found pre-training tasks on video inputs (e.g.,masked frame modeling) not very effective, we design a new pre-training task,Masked Visual-token Modeling (MVM), for better video modeling. Specifically,the original video frame patches are "tokenized" into discrete visual tokens,and the goal is to recover the original visual tokens based on the maskedpatches. Comprehensive analysis demonstrates the effectiveness of both explicittemporal modeling via video transformer and MVM. As a result, VIOLET achievesnew state-of-the-art performance on 5 video question answering tasks and 4text-to-video retrieval tasks.

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling