English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we study a novel problem in egocentric action recognition,which we term as "Multimodal Generalization" (MMG). MMG aims to study howsystems can generalize when data from certain modalities is limited or evencompletely missing. We thoroughly investigate MMG in the context of standardsupervised action recognition and the more challenging few-shot setting forlearning new action categories. MMG consists of two novel scenarios, designedto support security, and efficiency considerations in real-world applications:(1) missing modality generalization where some modalities that were presentduring the train time are missing during the inference time, and (2)cross-modal zero-shot generalization, where the modalities present during theinference time and the training time are disjoint. To enable thisinvestigation, we construct a new dataset MMG-Ego4D containing data points withvideo, audio, and inertial motion sensor (IMU) modalities. Our dataset isderived from Ego4D dataset, but processed and thoroughly re-annotated by humanexperts to facilitate research in the MMG problem. We evaluate a diverse arrayof models on MMG-Ego4D and propose new methods with improved generalizationability. In particular, we introduce a new fusion module with modality dropouttraining, contrastive-based alignment training, and a novel cross-modalprototypical loss for better few-shot performance. We hope this study willserve as a benchmark and guide future research in multimodal generalizationproblems. The benchmark and code will be available athttps://github.com/facebookresearch/MMG_Ego4D.

MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition