English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Masked Autoencoders learn strong visual representations and achievestate-of-the-art results in several independent modalities, yet very few workshave addressed their capabilities in multi-modality settings. In this work, wefocus on point cloud and RGB image data, two modalities that are oftenpresented together in the real world, and explore their meaningfulinteractions. To improve upon the cross-modal synergy in existing works, wepropose PiMAE, a self-supervised pre-training framework that promotes 3D and 2Dinteraction through three aspects. Specifically, we first notice the importanceof masking strategies between the two sources and utilize a projection moduleto complementarily align the mask and visible tokens of the two modalities.Then, we utilize a well-crafted two-branch MAE pipeline with a novel shareddecoder to promote cross-modality interaction in the mask tokens. Finally, wedesign a unique cross-modal reconstruction module to enhance representationlearning for both modalities. Through extensive experiments performed onlarge-scale RGB-D scene understanding benchmarks (SUN RGB-D and ScannetV2), wediscover it is nontrivial to interactively learn point-image features, where wegreatly improve multiple 3D detectors, 2D detectors, and few-shot classifiersby 2.9%, 6.7%, and 2.4%, respectively. Code is available athttps://github.com/BLVLab/PiMAE.

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection