English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Self-supervised learning for computer vision has achieved tremendous progressand improved many downstream vision tasks such as image classification,semantic segmentation, and object detection. Among these, generativeself-supervised vision learning approaches such as MAE and BEiT show promisingperformance. However, their global masked reconstruction mechanism iscomputationally demanding. To address this issue, we propose local maskedreconstruction (LoMaR), a simple yet effective approach that performs maskedreconstruction within a small window of 7$\times$7 patches on a simpleTransformer encoder, improving the trade-off between efficiency and accuracycompared to global masked reconstruction over the entire image. Extensiveexperiments show that LoMaR reaches 84.1% top-1 accuracy on ImageNet-1Kclassification, outperforming MAE by 0.5%. After finetuning the pretrainedLoMaR on 384$\times$384 images, it can reach 85.4% top-1 accuracy, surpassingMAE by 0.6%. On MS COCO, LoMaR outperforms MAE by 0.5 $\text{AP}^\text{box}$ onobject detection and 0.5 $\text{AP}^\text{mask}$ on instance segmentation.LoMaR is especially more computation-efficient on pretraining high-resolutionimages, e.g., it is 3.1$\times$ faster than MAE with 0.2% higher classificationaccuracy on pretraining 448$\times$448 images. This local masked reconstructionlearning mechanism can be easily integrated into any other generativeself-supervised learning approach. Our code is publicly available inhttps://github.com/junchen14/LoMaR.

Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction