English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The combination of transformers and masked image modeling (MIM) pre-trainingframework has shown great potential in various vision tasks. However, thepre-training computational budget is too heavy and withholds the MIM frombecoming a practical training paradigm. This paper presents FastMIM, a simpleand generic framework for expediting masked image modeling with the followingtwo steps: (i) pre-training vision backbones with low-resolution input images;and (ii) reconstructing Histograms of Oriented Gradients (HOG) feature insteadof original RGB values of the input images. In addition, we propose FastMIM-Pto progressively enlarge the input resolution during pre-training stage tofurther enhance the transfer results of models with high capacity. We point outthat: (i) a wide range of input resolutions in pre-training phase can lead tosimilar performances in fine-tuning phase and downstream tasks such asdetection and segmentation; (ii) the shallow layers of encoder are moreimportant during pre-training and discarding last several layers can speed upthe training stage with no harm to fine-tuning performance; (iii) the decodershould match the size of selected network; and (iv) HOG is more stable than RGBvalues when resolution transfers;. Equipped with FastMIM, all kinds of visionbackbones can be pre-trained in an efficient way. For example, we can achieve83.8%/84.1% top-1 accuracy on ImageNet-1K with ViT-B/Swin-B as backbones.Compared to previous relevant approaches, we can achieve comparable or bettertop-1 accuracy while accelerate the training procedure by $\sim$5$\times$. Codecan be found in https://github.com/ggjy/FastMIM.pytorch.

FastMIM: Expediting Masked Image Modeling Pre-training for Vision