English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Autoregressive language modeling (ALM) have been successfully used inself-supervised pre-training in Natural language processing (NLP). However,this paradigm has not achieved comparable results with other self-supervisedapproach in computer vision (e.g., contrastive learning, mask image modeling).In this paper, we try to find the reason why autoregressive modeling does notwork well on vision tasks. To tackle this problem, we fully analyze thelimitation of visual autoregressive methods and proposed a novel stochasticautoregressive image modeling (named SAIM) by the two simple designs. First, weemploy stochastic permutation strategy to generate effective and robust imagecontext which is critical for vision tasks. Second, we create a parallelencoder-decoder training process in which the encoder serves a similar role tothe standard vision transformer focus on learning the whole contextualinformation, and meanwhile the decoder predicts the content of the currentposition, so that the encoder and decoder can reinforce each other. Byintroducing stochastic prediction and the parallel encoder-decoder, SAIMsignificantly improve the performance of autoregressive image modeling. Ourmethod achieves the best accuracy (83.9%) on the vanilla ViT-Base model amongmethods using only ImageNet-1K data. Transfer performance in downstream tasksalso show that our model achieves competitive performance.

Exploring Stochastic Autoregressive Image Modeling for Visual Representation