English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

There are two de facto standard architectures in recent computer vision:Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Stronginductive biases of convolutions help the model learn sample effectively, butsuch strong biases also limit the upper bound of CNNs when sufficient data areavailable. On the contrary, ViT is inferior to CNNs for small data but superiorfor sufficient data. Recent approaches attempt to combine the strengths ofthese two architectures. However, we show these approaches overlook that theoptimal inductive bias also changes according to the target data scale changesby comparing various models' accuracy on subsets of sampled ImageNet atdifferent ratios. In addition, through Fourier analysis of feature maps, themodel's response patterns according to signal frequency changes, we observewhich inductive bias is advantageous for each data scale. The moreconvolution-like inductive bias is included in the model, the smaller the datascale is required where the ViT-like model outperforms the ResNet performance.To obtain a model with flexible inductive bias on the data scale, we showreparameterization can interpolate inductive bias between convolution andself-attention. By adjusting the number of epochs the model stays in theconvolution, we show that reparameterization from convolution to self-attentioninterpolates the Fourier analysis pattern between CNNs and ViTs. Adapting thesefindings, we propose Progressive Reparameterization Scheduling (PRS), in whichreparameterization adjusts the required amount of convolution-like orself-attention-like inductive bias per layer. For small-scale datasets, our PRSperforms reparameterization from convolution to self-attention linearly fasterat the late stage layer. PRS outperformed previous studies on the small-scaledataset, e.g., CIFAR-100.

Towards Flexible Inductive Bias via Progressive Reparameterization Scheduling