English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper reveals a new appeal of the recently emerged large-kernelConvolutional Neural Networks (ConvNets): as the teacher in KnowledgeDistillation (KD) for small-kernel ConvNets. While Transformers have ledstate-of-the-art (SOTA) performance in various fields with ever-larger modelsand labeled data, small-kernel ConvNets are considered more suitable forresource-limited applications due to the efficient convolution operation andcompact weight sharing. KD is widely used to boost the performance ofsmall-kernel ConvNets. However, previous research shows that it is not quiteeffective to distill knowledge (e.g., global information) from Transformers tosmall-kernel ConvNets, presumably due to their disparate architectures. Wehereby carry out a first-of-its-kind study unveiling that modern large-kernelConvNets, a compelling competitor to Vision Transformers, are remarkably moreeffective teachers for small-kernel ConvNets, due to more similararchitectures. Our findings are backed up by extensive experiments on bothlogit-level and feature-level KD ``out of the box", with no dedicatedarchitectural nor training recipe modifications. Notably, we obtain the\textbf{best-ever pure ConvNet} under 30M parameters with \textbf{83.1\%} top-1accuracy on ImageNet, outperforming current SOTA methods including ConvNeXt V2and Swin V2. We also find that beneficial characteristics of large-kernelConvNets, e.g., larger effective receptive fields, can be seamlesslytransferred to students through this large-to-small kernel distillation. Codeis available at: \url{https://github.com/VITA-Group/SLaK}.

Are Large Kernels Better Teachers than Transformers for ConvNets?