English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Small CNN-based models usually require transferring knowledge from a largemodel before they are deployed in computationally resource-limited edgedevices. Masked image modeling (MIM) methods achieve great success in variousvisual tasks but remain largely unexplored in knowledge distillation forheterogeneous deep models. The reason is mainly due to the significantdiscrepancy between the Transformer-based large model and the CNN-based smallnetwork. In this paper, we develop the first Heterogeneous Generative KnowledgeDistillation (H-GKD) based on MIM, which can efficiently transfer knowledgefrom large Transformer models to small CNN-based models in a generativeself-supervised fashion. Our method builds a bridge between Transformer-basedmodels and CNNs by training a UNet-style student with sparse convolution, whichcan effectively mimic the visual representation inferred by a teacher overmasked modeling. Our method is a simple yet effective learning paradigm tolearn the visual representation and distribution of data from heterogeneousteacher models, which can be pre-trained using advanced generative methods.Extensive experiments show that it adapts well to various models and sizes,consistently achieving state-of-the-art performance in image classification,object detection, and semantic segmentation tasks. For example, in the Imagenet1K dataset, H-GKD improves the accuracy of Resnet50 (sparse) from 76.98% to80.01%.

Heterogeneous Generative Knowledge Distillation with Masked Image Modeling