English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation is an effective method for training lightweightmodels, but it introduces a significant amount of computational overhead to thetraining cost, as the method requires acquiring teacher supervisions ontraining samples. This additional cost -- called distillation cost -- is mostpronounced when we employ large-scale teacher models such as visiontransformers (ViTs). We present MaskedKD, a simple yet effective strategy thatcan significantly reduce the cost of distilling ViTs without sacrificing theprediction accuracy of the student model. Specifically, MaskedKD diminishes thecost of running teacher at inference by masking a fraction of image patchtokens fed to the teacher, and therefore skipping the computations required toprocess those patches. The mask locations are selected to prevent masking awaythe core features of an image that the student model uses for prediction. Thismask selection mechanism operates based on some attention score of the studentmodel, which is already computed during the student forward pass, and thusincurs almost no additional computation. Without sacrificing the final studentaccuracy, MaskedKD dramatically reduces the amount of computations required fordistilling ViTs. We demonstrate that MaskedKD can save up the distillation costby $50\%$ without any student performance drop, leading to approximately $28\%$drop in the overall training FLOPs.

MaskedKD: Efficient Distillation of Vision Transformers with Masked Images