English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

To apply the latest computer vision techniques that require a largecomputational cost in real industrial applications, knowledge distillationmethods (KDs) are essential. Existing logit-based KDs apply the constanttemperature scaling to all samples in dataset, limiting the utilization ofknowledge inherent in each sample individually. In our approach, we classifythe dataset into two categories (i.e., low energy and high energy samples)based on their energy score. Through experiments, we have confirmed that lowenergy samples exhibit high confidence scores, indicating certain predictions,while high energy samples yield low confidence scores, meaning uncertainpredictions. To distill optimal knowledge by adjusting non-target classpredictions, we apply a higher temperature to low energy samples to createsmoother distributions and a lower temperature to high energy samples toachieve sharper distributions. When compared to previous logit-based andfeature-based methods, our energy-based KD (Energy KD) achieves betterperformance on various datasets. Especially, Energy KD shows significantimprovements on CIFAR-100-LT and ImageNet datasets, which contain manychallenging samples. Furthermore, we propose high energy-based dataaugmentation (HE-DA) for further improving the performance. We demonstrate thatmeaningful performance improvement could be achieved by augmenting only 20-50%of dataset, suggesting that it can be employed on resource-limited devices. Tothe best of our knowledge, this paper represents the first attempt to make useof energy scores in KD and DA, and we believe it will greatly contribute tofuture research.

Maximizing Discrimination Capability of Knowledge Distillation with Energy-based Score