English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Most existing online knowledge distillation(OKD) techniques typically requiresophisticated modules to produce diverse knowledge for improving students'generalization ability. In this paper, we strive to fully utilize multi-modelsettings instead of well-designed modules to achieve a distillation effect withexcellent generalization performance. Generally, model generalization can bereflected in the flatness of the loss landscape. Since averaging parameters ofmultiple models can find flatter minima, we are inspired to extend the processto the sampled convex combinations of multi-student models in OKD.Specifically, by linearly weighting students' parameters in each trainingbatch, we construct a Hybrid-Weight Model(HWM) to represent the parameterssurrounding involved students. The supervision loss of HWM can estimate thelandscape's curvature of the whole region around students to measure thegeneralization explicitly. Hence we integrate HWM's loss into students'training and propose a novel OKD framework via parameter hybridization(OKDPH)to promote flatter minima and obtain robust solutions. Considering theredundancy of parameters could lead to the collapse of HWM, we furtherintroduce a fusion operation to keep the high similarity of students. Comparedto the state-of-the-art(SOTA) OKD methods and SOTA methods of seeking flatminima, our OKDPH achieves higher performance with fewer parameters, benefitingOKD with lightweight and robust characteristics. Our code is publicly availableat https://github.com/tianlizhang/OKDPH.

Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation