English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Distillation is an effective knowledge-transfer technique that uses predicteddistributions of a powerful teacher model as soft targets to train aless-parameterized student model. A pre-trained high capacity teacher, however,is not always available. Recently proposed online variants use the aggregatedintermediate predictions of multiple student models as targets to train eachstudent model. Although group-derived targets give a good recipe forteacher-free distillation, group members are homogenized quickly with simpleaggregation functions, leading to early saturated solutions. In this work, wepropose Online Knowledge Distillation with Diverse peers (OKDDip), whichperforms two-level distillation during training with multiple auxiliary peersand one group leader. In the first-level distillation, each auxiliary peerholds an individual set of aggregation weights generated with anattention-based mechanism to derive its own targets from predictions of otherauxiliary peers. Learning from distinct target distributions helps to boostpeer diversity for effectiveness of group-based distillation. The second-leveldistillation is performed to transfer the knowledge in the ensemble ofauxiliary peers further to the group leader, i.e., the model used forinference. Experimental results show that the proposed framework consistentlygives better performance than state-of-the-art approaches without sacrificingtraining or inference complexity, demonstrating the effectiveness of theproposed two-level distillation framework.

Online Knowledge Distillation with Diverse Peers