English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Ensembles of machine learning models yield improved system performance aswell as robust and interpretable uncertainty estimates; however, theirinference costs may often be prohibitively high. \emph{Ensemble DistributionDistillation} is an approach that allows a single model to efficiently captureboth the predictive performance and uncertainty estimates of an ensemble. Forclassification, this is achieved by training a Dirichlet distribution over theensemble members' output distributions via the maximum likelihood criterion.Although theoretically principled, this criterion exhibits poor convergencewhen applied to large-scale tasks where the number of classes is very high. Inour work, we analyze this effect and show that the Dirichlet log-likelihoodcriterion classes with low probability induce larger gradients thanhigh-probability classes. This forces the model to focus on the distribution ofthe ensemble tail-class probabilities. We propose a new training objective thatminimizes the reverse KL-divergence to a \emph{Proxy-Dirichlet} target derivedfrom the ensemble. This loss resolves the gradient issues of EnsembleDistribution Distillation, as we demonstrate both theoretically and empiricallyon the ImageNet and WMT17 En-De datasets containing 1000 and 40,000 classes,respectively.

Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets