English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The holy grail in deep neural network research is porting the memory- andcomputation-intensive network models on embedded platforms with a minimalcompromise in model accuracy. To this end, we propose a novel approach, termedas Variational Student, where we reap the benefits of compressibility of theknowledge distillation (KD) framework, and sparsity inducing abilities ofvariational inference (VI) techniques. Essentially, we build a sparse studentnetwork, whose sparsity is induced by the variational parameters found viaoptimizing a loss function based on VI, leveraging the knowledge learnt by anaccurate but complex pre-trained teacher network. Further, for sparsityenhancement, we also employ a Block Sparse Regularizer on a concatenated tensorof teacher and student network weights. We demonstrate that the marriage of KDand the VI techniques inherits compression properties from the KD framework,and enhances levels of sparsity from the VI approach, with minimal compromisein the model accuracy. We benchmark our results on LeNet MLP and VGGNet (CNN)and illustrate a memory footprint reduction of 64x and 213x on these MLP andCNN variants, respectively, without a need to retrain the teacher network.Furthermore, in the low data regime, we observed that our method outperformsstate-of-the-art Bayesian techniques in terms of accuracy.

Variational Student: Learning Compact and Sparser Networks in Knowledge Distillation Framework