English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation (KD) is a popular method for reducing thecomputational overhead of deep network inference, in which the output of ateacher model is used to train a smaller, faster student model. Hint training(i.e., FitNets) extends KD by regressing a student model's intermediaterepresentation to a teacher model's intermediate representation. In this work,we introduce bLock-wise Intermediate representation Training (LIT), a novelmodel compression technique that extends the use of intermediaterepresentations in deep network compression, outperforming KD and hinttraining. LIT has two key ideas: 1) LIT trains a student of the same width (butshallower depth) as the teacher by directly comparing the intermediaterepresentations, and 2) LIT uses the intermediate representation from theprevious block in the teacher model as an input to the current student blockduring training, avoiding unstable intermediate representations in the studentnetwork. We show that LIT provides substantial reductions in network depthwithout loss in accuracy -- for example, LIT can compress a ResNeXt-110 to aResNeXt-20 (5.5x) on CIFAR10 and a VDCNN-29 to a VDCNN-9 (3.2x) on AmazonReviews without loss in accuracy, outperforming KD and hint training in networksize for a given accuracy. We also show that applying LIT to identicalstudent/teacher architectures increases the accuracy of the student model abovethe teacher model, outperforming the recently-proposed Born Again Networksprocedure on ResNet, ResNeXt, and VDCNN. Finally, we show that LIT caneffectively compress GAN generators, which are not supported in the KDframework because GANs output pixels as opposed to probabilities.

LIT: Block-wise Intermediate Representation Training for Model Compression