Open AccessResearch on the Theory and Application of Deep Interactive Learning
Author(s) -
Ziyuan Wang,
Fan Guo
Publication year - 2021
Publication title -
journal of physics. conference series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.21
H-Index - 85
eISSN - 1742-6596
pISSN - 1742-6588
DOI - 10.1088/1742-6596/1982/1/012085
Subject(s) - distillation , construct (python library) , computer science , artificial intelligence , deep learning , function (biology) , machine learning , chemistry , organic chemistry , evolutionary biology , biology , programming language
Knowledge distillation (KD), in which a small network (students) is trained to mimic a larger one(teachers), with high precision, has been widely used in various fields. However, the interaction between teachers and students is still weak. It is found in this study that most existing methods, such as Deep Mutual Learning (DML), mainly construct loss function through soft weight indexes. Few researchers pay attention to the sharing of hard and heavy ones. As an improvement of DML, a new online learning distillation method, namely, Deep Interactive Learning (hereinafter DIL), was proposed in this research, which has deeper interaction than DML. We not only output the features of layers, but also disclose the features of hidden layers. We transfer the features to other models to obtain the corresponding softer distribution or features for distillation. Extensive experiments on various data sets show that the accuracy of our method is improved by almost 3% in CIFAR and 2% in ImageNet, which proves the validity of our method.