Affiliations: [a] Science and Parallel and Distributed Processing Laboratory, National University of Defense Technology, Changsha, China
| [b] Laboratory of Software Engineering for Complex Systems, National University of Defense Technology, Changsha, China
Correspondence:
[*]
Corresponding author. Jie Liu, National University of Defense Technology
China. E-mail: [email protected].
Abstract: The prime focus of knowledge distillation (KD) seeks a light proxy termed student to mimic the outputs of its heavy neural networks termed teacher, and makes the student run real-time on the resource-limited devices. This paradigm requires aligning the soft logits of both teacher and student. However, few doubts whether the process of softening the logits truly give full play to the teacher-student paradigm. In this paper, we launch several analyses to delve into this issue from scratch. Subsequently, several simple yet effective functions are devised to replace the vanilla KD. The ultimate function can be an effective alternative to its original counterparts and work well with other skills like FitNets. To claim this point, we conduct several visual tasks on individual benchmarks, and experimental results verify the potential of our proposed function in terms of performance gains. For example, when the teacher and student networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, our proposed method achieves 40.88%top-1 error rate on Tiny ImageNet.
Keywords: Neural network compression, knowledge distillation, knowledge transfer
