Decoupled Knowledge Distillation Based on Inter-class Ranking Correlation
Knowledge distillation has achieved great success since it was proposed,but many distillation strategies focus on the characteristics of the hidden layers and ignore the developability of logit distillation.The decoupled of knowledge distillation makes logit distillation return to public view.Both knowledge distillation and decoupled knowledge distillation use strong consistency constraints to make the distillation effect sub-optimal,especially when the teacher network and student network structure are different.To solve this problem,a method based on consistency of ranking relation between inter-classes is proposed.In this method,the relationship between teacher and student non-target class prediction is retained,and the correlation between class ranking is used as the relationship between agent loss and evaluation index in knowledge distillation model,so as to match the relationship between teacher network and student network.In this method,the relatively easy relation matching is extended to decoupled knowledge distillation and verified in CIFAR-100 and ImageNet-1K datasets.The experimental results show that the classification accuracy of the proposed method for dataset CIFAR-100 reaches77.38%,which is0.93%higher than that of the benchmark method.The effect of decoupling knowledge distillation image classification is improved,which verifies the effectiveness of the proposed method.At the same time,the results of comparative experiments show that it is more competitive.
国家科技期刊平台
NSTL
万方数据
维普
