Graph-Based Representation Knowledge Distillation for Image Classification
杨传广 1陈路明 2赵二虎 1安竹林 1徐勇军1
扫码查看
点击上方二维码区域，可以放大扫码查看
作者信息
1. 中国科学院计算技术研究所,北京 100190
2. 93114部队,北京 100080
折叠
摘要
知识蒸馏的核心思想是利用1个作为教师网络的大型模型来指导1个作为学生网络的小型模型,提升学生网络在图像分类任务上的性能.现有知识蒸馏方法通常从单一的输入样本中提取类别概率或特征信息作为知识,并没有对样本间关系进行建模,造成网络的表征学习能力下降.为解决此问题,本文引入图卷积神经网络,将输入样本集视为图结点构建关系图,图中的每个样本都可以聚合其他样本信息,提升样本的表征能力.本文从图结点和图关系2个角度构建图表征知识蒸馏误差,利用元学习引导学生网络自适应学习教师网络更佳的图表征,提升学生网络的图建模能力.相比于基线方法,本文提出的图表征知识蒸馏方法在加拿大高等研究院(Canadian Institute For Ad-vanced Research,CIFAR)发布的100种分类数据集上提升了3.70%的分类准确率,表明本文方法引导学生网络学习到了更具有判别性的特征空间,提升了图像分类能力.
Abstract
The core idea of knowledge distillation is to use a large model as the teacher network to guide a small mod-el as the student network,improving the performance of the student network in image classification tasks.Existing knowl-edge distillation methods often extract category probability or feature information as knowledge from a single input sample.They could not model the relationships between samples,decreasing the network's representation learning ability.To solve this problem,this paper introduces a graph convolutional neural network,which treats the input sample set as graph nodes to construct a relationship graph.Each sample in the graph could aggregate information from other samples,improving its own representation ability.This paper constructs the distillation loss of graph representation knowledge from the perspec-tives of graph nodes and relationships.It uses meta-learning to guide the student network to adaptively learn better graph representations from a teacher network,thereby improving the graph modeling ability of the student network.Compared to the baseline method,the graph-based representation knowledge distillation method improves the classification accuracy by 3.70%on the 100-classification dataset published by Canadian Institute For Advanced Research.The result indicates that the proposed method makes the student network learn a more discriminative feature space,thereby improving its image clas-sification ability.
维普
万方数据