Multi-exit self-distillation with appropriate teachers
Multi-exit architecture allows early-stop inference to reduce computational cost,which can be used in resource-constrained circumstances.Recent works combine the multi-exit architecture with self-distillation to simultaneously achieve high efficiency and decent performance at different network depths.However,existing methods mainly transfer knowledge from deep exits or a single ensemble to guide all exits,without considering that inappropriate learning gaps between students and teachers may degrade the model performance,especially in shallow exits.To address this issue,we propose Multi-exit self-distillation with Appropriate TEachers(MATE)to provide diverse and appropriate teacher knowledge for each exit.In MATE,multiple ensemble teachers are obtained from all exits with different trainable weights.Each exit subsequently receives knowledge from all teachers,while focusing mainly on its primary teacher to keep an appropriate gap for efficient knowledge transfer.In this way,MATE achieves diversity in knowledge distillation while ensuring learning efficiency.Experimental results on CIFAR-100,TinyImageNet,and three fine-grained datasets demonstrate that MATE consistently outperforms state-of-the-art multi-exit self-distillation methods with various network architectures.
Multi-exit architectureKnowledge distillationLearning gap
孙武杰、陈德仿、王灿、叶德仕、冯雁、陈纯
展开 >
浙江大学计算机科学与技术学院,中国 杭州市,310000
多出口架构 知识蒸馏 学习差距
国家自然科学基金Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study,China
NETL
NSTL
万方数据
