Deep Mutual Learning

Ying Zhang, Tao Xiang, Timothy Hospedales, Huchuan Lu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy. Different from the one-way transfer between a static pre-defined teacher and a student in model distillation, with DML, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on both category and instance recognition tasks. Surprisingly, it is revealed that no prior powerful teacher network is necessary – mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher
Original languageEnglish
Title of host publication2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages4320-4328
Number of pages9
ISBN (Electronic)978-1-5386-6420-9
DOIs
Publication statusPublished - 17 Dec 2018
EventComputer Vision and Pattern Recognition 2018 - Salt Lake City, United States
Duration: 18 Jun 201822 Jun 2018
http://cvpr2018.thecvf.com/
http://cvpr2018.thecvf.com/
http://cvpr2018.thecvf.com/

Publication series

Name
ISSN (Electronic)2575-7075

Conference

ConferenceComputer Vision and Pattern Recognition 2018
Abbreviated titleCVPR 2018
Country/TerritoryUnited States
CitySalt Lake City
Period18/06/1822/06/18
Internet address

Fingerprint

Dive into the research topics of 'Deep Mutual Learning'. Together they form a unique fingerprint.

Cite this