Efficient parametrization of multi-domain deep neural networks

Sylvestre-Alvise Rebuffi, Hakan Bilen, Andrea Vedaldi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A practical limitation of deep neural networks is their high degree of specialization to a single task and visual domain. Recently, inspired by the successes of transfer learning, several authors have proposed to learn instead universal feature extractors that, used as the first stage of any deep network, work well for several tasks and domains simultaneously. Nevertheless, such universal features are still somewhat inferior to specialized networks. To overcome this limitation, in this paper we propose to consider instead universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters. We study different designs for such parametrizations, including series and parallel residual adapters, joint adapter compression, and parameter allocations, and empirically identify the ones that yield the highest compression. We show that, in order to maximize performance, it is necessary to adapt both shallow and deep layers of a deep network, but the required changes are very small. We also show that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques.
Original languageEnglish
Title of host publicationIEEE Conference on Computer Vision and Pattern Recognition
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages8119-8127
Number of pages9
ISBN (Electronic)978-1-5386-6420-9
DOIs
Publication statusPublished - 17 Dec 2018
EventComputer Vision and Pattern Recognition 2018 - Salt Lake City, United States
Duration: 18 Jun 201822 Jun 2018
http://cvpr2018.thecvf.com/
http://cvpr2018.thecvf.com/
http://cvpr2018.thecvf.com/

Publication series

Name
ISSN (Electronic)2575-7075

Conference

ConferenceComputer Vision and Pattern Recognition 2018
Abbreviated titleCVPR 2018
CountryUnited States
CitySalt Lake City
Period18/06/1822/06/18
Internet address

Fingerprint Dive into the research topics of 'Efficient parametrization of multi-domain deep neural networks'. Together they form a unique fingerprint.

Cite this