Edinburgh Research Explorer

BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Original languageEnglish
Title of host publicationProceedings to the International Conference on Learning Representations 2020
Number of pages15
Publication statusAccepted/In press - 1 Jan 2020
EventEighth International Conference on Learning Representations - Millennium Hall, Addis Ababa, Ethiopia
Duration: 26 Apr 202030 Apr 2020
https://iclr.cc/Conferences/2020

Conference

ConferenceEighth International Conference on Learning Representations
Abbreviated titleICLR 2020
CountryEthiopia
CityAddis Ababa
Period26/04/2030/04/20
Internet address

Abstract

The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks. However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive. In this work, we develop BlockSwap: a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential. These networks can then be used as students and distilled with the original large network as a teacher. We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classification, and COCO for detection, and provide a comprehensive ablation study of our approach. BlockSwap quickly explores possible block configurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. under 5 minutes on a single GPU for CIFAR-10). Code is available at https://github.com/BayesWatch/pytorch-blockswap.

Event

Eighth International Conference on Learning Representations

26/04/2030/04/20

Addis Ababa, Ethiopia

Event: Conference

ID: 132577772