Quality Diversity for Visual Pre-Training

Ruchika Chavhan, Henry Gouk, Da Li, Timothy Hospedales

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Models pre-trained on large datasets such as ImageNet provide the de-facto standard for transfer learning, with both supervised and self-supervised approaches proving effective. However, emerging evidence suggests that any single pre-trained feature will not perform well on diverse downstream tasks. Each pre-training strategy encodes a certain inductive bias, which may suit some downstream tasks but not others. Notably, the augmentations used in both supervised and self-supervised training lead to features with high invariance to spatial and appearance transformations. This renders them sub-optimal for tasks that demand sensitivity to these factors. In this paper we develop a feature that better supports diverse downstream tasks by providing a diverse set of sensitivities and invariances. In particular, we are inspired by Quality-Diversity in evolution, to define a pre-training objective that requires high quality yet diverse features -- where diversity is defined in terms of transformation (in)variances. Our framework plugs in to both supervised and self-supervised pre-training, and produces a small ensemble of features. We further show how downstream tasks can easily and efficiently select their preferred (in)variances. Both empirical and theoretical analysis show the efficacy of our representation and transfer learning approach for diverse downstream tasks.
Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
PublisherIEEE
Pages5384-5394
Number of pages11
ISBN (Electronic)979-8-3503-0718-4
ISBN (Print)979-8-3503-0719-1
DOIs
Publication statusAccepted/In press - 11 Aug 2023
EventInternational Conference on Computer Vision 2023 - Paris, France
Duration: 2 Oct 20236 Oct 2023
https://iccv2023.thecvf.com/

Publication series

NameInternational Conference on Computer Vision (ICCV)
PublisherIEEE
ISSN (Print)1550-5499
ISSN (Electronic)2380-7504

Conference

ConferenceInternational Conference on Computer Vision 2023
Abbreviated titleICCV 2023
Country/TerritoryFrance
CityParis
Period2/10/236/10/23
Internet address

Fingerprint

Dive into the research topics of 'Quality Diversity for Visual Pre-Training'. Together they form a unique fingerprint.

Cite this