Quality Diversity for Visual Pre-Training

Ruchika Chavhan, Henry Gouk, Da Li, Timothy Hospedales

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Models pre-trained on large datasets such as ImageNet provide the de-facto standard for transfer learning, with both supervised and self-supervised approaches proving effective. However, emerging evidence suggests that any single pre-trained feature will not perform well on diverse downstream tasks. Each pre-training strategy encodes a certain inductive bias, which may suit some downstream tasks but not others. Notably, the augmentations used in both supervised and self-supervised training lead to features with high invariance to spatial and appearance transformations. This renders them sub-optimal for tasks that demand sensitivity to these factors. In this paper we develop a feature that better supports diverse downstream tasks by providing a diverse set of sensitivities and invariances. In particular, we are inspired by Quality-Diversity in evolution, to define a pre-training objective that requires high quality yet diverse features -- where diversity is defined in terms of transformation (in)variances. Our framework plugs in to both supervised and self-supervised pre-training, and produces a small ensemble of features. We further show how downstream tasks can easily and efficiently select their preferred (in)variances. Both empirical and theoretical analysis show the efficacy of our representation and transfer learning approach for diverse downstream tasks.
Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Number of pages11
ISBN (Electronic)979-8-3503-0718-4
ISBN (Print)979-8-3503-0719-1
Publication statusAccepted/In press - 11 Aug 2023
EventInternational Conference on Computer Vision 2023 - Paris, France
Duration: 2 Oct 20236 Oct 2023

Publication series

NameInternational Conference on Computer Vision (ICCV)
ISSN (Print)1550-5499
ISSN (Electronic)2380-7504


ConferenceInternational Conference on Computer Vision 2023
Abbreviated titleICCV 2023
Internet address


Dive into the research topics of 'Quality Diversity for Visual Pre-Training'. Together they form a unique fingerprint.

Cite this