Towards Unsupervised Sketch-based Image Retrieval

Conghui Hu, Yongxin Yang, Yunpeng Li, Timothy Hospedales, Yi-Zhe Song

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

The practical value of existing supervised sketch-based image retrieval (SBIR) algorithms is largely limited by the requirement for intensive data collection and labeling. In this paper, we present the first attempt at unsupervised SBIR to remove the labeling cost (both category annotations and sketch-photo pairings) that is conventionally needed for training. Existing single-domain unsupervised representation learning methods perform poorly in this application, due to the unique cross-domain (sketch and photo) nature of the problem. We therefore introduce a novel framework that simultaneously performs sketch-photo domain alignment and semantic-aware representation learning. Technically this is underpinned by introducing joint distribution optimal transport (JDOT) to align data from different domains, which we extend with trainable cluster prototypes and feature memory banks to further improve scalability and efficacy. Extensive experiments show that our framework achieves excellent performance in the new unsupervised setting, and performs comparably to existing zero-shot SBIR methods.
Original languageEnglish
Title of host publicationProceedings of the 33rd British Machine Vision Conference 2022, (BMVC 2022)
PublisherBMVA Press
Number of pages14
Publication statusPublished - 25 Nov 2022
EventThe 33rd British Machine Vision Conference, 2022 - London, United Kingdom
Duration: 21 Nov 202224 Nov 2022
Conference number: 33


ConferenceThe 33rd British Machine Vision Conference, 2022
Abbreviated titleBMVC 2022
Country/TerritoryUnited Kingdom
Internet address


Dive into the research topics of 'Towards Unsupervised Sketch-based Image Retrieval'. Together they form a unique fingerprint.

Cite this