Generalising Fine-Grained Sketch-Based Image Retrieval

Kaiyue Pang, Ke Li, Yongxin Yang, Hongang Zhan, Timothy Hospedales, Tao Xiang, Yi-Zhe Song

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Fine-grained sketch-based image retrieval (FG-SBIR) addresses matching specific photo instance using free-handsketch as a query modality. Existing models aim to learnan embedding space in which sketch and photo can be directly compared. While successful, they require instance-level pairing within each coarse-grained category as annotated training data. Since the learned embedding space is domain-specific, these models do not generalise well across categories. This limits the practical applicability of FGSBIR. In this paper, we identify cross-category generalisation for FG-SBIR as a domain generalisation problem, and propose the first solution. Our key contribution is a novel unsupervised learning approach to model a universal manifold of prototypical visual sketch traits. This manifold can then be used to paramaterise the learning of a sketch/photo representation. Model adaptation to novel categories then becomes automatic via embedding the novel sketch in the manifold and updating the representation and retrieval function accordingly. Experiments on the two largest FG-SBIR datasets, Sketchy and QMUL-Shoe-V2, demonstrate the efficacy of our approach in enabling crosscategory generalisation of FG-SBIR.
Original languageEnglish
Title of host publication2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)978-1-7281-3293-8
ISBN (Print)978-1-7281-3294-5
Publication statusPublished - 9 Jan 2020
Event2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition - Long Beach, United States
Duration: 16 Jun 201920 Jun 2019

Publication series

PublisherInstitute of Electrical and Electronics Engineers (IEEE)
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075


Conference2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Abbreviated titleCVPR 2019
CountryUnited States
CityLong Beach
Internet address


Dive into the research topics of 'Generalising Fine-Grained Sketch-Based Image Retrieval'. Together they form a unique fingerprint.

Cite this