Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval

Ayan Kumar Bhunia, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Yi-Zhe Song

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user’s query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early-retrieval efficiency over state-of-the art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.
Original languageEnglish
Title of host publication2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Place of PublicationSeattle, WA, USA
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages9776-9785
Number of pages10
ISBN (Electronic)978-1-7281-7168-5
ISBN (Print)978-1-7281-7169-2
DOIs
Publication statusPublished - 5 Aug 2020
EventIEEE Conference on Computer Vision and Pattern Recognition 2020 - Seattle, United States
Duration: 16 Jun 202018 Jun 2020
http://cvpr2020.thecvf.com/

Publication series

Name
PublisherIEEE
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075

Conference

ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2020
Abbreviated titleCVPR 2020
Country/TerritoryUnited States
CitySeattle
Period16/06/2018/06/20
Internet address

Fingerprint

Dive into the research topics of 'Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval'. Together they form a unique fingerprint.

Cite this