Fine-grained sketch-based image retrieval: The role of part-aware attributes

Ki Li, Kaiyue Pang, Yi-Zhe Song, Timothy Hospedales, Honggang Zhang, Yichuan Hu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We study the problem of fine-grained sketch-based image retrieval. By performing instance-level (rather than category-level) retrieval, it embodies a timely and practical application, particularly with the ubiquitous availability of touchscreens. Three factors contribute to the challenging nature of the problem: (i) free-hand sketches are inherently abstract and iconic, making visual comparisons with photos more difficult, (ii) sketches and photos are in two different visual domains, i.e. black and white lines vs. color pixels, and (iii) fine-grained distinctions are especially challenging when executed across domain and abstraction-level. To address this, we propose to detect visual attributes at part-level, in order to build a new representation that not only captures fine-grained characteristics but also traverses across visual domains. More specifically, (i) we propose a dataset with 304 photos and 912 sketches, where each sketch and photo is annotated with its semantic parts and associated part-level attributes, and with the help of this dataset, we investigate (ii) how strongly-supervised deformable part-based models can be learned that subsequently enable automatic detection of part-level attributes, and (iii) a novel matching framework that synergistically integrates low-level features, mid-level geometric structure and high-level semantic attributes to boost retrieval performance. Extensive experiments conducted on our new dataset demonstrate value of the proposed method.
Original languageEnglish
Title of host publicationApplications of Computer Vision (WACV), 2016 IEEE Winter Conference on
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages9
DOIs
Publication statusPublished - 26 May 2016
EventIEEE Winter Conference on Application of Computer Vision 2016 - Lake Placid, New York City, United States
Duration: 7 Mar 20169 Mar 2016

Conference

ConferenceIEEE Winter Conference on Application of Computer Vision 2016
Abbreviated titleWACV 2016
CountryUnited States
CityNew York City
Period7/03/169/03/16

Fingerprint Dive into the research topics of 'Fine-grained sketch-based image retrieval: The role of part-aware attributes'. Together they form a unique fingerprint.

Cite this