Intra-category sketch-based image retrieval by matching deformable part models

An important characteristic of sketches, compared with text, rests with their ability of intrinsically capturing structure and appearance detail of objects. Nonetheless, akin to traditional text-based image retrieval, conventional sketch-based image retrieval (SBIR) principally focuses on retrieving photos of the same category, neglecting the fine-grained characteristics of sketches. In this paper, we further advocate the expressiveness of sketches and examine their efficacy under a novel intra-category SBIR framework. In particular, we study how sketches can be adopted to permit pose-specific retrieval within object categories. Key challenge to this problem is introducing a mid-level sketch representation that not only captures object pose, but also possess the ability to traverse sketch and photo domains. More specifically, we learn deformable part-based model (DPM) as a mid-level representation to discover and encode the various poses and parts in sketch and image domains independently, after which graph matching is utilized to establish component and part-level correspondences across the two domains. We further propose an SBIR dataset that covers the unique aspects of fine-grained SBIR. Through in-depth experiments, we demonstrate the superior performance of our proposed SBIR framework, and showcase its unique ability in pose-specific retrieval.