Sketch-a-Segmenter: Sketch-Based Photo Segmenter Generation

Conghui Hu, Da Li, Yongxin Yang, Timothy M. Hospedales, Yi-Zhe Song

Research output: Contribution to journalArticlepeer-review


Given pixel-level annotated data, traditional photo segmentation techniques have achieved promising results. However, these photo segmentation models can only identify objects in categories for which data annotation and training have been carried out. This limitation has inspired recent work on few-shot and zero-shot learning for image segmentation. In this article, we show the value of sketch for photo segmentation, in particular as a transferable representation to describe a concept to be segmented. We show, for the first time, that it is possible to generate a photo-segmentation model of a novel category using just a single sketch and furthermore exploit the unique fine-grained characteristics of sketch to produce more detailed segmentation. More specifically, we propose a sketch-based photo segmentation method that takes sketch as input and synthesizes the weights required for a neural network to segment the corresponding region of a given photo. Our framework can be applied at both the category-level and the instance-level, and fine-grained input sketches provide more accurate segmentation in the latter. This framework generalizes across categories via sketch and thus provides an alternative to zero-shot learning when segmenting a photo from a category without annotated training data. To investigate the instance-level relationship across sketch and photo, we create the SketchySeg dataset which contains segmentation annotations for photos corresponding to paired sketches in the Sketchy Dataset.
Original languageEnglish
Pages (from-to)9470-9481
Number of pages12
JournalIEEE Transactions on Image Processing
Publication statusPublished - 7 Oct 2020


  • Sketch-based photo segmentation
  • category-level segmentation
  • fine-grained segmentation
  • dataset

Fingerprint Dive into the research topics of 'Sketch-a-Segmenter: Sketch-Based Photo Segmenter Generation'. Together they form a unique fingerprint.

Cite this