Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting

Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M Hospedales, Tao Xiang, Yi-Zhe Song

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Self-supervised learning has gained prominence due to its efficacy at learning powerful representations from unlabelled data that achieve excellent performance on many challenging downstream tasks. However, supervision-free pre-text tasks are challenging to design and usually modality specific. Although there is a rich literature of self-supervised methods for either spatial (such as images) or temporal data (sound or text) modalities, a common pre-text task that benefits both modalities is largely missing. In this paper, we are interested in defining a self-supervised pre-text task for sketches and handwriting data. This data is uniquely characterised by its existence in dual modalities of rasterized images and vector coordinate sequences. We address and exploit this dual representation by proposing two novel cross-modal translation pre-text tasks for self-supervised feature learning: Vectorization and Rasterization. Vectorization learns to map image space to vector coordinates and rasterization maps vector coordinates to image space. We show that our learned encoder modules benefit both raster-based and vector-based downstream approaches to analysing hand-drawn data. Empirical evidence shows that our novel pre-text tasks surpass existing single and multi-modal self-supervision methods.
Original languageEnglish
Title of host publication2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)978-1-6654-4509-2
ISBN (Print)978-1-6654-4510-8
Publication statusPublished - 13 Nov 2021
EventIEEE Conference on Computer Vision and Pattern Recognition 2021 - Virtual
Duration: 19 Jun 202125 Jun 2021

Publication series

ISSN (Print)1063-6919
ISSN (Electronic)2575-7075


ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2021
Abbreviated titleCVPR 2021
Internet address


Dive into the research topics of 'Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting'. Together they form a unique fingerprint.

Cite this