Self-Supervised Video Representation Learning with Odd-One-Out Networks

Basura Fernando, Hakan Bilen, Efstratios Gavves, Stephen Gould

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We propose a new self-supervised CNN pre-training technique based on a novel auxiliary task called odd-one-out learning. In this task, the machine is asked to identify the unrelated or odd element from a set of otherwise related elements. We apply this technique to self-supervised video representation learning where we sample subsequences from videos and ask the network to learn to predict the odd video subsequence. The odd video subsequence is sampled such that it has wrong temporal order of frames while the even ones have the correct temporal order. Therefore, to generate a odd-one-out question no manual annotation is required. Our learning machine is implemented as multi-stream convolutional neural network, which is learned end-to-end. Using odd-one-out networks, we learn temporal representations for videos that generalizes to other related tasks such as action recognition. On action classification, our method obtains 60.3% on the UCF101 dataset using only UCF101 data for training which is approximately 10% better than current state-of-the-art self-supervised learning methods. Similarly, on HMDB51 dataset we outperform self-supervised state-of-the art methods by 12.7% on action classification task.
Original languageEnglish
Title of host publication2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Place of PublicationHonolulu, HI, USA
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)978-1-5386-0457-1
ISBN (Print)978-1-5386-0458-8
Publication statusPublished - 9 Nov 2017
Event2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops - Honolulu, United States
Duration: 21 Jul 201726 Jul 2017

Publication series

ISSN (Print)1063-6919


Conference2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops
Abbreviated titleCVPR 2017
Country/TerritoryUnited States
Internet address


Dive into the research topics of 'Self-Supervised Video Representation Learning with Odd-One-Out Networks'. Together they form a unique fingerprint.

Cite this