Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition

Shreyank Narayana Gowda, Marcus Rohrbach, Frank Keller, Laura Sevilla-Lara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

We address the problem of data augmentation for video action recognition. Standard augmentation strategies in video are handdesigned and sample the space of possible augmented data points either at random, without knowing which augmented points will be better, or through heuristics. We propose to learn what makes a “good” video for action recognition and select only high-quality samples for augmentation. In particular, we choose video compositing of a foreground and a background video as the data augmentation process, which results in diverse and realistic new samples. We learn which pairs of videos to augment without having to actually composite them. This reduces the space of possible augmentations, which has two advantages: it saves computational cost and increases the accuracy of the final trained classifier, as the augmented pairs are of higher quality than average. We present experimental results on the entire spectrum of training settings: few-shot, semisupervised and fully supervised. We observe consistent improvements across all of them over prior work and baselines on Kinetics, UCF101, HMDB51, and achieve a new state-of-the-art on settings with limited data. We see improvements of up to 8.6% in the semi-supervised setting.
Project Page: https://sites.google.com/view/learn2augment/home
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXI
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer, Cham
Number of pages23
ISBN (Electronic)978-3-031-19821-2
ISBN (Print)978-3-031-19820-5
Publication statusPublished - 23 Oct 2022
EventEuropean Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Cham
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceEuropean Conference on Computer Vision 2022
Abbreviated titleECCV 2022
CityTel Aviv
Internet address


Dive into the research topics of 'Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition'. Together they form a unique fingerprint.

Cite this