Action Recognition with Dynamic Image Networks

Hakan Bilen, Basura Fernando, Efstratios Gravves, Andrea Vedaldi

Research output: Contribution to journalArticlepeer-review

Abstract

We introduce the concept of dynamic image, a novel compact representation of videos useful for video analysis, particularly in combination with convolutional neural networks (CNNs). A dynamic image encodes temporal data such as RGB or optical flow videos by using the concept of 'rank pooling'. The idea is to learn a ranking machine that captures the temporal evolution of the data and to use the parameters of the latter as a representation. We call the resulting representation dynamic image because it summarizes the video dynamics in addition to appearance. This powerful idea allows to convert any video to an image so that existing CNN models pre-trained with still images can be immediately extended to videos. We also present an efficient approximate rank pooling operator that runs two orders of magnitude faster than the standard ones with any loss in ranking performance and can be formulated as a CNN layer. To demonstrate the power of the representation, we introduce a novel four stream CNN architecture which can learn from RGB and optical flow frames as well as from their dynamic image representations. We show that the proposed network achieves state-of-the-art performance, 95.5% and 72.5% accuracy, in the UCF101 and HMDB51 respectively.
Original languageEnglish
Pages (from-to)2799 - 2813
Number of pages14
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume40
Issue number12
Early online date2 Nov 2017
DOIs
Publication statusPublished - 1 Dec 2018

Fingerprint

Dive into the research topics of 'Action Recognition with Dynamic Image Networks'. Together they form a unique fingerprint.

Cite this