CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition

Shreyank Narayana Gowda, Laura Sevilla-Lara, Frank Keller, Marcus Rohrbach

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Zero-Shot action recognition is the task of recognizing action classes without visual examples. The problem can be seen as learning a representation on seen classes which generalizes well to instances of unseen classes, without losing discriminability between classes. Neural networks are able to model highly complex boundaries between visual classes, which explains their success as supervised models. However, in Zero-Shot learning, these highly specialized class boundaries may overfit to the seen classes and not transfer well from seen to unseen classes. We propose a novel cluster-based representation, which regularizes the learning process, yielding a representation that generalizes well to instances from unseen classes. We optimize the clustering using reinforcement learning, which we observe is critical. We call the proposed method CLASTER and observe that it consistently outperforms the state-of-the-art in all standard Zero-Shot video datasets, including UCF101, HMDB51 and Olympic Sports; both in the standard Zero-Shot evaluation and the generalized Zero-Shot learning. We see improvements of up to 11.9% over SOTA.
Project Page: https://sites.google.com/view/claster-zsl/home
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer
Pages187-203
Number of pages17
ISBN (Electronic)978-3-031-20044-1
ISBN (Print)978-3-031-20043-4
DOIs
Publication statusPublished - 20 Oct 2022
EventEuropean Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022
https://eccv2022.ecva.net/

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Cham
Volume13680
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceEuropean Conference on Computer Vision 2022
Abbreviated titleECCV 2022
Country/TerritoryIsrael
CityTel Aviv
Period23/10/2227/10/22
Internet address

Keywords / Materials (for Non-textual outputs)

  • Zero-Shot
  • Clustering
  • Action Recognition

Fingerprint

Dive into the research topics of 'CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition'. Together they form a unique fingerprint.

Cite this