CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition

Shreyank Narayana Gowda, Laura Sevilla-Lara, Frank Keller, Marcus Rohrbach

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Zero-Shot action recognition is the task of recognizing action classes without visual examples. The problem can be seen as learning a representation on seen classes which generalizes well to instances of unseen classes, without losing discriminability between classes. Neural networks are able to model highly complex boundaries between visual classes, which explains their success as supervised models. However, in Zero-Shot learning, these highly specialized class boundaries may overfit to the seen classes and not transfer well from seen to unseen classes. We propose a novel cluster-based representation, which regularizes the learning process, yielding a representation that generalizes well to instances from unseen classes. We optimize the clustering using reinforcement learning, which we observe is critical. We call the proposed method CLASTER and observe that it consistently outperforms the state-of-the-art in all standard Zero-Shot video datasets, including UCF101, HMDB51 and Olympic Sports; both in the standard Zero-Shot evaluation and the generalized Zero-Shot learning. We see improvements of up to 11.9% over SOTA.
Project Page:
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer, Cham
Number of pages17
ISBN (Electronic)978-3-031-20044-1
ISBN (Print)978-3-031-20043-4
Publication statusPublished - 20 Oct 2022
EventEuropean Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Cham
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceEuropean Conference on Computer Vision 2022
Abbreviated titleECCV 2022
CityTel Aviv
Internet address

Keywords / Materials (for Non-textual outputs)

  • Zero-Shot
  • Clustering
  • Action Recognition


Dive into the research topics of 'CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition'. Together they form a unique fingerprint.

Cite this