Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton Based Action Recognition

Ruijie Hou, Yanran Li, Ningyu Zhang, Yulin Zhou, Xiaosong Yang, Zhao Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Skeleton-based human action recognition is a longstanding challenge due to its complex dynamics. Some fine-grain details of the dynamics play a vital role in classification. The existing work largely focuses on designing incremental neural networks with more complicated adjacent matrices to capture the details of joints relationships. However, they still have difficulties distinguishing actions that have broadly similar motion patterns but belong to different categories. Interestingly, we found that the subtle differences in motion patterns can be significantly amplified and become easy for audience to distinct through specified view directions, where this property haven't been fully explored before. Drastically different from previous work, we boost the performance by proposing a conceptually simple yet effective Multi-view strategy that recognizes actions from a collection of dynamic view features. Specifically, we design a novel Skeleton-Anchor Proposal (SAP) module which contains a Multi-head structure to learn a set of views. For feature learning of different views, we introduce a novel Angle Representation to transform the actions under different views and feed the transformations into the baseline model. Our module can work seamlessly with the existing action classification model. Incorporated with baseline models, our SAP module exhibits clear performance gains on many challenging benchmarks. Moreover, comprehensive experiments show that our model consistently beats down the state-of-the-art and remains effective and robust especially when dealing with corrupted data. Related code will be available on
Original languageEnglish
Title of host publicationProceedings of the 30th ACM International Conference on Multimedia
Place of PublicationNew York, NY, USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages9
ISBN (Print)9781450392037
Publication statusPublished - 10 Oct 2022
Event30th ACM International Conference on Multimedia, 2022 - Lisbon, Portugal
Duration: 10 Oct 202214 Oct 2022
Conference number: 30

Publication series

NameMM '22
PublisherAssociation for Computing Machinery


Conference30th ACM International Conference on Multimedia, 2022
Abbreviated titleACMMM 2022
Internet address


  • action recognition
  • graph neural networks
  • multi-view


Dive into the research topics of 'Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton Based Action Recognition'. Together they form a unique fingerprint.

Cite this