Human Parsing with Joint Learning for Dynamic mmWave Radar Point Cloud

Shuai Wang, Dongjiang Cao, Ruofeng Liu, Wenchao Jiang, Tianshun Yao, Chris Xiaoxuan Lu

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

Human sensing and understanding is a key requirement for many intelligent systems, such as smart monitoring, human-computer interaction, and activity analysis, etc. In this paper, we present mmParse, the first human parsing design for dynamic point cloud from commercial millimeter-wave radar devices. mmParse proposes an end-to-end neural network design that addresses the inherent challenges in parsing mmWave point cloud (e.g., sparsity and specular reflection). First, we design a novel multi-task learning approach, in which an auxiliary task can guide the network to understand human structural features. Secondly, we introduce a multi-task feature fusion method that incorporates both intra-task and inter-task attention to aggregate spatio-temporal features of the subject from a global view. Through extensive experiments in both indoor and outdoor environments, we demonstrate that our proposed system is able to achieve ∼ 92% accuracy and ∼ 84% IoU accuracy. We also show that the predicted semantic labels can increase the performance of two downstream tasks (pose estimation and action recognition) by ∼ 18% and ∼ 6% respectively
Original languageEnglish
Article number34
Pages (from-to)1-22
Number of pages22
JournalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Issue number1
Publication statusPublished - 28 Mar 2023

Keywords / Materials (for Non-textual outputs)

  • Human Parsing
  • Joint Learning
  • Pose Estimation
  • Millimeter Wave Sensing


Dive into the research topics of 'Human Parsing with Joint Learning for Dynamic mmWave Radar Point Cloud'. Together they form a unique fingerprint.

Cite this