AcousticFusion: Fusing Sound Source Localization to Visual SLAM in Dynamic Environments

Tianwei Zhang, Huayan Zhang, Xiaofei Li, Junfeng Chen, Tin Lun Lam, Sethu Vijayakumar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Dynamic objects in the environment, such as people and other agents, lead to challenges for existing simultaneous localization and mapping (SLAM) approaches. To deal with dynamic environments, computer vision researchers usually apply some learning-based object detectors to remove these dynamic objects. However, these object detectors are computationally too expensive for mobile robot on-board processing. In practical applications, these objects output noisy sounds that can be effectively detected by on-board sound source localization. The directional information of the sound source object can be efficiently obtained by direction of sound arrival (DoA) estimation, but the depth estimation is difficult. Therefore, in this paper, we propose a novel audio-visual fusion approach that fuses sound source direction into the RGB-D image and thus removes the effect of dynamic obstacles on the multi-robot SLAM system. Experimental results of multi-robot SLAM in different dynamic environments show that the proposed method uses very small computational resources to obtain very stable self-localization results.
Original languageEnglish
Title of host publication2021 IEEE/RSJ International Confence on Intelligent Robots and Systems
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages8
ISBN (Electronic)978-1-6654-1714-3
ISBN (Print)978-1-6654-1715-0
Publication statusPublished - 16 Dec 2021
Event2021 IEEE/RSJ International Conference on Intelligent Robots and Systems - Online, Prague, Czech Republic
Duration: 27 Sept 20211 Oct 2021

Publication series

ISSN (Print)2153-0858
ISSN (Electronic)2153-0866


Conference2021 IEEE/RSJ International Conference on Intelligent Robots and Systems
Abbreviated titleIROS 2021
Country/TerritoryCzech Republic
Internet address


Dive into the research topics of 'AcousticFusion: Fusing Sound Source Localization to Visual SLAM in Dynamic Environments'. Together they form a unique fingerprint.

Cite this