Heightfields for Efficient Scene Reconstruction for AR

Jamie Watson, Sara Vicente, Oisin Mac Aodha, Clément Godard, Gabriel Brostow, Michael Firman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

3D scene reconstruction from a sequence of posed RGB images is a cornerstone task for computer vision and augmented reality (AR). While depth-based fusion is the foundation of most real-time approaches for 3D reconstruction, recent learning based methods that operate directly on RGB images can achieve higher quality reconstructions, but at the cost of increased runtime and memory requirements, making them unsuitable for AR applications. We propose an efficient learning-based method that refines the 3D reconstruction obtained by a traditional fusion approach. By leveraging a top-down heightfield representation, our method remains real-time while approaching the quality of other learning-based methods. Despite being a simplification, our heightfield is perfectly appropriate for robotic path planning or augmented reality character placement. We outline several innovations that push the performance beyond existing top-down prediction baselines, and we present an evaluation framework on the challenging ScanNetV2 dataset, targeting AR tasks. Ultimately, we show that our method improves over the baselines for AR applications. Full code and pretrained models will be released on acceptance.
Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Number of pages11
ISBN (Electronic)978-1-6654-9346-8
ISBN (Print)978-1-6654-9347-5
Publication statusPublished - 6 Feb 2023
EventIEEE/CVF Winter Conference on Applications of Computer Vision, 2023 - Waikoloa, United States
Duration: 3 Jan 20237 Jan 2023

Publication series

NameIEEE Workshop on Applications of Computer Vision (WACV)
ISSN (Print)2472-6737
ISSN (Electronic)2642-9381


ConferenceIEEE/CVF Winter Conference on Applications of Computer Vision, 2023
Abbreviated titleWACV 2023
Country/TerritoryUnited States
Internet address


Dive into the research topics of 'Heightfields for Efficient Scene Reconstruction for AR'. Together they form a unique fingerprint.

Cite this