Heightfields for Efficient Scene Reconstruction for AR

Jamie Watson, Sara Vicente, Oisin Mac Aodha, Clément Godard, Gabriel Brostow, Michael Firman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

3D scene reconstruction from a sequence of posed RGB images is a cornerstone task for computer vision and augmented reality (AR). While depth-based fusion is the foundation of most real-time approaches for 3D reconstruction, recent learning based methods that operate directly on RGB images can achieve higher quality reconstructions, but at the cost of increased runtime and memory requirements, making them unsuitable for AR applications. We propose an efficient learning-based method that refines the 3D reconstruction obtained by a traditional fusion approach. By leveraging a top-down heightfield representation, our method remains real-time while approaching the quality of other learning-based methods. Despite being a simplification, our heightfield is perfectly appropriate for robotic path planning or augmented reality character placement. We outline several innovations that push the performance beyond existing top-down prediction baselines, and we present an evaluation framework on the challenging ScanNetV2 dataset, targeting AR tasks. Ultimately, we show that our method improves over the baselines for AR applications. Full code and pretrained models will be released on acceptance.
Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
PublisherInstitute of Electrical and Electronics Engineers
Pages5850-5860
Number of pages11
ISBN (Electronic)978-1-6654-9346-8
ISBN (Print)978-1-6654-9347-5
DOIs
Publication statusPublished - 6 Feb 2023
EventIEEE/CVF Winter Conference on Applications of Computer Vision, 2023 - Waikoloa, United States
Duration: 3 Jan 20237 Jan 2023
https://wacv2023.thecvf.com/

Publication series

NameIEEE Workshop on Applications of Computer Vision (WACV)
PublisherIEEE
ISSN (Print)2472-6737
ISSN (Electronic)2642-9381

Conference

ConferenceIEEE/CVF Winter Conference on Applications of Computer Vision, 2023
Abbreviated titleWACV 2023
Country/TerritoryUnited States
CityWaikoloa
Period3/01/237/01/23
Internet address

Fingerprint

Dive into the research topics of 'Heightfields for Efficient Scene Reconstruction for AR'. Together they form a unique fingerprint.

Cite this