VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations

Nikolaos Tsagkas; Oisin Mac Aodha; Chris Xiaoxuan Lu

VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations

Nikolaos Tsagkas, Oisin Mac Aodha, Chris Xiaoxuan Lu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present Visual-Language Fields (VL-Fields), a neural implicit spatial representation that enables open-vocabulary semantic queries. Our model encodes and fuses the geometry of a scene with vision-language trained latent features by distilling information from a language-driven segmentation model. VL-Fields is trained without requiring any prior knowledge of the scene object classes, which makes it a promising representation for the field of robotics. Our model outperformed the similar CLIP-Fields model in the task of semantic segmentation by almost 10%.

Original language	English
Title of host publication	Workshop on Representations, Abstractions, and Priors for Robot Learning Workshop at International Conference on Robotics and Automation 2023
Pages	1-6
Number of pages	6
Publication status	Published - 29 May 2023
Event	2023 IEEE International Conference on Robotics and Automation - London, United Kingdom Duration: 29 May 2023 → 2 Jun 2023 https://www.icra2023.org

Conference

Conference	2023 IEEE International Conference on Robotics and Automation
Abbreviated title	ICRA 2023
Country/Territory	United Kingdom
City	London
Period	29/05/23 → 2/06/23
Internet address	https://www.icra2023.org

Access to Document

VL-Fields_TSAGKAS_DOA05052023_AFV_CC BYAccepted author manuscript, 7.72 MBLicence: Creative Commons: Attribution (CC-BY)

https://sites.google.com/view/rap4robotsLicence: All Rights Reserved

Cite this

@inproceedings{ee403c3434b143f0bbcbd14570f2269b,

title = "VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations",

abstract = "We present Visual-Language Fields (VL-Fields), a neural implicit spatial representation that enables open-vocabulary semantic queries. Our model encodes and fuses the geometry of a scene with vision-language trained latent features by distilling information from a language-driven segmentation model. VL-Fields is trained without requiring any prior knowledge of the scene object classes, which makes it a promising representation for the field of robotics. Our model outperformed the similar CLIP-Fields model in the task of semantic segmentation by almost 10%.",

author = "Nikolaos Tsagkas and {Mac Aodha}, Oisin and Lu, {Chris Xiaoxuan}",

year = "2023",

month = may,

day = "29",

language = "English",

pages = "1--6",

booktitle = "Workshop on Representations, Abstractions, and Priors for Robot Learning Workshop at International Conference on Robotics and Automation 2023",

note = "2023 IEEE International Conference on Robotics and Automation, ICRA 2023 ; Conference date: 29-05-2023 Through 02-06-2023",

url = "https://www.icra2023.org",

}

Tsagkas, N, Mac Aodha, O & Lu, CX 2023, VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations. in Workshop on Representations, Abstractions, and Priors for Robot Learning Workshop at International Conference on Robotics and Automation 2023. pp. 1-6, 2023 IEEE International Conference on Robotics and Automation, London, United Kingdom, 29/05/23. <https://sites.google.com/view/rap4robots>

VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations. / Tsagkas, Nikolaos; Mac Aodha, Oisin; Lu, Chris Xiaoxuan.
Workshop on Representations, Abstractions, and Priors for Robot Learning Workshop at International Conference on Robotics and Automation 2023. 2023. p. 1-6.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations

AU - Tsagkas, Nikolaos

AU - Mac Aodha, Oisin

AU - Lu, Chris Xiaoxuan

PY - 2023/5/29

Y1 - 2023/5/29

N2 - We present Visual-Language Fields (VL-Fields), a neural implicit spatial representation that enables open-vocabulary semantic queries. Our model encodes and fuses the geometry of a scene with vision-language trained latent features by distilling information from a language-driven segmentation model. VL-Fields is trained without requiring any prior knowledge of the scene object classes, which makes it a promising representation for the field of robotics. Our model outperformed the similar CLIP-Fields model in the task of semantic segmentation by almost 10%.

AB - We present Visual-Language Fields (VL-Fields), a neural implicit spatial representation that enables open-vocabulary semantic queries. Our model encodes and fuses the geometry of a scene with vision-language trained latent features by distilling information from a language-driven segmentation model. VL-Fields is trained without requiring any prior knowledge of the scene object classes, which makes it a promising representation for the field of robotics. Our model outperformed the similar CLIP-Fields model in the task of semantic segmentation by almost 10%.

M3 - Conference contribution

SP - 1

EP - 6

BT - Workshop on Representations, Abstractions, and Priors for Robot Learning Workshop at International Conference on Robotics and Automation 2023

T2 - 2023 IEEE International Conference on Robotics and Automation

Y2 - 29 May 2023 through 2 June 2023

ER -

VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this