Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Miltiadis Marios Katsakioris; Ioannis Konstas; Pierre Yves Mignotte; Helen Hastie

doi:10.18653/v1/2021.splurobonlp-1.2

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Miltiadis Marios Katsakioris, Ioannis Konstas, Pierre Yves Mignotte, Helen Hastie

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on OpenStreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for data-driven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

Original language	English
Title of host publication	Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics
Editors	Malihe Alikhani, Valts Blukis, Parisa Kordjamshidi, Aishwarya Padmakumar, Hao Tan
Place of Publication	Online
Publisher	Association for Computational Linguistics
Pages	11-21
Number of pages	11
DOIs	https://doi.org/10.18653/v1/2021.splurobonlp-1.2
Publication status	Published - 1 Aug 2021
Event	Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics - Online @ ACL-IJCNLP 2021 Duration: 5 Aug 2021 → 6 Aug 2021 https://splu-robonlp2021.github.io/

Workshop

Workshop	Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics
Abbreviated title	SpLU-RoboNLP 2021
Period	5/08/21 → 6/08/21
Internet address	https://splu-robonlp2021.github.io/

Access to Document

10.18653/v1/2021.splurobonlp-1.2Licence: Creative Commons: Attribution (CC-BY)

Learning to_KATSAKIORIS_DOA07062021_VOR_CC BYFinal published version, 926 KBLicence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/2021.splurobonlp-1.2Licence: Creative Commons: Attribution (CC-BY)

UK Robotics and Artificial Intelligence Hub for Offshore Energy Asset Integrity Management (ORCA)
Vijayakumar, S. (Principal Investigator), Mistry, M. (Co-investigator), Ramamoorthy, R. (Co-investigator) & Williams, C. (Co-investigator)
EPSRC
1/10/17 → 31/03/22
Project: Research

Cite this

Katsakioris, M. M., Konstas, I., Mignotte, P. Y., & Hastie, H. (2021). Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps. In M. Alikhani, V. Blukis, P. Kordjamshidi, A. Padmakumar, & H. Tan (Eds.), Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics (pp. 11-21). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.splurobonlp-1.2

Katsakioris, Miltiadis Marios ; Konstas, Ioannis ; Mignotte, Pierre Yves et al. / Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps. Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics. editor / Malihe Alikhani ; Valts Blukis ; Parisa Kordjamshidi ; Aishwarya Padmakumar ; Hao Tan. Online : Association for Computational Linguistics, 2021. pp. 11-21

@inproceedings{8d9b19aa09354894b26e738b234a23ba,

title = "Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps",

abstract = "Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on OpenStreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for data-driven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.",

author = "Katsakioris, {Miltiadis Marios} and Ioannis Konstas and Mignotte, {Pierre Yves} and Helen Hastie",

note = "Katsakioris{\textquoteright} PhD was funded by SeeByte, Datalab and MASTS-S. This work was also supported by the EPSRC funded ORCA Hub (EP/R026173/1, 2017-2021).; Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics, SpLU-RoboNLP 2021 ; Conference date: 05-08-2021 Through 06-08-2021",

year = "2021",

month = aug,

day = "1",

doi = "10.18653/v1/2021.splurobonlp-1.2",

language = "English",

pages = "11--21",

editor = "Malihe Alikhani and Valts Blukis and Parisa Kordjamshidi and Aishwarya Padmakumar and Hao Tan",

booktitle = "Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics",

publisher = "Association for Computational Linguistics",

url = "https://splu-robonlp2021.github.io/",

}

Katsakioris, MM, Konstas, I, Mignotte, PY & Hastie, H 2021, Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps. in M Alikhani, V Blukis, P Kordjamshidi, A Padmakumar & H Tan (eds), Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics. Association for Computational Linguistics, Online, pp. 11-21, Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics, 5/08/21. https://doi.org/10.18653/v1/2021.splurobonlp-1.2

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps. / Katsakioris, Miltiadis Marios; Konstas, Ioannis; Mignotte, Pierre Yves et al.
Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics. ed. / Malihe Alikhani; Valts Blukis; Parisa Kordjamshidi; Aishwarya Padmakumar; Hao Tan. Online: Association for Computational Linguistics, 2021. p. 11-21.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

AU - Katsakioris, Miltiadis Marios

AU - Konstas, Ioannis

AU - Mignotte, Pierre Yves

AU - Hastie, Helen

N1 - Katsakioris’ PhD was funded by SeeByte, Datalab and MASTS-S. This work was also supported by the EPSRC funded ORCA Hub (EP/R026173/1, 2017-2021).

PY - 2021/8/1

Y1 - 2021/8/1

N2 - Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on OpenStreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for data-driven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

AB - Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on OpenStreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for data-driven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

U2 - 10.18653/v1/2021.splurobonlp-1.2

DO - 10.18653/v1/2021.splurobonlp-1.2

M3 - Conference contribution

SP - 11

EP - 21

BT - Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

A2 - Alikhani, Malihe

A2 - Blukis, Valts

A2 - Kordjamshidi, Parisa

A2 - Padmakumar, Aishwarya

A2 - Tan, Hao

PB - Association for Computational Linguistics

CY - Online

T2 - Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

Y2 - 5 August 2021 through 6 August 2021

ER -

Katsakioris MM, Konstas I, Mignotte PY, Hastie H. Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps. In Alikhani M, Blukis V, Kordjamshidi P, Padmakumar A, Tan H, editors, Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics. Online: Association for Computational Linguistics. 2021. p. 11-21 doi: 10.18653/v1/2021.splurobonlp-1.2

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Abstract

Workshop

Access to Document

Fingerprint

Projects

UK Robotics and Artificial Intelligence Hub for Offshore Energy Asset Integrity Management (ORCA)

Cite this