Interactive Symbol Grounding with Complex Referential Expressions

Rimvydas Rubavicius, Alex Lascarides

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. “all but one green square above both red circles.”) and its designation in the visual scene. Our distinctive approach combines: a) lazy fewshot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new training examples for grounding symbols that occur within a noun phrase but aren’t designated by that noun phase (e.g, red in the above example), thereby potentially gaining data efficiency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts.
Original languageEnglish
Title of host publicationProceedings of The 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
EditorsMarine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Place of PublicationStroudsburg, PA, USA
PublisherAssociation for Computational Linguistics (ACL)
Pages4863-4874
Number of pages12
ISBN (Electronic)978-1-955917-71-1
Publication statusPublished - 1 Jul 2022
Event2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics
- Seattle, United States
Duration: 10 Jul 202215 Jul 2022
https://2022.naacl.org/

Conference

Conference2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Abbreviated titleNAACL 2022
Country/TerritoryUnited States
CitySeattle
Period10/07/2215/07/22
Internet address

Fingerprint

Dive into the research topics of 'Interactive Symbol Grounding with Complex Referential Expressions'. Together they form a unique fingerprint.

Cite this