References to graphical objects in interactive multimodal queries

Daqing He, G. Ritchie, John Lee

Research output: Contribution to journalArticlepeer-review


Traditionally, interactive natural language systems assume a semantic model in which the entities referred to are in some abstract representation of a real or imagined world. In a system where graphical objects such as diagrams may be on the screen during the language interaction, there is possibility that the user may want to allude to these visual entities. Since graphical objects have their own properties (colour, shape, position on the screen, etc.) but may also represent items in a knowledge base which have other associated properties (price, geographical location, technical specifications, etc.), some systematic way is needed to enable such objects to be referred to in terms of either their screen properties or their associated attributes from the domain under discussion. In this paper, we present a formalisation for these arrangements, and show how our logical definitions can be used to generate constraints suitable for reference resolution within a natural language interpreter.
Original languageEnglish
Pages (from-to)617-628
Number of pages12
JournalKnowledge-Based Systems
Issue number7
Publication statusPublished - Oct 2008


  • Source ambiguities
  • Intelligence multimodal interfaces
  • Constraint satisfaction problems
  • Reference resolution


Dive into the research topics of 'References to graphical objects in interactive multimodal queries'. Together they form a unique fingerprint.

Cite this