Resolving References to Graphical Objects in Multimodal Queries by Constraint Satisfaction

Abstract

In natural language queries to an intelligent multimodal system, ambiguities related to referring expressions -- source ambiguities -- can occur between items in the visual display and objects in the domain being represented. A multimodal interface has to be able to resolve these ambiguities in order to provide satisfactory communication with a user. In this paper, we briefly introduce source ambiguities, and present the formalisation of a constraint satisfaction approach to interpreting singular referring expressions with source ambiguities. In our approach, source ambiguities are resolved simultaneously with other referent ambiguities, allowing flexible access to various sorts of knowledge.