What Would You Like to Drink? Recognising and Planning with Social States in a Robot Bartender Domain

A robot coexisting with humans must not only be able to successfully perform physical tasks, but must also be able to interact with humans in a socially appropriate manner. In many social settings, this involves the use of social signals like gaze, facial expression, and language. In this paper we discuss preliminary work focusing on the problem of combining social interaction with task-based action in a dynamic, multiagent bartending domain, using an embodied robot. We discuss how social states are inferred from low-level sensors, using vision and speech as input modalities, and present a planning approach that models task, dialogue, and social actions in a simple bartending scenario. This approach allows us to build interesting plans, which have been evaluated in a real-world study with human subjects, using a general purpose, off-the-shelf planner, as an alternative to more mainstream methods of interaction management.