Reasoning and explanation generation in ad hoc collaboration between humans and embodied AI

An assistive embodied AI agent often has to collaborate with previously unseen humans. State of the art frameworks for such ad hoc teamwork use a large labeled dataset of prior observations to model the behavior of other agents and to determine the ad hoc (i.e., embodied AI) agent’s behavior. These approaches do not support rapid incremental revisions or transparency, and the necessary resources (e.g., training examples, computation) are not readily available in practical domains. Our previous work introduced an architecture that enabled an ad hoc agent to choose its actions in simple simulated domains based on non-monotonic logical reasoning with prior domain knowledge and models learned from limited examples to predict the behavior of other agents. Here, we extend this architecture to enable an ad hoc embodied AI agent to collaborate with a human performing household tasks in a complex indoor environment, focusing on the ad hoc agent’s ability to identify and reason with relevant knowledge, and provide relational descriptions as explanations of its behavior and that of the human. We evaluate our architecture’s capabilities in VirtualHome, a realistic 3D simulation environment.