Learning physically-instantiated game play through visual observation

Andrei Barbu, Siddharth Narayanaswamy, Jeffrey Mark Siskind

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We present an integrated vision and robotic system that plays, and learns to play, simple physically-instantiated board games that are variants of TIC TAC TOE and HEXA-PAWN. We employ novel custom vision and robotic hardware designed specifically for this learning task. The game rules can be parametrically specified. Two independent computational agents alternate playing the two opponents with the shared vision and robotic hardware, using pre-specified rule sets. A third independent computational agent, sharing the same hardware, learns the game rules solely by observing the physical play, without access to the pre-specified rule set, using inductive logic programming with minimal background knowledge possessed by human children. The vision component of our integrated system reliably detects the position of the board in the image and reconstructs the game state after every move, from a single image. The robotic component reliably moves pieces both between board positions and to and from off-board positions as needed by an arbitrary parametrically-specified legal-move generator. Thus the rules of games learned solely by observing physical play can drive further physical play. We demonstrate our system learning to play six different games.
Original languageEnglish
Title of host publication2010 IEEE International Conference on Robotics and Automation
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages8
ISBN (Print)978-1-4244-5038-1
Publication statusPublished - 15 Jul 2010
Event2010 IEEE International Conference on Robotics and Automation - Anchorage, United States
Duration: 3 May 20108 May 2010

Publication series

ISSN (Print)1050-4729


Conference2010 IEEE International Conference on Robotics and Automation
Country/TerritoryUnited States


Dive into the research topics of 'Learning physically-instantiated game play through visual observation'. Together they form a unique fingerprint.

Cite this