Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video

Miguel Jaques, Michael Burke, Timothy Hospedales

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We propose a model that is able to perform unsupervised physical parameter estimation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states or objects are not available. Existing physical scene understanding methods require either object state supervision, or do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through a physics-as-inverse-graphics approach that brings together vision-as-inverse-graphics and differentiable physics engines, enabling objects and explicit state and velocity representations to be discovered. This framework allows us to perform long term extrapolative video prediction, as well as vision-based model-predictive control. Our approach significantly out-performs related unsupervised methods in long-term future frame prediction of systems with interacting objects (such as ball-spring or 3-body gravitational systems), due to its ability to build dynamics into the model as an inductive bias. We further show the value of this tight vision-physics integration by demonstrating data-efficient learning of vision-actuated model-based control for a pendulum system. We also show that the controller’s interpretability provides unique capabilities in goal-driven control and physical reasoning for zero-data adaptation.
Original languageEnglish
Title of host publicationProceedings of the International Conference on Learning Representations (ICLR 2020)
Number of pages16
Publication statusPublished - 30 Apr 2020
EventEighth International Conference on Learning Representations - Millennium Hall, Virtual conference formerly Addis Ababa, Ethiopia
Duration: 26 Apr 202030 Apr 2020


ConferenceEighth International Conference on Learning Representations
Abbreviated titleICLR 2020
CityVirtual conference formerly Addis Ababa
Internet address


Dive into the research topics of 'Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video'. Together they form a unique fingerprint.

Cite this