Vision-based system identification and 3D keypoint discovery using dynamics constraints

Miguel Jaques; Martin Asenov; Michael Burke; Timothy M Hospedales

Vision-based system identification and 3D keypoint discovery using dynamics constraints

Miguel Jaques, Martin Asenov, Michael Burke, Timothy M Hospedales

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.

Original language	English
Title of host publication	Proceedings of The 4th Annual Learning for Dynamics and Control Conference
Subtitle of host publication	Volume 168: Learning for Dynamics and Control Conference, 23-24 June 2022, Stanford University, Stanford, CA, USA
Editors	Roya Firoozi, Negar Mehr, Esen Yel, Rika Antonova, Jeannette Bohg, Mac Schwager, Mykel Kochendorfer
Publisher	PMLR
Pages	316-329
Number of pages	14
Volume	168
Publication status	Published - 11 May 2022
Event	4th Annual Learning for Dynamics & Control Conference - Stanford, United States Duration: 23 Jun 2022 → 24 Jun 2022 Conference number: 4 https://l4dc.stanford.edu/

Conference

Conference	4th Annual Learning for Dynamics & Control Conference
Abbreviated title	L4DC 2022
Country/Territory	United States
City	Stanford
Period	23/06/22 → 24/06/22
Internet address	https://l4dc.stanford.edu/

Access to Document

Vision-Based_JAQUES_DOA01032022_AFVAccepted author manuscript, 12.2 MBLicence: Creative Commons: Attribution (CC-BY)

https://proceedings.mlr.press/v168/jaques22a.htmlLicence: All Rights Reserved

Cite this

Jaques, M., Asenov, M., Burke, M., & Hospedales, T. M. (2022). Vision-based system identification and 3D keypoint discovery using dynamics constraints. In R. Firoozi, N. Mehr, E. Yel, R. Antonova, J. Bohg, M. Schwager, & M. Kochendorfer (Eds.), Proceedings of The 4th Annual Learning for Dynamics and Control Conference: Volume 168: Learning for Dynamics and Control Conference, 23-24 June 2022, Stanford University, Stanford, CA, USA (Vol. 168, pp. 316-329). PMLR. https://proceedings.mlr.press/v168/jaques22a.html

Jaques, Miguel ; Asenov, Martin ; Burke, Michael et al. / Vision-based system identification and 3D keypoint discovery using dynamics constraints. Proceedings of The 4th Annual Learning for Dynamics and Control Conference: Volume 168: Learning for Dynamics and Control Conference, 23-24 June 2022, Stanford University, Stanford, CA, USA. editor / Roya Firoozi ; Negar Mehr ; Esen Yel ; Rika Antonova ; Jeannette Bohg ; Mac Schwager ; Mykel Kochendorfer. Vol. 168 PMLR, 2022. pp. 316-329

@inproceedings{6699363c606e479ca5cdb5a2ea3c6016,

title = "Vision-based system identification and 3D keypoint discovery using dynamics constraints",

abstract = "This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.",

author = "Miguel Jaques and Martin Asenov and Michael Burke and Hospedales, {Timothy M}",

year = "2022",

month = may,

day = "11",

language = "English",

volume = "168",

pages = "316--329",

editor = "Roya Firoozi and Negar Mehr and Esen Yel and Rika Antonova and Jeannette Bohg and Mac Schwager and Mykel Kochendorfer",

booktitle = "Proceedings of The 4th Annual Learning for Dynamics and Control Conference",

publisher = "PMLR",

note = "4th Annual Learning for Dynamics & Control Conference<br/>, L4DC 2022 ; Conference date: 23-06-2022 Through 24-06-2022",

url = "https://l4dc.stanford.edu/",

}

Jaques, M, Asenov, M, Burke, M & Hospedales, TM 2022, Vision-based system identification and 3D keypoint discovery using dynamics constraints. in R Firoozi, N Mehr, E Yel, R Antonova, J Bohg, M Schwager & M Kochendorfer (eds), Proceedings of The 4th Annual Learning for Dynamics and Control Conference: Volume 168: Learning for Dynamics and Control Conference, 23-24 June 2022, Stanford University, Stanford, CA, USA. vol. 168, PMLR, pp. 316-329, 4th Annual Learning for Dynamics & Control Conference
, Stanford, California, United States, 23/06/22. <https://proceedings.mlr.press/v168/jaques22a.html>

Vision-based system identification and 3D keypoint discovery using dynamics constraints. / Jaques, Miguel; Asenov, Martin; Burke, Michael et al.
Proceedings of The 4th Annual Learning for Dynamics and Control Conference: Volume 168: Learning for Dynamics and Control Conference, 23-24 June 2022, Stanford University, Stanford, CA, USA. ed. / Roya Firoozi; Negar Mehr; Esen Yel; Rika Antonova; Jeannette Bohg; Mac Schwager; Mykel Kochendorfer. Vol. 168 PMLR, 2022. p. 316-329.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Vision-based system identification and 3D keypoint discovery using dynamics constraints

AU - Jaques, Miguel

AU - Asenov, Martin

AU - Burke, Michael

AU - Hospedales, Timothy M

N1 - Conference code: 4

PY - 2022/5/11

Y1 - 2022/5/11

N2 - This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.

AB - This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.

M3 - Conference contribution

VL - 168

SP - 316

EP - 329

BT - Proceedings of The 4th Annual Learning for Dynamics and Control Conference

A2 - Firoozi, Roya

A2 - Mehr, Negar

A2 - Yel, Esen

A2 - Antonova, Rika

A2 - Bohg, Jeannette

A2 - Schwager, Mac

A2 - Kochendorfer, Mykel

PB - PMLR

T2 - 4th Annual Learning for Dynamics & Control Conference<br/>

Y2 - 23 June 2022 through 24 June 2022

ER -

Jaques M, Asenov M, Burke M, Hospedales TM. Vision-based system identification and 3D keypoint discovery using dynamics constraints. In Firoozi R, Mehr N, Yel E, Antonova R, Bohg J, Schwager M, Kochendorfer M, editors, Proceedings of The 4th Annual Learning for Dynamics and Control Conference: Volume 168: Learning for Dynamics and Control Conference, 23-24 June 2022, Stanford University, Stanford, CA, USA. Vol. 168. PMLR. 2022. p. 316-329

Vision-based system identification and 3D keypoint discovery using dynamics constraints

Abstract

Conference

Access to Document

Fingerprint

Cite this