Learning rewards from exploratory demonstrations using probabilistic temporal ranking

Michael Burke*, Katie Lu, Daniel Angelov, Arturas Straizys, Craig Innes, Kartic Subr, Subramanian Ramamoorthy

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

Informative path-planning is a well established approach to visual-servoing and active viewpoint selection in robotics, but typically assumes that a suitable cost function or goal state is known. This work considers the inverse problem, where the goal of the task is unknown, and a reward function needs to be inferred from exploratory example demonstrations provided by a demonstrator, for use in a downstream informative path-planning policy. Unfortunately, many existing reward inference strategies are unsuited to this class of problems, due to the exploratory nature of the demonstrations. In this paper, we propose an alternative approach to cope with the class of problems where these sub-optimal, exploratory demonstrations occur. We hypothesise that, in tasks which require discovery, successive states of any demonstration are progressively more likely to be associated with a higher reward, and use this hypothesis to generate time-based binary comparison outcomes and infer reward functions that support these ranks, under a probabilistic generative model. We formalise this probabilistic temporal ranking approach and show that it improves upon existing approaches to perform reward inference for autonomous ultrasound scanning, a novel application of learning from demonstration in medical imaging while also being of value across a broad range of goal-oriented learning from demonstration tasks.
Original languageEnglish
Pages (from-to)733-751
JournalAutonomous Robots
Issue number6
Early online date10 Jul 2023
Publication statusPublished - Aug 2023

Keywords / Materials (for Non-textual outputs)

  • visual servoing
  • reward inference
  • probabilistic temporal ranking


Dive into the research topics of 'Learning rewards from exploratory demonstrations using probabilistic temporal ranking'. Together they form a unique fingerprint.

Cite this