On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We present a reformulation of the stochastic optimal control problem in terms of KL divergence minimisation, not only providing a unifying perspective of previous approaches in this area, but also demonstrating that the formalism leads to novel practical approaches to the control problem. Specifically, a natural relaxation of the dual formulation gives rise to exact iterative solutions to the finite and infinite horizon stochastic optimal control problem, while direct application of Bayesian inference methods yields instances of risk sensitive control. We furthermore study corresponding formulations in the reinforcement learning setting and present model free algorithms for problems with both discrete and continuous state and action spaces. Evaluation of the proposed methods on the standard Gridworld and Cart-Pole benchmarks verifies the theoretical insights and shows that the proposed methods improve upon current approaches.
Original languageEnglish
Title of host publicationRobotics: Science and Systems VIII (RSS 2012)
Number of pages8
Publication statusPublished - 2012


Dive into the research topics of 'On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference'. Together they form a unique fingerprint.

Cite this