DynoPlan: Combining Motion Planning and Deep Neural Network based Controllers for Safe HRL

Daniel Angelov, Yordan Hristov, Subramanian Ramamoorthy

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many realistic robotics tasks are best solved compositionally, through control architectures that sequentially invoke primitives and achieve error correction through the use of loops and conditionals taking the system back to alternative earlier states. Recent end-to-end approaches to task learning attempt to directly learn a single controller that solves an entire task, but this has been difficult for complex control tasks that would have otherwise required a diversity of local primitive moves, and the resulting solutions are also not easy to inspect for plan monitoring purposes. In this work, we aim to bridge the gap between hand designed and learned controllers, by representing each as an option in a hybrid hierarchical Reinforcement Learning framework - DynoPlan. We extend the options framework by adding a dynamics model and the use of a nearness-to-goal heuristic, derived from demonstrations. This translates the optimization of a hierarchical policy controller to a problem of planning with a model predictive controller. By unrolling the dynamics of each option and assessing the expected value of each future state, we can create a simple switching controller for choosing the optimal policy within a constrained time horizon similarly to hill climbing heuristic search. The individual dynamics model allows each option to iterate and be activated independently of the specific underlying instantiation, thus allowing for a mix of motion planning and deep neural network based primitives. We can assess the safety regions of the resulting hybrid controller by investigating the initiation sets of the different options, and also by reasoning about the completeness and performance guarantees of the underpinning motion planners.
Original languageEnglish
Title of host publicationMulti-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM) Proceedings
Number of pages5
Publication statusE-pub ahead of print - 10 Jul 2019
Event4th Multi-disciplinary Conference on Reinforcement Learning and Decision Making - McGill, Montreal, Canada
Duration: 7 Jul 201910 Jul 2019
Conference number: 4
http://rldm.org/

Conference

Conference4th Multi-disciplinary Conference on Reinforcement Learning and Decision Making
Abbreviated titleRLDM 2019
Country/TerritoryCanada
CityMontreal
Period7/07/1910/07/19
Internet address

Keywords

  • hierarchical options learning
  • safe motion planning
  • dynamics model

Fingerprint

Dive into the research topics of 'DynoPlan: Combining Motion Planning and Deep Neural Network based Controllers for Safe HRL'. Together they form a unique fingerprint.

Cite this