Importance sampling in reinforcement learning with an estimated behavior policy

Josiah P. Hanna, Scott Niekum, Peter Stone

Research output: Contribution to journalArticlepeer-review


In reinforcement learning, importance sampling is a widely used method for evaluating an expectation under the distribution of data of one policy when the data has in fact been generated by a different policy. Importance sampling requires computing the likelihood ratio between the action probabilities of a target policy and those of the data-producing behavior policy. In this article, we study importance sampling where the behavior policy action probabilities are replaced by their maximum likelihood estimate of these probabilities under the observed data. We show this general technique reduces variance due to sampling error in Monte Carlo style estimators. We introduce two novel estimators that use this technique to estimate expected values that arise in the RL literature. We find that these general estimators reduce the variance of Monte Carlo sampling methods, leading to faster learning for policy gradient algorithms and more accurate off-policy policy evaluation. We also provide theoretical analysis showing that our new estimators are consistent and have asymptotically lower variance than Monte Carlo estimators.
Original languageEnglish
Pages (from-to)1267-1317
Number of pages51
JournalMachine Learning
Issue number6
Early online date7 May 2021
Publication statusPublished - 1 Jun 2021


  • Reinforcement learning
  • Policy evaluation
  • Importance sampling


Dive into the research topics of 'Importance sampling in reinforcement learning with an estimated behavior policy'. Together they form a unique fingerprint.

Cite this