Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

Chenyang Zhao; Timothy M Hospedales

Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback–Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.

Original language	English
Title of host publication	Proceedings of The 13th Asian Conference on Machine Learning
Editors	Vineeth N. Balasubramanian, Ivor Tsang
Publisher	PMLR
Pages	1237-1252
Number of pages	16
Publication status	Published - 17 Nov 2021
Event	13th Asian Conference on Machine Learning - Virtual Duration: 17 Nov 2021 → 19 Nov 2021 http://www.acml-conf.org/2021/

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR
Volume	157
ISSN (Electronic)	2640-3498

Conference

Conference	13th Asian Conference on Machine Learning
Abbreviated title	ACML 2021
Period	17/11/21 → 19/11/21
Internet address	http://www.acml-conf.org/2021/

Keywords / Materials (for Non-textual outputs)

domain randomisation
deep reinforcement learning
mutual learning

Access to Document

Robust Domain Randomised_ZHAO_DOA18092021_AFVAccepted author manuscript, 904 KBLicence: Creative Commons: Attribution (CC-BY)

https://proceedings.mlr.press/v157/zhao21b.htmlLicence: All Rights Reserved

Cite this

@inproceedings{ae94e3343e2e43ac91825c72febdf496,

title = "Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation",

abstract = "In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback–Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.",

keywords = "domain randomisation, deep reinforcement learning, mutual learning",

author = "Chenyang Zhao and Hospedales, {Timothy M}",

year = "2021",

month = nov,

day = "17",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "1237--1252",

editor = "Balasubramanian, {Vineeth N.} and Ivor Tsang",

booktitle = "Proceedings of The 13th Asian Conference on Machine Learning",

note = "13th Asian Conference on Machine Learning, ACML 2021 ; Conference date: 17-11-2021 Through 19-11-2021",

url = "http://www.acml-conf.org/2021/",

}

Zhao, C & Hospedales, TM 2021, Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation. in VN Balasubramanian & I Tsang (eds), Proceedings of The 13th Asian Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 157, PMLR, pp. 1237-1252, 13th Asian Conference on Machine Learning, 17/11/21. <https://proceedings.mlr.press/v157/zhao21b.html>

Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation. / Zhao, Chenyang; Hospedales, Timothy M.
Proceedings of The 13th Asian Conference on Machine Learning. ed. / Vineeth N. Balasubramanian; Ivor Tsang. PMLR, 2021. p. 1237-1252 (Proceedings of Machine Learning Research; Vol. 157).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

AU - Zhao, Chenyang

AU - Hospedales, Timothy M

PY - 2021/11/17

Y1 - 2021/11/17

N2 - In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback–Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.

AB - In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback–Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.

KW - domain randomisation

KW - deep reinforcement learning

KW - mutual learning

M3 - Conference contribution

T3 - Proceedings of Machine Learning Research

SP - 1237

EP - 1252

BT - Proceedings of The 13th Asian Conference on Machine Learning

A2 - Balasubramanian, Vineeth N.

A2 - Tsang, Ivor

PB - PMLR

T2 - 13th Asian Conference on Machine Learning

Y2 - 17 November 2021 through 19 November 2021

ER -

Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

Abstract

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this