Multitask Soft Option Learning

Maximilian Igl; Andrew Gambardell; Nantas Nardelli; Jinke He; N. Siddharth; Wendelinv Böhmer; Shimon Whiteson

Multitask Soft Option Learning

Maximilian Igl, Andrew Gambardell, Nantas Nardelli, Jinke He, N. Siddharth, Wendelinv Böhmer, Shimon Whiteson

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present Multitask Soft Option Learning (MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This “soft” version of options avoids several instabilities during training in a multitask setting, and provides a natural way to learn both intra-option policies and their terminations. Furthermore, it allows fine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines.

Original language	English
Title of host publication	Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)
Publisher	PMLR
Pages	969-978
Number of pages	10
Publication status	Published - 6 Aug 2020
Event	36th Conference on Uncertainty in Artificial Intelligence 2020 - Virtual conference, Canada Duration: 3 Aug 2020 → 6 Aug 2020 http://www.auai.org/~w-auai/uai2020/index.php

Publication series

Name	Proceedings of Machine Learning Research
Volume	124
ISSN (Electronic)	2640-3498

Conference

Conference	36th Conference on Uncertainty in Artificial Intelligence 2020
Abbreviated title	UAI 2020
Country/Territory	Canada
City	Virtual conference
Period	3/08/20 → 6/08/20
Internet address	http://www.auai.org/~w-auai/uai2020/index.php

Access to Document

http://proceedings.mlr.press/v124/igl20a.htmlLicence: All Rights Reserved

Cite this

@inproceedings{28b49e1ca90041afbda3b1313d426dd8,

title = "Multitask Soft Option Learning",

abstract = "We present Multitask Soft Option Learning (MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This “soft” version of options avoids several instabilities during training in a multitask setting, and provides a natural way to learn both intra-option policies and their terminations. Furthermore, it allows fine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines. ",

author = "Maximilian Igl and Andrew Gambardell and Nantas Nardelli and Jinke He and N. Siddharth and Wendelinv B{\"o}hmer and Shimon Whiteson",

year = "2020",

month = aug,

day = "6",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "969--978",

booktitle = "Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)",

note = "36th Conference on Uncertainty in Artificial Intelligence 2020, UAI 2020 ; Conference date: 03-08-2020 Through 06-08-2020",

url = "http://www.auai.org/~w-auai/uai2020/index.php",

}

Igl, M, Gambardell, A, Nardelli, N, He, J, Siddharth, N, Böhmer, W & Whiteson, S 2020, Multitask Soft Option Learning. in Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI). Proceedings of Machine Learning Research, vol. 124, PMLR, pp. 969-978, 36th Conference on Uncertainty in Artificial Intelligence 2020, Virtual conference, Ontario, Canada, 3/08/20. <http://proceedings.mlr.press/v124/igl20a.html>

TY - GEN

T1 - Multitask Soft Option Learning

AU - Igl, Maximilian

AU - Gambardell, Andrew

AU - Nardelli, Nantas

AU - He, Jinke

AU - Siddharth, N.

AU - Böhmer, Wendelinv

AU - Whiteson, Shimon

PY - 2020/8/6

Y1 - 2020/8/6

N2 - We present Multitask Soft Option Learning (MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This “soft” version of options avoids several instabilities during training in a multitask setting, and provides a natural way to learn both intra-option policies and their terminations. Furthermore, it allows fine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines.

AB - We present Multitask Soft Option Learning (MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This “soft” version of options avoids several instabilities during training in a multitask setting, and provides a natural way to learn both intra-option policies and their terminations. Furthermore, it allows fine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines.

M3 - Conference contribution

T3 - Proceedings of Machine Learning Research

SP - 969

EP - 978

BT - Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)

PB - PMLR

T2 - 36th Conference on Uncertainty in Artificial Intelligence 2020

Y2 - 3 August 2020 through 6 August 2020

ER -

Multitask Soft Option Learning

Abstract / Description of output

Publication series

Conference

Access to Document

Fingerprint

Cite this