TY - JOUR
T1 - Taming Hyper-Parameters in Deep Learning Systems
AU - Mai, Luo
AU - Koliousis, Alexandros
AU - Li, Guo
AU - Brabete, Andrei-Octavian
AU - Pietzuch, Peter
PY - 2019/7/25
Y1 - 2019/7/25
N2 - Deep learning (DL) systems expose many tuning parameters ("hyper-parameters") that affect the performance and accuracy of trained models. Increasingly users struggle to configure hyper-parameters, and a substantial portion of time is spent tuning them empirically. We argue that future DL systems should be designed to help manage hyper-parameters. We describe how a distributed DL system can (i) remove the impact of hyper-parameters on both performance and accuracy, thus making it easier to decide on a good setting, and (ii) support more powerful dynamic policies for adapting hyper-parameters, which take monitored training metrics into account. We report results from prototype implementations that show the practicality of DL system designs that are hyper-parameter-friendly.
AB - Deep learning (DL) systems expose many tuning parameters ("hyper-parameters") that affect the performance and accuracy of trained models. Increasingly users struggle to configure hyper-parameters, and a substantial portion of time is spent tuning them empirically. We argue that future DL systems should be designed to help manage hyper-parameters. We describe how a distributed DL system can (i) remove the impact of hyper-parameters on both performance and accuracy, thus making it easier to decide on a good setting, and (ii) support more powerful dynamic policies for adapting hyper-parameters, which take monitored training metrics into account. We report results from prototype implementations that show the practicality of DL system designs that are hyper-parameter-friendly.
U2 - 10.1145/3352020.3352029
DO - 10.1145/3352020.3352029
M3 - Article
VL - 53
SP - 52
EP - 58
JO - Operating Systems Review
JF - Operating Systems Review
SN - 0163-5980
IS - 1
ER -