Taming Hyper-Parameters in Deep Learning Systems

Luo Mai, Alexandros Koliousis, Guo Li, Andrei-Octavian Brabete, Peter Pietzuch

Research output: Contribution to journalArticlepeer-review


Deep learning (DL) systems expose many tuning parameters ("hyper-parameters") that affect the performance and accuracy of trained models. Increasingly users struggle to configure hyper-parameters, and a substantial portion of time is spent tuning them empirically. We argue that future DL systems should be designed to help manage hyper-parameters. We describe how a distributed DL system can (i) remove the impact of hyper-parameters on both performance and accuracy, thus making it easier to decide on a good setting, and (ii) support more powerful dynamic policies for adapting hyper-parameters, which take monitored training metrics into account. We report results from prototype implementations that show the practicality of DL system designs that are hyper-parameter-friendly.
Original languageEnglish
Pages (from-to)52–58
Number of pages7
JournalOperating Systems Review
Issue number1
Publication statusPublished - 25 Jul 2019


Dive into the research topics of 'Taming Hyper-Parameters in Deep Learning Systems'. Together they form a unique fingerprint.

Cite this