Regularisation of Neural Networks by Enforcing Lipschitz Continuity

Henry Gouk, Eibe Frank, Bernhard Pfahringer, Michael J. Cree

Research output: Contribution to journalArticlepeer-review

Abstract

We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant—for multiple p-norms—of a feed forward neural network composed of commonly used layer types. Our technique is then used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods. Our evaluation study shows that the performance of the resulting models exceeds that of models trained with other common regularisers. We also provide evidence that the hyperparameters are intuitive to tune, demonstrate how the choice of norm for computing the Lipschitz constant impacts the resulting model, and show that the performance gains provided by our method are particularly noticeable when only a small amount of training data is available.
Original languageEnglish
Pages (from-to)393 - 416
Number of pages24
JournalMachine Learning
Volume110
Issue number2
Early online date6 Dec 2020
DOIs
Publication statusPublished - 1 Feb 2021

Keywords

  • Neural Networks
  • Regularisation
  • Lipschitz Continuity

Fingerprint

Dive into the research topics of 'Regularisation of Neural Networks by Enforcing Lipschitz Continuity'. Together they form a unique fingerprint.

Cite this