MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes

Henry Gouk, Bernhard Pfahringer, Eibe Frank, Michael J. Cree

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will have a regularising effect, similar to how constraining the Lipschitz constant of a network has been shown to improve generalisation. A simple algorithm is provided that involves rescaling the weight matrix of each layer after each parameter update. We conduct a series of studies on common benchmark datasets, and also a novel dataset that we introduce to enable easier significance testing for experiments using convolutional networks. Performance on these datasets compares favourably with other common regularisation techniques. Data related to this paper is available at: https://www.cs.waikato.ac.nz/textasciitildeml/sins10/.
Original languageEnglish
Title of host publicationECML PKDD 2018: Machine Learning and Knowledge Discovery in Databases
EditorsMichele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, Georgiana Ifrim
Place of PublicationCham
PublisherSpringer International Publishing AG
Pages541-556
Number of pages16
Volume11051
ISBN (Electronic)978-3-030-10925-7
ISBN (Print)978-3-030-10924-0
DOIs
Publication statusPublished - 2019
EventEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Dublin, Ireland
Duration: 10 Sept 201814 Sept 2018
http://www.ecmlpkdd2018.org/

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume11051
ISSN (Print)0302-9743

Conference

ConferenceEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Abbreviated titleECML PKDD
Country/TerritoryIreland
CityDublin
Period10/09/1814/09/18
Internet address

Fingerprint

Dive into the research topics of 'MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes'. Together they form a unique fingerprint.

Cite this