Navigating noise: A study of how noise influences generalisation and calibration of neural networks

Martin Ferianc, Ondrej Bohdal, Timothy M. Hospedales, Miguel R. D. Rodrigues

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

Enhancing the generalisation abilities of neural networks (NNs) through integrating noise such as MixUp or Dropout during training has emerged as a powerful and adaptable technique. Despite the proven efficacy of noise in NN training, there is no consensus regarding which noise sources, types and placements yield maximal benefits in generalisation and confidence calibration. This study thoroughly explores diverse noise modalities to evaluate their impacts on NN's generalisation and calibration under in-distribution or out-of-distribution settings, paired with experiments investigating the metric landscapes of the learnt representations, across a spectrum of NN architectures, tasks, and datasets. Our study shows that AugMix and weak augmentation exhibit cross-task effectiveness in computer vision, emphasising the need to tailor noise to specific domains. Our findings emphasise the efficacy of combining noises and successful hyperparameter transfer within a single domain but the difficulties in transferring the benefits to other domains. Furthermore, the study underscores the complexity of simultaneously optimising for both generalisation and calibration, emphasising the need for practitioners to carefully consider noise combinations and hyperparameter tuning for optimal performance in specific tasks and datasets.
Original languageEnglish
Pages (from-to)1-44
Number of pages44
JournalTransactions on Machine Learning Research
Publication statusPublished - 3 Apr 2024

Fingerprint

Dive into the research topics of 'Navigating noise: A study of how noise influences generalisation and calibration of neural networks'. Together they form a unique fingerprint.

Cite this