Metric space magnitude and generalisation in neural networks

Rayna Andreeva; Katharina Limbeck; Bastian Rieck; Rik Sarkar

Metric space magnitude and generalisation in neural networks

Rayna Andreeva, Katharina Limbeck, Bastian Rieck, Rik Sarkar

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Deep learning models have seen significant successes in numerous applications, but their inner workings remain elusive. The purpose of this work is to quantify the learning process of deep neural networks through the lens of a novel topological invariant called magnitude. Magnitude is an isometry invariant; its properties are an active area of research as it encodes many known invariants of a metric space. We use magnitude to study the internal representations of neural networks and propose a new method for determining their generalisation capabilities. Moreover, we theoretically connect magnitude dimension and the generalisation error, and demonstrate experimentally that the proposed framework can be a good indicator of the latter.

Original language	English
Title of host publication	Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML)
Publisher	Journal of Machine Learning Research: Workshop and Conference Proceedings
Pages	242-253
Number of pages	11
Volume	221
Edition	TAG-ML, ICML
Publication status	Published - 28 Jul 2023
Event	The Fortieth International Conference on Machine Learning - Honolulu, United States Duration: 23 Jul 2023 → 29 Jul 2023 Conference number: 40 https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Publisher	MIT-Press
Volume	221
ISSN (Electronic)	2640-3498

Conference

Conference	The Fortieth International Conference on Machine Learning
Abbreviated title	ICML 2023
Country/Territory	United States
City	Honolulu
Period	23/07/23 → 29/07/23
Internet address	https://icml.cc/

Access to Document

Metric Space_ANDREEVA_DOA16062023_VOR_CC BYFinal published version, 955 KBLicence: Creative Commons: Attribution (CC-BY)

Cite this

Andreeva, R., Limbeck, K., Rieck, B., & Sarkar, R. (2023). Metric space magnitude and generalisation in neural networks. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML) (TAG-ML, ICML ed., Vol. 221, pp. 242-253). (Proceedings of Machine Learning Research; Vol. 221). Journal of Machine Learning Research: Workshop and Conference Proceedings. https://proceedings.mlr.press/v221/andreeva23a/andreeva23a.pdf

Andreeva, Rayna ; Limbeck, Katharina ; Rieck, Bastian et al. / Metric space magnitude and generalisation in neural networks. Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML). Vol. 221 TAG-ML, ICML. ed. Journal of Machine Learning Research: Workshop and Conference Proceedings, 2023. pp. 242-253 (Proceedings of Machine Learning Research).

@inproceedings{ec2582521ca34932a90d5b40e9b228e8,

title = "Metric space magnitude and generalisation in neural networks",

abstract = "Deep learning models have seen significant successes in numerous applications, but their inner workings remain elusive. The purpose of this work is to quantify the learning process of deep neural networks through the lens of a novel topological invariant called magnitude. Magnitude is an isometry invariant; its properties are an active area of research as it encodes many known invariants of a metric space. We use magnitude to study the internal representations of neural networks and propose a new method for determining their generalisation capabilities. Moreover, we theoretically connect magnitude dimension and the generalisation error, and demonstrate experimentally that the proposed framework can be a good indicator of the latter.",

author = "Rayna Andreeva and Katharina Limbeck and Bastian Rieck and Rik Sarkar",

year = "2023",

month = jul,

day = "28",

language = "English",

volume = "221",

series = "Proceedings of Machine Learning Research",

publisher = "Journal of Machine Learning Research: Workshop and Conference Proceedings",

pages = "242--253",

booktitle = "Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML)",

edition = "TAG-ML, ICML",

note = "The Fortieth International Conference on Machine Learning, ICML 2023 ; Conference date: 23-07-2023 Through 29-07-2023",

url = "https://icml.cc/",

}

Andreeva, R, Limbeck, K, Rieck, B & Sarkar, R 2023, Metric space magnitude and generalisation in neural networks. in Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML). TAG-ML, ICML edn, vol. 221, Proceedings of Machine Learning Research, vol. 221, Journal of Machine Learning Research: Workshop and Conference Proceedings, pp. 242-253, The Fortieth International Conference on Machine Learning, Honolulu, Hawaii, United States, 23/07/23. <https://proceedings.mlr.press/v221/andreeva23a/andreeva23a.pdf>

Metric space magnitude and generalisation in neural networks. / Andreeva, Rayna; Limbeck, Katharina; Rieck, Bastian et al.
Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML). Vol. 221 TAG-ML, ICML. ed. Journal of Machine Learning Research: Workshop and Conference Proceedings, 2023. p. 242-253 (Proceedings of Machine Learning Research; Vol. 221).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Metric space magnitude and generalisation in neural networks

AU - Andreeva, Rayna

AU - Limbeck, Katharina

AU - Rieck, Bastian

AU - Sarkar, Rik

N1 - Conference code: 40

PY - 2023/7/28

Y1 - 2023/7/28

N2 - Deep learning models have seen significant successes in numerous applications, but their inner workings remain elusive. The purpose of this work is to quantify the learning process of deep neural networks through the lens of a novel topological invariant called magnitude. Magnitude is an isometry invariant; its properties are an active area of research as it encodes many known invariants of a metric space. We use magnitude to study the internal representations of neural networks and propose a new method for determining their generalisation capabilities. Moreover, we theoretically connect magnitude dimension and the generalisation error, and demonstrate experimentally that the proposed framework can be a good indicator of the latter.

AB - Deep learning models have seen significant successes in numerous applications, but their inner workings remain elusive. The purpose of this work is to quantify the learning process of deep neural networks through the lens of a novel topological invariant called magnitude. Magnitude is an isometry invariant; its properties are an active area of research as it encodes many known invariants of a metric space. We use magnitude to study the internal representations of neural networks and propose a new method for determining their generalisation capabilities. Moreover, we theoretically connect magnitude dimension and the generalisation error, and demonstrate experimentally that the proposed framework can be a good indicator of the latter.

M3 - Conference contribution

VL - 221

T3 - Proceedings of Machine Learning Research

SP - 242

EP - 253

BT - Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML)

PB - Journal of Machine Learning Research: Workshop and Conference Proceedings

T2 - The Fortieth International Conference on Machine Learning

Y2 - 23 July 2023 through 29 July 2023

ER -

Andreeva R, Limbeck K, Rieck B, Sarkar R. Metric space magnitude and generalisation in neural networks. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML). TAG-ML, ICML ed. Vol. 221. Journal of Machine Learning Research: Workshop and Conference Proceedings. 2023. p. 242-253. (Proceedings of Machine Learning Research).

Metric space magnitude and generalisation in neural networks

Abstract / Description of output

Publication series

Conference

Access to Document

Fingerprint

Cite this