Understanding metric-related pitfalls in image analysis validation

Annika Reinke; Minu D. Tizabi; Michael Baumgartner; Matthias Eisenmann; Doreen Heckmann-Nötzel; A. Emre Kavur; Tim Rädsch; Carole H. Sudre; Laura Acion; Michela Antonelli; Tal Arbel; Spyridon Bakas; Arriel Benis; Matthew Blaschko; Florian Büttner; M. Jorge Cardoso; Veronika Cheplygina; Jianxu Chen; Evangelia Christodoulou; Beth A. Cimini; Gary S. Collins; Keyvan Farahani; Luciana Ferrer; Adrian Galdran; Bram van Ginneken; Ben Glocker; Patrick Godau; Robert Haase; Daniel A. Hashimoto; Michael M. Hoffman; Merel Huisman; Fabian Isensee; Pierre Jannin; Charles E. Kahn; Dagmar Kainmueller; Bernhard Kainz; Alexandros Karargyris; Alan Karthikesalingam; Hannes Kenngott; Jens Kleesiek; Florian Kofler; Thijs Kooi; Annette Kopp-Schneider; Michal Kozubek; Anna Kreshuk; Tahsin Kurc; Bennett A. Landman; Geert Litjens; Amin Madani; Klaus Maier-Hein; Anne L. Martel; Peter Mattson; Erik Meijering; Bjoern Menze; Karel G. M. Moons; Henning Müller; Brennan Nichyporuk; Felix Nickel; Jens Petersen; Susanne M. Rafelski; Nasir Rajpoot; Mauricio Reyes; Michael A. Riegler; Nicola Rieke; Julio Saez-Rodriguez; Clara I. Sánchez; Shravya Shetty; Maarten van Smeden; Ronald M. Summers; Abdel A. Taha; Aleksei Tiulpin; Sotirios A. Tsaftaris; Ben Van Calster; Gaël Varoquaux; Manuel Wiesenfarth; Ziv R. Yaniv; Paul F. Jäger; Lena Maier-Hein

doi:10.48550/arXiv.2302.01790

Understanding metric-related pitfalls in image analysis validation

Annika Reinke, Minu D. Tizabi, Michael Baumgartner, Matthias Eisenmann, Doreen Heckmann-Nötzel, A. Emre Kavur, Tim Rädsch, Carole H. Sudre, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, Arriel Benis, Matthew Blaschko, Florian Büttner, M. Jorge Cardoso, Veronika Cheplygina, Jianxu Chen, Evangelia Christodoulou, Beth A. CiminiGary S. Collins, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Ben Glocker, Patrick Godau, Robert Haase, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Alan Karthikesalingam, Hannes Kenngott, Jens Kleesiek, Florian Kofler, Thijs Kooi, Annette Kopp-Schneider, Michal Kozubek, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern Menze, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, Jens Petersen, Susanne M. Rafelski, Nasir Rajpoot, Mauricio Reyes, Michael A. Riegler, Nicola Rieke, Julio Saez-Rodriguez, Clara I. Sánchez, Shravya Shetty, Maarten van Smeden, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Manuel Wiesenfarth, Ziv R. Yaniv, Paul F. Jäger, Lena Maier-Hein

School of Engineering

Research output: Working paper › Preprint

Abstract

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Original language	English
Publisher	ArXiv
DOIs	https://doi.org/10.48550/arXiv.2302.01790
Publication status	Published - 2023

Keywords / Materials (for Non-textual outputs)

cs.CV

Access to Document

10.48550/arXiv.2302.01790Licence: Creative Commons: Attribution-NonCommercial-NoDerivatives (CC BY-NC-ND)

2302.01790v2

https://arxiv.org/abs/2302.01790Licence: Creative Commons: Attribution-NonCommercial-NoDerivatives (CC BY-NC-ND)

1 Article

Understanding metric-related pitfalls in image analysis validation
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Bakas, S., Benis, A., Buettner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B. A., Farahani, K. & Ferrer, L. & 49 others, Galdran, A., van Ginneken, B., Glocker, B., Godau, P., Hashimoto, D. A., Hoffman, M. M., Huisman, M., Isensee, F., Jannin, P., Kahn, C. E., Kainmueller, D., Kainz, B., Karargyris, A., Kleesiek, J., Kofler, F., Kooi, T., Kopp-Schneider, A., Kozubek, M., Kreshuk, A., Kurc, T., Landman, B. A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A. L., Meijering, E., Menze, B., Moons, K. G. M., Müller, H., Nichyporuk, B., Nickel, F., Petersen, J., Rafelski, S. M., Rajpoot, N., Reyes, M., Riegler, M. A., Rieke, N., Saez-Rodriguez, J., Sánchez, C. I., Shetty, S., Summers, R. M., Taha, A. A., Tiulpin, A., Tsaftaris, S. A., Van Calster, B., Varoquaux, G., Yaniv, Z. R., Jäger, P. F. & Maier-Hein, L., 12 Feb 2024, (E-pub ahead of print) In: Nature Methods. 21, 2, p. 182-194
Research output: Contribution to journal › Article › peer-review

Open Access
File

Cite this

Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Blaschko, M., Büttner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., ... Maier-Hein, L. (2023). Understanding metric-related pitfalls in image analysis validation. ArXiv. https://doi.org/10.48550/arXiv.2302.01790

@techreport{844fb8cfe2ca4f2f906a9df5441c16e7,

title = "Understanding metric-related pitfalls in image analysis validation",

abstract = " Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation. ",

keywords = "cs.CV",

author = "Annika Reinke and Tizabi, {Minu D.} and Michael Baumgartner and Matthias Eisenmann and Doreen Heckmann-N{\"o}tzel and Kavur, {A. Emre} and Tim R{\"a}dsch and Sudre, {Carole H.} and Laura Acion and Michela Antonelli and Tal Arbel and Spyridon Bakas and Arriel Benis and Matthew Blaschko and Florian B{\"u}ttner and Cardoso, {M. Jorge} and Veronika Cheplygina and Jianxu Chen and Evangelia Christodoulou and Cimini, {Beth A.} and Collins, {Gary S.} and Keyvan Farahani and Luciana Ferrer and Adrian Galdran and Ginneken, {Bram van} and Ben Glocker and Patrick Godau and Robert Haase and Hashimoto, {Daniel A.} and Hoffman, {Michael M.} and Merel Huisman and Fabian Isensee and Pierre Jannin and Kahn, {Charles E.} and Dagmar Kainmueller and Bernhard Kainz and Alexandros Karargyris and Alan Karthikesalingam and Hannes Kenngott and Jens Kleesiek and Florian Kofler and Thijs Kooi and Annette Kopp-Schneider and Michal Kozubek and Anna Kreshuk and Tahsin Kurc and Landman, {Bennett A.} and Geert Litjens and Amin Madani and Klaus Maier-Hein and Martel, {Anne L.} and Peter Mattson and Erik Meijering and Bjoern Menze and Moons, {Karel G. M.} and Henning M{\"u}ller and Brennan Nichyporuk and Felix Nickel and Jens Petersen and Rafelski, {Susanne M.} and Nasir Rajpoot and Mauricio Reyes and Riegler, {Michael A.} and Nicola Rieke and Julio Saez-Rodriguez and S{\'a}nchez, {Clara I.} and Shravya Shetty and Smeden, {Maarten van} and Summers, {Ronald M.} and Taha, {Abdel A.} and Aleksei Tiulpin and Tsaftaris, {Sotirios A.} and Calster, {Ben Van} and Ga{\"e}l Varoquaux and Manuel Wiesenfarth and Yaniv, {Ziv R.} and J{\"a}ger, {Paul F.} and Lena Maier-Hein",

year = "2023",

doi = "10.48550/arXiv.2302.01790",

language = "English",

publisher = "ArXiv",

type = "WorkingPaper",

institution = "ArXiv",

}

Reinke, A, Tizabi, MD, Baumgartner, M, Eisenmann, M, Heckmann-Nötzel, D, Kavur, AE, Rädsch, T, Sudre, CH, Acion, L, Antonelli, M, Arbel, T, Bakas, S, Benis, A, Blaschko, M, Büttner, F, Cardoso, MJ, Cheplygina, V, Chen, J, Christodoulou, E, Cimini, BA, Collins, GS, Farahani, K, Ferrer, L, Galdran, A, Ginneken, BV, Glocker, B, Godau, P, Haase, R, Hashimoto, DA, Hoffman, MM, Huisman, M, Isensee, F, Jannin, P, Kahn, CE, Kainmueller, D, Kainz, B, Karargyris, A, Karthikesalingam, A, Kenngott, H, Kleesiek, J, Kofler, F, Kooi, T, Kopp-Schneider, A, Kozubek, M, Kreshuk, A, Kurc, T, Landman, BA, Litjens, G, Madani, A, Maier-Hein, K, Martel, AL, Mattson, P, Meijering, E, Menze, B, Moons, KGM, Müller, H, Nichyporuk, B, Nickel, F, Petersen, J, Rafelski, SM, Rajpoot, N, Reyes, M, Riegler, MA, Rieke, N, Saez-Rodriguez, J, Sánchez, CI, Shetty, S, Smeden, MV, Summers, RM, Taha, AA, Tiulpin, A, Tsaftaris, SA, Calster, BV, Varoquaux, G, Wiesenfarth, M, Yaniv, ZR, Jäger, PF & Maier-Hein, L 2023 'Understanding metric-related pitfalls in image analysis validation' ArXiv. https://doi.org/10.48550/arXiv.2302.01790

TY - UNPB

T1 - Understanding metric-related pitfalls in image analysis validation

AU - Reinke, Annika

AU - Tizabi, Minu D.

AU - Baumgartner, Michael

AU - Eisenmann, Matthias

AU - Heckmann-Nötzel, Doreen

AU - Kavur, A. Emre

AU - Rädsch, Tim

AU - Sudre, Carole H.

AU - Acion, Laura

AU - Antonelli, Michela

AU - Arbel, Tal

AU - Bakas, Spyridon

AU - Benis, Arriel

AU - Blaschko, Matthew

AU - Büttner, Florian

AU - Cardoso, M. Jorge

AU - Cheplygina, Veronika

AU - Chen, Jianxu

AU - Christodoulou, Evangelia

AU - Cimini, Beth A.

AU - Collins, Gary S.

AU - Farahani, Keyvan

AU - Ferrer, Luciana

AU - Galdran, Adrian

AU - Ginneken, Bram van

AU - Glocker, Ben

AU - Godau, Patrick

AU - Haase, Robert

AU - Hashimoto, Daniel A.

AU - Hoffman, Michael M.

AU - Huisman, Merel

AU - Isensee, Fabian

AU - Jannin, Pierre

AU - Kahn, Charles E.

AU - Kainmueller, Dagmar

AU - Kainz, Bernhard

AU - Karargyris, Alexandros

AU - Karthikesalingam, Alan

AU - Kenngott, Hannes

AU - Kleesiek, Jens

AU - Kofler, Florian

AU - Kooi, Thijs

AU - Kopp-Schneider, Annette

AU - Kozubek, Michal

AU - Kreshuk, Anna

AU - Kurc, Tahsin

AU - Landman, Bennett A.

AU - Litjens, Geert

AU - Madani, Amin

AU - Maier-Hein, Klaus

AU - Martel, Anne L.

AU - Mattson, Peter

AU - Meijering, Erik

AU - Menze, Bjoern

AU - Moons, Karel G. M.

AU - Müller, Henning

AU - Nichyporuk, Brennan

AU - Nickel, Felix

AU - Petersen, Jens

AU - Rafelski, Susanne M.

AU - Rajpoot, Nasir

AU - Reyes, Mauricio

AU - Riegler, Michael A.

AU - Rieke, Nicola

AU - Saez-Rodriguez, Julio

AU - Sánchez, Clara I.

AU - Shetty, Shravya

AU - Smeden, Maarten van

AU - Summers, Ronald M.

AU - Taha, Abdel A.

AU - Tiulpin, Aleksei

AU - Tsaftaris, Sotirios A.

AU - Calster, Ben Van

AU - Varoquaux, Gaël

AU - Wiesenfarth, Manuel

AU - Yaniv, Ziv R.

AU - Jäger, Paul F.

AU - Maier-Hein, Lena

PY - 2023

Y1 - 2023

N2 - Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

AB - Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

KW - cs.CV

U2 - 10.48550/arXiv.2302.01790

DO - 10.48550/arXiv.2302.01790

M3 - Preprint

BT - Understanding metric-related pitfalls in image analysis validation

PB - ArXiv

ER -

Understanding metric-related pitfalls in image analysis validation

Abstract

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Research output

Understanding metric-related pitfalls in image analysis validation

Cite this