Understanding metric-related pitfalls in image analysis validation

Annika Reinke; Minu D. Tizabi; Michael Baumgartner; Matthias Eisenmann; Doreen Heckmann-Nötzel; A. Emre Kavur; Tim Rädsch; Carole H. Sudre; Laura Acion; Michela Antonelli; Spyridon Bakas; Arriel Benis; Florian Buettner; M. Jorge Cardoso; Veronika Cheplygina; Jianxu Chen; Evangelia Christodoulou; Beth A. Cimini; Keyvan Farahani; Luciana Ferrer; Adrian Galdran; Bram van Ginneken; Ben Glocker; Patrick Godau; Daniel A. Hashimoto; Michael M. Hoffman; Merel Huisman; Fabian Isensee; Pierre Jannin; Charles E. Kahn; Dagmar Kainmueller; Bernhard Kainz; Alexandros Karargyris; Jens Kleesiek; Florian Kofler; Thijs Kooi; Annette Kopp-Schneider; Michal Kozubek; Anna Kreshuk; Tahsin Kurc; Bennett A. Landman; Geert Litjens; Amin Madani; Klaus Maier-Hein; Anne L. Martel; Erik Meijering; Bjoern Menze; Karel G. M. Moons; Henning Müller; Brennan Nichyporuk; Felix Nickel; Jens Petersen; Susanne M. Rafelski; Nasir Rajpoot; Mauricio Reyes; Michael A. Riegler; Nicola Rieke; Julio Saez-Rodriguez; Clara I. Sánchez; Shravya Shetty; Ronald M. Summers; Abdel A. Taha; Aleksei Tiulpin; Sotirios A. Tsaftaris; Ben Van Calster; Gaël Varoquaux; Ziv R. Yaniv; Paul F. Jäger; Lena Maier-Hein

doi:10.1038/s41592-023-02150-0

Understanding metric-related pitfalls in image analysis validation

Annika Reinke, Minu D. Tizabi, Michael Baumgartner, Matthias Eisenmann, Doreen Heckmann-Nötzel, A. Emre Kavur, Tim Rädsch, Carole H. Sudre, Laura Acion, Michela Antonelli, Spyridon Bakas, Arriel Benis, Florian Buettner, M. Jorge Cardoso, Veronika Cheplygina, Jianxu Chen, Evangelia Christodoulou, Beth A. Cimini, Keyvan Farahani, Luciana FerrerAdrian Galdran, Bram van Ginneken, Ben Glocker, Patrick Godau, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Jens Kleesiek, Florian Kofler, Thijs Kooi, Annette Kopp-Schneider, Michal Kozubek, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Erik Meijering, Bjoern Menze, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, Jens Petersen, Susanne M. Rafelski, Nasir Rajpoot, Mauricio Reyes, Michael A. Riegler, Nicola Rieke, Julio Saez-Rodriguez, Clara I. Sánchez, Shravya Shetty, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Ziv R. Yaniv, Paul F. Jäger, Lena Maier-Hein

School of Engineering

Research output: Contribution to journal › Article › peer-review

Abstract

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Original language	English
Pages (from-to)	182-194
Journal	Nature Methods
Volume	21
Issue number	2
Early online date	12 Feb 2024
DOIs	https://doi.org/10.1038/s41592-023-02150-0
Publication status	E-pub ahead of print - 12 Feb 2024

Keywords / Materials (for Non-textual outputs)

Artificial Intelligence

Access to Document

10.1038/s41592-023-02150-0

Understanding metric-related pitfalls in image analysis validationAccepted author manuscript, 10.1 MBLicence: Creative Commons: Attribution (CC-BY)

http://rdcu.be/dysgX

1 Active

Canon Medical / RAEng Senior Research Fellow in Healthcare AI
Tsaftaris, S. (Principal Investigator)
Canon Medical Research Europe
31/03/19 → 30/06/26
Project: Research

1 Preprint

Understanding metric-related pitfalls in image analysis validation
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Blaschko, M., Büttner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E. & Cimini, B. A. & 58 others, Collins, G. S., Farahani, K., Ferrer, L., Galdran, A., Ginneken, B. V., Glocker, B., Godau, P., Haase, R., Hashimoto, D. A., Hoffman, M. M., Huisman, M., Isensee, F., Jannin, P., Kahn, C. E., Kainmueller, D., Kainz, B., Karargyris, A., Karthikesalingam, A., Kenngott, H., Kleesiek, J., Kofler, F., Kooi, T., Kopp-Schneider, A., Kozubek, M., Kreshuk, A., Kurc, T., Landman, B. A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A. L., Mattson, P., Meijering, E., Menze, B., Moons, K. G. M., Müller, H., Nichyporuk, B., Nickel, F., Petersen, J., Rafelski, S. M., Rajpoot, N., Reyes, M., Riegler, M. A., Rieke, N., Saez-Rodriguez, J., Sánchez, C. I., Shetty, S., Smeden, M. V., Summers, R. M., Taha, A. A., Tiulpin, A., Tsaftaris, S. A., Calster, B. V., Varoquaux, G., Wiesenfarth, M., Yaniv, Z. R., Jäger, P. F. & Maier-Hein, L., 2023, ArXiv.
Research output: Working paper › Preprint

File

Cite this

Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Bakas, S., Benis, A., Buettner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B. A., Farahani, K., ... Maier-Hein, L. (2024). Understanding metric-related pitfalls in image analysis validation. Nature Methods, 21(2), 182-194. Advance online publication. https://doi.org/10.1038/s41592-023-02150-0

@article{78c001a8914240bf9e7ff52b1784f7c4,

title = "Understanding metric-related pitfalls in image analysis validation",

abstract = "Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.",

keywords = "Artificial Intelligence",

author = "Annika Reinke and Tizabi, \{Minu D.\} and Michael Baumgartner and Matthias Eisenmann and Doreen Heckmann-N{\"o}tzel and Kavur, \{A. Emre\} and Tim R{\"a}dsch and Sudre, \{Carole H.\} and Laura Acion and Michela Antonelli and Spyridon Bakas and Arriel Benis and Florian Buettner and Cardoso, \{M. Jorge\} and Veronika Cheplygina and Jianxu Chen and Evangelia Christodoulou and Cimini, \{Beth A.\} and Keyvan Farahani and Luciana Ferrer and Adrian Galdran and \{van Ginneken\}, Bram and Ben Glocker and Patrick Godau and Hashimoto, \{Daniel A.\} and Hoffman, \{Michael M.\} and Merel Huisman and Fabian Isensee and Pierre Jannin and Kahn, \{Charles E.\} and Dagmar Kainmueller and Bernhard Kainz and Alexandros Karargyris and Jens Kleesiek and Florian Kofler and Thijs Kooi and Annette Kopp-Schneider and Michal Kozubek and Anna Kreshuk and Tahsin Kurc and Landman, \{Bennett A.\} and Geert Litjens and Amin Madani and Klaus Maier-Hein and Martel, \{Anne L.\} and Erik Meijering and Bjoern Menze and Moons, \{Karel G. M.\} and Henning M{\"u}ller and Brennan Nichyporuk and Felix Nickel and Jens Petersen and Rafelski, \{Susanne M.\} and Nasir Rajpoot and Mauricio Reyes and Riegler, \{Michael A.\} and Nicola Rieke and Julio Saez-Rodriguez and S{\'a}nchez, \{Clara I.\} and Shravya Shetty and Summers, \{Ronald M.\} and Taha, \{Abdel A.\} and Aleksei Tiulpin and Tsaftaris, \{Sotirios A.\} and \{Van Calster\}, Ben and Ga{\"e}l Varoquaux and Yaniv, \{Ziv R.\} and J{\"a}ger, \{Paul F.\} and Lena Maier-Hein",

year = "2024",

month = feb,

day = "12",

doi = "10.1038/s41592-023-02150-0",

language = "English",

volume = "21",

pages = "182--194",

journal = "Nature Methods",

issn = "1548-7105",

publisher = "Nature Publishing Group",

number = "2",

}

Reinke, A, Tizabi, MD, Baumgartner, M, Eisenmann, M, Heckmann-Nötzel, D, Kavur, AE, Rädsch, T, Sudre, CH, Acion, L, Antonelli, M, Bakas, S, Benis, A, Buettner, F, Cardoso, MJ, Cheplygina, V, Chen, J, Christodoulou, E, Cimini, BA, Farahani, K, Ferrer, L, Galdran, A, van Ginneken, B, Glocker, B, Godau, P, Hashimoto, DA, Hoffman, MM, Huisman, M, Isensee, F, Jannin, P, Kahn, CE, Kainmueller, D, Kainz, B, Karargyris, A, Kleesiek, J, Kofler, F, Kooi, T, Kopp-Schneider, A, Kozubek, M, Kreshuk, A, Kurc, T, Landman, BA, Litjens, G, Madani, A, Maier-Hein, K, Martel, AL, Meijering, E, Menze, B, Moons, KGM, Müller, H, Nichyporuk, B, Nickel, F, Petersen, J, Rafelski, SM, Rajpoot, N, Reyes, M, Riegler, MA, Rieke, N, Saez-Rodriguez, J, Sánchez, CI, Shetty, S, Summers, RM, Taha, AA, Tiulpin, A, Tsaftaris, SA, Van Calster, B, Varoquaux, G, Yaniv, ZR, Jäger, PF & Maier-Hein, L 2024, 'Understanding metric-related pitfalls in image analysis validation', Nature Methods, vol. 21, no. 2, pp. 182-194. https://doi.org/10.1038/s41592-023-02150-0

TY - JOUR

T1 - Understanding metric-related pitfalls in image analysis validation

AU - Reinke, Annika

AU - Tizabi, Minu D.

AU - Baumgartner, Michael

AU - Eisenmann, Matthias

AU - Heckmann-Nötzel, Doreen

AU - Kavur, A. Emre

AU - Rädsch, Tim

AU - Sudre, Carole H.

AU - Acion, Laura

AU - Antonelli, Michela

AU - Bakas, Spyridon

AU - Benis, Arriel

AU - Buettner, Florian

AU - Cardoso, M. Jorge

AU - Cheplygina, Veronika

AU - Chen, Jianxu

AU - Christodoulou, Evangelia

AU - Cimini, Beth A.

AU - Farahani, Keyvan

AU - Ferrer, Luciana

AU - Galdran, Adrian

AU - van Ginneken, Bram

AU - Glocker, Ben

AU - Godau, Patrick

AU - Hashimoto, Daniel A.

AU - Hoffman, Michael M.

AU - Huisman, Merel

AU - Isensee, Fabian

AU - Jannin, Pierre

AU - Kahn, Charles E.

AU - Kainmueller, Dagmar

AU - Kainz, Bernhard

AU - Karargyris, Alexandros

AU - Kleesiek, Jens

AU - Kofler, Florian

AU - Kooi, Thijs

AU - Kopp-Schneider, Annette

AU - Kozubek, Michal

AU - Kreshuk, Anna

AU - Kurc, Tahsin

AU - Landman, Bennett A.

AU - Litjens, Geert

AU - Madani, Amin

AU - Maier-Hein, Klaus

AU - Martel, Anne L.

AU - Meijering, Erik

AU - Menze, Bjoern

AU - Moons, Karel G. M.

AU - Müller, Henning

AU - Nichyporuk, Brennan

AU - Nickel, Felix

AU - Petersen, Jens

AU - Rafelski, Susanne M.

AU - Rajpoot, Nasir

AU - Reyes, Mauricio

AU - Riegler, Michael A.

AU - Rieke, Nicola

AU - Saez-Rodriguez, Julio

AU - Sánchez, Clara I.

AU - Shetty, Shravya

AU - Summers, Ronald M.

AU - Taha, Abdel A.

AU - Tiulpin, Aleksei

AU - Tsaftaris, Sotirios A.

AU - Van Calster, Ben

AU - Varoquaux, Gaël

AU - Yaniv, Ziv R.

AU - Jäger, Paul F.

AU - Maier-Hein, Lena

PY - 2024/2/12

Y1 - 2024/2/12

N2 - Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

AB - Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

KW - Artificial Intelligence

U2 - 10.1038/s41592-023-02150-0

DO - 10.1038/s41592-023-02150-0

M3 - Article

C2 - 38347140

SN - 1548-7105

VL - 21

SP - 182

EP - 194

JO - Nature Methods

JF - Nature Methods

IS - 2

ER -

Understanding metric-related pitfalls in image analysis validation

Abstract

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Projects

Canon Medical / RAEng Senior Research Fellow in Healthcare AI

Research output

Understanding metric-related pitfalls in image analysis validation

Cite this