Common Limitations of Image Processing Metrics: A Picture Story

Annika Reinke; Minu D. Tizabi; Carole H. Sudre; Matthias Eisenmann; Tim Rädsch; Michael Baumgartner; Laura Acion; Michela Antonelli; Spyridon Bakas; Peter Bankhead; Arriel Benis; M. Jorge Cardoso; Veronika Cheplygina; Evangelia Christodoulou; Beth Cimini; Gary S. Collins; Keyvan Farahani; Bram van Ginneken; Ben Glocker; Patrick Godau; Fred Hamprecht; Daniel A. Hashimoto; Doreen Heckmann-Nötzel; Michael M. Hoffman; Merel Huisman; Fabian Isensee; Pierre Jannin; Charles E. Kahn; Alexandros Karargyris; Alan Karthikesalingam; Bernhard Kainz; Emre Kavur; Hannes Kenngott; Jens Kleesiek; Thijs Kooi; Michal Kozubek; Anna Kreshuk; Tahsin Kurc; Bennett A. Landman; Geert Litjens; Amin Madani; Klaus Maier-Hein; Anne L. Martel; Peter Mattson; Erik Meijering; Bjoern Menze; David Moher; Karel G. M. Moons; Henning Müller; Brennan Nichyporuk; Felix Nickel; M. Alican Noyan; Jens Petersen; Gorkem Polat; Nasir Rajpoot; Mauricio Reyes; Nicola Rieke; Michael Riegler; Hassan Rivaz; Julio Saez-Rodriguez; Clarisa Sanchez Gutierrez; Julien Schroeter; Anindo Saha; Shravya Shetty; Maarten van Smeden; Bram Stieltjes; Ronald M. Summers; Abdel A. Taha; Sotirios A. Tsaftaris; Ben Van Calster; Gaël Varoquaux; Manuel Wiesenfarth; Ziv R. Yaniv; Annette Kopp-Schneider; Paul Jäger; Lena Maier-Hein

doi:10.48550/arXiv.2104.05642

Common Limitations of Image Processing Metrics: A Picture Story

Annika Reinke, Minu D. Tizabi, Carole H. Sudre, Matthias Eisenmann, Tim Rädsch, Michael Baumgartner, Laura Acion, Michela Antonelli, Spyridon Bakas, Peter Bankhead, Arriel Benis, M. Jorge Cardoso, Veronika Cheplygina, Evangelia Christodoulou, Beth Cimini, Gary S. Collins, Keyvan Farahani, Bram van Ginneken, Ben Glocker, Patrick GodauFred Hamprecht, Daniel A. Hashimoto, Doreen Heckmann-Nötzel, Michael M. Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Alexandros Karargyris, Alan Karthikesalingam, Bernhard Kainz, Emre Kavur, Hannes Kenngott, Jens Kleesiek, Thijs Kooi, Michal Kozubek, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern Menze, David Moher, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, M. Alican Noyan, Jens Petersen, Gorkem Polat, Nasir Rajpoot, Mauricio Reyes, Nicola Rieke, Michael Riegler, Hassan Rivaz, Julio Saez-Rodriguez, Clarisa Sanchez Gutierrez, Julien Schroeter, Anindo Saha, Shravya Shetty, Maarten van Smeden, Bram Stieltjes, Ronald M. Summers, Abdel A. Taha, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Manuel Wiesenfarth, Ziv R. Yaniv, Annette Kopp-Schneider, Paul Jäger, Lena Maier-Hein

Research output: Working paper › Preprint

Abstract / Description of output

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

Original language	English
Publisher	ArXiv
DOIs	https://doi.org/10.48550/arXiv.2104.05642
Publication status	Published - 2021

Access to Document

10.48550/arXiv.2104.05642

2104.05642v6Submitted manuscript, 23.3 MB

https://arxiv.org/abs/2104.05642

Cite this

Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Bakas, S., Bankhead, P., Benis, A., Cardoso, M. J., Cheplygina, V., Christodoulou, E., Cimini, B., Collins, G. S., Farahani, K., Ginneken, B. V., Glocker, B., ... Maier-Hein, L. (2021). Common Limitations of Image Processing Metrics: A Picture Story. ArXiv. https://doi.org/10.48550/arXiv.2104.05642

@techreport{f96b2c4fc9b44019bc12d4472b2e20c6,

title = "Common Limitations of Image Processing Metrics: A Picture Story",

abstract = " While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide. ",

author = "Annika Reinke and Tizabi, {Minu D.} and Sudre, {Carole H.} and Matthias Eisenmann and Tim R{\"a}dsch and Michael Baumgartner and Laura Acion and Michela Antonelli and Spyridon Bakas and Peter Bankhead and Arriel Benis and Cardoso, {M. Jorge} and Veronika Cheplygina and Evangelia Christodoulou and Beth Cimini and Collins, {Gary S.} and Keyvan Farahani and Ginneken, {Bram van} and Ben Glocker and Patrick Godau and Fred Hamprecht and Hashimoto, {Daniel A.} and Doreen Heckmann-N{\"o}tzel and Hoffman, {Michael M.} and Merel Huisman and Fabian Isensee and Pierre Jannin and Kahn, {Charles E.} and Alexandros Karargyris and Alan Karthikesalingam and Bernhard Kainz and Emre Kavur and Hannes Kenngott and Jens Kleesiek and Thijs Kooi and Michal Kozubek and Anna Kreshuk and Tahsin Kurc and Landman, {Bennett A.} and Geert Litjens and Amin Madani and Klaus Maier-Hein and Martel, {Anne L.} and Peter Mattson and Erik Meijering and Bjoern Menze and David Moher and Moons, {Karel G. M.} and Henning M{\"u}ller and Brennan Nichyporuk and Felix Nickel and Noyan, {M. Alican} and Jens Petersen and Gorkem Polat and Nasir Rajpoot and Mauricio Reyes and Nicola Rieke and Michael Riegler and Hassan Rivaz and Julio Saez-Rodriguez and Gutierrez, {Clarisa Sanchez} and Julien Schroeter and Anindo Saha and Shravya Shetty and Smeden, {Maarten van} and Bram Stieltjes and Summers, {Ronald M.} and Taha, {Abdel A.} and Tsaftaris, {Sotirios A.} and Calster, {Ben Van} and Ga{\"e}l Varoquaux and Manuel Wiesenfarth and Yaniv, {Ziv R.} and Annette Kopp-Schneider and Paul J{\"a}ger and Lena Maier-Hein",

note = "This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact [email protected] or [email protected]. Substantial contributions to this document will be acknowledged with a co-authorship",

year = "2021",

doi = "10.48550/arXiv.2104.05642",

language = "English",

publisher = "ArXiv",

type = "WorkingPaper",

institution = "ArXiv",

}

Reinke, A, Tizabi, MD, Sudre, CH, Eisenmann, M, Rädsch, T, Baumgartner, M, Acion, L, Antonelli, M, Bakas, S, Bankhead, P, Benis, A, Cardoso, MJ, Cheplygina, V, Christodoulou, E, Cimini, B, Collins, GS, Farahani, K, Ginneken, BV, Glocker, B, Godau, P, Hamprecht, F, Hashimoto, DA, Heckmann-Nötzel, D, Hoffman, MM, Huisman, M, Isensee, F, Jannin, P, Kahn, CE, Karargyris, A, Karthikesalingam, A, Kainz, B, Kavur, E, Kenngott, H, Kleesiek, J, Kooi, T, Kozubek, M, Kreshuk, A, Kurc, T, Landman, BA, Litjens, G, Madani, A, Maier-Hein, K, Martel, AL, Mattson, P, Meijering, E, Menze, B, Moher, D, Moons, KGM, Müller, H, Nichyporuk, B, Nickel, F, Noyan, MA, Petersen, J, Polat, G, Rajpoot, N, Reyes, M, Rieke, N, Riegler, M, Rivaz, H, Saez-Rodriguez, J, Gutierrez, CS, Schroeter, J, Saha, A, Shetty, S, Smeden, MV, Stieltjes, B, Summers, RM, Taha, AA, Tsaftaris, SA, Calster, BV, Varoquaux, G, Wiesenfarth, M, Yaniv, ZR, Kopp-Schneider, A, Jäger, P & Maier-Hein, L 2021 'Common Limitations of Image Processing Metrics: A Picture Story' ArXiv. https://doi.org/10.48550/arXiv.2104.05642

TY - UNPB

T1 - Common Limitations of Image Processing Metrics

T2 - A Picture Story

AU - Reinke, Annika

AU - Tizabi, Minu D.

AU - Sudre, Carole H.

AU - Eisenmann, Matthias

AU - Rädsch, Tim

AU - Baumgartner, Michael

AU - Acion, Laura

AU - Antonelli, Michela

AU - Bakas, Spyridon

AU - Bankhead, Peter

AU - Benis, Arriel

AU - Cardoso, M. Jorge

AU - Cheplygina, Veronika

AU - Christodoulou, Evangelia

AU - Cimini, Beth

AU - Collins, Gary S.

AU - Farahani, Keyvan

AU - Ginneken, Bram van

AU - Glocker, Ben

AU - Godau, Patrick

AU - Hamprecht, Fred

AU - Hashimoto, Daniel A.

AU - Heckmann-Nötzel, Doreen

AU - Hoffman, Michael M.

AU - Huisman, Merel

AU - Isensee, Fabian

AU - Jannin, Pierre

AU - Kahn, Charles E.

AU - Karargyris, Alexandros

AU - Karthikesalingam, Alan

AU - Kainz, Bernhard

AU - Kavur, Emre

AU - Kenngott, Hannes

AU - Kleesiek, Jens

AU - Kooi, Thijs

AU - Kozubek, Michal

AU - Kreshuk, Anna

AU - Kurc, Tahsin

AU - Landman, Bennett A.

AU - Litjens, Geert

AU - Madani, Amin

AU - Maier-Hein, Klaus

AU - Martel, Anne L.

AU - Mattson, Peter

AU - Meijering, Erik

AU - Menze, Bjoern

AU - Moher, David

AU - Moons, Karel G. M.

AU - Müller, Henning

AU - Nichyporuk, Brennan

AU - Nickel, Felix

AU - Noyan, M. Alican

AU - Petersen, Jens

AU - Polat, Gorkem

AU - Rajpoot, Nasir

AU - Reyes, Mauricio

AU - Rieke, Nicola

AU - Riegler, Michael

AU - Rivaz, Hassan

AU - Saez-Rodriguez, Julio

AU - Gutierrez, Clarisa Sanchez

AU - Schroeter, Julien

AU - Saha, Anindo

AU - Shetty, Shravya

AU - Smeden, Maarten van

AU - Stieltjes, Bram

AU - Summers, Ronald M.

AU - Taha, Abdel A.

AU - Tsaftaris, Sotirios A.

AU - Calster, Ben Van

AU - Varoquaux, Gaël

AU - Wiesenfarth, Manuel

AU - Yaniv, Ziv R.

AU - Kopp-Schneider, Annette

AU - Jäger, Paul

AU - Maier-Hein, Lena

N1 - This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact [email protected] or [email protected]. Substantial contributions to this document will be acknowledged with a co-authorship

PY - 2021

Y1 - 2021

N2 - While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

AB - While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

U2 - 10.48550/arXiv.2104.05642

DO - 10.48550/arXiv.2104.05642

M3 - Preprint

BT - Common Limitations of Image Processing Metrics

PB - ArXiv

ER -

Common Limitations of Image Processing Metrics: A Picture Story

Abstract / Description of output

Access to Document

Fingerprint

Cite this