Evaluation of lung MDCT nodule annotation across radiologists and methods

Charles R Meyer, Timothy D Johnson, Geoffrey McLennan, Denise R Aberle, Ella A Kazerooni, Heber Macmahon, Brian F Mullan, David F Yankelevitz, Edwin J R van Beek, Samuel G Armato, Michael F McNitt-Gray, Anthony P Reeves, David Gur, Claudia I Henschke, Eric A Hoffman, Peyton H Bland, Gary Laderach, Richie Pais, David Qing, Chris PikerJunfeng Guo, Adam Starkey, Daniel Max, Barbara Y Croft, Laurence P Clarke

Research output: Contribution to journalArticlepeer-review


RATIONALE AND OBJECTIVES: Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.

MATERIALS AND METHODS: The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.

RESULTS: Across the combination of all nodules, volume and p-map model parameters were found to be significant at P <.05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively.

CONCLUSION: Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.
Original languageEnglish
Pages (from-to)1254-65
Number of pages12
JournalAcademic Radiology
Issue number10
Publication statusPublished - Oct 2006


  • Artificial Intelligence
  • Humans
  • Image Interpretation, Computer-Assisted
  • Lung Neoplasms
  • Observer Variation
  • Pattern Recognition, Automated
  • Physicians
  • Professional Competence
  • Radiology
  • Reproducibility of Results
  • Sensitivity and Specificity
  • Solitary Pulmonary Nodule
  • Task Performance and Analysis
  • Tomography, X-Ray Computed


Dive into the research topics of 'Evaluation of lung MDCT nodule annotation across radiologists and methods'. Together they form a unique fingerprint.

Cite this