Assessment of Radiologist Performance in the Detection of Lung Nodules

Samuel G. Armato, Rachael Y. Roberts, Masha Kocherginsky, Denise R. Aberle, Ella A. Kazerooni, Heber Macmahon, Edwin J.r. Van Beek, David Yankelevitz, Geoffrey Mclennan, Michael F. Mcnitt-gray, Charles R. Meyer, Anthony P. Reeves, Philip Caligiuri, Leslie E. Quint, Baskaran Sundaram, Barbara Y. Croft, Laurence P. Clarke

Research output: Contribution to journalArticlepeer-review


Rationale and Objectives

Studies that evaluate the lung-nodule-detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this study was to analyze (1) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the context of lung nodule detection on computed tomography (CT) scans.

Materials and Methods

Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥ 3 mm in maximum diameter. Panel “truth” sets of nodules then were derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule-detection performance of the other radiologists was evaluated based on these panel “truth” sets.


The number of “true” nodules in the different panel “truth” sets ranged from 15–89 (mean: 49.8±25.6). The mean radiologist nodule-detection sensitivities across radiologists and panel “truth” sets for different panel “truth” conditions ranged from 51.0–83.2%; mean false-positive rates ranged from 0.33–1.39 per case.


Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of “truth” on which lung nodule detection studies are based must be carefully considered, since even experienced thoracic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.
Keywords: lung nodule, computed tomography (CT), thoracic imaging, inter-observer variability, computer-aided diagnosis (CAD)
Original languageEnglish
Pages (from-to)28-38
Number of pages11
JournalAcademic Radiology
Issue number1
Publication statusPublished - 1 Jan 2009


  • Adult
  • Aged
  • Artifacts
  • Female
  • Humans
  • Lung Neoplasms
  • Male
  • Middle Aged
  • Observer Variation
  • Professional Competence
  • Radiographic Image Interpretation, Computer-Assisted
  • Reproducibility of Results
  • Sensitivity and Specificity
  • Solitary Pulmonary Nodule
  • Tomography, X-Ray Computed


Dive into the research topics of 'Assessment of Radiologist Performance in the Detection of Lung Nodules'. Together they form a unique fingerprint.

Cite this