A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators |
| |
| Authors: | Brandon D. Gallas Andriy Bandos Frank W. Samuelson Robert F. Wagner |
| |
| Affiliation: | 1. NIBIB/CDRH Laboratory for the Assessment of Medical Imaging Systems , Silver Spring, Maryland, USA brandon.gallas@fda.hhs.gov;3. Department of Biostatistics , University of Pittsburgh , Pittsburgh, Pennsylvania, USA;4. NIBIB/CDRH Laboratory for the Assessment of Medical Imaging Systems , Silver Spring, Maryland, USA |
| |
| Abstract: | In this article, we analyze the three-way bootstrap estimate of the variance of the reader-averaged nonparametric area under the receiver operating characteristic (ROC) curve. The setting for this work is medical imaging, and the experimental design involves sampling from three distributions: a set of normal and diseased cases (patients), and a set of readers (doctors). The experiment we consider is fully crossed in that each reader reads each case. A reading generates a score that indicates the reader's level of suspicion that the patient is diseased. The distribution of scores for the normal patients is compared to the distribution of scores for the diseased patients via an ROC curve, and the area under the ROC curve (AUC) summarizes the reader's diagnostic ability to separate the normal patients from the diseased ones. We find that the bootstrap estimate of the variance of the reader-averaged AUC is biased, and we represent this bias in terms of moments of success outcomes. This representation helps unify and improve several current methods for multi-reader multi-case (MRMC) ROC analysis. |
| |
| Keywords: | Bias Multi-reader Multi-case (MRMC) Nonparametric AUC ROC analysis Three-way bootstrap Wilcoxon–Mann–Whitney statistic |
|
|
