Generalized Confidence Interval Estimation for the Difference in Paired Areas Under the ROC Curves in the Absence of a Gold Standard

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may be too expensive or infeasible. In many medical researches, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, we propose a heuristic method of estimating confidence intervals for the difference in paired AUCs of two diagnostic tests in the absence of a GS reference. This heuristic method is a three-stage method by combining the expectation-maximization (EM) algorithm, bootstrap method, and an estimation based on asymptotic generalized pivotal quantities (GPQs) to construct generalized confidence intervals for the difference in paired AUCs in the absence of a GS. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and expected interval lengths. The numerical example using a published dataset illustrates the proposed method.     

Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions

The receiver operating characteristic (ROC) curve is a graphical representation of the relationship between false positive and true positive rates. It is a widely used statistical tool for describing the accuracy of a diagnostic test. In this paper we propose a new nonparametric ROC curve estimator based on the smoothed empirical distribution functions. We prove its strong consistency and perform a simulation study to compare it with some other popular nonparametric estimators of the ROC curve. We also apply the proposed method to a real data set.     

Compare diagnostic tests using transformation-invariant smoothed ROC curves()

Receiver operating characteristic (ROC) curve, plotting true positive rates against false positive rates as threshold varies, is an important tool for evaluating biomarkers in diagnostic medicine studies. By definition, ROC curve is monotone increasing from 0 to 1 and is invariant to any monotone transformation of test results. And it is often a curve with certain level of smoothness when test results from the diseased and non-diseased subjects follow continuous distributions. Most existing ROC curve estimation methods do not guarantee all of these properties. One of the exceptions is Du and Tang (2009) which applies certain monotone spline regression procedure to empirical ROC estimates. However, their method does not consider the inherent correlations between empirical ROC estimates. This makes the derivation of the asymptotic properties very difficult. In this paper we propose a penalized weighted least square estimation method, which incorporates the covariance between empirical ROC estimates as a weight matrix. The resulting estimator satisfies all the aforementioned properties, and we show that it is also consistent. Then a resampling approach is used to extend our method for comparisons of two or more diagnostic tests. Our simulations show a significantly improved performance over the existing method, especially for steep ROC curves. We then apply the proposed method to a cancer diagnostic study that compares several newly developed diagnostic biomarkers to a traditional one.     

Applications of the Bootstrap in ROC Analysis

The problem of estimating standard errors for diagnostic accuracy measures might be challenging for many complicated models. We can address such a problem by using the Bootstrap methods to blunt its technical edge with resampled empirical distributions. We consider two cases where bootstrap methods can successfully improve our knowledge of the sampling variability of the diagnostic accuracy estimators. The first application is to make inference for the area under the ROC curve resulted from a functional logistic regression model which is a sophisticated modelling device to describe the relationship between a dichotomous response and multiple covariates. We consider using this regression method to model the predictive effects of multiple independent variables on the occurrence of a disease. The accuracy measures, such as the area under the ROC curve (AUC) are developed from the functional regression. Asymptotical results for the empirical estimators are provided to facilitate inferences. The second application is to test the difference of two weighted areas under the ROC curve (WAUC) from a paired two sample study. The correlation between the two WAUC complicates the asymptotic distribution of the test statistic. We then employ the bootstrap methods to gain satisfactory inference results. Simulations and examples are supplied in this article to confirm the merits of the bootstrap methods.     

Extending the Barnard’s test to non-inferiority

In 1945, George Alfred Barnard presented an unconditional exact test to compare two independent proportions. Critical regions for this test, by construction accomplish the very useful property of being Barnard convex sets. Besides, there are empirical findings suggesting that Barnard’s test is the most generally powerful. For Barnard’s test, calculation of critical regions is complicated due that they are constructed in an iterative form until is obtained a test size, as close as possible to the nominal significance level and less than or equal to it. In this article we present an extension to non-inferiority of this very leading test. This extension was contructed for any dissimilarity measure and tables were constructed for the difference between proportions. Also we calculate the critical regions for this extended test for sample sizes less or equal than 30, nominal significance level 0.01, 0.025, 0.05, and 0.10 and for non-inferiority margins 0.05, 0.10, 0.15, and 0.20. Additionally, we computed test sizes for the mentioned configurations. To do this calculations, we have written a program in the R environment.     

Receiver operating characteristic surfaces in the presence of verification bias

Summary.  In diagnostic medicine, the receiver operating characteristic (ROC) surface is one of the established tools for assessing the accuracy of a diagnostic test in discriminating three disease states, and the volume under the ROC surface has served as a summary index for diagnostic accuracy. In practice, the selection for definitive disease examination may be based on initial test measurements and induces verification bias in the assessment. We propose a non-parametric likelihood-based approach to construct the empirical ROC surface in the presence of differential verification, and to estimate the volume under the ROC surface. Estimators of the standard deviation are derived by both the Fisher information and the jackknife method, and their relative accuracy is evaluated in an extensive simulation study. The methodology is further extended to incorporate discrete baseline covariates in the selection process, and to compare the accuracy of a pair of diagnostic tests. We apply the proposed method to compare the diagnostic accuracy between mini-mental state examination and clinical evaluation of dementia, in discriminating between three disease states of Alzheimer's disease.     

Detecting diagnostic accuracy of two biomarkers through a bivariate log-normal ROC curve

In biomedical research, two or more biomarkers may be available for diagnosis of a particular disease. Selecting one single biomarker which ideally discriminate a diseased group from a healthy group is confront in a diagnostic process. Frequently, most of the people use the accuracy measure, area under the receiver operating characteristic (ROC) curve to choose the best diagnostic marker among the available markers for diagnosis. Some authors have tried to combine the multiple markers by an optimal linear combination to increase the discriminatory power. In this paper, we propose an alternative method that combines two continuous biomarkers by direct bivariate modeling of the ROC curve under log-normality assumption. The proposed method is applied to simulated data set and prostate cancer diagnostic biomarker data set.     

Adjusting ROC curves for covariates in the presence of verification bias

The ROC (receiver operating characteristic) curve is frequently used for describing effectiveness of a diagnostic marker or test. Classical estimation of the ROC curve uses independent identically distributed samples taken randomly from the healthy and diseased populations. Frequently not all subjects undergo a definitive gold standard assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased (verification bias). In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve adjusted for covariates (ROC regression) under verification bias. We develop the estimator's asymptotic distribution and examine its finite sample size properties via a simulation study. We apply this procedure to fingerstick postprandial blood glucose measurement data adjusting for age.     

Local linear smoothing of receiver operating characteristic (ROC) curves

For measuring the accuracy of a continuous diagnostic test, the receiver operating characteristic (ROC) curve is often used. The empirical ROC curve is the most commonly used non-parametric estimator for the ROC curve. Recently, Lloyd (J. Amer. Statist. Assoc. 93(1998) 1356) proposed a kernel smoothing estimator for the ROC curve and showed his estimator has better mean square error than the empirical ROC curve estimator. However, Lloyd's estimator involves two bandwidths and has a boundary problem. In addition, his choice of bandwidths is ad hoc. In this paper we propose another kernel smoothing estimator which involves only one bandwidth and does not have the boundary problem. Furthermore, our choice of the bandwidth is asymptotically optimal.     

Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface

In many situations the diagnostic decision is not limited to a binary choice. Binary statistical tools such as receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) need to be expanded to address three-category classification problem. Previous authors have suggest various ways to model the extension of AUC but not the ROC surface. Only simple parametric approaches are proposed for modeling the ROC measure under the assumption that test results all follow normal distributions. We study the estimation methods of three-dimensional ROC surfaces with nonparametric and semiparametric estimators. Asymptotical results are provided as a basis for statistical inference. Simulation studies are performed to assess the validity of our proposed methods in finite samples. We consider an Alzheimer's disease example from a clinical study in the US as an illustration. The nonparametric and semiparametric modelling approaches for the three way ROC analysis can be readily generalized to diagnostic problems with more than three classes.     

Assessing accuracy of a continuous screening test in the presence of verification bias

Summary.  In studies to assess the accuracy of a screening test, often definitive disease assessment is too invasive or expensive to be ascertained on all the study subjects. Although it may be more ethical or cost effective to ascertain the true disease status with a higher rate in study subjects where the screening test or additional information is suggestive of disease, estimates of accuracy can be biased in a study with such a design. This bias is known as verification bias. Verification bias correction methods that accommodate screening tests with binary or ordinal responses have been developed; however, no verification bias correction methods exist for tests with continuous results. We propose and compare imputation and reweighting bias-corrected estimators of true and false positive rates, receiver operating characteristic curves and area under the receiver operating characteristic curve for continuous tests. Distribution theory and simulation studies are used to compare the proposed estimators with respect to bias, relative efficiency and robustness to model misspecification. The bias correction estimators proposed are applied to data from a study of screening tests for neonatal hearing loss.     

Estimation of the area under ROC curve with censored data

The area under the receiver operating characteristic curve is the most commonly used summary measure of diagnostic accuracy for a continuous-scale diagnostic test. In this paper, we develop methods to estimate the area under the curve (AUC) with censored data. Based on two different integration representations of this parameter, two nonparametric estimators are defined by the “plug in” method. Both the proposed estimators are shown to be asymptotically normal based on counting process and martingale theory. A simulation study is conducted to evaluate the performances of the proposed estimators.     

Nonparametric predictive inference for diagnostic test thresholds

Abstract

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning and credit scoring. The receiver operating characteristic (ROC) curve and surface are useful tools to assess the ability of diagnostic tests to discriminate between ordered classes or groups. To define these diagnostic tests, selecting the optimal thresholds that maximize the accuracy of these tests is required. One procedure that is commonly used to find the optimal thresholds is by maximizing what is known as Youden’s index. This article presents nonparametric predictive inference (NPI) for selecting the optimal thresholds of a diagnostic test. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. Based on multiple future observations, the NPI approach is presented for selecting the optimal thresholds for two-group and three-group scenarios. In addition, a pairwise approach has also been presented for the three-group scenario. The article ends with an example to illustrate the proposed methods and a simulation study of the predictive performance of the proposed methods along with some classical methods such as Youden index. The NPI-based methods show some interesting results that overcome some of the issues concerning the predictive performance of Youden’s index.     

Optimal Threshold from ROC and CAP Curves

Receiver Operating Characteristic (ROC) and Cumulative Accuracy Profile (CAP) curves are used to assess the discriminatory power of different credit-rating approaches. The thresholds of optimal classification accuracy on an ROC curve and of maximal profit on a CAP curve can be found by using iso-performance tangent lines, which are based on the standard notion of accuracy. In this article, we propose another accuracy measure called the true rate. Using this rate, one can obtain alternative optimal thresholds on both ROC and CAP curves. For most real populations of borrowers, the number of the defaults is much less than that of the non defaults, and in such cases using the true rate may be more efficient than using the accuracy rate in terms of cost functions. Moreover, it is shown that both alternative optimal thresholds by using the true rate are the identical, and this single threshold coincides with the score corresponding to Kolmogorov–Smirnov statistic used to test the homogeneous distribution functions of the defaults and non defaults, whereas the optimal threshold by using the accuracy does not the same as the score corresponding to Kolmogorov–Smirnov statistic. These facts are explored with some simulation and illustrative examples.     

Detection and Localization in Test Accuracy: A Bayesian Perspective

In assessing the area under the ROC curve for the accuracy of a diagnostic test, it is imperative to detect and locate multiple abnormalities per image. This approach takes that into account by adopting a statistical model that allows for correlation between the reader scores of several regions of interest (ROI).

The ROI method of partitioning the image is taken. The readers give a score to each ROI in the image and the statistical model takes into account the correlation between the scores of the ROI's of an image in estimating test accuracy. The test accuracy is given by Pr[Y > Z] + (1/2)Pr[Y = Z], where Y is an ordinal diagnostic measurement of an affected ROI, and Z is the diagnostic measurement of an unaffected ROI. This way of measuring test accuracy is equivalent to the area under the ROC curve. The parameters are the parameters of a multinomial distribution, then based on the multinomial distribution, a Bayesian method of inference is adopted for estimating the test accuracy.

Using a multinomial model for the test results, a Bayesian method based on the predictive distribution of future diagnostic scores is employed to find the test accuracy. By resampling from the posterior distribution of the model parameters, samples from the posterior distribution of test accuracy are also generated. Using these samples, the posterior mean, standard deviation, and credible intervals are calculated in order to estimate the area under the ROC curve. This approach is illustrated by estimating the area under the ROC curve for a study of the diagnostic accuracy of magnetic resonance angiography for diagnosis of arterial atherosclerotic stenosis. A generalization to multiple readers and/or modalities is proposed.

A Bayesian way to estimate test accuracy is easy to perform with standard software packages and has the advantage of employing the efficient inclusion of information from prior related imaging studies.     

A test for crossing receiver operating characteristic (roc) curves

The receiver operating characteristic (ROC) curve gives a graphical representation of sensitivity and specificity of a prediction model when varying the decision treshold on a diagnostic criterion. A classical test for comparing the overall accuracies for two models -1 and 2- is based on the difference between ROC curves areas - related to its standard error. This test is designed for the situation where ROC curve 1 caps ROC curve 2. Often both curves cross :in this paper, a new test, based on the integrated difference between the curves, is proposed to deal with this situation. In a simulation experiment, the new test was less powerful than the old test for detecting an overall superiority, but much more powerfull against the crossing alternative.     

Consistency of non parametric estimators of the area under the ROC curve

ABSTRACT

The area under the receiver operating characteristic (ROC) curve is a popular summary index that measures the accuracy of a continuous-scale diagnostic test to measure its accuracy. Under certain conditions on estimators of distribution functions, we prove a theorem on strong consistency of the non parametric “plugin” estimators of the area under the ROC curve. Based on this theorem, we construct some new “plugin” consistent estimators. The performance of the non parametric estimators considered is illustrated numerically and the estimators are compared in terms of bias, variance, and mean square error.     

An exact test for comparing two predictive values in small‐size clinical trials

Positive and negative predictive values describe the performance of a diagnostic test. There are several methods to test the equality of predictive values in paired designs. However, these methods were premised on large sample theory, and they may not be suitable for small‐size clinical trials because of inflation of the type 1 error rate. In this study, we propose an exact test to control the type 1 error rate strictly for conducting a small‐size clinical trial that investigates the equality of predictive values in paired designs. In addition, we execute simulation studies to evaluate the performance of the proposed exact test and existing methods in small‐size clinical trials. The proposed test can calculate the exact P value, and as a result of simulations, the empirical type 1 error rate for the proposed test did not exceed the significance level regardless of the setting, and the empirical power for the proposed test is not much different from the other methods based on large‐sample theory. Therefore, it is considered that the proposed exact test is useful when the type 1 error rate needs to be controlled strictly.     

ROC curve estimation based on local smoothing

ROC curve is a graphical representation of the relationship between sensitivity and specificity of a diagnostic test. It is a popular tool for evaluating and comparing different diagnostic tests in medical sciences. In the literature,the ROC curve is often estimated empirically based on an empirical distribution function estimator and an empirical quantile function estimator. In this paper an alternative nonparametric procedure to estimate the ROC Curve is suggested which is based on local smoothing techniques. Several numerical examples are presented to evaluate the performance of this procedure.     

Empirical Likelihood-Based Confidence Intervals for the Sensitivity of a Continuous-Scale Diagnostic Test with Missing Data

In a continuous-scale diagnostic test, the receiver operating characteristic (ROC) curve is useful to evaluate the range of the sensitivity at the cut-off point that yields a desired specificity. Many current studies on inference of the ROC curve focus on the complete data case. In this paper, an imputation-based profile empirical likelihood ratio for the sensitivity, which is free of bandwidth selection, is defined and shown to follow an asymptotically scaled Chi-square distribution. Two new confidence intervals are proposed for the sensitivity with missing data. Simulation studies are conducted to evaluate the finite sample performance of the proposed intervals in terms of coverage probability. A real example is used to illustrate the new methods.