Consistency of non parametric estimators of the area under the ROC curve

ABSTRACT

The area under the receiver operating characteristic (ROC) curve is a popular summary index that measures the accuracy of a continuous-scale diagnostic test to measure its accuracy. Under certain conditions on estimators of distribution functions, we prove a theorem on strong consistency of the non parametric “plugin” estimators of the area under the ROC curve. Based on this theorem, we construct some new “plugin” consistent estimators. The performance of the non parametric estimators considered is illustrated numerically and the estimators are compared in terms of bias, variance, and mean square error.     

Applications of the Bootstrap in ROC Analysis

The problem of estimating standard errors for diagnostic accuracy measures might be challenging for many complicated models. We can address such a problem by using the Bootstrap methods to blunt its technical edge with resampled empirical distributions. We consider two cases where bootstrap methods can successfully improve our knowledge of the sampling variability of the diagnostic accuracy estimators. The first application is to make inference for the area under the ROC curve resulted from a functional logistic regression model which is a sophisticated modelling device to describe the relationship between a dichotomous response and multiple covariates. We consider using this regression method to model the predictive effects of multiple independent variables on the occurrence of a disease. The accuracy measures, such as the area under the ROC curve (AUC) are developed from the functional regression. Asymptotical results for the empirical estimators are provided to facilitate inferences. The second application is to test the difference of two weighted areas under the ROC curve (WAUC) from a paired two sample study. The correlation between the two WAUC complicates the asymptotic distribution of the test statistic. We then employ the bootstrap methods to gain satisfactory inference results. Simulations and examples are supplied in this article to confirm the merits of the bootstrap methods.     

Exact Bootstrap Variances of the Area Under ROC Curve

The area under the Receiver Operating Characteristic (ROC) curve (AUC) and related summary indices are widely used for assessment of accuracy of an individual and comparison of performances of several diagnostic systems in many areas including studies of human perception, decision making, and the regulatory approval process for new diagnostic technologies. Many investigators have suggested implementing the bootstrap approach to estimate variability of AUC-based indices. Corresponding bootstrap quantities are typically estimated by sampling a bootstrap distribution. Such a process, frequently termed Monte Carlo bootstrap, is often computationally burdensome and imposes an additional sampling error on the resulting estimates. In this article, we demonstrate that the exact or ideal (sampling error free) bootstrap variances of the nonparametric estimator of AUC can be computed directly, i.e., avoiding resampling of the original data, and we develop easy-to-use formulas to compute them. We derive the formulas for the variances of the AUC corresponding to a single given or random reader, and to the average over several given or randomly selected readers. The derived formulas provide an algorithm for computing the ideal bootstrap variances exactly and hence improve many bootstrap methods proposed earlier for analyzing AUCs by eliminating the sampling error and sometimes burdensome computations associated with a Monte Carlo (MC) approximation. In addition, the availability of closed-form solutions provides the potential for an analytical assessment of the properties of bootstrap variance estimators. Applications of the proposed method are shown on two experimentally ascertained datasets that illustrate settings commonly encountered in diagnostic imaging. In the context of the two examples we also demonstrate the magnitude of the effect of the sampling error of the MC estimators on the resulting inferences.     

Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface

In many situations the diagnostic decision is not limited to a binary choice. Binary statistical tools such as receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) need to be expanded to address three-category classification problem. Previous authors have suggest various ways to model the extension of AUC but not the ROC surface. Only simple parametric approaches are proposed for modeling the ROC measure under the assumption that test results all follow normal distributions. We study the estimation methods of three-dimensional ROC surfaces with nonparametric and semiparametric estimators. Asymptotical results are provided as a basis for statistical inference. Simulation studies are performed to assess the validity of our proposed methods in finite samples. We consider an Alzheimer's disease example from a clinical study in the US as an illustration. The nonparametric and semiparametric modelling approaches for the three way ROC analysis can be readily generalized to diagnostic problems with more than three classes.     

Small sample comparison of six bivariate survival curve estimators 1

We study the performance of six proposed bivariate survival curve estimators on simulated right censored data. The performance of the estimators is compared for data generated by three bivariate models with exponential marginal distributions. The estimators are compared in their ability to estimate correlations and survival functions probabilities. Simulated data results are presented so that the proposed estimators in this relatively new area of analysis can be explicitly compared to the known distribution of the data and the parameters of the underlying model. The results show clear differences in the performance of the estimators.     

Nonparametric methodology for the time-dependent partial area under the ROC curve

To assess the classification accuracy of a continuous diagnostic result, the receiver operating characteristic (ROC) curve is commonly used in applications. The partial area under the ROC curve (pAUC) is one of the widely accepted summary measures due to its generality and ease of probability interpretation. In the field of life science, a direct extension of the pAUC into the time-to-event setting can be used to measure the usefulness of a biomarker for disease detection over time. Without using a trapezoidal rule, we propose nonparametric estimators, which are easily computed and have closed-form expressions, for the time-dependent pAUC. The asymptotic Gaussian processes of the estimators are established and the estimated variance-covariance functions are provided, which are essential in the construction of confidence intervals. The finite sample performance of the proposed inference procedures are investigated through a series of simulations. Our method is further applied to evaluate the classification ability of CD4 cell counts on patient's survival time in the AIDS Clinical Trials Group (ACTG) 175 study. In addition, the inferences can be generalized to compare the time-dependent pAUCs between patients received the prior antiretroviral therapy and those without it.     

The average area under correlated receiver operating characteristic curves: a nonparametric approach based on generalized two-sample Wilcoxon statistics

It is well known that, when sample observations are independent, the area under the receiver operating characteristic (ROC) curve corresponds to the Wilcoxon statistics if the area is calculated by the trapezoidal rule. Correlated ROC curves arise often in medical research and have been studied by various parametric methods. On the basis of the Mann–Whitney U-statistics for clustered data proposed by Rosner and Grove, we construct an average ROC curve and derive nonparametric methods to estimate the area under the average curve for correlated ROC curves obtained from multiple readers. For the more complicated case where, in addition to multiple readers examining results on the same set of individuals, two or more diagnostic tests are involved, we derive analytic methods to compare the areas under correlated average ROC curves for these diagnostic tests. We demonstrate our methods in an example and compare our results with those obtained by other methods. The nonparametric average ROC curve and the analytic methods that we propose are easy to explain and simple to implement.     

Non‐parametric estimation for time-dependent AUC

The area under the receiver operating characteristic (ROC) curve (AUC) is one of the commonly used measure to evaluate or compare the predictive ability of markers to the disease status. Motivated by an angiographic coronary artery disease (CAD) study, our objective is mainly to evaluate and compare the performance of several baseline plasma levels in the prediction of CAD-related vital status over time. Based on censored survival data, the non-parametric estimators are proposed for the time-dependent AUC. The limiting Gaussian processes of the estimators and the estimated asymptotic variance–covariance functions enable us to further construct confidence bands and develop testing procedures. Applications and finite sample properties of the proposed estimation methods and inference procedures are demonstrated through the CAD-related death data from the British Columbia Vital Statistics Agency and Monte Carlo simulations.     

Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions

The receiver operating characteristic (ROC) curve is a graphical representation of the relationship between false positive and true positive rates. It is a widely used statistical tool for describing the accuracy of a diagnostic test. In this paper we propose a new nonparametric ROC curve estimator based on the smoothed empirical distribution functions. We prove its strong consistency and perform a simulation study to compare it with some other popular nonparametric estimators of the ROC curve. We also apply the proposed method to a real data set.     

Nonparametric covariate adjustment for receiver operating characteristic curves

The accuracy of a diagnostic test is typically characterized using the receiver operating characteristic (ROC) curve. Summarizing indexes such as the area under the ROC curve (AUC) are used to compare different tests as well as to measure the difference between two populations. Often additional information is available on some of the covariates which are known to influence the accuracy of such measures. The authors propose nonparametric methods for covariate adjustment of the AUC. Models with normal errors and possibly non‐normal errors are discussed and analyzed separately. Nonparametric regression is used for estimating mean and variance functions in both scenarios. In the model that relaxes the assumption of normality, the authors propose a covariate‐adjusted Mann–Whitney estimator for AUC estimation which effectively uses available data to construct working samples at any covariate value of interest and is computationally efficient for implementation. This provides a generalization of the Mann–Whitney approach for comparing two populations by taking covariate effects into account. The authors derive asymptotic properties for the AUC estimators in both settings, including asymptotic normality, optimal strong uniform convergence rates and mean squared error (MSE) consistency. The MSE of the AUC estimators was also assessed in smaller samples by simulation. Data from an agricultural study were used to illustrate the methods of analysis. The Canadian Journal of Statistics 38:27–46; 2010 © 2009 Statistical Society of Canada     

Assessing accuracy of a continuous screening test in the presence of verification bias

Summary.  In studies to assess the accuracy of a screening test, often definitive disease assessment is too invasive or expensive to be ascertained on all the study subjects. Although it may be more ethical or cost effective to ascertain the true disease status with a higher rate in study subjects where the screening test or additional information is suggestive of disease, estimates of accuracy can be biased in a study with such a design. This bias is known as verification bias. Verification bias correction methods that accommodate screening tests with binary or ordinal responses have been developed; however, no verification bias correction methods exist for tests with continuous results. We propose and compare imputation and reweighting bias-corrected estimators of true and false positive rates, receiver operating characteristic curves and area under the receiver operating characteristic curve for continuous tests. Distribution theory and simulation studies are used to compare the proposed estimators with respect to bias, relative efficiency and robustness to model misspecification. The bias correction estimators proposed are applied to data from a study of screening tests for neonatal hearing loss.     

Simex approaches to measurement error in roc studies

This paper explores the estimation of the area under the ROC curve when test scores are subject to errors. The naive approach that ignores measurement errors generally yields inconsistent estimates. Finding the asymptotic bias of the naive estimator, Coffin and Sukhatme (1995, 1997) proposed bias-corrected estimators for parametric and nonparametric cases. However, the asymptotic distributions of these estimators have not been developed because of their complexity. We propose several alternative approaches, including the SIMEX procedure of Cook and Stefanski (1994). We also provide the asymptotic distributions of the SIMEX estimators for use in statistical inference. Small simulation studies illustrate that the SIMEX estimators perform reasonably well when compared to the bias-corrected estimators.     

Exact confidence interval estimation for the difference in diagnostic accuracy with three ordinal diagnostic groups

In the cases with three ordinal diagnostic groups, the important measures of diagnostic accuracy are the volume under surface (VUS) and the partial volume under surface (PVUS) which are the extended forms of the area under curve (AUC) and the partial area under curve (PAUC). This article addresses confidence interval estimation of the difference in paired VUS s and the difference in paired PVUS s. To focus especially on studies with small to moderate sample sizes, we propose an approach based on the concepts of generalized inference. A Monte Carlo study demonstrates that the proposed approach generally can provide confidence intervals with reasonable coverage probabilities even at small sample sizes. The proposed approach is compared to a parametric bootstrap approach and a large sample approach through simulation. Finally, the proposed approach is illustrated via an application to a data set of blood test results of anemia patients.     

Marshall–Olkin Extended Lomax Distribution and Its Application to Censored Data

The problem of making statistical inference about θ =P(X > Y) has been under great investigation in the literature using simple random sampling (SRS) data. This problem arises naturally in the area of reliability for a system with strength X and stress Y. In this study, we will consider making statistical inference about θ using ranked set sampling (RSS) data. Several estimators are proposed to estimate θ using RSS. The properties of these estimators are investigated and compared with known estimators based on simple random sample (SRS) data. The proposed estimators based on RSS dominate those based on SRS. A motivated example using real data set is given to illustrate the computation of the newly suggested estimators.     

Detecting diagnostic accuracy of two biomarkers through a bivariate log-normal ROC curve

In biomedical research, two or more biomarkers may be available for diagnosis of a particular disease. Selecting one single biomarker which ideally discriminate a diseased group from a healthy group is confront in a diagnostic process. Frequently, most of the people use the accuracy measure, area under the receiver operating characteristic (ROC) curve to choose the best diagnostic marker among the available markers for diagnosis. Some authors have tried to combine the multiple markers by an optimal linear combination to increase the discriminatory power. In this paper, we propose an alternative method that combines two continuous biomarkers by direct bivariate modeling of the ROC curve under log-normality assumption. The proposed method is applied to simulated data set and prostate cancer diagnostic biomarker data set.     

Robust small area estimation

Small area estimation has received considerable attention in recent years because of growing demand for small area statistics. Basic area‐level and unit‐level models have been studied in the literature to obtain empirical best linear unbiased prediction (EBLUP) estimators of small area means. Although this classical method is useful for estimating the small area means efficiently under normality assumptions, it can be highly influenced by the presence of outliers in the data. In this article, the authors investigate the robustness properties of the classical estimators and propose a resistant method for small area estimation, which is useful for downweighting any influential observations in the data when estimating the model parameters. To estimate the mean squared errors of the robust estimators of small area means, a parametric bootstrap method is adopted here, which is applicable to models with block diagonal covariance structures. Simulations are carried out to study the behaviour of the proposed robust estimators in the presence of outliers, and these estimators are also compared to the EBLUP estimators. Performance of the bootstrap mean squared error estimator is also investigated in the simulation study. The proposed robust method is also applied to some real data to estimate crop areas for counties in Iowa, using farm‐interview data on crop areas and LANDSAT satellite data as auxiliary information. The Canadian Journal of Statistics 37: 381–399; 2009 © 2009 Statistical Society of Canada     

Wavelet estimation of density for censored data with censoring indicator missing at random

In this paper, we define the nonlinear wavelet estimator of density for the right censoring model with the censoring indicator missing at random (MAR), and develop its asymptotic expression for mean integrated squared error (MISE). Unlike for kernel estimator, the MISE expression of the estimator is not affected by the presence of discontinuities in the curve. Meanwhile, asymptotic normality of the estimator is established. The proposed estimator can reduce to the estimator defined by Li [Non-linear wavelet-based density estimators under random censorship. J Statist Plann Inference. 2003;117(1):35–58] when the censoring indicator MAR does not occur and a bandwidth in non-parametric estimation is close to zero. Also, we define another two nonlinear wavelet estimators of the density. A simulation is done to show the performance of the three proposed estimators.     

A general class of chain estimators for ratio and product of two means of a finite population

This paper proposes a class of estimators for estimating ratio and product of two means of a finite population using information on two auxiliary characters. Asymptotic expression to terms of order 0(n^-1) for bias and mean square error (MSE) of the proposed class of estimators are derived. Optimum conditions are obtained under which the proposed class of estimators has the minimum MSE. An empirical study is carried out to compare the performance of various estimators of ratio with the conventional estimators.     

Robust estimation of AR coefficients under simultaneously influencing outliers and missing values

A family of robust estimators for coefficients of Gaussian AR(p) time series under simultaneously influencing distortions of two types: outliers and missing values, is proposed. The estimators are based on special properties of the Cauchy probability distribution; consistency and the asymptotic normality of these estimators are proven. An approximate solution of the problem of minimization of the asymptotic variance within the proposed family of estimators is found. Performance of the proposed estimators is illustrated for simulated time series and for real data sets.     

Simultaneous confidexce intervals in an extended crow iii curve model

This paper deals with estimation problems under an extended growth curve model with two hierarchical within-individuals design matrices. The model in cludes the one whose mean structure consists of polynomial growth curves with two different degrees. First we propose certain simple estimators of the mean and covariance parameters which are closely related to the MEE's. Some basic properties of the estimators are given. Simultaneous confidence intervals are constructed, based on the estimators, for each and both of two growth curves. We give asymptotic approximations for the corresponding critical points. A numerical example is also given.