Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions

The receiver operating characteristic (ROC) curve is a graphical representation of the relationship between false positive and true positive rates. It is a widely used statistical tool for describing the accuracy of a diagnostic test. In this paper we propose a new nonparametric ROC curve estimator based on the smoothed empirical distribution functions. We prove its strong consistency and perform a simulation study to compare it with some other popular nonparametric estimators of the ROC curve. We also apply the proposed method to a real data set.     

A modified area under the ROC curve and its application to marker selection and classification

The area under the ROC curve (AUC) can be interpreted as the probability that the classification scores of a diseased subject is larger than that of a non-diseased subject for a randomly sampled pair of subjects. From the perspective of classification, we want to find a way to separate two groups as distinctly as possible via AUC. When the difference of the scores of a marker is small, its impact on classification is less important. Thus, a new diagnostic/classification measure based on a modified area under the ROC curve (mAUC) is proposed, which is defined as a weighted sum of two AUCs, where the AUC with the smaller difference is assigned a lower weight, and vice versa. Using mAUC is robust in the sense that mAUC gets larger as AUC gets larger as long as they are not equal. Moreover, in many diagnostic situations, only a specific range of specificity is of interest. Under normal distributions, we show that if the AUCs of two markers are within similar ranges, the larger mAUC implies the larger partial AUC for a given specificity. This property of mAUC will help to identify the marker with the higher partial AUC, even when the AUCs are similar. Two nonparametric estimates of an mAUC and their variances are given. We also suggest the use of mAUC as the objective function for classification, and the use of the gradient Lasso algorithm for classifier construction and marker selection. Application to simulation datasets and real microarray gene expression datasets show that our method finds a linear classifier with a higher ROC curve than some other existing linear classifiers, especially in the range of low false positive rates.     

Combining ROC curves using MAMSE weighted distributions

The receiver operating characteristic (ROC) curve can be used to evaluate the properties of a diagnostic test from the distribution of a variable on the healthy and diseased populations. The minimum averaged mean squared error (MAMSE) weights were developed to handle data from different sources by adjusting the relative contribution of each data source. The authors use the MAMSE weights to infer the ROC curve of a diagnostic test based on raw data from multiple studies. The proposed estimates are consistent and Monte Carlo simulations show favourable finite sample performance. The method is illustrated in a case study where progesterone level is used to detect ectopic pregnancies and abortions from other natural causes. The Canadian Journal of Statistics 46: 298–315; 2018 © 2018 Statistical Society of Canada     

Non‐parametric interval estimation for the partial area under the ROC curve

Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased conditions from non‐diseased conditions. For a continuous‐scale diagnostic test, a popular summary index of the receiver operating characteristic (ROC) curve is the area under the curve (AUC). However, when our focus is on a certain region of false positive rates, we often use the partial AUC instead. In this paper we have derived the asymptotic normal distribution for the non‐parametric estimator of the partial AUC with an explicit variance formula. The empirical likelihood (EL) ratio for the partial AUC is defined and it is shown that its limiting distribution is a scaled chi‐square distribution. Hybrid bootstrap and EL confidence intervals for the partial AUC are proposed by using the newly developed EL theory. We also conduct extensive simulation studies to compare the relative performance of the proposed intervals and existing intervals for the partial AUC. A real example is used to illustrate the application of the recommended intervals. The Canadian Journal of Statistics 39: 17–33; 2011 © 2011 Statistical Society of Canada     

Adjusting ROC curves for covariates in the presence of verification bias

The ROC (receiver operating characteristic) curve is frequently used for describing effectiveness of a diagnostic marker or test. Classical estimation of the ROC curve uses independent identically distributed samples taken randomly from the healthy and diseased populations. Frequently not all subjects undergo a definitive gold standard assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased (verification bias). In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve adjusted for covariates (ROC regression) under verification bias. We develop the estimator's asymptotic distribution and examine its finite sample size properties via a simulation study. We apply this procedure to fingerstick postprandial blood glucose measurement data adjusting for age.     

A non-inferiority test for diagnostic accuracy in the absence of the golden standard test based on the paired partial areas under receiver operating characteristic curves

Non-inferiority tests are often measured for the diagnostic accuracy in medical research. The area under the receiver operating characteristic (ROC) curve is a familiar diagnostic measure for the overall diagnostic accuracy. Nevertheless, since it may not differentiate the diverse shapes of the ROC curves with different diagnostic significance, the partial area under the ROC (PAUROC) curve, another summary measure emerges for such diagnostic processes that require the false-positive rate to be in the clinically interested range. Traditionally, to estimate the PAUROC, the golden standard (GS) test on the true disease status is required. Nevertheless, the GS test may sometimes be infeasible. Besides, in a lot of research fields such as the epidemiology field, the true disease status of the patients may not be known or available. Under the normality assumption on diagnostic test results, based on the expectation-maximization algorithm in combination with the bootstrap method, we propose the heuristic method to construct a non-inferiority test for the difference in the paired PAUROCs without the GS test. Through the simulation study, although the proposed method might provide a liberal test, as a whole, the empirical size of the proposed method sufficiently controls the size at the significance level, and the empirical power of the proposed method in the absence of the GS is as good as that of the non-inferiority in the presence of the GS. The proposed method is illustrated with the published data.     

Detecting diagnostic accuracy of two biomarkers through a bivariate log-normal ROC curve

In biomedical research, two or more biomarkers may be available for diagnosis of a particular disease. Selecting one single biomarker which ideally discriminate a diseased group from a healthy group is confront in a diagnostic process. Frequently, most of the people use the accuracy measure, area under the receiver operating characteristic (ROC) curve to choose the best diagnostic marker among the available markers for diagnosis. Some authors have tried to combine the multiple markers by an optimal linear combination to increase the discriminatory power. In this paper, we propose an alternative method that combines two continuous biomarkers by direct bivariate modeling of the ROC curve under log-normality assumption. The proposed method is applied to simulated data set and prostate cancer diagnostic biomarker data set.     

Local linear smoothing of receiver operating characteristic (ROC) curves

For measuring the accuracy of a continuous diagnostic test, the receiver operating characteristic (ROC) curve is often used. The empirical ROC curve is the most commonly used non-parametric estimator for the ROC curve. Recently, Lloyd (J. Amer. Statist. Assoc. 93(1998) 1356) proposed a kernel smoothing estimator for the ROC curve and showed his estimator has better mean square error than the empirical ROC curve estimator. However, Lloyd's estimator involves two bandwidths and has a boundary problem. In addition, his choice of bandwidths is ad hoc. In this paper we propose another kernel smoothing estimator which involves only one bandwidth and does not have the boundary problem. Furthermore, our choice of the bandwidth is asymptotically optimal.     

ROC curve estimation based on local smoothing

ROC curve is a graphical representation of the relationship between sensitivity and specificity of a diagnostic test. It is a popular tool for evaluating and comparing different diagnostic tests in medical sciences. In the literature,the ROC curve is often estimated empirically based on an empirical distribution function estimator and an empirical quantile function estimator. In this paper an alternative nonparametric procedure to estimate the ROC Curve is suggested which is based on local smoothing techniques. Several numerical examples are presented to evaluate the performance of this procedure.     

ROC Curves in Non‐Parametric Location‐Scale Regression Models

Abstract. The receiver operating characteristic (ROC) curve is a tool of extensive use to analyse the discrimination capability of a diagnostic variable in medical studies. In certain situations, the presence of a covariate related to the diagnostic variable can increase the discriminating power of the ROC curve. In this article, we model the effect of the covariate over the diagnostic variable by means of non‐parametric location‐scale regression models. We propose a new non‐parametric estimator of the conditional ROC curve and study its asymptotic properties. We also present some simulations and an illustration to a data set concerning diagnosis of diabetes.     

Applications of the Bootstrap in ROC Analysis

The problem of estimating standard errors for diagnostic accuracy measures might be challenging for many complicated models. We can address such a problem by using the Bootstrap methods to blunt its technical edge with resampled empirical distributions. We consider two cases where bootstrap methods can successfully improve our knowledge of the sampling variability of the diagnostic accuracy estimators. The first application is to make inference for the area under the ROC curve resulted from a functional logistic regression model which is a sophisticated modelling device to describe the relationship between a dichotomous response and multiple covariates. We consider using this regression method to model the predictive effects of multiple independent variables on the occurrence of a disease. The accuracy measures, such as the area under the ROC curve (AUC) are developed from the functional regression. Asymptotical results for the empirical estimators are provided to facilitate inferences. The second application is to test the difference of two weighted areas under the ROC curve (WAUC) from a paired two sample study. The correlation between the two WAUC complicates the asymptotic distribution of the test statistic. We then employ the bootstrap methods to gain satisfactory inference results. Simulations and examples are supplied in this article to confirm the merits of the bootstrap methods.     

Fitting Roc Curves Using Non-linear Binomial Regression

The performance of a diagnostic test is summarized by its receiver operating characteristic (ROC) curve. Empirical data on a test's performance often come in the form of observed true positive and false positive relative frequencies, under varying conditions. This paper describes a family of models for analysing such data. The underlying ROC curves are specified by a shift parameter, a shape parameter and a link function. Both the position along the ROC curve and the shift parameter are modelled linearly. The shape parameter enters the model non-linearly but in a very simple manner. One simple application is to the meta-analysis of independent studies of the same diagnostic test, illustrated on some data of Moses, Shapiro & Littenberg (1993). A second application to so-called vigilance data is given, where ROC curves differ across subjects, and modelling of the position along the ROC curve is of primary interest.     

Selection and combination of biomarkers using ROC method for disease classification and prediction

Based on the SCAD penalty and the area under the ROC curve (AUC), we propose a new method for selecting and combining biomarkers for disease classification and prediction. The proposed estimator for the combination of the biomarkers has an oracle property; that is, the estimated combination of the biomarkers performs as well as it would have been if the biomarkers significantly associated with the outcome had been known in advance, in terms of discriminative power. The proposed estimator is computationally feasible, n^1/2‐consistent and asymptotically normal. Simulation studies show that the proposed method performs better than existing methods. We illustrate the proposed methodology in the acoustic startle response study. The Canadian Journal of Statistics 39: 324–343; 2011 © 2011 Statistical Society of Canada     

Two transformation models for estimating an ROC curve derived from continuous data

A receiver operating characteristic (ROC) curve is a plot of two survival functions, derived separately from the diseased and healthy samples. A special feature is that the ROC curve is invariant to any monotone transformation of the measurement scale. We propose and analyse semiparametric and parametric transformation models for this two-sample problem. Following an unspecified or specified monotone transformation, we assume that the healthy and diseased measurements have two normal distributions with different means and variances. Maximum likelihood algorithms for estimating ROC curve parameters are developed. The proposed methods are illustrated on the marker CA125 in the diagnosis of gastric cancer.     

Consistency of non parametric estimators of the area under the ROC curve

ABSTRACT

The area under the receiver operating characteristic (ROC) curve is a popular summary index that measures the accuracy of a continuous-scale diagnostic test to measure its accuracy. Under certain conditions on estimators of distribution functions, we prove a theorem on strong consistency of the non parametric “plugin” estimators of the area under the ROC curve. Based on this theorem, we construct some new “plugin” consistent estimators. The performance of the non parametric estimators considered is illustrated numerically and the estimators are compared in terms of bias, variance, and mean square error.     

Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach

Summary.  The false discovery rate (FDR) is a multiple hypothesis testing quantity that describes the expected proportion of false positive results among all rejected null hypotheses. Benjamini and Hochberg introduced this quantity and proved that a particular step-up p -value method controls the FDR. Storey introduced a point estimate of the FDR for fixed significance regions. The former approach conservatively controls the FDR at a fixed predetermined level, and the latter provides a conservatively biased estimate of the FDR for a fixed predetermined significance region. In this work, we show in both finite sample and asymptotic settings that the goals of the two approaches are essentially equivalent. In particular, the FDR point estimates can be used to define valid FDR controlling procedures. In the asymptotic setting, we also show that the point estimates can be used to estimate the FDR conservatively over all significance regions simultaneously, which is equivalent to controlling the FDR at all levels simultaneously. The main tool that we use is to translate existing FDR methods into procedures involving empirical processes. This simplifies finite sample proofs, provides a framework for asymptotic results and proves that these procedures are valid even under certain forms of dependence.     

A Flexible Method for Estimating the ROC Curve

In this paper we propose a flexible method for estimating a receiver operating characteristic (ROC) curve that is based on a continuous-scale test. The approach is easily understood and efficiently computed, and robust to the smooth parameter selection, which needs intensive computation when using local polynomial and smoothing spline techniques. The results from our simulation experiment indicate that the moderate-sample numerical performance of our estimator is better than the empirical ROC curve estimator and comparable to the local linear estimator. The availability of easy implementation is also illustrated by our simulation. We apply the proposed method to two real data sets.     

Generalized Confidence Interval Estimation for the Difference in Paired Areas Under the ROC Curves in the Absence of a Gold Standard

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may be too expensive or infeasible. In many medical researches, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, we propose a heuristic method of estimating confidence intervals for the difference in paired AUCs of two diagnostic tests in the absence of a GS reference. This heuristic method is a three-stage method by combining the expectation-maximization (EM) algorithm, bootstrap method, and an estimation based on asymptotic generalized pivotal quantities (GPQs) to construct generalized confidence intervals for the difference in paired AUCs in the absence of a GS. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and expected interval lengths. The numerical example using a published dataset illustrates the proposed method.     

Semi-parametric estimation of ROC curves based on binomial regression modelling

This paper presents a method of estimating a receiver operating characteristic (ROC) curve when the underlying diagnostic variable X is continuous and fully observed. The new method is based on modelling the probability of response given X , rather than the distribution of X given response. The method offers advantages in modelling flexibility and computational simplicity. The resulting ROC curve estimates are semi-parametric and can, in principle, take an infinite variety of shapes. Moreover, model selection can be based on standard methods within the binomial regression framework. Statistical accuracy of the curve estimate is provided by a simply implemented bootstrap approach.