Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface

In many situations the diagnostic decision is not limited to a binary choice. Binary statistical tools such as receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) need to be expanded to address three-category classification problem. Previous authors have suggest various ways to model the extension of AUC but not the ROC surface. Only simple parametric approaches are proposed for modeling the ROC measure under the assumption that test results all follow normal distributions. We study the estimation methods of three-dimensional ROC surfaces with nonparametric and semiparametric estimators. Asymptotical results are provided as a basis for statistical inference. Simulation studies are performed to assess the validity of our proposed methods in finite samples. We consider an Alzheimer's disease example from a clinical study in the US as an illustration. The nonparametric and semiparametric modelling approaches for the three way ROC analysis can be readily generalized to diagnostic problems with more than three classes.     

An alternative method for global and partial comparison of two diagnostic systems based on ROC curves

In this paper, an alternative method for the comparison of two diagnostic systems based on receiver operating characteristic (ROC) curves is presented. ROC curve analysis is often used as a statistical tool for the evaluation of diagnostic systems. However, in general, the comparison of ROC curves is not straightforward, in particular, when they cross each other. A similar difficulty is also observed in the multi-objective optimization field where sets of solutions defining fronts must be compared with a multi-dimensional space. Thus, the proposed methodology is based on a procedure used to compare the performance of distinct multi-objective optimization algorithms. In general, methods based on the area under the ROC curves are not sensitive to the existence of crossing points between the curves. The new approach can deal with this situation and also allows the comparison of partial portions of ROC curves according to particular values of sensitivity and specificity of practical interest. Simulations results are presented. For illustration purposes, considering real data from newborns with very low birthweight, the new method was applied in order to discriminate the better index for evaluating the risk of death.     

Compare diagnostic tests using transformation-invariant smoothed ROC curves()

Receiver operating characteristic (ROC) curve, plotting true positive rates against false positive rates as threshold varies, is an important tool for evaluating biomarkers in diagnostic medicine studies. By definition, ROC curve is monotone increasing from 0 to 1 and is invariant to any monotone transformation of test results. And it is often a curve with certain level of smoothness when test results from the diseased and non-diseased subjects follow continuous distributions. Most existing ROC curve estimation methods do not guarantee all of these properties. One of the exceptions is Du and Tang (2009) which applies certain monotone spline regression procedure to empirical ROC estimates. However, their method does not consider the inherent correlations between empirical ROC estimates. This makes the derivation of the asymptotic properties very difficult. In this paper we propose a penalized weighted least square estimation method, which incorporates the covariance between empirical ROC estimates as a weight matrix. The resulting estimator satisfies all the aforementioned properties, and we show that it is also consistent. Then a resampling approach is used to extend our method for comparisons of two or more diagnostic tests. Our simulations show a significantly improved performance over the existing method, especially for steep ROC curves. We then apply the proposed method to a cancer diagnostic study that compares several newly developed diagnostic biomarkers to a traditional one.     

Semiparametric ROC surface estimation for continuous diagnostic tests via polytomous logistic regression procedures

In this paper, we propose a semiparametric method of estimating receiver operating characteristic (ROC) surfaces for continuous diagnostic tests under density ratio models. Implementation of our method is easy since the usual polytomous logistic regression procedures in many statistical software packages can be employed. A simulated example is provided to facilitate the implementation of our method. Simulation results show that the proposed semiparametric ROC surface estimator is more efficient than the nonparametric counterpart and the parametric counterpart whether the normality assumption of data holds or not. Moreover, some simulation results on the underlying semiparametric distribution function estimators are also reported. In addition, some discussions on the proposed method as well as analysis of a real data set are provided.     

Three-group ROC predictive analysis for ordinal outcomes

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning, and credit scoring. The receiver operating characteristic (ROC) surface is a useful tool to assess the ability of a diagnostic test to discriminate among three-ordered classes or groups. In this article, nonparametric predictive inference (NPI) for three-group ROC analysis for ordinal outcomes is presented. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. This article also includes results on the volumes under the ROC surfaces and consideration of the choice of decision thresholds for the diagnosis. Two examples are provided to illustrate our method.     

Semi-empirical likelihood inference for the ROC curve with missing data

The receiver operating characteristic (ROC) curve is one of the most commonly used methods to compare the diagnostic performance of two or more laboratory or diagnostic tests. In this paper, we propose semi-empirical likelihood based confidence intervals for ROC curves of two populations, where one population is parametric and the other one is non-parametric and both have missing data. After imputing missing values, we derive the semi-empirical likelihood ratio statistic and the corresponding likelihood equations. It is shown that the log-semi-empirical likelihood ratio statistic is asymptotically scaled chi-squared. The estimating equations are solved simultaneously to obtain the estimated lower and upper bounds of semi-empirical likelihood confidence intervals. We conduct extensive simulation studies to evaluate the finite sample performance of the proposed empirical likelihood confidence intervals with various sample sizes and different missing probabilities.     

Nonparametric Predictive Inference for diagnostic accuracy

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine and health care. Good methods for determining diagnostic accuracy provide useful guidance on selection of patient treatment, and the ability to compare different diagnostic tests has a direct impact on quality of care. In this paper Nonparametric Predictive Inference (NPI) methods for accuracy of diagnostic tests with continuous test results are presented and discussed. For such tests, Receiver Operating Characteristic (ROC) curves have become popular tools for describing the performance of diagnostic tests. We present the NPI approach to ROC curves, and some important summaries of these curves. As NPI does not aim at inference for an entire population but instead explicitly considers a future observation, this provides an attractive alternative to standard methods. We show how NPI can be used to compare two continuous diagnostic tests.     

Between a ROC and a hard place: Teaching prevalence plots to understand real world biomarker performance in the clinic

The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) of the ROC curve are widely used in discovery to compare the performance of diagnostic and prognostic assays. The ROC curve has the advantage that it is independent of disease prevalence. However, in this note, we remind scientists and clinicians that the performance of an assay upon translation to the clinic is critically dependent upon that very same prevalence. Without an understanding of prevalence in the test population, even robust bioassays with excellent ROC characteristics may perform poorly in the clinic. While the exact prevalence in the target population is not always known, simple plots of candidate assay performance as a function of prevalence rate give a better understanding of the likely real‐world performance and a greater understanding of the likely impact of variation in that prevalence on translation to the clinic.     

Combining ROC curves using MAMSE weighted distributions

The receiver operating characteristic (ROC) curve can be used to evaluate the properties of a diagnostic test from the distribution of a variable on the healthy and diseased populations. The minimum averaged mean squared error (MAMSE) weights were developed to handle data from different sources by adjusting the relative contribution of each data source. The authors use the MAMSE weights to infer the ROC curve of a diagnostic test based on raw data from multiple studies. The proposed estimates are consistent and Monte Carlo simulations show favourable finite sample performance. The method is illustrated in a case study where progesterone level is used to detect ectopic pregnancies and abortions from other natural causes. The Canadian Journal of Statistics 46: 298–315; 2018 © 2018 Statistical Society of Canada     

ROC curve estimation based on local smoothing

ROC curve is a graphical representation of the relationship between sensitivity and specificity of a diagnostic test. It is a popular tool for evaluating and comparing different diagnostic tests in medical sciences. In the literature,the ROC curve is often estimated empirically based on an empirical distribution function estimator and an empirical quantile function estimator. In this paper an alternative nonparametric procedure to estimate the ROC Curve is suggested which is based on local smoothing techniques. Several numerical examples are presented to evaluate the performance of this procedure.     

Detection and Localization in Test Accuracy: A Bayesian Perspective

In assessing the area under the ROC curve for the accuracy of a diagnostic test, it is imperative to detect and locate multiple abnormalities per image. This approach takes that into account by adopting a statistical model that allows for correlation between the reader scores of several regions of interest (ROI).

The ROI method of partitioning the image is taken. The readers give a score to each ROI in the image and the statistical model takes into account the correlation between the scores of the ROI's of an image in estimating test accuracy. The test accuracy is given by Pr[Y > Z] + (1/2)Pr[Y = Z], where Y is an ordinal diagnostic measurement of an affected ROI, and Z is the diagnostic measurement of an unaffected ROI. This way of measuring test accuracy is equivalent to the area under the ROC curve. The parameters are the parameters of a multinomial distribution, then based on the multinomial distribution, a Bayesian method of inference is adopted for estimating the test accuracy.

Using a multinomial model for the test results, a Bayesian method based on the predictive distribution of future diagnostic scores is employed to find the test accuracy. By resampling from the posterior distribution of the model parameters, samples from the posterior distribution of test accuracy are also generated. Using these samples, the posterior mean, standard deviation, and credible intervals are calculated in order to estimate the area under the ROC curve. This approach is illustrated by estimating the area under the ROC curve for a study of the diagnostic accuracy of magnetic resonance angiography for diagnosis of arterial atherosclerotic stenosis. A generalization to multiple readers and/or modalities is proposed.

A Bayesian way to estimate test accuracy is easy to perform with standard software packages and has the advantage of employing the efficient inclusion of information from prior related imaging studies.     

Sensitivity to imputation models and assumptions in receiver operating characteristic analysis with incomplete data

Modern statistical methods using incomplete data have been increasingly applied in a wide variety of substantive problems. Similarly, receiver operating characteristic (ROC) analysis, a method used in evaluating diagnostic tests or biomarkers in medical research, has also been increasingly popular problem in both its development and application. While missing-data methods have been applied in ROC analysis, the impact of model mis-specification and/or assumptions (e.g. missing at random) underlying the missing data has not been thoroughly studied. In this work, we study the performance of multiple imputation (MI) inference in ROC analysis. Particularly, we investigate parametric and non-parametric techniques for MI inference under common missingness mechanisms. Depending on the coherency of the imputation model with the underlying data generation mechanism, our results show that MI generally leads to well-calibrated inferences under ignorable missingness mechanisms.     

Adjusting ROC curves for covariates in the presence of verification bias

The ROC (receiver operating characteristic) curve is frequently used for describing effectiveness of a diagnostic marker or test. Classical estimation of the ROC curve uses independent identically distributed samples taken randomly from the healthy and diseased populations. Frequently not all subjects undergo a definitive gold standard assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased (verification bias). In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve adjusted for covariates (ROC regression) under verification bias. We develop the estimator's asymptotic distribution and examine its finite sample size properties via a simulation study. We apply this procedure to fingerstick postprandial blood glucose measurement data adjusting for age.     

COMBINING MULTIPLE MARKERS FOR MULTI‐CATEGORY CLASSIFICATION: AN ROC SURFACE APPROACH

The Receiver Operating Characteristic (ROC) curve and the Area Under the ROC Curve (AUC) are effective statistical tools for evaluating the accuracy of diagnostic tests for binary‐class medical data. However, many real‐world biomedical problems involve more than two categories. The Volume Under the ROC Surface (VUS) and Hypervolume Under the ROC Manifold (HUM) measures are extensions for the AUC under three‐class and multiple‐class models. Inference methods for such measures have been proposed recently. We develop a method of constructing a linear combination of markers for which the VUS or HUM of the combined markers is maximized. Asymptotic validity of the estimator is justified by extending the results for maximum rank correlation estimation that are well known in econometrics. A bootstrap resampling method is then applied to estimate the sampling variability. Simulations and examples are provided to demonstrate our methods.     

Empirical Likelihood-Based Confidence Intervals for the Sensitivity of a Continuous-Scale Diagnostic Test with Missing Data

In a continuous-scale diagnostic test, the receiver operating characteristic (ROC) curve is useful to evaluate the range of the sensitivity at the cut-off point that yields a desired specificity. Many current studies on inference of the ROC curve focus on the complete data case. In this paper, an imputation-based profile empirical likelihood ratio for the sensitivity, which is free of bandwidth selection, is defined and shown to follow an asymptotically scaled Chi-square distribution. Two new confidence intervals are proposed for the sensitivity with missing data. Simulation studies are conducted to evaluate the finite sample performance of the proposed intervals in terms of coverage probability. A real example is used to illustrate the new methods.     

Consistency of non parametric estimators of the area under the ROC curve

ABSTRACT

The area under the receiver operating characteristic (ROC) curve is a popular summary index that measures the accuracy of a continuous-scale diagnostic test to measure its accuracy. Under certain conditions on estimators of distribution functions, we prove a theorem on strong consistency of the non parametric “plugin” estimators of the area under the ROC curve. Based on this theorem, we construct some new “plugin” consistent estimators. The performance of the non parametric estimators considered is illustrated numerically and the estimators are compared in terms of bias, variance, and mean square error.     

A non-inferiority test for diagnostic accuracy in the absence of the golden standard test based on the paired partial areas under receiver operating characteristic curves

Non-inferiority tests are often measured for the diagnostic accuracy in medical research. The area under the receiver operating characteristic (ROC) curve is a familiar diagnostic measure for the overall diagnostic accuracy. Nevertheless, since it may not differentiate the diverse shapes of the ROC curves with different diagnostic significance, the partial area under the ROC (PAUROC) curve, another summary measure emerges for such diagnostic processes that require the false-positive rate to be in the clinically interested range. Traditionally, to estimate the PAUROC, the golden standard (GS) test on the true disease status is required. Nevertheless, the GS test may sometimes be infeasible. Besides, in a lot of research fields such as the epidemiology field, the true disease status of the patients may not be known or available. Under the normality assumption on diagnostic test results, based on the expectation-maximization algorithm in combination with the bootstrap method, we propose the heuristic method to construct a non-inferiority test for the difference in the paired PAUROCs without the GS test. Through the simulation study, although the proposed method might provide a liberal test, as a whole, the empirical size of the proposed method sufficiently controls the size at the significance level, and the empirical power of the proposed method in the absence of the GS is as good as that of the non-inferiority in the presence of the GS. The proposed method is illustrated with the published data.     

Generalized Confidence Interval Estimation for the Difference in Paired Areas Under the ROC Curves in the Absence of a Gold Standard

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may be too expensive or infeasible. In many medical researches, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, we propose a heuristic method of estimating confidence intervals for the difference in paired AUCs of two diagnostic tests in the absence of a GS reference. This heuristic method is a three-stage method by combining the expectation-maximization (EM) algorithm, bootstrap method, and an estimation based on asymptotic generalized pivotal quantities (GPQs) to construct generalized confidence intervals for the difference in paired AUCs in the absence of a GS. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and expected interval lengths. The numerical example using a published dataset illustrates the proposed method.     

Cumulative/dynamic ROC curve estimation

ABSTRACT

Receiver operating-characteristic (ROC) curve is a popular graphical method frequently used in order to study the diagnostic capacity of continuous (bio)markers. When the considered outcome is a time-dependent variable, the direct generalization is known as cumulative/dynamic ROC curve. For a fixed point of time, t, one subject is allocated into the positive group if the event happens before t and into the negative group if the event is not happened at t. The presence of censored subject, which can not be directly assigned into a group, is the main handicap of this approach. The proposed cumulative/dynamic ROC curve estimator assigns a probability to belong to the negative (positive) group to the subjects censored previously to t. The performance of the resulting estimator is studied from Monte Carlo simulations. Some real-world applications are reported. Results suggest that the new estimators provide a good approximation to the real cumulative/dynamic ROC curve.     

A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators

In this article, we analyze the three-way bootstrap estimate of the variance of the reader-averaged nonparametric area under the receiver operating characteristic (ROC) curve. The setting for this work is medical imaging, and the experimental design involves sampling from three distributions: a set of normal and diseased cases (patients), and a set of readers (doctors). The experiment we consider is fully crossed in that each reader reads each case. A reading generates a score that indicates the reader's level of suspicion that the patient is diseased. The distribution of scores for the normal patients is compared to the distribution of scores for the diseased patients via an ROC curve, and the area under the ROC curve (AUC) summarizes the reader's diagnostic ability to separate the normal patients from the diseased ones. We find that the bootstrap estimate of the variance of the reader-averaged AUC is biased, and we represent this bias in terms of moments of success outcomes. This representation helps unify and improve several current methods for multi-reader multi-case (MRMC) ROC analysis.