Receiver operating characteristic surfaces in the presence of verification bias

Summary.  In diagnostic medicine, the receiver operating characteristic (ROC) surface is one of the established tools for assessing the accuracy of a diagnostic test in discriminating three disease states, and the volume under the ROC surface has served as a summary index for diagnostic accuracy. In practice, the selection for definitive disease examination may be based on initial test measurements and induces verification bias in the assessment. We propose a non-parametric likelihood-based approach to construct the empirical ROC surface in the presence of differential verification, and to estimate the volume under the ROC surface. Estimators of the standard deviation are derived by both the Fisher information and the jackknife method, and their relative accuracy is evaluated in an extensive simulation study. The methodology is further extended to incorporate discrete baseline covariates in the selection process, and to compare the accuracy of a pair of diagnostic tests. We apply the proposed method to compare the diagnostic accuracy between mini-mental state examination and clinical evaluation of dementia, in discriminating between three disease states of Alzheimer's disease.     

Consistency of non parametric estimators of the area under the ROC curve

ABSTRACT

The area under the receiver operating characteristic (ROC) curve is a popular summary index that measures the accuracy of a continuous-scale diagnostic test to measure its accuracy. Under certain conditions on estimators of distribution functions, we prove a theorem on strong consistency of the non parametric “plugin” estimators of the area under the ROC curve. Based on this theorem, we construct some new “plugin” consistent estimators. The performance of the non parametric estimators considered is illustrated numerically and the estimators are compared in terms of bias, variance, and mean square error.     

Comparing accuracies of two screening tests in a two-phase study for dementia

A two-phase design has been widely used in epidemiological studies of dementia. The first phase assesses a large sample with screening tests. The second, based on the screening test results and possibly on other observed patient's factors, selects a subset of the study sample for a more definitive disease verification assessment. In comparing the accuracies of two screening tests in a two-phase study of dementia, inferences are commonly made from a sample of verified cases. The omission of non-verified cases can seriously bias comparison results. To correct for this bias, we derive the maximum likelihood (ML) estimators for the accuracies of two screening tests and their corresponding correlation. The p -values and confidence intervals are computed using the asymptotic normality of the ML estimators. Our method is used to compare the accuracies of two screening tests in a two-phase epidemiological study of dementia. We found that, although the sensitivities of the new and standard screening tests in detecting a diseased subject are not different, the new screening test performs better in detecting a non-diseased subject.     

Adjusting ROC curves for covariates in the presence of verification bias

The ROC (receiver operating characteristic) curve is frequently used for describing effectiveness of a diagnostic marker or test. Classical estimation of the ROC curve uses independent identically distributed samples taken randomly from the healthy and diseased populations. Frequently not all subjects undergo a definitive gold standard assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased (verification bias). In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve adjusted for covariates (ROC regression) under verification bias. We develop the estimator's asymptotic distribution and examine its finite sample size properties via a simulation study. We apply this procedure to fingerstick postprandial blood glucose measurement data adjusting for age.     

The effect of verification bias on the comparison of predictive values of two binary diagnostic tests

The comparison of the accuracy of two binary diagnostic tests has traditionally required knowledge of the disease status in all of the patients in the sample via the application of a gold standard. In practice, the gold standard is not always applied to all patients in a sample, and the problem of partial verification of the disease arises. The accuracy of a binary diagnostic test can be measured in terms of positive and negative predictive values, which represent the accuracy of a diagnostic test when it is applied to a cohort of patients. In this paper, we deduce the maximum likelihood estimators of predictive values (PVs) of two binary diagnostic tests, and the hypothesis tests to compare these measures when, in the presence of partial disease verification, the verification process only depends on the results of the two diagnostic tests. The effect of verification bias on the naïve estimators of PVs of two diagnostic tests is studied, and simulation experiments are performed in order to investigate the small sample behaviour of hypothesis tests. The hypothesis tests which we have deduced can be applied when all of the patients are verified with the gold standard. The results obtained have been applied to the diagnosis of coronary stenosis.     

Generalized Confidence Interval Estimation for the Difference in Paired Areas Under the ROC Curves in the Absence of a Gold Standard

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may be too expensive or infeasible. In many medical researches, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, we propose a heuristic method of estimating confidence intervals for the difference in paired AUCs of two diagnostic tests in the absence of a GS reference. This heuristic method is a three-stage method by combining the expectation-maximization (EM) algorithm, bootstrap method, and an estimation based on asymptotic generalized pivotal quantities (GPQs) to construct generalized confidence intervals for the difference in paired AUCs in the absence of a GS. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and expected interval lengths. The numerical example using a published dataset illustrates the proposed method.     

Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions

The receiver operating characteristic (ROC) curve is a graphical representation of the relationship between false positive and true positive rates. It is a widely used statistical tool for describing the accuracy of a diagnostic test. In this paper we propose a new nonparametric ROC curve estimator based on the smoothed empirical distribution functions. We prove its strong consistency and perform a simulation study to compare it with some other popular nonparametric estimators of the ROC curve. We also apply the proposed method to a real data set.     

Asymptotic hypothesis test to simultaneously compare the weighted kappa coefficients of multiple binary diagnostic tests in the presence of ignorable missing data

The weighted kappa coefficient of a binary diagnostic test is a measure of the beyond-chance agreement between the diagnostic test and the gold standard, and is a measure that allows us to assess and compare the performance of binary diagnostic tests. In the presence of partial disease verification, the comparison of the weighted kappa coefficients of two or more binary diagnostic tests cannot be carried out ignoring the individuals with an unknown disease status, since the estimators obtained would be affected by verification bias. In this article, we propose a global hypothesis test based on the chi-square distribution to simultaneously compare the weighted kappa coefficients when in the presence of partial disease verification the missing data mechanism is ignorable. Simulation experiments have been carried out to study the type I error and the power of the global hypothesis test. The results have been applied to the diagnosis of coronary disease.     

Estimation of the area under ROC curve with censored data

The area under the receiver operating characteristic curve is the most commonly used summary measure of diagnostic accuracy for a continuous-scale diagnostic test. In this paper, we develop methods to estimate the area under the curve (AUC) with censored data. Based on two different integration representations of this parameter, two nonparametric estimators are defined by the “plug in” method. Both the proposed estimators are shown to be asymptotically normal based on counting process and martingale theory. A simulation study is conducted to evaluate the performances of the proposed estimators.     

An application of multinomial logistic regression to estimating performance of a multiple-screening test with incomplete verification

Summary.  The paper describes a method of estimating the performance of a multiple-screening test where those who test negatively do not have their true disease status determined. The methodology is motivated by a data set on 49927 subjects who were given K =6 binary tests for bowel cancer. A complicating factor is that individuals may have polyps in the bowel, a condition that the screening test is not designed to detect but which may be worth diagnosing. The methodology is based on a multinomial logit model for Pr( S | R ₆), the probability distribution of patient status S (healthy, polyps or diseased) conditional on the results R ₆ from six binary tests. An advantage of the methodology described is that the modelling is data driven. In particular, we require no assumptions about correlation within subjects, the relative sensitivity of the K tests or the conditional independence of the tests. The model leads to simple estimates of the trade-off between different errors as the number of tests is varied, presented graphically by using receiver operating characteristic curves. Finally, the model allows us to estimate better protocols for assigning subjects to the disease group, as well as the gains in accuracy from these protocols.     

CONFIDENCE INTERVALS FOR THE DIFFERENCE BETWEEN TWO PARTIAL AUCS

As new diagnostic tests are developed and marketed, it is very important to be able to compare the accuracy of a given two continuous‐scale diagnostic tests. An effective method to evaluate the difference between the diagnostic accuracy of two tests is to compare partial areas under the receiver operating characteristic curves (AUCs). In this paper, we review existing parametric methods. Then, we propose a new semiparametric method and a new nonparametric method to investigate the difference between two partial AUCs. For the difference between two partial AUCs under each method, we derive a normal approximation, define an empirical log‐likelihood ratio, and show that the empirical log‐likelihood ratio follows a scaled chi‐square distribution. We construct five confidence intervals for the difference based on normal approximation, bootstrap, and empirical likelihood methods. Finally, extensive simulation studies are conducted to compare the finite‐sample performances of these intervals, and a real example is used as an application of our recommended intervals. The simulation results indicate that the proposed hybrid bootstrap and empirical likelihood intervals outperform other existing intervals in most cases.     

Dependence Between Two Diagnostic Tests with Copula Function Approach: A Simulation Study

The study of the dependence between two medical diagnostic tests is an important issue in health research since it can modify the diagnosis and, therefore, the decision regarding a therapeutic treatment for an individual. In many practical situations, the diagnostic procedure includes the use of two tests, with outcomes on a continuous scale. For final classification, usually there is an additional “gold standard” or reference test. Considering binary test responses, we usually assume independence between tests or a joint binary structure for dependence. In this article, we introduce a simulation study assuming two dependent dichotomized tests using two copula function dependence structures in the presence or absence of verification bias. We compare the test parameter estimators obtained under copula structure dependence with those obtained assuming binary dependence or assuming independent tests.     

Semiparametric ROC surface estimation for continuous diagnostic tests via polytomous logistic regression procedures

In this paper, we propose a semiparametric method of estimating receiver operating characteristic (ROC) surfaces for continuous diagnostic tests under density ratio models. Implementation of our method is easy since the usual polytomous logistic regression procedures in many statistical software packages can be employed. A simulated example is provided to facilitate the implementation of our method. Simulation results show that the proposed semiparametric ROC surface estimator is more efficient than the nonparametric counterpart and the parametric counterpart whether the normality assumption of data holds or not. Moreover, some simulation results on the underlying semiparametric distribution function estimators are also reported. In addition, some discussions on the proposed method as well as analysis of a real data set are provided.     

Comparison of the accuracy of multiple binary tests in the presence of partial disease verification

In the presence of partial disease verification, the comparison of the accuracy of binary diagnostic tests cannot be carried out through the paired comparison of the diagnostic tests applying McNemar's test, since for a subsample of patients the disease status is unknown. In this study, we have deduced the maximum likelihood estimators for the sensitivities and specificities of multiple binary diagnostic tests and we have studied various joint hypothesis tests based on the chi-square distribution to compare simultaneously the accuracy of these binary diagnostic tests when for some patients in the sample the disease status is unknown. Simulation experiments were carried out to study the type I error and the power of each hypothesis test deduced. The results obtained were applied to the diagnosis of coronary stenosis.     

Reducing Bias and Mean Squared Error Associated With Regression-Based Odds Ratio Estimators

Ratio estimators of effect are ordinarily obtained by exponentiating maximum-likelihood estimators (MLEs) of log-linear or logistic regression coefficients. These estimators can display marked positive finite-sample bias, however. We propose a simple correction that removes a substantial portion of the bias due to exponentiation. By combining this correction with bias correction on the log scale, we demonstrate that one achieves complete removal of second-order bias in odds ratio estimators in important special cases. We show how this approach extends to address bias in odds or risk ratio estimators in many common regression settings. We also propose a class of estimators that provide reduced mean bias and squared error, while allowing the investigator to control the risk of underestimating the true ratio parameter. We present simulation studies in which the proposed estimators are shown to exhibit considerable reduction in bias, variance, and mean squared error compared to MLEs. Bootstrapping provides further improvement, including narrower confidence intervals without sacrificing coverage.     

Non‐parametric estimation for time-dependent AUC

The area under the receiver operating characteristic (ROC) curve (AUC) is one of the commonly used measure to evaluate or compare the predictive ability of markers to the disease status. Motivated by an angiographic coronary artery disease (CAD) study, our objective is mainly to evaluate and compare the performance of several baseline plasma levels in the prediction of CAD-related vital status over time. Based on censored survival data, the non-parametric estimators are proposed for the time-dependent AUC. The limiting Gaussian processes of the estimators and the estimated asymptotic variance–covariance functions enable us to further construct confidence bands and develop testing procedures. Applications and finite sample properties of the proposed estimation methods and inference procedures are demonstrated through the CAD-related death data from the British Columbia Vital Statistics Agency and Monte Carlo simulations.     

A non-inferiority test for diagnostic accuracy in the absence of the golden standard test based on the paired partial areas under receiver operating characteristic curves

Non-inferiority tests are often measured for the diagnostic accuracy in medical research. The area under the receiver operating characteristic (ROC) curve is a familiar diagnostic measure for the overall diagnostic accuracy. Nevertheless, since it may not differentiate the diverse shapes of the ROC curves with different diagnostic significance, the partial area under the ROC (PAUROC) curve, another summary measure emerges for such diagnostic processes that require the false-positive rate to be in the clinically interested range. Traditionally, to estimate the PAUROC, the golden standard (GS) test on the true disease status is required. Nevertheless, the GS test may sometimes be infeasible. Besides, in a lot of research fields such as the epidemiology field, the true disease status of the patients may not be known or available. Under the normality assumption on diagnostic test results, based on the expectation-maximization algorithm in combination with the bootstrap method, we propose the heuristic method to construct a non-inferiority test for the difference in the paired PAUROCs without the GS test. Through the simulation study, although the proposed method might provide a liberal test, as a whole, the empirical size of the proposed method sufficiently controls the size at the significance level, and the empirical power of the proposed method in the absence of the GS is as good as that of the non-inferiority in the presence of the GS. The proposed method is illustrated with the published data.     

Modelling stochastic order in the analysis of receiver operating characteristic data: Bayesian non-parametric approaches

Summary.  The evaluation of the performance of a continuous diagnostic measure is a commonly encountered task in medical research. We develop Bayesian non-parametric models that use Dirichlet process mixtures and mixtures of Polya trees for the analysis of continuous serologic data. The modelling approach differs from traditional approaches to the analysis of receiver operating characteristic curve data in that it incorporates a stochastic ordering constraint for the distributions of serologic values for the infected and non-infected populations. Biologically such a constraint is virtually always feasible because serologic values from infected individuals tend to be higher than those for non-infected individuals. The models proposed provide data-driven inferences for the infected and non-infected population distributions, and for the receiver operating characteristic curve and corresponding area under the curve. We illustrate and compare the predictive performance of the Dirichlet process mixture and mixture of Polya trees approaches by using serologic data for Johne's disease in dairy cattle.     

Nonparametric methodology for the time-dependent partial area under the ROC curve

To assess the classification accuracy of a continuous diagnostic result, the receiver operating characteristic (ROC) curve is commonly used in applications. The partial area under the ROC curve (pAUC) is one of the widely accepted summary measures due to its generality and ease of probability interpretation. In the field of life science, a direct extension of the pAUC into the time-to-event setting can be used to measure the usefulness of a biomarker for disease detection over time. Without using a trapezoidal rule, we propose nonparametric estimators, which are easily computed and have closed-form expressions, for the time-dependent pAUC. The asymptotic Gaussian processes of the estimators are established and the estimated variance-covariance functions are provided, which are essential in the construction of confidence intervals. The finite sample performance of the proposed inference procedures are investigated through a series of simulations. Our method is further applied to evaluate the classification ability of CD4 cell counts on patient's survival time in the AIDS Clinical Trials Group (ACTG) 175 study. In addition, the inferences can be generalized to compare the time-dependent pAUCs between patients received the prior antiretroviral therapy and those without it.     

Nonparametric covariate adjustment for receiver operating characteristic curves

The accuracy of a diagnostic test is typically characterized using the receiver operating characteristic (ROC) curve. Summarizing indexes such as the area under the ROC curve (AUC) are used to compare different tests as well as to measure the difference between two populations. Often additional information is available on some of the covariates which are known to influence the accuracy of such measures. The authors propose nonparametric methods for covariate adjustment of the AUC. Models with normal errors and possibly non‐normal errors are discussed and analyzed separately. Nonparametric regression is used for estimating mean and variance functions in both scenarios. In the model that relaxes the assumption of normality, the authors propose a covariate‐adjusted Mann–Whitney estimator for AUC estimation which effectively uses available data to construct working samples at any covariate value of interest and is computationally efficient for implementation. This provides a generalization of the Mann–Whitney approach for comparing two populations by taking covariate effects into account. The authors derive asymptotic properties for the AUC estimators in both settings, including asymptotic normality, optimal strong uniform convergence rates and mean squared error (MSE) consistency. The MSE of the AUC estimators was also assessed in smaller samples by simulation. Data from an agricultural study were used to illustrate the methods of analysis. The Canadian Journal of Statistics 38:27–46; 2010 © 2009 Statistical Society of Canada