Bayesian ROC curve estimation under binormality using an ordinal category likelihood

Receiver operating characteristic (ROC) curve has been widely used in medical diagnosis. Various methods are proposed to estimate ROC curve parameters under the binormal model. In this paper, we propose a Bayesian estimation method from the continuously distributed data which is constituted by the truth-state-runs in the rank-ordered data. By using an ordinal category data likelihood and following the Metropolis–Hastings (M–H) procedure, we compute the posterior distribution of the binormal parameters, as well as the group boundaries parameters. Simulation studies and real data analysis are conducted to evaluate our Bayesian estimation method.     

An application of lomax distributions in receiver operating characteristic(roc)curve analysis

Receiver operating characteristic(ROC)curves are useful for studying the performance of diagnostic tests. ROC curves occur in many fields of applications including psychophysics, quality control and medical diagnostics. In practical situations, often the responses to a diagnostic test are classified into a number of ordered categories. Such data are referred to as ratings data. It is typically assumed that the underlying model is based on a continuous probability distribution. The ROC curve is then constructed from such data using this probability model. Properties of the ROC curve are inherited from the model. Therefore, understanding the role of different probability distributions in ROC modeling is an interesting and important area of research. In this paper the Lomax distribution is considered as a model for ratings data and the corresponding ROC curve is derived. The maximum likelihood estimation procedure for the related parameters is discussed. This procedure is then illustrated in the analysis of a neurological data example.     

Testing the difference between two Kolmogorov–Smirnov values in the context of receiver operating characteristic curves

The maximum vertical distance between a receiver operating characteristic (ROC) curve and its chance diagonal is a common measure of effectiveness of the classifier that gives rise to this curve. This measure is known to be equivalent to a two-sample Kolmogorov–Smirnov statistic; so the absolute difference D between two such statistics is often used informally as a measure of difference between the corresponding classifiers. A significance test of D is of great practical interest, but the available Kolmogorov–Smirnov distribution theory precludes easy analytical construction of such a significance test. We, therefore, propose a Monte Carlo procedure for conducting the test, using the binormal model for the underlying ROC curves. We provide Splus/R routines for the computation, tabulate the results for a number of illustrative cases, apply the methods to some practical examples and discuss some implications.     

Improving an estimator of Hsieh and Turnbull for the binormal ROC curve

The estimator of Hsieh and Turnbull (1996) for the binormal receiver operating characteristic (ROC) curve is extended from grouped to ungrouped data. The new estimator is shown to be consistent and asymptotically normally distributed, and simulation results show that it outperforms Hsieh and Turnbull's original estimator.     

Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions

The receiver operating characteristic (ROC) curve is a graphical representation of the relationship between false positive and true positive rates. It is a widely used statistical tool for describing the accuracy of a diagnostic test. In this paper we propose a new nonparametric ROC curve estimator based on the smoothed empirical distribution functions. We prove its strong consistency and perform a simulation study to compare it with some other popular nonparametric estimators of the ROC curve. We also apply the proposed method to a real data set.     

Two-sided tests for change in level for correlated data

The classical problem of change point is considered when the data are assumed to be correlated. The nuisance parameters in the model are the initial level μ and the common variance σ². The four cases, based on none, one, and both of the parameters are known are considered. Likelihood ratio tests are obtained for testing hypotheses regarding the change in level, δ, in each case. Following Henderson (1986), a Bayesian test is obtained for the two sided alternative. Under the Bayesian set up, a locally most powerful unbiased test is derived for the case μ=0 and σ²=1. The exact null distribution function of the Bayesian test statistic is given an integral representation. Methods to obtain exact and approximate critical values are indicated.     

Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface

In many situations the diagnostic decision is not limited to a binary choice. Binary statistical tools such as receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) need to be expanded to address three-category classification problem. Previous authors have suggest various ways to model the extension of AUC but not the ROC surface. Only simple parametric approaches are proposed for modeling the ROC measure under the assumption that test results all follow normal distributions. We study the estimation methods of three-dimensional ROC surfaces with nonparametric and semiparametric estimators. Asymptotical results are provided as a basis for statistical inference. Simulation studies are performed to assess the validity of our proposed methods in finite samples. We consider an Alzheimer's disease example from a clinical study in the US as an illustration. The nonparametric and semiparametric modelling approaches for the three way ROC analysis can be readily generalized to diagnostic problems with more than three classes.     

Fitting Roc Curves Using Non-linear Binomial Regression

The performance of a diagnostic test is summarized by its receiver operating characteristic (ROC) curve. Empirical data on a test's performance often come in the form of observed true positive and false positive relative frequencies, under varying conditions. This paper describes a family of models for analysing such data. The underlying ROC curves are specified by a shift parameter, a shape parameter and a link function. Both the position along the ROC curve and the shift parameter are modelled linearly. The shape parameter enters the model non-linearly but in a very simple manner. One simple application is to the meta-analysis of independent studies of the same diagnostic test, illustrated on some data of Moses, Shapiro & Littenberg (1993). A second application to so-called vigilance data is given, where ROC curves differ across subjects, and modelling of the position along the ROC curve is of primary interest.     

Compare diagnostic tests using transformation-invariant smoothed ROC curves()

Receiver operating characteristic (ROC) curve, plotting true positive rates against false positive rates as threshold varies, is an important tool for evaluating biomarkers in diagnostic medicine studies. By definition, ROC curve is monotone increasing from 0 to 1 and is invariant to any monotone transformation of test results. And it is often a curve with certain level of smoothness when test results from the diseased and non-diseased subjects follow continuous distributions. Most existing ROC curve estimation methods do not guarantee all of these properties. One of the exceptions is Du and Tang (2009) which applies certain monotone spline regression procedure to empirical ROC estimates. However, their method does not consider the inherent correlations between empirical ROC estimates. This makes the derivation of the asymptotic properties very difficult. In this paper we propose a penalized weighted least square estimation method, which incorporates the covariance between empirical ROC estimates as a weight matrix. The resulting estimator satisfies all the aforementioned properties, and we show that it is also consistent. Then a resampling approach is used to extend our method for comparisons of two or more diagnostic tests. Our simulations show a significantly improved performance over the existing method, especially for steep ROC curves. We then apply the proposed method to a cancer diagnostic study that compares several newly developed diagnostic biomarkers to a traditional one.     

Confidence Bands for ROC Curves With Serially Dependent Data

We propose serial correlation-robust asymptotic confidence bands for the receiver operating characteristic (ROC) curve and its functional, viz., the area under ROC curve (AUC), estimated by quasi-maximum likelihood in the binormal model. Our simulation experiments confirm that this new method performs fairly well in finite samples, and confers an additional measure of robustness to nonnormality. The conventional procedure is found to be markedly undersized in terms of yielding empirical coverage probabilities lower than the nominal level, especially when the serial correlation is strong. An example from macroeconomic forecasting demonstrates the importance of accounting for serial correlation when the probability forecasts for real GDP declines are evaluated using ROC. Supplementary materials for this article are available online.     

A non-inferiority test for diagnostic accuracy in the absence of the golden standard test based on the paired partial areas under receiver operating characteristic curves

Non-inferiority tests are often measured for the diagnostic accuracy in medical research. The area under the receiver operating characteristic (ROC) curve is a familiar diagnostic measure for the overall diagnostic accuracy. Nevertheless, since it may not differentiate the diverse shapes of the ROC curves with different diagnostic significance, the partial area under the ROC (PAUROC) curve, another summary measure emerges for such diagnostic processes that require the false-positive rate to be in the clinically interested range. Traditionally, to estimate the PAUROC, the golden standard (GS) test on the true disease status is required. Nevertheless, the GS test may sometimes be infeasible. Besides, in a lot of research fields such as the epidemiology field, the true disease status of the patients may not be known or available. Under the normality assumption on diagnostic test results, based on the expectation-maximization algorithm in combination with the bootstrap method, we propose the heuristic method to construct a non-inferiority test for the difference in the paired PAUROCs without the GS test. Through the simulation study, although the proposed method might provide a liberal test, as a whole, the empirical size of the proposed method sufficiently controls the size at the significance level, and the empirical power of the proposed method in the absence of the GS is as good as that of the non-inferiority in the presence of the GS. The proposed method is illustrated with the published data.     

Generalized Confidence Interval Estimation for the Difference in Paired Areas Under the ROC Curves in the Absence of a Gold Standard

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may be too expensive or infeasible. In many medical researches, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, we propose a heuristic method of estimating confidence intervals for the difference in paired AUCs of two diagnostic tests in the absence of a GS reference. This heuristic method is a three-stage method by combining the expectation-maximization (EM) algorithm, bootstrap method, and an estimation based on asymptotic generalized pivotal quantities (GPQs) to construct generalized confidence intervals for the difference in paired AUCs in the absence of a GS. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and expected interval lengths. The numerical example using a published dataset illustrates the proposed method.     

Bayesian ROC curve estimation under binormality using a rank likelihood

There are various methods to estimate the parameters in the binormal model for the ROC curve. In this paper, we propose a conceptually simple and computationally feasible Bayesian estimation method using a rank-based likelihood. Posterior consistency is also established. We compare the new method with other estimation methods and conclude that our estimator generally performs better than its competitors.     

Grouped Variable Selection Using Area under the ROC with Imbalanced Data

Imbalanced data brings biased classification and causes the low accuracy of the classification of the minority class. In this article, we propose a methodology to select grouped variables using the area under the ROC with an adjustable prediction cut point. The proposed method enhance the accuracy of classification for the minority class by maximizing the true positive rate. Simulation results show that the proposed method is appropriate for both the categorical and continuous covariates. An illustrative example of the analysis of the SHS data in TCM is discussed to show the reasonable application of the proposed method.     

Retrospective change detection in categorical time series

The aim of this paper is to propose methods of detecting change in the coefficients of a multinomial logistic regression model for categorical time series offline. The alternatives to the null hypothesis of stationarity can be either the hypothesis that it is not true, or that there is a temporary change in the sequence. We use the efficient score vector of the partial likelihood function. This has several advantages. First, the alternative value of the parameter does not have to be estimated; hence, we have a procedure that has a simple structure with only one parameter estimation using all available observations. This is in contrast with the generalized likelihood ratio-based change point tests. The efficient score vector is used in various ways. As a vector, its components correspond to the different components of the multinomial logistic regression model’s parameter vector. Using its quadratic form a test can be defined, where the presence of a change in any or all parameters is tested for. If there are too many parameters one can test for any subset while treating the rest as nuisance parameters. Our motivating example is a DNA sequence of four categories, and our test result shows that in the published data the distribution of the four categories is not stationary.     

Prediction bands for the EDF of a future sample

We develop both nonparametric and parametric methods for obtaining prediction bands for the empirical distribution function (EDF) of a future sample. These methods yield simultaneous prediction intervals for all order statistics of the future sample, and they also correspond to tests for the two-sample problem. The nonparametric prediction bands correspond to the two-sample Kolmogorov-Smirnov test and related nonparametric tests, but the parametric prediction bands correspond to entirely new parametric two-sample tests. The parametric prediction bands tend to outperform the nonparametric bands when the parametric assumptions hold, but they may have true coverage probabilities well below their nominal levels when the parametric assumptions fail. A new computational algorithm is used to obtain critical values in the nonparametric case.     

Nonparametric tests for multivariate multi-sample locations based on data depth

Several nonparametric tests for multivariate multi-sample location problem are proposed in this paper. These tests are based on the notion of data depth, which is used to measure the centrality/outlyingness of a given point with respect to a given distribution or a data cloud. Proposed tests are completely nonparametric and implemented through the idea of permutation tests. Performance of the proposed tests is compared with existing parametric test and nonparametric test based on data depth. An extensive simulation study reveals that proposed tests are superior to the existing tests based on data depth with regard to power. Illustrations with real data are provided.     

Assessing accuracy of a continuous screening test in the presence of verification bias

Summary.  In studies to assess the accuracy of a screening test, often definitive disease assessment is too invasive or expensive to be ascertained on all the study subjects. Although it may be more ethical or cost effective to ascertain the true disease status with a higher rate in study subjects where the screening test or additional information is suggestive of disease, estimates of accuracy can be biased in a study with such a design. This bias is known as verification bias. Verification bias correction methods that accommodate screening tests with binary or ordinal responses have been developed; however, no verification bias correction methods exist for tests with continuous results. We propose and compare imputation and reweighting bias-corrected estimators of true and false positive rates, receiver operating characteristic curves and area under the receiver operating characteristic curve for continuous tests. Distribution theory and simulation studies are used to compare the proposed estimators with respect to bias, relative efficiency and robustness to model misspecification. The bias correction estimators proposed are applied to data from a study of screening tests for neonatal hearing loss.     

Adjusting ROC curves for covariates in the presence of verification bias

The ROC (receiver operating characteristic) curve is frequently used for describing effectiveness of a diagnostic marker or test. Classical estimation of the ROC curve uses independent identically distributed samples taken randomly from the healthy and diseased populations. Frequently not all subjects undergo a definitive gold standard assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased (verification bias). In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve adjusted for covariates (ROC regression) under verification bias. We develop the estimator's asymptotic distribution and examine its finite sample size properties via a simulation study. We apply this procedure to fingerstick postprandial blood glucose measurement data adjusting for age.     

Optimal Threshold from ROC and CAP Curves

Receiver Operating Characteristic (ROC) and Cumulative Accuracy Profile (CAP) curves are used to assess the discriminatory power of different credit-rating approaches. The thresholds of optimal classification accuracy on an ROC curve and of maximal profit on a CAP curve can be found by using iso-performance tangent lines, which are based on the standard notion of accuracy. In this article, we propose another accuracy measure called the true rate. Using this rate, one can obtain alternative optimal thresholds on both ROC and CAP curves. For most real populations of borrowers, the number of the defaults is much less than that of the non defaults, and in such cases using the true rate may be more efficient than using the accuracy rate in terms of cost functions. Moreover, it is shown that both alternative optimal thresholds by using the true rate are the identical, and this single threshold coincides with the score corresponding to Kolmogorov–Smirnov statistic used to test the homogeneous distribution functions of the defaults and non defaults, whereas the optimal threshold by using the accuracy does not the same as the score corresponding to Kolmogorov–Smirnov statistic. These facts are explored with some simulation and illustrative examples.