Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.     

A sequential conditional probability ratio test procedure for comparing diagnostic tests

In this paper, we derive sequential conditional probability ratio tests to compare diagnostic tests without distributional assumptions on test results. The test statistics in our method are nonparametric weighted areas under the receiver-operating characteristic curves. By using the new method, the decision of stopping the diagnostic trial early is unlikely to be reversed should the trials continue to the planned end. The conservatism reflected in this approach to have more conservative stopping boundaries during the course of the trial is especially appealing for diagnostic trials since the end point is not death. In addition, the maximum sample size of our method is not greater than a fixed sample test with similar power functions. Simulation studies are performed to evaluate the properties of the proposed sequential procedure. We illustrate the method using data from a thoracic aorta imaging study.     

Optimal Composite Markers for Time‐Dependent Receiver Operating Characteristic Curves with Censored Survival Data

Abstract. To increase the predictive abilities of several plasma biomarkers on the coronary artery disease (CAD)‐related vital statuses over time, our research interest mainly focuses on seeking combinations of these biomarkers with the highest time‐dependent receiver operating characteristic curves. An extended generalized linear model (EGLM) with time‐varying coefficients and an unknown bivariate link function is used to characterize the conditional distribution of time to CAD‐related death. Based on censored survival data, two non‐parametric procedures are proposed to estimate the optimal composite markers, linear predictors in the EGLM model. Estimation methods for the classification accuracies of the optimal composite markers are also proposed. In the article we establish theoretical results of the estimators and examine the corresponding finite‐sample properties through a series of simulations with different sample sizes, censoring rates and censoring mechanisms. Our optimization procedures and estimators are further shown to be useful through an application to a prospective cohort study of patients undergoing angiography.     

A modified area under the ROC curve and its application to marker selection and classification

The area under the ROC curve (AUC) can be interpreted as the probability that the classification scores of a diseased subject is larger than that of a non-diseased subject for a randomly sampled pair of subjects. From the perspective of classification, we want to find a way to separate two groups as distinctly as possible via AUC. When the difference of the scores of a marker is small, its impact on classification is less important. Thus, a new diagnostic/classification measure based on a modified area under the ROC curve (mAUC) is proposed, which is defined as a weighted sum of two AUCs, where the AUC with the smaller difference is assigned a lower weight, and vice versa. Using mAUC is robust in the sense that mAUC gets larger as AUC gets larger as long as they are not equal. Moreover, in many diagnostic situations, only a specific range of specificity is of interest. Under normal distributions, we show that if the AUCs of two markers are within similar ranges, the larger mAUC implies the larger partial AUC for a given specificity. This property of mAUC will help to identify the marker with the higher partial AUC, even when the AUCs are similar. Two nonparametric estimates of an mAUC and their variances are given. We also suggest the use of mAUC as the objective function for classification, and the use of the gradient Lasso algorithm for classifier construction and marker selection. Application to simulation datasets and real microarray gene expression datasets show that our method finds a linear classifier with a higher ROC curve than some other existing linear classifiers, especially in the range of low false positive rates.     

Non‐parametric interval estimation for the partial area under the ROC curve

Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased conditions from non‐diseased conditions. For a continuous‐scale diagnostic test, a popular summary index of the receiver operating characteristic (ROC) curve is the area under the curve (AUC). However, when our focus is on a certain region of false positive rates, we often use the partial AUC instead. In this paper we have derived the asymptotic normal distribution for the non‐parametric estimator of the partial AUC with an explicit variance formula. The empirical likelihood (EL) ratio for the partial AUC is defined and it is shown that its limiting distribution is a scaled chi‐square distribution. Hybrid bootstrap and EL confidence intervals for the partial AUC are proposed by using the newly developed EL theory. We also conduct extensive simulation studies to compare the relative performance of the proposed intervals and existing intervals for the partial AUC. A real example is used to illustrate the application of the recommended intervals. The Canadian Journal of Statistics 39: 17–33; 2011 © 2011 Statistical Society of Canada     

Factors effecting the model performance measures area under the ROC curve,net reclassification improvement and integrated discrimination improvement

ABSTRACT

The aim of this study is to investigate the impact of correlation structure, prevalence and effect size on the risk prediction model by using the change in the area under the receiver operating characteristic curve (ΔAUC), net reclassification improvement (NRI), and integrated discrimination improvement (IDI). In simulation study, the dataset is generated under different correlation structures, prevalences and effect sizes. We verify the simulation results with the real-data application. In conclusion, the correlation structure between the variables should be taken into account while composing a multivariable model. Negative correlation structure between independent variables is more beneficial while constructing a model.     

A new mathematical approach for the estimation of the AUC and its variability under different experimental designs in preclinical studies

The aim of the present work was to develop a new mathematical method for estimating the area under the curve (AUC) and its variability that could be applied in different preclinical experimental designs and amenable to be implemented in standard calculation worksheets. In order to assess the usefulness of the new approach, different experimental scenarios were studied and the results were compared with those obtained with commonly used software: WinNonlin® and Phoenix WinNonlin®. The results do not show statistical differences among the AUC values obtained by both procedures, but the new method appears to be a better estimator of the AUC standard error, measured as the coverage of 95% confidence interval. In this way, the new proposed method demonstrates to be as useful as WinNonlin® software when it was applicable. Copyright © 2011 John Wiley & Sons, Ltd.     

Criteria for reporting noncompartmental estimates of half‐life and area under the curve extrapolated to infinity

Various criteria have been proposed for determining the reliability of noncompartmental pharmacokinetic estimates of the terminal disposition phase half‐life (t_1/2) and the extrapolated area under the curve (AUC_extrap). This simulation study assessed the performance of two frequently used reportability rules: the terminal disposition phase regression adjusted‐r² classification rule and the regression data point time span classification rule. Using simulated data, these rules were assessed in relation to the magnitude of the variability in the terminal disposition phase slope, the length of the terminal disposition phase captured in the concentration‐time profile (data span), the number of data points present in the terminal disposition phase, and the type and level of variability in concentration measurement. The accuracy of estimating t_1/2 was satisfactory for data spans of 1.5 and longer, given low measurement variability; and for spans of 2.5 and longer, given high measurement variability. Satisfactory accuracy in estimating AUC_extrap was only achieved with low measurement variability and spans of 2.5 and longer. Neither of the classification rules improved the identification of accurate t_1/2 and AUC_extrap estimates. Based on the findings of this study, a strategy is proposed for determining the reportability of estimates of t_1/2 and area under the curve extrapolated to infinity.