Testing the difference between two Kolmogorov–Smirnov values in the context of receiver operating characteristic curves

The maximum vertical distance between a receiver operating characteristic (ROC) curve and its chance diagonal is a common measure of effectiveness of the classifier that gives rise to this curve. This measure is known to be equivalent to a two-sample Kolmogorov–Smirnov statistic; so the absolute difference D between two such statistics is often used informally as a measure of difference between the corresponding classifiers. A significance test of D is of great practical interest, but the available Kolmogorov–Smirnov distribution theory precludes easy analytical construction of such a significance test. We, therefore, propose a Monte Carlo procedure for conducting the test, using the binormal model for the underlying ROC curves. We provide Splus/R routines for the computation, tabulate the results for a number of illustrative cases, apply the methods to some practical examples and discuss some implications.     

Powers of Discrete Goodness-of-Fit Test Statistics for a Uniform Null Against a Selection of Alternative Distributions

The comparative powers of six discrete goodness-of-fit test statistics for a uniform null distribution against a variety of fully specified alternative distributions are discussed. The results suggest that the test statistics based on the empirical distribution function for ordinal data (Kolmogorov–Smirnov, Cramér–von Mises, and Anderson–Darling) are generally more powerful for trend alternative distributions. The test statistics for nominal (Pearson's chi-square and the nominal Kolmogorov–Smirnov) and circular data (Watson's test statistic) are shown to be generally more powerful for the investigated triangular (∨), flat (or platykurtic type), sharp (or leptokurtic type), and bimodal alternative distributions.     

Rank-Based Classification Using Robust Discriminant Functions

The Pareto distribution model assumption in the peaks over threshold method, will be tested by making using of the Kolmogorov–Smirnov goodness of fit method. Pareto distributed variables can be transformed to exponential, and the test will be for exponentiality. It was found that the statistic can be used as an indication of where to choose the threshold and to check the Pareto model assumption.     

Kolmogorov–Smirnov test for life test data with hybrid censoring

This work considers goodness-of-fit for the life test data with hybrid censoring. An alternative representation of the Kolmogorov–Smirnov (KS) statistics is provided under Type-I censoring. The alternative representation leads us to approximate the limiting distributions of the KS statistic as a functional of the Brownian bridge for Type-II, Type-I hybrid, and Type-II hybrid censored data. The approximated distributions are used to obtain the critical values of the tests in this context. We found that the proposed KS test procedure for Type-II censoring has more power than the available one(s) in literature.     

Threshold selection for regional peaks-over-threshold data

A hurdle in the peaks-over-threshold approach for analyzing extreme values is the selection of the threshold. A method is developed to reduce this obstacle in the presence of multiple, similar data samples. This is for instance the case in many environmental applications. The idea is to combine threshold selection methods into a regional method. Regionalized versions of the threshold stability and the mean excess plot are presented as graphical tools for threshold selection. Moreover, quantitative approaches based on the bootstrap distribution of the spatially averaged Kolmogorov–Smirnov and Anderson–Darling test statistics are introduced. It is demonstrated that the proposed regional method leads to an increased sensitivity for too low thresholds, compared to methods that do not take into account the regional information. The approach can be used for a wide range of univariate threshold selection methods. We test the methods using simulated data and present an application to rainfall data from the Dutch water board Vallei en Veluwe.     

A new method for the comparison of survival distributions

The assessment of overall homogeneity of time‐to‐event curves is a key element in survival analysis in biomedical research. The currently commonly used testing methods, e.g. log‐rank test, Wilcoxon test, and Kolmogorov–Smirnov test, may have a significant loss of statistical testing power under certain circumstances. In this paper we propose a new testing method that is robust for the comparison of the overall homogeneity of survival curves based on the absolute difference of the area under the survival curves using normal approximation by Greenwood's formula. Monte Carlo simulations are conducted to investigate the performance of the new testing method compared against the log‐rank, Wilcoxon, and Kolmogorov–Smirnov tests under a variety of circumstances. The proposed new method has robust performance with greater power to detect the overall differences than the log‐rank, Wilcoxon, and Kolmogorov–Smirnov tests in many scenarios in the simulations. Furthermore, the applicability of the new testing approach is illustrated in a real data example from a kidney dialysis trial. Copyright © 2009 John Wiley & Sons, Ltd.     

Optimal cutoffs for continuous biomarkers for survival data under competing risks

Identifying an optimal cutoff value for a continuous biomarker is often useful for medical applications. For binary outcome, commonly used cutoff finding criteria include Youden's index, classification accuracy, and the Euclidean distance to the upper left corner on the ROC curve. We extend these three criteria to accommodate censored survival time that subjected to competing risks. We provide various definitions of time-dependent true positive rate and false positive rate and estimate those quantities using nonparametric methods. In simulation studies, the Euclidean distance to the upper left corner on the ROC curve shows the best overall performance.     

On a goodness-of-fit test for normality with unknown parameters and type-II censored data

We propose a new goodness-of-fit test for normal and lognormal distributions with unknown parameters and type-II censored data. This test is a generalization of Michael's test for censored samples, which is based on the empirical distribution and a variance stabilizing transformation. We estimate the parameters of the model by using maximum likelihood and Gupta's methods. The quantiles of the distribution of the test statistic under the null hypothesis are obtained through Monte Carlo simulations. The power of the proposed test is estimated and compared to that of the Kolmogorov–Smirnov test also using simulations. The new test is more powerful than the Kolmogorov–Smirnov test in most of the studied cases. Acceptance regions for the PP, QQ and Michael's stabilized probability plots are derived, making it possible to visualize which data contribute to the decision of rejecting the null hypothesis. Finally, an illustrative example is presented.     

Assessing goodness-of-fit of categorical regression models based on case-control data

Demonstrated equivalence between a categorical regression model based on case‐control data and an I‐sample semiparametric selection bias model leads to a new goodness‐of‐fit test. The proposed test statistic is an extension of an existing Kolmogorov–Smirnov‐type statistic and is the weighted average of the absolute differences between two estimated distribution functions in each response category. The paper establishes an optimal property for the maximum semiparametric likelihood estimator of the parameters in the I‐sample semiparametric selection bias model. It also presents a bootstrap procedure, some simulation results and an analysis of two real datasets.     

On the confusion matrix in credit scoring and its analytical properties

Abstract

Confusion Matrix is an important measure to evaluate the accuracy of credit scoring models. However, the literature about Confusion Matrix is limited. The analytical properties of Confusion Matrix are ignored. Moreover, the concept of Confusion Matrix is confusing. In this article, we systematically study Confusion Matrix and its analytical properties. We enumerate 16 possible variants of Confusion Matrix and show that only 8 are reasonable. We study the relationship between Confusion Matrix and 2 other performance measures: the receiver operating characteristic curve (ROC) and Kolmogorov-Smirnov statistic (KS). We show that an optimal cutoff score can be attained by KS.     

Goodness-of-Fit Tests for the Skew-Normal Distribution When the Parameters Are Estimated from the Data

In this article, tests are developed which can be used to investigate the goodness-of-fit of the skew-normal distribution in the context most relevant to the data analyst, namely that in which the parameter values are unknown and are estimated from the data. We consider five test statistics chosen from the broad Cramér–von Mises and Kolmogorov–Smirnov families, based on measures of disparity between the distribution function of a fitted skew-normal population and the empirical distribution function. The sampling distributions of the proposed test statistics are approximated using Monte Carlo techniques and summarized in easy to use tabular form. We also present results obtained from simulation studies designed to explore the true size of the tests and their power against various asymmetric alternative distributions.     

An exact Kolmogorov–Smirnov test for the Poisson distribution with unknown mean

We develop an exact Kolmogorov–Smirnov goodness-of-fit test for the Poisson distribution with an unknown mean. This test is conditional, with the test statistic being the maximum absolute difference between the empirical distribution function and its conditional expectation given the sample total. Exact critical values are obtained using a new algorithm. We explore properties of the test, and we illustrate it with three examples. The new test seems to be the first exact Poisson goodness-of-fit test for which critical values are available without simulation or exhaustive enumeration.     

Nonparametric predictive inference for diagnostic test thresholds

Abstract

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning and credit scoring. The receiver operating characteristic (ROC) curve and surface are useful tools to assess the ability of diagnostic tests to discriminate between ordered classes or groups. To define these diagnostic tests, selecting the optimal thresholds that maximize the accuracy of these tests is required. One procedure that is commonly used to find the optimal thresholds is by maximizing what is known as Youden’s index. This article presents nonparametric predictive inference (NPI) for selecting the optimal thresholds of a diagnostic test. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. Based on multiple future observations, the NPI approach is presented for selecting the optimal thresholds for two-group and three-group scenarios. In addition, a pairwise approach has also been presented for the three-group scenario. The article ends with an example to illustrate the proposed methods and a simulation study of the predictive performance of the proposed methods along with some classical methods such as Youden index. The NPI-based methods show some interesting results that overcome some of the issues concerning the predictive performance of Youden’s index.     

Semi-empirical likelihood inference for the ROC curve with missing data

The receiver operating characteristic (ROC) curve is one of the most commonly used methods to compare the diagnostic performance of two or more laboratory or diagnostic tests. In this paper, we propose semi-empirical likelihood based confidence intervals for ROC curves of two populations, where one population is parametric and the other one is non-parametric and both have missing data. After imputing missing values, we derive the semi-empirical likelihood ratio statistic and the corresponding likelihood equations. It is shown that the log-semi-empirical likelihood ratio statistic is asymptotically scaled chi-squared. The estimating equations are solved simultaneously to obtain the estimated lower and upper bounds of semi-empirical likelihood confidence intervals. We conduct extensive simulation studies to evaluate the finite sample performance of the proposed empirical likelihood confidence intervals with various sample sizes and different missing probabilities.     

Asymptotic properties of a goodness-of-fit test based on maximum correlations

We study the efficiency properties of the goodness-of-fit test based on the Q _n statistic introduced in Fortiana and Grané [Goodness-of-fit tests based on maximum correlations and their orthogonal decompositions, J. R. Stat. Soc. B 65 (2003), pp. 115–126] using the concepts of Bahadur asymptotic relative efficiency and Bahadur asymptotic optimality. We compare the test based on this statistic with those based on the Kolmogorov–Smirnov, the Cramér-von Mises criterion and the Anderson–Darling statistics. We also describe the distribution families for which the test based on Q _n is locally asymptotically optimal in the Bahadur sense and, as an application, we use this test to detect the presence of hidden periodicities in a stationary time series.     

Bootstrap Method for Order Statistics and Modeling Study of the Air Pollution

In this article, the block maxima (BM) and the peak over threshold (POT) methods are used to model the air pollution. A simulation technique is suggested to choose a suitable threshold value. The validity of the estimated models is checked by the Kolmogorov–Smirnov (K-S) test. A new efficient approach for modeling extreme values is suggested. Finally, the inconsistency and weak consistency of bootstrapping central and intermediate order statistics for an appropriate choice of re-sample size are investigated.     

Estimation based on progressive first-failure censoring from exponentiated exponential distribution

In this paper, point and interval estimations for the parameters of the exponentiated exponential (EE) distribution are studied based on progressive first-failure-censored data. The Bayes estimates are computed based on squared error and Linex loss functions and using Markov Chain Monte Carlo (MCMC) algorithm. Also, based on this censoring scheme, approximate confidence intervals for the parameters of EE distribution are developed. Monte Carlo simulation study is carried out to compare the performances of the different methods by computing the estimated risks (ERs), as well as Akaike's information criteria (AIC) and Bayesian information criteria (BIC) of the estimates. Finally, a real data set is introduced and analyzed using EE and Weibull distributions. A comparison is carried out between the mentioned models based on the corresponding Kolmogorov–Smirnov (K–S) test statistic to emphasize that the EE model fits the data with the same efficiency as the other model. Point and interval estimation of all parameters are studied based on this real data set as illustrative example.     

An application of lomax distributions in receiver operating characteristic(roc)curve analysis

Receiver operating characteristic(ROC)curves are useful for studying the performance of diagnostic tests. ROC curves occur in many fields of applications including psychophysics, quality control and medical diagnostics. In practical situations, often the responses to a diagnostic test are classified into a number of ordered categories. Such data are referred to as ratings data. It is typically assumed that the underlying model is based on a continuous probability distribution. The ROC curve is then constructed from such data using this probability model. Properties of the ROC curve are inherited from the model. Therefore, understanding the role of different probability distributions in ROC modeling is an interesting and important area of research. In this paper the Lomax distribution is considered as a model for ratings data and the corresponding ROC curve is derived. The maximum likelihood estimation procedure for the related parameters is discussed. This procedure is then illustrated in the analysis of a neurological data example.     

Fitting Roc Curves Using Non-linear Binomial Regression

The performance of a diagnostic test is summarized by its receiver operating characteristic (ROC) curve. Empirical data on a test's performance often come in the form of observed true positive and false positive relative frequencies, under varying conditions. This paper describes a family of models for analysing such data. The underlying ROC curves are specified by a shift parameter, a shape parameter and a link function. Both the position along the ROC curve and the shift parameter are modelled linearly. The shape parameter enters the model non-linearly but in a very simple manner. One simple application is to the meta-analysis of independent studies of the same diagnostic test, illustrated on some data of Moses, Shapiro & Littenberg (1993). A second application to so-called vigilance data is given, where ROC curves differ across subjects, and modelling of the position along the ROC curve is of primary interest.