Nonparametric predictive inference for diagnostic test thresholds

Abstract

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning and credit scoring. The receiver operating characteristic (ROC) curve and surface are useful tools to assess the ability of diagnostic tests to discriminate between ordered classes or groups. To define these diagnostic tests, selecting the optimal thresholds that maximize the accuracy of these tests is required. One procedure that is commonly used to find the optimal thresholds is by maximizing what is known as Youden’s index. This article presents nonparametric predictive inference (NPI) for selecting the optimal thresholds of a diagnostic test. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. Based on multiple future observations, the NPI approach is presented for selecting the optimal thresholds for two-group and three-group scenarios. In addition, a pairwise approach has also been presented for the three-group scenario. The article ends with an example to illustrate the proposed methods and a simulation study of the predictive performance of the proposed methods along with some classical methods such as Youden index. The NPI-based methods show some interesting results that overcome some of the issues concerning the predictive performance of Youden’s index.     

Confidence Intervals for the Youden Index and Corresponding Optimal Cut-Point

A global measure of biomarker effectiveness is the Youden index, the maximum difference between sensitivity, the probability of correctly classifying diseased individuals, and 1-specificity, the probability of incorrectly classifying healthy individuals. The cut-point leading to the index is the optimal cut-point when equal weight is given to sensitivity and specificity. Using the delta method, we present approaches for estimating confidence intervals for the Youden index and corresponding optimal cut-point for normally distributed biomarkers and also those following gamma distributions. We also provide confidence intervals using various bootstrapping methods. A comparison of interval width and coverage probability is conducted through simulation over a variety of parametric situations. Confidence intervals via delta method are shown to have both closer to nominal coverage and shorter interval widths than confidence intervals from the bootstrapping methods.     

Combination of multiple functional markers to improve diagnostic accuracy

Combination of multiple biomarkers to improve diagnostic accuracy is meaningful for practitioners and clinicians, and are attractive to lots of researchers. Nowadays, with development of modern techniques, functional markers such as curves or images, play an important role in diagnosis. There exists rich literature developing combination methods for continuous scalar markers. Unfortunately, only sporadic works have studied how functional markers affect diagnosis in the literature. Moreover, no publication can be found to do combination of multiple functional markers to improve the diagnostic accuracy. It is impossible to apply scalar combination methods to the multiple functional markers directly because of infinite dimensionality of functional markers. In this article, we propose a one-dimension scalar feature motivated by square loss distance, as an alternative of the original functional curve in the sense that, it can retain information to the most extent. The square loss distance is defined as the function of projection scores generated from functional principal component decomposition. Then existing variety of scalar combination methods can be applied to scalar features of functional markers after dimension reduction to improve the diagnostic accuracy. Area under the receiver operating characteristic curve and Youden index are used to assess performances of various methods in numerical studies. We also analyzed the high- or low- hospital admissions due to respiratory diseases between 2010 and 2017 in Hong Kong by combining weather conditions and media information, which are regarded as functional markers. Finally, we provide an R function for convenient application.     

Confidence intervals for the symmetry point: an optimal cutpoint in continuous diagnostic tests

Continuous diagnostic tests are often used for discriminating between healthy and diseased populations. For this reason, it is useful to select an appropriate discrimination threshold. There are several optimality criteria: the North‐West corner, the Youden index, the concordance probability and the symmetry point, among others. In this paper, we focus on the symmetry point that maximizes simultaneously the two types of correct classifications. We construct confidence intervals for this optimal cutpoint and its associated specificity and sensitivity indexes using two approaches: one based on the generalized pivotal quantity and the other on empirical likelihood. We perform a simulation study to check the practical behaviour of both methods and illustrate their use by means of three real biomedical datasets on melanoma, prostate cancer and coronary artery disease. Copyright © 2016 John Wiley & Sons, Ltd.     

Construction of Youden hyperrectangles

Youden hyperrectangles are higher-dimensional generalizations of balanced block designs and generalized Youden designs. This kind of design has been shown to be optimal for the elimination of multi-way heterogeneity. In this paper, patchwork and geometric methods are combined to construct Youden hyperrectangles for many parameter values.     

Confidence intervals for the difference in paired Youden indices

Comparison of accuracy between two diagnostic tests can be implemented by investigating the difference in paired Youden indices. However, few literature articles have discussed the inferences for the difference in paired Youden indices. In this paper, we propose an exact confidence interval for the difference in paired Youden indices based on the generalized pivotal quantities. For comparison, the maximum likelihood estimate‐based interval and a bootstrap‐based interval are also included in the study for the difference in paired Youden indices. Abundant simulation studies are conducted to compare the relative performance of these intervals by evaluating the coverage probability and average interval length. Our simulation results demonstrate that the exact confidence interval outperforms the other two intervals even with small sample size when the underlying distributions are normal. A real application is also used to illustrate the proposed intervals. Copyright © 2012 John Wiley & Sons, Ltd.     

Sample size determination and treatment screening in two‐stage phase II clinical trials via ROC curve

In pharmaceutical‐related research, we usually use clinical trials methods to identify valuable treatments and compare their efficacy with that of a standard control therapy. Although clinical trials are essential for ensuring the efficacy and postmarketing safety of a drug, conducting clinical trials is usually costly and time‐consuming. Moreover, to allocate patients to the little therapeutic effect treatments is inappropriate due to the ethical and cost imperative. Hence, there are several 2‐stage designs in the literature where, for reducing cost and shortening duration of trials, they use the conditional power obtained from interim analysis results to appraise whether we should continue the lower efficacious treatments in the next stage. However, there is a lack of discussion about the influential impacts on the conditional power of a trial at the design stage in the literature. In this article, we calculate the optimal conditional power via the receiver operating characteristic curve method to assess the impacts on the quality of a 2‐stage design with multiple treatments and propose an optimal design using the minimum expected sample size for choosing the best or promising treatment(s) among several treatments under an optimal conditional power constraint. In this paper, we provide tables of the 2‐stage design subject to optimal conditional power for various combinations of design parameters and use an example to illustrate our methods.     

Choosing between the BP and BN sequential strategies

Cost and burden of diagnostic testing may be reduced if fewer tests can be applied. Sequential testing involves selecting a sequence of tests, but only administering subsequent tests dependent on results of previous tests. This research provides guidance to choosing between single tests or the believe the positive (BP) and believe the negative (BN) sequential testing strategies, using accuracy (as measured by the Youden Index) as the primary determinant. Approximately 75% of the parameter combinations examined resulted in either BP or BN being recommended based on a higher accuracy at the optimal point. In about half of the scenarios BP was preferred, and the other half, BN, with the choice often a function of the value of the ratio of standard deviations of those without and with disease (b). Large values of b for the first test of the sequence tended to be associated with preference for BN as opposed to BP, while small values of b appear to favor BP. When there was no preference between sequences and/or single tests based on the Youden Index, cost of the sequence was considered. In this case, disease prevalence plays a large role in the selection of strategies, with lower values favoring BN and sometimes higher values favoring BP. The cost threshold for the sequential strategy to be preferred over a single, more accurate test, was often quite high. It appears that while sequential strategies most often increase diagnostic accuracy over a single test, sequential strategies are not always preferred.     

Weighted product index and its two-independent-sample comparison based on weighted sensitivity and specificity

Background: On the basis of statistical methods about index S (S = SEN × SPE), we develop a new weighted ways (weighted product index S_w) of combining sensitivity and specificity with user-defined weights. Methods: The new weighted product index S_w is defined as S_w = (SEN) (Youden 1950)^2w × (SPE) (Youden 1950) ^2(1?w) Results: For the large sample, the test statistics Z of two-independent-sample weighted product indices can either be a monotonous increasing/decreasing function or a no-monotonous function of weight w. Type I error of this statistics can be guaranteed close to the nominal level of 5%, which is more conservative than the weighted Youden index from simulation.     

The family of t-designs—part I

The family of t-designs is, without any doubt, the most important family of statistical designs. Their importance is due to their statistical optimalities, desirable symmetries for analyses and interpretations, and uses for constructing other important designs and structures such as Youden designs, generalized Youden designs, optimal fractional factorial designs, error defecting and correcting binary codes, balanced arrays, combinatorial filing systems, Hadamard matrices, finite projective and affine planes, strongly regular graphs, and so on. Research in the area of t-designs has been steadily and rapidly growing, especially during the last three decades. The number of publications in this area is in the several hundreds. Since papers on t-designs are published in a variety of journals, and because of the extensive role of these designs in design of experiments and other areas we believe it is imperative to gather these results and present them in varied form to suit diverse interests. This paper is an instance of such an attempt.     

Evaluation of incomplete multiple diagnostic tests,with an application in the colon cancer family registry study

Accurate diagnosis of a molecularly defined subtype of cancer is often an important step toward its effective control and treatment. For the diagnosis of some subtypes of a cancer, a gold standard with perfect sensitivity and specificity may be unavailable. In those scenarios, tumor subtype status is commonly measured by multiple imperfect diagnostic markers. Additionally, in many such studies, some subjects are only measured by a subset of diagnostic tests and the missing probabilities may depend on the unknown disease status. In this paper, we present statistical methods based on the EM algorithm to evaluate incomplete multiple imperfect diagnostic tests under a missing at random assumption and one missing not at random scenario. We apply the proposed methods to a real data set from the National Cancer Institute (NCI) colon cancer family registry on diagnosing microsatellite instability for hereditary non-polyposis colorectal cancer to estimate diagnostic accuracy parameters (i.e. sensitivities and specificities), prevalence, and potential differential missing probabilities for 11 biomarker tests. Simulations are also conducted to evaluate the small-sample performance of our methods.     

Detecting diagnostic accuracy of two biomarkers through a bivariate log-normal ROC curve

In biomedical research, two or more biomarkers may be available for diagnosis of a particular disease. Selecting one single biomarker which ideally discriminate a diseased group from a healthy group is confront in a diagnostic process. Frequently, most of the people use the accuracy measure, area under the receiver operating characteristic (ROC) curve to choose the best diagnostic marker among the available markers for diagnosis. Some authors have tried to combine the multiple markers by an optimal linear combination to increase the discriminatory power. In this paper, we propose an alternative method that combines two continuous biomarkers by direct bivariate modeling of the ROC curve under log-normality assumption. The proposed method is applied to simulated data set and prostate cancer diagnostic biomarker data set.     

Joint hypothesis testing of the area under the receiver operating characteristic curve and the Youden index

In the receiver operating characteristic (ROC) analysis, the area under the ROC curve (AUC ) serves as an overall measure of diagnostic accuracy. Another popular ROC index is the Youden index (J ), which corresponds to the maximum sum of sensitivity and specificity minus one. Since the AUC and J describe different aspects of diagnostic performance, we propose to test if a biomarker beats the pre-specified targeting values of AUC₀ and J₀ simultaneously with H₀ : AUC ≤ AUC₀ or J ≤ J₀ against H_a : AUC > AUC₀ and J > J₀ . This is a multivariate order restrictive hypothesis with a non-convex space in H_a , and traditional likelihood ratio-based tests cannot apply. The intersection–union test (IUT) and the joint test are proposed for such test. While the IUT test independently tests for the AUC and the Youden index, the joint test is constructed based on the joint confidence region. Findings from the simulation suggest both tests yield similar power estimates. We also illustrated the tests using a real data example and the results of both tests are consistent. In conclusion, testing jointly on AUC and J gives more reliable results than using a single index, and the IUT is easy to apply and have similar power as the joint test.     

An evaluation of common methods for dichotomization of continuous variables to discriminate disease status

Dichotomization of continuous variables to discriminate a dichotomous outcome is often useful in statistical applications. If a true threshold for a continuous variable exists, the challenge is identifying it. This paper examines common methods for dichotomization to identify which ones recover a true threshold. We provide mathematical and numeric proofs demonstrating that maximizing the odds ratio, Youden’s statistic, Gini Index, chi-square statistic, relative risk and kappa statistic all theoretically recover a true threshold. A simulation study evaluating the ability of these statistics to recover a threshold when sampling from a population indicates that maximizing the chi-square statistic and Gini Index have the smallest bias and variability when the probability of being larger than the threshold is small while maximizing Kappa or Youden’s statistics is best when this probability is larger. Maximizing odds ratio is the most variable and biased of the methods.     

Approximate theory of multiway block designs

Optimality properties of multiway block designs are deduced from the general results of J. Kiefer's approximate-design theory. In the model with additive effects these optimality properties solely depend on the two-dimensional marginals of the designs. Uniform designs, and designs whose two-dimensional marginals are products of the one-dimensional marginals, are shown to be optimal. Approximate Youden designs are introduced for the case when the support sets of the two-dimensional marginals are prescribed in advance. They are optimal in a relatively small class of competing designs only.     

基于均方根误差建模的多响应稳健参数设计

针对复杂产品质量设计阶段的静态多响应稳健参数设计中的模型参数不确定性问题,现有的多响应优化方法大都对响应的样本均值与样本方差采用双响应曲面法分别建模以考虑多响应的最优性与稳健性。文章在此基础上分析构建了均方根误差响应这一新的稳健性度量指标,提出了考虑模型参数不确定性的满意度函数方法,结合置信区间思想分析了模型参数不确定性对均方根误差响应的影响,并依据复杂产品质量设计阶段的实例进行分析研究,验证了该方法能够得到多响应系统在模型参数不确定性情况下更为稳健的全局最优解。     

A joint model of cancer incidence,metastasis, and mortality

Many diseases, especially cancer, are not static, but rather can be summarized by a series of events or stages (e.g. diagnosis, remission, recurrence, metastasis, death). Most available methods to analyze multi-stage data ignore intermediate events and focus on the terminal event or consider (time to) multiple events as independent. Competing-risk or semi-competing-risk models are often deficient in describing the complex relationship between disease progression events which are driven by a shared progression stochastic process. A multi-stage model can only examine two stages at a time and thus fails to capture the effect of one stage on the time spent between other stages. Moreover, most models do not account for latent stages. We propose a semi-parametric joint model of diagnosis, latent metastasis, and cancer death and use nonparametric maximum likelihood to estimate covariate effects on the risks of intermediate events and death and the dependence between them. We illustrate the model with Monte Carlo simulations and analysis of real data on prostate cancer from the SEER database.     

Block designs robust against the presence of positional effects

The problem considered is to find optimum designs for treatment effects in a block design (BD) setup, when positional effects are also present besides treatment and block effects, but they are ignored while formulating the model. In the class of symmetric balanced incomplete block designs, the Youden square design is shown to be optimal in the sense of minimizing the bias term in the mean squared error (MSE) of the best linear unbiased estimators of the full set of orthonormal treatment contrasts, irrespective of the value of the positional effects.     

A mixed double sampling plan based on Cpk

Abstract

Acceptance sampling plans are quality tools for the manufacturer and the customer. The ultimate result of reduction of nonconforming items will increase the profit of the manufacturer and enhance the satisfaction of the consumer. In this article, a mixed double sampling plan is proposed in which the attribute double sampling inspection is used in the first stage and a variables sampling plan based on the process capability index C_pk is used in the second stage. The optimal parameters are determined so that the producer’s and the consumer’s risks are to be satisfied with minimum average sample number. The optimal parameters of the proposed plan are estimated using different plan settings using two points on the operating characteristic curve approach. In designing the proposed mixed double sampling plan, we consider the symmetric and asymmetric nonconforming cases under variables inspection. The efficiency of the proposed plan is discussed and compared with the existing sampling plans. Tables are constructed for easy selection of the optimal plan parameters and an industrial example is also included for implementation of the proposed plan.