Weighted kappa as a function of unweighted kappas

The kappa coefficient is a widely used measure for assessing agreement on a nominal scale. Weighted kappa is an extension of Cohen's kappa that is commonly used for measuring agreement on an ordinal scale. In this article, it is shown that weighted kappa can be computed as a function of unweighted kappas. The latter coefficients are kappa coefficients that correspond to smaller contingency tables that are obtained by merging categories.     

Behavior of agreement measures in the presence of zero cells and biased marginal distributions

Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k=4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.     

Asymptotic hypothesis test to simultaneously compare the weighted kappa coefficients of multiple binary diagnostic tests in the presence of ignorable missing data

The weighted kappa coefficient of a binary diagnostic test is a measure of the beyond-chance agreement between the diagnostic test and the gold standard, and is a measure that allows us to assess and compare the performance of binary diagnostic tests. In the presence of partial disease verification, the comparison of the weighted kappa coefficients of two or more binary diagnostic tests cannot be carried out ignoring the individuals with an unknown disease status, since the estimators obtained would be affected by verification bias. In this article, we propose a global hypothesis test based on the chi-square distribution to simultaneously compare the weighted kappa coefficients when in the presence of partial disease verification the missing data mechanism is ignorable. Simulation experiments have been carried out to study the type I error and the power of the global hypothesis test. The results have been applied to the diagnosis of coronary disease.     

Assessing and comparing the accuracy of various bootstrap methods

We evaluate the performance of various bootstrap methods for constructing confidence intervals for mean and median of several common distributions. Using Monte Carlo simulation, we assessed performance by looking at coverage percentages and average confidence interval lengths. Poor performance is characterized by coverage deviating from 0.95 and large confidence interval lengths. Undercoverage is of greater concern than overcoverage. We also assess the performance of bootstrap methods in estimating the parameters of the Cox Proportional Hazard model and Accelerated Failure Time model.     

On population‐based measures of agreement for binary classifications

The authors describe a model‐based kappa statistic for binary classifications which is interpretable in the same manner as Scott's pi and Cohen's kappa, yet does not suffer from the same flaws. They compare this statistic with the data‐driven and population‐based forms of Scott's pi in a population‐based setting where many raters and subjects are involved, and inference regarding the underlying diagnostic procedure is of interest. The authors show that Cohen's kappa and Scott's pi seriously underestimate agreement between experts classifying subjects for a rare disease; in contrast, the new statistic is robust to changes in prevalence. The performance of the three statistics is illustrated with simulations and prostate cancer data.     

A bootstrap test for time series linearity

A bootstrap algorithm is proposed for testing Gaussianity and linearity in stationary time series, and consistency of the relevant bootstrap approximations is proven rigorously for the first time. Subba Rao and Gabr (1980) and Hinich (1982) have formulated some well-known nonparametric tests for Gaussianity and linearity based on the asymptotic distribution of the normalized bispectrum. The proposed bootstrap procedure gives an alternative way to approximate the finite-sample null distribution of such test statistics. We revisit a modified form of Hinich's test utilizing kernel smoothing, and compare its performance to the bootstrap test on several simulated data sets and two real data sets—the S&P 500 returns and the quarterly US real GNP growth rate. Interestingly, Hinich's test and the proposed bootstrapped version yield substantially different results when testing Gaussianity and linearity of the GNP data.     

A bootstrap test for single index models

Single index models are frequently used in econometrics and biometrics. Logit and Probit models arc special cases with fixed link functions. In this paper we consider a bootstrap specification test that detects nonparametric deviations of the link function. The bootstrap is used with the aim to rind a more accurate distribution under the null than the normal approximation. We prove that the statistic and its bootstrapped version have the same asymptotic distribution. In a simulation study we show that the bootstrap is able to capture the negative bias and the skewness of the test statistic. It yields better approximations to the true critical values and consequently it has a more accurate level than the normal approximation.     

Comparison of bootstrap and generalized bootstrap methods for estimating high quantiles

The generalized bootstrap is a parametric bootstrap method in which the underlying distribution function is estimated by fitting a generalized lambda distribution to the observed data. In this study, the generalized bootstrap is compared with the traditional parametric and non-parametric bootstrap methods in estimating the quantiles at different levels, especially for high quantiles. The performances of the three methods are evaluated in terms of cover rate, average interval width and standard deviation of width of the 95% bootstrap confidence intervals. Simulation results showed that the generalized bootstrap has overall better performance than the non-parametric bootstrap in high quantile estimation.     

New bootstrap confidence intervals for means of positively skewed distributions

ABSTRACT

In this paper, we consider the problem of constructing non parametric confidence intervals for the mean of a positively skewed distribution. We suggest calibrated, smoothed bootstrap upper and lower percentile confidence intervals. For the theoretical properties, we show that the proposed one-sided confidence intervals have coverage probability α + O(n^{? 3/2}). This is an improvement upon the traditional bootstrap confidence intervals in terms of coverage probability. A version smoothed approach is also considered for constructing a two-sided confidence interval and its theoretical properties are also studied. A simulation study is performed to illustrate the performance of our confidence interval methods. We then apply the methods to a real data set.     

A control chart based on weighted bootstrap with strata

The study proposes a weighted bootstrap control chart based on the strata and determines suitable weights for each stratum by measuring the relative error of estimated Type I risks (Err). In addition to validating the detection effectiveness of the proposed control chart, this study also compares several types of control charts for detecting the Type I, Type II risks, and median run lengths. The proposed control chart shows superior detection effectiveness in six skewed distributions compared to other types of control charts.     

Adaptive resampling algorithms for estimating bootstrap distributions

Based on recent developments in the field of operations research, we propose two adaptive resampling algorithms for estimating bootstrap distributions. One algorithm applies the principle of the recently proposed cross-entropy (CE) method for rare event simulation, and does not require calculation of the resampling probability weights via numerical optimization methods (e.g., Newton's method), whereas the other algorithm can be viewed as a multi-stage extension of the classical two-step variance minimization approach. The two algorithms can be easily used as part of a general algorithm for Monte Carlo calculation of bootstrap confidence intervals and tests, and are especially useful in estimating rare event probabilities. We analyze theoretical properties of both algorithms in an idealized setting and carry out simulation studies to demonstrate their performance. Empirical results on both one-sample and two-sample problems as well as a real survival data set show that the proposed algorithms are not only superior to traditional approaches, but may also provide more than an order of magnitude of computational efficiency gains.     

Kernel estimations for multivariate density functional with bootstrap

In this article the bootstrap method is discussed for the kernel estimation of the multivariate density function. We have considered sample mean functional and constructed its consistency and asymptotic normality by bootstrap estimator. It has been shown that the bootstrap works for kernel estimates of multivariate density functional. The convergence rate with bootstrap for density has been proved. Finally, two simulations of application are given.     

Inferences on correlation coefficients: One-sample,independent and correlated cases

This article concerns inference on the correlation coefficients of a multivariate normal distribution. Inferential procedures based on the concepts of generalized variables (GVs) and generalized p$p$-values are proposed for elements of a correlation matrix. For the simple correlation coefficient, the merits of the generalized confidence limits and other approximate methods are evaluated using a numerical study. The study indicates that the proposed generalized confidence limits are uniformly most accurate even for samples as small as three. The results are extended for comparing two independent correlations, overlapping and non-overlapping dependent correlations. For each problem, the properties of the GV approach and other asymptotic methods are evaluated using Monte Carlo simulation. The GV approach produces satisfactory results for all the problems considered. The methods are illustrated using a few practical examples.     

Importance resampling for the smoothed bootstrap

Alternative methods of estimating properties of unknown distributions include the bootstrap and the smoothed bootstrap. In the standard bootstrap setting, Johns (1988) introduced an importance resam¬pling procedure that results in more accurate approximation to the bootstrap estimate of a distribution function or a quantile. With a suitable “exponential tilting” similar to that used by Johns, we derived a smoothed version of importance resampling in the framework of the smoothed bootstrap. Smoothed importance resampling procedures were developed for the estimation of distribution functions of the Studentized mean, the Studentized variance, and the correlation coefficient. Implementation of these procedures are presented via simulation results which concentrate on the problem of estimation of distribution functions of the Studentized mean and Studentized variance for different sample sizes and various pre-specified smoothing bandwidths for the normal data; additional simulations were conducted for the estimation of quantiles of the distribution of the Studentized mean under an optimal smoothing bandwidth when the original data were simulated from three different parent populations: lognormal, t(3) and t(10). These results suggest that in cases where it is advantageous to use the smoothed bootstrap rather than the standard bootstrap, the amount of resampling necessary might be substantially reduced by the use of importance resampling methods and the efficiency gains depend on the bandwidth used in the kernel density estimation.     

Parametric bootstrap inferences for unbalanced panel data models

This article presents parametric bootstrap (PB) approaches for hypothesis testing and interval estimation for the regression coefficients of panel data regression models with incomplete panels. Some simulation results are presented to compare the performance of the PB approaches with the approximate inferences. Our studies show that the PB approaches perform satisfactorily for various sample sizes and parameter configurations, and the performance of PB approaches is mostly better than the approximate methods with respect to the coverage probabilities and the Type I error rates. The PB inferences have almost exact coverage probabilities and Type I error rates. Furthermore, the PB procedure can be simply carried out by a few simulation steps, and the derivation is easier to understand and to be extended to the multi-way error component regression models with unbalanced panels. Finally, the proposed approaches are illustrated by using a real data example.     

A weighted bootstrap approximation for comparing the error distributions in nonparametric regression

Several procedures have been proposed for testing the equality of error distributions in two or more nonparametric regression models. Here we deal with methods based on comparing estimators of the cumulative distribution function (CDF) of the errors in each population to an estimator of the common CDF under the null hypothesis. The null distribution of the associated test statistics has been approximated by means of a smooth bootstrap (SB) estimator. This paper proposes to approximate their null distribution through a weighted bootstrap. It is shown that it produces a consistent estimator. The finite sample performance of this approximation is assessed by means of a simulation study, where it is also compared to the SB. This study reveals that, from a computational point of view, the proposed approximation is more efficient than the one provided by the SB.     

Measuring effect size: a robust heteroscedastic approach for two or more groups

Motivated by involvement in an intervention study, the paper proposes a robust, heteroscedastic generalization of what is popularly known as Cohen's d. The approach has the additional advantage of being readily extended to situations where the goal is to compare more than two groups. The method arises quite naturally from a regression perspective in conjunction with a robust version of explanatory power. Moreover, it provides a single numeric summary of how the groups compare in contrast to other strategies aimed at dealing with heteroscedasticity. Kulinskaya and Staudte [16 Rousseeuw, P. J. and Leroy, A. M. 1987. Robust Regression & Outlier Detection, New York: Wiley. [Crossref] [Google Scholar]] studied a heteroscedastic measure of effect size similar to the one proposed here, but their measure of effect size depends on the sample sizes making it difficult for applied researchers to interpret the results. The approach used here is based on a generalization of Cohen's d that obviates the issue of unequal sample sizes. Simulations and illustrations demonstrate that the new measure of effect size can make a practical difference regarding the conclusions reached.     

Comparing three bootstrap methods for survey data

Various bootstrap methods for variance estimation and confidence intervals in complex survey data, where sampling is done without replacement, have been proposed in the literature. The oldest, and perhaps the most intuitively appealing, is the without-replacement bootstrap (BWO) method proposed by Gross (1980). Unfortunately, the BWO method is only applicable to very simple sampling situations. We first introduce extensions of the BWO method to more complex sampling designs. The performance of the BWO and two other bootstrap methods, the rescaling bootstrap (Rao and Wu 1988) and the mirror-match bootstrap (Sitter 1992), are then compared through a simulation study. Together these three methods encompass the various bootstrap proposals.