Weighted kappa as a function of unweighted kappas

The kappa coefficient is a widely used measure for assessing agreement on a nominal scale. Weighted kappa is an extension of Cohen's kappa that is commonly used for measuring agreement on an ordinal scale. In this article, it is shown that weighted kappa can be computed as a function of unweighted kappas. The latter coefficients are kappa coefficients that correspond to smaller contingency tables that are obtained by merging categories.     

Four Correlation Coefficients with a Third Blocking Variable: Their Efficacy,Relative Efficiency,and Test Statistics

Abstract

The efficacy and the asymptotic relative efficiency (ARE) of a weighted sum of Kendall's taus, a weighted sum of Spearman's rhos, a weighted sum of Pearson's r's, and a weighted sum of z-transformation of the Fisher–Yates correlation coefficients, in the presence of a blocking variable, are discussed. The method of selecting the weighting constants that maximize the efficacy of these four correlation coefficients is proposed. The estimate, test statistics and confidence interval of the four correlation coefficients with weights are also developed. To compare the small-sample properties of the four tests, a simulation study is performed. The theoretical and simulated results all prefer the weighted sum of the Pearson correlation coefficients with the optimal weights, as well as the weighted sum of z-transformation of the Fisher–Yates correlation coefficients with the optimal weights.     

The challenges of evaluating dose response in flexible‐dose trials using marginal structural models

Assessing dose response from flexible‐dose clinical trials is problematic. The true dose effect may be obscured and even reversed in observed data because dose is related to both previous and subsequent outcomes. To remove selection bias, we propose marginal structural models, inverse probability of treatment‐weighting (IPTW) methodology. Potential clinical outcomes are compared across dose groups using a marginal structural model (MSM) based on a weighted pooled repeated measures analysis (generalized estimating equations with robust estimates of standard errors), with dose effect represented by current dose and recent dose history, and weights estimated from the data (via logistic regression) and determined as products of (i) inverse probability of receiving dose assignments that were actually received and (ii) inverse probability of remaining on treatment by this time. In simulations, this method led to almost unbiased estimates of true dose effect under various scenarios. Results were compared with those obtained by unweighted analyses and by weighted analyses under various model specifications. The simulation showed that the IPTW MSM methodology is highly sensitive to model misspecification even when weights are known. Practitioners applying MSM should be cautious about the challenges of implementing MSM with real clinical data. Clinical trial data are used to illustrate the methodology. Copyright © 2012 John Wiley & Sons, Ltd.     

A bootstrap method for comparing correlated kappa coefficients

Cohen's kappa coefficient is traditionally used to quantify the degree of agreement between two raters on a nominal scale. Correlated kappas occur in many settings (e.g., repeated agreement by raters on the same individuals, concordance between diagnostic tests and a gold standard) and often need to be compared. While different techniques are now available to model correlated κ coefficients, they are generally not easy to implement in practice. The present paper describes a simple alternative method based on the bootstrap for comparing correlated kappa coefficients. The method is illustrated by examples and its type I error studied using simulations. The method is also compared with the generalized estimating equations of the second order and the weighted least-squares methods.     

Symmetric kappa as a function of unweighted kappas

It is shown that a symmetric kappa corresponding to a c × c table with c ? 3 categories can be written as a function of the unweighted kappa corresponding to the same table and the c(c ? 1)/2 distinct unweighted kappas associated with the (c ? 1) × (c ? 1) tables that are obtained by combining two categories. The result is a new MGB-type result.     

ON THE EQUIVALENCE OF SOME INDICES OF SIMILARITY: IMPLICATION FOR BINARY PRESENCE/ABSENCE DATA

Cohen’s kappa, a special case of the weighted kappa, is a chance‐corrected index used extensively to quantify inter‐rater agreement in validation and reliability studies. In this paper, it is shown that in inter‐rater agreement for 2 × 2 tables, for two raters having the same number of opposite ratings, the weighted kappa, Cohen’s kappa, Peirce, Yule, Maxwell and Pilliner and Fleiss indices are identical. This implies that the weights in the weighted kappa are less important under such assumptions. Equivalently, it is shown that for two partitions of the same data set, resulting from two clustering algorithms having the same number of clusters with equal cluster sizes, these similarity indices are identical. Hence, an important characterisation is formulated relating equal numbers of clusters with the same cluster sizes to the presence/absence of a trait in a reliability study. Two numerical examples that exemplify the implication of this relationship are presented.     

Weighted combinations of rank tests for location-scale alternatives

All existing location-scale rank tests use equal weights for the components. We advocate the use of weighted combinations of statistics. This approach can partly be substantiated by the theory of locally most powerful tests. We specifically investi= gate a Wilcoxon-Mood combination. We give exact critical values for a range of weights. The asymptotic normality of the test statistic is proved under a general hypothesis and Chernoff-Savage conditions. The asymptotic relative efficiency of this test with respect to unweighted combinations shows that a careful choice of weights results in a gain in efficiency.     

Asymptotic hypothesis test to simultaneously compare the weighted kappa coefficients of multiple binary diagnostic tests in the presence of ignorable missing data

The weighted kappa coefficient of a binary diagnostic test is a measure of the beyond-chance agreement between the diagnostic test and the gold standard, and is a measure that allows us to assess and compare the performance of binary diagnostic tests. In the presence of partial disease verification, the comparison of the weighted kappa coefficients of two or more binary diagnostic tests cannot be carried out ignoring the individuals with an unknown disease status, since the estimators obtained would be affected by verification bias. In this article, we propose a global hypothesis test based on the chi-square distribution to simultaneously compare the weighted kappa coefficients when in the presence of partial disease verification the missing data mechanism is ignorable. Simulation experiments have been carried out to study the type I error and the power of the global hypothesis test. The results have been applied to the diagnosis of coronary disease.     

Some Related Minima Stability and Minima Infinite Divisibility of the General Multivariate Pareto Distributions

This article studies the minima stable property of the general multivariate Pareto distributions MP^(k)(I), MP^(k)(II), MP^(k)(III), MP^(k)(IV) which can be applied to characterize the MP^(k) distribution via its weighted ordered coordinates minima and marginal distribution. Also, the multivariate semi-Pareto distribution (denoted by MSP) is discerned in the class of geometric minima infinite divisible and geometric minima stable distributions. If the exponent measure is satisfied by some functional equation, then the geometric minima stable property can be used to characterize the MSP distribution. Finally, the finite sample minima infinite divisible property of the MP^(k)(I), (II), and (IV) distributions is also discussed.     

Linear hypothesis testing for weighted functional data with applications

In socioeconomic areas, functional observations may be collected with weights, called weighted functional data. In this paper, we deal with a general linear hypothesis testing (GLHT) problem in the framework of functional analysis of variance with weighted functional data. With weights taken into account, we obtain unbiased and consistent estimators of the group mean and covariance functions. For the GLHT problem, we obtain a pointwise F-test statistic and build two global tests, respectively, via integrating the pointwise F-test statistic or taking its supremum over an interval of interest. The asymptotic distributions of test statistics under the null and some local alternatives are derived. Methods for approximating their null distributions are discussed. An application of the proposed methods to density function data is also presented. Intensive simulation studies and two real data examples show that the proposed tests outperform the existing competitors substantially in terms of size control and power.     

Concordance coefficients to measure the agreement among several sets of ranks

In this paper, two measures of agreement among several sets of ranks, Kendall's concordance coefficient and top-down concordance coefficient, are reviewed. In order to illustrate the utility of these measures, two examples, in the fields of health and sports, are presented. A Monte Carlo simulation study was carried out to compare the performance of Kendall's and top-down concordance coefficients in detecting several types and magnitudes of agreements. The data generation scheme was developed in order to induce an agreement with different intensities among m (m>2) sets of ranks in non-directional and directional rank agreement scenarios. The performance of each coefficient was estimated by the proportion of rejected null hypotheses, assessed at 5% significance level, when testing whether the underlying population concordance coefficient is sufficiently greater than zero. For the directional rank agreement scenario, the top-down concordance coefficient allowed to achieve a percentage of significant concordances that was higher than the one achieved by Kendall's concordance coefficient. Mainly, when the degree of agreement was small, the results of the simulation study pointed to the advantage of using a weighted rank concordance, namely the top-down concordance coefficient, simultaneously with Kendall's concordance coefficient, enabling the detection of agreement (in a top-down sense) in situations not detected by Kendall's concordance coefficient.     

Robust extreme ranked set sampling

In this paper, a robust extreme ranked set sampling (RERSS) procedure for estimating the population mean is introduced. It is shown that the proposed method gives an unbiased estimator with smaller variance, provided the underlying distribution is symmetric. However, for asymmetric distributions a weighted mean is given, where the optimal weights are computed by using Shannon's entropy. The performance of the population mean estimator is discussed along with its properties. Monte Carlo simulations are used to demonstrate the performance of the RERSS estimator relative to the simple random sample (SRS), ranked set sampling (RSS) and extreme ranked set sampling (ERSS) estimators. The results indicate that the proposed estimator is more efficient than the estimators based on the traditional sampling methods.     

Multivariate hypothesis testing using generalized and {2}-inverses – with applications

The use of generalized inverses in Wald's-type quadratic forms of test statistics having singular normal limiting distributions does not guarantee to obtain chi-square limiting distributions. In this article, the use of {2} -inverses for that problem is investigated. Alternatively, Imhof-based test statistics can also be defined, which converge in distribution to weighted sum of chi-square variables. The asymptotic distributions of these test statistics under the null and alternative hypotheses are discussed. Under fixed and local alternatives, the asymptotic powers are compared theoretically. Simulation studies are also performed to compare the exact powers of the test statistics in finite samples. A data analysis on the temperature and precipitation variability in the European Alps illustrates the proposed methods.     

Comparison of clustering algorithms on generalized propensity score in observational studies: a simulation study

In observational studies, unbalanced observed covariates between treatment groups often cause biased inferences on the estimation of treatment effects. Recently, generalized propensity score (GPS) has been proposed to overcome this problem; however, a practical technique to apply the GPS is lacking. This study demonstrates how clustering algorithms can be used to group similar subjects based on transformed GPS. We compare four popular clustering algorithms: k-means clustering (KMC), model-based clustering, fuzzy c-means clustering and partitioning around medoids based on the following three criteria: average dissimilarity between subjects within clusters, average Dunn index and average silhouette width under four various covariate scenarios. Simulation studies show that the KMC algorithm has overall better performance compared with the other three clustering algorithms. Therefore, we recommend using the KMC algorithm to group similar subjects based on the transformed GPS.     

Regression modelling of weighted κ by using generalized estimating equations

In many clinical studies more than one observer may be rating a characteristic measured on an ordinal scale. For example, a study may involve a group of physicians rating a feature seen on a pathology specimen or a computer tomography scan. In clinical studies of this kind, the weighted κ coefficient is a popular measure of agreement for ordinally scaled ratings. Our research stems from a study in which the severity of inflammatory skin disease was rated. The investigators wished to determine and evaluate the strength of agreement between a variable number of observers taking into account patient-specific (age and gender) as well as rater-specific (whether board certified in dermatology) characteristics. This suggested modelling κ as a function of these covariates. We propose the use of generalized estimating equations to estimate the weighted κ coefficient. This approach also accommodates unbalanced data which arise when some subjects are not judged by the same set of observers. Currently an estimate of overall κ for a simple unbalanced data set without covariates involving more than two observers is unavailable. In the inflammatory skin disease study none of the covariates were significantly associated with κ, thus enabling the calculation of an overall weighted κ for this unbalanced data set. In the second motivating example (multiple sclerosis), geographic location was significantly associated with κ. In addition we also compared the results of our method with current methods of testing for heterogeneity of weighted κ coefficients across strata (geographic location) that are available for balanced data sets.     

Interpretation of Kappa and B statistics measures of agreement

SUMMARY The Kappa statistic proposed by Cohen and the B statistic proposed by Bangdiwala are used to quantify the agreement between two observers, independently classifying the same n units into the same k categories. Both statistics correct for the agreement expected to result from chance alone, but the Kappa statistic is a measure that adjusts the observed proportion of agreement and ranges from- pc/(1- pc) to 1, where pc is the expected agreement that results from chance, and the B statistic is a measure that adjusts the observed area of agreement with that expected to result from chance, and ranges from 0 to 1. Statistical guidelines for the interpretation of either statistic are not available. For the Kappa statistic, the suggested arbitrary interpretation given by Landis and Koch is commonly quoted. This paper compares the behavior of the Kappa statistic and the B statistic in 3 3 and 4 4 contingency tables, under different agreement patterns. Based on simulation results, non-arbitrary guidelines for the interpretation of both statistics are provided.     

Remedying the Neyman–Scott phenomenon in model discrimination

The objective of this paper is to investigate through simulation the possible presence of the incidental parameters problem when performing frequentist model discrimination with stratified data. In this context, model discrimination amounts to considering a structural parameter taking values in a finite space, with k points, k≥2. This setting seems to have not yet been considered in the literature about the Neyman–Scott phenomenon. Here we provide Monte Carlo evidence of the severity of the incidental parameters problem also in the model discrimination setting and propose a remedy for a special class of models. In particular, we focus on models that are scale families in each stratum. We consider traditional model selection procedures, such as the Akaike and Takeuchi information criteria, together with the best frequentist selection procedure based on maximization of the marginal likelihood induced by the maximal invariant, or of its Laplace approximation. Results of two Monte Carlo experiments indicate that when the sample size in each stratum is fixed and the number of strata increases, correct selection probabilities for traditional model selection criteria may approach zero, unlike what happens for model discrimination based on exact or approximate marginal likelihoods. Finally, two examples with real data sets are given.     

Harmonic mean approach to unbalanced mixed effects models under heteroscedastic variances

In this paper we consider unbalanced mixed models (Scheffe's model) under heteroscedastic variances. By using the harmonic mean approach, It is shown that the problems appear to be anologous to those problems from balanced mixed models under homoscedastic variance. Thus, by using harmonic mean approach, statistical inferences about fixed effects and variance components are derived by using those from balanced models under homoscedastic variance. Laguerre polynomial expansion is used Lo approximate sampling distributions of relevant statistics.