Reproducibility of clinical data II: categorical outcomes

In this second part of this paper, reproducibility of discrete ordinal and nominal outcomes is addressed. The first part deals with continuous outcomes, concentrating on intraclass correlation (ρ) in the context of one‐way analysis of variance. For categorical data, the focus has generally not been on a meaningful population parameter such as ρ. However, intraclass correlation has been defined for discrete ordinal data, ρ_c, and for nominal data, κ_I. Therefore, a unified approach to reproducibility is proposed. The relevance of these parameters is outlined. Estimation and inferential procedures for ρ_c and κ_I are reviewed, together with worked examples. Topics related to reproducibility that are not addressed in either this or the previous paper are highlighted. Considerations for designing reproducibility studies and for interpreting their results are provided. Copyright © 2004 John Wiley & Sons, Ltd.     

A robust permutation test for the concordance correlation coefficient

In this work, we developed a robust permutation test for the concordance correlation coefficient (ρ_c) for testing the general hypothesis H₀ : ρ_c = ρ_c(0). The proposed test is based on an appropriately studentized statistic. Theoretically, the test is proven to be asymptotically valid in the general setting when two paired variables are uncorrelated but dependent. This desired property was demonstrated across a range of distributional assumptions and sample sizes in simulation studies, where the test exhibits robust type I error control in all settings tested, even when the sample size is small. We demonstrated the application of this test in two real world examples across cardiac output measurements and endocardiographic imaging.     

Estimation of Correlation for Continuous Semimartingales

Abstract. In this study we are concerned with inference on the correlation parameter ρ of two Brownian motions, when only high‐frequency observations from two one‐dimensional continuous Itô semimartingales, driven by these particular Brownian motions, are available. Estimators for ρ are constructed in two situations: either when both components are observed (at the same time), or when only one component is observed and the other one represents its volatility process and thus has to be estimated from the data as well. In the first case it is shown that our estimator has the same asymptotic behaviour as the standard one for i.i.d. normal observations, whereas a feasible estimator can still be defined in the second framework, but with a slower rate of convergence.     

The effects of missing serial effects and/or heteroscedastic errors on mixed models using repeated growth data

When a two-level multilevel model (MLM) is used for repeated growth data, the individuals constitute level 2 and the successive measurements constitute level 1, which is nested within the individuals that make up level 2. The heterogeneity among individuals is represented by either the random-intercept or random-coefficient (slope) model. The variance components at level 1 involve serial effects and measurement errors under constant variance or heteroscedasticity. This study hypothesizes that missing serial effects or/and heteroscedasticity may bias the results obtained from two-level models. To illustrate this effect, we conducted two simulation studies, where the simulated data were based on the characteristics of an empirical mouse tumour data set. The results suggest that for repeated growth data with constant variance (measurement error) and misspecified serial effects (ρ > 0.3), the proportion of level-2 variation (intra-class correlation coefficient) increases with ρ and the two-level random-coefficient model is the minimum AIC (or AIC_c) model when compared with the fixed model, heteroscedasticity model, and random-intercept model. In addition, the serial effect (ρ > 0.1) and heteroscedasticity are both misspecified, implying that the two-level random-coefficient model is the minimum AIC (or AIC_c) model when compared with the fixed model and random-intercept model. This study demonstrates that missing serial effects and/or heteroscedasticity may indicate heterogeneity among individuals in repeated growth data (mixed or two-level MLM). This issue is critical in biomedical research.     

Variance-stabilizing and Confidence-stabilizing Transformations for the Normal Correlation Coefficient with Known Variances

Fosdick and Raftery (2012) recently encountered the problem of inference for a bivariate normal correlation coefficient ρ with known variances. We derive a variance-stabilizing transformation y(ρ) analogous to Fisher’s classical z-transformation for the unknown-variance case. Adjusting y for the sample size n produces an improved “confidence-stabilizing” transformation y_n(ρ) that provides more accurate interval estimates for ρ than the known-variance MLE. Interestingly, the z transformation applied to the unknown-but-equal-variance MLE performs well in the known-variance case for smaller values of |ρ|. Both methods are useful for comparing two or more correlation coefficients in the known-variance case.     

On Small Sample Properties of the Wald,LR and LM Tests in a Linear Model with AR(1) Errors

When the error terms are autocorrelated, the conventional t-tests for individual regression coefficients mislead us to over-rejection of the null hypothesis. We examine, by Monte Carlo experiments, the small sample properties of the unrestricted estimator of ρ and of the estimator of ρ restricted by the null hypothesis. We compare the small sample properties of the Wald, likelihood ratio and Lagrange multiplier test statistics for individual regression coefficients. It is shown that when the null hypothesis is true, the unrestricted estimator of ρ is biased. It is also shown that the Lagrange multiplier test using the maximum likelihood estimator of ρ performs better than the Wald and likelihood ratio tests.     

Efficiency and Validity Analyses of Two-Stage Estimation Procedures and Derived Testing Procedures in Quantitative Linear Models with AR(1) Errors

Abstract

In a quantitative linear model with errors following a stationary Gaussian, first-order autoregressive or AR(1) process, Generalized Least Squares (GLS) on raw data and Ordinary Least Squares (OLS) on prewhitened data are efficient methods of estimation of the slope parameters when the autocorrelation parameter of the error AR(1) process, ρ, is known. In practice, ρ is generally unknown. In the so-called two-stage estimation procedures, ρ is then estimated first before using the estimate of ρ to transform the data and estimate the slope parameters by OLS on the transformed data. Different estimators of ρ have been considered in previous studies. In this article, we study nine two-stage estimation procedures for their efficiency in estimating the slope parameters. Six of them (i.e., three noniterative, three iterative) are based on three estimators of ρ that have been considered previously. Two more (i.e., one noniterative, one iterative) are based on a new estimator of ρ that we propose: it is provided by the sample autocorrelation coefficient of the OLS residuals at lag 1, denoted r(1). Lastly, REstricted Maximum Likelihood (REML) represents a different type of two-stage estimation procedure whose efficiency has not been compared to the others yet. We also study the validity of the testing procedures derived from GLS and the nine two-stage estimation procedures. Efficiency and validity are analyzed in a Monte Carlo study. Three types of explanatory variable x in a simple quantitative linear model with AR(1) errors are considered in the time domain: Case 1, x is fixed; Case 2, x is purely random; and Case 3, x follows an AR(1) process with the same autocorrelation parameter value as the error AR(1) process. In a preliminary step, the number of inadmissible estimates and the efficiency of the different estimators of ρ are compared empirically, whereas their approximate expected value in finite samples and their asymptotic variance are derived theoretically. Thereafter, the efficiency of the estimation procedures and the validity of the derived testing procedures are discussed in terms of the sample size and the magnitude and sign of ρ. The noniterative two-stage estimation procedure based on the new estimator of ρ is shown to be more efficient for moderate values of ρ at small sample sizes. With the exception of small sample sizes, REML and its derived F-test perform the best overall. The asymptotic equivalence of two-stage estimation procedures, besides REML, is observed empirically. Differences related to the nature, fixed or random (uncorrelated or autocorrelated), of the explanatory variable are also discussed.     

Statistical methods for continuous outcomes in partially clustered designs

We address statistical issues involved in the partially clustered design where clusters are only employed in the intervention arm, but not in the control arm. We develop a cluster adjusted t-test to compare group treatment effects with individual treatment effects for continuous outcomes in which the individual level data are used as the unit of the analysis in both arms, we develop an approach for determining sample sizes using this cluster adjusted t-test, and use simulation to demonstrate the consistent accuracy of the proposed cluster adjusted t-test and power estimation procedures. Two real examples illustrate how to use the proposed methods.     

Some Statistical Methods in Assessing Linear Relational Agreement: An Empirical Comparative Study

Although multiple indices were introduced in the area of agreement measurements, the only documented index for linear relational agreement, which is for interval scale data, is the Pearson product-moment correlation coefficient. Despite its meaningfulness, the Pearson product-moment correlation coefficient does not convey the practical information such as what proportion of observations is within a certain boundary of the target value. To address this need, based on the inverse regression, we proposed the adjusted mean squared deviation (AMSD), adjusted coverage probability (ACP), and adjusted total deviation index (ATDI) for the measurement of the relational agreement. They can serve as reasonable and practically meaningful measurements for relational agreement. Real life data are considered to illustrate the performance of the methods.     

Sunset Salvo

The Wilcoxon—Mann—Whitney test enjoys great popularity among scientists comparing two groups of observations, especially when measurements made on a continuous scale are non-normally distributed. Triggered by different results for the procedure from two statistics programs, we compared the outcomes from 11 PC-based statistics packages. The findings were that the delivered p values ranged from significant to nonsignificant at the 5% level, depending on whether a large-sample approximation or an exact permutation form of the test was used and, in the former case, whether or not a correction for continuity was used and whether or not a correction for ties was made. Some packages also produced pseudo-exact p values, based on the null distribution under the assumption of no ties. A further crucial point is that the variant of the algorithm used for computation by the packages is rarely indicated in the output or documented in the Help facility and the manuals. We conclude that the only accurate form of the Wilcoxon—Mann—Whitney procedure is one in which the exact permutation null distribution is compiled for the actual data.     

Parallel, AA/BB, AB/BA and Balaam's design: efficient and maximin choices when testing the treatment effect in a mixed effects linear regression

When examining the effect of treatment A versus B, there may be a choice between a parallel group design, an AA/BB design, an AB/BA cross‐over and Balaam's design. In case of a linear mixed effects regression, it is examined, starting from a flexible function of the costs involved and allowing for subject dropout, which design is most efficient in estimating this effect. For no carry‐over, the AB/BA cross‐over design is most efficient as long as the dropout rate at the second measurement does not exceed 2ρ/(1 + ρ), ρ being the intraclass correlation. For steady‐state carry‐over, depending on the costs involved, the dropout rate and ρ, either a parallel design or an AA/BB design is most efficient. For types of carry‐over that allow for self carry‐over, interest is in the direct treatment effect plus the self carry‐over effect, with either an AA/BB or Balaam's design being most efficient. In case of insufficient knowledge on the dropout rate or ρ, a maximin strategy is devised: choose the design that minimizes the maximum variance of the treatment estimator. Such maximin designs are derived for each type of carry‐over. Copyright © 2012 John Wiley & Sons, Ltd.     

Hypothesis Testing Problems in an Unbalanced Longitudinal Ophthalmology Study

ABSTRACT

The analysis of clustered data in a longitudinal ophthalmology study is complicated by correlations between repeatedly measured visual outcomes of paired eyes in a participant and missing observations due to the loss of follow-up. In the present article we consider hypothesis testing problems in an ophthalmology study, where eligible eyes are randomized to two treatments (when two eyes of a participant are eligible, the paired eyes are assigned to different treatments), and vision function outcomes are repeatedly measured over time. A large sample-based nonparametric test statistic and a nonparametric Bootstrap test analog are proposed for testing an interaction effect of two factors and testing an effect of a eye-specific factor within a level of the other person-specific factor on visual function outcomes. Both test statistics allow for missing observations, correlations between repeatedly measured outcomes on individual eyes, and correlations between repeatedly measured outcomes on both eyes of each participant. A simulation study shows that these proposed test statistics maintain nominal significance levels approximately and comparable powers to each other, as well as higher powers than the naive test statistic ignoring correlations between repeated bilateral measurements of both eyes in the same person. For illustration, we apply the proposed test statistics to the changes of visual field defect score in the Advanced Glaucoma Intervention Study.     

Detecting a Change in the Correlation Coefficient in a Sequence of Bivariate Normal Variables

We consider the problem of testing the hypothesis that the correlation coefficient is stable in a sequence of n observations of independent, bivariate normal random variables against the alternative that the correlation coefficient changes after an unknown point t(t < n). We propose an estimate of the changepoint t and report on power comparisons between the commonly used test for this problem and our proposed test. Some applications to finance are discussed.     

Simple moment-based inferences of generalized concordance correlation

We proposed two simple moment-based procedures, one with (GCCC1) and one without (GCCC2) normality assumptions, to generalize the inference of concordance correlation coefficient for the evaluation of agreement among multiple observers for measurements on a continuous scale. A modified Fisher's Z-transformation was adapted to further improve the inference. We compared the proposed methods with U-statistic-based inference approach. Simulation analysis showed desirable statistical properties of the simplified approach GCCC1, in terms of coverage probabilities and coverage balance, especially for small samples. GCCC2, which is distribution-free, behaved comparably with the U-statistic-based procedure, but had a more intuitive and explicit variance estimator. The utility of these approaches were illustrated using two clinical data examples.     

The function space D([O, ∞)ρ, E)

The Skorokhod topology is extended to the function space D([0, ∞)ρ, E) of functions, from [0, ∞)ρ to a complete separable metric space E, which are “continuous from above with limits from below. Criteria for tightness are developed. The case in which E is a product space is considered, and conditions under which tightness may be proven componentwise are given. Various applications are studied, including a multidimensional version of Donsker's Theorem, and a functional Central Limit Theorem for a multitype Poisson cluster process.     

On The Role of a Certain Eigenvalue in Estimating the Growth Rate of a Branching Process

In many situations, the data given on a p-type Galton-Watson process Z_n eP N^p will consist of the total generation sizes |Z_n| only. In that case, the maximum likelihood estimator ρ_ML of the growth rate ρ is not observable, and the asymptotic properties of the most obvious estimators of ρ based on the |Z_n|, as studied by Asmussen & Keiding (1978), show a crucial dependence on |ρ₁|,ρ₁ being a certain other eigenvalue of the offspring mean matrix. In fact, if |ρ₁|²≤ρ, then the speed of convergence compares badly with ρ_ML. In the present note, it is pointed out that recent results of Heyde (1981) on so-called Fibonacci branching processes provide further examples of this phenomenon, and an estimator with the same speed of convergence as ρ_ML and based on the |Z_n| alone is exhibited for the case p= 2, ρ₁²≥ρ.     

A weighted least-squares approach to clusterwise regression

Clusterwise regression aims to cluster data sets where the clusters are characterized by their specific regression coefficients in a linear regression model. In this paper, we propose a method for determining a partition which uses an idea of robust regression. We start with some random weighting to determine a start partition and continue in the spirit of M-estimators. The residuals for all regressions are used to assign the observations to the different groups. As target function we use the determination coefficient R²_wR^{2}_{w} for the overall model. This coefficient is suitably defined for weighted regression.     

A Bivariate Distribution Family with Specified Correlation Coefficient and Given Marginals

For given continuous distribution functions F(x) and G(y) and a Pearson correlation coefficient ρ, an algorithm is provided to construct a sequence of continuous bivariate distributions with marginals equal to F(x) and G(y) and the corresponding correlation coefficient converges to ρ. The algorithm can be easily implemented using S-Plus or R. Applications are given to generate bivariate random variables with marginals including Gamma, Beta, Weibull, and uniform distributions.     

Quality Control for Scientific Research: Addressing Reproducibility,Responsiveness, and Relevance

ABSTRACT

Efforts to address a reproducibility crisis have generated several valid proposals for improving the quality of scientific research. We argue there is also need to address the separate but related issues of relevance and responsiveness. To address relevance, researchers must produce what decision makers actually need to inform investments and public policy—that is, the probability that a claim is true or the probability distribution of an effect size given the data. The term responsiveness refers to the irregularity and delay in which issues about the quality of research are brought to light. Instead of relying on the good fortune that some motivated researchers will periodically conduct efforts to reveal potential shortcomings of published research, we could establish a continuous quality-control process for scientific research itself. Quality metrics could be designed through the application of this statistical process control for the research enterprise. We argue that one quality control metric—the probability that a research hypothesis is true—is required to address at least relevance and may also be part of the solution for improving responsiveness and reproducibility. This article proposes a “straw man” solution which could be the basis of implementing these improvements. As part of this solution, we propose one way to “bootstrap” priors. The processes required for improving reproducibility and relevance can also be part of a comprehensive statistical quality control for science itself by making continuously monitored metrics about the scientific performance of a field of research.     

Nonparametric estimation of copula-based measures of multivariate association from contingency tables

Nonparametric estimation of copula-based measures of multivariate association in a continuous random vector X=(X₁, …, X_d) is usually based on complete continuous data. In many practical applications, however, these types of data are not readily available; instead aggregated ordinal observations are given, for example, ordinal ratings based on a latent continuous scale. This article introduces a purely nonparametric and data-driven estimator of the unknown copula density and the corresponding copula based on multivariate contingency tables. Estimators for multivariate Spearman's rho and Kendall's tau are based thereon. The properties of these estimators in samples of medium and large size are evaluated in a simulation study. An increasing bias can be observed along with an increasing degree of association between the components. As it is to be expected, the bias is severely influenced by the amount of information available. Additionally, the influence of sample size is only marginal. We further give an empirical illustration based on daily returns of five German stocks.