Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models

ABSTRACT

In panel data models and other regressions with unobserved effects, fixed effects estimation is often paired with cluster-robust variance estimation (CRVE) to account for heteroscedasticity and un-modeled dependence among the errors. Although asymptotically consistent, CRVE can be biased downward when the number of clusters is small, leading to hypothesis tests with rejection rates that are too high. More accurate tests can be constructed using bias-reduced linearization (BRL), which corrects the CRVE based on a working model, in conjunction with a Satterthwaite approximation for t-tests. We propose a generalization of BRL that can be applied in models with arbitrary sets of fixed effects, where the original BRL method is undefined, and describe how to apply the method when the regression is estimated after absorbing the fixed effects. We also propose a small-sample test for multiple-parameter hypotheses, which generalizes the Satterthwaite approximation for t-tests. In simulations covering a wide range of scenarios, we find that the conventional cluster-robust Wald test can severely over-reject while the proposed small-sample test maintains Type I error close to nominal levels. The proposed methods are implemented in an R package called clubSandwich. This article has online supplementary materials.     

Approximate F-tests of multiple degree of freedom hypotheses in generalized least squares analyses of unbalanced split-plot experiments

Approximate t-tests of single degree of freedom hypotheses in generalized least squares analyses (GLS) of mixed linear models using restricted maximum likelihood (REML) estimates of variance components have been previously developed by Giesbrecht and Burns (GB), and by Jeske and Harville (JH), using method of moment approximations for the degrees of freedom (df) for the tstatistics. This paper proposes approximate Fstatistics for tests of multiple df hypotheses using one-moment and two-moment approximations which may be viewed as extensions of the GB and JH methods. The paper focuses specifically on tests of hypotheses concerning the main-plot treatment factor in split-plot experiments with missing data. Simulation results indicate usually satisfactory control of Type I error rates.     

Some factors affecting statistical power of approximate tests in the linear mixed model for longitudinal data

Different longitudinal study designs require different statistical analysis methods and different methods of sample size determination. Statistical power analysis is a flexible approach to sample size determination for longitudinal studies. However, different power analyses are required for different statistical tests which arises from the difference between different statistical methods. In this paper, the simulation-based power calculations of F-tests with Containment, Kenward-Roger or Satterthwaite approximation of degrees of freedom are examined for sample size determination in the context of a special case of linear mixed models (LMMs), which is frequently used in the analysis of longitudinal data. Essentially, the roles of some factors, such as variance–covariance structure of random effects [unstructured UN or factor analytic FA0], autocorrelation structure among errors over time [independent IND, first-order autoregressive AR1 or first-order moving average MA1], parameter estimation methods [maximum likelihood ML and restricted maximum likelihood REML] and iterative algorithms [ridge-stabilized Newton-Raphson and Quasi-Newton] on statistical power of approximate F-tests in the LMM are examined together, which has not been considered previously. The greatest factor affecting statistical power is found to be the variance–covariance structure of random effects in the LMM. It appears that the simulation-based analysis in this study gives an interesting insight into statistical power of approximate F-tests for fixed effects in LMMs for longitudinal data.     

Inverse prediction for heteroscedastic response using mixed models software

Distributions of a response y (height, for example) differ with values of a factor t (such as age). Given a response y_* for a subject of unknown t_*, the objective of inverse prediction is to infer the value of t_* and to provide a defensible confidence set for it. Training data provide values of y observed on subjects at known values of t. Models relating the mean and variance of y to t can be formulated as mixed (fixed and random) models in terms of sets of functions of t, such as polynomial spline functions. A confidence set on t_* can then be had as those hypothetical values of t for which y_* is not detected as an outlier when compared to the model fit to the training data. With nonconstant variance, the p-values for these tests are approximate. This article describes how versatile models for this problem can be formulated in such a way that the computations can be accomplished with widely available software for mixed models, such as SAS PROC MIXED. Coverage probabilities of confidence sets on t_* are illustrated in an example.     

On hsu's model in regression analysis

The paper examplifies with Hsu’s model a general pattern as how to derive results of variance component estimation from well known results on mean estimation, as far as linear model theory is concerned. This ’ dispersion-mean-correspondence‘provides new and short proofs for various theorems from the literature, concerning unbiased invariant quadratic estimators with minimum BAYES risk or minimum variance. For pure variance component models, unbiased non-negative quadratic estimability is characterized in terms of the design matrices.     

Preliminary test almost unbiased two-parameter estimators with student’s t errors and conflicting test statistics

Abstract

In this paper, we consider the preliminary test approach to the estimation of the regression parameter in a multiple regression model under multicollinearity situation. The preliminary test almost unbiased two-parameter estimators based on the Wald, the Likelihood ratio, and the Lagrangian multiplier tests are given, when it is suspected that the regression parameter may be restricted to a subspace and the regression error is distributed with multivariate Student’s t errors. The bias and quadratic risk of the proposed estimators are derived and compared. Furthermore, a Monte Carlo simulation is provided to illustrate some of the theoretical results.     

The split sample permutation t-tests

Without the exchangeability assumption, permutation tests for comparing two population means do not provide exact control of the probability of making a Type I error. Another drawback of permutation tests is that it cannot be used to test hypothesis about one population. In this paper, we propose a new type of permutation tests for testing the difference between two population means: the split sample permutation t-tests. We show that the split sample permutation t-tests do not require the exchangeability assumption, are asymptotically exact and can be easily extended to testing hypothesis about one population. Extensive simulations were carried out to evaluate the performance of two specific split sample permutation t-tests: the split in the middle permutation t-test and the split in the end permutation t-test. The simulation results show that the split in the middle permutation t-test has comparable performance to the permutation test if the population distributions are symmetric and satisfy the exchangeability assumption. Otherwise, the split in the end permutation t-test has significantly more accurate control of level of significance than the split in the middle permutation t-test and other existing permutation tests.     

Tests of regression coefficients under ridge regression models

This paper investigates two “non-exact” t-type tests, t( k₂) and t(k₂), of the individual coefficients of a linear regression model, based on two ordinary ridge estimators. The reported results are built on a simulation study covering 84 different models. For models with large standard errors, the ridge-based t-tests have correct levels with considerable gain in powers over those of the least squares t-test, t(0). For models with small standard errors, t(k₁) is found to be liberal and is not safe to use while, t(k₂) is found to slightly exceed the nominal level in few cases. When tie two ridge tests art: not winners, the results indicate that they don't loose much against t(0).     

Heterogeneity of variance and biased hypothesis tests

This study examined the influence of heterogeneity of variance on Type I error rates and power of the independent-samples Student's t-test of equality of means on samples of scores from normal and 10 non-normal distributions. The same test of equality of means was performed on corresponding rank-transformed scores. For many non-normal distributions, both versions produced anomalous power functions, resulting partly from the fact that the hypothesis test was biased, so that under some conditions, the probability of rejecting H ₀ decreased as the difference between means increased. In all cases where bias occurred, the t-test on ranks exhibited substantially greater bias than the t-test on scores. This anomalous result was independent of the more familiar changes in Type I error rates and power attributable to unequal sample sizes combined with unequal variances.     

Two separate effects of variance heterogeneity on the validity and power of significance tests of location

Heterogeneity of variances of treatment groups influences the validity and power of significance tests of location in two distinct ways. First, if sample sizes are unequal, the Type I error rate and power are depressed if a larger variance is associated with a larger sample size, and elevated if a larger variance is associated with a smaller sample size. This well-established effect, which occurs in t and F tests, and to a lesser degree in nonparametric rank tests, results from unequal contributions of pooled estimates of error variance in the computation of test statistics. It is observed in samples from normal distributions, as well as non-normal distributions of various shapes. Second, transformation of scores from skewed distributions with unequal variances to ranks produces differences in the means of the ranks assigned to the respective groups, even if the means of the initial groups are equal, and a subsequent inflation of Type I error rates and power. This effect occurs for all sample sizes, equal and unequal. For the t test, the discrepancy diminishes, and for the Wilcoxon–Mann–Whitney test, it becomes larger, as sample size increases. The Welch separate-variance t test overcomes the first effect but not the second. Because of interaction of these separate effects, the validity and power of both parametric and nonparametric tests performed on samples of any size from unknown distributions with possibly unequal variances can be distorted in unpredictable ways.     

JAMES-STEIN RULE ESTIMATORS IN LINEAR REGRESSION MODELS WITH MULTIVARIATE-t DISTRIBUTED ERROR

This paper considers estimation of β in the regression model y =Xβ+μ, where the error components in μ have the jointly multivariate Student-t distribution. A family of James-Stein type estimators (characterised by nonstochastic scalars) is presented. Sufficient conditions involving only X are given, under which these estimators are better (with respect to the risk under a general quadratic loss function) than the usual minimum variance unbiased estimator (MVUE) of β. Approximate expressions for the bias, the risk, the mean square error matrix and the variance-covariance matrix for the estimators in this family are obtained. A necessary and sufficient condition for the dominance of this family over MVUE is also given.     

Robustness without rank order statistics

An alternative to conventional rank tests based on a Euclidean distance analysis space is described. Comparisons based on exact probability values among classical two-sample t-tests and the Wilcoxon–Mann–Whitney test illustrate the advantages of the Euclidean distance analysis space alternative.     

Approximate testing in two-stage nonlinear mixed models

We investigate here small sample properties of approximate F-tests about fixed effects parameters in nonlinear mixed models. For estimation of population fixed effects parameters as well as variance components, we apply the two-stage approach. This method is useful and popular when the number of observations per sampling unit is large enough. The approximate F-test is constructed based on large-sample approximation to the distribution of nonlinear least-squares estimates of subject-specific parameters. We recommend a modified test statistic that takes into consideration approximation to the large-sample Fisher information matrix (See [Volaufova J, Burton JH. Note on hypothesis testing in mixed models. Oral presentation at: LINSTAT 2012/21st IWMS; 2012; Bedlewo, Poland]). Our main focus is on comparing finite sample properties of broadly used approximate tests (Wald test and likelihood ratio test) and the modified F-test under the null hypothesis, especially accuracy of p-values (See [Volaufova J, LaMotte L. Comparison of approximate tests of fixed effects in linear repeated measures design models with covariates. Tatra Mountains. 2008;39:17–25]). For that purpose two extensive simulation studies are conducted based on pharmacokinetic models (See [Hartford A, Davidian M. Consequences of misspecifying assumptions in nonlinear mixed effects models. Comput Stat and Data Anal. 2000;34:139–164; Pinheiro J, Bates D. Approximations to the log-likelihood function in the non-linear mixed-effects model. J Comput Graph Stat. 1995;4(1):12–35]).     

Asymptotic Distributions of Unit-Root Tests When the Process Is Nearly Stationary

Several test criteria are available for testing the hypothesis that the autoregressive polynomial of an autoregressive moving average process has a single unit root. Schwert (1989), using a Monte Carlo study, investigated the performance of some of the available test criteria. He concluded that the actual levels of the test criteria considered in his study are far from the specified levels when the moving average polynomial also has a root close to 1. This article studies the asymptotic null distribution of the test statistics for testing p = 1 in the model Y_t = pY _t-1 + e _t – 0e _t-1 as 0 approaches 1. It is shown that the test statistics differ from one another in their asymptotic properties depending on the rate at which 0 converges to 1.     

Testing the Equality of the Noncentrality Parameters of Two Noncentral t-Distributions with Identical Degrees of Freedom

The problem of testing the equality of the noncentrality parameters of two noncentral t-distributions with identical degrees of freedom is considered, which arises from the comparison of two signal-to-noise ratios for simple linear regression models. A test procedure is derived that is guaranteed to maintain Type I error while having only minimal amounts of conservativeness, and comparisons are made with several other approaches to this problem based on variance stabilizing transformations. The new procedure derived in this article is shown to have good properties and will be useful for practitioners.     

The effect of nonnormality on near-replicate lack-of-fit tests

We examine the effect of nonnormality on the distributions of near-replicate lack-of-fit F-tests. We show that when the number of clusters is large, the distributions of the lack-of-fit tests depend on the kurtosis of the error distribution, and that heavy-tailed error distributions can inflate significantly the sizes of the tests. This behaviour is also evident in small samples, where some lack-of-fit tests are clearly more affected than others by nonnormality. Two modifications of the F-tests are suggested to eliminate the effect of the kurtosis on the limiting null distributions, and their behaviour is studied for small samples.     

A Nonrandomized,Nonconservative Version of the Fisher Exact Test

The Fisher exact test has been unjustly dismissed by some as ‘only conditional,’ whereas it is unconditionally the uniform most powerful test among all unbiased tests, tests of size α and with power greater than its nominal level of significance α. The problem with this truly optimal test is that it requires randomization at the critical value(s) to be of size α. Obviously, in practice, one does not want to conclude that ‘with probability x the we have a statistical significant result.’ Usually, the hypothesis is rejected only if the test statistic's outcome is more extreme than the critical value, reducing the actual size considerably.

The randomized unconditional Fisher exact is constructed (using Neyman–structure arguments) by deriving a conditional randomized test randomizing at critical values c(t) by probabilities γ(t), that both depend on the total number of successes T (the complete-sufficient statistic for the nuisance parameter—the common success probability) conditioned upon.

In this paper, the Fisher exact is approximated by deriving nonrandomized conditional tests with critical region including the critical value only if γ (t) > γ₀, for a fixed threshold value γ₀, such that the size of the unconditional modified test is for all value of the nuisance parameter—the common success probability—smaller, but as close as possible to α. It will be seen that this greatly improves the size of the test as compared with the conservative nonrandomized Fisher exact test.

Size, power, and p value comparison with the (virtual) randomized Fisher exact test, and the conservative nonrandomized Fisher exact, Pearson's chi-square test, with the more competitive mid-p value, the McDonald's modification, and Boschloo's modifications are performed under the assumption of two binomial samples.     

Testing the Goodness of Fit of Parametric Regression Models with Random Toeplitz Forms

ABSTRACT: We introduce a class of Toeplitz‐band matrices for simple goodness of fit tests for parametric regression models. For a given length r of the band matrix the asymptotic optimal solution is derived. Asymptotic normality of the corresponding test statistic is established under a fixed and random design assumption as well as for linear and non‐linear models, respectively. This allows testing at any parametric assumption as well as the computation of confidence intervals for a quadratic measure of discrepancy between the parametric model and the true signal g;. Furthermore, the connection between testing the parametric goodness of fit and estimating the error variance is highlighted. As a by‐product we obtain a much simpler proof of a result of 34 ) concerning the optimality of an estimator for the variance. Our results unify and generalize recent results by 9 ) and 15 , 16 ) in several directions. Extensions to multivariate predictors and unbounded signals are discussed. A simulation study shows that a simple jacknife correction of the proposed test statistics leads to reasonable finite sample approximations.     

Which Measure Should be Used for Testing in a Paired Design: Simple Difference,Percent Change,or Symmetrized Percent Change?

We aimed to determine the most proper change measure among simple difference, percent, or symmetrized percent changes in simple paired designs. For this purpose, we devised a computer simulation program. Since distributions of percent and symmetrized percent change values are skewed and bimodal, paired t-test did not give good results according to Type I error and the test power. To be to able use percent change or symmetrized percent change as change measure, either the distribution of test statistics should be transformed to a known theoretical distribution by transformation methods or a new test statistic for these values should be developed.     

Parametric bootstrap tests for unbalanced nested designs under heteroscedasticity

In this article, we consider the two-factor unbalanced nested design model without the assumption of equal error variance. For the problem of testing ‘main effects’ of both factors, we propose a parametric bootstrap (PB) approach and compare it with the existing generalized F (GF) test. The Type I error rates of the tests are evaluated using Monte Carlo simulation. Our studies show that the PB test performs better than the GF test. The PB test performs very satisfactorily even for small samples while the GF test exhibit poor Type I error properties when the number of factorial combinations or treatments goes up. It is also noted that the same tests can be used to test the significance of the random effect variance component in a two-factor mixed effects nested model under unequal error variances.