Two separate effects of variance heterogeneity on the validity and power of significance tests of location

Heterogeneity of variances of treatment groups influences the validity and power of significance tests of location in two distinct ways. First, if sample sizes are unequal, the Type I error rate and power are depressed if a larger variance is associated with a larger sample size, and elevated if a larger variance is associated with a smaller sample size. This well-established effect, which occurs in t and F tests, and to a lesser degree in nonparametric rank tests, results from unequal contributions of pooled estimates of error variance in the computation of test statistics. It is observed in samples from normal distributions, as well as non-normal distributions of various shapes. Second, transformation of scores from skewed distributions with unequal variances to ranks produces differences in the means of the ranks assigned to the respective groups, even if the means of the initial groups are equal, and a subsequent inflation of Type I error rates and power. This effect occurs for all sample sizes, equal and unequal. For the t test, the discrepancy diminishes, and for the Wilcoxon–Mann–Whitney test, it becomes larger, as sample size increases. The Welch separate-variance t test overcomes the first effect but not the second. Because of interaction of these separate effects, the validity and power of both parametric and nonparametric tests performed on samples of any size from unknown distributions with possibly unequal variances can be distorted in unpredictable ways.     

Parametric bootstrap tests for unbalanced nested designs under heteroscedasticity

In this article, we consider the two-factor unbalanced nested design model without the assumption of equal error variance. For the problem of testing ‘main effects’ of both factors, we propose a parametric bootstrap (PB) approach and compare it with the existing generalized F (GF) test. The Type I error rates of the tests are evaluated using Monte Carlo simulation. Our studies show that the PB test performs better than the GF test. The PB test performs very satisfactorily even for small samples while the GF test exhibit poor Type I error properties when the number of factorial combinations or treatments goes up. It is also noted that the same tests can be used to test the significance of the random effect variance component in a two-factor mixed effects nested model under unequal error variances.     

Distribution of variance ratio in unbalanced random model under heterogeneity of error variances

The effects of heteroscedasticity have been studied on the mean and variance of F ratio and on the power of F-test in unbalanced one-way random model, numerically. The computed results reveal that the heteroscedasticity and unbalanoedness have combined effects. The mean and variance of F as well as the power of F-test increase with inequality of error variances under balanced and those unbalanced situations where more variable groups have larger size. The effects are of serious nature when more variable groups have smaller size.     

Two non-parametric, analysis-of-means-type tests for homogeneity of variances

After a brief review of the literature, two non-parametric tests for homogeneity of variances are presented. The first test is based on the analysis of means for ranks, which is a non-parametric version of the analysis of means (ANOM) that uses ranks as input for an ANOM test. The second test uses inverse normal scores of the ranks of scale transformations of the observations as input to the ANOM. Both homogeneity of variances tests can be presented in a graphical form, which makes it easy for practitioners to assess the practical and the statistical significance. A Monte Carlo study is used to show that these tests have power comparable with that of well-known robust tests for homogeneity of variances.     

Approximate tests in unbalanced two-way random models without interaction

In the presence of non-normality, we consider testing for the significance of the variance components in the unbalanced two-way random model without interaction. The approximate test is based on the F-statistic for this model. The asymptotic distribution of the F-statistic is derived as the number of treatments tends to infinity while the number of observations for a treatment in any block takes value from a finite set of positive integers. Robustness of the approximate test is given.     

Testing for equivalence of variances using Hartley's ratio

Hartley's test for homogeneity of k normal‐distribution variances is based on the ratio between the maximum sample variance and the minimum sample variance. In this paper, the author uses the same statistic to test for equivalence of k variances. Equivalence is defined in terms of the ratio between the maximum and minimum population variances, and one concludes equivalence when Hartley's ratio is small. Exact critical values for this test are obtained by using an integral expression for the power function and some theoretical results about the power function. These exact critical values are available both when sample sizes are equal and when sample sizes are unequal. One related result in the paper is that Hartley's test for homogeneity of variances is no longer unbiased when the sample sizes are unequal. The Canadian Journal of Statistics 38: 647–664; 2010 © 2010 Statistical Society of Canada     

Analysis of means approach for random factor analysis

Analysis of means (ANOM) is a powerful tool for comparing means and variances in fixed-effects models. The graphical exhibit of ANOM is considered as a great advantage because of its interpretability and its ability to evaluate the practical significance of the mean effects. However, the presence of random factors may be problematic for the ANOM method. In this paper, we propose an ANOM approach that can be applied to test random effects in many different balanced statistical models including fixed-, random- and mixed-effects models. The proposed approach utilizes the range of the treatment averages for identifying the dispersions of the underlying populations. The power performance of the proposed procedure is compared to the analysis of variance (ANOVA) approach in a wide range of situations via a Monte Carlo simulation study. Illustrative examples are used to demonstrate the usefulness of the proposed approach and its graphical exhibits, provide meaningful interpretations, and discuss the statistical and practical significance of factor effects.     

The limiting distribution of the F-statistic from nonnormal universes

We consider a linear regression model with an unbalanced 1-fold nested error structure, where group effect and error are from nonnormal universes. The limiting distribution of the F-statistic in this model is derived, as the sample size is large and group sizes take values from a finite set of distinct integers. The result is used to approximate the F-distribution quantile and to test the significance of the random effect variance component. Results are also applicable to the F-statistic in the one-way random-effects model. The effects of departure from normality on the F-statistic distribution are given.     

Comparison of nonparametric analysis of variance methods: A vote for van der Waerden

ABSTRACT

For two-way layouts in a between-subjects analysis of variance design, the parametric F-test is compared with seven nonparametric methods: rank transform (RT), inverse normal transform (INT), aligned rank transform (ART), a combination of ART and INT, Puri & Sen's L statistic, Van der Waerden, and Akritas and Brunners ANOVA-type statistics (ATS). The type I error rates and the power are computed for 16 normal and nonnormal distributions, with and without homogeneity of variances, for balanced and unbalanced designs as well as for several models including the null and the full model. The aim of this study is to identify a method that is applicable without too much testing for all the attributes of the plot. The Van der Waerden test shows the overall best performance though there are some situations in which it is disappointing. The Puri & Sen's and the ATS tests show generally very low power. These two and the other methods cannot keep the type I error rate under control in too many situations. Especially in the case of lognormal distributions, the use of any of the rank-based procedures can be dangerous for cell sizes above 10. As already shown by many other authors, nonnormal distributions do not violate the parametric F-test, but unequal variances do, and heterogeneity of variances leads to an inflated error rate more or less also for the nonparametric methods. Finally, it should be noted that some procedures show rising error rates with increasing cell sizes, the ART, especially for discrete variables, and the RT, Puri & Sen, and the ATS in the cases of heteroscedasticity.     

The Evaluation of the Performance of IPWGEE,a Simulation Study

This article compares two recently proposed test statistics for unobserved cluster effects (C, SSR _w ) with three statistics frequently mentioned in panel econometrics (BP, SLM, F). Simulations include data generating processes with a cluster-level explanatory variable, scenarios with unequally sized clusters, processes that have an incorrectly specified cluster structure, and processes that have no cluster structure but rather spatial correlation. All but the F test exhibit small-sample deviation from the asymptotic distribution. The SLM, F, and SSR _w tests show equivalent power when cluster sizes are balanced. SLM has greatest power when cluster sizes are unbalanced.     

Choosing the best pairwise comparisons of means from non-normal populations,with unequal variances,but equal sample sizes

A Monte Carlo simulation evaluated five pairwise multiple comparison procedures for controlling Type I error rates, any-pair power, and all-pairs power. Realistic conditions of non-normality were based on a previous survey. Variance ratios were varied from 1:1 to 64:1. Procedures evaluated included Tukey's honestly significant difference (HSD) preceded by an F test, the Hayter–Fisher, the Games–Howell preceded by an F test, the Pertiz with F tests, and the Peritz with Alexander–Govern tests. Tukey's procedure shows the greatest robustness in Type I error control. Any-pair power is generally best with one of the Peritz procedures. All-pairs power is best with the Pertiz F test procedure. However, Tukey's HSD preceded by the Alexander–Govern F test may provide the best combination for controlling Type I and power rates in a variety of conditions of non-normality and variance heterogeneity.     

Analysis of means: a generalized approach using R

Papers on the analysis of means (ANOM) have been circulating in the quality control literature for decades, routinely describing it as a statistical stand-alone concept. Therefore, we clarify that ANOM should rather be regarded as a special case of a much more universal approach known as multiple contrast tests (MCTs). Perceiving ANOM as a grand-mean-type MCT paves the way for implementing it in the open-source software R. We give a brief tutorial on how to exploit R's versatility and introduce the R package ANOM for drawing the familiar decision charts. Beyond that, we illustrate two practical aspects of data analysis with ANOM: firstly, we compare merits and drawbacks of ANOM-type MCTs and ANOVA F-test and assess their respective statistical powers, and secondly, we show that the benefit of using critical values from multivariate t-distributions for ANOM instead of simple Bonferroni quantiles is oftentimes negligible.     

Performance of confidence interval tests for the ratio of two lognormal means applied to Weibull and gamma distribution data

Five estimation approaches have been developed to compute the confidence interval (CI) for the ratio of two lognormal means: (1) T, the CI based on the t-test procedure; (2) ML, a traditional maximum likelihood-based approach; (3) BT, a bootstrap approach; (4) R, the signed log-likelihood ratio statistic; and (5) R*, the modified signed log-likelihood ratio statistic. The purpose of this study was to assess the performance of these five approaches when applied to distributions other than lognormal distribution, for which they were derived. Performance was assessed in terms of average length and coverage probability of the CIs for each estimation approaches (i.e., T, ML, BT, R, and R*) when data followed a Weibull or gamma distribution. Four models were discussed in this study. In Model 1, the sample sizes and variances were equal within the two groups. In Model 2, the sample sizes were equal but variances were different within the two groups. In Model 3, the variances were different within the two groups and the larger variance was paired with the larger sample size. In Model 4, the variances were different within the two groups and the larger variance was paired with the smaller sample size. The results showed that when the variances of the two groups were equal, the t-test performed well, no matter what the underlying distribution was and how large the variances of the two groups were. The BT approach performed better than the others when the underlying distribution was not lognormal distribution, although it was inaccurate when the variances were large. The R* test did not perform well when the underlying distribution was Weibull or gamma distributed data, but it performed best when the data followed a lognormal distribution.     

A new distribution-free k-sample test: Analysis of kernel density functionals

A novel distribution-free k-sample test of differences in location shifts based on the analysis of kernel density functional estimation is introduced and studied. The proposed test parallels one-way analysis of variance and the Kruskal–Wallis (KW) test aiming at testing locations of unknown distributions. In contrast to the rank (score)-transformed non-parametric approach, such as the KW test, the proposed F-test uses the measurement responses along with well-known kernel density estimation (KDE) to estimate the locations and construct the test statistic. A practical optimal bandwidth selection procedure is also provided. Our simulation studies and real data example indicate that the proposed analysis of kernel density functional estimate (ANDFE) test is superior to existing competitors for fat-tailed or heavy-tailed distributions when the k groups differ mainly in location rather than shape, especially with unbalanced data. ANDFE is also highly recommended when it is unclear whether test groups differ mainly in shape or location. The Canadian Journal of Statistics 48: 167–186; 2020 © 2019 Statistical Society of Canada     

Performance of nonparametric multiple comparison tests under heteroscedasticity,dependency, and skewed error distribution

In this article, an extensive Monte Carlo simulation study is conducted to evaluate and compare nonparametric multiple comparison tests under violations of classical analysis of variance assumptions. Simulation space of the Monte Carlo study is composed of 288 different combinations of balanced and unbalanced sample sizes, number of groups, treatment effects, various levels of heterogeneity of variances, dependence between subgroup levels, and skewed error distributions under the single factor experimental design. By this large simulation space, we present a detailed analysis of effects of the violations of assumptions on the performance of nonparametric multiple comparison tests in terms of three error and four power measures. Observations of this study are beneficial to decide the optimal nonparametric test according to requirements and conditions of undertaken experiments. When some of the assumptions of analysis of variance are violated and number of groups is small, use of stepwise Steel-Dwass procedure with Holm's approach is appropriate to control type I error at a desired level. Dunn's method should be employed for greater number of groups. When subgroups are unbalanced and number of groups is small, Nemenyi's procedure with Duncan's approach produces high power values. Conover's procedure successfully provides high power values with a small number of unbalanced groups or with a greater number of balanced or unbalanced groups. At the same time, Conover's procedure is unable to control type I error rates.     

Comparison of the estimators of the intra-cluster correlation for the nested error regression model

The intra-cluster correlation is insisted on nested error regression model that, in practice, is rarely known. This article demonstrates the size in generalized least squares (GLS) F-test using Fuller–Battese transformation and modification F-test. For the balanced case, the former using strictly positive, analysis of covariance (ANCOVA) and analysis of variance (ANOVA) estimators of intra-cluster correlation can control the size for moderate intra-cluster correlations. For small intra-cluster correlation, they perform well when the numbers of cluster are large. The latter using the ANOVA estimator performs well except for small numbers of cluster. When intra-cluster correlation is large, it cannot control the size. For the unbalanced case, the GLS F-test using the Fuller–Battese transformation and the modification F-test using the strictly positive, the ANCOVA and the ANOVA estimators maintain the significance level for small total sample size and small intra-cluster correlations when there is a large variation in cluster sizes, but they perform well in controlling the size for large total sample size and small different variation in cluster sizes. Besides, Henderson’s method 3 estimator maintains the significance level for a few situations.     

A Parametric Bootstrap Test for Two-Way ANOVA Model Without Interaction Under Heteroscedasticity

In this article we consider the two-way ANOVA model without interaction under heteroscedasticity. For the problem of testing equal effects of factors, we propose a parametric bootstrap (PB) approach and compare it with existing the generalized F (GF) test. The Type I error rates and powers of the tests are evaluated using Monte Carlo simulation. Our studies show that the PB test performs better than the GF test. The PB test performs very satisfactorily even for small samples while the GF test exhibits poor Type I error properties when the number of factorial combinations or treatments goes up. It is also noted that the same tests can be used to test the significance of random effect variance component in a two-way mixed-effects model under unequal error variances.     

Robust I-sample analysis of means type randomization tests for variances

Four Analysis of Means (ANOM) type randomization tests for testing the equality of I variances are presented. Randomization techniques for testing statistical hypotheses can be used when parametric tests are inappropriate. Suppose that I independent samples have been collected. Randomization tests are based on shuffles or rearrangements of the (combined) sample. Putting each of the I samples "in a bowl" forms the combined sample. Drawing samples "from the bowl" forms a shuffle. Shuffles can be made with replacement (bootstrap shuffling) or without replacement (permutation shuffling). The tests that are presented offer two advantages. They are robust to non-normality and they allow the user to graphically present the results via a decision chart similar to a Shewhart control chart. The decision chart facilitates easy assessment of both statistical and practical significance. A Monte Carlo study is used to identify robust randomization tests that exhibit excellent power when compared to other robust tests.     

An adaptive test for the one-way layout

An adaptive test is proposed for the one-way layout. This test procedure uses the order statistics of the combined data to obtain estimates of percentiles, which are used to select an appropriate set of rank scores for the one-way test statistic. This test is designed to have reasonably high power over a range of distributions. The adaptive procedure proposed for a one-way layout is a generalization of an existing two-sample adaptive test procedure. In this Monte Carlo study, the power and significance level of the F-test, the Kruskal-Wallis test, the normal scores test, and the adaptive test were evaluated for the one-way layout. All tests maintained their significance level for data sets having at least 24 observations. The simulation results show that the adaptive test is more powerful than the other tests for skewed distributions if the total number of observations equals or exceeds 24. For data sets having at least 60 observations the adaptive test is also more powerful than the F-test for some symmetric distributions.     

Finite mixtures of natural exponential families

Let μ be a positive measure concentrated on R⁺ generating a natural exponential family (NEF) F with quadratic variance function V_F(m), m being the mean parameter of F. It is shown that v(dx) = (γ+x)μ(γ ≥ 0) (γ ≥ 0) generates a NEF G whose variance function is of the form l(m)Δ+cΔ(m), where l(m) is an affine function of m, Δ(m) is a polynomial in m (the mean of G) of degree 2, and c is a constant. The family G turns out to be a finite mixture of F and its length-biased family. We also examine the cases when F has cubic variance function and show that for suitable choices of γ the family G has variance function of the form P(m) + Q(m)m where P, Q are polynomials in m of degree m2 while Δ is an affine function of m. Finally we extend the idea to two dimensions by considering a bivariate Poisson and bivariate gamma mixture distribution.