Testing the equality of several means when the population variances are unequal

A comparative study is made of three tests, developed by James (1951), Welch (1951) and Brown & Forsythe (1974). James presented two methods of which only one is considered in this paper. It is shown that this method gives better control over the size than the other two tests. None of these methods is uniformly more powerful than the other two. In some cases the tests of James and Welch reject a false null hypothesis more often than the test of Brown & Forsythe, but there are also situations in which it is the other way around.

We conclude that for implementation in a statistical software package the very complicated test of James is the most attractive. A practical disadvantage of this method can be overcome by a minor modification.     

Robust tests for the equality of variances for clustered data

Tests for the equality of variances are often needed in applications. In genetic studies the assumption of equal variances of continuous traits, measured in identical and fraternal twins, is crucial for heritability analysis. To test the equality of variances of traits, which are non-normally distributed, Levene [H. Levene, Robust tests for equality of variances, in Contributions to Probability and Statistics, I. Olkin, ed. Stanford University Press, Palo Alto, California, 1960, pp. 278–292] suggested a method that was surprisingly robust under non-normality, and the procedure was further improved by Brown and Forsythe [M.B. Brown and A.B. Forsythe, Robust tests for the equality of variances, J. Amer. Statis. Assoc. 69 (1974), pp. 364–367]. These tests assumed independence of observations. However, twin data are clustered – observations within a twin pair may be dependent due to shared genes and environmental factors. Uncritical application of the tests of Brown and Forsythe to clustered data may result in much higher than nominal Type I error probabilities. To deal with clustering we developed an extended version of Levene's test, where the ANOVA step is replaced with a regression analysis followed by a Wald-type test based on a clustered version of the robust Huber–White sandwich estimator of the covariance matrix. We studied the properties of our procedure using simulated non-normal clustered data and obtained Type I error rates close to nominal as well as reasonable powers. We also applied our method to oral glucose tolerance test data obtained from a twin study of the metabolic syndrome and related components and compared the results with those produced by the traditional approaches.     

New monte carlo results on the robustness of the anova f,w and f statistics

Because the usual F test for equal means is not robust to unequal variances, Brown and Forsythe (1974a) suggest replacing F with the statistics F or W which are based on the Satterthwaite and Welch adjusted degrees of freedom procedures. This paper reports practical situations where both F and W give * unsatisfactory results. In particular, both F and W may not provide adequate control over Type I errors. Moreover, for equal variances, but unequal sample sizes, W should be avoided in favor of F (or F ), but for equal sample sizes, and possibly unequal variances, W was the only satisfactory statistic. New results on power are included as well. The paper also considers the effect of using F or W only after a significant test for equal variances has been obtained, and new results on the robustness of the F test are described. It is found that even for equal sample sizes as large as 50 per treatment group, there are practical situations where the F test does not provide adequately control over the probability of a Type I error.     

The 'improved' brown and forsythe test for mean equality: some things can't be fixed

Mehrotra (1997) presented an ‘;improved’ Brown and Forsythe (1974) statistic which is designed to provide a valid test of mean equality in independent groups designs when variances are heterogeneous. In particular, the usual Brown and Fosythe procedure was modified by using a Satterthwaite approximation for numerator degrees of freedom instead of the usual value of number of groups minus one. Mehrotra then, through Monte Carlo methods, demonstrated that the ‘improved’ method resulted in a robust test of significance in cases where the usual Brown and Forsythe method did not. Accordingly, this ‘improved’ procedure was recommended. We show that under conditions likely to be encountered in applied settings, that is, conditions involving heterogeneous variances as well as nonnormal data, the ‘improved’ Brown and Forsythe procedure results in depressed or inflated rates of Type I error in unbalanced designs. Previous findings indicate, however, that one can obtain a robust test by adopting a heteroscedastic statistic with the robust estimators, rather than the usual least squares estimators, and further improvement can be expected when critical significance values are obtained through bootstrapping methods.     

Testing methods for the one-way fixed effects ANOVA models of log-normal samples

For one-way fixed effects of log-normal data with unequal variance, the present study proposes a method to deal with heterogeneity. An appropriate hypothesis testing is demonstrated; and one of the approximate tests, such as the Alexander-Govern test, Welch test or James second-order test, is applied to control Type I error rate. Monte Carlo simulation is used to investigate the performance of the F test for log-scale, the F test for original scale, the James second-order test, the Welch test, and the Alexander-Govern test. The simulated results and real data analysis show that the proposed method is valid and powerful.     

Comparison of k independent,zero‐heavy lognormal distributions

Lachenbruch ( 1976 , 2001 ) introduced two‐part tests for comparison of two means in zero‐inflated continuous data. We are extending this approach and compare k independent distributions (by comparing their means, either overall or the departure from equal proportion of zeros and equal means of nonzero values) by introducing two tests: a two‐part Wald test and a two‐part likelihood ratio test. If the continuous part of the distributions is lognormal then the proposed two test statistics have asymptotically chi‐square distribution with $2(k-1)$ degrees of freedom. A simulation study was conducted to compare the performance of the proposed tests with several well‐known tests such as ANOVA, Welch ( 1951 ), Brown & Forsythe ( 1974 ), Kruskal–Wallis, and one‐part Wald test proposed by Tu & Zhou ( 1999 ). Results indicate that the proposed tests keep the nominal type I error and have consistently best power among all tests being compared. An application to rainfall data is provided as an example. The Canadian Journal of Statistics 39: 690–702; 2011. © 2011 Statistical Society of Canada     

A robust test for the homogeneity of scales

Many robust tests for the equality of variances have been proposed recently. Brown and Forsythe (1974) and Layard (1973) review some of the well-known procedures and compare them by simulation methods. Brown and Forsythe’s alternative formulation of Levene’s test statistic is found to be quite robust under certain nonnormal distributions. The performance of the methods, however, suffers in the presence of heavy tailed distributions such as the Cauchy distribution.

In this paper, we propose and study a simple robust test. The results obtained from the Monte Carlo study compare favorably with those of the existing procedures.     

Choosing the best pairwise comparisons of means from non-normal populations,with unequal variances,but equal sample sizes

A Monte Carlo simulation evaluated five pairwise multiple comparison procedures for controlling Type I error rates, any-pair power, and all-pairs power. Realistic conditions of non-normality were based on a previous survey. Variance ratios were varied from 1:1 to 64:1. Procedures evaluated included Tukey's honestly significant difference (HSD) preceded by an F test, the Hayter–Fisher, the Games–Howell preceded by an F test, the Pertiz with F tests, and the Peritz with Alexander–Govern tests. Tukey's procedure shows the greatest robustness in Type I error control. Any-pair power is generally best with one of the Peritz procedures. All-pairs power is best with the Pertiz F test procedure. However, Tukey's HSD preceded by the Alexander–Govern F test may provide the best combination for controlling Type I and power rates in a variety of conditions of non-normality and variance heterogeneity.     

A closer examination on some parametric alternatives to the ANOVA F-test

In experiments, the classical (ANOVA) F-test is often used to test the omnibus null-hypothesis μ₁ = μ₂ ... = μ _j = ... = μ _n (all n population means are equal) in a one-way ANOVA design, even when one or more basic assumptions are being violated. In the first part of this article, we will briefly discuss the consequences of the different types of violations of the basic assumptions (dependent measurements, non-normality, heteroscedasticity) on the validity of the F-test. Secondly, we will present a simulation experiment, designed to compare the type I-error and power properties of both the F-test and some of its parametric adaptations: the Brown & Forsythe F^*-test and Welch’s V_w-test. It is concluded that the Welch V_w-test offers acceptable control over the type I-error rate in combination with (very) high power in most of the experimental conditions. Therefore, its use is highly recommended when one or more basic assumptions are being violated. In general, the use of the Brown & Forsythe F^*-test cannot be recommended on power considerations unless the design is balanced and the homoscedasticity assumption holds.     

Effects of the generalized Box–Cox transformation on Type I error rate and power of Hotelling's T 2

Most multivariate statistical techniques rely on the assumption of multivariate normality. The effects of nonnormality on multivariate tests are assumed to be negligible when variance–covariance matrices and sample sizes are equal. Therefore, in practice, investigators usually do not attempt to assess multivariate normality. In this simulation study, the effects of skewed and leptokurtic multivariate data on the Type I error and power of Hotelling's T ² were examined by manipulating distribution, sample size, and variance–covariance matrix. The empirical Type I error rate and power of Hotelling's T ² were calculated before and after the application of generalized Box–Cox transformation. The findings demonstrated that even when variance–covariance matrices and sample sizes are equal, small to moderate changes in power still can be observed.     

Exact methods for testing homogeneity of proportions for multiple groups of paired binary data

We consider four exact procedures to test the homogeneity of proportions for correlated multiple clustered data. Exact procedures are compared with the asymptotic approach based on the score statistic. We use a real example from a double-blind clinical trial studying the treatment of otitis media to illustrate the various test procedures and provide extensive numerical studies to compare procedures with regards to Type I error rates and powers under the unconditional framework. The exact unconditional procedure based on estimation followed by maximization is generally more powerful than other procedures.     

Testing Variability in the Two-Sample Case

A number of robust methods for testing variability have been reported in previous literature. An examination of these procedures for a wide variety of populations confirms their general robustness. Shoemaker's improvement of the F test extends that test use to a realistic variety of population shapes. However, a combination of the Brown–Forsythe and O'Brien methods based on testing kurtosis is shown to be conservative for a wide range of sample sizes and population distributions. The composite test is also shown to be more powerful in most conditions than other conservative procedures.     

Comparison of Nonparametric and Parametric Methods in Repeated Measures Designs - A Simulation Study

This study investigates the performance of parametric and nonparametric tests to analyze repeated measures designs. Both multivariate normal and exponential distributions were simulated for varying values of the correlation and ten or twenty subjects within each cell. For multivariate normal distributions, the type I error rates were lower than the usual 0.05 level for nonparametric tests, whereas the parametric tests without the Greenhouse-Geisser or the Huynh-Feldt adjustment produced slightly higher type I error rates. Type I error rates for nonparametric tests, for multivariate exponential distributions, were more stable than parametric, Greenhouse-Geisser or Huynh-Feldt adjusted tests. For ten subjects within each cell, the parametric tests were more powerful than nonparametric tests. For twenty subjects per cell, the power of the nonparametric and parametric tests was comparable.     

Analytic approach to coset–leader decoding leads to extension of the Welch–Berlekamp theorem

This paper provides a complete proof of the Welch–Berlekamp theorem on which the Welch–Berlekamp algorithm was founded. By introducing an analytic approach to coset–leader decoders for Reed–Solomon codes, the Welch–Berlekamp key-equation of error corrections is enlarged and a complete proof of the Welch–Berlekamp theorem is derived in a natural way, and the theorem is extended such that the BCH-bound constraint is moved.     

A parametric bootstrap solution to the MANOVA under heteroscedasticity

In this article, we consider the problem of comparing several multivariate normal mean vectors when the covariance matrices are unknown and arbitrary positive definite matrices. We propose a parametric bootstrap (PB) approach and develop an approximation to the distribution of the PB pivotal quantity for comparing two mean vectors. This approximate test is shown to be the same as the invariant test given in [Krishnamoorthy and Yu, Modified Nel and Van der Merwe test for the multivariate Behrens–Fisher problem, Stat. Probab. Lett. 66 (2004), pp. 161–169] for the multivariate Behrens–Fisher problem. Furthermore, we compare the PB test with two existing invariant tests via Monte Carlo simulation. Our simulation studies show that the PB test controls Type I error rates very satisfactorily, whereas other tests are liberal especially when the number of means to be compared is moderate and/or sample sizes are small. The tests are illustrated using an example.     

A Parametric Bootstrap Solution to Two-way MANOVA without Interaction under Heteroscedasticity: Fixed and Mixed Models

In this article, we propose a parametric bootstrap (PB) test for heteroscedastic two-way multivariate analysis of variance without Interaction. For the problem of testing equal main effects of factors, we obtain a PB approach and compare it with existing modified Brown–Forsythe (MBF) test and approximate Hotelling T² (AHT) test by an extensive simulation study. The PB test is a symmetric function in samples, and does not depend on the chosen weights used to define the parameters uniquely. Simulation results indicate that the PB test performs satisfactorily for various cell sizes and parameter configurations when the homogeneity assumption is seriously violated, and tends to outperform the AHT test for moderate or larger samples in terms of power and controlling size. The MBF test, the AHT test, and the PB test have similar robustness to violations of underlying assumptions. It is also noted that the same PB test can be used to test the significance of random effect vector in a two-way multivariate mixed effects model with unequal cell covariance matrices.     

A robust procedure for comparing several means under heteroscedasticity and nonnormality

By applying Tiku's MML robust procedure to Brown and Forsythe's (1974) statistic, this paper derives a robust and more powerful procedure for comparing several means under hetero-scedasticity and nonnormality. Some Monte Carlo studies indicate clearly that among five nonnormal distributions, except for the uniform distribution, the new test is more powerful than the Brown and Forsythe test under nonnormal distributions in all cases investigated and has substantially the same power as the Brown and Forsythe test under normal distribution.     

Type I error rate control in adaptive designs for confirmatory clinical trials with treatment selection at interim

Interest in confirmatory adaptive combined phase II/III studies with treatment selection has increased in the past few years. These studies start comparing several treatments with a control. One (or more) treatment(s) is then selected after the first stage based on the available information at an interim analysis, including interim data from the ongoing trial, external information and expert knowledge. Recruitment continues, but now only for the selected treatment(s) and the control, possibly in combination with a sample size reassessment. The final analysis of the selected treatment(s) includes the patients from both stages and is performed such that the overall Type I error rate is strictly controlled, thus providing confirmatory evidence of efficacy at the final analysis. In this paper we describe two approaches to control the Type I error rate in adaptive designs with sample size reassessment and/or treatment selection. The first method adjusts the critical value using a simulation-based approach, which incorporates the number of patients at an interim analysis, the true response rates, the treatment selection rule, etc. We discuss the underlying assumptions of simulation-based procedures and give several examples where the Type I error rate is not controlled if some of the assumptions are violated. The second method is an adaptive Bonferroni-Holm test procedure based on conditional error rates of the individual treatment-control comparisons. We show that this procedure controls the Type I error rate, even if a deviation from a pre-planned adaptation rule or the time point of such a decision is necessary.     

An empirical study of five multivariate tests for the single-factor repeated measures model

A Monte Carlo study was used to examine the Type I error and power values of five multivariate tests for the single-factor repeated measures model The performance of Hotelling's T2 and four nonparametric tests, including a chi-square and an F-test version of a rank-transform procedure, were investigated for different distributions, sample sizes, and numbers of repeated measures. The results indicated that both Hotellings T* and the F-test version of the rank-transform performed well, producing Type I error rates which were close to the nominal value. The chi-square version of the rank-transform test, on the other hand, produced inflated Type I error rates for every condition studied. The Hotelling and F-test version of the rank-transform procedure showed similar power for moderately-skewed distributions, but for strongly skewed distributions the F-test showed much better power. The performance of the other nonparametric tests depended heavily on sample size. Based on these results, the F-test version of the rank-transform procedure is recommended for the single-factor repeated measures model.     

Selecting the Optimal Transformation of a Continuous Covariate in Cox's Regression: Implications for Hypothesis Testing

ABSTRACT

Background: Many exposures in epidemiological studies have nonlinear effects and the problem is to choose an appropriate functional relationship between such exposures and the outcome. One common approach is to investigate several parametric transformations of the covariate of interest, and to select a posteriori the function that fits the data the best. However, such approach may result in an inflated Type I error. Methods: Through a simulation study, we generated data from Cox's models with different transformations of a single continuous covariate. We investigated the Type I error rate and the power of the likelihood ratio test (LRT) corresponding to three different procedures that considered the same set of parametric dose-response functions. The first unconditional approach did not involve any model selection, while the second conditional approach was based on a posteriori selection of the parametric function. The proposed third approach was similar to the second except that it used a corrected critical value for the LRT to ensure a correct Type I error. Results: The Type I error rate of the second approach was two times higher than the nominal size. For simple monotone dose-response, the corrected test had similar power as the unconditional approach, while for non monotone, dose-response, it had a higher power. A real-life application that focused on the effect of body mass index on the risk of coronary heart disease death, illustrated the advantage of the proposed approach. Conclusion: Our results confirm that a posteriori selecting the functional form of the dose-response induces a Type I error inflation. The corrected procedure, which can be applied in a wide range of situations, may provide a good trade-off between Type I error and power.