Performance of nonparametric multiple comparison tests under heteroscedasticity,dependency, and skewed error distribution

In this article, an extensive Monte Carlo simulation study is conducted to evaluate and compare nonparametric multiple comparison tests under violations of classical analysis of variance assumptions. Simulation space of the Monte Carlo study is composed of 288 different combinations of balanced and unbalanced sample sizes, number of groups, treatment effects, various levels of heterogeneity of variances, dependence between subgroup levels, and skewed error distributions under the single factor experimental design. By this large simulation space, we present a detailed analysis of effects of the violations of assumptions on the performance of nonparametric multiple comparison tests in terms of three error and four power measures. Observations of this study are beneficial to decide the optimal nonparametric test according to requirements and conditions of undertaken experiments. When some of the assumptions of analysis of variance are violated and number of groups is small, use of stepwise Steel-Dwass procedure with Holm's approach is appropriate to control type I error at a desired level. Dunn's method should be employed for greater number of groups. When subgroups are unbalanced and number of groups is small, Nemenyi's procedure with Duncan's approach produces high power values. Conover's procedure successfully provides high power values with a small number of unbalanced groups or with a greater number of balanced or unbalanced groups. At the same time, Conover's procedure is unable to control type I error rates.     

Robust analysis of variance

SUMMARY For the c -sample location problem with equal and unequal variances, we compare the classical F -test and its robustified version-the Welch test-with some nonparametric counterparts defined for two-sided and one-sided ordered alternatives, such as trend and umbrella alternatives. A new rank test for long-tailed distributions is proposed. The comparison is referred to level alpha and power beta of the tests, and is carried out via Monte Carlo simulation, assuming short-, medium- and long-tailed as well as asymmetric distributions. It turns out that the Welch test is the best one in the case of unequal variances but in the case of equal variances special non-parametric tests are to prefer.     

A test statistic based on ranked set sampling for two normal means

In this study, we considered a hypothesis test for the difference of two population means using ranked set sampling. We proposed a test statistic for this hypothesis test with more than one cycle under normality. We also investigate the performance of this test statistic, when the assumptions hold and are violated. For this reason, we investigate the type I error and power rates of tests under normality with equal and unequal variances, non-normality with equal and unequal variances. We also examine the performance of this test under imperfect ranking case. The simulation results show that derived test performs quite well.     

Contrast analysis for scale differences

Research on tests for scale equality, that are robust to violations of the distributional normality assumption, have focused exclusively on an overall test statistic and have not examined procedures for identifying specific differences in multiple group designs. The present study compares four contrast analysis procedures for scale differences in the single factor four group design. Two data transformations are considered under several conbinations of variance difference, sample sizes, and distributional forms.The results indicate that no single transformation or analysis procedure is uniformly superior in controlling the familywise error rate or in statistical power. The relationship between sample size and variances is a major factor in selecting a contrast analysis procedure.     

Performance of threshold cointegration tests

In this paper, we examine and compare the performance of ordinary least squares (OLS) and instrumental variables (IVs) based single-equation Engle–Granger, error correction model (ECM), and autoregressive-distributed lag (ADL) threshold cointegration tests. Results are additionally compared with the system-equation OLS–ECM test. Finite sample properties of size and power are examined in simulations. The single-equation ECM test performs well, but has certain limitations to resolve. The system-equation ECM test has good power, but suffers from mild size distortions that lead to over-rejections. Overall, the single-equation OLS–ADL test has the most desirable properties among the OLS-based tests and the IV–ADL test has the most desirable properties among the IV-based tests. The IV-based tests are invariant to nuisance parameters and have a standard normal distribution in each case, while the OLS-based tests are non-standard so that critical values must be simulated in different models.     

Performance of some multiple comparison tests under heteroscedasticity and dependency

In this article, 18 multiple comparison tests are compared according to powers and type I error measures under some violations of analysis of variance assumptions with a Monte Carlo simulation study. Considered violations of assumptions are heterogeneity in subgroup variances and dependency between subgroups. Various numbers of subgroups and subgroup sizes are considered simultaneously with the violations of assumptions. Simulation results are analysed by using visual inspection, graphical representations, decision-tree and correspondence analyses. Wide inferences are drawn on the behaviour of considered tests with respect to measures used. Some general suggestions are given on which tests should be used or avoided under violations of assumptions.     

A Nonparametric Test for Interval-Censored Failure Time Data with Unequal Censoring

This article considers nonparametric comparison of survival functions, one of the most commonly required task in survival studies. For this, several test procedures have been proposed for interval-censored failure time data in which distributions of censoring intervals are identical among different treatment groups. Sometimes the distributions may depend on treatments and thus not be the same. A class of test statistics is proposed for situations where the distributions may be different for subjects in different treatment groups. The asymptotic normality of the test statistics is established and the test procedure is evaluated by simulations, which suggest that it works well for practical situations. An illustrative example is provided.     

Comparison of nonparametric analysis of variance methods: A vote for van der Waerden

ABSTRACT

For two-way layouts in a between-subjects analysis of variance design, the parametric F-test is compared with seven nonparametric methods: rank transform (RT), inverse normal transform (INT), aligned rank transform (ART), a combination of ART and INT, Puri & Sen's L statistic, Van der Waerden, and Akritas and Brunners ANOVA-type statistics (ATS). The type I error rates and the power are computed for 16 normal and nonnormal distributions, with and without homogeneity of variances, for balanced and unbalanced designs as well as for several models including the null and the full model. The aim of this study is to identify a method that is applicable without too much testing for all the attributes of the plot. The Van der Waerden test shows the overall best performance though there are some situations in which it is disappointing. The Puri & Sen's and the ATS tests show generally very low power. These two and the other methods cannot keep the type I error rate under control in too many situations. Especially in the case of lognormal distributions, the use of any of the rank-based procedures can be dangerous for cell sizes above 10. As already shown by many other authors, nonnormal distributions do not violate the parametric F-test, but unequal variances do, and heterogeneity of variances leads to an inflated error rate more or less also for the nonparametric methods. Finally, it should be noted that some procedures show rising error rates with increasing cell sizes, the ART, especially for discrete variables, and the RT, Puri & Sen, and the ATS in the cases of heteroscedasticity.     

Improved tests for homogeneity of variances

Equality of variances is one of the key assumptions of analysis of variances (ANOVA). There are several testing procedures available to validate this assumption, but it is rare to find a test procedure which controls the type I error rate while providing high statistical power. In this article, we introduce a bootstrap test based on the ratio of mean absolute deviances (RMD). We also propose a two-stage testing procedure where we first quantify the skewness of the distributions and then choose an appropriate test for homogeneity of variances. The performances of these test procedures are studied via a simulation study.     

Two separate effects of variance heterogeneity on the validity and power of significance tests of location

Heterogeneity of variances of treatment groups influences the validity and power of significance tests of location in two distinct ways. First, if sample sizes are unequal, the Type I error rate and power are depressed if a larger variance is associated with a larger sample size, and elevated if a larger variance is associated with a smaller sample size. This well-established effect, which occurs in t and F tests, and to a lesser degree in nonparametric rank tests, results from unequal contributions of pooled estimates of error variance in the computation of test statistics. It is observed in samples from normal distributions, as well as non-normal distributions of various shapes. Second, transformation of scores from skewed distributions with unequal variances to ranks produces differences in the means of the ranks assigned to the respective groups, even if the means of the initial groups are equal, and a subsequent inflation of Type I error rates and power. This effect occurs for all sample sizes, equal and unequal. For the t test, the discrepancy diminishes, and for the Wilcoxon–Mann–Whitney test, it becomes larger, as sample size increases. The Welch separate-variance t test overcomes the first effect but not the second. Because of interaction of these separate effects, the validity and power of both parametric and nonparametric tests performed on samples of any size from unknown distributions with possibly unequal variances can be distorted in unpredictable ways.     

An outlier problem in the determination of ore grade

Data from recordings of ore assays from the Western Australian goldfields provide motivation to devise new tests for outliers when observations are distributed with the same mean but diff ering variances. In the case of equal variances, tests for a single outlier reduce to well-known tests of discordancy. A block discordancy test for k outliers is also described. The question of whether or not one should omit any observation(s) in the calculation of the mean recoverable gold content is addressed in the context of whether or not the data contain outliers, as judged by a normal model for the 'logged' ore assay values. The given data suggest that models with 'logged' values that follow long-tailed approximately normal distributions may be appropriate.     

New monte carlo results on the robustness of the anova f,w and f statistics

Because the usual F test for equal means is not robust to unequal variances, Brown and Forsythe (1974a) suggest replacing F with the statistics F or W which are based on the Satterthwaite and Welch adjusted degrees of freedom procedures. This paper reports practical situations where both F and W give * unsatisfactory results. In particular, both F and W may not provide adequate control over Type I errors. Moreover, for equal variances, but unequal sample sizes, W should be avoided in favor of F (or F ), but for equal sample sizes, and possibly unequal variances, W was the only satisfactory statistic. New results on power are included as well. The paper also considers the effect of using F or W only after a significant test for equal variances has been obtained, and new results on the robustness of the F test are described. It is found that even for equal sample sizes as large as 50 per treatment group, there are practical situations where the F test does not provide adequately control over the probability of a Type I error.     

A non-parametric maximum test for the Behrens–Fisher problem

Non-normality and heteroscedasticity are common in applications. For the comparison of two samples in the non-parametric Behrens–Fisher problem, different tests have been proposed, but no single test can be recommended for all situations. Here, we propose combining two tests, the Welch t test based on ranks and the Brunner–Munzel test, within a maximum test. Simulation studies indicate that this maximum test, performed as a permutation test, controls the type I error rate and stabilizes the power. That is, it has good power characteristics for a variety of distributions, and also for unbalanced sample sizes. Compared to the single tests, the maximum test shows acceptable type I error control.     

Testing equality of variances for several normal populations

In this study, we develop a test based on computational approach for the equality of variances of several normal populations. The proposed method is numerically compared with the existing methods. The numeric results demonstrate that the proposed method performs very well in terms of type I error rate and power of test. Furthermore we study the robustness of the tests by using simulation study when the underlying data are from t, exponential and uniform distributions. Finally we analyze a real dataset that motivated our study using the proposed test.     

A cautionary note about assessing the fit of logistic regression models

Logistic regression is a popular method of relating a binary response to one or more potential covariables or risk factors. In 1980, Hosmer and Lemeshow proposed a method for assessing the goodness of fit of logistic regression models. This test is based on a chi-squared statistic that compares the observed and expected cell frequencies in the 2 g table, as found by sorting the observations by predicted probabilities and forming g groups. We have noted that the test may be sensitive to situations where there are low expected cell frequencies. Further, several commonly used statistical packages apply the Hosmer-Lemeshow test, but do so in diff erent ways, and none of the packages we considered alerted the user to the potential difficulty with low expected cell frequencies. An alternative goodness-of-fit test is illustrated which seems to off er an advantage over the popular Hosmer-Lemeshow test, by reducing the likelihood of small expected counts and, potentially, sharpening the interpretation. An example is provided which demonstrates these ideas.     

On equivalence of two variances of a bivariate normal vector

A test for assessing the equivalence of two variances of a bivariate normal vector is constructed. It is uniformly more powerful than the two one-sided tests procedure and the power improvement is substantial. Numerical studies show that it has a type I error close to the test level at most boundary points of the null hypothesis space. One can apply this test to paired difference experiments or 2×2 crossover designs to compare the variances of two populations with two correlated samples. The application of this test on bioequivalence in variability is presented. We point out that bioequivalence in intra-variability implies bioequivalence in variability, however, the latter has a larger power.     

Exact tests and confidence regions in nonlinear regression

Nonlinear regression models with spherically symmetric error vectors and a single nonlinear parameter are considered. On the basis of a new geometric approach, exact one- and two-sided tests and confidence regions for the nonlinear parameter are derived in the cases of known and unknown error variances. A geometric measure representation formula is used to determine the power functions of the tests if the error variance is known and to derive different lower bounds for the power function of a one-sided test in the case of an unknown error variance. The latter can be done quite effectively by constructing and measuring several balls inside the critical region. A numerical study compares the results for different density generating functions of the error distribution.     

A stein two-sample procedure for the general linear model with unequal error variances

The pronerties of the tests and confidence regions for the parameters in the classical general linear model depend upon the equality of the variances of the error terms. The level and power of tests and the confidence coefficients associated with confidence regions are vitiated when the assumption of equality is not true. Even when the error variances are equal the power of tests and the size of confidence regions depend upon the unknown common variance and hence are uncontrollable. This paper presents a two-stage procedure which yields tests and confidence regions which are completely independent of the variances of the errors and hence tests with controllable power and confidence regions of fixed controllable size are obtained.     

A new computational approach test for one-way ANOVA under heteroscedasticity

This paper proposes a new test statistic based on the computational approach test (CAT) for one-way analysis of variance (ANOVA) under heteroscedasticity. The proposed test was compared with other popular tests according to type I error and power of tests under different combinations of variances, means, number of groups and sample sizes. As a result, it was observed that the proposed test yields better results than other tests in many cases.     

Regression model checking with Berkson measurement errors

This paper discusses asymptotically distribution free tests for the lack-of-fit of a parametric regression model in the Berkson measurement error model. These tests are based on a martingale transform of a certain marked empirical process of calibrated residuals. A simulation study is included to assess the effect of measurement error on the proposed test. It is observed that empirical level is more stable across the chosen measurement error variances when fitting a linear model compared to when fitting a nonlinear model, while, in both cases, the empirical power decreases as this error variance increases, against all chosen alternatives.