A comparison of recent nonparametric methods for testing effects in two-by-two factorial designs

The two-way two-levels crossed factorial design is a commonly used design by practitioners at the exploratory phase of industrial experiments. The F-test in the usual linear model for analysis of variance (ANOVA) is a key instrument to assess the impact of each factor and of their interactions on the response variable. However, if assumptions such as normal distribution and homoscedasticity of errors are violated, the conventional wisdom is to resort to nonparametric tests. Nonparametric methods, rank-based as well as permutation, have been a subject of recent investigations to make them effective in testing the hypotheses of interest and to improve their performance in small sample situations. In this study, we assess the performances of some nonparametric methods and, more importantly, we compare their powers. Specifically, we examine three permutation methods (Constrained Synchronized Permutations, Unconstrained Synchronized Permutations and Wald-Type Permutation Test), a rank-based method (Aligned Rank Transform) and a parametric method (ANOVA-Type Test). In the simulations, we generate datasets with different configurations of distribution of errors, variance, factor's effect and number of replicates. The objective is to elicit practical advice and guides to practitioners regarding the sensitivity of the tests in the various configurations, the conditions under which some tests cannot be used, the tradeoff between power and type I error, and the bias of the power on one main factor analysis due to the presence of effect of the other factor. A dataset from an industrial engineering experiment for thermoformed packaging production is used to illustrate the application of the various methods of analysis, taking into account the power of the test suggested by the objective of the experiment.     

A progressive shift alternative to evaluate nonparametric tests for skewed data

ABSTRACT

When the power of different nonparametric tests is evaluated by simulation, the alternative hypothesis should be carefully designed to ensure validity of the results in the specific research field. In the article, we propose a probit-based progressive shift alternative that is more realistic than the simple shift alternative for skewed non-negative data that occur in many research areas. Our motivation comes from parasitology. The progressive shift alternative is used to compare the power of six location-scale tests and seven commonly used location tests for several skewed theoretical and empirical parasite distributions. It is shown that location-scale tests are more powerful than location tests. Programs for applying the methods studied in the article are freely available for download.     

Multivariate generalizations of jonckheere's test for ordered alternatives

In many dose-response studies, each of several independent groups of animals is treated with a different dose of a substance. Many response variables are then measured on each animal. The distributions of the response variables may be nonnormal, and Jonckheere's (1954) test for ordered alternatives in the one-way layout is sometimes used to test whether the level of a single variable increases with increasing dose. In some applications, however, it is important to consider a set of response variables simultaneously. For instance, an increase in each of certain enzymes in the blood serum may suggest liver damage. To test whether these enzyme levels increase with increasing dose, it may be preferable to consider these enzymes as a group, rather than individually.

I propose two multivariate generalizations of Jonckheere's univariate test. Each multivariate test statistic is a function of coordinate-wise Jonckheere statistics—one a sum, the other a quadratic form. The sum statistic can be used to test the alternative hypothesis that each variable is stochastically increasing with increasing dose. The quadratic form statistic is designed for the more general alternative hypothesis that each variable is stochastically ordered with increasing dose.

For each of these two alternatives, I also propose a multivariate generalization of a normal theory test described by Puri (1965). I examine the asymptotic distributions of the four test statistics under the null hypothesis and under translation alternatives and compare each distribution-free test to the corresponding normal theory test in terms of asymptotic relative efficiency.

The multivariate Jonckheere tests are illustrated using does-response data from a subchronic toxicology study carried out by the National Toxicology Program. Four groups of ten male rats each were treated with increasing doses of vinylidene flouride, and the serum enzymes SDH, SGOT, and SGPT were measured. A comparison of univariate Jonckheere tests on each variable, bivariate tests on SDH and SGOT, and multivariate tests on all three variables gives insight into the behavior of the various procedures.     

Power comparison of data depth-based nonparametric tests for testing equality of locations

In the recent years, the notion of data depth has been used in nonparametric multivariate data analysis since it gives natural ‘centre-outward’ ordering of multivariate data points with respect to the given data cloud. In the literature, various nonparametric tests are developed for testing equality of location of two multivariate distributions based on data depth. Here, we define two nonparametric tests based on two different test statistic for testing equality of locations of two multivariate distributions. In the present work, we compare the performance of these tests with the tests developed by Li and Liu [New nonparametric tests of multivariate locations and scales using data depth. Statist Sci. 2004;(1):686–696] for testing equality of locations of two multivariate distributions. Comparison in terms of power is done for multivariate symmetric and skewed distributions using simulation for three popular depth functions. Application of tests to real life data is provided. Conclusion and recommendations are also provided.     

A review and some new results on permutation testing for multivariate problems

In recent years permutation testing methods have increased both in number of applications and in solving complex multivariate problems. When available permutation tests are essentially of an exact nonparametric nature in a conditional context, where conditioning is on the pooled observed data set which is often a set of sufficient statistics in the null hypothesis. Whereas, the reference null distribution of most parametric tests is only known asymptotically. Thus, for most sample sizes of practical interest, the possible lack of efficiency of permutation solutions may be compensated by the lack of approximation of parametric counterparts. There are many complex multivariate problems, quite common in empirical sciences, which are difficult to solve outside the conditional framework and in particular outside the method of nonparametric combination (NPC) of dependent permutation tests. In this paper we review such a method and its main properties along with some new results in experimental and observational situations (robust testing, multi-sided alternatives and testing for survival functions).     

Performance of nonparametric multiple comparison tests under heteroscedasticity,dependency, and skewed error distribution

In this article, an extensive Monte Carlo simulation study is conducted to evaluate and compare nonparametric multiple comparison tests under violations of classical analysis of variance assumptions. Simulation space of the Monte Carlo study is composed of 288 different combinations of balanced and unbalanced sample sizes, number of groups, treatment effects, various levels of heterogeneity of variances, dependence between subgroup levels, and skewed error distributions under the single factor experimental design. By this large simulation space, we present a detailed analysis of effects of the violations of assumptions on the performance of nonparametric multiple comparison tests in terms of three error and four power measures. Observations of this study are beneficial to decide the optimal nonparametric test according to requirements and conditions of undertaken experiments. When some of the assumptions of analysis of variance are violated and number of groups is small, use of stepwise Steel-Dwass procedure with Holm's approach is appropriate to control type I error at a desired level. Dunn's method should be employed for greater number of groups. When subgroups are unbalanced and number of groups is small, Nemenyi's procedure with Duncan's approach produces high power values. Conover's procedure successfully provides high power values with a small number of unbalanced groups or with a greater number of balanced or unbalanced groups. At the same time, Conover's procedure is unable to control type I error rates.     

Nonparametric multiple comparison procedures for unbalanced two-way layouts

In this paper, we consider nonparametric multiple comparison procedures for unbalanced two-way factorial designs under a pure nonparametric framework. For multiple comparisons of treatments versus a control concerning the main effects or the simple factor effects, the limiting distribution of the associated rank statistics is proven to satisfy the multivariate totally positive of order two condition. Hence, asymptotically the proposed Hochberg procedure strongly controls the familywise type I error rate for the simultaneous testing of the individual hypotheses. In addition, we propose to employ Shaffer's modified version of Holm's stepdown procedure to perform simultaneous tests on all pairwise comparisons regarding the main or simple factor effects and to perform simultaneous tests on all interaction effects. The logical constraints in the corresponding hypothesis families are utilized to sharpen the rejective thresholds and improve the power of the tests.     

Small sample power of multivariate nonparametric tests for monotone trend

Bhattacharyya and Kioiz (1966) propose two multivariate nonparametric tests for monotone trend, one involving coordinate-wise Mann statistics and the other, coordinate-wise Spearman statistics. Dietz and Killeen (1981) propose a different test statistic based on coordinate-wise Mann statistics. The Pitman asymptotic relative efficiency of all three tests with respect to a normal theory competitor equals the cube root of the efficiency of a multivariate signed rank test with respect to Hotelling's T2. In this article, the small sample power of the nonparametric tests, the normal theory test, and a Bonferroni approach involving coordinate-wise univariate Mann or Spearman tests is examined in a simulation study. The Mann statistic of Dietz and Killeen and the Spearman statistic of Bhattacharyya and Klotz are found to perform well under both null and alternative hypotheses     

Tests based on weighted rankings in complete blocks: exact distribution and monte carlo simulation

Quade (1972, 1979) proposed a family of nonparametric tests based on weighted within-block rankings, for testing the hypothesis of no treatment effects in a complete randomized blocks layout. In this paper we give a table of the exact null distribution of these tests when the number of treatments is 3, the number of blocks is less than or equal to 14 and the block scores are linear. Moreover, a Monte Carlo study was performed to compare the powers of these tests with parametric and nonparametric competitors     

Bayesian density regression for discrete outcomes

We develop Bayesian models for density regression with emphasis on discrete outcomes. The problem of density regression is approached by considering methods for multivariate density estimation of mixed scale variables, and obtaining conditional densities from the multivariate ones. The approach to multivariate mixed scale outcome density estimation that we describe represents discrete variables, either responses or covariates, as discretised versions of continuous latent variables. We present and compare several models for obtaining these thresholds in the challenging context of count data analysis where the response may be over‐ and/or under‐dispersed in some of the regions of the covariate space. We utilise a nonparametric mixture of multivariate Gaussians to model the directly observed and the latent continuous variables. The paper presents a Markov chain Monte Carlo algorithm for posterior sampling, sufficient conditions for weak consistency, and illustrations on density, mean and quantile regression utilising simulated and real datasets.     

Bootstrap hypothesis testing in generalized additive models for comparing curves of treatments in longitudinal studies

The study of the effect of a treatment may involve the evaluation of a variable at a number of moments. When assuming a smooth curve for the mean response along time, estimation can be afforded by spline regression, in the context of generalized additive models. The novelty of our work lies in the construction of hypothesis tests to compare two curves of treatments in any interval of time for several types of response variables. The within-subject correlation is not modeled but is considered to obtain valid inferences by the use of bootstrap. We propose both semiparametric and nonparametric bootstrap approaches, based on resampling vectors of residuals or responses, respectively. Simulation studies revealed a good performance of the tests, considering, for the outcome, different distribution functions in the exponential family and varying the correlation between observations along time. We show that the sizes of bootstrap tests are close to the nominal value, with tests based on a standardized statistic having slightly better size properties. The power increases as the distance between curves increases and decreases when correlation gets higher. The usefulness of these statistical tools was confirmed using real data, thus allowing to detect changes in fish behavior when exposed to the toxin microcystin-RR.     

Nonparametric estimation in the illness-death model using prevalent data

We study nonparametric estimation of the illness-death model using left-truncated and right-censored data. The general aim is to estimate the multivariate distribution of a progressive multi-state process. Maximum likelihood estimation under censoring suffers from problems of uniqueness and consistency, so instead we review and extend methods that are based on inverse probability weighting. For univariate left-truncated and right-censored data, nonparametric maximum likelihood estimation can be considerably improved when exploiting knowledge on the truncation distribution. We aim to examine the gain in using such knowledge for inverse probability weighting estimators in the illness-death framework. Additionally, we compare the weights that use truncation variables with the weights that integrate them out, showing, by simulation, that the latter performs more stably and efficiently. We apply the methods to intensive care units data collected in a cross-sectional design, and discuss how the estimators can be easily modified to more general multi-state models.     

Some new results on univariate and multivariate permutation tests for ordinal categorical variables under restricted alternatives

In several sciences, especially when dealing with performance evaluation, complex testing problems may arise due in particular to the presence of multidimensional categorical data. In such cases the application of nonparametric methods can represent a reasonable approach. In this paper, we consider the problem of testing whether a “treatment” is stochastically larger than a “control” when univariate and multivariate ordinal categorical data are present. We propose a solution based on the nonparametric combination of dependent permutation tests (Pesarin in Multivariate permutation test with application to biostatistics. Wiley, Chichester, 2001), on variable transformation, and on tests on moments. The solution requires the transformation of categorical response variables into numeric variables and the breaking up of the original problem’s hypotheses into partial sub-hypotheses regarding the moments of the transformed variables. This type of problem is considered to be almost impossible to analyze within likelihood ratio tests, especially in the multivariate case (Wang in J Am Stat Assoc 91:1676–1683, 1996). A comparative simulation study is also presented along with an application example.     

Two-Step Hypothesis Testing When the Number of Variables Exceeds the Sample Size

Medical images and genetic assays typically generate data with more variables than subjects. Scientists may use a two-step approach for testing hypotheses about Gaussian mean vectors. In the first step, principal components analysis (PCA) selects a set of sample components fewer in number than the sample size. In the second step, applying classical multivariate analysis of variance (MANOVA) methods to the reduced set of variables provides the desired hypothesis tests. Simulation results presented here indicate that success of the PCA in the first step requires nearly all variation to occur in population components far fewer in number than the number of subjects. In the second step, multivariate tests fail to attain reasonable power except in restrictive, favorable cases. The results encourage using other approaches discussed in the article to provide dependable hypothesis testing with high dimension, low sample size data (HDLSS).     

The asymptotic distributions of maximum likelihood ratio test and maximally selected χ2-test in binomial observations

We considered binomial distributed random variables whose parameters are unknown and some of those parameters need to be estimated. We studied the maximum likelihood ratio test and the maximally selected χ²-test to detect if there is a change in the distributions among the random variables. Their limit distributions under the null hypothesis and their asymptotic distributions under the alternative hypothesis were obtained when the number of the observations is fixed. We discussed the properties of the limit distribution and found an efficient way to calculate the probability of multivariate normal random variables. Finally, those results for both tests have been applied to examples of Lindisfarne's data, the Talipes Data. Our conclusions are consistent with other researchers' findings.     

New tests for multivariate normality based on Small's and Srivastava's graphical methods

In this paper, we propose a new measure of fit which can be used in the case of quantile–quantile plots. This measure, when applied to Small's and Srivastava's graphical methods provides two new tests for assessing multivariate normality. For different sample sizes and numbers of variables, the critical values of these tests were evaluated via simulations. The power of the new tests and its comparison with some other tests for multivariate normality are presented herein.     

Permutation Solutions for Multivariate Ranking and Testing with Applications

In this article, we consider permutation methods for multivariate testing on ordered categorical variables based on the nonparametric combination of permutation dependent tests (NPC; Pesarin and Salmaso, 2010). Furthermore, an extension of the nonparametric combination of dependent rankings (Arboretti et al., 2007) is proposed in order to construct a synthesis of composite indicators.

The methodological approaches are applied to a study of risk factors for skin cancer in a cohort of adult patients with heart transplants followed for a minimum of three years after transplantation (Belloni et al, 2004) and to a survey on tourist's opinions about “Tre Cime” Park (District of Sesto Dolomites/Alta Pusteria, Italy).     

Simulated Power Curves for some Location Tests

In this paper we compare the power properties of some location tests. The most widely used such test is Student's t. Recently bootstrap-based tests have received much attention in the literature. A bootstrap version of the t-test will be included in our comparison. Finally, the nonparametric tests based on the idea of permuting the signs will be represented in our comparison. Again, we will initially concentrate on a version of that test based on the mean. The permutation tests predate the bootstrap by about fourty years. Theoretical results of Pitman (1937) and Bickel & Freedman (1981) show that these three methods are asymptotically equivalent if the underlying distribution is symmetric and has finite second moment. In the modern literature, the use of the nonparametric techniques is advocated on the grounds that the size of the test would be either exact, or more nearly exact. In this paper we report on a simulation study that compares the power curves and we show that it is not necessary to use resampling tests with a statistic based on the mean of the sample.     

Bootstrap tests for multivariate directional alternatives

Tests on multivariate means that are hypothesized to be in a specified direction have received attention from both theoretical and applied points of view. One of the most common procedures used to test this cone alternative is the likelihood ratio test (LRT) assuming a multivariate normal model for the data. However, the resulting test for an ordered alternative is biased in that the only usable critical values are bounds on the null distribution. The present paper provides empirical evidence that bootstrapping the null distribution of the likelihood ratio statistic results in a bootstrap test (BT) with comparable power properties without the additional burden of assuming multivariate normality. Additionally, the tests based on the LRT statistic can reject the null hypothesis in favor of the alternative even though the true means are far from the alternative region. The BT also has similar properties for normal and nonnormal data. This anomalous behavior is due to the formulation of the null hypothesis and a possible remedy is to reformulate the null to be the complement of the alternative hypothesis. We discuss properties of a BT for the modified set of hypotheses (MBT) based on a simulation study. The resulting test is conservative in general and in some specific cases has power estimates comparable to those for existing methods. The BT has higher sensitivity but relatively lower specificity, whereas the MBT has higher specificity but relatively lower sensitivity.     

A monte carlo study of the friedman and conover tests in the single-factor repeated measures design

A Monte Carlo study was used to compare the Type I error rates and power of two nonparametric tests against the F test for the single-factor repeated measures model. The performance of the nonparametric Friedman and Conover tests was investigated for different distributions, numbers of blocks and numbers of repeated measures. The results indicated that the type of the distribution has little effect on the ability of the Friedman and Conover tests to control Type error rates. For power, the Friedman and Conover tests tended to agree in rejecting the same false hyporhesis when the design consisted of three repeated measures. However, the Conover test was more powerful than the Friedman test when the number of repeated measures was 4 or 5. Still, the F test is recommended for the single-factor repeated measures model because of its robustness to non-normality and its good power across a range of conditions.