A two-step rejection procedure for testing multiple hypotheses

This paper considers p-value based step-wise rejection procedures for testing multiple hypotheses. The existing procedures have used constants as critical values at all steps. With the intention of incorporating the exact magnitude of the p-values at the earlier steps into the decisions at the later steps, this paper applies a different strategy that the critical values at the later steps are determined as functions of the p-values from the earlier steps. As a result, we have derived a new equality and developed a two-step rejection procedure following that. The new procedure is a short-cut of a step-up procedure, and it possesses great simplicity. In terms of power, the proposed procedure is generally comparable to the existing ones and exceptionally superior when the largest p-value is anticipated to be less than 0.5.     

Two-sample nonparametric stochastic order inference with an application in plant physiology

In this paper, a new nonparametric methodology is developed for testing whether the changing pattern of a response variable over multiple ordered sub-populations from one treatment group differs with the one from another treatment group. The question is formalized into a nonparametric two-sample comparison problem for the stochastic order among subsamples, through U-statistics with accommodations for zero-inflated distributions. A novel bootstrap procedure is proposed to obtain the critical values with given type I error. Following the procedure, bootstrapped p-values are obtained through simulated samples. It is proven that the distribution of the test statistics is independent from the underlying distributions of the subsamples, when certain sufficient statistics provided. Furthermore, this study also develops a feasible framework for power studies to determine sample sizes, which is necessary in real-world applications. Simulation results suggest that the test is consistent. The methodology is illustrated using a biological experiment with a split-plot design, and significant differences in changing patterns of seed weight between treatments are found with relative small subsample sizes.     

On the Critical Value Behaviour of Multiple Decision Procedures

In this paper we investigate the asymptotic critical value behaviour of certain multiple decision procedures as e.g. simultaneous confidence intervals and simultaneous as well as stepwise multiple test procedures. Supposing that n hypotheses or parameters of interest are under consideration we investigate the critical value behaviour when n increases. More specifically, we answer e.g. the question by which amount the lengths of confidence intervals increase when an additional parameter is added to the statistical analysis. Furthermore, critical values of different multiple decision procedures as for instance step-down and step-up procedures will be compared. Some general theoretic results are derived and applied for various distributions.     

A generalized step-up-down multiple test procedure

A generalization of step-up and step-down multiple test procedures is proposed. This step-up-down procedure is useful when the objective is to reject a specified minimum number, q, out of a family of k hypotheses. If this basic objective is met at the first step, then it proceeds in a step-down manner to see if more than q hypotheses can be rejected. Otherwise it proceeds in a step-up manner to see if some number less than q hypotheses can be rejected. The usual step-down procedure is the special case where q = 1, and the usual step-up procedure is the special case where q = k. Analytical and numerical comparisons between the powers of the step-up-down procedures with different choices of q are made to see how these powers depend on the actual number of false hypotheses. Examples of application include comparing the efficacy of a treatment to a control for multiple endpoints and testing the sensitivity of a clinical trial for comparing the efficacy of a new treatment with a set of standard treatments.     

STEPWISE TESTS WHEN THE TEST STATISTICS ARE INDEPENDENT

Consider the problem of simultaneously testing a nonhierarchical finite family of hypotheses based on independent test statistics. A general stepwise test is defined, of which the well known step-down and step-up tests are special cases. The step-up test is shown to dominate the other stepwise tests, including the step-down test, for situations of practical importance. When testing against two-sided alternatives, it is pointed out that if the step-up test is augmented to include directional decisions then the augmented step-up test controls the type I and III familywise error jointly at the original level q. The definition of the adjusted p values for the step-up test is justified. The results are illustrated by a numerical example.     

Global and multiple test procedures using ordered p-values—a review

This paper reviews global and multiple tests for the combination ofn hypotheses using the orderedp-values of then individual tests. In 1987, Röhmel and Streitberg presented a general method to construct global level α tests based on orderedp-values when there exists no prior knowledge regarding the joint distribution of the corresponding test statistics. In the case of independent test statistics, construction of global tests is available by means of recursive formulae presented by Bicher (1989), Kornatz (1994) and Finner and Roters (1994). Multiple test procedures can be developed by applying the closed test principle using these global tests as building blocks. Liu (1996) proposed representing closed tests by means of “critical matrices” which contain the critical values of the global tests. Within the framework of these theoretical concepts, well-known global tests and multiple test procedures are classified and the relationships between the different tests are characterised.     

False discovery rate control for non-positively regression dependent test statistics

In this paper we present a modification of the Benjamini and Hochberg false discovery rate controlling procedure for testing non-positive dependent test statistics. The new testing procedure makes use of the same series of linearly increasing critical values. Yet, in the new procedure the set of p-values is divided into subsets of positively dependent p-values, and each subset of p-values is separately sorted and compared to the series of critical values. In the first part of the paper we introduce the new testing methodology, discuss the technical issues needed to apply the new approach, and apply it to data from a genetic experiment.     

Improved bonferroni procedures for testing overall and pairwise homogeneity hypotheses

Simes' (1986) improved Bonferroni test is verified by simulations ^?to control the α-level when testing the overall homogeneity hypothesis with all pairwise t statistics in a balanced parallel group design. Similarly, this result was found to hold (for practical purposes) in various underlying distributions other than the normal and in some unbalanced designs. To allow the use of step-up procedures based on pairwise t statistics, simulations were used to verify that Simes' test, when applied to testing multiple subset homogeneity hypotheses with pairwise t statistics also keeps the level ? α. Some robustness as above was found here too. Tables of the simulation results are provided and an example of a step-up Hommel-Shaffer type procedure with pairwise comparisons is given.     

Modified Simes’ critical values under positive dependence

A modification of the critical values of Simes’ test is suggested in this article when the underlying test statistics are multivariate normal with a common non-negative correlation, yielding a more powerful test than the original Simes’ test. A step-up multiple testing procedure with these modified critical values, which is shown to control false discovery rate (FDR), is presented as a modification of the traditional Benjamini–Hochberg (BH) procedure. Simulations were carried out to compare this modified BH procedure with the BH and other modified BH procedures in terms of false non-discovery rate (FNR), 1–FDR–FNR and average power. The present modified BH procedure is observed to perform well compared to others when the test statistics are highly correlated and most of the hypotheses are true.     

A step-down heteroscedastic multiple comparison procedure

All-pairs power in a one-way ANOVA is the probability of detecting all true differences between pairs of means. Ramsey (1978) found that for normal distributions having equal variances, step-down multiple comparison procedures can have substantially more all-pairs power than single-step procedures, such as Tukey’s HSD, when equal sample sizes are randomly sampled from each group. This paper suggests a step-down procedure for the case of unequal variances and compares it to Dunnett's T3 technique. The new procedure is similar in spirit to one of the heteroscedastic procedures described by Hochberg and Tamhane (1987), but it has certain advantages that are discussed in the paper. Included are results on unequal sample sizes.     

Controlling the familywise error rate with plug-in estimator for the proportion of true null hypotheses

Summary.  Estimation of the number or proportion of true null hypotheses in multiple-testing problems has become an interesting area of research. The first important work in this field was performed by Schweder and Spjøtvoll. Among others, they proposed to use plug-in estimates for the proportion of true null hypotheses in multiple-test procedures to improve the power. We investigate the problem of controlling the familywise error rate FWER when such estimators are used as plug-in estimators in single-step or step-down multiple-test procedures. First we investigate the case of independent p -values under the null hypotheses and show that a suitable choice of plug-in estimates leads to control of FWER in single-step procedures. We also investigate the power and study the asymptotic behaviour of the number of false rejections. Although step-down procedures are more difficult to handle we briefly consider a possible solution to this problem. Anyhow, plug-in step-down procedures are not recommended here. For dependent p -values we derive a condition for asymptotic control of FWER and provide some simulations with respect to FWER and power for various models and hypotheses.     

Applications and asymptotic power of marginal-free tests of stochastic vectorial independence

Fully nonparametric tests for the independence between random vectors are studied in this paper. The test statistics are functionals of an empirical process defined as the difference between the joint empirical copula and the product of the empirical copulas associated to the vectors that are suspected to be independent. The validity of a weighted bootstrap procedure is established, which allows for a quick computation of p-values. A special attention is given to the asymptotic behavior of the tests under contiguous sequences of distributions. Finally, a characteristic of the copulas in the Archimedean class in terms of independence of vectors is exploited in order to propose a new goodness-of-fit procedure.     

Sample size determination in step-down and step-up multiple tests for comparing treatments with a control

We address the problem of sample size determination in multiple comparisons of k treatments with a control for step-down and step-up testing, assuming normal data and homogeneous variances. We define power as the probability of correctly rejecting all hypotheses for which the treatment vs. control difference exceeds a specified value. Our paper supplements papers by Hayter and Tamhane (J. Statist. Plann. Inference 27 (1991) 271–290) who solved the problem for one-sided comparisons using the step-down procedure and by Liu (J. Statist. Plann. Inference 62 (1997b) 255–261) who considered the two-sided case using the single-step method. We provide expressions that allow computer evaluation of the power and necessary sample sizes for one- and two-sided tests using either step-down or step-up procedures. Tables are given from which sample sizes to guarantee a specified power can be determined.     

Methods to test for equality of two normal distributions

Statistical tests for two independent samples under the assumption of normality are applied routinely by most practitioners of statistics. Likewise, presumably each introductory course in statistics treats some statistical procedures for two independent normal samples. Often, the classical two-sample model with equal variances is introduced, emphasizing that a test for equality of the expected values is a test for equality of both distributions as well, which is the actual goal. In a second step, usually the assumption of equal variances is discarded. The two-sample t test with Welch correction and the F test for equality of variances are introduced. The first test is solely treated as a test for the equality of central location, as well as the second as a test for the equality of scatter. Typically, there is no discussion if and to which extent testing for equality of the underlying normal distributions is possible, which is quite unsatisfactorily regarding the motivation and treatment of the situation with equal variances. It is the aim of this article to investigate the problem of testing for equality of two normal distributions, and to do so using knowledge and methods adequate to statistical practitioners as well as to students in an introductory statistics course. The power of the different tests discussed in the article is examined empirically. Finally, we apply the tests to several real data sets to illustrate their performance. In particular, we consider several data sets arising from intelligence tests since there is a large body of research supporting the existence of sex differences in mean scores or in variability in specific cognitive abilities.     

QuickMMCTest: quick multiple Monte Carlo testing

Multiple hypothesis testing is widely used to evaluate scientific studies involving statistical tests. However, for many of these tests, p values are not available and are thus often approximated using Monte Carlo tests such as permutation tests or bootstrap tests. This article presents a simple algorithm based on Thompson Sampling to test multiple hypotheses. It works with arbitrary multiple testing procedures, in particular with step-up and step-down procedures. Its main feature is to sequentially allocate Monte Carlo effort, generating more Monte Carlo samples for tests whose decisions are so far less certain. A simulation study demonstrates that for a low computational effort, the new approach yields a higher power and a higher degree of reproducibility of its results than previously suggested methods.     

A survey on multivariate chi-square distributions and their applications in testing multiple hypotheses

We are concerned with three different types of multivariate chi-square distributions. Their members play important roles as limiting distributions of vectors of test statistics in several applications of multiple hypotheses testing. We explain these applications and consider the computation of multiplicity-adjusted p-values under the respective global hypothesis. By means of numerical examples, we demonstrate how much gain in level exhaustion or, equivalently, power can be achieved with corresponding multivariate multiple tests compared with approaches which are only based on univariate marginal distributions and do not take the dependence structure among the test statistics into account. As a further contribution of independent value, we provide an overview of essentially all analytic formulas for computing multivariate chi-square probabilities of the considered types which are available up to present. These formulas were scattered in the previous literature and are presented here in a unified manner.     

Closed testing procedures for all pairwise comparisons in a randomized block design

We consider multiple comparison test procedures among treatment effects in a randomized block design. We propose closed testing procedures based on maximum values of some two-sample t test statistics and based on F test statistics. It is shown that the proposed procedures are more powerful than single-step procedures and the REGW (Ryan/Einot–Gabriel/Welsch)-type tests. Next, we consider the randomized block design under simple ordered restrictions of treatment effects. We propose closed testing procedures based on maximum values of two-sample one-sided t test statistics and based on Batholomew’s statistics for all pairwise comparisons of treatment effects. Although single-step multiple comparison procedures are utilized in general, the power of these procedures is low for a large number of groups. The closed testing procedures stated in the present article are more powerful than the single-step procedures. Simulation studies are performed under the null hypothesis and some alternative hypotheses. In this studies, the proposed procedures show a good performance.     

Multiple comparison procedures for finding non-maximum normal means

Abstract

In this study, we discuss multiple comparison procedures for finding normal means which are not maximum among several normal means. Specifically, we propose the single step procedure, the sequentially rejective step down procedure and the step up procedure. For the single step procedure we determine the critical value for a specified significance level. For the sequentially rejective step down procedure and the step up procedure we determine the critical value at each step of the test for a specified significance level. For three procedures we formulate the power of the test under a specified alternative hypothesis. We give some numerical examples regarding critical values and power of the test intended to compare three procedures.     

Recent and classical goodness-of-fit tests for the Poisson distribution

We give a critical synopsis of classical and recent tests for Poissonity, our emphasis being on procedures which are consistent against general alternatives. Two classes of weighted Cramér–von Mises type test statistics, based on the empirical probability generating function process, are studied in more detail. Both of them generalize already known test statistics by introducing a weighting parameter, thus providing more flexibility with regard to power against specific alternatives. In both cases, we prove convergence in distribution of the statistics under the null hypothesis in the setting of a triangular array of rowwise independent and identically distributed random variables as well as consistency of the corresponding test against general alternatives. Therefore, a sound theoretical basis is provided for the parametric bootstrap procedure, which is applied to obtain critical values in a large-scale simulation study. Each of the tests considered in this study, when implemented via the parametric bootstrap method, maintains a nominal level of significance very closely, even for small sample sizes. The procedures are applied to four well-known data sets.