A method of unconstrained multiple comparisons with the best

There are three types of multiple comparisons: all-pairwise multiple comparisons (MCA), multiple comparisons with the best (MCB), and multiple comparisons with a control (MCC). There are also three levels of multiple comparisons inference: confidence sets, subset comparisons, test of homogeneity. In current practice, MCA procedures dominate. In correct attempts at more efficient comparisons, in the form of employing lower level MCA procedures for higher level inference, account for the most frequent abuses in multiple comparisons. A better strategy is to choose the correct type of inference at the level of inference desired. In particular, very often the simulataneous comparisons of each treatment with the best of the other treatments (MCB) suffice. Hsu (1984b) gave simultaneous confidence intervals for θ_i ? max_j≠iθ_j having the simple form [? (Y_i ?max_j≠i Y_j ? C) (Y_i?max_j≠i Y_j + C)⁺]. Those intervals were constrained, sothat even if a treatment is inferred to be the best, no positive bound on how much it is better thatn the rest is given, a somewhat undesirable property. In this article it is shown that by employing a slightly larger critical value, the nonpositivity constraint on the lower bound is removed.     

Sample sizes for trials involving multiple correlated must-win comparisons

In Clinical trials involving multiple comparisons of interest, the importance of controlling the trial Type I error is well-understood and well-documented. Moreover, when these comparisons are themselves correlated, methodologies exist for accounting for the correlation in the trial design, when calculating the trial significance levels. However, less well-documented is the fact that there are some circumstances where multiple comparisons affect the Type II error rather than the Type I error, and failure to account for this, can result in a reduction in the overall trial power. In this paper, we describe sample size calculations for clinical trials involving multiple correlated comparisons, where all the comparisons must be statistically significant for the trial to provide evidence of effect, and show how such calculations have to account for multiplicity in the Type II error. For the situation of two comparisons, we provide a result which assumes a bivariate Normal distribution. For the general case of two or more comparisons we provide a solution using inflation factors to increase the sample size relative to the case of a single outcome. We begin with a simple case of two comparisons assuming a bivariate Normal distribution, show how to factor in correlation between comparisons and then generalise our findings to situations with two or more comparisons. These methods are easy to apply, and we demonstrate how accounting for the multiplicity in the Type II error leads, at most, to modest increases in the sample size.     

Multiple Comparisons Among Mean Vectors When the Dimension is Larger Than the Total Sample Size

We consider pairwise multiple comparisons and multiple comparisons with a control among mean vectors for high-dimensional data under the multivariate normality. For such cases, the statistics based on the Dempster trace criterion are given, and also their approximate upper percentiles are derived by using the Bonferroni’s inequality. Finally, the accuracy of their approximate values is evaluated by Monte Carlo simulation.     

On the conservative simultaneous confidence procedures for multiple comparisons among mean vectors

In this paper, conservative simultaneous confidence intervals for multiple comparisons among mean vectors in multivariate normal distributions are considered. Some properties of the multivariate Tukey–Kramer procedure for pairwise comparisons and the conservative simultaneous confidence procedure for comparisons with a control are presented. Particularly, the upper bound for the conservativeness of the simultaneous confidence procedure for comparisons with a control is obtained. Finally, numerical results by Monte Carlo simulations and an example to illustrate the procedure are given.     

On the Skew-Normal-Cauchy Distribution

A fixed effects one-way layout model of analysis of variance is considered where the variances are taken to be possibly unequal. Conservative single-stage procedures based on Banerjee’s method for the solution of the Behrens-Fisher problem are proposed for the following multiple comparisons problems: 1) all pairwise comparisons with a control population mean, and 2) all pairwise comparisons and all linear contrasts among the means. Since these procedures are likely to be very conservative in practice, approximate procedures based on Welch’s method for the solution of the Behrens-Fisher problem are suggested as alternatives. Monte Carlo studies indicate that the latter are much less conservative and hence may be better in practice. Both these sets of procedures need only the tables of the Student’s t-distribution for their application and are very simple to use. Exact two-stage procedures are proposed for the following multiple comparisons problems: 1) all pairwise comparisons and all linear contrasts among the means, and 2) all linear combinations of the means.     

On multiple comparisons among K samples subject to unequal patterns of censorship

Two multiple comparisons procedures for determining which of K arbitrarily censored populations differ from each other are proposed. The procedures are based on multiple comparisons using the generalized Wilcoxon and log-rank statistics. The procedures incorporate a pairwise ranking scheme, rather than the joint ranking scheme proposed by Breslow (1970) and Crowley and Thomas (1975). A conservative testing method suggested by an inequality due to ?idák (1967) is given; a numerical example is presented.     

Multiple comparisons of mean vectors with large dimension under general conditions

ABSTRACT

Multiple comparisons for two or more mean vectors are considered when the dimension of the vectors may exceed the sample size, the design may be unbalanced, populations need not be normal, and the true covariance matrices may be unequal. Pairwise comparisons, including comparisons with a control, and their linear combinations are considered. Under fairly general conditions, the asymptotic multivariate distribution of the vector of test statistics is derived whose quantiles can be used in multiple testing. Simulations are used to show the accuracy of the tests. Real data applications are also demonstrated.     

Overview of Friedman’s Test and Post-hoc Analysis

When the null hypothesis of Friedman’s test is rejected, there is a wide variety of multiple comparisons that can be used to determine which treatments differ from each other. We will discuss the contexts where different multiple comparisons should be applied, when the population follows some discrete distributions commonly used to model count data in biological and ecological fields. Our simulation study shows that sign test is very conservative. Fisher’s LSD and Tukey’s HSD tests computed with ranks are the most liberal. Theoretical considerations are illustrated with data of the Azores Buzzard (Buteo buteo rothschildi) population from Azores, Portugal.     

A revision of the tukey multiple comparisons procedure to control the probability of committing at most one type i error

Halperin et al. (1988) suggested an approach which allows for k Type I errors while using Scheffe's method of multiple comparisons for linear combinations of p means. In this paper we apply the same type of error control to Tukey's method of multiple pairwise comparisons. In fact, the variant of the Tukey (1953) approach discussed here defines the error control objective as assuring with a specified probability that at most one out of the p(p-l)/2 comparisons between all pairs of the treatment means is significant in two-sided tests when an overall null hypothesis (all p means are equal) is true or, from a confidence interval point of view, that at most one of a set of simultaneous confidence intervals for all of the pairwise differences of the treatment means is incorrect. The formulae which yield the critical values needed to carry out this new procedure are derived and the critical values are tabulated. A Monte Carlo study was conducted and several tables are presented to demonstrate the experimentwise Type I error rates and the gains in power furnished by the proposed procedure     

Robust pairwise multiple comparisons under short-tailed symmetric distributions

In one-way ANOVA, most of the pairwise multiple comparison procedures depend on normality assumption of errors. In practice, errors have non-normal distributions so frequently. Therefore, it is very important to develop robust estimators of location and the associated variance under non-normality. In this paper, we consider the estimation of one-way ANOVA model parameters to make pairwise multiple comparisons under short-tailed symmetric (STS) distribution. The classical least squares method is neither efficient nor robust and maximum likelihood estimation technique is problematic in this situation. Modified maximum likelihood (MML) estimation technique gives the opportunity to estimate model parameters in closed forms under non-normal distributions. Hence, the use of MML estimators in the test statistic is proposed for pairwise multiple comparisons under STS distribution. The efficiency and power comparisons of the test statistic based on sample mean, trimmed mean, wave and MML estimators are given and the robustness of the test obtained using these estimators under plausible alternatives and inlier model are examined. It is demonstrated that the test statistic based on MML estimators is efficient and robust and the corresponding test is more powerful and having smallest Type I error.     

Simultaneous confidence intervals for pairwise multiple comparisons in a two-way unbalanced design with unequal variances

In this research, we propose simultaneous confidence intervals for all pairwise multiple comparisons in a two-way unbalanced design with unequal variances, using a parametric bootstrap approach. Simulation results show that Type 1 error of the multiple comparison test is close to the nominal level even for small samples. They also show that the proposed method outperforms Tukey–Kramer procedure when variances are heteroscedastic and group sizes are unequal.     

Statistics in the High School Curriculum

Recent work, spearheaded by Charles Dunnett (1980a), leads to the conclusion that the Tukey-Kramer (TK) method (popularly known as “Kramer's Method”) is the recommended multiple comparisons procedure for the simultaneous estimation of all pairwise differences of means in an imbalanced one-way ANOVA design with homogeneous variances. Nine other multiple comparisons methods are compared to each other and to the TK method using the criteria of conservativeness, narrowness of confidence intervals, robustness, and ease of use. The degree of superiority of the TK method over these methods, especially over the popular Bonferroni method, is sufficient to warrant recommending its use. Because of the lack of robustness of the TK method in heterogeneous variance cases, other methods are recommended.     

Step-up and step-down methods for testing multiple hypotheses in sequential experiments

Sequential methods are developed for testing multiple hypotheses, resulting in a statistical decision for each individual test and controlling the familywise error rate and the familywise power in the strong sense. Extending the ideas of step-up and step-down methods for multiple comparisons to sequential designs, the new techniques improve over the Bonferroni and closed testing methods proposed earlier by a substantial reduction of the expected sample size.     

Multiple Comparisons Controlling Expected Number of False Discoveries

In this paper, it is put forward that the task of designing a procedure for a set of multiple comparisons should be considered as a decision-making under uncertainty. Due to this motivation, for the problem of multiple comparisons, we considered another error rate to be controlled, called PFER (per-family error rate), which requests that the expected number of false rejections of a test procedure should be bounded no more than a prespecified level k. Although PFER was proposed by Tukey in 1953, there is not much studying about it so far. We first present Bonferroni procedure (single-step) and then build two step-up procedures with one having generic critical values and another using critical values in BH (Benjamini and Hochberg) type. These procedures are compared through simulations.     

A generalized scheffé method of multiple comparisons

Standard resulrs on the extrema of quotients of quadratic forms are extended to the non-negative definite case. The maximum and the set over which it is achieved are characterized explicitly both in terms of generalized inverse matrices and generalized eigenvalues. These results become the basis of Scheffe type multiple comparisons in the usual way. To demonstrate their application to statistics with singular covariance matrices, the method is detailed for Mantel-Haenszel, Breslow, and Cox statistics. An example is presented illustrating a situation where the proposed Scheffe type comparisons may be better than the pairwise method.     

An exploration of aspects of Bayesian multiple testing

There has been increased interest of late in the Bayesian approach to multiple testing (often called the multiple comparisons problem), motivated by the need to analyze DNA microarray data in which it is desired to learn which of potentially several thousand genes are activated by a particular stimulus. We study the issue of prior specification for such multiple tests; computation of key posterior quantities; and useful ways to display these quantities. A decision-theoretic approach is also considered.     

Gauss-laguerre dynamics:an efficient method of computing integrals in multiple comparisons

Computing with Gauss-Laguerre quadratures is a common practice when computing the distribution of the studentized version of many statistics used in normal theory multiple comparisons. We study the dynamics of the points associated with such quadrature and apply the results to the simple example of the studentized maximum modulus. Because most distributions used in multiple comparisons require relatively expensive numerical quadrature for evaluation of their non-studentized integrand, the efficiency of the suggested method is clear as it requires fewer integrand evaluations than the standard Gauss-Laguerre quadrature.     

Nonparametric approaches to testing and multiple comparisons in a one-way anova

Two questions of interest involving nonparametric multiple comparisons are considered. The first question concerns whether it is appropriate to use a multiple comparison procedure as a test of the equality of k treatments, and if it is, which procedure performs best as a test. Our results show that for smaller k values some multiple comparison procedures perform well as tests. The second question concerns whether a joint ranking or a separate ranking multiple comparison procedure performs better as a test and as a device for treatment separation. We find that the joint ranking procedure does slightly better as a test, but for treatment separation the answer depends on the situation.     

Multiple comparisons for an unbalanced a × b design under mixed models with interaction

Consider a two-way factorial experiment involving a “treatment” factor A with fixed effects, a “blocking” factor B with random effects, and interaction effects perhaps non-negligible. The degree of balance required for multiple comparison procedures to be applicable for the comparison of the treatment effects using ordinary least-squares estimates is investigated. For main effects to be estimated independently of MSAB, a sufficient condition is that the design consist of identical blocks, a strong condition of proportional frequencies. Surprisingly, under this condition of proportional frequencies, MSAB does not provide an appropriate variance estimate for inferences on each treatment contrast, even though the statistics F = MSA/MSAB is appropriate for testing equality of the treatment effects. In short, when factor B is random, standard methods of multiple comparisons apply using the interaction mean square MSAB as a variance estimator only when the treatment-block incidences n_n are constant. Nevertheless, for designs with identical blocks, appropriate variance estimates can be identified to allow for conservative or approximate multiple comparisons. This is illustrated for certain treatment balanced designs for comparisons with a control.     

Nonparametric multiple comparison procedures for unbalanced two-way layouts

In this paper, we consider nonparametric multiple comparison procedures for unbalanced two-way factorial designs under a pure nonparametric framework. For multiple comparisons of treatments versus a control concerning the main effects or the simple factor effects, the limiting distribution of the associated rank statistics is proven to satisfy the multivariate totally positive of order two condition. Hence, asymptotically the proposed Hochberg procedure strongly controls the familywise type I error rate for the simultaneous testing of the individual hypotheses. In addition, we propose to employ Shaffer's modified version of Holm's stepdown procedure to perform simultaneous tests on all pairwise comparisons regarding the main or simple factor effects and to perform simultaneous tests on all interaction effects. The logical constraints in the corresponding hypothesis families are utilized to sharpen the rejective thresholds and improve the power of the tests.