Robust pairwise multiple comparisons under short-tailed symmetric distributions

In one-way ANOVA, most of the pairwise multiple comparison procedures depend on normality assumption of errors. In practice, errors have non-normal distributions so frequently. Therefore, it is very important to develop robust estimators of location and the associated variance under non-normality. In this paper, we consider the estimation of one-way ANOVA model parameters to make pairwise multiple comparisons under short-tailed symmetric (STS) distribution. The classical least squares method is neither efficient nor robust and maximum likelihood estimation technique is problematic in this situation. Modified maximum likelihood (MML) estimation technique gives the opportunity to estimate model parameters in closed forms under non-normal distributions. Hence, the use of MML estimators in the test statistic is proposed for pairwise multiple comparisons under STS distribution. The efficiency and power comparisons of the test statistic based on sample mean, trimmed mean, wave and MML estimators are given and the robustness of the test obtained using these estimators under plausible alternatives and inlier model are examined. It is demonstrated that the test statistic based on MML estimators is efficient and robust and the corresponding test is more powerful and having smallest Type I error.     

Closed testing procedures for all pairwise comparisons in a randomized block design

We consider multiple comparison test procedures among treatment effects in a randomized block design. We propose closed testing procedures based on maximum values of some two-sample t test statistics and based on F test statistics. It is shown that the proposed procedures are more powerful than single-step procedures and the REGW (Ryan/Einot–Gabriel/Welsch)-type tests. Next, we consider the randomized block design under simple ordered restrictions of treatment effects. We propose closed testing procedures based on maximum values of two-sample one-sided t test statistics and based on Batholomew’s statistics for all pairwise comparisons of treatment effects. Although single-step multiple comparison procedures are utilized in general, the power of these procedures is low for a large number of groups. The closed testing procedures stated in the present article are more powerful than the single-step procedures. Simulation studies are performed under the null hypothesis and some alternative hypotheses. In this studies, the proposed procedures show a good performance.     

A sample size selection procedure for pairwise comparisons of means with unequal variances

A sample size selection procedure for paired comparisons of means is presented which controls the half width of the confidence intervals while allowing for unequal variances of treatment means.     

Multiple comparisons of mean vectors with large dimension under general conditions

ABSTRACT

Multiple comparisons for two or more mean vectors are considered when the dimension of the vectors may exceed the sample size, the design may be unbalanced, populations need not be normal, and the true covariance matrices may be unequal. Pairwise comparisons, including comparisons with a control, and their linear combinations are considered. Under fairly general conditions, the asymptotic multivariate distribution of the vector of test statistics is derived whose quantiles can be used in multiple testing. Simulations are used to show the accuracy of the tests. Real data applications are also demonstrated.     

On the use of extreme value tail modeling for generalized pairwise comparisons with censored outcomes

In randomized clinical trials, methods of pairwise comparisons such as the ‘Net Benefit’ or the ‘win ratio’ have recently gained much attention when interests lies in assessing the effect of a treatment as compared to a standard of care. Among other advantages, these methods are usually praised for delivering a treatment measure that can easily handle multiple outcomes of different nature, while keeping a meaningful interpretation for patients and clinicians. For time-to-event outcomes, a recent suggestion emerged in the literature for estimating these treatment measures by providing a natural handling of censored outcomes. However, this estimation procedure may lead to biased estimates when tails of survival functions cannot be reliably estimated using Kaplan–Meier estimators. The problem then extrapolates to the other outcomes incorporated in the pairwise comparison construction. In this work, we suggest to extend the procedure by the consideration of a hybrid survival function estimator that relies on an extreme value tail model through the Generalized Pareto distribution. We provide an estimator of treatment effect measures that notably improves on bias and remains easily apprehended for practical implementation. This is illustrated in an extensive simulation study as well as in an actual trial of a new cancer immunotherapy.     

A revision of the tukey multiple comparisons procedure to control the probability of committing at most one type i error

Halperin et al. (1988) suggested an approach which allows for k Type I errors while using Scheffe's method of multiple comparisons for linear combinations of p means. In this paper we apply the same type of error control to Tukey's method of multiple pairwise comparisons. In fact, the variant of the Tukey (1953) approach discussed here defines the error control objective as assuring with a specified probability that at most one out of the p(p-l)/2 comparisons between all pairs of the treatment means is significant in two-sided tests when an overall null hypothesis (all p means are equal) is true or, from a confidence interval point of view, that at most one of a set of simultaneous confidence intervals for all of the pairwise differences of the treatment means is incorrect. The formulae which yield the critical values needed to carry out this new procedure are derived and the critical values are tabulated. A Monte Carlo study was conducted and several tables are presented to demonstrate the experimentwise Type I error rates and the gains in power furnished by the proposed procedure     

A generalized step-up-down multiple test procedure

A generalization of step-up and step-down multiple test procedures is proposed. This step-up-down procedure is useful when the objective is to reject a specified minimum number, q, out of a family of k hypotheses. If this basic objective is met at the first step, then it proceeds in a step-down manner to see if more than q hypotheses can be rejected. Otherwise it proceeds in a step-up manner to see if some number less than q hypotheses can be rejected. The usual step-down procedure is the special case where q = 1, and the usual step-up procedure is the special case where q = k. Analytical and numerical comparisons between the powers of the step-up-down procedures with different choices of q are made to see how these powers depend on the actual number of false hypotheses. Examples of application include comparing the efficacy of a treatment to a control for multiple endpoints and testing the sensitivity of a clinical trial for comparing the efficacy of a new treatment with a set of standard treatments.     

Improved bonferroni procedures for testing overall and pairwise homogeneity hypotheses

Simes' (1986) improved Bonferroni test is verified by simulations ^?to control the α-level when testing the overall homogeneity hypothesis with all pairwise t statistics in a balanced parallel group design. Similarly, this result was found to hold (for practical purposes) in various underlying distributions other than the normal and in some unbalanced designs. To allow the use of step-up procedures based on pairwise t statistics, simulations were used to verify that Simes' test, when applied to testing multiple subset homogeneity hypotheses with pairwise t statistics also keeps the level ? α. Some robustness as above was found here too. Tables of the simulation results are provided and an example of a step-up Hommel-Shaffer type procedure with pairwise comparisons is given.     

Objective Bayesian multiple comparisons for normal variances

This paper considers the multiple comparisons problem for normal variances. We propose a solution based on a Bayesian model selection procedure to this problem in which no subjective input is considered. We construct the intrinsic and fractional priors for which the Bayes factors and model selection probabilities are well defined. The posterior probability of each model is used as a model selection tool. The behaviour of these Bayes factors is compared with the Bayesian information criterion of Schwarz and some frequentist tests.     

Choosing the best pairwise comparisons of means from non-normal populations,with unequal variances,but equal sample sizes

A Monte Carlo simulation evaluated five pairwise multiple comparison procedures for controlling Type I error rates, any-pair power, and all-pairs power. Realistic conditions of non-normality were based on a previous survey. Variance ratios were varied from 1:1 to 64:1. Procedures evaluated included Tukey's honestly significant difference (HSD) preceded by an F test, the Hayter–Fisher, the Games–Howell preceded by an F test, the Pertiz with F tests, and the Peritz with Alexander–Govern tests. Tukey's procedure shows the greatest robustness in Type I error control. Any-pair power is generally best with one of the Peritz procedures. All-pairs power is best with the Pertiz F test procedure. However, Tukey's HSD preceded by the Alexander–Govern F test may provide the best combination for controlling Type I and power rates in a variety of conditions of non-normality and variance heterogeneity.     

Robust multiple comparisons

The use of several robust estimators of location with their associated variance estimates in a modified T-method for pairwise multiple comparisons between treatment means was compared with the sample mean and variance and with the k-sample rank sum test. The methods were compared with respect to the stability of their experimentwise error rates under a variety of non-normal situations (robustness of validity) and their average confidence interval lengths (robustness of efficiency).     

A fiducial approach to multiple comparisons

Comparing treatment means from populations that follow independent normal distributions is a common statistical problem. Many frequentist solutions exist to test for significant differences amongst the treatment means. A different approach would be to determine how likely it is that particular means are grouped as equal. We developed a fiducial framework for this situation. Our method provides fiducial probabilities that any number of means are equal based on the data and the assumed normal distributions. This methodology was developed when there is constant and non-constant variance across populations. Simulations suggest that our method selects the correct grouping of means at a relatively high rate for small sample sizes and asymptotic calculations demonstrate good properties. Additionally, we have demonstrated the flexibility in the methods ability to calculate the fiducial probability for any number of equal means. This was done by analyzing a simulated data set and a data set measuring the nitrogen levels of red clover plants that were inoculated with different treatments.     

A method of unconstrained multiple comparisons with the best

There are three types of multiple comparisons: all-pairwise multiple comparisons (MCA), multiple comparisons with the best (MCB), and multiple comparisons with a control (MCC). There are also three levels of multiple comparisons inference: confidence sets, subset comparisons, test of homogeneity. In current practice, MCA procedures dominate. In correct attempts at more efficient comparisons, in the form of employing lower level MCA procedures for higher level inference, account for the most frequent abuses in multiple comparisons. A better strategy is to choose the correct type of inference at the level of inference desired. In particular, very often the simulataneous comparisons of each treatment with the best of the other treatments (MCB) suffice. Hsu (1984b) gave simultaneous confidence intervals for θ_i ? max_j≠iθ_j having the simple form [? (Y_i ?max_j≠i Y_j ? C) (Y_i?max_j≠i Y_j + C)⁺]. Those intervals were constrained, sothat even if a treatment is inferred to be the best, no positive bound on how much it is better thatn the rest is given, a somewhat undesirable property. In this article it is shown that by employing a slightly larger critical value, the nonpositivity constraint on the lower bound is removed.     

A Rejection Principle for Sequential Tests of Multiple Hypotheses Controlling Familywise Error Rates

We present a unifying approach to multiple testing procedures for sequential (or streaming) data by giving sufficient conditions for a sequential multiple testing procedure to control the familywise error rate (FWER). Together, we call these conditions a ‘rejection principle for sequential tests’, which we then apply to some existing sequential multiple testing procedures to give simplified understanding of their FWER control. Next, the principle is applied to derive two new sequential multiple testing procedures with provable FWER control, one for testing hypotheses in order and another for closed testing. Examples of these new procedures are given by applying them to a chromosome aberration data set and finding the maximum safe dose of a treatment.     

Truncation of the Bechhofer-Kiefer-Sobel sequential procedure for selecting the multinomial event which has the largest probability

In this article we study the effect of truncation on the performance of an open vector-at-a-time sequential sampling procedure (P^* _B) proposed by Bechhofer, Kiefer and Sobel , for selecting the multinomial event which has the largest probability. The performance of the truncated version (P^{* B} _T) is compared to that of the original basic procedure (P^* _B). The performance characteristics studied include the probability of a correct selection, the expected number of vector-observations (n) to terminate sampling, and the variance of n. Both procedures guarantee the specified probability of a correct selection. Exact results and Monte Carlo sampling results are obtained. It is shown that P^* _{B T}is far superior to P^* _B in terms of E{n} and Var{n}, particularly when the event probabilities are equal.The performance of P^* _{B T} is also compared to that of a closed vector-at-a-time sequential sampling procedure proposed for the same problem by Ramey and Alam; this procedure has here to fore been claimed to be the best one for this problem. It is shown that p^* _{B T} is superior to the Ramey-Alam procedure for most of the specifications of practical interest.     

Pairwise comparisons of treatments with ordinal responses in a two-way setting

In clinical studies, pairwise comparisons are frequently performed to examine differences in efficacy between treatments. The statistical methods of pairwise comparisons are available when treatment responses are measured on an ordinal scale. The Wilcoxon–Mann–Whitney test and the latent normal model are popular examples. However, these procedures cannot be used to compare treatments in parallel groups (a two-way design) when overall type I error must be controlled. In this paper, we explore statistical approaches to the pairwise testing of treatments that satisfy the requirements of a two-way layout. The results of our simulation indicate that the latent normal approach is superior to the Wilcoxon–Mann–Whitney test. Clinical examples are used to illustrate our suggested testing methods.     

Simultaneous pairwise multiple comparisons in a two-way analysis of covariance model

Pairwise comparison procedures are important and popular statistical techniques in many disciplines, such as physiology and agrobiology. In this paper, we seek to derive the statistical methods which enable one to perform pairwise comparisons in a two-way analysis of covariance model. The overall family-wise type I error rate is controlled at a designated level. The procedures are outlined for simultaneous inferences among treatment means. Numerical examples are given to illustrate our testing procedure.     

A Bayesian multiple comparison procedure for simply ordered treatment medians

This article considers a Bayesian hierarchical model for multiple comparisons in linear models where the population medians satisfy a simple order restriction. Representing the asymmetric Laplace distribution as a scale mixture of normals with an exponential mixing density and a continuous prior restricted to order constraints, a Gibbs sampling algorithm for parameter estimation and simultaneous comparison of treatment medians is proposed. Posterior probabilities of all possible hypotheses on the equality/inequality of treatment medians are estimated using Bayes factors that are computed via the Savage-Dickey density ratios. The performance of the proposed median-based model is investigated in the simulated and real datasets. The results show that the proposed method can outperform the commonly used method that is based on treatment means, when data are from nonnormal distributions.     

Nonparametric repeated significance tests for one-way anova with adoptation to multiple comparisons

Based on two-sample rank order statistics, a repeated significance testing procedure for a multi-sample location problem is considered. The asymptotic distribution theory of the proposed tests is given under the null hypothesis as well as under local alternatives. A Bahadur efficiency result of the repeated significance test relative to the terminal test based solely on the target sample size is presented. In the adaptation of the proposed tests to multiple comparisons, an asymptotically equivalent test statistic in terms of the rank estimators of the location parameters is derived from which the Scheffé method of multiple comparisons can be obtained in a convinient way.     

A combination multiple comparisons and subset selection procedure to identify treatments that are strictly inferior to the best

This paper considers the problem of identifying which treatments are strictly worse than the best treatment or treatments in a one-way layout, which has many important applications in screening trials for new product development. A procedure is proposed that selects a subset of the treatments containing only treatments that are known to be strictly worse than the best treatment or treatments. In addition, simultaneous confidence intervals are obtained which provide upper bounds on how inferior the treatments are compared with these best treatments. In this way, the new procedure shares the characteristics of both subset selection procedures and multiple comparison procedures. Some tables of critical points are provided for implementing the new procedure, and some examples of its use are given.