Choosing the best pairwise comparisons of means from non-normal populations,with unequal variances,but equal sample sizes

A Monte Carlo simulation evaluated five pairwise multiple comparison procedures for controlling Type I error rates, any-pair power, and all-pairs power. Realistic conditions of non-normality were based on a previous survey. Variance ratios were varied from 1:1 to 64:1. Procedures evaluated included Tukey's honestly significant difference (HSD) preceded by an F test, the Hayter–Fisher, the Games–Howell preceded by an F test, the Pertiz with F tests, and the Peritz with Alexander–Govern tests. Tukey's procedure shows the greatest robustness in Type I error control. Any-pair power is generally best with one of the Peritz procedures. All-pairs power is best with the Pertiz F test procedure. However, Tukey's HSD preceded by the Alexander–Govern F test may provide the best combination for controlling Type I and power rates in a variety of conditions of non-normality and variance heterogeneity.     

Asymptotic Inference for Performance Fees and the Predictability of Asset Returns

In this article, we provide analytical, simulation, and empirical evidence on a test of equal economic value from competing predictive models of asset returns. We define economic value using the concept of a performance fee—the amount an investor would be willing to pay to have access to an alternative predictive model used to make investment decisions. We establish that this fee can be asymptotically normal under modest assumptions. Monte Carlo evidence shows that our test can be accurately sized in reasonably large samples. We apply the proposed test to predictions of the U.S. equity premium.     

Tables for pairwise multiple comparisons using shaffer's modifiedsequentially-rejective procedure

A sequentially rejective (SR) testing procedure introduced by Holm (1979) and modified (MSR) by Shaffer (1986) is considered for testing all pairwise mean comparisons.For such comparisons, both the SR and MSR methods require that the observed test statistics be ordered and compared, each in turn, to appropriate percentiles on Student's t distribution.For the MSR method these percentiles are based on the maximum number of true null hypotheses remaining at each stage of the sequential procedure, given prior significance at previous stages, A function is developed for determining this number from the number of means being tested and the stage of the test.For a test of all pairwise comparisons, the logical implications which follow the rejection of a null hypothesis renders the MSR procedure uniformly more powerful than the SR procedure.Tables of percentiles for comparing K means, 3 < K < 6, using the MSR method are presented.These tables use Sidak's (1967) multiplicative inequality and simplify the use of t he MSR procedure.Several modifications to the MSR are suggested as a means of further increasing the power for testing the pairwise comparisons.General use of the MSR and the corresponding function for testing other parameters besides the mean is discussed.     

Nonparametric Applications of Shaffer's Extension of Dennett's Procedure

Shaffer's extensions and generalization of Dunnett's procedure are shown to be applicable in several nonparametric data analyses. Applications are considered within the context of the Kruskal-Wallis one-way analysis of variance (ANOVA) test for ranked data, Friedman's two-way ANOVA test for ranked data, and Cochran's test of change for dichotomous data.     

Improved bonferroni procedures for testing overall and pairwise homogeneity hypotheses

Simes' (1986) improved Bonferroni test is verified by simulations ^?to control the α-level when testing the overall homogeneity hypothesis with all pairwise t statistics in a balanced parallel group design. Similarly, this result was found to hold (for practical purposes) in various underlying distributions other than the normal and in some unbalanced designs. To allow the use of step-up procedures based on pairwise t statistics, simulations were used to verify that Simes' test, when applied to testing multiple subset homogeneity hypotheses with pairwise t statistics also keeps the level ? α. Some robustness as above was found here too. Tables of the simulation results are provided and an example of a step-up Hommel-Shaffer type procedure with pairwise comparisons is given.     

Nonparametric approaches to testing and multiple comparisons in a one-way anova

Two questions of interest involving nonparametric multiple comparisons are considered. The first question concerns whether it is appropriate to use a multiple comparison procedure as a test of the equality of k treatments, and if it is, which procedure performs best as a test. Our results show that for smaller k values some multiple comparison procedures perform well as tests. The second question concerns whether a joint ranking or a separate ranking multiple comparison procedure performs better as a test and as a device for treatment separation. We find that the joint ranking procedure does slightly better as a test, but for treatment separation the answer depends on the situation.     

Manipulating Statistical Hypotheses and Tests as Boolean Functions

Statistical hypotheses and test statistics are Boolean functions that can be manipulated using the tools of Boolean algebra. These tools are particularly useful for exploring multiple comparisons or simultaneous inference theory, in which multiparameter hypotheses or multiparameter test statistics may be decomposed into combinations of uniparameter hypotheses or uniparameter tests. These concepts are illustrated with both finite and infinite decompositions of familiar multiparameter hypotheses and tests. The corresponding decompositions of acceptance regions and rejection regions are also shown. Finally, the close relationship between hypothesis and test decompositions and Roy's union—intersection principle is demonstrated by a derivation of the union—intersection test of the univariate general linear hypothesis.     

Generalized confidence intervals compatible with the Min test for simultaneous comparisons of one subpopulation to several other subpopulations

A problem where one subpopulation is compared with several other subpopulations in terms of means with the goal of estimating the smallest difference between the means commonly arises in biology, medicine, and many other scientific fields. A generalization of Strass-burger-Bretz-Hochberg approach for two comparisons is presented for cases with three and more comparisons. The method allows constructing an interval estimator for the smallest mean difference, which is compatible with the Min test. An application to a fluency-disorder study is illustrated. Simulations confirmed adequate probability coverage for normally distributed outcomes for a number of designs.     

Multiple comparisons of several populations with more than one control with respectto scale parameter

In this paper, one-sided and two-sided test procedures for comparing several treatments with more than one control with respect to scale parameter are proposed. The proposed test procedures are inverted to obtain the associated simultaneous confidence intervals. The multiple comparisons of test treatments with the best control are also developed. The computation of the critical points, required to implement the proposed procedures, is discussed by taking the normal probability model. Applications of the proposed test procedures to two-parameter exponential probability model are also demonstrated.     

Multiple comparisons of mean vectors with large dimension under general conditions

ABSTRACT

Multiple comparisons for two or more mean vectors are considered when the dimension of the vectors may exceed the sample size, the design may be unbalanced, populations need not be normal, and the true covariance matrices may be unequal. Pairwise comparisons, including comparisons with a control, and their linear combinations are considered. Under fairly general conditions, the asymptotic multivariate distribution of the vector of test statistics is derived whose quantiles can be used in multiple testing. Simulations are used to show the accuracy of the tests. Real data applications are also demonstrated.     

A multi sample test for the equality of coefficients of variation in normal populations

Two tests are derived for the hypothesis that the coefficients of variation of k normal populations are equal. The k samples may be of unequal size. The first test is the likelihood ratio test with the usual X²-approximation. A simulation study shows that the small sample behaviour under the null hypothesis is unsatisfactory. An alternative test, based on the sample coefficients of variation, appears to have somewhat better properties.     

A Nonparametric Test for the Ordered Alternative Based on Kendall's Correlation Coefficient

Several methods have been developed for testing the ordered alternative. These include the Jonckheere–Terpstra (JT) test (Jonckheere, 1954 Jonckheere , A. R. ( 1954 ). A distribution free k-sample test against ordered alternatives . Biometrika 41 : 133 – 145 .[Crossref], [Web of Science ®] , [Google Scholar]; Terpstra, 1952 Terpstra , T. ( 1952 ). The asymptotic normality and consistency of Kendall's test against trend when ties are present in one ranking . Indigationes Mathematicae 14 : 327 – 333 . [Google Scholar]), a modified JT test (MJT) (Tryon and Hettmansperger, 1987 Tryon , V. P. , Hettmansperger , T. P. ( 1987 ). A class of nonparametric tests for homogeneity against ordered alternatives . Annals of Statistics 1 : 1061 – 1070 . [Google Scholar]), and a test proposed by Terpstra and Magel (TM) (Terpstra and Magel, 2003 Terpstra , J. T. , Magel , R. C. ( 2003 ). A new nonparametric test for the ordered alternative problem . Journal of Nonparametric Statistics 15 : 289 – 301 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]), among others. This article proposes a new method for testing the ordered alternative. The proposed test is based on Kendall's tau statistic. The asymptotic distribution of the test statistic is given. A Monte Carlo simulation study is conducted comparing the estimated powers of the proposed test with existing tests under a variety of sample sizes and distributions.     

C194. A nearly flat prior,some unsolved problexs,anb soxe comments about improper priors

Two different approaches to obtaining finite-sample corrections to score tests are the analytical and the computational approaches. The former is based either on a Bartletttype correction to the test statistic or on the inversion of an Edgeworth expansion to its null distribution. The latter, on the other hand, is usually based on a bootstrapping resampling scheme. This paper provides a numerical comparison of the size and power properties of these two approaches both under correct model specification and under model misspecification.     

The optimal size of choice sets in choice experiments

In this paper, we establish the optimal size of the choice sets in generic choice experiments for asymmetric attributes when estimating main effects only. We give an upper bound for the determinant of the information matrix when estimating main effects and all two-factor interactions for binary attributes. We also derive the information matrix for a choice experiment in which the choice sets are of different sizes and use this to determine the optimal sizes for the choice sets.     

An empirical comparison of several methods for testing the equality of dependent proportions

Three methods for testing the equality of nonindependent proportions were compared with, the use of Monte Carlo techniques. The three methods included Cochran's test, an ANOVA F test, and Hotelling's T² test. With respect to empirical significance levels, the ANOVA F test is recommended as the preferred method of analysis.

Oftentimes an experimenter is interested in testing the equality of several proportions. When the proportions are independent Kemp and Butcher (1972) and Butcher and Kemp (1974) compared several methods for analysing large sample binomial data for the case of a 3 x 3 factorial design without replication. In addition, Levy and Narula (1977) compared many of the same methods for analyzing binomial data; however, Levy and Narula investigated the relative utility of the methods for small sample sizes.     

False discovery rate control for non-positively regression dependent test statistics

In this paper we present a modification of the Benjamini and Hochberg false discovery rate controlling procedure for testing non-positive dependent test statistics. The new testing procedure makes use of the same series of linearly increasing critical values. Yet, in the new procedure the set of p-values is divided into subsets of positively dependent p-values, and each subset of p-values is separately sorted and compared to the series of critical values. In the first part of the paper we introduce the new testing methodology, discuss the technical issues needed to apply the new approach, and apply it to data from a genetic experiment.     

A comparison of some conditional and unconditional exact tests for 2x2 contingency tables

In 1935, R.A. Fisher published his well-known “exact” test for 2x2 contingency tables. This test is based on the conditional distribution of a cell entry when the rows and columns marginal totals are held fixed. Tocher (1950) and Lehmann (1959) showed that Fisher s test, when supplemented by randomization, is uniformly most powerful among all the unbiased tests UMPU). However, since all the practical tests for 2x2 tables are nonrandomized - and therefore biased the UMPU test is not necessarily more powerful than other tests of the same or lower size. Inthis work, the two-sided Fisher exact test and the UMPU test are compared with six nonrandomized unconditional exact tests with respect to their power. In both the two-binomial and double dichotomy models, the UMPU test is often less powerful than some of the unconditional tests of the same (or even lower) size. Thus, the assertion that the Tocher-Lehmann modification of Fisher's conditional test is the optimal test for 2x2 tables is unjustified.     

Multiple Comparisons Controlling Expected Number of False Discoveries

In this paper, it is put forward that the task of designing a procedure for a set of multiple comparisons should be considered as a decision-making under uncertainty. Due to this motivation, for the problem of multiple comparisons, we considered another error rate to be controlled, called PFER (per-family error rate), which requests that the expected number of false rejections of a test procedure should be bounded no more than a prespecified level k. Although PFER was proposed by Tukey in 1953, there is not much studying about it so far. We first present Bonferroni procedure (single-step) and then build two step-up procedures with one having generic critical values and another using critical values in BH (Benjamini and Hochberg) type. These procedures are compared through simulations.     

An extreme null distribution approach to the problem of paired comparisons

David (1963) and Davidson & Farquhar(1976) contain extensive bibliographies of proposed approaches to problems involving paired comparisons. However, each of the discussed methods that is based on a hypothesis test, relies heavily on the assumption that all paired comparisons are made independently. In this paper we eliminate this assumption and develop a new procedure based on an adaptation of a statistic considered by Kendall & Babington Smith (1940). We show that their original test procedure substantially underestimates the true significance level if the comparisons are not made independently. Our modification utilizes the approach developed in Costello & Wolfe (1985) for the problem of agreement between two groups of judges and relies heavily on computer-generated tables.     

Omnibus testing and gene filtration in microarray data analysis

When thousands of tests are performed simultaneously to detect differentially expressed genes in microarray analysis, the number of Type I errors can be immense if a multiplicity adjustment is not made. However, due to the large scale, traditional adjustment methods require very stringen significance levels for individual tests, which yield low power for detecting alterations. In this work, we describe how two omnibus tests can be used in conjunction with a gene filtration process to circumvent difficulties due to the large scale of testing. These two omnibus tests, the D-test and the modified likelihood ratio test (MLRT), can be used to investigate whether a collection of P-values has arisen from the Uniform(0,1) distribution or whether the Uniform(0,1) distribution contaminated by another Beta distribution is more appropriate. In the former case, attention can be directed to a smaller part of the genome; in the latter event, parameter estimates for the contamination model provide a frame of reference for multiple comparisons. Unlike the likelihood ratio test (LRT), both the D-test and MLRT enjoy simple limiting distributions under the null hypothesis of no contamination, so critical values can be obtained from standard tables. Simulation studies demonstrate that the D-test and MLRT are superior to the AIC, BIC, and Kolmogorov-Smirnov test. A case study illustrates omnibus testing and filtration.