Consistent variable selection via the optimal discovery procedure in multiple testing

In this paper, we translate variable selection for linear regression into multiple testing, and select significant variables according to testing result. New variable selection procedures are proposed based on the optimal discovery procedure (ODP) in multiple testing. Due to ODP’s optimality, if we guarantee the number of significant variables included, it will include less non significant variables than marginal p-value based methods. Consistency of our procedures is obtained in theory and simulation. Simulation results suggest that procedures based on multiple testing have improvement over procedures based on selection criteria, and our new procedures have better performance than marginal p-value based procedures.     

Gatekeeping procedures with clinical trial applications

The objective of this paper is to give an overview of a relatively new area of multiplicity research that deals with the analysis of hierarchically ordered multiple objectives. Testing procedures for this problem are known as gatekeeping procedures and have found a variety of applications in clinical trials. This paper reviews main classes of these procedures, including serial and parallel gatekeeping procedures, and tree gatekeeping procedures that account for logical restrictions among multiple objectives. We focus on procedures based on marginal p-values; extensions to procedures that exploit the joint distribution of the p-values are also noted. Clinical trial examples are used to illustrate the procedures and their important properties.     

On multiple comparisons among K samples subject to unequal patterns of censorship

Two multiple comparisons procedures for determining which of K arbitrarily censored populations differ from each other are proposed. The procedures are based on multiple comparisons using the generalized Wilcoxon and log-rank statistics. The procedures incorporate a pairwise ranking scheme, rather than the joint ranking scheme proposed by Breslow (1970) and Crowley and Thomas (1975). A conservative testing method suggested by an inequality due to ?idák (1967) is given; a numerical example is presented.     

Closed testing procedures for all pairwise comparisons in a randomized block design

We consider multiple comparison test procedures among treatment effects in a randomized block design. We propose closed testing procedures based on maximum values of some two-sample t test statistics and based on F test statistics. It is shown that the proposed procedures are more powerful than single-step procedures and the REGW (Ryan/Einot–Gabriel/Welsch)-type tests. Next, we consider the randomized block design under simple ordered restrictions of treatment effects. We propose closed testing procedures based on maximum values of two-sample one-sided t test statistics and based on Batholomew’s statistics for all pairwise comparisons of treatment effects. Although single-step multiple comparison procedures are utilized in general, the power of these procedures is low for a large number of groups. The closed testing procedures stated in the present article are more powerful than the single-step procedures. Simulation studies are performed under the null hypothesis and some alternative hypotheses. In this studies, the proposed procedures show a good performance.     

Power and stability comparisons of multiple testing procedures with false discovery rate control

High-throughput data analyses are widely used for examining differential gene expression, identifying single nucleotide polymorphisms, and detecting methylation loci. False discovery rate (FDR) has been considered a proper type I error rate to control for discovery-based high-throughput data analysis. Various multiple testing procedures have been proposed to control the FDR. The power and stability properties of some commonly used multiple testing procedures have not been extensively investigated yet, however. Simulation studies were conducted to compare power and stability properties of five widely used multiple testing procedures at different proportions of true discoveries for various sample sizes for both independent and dependent test statistics. Storey's two linear step-up procedures showed the best performance among all tested procedures considering FDR control, power, and variance of true discoveries. Leukaemia and ovarian cancer microarray studies were used to illustrate the power and stability characteristics of these five multiple testing procedures with FDR control.     

Multiple Comparison Procedures: The Practical Solution

A practicing statistician looks at the multiple comparison controversy and related issues through the eyes of the users. The concept of consistency is introduced and discussed in relation to five of the more common multiple comparison procedures. All of the procedures are found to be inconsistent except the simplest procedure, the unrestricted least significant difference (LSD) procedure (or multiple t test). For this and other reasons the unrestricted LSD procedure is recommended for general use, with the proviso that it should be viewed as a hypothesis generator rather than as a method for simultaneous hypothesis generation and testing. The implications for Scheffé's test for general contrasts are also discussed, and a new recommendation is made.     

New one-stage multiple comparison procedures with the average for exponential location parameters under heteroscedasticity

ABSTRACT

In this paper, we present new one-stage multiple comparison procedures with the average for location parameters of two-parameter exponential distributions under heteroscedasticity by modifying the existing one proposed by Wu [One stage multiple comparisons with the average for exponential location parameters under heteroscedasticity. Comput Stat Data Anal. 2013;68:352–360] with unequal sample sizes. A simulation study is done and the results show that the proposed procedures have shorter confidence length with coverage probabilities closer to the nominal ones. At last, an example of comparing the survival days of patients for four categories of lung cancer is given to demonstrate the proposed procedures.     

Combined multiple testing by censored empirical likelihood

We propose a new procedure for combining multiple tests in samples of right-censored observations. The new method is based on multiple constrained censored empirical likelihood where the constraints are formulated as linear functionals of the cumulative hazard functions. We prove a version of Wilks’ theorem for the multiple constrained censored empirical likelihood ratio, which provides a simple reference distribution for the test statistic of our proposed method. A useful application of the proposed method is, for example, examining the survival experience of different populations by combining different weighted log-rank tests. Real data examples are given using the log-rank and Gehan-Wilcoxon tests. In a simulation study of two sample survival data, we compare the proposed method of combining tests to previously developed procedures. The results demonstrate that, in addition to its computational simplicity, the combined test performs comparably to, and in some situations more reliably than previously developed procedures. Statistical software is available in the R package ‘emplik’.     

A Rejection Principle for Sequential Tests of Multiple Hypotheses Controlling Familywise Error Rates

We present a unifying approach to multiple testing procedures for sequential (or streaming) data by giving sufficient conditions for a sequential multiple testing procedure to control the familywise error rate (FWER). Together, we call these conditions a ‘rejection principle for sequential tests’, which we then apply to some existing sequential multiple testing procedures to give simplified understanding of their FWER control. Next, the principle is applied to derive two new sequential multiple testing procedures with provable FWER control, one for testing hypotheses in order and another for closed testing. Examples of these new procedures are given by applying them to a chromosome aberration data set and finding the maximum safe dose of a treatment.     

Testing non-inferiority and superiority for two endpoints for several treatments with a control

Some multiple comparison procedures are described for multiple armed studies. The procedures are appropriate for testing all hypotheses for comparing two endpoints and multiple test arms to a single control group, for example three different fixed doses compared to a placebo. The procedure assumes that among the two endpoints, one is designated as a primary endpoint such that for a given treatment arm, no hypothesis for the secondary endpoint can be rejected unless the hypothesis for the primary endpoint was rejected. The procedures described control the family-wise error rate in the strong sense at a specified level α.     

Multiple Hypotheses Testing with Weights

In this paper we offer a multiplicity of approaches and procedures for multiple testing problems with weights. Some rationale for incorporating weights in multiple hypotheses testing are discussed. Various type-I error-rates and different possible formulations are considered, for both the intersection hypothesis testing and the multiple hypotheses testing problems. An optimal per family weighted error-rate controlling procedure a la Spjotvoll (1972) is obtained. This model serves as a vehicle for demonstrating the different implications of the approaches to weighting. Alternative approach es to that of Holm (1979) for family-wise error-rate control with weights are discussed, one involving an alternative procedure for family-wise error-rate control, and the other involving the control of a weighted family-wise error-rate. Extensions and modifications of the procedures based on Simes (1986) are given. These include a test of the overall intersec tion hypothesis with general weights, and weighted sequentially rejective procedures for testing the individual hypotheses. The false discovery rate controlling approach and procedure of Benjamini & Hochberg (1995) are extended to allow for different weights.     

A method of unconstrained multiple comparisons with the best

There are three types of multiple comparisons: all-pairwise multiple comparisons (MCA), multiple comparisons with the best (MCB), and multiple comparisons with a control (MCC). There are also three levels of multiple comparisons inference: confidence sets, subset comparisons, test of homogeneity. In current practice, MCA procedures dominate. In correct attempts at more efficient comparisons, in the form of employing lower level MCA procedures for higher level inference, account for the most frequent abuses in multiple comparisons. A better strategy is to choose the correct type of inference at the level of inference desired. In particular, very often the simulataneous comparisons of each treatment with the best of the other treatments (MCB) suffice. Hsu (1984b) gave simultaneous confidence intervals for θ_i ? max_j≠iθ_j having the simple form [? (Y_i ?max_j≠i Y_j ? C) (Y_i?max_j≠i Y_j + C)⁺]. Those intervals were constrained, sothat even if a treatment is inferred to be the best, no positive bound on how much it is better thatn the rest is given, a somewhat undesirable property. In this article it is shown that by employing a slightly larger critical value, the nonpositivity constraint on the lower bound is removed.     

Control-variate methods for comparison with a standard

Comparison with a standard is a general multiple comparison problem, where each system is required to be compared with a single system, referred to as a ‘standard’, as well as with other alternative systems. Screening procedures specially designed to be used for comparison with a standard have been proposed to find a subset that includes all the systems better than the standard in terms of the expected performance. Selection procedures are derived to determine the best system among a number of systems that are better than the standard, or to select the standard when it is equal to or better than the other alternatives. We develop new procedures for screening and selection through the use of two variance reduction techniques, common random numbers and control variates, which are particularly useful in the context of simulation experiments. Empirical results and a realistic example are also provided to compare our procedures with the existing ones.     

On the Critical Value Behaviour of Multiple Decision Procedures

In this paper we investigate the asymptotic critical value behaviour of certain multiple decision procedures as e.g. simultaneous confidence intervals and simultaneous as well as stepwise multiple test procedures. Supposing that n hypotheses or parameters of interest are under consideration we investigate the critical value behaviour when n increases. More specifically, we answer e.g. the question by which amount the lengths of confidence intervals increase when an additional parameter is added to the statistical analysis. Furthermore, critical values of different multiple decision procedures as for instance step-down and step-up procedures will be compared. Some general theoretic results are derived and applied for various distributions.     

Multiple comparisons with a control in repeated measures incomplete block designs using R-estimators

New multiple comparison with a control (MCC) procedures are developed in repeated measures incomplete block design settings based on R-estimates. It is assumed that the errors within each subject are exchangeable random variables. The R-estimators of the treatment effects are obtained by minimizing a sum of Jaeckel (1972)-type dispersion functions. Based on the R-estimators, Dunnett-type multiple comparison procedures are developed for comparing test-treatments with a control-treatment. Under exchangeable errors, it is demonstrated that for Cox-type designs, the new procedures are more efficient than the existing nonparametric procedures. The new MCC procedures are applied to a data set in a clinical trial which consists of patients with reversible obstructive pulmonary disease.     

Adaptive graph‐based multiple testing procedures

Multiple testing procedures defined by directed, weighted graphs have recently been proposed as an intuitive visual tool for constructing multiple testing strategies that reflect the often complex contextual relations between hypotheses in clinical trials. Many well‐known sequentially rejective tests, such as (parallel) gatekeeping tests or hierarchical testing procedures are special cases of the graph based tests. We generalize these graph‐based multiple testing procedures to adaptive trial designs with an interim analysis. These designs permit mid‐trial design modifications based on unblinded interim data as well as external information, while providing strong family wise error rate control. To maintain the familywise error rate, it is not required to prespecify the adaption rule in detail. Because the adaptive test does not require knowledge of the multivariate distribution of test statistics, it is applicable in a wide range of scenarios including trials with multiple treatment comparisons, endpoints or subgroups, or combinations thereof. Examples of adaptations are dropping of treatment arms, selection of subpopulations, and sample size reassessment. If, in the interim analysis, it is decided to continue the trial as planned, the adaptive test reduces to the originally planned multiple testing procedure. Only if adaptations are actually implemented, an adjusted test needs to be applied. The procedure is illustrated with a case study and its operating characteristics are investigated by simulations. Copyright © 2014 John Wiley & Sons, Ltd.     

Closure properties of classes of multiple testing procedures

Statistical discoveries are often obtained through multiple hypothesis testing. A variety of procedures exists to evaluate multiple hypotheses, for instance the ones of Benjamini–Hochberg, Bonferroni, Holm or Sidak. We are particularly interested in multiple testing procedures with two desired properties: (solely) monotonic and well-behaved procedures. This article investigates to which extent the classes of (monotonic or well-behaved) multiple testing procedures, in particular the subclasses of so-called step-up and step-down procedures, are closed under basic set operations, specifically the union, intersection, difference and the complement of sets of rejected or non-rejected hypotheses. The present article proves two main results: First, taking the union or intersection of arbitrary (monotonic or well-behaved) multiple testing procedures results in new procedures which are monotonic but not well-behaved, whereas the complement or difference generally preserves neither property. Second, the two classes of (solely monotonic or well-behaved) step-up and step-down procedures are closed under taking the union or intersection, but not the complement or difference.     

Exact methods for testing homogeneity of proportions for multiple groups of paired binary data

We consider four exact procedures to test the homogeneity of proportions for correlated multiple clustered data. Exact procedures are compared with the asymptotic approach based on the score statistic. We use a real example from a double-blind clinical trial studying the treatment of otitis media to illustrate the various test procedures and provide extensive numerical studies to compare procedures with regards to Type I error rates and powers under the unconditional framework. The exact unconditional procedure based on estimation followed by maximization is generally more powerful than other procedures.     

A step-down heteroscedastic multiple comparison procedure

All-pairs power in a one-way ANOVA is the probability of detecting all true differences between pairs of means. Ramsey (1978) found that for normal distributions having equal variances, step-down multiple comparison procedures can have substantially more all-pairs power than single-step procedures, such as Tukey’s HSD, when equal sample sizes are randomly sampled from each group. This paper suggests a step-down procedure for the case of unequal variances and compares it to Dunnett's T3 technique. The new procedure is similar in spirit to one of the heteroscedastic procedures described by Hochberg and Tamhane (1987), but it has certain advantages that are discussed in the paper. Included are results on unequal sample sizes.