Relations among three parametric multiple testing methods for correlated tests

Multiple endpoints in clinical trials are usually correlated. To control the family-wise type I error rate, both Huque and Alosh's flexible fixed-sequence (FFS) testing method and Li and Mehrotra's adaptive α allocation approach (4A) have taken into account correlations among endpoints. I suggested a weighted multiple testing correction (WMTC) for correlated tests and compared it with FFS. However, the relationship between the 4A method and the FFS method or the relationship between the 4A method and the WMTC method has not been studied. In this paper, simulations are conducted to investigate these relationships. Tentative guidelines to help choosing an appropriate method are provided.     

A flexible fixed-sequence testing method for hierarchically ordered correlated multiple endpoints in clinical trials

Statistical approaches for addressing multiplicity in clinical trials range from the very conservative (the Bonferroni method) to the least conservative the fixed sequence approach. Recently, several authors proposed methods that combine merits of the two extreme approaches. Wiens [2003. A fixed sequence Bonferroni procedure for testing multiple endpoints. Pharmaceutical Statist. 2003, 2, 211–215], for example, considered an extension of the Bonferroni approach where the type I error rate (α)$(\alpha )$ is allocated among the endpoints, however, testing proceeds in a pre-determined order allowing the type I error rate to be saved for later use as long as the null hypotheses are rejected. This leads to a higher power of the test in testing later null hypotheses. In this paper, we consider an extension of Wiens’ approach by taking into account correlations among endpoints for achieving higher flexibility in testing. We show strong control of the family-wise type I error rate for this extension and provide critical values and significance levels for testing up to three endpoints with equal correlations and show how to calculate them for other correlation structures. We also present results of a simulation experiment for comparing the power of the proposed method with those of Wiens’ and others. The results of this experiment show that the magnitude of the gain in power of the proposed method depends on the prospective ordering of testing of the endpoints, the magnitude of the treatment effects of the endpoints and the magnitude of correlation between endpoints. Finally, we consider applications of the proposed method for clinical trials with multiple time points and multiple doses, where correlations among endpoints frequently arise.     

Gatekeeping procedures in dose-response clinical trials based on the Dunnett test

This paper discusses multiple testing procedures in dose-response clinical trials with primary and secondary endpoints. A general gatekeeping framework for constructing multiple tests is proposed, which extends the Dunnett test [Journal of the American Statistical Association 1955; 50: 1096-1121] and Bonferroni-based gatekeeping tests developed by Dmitrienko et al. [Statistics in Medicine 2003; 22:2387-2400]. The proposed procedure accounts for the hierarchical structure of the testing problem; for example, it restricts testing of secondary endpoints to the doses for which the primary endpoint is significant. The multiple testing approach is illustrated using a dose-response clinical trial in patients with diabetes. Monte-Carlo simulations demonstrate that the proposed procedure provides a power advantage over the Bonferroni gatekeeping procedure. The power gain generally increases with increasing correlation among the endpoints, especially when all primary dose-control comparisons are significant.     

A fixed sequence Bonferroni procedure for testing multiple endpoints

A method for controlling the familywise error rate combining the Bonferroni adjustment and fixed testing sequence procedures is proposed. This procedure allots Type I error like the Bonferroni adjustment, but allows the Type I error to accumulate whenever a null hypothesis is rejected. In this manner, power for hypotheses tested later in a prespecified order will be increased. The order of the hypothesis tests needs to be prespecified as in a fixed sequence testing procedure, but unlike the fixed sequence testing procedure all hypotheses can always be tested, allowing for an a priori method of concluding a difference in the various endpoints. An application will be in clinical trials in which mortality is a concern, but it is expected that power to distinguish a difference in mortality will be low. If the effect on mortality is larger than anticipated, this method allows a test with a prespecified method of controlling the Type I error rate. Copyright © 2003 John Wiley & Sons, Ltd.     

Testing noninferiority in three‐armed clinical trials based on likelihood ratio statistics

Clinical noninferiority trials with at least three groups have received much attention recently, perhaps due to the fact that regulatory agencies often require that a placebo group be evaluated along with a new experimental drug and an active control. The authors discuss likelihood ratio tests for binary endpoints and various noninferiority hypotheses. They find that, depending on the particular hypothesis, the test reduces asymptotically either to the intersection‐union test or to a test which follows asymptotically a mixture of generalized chi‐squared distributions. They investigate the performance of this asymptotic test and provide an exact modification. They show that this test considerably outperforms multiple testing methods such as the Bonferroni adjustment with respect to power. They illustrate their methods with a cancer study to compare antiemetic agents. Finally, they discuss the extension of the results to other settings, such as Gaussian endpoints.     

Evaluation of a weighted multiple comparison procedure

We consider the problem of accounting for multiplicity for two correlated endpoints in the comparison of two treatments using weighted hypothesis tests. Various weighted testing procedures are reviewed, and a more powerful method (a variant of the weighted Simes test) is evaluated for the general bivariate normal case and for a particular clinical trial example. Results from these evaluations are summarized and indicate that the weighted methods perform in a manner similar to unweighted methods. Copyright © 2005 John Wiley & Sons, Ltd.     

A simple approach for generating correlated binary variates ∗

Correlated binary data arise frequently in medical as well as other scientific disciplines; and statistical methods, such as generalized estimating equation (GEE), have been widely used for their analysis. The need for simulating correlated binary variates arises for evaluating small sample properties of the GEE estimators when modeling such data. Also, one might generate such data to simulate and study biological phenomena such as tooth decay or periodontal disease. This article introduces a simple method for generating pairs of correlated binary data. A simple algorithm is also provided for generating an arbitrary dimensional random vector of non-negatively correlated binary variates. The method relies on the idea that correlations among the random variables arise as a result of their sharing some common components that induce such correlations. It then uses some properties of the binary variates to represent each variate in terms of these common components in addition to its own elements. Unlike most previous approaches that require solving nonlinear equations or use some distributional properties of other random variables, this method uses only some properties of the binary variate. As no intermediate random variables are required for generating the binary variates, the proposed method is shown to be faster than the other methods. To verify this claim, we compare the computational efficiency of the proposed method with those of other procedures.     

Order restricted inference for multivariate binary data with application to toxicology

In many applications researchers collect multivariate binary response data under two or more, naturally ordered, experimental conditions. In such situations one is often interested in using all binary outcomes simultaneously to detect an ordering among the experimental conditions. To make such comparisons we develop a general methodology for testing for the multivariate stochastic order between K ≥ 2 multivariate binary distributions. The proposed test uses order restricted estimators which, according to our simulation study, are more efficient than the unrestricted estimators in terms of mean squared error. The power of the proposed test was compared with several alternative tests. These included procedures which combine individual univariate tests for order, such as union intersection tests and a Bonferroni based test. We also compared the proposed test with unrestricted Hotelling's T(2) type test. Our simulations suggest that the proposed method competes well with these alternatives. The gain in power is often substantial. The proposed methodology is illustrated by applying it to a two-year rodent cancer bioassay data obtained from the US National Toxicology Program (NTP). Supplemental materials are available online.     

Permutation tests for between-unit fixed effects in multivariate generalized linear mixed models

A permutation testing approach in multivariate mixed models is presented. The solutions proposed allow for testing between-unit effect; they are exact under some assumptions, while approximated in the more general case. The classes of models comprised by this approach include generalized linear models, vector generalized additive models and other nonparametric models based on smoothing. Moreover it does not assume observations of different units to have the same distribution. The extensions to a multivariate framework are presented and discussed. The proposed multivariate tests exploit the dependence among variables, hence increasing the power with respect to other standard solutions (e.g. Bonferroni correction) which combine many univariate tests in an overall one. Examples are given of two applications to real data from psychological and ecological studies; a simulation study provides some insight into the unbiasedness of the tests and their power. The methods were implemented in the R package flip, freely available on CRAN.     

Resampling-based stepwise multiple testing procedures with applications to clinical trial data

Endpoints in clinical trials are often highly correlated. However, the commonly used multiple testing procedures in clinical trials either do not take into consideration the correlations among test statistics or can only exploit known correlations. Westfall and Young constructed a resampling-based stepdown method that implicitly utilizes the correlation structure of test statistics in situations with unknown correlations. However, their method requires a “subset pivotality” assumption. Romano and Wolf proposed a more general stepdown method, which does not require such an assumption. There is at present little experience with the application of such methods in analyzing clinical trial data. We advocate the application of resampling-based multiple testing procedures to clinical trials data when appropriate. We have conjectured that the resampling-based stepdown methods can be extended to a stepup procedure under appropriate assumptions and examined the performance of both stepdown and stepup methods under a variety of correlation structures and distribution types. Results from our simulation studies support the use of the resampling-based methods under various scenarios, including binary data and small samples, with strong control of Family wise type I error rate (FWER). Under positive dependence and for binary data even under independence, the resampling-based methods are more powerful than the Holm and Hochberg methods. Last, we illustrate the advantage of the resampling-based stepwise methods with two clinical trial data examples: a cardiovascular outcome trial and an oncology trial.     

BinNor: An R Package for Concurrent Generation of Binary and Normal Data

This article describes the package BinNor, which is designed for generating multiple binary and normal variables simultaneously given marginal characteristics and association structure via combining well-established results from the random number generation literature, based on the methodology proposed by Demirtas and Doganay.     

Multiple-arm superiority and non-inferiority designs with various endpoints

Multiple-arm dose-response superiority trials are widely studied for continuous and binary endpoints, while non-inferiority designs have been studied recently in two-arm trials. In this paper, a unified asymptotic formulation of a sample size calculation for k-arm (k>0) trials with different endpoints (continuous, binary and survival endpoints) is derived for both superiority and non-inferiority designs. The proposed method covers the sample size calculation for single-arm and k-arm (k> or =2) designs with survival endpoints, which has not been covered in the statistic literature. A simple, closed form for power and sample size calculations is derived from a contrast test. Application examples are provided. The effect of the contrasts on the power is discussed, and a SAS program for sample size calculation is provided and ready to use.     

MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests

Consider testing multiple hypotheses using tests that can only be evaluated by simulation, such as permutation tests or bootstrap tests. This article introduces MMCTest , a sequential algorithm that gives, with arbitrarily high probability, the same classification as a specific multiple testing procedure applied to ideal p‐values. The method can be used with a class of multiple testing procedures that include the Benjamini and Hochberg false discovery rate procedure and the Bonferroni correction controlling the familywise error rate. One of the key features of the algorithm is that it stops sampling for all the hypotheses that can already be decided as being rejected or non‐rejected. MMCTest can be interrupted at any stage and then returns three sets of hypotheses: the rejected, the non‐rejected and the undecided hypotheses. A simulation study motivated by actual biological data shows that MMCTest is usable in practice and that, despite the additional guarantee, it can be computationally more efficient than other methods.     

An evaluation of methods for testing hypotheses relating to two endpoints in a single clinical trial

The issues and dangers involved in testing multiple hypotheses are well recognised within the pharmaceutical industry. In reporting clinical trials, strenuous efforts are taken to avoid the inflation of type I error, with procedures such as the Bonferroni adjustment and its many elaborations and refinements being widely employed. Typically, such methods are conservative. They tend to be accurate if the multiple test statistics involved are mutually independent and achieve less than the type I error rate specified if these statistics are positively correlated. An alternative approach is to estimate the correlations between the test statistics and to perform a test that is conditional on those estimates being the true correlations. In this paper, we begin by assuming that test statistics are normally distributed and that their correlations are known. Under these circumstances, we explore several approaches to multiple testing, adapt them so that type I error is preserved exactly and then compare their powers over a range of true parameter values. For simplicity, the explorations are confined to the bivariate case. Having described the relative strengths and weaknesses of the approaches under study, we use simulation to assess the accuracy of the approximate theory developed when the correlations are estimated from the study data rather than being known in advance and when data are binary so that test statistics are only approximately normally distributed.