Multiplicity: discussion points from the Statisticians in the Pharmaceutical Industry multiplicity expert group

In May 2012, the Committee of Health and Medicinal Products issued a concept paper on the need to review the points to consider document on multiplicity issues in clinical trials. In preparation for the release of the updated guidance document, Statisticians in the Pharmaceutical Industry held a one‐day expert group meeting in January 2013. Topics debated included multiplicity and the drug development process, the usefulness and limitations of newly developed strategies to deal with multiplicity, multiplicity issues arising from interim decisions and multiregional development, and the need for simultaneous confidence intervals (CIs) corresponding to multiple test procedures. A clear message from the meeting was that multiplicity adjustments need to be considered when the intention is to make a formal statement about efficacy or safety based on hypothesis tests. Statisticians have a key role when designing studies to assess what adjustment really means in the context of the research being conducted. More thought during the planning phase needs to be given to multiplicity adjustments for secondary endpoints given these are increasing in importance in differentiating products in the market place. No consensus was reached on the role of simultaneous CIs in the context of superiority trials. It was argued that unadjusted intervals should be employed as the primary purpose of the intervals is estimation, while the purpose of hypothesis testing is to formally establish an effect. The opposing view was that CIs should correspond to the test decision whenever possible. Copyright © 2013 John Wiley & Sons, Ltd.     

A power study of goodness-of-fit tests for multivariate normality implemented in R

Multivariate statistical analysis procedures often require data to be multivariate normally distributed. Many tests have been developed to verify if a sample could indeed have come from a normally distributed population. These tests do not all share the same sensitivity for detecting departures from normality, and thus a choice of test is of central importance. This study investigates through simulated data the power of those tests for multivariate normality implemented in the statistic software R and pits them against the variant of testing each marginal distribution for normality. The results of testing two-dimensional data at a level of significance α=5% showed that almost one-third of those tests implemented in R do not have a type I error below this. Other tests outperformed the naive variant in terms of power even when the marginals were not normally distributed. Even though no test was consistently better than all alternatives with every alternative distribution, the energy-statistic test always showed relatively good power across all tested sample sizes.     

A comparison of alternative unit root tests

In this paper we evaluate the performance of three methods for testing the existence of a unit root in a time series, when the models under consideration in the null hypothesis do not display autocorrelation in the error term. In such cases, simple versions of the Dickey-Fuller test should be used as the most appropriate ones instead of the known augmented Dickey-Fuller or Phillips-Perron tests. Through Monte Carlo simulations we show that, apart from a few cases, testing the existence of a unit root we obtain actual type I error and power very close to their nominal levels. Additionally, when the random walk null hypothesis is true, by gradually increasing the sample size, we observe that p-values for the drift in the unrestricted model fluctuate at low levels with small variance and the Durbin-Watson (DW) statistic is approaching 2 in both the unrestricted and restricted models. If, however, the null hypothesis of a random walk is false, taking a larger sample, the DW statistic in the restricted model starts to deviate from 2 while in the unrestricted model it continues to approach 2. It is also shown that the probability not to reject that the errors are uncorrelated, when they are indeed not correlated, is higher when the DW test is applied at 1% nominal level of significance.     

判别分析统计检验体系的探讨

判别分析已越来越受到人们的重视并取得了重要的应用成果，但应用中存在着简单套用的情况，对判别分析的适用性、判别效果的显著性、判别变量的判别能力以及判别函数的判别能力的检验等问题重视不够。为了更好地应用判别分析，就应对判别分析进行统计检验并建立统计检验体系，统计检验体系应包括：判别分析适用性检验，判别效果显著性检验，判别变量的判别能力检验和判别函数的判别能力检验。     

Tests of hypotheses based on ranks in the general linear model

A unified approach is developed for testing hypotheses in the general linear model based on the ranks of the residuals. It complements the nonparametric estimation procedures recently reported in the literature. The testing and estimation procedures together provide a robust alternative to least squares. The methods are similar in spirit to least squares so that results are simple to interpret. Hypotheses concerning a subset of specified parameters can be tested, while the remaining parameters are treated as nuisance parameters. Asymptotically, the test statistic is shown to have a chi-square distribution under the null hypothesis. This result is then extended to cover a sequence of contiguous alternatives from which the Pitman efficacy is derived. The general application of the test requires the consistent estimation of a functional of the underlying distribution and one such estimate is furnished.     

On Non-parametric Testing, the Uniform Behaviour of the t-test, and Related Problems

Abstract.  In this article, we revisit some problems in non-parametric hypothesis testing. First, we extend the classical result of Bahadur & Savage [ Ann. Math. Statist . 25 (1956) 1115] to other testing problems, and we answer a conjecture of theirs. Other examples considered are testing whether or not the mean is rational, testing goodness-of-fit, and equivalence testing. Next, we discuss the uniform behaviour of the classical t -test. For most non-parametric models, the Bahadur–Savage result yields that the size of the t -test is one for every sample size. Even if we restrict attention to the family of symmetric distributions supported on a fixed compact set, the t -test is not even uniformly asymptotically level α . However, the convergence of the rejection probability is established uniformly over a large family with a very weak uniform integrability type of condition. Furthermore, under such a restriction, the t -test possesses an asymptotic maximin optimality property.     

Robustness of the two-sample t-test umdeh violations of the homogeneity of farjance assumptioi

When the two-sample t-test has equal sample slies, it is widely considered to be a robust procedure (with respect to the significaoce level) under violatioa of the assuaptioo of equal variances. This paper is coa-earned with a quantification of the amount of robustness which this procedure has under such violations, The approach is through the concept of "religion of robustness" and the resluts show an extremely strong degree of robustness for the equal an extremely strong degree of robustness for the equal sample size t-test, probably more so than most statistyicians realise. This extremely high level of robustness, however, reduces quickly as the sample sizes begin to vary from equality. The regions of robustnes obtained show that while most users would likely be satisfied with the degree of robustness inherent when the two sample sizes each vary by 10% from equality, most would wish to be much more cautions when the variation is 20%. The study covers sample sizes n₁ -= n ² = 5(5)30(10)50 plus 10% and 20% variations thereof for the two-tailed test and nominal significance levels of 0.01 and 0.05.     

A robust test for testing the correlation coefficient

A test based on Tiku's MML (modified maximum likelihood) estimators is developed for testing that the population correlation coefficient is zero. The test is compared with various other tests and shown to have good Type I error robustness and power for numerous symmetric and skew bivariate populations.     

Computing Probabilities of Type I Error and Type II Error of Sequential Bayesian Procedures for Testing One-Sided Hypotheses

In this article, it is shown how to compute, in an approximated way, probabilities of Type I error and Type II error of sequential Bayesian procedures for testing one-sided null hypotheses. First, some theoretical results are obtained, and then an algorithm is developed for applying these results. The prior predictive density plays a central role in this study.     

A two-step rejection procedure for testing multiple hypotheses

This paper considers p-value based step-wise rejection procedures for testing multiple hypotheses. The existing procedures have used constants as critical values at all steps. With the intention of incorporating the exact magnitude of the p-values at the earlier steps into the decisions at the later steps, this paper applies a different strategy that the critical values at the later steps are determined as functions of the p-values from the earlier steps. As a result, we have derived a new equality and developed a two-step rejection procedure following that. The new procedure is a short-cut of a step-up procedure, and it possesses great simplicity. In terms of power, the proposed procedure is generally comparable to the existing ones and exceptionally superior when the largest p-value is anticipated to be less than 0.5.     

Comparing the overlapping of two independent confidence intervals with a single confidence interval for two normal population parameters

Two overlapping confidence intervals have been used in the past to conduct statistical inferences about two population means and proportions. Several authors have examined the shortcomings of Overlap procedure and have determined that such a method distorts the significance level of testing the null hypothesis of two population means and reduces the statistical power of the test. Nearly all results for small samples in Overlap literature have been obtained either by simulation or by formulas that may need refinement for small sample sizes, but accurate large sample information exists. Nevertheless, there are aspects of Overlap that have not been presented and compared against the standard statistical procedure. This article will present exact formulas for the maximum % overlap of two independent confidence intervals below which the null hypothesis of equality of two normal population means or variances must still be rejected for any sample sizes. Further, the impact of Overlap on the power of testing the null hypothesis of equality of two normal variances will be assessed. Finally, the noncentral t-distribution is used to assess the Overlap impact on type II error probability when testing equality of means for sample sizes larger than 1.     

Deriving a lower bound for the proportion of perceivers in replicated difference tests by means of multiple test theory

Analyzing repeated difference tests aims in significance testing for differences as well as in estimating the mean discrimination ability of the consumers. In addition to the average success probability, the proportion of consumers that may detect the difference between two products and therefore account for any increase of this probability is of interest. While some authors address the first two goals, for the latter one only an estimator directly linked to the average probability seems to be used. However, this may lead to unreasonable results. Therefore we propose a new approach based on multiple test theory. We define a suitable set of hypotheses that is closed under intersection. From this, we derive a series of hypotheses that may be sequentially tested while the overall significance level will not be violated. By means of this procedure we may determine a minimal number of assessors that must have perceived the difference between the products at least once in a while. From this, we can find a conservative lower bound for the proportion of perceivers within the consumers. In several examples, we give some insight into the properties of this new method and show that the knowledge about this lower bound might indeed be valuable for the investigator. Finally, an adaption of this approach for similarity tests will be proposed.     

对应分析统计检验体系探讨

对应分析因其结果的易读性，近些年得到了越来越广泛的应用。为了更好地应用对应分析，提出建立对应分析统计检验体系，包括对应分析适用性的统计检验以及对应分析效果的检验，同时还提出应用对应分析时应注意的其它问题。     

On concurrence of several linear regressions and applications to problems of enzyme kinetics

In comparing several regressions E(y_ij) =α_i + β_ix_ij i = 1, 2, ..., k, j = 1,2, ..., n_i, researchers are generally interested in the following five problems: whether they have (1) equal slope, (2) equal intercept, (3) coincidence, (4) common intersection on X-axis, and (5) common intersection on (X,Y) - plane. Problems (1) - (3) can be put into the framework of the general linear hypothesis and the F-test can be used. However, problems (4) and (5) cannot be put into the general linear hypothesis because they are ratios of parameters. Hence, in this paper we consider the generalized likelihood ratio test for hypothesis testing. An application to an enzyme kinetics problem in Aniline Metabolism is demonstrated     

Multivariate association test for rare variants controlling for cryptic and family relatedness

In genetic studies of complex diseases, multiple measures of related phenotypes are often collected. Jointly analyzing these phenotypes may improve statistical power to detect sets of rare variants affecting multiple traits. In this work, we consider association testing between a set of rare variants and multiple phenotypes in family‐based designs. We use a mixed linear model to express the correlations among the phenotypes and between related individuals. Given the many sources of correlations in this situation, deriving an appropriate test statistic is not straightforward. We derive a vector of score statistics, whose joint distribution is approximated using a copula. This allows us to have closed‐form expressions for the p‐values of several test statistics. A comprehensive simulation study and an application to Genetic Analysis Workshop 18 data highlight the gains associated with joint testing over univariate approaches, especially in the presence of pleiotropy or highly correlated phenotypes. The Canadian Journal of Statistics 47: 90–107; 2019 © 2018 Statistical Society of Canada     

Powerful Parametric Tests Based on Sum‐Functions of Spacings

Assume that we have a sequence of n independent and identically distributed random variables with a continuous distribution function F, which is specified up to a few unknown parameters. In this paper, tests based on sum‐functions of sample spacings are proposed, and large sample theory of the tests are presented under simple null hypotheses as well as under close alternatives. Tests, which are optimal within this class, are constructed, and it is noted that these tests have properties that closely parallel those of the likelihood ratio test in regular parametric models. Some examples are given, which show that the proposed tests work also in situations where the likelihood ratio test breaks down. Extensions to more general hypotheses are discussed.     

A Five-Decision Testing Procedure to Infer the Value of a Unidimensional Parameter

ABSTRACT

A statistical test can be seen as a procedure to produce a decision based on observed data, where some decisions consist of rejecting a hypothesis (yielding a significant result) and some do not, and where one controls the probability to make a wrong rejection at some prespecified significance level. Whereas traditional hypothesis testing involves only two possible decisions (to reject or not a null hypothesis), Kaiser’s directional two-sided test as well as the more recently introduced testing procedure of Jones and Tukey, each equivalent to running two one-sided tests, involve three possible decisions to infer the value of a unidimensional parameter. The latter procedure assumes that a point null hypothesis is impossible (e.g., that two treatments cannot have exactly the same effect), allowing a gain of statistical power. There are, however, situations where a point hypothesis is indeed plausible, for example, when considering hypotheses derived from Einstein’s theories. In this article, we introduce a five-decision rule testing procedure, equivalent to running a traditional two-sided test in addition to two one-sided tests, which combines the advantages of the testing procedures of Kaiser (no assumption on a point hypothesis being impossible) and Jones and Tukey (higher power), allowing for a nonnegligible (typically 20%) reduction of the sample size needed to reach a given statistical power to get a significant result, compared to the traditional approach.     

A comparison of uniformity tests

Problems of goodness-of-fit to a given distribution can usually be reduced to test uniformity. The uniform distribution appears due to natural random events or due to the application of methods for transforming samples from any other distribution to the samples with values uniformly distributed in the interval (0, 1). Thus, one can solve the problem of testing if a sample comes from a given distribution by testing whether its transformed sample is distributed according to the uniform distribution. For this reason, the methods of testing for goodness-of-fit to a uniform distribution have been widely investigated. In this paper, a comparative power analysis of a selected set of statistics is performed in order to give suggestions on which one to use for testing uniformity against the families of alternatives proposed by Stephens [Stephens, M.A., 1974, EDF statistics for goodness of fit and some comparisons. Journal of the American Statistical Association, 69, 730–737.]. Definition and some relevant features of the considered test statistics are given in section 1. Implemented numerical processes to calculate percentage points of every considered statistic are described in section 2. Finally, a Monte Carlo simulation experiment has been carried out to fulfill the mentioned target of this paper.     

Dependency and truncated forms of combinations in multivariate combination-based permutation tests and ordered categorical variables

ABSTRACT

Quite an important problem usually occurs in several multi-dimensional hypotheses testing problems when variables are correlated. In this framework the non-parametric combination (NPC) of a finite number of dependent permutation tests is suitable to cover almost all real situations of practical interest since the dependence relations among partial tests are implicitly captured by the combining procedure itself without the need to specify them [Pesarin F, Salmaso L. Permutation tests for complex data: theory, applications and software. Chichester: Wiley; 2010a]. An open problem related to NPC-based tests is the impact of the dependency structure on combined tests, especially in the presence of categorical variables. This paper’s goal is firstly to investigate the impact of the dependency structure on the possible significance of combined tests in cases of ordered categorical responses using Monte Carlo simulations, then to propose some specific procedures aimed at improving the power of multivariate combination-based permutation tests. The results show that an increasing level of correlation/association among responses negatively affects the power of combination-based multivariate permutation tests. The application of special forms of combination functions based on the truncated product method [Zaykin DV, Zhivotovsky LA, Westfall PH, Weir BS. Truncated product method for combining p-values. Genet Epidemiol. 2002;22:170–185; Dudbridge F, Koeleman BPC. Rank truncated product of p-values, with application to genomewide association scans. Genet Epidemiol. 2003;25:360–366] or on Liptak combination allowed us, using Monte Carlo simulations, to demonstrate the possibility of mitigating the negative effect on power of combination-based multivariate permutation tests produced by an increasing level of correlation/association among responses.