QuickMMCTest: quick multiple Monte Carlo testing

Multiple hypothesis testing is widely used to evaluate scientific studies involving statistical tests. However, for many of these tests, p values are not available and are thus often approximated using Monte Carlo tests such as permutation tests or bootstrap tests. This article presents a simple algorithm based on Thompson Sampling to test multiple hypotheses. It works with arbitrary multiple testing procedures, in particular with step-up and step-down procedures. Its main feature is to sequentially allocate Monte Carlo effort, generating more Monte Carlo samples for tests whose decisions are so far less certain. A simulation study demonstrates that for a low computational effort, the new approach yields a higher power and a higher degree of reproducibility of its results than previously suggested methods.     

TESTING FOR CONTINUOUS LOCAL MARTINGALES USING THE CROSSING TREE

We present statistical tests for the continuous martingale hypothesis; that is, for whether an observed process is a continuous local martingale, or equivalently a continuous time‐changed Brownian motion. Our technique is based on the concept of the crossing tree. Simulation experiments are used to assess the power of the tests, which is generally higher than that of recently proposed tests using the estimated quadratic variation (i.e. realized volatility). In particular, the crossing tree shows significantly higher power with shorter data sets. We then show results from applying the methodology to five high‐frequency currency exchange rate data sets from 2003. For four of them we show that at small time‐scales (less than 15 minutes or so) the continuous martingale hypothesis is rejected, but not so at larger time‐scales. For the fifth, the hypothesis is rejected at small time‐scales and at some moderate time‐scales, but not all.     

Testing nonparametric statistical functionals with applications to rank tests

The present paper discusses how nonparametric tests can be deduced from statistical functionals. Efficient and asymptotically most powerful maximin tests are derived. Their power function is calculated under implicit alternatives given by the functional for one – and two – sample testing problems. It is shown that the asymptotic power function does not depend on the special implicit direction of the alternatives but only on quantities of the functional. The present approach offers a nonparametric principle how to construct common rank tests as the Wilcoxon test, the log rank test, and the median test from special two-sample functionals. In addition it is shown that studentized permutation tests yield asymptotically valid tests for certain extended null hypotheses given by functionals which are strictly larger than the common i.i.d. null hypothesis. As example tests concerning the von Mises functional and the Wilcoxon two-sample test are treated.     

Asymptotic hypothesis test to compare likelihood ratios of multiple diagnostic tests in unpaired designs

The accuracy of a binary diagnostic test is usually measured in terms of its sensitivity and its specificity. Other measures of the performance of a diagnostic test are the positive and negative likelihood ratios, which quantify the increase in knowledge about the presence of the disease through the application of a diagnostic test, and which depend on the sensitivity and specificity of the diagnostic test. In this article, we construct an asymptotic hypothesis test to simultaneously compare the positive and negative likelihood ratios of two or more diagnostic tests in unpaired designs. The hypothesis test is based on the logarithmic transformation of the likelihood ratios and on the chi-square distribution. Simulation experiments have been carried out to study the type I error and the power of the constructed hypothesis test when comparing two and three binary diagnostic tests. The method has been extended to the case of multiple multi-level diagnostic tests.     

Nonparametric Two‐Sample Tests of the Marginal Mark Distribution with Censored Marks

Occasionally, investigators collect auxiliary marks at the time of failure in a clinical study. Because the failure event may be censored at the end of the follow‐up period, these marked endpoints are subject to induced censoring. We propose two new families of two‐sample tests for the null hypothesis of no difference in mark‐scale distribution that allows for arbitrary associations between mark and time. One family of proposed tests is a nonparametric extension of an existing semi‐parametric linear test of the same null hypothesis while a second family of tests is based on novel marked rank processes. Simulation studies indicate that the proposed tests have the desired size and possess adequate statistical power to reject the null hypothesis under a simple change of location in the marginal mark distribution. When the marginal mark distribution has heavy tails, the proposed rank‐based tests can be nearly twice as powerful as linear tests.     

A unified sequential test procedure for simultaneous testing the equality of several binomial proportions to a specified standard

In this article, we propose a unified sequentially rejective test procedure for testing simultaneously the equality of several independent binomial proportions to a specified standard. The proposed test procedure is general enough to include some well-known multiple testing procedures such as the Ordinary Bonferroni procedure, Hochberg procedure and Rom procedure. It involves multiple tests of significance based on the simple binomial tests (exact or approximate) which can be easily found in many elementary standard statistics textbooks. Unlike the traditional Chi-square test of the overall hypothesis, the procedure can identify the subset of the binomial proportions, which are different from the prespecified standard with the control of the familywise type I error rate. Moreover, the power computation of the procedure is provided and the procedure is illustrated by two real examples from an ecological study and a carcinogenicity study.     

Estimating and Testing a Structured Covariance Matrix for Three-Level Multivariate Data

This article considers an approach to estimating and testing a new Kronecker product covariance structure for three-level (multiple time points (p), multiple sites (u), and multiple response variables (q)) multivariate data. Testing of such covariance structure is potentially important for high dimensional multi-level multivariate data. The hypothesis testing procedure developed in this article can not only test the hypothesis for three-level multivariate data, but also can test many different hypotheses, such as blocked compound symmetry, for two-level multivariate data as special cases. The tests are implemented with two real data sets.     

Model specification tests against non-nested alternatives

Non-nested hypothesis tests provide a way to test the specification of an econometric model against the evidence provided by one or more non-nested alternatives. This paper surveys the recent literature on non-nested hypothesis testing in the context of regression and related models. Much of the purely statistical 1iterature which has evolved from the fundamental work of Cox (1961, 1962) is discussed briefly or not at all. Instead, emphasis is placed on those techniques which are easy to employ in practice and are likely to be useful to applied workers.     

RAO'S SCORE TESTS IN SURVIVAL ANALYSIS: EXAMPLES AND BARTLETT TYPE ADJUSTMENTS

Score method in hypothesis testing is one of Professor C. R. Rao's great contributions to statistics. It provides a simple and unified way to test some simple and composite hypotheses in many statistical problems. Some popular tests in statistical practice derived with the help of intuitions can be shown as score tests under some statistical models. The subject-years test and log-rank test in survival analysis are two of the examples. In this paper, we first introduce these two examples. After formulating these two tests as score tests, we then review some recent results on the Bartlett type adjustments for these tests.     

Model specification tests against non-nested alternatives

Non-nested hypothesis tests provide a way to test the specification of an econometric model against the evidence provided by one or more non-nested alternatives. This paper surveys the recent literature on non-nested hypothesis testing in the context of regression and related models. Much of the purely statistical 1iterature which has evolved from the fundamental work of Cox (1961, 1962) is discussed briefly or not at all. Instead, emphasis is placed on those techniques which are easy to employ in practice and are likely to be useful to applied workers.     

The teaching of the concepts of statistical tests of hypotheses to non-statisticians

The logic underlying the formulation of statistical tests of hypothesis can be counterintuitive for the non-mathematician, e.g. to test whether two treatments are different, why suppose they are equal? When introducing the topic of hypothesis testing, it is easy to present the formal fiamework for the testing procedure without explaining the logic behind it. In courses for statisticians, one may often (unjustifiably) rely on the understanding of probability concepts as a foundation for understanding statistical inference, but in courses taught to non-statisticians where there is minimal discussion of probability, it is essential that explanations must be based on concepts the students can readily understand. The method proposed here for teaching the concept of hypothesis testing makes an analogy to the judicial system, whereby a person is assumed innocent until sufficient evidence warrants a verdict of guilty. Analogies for the different elements of statistical tests are presented and discussed, together with a classroom fiamework for discussion of statistical tests.     

Multiple hypothesis test for parameter constancy based on recursive residuals

This article presents a multiple hypothesis test procedure that combines two well known tests for structural change in the linear regression model, the CUSUM test and the recursive t test. The CUSUM test is run through the sequence of recursive residuals as usual; if the CUSUM plot does not violate the critical lines, one more step is taken to perform the t test for hypothesis of zero mean based on all recursive residuals. The asymptotic size of this multiple hypothesis test is derived; power simulation results suggest that it outperforms the traditional CUSUM test and complements other tests that are currently stressed in econometrics.     

Multiple hypothesis test for parameter constancy based on recursive residuals

This article presents a multiple hypothesis test procedure that combines two well known tests for structural change in the linear regression model, the CUSUM test and the recursive t test. The CUSUM test is run through the sequence of recursive residuals as usual; if the CUSUM plot does not violate the critical lines, one more step is taken to perform the t test for hypothesis of zero mean based on all recursive residuals. The asymptotic size of this multiple hypothesis test is derived; power simulation results suggest that it outperforms the traditional CUSUM test and complements other tests that are currently stressed in econometrics.     

Two new estimators for the proportion of true null hypotheses in multiple test

In multiple hypothesis test, an important problem is estimating the proportion of true null hypotheses. Existing methods are mainly based on the p-values of the single tests. In this paper, we propose two new estimations for this proportion. One is a natural extension of the commonly used methods based on p-values and the other is based on a mixed distribution. Simulations show that the first method is comparable with existing methods and performs better under some cases. And the method based on a mixed distribution can get accurate estimators even if the variance of data is large or the difference between the null hypothesis and alternative hypothesis is very small.     

LOPSIDED REASONING ON LOPSIDED TESTS AND MULTIPLE COMPARISONS

For those who have not recognized the disparate natures of tests of statistical hypotheses and tests of scientific hypotheses, one‐tailed statistical tests of null hypotheses such as ?≤ 0 or ?≥ 0 have often seemed a reasonable procedure. We earlier reviewed the many grounds for not regarding them as such. To have at least some power for detection of effects in the unpredicted direction, several authors have independently proposed the use of lopsided (also termed split‐tailed, directed or one‐and‐a‐half‐tailed) tests, two‐tailed tests with α partitioned unequally between the two tails of the test statistic distribution. We review the history of these proposals and conclude that lopsided tests are never justified. They are based on the same misunderstandings that have led to massive misuse of one‐tailed tests as well as to much needless worry, for more than half a century, over the various so‐called ‘multiplicity problems’. We discuss from a neo‐Fisherian point of view the undesirable properties of multiple comparison procedures based on either (i) maximum potential set‐wise (or family‐wise) type I error rates (SWERs), or (ii) the increasingly fashionable, maximum potential false discovery rates (FDRs). Neither the classical nor the newer multiple comparison procedures based on fixed maximum potential set‐wise error rates are helpful to the cogent analysis and interpretation of scientific data.     

A Five-Decision Testing Procedure to Infer the Value of a Unidimensional Parameter

ABSTRACT

A statistical test can be seen as a procedure to produce a decision based on observed data, where some decisions consist of rejecting a hypothesis (yielding a significant result) and some do not, and where one controls the probability to make a wrong rejection at some prespecified significance level. Whereas traditional hypothesis testing involves only two possible decisions (to reject or not a null hypothesis), Kaiser’s directional two-sided test as well as the more recently introduced testing procedure of Jones and Tukey, each equivalent to running two one-sided tests, involve three possible decisions to infer the value of a unidimensional parameter. The latter procedure assumes that a point null hypothesis is impossible (e.g., that two treatments cannot have exactly the same effect), allowing a gain of statistical power. There are, however, situations where a point hypothesis is indeed plausible, for example, when considering hypotheses derived from Einstein’s theories. In this article, we introduce a five-decision rule testing procedure, equivalent to running a traditional two-sided test in addition to two one-sided tests, which combines the advantages of the testing procedures of Kaiser (no assumption on a point hypothesis being impossible) and Jones and Tukey (higher power), allowing for a nonnegligible (typically 20%) reduction of the sample size needed to reach a given statistical power to get a significant result, compared to the traditional approach.     

Decisions on Seasonal Unit Roots

Decisions on the presence of seasonal unit roots in economic time series are commonly taken on the basis of statistical hypothesis tests. Some of these tests have absence of unit roots as the null hypothesis, while others use unit roots as their null. Following a suggestion by Hylleberg (1995) to combine such tests in order to reach a clearer conclusion, we evaluate the merits of such test combinations on the basis of a Bayesian decision setup. We find that the potential gains over a pure application of the most common test due to Hylleberg et al. (1990) can be small.     

A comparative study of confidence intervals to assess biosimilarity from analytical data

Assessment of analytical similarity of tier 1 quality attributes is based on a set of hypotheses that tests the mean difference of reference and test products against a margin adjusted for standard deviation of the reference product. Thus, proper assessment of the biosimilarity hypothesis requires statistical tests that account for the uncertainty associated with the estimations of the mean differences and the standard deviation of the reference product. Recently, a linear reformulation of the biosimilarity hypothesis has been proposed, which facilitates development and implementation of statistical tests. These statistical tests account for the uncertainty in the estimation process of all the unknown parameters. In this paper, we survey methods for constructing confidence intervals for testing the linearized reformulation of the biosimilarity hypothesis and also compare the performance of the methods. We discuss test procedures using confidence intervals to make possible comparison among recently developed methods as well as other previously developed methods that have not been applied for demonstrating analytical similarity. A computer simulation study was conducted to compare the performance of the methods based on the ability to maintain the test size and power, as well as computational complexity. We demonstrate the methods using two example applications. At the end, we make recommendations concerning the use of the methods.     

Testing for Common Features

This article introduces a class of statistical tests for the hypothesis that some feature that is present in each of several variables is common to them. Features are data properties such as serial correlation, trends, seasonality, heteroscedasticity, autoregressive conditional hetero-scedasticity, and excess kurtosis. A feature is detected by a hypothesis test taking no feature as the null, and a common feature is detected by a test that finds linear combinations of variables with no feature. Often, an exact asymptotic critical value can be obtained that is simply a test of overidentifying restrictions in an instrumental variable regression. This article tests for a common international business cycle.     

Nonparametric repeated significance tests for one-way anova with adoptation to multiple comparisons

Based on two-sample rank order statistics, a repeated significance testing procedure for a multi-sample location problem is considered. The asymptotic distribution theory of the proposed tests is given under the null hypothesis as well as under local alternatives. A Bahadur efficiency result of the repeated significance test relative to the terminal test based solely on the target sample size is presented. In the adaptation of the proposed tests to multiple comparisons, an asymptotically equivalent test statistic in terms of the rank estimators of the location parameters is derived from which the Scheffé method of multiple comparisons can be obtained in a convinient way.