EXACT P‐VALUES FOR DISCRETE MODELS OBTAINED BY ESTIMATION AND MAXIMIZATION

In constructing exact tests from discrete data, one must deal with the possible dependence of the P‐value on nuisance parameter(s) ψ as well as the discreteness of the sample space. A classical but heavy‐handed approach is to maximize over ψ. We prove what has previously been understood informally, namely that maximization produces the unique and smallest possible P‐value subject to the ordering induced by the underlying test statistic and test validity. On the other hand, allowing for the worst case will be more attractive when the P‐value is less dependent on ψ. We investigate the extent to which estimating ψ under the null reduces this dependence. An approach somewhere between full maximization and estimation is partial maximization, with appropriate penalty, as introduced by Berger & Boos (1994, P values maximized over a confidence set for the nuisance parameter. J. Amer. Statist. Assoc. 89 , 1012–1016). It is argued that estimation followed by maximization is an attractive, but computationally more demanding, alternative to partial maximization. We illustrate the ideas on a range of low‐dimensional but important examples for which the alternative methods can be investigated completely numerically.     

Exact tests based on pre-estimation and second order pivotals: non-inferiority trials

Pre-estimation is a technique for adjusting a standard approximate P-value to be close to exact. While conceptually simple, it can become computationally intensive. Second order pivotals [N. Reid, Asymptotics and the theory of inference, Ann. Statist. 31 (2003), pp. 1695–1731] are constructed to be closer to exact than standard approximate pivotals. The theory behind these pivotals is complex, and their properties are unclear for discrete models. However, since they are typically given in closed form they are easy to compute. For the special case of non-inferiority trials, we investigate Wald, Score, likelihood ratio and second order pivotals. Each of the basic pivotals are used to generate an exact test by maximising with respect to the nuisance parameter. We also study the effect of pre-estimating the nuisance parameter, as described in Lloyd [C.J. Lloyd, Exact P-values for discrete models obtained by estimation and maximisation, Aust. N. Z. J. Statist. 50 (2008), pp. 329–346]. It appears that second order methods are not as close to exact as might have been hoped. On the other hand, P-values, based on pre-estimation are very close to exact, are more powerful than competitors and are hardly affected by the basic generating statistic chosen.     

Effective directed tests for models with ordered categorical data

This paper offers a new method for testing one‐sided hypotheses in discrete multivariate data models. One‐sided alternatives mean that there are restrictions on the multidimensional parameter space. The focus is on models dealing with ordered categorical data. In particular, applications are concerned with R×C contingency tables. The method has advantages over other general approaches. All tests are exact in the sense that no large sample theory or large sample distribution theory is required. Testing is unconditional although its execution is done conditionally, section by section, where a section is determined by marginal totals. This eliminates any potential nuisance parameter issues. The power of the tests is more robust than the power of the typical linear tests often recommended. Furthermore, computer programs are available to carry out the tests efficiently regardless of the sample sizes or the order of the contingency tables. Both censored data and uncensored data models are discussed.     

Exact calculation of power and sample size in bioequivalence studies using two one‐sided tests

The number of subjects in a pharmacokinetic two‐period two‐treatment crossover bioequivalence study is typically small, most often less than 60. The most common approach to testing for bioequivalence is the two one‐sided tests procedure. No explicit mathematical formula for the power function in the context of the two one‐sided tests procedure exists in the statistical literature, although the exact power based on Owen's special case of bivariate noncentral t‐distribution has been tabulated and graphed. Several approximations have previously been published for the probability of rejection in the two one‐sided tests procedure for crossover bioequivalence studies. These approximations and associated sample size formulas are reviewed in this article and compared for various parameter combinations with exact power formulas derived here, which are computed analytically as univariate integrals and which have been validated by Monte Carlo simulations. The exact formulas for power and sample size are shown to improve markedly in realistic parameter settings over the previous approximations. Copyright © 2014 John Wiley & Sons, Ltd.     

Exact unconditional testing procedures for comparing two independent Poisson rates

We consider seven exact unconditional testing procedures for comparing adjusted incidence rates between two groups from a Poisson process. Exact tests are always preferable due to the guarantee of test size in small to medium sample settings. Han [Comparing two independent incidence rates using conditional and unconditional exact tests. Pharm Stat. 2008;7(3):195–201] compared the performance of partial maximization p-values based on the Wald test statistic, the likelihood ratio test statistic, the score test statistic, and the conditional p-value. These four testing procedures do not perform consistently, as the results depend on the choice of test statistics for general alternatives. We consider the approach based on estimation and partial maximization, and compare these to the ones studied by Han (2008) for testing superiority. The procedures are compared with regard to the actual type I error rate and power under various conditions. An example from a biomedical research study is provided to illustrate the testing procedures. The approach based on partial maximization using the score test is recommended due to the comparable performance and computational advantage in large sample settings. Additionally, the approach based on estimation and partial maximization performs consistently for all the three test statistics, and is also recommended for use in practice.     

Confidence and Likelihood*

Confidence intervals for a single parameter are spanned by quantiles of a confidence distribution, and one‐sided p‐values are cumulative confidences. Confidence distributions are thus a unifying format for representing frequentist inference for a single parameter. The confidence distribution, which depends on data, is exact (unbiased) when its cumulative distribution function evaluated at the true parameter is uniformly distributed over the unit interval. A new version of the Neyman–Pearson lemma is given, showing that the confidence distribution based on the natural statistic in exponential models with continuous data is less dispersed than all other confidence distributions, regardless of how dispersion is measured. Approximations are necessary for discrete data, and also in many models with nuisance parameters. Approximate pivots might then be useful. A pivot based on a scalar statistic determines a likelihood in the parameter of interest along with a confidence distribution. This proper likelihood is reduced of all nuisance parameters, and is appropriate for meta‐analysis and updating of information. The reduced likelihood is generally different from the confidence density. Confidence distributions and reduced likelihoods are rooted in Fisher–Neyman statistics. This frequentist methodology has many of the Bayesian attractions, and the two approaches are briefly compared. Concepts, methods and techniques of this brand of Fisher–Neyman statistics are presented. Asymptotics and bootstrapping are used to find pivots and their distributions, and hence reduced likelihoods and confidence distributions. A simple form of inverting bootstrap distributions to approximate pivots of the abc type is proposed. Our material is illustrated in a number of examples and in an application to multiple capture data for bowhead whales.     

A Nonrandomized,Nonconservative Version of the Fisher Exact Test

The Fisher exact test has been unjustly dismissed by some as ‘only conditional,’ whereas it is unconditionally the uniform most powerful test among all unbiased tests, tests of size α and with power greater than its nominal level of significance α. The problem with this truly optimal test is that it requires randomization at the critical value(s) to be of size α. Obviously, in practice, one does not want to conclude that ‘with probability x the we have a statistical significant result.’ Usually, the hypothesis is rejected only if the test statistic's outcome is more extreme than the critical value, reducing the actual size considerably.

The randomized unconditional Fisher exact is constructed (using Neyman–structure arguments) by deriving a conditional randomized test randomizing at critical values c(t) by probabilities γ(t), that both depend on the total number of successes T (the complete-sufficient statistic for the nuisance parameter—the common success probability) conditioned upon.

In this paper, the Fisher exact is approximated by deriving nonrandomized conditional tests with critical region including the critical value only if γ (t) > γ₀, for a fixed threshold value γ₀, such that the size of the unconditional modified test is for all value of the nuisance parameter—the common success probability—smaller, but as close as possible to α. It will be seen that this greatly improves the size of the test as compared with the conservative nonrandomized Fisher exact test.

Size, power, and p value comparison with the (virtual) randomized Fisher exact test, and the conservative nonrandomized Fisher exact, Pearson's chi-square test, with the more competitive mid-p value, the McDonald's modification, and Boschloo's modifications are performed under the assumption of two binomial samples.     

Log‐rank permutation tests for trend: saddlepoint p‐values and survival rate confidence intervals

Suppose p + 1 experimental groups correspond to increasing dose levels of a treatment and all groups are subject to right censoring. In such instances, permutation tests for trend can be performed based on statistics derived from the weighted log‐rank class. This article uses saddlepoint methods to determine the mid‐P‐values for such permutation tests for any test statistic in the weighted log‐rank class. Permutation simulations are replaced by analytical saddlepoint computations which provide extremely accurate mid‐P‐values that are exact for most practical purposes and almost always more accurate than normal approximations. The speed of mid‐P‐value computation allows for the inversion of such tests to determine confidence intervals for the percentage increase in mean (or median) survival time per unit increase in dosage. The Canadian Journal of Statistics 37: 5‐16; 2009 © 2009 Statistical Society of Canada     

A more powerful unconditional exact test of homogeneity for 2 × c contingency table analysis

The classical unconditional exact p-value test can be used to compare two multinomial distributions with small samples. This general hypothesis requires parameter estimation under the null which makes the test severely conservative. Similar property has been observed for Fisher's exact test with Barnard and Boschloo providing distinct adjustments that produce more powerful testing approaches. In this study, we develop a novel adjustment for the conservativeness of the unconditional multinomial exact p-value test that produces nominal type I error rate and increased power in comparison to all alternative approaches. We used a large simulation study to empirically estimate the 5th percentiles of the distributions of the p-values of the exact test over a range of scenarios and implemented a regression model to predict the values for two-sample multinomial settings. Our results show that the new test is uniformly more powerful than Fisher's, Barnard's, and Boschloo's tests with gains in power as large as several hundred percent in certain scenarios. Lastly, we provide a real-life data example where the unadjusted unconditional exact test wrongly fails to reject the null hypothesis and the corrected unconditional exact test rejects the null appropriately.     

Two-sided tests for change in level for correlated data

The classical problem of change point is considered when the data are assumed to be correlated. The nuisance parameters in the model are the initial level μ and the common variance σ². The four cases, based on none, one, and both of the parameters are known are considered. Likelihood ratio tests are obtained for testing hypotheses regarding the change in level, δ, in each case. Following Henderson (1986), a Bayesian test is obtained for the two sided alternative. Under the Bayesian set up, a locally most powerful unbiased test is derived for the case μ=0 and σ²=1. The exact null distribution function of the Bayesian test statistic is given an integral representation. Methods to obtain exact and approximate critical values are indicated.     

How Many Iterations are Sufficient for Efficient Semiparametric Estimation?

Abstract. A common practice in obtaining an efficient semiparametric estimate is through iteratively maximizing the (penalized) full log‐likelihood w.r.t. its Euclidean parameter and functional nuisance parameter. A rigorous theoretical study of this semiparametric iterative estimation approach is the main purpose of this study. We first show that the grid search algorithm produces an initial estimate with the proper convergence rate. Our second contribution is to provide a formula in calculating the minimal number of iterations k ^* needed to produce an efficient estimate . We discover that (i) k ^* depends on the convergence rates of the initial estimate and the nuisance functional estimate, and (ii) k ^* iterations are also sufficient for recovering the estimation sparsity in high dimensional data. The last contribution is the novel construction of which does not require knowing the explicit expression of the efficient score function. The above general conclusions apply to semiparametric models estimated under various regularizations, for example, kernel or penalized estimation. As far as we are aware, this study provides a first general theoretical justification for the ‘one‐/two‐step iteration’ phenomena observed in the semiparametric literature.     

Computing Critical Values of Exact Tests by Incorporating Monte Carlo Simulations Combined with Statistical Tables

Various exact tests for statistical inference are available for powerful and accurate decision rules provided that corresponding critical values are tabulated or evaluated via Monte Carlo methods. This article introduces a novel hybrid method for computing p‐values of exact tests by combining Monte Carlo simulations and statistical tables generated a priori. To use the data from Monte Carlo generations and tabulated critical values jointly, we employ kernel density estimation within Bayesian‐type procedures. The p‐values are linked to the posterior means of quantiles. In this framework, we present relevant information from the Monte Carlo experiments via likelihood‐type functions, whereas tabulated critical values are used to reflect prior distributions. The local maximum likelihood technique is employed to compute functional forms of prior distributions from statistical tables. Empirical likelihood functions are proposed to replace parametric likelihood functions within the structure of the posterior mean calculations to provide a Bayesian‐type procedure with a distribution‐free set of assumptions. We derive the asymptotic properties of the proposed nonparametric posterior means of quantiles process. Using the theoretical propositions, we calculate the minimum number of needed Monte Carlo resamples for desired level of accuracy on the basis of distances between actual data characteristics (e.g. sample sizes) and characteristics of data used to present corresponding critical values in a table. The proposed approach makes practical applications of exact tests simple and rapid. Implementations of the proposed technique are easily carried out via the recently developed STATA and R statistical packages.     

Evaluating the Accuracy of Small P‐Values In Genetic Association Studies Using Edgeworth Expansions

The asymptotic distributions of many classical test statistics are normal. The resulting approximations are often accurate for commonly used significance levels, 0.05 or 0.01. In genome‐wide association studies, however, the significance level can be as low as 1×10⁻⁷, and the accuracy of the p‐values can be challenging. We study the accuracies of these small p‐values are using two‐term Edgeworth expansions for three commonly used test statistics in GWAS. These tests have nuisance parameters not defined under the null hypothesis but estimable. We derive results for this general form of testing statistics using Edgeworth expansions, and find that the commonly used score test, maximin efficiency robust test and the chi‐squared test are second order accurate in the presence of the nuisance parameter, justifying the use of the p‐values obtained from these tests in the genome‐wide association studies.     

Implications of random cut‐points theory for the Mann‐Whitney and binomial tests

Through random cut‐points theory, the author extends inference for ordered categorical data to the unspecified continuum underlying the ordered categories. He shows that a random cut‐point Mann‐Whitney test yields slightly smaller p‐values than the conventional test for most data. However, when at least P% of the data lie in one of the k categories (with P = 80 for k = 2, P = 67 for k = 3,…, P = 18 for k = 30), he also shows that the conventional test can yield much smaller p‐values, and hence misleadingly liberal inference for the underlying continuum. The author derives formulas for exact tests; for k = 2, the Mann‐Whitney test is but a binomial test.     

Nonparametric equivalence testing with respect to the median difference

The problem of comparing two independent groups of univariate data in the sense of testing for equivalence is considered for a fully nonparametric setting. The distribution of the data within each group may be a mixture of both a continuous and a discrete component, and no assumptions are made regarding the way in which the distributions of the two groups of data may differ from each other – in particular, the assumption of a shift model is avoided. The proposed equivalence testing procedure for this scenario refers to the median of the independent difference distribution, i.e. to the median of the differences between independent observations from the test group and the reference group, respectively. The procedure provides an asymptotic equivalence test, which is symmetric with respect to the roles of ‘test’ and ‘reference’. It can be described either as a two‐one‐sided‐tests (TOST) approach, or equivalently as a confidence interval inclusion rule. A one‐sided variant of the approach can be applied analogously to non‐inferiority testing problems. The procedure may be generalised to equivalence testing with respect to quantiles other than the median, and is closely related to tolerance interval type inference. Copyright © 2009 John Wiley & Sons, Ltd.     

Accurate unconditional p-values for a two-arm study with binary endpoints

Unconditional exact tests are increasingly used in practice for categorical data to increase the power of a study and to make the data analysis approach being consistent with the study design. In a two-arm study with a binary endpoint, p-value based on the exact unconditional Barnard test is computed by maximizing the tail probability over a nuisance parameter with a range from 0 to 1. The traditional grid search method is able to find an approximate maximum with a partition of the parameter space, but it is not accurate and this approach becomes computationally intensive for a study beyond two groups. We propose using a polynomial method to rewrite the tail probability as a polynomial. The solutions from the derivative of the polynomial contain the solution for the global maximum of the tail probability. We use an example from a double-blind randomized Phase II cancer clinical trial to illustrate the application of the proposed polynomial method to achieve an accurate p-value. We also compare the performance of the proposed method and the traditional grid search method under various conditions. We would recommend using this new polynomial method in computing accurate exact unconditional p-values.     

Simple and accurate one‐sided inference from signed roots of likelihood ratios

The authors propose two methods based on the signed root of the likelihood ratio statistic for one‐sided testing of a simple null hypothesis about a scalar parameter in the présence of nuisance parameters. Both methods are third‐order accurate and utilise simulation to avoid the need for onerous analytical calculations characteristic of competing saddlepoint procedures. Moreover, the new methods do not require specification of ancillary statistics. The methods respect the conditioning associated with similar tests up to an error of third order, and conditioning on ancillary statistics to an error of second order.     

Estimation of the smallest scale parameter of two-parameter exponential distributions

Improved point and interval estimation of the smallest scale parameter of n independent populations following two-parameter exponential distributions are studied. The model is formulated in such a way that allows for treating the estimation of the smallest scale parameter as a problem of estimating an unrestricted scale parameter in the presence of a nuisance parameter. The classes of improved point and interval estimators are enriched with Stein-type, Brewster and Zidek-type, Maruyama-type and Strawderman-type improved estimators under both quadratic and entropy losses, whereas using as a criterion the coverage probability, with Stein-type, Brewster and Zidek-type, and Maruyama-type improved intervals. The sampling framework considered incorporates important life-testing schemes such as i.i.d. sampling, type-II censoring, progressive type-II censoring, adaptive progressive type-II censoring, and record values.     

Distribution-free tests for cluster samples of ordinal responses

A simulation comparison is done of Mann–Whitney U test extensions recently proposed for simple cluster samples or for repeated ordinal responses. These are based on two approaches: the permutation approach of Fay and Gennings (four tests, two exact), and Edwardes’ approach (two asymptotic tests, one new). Edwardes’ approach permits confidence interval estimation, unlike the permutation approach. An appropriate parameter for estimation is P(X<Y)−P(X>Y), where X is the rank of a response from group 1 and Y is from group 2. The permutation tests are shown to be unsuitable for some survey data, since they are sensitive to a difference in cluster intra-correlations when there is no distribution difference between groups at the individual level. The exact permutation tests are of little use for less than seven clusters, precisely where they are most needed. Otherwise, the permutation tests perform well.     

Mid-P confidence intervals based on the likelihood ratio for proportions estimated by group testing

Group testing has been used in many fields of study to estimate proportions. When groups are of different size, the derivation of exact confidence intervals is complicated by the lack of a unique ordering of the event space. An exact interval estimation method is described here, in which outcomes are ordered according to a likelihood ratio statistic. The method is compared with another exact method, in which outcomes are ordered by their associated MLE. Plots of the P‐value against the proportion are useful in examining the properties of the methods. Coverage provided by the intervals is assessed using several realistic grouptesting procedures. The method based on the likelihood ratio, with a mid‐P correction, is shown to give very good coverage in terms of closeness to the nominal level, and is recommended for this type of problem.