Adaptive procedure for generalized familywise error rate control

This article considers multiple hypotheses testing with the generalized familywise error rate k-FWER control, which is the probability of at least k false rejections. We first assume the p-values corresponding to the true null hypotheses are independent, and propose adaptive generalized Bonferroni procedure with k-FWER control based on the estimation of the number of true null hypotheses. Then, we assume the p-values are dependent, satisfying block dependence, and propose adaptive procedure with k-FWER control. Extensive simulations compare the performance of the adaptive procedures with different estimators.     

An improved two-stage procedure to compare hazard curves

Qiu and Sheng has proposed a powerful and robust two-stage procedure to compare two hazard rate functions. In this paper we improve their method by using the Fisher test to combine the asymptotically independent p-values obtained from the two stages of their procedure. In addition, we extend the procedure to situations with multiple hazard rate functions. Our comprehensive simulation study shows that the proposed method has a good performance in terms of controlling the type I error rate and of detecting power. Three real data applications are considered for illustrating the use of the new method.     

Bonferroni-type Plug-in Procedure Controlling Generalized Familywise Error Rate

Consider the multiple hypotheses testing problem controlling the generalized familywise error rate k-FWER, the probability of at least k false rejections. We propose a plug-in procedure based on the estimation of the number of true null hypotheses. Under the independence assumption of the p-values corresponding to the true null hypotheses, we first introduce the least favorable configuration (LFC) of k-FWER for Bonferroni-type plug-in procedure, then we construct a plug-in k-FWER-controlled procedure based on LFC. For dependent p-values, we establish the asymptotic k-FWER control under some mild conditions. Simulation studies suggest great improvement over generalized Bonferroni test and generalized Holm test.     

Comparison of Nonparametric Regression Curves by Spline Smoothing

In this article, procedures are proposed to test the hypothesis of equality of two or more regression functions. Tests are proposed by p-values, first under homoscedastic regression model, which are derived using fiducial method based on cubic spline interpolation. Then, we construct a test in the heteroscedastic case based on Fisher's method of combining independent tests. We study the behaviors of the tests by simulation experiments, in which comparisons with other tests are also given. The proposed tests have good performances. Finally, an application to a data set are given to illustrate the usefulness of the proposed test in practice.     

The adaptive calibration of testing p-values

Abstract

In statistical hypothesis testing, a p-value is expected to be distributed as the uniform distribution on the interval (0, 1) under the null hypothesis. However, some p-values, such as the generalized p-value and the posterior predictive p-value, cannot be assured of this property. In this paper, we propose an adaptive p-value calibration approach, and show that the calibrated p-value is asymptotically distributed as the uniform distribution. For Behrens–Fisher problem and goodness-of-fit test under a normal model, the calibrated p-values are constructed and their behavior is evaluated numerically. Simulations show that the calibrated p-values are superior than original ones.     

Three Recommendations for Improving the Use of p-Values

ABSTRACT

Researchers commonly use p-values to answer the question: How strongly does the evidence favor the alternative hypothesis relative to the null hypothesis? p-Values themselves do not directly answer this question and are often misinterpreted in ways that lead to overstating the evidence against the null hypothesis. Even in the “post p?<?0.05 era,” however, it is quite possible that p-values will continue to be widely reported and used to assess the strength of evidence (if for no other reason than the widespread availability and use of statistical software that routinely produces p-values and thereby implicitly advocates for their use). If so, the potential for misinterpretation will persist. In this article, we recommend three practices that would help researchers more accurately interpret p-values. Each of the three recommended practices involves interpreting p-values in light of their corresponding “Bayes factor bound,” which is the largest odds in favor of the alternative hypothesis relative to the null hypothesis that is consistent with the observed data. The Bayes factor bound generally indicates that a given p-value provides weaker evidence against the null hypothesis than typically assumed. We therefore believe that our recommendations can guard against some of the most harmful p-value misinterpretations. In research communities that are deeply attached to reliance on “p?<?0.05,” our recommendations will serve as initial steps away from this attachment. We emphasize that our recommendations are intended merely as initial, temporary steps and that many further steps will need to be taken to reach the ultimate destination: a holistic interpretation of statistical evidence that fully conforms to the principles laid out in the ASA statement on statistical significance and p-values.     

A note on fiducial model averaging as an alternative to checking Bayesian and frequentist models

Just as frequentist hypothesis tests have been developed to check model assumptions, prior predictive p-values and other Bayesian p-values check prior distributions as well as other model assumptions. These model checks not only suffer from the usual threshold dependence of p-values, but also from the suppression of model uncertainty in subsequent inference. One solution is to transform Bayesian and frequentist p-values for model assessment into a fiducial distribution across the models. Averaging the Bayesian or frequentist posterior distributions with respect to the fiducial distribution can reproduce results from Bayesian model averaging or classical fiducial inference.     

Inferences on a linear combination of K multivariate normal mean vectors

In this paper, the hypothesis testing and confidence region construction for a linear combination of mean vectors for K independent multivariate normal populations are considered. A new generalized pivotal quantity and a new generalized test variable are derived based on the concepts of generalized p-values and generalized confidence regions. When only two populations are considered, our results are equivalent to those proposed by Gamage et al. [Generalized p-values and confidence regions for the multivariate Behrens–Fisher problem and MANOVA, J. Multivariate Aanal. 88 (2004), pp. 117–189] in the bivariate case, which is also known as the bivariate Behrens–Fisher problem. However, in some higher dimension cases, these two results are quite different. The generalized confidence region is illustrated with two numerical examples and the merits of the proposed method are numerically compared with those of the existing methods with respect to their expected areas, coverage probabilities under different scenarios.     

Distribution of the Λ-Criterion for Sphericity Test and Its Connection to a Generalized Dirichlet Model

It is shown that the exact null distribution of the likelihood ratio criterion for sphericity test in the p-variate normal case and the marginal distribution of the first component of a (p ? 1)-variate generalized Dirichlet model with a given set of parameters are identical. The exact distribution of the likelihood ratio criterion so obtained has a general format for every p. A novel idea is introduced here through which the complicated exact null distribution of the sphericity test criterion in multivariate statistical analysis is converted into an easily tractable marginal density in a generalized Dirichlet model. It provides a direct and easiest method of computation of p-values. The computation of p-values and a table of critical points corresponding to p = 3 and 4 are also presented.     

A two-stage procedure with accelerated test for selecting the best exponential population

In this article, we discuss a two-stage procedure for selecting the largest location parameter among k(k≥2) two-parameter exponential populations(or products) from an accelerated test. The accelerated test will be conducted at a higher stress level than that of normal in the second stage. under certain assumptions between parameter and stress leveL, the two-stage selection procedure, which guarantees that the probability of correct selection is at least p^*, is proposed. At the end of the paper , we present some useful tables that serve as a guide for the needed sample size in the second stage.     

A method for combining p-values in meta-analysis by gamma distributions

Combining p-values from statistical tests across different studies is the most commonly used approach in meta-analysis for evolutionary biology. The most commonly used p-value combination methods mainly incorporate the z-transform tests (e.g., the un-weighted z-test and the weighted z-test) and the gamma-transform tests (e.g., the CZ method [Z. Chen, W. Yang, Q. Liu, J.Y. Yang, J. Li, and M.Q. Yang, A new statistical approach to combining p-values using gamma distribution and its application to genomewide association study, Bioinformatics 15 (2014), p. S3]). However, among these existing p-value combination methods, no method is uniformly most powerful in all situations [Chen et al. 2014]. In this paper, we propose a meta-analysis method based on the gamma distribution, MAGD, by pooling the p-values from independent studies. The newly proposed test, MAGD, allows for flexible accommodating of the different levels of heterogeneity of effect sizes across individual studies. The MAGD simultaneously retains all the characters of the z-transform tests and the gamma-transform tests. We also propose an easy-to-implement resampling approach for estimating the empirical p-values of MAGD for the finite sample size. Simulation studies and two data applications show that the proposed method MAGD is essentially as powerful as the z-transform tests (the gamma-transform tests) under the circumstance with the homogeneous (heterogeneous) effect sizes across studies.     

A generalized test variable approach for grain yield comparisons of rice

Traditionally, an assessment for grain yield of rice is to split it into the yield components, including the number of panicles per plant, the number of spikelets per panicle, the 1000-grain weight and the filled-spikelet percentage, such that the yield performance can be individually evaluated through each component, and the products of yield components are employed for grain yield comparisons. However, when using the standard statistical methods, such as the two-sample t-test and analysis of variance, the assumptions of normality and variance homogeneity cannot be fully justified for comparing the grain yields, leading to that the empirical sizes cannot be adequately controlled. In this study, based on the concepts of generalized test variables and generalized p-values, a novel statistical testing procedure is developed for grain yield comparisons of rice. The proposed method is assessed by a series of numerical simulations. According to the simulation results, the proposed method performs reasonably well in Type I error control and empirical power. In addition, a real-life field experiment is analyzed by the proposed method, some productive rice varieties are screened out and suggested for a follow-up investigation.     

P-Value Precision and Reproducibility

P-values are useful statistical measures of evidence against a null hypothesis. In contrast to other statistical estimates, however, their sample-to-sample variability is usually not considered or estimated, and therefore not fully appreciated. Via a systematic study of log-scale p-value standard errors, bootstrap prediction bounds, and reproducibility probabilities for future replicate p-values, we show that p-values exhibit surprisingly large variability in typical data situations. In addition to providing context to discussions about the failure of statistical results to replicate, our findings shed light on the relative value of exact p-values vis-a-vis approximate p-values, and indicate that the use of *, **, and *** to denote levels 0.05, 0.01, and 0.001 of statistical significance in subject-matter journals is about the right level of precision for reporting p-values when judged by widely accepted rules for rounding statistical estimates.     

Two new estimators for the proportion of true null hypotheses in multiple test

In multiple hypothesis test, an important problem is estimating the proportion of true null hypotheses. Existing methods are mainly based on the p-values of the single tests. In this paper, we propose two new estimations for this proportion. One is a natural extension of the commonly used methods based on p-values and the other is based on a mixed distribution. Simulations show that the first method is comparable with existing methods and performs better under some cases. And the method based on a mixed distribution can get accurate estimators even if the variance of data is large or the difference between the null hypothesis and alternative hypothesis is very small.     

A two-step rejection procedure for testing multiple hypotheses

This paper considers p-value based step-wise rejection procedures for testing multiple hypotheses. The existing procedures have used constants as critical values at all steps. With the intention of incorporating the exact magnitude of the p-values at the earlier steps into the decisions at the later steps, this paper applies a different strategy that the critical values at the later steps are determined as functions of the p-values from the earlier steps. As a result, we have derived a new equality and developed a two-step rejection procedure following that. The new procedure is a short-cut of a step-up procedure, and it possesses great simplicity. In terms of power, the proposed procedure is generally comparable to the existing ones and exceptionally superior when the largest p-value is anticipated to be less than 0.5.     

Efficient Stratified Testing Procedure for a False Discovery Rate

The false discovery rate (FDR) has become a popular error measure in the large-scale simultaneous testing. When data are collected from heterogenous sources and form grouped hypotheses testing, it may be beneficial to use the distinct feature of groups to conduct the multiple hypotheses testing. We propose a stratified testing procedure that uses different FDR levels according to the stratification features based on p-values. Our proposed method is easy to implement in practice. Simulations studies show that the proposed method produces more efficient testing results. The stratified testing procedure minimizes the overall false negative rate (FNR) level, while controlling the overall FDR. An example from a type II diabetes mice study further illustrates the practical advantages of this new approach.     

On a Multiple Three-Decision Problem for Comparing Several Treatments with the Best Control

In this article, a multiple three-decision procedure is proposed to classify p (≥2) treatments as better or worse than the best of q (≥2) control treatments in one way layout. Critical constants required for the implementation of the proposed procedure are tabulated for some pre-specified values of probability of no misclassification. Power function of the proposed procedure is defined and a common sample size necessary to guarantee various pre-specified power levels are tabulated under two optimal allocation schemes. Finally the implementation of the proposed methodology is demonstrated through numerical examples based on real life data.     

Exact Tests in Panel Data Using Generalized p-Values

In this article, we consider exact tests in panel data regression model with one-way and two-way error component for which no exact tests are available. Exact inferences using generalized p-values are obtained. When there are several groups of panel data, test for equal coefficients under one-way and two-way error component are derived.     

Modified Stationarity Tests With Data-Dependent Model-Selection Rules

We describe some simple methods for improving the performance of stationarity tests (i.e., tests that have a stationary null and a unit-root alternative). Specifically, we increase the rate of convergence of the test under the unit-root alternative from O _p(T) to O _p (T ²), then suggest an optimal method of selecting the order of the autoregressive component in the fitted autoregressive integrated moving average model on which the test is based. Simulation evidence suggests that these modifications work well. We apply the modified procedure to U.S. monthly macroeconomic data and uncover new evidence of a unit root in unemployment.     

Algorithms for approximate conditional inference

This paper presents a method for listing the sample space for a conditional distribution in a discrete generalized linear model. This tabulation is used in conjunction with saddlepoint methods to approximate the associated conditional probabilities. These probabilities are used to calculate conditional p-values.