The adaptive calibration of testing p-values

Abstract

In statistical hypothesis testing, a p-value is expected to be distributed as the uniform distribution on the interval (0, 1) under the null hypothesis. However, some p-values, such as the generalized p-value and the posterior predictive p-value, cannot be assured of this property. In this paper, we propose an adaptive p-value calibration approach, and show that the calibrated p-value is asymptotically distributed as the uniform distribution. For Behrens–Fisher problem and goodness-of-fit test under a normal model, the calibrated p-values are constructed and their behavior is evaluated numerically. Simulations show that the calibrated p-values are superior than original ones.     

Comparison of confidence intervals for the Behrens-Fisher problem

The Behrens–Fisher problem concerns the inferences for the difference between means of two independent normal populations without the assumption of equality of variances. In this article, we compare three approximate confidence intervals and a generalized confidence interval for the Behrens–Fisher problem. We also show how to obtain simultaneous confidence intervals for the three population case (analysis of variance, ANOVA) by the Bonferroni correction factor. We conduct an extensive simulation study to evaluate these methods in respect to their type I error rate, power, expected confidence interval width, and coverage probability. Finally, the considered methods are applied to two real dataset.     

A parametric bootstrap solution to the MANOVA under heteroscedasticity

In this article, we consider the problem of comparing several multivariate normal mean vectors when the covariance matrices are unknown and arbitrary positive definite matrices. We propose a parametric bootstrap (PB) approach and develop an approximation to the distribution of the PB pivotal quantity for comparing two mean vectors. This approximate test is shown to be the same as the invariant test given in [Krishnamoorthy and Yu, Modified Nel and Van der Merwe test for the multivariate Behrens–Fisher problem, Stat. Probab. Lett. 66 (2004), pp. 161–169] for the multivariate Behrens–Fisher problem. Furthermore, we compare the PB test with two existing invariant tests via Monte Carlo simulation. Our simulation studies show that the PB test controls Type I error rates very satisfactorily, whereas other tests are liberal especially when the number of means to be compared is moderate and/or sample sizes are small. The tests are illustrated using an example.     

Inferences on the Coefficients of Variation in a Multivariate Normal Population

In this article, the problem of testing the equality of coefficients of variation in a multivariate normal population is considered, and an asymptotic approach and a generalized p-value approach based on the concepts of generalized test variable are proposed. Monte Carlo simulation studies show that the proposed generalized p-value test has good empirical sizes, and it is better than the asymptotic approach. In addition, the problem of hypothesis testing and confidence interval for the common coefficient variation of a multivariate normal population are considered, and a generalized p-value and a generalized confidence interval are proposed. Using Monte Carlo simulation, we find that the coverage probabilities and expected lengths of this generalized confidence interval are satisfactory, and the empirical sizes of the generalized p-value are close to nominal level. We illustrate our approaches using a real data.     

Generalized inferences on the common mean vector of several multivariate normal populations

The hypothesis testing and confidence region are considered for the common mean vector of several multivariate normal populations when the covariance matrices are unknown and possibly unequal. A generalized confidence region is derived using the concepts of generalized method based on the generalized p$p$-value. The generalized confidence region is illustrated with two numerical examples. The merits of the proposed method are numerically compared with those of existing methods with respect to their expected area or expected d-dimensional volumes and coverage probabilities under different scenarios.     

Inferences on correlation coefficients: One-sample,independent and correlated cases

This article concerns inference on the correlation coefficients of a multivariate normal distribution. Inferential procedures based on the concepts of generalized variables (GVs) and generalized p$p$-values are proposed for elements of a correlation matrix. For the simple correlation coefficient, the merits of the generalized confidence limits and other approximate methods are evaluated using a numerical study. The study indicates that the proposed generalized confidence limits are uniformly most accurate even for samples as small as three. The results are extended for comparing two independent correlations, overlapping and non-overlapping dependent correlations. For each problem, the properties of the GV approach and other asymptotic methods are evaluated using Monte Carlo simulation. The GV approach produces satisfactory results for all the problems considered. The methods are illustrated using a few practical examples.     

Three Approximate Solutions to the Multivariate Behrens–Fisher Problem

ABSTRACT

This article provides three approximate solutions to the multivariate Behrens–Fisher problem: the F statistic, the Bartlett, as well as the modified Bartlett corrected statistics. Empirical results indicate that the F statistic outperforms the other two and five existing procedures. The modified Bartlett corrected statistic is also very competitive.     

Heteroscedastic ANOVA: old p values, new views

The generalization of the Behrens–Fisher problem to comparing more than two means from nonhomogeneous populations has attracted the attention of statisticians for many decades. Several approaches offer different approximations to the distribution of the test statistic. The question of statistical properties of these approximations is still alive. Here, we present a brief overview of several approaches suggested in the literature and implemented in software with a focus on investigating the accuracy of p values as well as their dependence on nuisance parameters and on the underlying assumption of normality. We illustrate by simulation the behavior of p values. In addition to the Satterthwaite–Fai–Cornelius test, the Kenward–Roger test, the simple ANOVA F test, the parametric bootstrap test, and the generalized F test will be briefly discussed.     

Confidence regions for the common mean vector of several multivariate normal populations

Suppose that there are independent samples available from several multivariate normal populations with the same mean vector m? but possibly different covariance matrices. The problem of developing a confidence region for the common mean vector based on all the samples is considered. An exact confidence region centered at a generalized version of the well-known Graybill-Deal estimator of m? is developed, and a multiple comparison procedure based on this confidence region is outlined. Necessary percentile points for constructing the confidence region are given for the two-sample case. For more than two samples, a convenient method of approximating the percentile points is suggested. Also, a numerical example is presented to illustrate the methods. Further, for the bivariate case, the proposed confidence region and the ones based on individual samples are compared numerically with respect to their expected areas. The numerical results indicate that the new confidence region is preferable to the single-sample versions for practical use.     

Inferences on the reliability in balanced and unbalanced one-way random models

In this article, the hypothesis testing and interval estimation for the reliability parameter are considered in balanced and unbalanced one-way random models. The tests and confidence intervals for the reliability parameter are developed using the concepts of generalized p-value and generalized confidence interval. Furthermore, some simulation results are presented to compare the performances between the proposed approach and the existing approach. For balanced models, the simulation results indicate that the proposed approach can provide satisfactory coverage probabilities and performs better than the existing approaches across the wide array of scenarios, especially for small sample sizes. For unbalanced models, the simulation results show that the two proposed approaches perform more satisfactorily than the existing approach in most cases. Finally, the proposed approaches are illustrated using two real examples.     

Default Bayesian analysis of the Behrens–Fisher problem

In the Bayesian approach, the Behrens–Fisher problem has been posed as one of estimation for the difference of two means. No Bayesian solution to the Behrens–Fisher testing problem has yet been given due, perhaps, to the fact that the conventional priors used are improper. While default Bayesian analysis can be carried out for estimation purposes, it poses difficulties for testing problems. This paper generates sensible intrinsic and fractional prior distributions for the Behrens–Fisher testing problem from the improper priors commonly used for estimation. It allows us to compute the Bayes factor to compare the null and the alternative hypotheses. This default procedure of model selection is compared with a frequentist test and the Bayesian information criterion. We find discrepancy in the sense that frequentist and Bayesian information criterion reject the null hypothesis for data, that the Bayes factor for intrinsic or fractional priors do not.     

A Revisit to the Behrens–Fisher Problem: Comparison of Five Test Methods

We revisit the well-known Behrens–Fisher problem and apply a newly developed ‘Computational Approach Test’ (CAT) to test the equality of two population means where the populations are assumed to be normal with unknown and possibly unequal variances. An advantage of the CAT is that it does not require the explicit knowledge of the sampling distribution of the test statistic. The CAT is then compared with three widely accepted tests—Welch–Satterthwaite test (WST), Cochran–Cox test (CCT), ‘Generalized p-value’ test (GPT)—and a recently suggested test based on the jackknife procedure, called Singh–Saxena–Srivastava test (SSST). Further, model robustness of these five tests are studied when the data actually came from t-distributions, but wrongly perceived as normal ones. Our detailed study based on a comprehensive simulation indicate some interesting results including the facts that the GPT is quite conservative, and the SSST is not as good as it has been claimed in the literature. To the best of our knowledge, the trends observed in our study have not been reported earlier in the existing literature.     

A non-parametric maximum test for the Behrens–Fisher problem

Non-normality and heteroscedasticity are common in applications. For the comparison of two samples in the non-parametric Behrens–Fisher problem, different tests have been proposed, but no single test can be recommended for all situations. Here, we propose combining two tests, the Welch t test based on ranks and the Brunner–Munzel test, within a maximum test. Simulation studies indicate that this maximum test, performed as a permutation test, controls the type I error rate and stabilizes the power. That is, it has good power characteristics for a variety of distributions, and also for unbalanced sample sizes. Compared to the single tests, the maximum test shows acceptable type I error control.     

Empirical approximations for Hoeffding’s test of bivariate independence using two Weibull extensions

The sampling distributions are generally unavailable in exact form and are approximated either in terms of the asymptotic distributions, or their correction using expansions such as Edgeworth, Laguerre or Cornish–Fisher; or by using transformations analogous to that of Wilson and Hilferty. However, when theoretical routes are intractable, in this electronic age, the sampling distributions can be reasonably approximated using empirical methods. The point is illustrated using the null distribution of Hoeffding’s test of bivariate independence which is important because of its consistency against all dependence alternatives. For constructing the approximations we employ two Weibull extensions, the generalized Weibull and the exponentiated Weibull families, which contain a rich variety of density shapes and tail lengths, and have their distribution functions and quantile functions available in closed form, making them convenient for obtaining the necessary percentiles and p-values. Both approximations are seen to be excellent in terms of accuracy, but that based on the generalized Weibull is more portable.     

Estimation of P(Y < X) for progressively first-failure-censored generalized inverted exponential distribution

In this article, we consider the problem of estimation of the stress–strength parameter δ?=?P(Y?<?X) based on progressively first-failure-censored samples, when X and Y both follow two-parameter generalized inverted exponential distribution with different and unknown shape and scale parameters. The maximum likelihood estimator of δ and its asymptotic confidence interval based on observed Fisher information are constructed. Two parametric bootstrap boot-p and boot-t confidence intervals are proposed. We also apply Markov Chain Monte Carlo techniques to carry out Bayes estimation procedures. Bayes estimate under squared error loss function and the HPD credible interval of δ are obtained using informative and non-informative priors. A Monte Carlo simulation study is carried out for comparing the proposed methods of estimation. Finally, the methods developed are illustrated with a couple of real data examples.     

Comparing the mean vectors of two independent multivariate log-normal distributions

The multivariate log-normal distribution is a good candidate to describe data that are not only positive and skewed, but also contain many characteristic values. In this study, we apply the generalized variable method to compare the mean vectors of two independent multivariate log-normal populations that display heteroscedasticity. Two generalized pivotal quantities are derived for constructing the generalized confidence region and for testing the difference between two mean vectors. Simulation results indicate that the proposed procedures exhibit satisfactory performance regardless of the sample sizes and heteroscedasticity. The type I error rates obtained are consistent with expectations and the coverage probabilities are close to the nominal level when compared with the other method which is currently available. These features make the proposed method a worthy alternative for inferential analysis of problems involving multivariate log-normal means. The results are illustrated using three examples.     

An improved two-stage procedure to compare hazard curves

Qiu and Sheng has proposed a powerful and robust two-stage procedure to compare two hazard rate functions. In this paper we improve their method by using the Fisher test to combine the asymptotically independent p-values obtained from the two stages of their procedure. In addition, we extend the procedure to situations with multiple hazard rate functions. Our comprehensive simulation study shows that the proposed method has a good performance in terms of controlling the type I error rate and of detecting power. Three real data applications are considered for illustrating the use of the new method.     

A generalized Q–Q plot for longitudinal data

Most biomedical research is carried out using longitudinal studies. The method of generalized estimating equations (GEEs) introduced by Liang and Zeger [Longitudinal data analysis using generalized linear models, Biometrika 73 (1986), pp. 13–22] and Zeger and Liang [Longitudinal data analysis for discrete and continuous outcomes, Biometrics 42 (1986), pp. 121–130] has become a standard method for analyzing non-normal longitudinal data. Since then, a large variety of GEEs have been proposed. However, the model diagnostic problem has not been explored intensively. Oh et al. [Modeldiagnostic plots for repeated measures data using the generalized estimating equations approach, Comput. Statist. Data Anal. 53 (2008), pp. 222–232] proposed residual plots based on the quantile–quantile (Q–Q) plots of the χ²-distribution for repeated-measures data using the GEE methodology. They considered the Pearson, Anscombe and deviance residuals. In this work, we propose to extend this graphical diagnostic using a generalized residual. A simulation study is presented as well as two examples illustrating the proposed generalized Q–Q plots.     

Confidence intervals for product of powers of the generalized variances of k multivariate normal populations

In this paper, some confidence intervals (CIs) for the product of powers of the generalized variances of k multivariate normal populations with possibly different dimensions are proposed. The performance of these CIs in terms of the coverage probabilities and average lengths were evaluated via a Monte Carlo simulation study. The results were found to be satisfactory. To demonstrate utility of the proposed CIs, applications on three real data sets were provided.     

The behrens‐fisher problem revisited: A bayes‐frequentist synthesis

The Behrens‐Fisher problem concerns the inference for the difference between the means of two normal populations whose ratio of variances is unknown. In this situation, Fisher's fiducial interval differs markedly from the Neyman‐Pearson confidence interval. A prior proposed by Jeffreys leads to a credible interval that is equivalent to Fisher's solution but it carries a different interpretation. The authors propose an alternative prior leading to a credible interval whose asymptotic coverage probability matches the frequentist coverage probability more accurately than the interval of Jeffreys. Their simulation results indicate excellent matching even in small samples.