Significance test for linear regression: how to test without P-values?

The discussion on the use and misuse of p-values in 2016 by the American Statistician Association was a timely assertion that statistical concept should be properly used in science. Some researchers, especially the economists, who adopt significance testing and p-values to report their results, may felt confused by the statement, leading to misinterpretations of the statement. In this study, we aim to re-examine the accuracy of the p-value and introduce an alternative way for testing the hypothesis. We conduct a simulation study to investigate the reliability of the p-value. Apart from investigating the performance of p-value, we also introduce some existing approaches, Minimum Bayes Factors and Belief functions, for replacing p-value. Results from the simulation study confirm unreliable p-value in some cases and that our proposed approaches seem to be useful as the substituted tool in the statistical inference. Moreover, our results show that the plausibility approach is more accurate for making decisions about the null hypothesis than the traditionally used p-values when the null hypothesis is true. However, the MBFs of Edwards et al. [Bayesian statistical inference for psychological research. Psychol. Rev. 70(3) (1963), pp. 193–242]; Vovk [A logic of probability, with application to the foundations of statistics. J. Royal Statistical Soc. Series B (Methodological) 55 (1993), pp. 317–351] and Sellke et al. [Calibration of p values for testing precise null hypotheses. Am. Stat. 55(1) (2001), pp. 62–71] provide more reliable results compared to all other methods when the null hypothesis is false.KEYWORDS: Ban of P-value, Minimum Bayes Factors, belief functions     

The split sample permutation t-tests

Without the exchangeability assumption, permutation tests for comparing two population means do not provide exact control of the probability of making a Type I error. Another drawback of permutation tests is that it cannot be used to test hypothesis about one population. In this paper, we propose a new type of permutation tests for testing the difference between two population means: the split sample permutation t-tests. We show that the split sample permutation t-tests do not require the exchangeability assumption, are asymptotically exact and can be easily extended to testing hypothesis about one population. Extensive simulations were carried out to evaluate the performance of two specific split sample permutation t-tests: the split in the middle permutation t-test and the split in the end permutation t-test. The simulation results show that the split in the middle permutation t-test has comparable performance to the permutation test if the population distributions are symmetric and satisfy the exchangeability assumption. Otherwise, the split in the end permutation t-test has significantly more accurate control of level of significance than the split in the middle permutation t-test and other existing permutation tests.     

Convergence of posterior odds

The problem of approximating an interval null or imprecise hypothesis test by a point null or precise hypothesis test under a Bayesian framework is considered. In the literature, some of the methods for solving this problem have used the Bayes factor for testing a point null and justified it as an approximation to the interval null. However, many authors recommend evaluating tests through the posterior odds, a Bayesian measure of evidence against the null hypothesis. It is of interest then to determine whether similar results hold when using the posterior odds as the primary measure of evidence. For the prior distributions under which the approximation holds with respect to the Bayes factor, it is shown that the posterior odds for testing the point null hypothesis does not approximate the posterior odds for testing the interval null hypothesis. In fact, in order to obtain convergence of the posterior odds, a number of restrictive conditions need to be placed on the prior structure. Furthermore, under a non-symmetrical prior setup, neither the Bayes factor nor the posterior odds for testing the imprecise hypothesis converges to the Bayes factor or posterior odds respectively for testing the precise hypothesis. To rectify this dilemma, it is shown that constraints need to be placed on the priors. In both situations, the class of priors constructed to ensure convergence of the posterior odds are not practically useful, thus questioning, from a Bayesian perspective, the appropriateness of point null testing in a problem better represented by an interval null. The theories developed are also applied to an epidemiological data set from White et al. (Can. Veterinary J. 30 (1989) 147–149.) in order to illustrate and study priors for which the point null hypothesis test approximates the interval null hypothesis test. AMS Classification: Primary 62F15; Secondary 62A15     

Another look at life testing

In life testing, n identical testing items are placed on test. Instead of doing a complete life testing with all n outcomes, a Type II censored life testing, consisting of the first m outcomes, is usually employed. Although statistical analysis for the life testing based on censored data is less efficient than the complete life testing, the expected length of the censored life testing is less than that of the complete life testing. In this paper, we compare censored and complete life testing and suggest ways to improve time saving and efficiency. Instead of doing a complete life testing with all n outcomes, we put N>n items on test, which continues until we observe the nth outcome. With both the censored life testing and the complete life testing containing the same number of observations, we show that the expected length of the censored life testing is less than that of the complete life testing and that the censored life testing may be also more efficient than the complete life testing with the same number of observations.     

Testing time series linearity via goodness-of-fit methods

We consider the problem of testing time series linearity. Existing time domain and spectral domain tests are discussed. A new approach relying on spectral domain properties of a time series under the null hypothesis of linearity is suggested. Under linearity, the normalized bispectral density function Z is a constant. Under the null hypothesis of linearity, properly constructed estimators of 2|Z|² have a non-central chi-squared distribution with two degrees of freedom and constant non-centrality parameter 2|Z|². If the null hypothesis is false, the non-centrality parameter is non-constant. This suggests goodness-of-fit tests might be effective in diagnosing non-linearity. Several approaches are introduced.     

The Bayesian and frequentist approaches to testing a one-sided hypothesis about a multivariate mean

This paper compares the Bayesian and frequentist approaches to testing a one-sided hypothesis about a multivariate mean. First, this paper proposes a simple way to assign a Bayesian posterior probability to one-sided hypotheses about a multivariate mean. The approach is to use (almost) the exact posterior probability under the assumption that the data has multivariate normal distribution, under either a conjugate prior in large samples or under a vague Jeffreys prior. This is also approximately the Bayesian posterior probability of the hypothesis based on a suitably flat Dirichlet process prior over an unknown distribution generating the data. Then, the Bayesian approach and a frequentist approach to testing the one-sided hypothesis are compared, with results that show a major difference between Bayesian reasoning and frequentist reasoning. The Bayesian posterior probability can be substantially smaller than the frequentist p-value. A class of example is given where the Bayesian posterior probability is basically 0, while the frequentist p-value is basically 1. The Bayesian posterior probability in these examples seems to be more reasonable. Other drawbacks of the frequentist p-value as a measure of whether the one-sided hypothesis is true are also discussed.     

Distribution-free tests for parallelism and concurrence in two-sample regression problem

This paper suggests some distribution-free methods for testing hypothesis of parallelism and concurrence of two linear regressions. We assume that the independent variable x is equally spaced. The proposed procedures are compared with nonparametric competitors and the normal theory t-test.     

Testing of two sample proportional intensity assumption for non-homogeneous Poisson processes

For two independent non-homogeneous Poisson processes with unknown intensities we propose a test for testing the hypothesis that the ratio of the intensities is constant versus it is increasing on (0,t]. The existing test procedures for testing such relative trends are based on conditioning on the number of failures observed in (0,t] from the two processes. Our test is unconditional and is based on the original time truncated data which enables us to have meaningful asymptotics. We obtain the asymptotic null distribution (as t becomes large) of the proposed test statistic and show that the proposed test is consistent against several large classes of alternatives. It was observed by Park and Kim (IEEE. Trans. Rehab. 40 (1), 1992, 107–111) that it is difficult to distinguish between the power-law and log-linear processes for certain parameter values. We show that our test is consistent for such alternatives also.     

Statistical inferences for partially linear single-index models with error-prone linear covariates

We consider statistical inference for partially linear single-index models (PLSIM) when some linear covariates are not observed, but ancillary variables are available. Based on the profile least-squared estimators of the unknowns, we study the testing problems for parametric components in the proposed models. It is to see whether the generalized likelihood ratio (GLR) tests proposed by Fan et al. (2001) are applicable to testing for the parametric components. We show that under the null hypothesis the proposed GLR statistics follow asymptotically the χ²-distributions with the scale constants and the degrees of freedom being independent of the nuisance parameters or functions, which is called the Wilks phenomenon. Simulated experiments are conducted to illustrate our proposed methodology.     

On large deviation asymptotics of some tests in time series

Let {X_t} be a Gaussian stationary process with spectral density f_θ(λ). The problem considered is that of testing a simple hypothesis H₀:θ=θ₀ against the alternative A:θ≠θ₀. For this we investigate the Bahadur efficiency of the likelihood ratio, Rao, modified Wald and Wald tests. The Bahadur efficiency is based on the large deviation theory. Then it is shown that the asymptotics of the above tests are identical up to second-order in a certain sense. We show that this result makes a sharp contrast with the ordinary higher-order asymptotic theory for tests.     

A note on multiple testing for composite null hypotheses

Multiple hypothesis testing literature has recently experienced a growing development with particular attention to the control of the false discovery rate (FDR) based on p-values. While these are not the only methods to deal with multiplicity, inference with small samples and large sets of hypotheses depends on the specific choice of the p-value used to control the FDR in the presence of nuisance parameters. In this paper we propose to use the partial posterior predictive p-value [Bayarri, M.J., Berger, J.O., 2000. p-values for composite null models. J. Amer. Statist. Assoc. 95, 1127–1142] that overcomes this difficulty. This choice is motivated by theoretical considerations and examples. Finally, an application to a controlled microarray experiment is presented.     

Large and moderate deviations for modified Engel series

The modified Engel series of real numbers introduced by Rényi (1962 Rényi, A. (1962). A new approach to the theory of Engel’s series. Ann. Univ. Sci. Budapest. Eötvös Sect. Math. 5:25–32. [Google Scholar]) is a simple modification of Engel series, and they have the same classical limit theorems, such as the law of large numbers, central limit theorem, and law of the iterated logarithm. In this paper, we studied the large and moderate deviations for modified Engel series, which indicate that the large deviations for modified Engel series and Engel series are different.     

Characterization of diagonal symmetry: location unknown,and a test based on allied U-processes

For a (possibly multivariate) distribution F, a characterization of (diagonal) symmetry is made with respect to a kernel of degree 2; this is incorporated in the formulation of appropriate U-processes that provide the access to a suitable test statistic for testing the hypothesis of diagonal symmetry when the location is treated as unknown. Asymptotic properties of the test are studied.     

On the use of the t-statistic in preliminary testing procedures for the Behrens-Fisher problem

Preliminary testing procedures for the two means problem traditionally employ the pooled variance t-statistic. In this paper we show that bias of the t-statistic under conditions of heterogeneity of variance may be increased if use of the t-statistic is conditional on an affirmative F-test. For this reason we conclude that use of the t-statistic in preliminary testing procedures is inappropriate.     

False discovery rate control for non-positively regression dependent test statistics

In this paper we present a modification of the Benjamini and Hochberg false discovery rate controlling procedure for testing non-positive dependent test statistics. The new testing procedure makes use of the same series of linearly increasing critical values. Yet, in the new procedure the set of p-values is divided into subsets of positively dependent p-values, and each subset of p-values is separately sorted and compared to the series of critical values. In the first part of the paper we introduce the new testing methodology, discuss the technical issues needed to apply the new approach, and apply it to data from a genetic experiment.     

Optimum designs in complete two-way layouts

In the usual two-way layout of ANOVA (interactions are admitted) let nij ? 1 be the number of observations for the factor-level combination(i, j). For testing the hypothesis that all main effects of the first factor vanish numbers $\text{n}{}^1\text{ij}$ are given such that the power function of the F-test is uniformly maximized (U-optimality), if one considers only designs (nij) for which the row-sums ni are prescribed. Furthermore, in the (larger) set of all designs for which the total number of observations is given, all D-optimum designs are constructed.     

Some asymptotic results for hypothesis testing in multivariate failure time data

This paper presents a class of generalized Wald, generalized score and generalized likelihood ratio statistics for hypothesis testing and model selection for multivariate failure time data. These statistics are based on a marginal hazard model with a common baseline hazard function. The large sample distributions of these statistics are examined. It is shown that the proposed test statistics follow asymptotically a weighted sum of independent χ₁² distributions.