Conditional probability and improper priors

ABSTRACT

According to Jeffreys improper priors are needed to get the Bayesian machine up and running. This may be disputed, but usage of improper priors flourish. Arguments based on symmetry or information theoretic reference analysis can be most convincing in concrete cases. The foundations of statistics as usually formulated rely on the axioms of a probability space, or alternative information theoretic axioms that imply the axioms of a probability space. These axioms do not include improper laws, but this is typically ignored in papers that consider improper priors.

The purpose of this paper is to present a mathematical theory that can be used as a foundation for statistics that include improper priors. This theory includes improper laws in the initial axioms and has in particular Bayes theorem as a consequence. Another consequence is that some of the usual calculation rules are modified. This is important in relation to common statistical practice which usually include improper priors, but tends to use unaltered calculation rules. In some cases, the results are valid, but in other cases inconsistencies may appear. The famous marginalization paradoxes exemplify this latter case.

An alternative mathematical theory for the foundations of statistics can be formulated in terms of conditional probability spaces. In this case, the appearance of improper laws is a consequence of the theory. It is proved here that the resulting mathematical structures for the two theories are equivalent. The conclusion is that the choice of the first or the second formulation for the initial axioms can be considered a matter of personal preference. Readers that initially have concerns regarding improper priors can possibly be more open toward a formulation of the initial axioms in terms of conditional probabilities. The interpretation of an improper law is given by the corresponding conditional probabilities.     

A Note on Priors for the Multinomial Model

An “overall objective” prior proposed for the multinomial model is shown to be inadequate in the presence of zero counts. An earlier proposed reference prior for when interest is in a particular category suffers from similar problems. It is argued that there is no need to deviate from the uniform prior proposed by Jeffreys, for which links with a non-Bayesian approach, when prediction is of interest, are shown.     

On predictive distributions and Bayesian networks

In this paper we are interested in discrete prediction problems for a decision-theoretic setting, where the task is to compute the predictive distribution for a finite set of possible alternatives. This question is first addressed in a general Bayesian framework, where we consider a set of probability distributions defined by some parametric model class. Given a prior distribution on the model parameters and a set of sample data, one possible approach for determining a predictive distribution is to fix the parameters to the instantiation with the maximum a posteriori probability. A more accurate predictive distribution can be obtained by computing the evidence (marginal likelihood), i.e., the integral over all the individual parameter instantiations. As an alternative to these two approaches, we demonstrate how to use Rissanen's new definition of stochastic complexity for determining predictive distributions, and show how the evidence predictive distribution with Jeffrey's prior approaches the new stochastic complexity predictive distribution in the limit with increasing amount of sample data. To compare the alternative approaches in practice, each of the predictive distributions discussed is instantiated in the Bayesian network model family case. In particular, to determine Jeffrey's prior for this model family, we show how to compute the (expected) Fisher information matrix for a fixed but arbitrary Bayesian network structure. In the empirical part of the paper the predictive distributions are compared by using the simple tree-structured Naive Bayes model, which is used in the experiments for computational reasons. The experimentation with several public domain classification datasets suggest that the evidence approach produces the most accurate predictions in the log-score sense. The evidence-based methods are also quite robust in the sense that they predict surprisingly well even when only a small fraction of the full training set is used.     

Bayesian analysis of the generalized gamma distribution using non-informative priors

The Generalized gamma (GG) distribution plays an important role in statistical analysis. For this distribution, we derive non-informative priors using formal rules, such as Jeffreys prior, maximal data information prior and reference priors. We have shown that these most popular formal rules with natural ordering of parameters, lead to priors with improper posteriors. This problem is overcome by considering a prior averaging approach discussed in Berger et al. [Overall objective priors. Bayesian Analysis. 2015;10(1):189–221]. The obtained hybrid Jeffreys-reference prior is invariant under one-to-one transformations and yields a proper posterior distribution. We obtained good frequentist properties of the proposed prior using a detailed simulation study. Finally, an analysis of the maximum annual discharge of the river Rhine at Lobith is presented.     

Objective Bayesian hypothesis testing in regression models with first-order autoregressive residuals

This article considers the objective Bayesian testing in the normal regression models with first-order autoregressive residuals. We propose some solutions based on a Bayesian model selection procedure to this problem where no subjective input is considered. We construct the proper priors for testing the autocorrelation coefficient based on measures of divergence between competing models, which is called the divergence-based (DB) priors and then propose the objective Bayesian decision-theoretic rule, which is called the Bayesian reference criterion (BRC). Finally, we derive the intrinsic test statistic for testing the autocorrelation coefficient. The behavior of the Bayes factor-based DB priors is examined by comparing with the BRC in a simulation study and an example.     

Non parametric Bayesian analysis of the two-sample problem with censoring

Testing for differences between two groups is a fundamental problem in statistics, and due to developments in Bayesian non parametrics and semiparametrics there has been renewed interest in approaches to this problem. Here we describe a new approach to developing such tests and introduce a class of such tests that take advantage of developments in Bayesian non parametric computing. This class of tests uses the connection between the Dirichlet process (DP) prior and the Wilcoxon rank sum test but extends this idea to the DP mixture prior. Here tests are developed that have appropriate frequentist sampling procedures for large samples but have the potential to outperform the usual frequentist tests. Extensions to interval and right censoring are considered and an application to a high-dimensional data set obtained from an RNA-Seq investigation demonstrates the practical utility of the method.     

Bayesian analysis for a stress-strength system under noninformative priors

Noninformative priors are used for estimating the reliability of a stress-strength system. Several reference priors (cf. Berger and Bernardo 1989, 1992) are derived. A class of priors is found by matching the coverage probabilities of one-sided Bayesian credible intervals with the corresponding frequentist coverage probabilities. It turns out that none of the reference priors is a matching prior. Sufficient conditions for propriety of posteriors under reference priors and matching priors are provided. A simple matching prior is compared with three reference priors when sample sizes are small. The study shows that the matching prior performs better than Jeffreys's prior and reference priors in meeting the target coverage probabilities.     

Computing some bayes estimates for the mean of a normal distribution

For sampling from a normal population with unknown mean, two families of prior densities for the mean are discussed. The corresponding posterior densities are found. A data analyst may choose a prior from these families to represent prior beliefs and then compute the corresponding Bayes estimator, using the techniques discussed.     

Reference priors for exponential families

Reference analysis, introduced by Bernardo (J. Roy. Statist. Soc. 41 (1979) 113) and further developed by Berger and Bernardo (On the development of reference priors (with discussion). In: J.M. Bernardo, J.O. Berger, A.P. Dawid, A.F.M. Smith (Eds.), Bayesian Statistics, Vol. 4, Clarendon Press, Oxford, pp. 35–60), has proved to be one of the most successful general methods to derive noninformative prior distributions. In practice, however, reference priors are typically difficult to obtain. In this paper we show how to find reference priors for a wide class of exponential family likelihoods.     

On probability matching priors

First‐order probability matching priors are priors for which Bayesian and frequentist inference, in the form of posterior quantiles, or confidence intervals, agree to a second order of approximation. The authors show that the matching priors developed by Peers (1965) and Tibshirani (1989) are readily and uniquely implemented in a third‐order approximation to the posterior marginal density. The authors further show how strong orthogonality of parameters simplifies the arguments. Several examples illustrate their results.     

Optimally and computations for relative surprise inferences

Relative surprise inferences are based on how beliefs change from a priori to a posteriori. As they are based on the posterior distribution of the integrated likelihood, inferences of this type are invariant under relabellings of the parameter of interest. The authors demonstrate that these inferences possess a certain optimality property. Further, they develop computational techniques for implementing them, provided that algorithms are available to sample from the prior and posterior distributions.     

A Method to Handle Zero Counts in the Multinomial Model

In the context of an objective Bayesian approach to the multinomial model, Dirichlet(a, …, a) priors with a < 1 have previously been shown to be inadequate in the presence of zero counts, suggesting that the uniform prior (a = 1) is the preferred candidate. In the presence of many zero counts, however, this prior may not be satisfactory either. A model selection approach is proposed, allowing for the possibility of zero parameters corresponding to zero count categories. This approach results in a posterior mixture of Dirichlet distributions and marginal mixtures of beta distributions, which seem to avoid the problems that potentially result from the various proposed Dirichlet priors, in particular in the context of extreme data with zero counts.     

Bayesian estimation and prediction for some life distributions based on record values

Some statistical data are most easily accessed in terms of record values. Examples include meteorology, hydrology and athletic events. Also, there are a number of industrial situations where experimental outcomes are a sequence of record-breaking observations. In this paper, Bayesian estimation for the two parameters of some life distributions, including Exponential, Weibull, Pareto and Burr type XII, are obtained based on upper record values. Prediction, either point or interval, for future upper record values is also presented from a Bayesian view point. Some of the non-Bayesian results can be achieved as limiting cases from our results. Numerical computations are given to illustrate the results.     

On the construction of Bayes–confidence regions

We obtain approximate Bayes–confidence intervals for a scalar parameter based on directed likelihood. The posterior probabilities of these intervals agree with their unconditional coverage probabilities to fourth order, and with their conditional coverage probabilities to third order. These intervals are constructed for arbitrary smooth prior distributions. A key feature of the construction is that log-likelihood derivatives beyond second order are not required, unlike the asymptotic expansions of Severini.     

Bayesian inference and prediction analysis of the power law process based on a natural conjugate prior

Under a natural conjugate prior with four hyperparameters, the importance sampling (IS) technique is applied to the Bayesian analysis of the power law process (PLP). Samples of the parameters of the PLP are obtained from IS. Based on these samples, not only the posterior analysis of parameters and some parameter functions in the PLP are performed conveniently, but also single-sample and two-sample prediction procedures are constructed easily. Furthermore, the sensitivity of the posterior mean of the parameter functions in the PLP is studied with respect to the hyperparameters of the natural conjugate prior and it can guide the selections of the hyperparameters directly. Coupled this sensitivity with the relations between the prior moments and the hyperparameters in the natural conjugate prior, it is possible to give directions about the selections of the prior moments to a certain degree. After some numerical experiments illustrate the rationality and feasibility of the proposed methods, an engineering example demonstrates its application.     

On a theorem of Stein relating Bayesian and classical inferences in group models

Let a group G act on the sample space. This paper gives another proof of a theorem of Stein relating a group invariant family of posterior Bayesian probability regions to classical confidence regions when an appropriate prior is used. The example of the central multivariate normal distribution is discussed.     

Dynamic models for spatiotemporal data

We propose a model for non-stationary spatiotemporal data. To account for spatial variability, we model the mean function at each time period as a locally weighted mixture of linear regressions. To incorporate temporal variation, we allow the regression coefficients to change through time. The model is cast in a Gaussian state space framework, which allows us to include temporal components such as trends, seasonal effects and autoregressions, and permits a fast implementation and full probabilistic inference for the parameters, interpolations and forecasts. To illustrate the model, we apply it to two large environmental data sets: tropical rainfall levels and Atlantic Ocean temperatures.     

Bayesian and fiducial inference for the inverse gaussian distribution via Gibbs sampler

This paper presents a kernel estimation of the distribution of the scale parameter of the inverse Gaussian distribution under type II censoring together with the distribution of the remaining time. Estimation is carried out via the Gibbs sampling algorithm combined with a missing data approach. Estimates and confidence intervals for the parameters of interest are also presented.     

A default Bayesian procedure for the generalized Pareto distribution

The generalized Pareto distribution is used to model the exceedances over a threshold in a number of fields, including the analysis of environmental extreme events and financial data analysis. We use this model in a default Bayesian framework where no prior information is available on unknown model parameters. Using a large simulation study, we compare the performance of our posterior estimations of parameters with other methods proposed in the literature. We show that our procedure also allows to make inferences in other quantities of interest in extreme value analysis without asymptotic arguments. We apply the proposed methodology to a real data set.     

Bayesian inference in the triangular cointegration model using a jeffreys prior

This paper presents a strategy for conducting Bayesian inference in the triangular cointegration model. A Jeffreys prior is used to circumvent an identification problem in the parameter region in which there is a near lack of cointegration. Sampling experiments are used to compare the repeated sampling performance of the approach with alternative classical cointegration methods. The Bayesian procedure is applied to testing for substitution between private and public consumption for a range of countries, with posterior estimates produced via Markov Chain Monte Carlo simulators.