Objective Bayes Factors for Gaussian Directed Acyclic Graphical Models

Abstract. We propose an objective Bayesian method for the comparison of all Gaussian directed acyclic graphical models defined on a given set of variables. The method, which is based on the notion of fractional Bayes factor (BF), requires a single default (typically improper) prior on the space of unconstrained covariance matrices, together with a prior sample size hyper‐parameter, which can be set to its minimal value. We show that our approach produces genuine BFs. The implied prior on the concentration matrix of any complete graph is a data‐dependent Wishart distribution, and this in turn guarantees that Markov equivalent graphs are scored with the same marginal likelihood. We specialize our results to the smaller class of Gaussian decomposable undirected graphical models and show that in this case they coincide with those recently obtained using limiting versions of hyper‐inverse Wishart distributions as priors on the graph‐constrained covariance matrices.     

Bayes model averaging with selection of regressors

Summary. When a number of distinct models contend for use in prediction, the choice of a single model can offer rather unstable predictions. In regression, stochastic search variable selection with Bayesian model averaging offers a cure for this robustness issue but at the expense of requiring very many predictors. Here we look at Bayes model averaging incorporating variable selection for prediction. This offers similar mean-square errors of prediction but with a vastly reduced predictor space. This can greatly aid the interpretation of the model. It also reduces the cost if measured variables have costs. The development here uses decision theory in the context of the multivariate general linear model. In passing, this reduced predictor space Bayes model averaging is contrasted with single-model approximations. A fast algorithm for updating regressions in the Markov chain Monte Carlo searches for posterior inference is developed, allowing many more variables than observations to be contemplated. We discuss the merits of absolute rather than proportionate shrinkage in regression, especially when there are more variables than observations. The methodology is illustrated on a set of spectroscopic data used for measuring the amounts of different sugars in an aqueous solution.     

Posterior Bayes factor analysis for an exponential regression model

In the exponential regression model, Bayesian inference concerning the non-linear regression parameter has proved extremely difficult. In particular, standard improper diffuse priors for the usual parameters lead to an improper posterior for the non-linear regression parameter. In a recent paper Ye and Berger (1991) applied the reference prior approach of Bernardo (1979) and Berger and Bernardo (1989) yielding a proper informative prior for . This prior depends on the values of the explanatory variable, goes to 0 as goes to 1, and depends on the specification of a hierarchical ordering of importance of the parameters.This paper explains the failure of the uniform prior to give a proper posterior: the reason is the appearance of the determinant of the information matrix in the posterior density for . We apply the posterior Bayes factor approach of Aitkin (1991) to this problem; in this approach we integrate out nuisance parameters with respect to their conditional posterior density given the parameter of interest. The resulting integrated likelihood for requires only the standard diffuse prior for all the parameters, and is unaffected by orderings of importance of the parameters. Computation of the likelihood for is extremely simple. The approach is applied to the three examples discussed by Berger and Ye and the likelihoods compared with their posterior densities.     

Bayesian analysis of the amended Davidson model for paired comparison using noninformative and informative priors

In recent years, numerous statisticians have focused their attention on the Bayesian analysis of different paired comparison models. While studying paired comparison techniques, the Davidson model is considered to be one of the famous paired comparison models in the available literature. In this article, we have introduced an amendment in the Davidson model which has been commenced to accommodate the option of not distinguishing the effects of two treatments when they are compared pairwise. Having made this amendment, the Bayesian analysis of the Amended Davidson model is performed using the noninformative (uniform and Jeffreys’) and informative (Dirichlet–gamma–gamma) priors. To study the model and to perform the Bayesian analysis with the help of an example, we have obtained the joint and marginal posterior distributions of the parameters, their posterior estimates, graphical presentations of the marginal densities, preference and predictive probabilities and the posterior probabilities to compare the treatment parameters.     

Bayes factors for goodness of fit testing

We propose the use of the generalized fractional Bayes factor for testing fit in multinomial models. This is a non-asymptotic method that can be used to quantify the evidence for or against a sub-model. We give expressions for the generalized fractional Bayes factor and we study its properties. In particular, we show that the generalized fractional Bayes factor has better properties than the fractional Bayes factor.     

Variational Bayes for Phase-Type Distribution

This article develops an algorithm for estimating parameters of general phase-type (PH) distribution based on Bayes estimation. The idea of Bayes estimation is to regard parameters as random variables, and the posterior distribution of parameters which is updated by the likelihood function provides estimators of parameters. One of the advantages of Bayes estimation is to evaluate uncertainty of estimators. In this article, we propose a fast algorithm for computing posterior distributions approximately, based on variational approximation. We formulate the optimal variational posterior distributions for PH distributions and develop the efficient computation algorithm for the optimal variational posterior distributions of discrete and continuous PH distributions.     

A Comparison of Bayes Factors for Separated Models: Some Simulation Results

Some alternative Bayes Factors: Intrinsic, Posterior, and Fractional have been proposed to overcome the difficulties presented when prior information is weak and improper prior are used. Additional difficulties also appear when the models are separated or non nested. This article presents both simulation results and some illustrative examples analysis comparing these alternative Bayes factors to discriminate among the Lognormal, the Weibull, the Gamma, and the Exponential distributions. Simulation results are obtained for different sample sizes generated from the distributions. Results from simulations indicates that these alternative Bayes factors are useful for comparing non nested models. The simulations also show some similar behavior and that when both models are true they choose the simplest model. Some illustrative example are also presented.     

Bayes and Empirical Bayes Inference in Changepoint Problems

ABSTRACT

In the case of the random design nonparametric regression, the double smoothing technique is applied to estimate the multivariate regression function. The proposed estimator has desirable properties in both the finite sample and the asymptotic cases. In the finite sample case, it has bounded conditional (and unconditional) bias and variance. On the other hand, in the asymptotic case, it has the same mean square error as the local linear estimator in Fan (Design-Adaptive Nonparametric Regression. Journal of the American Statistical Association 1992, 87, 998–1004; Local Linear Regression Smoothers and Their Minimax Efficiencies. Annals of Statistics 1993, 21, 196–216). Simulation studies demonstrate that the proposed estimator is better than the local linear estimator, because it has a smaller sample mean integrated square error and gives smoother estimates.     

Bayes and MCMC for Undergraduates

Students of statistics should be taught the ideas and methods that are widely used in practice and that will help them understand the world of statistics. Today, this means teaching them about Bayesian methods. In this article, I present ideas on teaching an undergraduate Bayesian course that uses Markov chain Monte Carlo and that can be a second course or, for strong students, a first course in statistics.     

Probability model choice in single samples from exponential families using Poisson log-linear modelling,and model comparison using Bayes and posterior Bayes factors

This paper describes a method due to Lindsey (1974a) for fitting different exponential family distributions for a single population to the same data, using Poisson log-linear modelling of the density or mass function. The method is extended to Efron's (1986) double exponential family, giving exact ML estimation of the two parameters not easily achievable directly. The problem of comparing the fit of the non-nested models is addressed by both Bayes and posterior Bayes factors (Aitkin, 1991). The latter allow direct comparisons of deviances from the fitted distributions.     

Variational Bayes for estimating the parameters of a hidden Potts model

Hidden Markov random field models provide an appealing representation of images and other spatial problems. The drawback is that inference is not straightforward for these models as the normalisation constant for the likelihood is generally intractable except for very small observation sets. Variational methods are an emerging tool for Bayesian inference and they have already been successfully applied in other contexts. Focusing on the particular case of a hidden Potts model with Gaussian noise, we show how variational Bayesian methods can be applied to hidden Markov random field inference. To tackle the obstacle of the intractable normalising constant for the likelihood, we explore alternative estimation approaches for incorporation into the variational Bayes algorithm. We consider a pseudo-likelihood approach as well as the more recent reduced dependence approximation of the normalisation constant. To illustrate the effectiveness of these approaches we present empirical results from the analysis of simulated datasets. We also analyse a real dataset and compare results with those of previous analyses as well as those obtained from the recently developed auxiliary variable MCMC method and the recursive MCMC method. Our results show that the variational Bayesian analyses can be carried out much faster than the MCMC analyses and produce good estimates of model parameters. We also found that the reduced dependence approximation of the normalisation constant outperformed the pseudo-likelihood approximation in our analysis of real and synthetic datasets.     

Bayesian inference method for model validation and confidence extrapolation

This paper presents a Bayesian-hypothesis-testing-based methodology for model validation and confidence extrapolation under uncertainty, using limited test data. An explicit expression of the Bayes factor is derived for the interval hypothesis testing. The interval method is compared with the Bayesian point null hypothesis testing approach. The Bayesian network with Markov Chain Monte Carlo simulation and Gibbs sampling is explored for extrapolating the inference from the validated domain at the component level to the untested domain at the system level. The effect of the number of experiments on the confidence in the model validation decision is investigated. The probabilities of Type I and Type II errors in decision-making during the model validation and confidence extrapolation are quantified. The proposed methodologies are applied to a structural mechanics problem. Numerical results demonstrate that the Bayesian methodology provides a quantitative approach to facilitate rational decisions in model validation and confidence extrapolation under uncertainty.     

Posterior robustness with more than one sampling model

In order to robustify posterior inference, besides the use of large classes of priors, it is necessary to consider uncertainty about the sampling model. In this article we suggest that a convenient and simple way to incorporate model robustness is to consider a discrete set of competing sampling models, and combine it with a suitable large class of priors. This set reflects foreseeable departures of the base model, like thinner or heavier tails or asymmetry. We combine the models with different classes of priors that have been proposed in the vast literature on Bayesian robustness with respect to the prior. Also we explore links with the related literature of stable estimation and precise measurement theory, now with more than one model entertained. To these ends it will be necessary to introduce a procedure for model comparison that does not depend on an arbitrary constant or scale. We utilize a recent development on automatic Bayes factors with self-adjusted scale, the ‘intrinsic Bayes factor’ (Berger and Pericchi, Technical Report, 1993).     

The calibration of P-values,posterior Bayes factors and the AIC from the posterior distribution of the likelihood

The posterior distribution of the likelihood is used to interpret the evidential meaning of P-values, posterior Bayes factors and Akaike's information criterion when comparing point null hypotheses with composite alternatives. Asymptotic arguments lead to simple re-calibrations of these criteria in terms of posterior tail probabilities of the likelihood ratio. (Prior) Bayes factors cannot be calibrated in this way as they are model-specific.     

Simpson's paradox and the Bayes factor

A counter-example presented by Lindley to Aitkin's posterior Bayes factor is examined. The paradoxical feature of the counter-example is found to be a simple case of Simpson's paradox.     

Performances of Bayesian model selection criteria for generalized linear models with non-ignorably missing covariates

This article deals with model comparison as an essential part of generalized linear modelling in the presence of covariates missing not at random (MNAR). We provide an evaluation of the performances of some of the popular model selection criteria, particularly of deviance information criterion (DIC) and weighted L (WL) measure, for comparison among a set of candidate MNAR models. In addition, we seek to provide deviance and quadratic loss-based model selection criteria with alternative penalty terms targeting directly the MNAR models. This work is motivated by the need in the literature to understand the performances of these important model selection criteria for comparison among a set of MNAR models. A Monte Carlo simulation experiment is designed to assess the finite sample performances of these model selection criteria in the context of interest under different scenarios for missingness amounts. Some naturally driven DIC and WL extensions are also discussed and evaluated.     

Asymptotic properties of Bayes risk for one-sided tests

Consider a given sequence {T_n} of estimators for a real-valued parameter θ. This paper studies asymptotic properties of restricted Bayes tests of the following form: reject H₀:θ ≤ θ₀ in favour of the alternative θ > θ₀ if T_n ≤ C_n, where the critical point C_n is determined to minimize among all tests of this form the expected probability of error with respect to the prior distribution. Such tests may or may not be fully Bayes tests, and so are called T_n-Bayes. Under fairly broad conditions it is shown that and the T_n-Bayes risk where a_n is the order of the standard error of T_n, - is the prior density, and μ is the median of F, the limit distribution of (T_n – θ)/a_nb(θ). Several examples are given.     

A comparison of Bayesian model selection based on MCMC with an application to GARCH-type models

This paper presents a comprehensive review and comparison of five computational methods for Bayesian model selection, based on MCMC simulations from posterior model parameter distributions. We apply these methods to a well-known and important class of models in financial time series analysis, namely GARCH and GARCH-t models for conditional return distributions (assuming normal and t-distributions). We compare their performance with the more common maximum likelihood-based model selection for simulated and real market data. All five MCMC methods proved reliable in the simulation study, although differing in their computational demands. Results on simulated data also show that for large degrees of freedom (where the t-distribution becomes more similar to a normal one), Bayesian model selection results in better decisions in favor of the true model than maximum likelihood. Results on market data show the instability of the harmonic mean estimator and reliability of the advanced model selection methods.     

Bayes factor consistency for unbalanced ANOVA models

In practical situations, most experimental designs often yield unbalanced data which have different numbers of observations per unit because of cost constraints, missing data, etc. In this paper, we consider the Bayesian approach to hypothesis testing or model selection under the one-way unbalanced fixed-effects analysis-of-variance (ANOVA) model. We adopt Zellner's g-prior with the beta-prime distribution for g, which results in an explicit closed-form expression of the Bayes factor without integral representation. Furthermore, we investigate the model selection consistency of the Bayes factor under three different asymptotic scenarios: either the number of units goes to infinity, the number of observations per unit goes to infinity, or both go to infinity. The results presented extend some existing ones of the Bayes factor for the balanced ANOVA models in the literature.