On predictive distributions and Bayesian networks

In this paper we are interested in discrete prediction problems for a decision-theoretic setting, where the task is to compute the predictive distribution for a finite set of possible alternatives. This question is first addressed in a general Bayesian framework, where we consider a set of probability distributions defined by some parametric model class. Given a prior distribution on the model parameters and a set of sample data, one possible approach for determining a predictive distribution is to fix the parameters to the instantiation with the maximum a posteriori probability. A more accurate predictive distribution can be obtained by computing the evidence (marginal likelihood), i.e., the integral over all the individual parameter instantiations. As an alternative to these two approaches, we demonstrate how to use Rissanen's new definition of stochastic complexity for determining predictive distributions, and show how the evidence predictive distribution with Jeffrey's prior approaches the new stochastic complexity predictive distribution in the limit with increasing amount of sample data. To compare the alternative approaches in practice, each of the predictive distributions discussed is instantiated in the Bayesian network model family case. In particular, to determine Jeffrey's prior for this model family, we show how to compute the (expected) Fisher information matrix for a fixed but arbitrary Bayesian network structure. In the empirical part of the paper the predictive distributions are compared by using the simple tree-structured Naive Bayes model, which is used in the experiments for computational reasons. The experimentation with several public domain classification datasets suggest that the evidence approach produces the most accurate predictions in the log-score sense. The evidence-based methods are also quite robust in the sense that they predict surprisingly well even when only a small fraction of the full training set is used.     

A non-iterative sampling Bayesian method for linear mixed models with normal independent distributions

In this paper, we utilize normal/independent (NI) distributions as a tool for robust modeling of linear mixed models (LMM) under a Bayesian paradigm. The purpose is to develop a non-iterative sampling method to obtain i.i.d. samples approximately from the observed posterior distribution by combining the inverse Bayes formulae, sampling/importance resampling and posterior mode estimates from the expectation maximization algorithm to LMMs with NI distributions, as suggested by Tan et al. [33 Tan, M., Tian, G. and Ng, K. 2003. A noniterative sampling method for computing posteriors in the structure of EM-type algorithms. Statist. Sinica, 13(3): 625–640. [Web of Science ®] , [Google Scholar]]. The proposed algorithm provides a novel alternative to perfect sampling and eliminates the convergence problems of Markov chain Monte Carlo methods. In order to examine the robust aspects of the NI class, against outlying and influential observations, we present a Bayesian case deletion influence diagnostics based on the Kullback–Leibler divergence. Further, some discussions on model selection criteria are given. The new methodologies are exemplified through a real data set, illustrating the usefulness of the proposed methodology.     

Understanding predictive information criteria for Bayesian models

We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this paper is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice.     

Bayesian analysis of censored linear regression models with scale mixtures of normal distributions

As is the case of many studies, the data collected are limited and an exact value is recorded only if it falls within an interval range. Hence, the responses can be either left, interval or right censored. Linear (and nonlinear) regression models are routinely used to analyze these types of data and are based on normality assumptions for the errors terms. However, those analyzes might not provide robust inference when the normality assumptions are questionable. In this article, we develop a Bayesian framework for censored linear regression models by replacing the Gaussian assumptions for the random errors with scale mixtures of normal (SMN) distributions. The SMN is an attractive class of symmetric heavy-tailed densities that includes the normal, Student-t, Pearson type VII, slash and the contaminated normal distributions, as special cases. Using a Bayesian paradigm, an efficient Markov chain Monte Carlo algorithm is introduced to carry out posterior inference. A new hierarchical prior distribution is suggested for the degrees of freedom parameter in the Student-t distribution. The likelihood function is utilized to compute not only some Bayesian model selection measures but also to develop Bayesian case-deletion influence diagnostics based on the q-divergence measure. The proposed Bayesian methods are implemented in the R package BayesCR. The newly developed procedures are illustrated with applications using real and simulated data.     

Sequential testing procedures for a class of distributions representing various life-testing models

A class of probability density functions is considered, which covers several life-testing models as specific cases. Sequential probability ratio tests are developed for testing simple and composite hypotheses regarding the parameters of the probabilistic model. Expressions for the operating characteristic and the average sample number functions are derived and their behaviour is studied by means of graph-plotting.     

Bayesian regression analysis with linked data using mixture normal distributions

The probabilistic uncertainty in record linkage affects statistical analysis such as regression analysis of linked data. This paper considers Bayesian regression analysis with linked data and shows that despite using the usual normal regression analysis, the least squares type estimators of regression coefficients are not always adequate. A method is proposed in which the distribution of the response variable is used. This method is related to finite mixture analysis and leads to more accurate estimations. A simple approach has been proposed to increase the tractability and reduce the number of mixture components. A Monte Carlo simulation study is also performed to assess the proposed approach.     

A class of weighted normal distributions and its variants useful for inequality constrained analysis

This article develops a class of the weighted normal distributions for which the probability density function has the form of a product of a normal density and a weight function. The class constitutes marginal distributions obtained from various kinds of doubly truncated bivariate normal distributions. This class of distributions strictly includes the normal, skew–normal and two-piece skew–normal and is useful for selection modelling and inequality constrained normal mean analysis. Some distributional properties and Bayesian perspectives of the class are given. Probabilistic representation of the distributions is also given. The representation is shown to be straightforward to specify distribution and to implement computation, with output readily adapted for required analysis. Necessary theories and illustrative examples are provided.     

Stochastic volatility in mean models with scale mixtures of normal distributions and correlated errors: A Bayesian approach

A stochastic volatility in mean model with correlated errors using the symmetrical class of scale mixtures of normal distributions is introduced in this article. The scale mixture of normal distributions is an attractive class of symmetric distributions that includes the normal, Student-t, slash and contaminated normal distributions as special cases, providing a robust alternative to estimation in stochastic volatility in mean models in the absence of normality. Using a Bayesian paradigm, an efficient method based on Markov chain Monte Carlo (MCMC) is developed for parameter estimation. The methods developed are applied to analyze daily stock return data from the São Paulo Stock, Mercantile & Futures Exchange index (IBOVESPA). The Bayesian predictive information criteria (BPIC) and the logarithm of the marginal likelihood are used as model selection criteria. The results reveal that the stochastic volatility in mean model with correlated errors and slash distribution provides a significant improvement in model fit for the IBOVESPA data over the usual normal model.     

Bayesian consistency for regression models under a supremum distance

This paper studies the consistency of Bayesian nonparametric regression models. We concentrate on the use of the sup metric and dealing with non-stochastic, i.e. designed, covariate values. We illustrate our results on a normal mean regression function and demonstrate the usefulness of a model based on piecewise constant functions.     

Application of a predictive distribution formula to Bayesian computation for incomplete data models

We consider exact and approximate Bayesian computation in the presence of latent variables or missing data. Specifically we explore the application of a posterior predictive distribution formula derived in Sweeting And Kharroubi (2003), which is a particular form of Laplace approximation, both as an importance function and a proposal distribution. We show that this formula provides a stable importance function for use within poor man’s data augmentation schemes and that it can also be used as a proposal distribution within a Metropolis-Hastings algorithm for models that are not analytically tractable. We illustrate both uses in the case of a censored regression model and a normal hierarchical model, with both normal and Student t distributed random effects. Although the predictive distribution formula is motivated by regular asymptotic theory, it is not necessary that the likelihood has a closed form or that it possesses a local maximum.     

Bayesian forecasting for AR(1) models with normal coefficients

This paper presents a Bayesian solution to the problem of time series forecasting, for the case in which the generating process is an autoregressive of order one, with a normal random coefficient. The proposed procedure is based on the predictive density of the future observation. Conjugate priors are used for some parameters, while improper vague priors are used for others.     

Stationary mixture transition distribution (MTD) models via predictive distributions

This paper combines two ideas to construct autoregressive processes of arbitrary order. The first idea is the construction of first order stationary processes described in Pitt et al. [(2002). Constructing first order autoregressive models via latent processes. Scand. J. Statist.29, 657–663] and the second idea is the construction of higher order processes described in Raftery [(1985). A model for high order Markov chains. J. Roy. Statist. Soc. B.47, 528–539]. The resulting models provide appealing alternatives to model non-linear and non-Gaussian time series.     

Bayesian multivariate GARCH models with dynamic correlations and asymmetric error distributions

The main goal in this paper is to develop and apply stochastic simulation techniques for GARCH models with multivariate skewed distributions using the Bayesian approach. Both parameter estimation and model comparison are not trivial tasks and several approximate and computationally intensive methods (Markov chain Monte Carlo) will be used to this end. We consider a flexible class of multivariate distributions which can model both skewness and heavy tails. Also, we do not fix tail behaviour when dealing with fat tail distributions but leave it subject to inference.     

Bayesian predictive density of order statistics based on finite mixture models

Bayesian predictive density functions, which are necessary to obtain bounds for predictive intervals of future order statistics, are obtained when the population density is a finite mixture of general components. Such components include, among others, the Weibull (exponential and Rayleigh as special cases), compound Weibull (three-parameter Burr type XII), Pareto, beta, Gompertz and compound Gompertz distributions. The prior belief of the experimenter is measured by a general distribution that was suggested by AL-Hussaini (J. Statist. Plann. Inf. 79 (1999b) 79). Applications to finite mixtures of Weibull and Burr type XII components are illustrated and comparison is made, in the special cases of the exponential and Pareto type II components, with previous results.     

Bayesian inference for truncated exponential distributions

Truncated normal distributions are considered as prior distributions for the truncation parameter in truncated exponential models. Posterior istributions re obtained, and inferenceforthe truncation parameter and for the reliability function is discussed. One and two parameter models are considered.     

Bayesian inference for multivariate gamma distributions

The paper considers the multivariate gamma distribution for which the method of moments has been considered as the only method of estimation due to the complexity of the likelihood function. With a non-conjugate prior, practical Bayesian analysis can be conducted using Gibbs sampling with data augmentation. The new methods are illustrated using artificial data for a trivariate gamma distribution as well as an application to technical inefficiency estimation.