Alternative posterior consistency results in nonparametric binary regression using Gaussian process priors

We establish consistency of posterior distribution when a Gaussian process prior is used as a prior distribution for the unknown binary regression function. Specifically, we take the work of Ghosal and Roy [2006. Posterior consistency of Gaussian process prior for nonparametric binary regression. Ann. Statist. 34, 2413–2429] as our starting point, and then weaken their assumptions on the smoothness of the Gaussian process kernel while retaining a stronger yet applicable condition about design points. Furthermore, we extend their results to multi-dimensional covariates under a weaker smoothness condition on the Gaussian process. Finally, we study the extent to which posterior consistency can be achieved under a general model where, when additional hyperparameters in the covariance function of a Gaussian process are involved.     

On asymptotic properties of Bayesian partially linear models

In this paper, we present large sample properties of a partially linear model from the Bayesian perspective, in which responses are explained by the semiparametric regression model with the additive form of the linear component and the nonparametric component. For this purpose, we investigate asymptotic behaviors of posterior distributions in terms of consistency. Specifically, we deal with a specific Bayesian partially linear regression model with additive noises in which the nonparametric component is modeled using Gaussian process priors. Under the Bayesian partially linear model using Gaussian process priors, we focus on consistency of posterior distribution and consistency of the Bayes factor, and extend these results to generalized additive regression models and study their asymptotic properties. In addition we illustrate the asymptotic properties based on empirical analysis through simulation studies.     

Consistency of Bernstein polynomial posteriors

A Bernstein prior is a probability measure on the space of all the distribution functions on [0, 1]. Under very general assumptions, it selects absolutely continuous distribution functions, whose densities are mixtures of known beta densities. The Bernstein prior is of interest in Bayesian nonparametric inference with continuous data. We study the consistency of the posterior from a Bernstein prior. We first show that, under mild assumptions, the posterior is weakly consistent for any distribution function P ₀ on [0, 1] with continuous and bounded Lebesgue density. With slightly stronger assumptions on the prior, the posterior is also Hellinger consistent. This implies that the predictive density from a Bernstein prior, which is a Bayesian density estimate, converges in the Hellinger sense to the true density (assuming that it is continuous and bounded). We also study a sieve maximum likelihood version of the density estimator and show that it is also Hellinger consistent under weak assumptions. When the order of the Bernstein polynomial, i.e. the number of components in the beta distribution mixture, is truncated, we show that under mild restrictions the posterior concentrates on the set of pseudotrue densities. Finally, we study the behaviour of the predictive density numerically and we also study a hybrid Bayes–maximum likelihood density estimator.     

Bayesian analysis of the scatterometer wind retrieval inverse problem: some new approaches

Summary.  The retrieval of wind vectors from satellite scatterometer observations is a non-linear inverse problem. A common approach to solving inverse problems is to adopt a Bayesian framework and to infer the posterior distribution of the parameters of interest given the observations by using a likelihood model relating the observations to the parameters, and a prior distribution over the parameters. We show how Gaussian process priors can be used efficiently with a variety of likelihood models, using local forward (observation) models and direct inverse models for the scatterometer. We present an enhanced Markov chain Monte Carlo method to sample from the resulting multimodal posterior distribution. We go on to show how the computational complexity of the inference can be controlled by using a sparse, sequential Bayes algorithm for estimation with Gaussian processes. This helps to overcome the most serious barrier to the use of probabilistic, Gaussian process methods in remote sensing inverse problems, which is the prohibitively large size of the data sets. We contrast the sampling results with the approximations that are found by using the sparse, sequential Bayes algorithm.     

Objective Bayes Factors for Gaussian Directed Acyclic Graphical Models

Abstract. We propose an objective Bayesian method for the comparison of all Gaussian directed acyclic graphical models defined on a given set of variables. The method, which is based on the notion of fractional Bayes factor (BF), requires a single default (typically improper) prior on the space of unconstrained covariance matrices, together with a prior sample size hyper‐parameter, which can be set to its minimal value. We show that our approach produces genuine BFs. The implied prior on the concentration matrix of any complete graph is a data‐dependent Wishart distribution, and this in turn guarantees that Markov equivalent graphs are scored with the same marginal likelihood. We specialize our results to the smaller class of Gaussian decomposable undirected graphical models and show that in this case they coincide with those recently obtained using limiting versions of hyper‐inverse Wishart distributions as priors on the graph‐constrained covariance matrices.     

A prior for the variance in hierarchical models

The choice of prior distributions for the variances can be important and quite difficult in Bayesian hierarchical and variance component models. For situations where little prior information is available, a ‘nonin-formative’ type prior is usually chosen. ‘Noninformative’ priors have been discussed by many authors and used in many contexts. However, care must be taken using these prior distributions as many are improper and thus, can lead to improper posterior distributions. Additionally, in small samples, these priors can be ‘informative’. In this paper, we investigate a proper ‘vague’ prior, the uniform shrinkage prior (Strawder-man 1971; Christiansen & Morris 1997). We discuss its properties and show how posterior distributions for common hierarchical models using this prior lead to proper posterior distributions. We also illustrate the attractive frequentist properties of this prior for a normal hierarchical model including testing and estimation. To conclude, we generalize this prior to the multivariate situation of a covariance matrix.     

Multivariate boundary kernels and a continuous least squares principle

Whereas there are many references on univariate boundary kernels, the construction of boundary kernels for multivariate density and curve estimation has not been investigated in detail. The use of multivariate boundary kernels ensures global consistency of multivariate kernel estimates as measured by the integrated mean-squared error or sup-norm deviation for functions with compact support. We develop a class of boundary kernels which work for any support, regardless of the complexity of its boundary. Our construction yields a boundary kernel for each point in the boundary region where the function is to be estimated. These boundary kernels provide a natural continuation of non-negative kernels used in the interior onto the boundary. They are obtained as solutions of the same kernel-generating variational problem which also produces the kernel function used in the interior as its solution. We discuss the numerical implementation of the proposed boundary kernels and their relationship to locally weighted least squares. Along the way we establish a continuous least squares principle and a continuous analogue of the Gauss–Markov theorem.     

Bayesian Goodness of Fit Testing with Mixtures of Triangular Distributions

Abstract.  We consider the consistency of the Bayes factor in goodness of fit testing for a parametric family of densities against a non-parametric alternative. Sufficient conditions for consistency of the Bayes factor are determined and demonstrated with priors using certain mixtures of triangular densities.     

Max-stable processes for modeling extremes observed in space and time

Max-stable processes have proved to be useful for the statistical modeling of spatial extremes. For statistical inference it is often assumed that there is no temporal dependence; i.e., that the observations at spatial locations are independent in time. In a first approach we construct max-stable space–time processes as limits of rescaled pointwise maxima of independent Gaussian processes, where the space–time covariance functions satisfy weak regularity conditions. This leads to so-called Brown–Resnick processes. In a second approach, we extend Smith’s storm profile model to a space–time setting. We provide explicit expressions for the bivariate distribution functions, which are equal under appropriate choice of the parameters. We also show how the space–time covariance function of the underlying Gaussian process can be interpreted in terms of the tail dependence function in the limiting max-stable space–time process.     

Density estimation using asymmetric kernels and Bayes bandwidths with censored data

We propose a modification to the regular kernel density estimation method that use asymmetric kernels to circumvent the spill over problem for densities with positive support. First a pivoting method is introduced for placement of the data relative to the kernel function. This yields a strongly consistent density estimator that integrates to one for each fixed bandwidth in contrast to most density estimators based on asymmetric kernels proposed in the literature. Then a data-driven Bayesian local bandwidth selection method is presented and lognormal, gamma, Weibull and inverse Gaussian kernels are discussed as useful special cases. Simulation results and a real-data example illustrate the advantages of the new methodology.     

Bayesian integrative analysis for multi-fidelity computer experiments

This paper proposes a Bayesian integrative analysis method for linking multi-fidelity computer experiments. Instead of assuming covariance structures of multivariate Gaussian process models, we handle the outputs from different levels of accuracy as independent processes and link them via a penalization method that controls the distance between their overall trends. Based on the priors induced by the penalty, we build Bayesian prediction models for the output at the highest accuracy. Simulated and real examples show that the proposed method is better than existing methods in terms of prediction accuracy for many cases.     

On the Posterior Consistency of Mixtures of Dirichlet Process Priors with Censored Data

Mixtures of Dirichlet process priors offer a reasonable compromise between purely parametric and purely non‐parametric models, and are popularly used in survival analysis and for testing problems with non‐parametric alternatives. In this paper, we study large sample properties of the posterior distribution with a mixture of Dirichlet process priors. We show that the posterior distribution of the survival function is consistent with right censored data.     

On probability matching priors

First‐order probability matching priors are priors for which Bayesian and frequentist inference, in the form of posterior quantiles, or confidence intervals, agree to a second order of approximation. The authors show that the matching priors developed by Peers (1965) and Tibshirani (1989) are readily and uniquely implemented in a third‐order approximation to the posterior marginal density. The authors further show how strong orthogonality of parameters simplifies the arguments. Several examples illustrate their results.     

Objective Bayesian analysis of spatial data with uncertain nugget and range parameters

The authors develop default priors for the Gaussian random field model that includes a nugget parameter accounting for the effects of microscale variations and measurement errors. They present the independence Jeffreys prior, the Jeffreys‐rule prior and a reference prior and study posterior propriety of these and related priors. They show that the uniform prior for the correlation parameters yields an improper posterior. In case of known regression and variance parameters, they derive the Jeffreys prior for the correlation parameters. They prove posterior propriety and obtain that the predictive distributions at ungauged locations have finite variance. Moreover, they show that the proposed priors have good frequentist properties, except for those based on the marginal Jeffreys‐rule prior for the correlation parameters, and illustrate their approach by analyzing a dataset of zinc concentrations along the river Meuse. The Canadian Journal of Statistics 40: 304–327; 2012 © 2012 Statistical Society of Canada     

Gains from joint cross validation bandwidth selection for derivatives of conditional multidimensional densities

This paper studies bandwidth selection for kernel estimation of derivatives of multidimensional conditional densities, a non-parametric realm unexplored in the literature. This paper extends Baird [Cross validation bandwidth selection for derivatives of multidimensional densities. RAND Working Paper series, WR-1060; 2014] in its examination of conditional multivariate densities, derives and presents criteria for arbitrary kernel order and density dimension, shows consistency of the estimators, and investigates a minimization criterion which jointly estimates numerator and denominator bandwidths. I conduct a Monte Carlo simulation study for various orders of kernels in the Gaussian family and compare the new cross validation criterion with those implied by Baird [Cross validation bandwidth selection for derivatives of multidimensional densities. RAND Working Paper series, WR-1060; 2014]. The paper finds that higher order kernels become increasingly important as the dimension of the distribution increases. I find that the cross validation criterion developed in this paper that jointly estimates the derivative of the joint density (numerator) and the marginal density (denominator) does orders of magnitude better than criteria that estimate the bandwidths separately. I further find that using the infinite order Dirichlet kernel tends to have the best results.     

Bayesian Kernel Mixtures for Counts

Although Bayesian nonparametric mixture models for continuous data are well developed, there is a limited literature on related approaches for count data. A common strategy is to use a mixture of Poissons, which unfortunately is quite restrictive in not accounting for distributions having variance less than the mean. Other approaches include mixing multinomials, which requires finite support, and using a Dirichlet process prior with a Poisson base measure, which does not allow smooth deviations from the Poisson. As a broad class of alternative models, we propose to use nonparametric mixtures of rounded continuous kernels. An efficient Gibbs sampler is developed for posterior computation, and a simulation study is performed to assess performance. Focusing on the rounded Gaussian case, we generalize the modeling framework to account for multivariate count data, joint modeling with continuous and categorical variables, and other complications. The methods are illustrated through applications to a developmental toxicity study and marketing data. This article has supplementary material online.     

Regularized Posteriors in Linear Ill‐Posed Inverse Problems

Abstract. We study the Bayesian solution of a linear inverse problem in a separable Hilbert space setting with Gaussian prior and noise distribution. Our contribution is to propose a new Bayes estimator which is a linear and continuous estimator on the whole space and is stronger than the mean of the exact Gaussian posterior distribution which is only defined as a measurable linear transformation. Our estimator is the mean of a slightly modified posterior distribution called regularized posterior distribution. Frequentist consistency of our estimator and of the regularized posterior distribution is proved. A Monte Carlo study and an application to real data confirm good small‐sample properties of our procedure.     

Asymptotic normality of the posterior given a statistic

The authors establish the asymptotic normality and determine the limiting variance of the posterior density for a multivariate parameter, given the value of a consistent and asymptotically Gaussian statistic satisfying a uniform local central limit theorem. Their proof is given in the continuous case but generalizes to lattice‐valued random variables. It hinges on a uniform Edgeworth expansion used to control the behaviour of the conditioning statistic. They provide examples and show how their result can help in identifying reference priors.     

Approximate reference priors for Gaussian random fields

Reference priors are theoretically attractive for the analysis of geostatistical data since they enable automatic Bayesian analysis and have desirable Bayesian and frequentist properties. But their use is hindered by computational hurdles that make their application in practice challenging. In this work, we derive a new class of default priors that approximate reference priors for the parameters of some Gaussian random fields. It is based on an approximation to the integrated likelihood of the covariance parameters derived from the spectral approximation of stationary random fields. This prior depends on the structure of the mean function and the spectral density of the model evaluated at a set of spectral points associated with an auxiliary regular grid. In addition to preserving the desirable Bayesian and frequentist properties, these approximate reference priors are more stable, and their computations are much less onerous than those of exact reference priors. Unlike exact reference priors, the marginal approximate reference prior of correlation parameter is always proper, regardless of the mean function or the smoothness of the correlation function. This property has important consequences for covariance model selection. An illustration comparing default Bayesian analyses is provided with a dataset of lead pollution in Galicia, Spain.     

Bayesian Instrumental Variables: Priors and Likelihoods

Instrumental variable (IV) regression provides a number of statistical challenges due to the shape of the likelihood. We review the main Bayesian literature on instrumental variables and highlight these pathologies. We discuss Jeffreys priors, the connection to the errors-in-the-variables problems and more general error distributions. We propose, as an alternative to the inverted Wishart prior, a new Cholesky-based prior for the covariance matrix of the errors in IV regressions. We argue that this prior is more flexible and more robust thanthe inverted Wishart prior since it is not based on only one tightness parameter and therefore can be more informative about certain components of the covariance matrix and less informative about others. We show how prior-posterior inference can be formulated in a Gibbs sampler and compare its performance in the weak instruments case for synthetic as well as two illustrations based on well-known real data.