Bayesian measures of model complexity and fit

Summary. We consider the problem of comparing complex hierarchical models in which the number of parameters is not clearly defined. Using an information theoretic argument we derive a measure p _D for the effective number of parameters in a model as the difference between the posterior mean of the deviance and the deviance at the posterior means of the parameters of interest. In general p _D approximately corresponds to the trace of the product of Fisher's information and the posterior covariance, which in normal models is the trace of the 'hat' matrix projecting observations onto fitted values. Its properties in exponential families are explored. The posterior mean deviance is suggested as a Bayesian measure of fit or adequacy, and the contributions of individual observations to the fit and complexity can give rise to a diagnostic plot of deviance residuals against leverages. Adding p _D to the posterior mean deviance gives a deviance information criterion for comparing models, which is related to other information criteria and has an approximate decision theoretic justification. The procedure is illustrated in some examples, and comparisons are drawn with alternative Bayesian and classical proposals. Throughout it is emphasized that the quantities required are trivial to compute in a Markov chain Monte Carlo analysis.     

Using a mixture model for multiple imputation in the presence of outliers: the 'Healthy for life' project

Summary.  We consider the problem of obtaining population-based inference in the presence of missing data and outliers in the context of estimating the prevalence of obesity and body mass index measures from the 'Healthy for life' study. Identifying multiple outliers in a multivariate setting is problematic because of problems such as masking, in which groups of outliers inflate the covariance matrix in a fashion that prevents their identification when included, and swamping, in which outliers skew covariances in a fashion that makes non-outlying observations appear to be outliers. We develop a latent class model that assumes that each observation belongs to one of K unobserved latent classes, with each latent class having a distinct covariance matrix. We consider the latent class covariance matrix with the largest determinant to form an 'outlier class'. By separating the covariance matrix for the outliers from the covariance matrices for the remainder of the data, we avoid the problems of masking and swamping. As did Ghosh-Dastidar and Schafer, we use a multiple-imputation approach, which allows us simultaneously to conduct inference after removing cases that appear to be outliers and to promulgate uncertainty in the outlier status through the model inference. We extend the work of Ghosh-Dastidar and Schafer by embedding the outlier class in a larger mixture model, consider penalized likelihood and posterior predictive distributions to assess model choice and model fit, and develop the model in a fashion to account for the complex sample design. We also consider the repeated sampling properties of the multiple imputation removal of outliers.     

A prior for the variance in hierarchical models

The choice of prior distributions for the variances can be important and quite difficult in Bayesian hierarchical and variance component models. For situations where little prior information is available, a ‘nonin-formative’ type prior is usually chosen. ‘Noninformative’ priors have been discussed by many authors and used in many contexts. However, care must be taken using these prior distributions as many are improper and thus, can lead to improper posterior distributions. Additionally, in small samples, these priors can be ‘informative’. In this paper, we investigate a proper ‘vague’ prior, the uniform shrinkage prior (Strawder-man 1971; Christiansen & Morris 1997). We discuss its properties and show how posterior distributions for common hierarchical models using this prior lead to proper posterior distributions. We also illustrate the attractive frequentist properties of this prior for a normal hierarchical model including testing and estimation. To conclude, we generalize this prior to the multivariate situation of a covariance matrix.     

Posterior moments of scalar functions of a covariance matrix

Given multivariate normal data and a certain spherically invariant prior distribution on the covariance matrix, it is desired to estimate the moments of the posterior marginal distributions of some scalar functions of the covariance matrix by importance sampling. To this end a family of distributions is defined on the group of orthogonal matrices and a procedure is proposed for selecting one of these distributions for use as a weighting distribution in the importance sampling process. In an example estimates are calculated for the posterior mean and variance of each element in the covariance matrix expressed in the original coordinates, for the posterior mean of each element in the correlation matrix expressed in the original coordinates, and for the posterior mean of each element in the covariance matrix expressed in the coordinates of the principal variables.     

Comments on a Paper by Lachenbruch and Flack

Razzaghi (1987) conjectured that a wrong choice of covariance matrix in a restricted linear model results in loss of efficiency. This conjecture was proved correct by Kabe and Gupta for a wrong choice of constant covariance matrix. The present paper demonstrates that this loss of efficiency persists even with an estimated covariance matrix, thereby resulting in inefficient estimation, prediction, and confidence intervals.     

a bayesian analysis of the multivariate mixed-linear model

The Bayesian analysis of the multivariate mixed linear model is considered. The exact posterior distribution for the fixed effects matrix and the error covariance matrix are obtained. The exact posterior means and variances of the Bayesian estimators for the covariance matrices of random effects are also derived. These posterior moments are computed without constrained optimization and numerical integration. The calculations are feasible for arbitrary models. Reasonable approximations for the posterior distributions for the covariance matrices associated with the random effects are obtained also. Results are illustrated with a numerical example.     

Bayesian covariance matrix estimation using a mixture of decomposable graphical models

We present a Bayesian approach to estimating a covariance matrix by using a prior that is a mixture over all decomposable graphs, with the probability of each graph size specified by the user and graphs of equal size assigned equal probability. Most previous approaches assume that all graphs are equally probable. We show empirically that the prior that assigns equal probability over graph sizes outperforms the prior that assigns equal probability over all graphs in more efficiently estimating the covariance matrix. The prior requires knowing the number of decomposable graphs for each graph size and we give a simulation method for estimating these counts. We also present a Markov chain Monte Carlo method for estimating the posterior distribution of the covariance matrix that is much more efficient than current methods. Both the prior and the simulation method to evaluate the prior apply generally to any decomposable graphical model.     

Stochastic volatility demand systems

We address the estimation of stochastic volatility demand systems. In particular, we relax the homoscedasticity assumption and instead assume that the covariance matrix of the errors of demand systems is time-varying. Since most economic and financial time series are nonlinear, we achieve superior modeling using parametric nonlinear demand systems in which the unconditional variance is constant but the conditional variance, like the conditional mean, is also a random variable depending on current and past information. We also prove an important practical result of invariance of the maximum likelihood estimator with respect to the choice of equation eliminated from a singular demand system. An empirical application is provided, using the BEKK specification to model the conditional covariance matrix of the errors of the basic translog demand system.     

On a statistic useful in dimensionality reduction in multivariable linear stochastic system

In this paper we study the sampling properties of a test statistic which has important applications in the area of linear stochastic control systems with multi-inputs and multi-outputs. The statistic is the ratio of a partial sum of the eigenvalues of a sample covariance matrix and its trace. It turns out that using a method due to Sugiura we may derive a useful approximation for its distribution up to and including terms of order l/n, where n denotes the appropriate size. Numerical illustrations using real data are given.     

Bayesian Inference on Multivariate Normal Covariance and Precision Matrices in a Star-Shaped Model with Missing Data

In this article, we study Bayesian estimation for the covariance matrix Σ and the precision matrix Ω (the inverse of the covariance matrix) in the star-shaped model with missing data. Based on a Cholesky-type decomposition of the precision matrix Ω = Ψ′Ψ, where Ψ is a lower triangular matrix with positive diagonal elements, we develop the Jeffreys prior and a reference prior for Ψ. We then introduce a class of priors for Ψ, which includes the invariant Haar measures, Jeffreys prior, and reference prior. The posterior properties are discussed and the closed-form expressions for Bayesian estimators for the covariance matrix Σ and the precision matrix Ω are derived under the Stein loss, entropy loss, and symmetric loss. Some simulation results are given for illustration.     

Modeling a mixture of ordinal and continuous repeated measures

We study the correlation structure for a mixture of ordinal and continuous repeated measures using a Bayesian approach. We assume a multivariate probit model for the ordinal variables and a normal linear regression for the continuous variables, where latent normal variables underlying the ordinal data are correlated with continuous variables in the model. Due to the probit model assumption, we are required to sample a covariance matrix with some of the diagonal elements equal to one. The key computational idea is to use parameter-extended data augmentation, which involves applying the Metropolis-Hastings algorithm to get a sample from the posterior distribution of the covariance matrix incorporating the relevant restrictions. The methodology is illustrated through a simulated example and through an application to data from the UCLA Brain Injury Research Center.     

Covariance Matrix Estimation via Network Structure

In this article, we employ a regression formulation to estimate the high-dimensional covariance matrix for a given network structure. Using prior information contained in the network relationships, we model the covariance as a polynomial function of the symmetric adjacency matrix. Accordingly, the problem of estimating a high-dimensional covariance matrix is converted to one of estimating low dimensional coefficients of the polynomial regression function, which we can accomplish using ordinary least squares or maximum likelihood. The resulting covariance matrix estimator based on the maximum likelihood approach is guaranteed to be positive definite even in finite samples. Under mild conditions, we obtain the theoretical properties of the resulting estimators. A Bayesian information criterion is also developed to select the order of the polynomial function. Simulation studies and empirical examples illustrate the usefulness of the proposed methods.     

Robust estimation of a high-dimensional integrated covariance matrix

In this article, we consider a robust method of estimating a realized covariance matrix calculated as the sum of cross products of intraday high-frequency returns. According to recent articles in financial econometrics, the realized covariance matrix is essentially contaminated with market microstructure noise. Although techniques for removing noise from the matrix have been studied since the early 2000s, they have primarily investigated a low-dimensional covariance matrix with statistically significant sample sizes. We focus on noise-robust covariance estimation under converse circumstances, that is, a high-dimensional covariance matrix possibly with a small sample size. For the estimation, we utilize a statistical hypothesis test based on the characteristic that the largest eigenvalue of the covariance matrix asymptotically follows a Tracy–Widom distribution. The null hypothesis assumes that log returns are not pure noises. If a sample eigenvalue is larger than the relevant critical value, then we fail to reject the null hypothesis. The simulation results show that the estimator studied here performs better than others as measured by mean squared error. The empirical analysis shows that our proposed estimator can be adopted to forecast future covariance matrices using real data.     

Adaptive robust estimation in joint mean–covariance regression model for bivariate longitudinal data

The estimation of the covariance matrix is important in the analysis of bivariate longitudinal data. A good estimator for the covariance matrix can improve the efficiency of the estimators of the mean regression coefficients. Furthermore, the covariance estimation itself is also of interest, but it is a challenging job to model the covariance matrix of bivariate longitudinal data due to the complex structure and positive definite constraint. In addition, most of existing approaches are based on the maximum likelihood, which is very sensitive to outliers or heavy-tail error distributions. In this article, an adaptive robust estimation method is proposed for bivariate longitudinal data. Unlike the existing likelihood-based methods, the proposed method can adapt to different error distributions. Specifically, at first, we utilize the modified Cholesky block decomposition to parameterize the covariance matrices. Secondly, we apply the bounded Huber's score function to develop a set of robust generalized estimating equations to estimate the parameters both in the mean and the covariance models simultaneously. A data-driven approach is presented to select the parameter c in the Huber's score function, which can ensure that the proposed method is robust and efficient. A simulation study and a real data analysis are conducted to illustrate the robustness and efficiency of the proposed approach.     

Bayesian analysis of vector-autoregressive models with noninformative priors

In this paper, we investigate the properties of Bayes estimators of vector autoregression (VAR) coefficients and the covariance matrix under two commonly employed loss functions. We point out that the posterior mean of the variances of the VAR errors under the Jeffreys prior is likely to have an over-estimation bias. Our Bayesian computation results indicate that estimates using the constant prior on the VAR regression coefficients and the reference prior of Yang and Berger (Ann. Statist. 22 (1994) 1195) on the covariance matrix dominate the constant-Jeffreys prior estimates commonly used in applications of VAR models in macroeconomics. We also estimate a VAR model of consumption growth using both constant-reference and constant-Jeffreys priors.     

Efficiency of generalized estimating equations for binary responses

Summary.  Using standard correlation bounds, we show that in generalized estimation equations (GEEs) the so-called 'working correlation matrix' R ( α ) for analysing binary data cannot in general be the true correlation matrix of the data. Methods for estimating the correlation param-eter in current GEE software for binary responses disregard these bounds. To show that the GEE applied on binary data has high efficiency, we use a multivariate binary model so that the covariance matrix from estimating equation theory can be compared with the inverse Fisher information matrix. But R ( α ) should be viewed as the weight matrix, and it should not be confused with the correlation matrix of the binary responses. We also do a comparison with more general weighted estimating equations by using a matrix Cauchy–Schwarz inequality. Our analysis leads to simple rules for the choice of α in an exchangeable or autoregressive AR(1) weight matrix R ( α ), based on the strength of dependence between the binary variables. An example is given to illustrate the assessment of dependence and choice of α .     

A Monte Carlo approach to quantifying discrepancies between intractable posterior distributions

The computational demand required to perform inference using Markov chain Monte Carlo methods often obstructs a Bayesian analysis. This may be a result of large datasets, complex dependence structures, or expensive computer models. In these instances, the posterior distribution is replaced by a computationally tractable approximation, and inference is based on this working model. However, the error that is introduced by this practice is not well studied. In this paper, we propose a methodology that allows one to examine the impact on statistical inference by quantifying the discrepancy between the intractable and working posterior distributions. This work provides a structure to analyse model approximations with regard to the reliability of inference and computational efficiency. We illustrate our approach through a spatial analysis of yearly total precipitation anomalies where covariance tapering approximations are used to alleviate the computational demand associated with inverting a large, dense covariance matrix.     

Strategies for Fitting Large,Geostatistical Data in MCMC Simulation

Models for geostatistical data introduce spatial dependence in the covariance matrix of location-specific random effects. This is usually defined to be a parametric function of the distances between locations. Bayesian formulations of such models overcome asymptotic inference and estimation problems involved in maximum likelihood-based approaches and can be fitted using Markov chain Monte Carlo (MCMC) simulation. The MCMC implementation, however, requires repeated inversions of the covariance matrix which makes the problem computationally intensive, especially for large number of locations. In the present work, we propose to convert the spatial covariance matrix to a sparse matrix and compare a number of numerical algorithms especially suited within the MCMC framework in order to accelerate large matrix inversion. The algorithms are assessed empirically on simulated datasets of different size and sparsity. We conclude that the band solver applied after ordering the distance matrix reduces the computational time in inverting covariance matrices substantially.     

Use of the Bootstrap and Cross-Validation in Ridge Regression

Several existing methods for the choice of the ridge parameter are reviewed, and a bootstrap method is proposed. The bootstrap provides independent measures of prediction errors based on multiple predictions along with an estimate of the standard error of prediction. The bootstrap and selected competitors are compared through Monte Carlo simulations for various degrees of design matrix collinearity and varying levels of signal-to-noise ratio. The procedure is also illustrated by application to two published data sets. In one case, the bootstrap choice of the ridge parameter leads to a smaller mean squared error of prediction than the ridge trace method. In the second case, an optimal choice of no perturbation is confirmed. Benefits of the bootstrap choice include its less subjective nature, ease of implementation, and robustness.     

Rank covariance matrix estimation of a partially known covariance matrix

Classical multivariate methods are often based on the sample covariance matrix, which is very sensitive to outlying observations. One alternative to the covariance matrix is the affine equivariant rank covariance matrix (RCM) that has been studied in Visuri et al. [2003. Affine equivariant multivariate rank methods. J. Statist. Plann. Inference 114, 161–185]. In this article we assume that the covariance matrix is partially known and study how to estimate the corresponding RCM. We use the properties that the RCM is affine equivariant and that the RCM is proportional to the inverse of the regular covariance matrix, and hence reduce the problem of estimating the original RCM to estimating marginal rank covariance matrices. This is a great computational advantage when the dimension of the original data vector is large.