A Monte Carlo approach to quantifying discrepancies between intractable posterior distributions

The computational demand required to perform inference using Markov chain Monte Carlo methods often obstructs a Bayesian analysis. This may be a result of large datasets, complex dependence structures, or expensive computer models. In these instances, the posterior distribution is replaced by a computationally tractable approximation, and inference is based on this working model. However, the error that is introduced by this practice is not well studied. In this paper, we propose a methodology that allows one to examine the impact on statistical inference by quantifying the discrepancy between the intractable and working posterior distributions. This work provides a structure to analyse model approximations with regard to the reliability of inference and computational efficiency. We illustrate our approach through a spatial analysis of yearly total precipitation anomalies where covariance tapering approximations are used to alleviate the computational demand associated with inverting a large, dense covariance matrix.     

A close look at the spatial structure implied by the CAR and SAR models

Modeling spatial interactions that arise in spatially referenced data is commonly done by incorporating the spatial dependence into the covariance structure either explicitly or implicitly via an autoregressive model. In the case of lattice (regional summary) data, two common autoregressive models used are the conditional autoregressive model (CAR) and the simultaneously autoregressive model (SAR). Both of these models produce spatial dependence in the covariance structure as a function of a neighbor matrix W and often a fixed unknown spatial correlation parameter. This paper examines in detail the correlation structures implied by these models as applied to an irregular lattice in an attempt to demonstrate their many counterintuitive or impractical results. A data example is used for illustration where US statewide average SAT verbal scores are modeled and examined for spatial structure using different spatial models.     

Strategies for Fitting Large,Geostatistical Data in MCMC Simulation

Models for geostatistical data introduce spatial dependence in the covariance matrix of location-specific random effects. This is usually defined to be a parametric function of the distances between locations. Bayesian formulations of such models overcome asymptotic inference and estimation problems involved in maximum likelihood-based approaches and can be fitted using Markov chain Monte Carlo (MCMC) simulation. The MCMC implementation, however, requires repeated inversions of the covariance matrix which makes the problem computationally intensive, especially for large number of locations. In the present work, we propose to convert the spatial covariance matrix to a sparse matrix and compare a number of numerical algorithms especially suited within the MCMC framework in order to accelerate large matrix inversion. The algorithms are assessed empirically on simulated datasets of different size and sparsity. We conclude that the band solver applied after ordering the distance matrix reduces the computational time in inverting covariance matrices substantially.     

Accent on Teaching Materials

Estimation of covariance components in the multivariate random-effect model with nested covariance structure is discussed. There are two covariance matrices to be estimated, namely, the between-group and the within-group covariance matrices. These two covariance matrices are most often estimated by forming a multivariate analysis of variance and equating mean square matrices to their expectations. Such a procedure involves taking the difference between the between-group mean square and the within-group mean square matrices, and often produces an estimated between-group covariance matrix that is not nonnegative definite. We present estimators of the two covariance matrices that are always proper covariance matrices. The estimators are the restricted maximum likelihood estimators if the random effects are normally distributed. The estimation procedure is extended to more complicated models, including the twofold nested and the mixed-effect models. A numerical example is presented to illustrate the use of the estimation procedure.     

A comparison of centring parameterisations of Gaussian process-based models for Bayesian computation using MCMC

Markov chain Monte Carlo (MCMC) algorithms for Bayesian computation for Gaussian process-based models under default parameterisations are slow to converge due to the presence of spatial- and other-induced dependence structures. The main focus of this paper is to study the effect of the assumed spatial correlation structure on the convergence properties of the Gibbs sampler under the default non-centred parameterisation and a rival centred parameterisation (CP), for the mean structure of a general multi-process Gaussian spatial model. Our investigation finds answers to many pertinent, but as yet unanswered, questions on the choice between the two. Assuming the covariance parameters to be known, we compare the exact rates of convergence of the two by varying the strength of the spatial correlation, the level of covariance tapering, the scale of the spatially varying covariates, the number of data points, the number and the structure of block updating of the spatial effects and the amount of smoothness assumed in a Matérn covariance function. We also study the effects of introducing differing levels of geometric anisotropy in the spatial model. The case of unknown variance parameters is investigated using well-known MCMC convergence diagnostics. A simulation study and a real-data example on modelling air pollution levels in London are used for illustrations. A generic pattern emerges that the CP is preferable in the presence of more spatial correlation or more information obtained through, for example, additional data points or by increased covariate variability.     

Modified linear projection for large spatial datasets

Recent developments in engineering techniques for spatial data collection such as geographic information systems have resulted in an increasing need for methods to analyze large spatial datasets. These sorts of datasets can be found in various fields of the natural and social sciences. However, model fitting and spatial prediction using these large spatial datasets are impractically time-consuming, because of the necessary matrix inversions. Various methods have been developed to deal with this problem, including a reduced rank approach and a sparse matrix approximation. In this article, we propose a modification to an existing reduced rank approach to capture both the large- and small-scale spatial variations effectively. We have used simulated examples and an empirical data analysis to demonstrate that our proposed approach consistently performs well when compared with other methods. In particular, the performance of our new method does not depend on the dependence properties of the spatial covariance functions.     

Discussing the “big n problem”

When a large amount of spatial data is available computational and modeling challenges arise and they are often labeled as “big n problem”. In this work we present a brief review of the literature. Then we focus on two approaches, respectively based on stochastic partial differential equations and integrated nested Laplace approximation, and on the tapering of the spatial covariance matrix. The fitting and predictive abilities of using the two methods in conjunction with Kriging interpolation are compared in a simulation study.     

Random effects selection in generalized linear mixed models via shrinkage penalty function

In this paper, we discuss the selection of random effects within the framework of generalized linear mixed models (GLMMs). Based on a reparametrization of the covariance matrix of random effects in terms of modified Cholesky decomposition, we propose to add a shrinkage penalty term to the penalized quasi-likelihood (PQL) function of the variance components for selecting effective random effects. The shrinkage penalty term is taken as a function of the variance of random effects, initiated by the fact that if the variance is zero then the corresponding variable is no longer random (with probability one). The proposed method takes the advantage of a convenient computation for the PQL estimation and appealing properties for certain shrinkage penalty functions such as LASSO and SCAD. We propose to use a backfitting algorithm to estimate the fixed effects and variance components in GLMMs, which also selects effective random effects simultaneously. Simulation studies show that the proposed approach performs quite well in selecting effective random effects in GLMMs. Real data analysis is made using the proposed approach, too.     

A general joint model for longitudinal measurements and competing risks survival data with heterogeneous random effects

This article studies a general joint model for longitudinal measurements and competing risks survival data. The model consists of a linear mixed effects sub-model for the longitudinal outcome, a proportional cause-specific hazards frailty sub-model for the competing risks survival data, and a regression sub-model for the variance–covariance matrix of the multivariate latent random effects based on a modified Cholesky decomposition. The model provides a useful approach to adjust for non-ignorable missing data due to dropout for the longitudinal outcome, enables analysis of the survival outcome with informative censoring and intermittently measured time-dependent covariates, as well as joint analysis of the longitudinal and survival outcomes. Unlike previously studied joint models, our model allows for heterogeneous random covariance matrices. It also offers a framework to assess the homogeneous covariance assumption of existing joint models. A Bayesian MCMC procedure is developed for parameter estimation and inference. Its performances and frequentist properties are investigated using simulations. A real data example is used to illustrate the usefulness of the approach.     

Semiparametric principal component poisson regression on clustered data

In modeling count data with multivariate predictors, we often encounter problems with clustering of observations and interdependency of predictors. We propose to use principal components of predictors to mitigate the multicollinearity problem and to abate information losses due to dimension reduction, a semiparametric link between the count dependent variable and the principal components is postulated. Clustering of observations is accounted into the model as a random component and the model is estimated via the backfitting algorithm. Simulation study illustrates the advantages of the proposed model over standard poisson regression in a wide range of scenarios.     

Testing the equality of two high‐dimensional spatial sign covariance matrices

This paper is concerned with testing the equality of two high‐dimensional spatial sign covariance matrices with applications to testing the proportionality of two high‐dimensional covariance matrices. It is interesting that these two testing problems are completely equivalent for the class of elliptically symmetric distributions. This paper develops a new test for testing the equality of two high‐dimensional spatial sign covariance matrices based on the Frobenius norm of the difference between two spatial sign covariance matrices. The asymptotic normality of the proposed testing statistic is derived under the null and alternative hypotheses when the dimension and sample sizes both tend to infinity. Moreover, the asymptotic power function is also presented. Simulation studies show that the proposed test performs very well in a wide range of settings and can be allowed for the case of large dimensions and small sample sizes.     

Covariance and variance evaluations of two estimators for drug–placebo difference in a trial with sequential parallel design

Chen et al. [Contemp. Clin. Trials, 32: 592–604 (2011)] heuristically proved that the covariance of two estimators is zero assuming equal correlation coefficients. In this article, the above covariance is analytically and rigorously re-derived without any strong assumption in equality between two correlation coefficients. Under rigorous analytic derivations plus assuming number of subjects continuing into Period 2 is a random variable, covariance is reconfirmed to be zero for both normal and binomial data.     

A permutation-based Bayesian approach for inverse covariance estimation

Abstract

Covariance estimation and selection for multivariate datasets in a high-dimensional regime is a fundamental problem in modern statistics. Gaussian graphical models are a popular class of models used for this purpose. Current Bayesian methods for inverse covariance matrix estimation under Gaussian graphical models require the underlying graph and hence the ordering of variables to be known. However, in practice, such information on the true underlying model is often unavailable. We therefore propose a novel permutation-based Bayesian approach to tackle the unknown variable ordering issue. In particular, we utilize multiple maximum a posteriori estimates under the DAG-Wishart prior for each permutation, and subsequently construct the final estimate of the inverse covariance matrix. The proposed estimator has smaller variability and yields order-invariant property. We establish posterior convergence rates under mild assumptions and illustrate that our method outperforms existing approaches in estimating the inverse covariance matrices via simulation studies.     

A Class of Convolution‐Based Models for Spatio‐Temporal Processes with Non‐Separable Covariance Structure

Abstract. In this article, we propose a new parametric family of models for real‐valued spatio‐temporal stochastic processes S ( x , t ) and show how low‐rank approximations can be used to overcome the computational problems that arise in fitting the proposed class of models to large datasets. Separable covariance models, in which the spatio‐temporal covariance function of S ( x , t ) factorizes into a product of purely spatial and purely temporal functions, are often used as a convenient working assumption but are too inflexible to cover the range of covariance structures encountered in applications. We define positive and negative non‐separability and show that in our proposed family we can capture positive, zero and negative non‐separability by varying the value of a single parameter.     

Independence distribution preserving joint covariance structures for the multivariate two-group case

We characterize the general nonnegative-definite and positive-definite joint observation covariance structures for the two-group case such that the two sample mean vectors are independent of the two corresponding sample covariance matrices. Also, the sample covariance matrices are distributed as independent noncentral or central Wishart random matrices. We derive and utilize a representation of the general common non-negative-definite solution to a particular system of matrix equations with idempotent coefficient matrices.     

Alternative covariance estimates in a replicated measurement error model with correlated,heteroscedastic errors

The article concerns covariance estimates in a replicated measurement error model with correlated, heteroscedastic errors. Freedman has conjectured that using more of the data will improve estimates of covariance matrices and result in a more efficient estimate of the coefficient of the regression model. The paper confirms the conjecture asymptotically for the case that all random variables are normally distributed, but the gain is not substantial.     

Computational algorithms for the factor model

Algorithms for computing the maximum likelihood estimators and the estimated covariance matrix of the estimators of the factor model are derived. The algorithms are particularly suitable for large matrices and for samples that give zero estimates of some error variances. A method of constructing estimators for reduced models is presented. The algorithms can also be used for the multivariate errors-in-variables model with known error covariance matrix.     

Identifiability and Comparison of Estimation Methods on Weibull Mixture Models

In this article, the finite mixture model of Weibull distributions is studied, the identifiability of the model with m components is proven, and the parameter estimators for the case of two components resulted by several algorithms are compared. The parameter estimators are obtained with maximum likelihood performing calculations with different algorithms: expectation-maximization (EM), Fisher scoring, backfitting, optimization of k-nearest neighbor approach, and random walk algorithm using Monte Carlo simulation. The Akaike information criterion and the log-likelihood value are used to compare models. In general, the proposed random walk algorithm shows better performance in mean square error and bias. Finally, the results are applied to electronic component lifetime data.     

Repeated measurements,log likelihood ratio tests and small samples

Various models have previously been proposed for data comprising m repeated measurements on each of N subjects. Log likelihood ratio tests may be used to help choose between possible models, but these tests are based on distributions which in theory apply only asymptotically. With small N , the log likelihood ratio approximation is unreliable, tending to reject the simpler of two models more often than it should. This is shown by reference to three datasets and analogous simulated data. For two of the three datasets, subjects fall into two groups. Log likelihood ratio tests confirm that for each of these two datasets group means over time differ. Tests suggest that group covariance structures also differ.     

Likelihood-based approach for analysis of longitudinal nominal data using marginalized random effects models

Likelihood-based marginalized models using random effects have become popular for analyzing longitudinal categorical data. These models permit direct interpretation of marginal mean parameters and characterize the serial dependence of longitudinal outcomes using random effects [12,22]. In this paper, we propose model that expands the use of previous models to accommodate longitudinal nominal data. Random effects using a new covariance matrix with a Kronecker product composition are used to explain serial and categorical dependence. The Quasi-Newton algorithm is developed for estimation. These proposed methods are illustrated with a real data set and compared with other standard methods.