Markov chain Monte Carlo estimation of a mixture item response theory model

Markov chain Monte Carlo (MCMC) algorithms have been shown to be useful for estimation of complex item response theory (IRT) models. Although an MCMC algorithm can be very useful, it also requires care in use and interpretation of results. In particular, MCMC algorithms generally make extensive use of priors on model parameters. In this paper, MCMC estimation is illustrated using a simple mixture IRT model, a mixture Rasch model (MRM), to demonstrate how the algorithm operates and how results may be affected by some commonly used priors. Priors on the probabilities of mixtures, label switching, model selection, metric anchoring, and implementation of the MCMC algorithm using WinBUGS are described, and their effects illustrated on parameter recovery in practical testing situations. In addition, an example is presented in which an MRM is fitted to a set of educational test data using the MCMC algorithm and a comparison is illustrated with results from three existing maximum likelihood estimation methods.     

Imputation techniques for incomplete data in quadratic discriminant analysis

We have compared the efficacy of five imputation algorithms readily available in SAS for the quadratic discriminant function. Here, we have generated several different parametric-configuration training data with missing data, including monotone missing-at-random observations, and used a Monte Carlo simulation to examine the expected probabilities of misclassification for the two-class quadratic statistical discrimination problem under five different imputation methods. Specifically, we have compared the efficacy of the complete observation-only method and the mean substitution, regression, predictive mean matching, propensity score, and Markov Chain Monte Carlo (MCMC) imputation methods. We found that the MCMC and propensity score multiple imputation approaches are, in general, superior to the other imputation methods for the configurations and training-sample sizes we considered.     

Bayesian sieve methods: approximation rates and adaptive posterior contraction rates

In the last 20 years, a lot of achievements have been made in the study of posterior contraction rates of nonparametric Bayesian methods, and plenty of them involve sieve priors, but mainly for specific models or sieves. We provide a posterior contraction theorem for general parametric sieve priors. The theorem has weaker and simpler conditions compared with the existing results, and indicates that the sieve prior is rate adaptive. We apply the general theorem to density estimations and nonparametric regression with jumps. We also provided a reversible jump MCMC (Markov Chain Monte Carlo) algorithm for the sieve prior.     

Bayesian model selection in linear mixed effects models with autoregressive(p) errors using mixture priors

In this article, we apply the Bayesian approach to the linear mixed effect models with autoregressive(p) random errors under mixture priors obtained with the Markov chain Monte Carlo (MCMC) method. The mixture structure of a point mass and continuous distribution can help to select the variables in fixed and random effects models from the posterior sample generated using the MCMC method. Bayesian prediction of future observations is also one of the major concerns. To get the best model, we consider the commonly used highest posterior probability model and the median posterior probability model. As a result, both criteria tend to be needed to choose the best model from the entire simulation study. In terms of predictive accuracy, a real example confirms that the proposed method provides accurate results.     

Bayesian optimal sequential design for nonparametric regression via inhomogeneous evolutionary MCMC

We develop a novel computational methodology for Bayesian optimal sequential design for nonparametric regression. This computational methodology, that we call inhomogeneous evolutionary Markov chain Monte Carlo, combines ideas of simulated annealing, genetic or evolutionary algorithms, and Markov chain Monte Carlo. Our framework allows optimality criteria with general utility functions and general classes of priors for the underlying regression function. We illustrate the usefulness of our novel methodology with applications to experimental design for nonparametric function estimation using Gaussian process priors and free-knot cubic splines priors.     

An efficient Monte Carlo EM algorithm for Bayesian lasso

The lasso is a popular technique of simultaneous estimation and variable selection in many research areas. The marginal posterior mode of the regression coefficients is equivalent to estimates given by the non-Bayesian lasso when the regression coefficients have independent Laplace priors. Because of its flexibility of statistical inferences, the Bayesian approach is attracting a growing body of research in recent years. Current approaches are primarily to either do a fully Bayesian analysis using Markov chain Monte Carlo (MCMC) algorithm or use Monte Carlo expectation maximization (MCEM) methods with an MCMC algorithm in each E-step. However, MCMC-based Bayesian method has much computational burden and slow convergence. Tan et al. [An efficient MCEM algorithm for fitting generalized linear mixed models for correlated binary data. J Stat Comput Simul. 2007;77:929–943] proposed a non-iterative sampling approach, the inverse Bayes formula (IBF) sampler, for computing posteriors of a hierarchical model in the structure of MCEM. Motivated by their paper, we develop this IBF sampler in the structure of MCEM to give the marginal posterior mode of the regression coefficients for the Bayesian lasso, by adjusting the weights of importance sampling, when the full conditional distribution is not explicit. Simulation experiments show that the computational time is much reduced with our method based on the expectation maximization algorithm and our algorithms and our methods behave comparably with other Bayesian lasso methods not only in prediction accuracy but also in variable selection accuracy and even better especially when the sample size is relatively large.     

Bayesian inference for joint location and scale nonlinear models with skew-normal errors

A regression model with skew-normal errors provides a useful extension for ordinary normal regression models when the dataset under consideration involves asymmetric outcomes. In this article, we explore the use of Markov Chain Monte Carlo (MCMC) methods to develop a Bayesian analysis for joint location and scale nonlinear models with skew-normal errors, which relax the normality assumption and include the normal one as a special case. The main advantage of these class of distributions is that they have a nice hierarchical representation that allows the implementation of MCMC methods to simulate samples from the joint posterior distribution. Finally, simulation studies and a real example are used to illustrate the proposed methodology.     

Stochastic approximation Monte Carlo EM for change-point analysis

In the expectation–maximization (EM) algorithm for maximum likelihood estimation from incomplete data, Markov chain Monte Carlo (MCMC) methods have been used in change-point inference for a long time when the expectation step is intractable. However, the conventional MCMC algorithms tend to get trapped in local mode in simulating from the posterior distribution of change points. To overcome this problem, in this paper we propose a stochastic approximation Monte Carlo version of EM (SAMCEM), which is a combination of adaptive Markov chain Monte Carlo and EM utilizing a maximum likelihood method. SAMCEM is compared with the stochastic approximation version of EM and reversible jump Markov chain Monte Carlo version of EM on simulated and real datasets. The numerical results indicate that SAMCEM can outperform among the three methods by producing much more accurate parameter estimates and the ability to achieve change-point positions and estimates simultaneously.     

Markov chain Monte Carlo with the Integrated Nested Laplace Approximation

The Integrated Nested Laplace Approximation (INLA) has established itself as a widely used method for approximate inference on Bayesian hierarchical models which can be represented as a latent Gaussian model (LGM). INLA is based on producing an accurate approximation to the posterior marginal distributions of the parameters in the model and some other quantities of interest by using repeated approximations to intermediate distributions and integrals that appear in the computation of the posterior marginals. INLA focuses on models whose latent effects are a Gaussian Markov random field. For this reason, we have explored alternative ways of expanding the number of possible models that can be fitted using the INLA methodology. In this paper, we present a novel approach that combines INLA and Markov chain Monte Carlo (MCMC). The aim is to consider a wider range of models that can be fitted with INLA only when some of the parameters of the model have been fixed. We show how new values of these parameters can be drawn from their posterior by using conditional models fitted with INLA and standard MCMC algorithms, such as Metropolis–Hastings. Hence, this will extend the use of INLA to fit models that can be expressed as a conditional LGM. Also, this new approach can be used to build simpler MCMC samplers for complex models as it allows sampling only on a limited number of parameters in the model. We will demonstrate how our approach can extend the class of models that could benefit from INLA, and how the R-INLA package will ease its implementation. We will go through simple examples of this new approach before we discuss more advanced applications with datasets taken from the relevant literature. In particular, INLA within MCMC will be used to fit models with Laplace priors in a Bayesian Lasso model, imputation of missing covariates in linear models, fitting spatial econometrics models with complex nonlinear terms in the linear predictor and classification of data with mixture models. Furthermore, in some of the examples we could exploit INLA within MCMC to make joint inference on an ensemble of model parameters.     

Bayesian estimation based on progressive Type-II censoring from two-parameter bathtub-shaped lifetime model: an Markov chain Monte Carlo approach

In this paper, maximum likelihood and Bayes estimators of the parameters, reliability and hazard functions have been obtained for two-parameter bathtub-shaped lifetime distribution when sample is available from progressive Type-II censoring scheme. The Markov chain Monte Carlo (MCMC) method is used to compute the Bayes estimates of the model parameters. It has been assumed that the parameters have gamma priors and they are independently distributed. Gibbs within the Metropolis–Hasting algorithm has been applied to generate MCMC samples from the posterior density function. Based on the generated samples, the Bayes estimates and highest posterior density credible intervals of the unknown parameters as well as reliability and hazard functions have been computed. The results of Bayes estimators are obtained under both the balanced-squared error loss and balanced linear-exponential (BLINEX) loss. Moreover, based on the asymptotic normality of the maximum likelihood estimators the approximate confidence intervals (CIs) are obtained. In order to construct the asymptotic CI of the reliability and hazard functions, we need to find the variance of them, which are approximated by delta and Bootstrap methods. Two real data sets have been analyzed to demonstrate how the proposed methods can be used in practice.     

Bayesian composite quantile regression for linear mixed-effects models

Longitudinal data are commonly modeled with the normal mixed-effects models. Most modeling methods are based on traditional mean regression, which results in non robust estimation when suffering extreme values or outliers. Median regression is also not a best choice to estimation especially for non normal errors. Compared to conventional modeling methods, composite quantile regression can provide robust estimation results even for non normal errors. In this paper, based on a so-called pseudo composite asymmetric Laplace distribution (PCALD), we develop a Bayesian treatment to composite quantile regression for mixed-effects models. Furthermore, with the location-scale mixture representation of the PCALD, we establish a Bayesian hierarchical model and achieve the posterior inference of all unknown parameters and latent variables using Markov Chain Monte Carlo (MCMC) method. Finally, this newly developed procedure is illustrated by some Monte Carlo simulations and a case analysis of HIV/AIDS clinical data set.     

Detecting homogenous predictors in high-dimensional panel model with an MCMC algorithm

In panel data analysis, predictors may impact response in substantially different manner. Some predictors are in homogenous effects across all individuals, while the others are in heterogenous way. How to effectively differentiate these two kinds of predictors is crucial, particularly in high-dimensional panel data, since the number of parameters should be greatly reduced and hence lead to better interpretability by homogenous assumption. In this article, based on a hierarchical Bayesian panel regression model, we propose a novel yet effective Markov chain Monte Carlo (MCMC) algorithm together with a simple maximum ratio criterion to detect the predictors in homogenous effects in high-dimensional panel data. Extensive Monte Carlo simulations show that this MCMC algorithm performs well. The usefulness of the proposed method is further demonstrated by a real example from China financial market.     

Bayesian Additive Machine: classification with a semiparametric discriminant function

In this paper, we propose a new Bayesian inference approach for classification based on the traditional hinge loss used for classical support vector machines, which we call the Bayesian Additive Machine (BAM). Unlike existing approaches, the new model has a semiparametric discriminant function where some feature effects are nonlinear and others are linear. This separation of features is achieved automatically during model fitting without user pre-specification. Following the literature on sparse regression of high-dimensional models, we can also identify the irrelevant features. By introducing spike-and-slab priors using two sets of indicator variables, these multiple goals are achieved simultaneously and automatically, without any parameter tuning such as cross-validation. An efficient partially collapsed Markov chain Monte Carlo algorithm is developed for posterior exploration based on a data augmentation scheme for the hinge loss. Our simulations and three real data examples demonstrate that the new approach is a strong competitor to some approaches that were proposed recently for dealing with challenging classification examples with high dimensionality.     

Re-examining informative prior elicitation through the lens of Markov chain Monte Carlo methods

Summary.  In recent years, advances in Markov chain Monte Carlo techniques have had a major influence on the practice of Bayesian statistics. An interesting but hitherto largely underexplored corollary of this fact is that Markov chain Monte Carlo techniques make it practical to consider broader classes of informative priors than have been used previously. Conjugate priors, long the workhorse of classic methods for eliciting informative priors, have their roots in a time when modern computational methods were unavailable. In the current environment more attractive alternatives are practicable. A reappraisal of these classic approaches is undertaken, and principles for generating modern elicitation methods are described. A new prior elicitation methodology in accord with these principles is then presented.     

Inference for Weibull distribution under adaptive Type-I progressive hybrid censored competing risks data

In this paper, a competing risks model is considered under adaptive type-I progressive hybrid censoring scheme (AT-I PHCS). The lifetimes of the latent failure times have Weibull distributions with the same shape parameter. We investigate the maximum likelihood estimation of the parameters. Bayes estimates of the parameters are obtained based on squared error and LINEX loss functions under the assumption of independent gamma priors. We propose to apply Markov Chain Monte Carlo (MCMC) techniques to carry out a Bayesian estimation procedure and in turn calculate the credible intervals. To evaluate the performance of the estimators, a simulation study is carried out.     

Model selection in finite mixture of regression models: a Bayesian approach with innovative weighted g priors and reversible jump Markov chain Monte Carlo implementation

Finite mixture of regression (FMR) models are aimed at characterizing subpopulation heterogeneity stemming from different sets of covariates that impact different groups in a population. We address the contemporary problem of simultaneously conducting covariate selection and determining the number of mixture components from a Bayesian perspective that can incorporate prior information. We propose a Gibbs sampling algorithm with reversible jump Markov chain Monte Carlo implementation to accomplish concurrent covariate selection and mixture component determination in FMR models. Our Bayesian approach contains innovative features compared to previously developed reversible jump algorithms. In addition, we introduce component-adaptive weighted g priors for regression coefficients, and illustrate their improved performance in covariate selection. Numerical studies show that the Gibbs sampler with reversible jump implementation performs well, and that the proposed weighted priors can be superior to non-adaptive unweighted priors.     

Adaptive evolutionary Monte Carlo algorithm for optimization with applications to sensor placement problems

In this paper, we present an adaptive evolutionary Monte Carlo algorithm (AEMC), which combines a tree-based predictive model with an evolutionary Monte Carlo sampling procedure for the purpose of global optimization. Our development is motivated by sensor placement applications in engineering, which requires optimizing certain complicated “black-box” objective function. The proposed method is able to enhance the optimization efficiency and effectiveness as compared to a few alternative strategies. AEMC falls into the category of adaptive Markov chain Monte Carlo (MCMC) algorithms and is the first adaptive MCMC algorithm that simulates multiple Markov chains in parallel. A theorem about the ergodicity property of the AEMC algorithm is stated and proven. We demonstrate the advantages of the proposed method by applying it to a sensor placement problem in a manufacturing process, as well as to a standard Griewank test function.     

A flexible statistical and efficient computational approach to object location applied to electrical tomography

In electrical tomography, multiple measurements of voltage are taken between electrodes on the boundary of a region with the aim of investigating the electrical conductivity distribution within the region. The relationship between conductivity and voltage is governed by an elliptic partial differential equation derived from Maxwell’s equations. Recent statistical approaches, combining Bayesian methods with Markov chain Monte Carlo (MCMC) algorithms, allow to greater flexibility than classical inverse solution approaches and require only the calculation of voltages from a conductivity distribution. However, solution of this forward problem still requires the use of the Finite Difference Method (FDM) or the Finite Element Method (FEM) and many thousands of forward solutions are needed which strains practical feasibility. Many tomographic applications involve locating the perimeter of some homogeneous conductivity objects embedded in a homogeneous background. It is possible to exploit this type of structure using the Boundary Element Method (BEM) to provide a computationally efficient alternative forward solution technique. A geometric model is then used to define the region boundary, with priors on boundary smoothness and on the range of feasible conductivity values. This paper investigates the use of a BEM/MCMC approach for electrical resistance tomography (ERT) data. The efficiency of the boundary-element method coupled with the flexibility of the MCMC technique gives a promising new approach to object identification in electrical tomography. Simulated ERT data are used to illustrate the procedures.     

Regenerative Markov Chain Importance Sampling

We introduce Markov Chain Importance Sampling (MCIS), which combines importance sampling (IS) and Markov Chain Monte Carlo (MCMC) to estimate some characteristics of a non-normalized multi-dimensional distribution. Especially, we introduce some importance functions whose variates are regeneratively generated by MCMC; these variates then are used to estimate the quantity of interest through IS. Because MCIS is regenerative, it overcomes the burn-in problem associated with MCMC. It could also speed up the mixing rate in MCMC.     

Geodesic Lagrangian Monte Carlo over the space of positive definite matrices: with application to Bayesian spectral density estimation

We present geodesic Lagrangian Monte Carlo, an extension of Hamiltonian Monte Carlo for sampling from posterior distributions defined on general Riemannian manifolds. We apply this new algorithm to Bayesian inference on symmetric or Hermitian positive definite (PD) matrices. To do so, we exploit the Riemannian structure induced by Cartan's canonical metric. The geodesics that correspond to this metric are available in closed-form and – within the context of Lagrangian Monte Carlo – provide a principled way to travel around the space of PD matrices. Our method improves Bayesian inference on such matrices by allowing for a broad range of priors, so we are not limited to conjugate priors only. In the context of spectral density estimation, we use the (non-conjugate) complex reference prior as an example modelling option made available by the algorithm. Results based on simulated and real-world multivariate time series are presented in this context, and future directions are outlined.