A comparison of Bayesian model selection based on MCMC with an application to GARCH-type models

This paper presents a comprehensive review and comparison of five computational methods for Bayesian model selection, based on MCMC simulations from posterior model parameter distributions. We apply these methods to a well-known and important class of models in financial time series analysis, namely GARCH and GARCH-t models for conditional return distributions (assuming normal and t-distributions). We compare their performance with the more common maximum likelihood-based model selection for simulated and real market data. All five MCMC methods proved reliable in the simulation study, although differing in their computational demands. Results on simulated data also show that for large degrees of freedom (where the t-distribution becomes more similar to a normal one), Bayesian model selection results in better decisions in favor of the true model than maximum likelihood. Results on market data show the instability of the harmonic mean estimator and reliability of the advanced model selection methods.     

An efficient Monte Carlo EM algorithm for Bayesian lasso

The lasso is a popular technique of simultaneous estimation and variable selection in many research areas. The marginal posterior mode of the regression coefficients is equivalent to estimates given by the non-Bayesian lasso when the regression coefficients have independent Laplace priors. Because of its flexibility of statistical inferences, the Bayesian approach is attracting a growing body of research in recent years. Current approaches are primarily to either do a fully Bayesian analysis using Markov chain Monte Carlo (MCMC) algorithm or use Monte Carlo expectation maximization (MCEM) methods with an MCMC algorithm in each E-step. However, MCMC-based Bayesian method has much computational burden and slow convergence. Tan et al. [An efficient MCEM algorithm for fitting generalized linear mixed models for correlated binary data. J Stat Comput Simul. 2007;77:929–943] proposed a non-iterative sampling approach, the inverse Bayes formula (IBF) sampler, for computing posteriors of a hierarchical model in the structure of MCEM. Motivated by their paper, we develop this IBF sampler in the structure of MCEM to give the marginal posterior mode of the regression coefficients for the Bayesian lasso, by adjusting the weights of importance sampling, when the full conditional distribution is not explicit. Simulation experiments show that the computational time is much reduced with our method based on the expectation maximization algorithm and our algorithms and our methods behave comparably with other Bayesian lasso methods not only in prediction accuracy but also in variable selection accuracy and even better especially when the sample size is relatively large.     

基于MCMC模拟和伪似然估计法的交叉分类信度模型费率厘定简

针对传统交叉分类信度模型计算复杂且在结构参数先验信息不足的情况下不能得到参数无偏后验估计的问题，利用MCMC模拟和GLMM方法，对交叉分类信度模型进行实证分析证明模型的有效性。结果表明：基于MCMC方法能够动态模拟参数的后验分布，并可提高模型估计的精度；基于GLMM能大大简化计算过程且操作方便，可利用图形和其它诊断工具选择模型，并对模型实用性做出评价。     

Bayesian quantile regression for longitudinal data models

In this paper, we discuss a fully Bayesian quantile inference using Markov Chain Monte Carlo (MCMC) method for longitudinal data models with random effects. Under the assumption of error term subject to asymmetric Laplace distribution, we establish a hierarchical Bayesian model and obtain the posterior distribution of unknown parameters at τ-th level. We overcome the current computational limitations using two approaches. One is the general MCMC technique with Metropolis–Hastings algorithm and another is the Gibbs sampling from the full conditional distribution. These two methods outperform the traditional frequentist methods under a wide array of simulated data models and are flexible enough to easily accommodate changes in the number of random effects and in their assumed distribution. We apply the Gibbs sampling method to analyse a mouse growth data and some different conclusions from those in the literatures are obtained.     

Model weights for model choice and averaging

A method is suggested to estimate posterior model probabilities and model averaged parameters via MCMC sampling under a Bayesian approach. The estimates use pooled output for J models (J>1) whereby all models are updated at each iteration. Posterior probabilities are based on averages of continuous weights obtained for each model at each iteration, while samples of averaged parameters are obtained from iteration specific averages that are based on these weights. Parallel sampling of models assists in deriving posterior densities for parameter contrasts between models and in assessing hypotheses regarding model averaged parameters. Four worked examples illustrate application of the approach, two involving fixed effect regression, and two involving random effects.     

Markov chain Monte Carlo with the Integrated Nested Laplace Approximation

The Integrated Nested Laplace Approximation (INLA) has established itself as a widely used method for approximate inference on Bayesian hierarchical models which can be represented as a latent Gaussian model (LGM). INLA is based on producing an accurate approximation to the posterior marginal distributions of the parameters in the model and some other quantities of interest by using repeated approximations to intermediate distributions and integrals that appear in the computation of the posterior marginals. INLA focuses on models whose latent effects are a Gaussian Markov random field. For this reason, we have explored alternative ways of expanding the number of possible models that can be fitted using the INLA methodology. In this paper, we present a novel approach that combines INLA and Markov chain Monte Carlo (MCMC). The aim is to consider a wider range of models that can be fitted with INLA only when some of the parameters of the model have been fixed. We show how new values of these parameters can be drawn from their posterior by using conditional models fitted with INLA and standard MCMC algorithms, such as Metropolis–Hastings. Hence, this will extend the use of INLA to fit models that can be expressed as a conditional LGM. Also, this new approach can be used to build simpler MCMC samplers for complex models as it allows sampling only on a limited number of parameters in the model. We will demonstrate how our approach can extend the class of models that could benefit from INLA, and how the R-INLA package will ease its implementation. We will go through simple examples of this new approach before we discuss more advanced applications with datasets taken from the relevant literature. In particular, INLA within MCMC will be used to fit models with Laplace priors in a Bayesian Lasso model, imputation of missing covariates in linear models, fitting spatial econometrics models with complex nonlinear terms in the linear predictor and classification of data with mixture models. Furthermore, in some of the examples we could exploit INLA within MCMC to make joint inference on an ensemble of model parameters.     

Data reconciliation of nonnormal observations with nonlinear constraints

This paper presents a new method for the reconciliation of data described by arbitrary continuous probability distributions, with the focus on nonlinear constraints. The main idea, already applied to linear constraints in a previous paper, is to restrict the joint prior probability distribution of the observed variables with model constraints to get a joint posterior probability distribution. Because in general the posterior probability density function cannot be calculated analytically, it is shown that it has decisive advantages to sample from the posterior distribution by a Markov chain Monte Carlo (MCMC) method. From the resulting sample of observed and unobserved variables various characteristics of the posterior distribution can be estimated, such as the mean, the full covariance matrix, marginal posterior densities, as well as marginal moments, quantiles, and HPD intervals. The procedure is illustrated by examples from material flow analysis and chemical engineering.     

On double hysteretic heteroskedastic model

ABSTRACT

This paper proposes a hysteretic autoregressive model with GARCH specification and a skew Student's t-error distribution for financial time series. With an integrated hysteresis zone, this model allows both the conditional mean and conditional volatility switching in a regime to be delayed when the hysteresis variable lies in a hysteresis zone. We perform Bayesian estimation via an adaptive Markov Chain Monte Carlo sampling scheme. The proposed Bayesian method allows simultaneous inferences for all unknown parameters, including threshold values and a delay parameter. To implement model selection, we propose a numerical approximation of the marginal likelihoods to posterior odds. The proposed methodology is illustrated using simulation studies and two major Asia stock basis series. We conduct a model comparison for variant hysteresis and threshold GARCH models based on the posterior odds ratios, finding strong evidence of the hysteretic effect and some asymmetric heavy-tailness. Versus multi-regime threshold GARCH models, this new collection of models is more suitable to describe real data sets. Finally, we employ Bayesian forecasting methods in a Value-at-Risk study of the return series.     

Quantifying uncertainty in transdimensional Markov chain Monte Carlo using discrete Markov models

Bayesian analysis often concerns an evaluation of models with different dimensionality as is necessary in, for example, model selection or mixture models. To facilitate this evaluation, transdimensional Markov chain Monte Carlo (MCMC) relies on sampling a discrete indexing variable to estimate the posterior model probabilities. However, little attention has been paid to the precision of these estimates. If only few switches occur between the models in the transdimensional MCMC output, precision may be low and assessment based on the assumption of independent samples misleading. Here, we propose a new method to estimate the precision based on the observed transition matrix of the model-indexing variable. Assuming a first-order Markov model, the method samples from the posterior of the stationary distribution. This allows assessment of the uncertainty in the estimated posterior model probabilities, model ranks, and Bayes factors. Moreover, the method provides an estimate for the effective sample size of the MCMC output. In two model selection examples, we show that the proposed approach provides a good assessment of the uncertainty associated with the estimated posterior model probabilities.

     

Regularized Bayesian quantile regression

A number of nonstationary models have been developed to estimate extreme events as function of covariates. A quantile regression (QR) model is a statistical approach intended to estimate and conduct inference about the conditional quantile functions. In this article, we focus on the simultaneous variable selection and parameter estimation through penalized quantile regression. We conducted a comparison of regularized Quantile Regression model with B-Splines in Bayesian framework. Regularization is based on penalty and aims to favor parsimonious model, especially in the case of large dimension space. The prior distributions related to the penalties are detailed. Five penalties (Lasso, Ridge, SCAD0, SCAD1 and SCAD2) are considered with their equivalent expressions in Bayesian framework. The regularized quantile estimates are then compared to the maximum likelihood estimates with respect to the sample size. A Markov Chain Monte Carlo (MCMC) algorithms are developed for each hierarchical model to simulate the conditional posterior distribution of the quantiles. Results indicate that the SCAD0 and Lasso have the best performance for quantile estimation according to Relative Mean Biais (RMB) and the Relative Mean-Error (RME) criteria, especially in the case of heavy distributed errors. A case study of the annual maximum precipitation at Charlo, Eastern Canada, with the Pacific North Atlantic climate index as covariate is presented.     

空间计量模型选择及其模拟分析

空间计量模型的选择是空间计量建模的一个重要组成部分,也是空间计量模型实证分析的关键步骤。本文对空间计量模型选择中的Moran指数检验、LM检验、似然函数、三大信息准则、贝叶斯后验概率、马尔可夫链蒙特卡罗方法做了详细的理论分析。并在此基础之上,通过Matlab编程进行模拟分析,结果表明：在扩充的空间计量模型族中进行模型选择时,基于OLS残差的Moran指数与LM检验均存在较大的局限性,对数似然值最大原则缺少区分度,LM检验只针对SEM和SAR模型的区分有效,信息准则对大多数模型有效,但是也会出现误选。而当给出恰当的M-H算法时,充分利用了似然函数和先验信息的MCMC方法,具有更高的检验效度,特别是在较大的样本条件下得到了完全准确的判断,且对不同阶空间邻接矩阵的空间计量模型的选择也非常有效。     

A Simple Analysis of the Exact Probability Matching Prior in the Location-Scale Model

It has long been asserted that in univariate location-scale models, when concerned with inference for either the location or scale parameter, the use of the inverse of the scale parameter as a Bayesian prior yields posterior credible sets that have exactly the correct frequentist confidence set interpretation. This claim dates to at least Peers, and has subsequently been noted by various authors, with varying degrees of justification. We present a simple, direct demonstration of the exact matching property of the posterior credible sets derived under use of this prior in the univariate location-scale model. This is done by establishing an equivalence between the conditional frequentist and posterior densities of the pivotal quantities on which conditional frequentist inferences are based.     

Markov chain Monte Carlo estimation of a mixture item response theory model

Markov chain Monte Carlo (MCMC) algorithms have been shown to be useful for estimation of complex item response theory (IRT) models. Although an MCMC algorithm can be very useful, it also requires care in use and interpretation of results. In particular, MCMC algorithms generally make extensive use of priors on model parameters. In this paper, MCMC estimation is illustrated using a simple mixture IRT model, a mixture Rasch model (MRM), to demonstrate how the algorithm operates and how results may be affected by some commonly used priors. Priors on the probabilities of mixtures, label switching, model selection, metric anchoring, and implementation of the MCMC algorithm using WinBUGS are described, and their effects illustrated on parameter recovery in practical testing situations. In addition, an example is presented in which an MRM is fitted to a set of educational test data using the MCMC algorithm and a comparison is illustrated with results from three existing maximum likelihood estimation methods.     

空间滞后模型的贝叶斯估计

本文采用Bayes方法对空间滞后模型进行全面分析。在构建模型的贝叶斯框架时,对模型系数与误差方差分别选取正态先验分布和逆伽玛先验分布,这样以便获得参数的联合后验分布和条件后验分布。在抽样估计时,文章主要使用MCMC方法,同时还设计了一个简单随机游动Metropolis抽样器,以方便从空间权重因子系数的条件后验分布中进行抽样。最后应用所建议的方法进行数值模拟。     

Collapsing of Non‐centred Parameterized MCMC Algorithms with Applications to Epidemic Models

Data augmentation is required for the implementation of many Markov chain Monte Carlo (MCMC) algorithms. The inclusion of augmented data can often lead to conditional distributions from well‐known probability distributions for some of the parameters in the model. In such cases, collapsing (integrating out parameters) has been shown to improve the performance of MCMC algorithms. We show how integrating out the infection rate parameter in epidemic models leads to efficient MCMC algorithms for two very different epidemic scenarios, final outcome data from a multitype SIR epidemic and longitudinal data from a spatial SI epidemic. The resulting MCMC algorithms give fresh insight into real‐life epidemic data sets.     

Estimation Under Inequality Constraints: Semiparametric Estimation of Conditional Duration Models

This article proposes a semiparametric estimator of the parameter in a conditional duration model when there are inequality constraints on some parameters and the error distribution may be unknown. We propose to estimate the parameter by a constrained version of an unrestricted semiparametrically efficient estimator. The main requirement for applying this method is that the initial unrestricted estimator converges in distribution. Apart from this, additional regularity conditions on the data generating process or the likelihood function, are not required. Hence the method is applicable to a broad range of models where the parameter space is constrained by inequality constraints, such as the conditional duration models. In a simulation study involving conditional duration models, the overall performance of the constrained estimator was better than its competitors, in terms of mean squared error. A data example is used to illustrate the method.     

A Bayesian multiple comparison procedure for simply ordered treatment medians

This article considers a Bayesian hierarchical model for multiple comparisons in linear models where the population medians satisfy a simple order restriction. Representing the asymmetric Laplace distribution as a scale mixture of normals with an exponential mixing density and a continuous prior restricted to order constraints, a Gibbs sampling algorithm for parameter estimation and simultaneous comparison of treatment medians is proposed. Posterior probabilities of all possible hypotheses on the equality/inequality of treatment medians are estimated using Bayes factors that are computed via the Savage-Dickey density ratios. The performance of the proposed median-based model is investigated in the simulated and real datasets. The results show that the proposed method can outperform the commonly used method that is based on treatment means, when data are from nonnormal distributions.     

Bayesian curve fitting and clustering with Dirichlet process mixture models for microarray data

In the field of molecular biology, it is often of interest to analyze microarray data for clustering genes based on similar profiles of gene expression to identify genes that are differentially expressed under multiple biological conditions. One of the notable characteristics of a gene expression profile is that it shows a cyclic curve over a course of time. To group sequences of similar molecular functions, we propose a Bayesian Dirichlet process mixture of linear regression models with a Fourier series for the regression coefficients, for each of which a spike and slab prior is assumed. A full Gibbs-sampling algorithm is developed for an efficient Markov chain Monte Carlo (MCMC) posterior computation. Due to the so-called “label-switching” problem and different numbers of clusters during the MCMC computation, a post-process approach of Fritsch and Ickstadt (2009) is additionally applied to MCMC samples for an optimal single clustering estimate by maximizing the posterior expected adjusted Rand index with the posterior probabilities of two observations being clustered together. The proposed method is illustrated with two simulated data and one real data of the physiological response of fibroblasts to serum of Iyer et al. (1999).     

Bayesian model selection and parameter estimation for possibly asymmetric and non-stationary time series using a reversible jump Markov chain Monte Carlo approach

A Markov chain Monte Carlo (MCMC) approach, called a reversible jump MCMC, is employed in model selection and parameter estimation for possibly non-stationary and non-linear time series data. The non-linear structure is modelled by the asymmetric momentum threshold autoregressive process (MTAR) of Enders & Granger (1998) or by the asymmetric self-exciting threshold autoregressive process (SETAR) of Tong (1990). The non-stationary and non-linear feature is represented by the MTAR (or SETAR) model in which one ( 𝜌 1 ) of the AR coefficients is greater than one, and the other ( 𝜌 2 ) is smaller than one. The other non-stationary and linear, stationary and nonlinear, and stationary and linear features, represented respectively by ( 𝜌 1 = 𝜌 2 = 1 ), ( 𝜌 1 p 𝜌 2 < 1 ) and ( 𝜌 1 = 𝜌 2 < 1 ), are also considered as possible models. The reversible jump MCMC provides estimates of posterior probabilities for these four different models as well as estimates of the AR coefficients 𝜌 1 and 𝜌 2 . The proposed method is illustrated by analysing six series of US interest rates in terms of model selection, parameter estimation, and forecasting.     

On Block Updating in Markov Random Field Models for Disease Mapping

Gaussian Markov random field (GMRF) models are commonly used to model spatial correlation in disease mapping applications. For Bayesian inference by MCMC, so far mainly single-site updating algorithms have been considered. However, convergence and mixing properties of such algorithms can be extremely poor due to strong dependencies of parameters in the posterior distribution. In this paper, we propose various block sampling algorithms in order to improve the MCMC performance. The methodology is rather general, allows for non-standard full conditionals, and can be applied in a modular fashion in a large number of different scenarios. For illustration we consider three different applications: two formulations for spatial modelling of a single disease (with and without additional unstructured parameters respectively), and one formulation for the joint analysis of two diseases. The results indicate that the largest benefits are obtained if parameters and the corresponding hyperparameter are updated jointly in one large block. Implementation of such block algorithms is relatively easy using methods for fast sampling of Gaussian Markov random fields ( Rue, 2001 ). By comparison, Monte Carlo estimates based on single-site updating can be rather misleading, even for very long runs. Our results may have wider relevance for efficient MCMC simulation in hierarchical models with Markov random field components.