Fully Bayesian logistic regression with hyper-LASSO priors for high-dimensional feature selection

Feature selection arises in many areas of modern science. For example, in genomic research, we want to find the genes that can be used to separate tissues of different classes (e.g. cancer and normal). One approach is to fit regression/classification models with certain penalization. In the past decade, hyper-LASSO penalization (priors) have received increasing attention in the literature. However, fully Bayesian methods that use Markov chain Monte Carlo (MCMC) for regression/classification with hyper-LASSO priors are still in lack of development. In this paper, we introduce an MCMC method for learning multinomial logistic regression with hyper-LASSO priors. Our MCMC algorithm uses Hamiltonian Monte Carlo in a restricted Gibbs sampling framework. We have used simulation studies and real data to demonstrate the superior performance of hyper-LASSO priors compared to LASSO, and to investigate the issues of choosing heaviness and scale of hyper-LASSO priors.     

Sequential Monte Carlo methods for mixtures with normalized random measures with independent increments priors

Normalized random measures with independent increments are a general, tractable class of nonparametric prior. This paper describes sequential Monte Carlo methods for both conjugate and non-conjugate nonparametric mixture models with these priors. A simulation study is used to compare the efficiency of the different algorithms for density estimation and comparisons made with Markov chain Monte Carlo methods. The SMC methods are further illustrated by applications to dynamically fitting a nonparametric stochastic volatility model and to estimation of the marginal likelihood in a goodness-of-fit testing example.     

Approximate exchangeability and de Finetti priors in 2022

This is a review paper, beginning with de Finetti's work on partial exchangeability, continuing with his approach to approximate exchangeability, and then his (surprising) approach to assigning informative priors in nonstandard situations. Recent progress on Markov chain Monte Carlo methods for drawing conclusions is supplemented by a review of work by Gerencsér and Ottolini on getting honest bounds for rates of convergence. The paper concludes with a speculative approach to combining classical asymptotics with Monte Carlo. This promises real speed-ups and makes a nice example of how theory and computation can interact.     

Robust Bayesian analysis with partially exchangeable priors

In this paper, we consider a general Bayesian model which allows multiple grouping of parameters, where the components within a subgroup are exchangeable. The general idea is then illustrated for the normal means estimation problem under priors which are scale mixture of normals. We discuss also implementation of the Bayes procedure via Markov chain Monte Carlo integration techniques. We illustrate the proposed methods with a numerical example.     

On Bayesian nonparametric modelling of two correlated distributions

In this paper, we consider the problem of modelling a pair of related distributions using Bayesian nonparametric methods. A representation of the distributions as weighted sums of distributions is derived through normalisation. This allows us to define several classes of nonparametric priors. The properties of these distributions are explored and efficient Markov chain Monte Carlo methods are developed. The methodology is illustrated on simulated data and an example concerning hospital efficiency measurement.     

Bayesian Survival Analysis Using Bernstein Polynomials

Abstract.  Bayesian survival analysis of right-censored survival data is studied using priors on Bernstein polynomials and Markov chain Monte Carlo methods. These priors easily take into consideration geometric information like convexity or initial guess on the cumulative hazard functions, select only smooth functions, can have large enough support, and can be easily specified and generated. Certain frequentist asymptotic properties of the posterior distribution are established. Simulation studies indicate that these Bayes methods are quite satisfactory.     

An objective Bayesian analysis of the two-parameter half-logistic distribution based on progressively type-II censored samples

This paper develops an objective Bayesian analysis method for estimating unknown parameters of the half-logistic distribution when a sample is available from the progressively Type-II censoring scheme. Noninformative priors such as Jeffreys and reference priors are derived. In addition, derived priors are checked to determine whether they satisfy probability-matching criteria. The Metropolis–Hasting algorithm is applied to generate Markov chain Monte Carlo samples from these posterior density functions because marginal posterior density functions of each parameter cannot be expressed in an explicit form. Monte Carlo simulations are conducted to investigate frequentist properties of estimated models under noninformative priors. For illustration purposes, a real data set is presented, and the quality of models under noninformative priors is evaluated through posterior predictive checking.     

Bayesian optimal sequential design for nonparametric regression via inhomogeneous evolutionary MCMC

We develop a novel computational methodology for Bayesian optimal sequential design for nonparametric regression. This computational methodology, that we call inhomogeneous evolutionary Markov chain Monte Carlo, combines ideas of simulated annealing, genetic or evolutionary algorithms, and Markov chain Monte Carlo. Our framework allows optimality criteria with general utility functions and general classes of priors for the underlying regression function. We illustrate the usefulness of our novel methodology with applications to experimental design for nonparametric function estimation using Gaussian process priors and free-knot cubic splines priors.     

BAYESIAN DETECTION AND ANALYSIS OF CHANGING TRANSITION MATRICES OF STATIONARY MARKOV CHAINS

This paper considers the detection of abrupt changes in the transition matrix of a Markov chain from a Bayesian viewpoint. It derives Bayes factors and posterior probabilities for unknown numbers of change‐points, as well as the positions of the change‐points, assuming non‐informative but proper priors on the parameters and fixed upper bound. The Markov chain Monte Carlo approach proposed by Chib in 1998 for estimating multiple change‐points models is adapted for the Markov chain model. It is especially useful when there are many possible change‐points. The method can be applied in a wide variety of disciplines and is particularly relevant in the social and behavioural sciences, for analysing the effects of events on the attitudes of people.     

Maximizing generalized linear mixed model likelihoods with an automated Monte Carlo EM algorithm

Two new implementations of the EM algorithm are proposed for maximum likelihood fitting of generalized linear mixed models. Both methods use random (independent and identically distributed) sampling to construct Monte Carlo approximations at the E-step. One approach involves generating random samples from the exact conditional distribution of the random effects (given the data) by rejection sampling, using the marginal distribution as a candidate. The second method uses a multivariate t importance sampling approximation. In many applications the two methods are complementary. Rejection sampling is more efficient when sample sizes are small, whereas importance sampling is better with larger sample sizes. Monte Carlo approximation using random samples allows the Monte Carlo error at each iteration to be assessed by using standard central limit theory combined with Taylor series methods. Specifically, we construct a sandwich variance estimate for the maximizer at each approximate E-step. This suggests a rule for automatically increasing the Monte Carlo sample size after iterations in which the true EM step is swamped by Monte Carlo error. In contrast, techniques for assessing Monte Carlo error have not been developed for use with alternative implementations of Monte Carlo EM algorithms utilizing Markov chain Monte Carlo E-step approximations. Three different data sets, including the infamous salamander data of McCullagh and Nelder, are used to illustrate the techniques and to compare them with the alternatives. The results show that the methods proposed can be considerably more efficient than those based on Markov chain Monte Carlo algorithms. However, the methods proposed may break down when the intractable integrals in the likelihood function are of high dimension.     

Estimation of P(Y < X) for progressively first-failure-censored generalized inverted exponential distribution

In this article, we consider the problem of estimation of the stress–strength parameter δ?=?P(Y?<?X) based on progressively first-failure-censored samples, when X and Y both follow two-parameter generalized inverted exponential distribution with different and unknown shape and scale parameters. The maximum likelihood estimator of δ and its asymptotic confidence interval based on observed Fisher information are constructed. Two parametric bootstrap boot-p and boot-t confidence intervals are proposed. We also apply Markov Chain Monte Carlo techniques to carry out Bayes estimation procedures. Bayes estimate under squared error loss function and the HPD credible interval of δ are obtained using informative and non-informative priors. A Monte Carlo simulation study is carried out for comparing the proposed methods of estimation. Finally, the methods developed are illustrated with a couple of real data examples.     

Stochastic approximation Monte Carlo importance sampling for approximating exact conditional probabilities

Importance sampling and Markov chain Monte Carlo methods have been used in exact inference for contingency tables for a long time, however, their performances are not always very satisfactory. In this paper, we propose a stochastic approximation Monte Carlo importance sampling (SAMCIS) method for tackling this problem. SAMCIS is a combination of adaptive Markov chain Monte Carlo and importance sampling, which employs the stochastic approximation Monte Carlo algorithm (Liang et al., J. Am. Stat. Assoc., 102(477):305–320, 2007) to draw samples from an enlarged reference set with a known Markov basis. Compared to the existing importance sampling and Markov chain Monte Carlo methods, SAMCIS has a few advantages, such as fast convergence, ergodicity, and the ability to achieve a desired proportion of valid tables. The numerical results indicate that SAMCIS can outperform the existing importance sampling and Markov chain Monte Carlo methods: It can produce much more accurate estimates in much shorter CPU time than the existing methods, especially for the tables with high degrees of freedom.     

A Bayesian approach for non-homogeneous gamma degradation processes

A Bayesian approach based on the Markov Chain Monte Carlo technique is proposed for the non-homogeneous gamma process with power-law shape function. Vague and informative priors, formalized on some quantities having a “physical” meaning, are provided. Point and interval estimation of process parameters and some functions thereof are developed, as well as prediction on some observable quantities that are useful in defining the maintenance strategy is proposed. Some useful approximations are derived for the conditional and unconditional mean and median of the residual life to reduce computational time. Finally, the proposed approach is applied to a real dataset.     

Monte Carlo integration with Markov chain

There are two conceptually distinct tasks in Markov chain Monte Carlo (MCMC): a sampler is designed for simulating a Markov chain and then an estimator is constructed on the Markov chain for computing integrals and expectations. In this article, we aim to address the second task by extending the likelihood approach of Kong et al. for Monte Carlo integration. We consider a general Markov chain scheme and use partial likelihood for estimation. Basically, the Markov chain scheme is treated as a random design and a stratified estimator is defined for the baseline measure. Further, we propose useful techniques including subsampling, regulation, and amplification for achieving overall computational efficiency. Finally, we introduce approximate variance estimators for the point estimators. The method can yield substantially improved accuracy compared with Chib's estimator and the crude Monte Carlo estimator, as illustrated with three examples.     

Reference priors for linear models with general covariance structures

We develop a new class of reference priors for linear models with general covariance structures. A general Markov chain Monte Carlo algorithm is also proposed for implementing the computation. We present several examples to demonstrate the results: Bayesian penalized spline smoothing, a Bayesian approach to bivariate smoothing for a spatial model, and prior specification for structural equation models.     

Stochastic approximation Monte Carlo EM for change-point analysis

In the expectation–maximization (EM) algorithm for maximum likelihood estimation from incomplete data, Markov chain Monte Carlo (MCMC) methods have been used in change-point inference for a long time when the expectation step is intractable. However, the conventional MCMC algorithms tend to get trapped in local mode in simulating from the posterior distribution of change points. To overcome this problem, in this paper we propose a stochastic approximation Monte Carlo version of EM (SAMCEM), which is a combination of adaptive Markov chain Monte Carlo and EM utilizing a maximum likelihood method. SAMCEM is compared with the stochastic approximation version of EM and reversible jump Markov chain Monte Carlo version of EM on simulated and real datasets. The numerical results indicate that SAMCEM can outperform among the three methods by producing much more accurate parameter estimates and the ability to achieve change-point positions and estimates simultaneously.     

Markov Random Fields with Higher-order Interactions

Discrete-state Markov random fields on regular arrays have played a significant role in spatial statistics and image analysis. For example, they are used to represent objects against background in computer vision and pixel-based classification of a region into different crop types in remote sensing. Convenience has generally favoured formulations that involve only pairwise interactions. Such models are in themselves unrealistic and, although they often perform surprisingly well in tasks such as the restoration of degraded images, they are unsatisfactory for many other purposes. In this paper, we consider particular forms of Markov random fields that involve higher-order interactions and therefore are better able to represent the large-scale properties of typical spatial scenes. Interpretations of the parameters are given and realizations from a variety of models are produced via Markov chain Monte Carlo. Potential applications are illustrated in two examples. The first concerns Bayesian image analysis and confirms that pairwise-interaction priors may perform very poorly for image functionals such as number of objects, even when restoration apparently works well. The second example describes a model for a geological dataset and obtains maximum-likelihood parameter estimates using Markov chain Monte Carlo. Despite the complexity of the formulation, realizations of the estimated model suggest that the representation is quite realistic.     

On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion)

New methodology for fully Bayesian mixture analysis is developed, making use of reversible jump Markov chain Monte Carlo methods that are capable of jumping between the parameter subspaces corresponding to different numbers of components in the mixture. A sample from the full joint distribution of all unknown variables is thereby generated, and this can be used as a basis for a thorough presentation of many aspects of the posterior distribution. The methodology is applied here to the analysis of univariate normal mixtures, using a hierarchical prior model that offers an approach to dealing with weak prior information while avoiding the mathematical pitfalls of using improper priors in the mixture context.     

Markov chain Monte Carlo algorithms with sequential proposals

We explore a general framework in Markov chain Monte Carlo (MCMC) sampling where sequential proposals are tried as a candidate for the next state of the Markov chain. This sequential-proposal framework can be applied to various existing MCMC methods, including Metropolis–Hastings algorithms using random proposals and methods that use deterministic proposals such as Hamiltonian Monte Carlo (HMC) or the bouncy particle sampler. Sequential-proposal MCMC methods construct the same Markov chains as those constructed by the delayed rejection method under certain circumstances. In the context of HMC, the sequential-proposal approach has been proposed as extra chance generalized hybrid Monte Carlo (XCGHMC). We develop two novel methods in which the trajectories leading to proposals in HMC are automatically tuned to avoid doubling back, as in the No-U-Turn sampler (NUTS). The numerical efficiency of these new methods compare favorably to the NUTS. We additionally show that the sequential-proposal bouncy particle sampler enables the constructed Markov chain to pass through regions of low target density and thus facilitates better mixing of the chain when the target density is multimodal.

     

Bayesian phylogenetic inference from animal mitochondrial genome arrangements

Summary. The determination of evolutionary relationships is a fundamental problem in evolutionary biology. Genome arrangement data are potentially more informative than deoxyribonucleic acid sequence data for inferring evolutionary relationships between distantly related taxa. We describe a Bayesian framework for phylogenetic inference from mitochondrial genome arrangement data using Markov chain Monte Carlo methods. We apply the method to assess evolutionary relationships between eight animal phyla.