Tree-structured prognostic classification for censored survival data: validation of computationally inexpensive model selection criteria

The performance of computationally inexpensive model selection criteria in the context of tree structured prediction is discussed. It is shown through a simulation study that no one model selection criterion exhibits a uniformly superior performance over a wide range of scenarios. Therefore, a two-stage approach for model selection is suggested and shown to perform satisfactorily. A computationally efficient method of tree-growing within the RECursive Partition and AMalgamation (RECPAM) framework is suggested. The computationally efficient algorithm gives identical results as the original RECPAM tree-growing algorithm. An example of medical data analysis for developing prognostic classification is presented.     

Particle filtering for partially observed Gaussian state space models

Summary. Solving Bayesian estimation problems where the posterior distribution evolves over time through the accumulation of data has many applications for dynamic models. A large number of algorithms based on particle filtering methods, also known as sequential Monte Carlo algorithms, have recently been proposed to solve these problems. We propose a special particle filtering method which uses random mixtures of normal distributions to represent the posterior distributions of partially observed Gaussian state space models. This algorithm is based on a marginalization idea for improving efficiency and can lead to substantial gains over standard algorithms. It differs from previous algorithms which were only applicable to conditionally linear Gaussian state space models. Computer simulations are carried out to evaluate the performance of the proposed algorithm for dynamic tobit and probit models.     

Parallel algorithms for Markov chain Monte Carlo methods in latent spatial Gaussian models

Markov chain Monte Carlo (MCMC) implementations of Bayesian inference for latent spatial Gaussian models are very computationally intensive, and restrictions on storage and computation time are limiting their application to large problems. Here we propose various parallel MCMC algorithms for such models. The algorithms' performance is discussed with respect to a simulation study, which demonstrates the increase in speed with which the algorithms explore the posterior distribution as a function of the number of processors. We also discuss how feasible problem size is increased by use of these algorithms.     

Sampling-based Inference of Time Deformation Models with Heavy Tail Distributions

This article focuses on simulation-based inference for the time-deformation models directed by a duration process. In order to better capture the heavy tail property of the time series of financial asset returns, the innovation of the observation equation is subsequently assumed to have a Student-t distribution. Suitable Markov chain Monte Carlo (MCMC) algorithms, which are hybrids of Gibbs and slice samplers, are proposed for estimation of the parameters of these models. In the algorithms, the parameters of the models can be sampled either directly from known distributions or through an efficient slice sampler. The states are simulated one at a time by using a Metropolis-Hastings method, where the proposal distributions are sampled through a slice sampler. Simulation studies conducted in this article suggest that our extended models and accompanying MCMC algorithms work well in terms of parameter estimation and volatility forecast.     

The penalized biclustering model and related algorithms

Biclustering is the simultaneous clustering of two related dimensions, for example, of individuals and features, or genes and experimental conditions. Very few statistical models for biclustering have been proposed in the literature. Instead, most of the research has focused on algorithms to find biclusters. The models underlying them have not received much attention. Hence, very little is known about the adequacy and limitations of the models and the efficiency of the algorithms. In this work, we shed light on associated statistical models behind the algorithms. This allows us to generalize most of the known popular biclustering techniques, and to justify, and many times improve on, the algorithms used to find the biclusters. It turns out that most of the known techniques have a hidden Bayesian flavor. Therefore, we adopt a Bayesian framework to model biclustering. We propose a measure of biclustering complexity (number of biclusters and overlapping) through a penalized plaid model, and present a suitable version of the deviance information criterion to choose the number of biclusters, a problem that has not been adequately addressed yet. Our ideas are motivated by the analysis of gene expression data.     

Maximum likelihood computation for fitting semiparametric mixture models

Three general algorithms that use different strategies are proposed for computing the maximum likelihood estimate of a semiparametric mixture model. They seek to maximize the likelihood function by, respectively, alternating the parameters, profiling the likelihood and modifying the support set. All three algorithms make a direct use of the recently proposed fast and stable constrained Newton method for computing the nonparametric maximum likelihood of a mixing distribution and employ additionally an optimization algorithm for unconstrained problems. The performance of the algorithms is numerically investigated and compared for solving the Neyman-Scott problem, overcoming overdispersion in logistic regression models and fitting two-level mixed effects logistic regression models. Satisfactory results have been obtained.     

A case study in non-centering for data augmentation: Stochastic epidemics

In this paper, we introduce non-centered and partially non-centered MCMC algorithms for stochastic epidemic models. Centered algorithms previously considered in the literature perform adequately well for small data sets. However, due to the high dependence inherent in the models between the missing data and the parameters, the performance of the centered algorithms gets appreciably worse when larger data sets are considered. Therefore non-centered and partially non-centered algorithms are introduced and are shown to out perform the existing centered algorithms.     

Orthogonal Stiefel manifold optimization for eigen-decomposed covariance parameter estimation in mixture models

Within the mixture model-based clustering literature, parsimonious models with eigen-decomposed component covariance matrices have dominated for over a decade. Although originally introduced as a fourteen-member family of models, the current state-of-the-art is to utilize just ten of these models; the rationale for not using the other four models usually centers around parameter estimation difficulties. Following close examination of these four models, we find that two are actually easily implemented using existing algorithms but that two benefit from a novel approach. We present and implement algorithms that use an accelerated line search for optimization on the orthogonal Stiefel manifold. Furthermore, we show that the ‘extra’ models that these decompositions facilitate outperform the current state-of-the art when applied to two benchmark data sets.     

Evaluating model misspecification in independent component analysis

Independent component analysis (ICA) is a popular blind source separation technique used in many scientific disciplines. Current ICA approaches have focused on developing efficient algorithms under specific ICA models, such as instantaneous or convolutive mixing conditions, intrinsically assuming temporal independence or autocorrelation of the sources. In practice, the true model is not known and different ICA algorithms can produce very different results. Although it is critical to choose an ICA model, there has not been enough research done on evaluating mixing models and assumptions, and how the associated algorithms may perform under different scenarios. In this paper, we investigate the performance of multiple ICA algorithms under various mixing conditions. We also propose a convolutive ICA algorithm for echoic mixing cases. Our simulation studies show that the performance of ICA algorithms is highly dependent on mixing conditions and temporal independence of the sources. Most instantaneous ICA algorithms fail to separate autocorrelated sources, while convolutive ICA algorithms depend highly on the model specification and approximation accuracy of unmixing filters.     

Algorithmic construction of nearly column-orthogonal designs

Computer models can describe complicated phenomena encountered in science and engineering fields. To use these models for scientific investigation, however, their generally long running time and mostly deterministic nature require a specially designed experiment. The column-orthogonal design (COD) is a popular choice for computer experiments. Because of the restriction on the orthogonality, however, only little CODs can be constructed. In this article, we propose two algorithms for constructing nearly CODs by rotating orthogonal arrays under two different criteria. Further, some obtained nearly CODs are nearly Latin hypercube designs. Some examples are provided to show the advantages of our algorithms. Some rotation matrices obtained via the algorithms are listed.     

Exact optimal experimental designs with constraints

The experimental design literature has produced a wide range of algorithms optimizing estimator variance for linear models where the design-space is finite or a convex polytope. But these methods have problems handling nonlinear constraints or constraints over multiple treatments. This paper presents Newton-type algorithms to compute exact optimal designs in models with continuous and/or discrete regressors, where the set of feasible treatments is defined by nonlinear constraints. We carry out numerical comparisons with other state-of-art methods to show the performance of this approach.     

Model‐based clustering of longitudinal data

A new family of mixture models for the model‐based clustering of longitudinal data is introduced. The covariance structures of eight members of this new family of models are given and the associated maximum likelihood estimates for the parameters are derived via expectation–maximization (EM) algorithms. The Bayesian information criterion is used for model selection and a convergence criterion based on the Aitken acceleration is used to determine the convergence of these EM algorithms. This new family of models is applied to yeast sporulation time course data, where the models give good clustering performance. Further constraints are then imposed on the decomposition to allow a deeper investigation of the correlation structure of the yeast data. These constraints greatly extend this new family of models, with the addition of many parsimonious models. The Canadian Journal of Statistics 38:153–168; 2010 © 2010 Statistical Society of Canada     

Bootstrap Confidence Intervals for Large-scale Multivariate Monotonic Regression Problems

Recently, the methods used to estimate monotonic regression (MR) models have been substantially improved, and some algorithms can now produce high-accuracy monotonic fits to multivariate datasets containing over a million observations. Nevertheless, the computational burden can be prohibitively large for resampling techniques in which numerous datasets are processed independently of each other. Here, we present efficient algorithms for estimation of confidence limits in large-scale settings that take into account the similarity of the bootstrap or jackknifed datasets to which MR models are fitted. In addition, we introduce modifications that substantially improve the accuracy of MR solutions for binary response variables. The performance of our algorithms is illustrated using data on death in coronary heart disease for a large population. This example also illustrates that MR can be a valuable complement to logistic regression.     

Random field models in image analysis

Image models are useful in quantitatively specifying natural constraints and general assumptions about the physical world and the imaging process. This review paper explains how Gibbs and Markov random field models provide a unifying theme for many contemporary problems in image analysis. Random field models permit the introduction of spatial context into pixel labeling problems, such as segmentation and restoration. Random field models also describe textured images and lead to algorithms for generating textured images, classifying textures, and segmenting textured images. In spite of some impressive model-based image restoration and texture segmentation results reported in the literature, a number of fundamental issues remain unexplored, such as the specification of MRF models, modeling noise processes, performance evaluation, parameter estimation, the phase transition phenomenon, and the comparative analysis of alternative procedures. The literature of random field models is filled with great promise, but a better mathematical understanding of these issues is needed as well as efficient algorithms for applications. These issues need to be resolved before random field models will be widely accepted as general tools in the image processing community.     

Random field models in image analysis*

Image models are useful in quantitatively specifying natural constraints and general assumptions about the physical world and the imaging process. This review paper explains how Gibbs and Markov random field models provide a unifying theme for many contemporary problems in image analysis. Random field models permit the introduction of spatial context into pixel labeling problems, such as segmentation and restoration. Random field models also describe textured images and lead to algorithms for generating textured images, classifying textures and segmenting textured images. In spite of some impressive model-based image restoration and texture segmentation results reported in the literature, a number of fundamental issues remain unexplored, such as the specification of MRF models, modeling noise processes, performance evaluation, parameter estimation, the phase transition phenomenon and the comparative analysis of alternative procedures. The literature of random field models is filled with great promise, but a better mathematical understanding of these issues is needed as well as efficient algorithms for applications. These issues need to be resolved before random field models will be widely accepted as general tools in the image-processing community.     

EM-Type algorithms for heavy-tailed logistic mixed models

This paper aims at evaluating different aspects of Monte Carlo expectation – maximization algorithm to estimate heavy-tailed mixed logistic regression (MLR) models. As a novelty it also proposes a multiple chain Gibbs sampler to generate of the latent variables distributions thus obtaining independent samples. In heavy-tailed MLR models, the analytical forms of the full conditional distributions for the random effects are unknown. Four different Metropolis–Hastings algorithms are assumed to generate from them. We also discuss stopping rules in order to obtain more efficient algorithms in heavy-tailed MLR models. The algorithms are compared through the analysis of simulated and Ascaris Suum data.     

Combinatorial EM algorithms

The complete-data model that underlies an Expectation-Maximization (EM) algorithm must have a parameter space that coincides with the parameter space of the observed-data model. Otherwise, maximization of the observed-data log-likelihood will be carried out over a space that does not coincide with the desired parameter space. In some contexts, however, a natural complete-data model may be defined only for parameter values within a subset of the observed-data parameter space. In this paper we discuss situations where this can still be useful if the complete-data model can be viewed as a member of a finite family of complete-data models that have parameter spaces which collectively cover the observed-data parameter space. Such a family of complete-data models defines a family of EM algorithms which together lead to a finite collection of constrained maxima of the observed-data log-likelihood. Maximization of the log-likelihood function over the full parameter space then involves identifying the constrained maximum that achieves the greatest log-likelihood value. Since optimization over a finite collection of candidates is referred to as combinatorial optimization, we refer to such a family of EM algorithms as a combinatorial EM (CEM) algorithm. As well as discussing the theoretical concepts behind CEM algorithms, we discuss strategies for improving the computational efficiency when the number of complete-data models is large. Various applications of CEM algorithms are also discussed, ranging from simple examples that illustrate the concepts, to more substantive examples that demonstrate the usefulness of CEM algorithms in practice.     

Constructing Efficient Experimental Designs for Generalized Linear Models

Exchange algorithms are popular for finding optimal or efficient designs for linear models, but there are few discussions of this type of algorithm for generalized linear models (GLMs) in literature. A new algorithm, generalized Coordinate Exchange Algorithm (gCEA), is developed in this article to construct efficient designs for GLMs. We compare the performance of the proposed algorithm with other optimization algorithms, including point exchange algorithm, columnwise-pairwise algorithm, simulated annealing and generic algorithm, and demonstrate the superior performance of this new algorithm.     

Sequential parameter learning and filtering in structured autoregressive state-space models

We present particle-based algorithms for sequential filtering and parameter learning in state-space autoregressive (AR) models with structured priors. Non-conjugate priors are specified on the AR coefficients at the system level by imposing uniform or truncated normal priors on the moduli and wavelengths of the reciprocal roots of the AR characteristic polynomial. Sequential Monte Carlo algorithms are considered and implemented for on-line filtering and parameter learning within this modeling framework. More specifically, three SMC approaches are considered and compared by applying them to data simulated from different state-space AR models. An analysis of a human electroencephalogram signal is also presented to illustrate the use of the structured state-space AR models in describing biomedical signals.     

Recursive Estimation of GARCH Models

This article develops three recursive on-line algorithms, based on a two-stage least squares scheme for estimating generalized autoregressive conditionally heteroskedastic (GARCH) models. The first one, denoted by 2S-RLS, is an adaptation of the recursive least squares method for estimating autoregressive conditionally heteroskedastic (ARCH) models. The second and the third ones (denoted, respectively, by 2S-PLR and 2S-RML) are adapted versions of the pseudolinear regression (PLR) and the recursive maximum likelihood (RML) methods to the GARCH case. We show that the proposed algorithms give consistent estimators and that the 2S-RLS and the 2S-RML estimators are asymptotically Gaussian. These methods seem very adequate for modeling the sequential feature of financial time series, which are observed on a high-frequency basis. The performance of these algorithms is shown via a simulation study.