Centered parameterizations and dependence limitations in Markov random field models

Markov random field models incorporate terms representing local statistical dependence among variables in a discrete-index random field. Traditional parameterizations for models based on one-parameter exponential family conditional distributions contain components that would appear to reflect large-scale and small-scale model behaviors, and it is natural to attempt to match these structures with large-scale and small-scale patterns in a set of data. Traditional manners of parameterizing Markov random field models do not allow such correspondence, however. We propose an alternative centered parameterization that, while not leading to different models, allows a correspondence between model structures and data structures to be successfully accomplished. The ability to make these connections is important when incorporating covariate information into a model or if a sequence of models is fit over time to investigate and interpret possible changes in data structure. We demonstrate the improved interpretation that results from use of centered parameterizations. Centered parameterizations also lend themselves to computation of an interpretable decomposition of mean squared error, and this is demonstrated both analytically and through a simulated example. A breakdown in model behavior occurs even with centered parameterizations if dependence parameters in Markov random field models are allowed to become too large. This phenomenon is discussed and illustrated using an auto-logistic model.     

Sparse Markov Chains for Sequence Data

Finite memory sources and variable‐length Markov chains have recently gained popularity in data compression and mining, in particular, for applications in bioinformatics and language modelling. Here, we consider denser data compression and prediction with a family of sparse Bayesian predictive models for Markov chains in finite state spaces. Our approach lumps transition probabilities into classes composed of invariant probabilities, such that the resulting models need not have a hierarchical structure as in context tree‐based approaches. This can lead to a substantially higher rate of data compression, and such non‐hierarchical sparse models can be motivated for instance by data dependence structures existing in the bioinformatics context. We describe a Bayesian inference algorithm for learning sparse Markov models through clustering of transition probabilities. Experiments with DNA sequence and protein data show that our approach is competitive in both prediction and classification when compared with several alternative methods on the basis of variable memory length.     

Generic Identifiability of Linear Structural Equation Models by Ancestor Decomposition

Linear structural equation models, which relate random variables via linear interdependencies and Gaussian noise, are a popular tool for modelling multivariate joint distributions. The models correspond to mixed graphs that include both directed and bidirected edges representing the linear relationships and correlations between noise terms, respectively. A question of interest for these models is that of parameter identifiability, whether or not it is possible to recover edge coefficients from the joint covariance matrix of the random variables. For the problem of determining generic parameter identifiability, we present an algorithm building upon the half‐trek criterion. Underlying our new algorithm is the idea that ancestral subsets of vertices in the graph can be used to extend the applicability of a decomposition technique.     

Focused information criteria for copulas

In this paper, we extend the focused information criterion (FIC) to copula models. Copulas are often used for applications where the joint tail behavior of the variables is of particular interest, and selecting a copula that captures this well is then essential. Traditional model selection methods such as the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) aim at finding the overall best‐fitting model, which is not necessarily the one best suited for the application at hand. The FIC, on the other hand, evaluates and ranks candidate models based on the precision of their point estimates of a context‐given focus parameter. This could be any quantity of particular interest, for example, the mean, a correlation, conditional probabilities, or measures of tail dependence. We derive FIC formulae for the maximum likelihood estimator, the two‐stage maximum likelihood estimator, and the so‐called pseudo‐maximum‐likelihood (PML) estimator combined with parametric margins. Furthermore, we confirm the validity of the AIC formula for the PML estimator combined with parametric margins. To study the numerical behavior of FIC, we have carried out a simulation study, and we have also analyzed a multivariate data set pertaining to abalones. The results from the study show that the FIC successfully ranks candidate models in terms of their performance, defined as how well they estimate the focus parameter. In terms of estimation precision, FIC clearly outperforms AIC, especially when the focus parameter relates to only a specific part of the model, such as the conditional upper‐tail probability.     

Non‐stationary Cross‐Covariance Models for Multivariate Processes on a Globe

Abstract. In geophysical and environmental problems, it is common to have multiple variables of interest measured at the same location and time. These multiple variables typically have dependence over space (and/or time). As a consequence, there is a growing interest in developing models for multivariate spatial processes, in particular, the cross‐covariance models. On the other hand, many data sets these days cover a large portion of the Earth such as satellite data, which require valid covariance models on a globe. We present a class of parametric covariance models for multivariate processes on a globe. The covariance models are flexible in capturing non‐stationarity in the data yet computationally feasible and require moderate numbers of parameters. We apply our covariance model to surface temperature and precipitation data from an NCAR climate model output. We compare our model to the multivariate version of the Matérn cross‐covariance function and models based on coregionalization and demonstrate the superior performance of our model in terms of AIC (and/or maximum loglikelihood values) and predictive skill. We also present some challenges in modelling the cross‐covariance structure of the temperature and precipitation data. Based on the fitted results using full data, we give the estimated cross‐correlation structure between the two variables.     

Exact or approximate inference in graphical models: why the choice is dictated by the treewidth,and how variable elimination can be exploited

Probabilistic graphical models offer a powerful framework to account for the dependence structure between variables, which is represented as a graph. However, the dependence between variables may render inference tasks intractable. In this paper, we review techniques exploiting the graph structure for exact inference, borrowed from optimisation and computer science. They are built on the principle of variable elimination whose complexity is dictated in an intricate way by the order in which variables are eliminated. The so‐called treewidth of the graph characterises this algorithmic complexity: low‐treewidth graphs can be processed efficiently. The first point that we illustrate is therefore the idea that for inference in graphical models, the number of variables is not the limiting factor, and it is worth checking the width of several tree decompositions of the graph before resorting to the approximate method. We show how algorithms providing an upper bound of the treewidth can be exploited to derive a ‘good' elimination order enabling to realise exact inference. The second point is that when the treewidth is too large, algorithms for approximate inference linked to the principle of variable elimination, such as loopy belief propagation and variational approaches, can lead to accurate results while being much less time consuming than Monte‐Carlo approaches. We illustrate the techniques reviewed in this article on benchmarks of inference problems in genetic linkage analysis and computer vision, as well as on hidden variables restoration in coupled Hidden Markov Models.     

Exact Goodness‐of‐Fit Testing for the Ising Model

The Ising model is one of the simplest and most famous models of interacting systems. It was originally proposed to model ferromagnetic interactions in statistical physics and is now widely used to model spatial processes in many areas such as ecology, sociology, and genetics, usually without testing its goodness of fit. Here, we propose various test statistics and an exact goodness‐of‐fit test for the finite‐lattice Ising model. The theory of Markov bases has been developed in algebraic statistics for exact goodness‐of‐fit testing using a Monte Carlo approach. However, finding a Markov basis is often computationally intractable. Thus, we develop a Monte Carlo method for exact goodness‐of‐fit testing for the Ising model that avoids computing a Markov basis and also leads to a better connectivity of the Markov chain and hence to a faster convergence. We show how this method can be applied to analyze the spatial organization of receptors on the cell membrane.     

Cure rate models: A unified approach

The authors propose a novel class of cure rate models for right‐censored failure time data. The class is formulated through a transformation on the unknown population survival function. It includes the mixture cure model and the promotion time cure model as two special cases. The authors propose a general form of the covariate structure which automatically satisfies an inherent parameter constraint and includes the corresponding binomial and exponential covariate structures in the two main formulations of cure models. The proposed class provides a natural link between the mixture and the promotion time cure models, and it offers a wide variety of new modelling structures as well. Within the Bayesian paradigm, a Markov chain Monte Carlo computational scheme is implemented for sampling from the full conditional distributions of the parameters. Model selection is based on the conditional predictive ordinate criterion. The use of the new class of models is illustrated with a set of real data involving a melanoma clinical trial.     

ESTIMATORS BASED ON KENDALL'S TAU IN MULTIVARIATE COPULA MODELS

The estimation of a real‐valued dependence parameter in a multivariate copula model is considered. Rank‐based procedures are often used in this context to guard against possible misspecification of the marginal distributions. A standard approach consists of maximizing the pseudo‐likelihood. Here, we investigate alternative estimators based on the inversion of two multivariate extensions of Kendall's tau developed by Kendall and Babington Smith, and by Joe. The former, which amounts to the average value of tau over all pairs of variables, is often referred to as the coefficient of agreement. Existing results concerning the finite‐ and large‐sample properties of this coefficient are summarized, and new, parallel findings are provided for the multivariate version of tau due to Joe, along with illustrations. The performance of the estimators resulting from the inversion of these two versions of Kendall's tau is compared in the context of copula models through simulations.     

Examination and visualisation of the simplifying assumption for vine copulas in three dimensions

Vine copulas are a highly flexible class of dependence models, which are based on the decomposition of the density into bivariate building blocks. For applications one usually makes the simplifying assumption that copulas of conditional distributions are independent of the variables on which they are conditioned. However this assumption has been criticised for being too restrictive. We examine both simplified and non‐simplified vine copulas in three dimensions and investigate conceptual differences. We show and compare contour surfaces of three‐dimensional vine copula models, which prove to be much more informative than the contour lines of the bivariate marginals. Our investigation shows that non‐simplified vine copulas can exhibit arbitrarily irregular shapes, whereas simplified vine copulas appear to be smooth extrapolations of their bivariate margins to three dimensions. In addition to a variety of constructed examples, we also investigate a three‐dimensional subset of the well‐known uranium data set and visually detect the fact that a non‐simplified vine copula is necessary to capture its complex dependence structure.     

Simultaneous inference in structured additive conditional copula regression models: a unifying Bayesian approach

While most regression models focus on explaining distributional aspects of one single response variable alone, interest in modern statistical applications has recently shifted towards simultaneously studying multiple response variables as well as their dependence structure. A particularly useful tool for pursuing such an analysis are copula-based regression models since they enable the separation of the marginal response distributions and the dependence structure summarised in a specific copula model. However, so far copula-based regression models have mostly been relying on two-step approaches where the marginal distributions are determined first whereas the copula structure is studied in a second step after plugging in the estimated marginal distributions. Moreover, the parameters of the copula are mostly treated as a constant not related to covariates and most regression specifications for the marginals are restricted to purely linear predictors. We therefore propose simultaneous Bayesian inference for both the marginal distributions and the copula using computationally efficient Markov chain Monte Carlo simulation techniques. In addition, we replace the commonly used linear predictor by a generic structured additive predictor comprising for example nonlinear effects of continuous covariates, spatial effects or random effects and furthermore allow to make the copula parameters covariate-dependent. To facilitate Bayesian inference, we construct proposal densities for a Metropolis–Hastings algorithm relying on quadratic approximations to the full conditionals of regression coefficients avoiding manual tuning. The performance of the resulting Bayesian estimates is evaluated in simulations comparing our approach with penalised likelihood inference, studying the choice of a specific copula model based on the deviance information criterion, and comparing a simultaneous approach with a two-step procedure. Furthermore, the flexibility of Bayesian conditional copula regression models is illustrated in two applications on childhood undernutrition and macroecology.     

BAYESIAN HIDDEN MARKOV MODELS FOR LONGITUDINAL COUNTS

This paper describes a Bayesian approach to modelling carcinogenity in animal studies where the data consist of counts of the number of tumours present over time. It compares two autoregressive hidden Markov models. One of them models the transitions between three latent states: an inactive transient state, a multiplying state for increasing counts and a reducing state for decreasing counts. The second model introduces a fourth tied state to describe non‐zero observations that are neither increasing nor decreasing. Both these models can model the length of stay upon entry of a state. A discrete constant hazards waiting time distribution is used to model the time to onset of tumour growth. Our models describe between‐animal‐variability by a single hierarchy of random effects and the within‐animal variation by first‐order serial dependence. They can be extended to higher‐order serial dependence and multi‐level hierarchies. Analysis of data from animal experiments comparing the influence of two genes leads to conclusions that differ from those of Dunson (2000). The observed data likelihood defines an information criterion to assess the predictive properties of the three‐ and four‐state models. The deviance information criterion is appropriately defined for discrete parameters.     

Multi-state Models: A Review

Multi-state models are models for a process, for example describing a life history of an individual, which at any time occupies one of a few possible states. This can describe several possible events for a single individual, or the dependence between several individuals. The events are the transitions between the states. This class of models allows for an extremely flexible approach that can model almost any kind of longitudinal failure time data. This is particularly relevant for modeling different events, which have an event-related dependence, like occurrence of disease changing the risk of death. It can also model paired data. It is useful for recurrent events, but has limitations. The Markov models stand out as much simpler than other models from a probability point of view, and this simplifies the likelihood evaluation. However, in many cases, the Markov models do not fit satisfactorily, and happily, it is reasonably simple to study non-Markov models, in particular the Markov extension models. This also makes it possible to consider, whether the dependence is of short-term or long-term nature. Applications include the effect of heart transplantation on the mortality and the mortality among Danish twins.     

Graphs for Margins of Bayesian Networks

Directed acyclic graph (DAG) models—also called Bayesian networks—are widely used in probabilistic reasoning, machine learning and causal inference. If latent variables are present, then the set of possible marginal distributions over the remaining (observed) variables is generally not represented by any DAG. Larger classes of mixed graphical models have been introduced to overcome this; however, as we show, these classes are not sufficiently rich to capture all the marginal models that can arise. We introduce a new class of hyper‐graphs, called mDAGs, and a latent projection operation to obtain an mDAG from the margin of a DAG. We show that each distinct marginal of a DAG model is represented by at least one mDAG and provide graphical results towards characterizing equivalence of these models. Finally, we show that mDAGs correctly capture the marginal structure of causally interpreted DAGs under interventions on the observed variables.     

Alternative Markov Properties for Chain Graphs

Graphical Markov models use graphs to represent possible dependences among statistical variables. Lauritzen, Wermuth, and Frydenberg (LWF) introduced a Markov property for chain graphs (CG): graphs that can be used to represent both structural and associative dependences simultaneously and that include both undirected graphs (UG) and acyclic directed graphs (ADG) as special cases. Here an alternative Markov property (AMP) for CGs is introduced and shown to be the Markov property satisfied by a block-recursive linear system with multivariate normal errors. This model can be decomposed into a collection of conditional normal models, each of which combines the features of multivariate linear regression models and covariance selection models, facilitating the estimation of its parameters. In the general case, necessary and sufficient conditions are given for the equivalence of the LWF and AMP Markov properties of a CG, for the AMP Markov equivalence of two CGs, for the AMP Markov equivalence of a CG to some ADG or decomposable UG, and for other equivalences. For CGs, in some ways the AMP property is a more direct extension of the ADG Markov property than is the LWF property.     

Optimal Bayesian estimators for latent variable cluster models

In cluster analysis interest lies in probabilistically capturing partitions of individuals, items or observations into groups, such that those belonging to the same group share similar attributes or relational profiles. Bayesian posterior samples for the latent allocation variables can be effectively obtained in a wide range of clustering models, including finite mixtures, infinite mixtures, hidden Markov models and block models for networks. However, due to the categorical nature of the clustering variables and the lack of scalable algorithms, summary tools that can interpret such samples are not available. We adopt a Bayesian decision theoretical approach to define an optimality criterion for clusterings and propose a fast and context-independent greedy algorithm to find the best allocations. One important facet of our approach is that the optimal number of groups is automatically selected, thereby solving the clustering and the model-choice problems at the same time. We consider several loss functions to compare partitions and show that our approach can accommodate a wide range of cases. Finally, we illustrate our approach on both artificial and real datasets for three different clustering models: Gaussian mixtures, stochastic block models and latent block models for networks.     

Building and Fitting Non‐Gaussian Latent Variable Models via the Moment‐Generating Function

Abstract. For certain classes of hierarchical models, it is easy to derive an expression for the joint moment‐generating function (MGF) of data, whereas the joint probability density has an intractable form which typically involves an integral. The most important example is the class of linear models with non‐Gaussian latent variables. Parameters in the model can be estimated by approximate maximum likelihood, using a saddlepoint‐type approximation to invert the MGF. We focus on modelling heavy‐tailed latent variables, and suggest a family of mixture distributions that behaves well under the saddlepoint approximation (SPA). It is shown that the well‐known normalization issue renders the ordinary SPA useless in the present context. As a solution we extend the non‐Gaussian leading term SPA to a multivariate setting, and introduce a general rule for choosing the leading term density. The approach is applied to mixed‐effects regression, time‐series models and stochastic networks and it is shown that the modified SPA is very accurate.     

Autoregressive Moving Average Infinite Hidden Markov-Switching Models

Markov-switching models are usually specified under the assumption that all the parameters change when a regime switch occurs. Relaxing this hypothesis and being able to detect which parameters evolve over time is relevant for interpreting the changes in the dynamics of the series, for specifying models parsimoniously, and may be helpful in forecasting. We propose the class of sticky infinite hidden Markov-switching autoregressive moving average models, in which we disentangle the break dynamics of the mean and the variance parameters. In this class, the number of regimes is possibly infinite and is determined when estimating the model, thus avoiding the need to set this number by a model choice criterion. We develop a new Markov chain Monte Carlo estimation method that solves the path dependence issue due to the moving average component. Empirical results on macroeconomic series illustrate that the proposed class of models dominates the model with fixed parameters in terms of point and density forecasts.     

ESTIMATING BAYESIAN NETWORKS FOR HIGH‐DIMENSIONAL DATA WITH COMPLEX MEAN STRUCTURE AND RANDOM EFFECTS

The estimation of Bayesian networks given high‐dimensional data, in particular gene expression data, has been the focus of much recent research. Whilst there are several methods available for the estimation of such networks, these typically assume that the data consist of independent and identically distributed samples. It is often the case, however, that the available data have a more complex mean structure, plus additional components of variance, which must then be accounted for in the estimation of a Bayesian network. In this paper, score metrics that take account of such complexities are proposed for use in conjunction with score‐based methods for the estimation of Bayesian networks. We propose first, a fully Bayesian score metric, and second, a metric inspired by the notion of restricted maximum likelihood. We demonstrate the performance of these new metrics for the estimation of Bayesian networks using simulated data with known complex mean structures. We then present the analysis of expression levels of grape‐berry genes adjusting for exogenous variables believed to affect the expression levels of the genes. Demonstrable biological effects can be inferred from the estimated conditional independence relationships and correlations amongst the grape‐berry genes.     

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

Summary.  Structured additive regression models are perhaps the most commonly used class of models in statistical applications. It includes, among others, (generalized) linear models, (generalized) additive models, smoothing spline models, state space models, semiparametric regression, spatial and spatiotemporal models, log-Gaussian Cox processes and geostatistical and geoadditive models. We consider approximate Bayesian inference in a popular subset of structured additive regression models, latent Gaussian models , where the latent field is Gaussian, controlled by a few hyperparameters and with non-Gaussian response variables. The posterior marginals are not available in closed form owing to the non-Gaussian response variables. For such models, Markov chain Monte Carlo methods can be implemented, but they are not without problems, in terms of both convergence and computational time. In some practical applications, the extent of these problems is such that Markov chain Monte Carlo sampling is simply not an appropriate tool for routine analysis. We show that, by using an integrated nested Laplace approximation and its simplified version, we can directly compute very accurate approximations to the posterior marginals. The main benefit of these approximations is computational: where Markov chain Monte Carlo algorithms need hours or days to run, our approximations provide more precise estimates in seconds or minutes. Another advantage with our approach is its generality, which makes it possible to perform Bayesian analysis in an automatic, streamlined way, and to compute model comparison criteria and various predictive measures so that models can be compared and the model under study can be challenged.