Kernel partial correlation: a novel approach to capturing conditional independence in graphical models for noisy data

Graphical models capture the conditional independence structure among random variables via existence of edges among vertices. One way of inferring a graph is to identify zero partial correlation coefficients, which is an effective way of finding conditional independence under a multivariate Gaussian setting. For more general settings, we propose kernel partial correlation which extends partial correlation with a combination of two kernel methods. First, a nonparametric function estimation is employed to remove effects from other variables, and then the dependence between remaining random components is assessed through a nonparametric association measure. The proposed approach is not only flexible but also robust under high levels of noise owing to the robustness of the nonparametric approaches.     

The Ranges of Limiting Values of Some Partial Correlations under Conditional Independence

Pearson's partial correlation, Kendall's partial tau, and a partial correlation based on Spearman's rho need not be consistent estimators of zero under conditional independence. The ranges of possible limiting values of these correlations are computed under multivariate normality and lognormality. Students should exercise caution when interpreting these partial correlations as a measure of conditional independence.     

AN EXACT CONDITIONAL TEST FOR COVARIANCE SELECTION MODELS

An exact conditional test is developed for testing the absence of an edge in a graphical covariance selection model and is shown to be equivalent to a test based on the partial correlation coefficient. An example is given.     

A Note on Sufficient Conditions for Valid Unmodified t Testing in Correlation Analysis with Autocorrelated and Heteroscedastic Sample Data

Traditionally, sphericity (i.e., independence and homoscedasticity for raw data) is put forward as the condition to be satisfied by the variance–covariance matrix of at least one of the two observation vectors analyzed for correlation, for the unmodified t test of significance to be valid under the Gaussian and constant population mean assumptions. In this article, the author proves that the sphericity condition is too strong and a weaker (i.e., more general) sufficient condition for valid unmodified t testing in correlation analysis is circularity (i.e., independence and homoscedasticity after linear transformation by orthonormal contrasts), to be satisfied by the variance–covariance matrix of one of the two observation vectors. Two other conditions (i.e., compound symmetry for one of the two observation vectors; absence of correlation between the components of one observation vector, combined with a particular pattern of joint heteroscedasticity in the two observation vectors) are also considered and discussed. When both observation vectors possess the same variance–covariance matrix up to a positive multiplicative constant, the circularity condition is shown to be necessary and sufficient. “Observation vectors” may designate partial realizations of temporal or spatial stochastic processes as well as profile vectors of repeated measures. From the proof, it follows that an effective sample size appropriately defined can measure the discrepancy from the more general sufficient condition for valid unmodified t testing in correlation analysis with autocorrelated and heteroscedastic sample data. The proof is complemented by a simulation study. Finally, the differences between the role of the circularity condition in the correlation analysis and its role in the repeated measures ANOVA (i.e., where it was first introduced) are scrutinized, and the link between the circular variance–covariance structure and the centering of observations with respect to the sample mean is emphasized.     

On testing for independence between the innovations of several time series

Test statistics for checking the independence between the innovations of several time series are developed. The time series models considered allow for general specifications for the conditional mean and variance functions that could depend on common explanatory variables. In testing for independence between more than two time series, checking pairwise independence does not lead to consistent procedures. Thus a finite family of empirical processes relying on multivariate lagged residuals are constructed, and we derive their asymptotic distributions. In order to obtain simple asymptotic covariance structures, Möbius transformations of the empirical processes are studied, and simplifications occur. Under the null hypothesis of independence, we show that these transformed processes are asymptotically Gaussian, independent, and with tractable covariance functions not depending on the estimated parameters. Various procedures are discussed, including Cramér–von Mises test statistics and tests based on non‐parametric measures. The ranks of the residuals are considered in the new methods, giving test statistics which are asymptotically margin‐free. Generalized cross‐correlations are introduced, extending the concept of cross‐correlation to an arbitrary number of time series; portmanteau procedures based on them are discussed. In order to detect the dependence visually, graphical devices are proposed. Simulations are conducted to explore the finite sample properties of the methodology, which is found to be powerful against various types of alternatives when the independence is tested between two and three time series. An application is considered, using the daily log‐returns of Apple, Intel and Hewlett‐Packard traded on the Nasdaq financial market. The Canadian Journal of Statistics 40: 447–479; 2012 © 2012 Statistical Society of Canada     

Posterior moments of scalar functions of a covariance matrix

Given multivariate normal data and a certain spherically invariant prior distribution on the covariance matrix, it is desired to estimate the moments of the posterior marginal distributions of some scalar functions of the covariance matrix by importance sampling. To this end a family of distributions is defined on the group of orthogonal matrices and a procedure is proposed for selecting one of these distributions for use as a weighting distribution in the importance sampling process. In an example estimates are calculated for the posterior mean and variance of each element in the covariance matrix expressed in the original coordinates, for the posterior mean of each element in the correlation matrix expressed in the original coordinates, and for the posterior mean of each element in the covariance matrix expressed in the coordinates of the principal variables.     

CONDITIONAL EXPECTATION FORMULAE FOR COPULAS

Not only are copula functions joint distribution functions in their own right, they also provide a link between multivariate distributions and their lower‐dimensional marginal distributions. Copulas have a structure that allows us to characterize all possible multivariate distributions, and therefore they have the potential to be a very useful statistical tool. Although copulas can be traced back to 1959, there is still much scope for new results, as most of the early work was theoretical rather than practical. We focus on simple practical tools based on conditional expectation, because such tools are not widely available. When dealing with data sets in which the dependence throughout the sample is variable, we suggest that copula‐based regression curves may be more accurate predictors of specific outcomes than linear models. We derive simple conditional expectation formulae in terms of copulas and apply them to a combination of simulated and real data.     

On symmetry of finite mixtures of normal distributions

In this note we derive a necessary and sufficient condition for a distribution obtained by taking a finite mixture of multivariate normal distributions to be symmetric about zero. The result derived also holds for mixtures of symmetric stable distributions, including the Cauchy distribution.     

Graphical models for skew-normal variates

This paper explores the usefulness of the multivariate skew-normal distribution in the context of graphical models. A slight extension of the family recently discussed by Azzalini & Dalla Valle (1996 ) and Azzalini & Capitanio (1999 ) is described, the main motivation being the additional property of closure under conditioning. After considerations of the main probabilistic features, the focus of the paper is on the construction of conditional independence graphs for skew-normal variables. Necessary and sufficient conditions for conditional independence are stated, and the admissible structures of a graph under restriction on univariate marginal distribution are studied. Finally, parameter estimation is considered. It is shown how the factorization of the likelihood function according to a graph can be rearranged in order to obtain a parameter based factorization.     

A comparison of methods for simulating correlated binary variables with specified marginal means and correlations

Simulation studies employed to study properties of estimators for parameters in population-average models for clustered or longitudinal data require suitable algorithms for data generation. Methods for generating correlated binary data that allow general specifications of the marginal mean and correlation structures are particularly useful. We compare an algorithm based on dichotomizing multi-normal variates to one based on a conditional linear family (CLF) of distributions [Qaqish BF. A family of multivariate binary distributions for simulating correlated binary variables with specified marginal means and correlations. Biometrika. 2003;90:455–463] with respect to range restrictions induced on correlations. Examples include generating longitudinal binary data and generating correlated binary data compatible with specified marginal means and covariance structures for bivariate, overdispersed binomial outcomes. Results show the CLF method gives a wider range of correlations for longitudinal data having autocorrelated within-subject associations, while the multivariate probit method gives a wider range of correlations for clustered data having exchangeable-type correlations. In the case of a decaying-product correlation structure, it is shown that the CLF method achieves the nonparametric limits on the range of correlations, which cannot be surpassed by any method.     

Applications of a new power normal family

The main purpose of this paper is to give an algorithm to attain joint normality of non-normal multivariate observations through a new power normal family introduced by the author (Isogai, 1999). The algorithm tries to transform each marginal variable simultaneously to joint normality, but due to a large number of parameters it repeats a maximization process with respect to the conditional normal density of one transformed variable given the other transformed variables. A non-normal data set is used to examine performance of the algorithm, and the degree of achievement of joint normality is evaluated by measures of multivariate skewness and kurtosis. Besides the above topic, making use of properties of our power normal family, we discuss not only a normal approximation formula of non-central F distributions in the frame of regression analysis but also some decomposition formulas of a power parameter, which appear in a Wilson-Hilferty power transformation setting.     

Explaining the behavior of joint and marginal Monte Carlo estimators in latent variable models with independence assumptions

In latent variable models parameter estimation can be implemented by using the joint or the marginal likelihood, based on independence or conditional independence assumptions. The same dilemma occurs within the Bayesian framework with respect to the estimation of the Bayesian marginal (or integrated) likelihood, which is the main tool for model comparison and averaging. In most cases, the Bayesian marginal likelihood is a high dimensional integral that cannot be computed analytically and a plethora of methods based on Monte Carlo integration (MCI) are used for its estimation. In this work, it is shown that the joint MCI approach makes subtle use of the properties of the adopted model, leading to increased error and bias in finite settings. The sources and the components of the error associated with estimators under the two approaches are identified here and provided in exact forms. Additionally, the effect of the sample covariation on the Monte Carlo estimators is examined. In particular, even under independence assumptions the sample covariance will be close to (but not exactly) zero which surprisingly has a severe effect on the estimated values and their variability. To address this problem, an index of the sample’s divergence from independence is introduced as a multivariate extension of covariance. The implications addressed here are important in the majority of practical problems appearing in Bayesian inference of multi-parameter models with analogous structures.     

On marginal likelihood inference for the intra-class correlation coefficient

A p-component set of responses have been constructed by a location-scale transformation to a p-component set of error variables, the covariance matrix of the set of error variables being of intra-class covariance structure:all variances being unity, and covariance being equal [IML0001]. A sample of size n has been described as a conditional structural model, conditional on the value of the intra-class correlation coefficient ρ. The conditional technique of structural inference provides the marginal likelihood function of ρ based on the standardized residuals. For the normal case, the marginal likelihood function of ρ is seen to be dependent on the standardized residuals through the sample intra-class correlation coefficient. By the likelihood modulation technique, the nonnull distribution of the sample intra-class correlation coefficient has also been obtained.     

A generalization of the bivariate Beta-Binomial distribution

This paper presents a new bivariate discrete distribution that generalizes the bivariate Beta-Binomial distribution. It is generated by Appell hypergeometric function F₁ and can be obtained as a Binomial mixture with an Exton's Generalized Beta distribution. The model has different marginal distributions which are, together with the conditional distributions, more flexible than the Beta-Binomial distribution. It has non-linear regression curves and is useful for random variables with positive correlation. These features make the model very adequate to fit observed data as the two applications included show.     

Stratified Gaussian graphical models

Gaussian graphical models represent the backbone of the statistical toolbox for analyzing continuous multivariate systems. However, due to the intrinsic properties of the multivariate normal distribution, use of this model family may hide certain forms of context-specific independence that are natural to consider from an applied perspective. Such independencies have been earlier introduced to generalize discrete graphical models and Bayesian networks into more flexible model families. Here, we adapt the idea of context-specific independence to Gaussian graphical models by introducing a stratification of the Euclidean space such that a conditional independence may hold in certain segments but be absent elsewhere. It is shown that the stratified models define a curved exponential family, which retains considerable tractability for parameter estimation and model selection.     

Singular Gaussian graphical models: Structure learning

ABSTRACT

The goal of this article is to introduce singular Gaussian graphical models and their conditional independence properties. In fact, we extend the concept of Gaussian Markov Random Field to the case of a multivariate normally distributed vector with a singular covariance matrix. We construct, then, the associated graph’s structure from the covariance matrix’s pseudo-inverse on the basis of a characterization of the pairwise conditional independence. The proposed approach can also be used when the covariance matrix is ill-conditioned, through projecting data on a smaller subspace. In this case, our method ensures numerical stability and consistency of the constructed graph and significantly reduces the inference problem’s complexity. These aspects are illustrated using numerical experiments.     

Simulation of truncated normal variables

We provide simulation algorithms for one-sided and two-sided truncated normal distributions. These algorithms are then used to simulate multivariate normal variables with convex restricted parameter space for any covariance structure.     

Examples of Uncorrelated Dependent Random Variables Using a Bivariate Mixture

A particular mixture of bivariate distributions is used to present examples of dependent uncorrelated random variables and independent random variables. A necessary and sufficient condition for the independence for such a bivariate distribution is given.     

Modeling a mixture of ordinal and continuous repeated measures

We study the correlation structure for a mixture of ordinal and continuous repeated measures using a Bayesian approach. We assume a multivariate probit model for the ordinal variables and a normal linear regression for the continuous variables, where latent normal variables underlying the ordinal data are correlated with continuous variables in the model. Due to the probit model assumption, we are required to sample a covariance matrix with some of the diagonal elements equal to one. The key computational idea is to use parameter-extended data augmentation, which involves applying the Metropolis-Hastings algorithm to get a sample from the posterior distribution of the covariance matrix incorporating the relevant restrictions. The methodology is illustrated through a simulated example and through an application to data from the UCLA Brain Injury Research Center.     

Sampling Methods for Wallenius' and Fisher's Noncentral Hypergeometric Distributions

Several methods for generating variates with univariate and multivariate Walleniu' and Fisher's noncentral hypergeometric distributions are developed. Methods for the univariate distributions include: simulation of urn experiments, inversion by binary search, inversion by chop-down search from the mode, ratio-of-uniforms rejection method, and rejection by sampling in the τ domain. Methods for the multivariate distributions include: simulation of urn experiments, conditional method, Gibbs sampling, and Metropolis-Hastings sampling. These methods are useful for Monte Carlo simulation of models of biased sampling and models of evolution and for calculating moments and quantiles of the distributions.