Maximum likelihood estimation of parameter structures for the Wishart distribution using constraints

Maximum likelihood estimation under constraints for estimation in the Wishart class of distributions, is considered. It provides a unified approach to estimation in a variety of problems concerning covariance matrices. Virtually all covariance structures can be translated to constraints on the covariances. This includes covariance matrices with given structure such as linearly patterned covariance matrices, covariance matrices with zeros, independent covariance matrices and structurally dependent covariance matrices. The methodology followed in this paper provides a useful and simple approach to directly obtain the exact maximum likelihood estimates. These maximum likelihood estimates are obtained via an estimation procedure for the exponential class using constraints.     

Testing the equality of two high‐dimensional spatial sign covariance matrices

This paper is concerned with testing the equality of two high‐dimensional spatial sign covariance matrices with applications to testing the proportionality of two high‐dimensional covariance matrices. It is interesting that these two testing problems are completely equivalent for the class of elliptically symmetric distributions. This paper develops a new test for testing the equality of two high‐dimensional spatial sign covariance matrices based on the Frobenius norm of the difference between two spatial sign covariance matrices. The asymptotic normality of the proposed testing statistic is derived under the null and alternative hypotheses when the dimension and sample sizes both tend to infinity. Moreover, the asymptotic power function is also presented. Simulation studies show that the proposed test performs very well in a wide range of settings and can be allowed for the case of large dimensions and small sample sizes.     

A hierarchical eigenmodel for pooled covariance estimation

Summary.  Although the covariance matrices corresponding to different populations are unlikely to be exactly equal they can still exhibit a high degree of similarity. For example, some pairs of variables may be positively correlated across most groups, whereas the correlation between other pairs may be consistently negative. In such cases much of the similarity across covariance matrices can be described by similarities in their principal axes, which are the axes that are defined by the eigenvectors of the covariance matrices. Estimating the degree of across-population eigenvector heterogeneity can be helpful for a variety of estimation tasks. For example, eigenvector matrices can be pooled to form a central set of principal axes and, to the extent that the axes are similar, covariance estimates for populations having small sample sizes can be stabilized by shrinking their principal axes towards the across-population centre. To this end, the paper develops a hierarchical model and estimation procedure for pooling principal axes across several populations. The model for the across-group heterogeneity is based on a matrix-valued antipodally symmetric Bingham distribution that can flexibly describe notions of 'centre' and 'spread' for a population of orthogonal matrices.     

Regularized covariance matrix estimation under the common principal components model

The common principal components (CPC) model provides a way to model the population covariance matrices of several groups by assuming a common eigenvector structure. When appropriate, this model can provide covariance matrix estimators of which the elements have smaller standard errors than when using either the pooled covariance matrix or the per group unbiased sample covariance matrix estimators. In this article, a regularized CPC estimator under the assumption of a common (or partially common) eigenvector structure in the populations is proposed. After estimation of the common eigenvectors using the Flury–Gautschi (or other) algorithm, the off-diagonal elements of the nearly diagonalized covariance matrices are shrunk towards zero and multiplied with the orthogonal common eigenvector matrix to obtain the regularized CPC covariance matrix estimates. The optimal shrinkage intensity per group can be estimated using cross-validation. The efficiency of these estimators compared to the pooled and unbiased estimators is investigated in a Monte Carlo simulation study, and the regularized CPC estimator is applied to a real dataset to demonstrate the utility of the method.     

Autocovariance Estimation in Regression with a Discontinuous Signal and m‐Dependent Errors: A Difference‐Based Approach

We discuss a class of difference‐based estimators for the autocovariance in nonparametric regression when the signal is discontinuous and the errors form a stationary m‐dependent process. These estimators circumvent the particularly challenging task of pre‐estimating such an unknown regression function. We provide finite‐sample expressions of their mean squared errors for piecewise constant signals and Gaussian errors. Based on this, we derive biased‐optimized estimates that do not depend on the unknown autocovariance structure. Notably, for positively correlated errors, that part of the variance of our estimators that depend on the signal is minimal as well. Further, we provide sufficient conditions for ‐consistency; this result is extended to piecewise Hölder regression with non‐Gaussian errors. We combine our biased‐optimized autocovariance estimates with a projection‐based approach and derive covariance matrix estimates, a method that is of independent interest. An R package, several simulations and an application to biophysical measurements complement this paper.     

Statistical Corrections of Invalid Correlation Matrices

Suppose estimates are available for correlations between pairs of variables but that the matrix of correlation estimates is not positive definite. In various applications, having a valid correlation matrix is important in connection with follow‐up analyses that might, for example, involve sampling from a valid distribution. We present new methods for adjusting the initial estimates to form a proper, that is, nonnegative definite, correlation matrix. These are based on constructing certain pseudo‐likelihood functions, formed by multiplying together exact or approximate likelihood contributions associated with the individual correlations. Such pseudo‐likelihoods may then be maximized over the range of proper correlation matrices. They may also be utilized to form pseudo‐posterior distributions for the unknown correlation matrix, by factoring in relevant prior information for the separate correlations. We illustrate our methods on two examples from a financial time series and genomic pathway analysis.     

A permutation-based Bayesian approach for inverse covariance estimation

Abstract

Covariance estimation and selection for multivariate datasets in a high-dimensional regime is a fundamental problem in modern statistics. Gaussian graphical models are a popular class of models used for this purpose. Current Bayesian methods for inverse covariance matrix estimation under Gaussian graphical models require the underlying graph and hence the ordering of variables to be known. However, in practice, such information on the true underlying model is often unavailable. We therefore propose a novel permutation-based Bayesian approach to tackle the unknown variable ordering issue. In particular, we utilize multiple maximum a posteriori estimates under the DAG-Wishart prior for each permutation, and subsequently construct the final estimate of the inverse covariance matrix. The proposed estimator has smaller variability and yields order-invariant property. We establish posterior convergence rates under mild assumptions and illustrate that our method outperforms existing approaches in estimating the inverse covariance matrices via simulation studies.     

Accent on Teaching Materials

Estimation of covariance components in the multivariate random-effect model with nested covariance structure is discussed. There are two covariance matrices to be estimated, namely, the between-group and the within-group covariance matrices. These two covariance matrices are most often estimated by forming a multivariate analysis of variance and equating mean square matrices to their expectations. Such a procedure involves taking the difference between the between-group mean square and the within-group mean square matrices, and often produces an estimated between-group covariance matrix that is not nonnegative definite. We present estimators of the two covariance matrices that are always proper covariance matrices. The estimators are the restricted maximum likelihood estimators if the random effects are normally distributed. The estimation procedure is extended to more complicated models, including the twofold nested and the mixed-effect models. A numerical example is presented to illustrate the use of the estimation procedure.     

Averaging estimation for conditional covariance models

Estimating conditional covariance matrices is important in statistics and finance. In this paper, we propose an averaging estimator for the conditional covariance, which combines the estimates of marginal conditional covariance matrices by Model Averaging MArginal Regression of Li, Linton, and Lu. This estimator avoids the “curse of dimensionality” problem that the local constant estimator of Yin et al. suffered from. We establish the asymptotic properties of the averaging weights and that of the proposed conditional covariance estimator. The finite sample performances are augmented by simulation. An application to portfolio allocation illustrates the practical superiority of the averaging estimator.     

Double shrinkage estimators for large sparse covariance matrices

Covariance matrices play an important role in many multivariate techniques and hence a good covariance estimation is crucial in this kind of analysis. In many applications a sparse covariance matrix is expected due to the nature of the data or for simple interpretation. Hard thresholding, soft thresholding, and generalized thresholding were therefore developed to this end. However, these estimators do not always yield well-conditioned covariance estimates. To have sparse and well-conditioned estimates, we propose doubly shrinkage estimators: shrinking small covariances towards zero and then shrinking covariance matrix towards a diagonal matrix. Additionally, a richness index is defined to evaluate how rich a covariance matrix is. According to our simulations, the richness index serves as a good indicator to choose relevant covariance estimator.     

Alternative covariance estimates in a replicated measurement error model with correlated,heteroscedastic errors

The article concerns covariance estimates in a replicated measurement error model with correlated, heteroscedastic errors. Freedman has conjectured that using more of the data will improve estimates of covariance matrices and result in a more efficient estimate of the coefficient of the regression model. The paper confirms the conjecture asymptotically for the case that all random variables are normally distributed, but the gain is not substantial.     

Independence distribution preserving joint covariance structures for the multivariate two-group case

We characterize the general nonnegative-definite and positive-definite joint observation covariance structures for the two-group case such that the two sample mean vectors are independent of the two corresponding sample covariance matrices. Also, the sample covariance matrices are distributed as independent noncentral or central Wishart random matrices. We derive and utilize a representation of the general common non-negative-definite solution to a particular system of matrix equations with idempotent coefficient matrices.     

Rank covariance matrix estimation of a partially known covariance matrix

Classical multivariate methods are often based on the sample covariance matrix, which is very sensitive to outlying observations. One alternative to the covariance matrix is the affine equivariant rank covariance matrix (RCM) that has been studied in Visuri et al. [2003. Affine equivariant multivariate rank methods. J. Statist. Plann. Inference 114, 161–185]. In this article we assume that the covariance matrix is partially known and study how to estimate the corresponding RCM. We use the properties that the RCM is affine equivariant and that the RCM is proportional to the inverse of the regular covariance matrix, and hence reduce the problem of estimating the original RCM to estimating marginal rank covariance matrices. This is a great computational advantage when the dimension of the original data vector is large.     

ON ESTIMATION OF THE POPULATION SPECTRAL DISTRIBUTION FROM A HIGH‐DIMENSIONAL SAMPLE COVARIANCE MATRIX

Sample covariance matrices play a central role in numerous popular statistical methodologies, for example principal components analysis, Kalman filtering and independent component analysis. However, modern random matrix theory indicates that, when the dimension of a random vector is not negligible with respect to the sample size, the sample covariance matrix demonstrates significant deviations from the underlying population covariance matrix. There is an urgent need to develop new estimation tools in such cases with high‐dimensional data to recover the characteristics of the population covariance matrix from the observed sample covariance matrix. We propose a novel solution to this problem based on the method of moments. When the parametric dimension of the population spectrum is finite and known, we prove that the proposed estimator is strongly consistent and asymptotically Gaussian. Otherwise, we combine the first estimation method with a cross‐validation procedure to select the unknown model dimension. Simulation experiments demonstrate the consistency of the proposed procedure. We also indicate possible extensions of the proposed estimator to the case where the population spectrum has a density.     

Joint Central Limit Theorem for Eigenvalue Statistics from Several Dependent Large Dimensional Sample Covariance Matrices with Application

Let X _n = (x _{i j} ) be a k ×n data matrix with complex‐valued, independent and standardized entries satisfying a Lindeberg‐type moment condition. We consider simultaneously R sample covariance matrices , where the Q _r 's are non‐random real matrices with common dimensions p ×k (k ≥p ). Assuming that both the dimension p and the sample size n grow to infinity, the limiting distributions of the eigenvalues of the matrices { B _{n r} } are identified, and as the main result of the paper, we establish a joint central limit theorem (CLT) for linear spectral statistics of the R matrices { B _{n r} }. Next, this new CLT is applied to the problem of testing a high‐dimensional white noise in time series modelling. In experiments, the derived test has a controlled size and is significantly faster than the classical permutation test, although it does have lower power. This application highlights the necessity of such joint CLT in the presence of several dependent sample covariance matrices. In contrast, all the existing works on CLT for linear spectral statistics of large sample covariance matrices deal with a single sample covariance matrix (R = 1).     

An Introduction to Modern Matrix Methods and Statistics

In this article, using the representation that the Kalman filter recursions in state-space models can be expressed as a matrix-weighted average of prior and sample estimates, we supplement the usual filtering algorithm by an extreme bounds analysis. Specifically, as the covariance matrix of the state error is varied in the class of symmetric and positive-definite matrices, the filtering estimates are shown to be in an ellipsoid.     

Objective Bayes Factors for Gaussian Directed Acyclic Graphical Models

Abstract. We propose an objective Bayesian method for the comparison of all Gaussian directed acyclic graphical models defined on a given set of variables. The method, which is based on the notion of fractional Bayes factor (BF), requires a single default (typically improper) prior on the space of unconstrained covariance matrices, together with a prior sample size hyper‐parameter, which can be set to its minimal value. We show that our approach produces genuine BFs. The implied prior on the concentration matrix of any complete graph is a data‐dependent Wishart distribution, and this in turn guarantees that Markov equivalent graphs are scored with the same marginal likelihood. We specialize our results to the smaller class of Gaussian decomposable undirected graphical models and show that in this case they coincide with those recently obtained using limiting versions of hyper‐inverse Wishart distributions as priors on the graph‐constrained covariance matrices.     

Structured Modeling for Post-Mortem Brain Tissue Data

Structured means have been used in studying possible covariate effects on responses, whereas patterned covariances deal with random effects, missing data, and differing study designs. In this article, we develop new multivariate models with patterned means and covariance matrices to deal with special structures of the post-mortem brain tissue data collected in the Conte Center for the Neuroscience of Mental Disorders at the University of Pittsburgh. We obtain maximum likelihood estimates via the method of scoring for these new structured models. One-iteration estimators from a consistent starting point are used to derive the asymptotic distributions. The model fitting algorithms, as well as the asymptotic distributions, are examined using simulated data, and are applied to data from post-mortem tissue studies in schizophrenia.