Estimation of parameters in a generalized GMANOVA model based on an outer product analogy and least squares

This paper investigates estimation of parameters in a combination of the multivariate linear model and growth curve model, called a generalized GMANOVA model. Making analogy between the outer product of data vectors and covariance yields an approach to directly do least squares to covariance. An outer product least squares estimator of covariance (COPLS estimator) is obtained and its distribution is presented if a normal assumption is imposed on the error matrix. Based on the COPLS estimator, two-stage generalized least squares estimators of the regression coefficients are derived. In addition, asymptotic normalities of these estimators are investigated. Simulation studies have shown that the COPLS estimator and two-stage GLS estimators are alternative competitors with more efficiency in the sense of sample mean, standard deviations and mean of the variance estimates to the existing ML estimator in finite samples. An example of application is also illustrated.     

Approaches to regression analysis with multiple measurements from individual sampling units

Application of ordinary least-squares regression to data sets which contain multiple measurements from individual sampling units produces an unbiased estimator of the parameters but a biased estimator of the covariance matrix of the parameter estimates. The present work considers a random coefficient, linear model to deal with such data sets: this model permits many senses in which multiple measurements are taken from a sampling unit, not just when it is measured at several times. Three procedures to estimate the covariance matrix of the error term of the model are considered. Given these, three procedures to estimate the parameters of the model and their covariance matrix are considered; these are ordinary least-squares, generalized least-squares, and an adjusted ordinary least-squares procedure which produces an unbiased estimator of the covariance matrix of the parameters with small samples. These various procedures are compared in simulation studies using three examples from the biological literature. The possibility of testing hypotheses about the vector of parameters is also considered. It is found that all three procedures for regression estimation produce estimators of the parameters with bias of no practical consequence, Both generalized least-squares and adjusted ordinary least-squares generally produce estimators of the covariance matrix of the parameter estimates with bias of no practical consequence, while ordinary least-squares produces a negatively biased estimator. Neither ordinary nor generalized least-squares provide satisfactory hypothesis tests of the vector of parameter estimates. It is concluded that adjusted ordinary least-squares, when applied with either of two of the procedures used to estimate the error coveriance matrix, shows promise for practical application with data sets of the nature considered here.     

Computational algorithms for the factor model

Algorithms for computing the maximum likelihood estimators and the estimated covariance matrix of the estimators of the factor model are derived. The algorithms are particularly suitable for large matrices and for samples that give zero estimates of some error variances. A method of constructing estimators for reduced models is presented. The algorithms can also be used for the multivariate errors-in-variables model with known error covariance matrix.     

Separability tests for high-dimensional,low-sample size multivariate repeated measures data

Longitudinal imaging studies have moved to the forefront of medical research due to their ability to characterize spatio-temporal features of biological structures across the lifespan. Valid inference in longitudinal imaging requires enough flexibility of the covariance model to allow reasonable fidelity to the true pattern. On the other hand, the existence of computable estimates demands a parsimonious parameterization of the covariance structure. Separable (Kronecker product) covariance models provide one such parameterization in which the spatial and temporal covariances are modeled separately. However, evaluating the validity of this parameterization in high dimensions remains a challenge. Here we provide a scientifically informed approach to assessing the adequacy of separable (Kronecker product) covariance models when the number of observations is large relative to the number of independent sampling units (sample size). We address both the general case, in which unstructured matrices are considered for each covariance model, and the structured case, which assumes a particular structure for each model. For the structured case, we focus on the situation where the within-subject correlation is believed to decrease exponentially in time and space as is common in longitudinal imaging studies. However, the provided framework equally applies to all covariance patterns used within the more general multivariate repeated measures context. Our approach provides useful guidance for high dimension, low-sample size data that preclude using standard likelihood-based tests. Longitudinal medical imaging data of caudate morphology in schizophrenia illustrate the approaches appeal.     

Efficient Estimation in Marginal Partially Linear Models for Longitudinal/Clustered Data Using Splines

Abstract.  We consider marginal semiparametric partially linear models for longitudinal/clustered data and propose an estimation procedure based on a spline approximation of the non-parametric part of the model and an extension of the parametric marginal generalized estimating equations (GEE). Our estimates of both parametric part and non-parametric part of the model have properties parallel to those of parametric GEE, that is, the estimates are efficient if the covariance structure is correctly specified and they are still consistent and asymptotically normal even if the covariance structure is misspecified. By showing that our estimate achieves the semiparametric information bound, we actually establish the efficiency of estimating the parametric part of the model in a stronger sense than what is typically considered for GEE. The semiparametric efficiency of our estimate is obtained by assuming only conditional moment restrictions instead of the strict multivariate Gaussian error assumption.     

Multivariate components of covariance model in unbalanced case

In this paper we are interested in the determination of the rank of the random effect covariance matrix in an unbalanced components of covariance model, and obtain some strongly consistent estimates of the rank based on eigenstructure methods.     

Regularized covariance matrix estimation under the common principal components model

The common principal components (CPC) model provides a way to model the population covariance matrices of several groups by assuming a common eigenvector structure. When appropriate, this model can provide covariance matrix estimators of which the elements have smaller standard errors than when using either the pooled covariance matrix or the per group unbiased sample covariance matrix estimators. In this article, a regularized CPC estimator under the assumption of a common (or partially common) eigenvector structure in the populations is proposed. After estimation of the common eigenvectors using the Flury–Gautschi (or other) algorithm, the off-diagonal elements of the nearly diagonalized covariance matrices are shrunk towards zero and multiplied with the orthogonal common eigenvector matrix to obtain the regularized CPC covariance matrix estimates. The optimal shrinkage intensity per group can be estimated using cross-validation. The efficiency of these estimators compared to the pooled and unbiased estimators is investigated in a Monte Carlo simulation study, and the regularized CPC estimator is applied to a real dataset to demonstrate the utility of the method.     

Maximum likelihood estimation of parameter structures for the Wishart distribution using constraints

Maximum likelihood estimation under constraints for estimation in the Wishart class of distributions, is considered. It provides a unified approach to estimation in a variety of problems concerning covariance matrices. Virtually all covariance structures can be translated to constraints on the covariances. This includes covariance matrices with given structure such as linearly patterned covariance matrices, covariance matrices with zeros, independent covariance matrices and structurally dependent covariance matrices. The methodology followed in this paper provides a useful and simple approach to directly obtain the exact maximum likelihood estimates. These maximum likelihood estimates are obtained via an estimation procedure for the exponential class using constraints.     

A cautionary note on generalized linear models for covariance of unbalanced longitudinal data

Missing data in longitudinal studies can create enormous challenges in data analysis when coupled with the positive-definiteness constraint on a covariance matrix. For complete balanced data, the Cholesky decomposition of a covariance matrix makes it possible to remove the positive-definiteness constraint and use a generalized linear model setup to jointly model the mean and covariance using covariates (Pourahmadi, 2000). However, this approach may not be directly applicable when the longitudinal data are unbalanced, as coherent regression models for the dependence across all times and subjects may not exist. Within the existing generalized linear model framework, we show how to overcome this and other challenges by embedding the covariance matrix of the observed data for each subject in a larger covariance matrix and employing the familiar EM algorithm to compute the maximum likelihood estimates of the parameters and their standard errors. We illustrate and assess the methodology using real data sets and simulations.     

Bayesian analysis of vector-autoregressive models with noninformative priors

In this paper, we investigate the properties of Bayes estimators of vector autoregression (VAR) coefficients and the covariance matrix under two commonly employed loss functions. We point out that the posterior mean of the variances of the VAR errors under the Jeffreys prior is likely to have an over-estimation bias. Our Bayesian computation results indicate that estimates using the constant prior on the VAR regression coefficients and the reference prior of Yang and Berger (Ann. Statist. 22 (1994) 1195) on the covariance matrix dominate the constant-Jeffreys prior estimates commonly used in applications of VAR models in macroeconomics. We also estimate a VAR model of consumption growth using both constant-reference and constant-Jeffreys priors.     

Polynomial estimation of eigenvalues

In estimating the eigenvalues of the covariance matrix of a multivariate normal population, the usual estimates are the eigenvalues of the sample covariance matrix. It is well known that these estimates are biased. This paper investigates obtaining improved eigenvalue estimates through improved estimates of the characteristic polynomial, which is a function of the sample eigenvalues. A numerical study investigates the improvements evaluated under both a square error and an entropy loss function.     

Quasi-likelihood estimation for GLM with random scales

This paper uses random scales similar to random effects used in the generalized linear mixed models to describe “inter-location” population variation in variance components for modeling complicated data obtained from applications such as antenna manufacturing. Our distribution studies lead to a complicated integrated extended quasi-likelihood (IEQL) for parameter estimations and large sample inference derivations. Laplace's expansion and several approximation methods are employed to simplify the IEQL estimation procedures. Asymptotic properties of the approximate IEQL estimates are derived for general structures of the covariance matrix of random scales. Focusing on a few special covariance structures in simpler forms, the authors further simplify IEQL estimates such that typically used software tools such as weighted regression can compute the estimates easily. Moreover, these special cases allow us to derive interesting asymptotic results in much more compact expressions. Finally, numerical simulation results show that IEQL estimates perform very well in several special cases studied.     

Dynamic Equicorrelation

A new covariance matrix estimator is proposed under the assumption that at every time period all pairwise correlations are equal. This assumption, which is pragmatically applied in various areas of finance, makes it possible to estimate arbitrarily large covariance matrices with ease. The model, called DECO, involves first adjusting for individual volatilities and then estimating correlations. A quasi-maximum likelihood result shows that DECO provides consistent parameter estimates even when the equicorrelation assumption is violated. We demonstrate how to generalize DECO to block equicorrelation structures. DECO estimates for U.S. stock return data show that (block) equicorrelated models can provide a better fit of the data than DCC. Using out-of-sample forecasts, DECO and Block DECO are shown to improve portfolio selection compared to an unrestricted dynamic correlation structure.     

Variance estimation for measures of change in probability sampling

Variance estimation of changes requires estimates of variances and covariances that would be relatively straightforward to make if the sample remained the same from one wave to the next, but this is rarely the case in practice as successive waves are usually different overlapping samples. The author proposes a design‐based estimator for covariance matrices that is adapted to this situation. Under certain conditions, he shows that his approach yields non‐negative definite estimates for covariance matrices and therefore positive variance estimates for a large class of measures of change.     

Mixed models for data from thorough QT studies: part 2. One-step assessment of conditional QT prolongation

We investigate mixed analysis of covariance models for the 'one-step' assessment of conditional QT prolongation. Initially, we consider three different covariance structures for the data, where between-treatment covariance of repeated measures is modelled respectively through random effects, random coefficients, and through a combination of random effects and random coefficients. In all three of those models, an unstructured covariance pattern is used to model within-treatment covariance. In a fourth model, proposed earlier in the literature, between-treatment covariance is modelled through random coefficients but the residuals are assumed to be independent identically distributed (i.i.d.). Finally, we consider a mixed model with saturated covariance structure. We investigate the precision and robustness of those models by fitting them to a large group of real data sets from thorough QT studies. Our findings suggest: (i) Point estimates of treatment contrasts from all five models are similar. (ii) The random coefficients model with i.i.d. residuals is not robust; the model potentially leads to both under- and overestimation of standard errors of treatment contrasts and therefore cannot be recommended for the analysis of conditional QT prolongation. (iii) The combined random effects/random coefficients model does not always converge; in the cases where it converges, its precision is generally inferior to the other models considered. (iv) Both the random effects and the random coefficients model are robust. (v) The random effects, the random coefficients, and the saturated model have similar precision and all three models are suitable for the one-step assessment of conditional QT prolongation.     

Maximum likelihood estimates in the multivariate normal with patterned mean and covariance via the em algorithm

The maximum likelihood equations for a multivariate normal model with structured mean and structured covariance matrix may not have an explicit solution. In some cases the model's error term may be decomposed as the sum of two independent error terms, each having a patterned covariance matrix, such that if one of the unobservable error terms is artificially treated as "missing data", the EM algorithm can be used to compute the maximum likelihood estimates for the original problem. Some decompositions produce likelihood equations which do not have an explicit solution at each iteration of the EM algorithm, but within-iteration explicit solutions are shown for two general classes of models including covariance component models used for analysis of longitudinal data.     

Correlation is first order independent of transformation

We show that the correlation between the estimates of two parameters is almost unchanged if they are each transformed in an arbitrary way. To be more specific, the correlation of two estimates is invariant (except for a possible sign change) up to a first order approximation, to smooth transformations of the estimates. There is a sign change if exactly one of the transformations is decreasing in a neighborhood of its parameter. In addition, we approximate the variance, covariance and correlation between functions of sample means and moments.     

The multiplier method in constrained estimation of covariance structure models

Based on the multiplier method of constrained minimization, an algorithm is developed to handle the constrained estimation problem in covariance structure analysis. In the context of a general model which has wide applicability in multivariate medical and behavioural researches, computer programs are implemented to produce the weighted least squares estimates and the maximum likelihood estimates. The multiplier method is compared with the penalty function method in terms of computer time, number of iterations and number of unconstrained minimizations. The indication is that the multiplier method is substantially better.     

Comparative GMM and GQL logistic regression models on hierarchical data

We often rely on the likelihood to obtain estimates of regression parameters but it is not readily available for generalized linear mixed models (GLMMs). Inferences for the regression coefficients and the covariance parameters are key in these models. We presented alternative approaches for analyzing binary data from a hierarchical structure that do not rely on any distributional assumptions: a generalized quasi-likelihood (GQL) approach and a generalized method of moments (GMM) approach. These are alternative approaches to the typical maximum-likelihood approximation approach in Statistical Analysis System (SAS) such as Laplace approximation (LAP). We examined and compared the performance of GQL and GMM approaches with multiple random effects to the LAP approach as used in PROC GLIMMIX, SAS. The GQL approach tends to produce unbiased estimates, whereas the LAP approach can lead to highly biased estimates for certain scenarios. The GQL approach produces more accurate estimates on both the regression coefficients and the covariance parameters with smaller standard errors as compared to the GMM approach. We found that both GQL and GMM approaches are less likely to result in non-convergence as opposed to the LAP approach. A simulation study was conducted and a numerical example was presented for illustrative purposes.     

Effects of the estimation of covariance matrix parameters in the generalized multivariate linear model

We Consider the generalized multivariate linear model and assume the covariance matrix of the p x 1 vector of responses on a given individual can be represented in the general linear structure form described by Anderson (1973). The effects of the use of estimates of the parameters of the covariance matrix on the generalized least squares estimator of the regression coefficients and on the prediction of a portion of a future vector, when only the first portion of the vector has been observed, are investigated. Approximations are derived for the covariance matrix of the generalized least squares estimator and for the mean square error matrix of the usual predictor, for the practical case where estimated parameters are used.