The covariance matrix of normal order statistics

An approximation is given to calculate V, the covariance matrix for normal order statistics. The approximation gives considerable improvement over previous approximations, and the computing algorithm is available from the authors.     

Linear model estimation errors with estimated design parameters

The problem of error estimation of parameters b in a linear model,Y = Xb+ e, is considered when the elements of the design matrix X are functions of an unknown ‘design’ parameter vector c. An estimated value c is substituted in X to obtain a derived design matrix [Xtilde]. Even though the usual linear model conditions are not satisfied with [Xtilde], there are situations in physical applications where the least squares solution to the parameters is used without concern for the magnitude of the resulting error. Such a solution can suffer from serious errors.

This paper examines bias and covariance errors of such estimators. Using a first-order Taylor series expansion, we derive approximations to the bias and covariance matrix of the estimated parameters. The bias approximation is a sum of two terms:One is due to the dependence between ? and Y; the other is due to the estimation errors of ? and is proportional to b, the parameter being estimated. The covariance matrix approximation, on the other hand, is composed of three omponents:One component is due to the dependence between ? and Y; the second is the covariance matrix ∑_b corresponding to the minimum variance unbiased b, as if the design parameters were known without error; and the third is an additional component due to the errors in the design parameters. It is shown that the third error component is directly proportional to bb'. Thus, estimation of large parameters with wrong design matrix [Xtilde] will have larger errors of estimation. The results are illustrated with a simple linear example.     

Approximate Small-Sample Tests of Fixed Effects in Nonlinear Mixed Models

Nonlinear mixed effect models have been studied extensively over several decades, particularly in pharmacokinetic and pharmacodynamic applications. Here, we focus on investigating the performance of commonly applied tests of linear hypotheses about the fixed effect parameters under different approximations to the likelihood function and to the estimated covariance matrix of the estimators. Included are the first-order approximation (FIRO), first-order conditional approximation (FOCE), and Gaussian quadrature approximation (AGQ) estimation methods. There is no straightforward way to mimic the approximations and adjustments taken in linear mixed models, such as the Kackar–Harville–Jeske–Kenward–Roger approach. By simulations, we illustrate the accuracy of p-values for the tests considered here. The observed results indicate that FOCE and AGQ estimation methods outperform FIRO. The test with an adjustment coefficient that takes into consideration the number of sampling units and the number of fixed effect parameters (Gallant-type) seems to perform closest to desirable even for small-sample sizes.     

Covariance structure approximation via gLasso in high-dimensional supervised classification

Recent work has shown that the Lasso-based regularization is very useful for estimating the high-dimensional inverse covariance matrix. A particularly useful scheme is based on penalizing the ?₁ norm of the off-diagonal elements to encourage sparsity. We embed this type of regularization into high-dimensional classification. A two-stage estimation procedure is proposed which first recovers structural zeros of the inverse covariance matrix and then enforces block sparsity by moving non-zeros closer to the main diagonal. We show that the block-diagonal approximation of the inverse covariance matrix leads to an additive classifier, and demonstrate that accounting for the structure can yield better performance accuracy. Effect of the block size on classification is explored, and a class of asymptotically equivalent structure approximations in a high-dimensional setting is specified. We suggest a variable selection at the block level and investigate properties of this procedure in growing dimension asymptotics. We present a consistency result on the feature selection procedure, establish asymptotic lower an upper bounds for the fraction of separative blocks and specify constraints under which the reliable classification with block-wise feature selection can be performed. The relevance and benefits of the proposed approach are illustrated on both simulated and real data.     

Spectral and circulant approximations to the likelihood for stationary Gaussian random fields

Consider a Gaussian random field model on , observed on a rectangular region. Suppose it is desired to estimate a set of parameters in the covariance function. Spectral and circulant approximations to the likelihood are often used to facilitate estimation of the parameters. The purpose of the paper is to give a careful treatment of the quality of these approximations. A spectral approximation for the likelihood was given by Guyon (Biometrika 69 (1982) 95–105) but without proof. The results given here generalize those of Guyon, and fill in the details of the proof. In addition some matrix results are derived which may be of independent interest. Applications are made to Fisher information and bias calculations for maximum likelihood estimates.     

Further theoretical results and a comparison between two methods for approximating eigenvalues of perturbed covariance matrices

Covariance matrices, or in general matrices of sums of squares and cross-products, are used as input in many multivariate analyses techniques. The eigenvalues of these matrices play an important role in the statistical analysis of data including estimation and hypotheses testing. It has been recognized that one or few observations can exert an undue influence on the eigenvalues of a covariance matrix. The relationship between the eigenvalues of the covariance matrix computed from all data and the eigenvalues of the perturbed covariance matrix (a covariance matrix computed after a small subset of the observations has been deleted) cannot in general be written in closed-form. Two methods for approximating the eigenvalues of a perturbed covariance matrix have been suggested by Hadi (1988) and Wang and Nyquist (1991) for the case of a perturbation by a single observation. In this paper we improve on these two methods and give some additional theoretical results that may give further insight into the problem. We also compare the two improved approximations in terms of their accuracies.     

A Monte Carlo approach to quantifying discrepancies between intractable posterior distributions

The computational demand required to perform inference using Markov chain Monte Carlo methods often obstructs a Bayesian analysis. This may be a result of large datasets, complex dependence structures, or expensive computer models. In these instances, the posterior distribution is replaced by a computationally tractable approximation, and inference is based on this working model. However, the error that is introduced by this practice is not well studied. In this paper, we propose a methodology that allows one to examine the impact on statistical inference by quantifying the discrepancy between the intractable and working posterior distributions. This work provides a structure to analyse model approximations with regard to the reliability of inference and computational efficiency. We illustrate our approach through a spatial analysis of yearly total precipitation anomalies where covariance tapering approximations are used to alleviate the computational demand associated with inverting a large, dense covariance matrix.     

General saddlepoint approximations to distributions under an elliptical population

General saddlepoint approximations are derived for the distributions of statistics under an elliptical population. The technique is applied to obtain the tail probabilities of latent roots of a sample covariance matrix. It is shown that the method based on normalizing transformations by Tsuchiya and Konishi (1997) is efficient for the sample correlation coefficient in an elliptical sample.     

a bayesian analysis of the multivariate mixed-linear model

The Bayesian analysis of the multivariate mixed linear model is considered. The exact posterior distribution for the fixed effects matrix and the error covariance matrix are obtained. The exact posterior means and variances of the Bayesian estimators for the covariance matrices of random effects are also derived. These posterior moments are computed without constrained optimization and numerical integration. The calculations are feasible for arbitrary models. Reasonable approximations for the posterior distributions for the covariance matrices associated with the random effects are obtained also. Results are illustrated with a numerical example.     

Building adaptive estimating equations when inverse of covariance estimation is difficult

Summary. To construct an optimal estimating function by weighting a set of score functions, we must either know or estimate consistently the covariance matrix for the individual scores. In problems with high dimensional correlated data the estimated covariance matrix could be unreliable. The smallest eigenvalues of the covariance matrix will be the most important for weighting the estimating equations, but in high dimensions these will be poorly determined. Generalized estimating equations introduced the idea of a working correlation to minimize such problems. However, it can be difficult to specify the working correlation model correctly. We develop an adaptive estimating equation method which requires no working correlation assumptions. This methodology relies on finding a reliable approximation to the inverse of the variance matrix in the quasi-likelihood equations. We apply a multivariate generalization of the conjugate gradient method to find estimating equations that preserve the information well at fixed low dimensions. This approach is particularly useful when the estimator of the covariance matrix is singular or close to singular, or impossible to invert owing to its large size.     

Accurate computation of single and product moments of order statistics under progressive censoring

An accurate procedure is proposed to calculate approximate moments of progressive order statistics in the context of statistical inference for lifetime models. The study analyses the performance of power series expansion to approximate the moments for location and scale distributions with high precision and smaller deviations with respect to the exact values. A comparative analysis between exact and approximate methods is shown using some tables and figures. The different approximations are applied in two situations. First, we consider the problem of computing the large sample variance–covariance matrix of maximum likelihood estimators. We also use the approximations to obtain progressively censored sampling plans for log-normal distributed data. These problems illustrate that the presented procedure is highly useful to compute the moments with precision for numerous censoring patterns and, in many cases, is the only valid method because the exact calculation may not be applicable.     

Error analysis of spatial representation and estimation of mobile robots

The problem of estimating the location of a mobile robot in an unstructured environment is discussed. This work extends earlier results in two important ways. First, the bias and variance of the estimation are analytically derived as functions of the angular error and distance between frames. Second, the uncertainty covariance matrix is derived and is compared to the first-order approximation previously used to estimate the result of compounding uncertain transformations to provide a framework in which the appropriateness of the first-order estimate can be formally studied. A simulation study, showing how the biases and expected distance between the estimate and true position of the robot vary as a function of measurement errors and different path plannings, is presented. Some possible improvements of the estimation method and future research topics are also given.     

Supervised classifiers for high-dimensional higher-order data with locally doubly exchangeable covariance structure

We explore the performance accuracy of the linear and quadratic classifiers for high-dimensional higher-order data, assuming that the class conditional distributions are multivariate normal with locally doubly exchangeable covariance structure. We derive a two-stage procedure for estimating the covariance matrix: at the first stage, the Lasso-based structure learning is applied to sparsifying the block components within the covariance matrix. At the second stage, the maximum-likelihood estimators of all block-wise parameters are derived assuming the doubly exchangeable within block covariance structure and a Kronecker product structured mean vector. We also study the effect of the block size on the classification performance in the high-dimensional setting and derive a class of asymptotically equivalent block structure approximations, in a sense that the choice of the block size is asymptotically negligible.     

Fisher information and maximum-likelihood estimation of covariance parameters in Gaussian stochastic processes

The inverse of the Fisher information matrix is commonly used as an approximation for the covariance matrix of maximum-likelihood estimators. We show via three examples that for the covariance parameters of Gaussian stochastic processes under infill asymptotics, the covariance matrix of the limiting distribution of their maximum-likelihood estimators equals the limit of the inverse information matrix. This is either proven analytically or justified by simulation. Furthermore, the limiting behaviour of the trace of the inverse information matrix indicates equivalence or orthogonality of the underlying Gaussian measures. Even in the case of singularity, the estimator of the process variance is seen to be unbiased, and also its variability is approximated accurately from the information matrix.     

A monte carlo study on two methods of calculating the mle's covariance matrix in a seemingly unrelated nonlinear regression.

Econometric techniques to estimate output supply systems, factor demand systems and consumer demand systems have often required estimating a nonlinear system of equations that have an additive error structure when written in reduced form. To calculate the ML estimate's covariance matrix of this nonlinear system one can either invert the Hessian of the concentrated log likelihood function, or invert the matrix calculated by pre-multiplying and post multiplying the inverted MLE of the disturbance covariance matrix by the Jacobian of the reduced form model. Malinvaud has shown that the latter of these methods is the actual limiting distribution's covariance matrix, while Barnett has shown that the former is only an approximation.

In this paper, we use a Monte Carlo simulation study to determine how these two covariance matrices differ with respect to the nonlinearity of the model, the number of observations in the dataet, and the residual process. We find that the covariance matrix calculated from the Hessian of the concentrated likelihood function produces Wald statistics that are distributed above those calculated with the other covariance matrix. This difference becomes insignificant as the sample size increases to one-hundred or more observations, suggesting that the asymptotics of the two covariance matrices are quickly reached.     

Generalized Laplacian approximations in Bayesian inference

This paper presents a new Laplacian approximation to the posterior density of η = g(θ). It has a simpler analytical form than that described by Leonard et al. (1989). The approximation derived by Leonard et al. requires a conditional information matrix R_η to be positive definite for every fixed η. However, in many cases, not all R_η are positive definite. In such cases, the computations of their approximations fail, since the approximation cannot be normalized. However, the new approximation may be modified so that the corresponding conditional information matrix can be made positive definite for every fixed η. In addition, a Bayesian procedure for contingency-table model checking is provided. An example of cross-classification between the educational level of a wife and fertility-planning status of couples is used for explanation. Various Laplacian approximations are computed and compared in this example and in an example of public school expenditures in the context of Bayesian analysis of the multiparameter Fisher-Behrens problem.     

A new approximation to the distribution function of the studentized maximum root

The Studentized maximum root (SMR) distribution is useful for constructing simultaneous confidence intervals around product interaction contrasts in replicated two-way ANOVA. A three-moment approximation to the SMR distribution is proposed. The approximation requires the first three moments of the maximum root of a central Wishart matrix. These values are obtained by means of numerical integration. The accuracy of the approximation is compared to the accuracy of a two-moment approximation for selected two-way table sizes. Both approximations are reasonably accurate. The three-moment approximation is generally superior.     

A monte carlo study on two methods of calculating the mle's covariance matrix in a seemingly unrelated nonlinear regression. *

Econometric techniques to estimate output supply systems, factor demand systems and consumer demand systems have often required estimating a nonlinear system of equations that have an additive error structure when written in reduced form. To calculate the ML estimate's covariance matrix of this nonlinear system one can either invert the Hessian of the concentrated log likelihood function, or invert the matrix calculated by pre-multiplying and post multiplying the inverted MLE of the disturbance covariance matrix by the Jacobian of the reduced form model. Malinvaud has shown that the latter of these methods is the actual limiting distribution's covariance matrix, while Barnett has shown that the former is only an approximation.

In this paper, we use a Monte Carlo simulation study to determine how these two covariance matrices differ with respect to the nonlinearity of the model, the number of observations in the dataet, and the residual process. We find that the covariance matrix calculated from the Hessian of the concentrated likelihood function produces Wald statistics that are distributed above those calculated with the other covariance matrix. This difference becomes insignificant as the sample size increases to one-hundred or more observations, suggesting that the asymptotics of the two covariance matrices are quickly reached.     

Empirical distribution function under heteroscedasticity

Neglecting heteroscedasticity of error terms may imply the wrong identification of a regression model (see appendix). Employment of (heteroscedasticity resistent) White's estimator of covariance matrix of estimates of regression coefficients may lead to the correct decision about the significance of individual explanatory variables under heteroscedasticity. However, White's estimator of covariance matrix was established for least squares (LS)-regression analysis (in the case when error terms are normally distributed, LS- and maximum likelihood (ML)-analysis coincide and hence then White's estimate of covariance matrix is available for ML-regression analysis, tool). To establish White's-type estimate for another estimator of regression coefficients requires Bahadur representation of the estimator in question, under heteroscedasticity of error terms. The derivation of Bahadur representation for other (robust) estimators requires some tools. As the key too proved to be a tight approximation of the empirical distribution function (d.f.) of residuals by the theoretical d.f. of the error terms of the regression model. We need the approximation to be uniform in the argument of d.f. as well as in regression coefficients. The present paper offers this approximation for the situation when the error terms are heteroscedastic.     

Analysis of multivariate repeated measures data with a Kronecker product structured covariance matrix

In this article we consider a set of t repeated measurements on p variables (or characteristics) on each of the n individuals. Thus, data on each individual is a p ×t matrix. The n individuals themselves may be divided and randomly assigned to g groups. Analysis of these data using a MANOVA model, assuming that the data on an individual has a covariance matrix which is a Kronecker product of two positive definite matrices, is considered. The well-known Satterthwaite type approximation to the distribution of a quadratic form in normal variables is extended to the distribution of a multivariate quadratic form in multivariate normal variables. The multivariate tests using this approximation are developed for testing the usual hypotheses. Results are illustrated on a data set. A method for analysing unbalanced data is also discussed.