Efficiency analysis of ten estimation procedures for quantitative linear models with autocorrelated errors

Many estimation procedures for quantitative linear models with autocorrelated errors have been proposed in the literature. A number of these procedures have been compared in various ways for different sample sizes and autocorrelation parameters values and for structured or random explanatory vaiables. In this paper, we revisit three situations that were considered to some extent in previous studies, by comparing ten estimation procedures: Ordinary Least Squares (OLS), Generalized Least Squares (GLS), estimated Generalized Least Squares (six procedures), Maximum Likelihood (ML), and First Differences (FD). The six estimated GLS procedures and the ML procedure differ in the way the error autocovariance matrix is estimated. The three situations can be defined as follows: Case 1, the explanatory variable x in the simple linear regression is fixed; Case 2,x is purely random; and Case 3x is first-order autoregressive. Following a theoretical presentation, the ten estimation procedures are compared in a Monte Carlo study conducted in the time domain, where the errors are first-order autoregressive in Cases 1-3. The measure of comparison for the estimation procedures is their efficiency relative to OLS. It is evaluated as a function of the time series length and the magnitude and sign of the error autocorrelation parameter. Overall, knowledge of the model of the time series process generating the errors enhances efficiency in estimated GLS. Differences in the efficiency of estimation procedures between Case 1 and Cases 2 and 3 as well as differences in efficiency among procedures in a given situation are observed and discussed.     

Ridge regression estimators in the linear regression models with non-spherical errors

The present paper considers a family of ordinary ridge regression estimators in the linear regression model when the disturbances covariance matrix depends upon a few unknown parameters. An asymptotic expansion for the distribution of the ridge regression estimator is developed and under the quadratic loss function its asymptotic risk is compared with that of the feasible GLS estimator.     

Modeling of covariance structures of random effects and random errors in linear mixed models

Abstract

In this paper, we discuss how to model the mean and covariancestructures in linear mixed models (LMMs) simultaneously. We propose a data-driven method to modelcovariance structures of the random effects and random errors in the LMMs. Parameter estimation in the mean and covariances is considered by using EM algorithm, and standard errors of the parameter estimates are calculated through Louis’ (1982 Louis, T.A. (1982). Finding observed information using the EM algorithm. J. Royal Stat. Soc. B 44:98–130. [Google Scholar]) information principle. Kenward’s (1987 Kenward, M.G. (1987). A method for comparing profiles of repeated measurements. Appl. Stat. 36:296–308.[Crossref], [Web of Science ®] , [Google Scholar]) cattle data sets are analyzed for illustration,and comparison to the literature work is made through simulation studies. Our numerical analysis confirms the superiority of the proposed method to existing approaches in terms of Akaike information criterion.     

Ridge regression methodology in partial linear models with correlated errors

In this article, a generalized restricted difference-based ridge estimator is defined for the vector parameter in a partial linear model when the errors are dependent. It is suspected that some additional linear constraints may hold on to the whole parameter space. The estimator is a generalization of the well-known restricted least-squares estimator and is confined to the (affine) subspace which is generated by the restrictions. The risk functions of the proposed estimators are derived under balanced loss function. Finally, the performance of the new estimators is evaluated by a simulated data set.     

Asymptotic results for estimation and testing variances in regression models

Usually the variance of independent observations resulting from a linear or a nonlinear relationship is estimated by the Least-Squares residual estimator. In this paper its asymptotic properties are investigated. Further the asymptotic behaviour of tests for one-sided hypotheses on the variance is studied. The paper splits into two parts, the first one concerned with linear and the second one with nonlinear models.     

The strong consistency of M-estimates in linear models with extended negatively dependent errors

In this paper, we first establish the strong convergence for weighted sums of extended negatively dependent (END) random variables. Based on the strong convergence and Bernstein inequality, we obtain the strong consistency of M-estimates of the regression parameters in a linear model for END random errors under some mild moment conditions. The results generalize and improve the ones obtained in the literature to the case of END random errors.     

Missing data in linear models with correlated errors

One common method for analyzing data in experimental designs when observations are missing was devised by Yates (1933), who developed his procedure based upon a suggestion by R. A. Fisher. Considering a linear model with independent, equi-variate errors, Yates substituted algebraic values for the missing data and then minimized the error sum of squares with respect to both the unknown parameters and the algebraic values. Yates showed that this procedure yielded the correct error sum of squares and a positively biased hypothesis sum of squares.

Others have elaborated on this technique. Chakrabarti (1962) gave a formal proof of Fisher's rule that produced a way to simplify the calculations of the auxiliary values to be used in place of the missing observations. Kshirsagar (1971) proved that the hypothesis sum of squares based on these values was biased, and developed an easy way to compute that bias. Sclove     

A new test for random effects in linear mixed models with longitudinal data

As it is known, testing the existence of random effects is often transferred to testing their zero variances/covariance matrices. It is a nonstandard testing problem because the hypothetical values are on the boundary of the whole space. In the literature, a difference-based test was proposed, which has asymptotically tractable null distribution and is then easy to implement. However, the projection method on which the difference-based test relies may affect and deteriorate its performance when covariates associated with fixed effects and covariates associated with random effects are highly correlated. In the paper, for linear mixed models (LMM) with longitudinal data, a new test is proposed to avoid this problem. The new test is also asymptotically distribution-free and more powerful than the difference-based test, particularly when the above correlation is high. The new test is consistent against all global alternatives and can detect local alternatives converging to the null at a rate as close as to m^−1/2$m^{-1/2}$ with m being the number of subjects. Simulations are carried out to examine the performance and a real data analysis is performed for illustration.     

Projectors and linear estimation in general linear models

The paper gives a self-contained account of minimum disper­sion linear unbiased estimation of the expectation vector in a linear model with the dispersion matrix belonging to some, rather arbitrary, set of nonnegative definite matrices. The approach to linear estimation in general linear models recommended here is a direct generalization of some ideas and results presented by Rao (1973, 19 74) for the case of a general Gauss-Markov model

A new insight into the nature of some estimation problems originaly arising in the context of a general Gauss-Markov model as well as the correspondence of results known in the literature to those obtained in the present paper for general linear models are also given. As preliminary results the theory of projectors defined by Rao (1973) is extended.     

Semi-parametric estimation of disequilibrium models

This paper presents an easy-to-compute semi-parametric (SP) method to estimate a simple disequilibrium model proposed by Fair and Jaffee (1972). The proposed approach is based on a non-parametric interpretation of the EM (Expectation and Maximization) principle (Dempster et al; 1977) and the least squares method. The simple disequilibrium model includes the demand equation, the supply equation, and the condition that only the minimum of quantity demanded and quantity supplied is observed. The method used here allows one to consistently estimate the disequilibrium model without fully specifying the distribution of error terms in both demand and supply equations. Our Monte Carlo study suggests that the proposedestimator is better than the normal maximum likelihood estimator under asymmetric error distributions. and comparable to the nlaximunl likelihood estimator under synirnetric error distributions in finite samples. Aggregate U.S. labor market data from Quandt and Rosen (1988) is used to illustrate the procedure.     

Estimation in Singular Linear Models with Stochastic Linear Restrictions and Linear Equality Restrictions

This article is concerned with the parameter estimation in a singular linear regression model with stochastic linear restrictions and linear equality restrictions simultaneously. A new estimator is introduced and it is proved that the proposed estimator is superior to the least squares estimator and singular mixed estimator in the mean squared error sense under certain conditions.     

Prediction of response values in linear regression models from replicated experiments

This paper considers the problem of prediction in a linear regression model when data sets are available from replicated experiments. Pooling the data sets for the estimation of regression parameters, we present three predictors — one arising from the least squares method and two stemming from Stein-rule method. Efficiency properties of these predictors are discussed when they are used to predict actual and average values of response variable within/outside the sample. Received: November 17, 1999; revised version: August 10, 2000     

Unbiased prediction in linear regression models with equi-correlated responses

This paper considers the problem of predicting the actual and mean values of response variable in a linear regression model with equi-correlated responses. Two such predictors are presented and their efficiency properties are studied with respect to the criterion of covariance matrix.     

Asymptotic properties of maximum quasi-likelihood estimator in quasi-likelihood non linear models with stochastic regression

In this paper, we establish the asymptotic properties of maximum quasi-likelihood estimator (MQLE) in quasi-likelihood non linear models (QLNMs) with stochastic regression under some mild regular conditions. We also investigate the existence, strong consistency, and asymptotic normality of MQLE in QLNMs with stochastic regression.     

LS estimation of periodic autoregressive models with non-Gaussian errors: a simulation study

In this article, the least squares (LS) estimates of the parameters of periodic autoregressive (PAR) models are investigated for various distributions of error terms via Monte-Carlo simulation. Beside the Gaussian distribution, this study covers the exponential, gamma, student-t, and Cauchy distributions. The estimates are compared for various distributions via bias and MSE criterion. The effect of other factors are also examined as the non-constancy of model orders, the non-constancy of the variances of seasonal white noise, the period length, and the length of the time series. The simulation results indicate that this method is in general robust for the estimation of AR parameters with respect to the distribution of error terms and other factors. However, the estimates of those parameters were, in some cases, noticeably poor for Cauchy distribution. It is also noticed that the variances of estimates of white noise variances are highly affected by the degree of skewness of the distribution of error terms.     

Posterior consistency of random effects models for binary data

In longitudinal studies or clustered designs, observations for each subject or cluster are dependent and exhibit intra-correlation. To account for this dependency, we consider Bayesian analysis for conditionally specified models, so-called generalized linear mixed model. In nonlinear mixed models, the maximum likelihood estimator of the regression coefficients is typically a function of the distribution of random effects, and so the misspecified choice of the distribution of random effects can cause bias in the estimator. To avoid the problem of the misspecification of the distribution of random effects, one can resort in nonparametric approaches. We give sufficient conditions for posterior consistency of the distribution of random effects as well as regression coefficients.     

On the correct regression function (in L2) and its applications when the dimension of the covariate vector is random

We derive the optimal regression function (i.e., the best approximation in the L₂ sense) when the vector of covariates has a random dimension. Furthermore, we consider applications of these results to problems in statistical regression and classification with missing covariates. It will be seen, perhaps surprisingly, that the correct regression function for the case with missing covariates can sometimes perform better than the usual regression function corresponding to the case with no missing covariates. This is because even if some of the covariates are missing, an indicator random variable δ$\delta$, which is always observable, and is equal to 1 if there are no missing values (and 0 otherwise), may have far more information and predictive power about the response variable Y than the missing covariates do. We also propose kernel-based procedures for estimating the correct regression function nonparametrically. As an alternative estimation procedure, we also consider the least-squares method.     

Shrinkage estimation for linear regression with ARMA errors

In this paper, we extend the modified lasso of Wang et al. (2007) to the linear regression model with autoregressive moving average (ARMA) errors. Such an extension is far from trivial because new devices need to be called for to establish the asymptotics due to the existence of the moving average component. A shrinkage procedure is proposed to simultaneously estimate the parameters and select the informative variables in the regression, autoregressive, and moving average components. We show that the resulting estimator is consistent in both parameter estimation and variable selection, and enjoys the oracle properties. To overcome the complexity in numerical computation caused by the existence of the moving average component, we propose a procedure based on a least squares approximation to implement estimation. The ordinary least squares formulation with the use of the modified lasso makes the computation very efficient. Simulation studies are conducted to evaluate the finite sample performance of the procedure. An empirical example of ground-level ozone is also provided.