Generalized least squares with misspecified serial correlation structures

Summary. The regression literature contains hundreds of studies on serially correlated disturbances. Most of these studies assume that the structure of the error covariance matrix Ω is known or can be estimated consistently from data. Surprisingly, few studies investigate the properties of estimated generalized least squares (GLS) procedures when the structure of Ω is incorrectly identified and the parameters are inefficiently estimated. We compare the finite sample efficiencies of ordinary least squares (OLS), GLS and incorrect GLS (IGLS) estimators. We also prove new theorems establishing theoretical efficiency bounds for IGLS relative to GLS and OLS. Results from an exhaustive simulation study are used to evaluate the finite sample performance and to demonstrate the robustness of IGLS estimates vis-à-vis OLS and GLS estimates constructed for models with known and estimated (but correctly identified) Ω. Some of our conclusions for finite samples differ from established asymptotic results.     

Estimation of parameters in a generalized GMANOVA model based on an outer product analogy and least squares

This paper investigates estimation of parameters in a combination of the multivariate linear model and growth curve model, called a generalized GMANOVA model. Making analogy between the outer product of data vectors and covariance yields an approach to directly do least squares to covariance. An outer product least squares estimator of covariance (COPLS estimator) is obtained and its distribution is presented if a normal assumption is imposed on the error matrix. Based on the COPLS estimator, two-stage generalized least squares estimators of the regression coefficients are derived. In addition, asymptotic normalities of these estimators are investigated. Simulation studies have shown that the COPLS estimator and two-stage GLS estimators are alternative competitors with more efficiency in the sense of sample mean, standard deviations and mean of the variance estimates to the existing ML estimator in finite samples. An example of application is also illustrated.     

SEPARATE VERSUS SYSTEM METHODS OF STEIN-RULE ESTIMATION IN SEEMINGLY UNRELATED REGRESSION MODELS

ABSTRACT

Despite the sizeable literature associated with the seemingly unrelated regression models, not much is known about the use of Stein-rule estimators in these models. This gap is remedied in this paper, in which two families of Stein-rule estimators in seemingly unrelated regression equations are presented and their large sample asymptotic properties explored and evaluated. One family of estimators uses a shrinkage factor obtained solely from the equation under study while the other has a shrinkage factor based on all the equations of the model. Using a quadratic loss measure and Monte-Carlo sampling experiments, the finite sample risk performance of these estimators is also evaluated and compared with the traditional feasible generalized least squares estimator.     

The unconditional distributions of the OLS,TSLS and LIML estimators in a simple structural equations model

The exact distributions of the standard estimators of the structural coe?cients in a linear structural equations model conditional on the exogenous variables have been shown to have some unexpected and quirky features. Since the argument for conditioning on exogenous (ancillary) variables has been weakened over the past 20 years by the discovery of an “ancillarity paradox,” it is natural to wonder whether such finite sample properties are in fact due to conditioning on the exogenous variables. This article studies the exact distributions of the ordinary least squares (OLS), two-stage least squares (TSLS), and limited information maximum likelihood (LIML) estimators of the structural coe?cients in a linear structural equation without conditioning on the exogenous variables.     

Estimation of parameters in heavy-tailed distribution when its second order tail parameter is known

Estimating parameters in heavy-tailed distribution plays a central role in extreme value theory. It is well known that classical estimators based on the first order asymptotics such as the Hill, rank-based and QQ estimators are seriously biased under finer second order regular variation framework. To reduce the bias, many authors proposed the so-called second order reduced bias estimators for both first and second order tail parameters. In this work, estimation of parameters in heavy-tailed distributions are studied under the second order regular variation framework when the second order parameter in the distribution tail is known. This is motivated in large part by a recent work by the authors showing that the second order tail parameter is known for a large class of popular random difference equations (for example, ARCH models). The focus is on least squares estimators that generalize rank-based and QQ estimators. Though other possible estimators are also briefly discussed, the least squares estimators are most simple to use and perform best for finite samples in Monte Carlo simulations.     

Least squares variogram fitting by spatial subsampling

Summary. Least squares methods are popular for fitting valid variogram models to spatial data. The paper proposes a new least squares method based on spatial subsampling for variogram model fitting. We show that the method proposed is statistically efficient among a class of least squares methods, including the generalized least squares method. Further, it is computationally much simpler than the generalized least squares method. The method produces valid variogram estimators under very mild regularity conditions on the underlying random field and may be applied with different choices of the generic variogram estimator without analytical calculation. An extension of the method proposed to a class of spatial regression models is illustrated with a real data example. Results from a simulation study on finite sample properties of the method are also reported.     

Inference under Heteroscedasticity of Unknown Form Using an Adaptive Estimator

The heteroscedasticity consistent covariance matrix estimators are commonly used for the testing of regression coefficients when error terms of regression model are heteroscedastic. These estimators are based on the residuals obtained from the method of ordinary least squares and this method yields inefficient estimators in the presence of heteroscedasticity. It is usual practice to use estimated weighted least squares method or some adaptive methods to find efficient estimates of the regression parameters when the form of heteroscedasticity is unknown. But HCCM estimators are seldom derived from such efficient estimators for testing purposes in the available literature. The current article addresses the same concern and presents the weighted versions of HCCM estimators. Our numerical work uncovers the performance of these estimators and their finite sample properties in terms of interval estimation and null rejection rate.     

The Risk of James–Stein and Lasso Shrinkage

This article compares the mean-squared error (or ?₂ risk) of ordinary least squares (OLS), James–Stein, and least absolute shrinkage and selection operator (Lasso) shrinkage estimators in simple linear regression where the number of regressors is smaller than the sample size. We compare and contrast the known risk bounds for these estimators, which shows that neither James–Stein nor Lasso uniformly dominates the other. We investigate the finite sample risk using a simple simulation experiment. We find that the risk of Lasso estimation is particularly sensitive to coefficient parameterization, and for a significant portion of the parameter space Lasso has higher mean-squared error than OLS. This investigation suggests that there are potential pitfalls arising with Lasso estimation, and simulation studies need to be more attentive to careful exploration of the parameter space.     

Fractional principal components regression: a general approach to biased estimators

Several biased estimators have been proposed as alternatives to the least squares estimator when multicollinearity is present in the multiple linear regression model. The ridge estimator and the principal components estimator are two techniques that have been proposed for such problems. In this paper the class of fractional principal component estimators is developed for the multiple linear regression model. This class contains many of the biased estimators commonly used to combat multicollinearity. In the fractional principal components framework, two new estimation techniques are introduced. The theoretical performances of the new estimators are evaluated and their small sample properties are compared via simulation with the ridge, generalized ridge and principal components estimators     

An efficient and fast algorithm for estimating the parameters of two-dimensional sinusoidal signals

In this paper we propose a computationally efficient algorithm to estimate the parameters of a 2-D sinusoidal model in the presence of stationary noise. The estimators obtained by the proposed algorithm are consistent and asymptotically equivalent to the least squares estimators. Monte Carlo simulations are performed for different sample sizes and it is observed that the performances of the proposed method are quite satisfactory and they are equivalent to the least squares estimators. The main advantage of the proposed method is that the estimators can be obtained using only finite number of iterations. In fact it is shown that starting from the average of periodogram estimators, the proposed algorithm converges in three steps only. One synthesized texture data and one original texture data have been analyzed using the proposed algorithm for illustrative purpose.     

Estimation of parameters of two-dimensional sinusoidal signal in heavy-tailed errors

In this paper, we consider a two-dimensional sinusoidal model observed in an additive random field. The proposed model has wide applications in statistical signal processing. The additive noise has mean zero but the variance may not be finite. We propose the least squares estimators to estimate the unknown parameters. It is observed that the least squares estimators are strongly consistent. We obtain the asymptotic distribution of the least squares estimators under the assumption that the additive errors are from a symmetric stable distribution. Some numerical experiments are performed to see how the results work for finite samples.     

A comparison of LS/ML and GMM estimation in a simple AR(1) model

This paper compares least squares (LS)/maximum likelihood (ML) and generalised method of moments (GMM) estimation in a simple. Gaussian autoregressive of order one (AR(1)) model. First, we show that the usual LS/ML estimator is a corner solution to a general minimisation problem that involves two moment conditions, while the new GMM we devise is not. Secondly, we examine asymptotic and finite sample properties of the new GMM estimator in comparison to the usual LS/ML estimator in a simple AR(1) model. For both stable and unstable (unit root) specifications, we show asymptotic equivalence of the distributions of the two estimators. However, in finite samples, the new GMM estimator performs better.     

Estimating the error variance after a pre-test for an inequality restriction on the coefficients

Estimation of the regression error variance after a preliminary test of an inequality constraint on the coefficient vector is considered. We derive the exact finite sample risk of several inequality restricted and pre-test estimators of σ². These estimators are associated with the maximum likelihood, least squares and minimum mean squared error component estimators. Optimal critical values for the pre-test according to a mini-max regret criterion are numerically calculated. Furthermore, we examine the robustness of the optimal choice of critical values and the risk properties of the estimators of σ² to model mis-specification through the exclusion of relevant regressors.     

Least squares estimation of varying‐coefficient hazard regression with application to breast cancer dose‐intensity data

To enhance modeling flexibility, the authors propose a nonparametric hazard regression model, for which the ordinary and weighted least squares estimation and inference procedures are studied. The proposed model does not assume any parametric specifications on the covariate effects, which is suitable for exploring the nonlinear interactions between covariates, time and some exposure variable. The authors propose the local ordinary and weighted least squares estimators for the varying‐coefficient functions and establish the corresponding asymptotic normality properties. Simulation studies are conducted to empirically examine the finite‐sample performance of the new methods, and a real data example from a recent breast cancer study is used as an illustration. The Canadian Journal of Statistics 37: 659–674; 2009 © 2009 Statistical Society of Canada     

Heteroskedasticity-consistent interval estimators

The linear regression model is commonly used in applications. One of the assumptions made is that the error variances are constant across all observations. This assumption, known as homoskedasticity, is frequently violated in practice. A commonly used strategy is to estimate the regression parameters by ordinary least squares and to compute standard errors that deliver asymptotically valid inference under both homoskedasticity and heteroskedasticity of an unknown form. Several consistent standard errors have been proposed in the literature, and evaluated in numerical experiments based on their point estimation performance and on the finite sample behaviour of associated hypothesis tests. We build upon the existing literature by constructing heteroskedasticity-consistent interval estimators and numerically evaluating their finite sample performance. Different bootstrap interval estimators are also considered. The numerical results favour the HC4 interval estimator.     

On robust forecasting in dynamic vector time series models

In this article, robust estimation and prediction in multivariate autoregressive models with exogenous variables (VARX) are considered. The conditional least squares (CLS) estimators are known to be non-robust when outliers occur. To obtain robust estimators, the method introduced in Duchesne [2005. Robust and powerful serial correlation tests with new robust estimates in ARX models. J. Time Ser. Anal. 26, 49–81] and Bou Hamad and Duchesne [2005. On robust diagnostics at individual lags using RA-ARX estimators. In: Duchesne, P., Rémillard, B. (Eds.), Statistical Modeling and Analysis for Complex Data Problems. Springer, New York] is generalized for VARX models. The asymptotic distribution of the new estimators is studied and from this is obtained in particular the asymptotic covariance matrix of the robust estimators. Classical conditional prediction intervals normally rely on estimators such as the usual non-robust CLS estimators. In the presence of outliers, such as additive outliers, these classical predictions can be severely biased. More generally, the occurrence of outliers may invalidate the usual conditional prediction intervals. Consequently, the new robust methodology is used to develop robust conditional prediction intervals which take into account parameter estimation uncertainty. In a simulation study, we investigate the finite sample properties of the robust prediction intervals under several scenarios for the occurrence of the outliers, and the new intervals are compared to non-robust intervals based on classical CLS estimators.     

Reweighting approximate GM estimators: Asymptotics and residual-based graphics

The iterative weighted least squares algorithm is handy for solving generalized estimating equations. In some situations it may be desirable to limit the number of iterations to a fixed finite number, for instance, to keep the breakdown point under control. Such a scheme is called reweighting. Usually reweighting leads to a different large sample theory than full iteration, and the reweighted estimator may inherit deficiencies of the starting value. When might the reweighting scheme work? To answer this question we define a broad class of estimators, namely, approximate GM estimators, and we show that reweighting leads to the same large sample theory as full iteration within this class. As an example, we provide conditions under which one-step Newton-Raphson estimators are approximate GM estimators. We then use the reweighting to construct residual-based graphics for approximate GM estimates, adapting weighted residual plots that have been proposed previously, and developing new plots to provide complementary views of the data.     

Estimation of dynamic panel data models with both individual and time-specific effects

This paper proposes a generalized least squares and a generalized method of moment estimators for dynamic panel data models with both individual-specific and time-specific effects. We also demonstrate that the common estimators ignoring the presence of time-specific effects are inconsistent when N→∞$N\to \infty$ but T is finite if the time-specific effects are indeed present. Monte Carlo studies are also conducted to investigate the finite sample properties of various estimators. It is found that the generalized least squares estimator has the smallest bias and root mean square error, and also has nominal size close to the empirical size. It is also found that even when there is no presence of time-specific effects, there is hardly any efficiency loss of the generalized least squares estimator assuming its presence compared to the generalized least squares estimator allowing only the presence of individual-specific effects.     

Statistical Analysis of Parameter Estimation for 2-D Harmonics in Multiplicative and Additive Noise

This article considers the problem of parameter estimation for two dimensional (2-D) multi-component harmonics in non zero-mean multiplicative and additive noise. The least squares estimators (LSEs) are proposed to estimate the coherent model parameters, and some statistical results of the LSEs are obtained, including strong consistency, strong convergence rate, and asymptotic normality. Furthermore, the LSEs-based estimators are proposed to estimate the noncoherent model parameters, and the strong consistency and the asymptotic normality are also proved. Finally, some numerical experiments are performed to see how the asymptotic results work for finite sample sizes.     

Asymptotic Inefficiency of Mean-Correction on Parameter Estimation for a Periodic First-Order Autoregressive Model

A common practice in time series analysis is to fit a centered model to the mean-corrected data set. For stationary autoregressive moving-average (ARMA) processes, as far as the parameter estimation is concerned, fitting an ARMA model without intercepts to the mean-corrected series is asymptotically equivalent to fitting an ARMA model with intercepts to the observed series. We show that, related to the parameter least squares estimation of periodic ARMA models, the second approach can be arbitrarily more efficient than the mean-corrected counterpart. This property is illustrated by means of a periodic first-order autoregressive model. The asymptotic variance of the estimators for both approaches is derived. Moreover, empirical experiments based on simulations investigate the finite sample properties of the estimators.