Efficiency of least squares estimators in the presence of spatial autocorrelation

The effect of spatial autocorrelation on inferences made using ordinary least squares estimation is considered. It is found, in some cases, that ordinary least squares estimators provide a reasonable alternative to the estimated generalized least squares estimators recommended in the spatial statistics literature. One of the most serious problems in using ordinary least squares is that the usual variance estimators are severely biased when the errors are correlated. An alternative variance estimator that adjusts for any observed correlation is proposed. The need to take autocorrelation into account in variance estimation negates much of the advantage that ordinary least squares estimation has in terms of computational simplicity     

Fractional principal components regression: a general approach to biased estimators

Several biased estimators have been proposed as alternatives to the least squares estimator when multicollinearity is present in the multiple linear regression model. The ridge estimator and the principal components estimator are two techniques that have been proposed for such problems. In this paper the class of fractional principal component estimators is developed for the multiple linear regression model. This class contains many of the biased estimators commonly used to combat multicollinearity. In the fractional principal components framework, two new estimation techniques are introduced. The theoretical performances of the new estimators are evaluated and their small sample properties are compared via simulation with the ridge, generalized ridge and principal components estimators     

A simulation study of ridge and other regression estimators

We consider a number of estimators of regression coefficients, all of generalized ridge, or 'shrinkage' type. Results of a simulation study indicate that with respect to two commonly used mean square error criteria, two ordinary ridge estimators, one proposed by Hoerl, Kennard and Baldwin, and the other introduced here, perform substantially better than both least squares and the other estimators discussed here     

Small sample properties of ridge estimators with normal and non-normal disturbances

In this paper we consider five well known and widely used ridge estimators when the convenient assumption of normality of the disturbances is abandoned and report on a Monte Carlo study of their small sample properties. The Monte Carlo experiment is applied to four different data sets with artificially varied degrees of multicollinearity, while the disturbances follow normal, lognormal, uniform and Laplace distributions with small and large variances. The results show that the best estimates are obtained for all ridge estimators when the disturbances follow the lognormal distribution. Also, none of the examined ridge estimators shows a consistent behavior under the different settings considered.     

The Kalman Filter from the Perspective of Goldberger—Theil Estimators

This article is designed to point out the close connection between recursive estimation procedures, such as Kalman filter theory, familiar to control engineers, and linear least squares estimators and estimators that include prior information in the form of linear restrictions, such as mixed estimators and ridge estimators, familiar to statisticians. The only difference between the two points of view seems to be a difference in terminology. To demonstrate this point, it is shown how the Kalman filter equations can be derived from an existing textbook account of linear least squares theory and the notion of combining prior information in linear models, that is, the Goldberger—Theil mixed estimators' point of view. The author advocates the inclusion of these ideas early when least squares estimation concepts are being taught.     

A simulation study of deterministic ridge estimators

Some deterministic ridge rules are proposed and their finite sample properties are studied. Further, a simulation study is also conducted. Based on the simulation results, the proposed ridge estimators can improve the mean squared error over the least squares estimator, provided that the condition number of correlation matrices in the regression model is large, say at least 1,000.     

An evaluation of biased estimators of regression coefficients—A simulation study

Different versions of generalized and ordinary ridge estimators and shrinkage estimators of regression coefficients are studied in comparison with least squares estimators using simulations. The results show that some of the biased estimators considered are better than the least squares estimator in general and the improvement is substantial in some cases.     

ROBUST RIDGE REGRESSION BASED ON AN M-ESTIMATOR

Consider the linear regression model y =β₀1 +Xβ+ in the usual notation. It is argued that the class of ordinary ridge estimators obtained by shrinking the least squares estimator by the matrix (X¹X + kI)^-1X'X is sensitive to outliers in the ^variable. To overcome this problem, we propose a new class of ridge-type M-estimators, obtained by shrinking an M-estimator (instead of the least squares estimator) by the same matrix. Since the optimal value of the ridge parameter k is unknown, we suggest a procedure for choosing it adaptively. In a reasonably large scale simulation study with a particular M-estimator, we found that if the conditions are such that the M-estimator is more efficient than the least squares estimator then the corresponding ridge-type M-estimator proposed here is better, in terms of a Mean Squared Error criteria, than the ordinary ridge estimator with k chosen suitably. An example illustrates that the estimators proposed here are less sensitive to outliers in the y-variable than ordinary ridge estimators.     

A Combined Nonlinear Programming Model and Kibria Method for Choosing Ridge Parameter Regression

Ridge regression solves multicollinearity problems by introducing a biasing parameter that is called ridge parameter; it shrinks the estimates as well as their standard errors in order to reach acceptable results. Many methods are available for estimating a ridge parameter. This article has considered some of these methods and also proposed a combined nonlinear programming model and Kibria method. A simulation study has been made to evaluate the performance of the proposed estimators based on the minimum mean squared error criterion. The simulation study indicates that under certain conditions the proposed estimators outperform the least squares (LS) estimators and other popular existing estimators. Moreover, the new proposed model is applied on dataset that suffers also from the presence of heteroscedastic errors.     

Ridge Regression – A Simulation Study

In this article we assess the suitability of two new ridge estimators by means of a simulation study. We compare these estimators with well-known ridge estimators. We also make direct comparisons between the ordinary least squares (OLS) estimator and the ridge estimators by using ratio of the average total mean square error of the OLS estimator and the ridge estimators. We find that the new estimators perform well under certain conditions.     

Robust Estimation for Parameters of the Extended Burr Type III Distribution

We consider various robust estimators for the extended Burr Type III (EBIII) distribution for complete data with outliers. The considered robust estimators are M-estimators, least absolute deviations, Theil, Siegel's repeated median, least trimmed squares, and least median of squares. Before we perform the aforementioned estimators for the EBIII, we adapt the quantiles method to the estimation of the shape parameter k of the EBIII. The simulation results show that the considered robust estimators generally outperform the existing estimation approaches for data with upper outliers, with certain of them retaining a relatively high degree of efficiency for small sample sizes.     

Using Heteroscedasticity-Consistent Standard Errors for the Linear Regression Model with Correlated Regressors

The use of heteroscedasticity-consistent covariance matrix (HCCM) estimators is very common in practice to draw correct inference for the coefficients of a linear regression model with heteroscedastic errors. However, in addition to the problem of heteroscedasticity, linear regression models may also be plagued with some considerable degree of collinearity among the regressors when two or more regressors are considered. This situation causes many adverse effects on the least squares measures and alternatively, the ordinary ridge regression method is used as a common practice. But in the available literature, the problems of multicollinearity and heteroscedasticity have not been discussed as a combined issue especially, for the inference of the regression coefficients. The present article addresses the inference about the regression coefficients taking both the issues of multicollinearity and heteroscedasticity into account and suggests the use of HCCM estimators for the ridge regression. This article proposes t- and F-tests, based on these HCCM estimators, that perform adequately well in the numerical evaluation of the Monte Carlo simulations.     

Relative squared error prediction in the generalized linear regression model

In linear regression models, predictors based on least squares or on generalized least squares estimators are usually applied which, however, fail in case of multicollinearity. As an alternative biased estimators like ridge estimators, Kuks-Olman estimators, Bayes or minimax estimators are sometimes suggested. In our analysis the relative instead of the generally used absolute squared error enters the objective function. An explicit minimax solution is derived which, in an important special case, can be viewed as a predictor based on a Kuks-Olman estimator.     

A new estimator combining the ridge regression and the restricted least squares methods of estimation

It is well-known in the literature on multicollinearity that one of the major consequences of multicollinearity on the ordinary least squares estimator is that the estimator produces large sampling variances, which in turn might inappropriately lead to exclusion of otherwise significant coefficients from the model. To circumvent this problem, two accepted estimation procedures which are often suggested are the restricted least squares method and the ridge regression method. While the former leads to a reduction in the sampling variance of the estimator, the later ensures a smaller mean square error value for the estimator. In this paper we have proposed a new estimator which is based on a criterion that combines the ideas underlying these two estimators. The standard properties of this new estimator have been studied in the paper. It has also been shown that this estimator is superior to both the restricted least squares as well as the ordinary ridge regression estimators by the criterion of mean sauare error of the estimator of the regression coefficients when the restrictions are indeed correct. The conditions for superiority of this estimator over the other two have also been derived for the situation when the restrictions are not correct.     

Admissibility of linear estimators with respect to restricted parameter sets

In this paper the admissibility of a linear estimator for a linear regression parameter is characterized for such cases, where the considered parameter varies in an ellipsoid. We obtain a certain subset of the set of all linear estimators which are admissible with respect to the unrestricted parameter set. Furthermore, various linear estimators which have been proposed for improving the least squares estimator in cases of a restricted parameter set are investigated for admissibility. It turns out that only some shrunken estimators and some estimators of ridge type are admissible, whereas the KUKS-OLMAN estimator and all estimators of MARQUARDT type can be improved.     

Simulation study of new estimators combining the SUR ridge regression and the restricted least squares methodologies

In this paper, we propose two SUR type estimators based on combining the SUR ridge regression and the restricted least squares methods. In the sequel these estimators are designated as the restricted ridge Liu estimator and the restricted ridge HK estimator (see Liu in Commun Statist Thoery Methods 22(2):393–402, 1993; Sarkar in Commun Statist A 21:1987–2000, 1992). The study has been made using Monte Carlo techniques, (1,000 replications), under certain conditions where a number of factors that may effect their performance have been varied. The performance of the proposed and some of the existing estimators are evaluated by means of the TMSE and the PR criteria. Our results indicate that the proposed SUR restricted ridge estimators based on K _SUR, K _Sratio, K _Mratio and [(K)\ddot]{\ddot{K}} produced smaller TMSE and/or PR values than the remaining estimators. In contrast with other ridge estimators, components of [(K)\ddot]{\ddot{K}} are defined in terms of the eigenvalues of X^*￠ X^*{X^{{\ast^{\prime}}} X^{\rm \ast}} and all lie in the open interval (0, 1).     

Comparisons among regression estimators under the generalized mean square error criterion

It is not always prossible to establish a preference ordering among regression estimators in terms of the generalized mean square error criterion. In the paper, we determine when it is feasible to use this criteion to couduct comparisons among ordinary least squares, principal components, ridge regression, and shrunken least squares estimators.     

A Seemingly Unrelated Nonparametric Additive Model with Autoregressive Errors

This article considers a nonparametric additive seemingly unrelated regression model with autoregressive errors, and develops estimation and inference procedures for this model. Our proposed method first estimates the unknown functions by combining polynomial spline series approximations with least squares, and then uses the fitted residuals together with the smoothly clipped absolute deviation (SCAD) penalty to identify the error structure and estimate the unknown autoregressive coefficients. Based on the polynomial spline series estimator and the fitted error structure, a two-stage local polynomial improved estimator for the unknown functions of the mean is further developed. Our procedure applies a prewhitening transformation of the dependent variable, and also takes into account the contemporaneous correlations across equations. We show that the resulting estimator possesses an oracle property, and is asymptotically more efficient than estimators that neglect the autocorrelation and/or contemporaneous correlations of errors. We investigate the small sample properties of the proposed procedure in a simulation study.     

Evaluation of the predictive performance of the Liu type estimator

This article discusses the predictive performance of the Liu type (LT) estimator compared to ordinary least squares, principal components, ridge regression, and Liu estimators. The theoretical results are illustrated by a numerical example and a region is established where the LT estimator is uniformly superior to the other mentioned estimators.     

A Comparison of Partially Adaptive and Reweighted Least Squares Estimation

The small sample performance of least median of squares, reweighted least squares, least squares, least absolute deviations, and three partially adaptive estimators are compared using Monte Carlo simulations. Two data problems are addressed in the paper: (1) data generated from non-normal error distributions and (2) contaminated data. Breakdown plots are used to investigate the sensitivity of partially adaptive estimators to data contamination relative to RLS. One partially adaptive estimator performs especially well when the errors are skewed, while another partially adaptive estimator and RLS perform particularly well when the errors are extremely leptokur-totic. In comparison with RLS, partially adaptive estimators are only moderately effective in resisting data contamination; however, they outperform least squares and least absolute deviation estimators.