A new estimator combining the ridge regression and the restricted least squares methods of estimation

It is well-known in the literature on multicollinearity that one of the major consequences of multicollinearity on the ordinary least squares estimator is that the estimator produces large sampling variances, which in turn might inappropriately lead to exclusion of otherwise significant coefficients from the model. To circumvent this problem, two accepted estimation procedures which are often suggested are the restricted least squares method and the ridge regression method. While the former leads to a reduction in the sampling variance of the estimator, the later ensures a smaller mean square error value for the estimator. In this paper we have proposed a new estimator which is based on a criterion that combines the ideas underlying these two estimators. The standard properties of this new estimator have been studied in the paper. It has also been shown that this estimator is superior to both the restricted least squares as well as the ordinary ridge regression estimators by the criterion of mean sauare error of the estimator of the regression coefficients when the restrictions are indeed correct. The conditions for superiority of this estimator over the other two have also been derived for the situation when the restrictions are not correct.     

On the equality of usual and amemiya's partially generalized least squares estimator

Equivalent conditions are derived for the equality of GLSE (generalized least squares estimator) and partially GLSE (PGLSE), the latter introduced by Amemiya (1983). By adopting a more general approach the ordinary least squares estimator (OLSE) can shown to be a special PGLSE. Furthcrmore, linearly restricted estimators proposed by Balestra (1983) are investigated in this context. To facilitate the comparison of estimators extensive use of oblique and orthogonal projectors is made.     

Trimmed least squares estimator as best trimmed linear conditional estimator for linear regression model

A class of trimmed linear conditional estimators based on regression quantiles for the linear regression model is introduced. This class serves as a robust analogue of non-robust linear unbiased estimators. Asymptotic analysis then shows that the trimmed least squares estimator based on regression quantiles ( Koenker and Bassett ( 1978 ) ) is the best in this estimator class in terms of asymptotic covariance matrices. The class of trimmed linear conditional estimators contains the Mallows-type bounded influence trimmed means ( see De Jongh et al ( 1988 ) ) and trimmed instrumental variables estimators. A large sample methodology based on trimmed instrumental variables estimator for confidence ellipsoids and hypothesis testing is also provided.     

Linearized Ridge Regression Estimator in Linear Regression

In this article, we aim to study the linearized ridge regression (LRR) estimator in a linear regression model motivated by the work of Liu (1993). The LRR estimator and the two types of generalized Liu estimators are investigated under the PRESS criterion. The method of obtaining the optimal generalized ridge regression (GRR) estimator is derived from the optimal LRR estimator. We apply the Hald data as a numerical example and then make a simulation study to show the main results. It is concluded that the idea of transforming the GRR estimator as a complicated function of the biasing parameters to a linearized version should be paid more attention in the future.     

Relative efficiency of first difference estimator in panel data regression with serially correlated error components

We consider the pooled cross-sectional and time series regression model when the disturbances follow a serially correlated one-way error components. In this context we discovered that the first difference estimator for the regression coefficients is equivalent to the generalized least squares estimator irrespective of the particular form of the regressor matrix when the disturbances are generated by a first order autoregressive process where the autocorrelation is close to unity.     

An asymptotic theory for semiparametric generalized least squares estimation in partially linear regression models

Consider a partially linear regression model with an unknown vector parameter β, an unknown functiong(·), and unknown heteroscedastic error variances. In this paper we develop an asymptotic semiparametric generalized least squares estimation theory under some weak moment conditions. These moment conditions are satisfied by many of the error distributions encountered in practice, and our theory does not require the number of replications to go to infinity.     

Least squares estimation of nonhomogeneous poisson processes

We formulate and evaluate weighted least squares (WLS) and ordinary least squares (OLS) procedures for estimating the parametric mean-value function of a nonhomogeneous Poisson process. We focus the development on processes having an exponential rate function, where the exponent may include a polynomial component or some trigonometric components. Unanticipated problems with the WLS procedure are explained by an analysis of the associated residuals. The OLS procedure is based on a square root transformation of the "detrended" event (arrival) times - that is, the fitted mean-value function evaluated at the observed event times; and under appropriate conditions, the corresponding residuals are proved to converge weakly to a normal distribution with mean 0 and variance 0.25. The results of a Monte Carlo study indicate the advantages of the OLS procedure with respect to estimation accuracy and computational efficiency.     

Semiparametric Sieve-Type Generalized Least Squares Inference

This article considers the problem of statistical inference in linear regression models with dependent errors. A sieve-type generalized least squares (GLS) procedure is proposed based on an autoregressive approximation to the generating mechanism of the errors. The asymptotic properties of the sieve-type GLS estimator are established under general conditions, including mixingale-type conditions as well as conditions which allow for long-range dependence in the stochastic regressors and/or the errors. A Monte Carlo study examines the finite-sample properties of the method for testing regression hypotheses.     

Least Trimmed Squares Estimator in the Errors-in-Variables Model

We propose a robust estimator in the errors-in-variables model using the least trimmed squares estimator. We call this estimator the orthogonal least trimmed squares (OLTS) estimator. We show that the OLTS estimator has the high breakdown point and appropriate equivariance properties. We develop an algorithm for the OLTS estimate. Simulations are performed to compare the efficiencies of the OLTS estimates with the total least squares (TLS) estimates and a numerical example is given to illustrate the effectiveness of the estimate.     

Least squares parameter estimation in multiplicative noise models

A simple multiplicative noise model with a constant signal has become a basic mathematical model in processing synthetic aperture radar images. The purpose of this paper is to examine a general multiplicative noise model with linear signals represented by a number of unknown parameters. The ordinary least squares (LS) and weighted LS methods are used to estimate the model parameters. The biases of the weighted LS estimates of the parameters are derived. The biases are then corrected to obtain a second-order unbiased estimator, which is shown to be exactly equivalent to the maximum log quasi-likelihood estimation, though the quasi-likelihood function is founded on a completely different theoretical consideration and is known, at the present time, to be a uniquely acceptable theory for multiplicative noise models. Synthetic simulations are carried out to confirm theoretical results and to illustrate problems in processing data contaminated by multiplicative noises. The sensitivity of the LS and weighted LS methods to extremely noisy data is analysed through the simulated examples.     

The peculiar shrinkage properties of partial least squares regression

Partial least squares regression has been widely adopted within some areas as a useful alternative to ordinary least squares regression in the manner of other shrinkage methods such as principal components regression and ridge regression. In this paper we examine the nature of this shrinkage and demonstrate that partial least squares regression exhibits some undesirable properties.     

Geometric characterization of planar regression

The geometric characterization of linear regression in terms of the ‘concentration ellipse’ by Galton [Galton, F., 1886, Family likeness in stature (with Appendix by Dickson, J.D.H.). Proceedings of the Royal Society of London, 40, 42–73.] and Pearson [Pearson, K., 1901, On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2, 559–572.] was extended to the case of unequal variances of the presumably uncorrelated errors in the experimental data [McCartin, B.J., 2003, A geometric characterization of linear regression. Statistics, 37(2), 101–117.]. In this paper, this geometric characterization is further extended to planar (and also linear) regression in three dimensions where a beautiful interpretation in terms of the concentration ellipsoid is developed.     

On necessary and sufficient condition for superiority of ridge estimator over least squares estimator

The necessary and sufficient condition is obtained such that ridge estimator is better than the least squares estimator relative to the matrix mean square error.     

Generalized least squares with misspecified serial correlation structures

Summary. The regression literature contains hundreds of studies on serially correlated disturbances. Most of these studies assume that the structure of the error covariance matrix Ω is known or can be estimated consistently from data. Surprisingly, few studies investigate the properties of estimated generalized least squares (GLS) procedures when the structure of Ω is incorrectly identified and the parameters are inefficiently estimated. We compare the finite sample efficiencies of ordinary least squares (OLS), GLS and incorrect GLS (IGLS) estimators. We also prove new theorems establishing theoretical efficiency bounds for IGLS relative to GLS and OLS. Results from an exhaustive simulation study are used to evaluate the finite sample performance and to demonstrate the robustness of IGLS estimates vis-à-vis OLS and GLS estimates constructed for models with known and estimated (but correctly identified) Ω. Some of our conclusions for finite samples differ from established asymptotic results.     

Quantile regression via iterative least squares computations

We present an estimating framework for quantile regression where the usual L ₁-norm objective function is replaced by its smooth parametric approximation. An exact path-following algorithm is derived, leading to the well-known ‘basic’ solutions interpolating exactly a number of observations equal to the number of parameters being estimated. We discuss briefly possible practical implications of the proposed approach, such as early stopping for large data sets, confidence intervals, and additional topics for future research.     

Study of partial least squares and ridge regression methods

This article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.     

MORE ON THE PRE-TEST ESTIMATOR IN RIDGE REGRESSION

ABSTRACT

The problem of estimation of the regression coefficients in a multiple regression model is considered under a multicollinearity situation when it is suspected that the regression coefficients may be restricted to a subspace. The objective of this paper is to compare the usual preliminary test estimator and the preliminary test ridge regression estimator in the sense of the dispersion matrix of one dominating that of the other. In particular we proved two results giving necessary and sufficient conditions for the superiority of the preliminary test ridge regression estimator over the preliminary test estimator associated with the δ = 0 (or Δ = 0) and δ ≠ 0 (or Δ ≠ 0).