Robust two parameter ridge M-estimator for linear regression

The problem of multicollinearity and outliers in the data set produce undesirable effects on the ordinary least squares estimator. Therefore, robust two parameter ridge estimation based on M-estimator (ME) is introduced to deal with multicollinearity and outliers in the y-direction. The proposed estimator outperforms ME, two parameter ridge estimator and robust ridge M-estimator according to mean square error criterion. Moreover, a numerical example and a Monte Carlo simulation experiment are presented.     

Variable selection in linear regression based on ridge estimator

In this article, we consider the problem of variable selection in linear regression when multicollinearity is present in the data. It is well known that in the presence of multicollinearity, performance of least square (LS) estimator of regression parameters is not satisfactory. Consequently, subset selection methods, such as Mallow's Cp, which are based on LS estimates lead to selection of inadequate subsets. To overcome the problem of multicollinearity in subset selection, a new subset selection algorithm based on the ridge estimator is proposed. It is shown that the new algorithm is a better alternative to Mallow's Cp when the data exhibit multicollinearity.     

Cross validation of ridge regression estimator in autocorrelated linear regression models

ABSTRACT

In this paper, we investigated the cross validation measures, namely OCV, GCV and Cp under the linear regression models when the error structure is autocorrelated and regressor data are correlated. The best performed ridge regression estimator is obtained by getting the optimal ridge parameter so as to minimize these measures. A Monte Carlo simulation study is given to see how the optimal ridge parameter is affected by autocorrelation and the strength of multicollinearity.     

A new estimator combining the ridge regression and the restricted least squares methods of estimation

It is well-known in the literature on multicollinearity that one of the major consequences of multicollinearity on the ordinary least squares estimator is that the estimator produces large sampling variances, which in turn might inappropriately lead to exclusion of otherwise significant coefficients from the model. To circumvent this problem, two accepted estimation procedures which are often suggested are the restricted least squares method and the ridge regression method. While the former leads to a reduction in the sampling variance of the estimator, the later ensures a smaller mean square error value for the estimator. In this paper we have proposed a new estimator which is based on a criterion that combines the ideas underlying these two estimators. The standard properties of this new estimator have been studied in the paper. It has also been shown that this estimator is superior to both the restricted least squares as well as the ordinary ridge regression estimators by the criterion of mean sauare error of the estimator of the regression coefficients when the restrictions are indeed correct. The conditions for superiority of this estimator over the other two have also been derived for the situation when the restrictions are not correct.     

Developing ridge estimation method for median regression

In this paper, the ridge estimation method is generalized to the median regression. Though the least absolute deviation (LAD) estimation method is robust in the presence of non-Gaussian or asymmetric error terms, it can still deteriorate into a severe multicollinearity problem when non-orthogonal explanatory variables are involved. The proposed method increases the efficiency of the LAD estimators by reducing the variance inflation and giving more room for the bias to get a smaller mean squared error of the LAD estimators. This paper includes an application of the new methodology and a simulation study as well.     

Robust ridge and robust Liu estimator for regression based on the LTS estimator

In the multiple linear regression analysis, the ridge regression estimator and the Liu estimator are often used to address multicollinearity. Besides multicollinearity, outliers are also a problem in the multiple linear regression analysis. We propose new biased estimators based on the least trimmed squares (LTS) ridge estimator and the LTS Liu estimator in the case of the presence of both outliers and multicollinearity. For this purpose, a simulation study is conducted in order to see the difference between the robust ridge estimator and the robust Liu estimator in terms of their effectiveness; the mean square error. In our simulations, the behavior of the new biased estimators is examined for types of outliers: X-space outlier, Y-space outlier, and X-and Y-space outlier. The results for a number of different illustrative cases are presented. This paper also provides the results for the robust ridge regression and robust Liu estimators based on a real-life data set combining the problem of multicollinearity and outliers.     

Improved robust ridge M-estimation

It is developed that non-sample prior information about regression vector-parameter, usually in the form of constraints, improves the risk performance of the ordinary least squares estimator (OLSE) when it is shrunken. However, in practice, it may happen that both multicollinearity and outliers exist simultaneously in the data. In such a situation, the use of robust ridge estimator is suggested to overcome the undesirable effects of the OLSE. In this article, some prior information in the form of constraints is employed to improve the performance of this estimator in the multiple regression model. In this regard, shrinkage ridge robust estimators are defined. Advantages of the proposed estimators over the usual robust ridge estimator are also investigated using Monte-Carlo simulation as well as a real data example.     

Some ridge regression estimators for the zero-inflated Poisson model

The zero-inflated Poisson regression model is commonly used when analyzing economic data that come in the form of non-negative integers since it accounts for excess zeros and overdispersion of the dependent variable. However, a problem often encountered when analyzing economic data that has not been addressed for this model is multicollinearity. This paper proposes ridge regression (RR) estimators and some methods for estimating the ridge parameter k for a non-negative model. A simulation study has been conducted to compare the performance of the estimators. Both mean squared error and mean absolute error are considered as the performance criteria. The simulation study shows that some estimators are better than the commonly used maximum-likelihood estimator and some other RR estimators. Based on the simulation study and an empirical application, some useful estimators are recommended for practitioners.     

Graphical methods for evaluating ridge regression estimator in mixture experiments

When the component proportions in mixture experiments are restricted by lower and upper bounds, multicollinearity appears all too frequently. Thus, we can suggest the use of ridge regression as a mean for stabilizing the coefficient estimates in the fitted model. We propose graphical methods for evaluating the effect of ridge regression estimator with respect to the predicted response value and the prediction variance.     

Shrinkage ridge regression in partial linear models

ABSTRACT

In this paper, shrinkage ridge estimator and its positive part are defined for the regression coefficient vector in a partial linear model. The differencing approach is used to enjoy the ease of parameter estimation after removing the non parametric part of the model. The exact risk expressions in addition to biases are derived for the estimators under study and the region of optimality of each estimator is exactly determined. The performance of the estimators is evaluated by simulated as well as real data sets.     

A note on the moments of stochastic shrinkage parameters in ridge regression

The purpose of this note is to gain insight on the performance of two well known operational Ridge Regression estimators by deriving the moments of their stochastic shrinkage parameters. We also show that, under certain conditions, one of them has bounded moments.     

Generalised Rank Regression Estimator with Standard Error Adjusted Lasso

One of the standard variable selection procedures in multiple linear regression is to use a penalisation technique in least‐squares (LS) analysis. In this setting, many different types of penalties have been introduced to achieve variable selection. It is well known that LS analysis is sensitive to outliers, and consequently outliers can present serious problems for the classical variable selection procedures. Since rank‐based procedures have desirable robustness properties compared to LS procedures, we propose a rank‐based adaptive lasso‐type penalised regression estimator and a corresponding variable selection procedure for linear regression models. The proposed estimator and variable selection procedure are robust against outliers in both response and predictor space. Furthermore, since rank regression can yield unstable estimators in the presence of multicollinearity, in order to provide inference that is robust against multicollinearity, we adjust the penalty term in the adaptive lasso function by incorporating the standard errors of the rank estimator. The theoretical properties of the proposed procedures are established and their performances are investigated by means of simulations. Finally, the estimator and variable selection procedure are applied to the Plasma Beta‐Carotene Level data set.     

Comment on hoerl and kennard's ridge regression simulation methodology

Ridge regression has received strong support in several Monte carlo studies, leading some authors to advocate its general use. It is argued, however, that these studies were strongly biased in favor of ridge regression by simulating regression coefficient vectors centered at the origin; a condition well suited for a shrinkage technique. Studies which modeled some non-zero regression coefficients and which showed only qualified support for ridge regression are cited in support of this argument. It is concluded that only to the extent that ridge regression type coefficient vectors actually underlie real data sets -a poorly understood phenomenon - will ridge regression be of use.     

Using improved estimation strategies to combat multicollinearity

In this approach, some generalized ridge estimators are defined based on shrinkage foundation. Completely under the suspicion that some sub-space restrictions may occur, we present the estimators of the regression coefficients combining the idea of preliminary test estimator and Stein-rule estimator with the ridge regression methodology for normal models. Their exact risk expressions in addition to biases are derived and the regions of optimality of the estimators are exactly determined along with some numerical analysis. In this regard, the ridge parameter is determined in different disciplines.     

An examination of recent criticisms of ridge regression simulation designs

In this paper we carefully examine three propositions asserting that certain simulation studies are biased in favour of ridge regression and find them to be ill-founded.     

Restricted minimum bias linear estimation in regression

In this article a class of restricted minimum bias linear estimators of the vector of unknown regression coefficients when multicollinearity among the columns of the design matrix exists, is obtained. The ordinary ridge regression, principal components and shrinkage estimators are members of this class. Moreover, our ap-proach can be used to improve, in some sense, certain classes of generalized ridge and shrinkage estimators of the vector of un-known parameters in linear models.     

Rank estimation for the functional linear model

This article discusses the estimation of the parameter function for a functional linear regression model under heavy-tailed errors' distributions and in the presence of outliers. Standard approaches of reducing the high dimensionality, which is inherent in functional data, are considered. After reducing the functional model to a standard multiple linear regression model, a weighted rank-based procedure is carried out to estimate the regression parameters. A Monte Carlo simulation and a real-world example are used to show the performance of the proposed estimator and a comparison made with the least-squares and least absolute deviation estimators.     

An example of ridge regression difficulties

A simple consumption function is used to illustrate two fundamental difficulties with ridge regression and similarly motivated procedures. The first is the ambiguity of multicollinearity measures for judging the data's “ill-conditioning”. The second is the sensitivity of the estimates to the arbitrary normalization of the model. Neither of these poses a problem for least squares or Bayesian estimates. The logical restructuring of ridge procedures to avoid these difficulties leads to a more explicitly Bayesian approach.     

Improved ridge estimators in a linear regression model

In this paper, the notion of the improved ridge estimator (IRE) is put forward in the linear regression model y=X β+e. The problem arises if augmenting the equation 0=c′α+ε instead of 0=C α+? to the model. Three special IREs are considered and studied under the mean-squared error criterion and the prediction error sum of squares criterion. The simulations demonstrate that the proposed estimators are effective and recommendable, especially when multicollinearity is severe.