Confidence Interval for Shrinkage Parameters in Ridge Regression

In ridge regression, the estimation of ridge parameter k is an important problem. There are several methods available in the literature to do this job some what efficiently. However, no attempts were made to suggest a confidence interval for the ridge parameter using the knwoledge from the data. In this article, we propose a data dependent confidence interval for the ridge parameter k. The method of obtaining the confidence interval is illustrated with the help of a data set. A simulation study indicates that the empirical coverage probability of the suggested confidence intervals are quite high.     

A new modified Jackknifed estimator for the Poisson regression model

The Poisson regression is very popular in applied researches when analyzing the count data. However, multicollinearity problem arises for the Poisson regression model when the independent variables are highly intercorrelated. Shrinkage estimator is a commonly applied solution to the general problem caused by multicollinearity. Recently, the ridge regression (RR) estimators and some methods for estimating the ridge parameter k in the Poisson regression have been proposed. It has been found that some estimators are better than the commonly used maximum-likelihood (ML) estimator and some other RR estimators. In this study, the modified Jackknifed Poisson ridge regression (MJPR) estimator is proposed to remedy the multicollinearity. A simulation study and a real data example are provided to evaluate the performance of estimators. Both mean-squared error and the percentage relative error are considered as the performance criteria. The simulation study and the real data example results show that the proposed MJPR method outperforms the Poisson ridge regression, Jackknifed Poisson ridge regression and the ML in all of the different situations evaluated in this paper.     

More on the restricted ridge regression estimation

Several alternative methods for derivation of the restricted ridge regression estimator (RRRE) are provided. Theoretical comparison and relationship of RRRE with related methods for regression with the multicollinearity problem are described. We also find inter-connections among RRRE, ordinary ridge regression estimator (ORRE), restricted least squares estimator (RLSE), modified ridge regression estimator (MRRE) and restricted modified generalized ridge estimator (RMGRE). Finally, numerical comparison, in addition to theoretical derivation, is also conducted with a Monte Carlo simulation and a real data example.     

The Ubiquity of Statistics

The use of biased estimation in data analysis and model building is discussed. A review of the theory of ridge regression and its relation to generalized inverse regression is presented along with the results of a simulation experiment and three examples of the use of ridge regression in practice. Comments on variable selection procedures, model validation, and ridge and generalized inverse regression computation procedures are included. The examples studied here show that when the predictor variables are highly correlated, ridge regression produces coefficients which predict and extrapolate better than least squares and is a safe procedure for selecting variables.     

On the ridge regression estimator with sub-space restriction

In the linear regression model with elliptical errors, a shrinkage ridge estimator is proposed. In this regard, the restricted ridge regression estimator under sub-space restriction is improved by incorporating a general function which satisfies Taylor’s series expansion. Approximate quadratic risk function of the proposed shrinkage ridge estimator is evaluated in the elliptical regression model. A Monte Carlo simulation study and analysis based on a real data example are considered for performance analysis. It is evident from the numerical results that the shrinkage ridge estimator performs better than both unrestricted and restricted estimators in the multivariate t-regression model, for some specific cases.     

Comment on hoerl and kennard's ridge regression simulation methodology

Ridge regression has received strong support in several Monte carlo studies, leading some authors to advocate its general use. It is argued, however, that these studies were strongly biased in favor of ridge regression by simulating regression coefficient vectors centered at the origin; a condition well suited for a shrinkage technique. Studies which modeled some non-zero regression coefficients and which showed only qualified support for ridge regression are cited in support of this argument. It is concluded that only to the extent that ridge regression type coefficient vectors actually underlie real data sets -a poorly understood phenomenon - will ridge regression be of use.     

Local Influence Analysis for The Ridge Regression Under Stochastic Linear Restrictions

In this article, we assess the local influence for the ridge regression of linear models with stochastic linear restrictions in the spirit of Cook by using the log-likelihood of the stochastic restricted ridge regression estimator. The diagnostics under the perturbations of constant variance, responses and individual explanatory variables are derived. We also assess the local influence of the stochastic restricted ridge regression estimator under the approach suggested by Billor and Loynes. At the end, a numerical example on the Longley data is given to illustrate the theoretic results.     

Cross validation of ridge regression estimator in autocorrelated linear regression models

ABSTRACT

In this paper, we investigated the cross validation measures, namely OCV, GCV and Cp under the linear regression models when the error structure is autocorrelated and regressor data are correlated. The best performed ridge regression estimator is obtained by getting the optimal ridge parameter so as to minimize these measures. A Monte Carlo simulation study is given to see how the optimal ridge parameter is affected by autocorrelation and the strength of multicollinearity.     

Use of the Bootstrap and Cross-Validation in Ridge Regression

Several existing methods for the choice of the ridge parameter are reviewed, and a bootstrap method is proposed. The bootstrap provides independent measures of prediction errors based on multiple predictions along with an estimate of the standard error of prediction. The bootstrap and selected competitors are compared through Monte Carlo simulations for various degrees of design matrix collinearity and varying levels of signal-to-noise ratio. The procedure is also illustrated by application to two published data sets. In one case, the bootstrap choice of the ridge parameter leads to a smaller mean squared error of prediction than the ridge trace method. In the second case, an optimal choice of no perturbation is confirmed. Benefits of the bootstrap choice include its less subjective nature, ease of implementation, and robustness.     

Improved robust ridge M-estimation

It is developed that non-sample prior information about regression vector-parameter, usually in the form of constraints, improves the risk performance of the ordinary least squares estimator (OLSE) when it is shrunken. However, in practice, it may happen that both multicollinearity and outliers exist simultaneously in the data. In such a situation, the use of robust ridge estimator is suggested to overcome the undesirable effects of the OLSE. In this article, some prior information in the form of constraints is employed to improve the performance of this estimator in the multiple regression model. In this regard, shrinkage ridge robust estimators are defined. Advantages of the proposed estimators over the usual robust ridge estimator are also investigated using Monte-Carlo simulation as well as a real data example.     

Robust two parameter ridge M-estimator for linear regression

The problem of multicollinearity and outliers in the data set produce undesirable effects on the ordinary least squares estimator. Therefore, robust two parameter ridge estimation based on M-estimator (ME) is introduced to deal with multicollinearity and outliers in the y-direction. The proposed estimator outperforms ME, two parameter ridge estimator and robust ridge M-estimator according to mean square error criterion. Moreover, a numerical example and a Monte Carlo simulation experiment are presented.     

Robust ridge and robust Liu estimator for regression based on the LTS estimator

In the multiple linear regression analysis, the ridge regression estimator and the Liu estimator are often used to address multicollinearity. Besides multicollinearity, outliers are also a problem in the multiple linear regression analysis. We propose new biased estimators based on the least trimmed squares (LTS) ridge estimator and the LTS Liu estimator in the case of the presence of both outliers and multicollinearity. For this purpose, a simulation study is conducted in order to see the difference between the robust ridge estimator and the robust Liu estimator in terms of their effectiveness; the mean square error. In our simulations, the behavior of the new biased estimators is examined for types of outliers: X-space outlier, Y-space outlier, and X-and Y-space outlier. The results for a number of different illustrative cases are presented. This paper also provides the results for the robust ridge regression and robust Liu estimators based on a real-life data set combining the problem of multicollinearity and outliers.     

An evaluation of ridge estimator in linear mixed models: an example from kidney failure data

This paper is concerned with the ridge estimation of fixed and random effects in the context of Henderson's mixed model equations in the linear mixed model. For this purpose, a penalized likelihood method is proposed. A linear combination of ridge estimator for fixed and random effects is compared to a linear combination of best linear unbiased estimator for fixed and random effects under the mean-square error (MSE) matrix criterion. Additionally, for choosing the biasing parameter, a method of MSE under the ridge estimator is given. A real data analysis is provided to illustrate the theoretical results and a simulation study is conducted to characterize the performance of ridge and best linear unbiased estimators approach in the linear mixed model.     

Two methods of evaluating hoerl and kennard's ridge regression

We formulate a modified version of the Hoerl-Kennard ridge regression method to solve the problem of estimating coefficients in economic relationships. We investigate two approaches for determining the biasing parameter One approach utilizes prior information in choosing jr, the other approach estimates y from the sample data. Monte Carlo experiments are used to evaluate the relative efficiencies of alternative ridge estimators.     

Some ridge regression estimators for the zero-inflated Poisson model

The zero-inflated Poisson regression model is commonly used when analyzing economic data that come in the form of non-negative integers since it accounts for excess zeros and overdispersion of the dependent variable. However, a problem often encountered when analyzing economic data that has not been addressed for this model is multicollinearity. This paper proposes ridge regression (RR) estimators and some methods for estimating the ridge parameter k for a non-negative model. A simulation study has been conducted to compare the performance of the estimators. Both mean squared error and mean absolute error are considered as the performance criteria. The simulation study shows that some estimators are better than the commonly used maximum-likelihood estimator and some other RR estimators. Based on the simulation study and an empirical application, some useful estimators are recommended for practitioners.     

Restricted Ridge Estimators of the Parameters in Semiparametric Regression Model

In this article, we introduce a ridge estimator for the vector of parameters β in a semiparametric model when additional linear restrictions on the parameter vector are assumed to hold. We also obtain the semiparametric restricted ridge estimator for the parametric component in the semiparametric regression model. The ideas in this article are illustrated with a data set consisting of housing prices and through a comparison of the performances of the proposed and related estimators via a Monte Carlo simulation.     

Small sample properties of ridge estimators with normal and non-normal disturbances

In this paper we consider five well known and widely used ridge estimators when the convenient assumption of normality of the disturbances is abandoned and report on a Monte Carlo study of their small sample properties. The Monte Carlo experiment is applied to four different data sets with artificially varied degrees of multicollinearity, while the disturbances follow normal, lognormal, uniform and Laplace distributions with small and large variances. The results show that the best estimates are obtained for all ridge estimators when the disturbances follow the lognormal distribution. Also, none of the examined ridge estimators shows a consistent behavior under the different settings considered.     

Choice of the ridge factor from the correlation matrix determinant

Ridge regression is the alternative method to ordinary least squares, which is mostly applied when a multiple linear regression model presents a worrying degree of collinearity. A relevant topic in ridge regression is the selection of the ridge parameter, and different proposals have been presented in the scientific literature. Since the ridge estimator is biased, its estimation is normally based on the calculation of the mean square error (MSE) without considering (to the best of our knowledge) whether the proposed value for the ridge parameter really mitigates the collinearity. With this goal and different simulations, this paper proposes to estimate the ridge parameter from the determinant of the matrix of correlation of the data, which verifies that the variance inflation factor (VIF) is lower than the traditionally established threshold. The possible relation between the VIF and the determinant of the matrix of correlation is also analysed. Finally, the contribution is illustrated with three real examples.     

A new linearized ridge Poisson estimator in the presence of multicollinearity

Poisson regression is a very commonly used technique for modeling the count data in applied sciences, in which the model parameters are usually estimated by the maximum likelihood method. However, the presence of multicollinearity inflates the variance of maximum likelihood (ML) estimator and the estimated parameters give unstable results. In this article, a new linearized ridge Poisson estimator is introduced to deal with the problem of multicollinearity. Based on the asymptotic properties of ML estimator, the bias, covariance and mean squared error of the proposed estimator are obtained and the optimal choice of shrinkage parameter is derived. The performance of the existing estimators and proposed estimator is evaluated through Monte Carlo simulations and two real data applications. The results clearly reveal that the proposed estimator outperforms the existing estimators in the mean squared error sense.KEYWORDS: Poisson regression, multicollinearity, ridge Poisson estimator, linearized ridge regression estimator, mean squared errorMathematics Subject Classifications: 62J07, 62F10     

Predictability measures for ridge regression models

Methods for flagging new points that are not similar to the original data used for developing a ridge regression equation are discussed. Using the regression equation for predictions should be avoided for these dissimilar points. An example quantifies the sample space limitations for using biased prediction when multicollinearity is present.