Ridge Estimation to the Restricted Linear Model

This article is concerned with the problem of multicollinearity in a linear model with linear restrictions. After introducing a spheral restricted condition, a new restricted ridge estimation method is proposed by minimizing the sum of squared residuals. The property of the new estimator in its superiority over the ordinary restricted least squares estimation is then theoretically analyzed. Furthermore, a sufficient and necessary condition for selecting the ridge parameter k is obtained. To simplify the selection of the ridge parameter, a sufficient condition is also given. Finally, a numerical example demonstrates the merit of the new method in the aspect of solving the multicollinearity over the ordinary restricted least squares estimation.     

Ridge Regression – A Simulation Study

In this article we assess the suitability of two new ridge estimators by means of a simulation study. We compare these estimators with well-known ridge estimators. We also make direct comparisons between the ordinary least squares (OLS) estimator and the ridge estimators by using ratio of the average total mean square error of the OLS estimator and the ridge estimators. We find that the new estimators perform well under certain conditions.     

Using Liu-Type Estimator to Combat Collinearity

Abstract

Linear regression model and least squares method are widely used in many fields of natural and social sciences. In the presence of collinearity, the least squares estimator is unstable and often gives misleading information. Ridge regression is the most common method to overcome this problem. We find that when there exists severe collinearity, the shrinkage parameter selected by existing methods for ridge regression may not fully address the ill conditioning problem. To solve this problem, we propose a new two-parameter estimator. We show using both theoretic results and simulation that our new estimator has two advantages over ridge regression. First, our estimator has less mean squared error (MSE). Second, our estimator can fully address the ill conditioning problem. A numerical example from literature is used to illustrate the results.     

Selection of the Ridge Parameter Using Mathematical Programming

Ridge regression solves multicollinearity problems by introducing a biasing parameter that is called ridge parameter; it shrinks the estimates and their standard errors in order to reach acceptable results. Selection of the ridge parameter was done using several subjective and objective techniques that are concerned with certain criteria. In this study, selection of the ridge parameter depends on other important statistical measures to reach a better value of the ridge parameter. The proposed ridge parameter selection technique depends on a mathematical programming model and the results are evaluated using a simulation study. The performance of the proposed method is good when the error variance is greater than or equal to one; the sample consists of 20 observations, the number of explanatory variables in the model is 2, and there is a very strong correlation between the two explanatory variables.     

A New Asymmetric Interaction Ridge (AIR) Regression Method

Despite that interaction terms are standard tools of regression analysis, the side effects of the inclusion of these terms in models estimated by ordinary least squares (OLS) are yet not fully penetrated. The inclusion of interaction effects induces multicollinearity problems since all non zero values are equal between the interaction term and the regressor. In this article, we propose a procedure to remedy this problem by the use of new ridge regression (RR) shrinkage parameters—which we call the asymmetric interaction ridge (AIR) regression method. By means of Monte Carlo simulations we evaluate both OLS and AIR using the mean square error (MSE) performance criterion. The result from the simulation study confirms our hypothesis that AIR always should be preferred to OLS since it has a lower estimated MSE. Moreover, the advantages of our new method are demonstrated in an empirical application where positive asymmetric price transmission effects are exposed for the mortgage interest rates of Handelsbanken Stadshypotek. It is observed that the mortgage interest rates increase more fully and rapidly to an increase in the bank's borrowing costs than to a decrease. This asymmetry is defined as positive asymmetric price transmission (APT).     

Confidence Interval for Shrinkage Parameters in Ridge Regression

In ridge regression, the estimation of ridge parameter k is an important problem. There are several methods available in the literature to do this job some what efficiently. However, no attempts were made to suggest a confidence interval for the ridge parameter using the knwoledge from the data. In this article, we propose a data dependent confidence interval for the ridge parameter k. The method of obtaining the confidence interval is illustrated with the help of a data set. A simulation study indicates that the empirical coverage probability of the suggested confidence intervals are quite high.     

Seemingly unrelated regressions with covariance matrix of cross-equation ridge regression residuals

Generalized least squares estimation of a system of seemingly unrelated regressions is usually a two-stage method: (1) estimation of cross-equation covariance matrix from ordinary least squares residuals for transforming data, and (2) application of least squares on transformed data. In presence of multicollinearity problem, conventionally ridge regression is applied at stage 2. We investigate the usage of ridge residuals at stage 1, and show analytically that the covariance matrix based on the least squares residuals does not always result in more efficient estimator. A simulation study and an application to a system of firms' gross investment support our finding.     

Computational Method for Jackknifed Generalized Ridge Tuning Parameter based on Generalized Maximum Entropy

In this article, a new method to estimate the Jackknifed generalized ridge tuning parameter, based on the Jackknifed Ridge-trace and an analytical method borrowed from generalized maximum entropy, is presented. The ideas in the article are illustrated and evaluated using to the well-known Portland cement data set and simulations.     

Two-parameter ridge estimator in the binary logistic regression

The binary logistic regression is a commonly used statistical method when the outcome variable is dichotomous or binary. The explanatory variables are correlated in some situations of the logit model. This problem is called multicollinearity. It is known that the variance of the maximum likelihood estimator (MLE) is inflated in the presence of multicollinearity. Therefore, in this study, we define a new two-parameter ridge estimator for the logistic regression model to decrease the variance and overcome multicollinearity problem. We compare the new estimator to the other well-known estimators by studying their mean squared error (MSE) properties. Moreover, a Monte Carlo simulation is designed to evaluate the performances of the estimators. Finally, a real data application is illustrated to show the applicability of the new method. According to the results of the simulation and real application, the new estimator outperforms the other estimators for all of the situations considered.     

A class of generalized ridge estimators

Presence of collinearity among the explanatory variables results in larger standard errors of parameters estimated. When multicollinearity is present among the explanatory variables, the ordinary least-square (OLS) estimators tend to be unstable due to larger variance of the estimators of the regression coefficients. As alternatives to OLS estimators few ridge estimators are available in the literature. This article presents some of the popular ridge estimators and attempts to provide (i) a generalized class of ridge estimators and (ii) a modified ridge estimator. The performance of the proposed estimators is investigated with the help of Monte Carlo simulation technique. Simulation results indicate that the suggested estimators perform better than the ordinary least-square (OLS) estimators and other estimators considered in this article.     

A Combined Nonlinear Programming Model and Kibria Method for Choosing Ridge Parameter Regression

Ridge regression solves multicollinearity problems by introducing a biasing parameter that is called ridge parameter; it shrinks the estimates as well as their standard errors in order to reach acceptable results. Many methods are available for estimating a ridge parameter. This article has considered some of these methods and also proposed a combined nonlinear programming model and Kibria method. A simulation study has been made to evaluate the performance of the proposed estimators based on the minimum mean squared error criterion. The simulation study indicates that under certain conditions the proposed estimators outperform the least squares (LS) estimators and other popular existing estimators. Moreover, the new proposed model is applied on dataset that suffers also from the presence of heteroscedastic errors.     

Use of the Bootstrap and Cross-Validation in Ridge Regression

Several existing methods for the choice of the ridge parameter are reviewed, and a bootstrap method is proposed. The bootstrap provides independent measures of prediction errors based on multiple predictions along with an estimate of the standard error of prediction. The bootstrap and selected competitors are compared through Monte Carlo simulations for various degrees of design matrix collinearity and varying levels of signal-to-noise ratio. The procedure is also illustrated by application to two published data sets. In one case, the bootstrap choice of the ridge parameter leads to a smaller mean squared error of prediction than the ridge trace method. In the second case, an optimal choice of no perturbation is confirmed. Benefits of the bootstrap choice include its less subjective nature, ease of implementation, and robustness.     

Jackknifing the Ridge Regression Estimator: A Revisit

Singh et al. (1986 Singh, B., Chaubey, Y.P., Dwivedi, T.D. (1986). An almost unbiased ridge estimator. Sankhya B48: 34236. [Google Scholar]) proposed an almost unbiased ridge estimator using Jackknife method that required transformation of the regression parameters. This article shows that the same method can be used to derive the Jackknifed ridge estimator of the original (untransformed) parameter without transformation. This method also leads in deriving easily the second-order Jackknifed ridge that may reduce the bias further. We further investigate the performance of these estimators along with a recent method by Batah et al. (2008 Batah, F. S.M., Ramanathan, T.V., Gore, S.D. (2008). The efficiency of modified Jack-knife and ridge type regression estimators: a comparison. Surv. Math. Applic. 3:111122. [Google Scholar]) called modified Jackknifed ridge theoretically as well as numerically.     

Marginal maximum likelihood estimation methods for the tuning parameters of ridge,power ridge,and generalized ridge regression

This study introduces fast marginal maximum likelihood (MML) algorithms for estimating the tuning (shrinkage) parameter(s) of the ridge and power ridge regression models, and an automatic plug-in MML estimator for the generalized ridge regression model, in a Bayesian framework. These methods are applicable to multicollinear or singular covariate design matrices, including matrices where the number of covariates exceeds the sample size. According to analyses of many real and simulated datasets, these MML-based ridge methods tend to compare favorably to other tuning parameter selection methods, in terms of computation speed, prediction accuracy, and ability to detect relevant covariates.     

An evaluation of ridge estimator in linear mixed models: an example from kidney failure data

This paper is concerned with the ridge estimation of fixed and random effects in the context of Henderson's mixed model equations in the linear mixed model. For this purpose, a penalized likelihood method is proposed. A linear combination of ridge estimator for fixed and random effects is compared to a linear combination of best linear unbiased estimator for fixed and random effects under the mean-square error (MSE) matrix criterion. Additionally, for choosing the biasing parameter, a method of MSE under the ridge estimator is given. A real data analysis is provided to illustrate the theoretical results and a simulation study is conducted to characterize the performance of ridge and best linear unbiased estimators approach in the linear mixed model.     

A Jackknifed estimators for the negative binomial regression model

Shrinkage estimator is a commonly applied solution to the general problem caused by multicollinearity. Recently, the ridge regression (RR) estimators for estimating the ridge parameter k in the negative binomial (NB) regression have been proposed. The Jackknifed estimators are obtained to remedy the multicollinearity and reduce the bias. A simulation study is provided to evaluate the performance of estimators. Both mean squared error (MSE) and the percentage relative error (PRE) are considered as the performance criteria. The simulated result indicated that some of proposed Jackknifed estimators should be preferred to the ML method and ridge estimators to reduce MSE and bias.     

Penalized Maximum Likelihood Principle for Choosing Ridge Parameter

We consider the problem of choosing the ridge parameter. Two penalized maximum likelihood (PML) criteria based on a distribution-free and a data-dependent penalty function are proposed. These PML criteria can be considered as “continuous” versions of AIC. A systematic simulation is conducted to compare the suggested criteria to several existing methods. The simulation results strongly support the use of our method. The method is also applied to two real data sets.     

Restricted ridge estimator in the logistic regression model

It is known that when the multicollinearity exists in the logistic regression model, variance of maximum likelihood estimator is unstable. As a remedy, Schaefer et al. presented a ridge estimator in the logistic regression model. Making use of the ridge estimator, when some linear restrictions are also present, we introduce a restricted ridge estimator in the logistic regression model. Statistical properties of this newly defined estimator will be studied and comparisons are done in the simulation study in the sense of mean squared error criterion. A real-data example and a simulation study are introduced to discuss the performance of this estimator.     

Performance of Some Ridge Parameters for Probit Regression: With Application to Swedish Job Search Data

In ridge regression, the estimation of the ridge parameter is an important issue. This article generalizes some methods for estimating the ridge parameter for probit ridge regression (PRR) model based on the work of Kibria et al. (2011 Kibria, B. M. G., Månsson, K. and Shukur, G. 2011. Performance of some logistic ridge regression parameters. Computational Economics, DOI: 10.1007/s10614-011-9275-x [Google Scholar]). The performance of these new estimators is judged by calculating the mean squared error (MSE) using Monte Carlo simulations. In the design of the experiment, we chose to vary the sample size and the number of regressors. Furthermore, we generate explanatory variables that are linear combinations of other regressors, which is a common situation in economics. In an empirical application regarding Swedish job search data, we also illustrate the benefits of the new method.