The Ubiquity of Statistics

The use of biased estimation in data analysis and model building is discussed. A review of the theory of ridge regression and its relation to generalized inverse regression is presented along with the results of a simulation experiment and three examples of the use of ridge regression in practice. Comments on variable selection procedures, model validation, and ridge and generalized inverse regression computation procedures are included. The examples studied here show that when the predictor variables are highly correlated, ridge regression produces coefficients which predict and extrapolate better than least squares and is a safe procedure for selecting variables.     

Unbiased ridge estimation with prior information and ridge trace

A procedure is illustrated to incorporate prior information in the ridge regression model. Unbiased ridge estimators with prior information are defined and a robust estimate of the ridge parameter k is proposed.     

Generalized ridge regression and a generalization of the CP statistic

We consider a generalization of ridge regression and demonstrate advantages over ridge regression. We provide an empirical Bayes method for determining the ridge constants, using the Bayesian interpretation of ridge estimators, and show that this coincides with a method based on a generalization of the C_P statistic and the non-negative garrote. These provide an automatic variable selection procedure for the canonical variables.     

On the ridge regression estimator with sub-space restriction

In the linear regression model with elliptical errors, a shrinkage ridge estimator is proposed. In this regard, the restricted ridge regression estimator under sub-space restriction is improved by incorporating a general function which satisfies Taylor’s series expansion. Approximate quadratic risk function of the proposed shrinkage ridge estimator is evaluated in the elliptical regression model. A Monte Carlo simulation study and analysis based on a real data example are considered for performance analysis. It is evident from the numerical results that the shrinkage ridge estimator performs better than both unrestricted and restricted estimators in the multivariate t-regression model, for some specific cases.     

Ridge estimation in generalized linear models and proportional hazards regressions

Conventionally, a ridge parameter is estimated as a function of regression parameters based on ordinary least squares. In this article, we proposed an iterative procedure instead of the one-step or conventional ridge method. Additionally, we construct an indicator that measures the potential degree of improvement in mean squared error when ridge estimates are employed. Simulations show that our methods are appropriate for a wide class of non linear models including generalized linear models and proportional hazards (PHs) regressions. The method is applied to a PH regression with highly collinear covariates in a cancer recurrence study.     

Robust symmetric distribution location estimation and regression

A procedure for estimating the location parameter of an unknown symmetric distribution is developed for application to samples from very light-tailed through very heavy-tailed distributions. This procedure has an easy extension to a technique for estimating the coefficients in a linear regression model whose error distribution is symmetric with arbitrary tail weights. The regression procedure is, in turn, extended to make it applicable to situations where the error distribution is either symmetric or skewed. The potentials of the procedures for robust location parameter and regression coefficient estimation are demonstrated by simulation studies.     

Diagnostic measures for kernel ridge regression on reproducing kernel Hilbert space

The aim of this paper is to define and develop diagnostic measures with respect to kernel ridge regression in a reproducing kernel Hilbert space (RKHS). To identify influential observations, we define a particular version of Cook’s distance for the kernel ridge regression model in RKHS, which is conceptually consistent with Cook’s distance in a classical regression model. Then, by using the perturbation formula for the regularized conditional expectation of the outcome in RKHS, we develop an approximate version of Cook”s distance in RKHS because the original definition requires intensive computations. Such an approximated Cook”s distance is represented in terms of basic building blocks such as residuals and leverages of the kernel ridge regression. The results of the simulation and real application demonstrate that our diagnostic measure successfully detects potentially influential observations on estimators in kernel ridge regression.     

Restricted ridge estimator in the logistic regression model

It is known that when the multicollinearity exists in the logistic regression model, variance of maximum likelihood estimator is unstable. As a remedy, Schaefer et al. presented a ridge estimator in the logistic regression model. Making use of the ridge estimator, when some linear restrictions are also present, we introduce a restricted ridge estimator in the logistic regression model. Statistical properties of this newly defined estimator will be studied and comparisons are done in the simulation study in the sense of mean squared error criterion. A real-data example and a simulation study are introduced to discuss the performance of this estimator.     

Performance of some new preliminary test ridge regression estimators and their properties

The problem of estimation of the regression coefficients in a multiple regression model is considered under multicollinearity situation when it is suspected that the regression coefficients may be restricted to a subspace. We present the estimators of the regression coefficients combining the idea of preliminary test and ridge regression methodology. Accordingly, we consider three estimators, namely, the unrestricted ridge regression estimator (URRE), the restricted ridge regression estimator (RRRE), and finally, the preliminary test ridge regression estimator (PTRRE). The biases, variancematrices and mean square errors (mse) of the estimators are derived and compared with the usual estimators. Regions of optimality of the estimators are determined by studying the mse criterion. The conditions of superiority of the estimators over the traditional estimators as in Saleh and Han (1990) and Ali and Saleh (1991) have also been discussed.     

Ridge Analysis 25 Years Later

The response surface technique called ridge analysis was originally introduced by Hoerl (1959) more than 25 years ago. Despite tremendous advantages over more conventional response surface procedures when more than two independent variables are present, ridge analysis has received little attention in the statistical literature since then, although numerous applications have appeared in engineering journals. This situation may be partially due to the fact that this procedure led to the discovery of ridge regression, which has completely overshadowed ridge analysis in the literature since. This discussion will briefly review the mathematics of ridge analysis, its literature, practical advantages, and relationship to ridge regression.     

Ridge regression estimators in the linear regression models with non-spherical errors

The present paper considers a family of ordinary ridge regression estimators in the linear regression model when the disturbances covariance matrix depends upon a few unknown parameters. An asymptotic expansion for the distribution of the ridge regression estimator is developed and under the quadratic loss function its asymptotic risk is compared with that of the feasible GLS estimator.     

A new modified Jackknifed estimator for the Poisson regression model

The Poisson regression is very popular in applied researches when analyzing the count data. However, multicollinearity problem arises for the Poisson regression model when the independent variables are highly intercorrelated. Shrinkage estimator is a commonly applied solution to the general problem caused by multicollinearity. Recently, the ridge regression (RR) estimators and some methods for estimating the ridge parameter k in the Poisson regression have been proposed. It has been found that some estimators are better than the commonly used maximum-likelihood (ML) estimator and some other RR estimators. In this study, the modified Jackknifed Poisson ridge regression (MJPR) estimator is proposed to remedy the multicollinearity. A simulation study and a real data example are provided to evaluate the performance of estimators. Both mean-squared error and the percentage relative error are considered as the performance criteria. The simulation study and the real data example results show that the proposed MJPR method outperforms the Poisson ridge regression, Jackknifed Poisson ridge regression and the ML in all of the different situations evaluated in this paper.     

A Dirichlet random coefficient regression model for quality indicators

We present a random coefficient regression model in which a response is linearly related to some explanatory variables with random coefficients following a Dirichlet distribution. These coefficients can be interpreted as weights because they are nonnegative and add up to one. The proposed estimation procedure combines iteratively reweighted least squares and the maximization on an approximated likelihood function. We also present a diagnostic tool based on a residual Q–Q plot and two procedures for estimating individual weights. The model is used to construct an index for measuring the quality of the railroad system in Spain.     

Fitting survival data with penalized Poisson regression

Cox’s proportional hazards model is the most common way to analyze survival data. The model can be extended in the presence of collinearity to include a ridge penalty, or in cases where a very large number of coefficients (e.g. with microarray data) has to be estimated. To maximize the penalized likelihood, optimal weights of the ridge penalty have to be obtained. However, there is no definite rule for choosing the penalty weight. One approach suggests maximization of the weights by maximizing the leave-one-out cross validated partial likelihood, however this is time consuming and computationally expensive, especially in large datasets. We suggest modelling survival data through a Poisson model. Using this approach, the log-likelihood of a Poisson model is maximized by standard iterative weighted least squares. We will illustrate this simple approach, which includes smoothing of the hazard function and move on to include a ridge term in the likelihood. We will then maximize the likelihood by considering tools from generalized mixed linear models. We will show that the optimal value of the penalty is found simply by computing the hat matrix of the system of linear equations and dividing its trace by a product of the estimated coefficients.     

Marginal maximum likelihood estimation methods for the tuning parameters of ridge,power ridge,and generalized ridge regression

This study introduces fast marginal maximum likelihood (MML) algorithms for estimating the tuning (shrinkage) parameter(s) of the ridge and power ridge regression models, and an automatic plug-in MML estimator for the generalized ridge regression model, in a Bayesian framework. These methods are applicable to multicollinear or singular covariate design matrices, including matrices where the number of covariates exceeds the sample size. According to analyses of many real and simulated datasets, these MML-based ridge methods tend to compare favorably to other tuning parameter selection methods, in terms of computation speed, prediction accuracy, and ability to detect relevant covariates.     

Choosing shrinkage estimators for regression problems

A Bayesian formulation of the canonical form of the standard regression model is used to compare various Stein-type estimators and the ridge estimator of regression coefficients, A particular (“constant prior”) Stein-type estimator having the same pattern of shrinkage as the ridge estimator is recommended for use.     

Choice of the ridge factor from the correlation matrix determinant

Ridge regression is the alternative method to ordinary least squares, which is mostly applied when a multiple linear regression model presents a worrying degree of collinearity. A relevant topic in ridge regression is the selection of the ridge parameter, and different proposals have been presented in the scientific literature. Since the ridge estimator is biased, its estimation is normally based on the calculation of the mean square error (MSE) without considering (to the best of our knowledge) whether the proposed value for the ridge parameter really mitigates the collinearity. With this goal and different simulations, this paper proposes to estimate the ridge parameter from the determinant of the matrix of correlation of the data, which verifies that the variance inflation factor (VIF) is lower than the traditionally established threshold. The possible relation between the VIF and the determinant of the matrix of correlation is also analysed. Finally, the contribution is illustrated with three real examples.     

Influence diagnostics in gamma ridge regression model

In this article, we proposed some influence diagnostics for the gamma regression model (GRM) and the gamma ridge regression model (GRRM). We assess the impact of influential observations on the GRM and GRRM estimates by extending the work of Pregibon [Logistic regression diagnostics. Ann Stat. 1981;9:705–724] and Walker and Birch [Influence measures in ridge regression. Technometrics. 1988;30:221–227]. Comparison of both models is made and demonstrated with the help of a simulation study and a real data set. We report some momentous results in detecting the influential observations and their effects on the GRM and GRRM estimates.     

Graphical methods for evaluating ridge regression estimator in mixture experiments

When the component proportions in mixture experiments are restricted by lower and upper bounds, multicollinearity appears all too frequently. Thus, we can suggest the use of ridge regression as a mean for stabilizing the coefficient estimates in the fitted model. We propose graphical methods for evaluating the effect of ridge regression estimator with respect to the predicted response value and the prediction variance.     

Influence in the Elliptical Modified Ridge Regression

In this article, we present diagnostic methods for the modified ridge regression under elliptical model based on the pseudo-likelihood function. The maximum likelihood estimators of the parameters in the modified ridge elliptical model are given and local influence measures are developed. Finally, illustration of our methodology is given through a numerical example.