A test of the mean square error criterion for shrinkage estimators

A test for choosing between a shrinkage estimator and the least squares estimator is described and a central-F approximation to the test statistic is considered. An example from the literature was analysed using the test procedure proposed here. The power of the test was studied by means of simulation.     

A simulation study of five biased estimators for straight line regression

Five biased estimators of the slope in straight line regression are considered. For each, the estimate of the “bias parameter”, k, is a function of N, the number of observations, and [rcirc]² , the square of the least squares estimate of the standardized slope, β. The estimators include that of Farebrother, the ridge estimator of Hoerl, Kennard, and Baldwin, Vinod's shrunken estimators., and a new modification of one of the latter. Properties of the estimators are studied for 13 combinations of N and 3. Results of simulation experiments provide empirical evidence concerning the values of means and variances of the biased estimators of the slope and estimates of the “bias parameter”, the mean square errors of the estimators, and the frequency of improvement relative to least squares. Adjustments to degrees of freedom in the biased regression analysis of variance table are also considered. An extension of the new modification to the case of p> 1 independent variables is presented in an Appendix.     

On the minimum mean squared error estimators in a regression model

In this paper we analyze the properties of two estimators oroposed by Farebrother (1975) for linear regression models.     

On the small sample properties of norm-restricted maximum likelihood estimators for logistic regression models

This paper develops alternatives to maximum likelihood estimators (MLE) for logistic regression models and compares the mean squared error (MSE) of the estimators. The MLE for the vector of underlying success probabilities has low MSE only when the true probabilities are extreme (i.e., near 0 or 1). Extreme probabilities correspond to logistic regression parameter vectors which are large in norm. A competing “restricted” MLE and an empirical version of it are suggested as estimators with better performance than the MLE for central probabilities. An approximate EM-algorithm for estimating the restriction is described. As in the case of normal theory ridge estimators, the proposed estimators are shown to be formally derivable by Bayes and empirical Bayes arguments. The small sample operating characteristics of the proposed estimators are compared to the MLE via a simulation study; both the estimation of individual probabilities and of logistic parameters are considered.     

Bayes and empirical Bayes shrinkage estimation of regression coefficients

For a moderate or large number of regression coefficients, shrinkage estimates towards an overall mean are obtained by Bayes and empirical Bayes methods. For a special case, the Bayes and empirical Bayes shrinking weights are shown to be asymptotically equivalent as the amount of shrinkage goes to zero. Based on comparisons between Bayes and empirical Bayes solutions, a modification of the empirical Bayes shrinking weights designed to guard against unreasonable overshrinking is suggested. A numerical example is given.     

A class of shrinkage estimators in linear regression

We consider the problem of using shrinkage estimators that shrink towards subspaces in linear regression, in particular subspaces spanned by principal components. This is especially important when multicollinearity is present and the number of predictors is not small compared to the sample size. New theoretical results about Stein estimation are used to get estimators with lower theoretical risk than standard Stein estimators used by Oman (1991). Application of the techniques to real data is largely successful.     

LASSO and shrinkage estimation in Weibull censored regression models

In this paper we address the problem of estimating a vector of regression parameters in the Weibull censored regression model. Our main objective is to provide natural adaptive estimators that significantly improve upon the classical procedures in the situation where some of the predictors may or may not be associated with the response. In the context of two competing Weibull censored regression models (full model and candidate submodel), we consider an adaptive shrinkage estimation strategy that shrinks the full model maximum likelihood estimate in the direction of the submodel maximum likelihood estimate. We develop the properties of these estimators using the notion of asymptotic distributional risk. The shrinkage estimators are shown to have higher efficiency than the classical estimators for a wide class of models. Further, we consider a LASSO type estimation strategy and compare the relative performance with the shrinkage estimators. Monte Carlo simulations reveal that when the true model is close to the candidate submodel, the shrinkage strategy performs better than the LASSO strategy when, and only when, there are many inactive predictors in the model. Shrinkage and LASSO strategies are applied to a real data set from Veteran's administration (VA) lung cancer study to illustrate the usefulness of the procedures in practice.     

Some properties of generalized ridge estimators

Exact expressions, in the form of infinite series expansions, are given for the first and second moments of two well known generalized ridge estimators. These series expansions are then evaluated using recursive formulas and computations are verified using approximations. Results are presented for the relative mean square error and bias of these estimators as well as their relative efficiency with respect to least squares.     

Bootstrapping estimators for the seemingly unrelated regressions model

In this paper we consider the possibility of using the bootstrap to estimate the finite sample variability of feasible generalized least squares and improved estimators applied to the seemingly unrelated regressions model. The improved estimators we employ include members of the Stein-rule family and a hierarchical Bayes estimator proposed by Blattberg and George (1991). Simulation experiments are carried out using several SUR examples as well as a very large example based on the price-promotion model, and data, from marketing research.     

The relative performances of improved ridge estimators and an empirical bayes estimator: some monte carlo results

The relative 'performances of improved ridge estimators and an empirical Bayes estimator are studied by means of Monte Carlo simulations. The empirical Bayes method is seen to perform consistently better in terms of smaller MSE and more accurate empirical coverage than any of the estimators considered here. A bootstrap method is proposed to obtain more reliable estimates of the MSE of ridge esimators. Some theorems on the bootstrap for the ridge estimators are also given and they are used to provide an analytical understanding of the proposed bootstrap procedure. Empirical coverages of the ridge estimators based on the proposed procedure are generally closer to the nominal coverage when compared to their earlier counterparts. In general, except for a few cases, these coverages are still less accurate than the empirical coverages of the empirical Bayes estimator.     

A Jackknifed estimators for the negative binomial regression model

Shrinkage estimator is a commonly applied solution to the general problem caused by multicollinearity. Recently, the ridge regression (RR) estimators for estimating the ridge parameter k in the negative binomial (NB) regression have been proposed. The Jackknifed estimators are obtained to remedy the multicollinearity and reduce the bias. A simulation study is provided to evaluate the performance of estimators. Both mean squared error (MSE) and the percentage relative error (PRE) are considered as the performance criteria. The simulated result indicated that some of proposed Jackknifed estimators should be preferred to the ML method and ridge estimators to reduce MSE and bias.     

Robustness of theil's mixed regression estimators

An efficient method for incorporating incomplete prior information in regression analysis was developed by Theil [1963]. In this paper we take up the estimator of coefficients given by this procedure and study its robustness to departures from normality of prior estimators of coefficients. The use of incomplete or biased prior information in regression analysis is also considered and a new estimator for the regression coefficient is suggested.     

Alternative estimators in logistic regression when the data are collinear

Logistic regression using conditional maximum likelihood estimation has recently gained widespread use. Many of the applications of logistic regression have been in situations in which the independent variables are collinear. It is shown that collinearity among the independent variables seriously effects the conditional maximum likelihood estimator in that the variance of this estimator is inflated in much the same way that collinearity inflates the variance of the least squares estimator in multiple regression. Drawing on the similarities between multiple and logistic regression several alternative estimators, which reduce the effect of the collinearity and are easy to obtain in practice, are suggested and compared in a simulation study.     

Operational ridge regression estimators under the prediction goal

In this paper we study the Mean Square Error and Conditional Mean Forecasting of Operational Ordinary Ridge Estimator. We use the G( ) functions to provide both the exact and the approximate bias and Mean Square Error of ordinary ridge estimator (ORE), We show, among other things, that ORE dominates OLS up to a certain order of approximation under the conditional mean forecasting sense.     

A simulation study of ridge and other regression estimators

We consider a number of estimators of regression coefficients, all of generalized ridge, or 'shrinkage' type. Results of a simulation study indicate that with respect to two commonly used mean square error criteria, two ordinary ridge estimators, one proposed by Hoerl, Kennard and Baldwin, and the other introduced here, perform substantially better than both least squares and the other estimators discussed here     

A linear regression with unobserved dependent variables

Let (?,X) be a random vector such that E(X|?) = ? and Var(x|?) a + b? + c?² for some known constants a, b and c. Assume X₁,…,X_n are independent observations which have the same distribution as X. Let t(X) be the linear regression of ? on X. The linear empirical Bayes estimator is used to approximate the linear regression function. It is shown that under appropriate conditions, the linear empirical Bayes estimator approximates the linear regression well in the sense of mean squared error.     

Interval shrinkage estimators

This paper considers estimation of an unknown distribution parameter in situations where we believe that the parameter belongs to a finite interval. We propose for such situations an interval shrinkage approach which combines in a coherent way an unbiased conventional estimator and non-sample information about the range of plausible parameter values. The approach is based on an infeasible interval shrinkage estimator which uniformly dominates the underlying conventional estimator with respect to the mean square error criterion. This infeasible estimator allows us to obtain useful feasible counterparts. The properties of these feasible interval shrinkage estimators are illustrated both in a simulation study and in empirical examples.     

Fractional principal components regression: a general approach to biased estimators

Several biased estimators have been proposed as alternatives to the least squares estimator when multicollinearity is present in the multiple linear regression model. The ridge estimator and the principal components estimator are two techniques that have been proposed for such problems. In this paper the class of fractional principal component estimators is developed for the multiple linear regression model. This class contains many of the biased estimators commonly used to combat multicollinearity. In the fractional principal components framework, two new estimation techniques are introduced. The theoretical performances of the new estimators are evaluated and their small sample properties are compared via simulation with the ridge, generalized ridge and principal components estimators     

Comparison of regression estimators using Pitman measures of nearness

Received: August 5, 1999; revised version: June 14, 2000