MSE performance of the weighted average estimators consisting of shrinkage estimators when each individual regression coefficient is estimated

In this paper, we analytically derive the exact formula for the mean squared error (MSE) of two weighted average (WA) estimators for each individual regression coefficient. Further, we execute numerical evaluations to investigate small sample properties of the WA estimators, and compare the MSE performance of the WA estimators with the other shrinkage estimators and the usual OLS estimator. Our numerical results show that (1) the WA estimators have smaller MSE than the other shrinkage estimators and the OLS estimator over a wide region of parameter space; (2) the range where the relative MSE of the WA estimator is smaller than that of the OLS estimator gets narrower as the number of explanatory variables k increases.     

Shrinkage and penalized estimators in weighted least absolute deviations regression models

In this paper, we consider the estimation problem of the weighted least absolute deviation (WLAD) regression parameter vector when there are some outliers or heavy-tailed errors in the response and the leverage points in the predictors. We propose the pretest and James–Stein shrinkage WLAD estimators when some of the parameters may be subject to certain restrictions. We derive the asymptotic risk of the pretest and shrinkage WLAD estimators and show that if the shrinkage dimension exceeds two, the asymptotic risk of the shrinkage WLAD estimator is strictly less than the unrestricted WLAD estimator. On the other hand, the risk of the pretest WLAD estimator depends on the validity of the restrictions on the parameters. Furthermore, we study the WLAD absolute shrinkage and selection operator (WLAD-LASSO) and compare its relative performance with the pretest and shrinkage WLAD estimators. A simulation study is conducted to evaluate the performance of the proposed estimators relative to that of the unrestricted WLAD estimator. A real-life data example using body fat study is used to illustrate the performance of the suggested estimators.     

Relative performance of stein-rule and preliminary test estimators in linear models least squares theory

In the classical (univariare) linear model, bearing the plausibility of a subset of the regression parameters being close to a pivot, shrinkage least squares estimation of the complementary subset is considered. Based on the usual James-Stein rule, shrinkage least squares estimators are constructed, and under an asymptotic setup (allowing the shrinkage parameters to be 'close to ' the pivot), the relative performance of such estimators and the prcliminary test estimators is studied. In this context, the normality of the errors is also avoided under the same asymptotic setup. None of the shrinkage and preliminary test estimators may dominate the other (in the light of the asymptotic distributional risk criterion, as has been developed here), though each of them fares well relative to the classical least squeres estimator. The chice of the shrinkage factor is also examined properly.     

Shrinkage estimation of location parameters in a multivariate skew-normal distribution

Abstract

This paper studies decision theoretic properties of Stein type shrinkage estimators in simultaneous estimation of location parameters in a multivariate skew-normal distribution with known skewness parameters under a quadratic loss. The benchmark estimator is the best location equivariant estimator which is minimax. A class of shrinkage estimators improving on the best location equivariant estimator is constructed when the dimension of the location parameters is larger than or equal to four. An empirical Bayes estimator is also derived, and motivated from the Bayesian procedure, we suggest a simple skew-adjusted shrinkage estimator and show its dominance property. The performances of these estimators are investigated by simulation.     

Model selection and post estimation based on a pretest for logistic regression models

ABSTRACT

This article addresses the problem of parameter estimation of the logistic regression model under subspace information via linear shrinkage, pretest, and shrinkage pretest estimators along with the traditional unrestricted maximum likelihood estimator and restricted estimator. We developed an asymptotic theory for the linear shrinkage and pretest estimators and compared their relative performance using the notion of asymptotic distributional bias and asymptotic quadratic risk. The analytical results demonstrated that the proposed estimation strategies outperformed the classical estimation strategies in a meaningful parameter space. Detailed Monte-Carlo simulation studies were conducted for different combinations and the performance of each estimation method was evaluated in terms of simulated relative efficiency. The results of the simulation study were in strong agreement with the asymptotic analytical findings. Two real-data examples are also given to appraise the performance of the estimators.     

Simulated deficiencies of robust estimators of a proportionality constant

In linear regression, robust methods are at the beginning of their use in practice. In the small sample case, such robust methods provide a necessary measure of protection against deviations from the assumed error distribution. This paper studies through simulation the deficiencies of bioptimal estimators and compares them with more common methods like Huber's estimator or Tukey's estimator. Polyoptimal estimators are convex combinations of Pitman estimators and are optimally robust for a confrontation containing several shapes. The word confrontation is due to J.W. Tukey. It expresses the situation when compromising two or several error distributions. The paper uses the confrontation containing the Gaussian distribution along with a symmetric heavy-tailed distribution having a tail of order 0(t^-2) as t→ ±∞.     

Application of shrinkage estimation in linear regression models with autoregressive errors

In this paper, we consider the shrinkage and penalty estimation procedures in the linear regression model with autoregressive errors of order p when it is conjectured that some of the regression parameters are inactive. We develop the statistical properties of the shrinkage estimation method including asymptotic distributional biases and risks. We show that the shrinkage estimators have a significantly higher relative efficiency than the classical estimator. Furthermore, we consider the two penalty estimators: least absolute shrinkage and selection operator (LASSO) and adaptive LASSO estimators, and numerically compare their relative performance with that of the shrinkage estimators. A Monte Carlo simulation experiment is conducted for different combinations of inactive predictors and the performance of each estimator is evaluated in terms of the simulated mean-squared error. This study shows that the shrinkage estimators are comparable to the penalty estimators when the number of inactive predictors in the model is relatively large. The shrinkage and penalty methods are applied to a real data set to illustrate the usefulness of the procedures in practice.     

A lower bound for the risk of classes of shrinkage estimators ina general multivariate estimation problem and some deduced estimators

Given a general statistical model and an arbitrary quadratic loss, we propose a lower bound for the associated risk of a class of shrinkage estimators. With respect to the considered class of shrinkage estimators, this bound is optimal.In the particular case of the estimation of the location parameter of an ellipti-cally symmetric distribution, this bound can be used to find the relative improvement brought by a given estimator and the remaining possible improvement, using a Monte-Carlo method. We deduce from these results a new type of shrinkage estimators whose risk can be as close as one wants of the lower bound near a chosen pole and yet remain bounded. Some of them are good alternatives to the positive-part James-Stein estimator.     

Comparison of causal effect estimators under exposure misclassification

Over the past decades, various principles for causal effect estimation have been proposed, all differing in terms of how they adjust for measured confounders: either via traditional regression adjustment, by adjusting for the expected exposure given those confounders (e.g., the propensity score), or by inversely weighting each subject's data by the likelihood of the observed exposure, given those confounders. When the exposure is measured with error, this raises the question whether these different estimation strategies might be differently affected and whether one of them is to be preferred for that reason. In this article, we investigate this by comparing inverse probability of treatment weighted (IPTW) estimators and doubly robust estimators for the exposure effect in linear marginal structural mean models (MSM) with G-estimators, propensity score (PS) adjusted estimators and ordinary least squares (OLS) estimators for the exposure effect in linear regression models. We find analytically that these estimators are equally affected when exposure misclassification is independent of the confounders, but not otherwise. Simulation studies reveal similar results for time-varying exposures and when the model of interest includes a logistic link.     

Admissibility of linear estimators with respect to restricted parameter sets

In this paper the admissibility of a linear estimator for a linear regression parameter is characterized for such cases, where the considered parameter varies in an ellipsoid. We obtain a certain subset of the set of all linear estimators which are admissible with respect to the unrestricted parameter set. Furthermore, various linear estimators which have been proposed for improving the least squares estimator in cases of a restricted parameter set are investigated for admissibility. It turns out that only some shrunken estimators and some estimators of ridge type are admissible, whereas the KUKS-OLMAN estimator and all estimators of MARQUARDT type can be improved.     

On the shrinkage of local linear curve estimators

Local linear curve estimators are typically constructed using a compactly supported kernel, which minimizes edge effects and (in the case of the Epanechnikov kernel) optimizes asymptotic performance in a mean square sense. The use of compactly supported kernels can produce numerical problems, however. A common remedy is ridging, which may be viewed as shrinkage of the local linear estimator towards the origin. In this paper we propose a general form of shrinkage, and suggest that, in practice, shrinkage be towards a proper curve estimator. For the latter we propose a local linear estimator based on an infinitely supported kernel. This approach is resistant against selection of too large a shrinkage parameter, which can impair performance when shrinkage is towards the origin. It also removes problems of numerical instability resulting from using a compactly supported kernel, and enjoys very good mean squared error properties.     

Interval shrinkage estimators

This paper considers estimation of an unknown distribution parameter in situations where we believe that the parameter belongs to a finite interval. We propose for such situations an interval shrinkage approach which combines in a coherent way an unbiased conventional estimator and non-sample information about the range of plausible parameter values. The approach is based on an infeasible interval shrinkage estimator which uniformly dominates the underlying conventional estimator with respect to the mean square error criterion. This infeasible estimator allows us to obtain useful feasible counterparts. The properties of these feasible interval shrinkage estimators are illustrated both in a simulation study and in empirical examples.     

Shrinkage and pretest estimators for longitudinal data analysis under partially linear models

In this paper, we develop marginal analysis methods for longitudinal data under partially linear models. We employ the pretest and shrinkage estimation procedures to estimate the mean response parameters as well as the association parameters, which may be subject to certain restrictions. We provide the analytic expressions for the asymptotic biases and risks of the proposed estimators, and investigate their relative performance to the unrestricted semiparametric least-squares estimator (USLSE). We show that if the dimension of association parameters exceeds two, the risk of the shrinkage estimators is strictly less than that of the USLSE in most of the parameter space. On the other hand, the risk of the pretest estimator depends on the validity of the restrictions of association parameters. A simulation study is conducted to evaluate the performance of the proposed estimators relative to that of the USLSE. A real data example is applied to illustrate the practical usefulness of the proposed estimation procedures.     

On a class of shrinkage estimators of vector of regression coefficients the

This paper studies a class of shrinkage estimators of the vector of regression coefficients. The small disturbance approximations for the bias and the mean squared error matrix of the estimator are derived. In the sense of mean squared error, these estimators dominate the least squares estimator and the generalized Stein estimator developed by Hosmane (1988).     

The relative efficiency of the restricted estimators in linear regression models

This paper deals with the problem of multicollinearity in a multiple linear regression model with linear equality restrictions. The restricted two parameter estimator which was proposed in case of multicollinearity satisfies the restrictions. The performance of the restricted two parameter estimator over the restricted least squares (RLS) estimator and the ordinary least squares (OLS) estimator is examined under the mean square error (MSE) matrix criterion when the restrictions are correct and not correct. The necessary and sufficient conditions for the restricted ridge regression, restricted Liu and restricted shrunken estimators, which are the special cases of the restricted two parameter estimator, to have a smaller MSE matrix than the RLS and the OLS estimators are derived when the restrictions hold true and do not hold true. Theoretical results are illustrated with numerical examples based on Webster, Gunst and Mason data and Gorman and Toman data. We conduct a final demonstration of the performance of the estimators by running a Monte Carlo simulation which shows that when the variance of the error term and the correlation between the explanatory variables are large, the restricted two parameter estimator performs better than the RLS estimator and the OLS estimator under the configurations examined.     

On the comparison of the pre-test and shrinkage estimators for the univariate normal mean

The estimation of the mean of an univariate normal population with unknown variance is considered when uncertain non-sample prior information is available. Alternative estimators are defined to incorporate both the sample as well as the non-sample information in the estimation process. Some of the important statistical properties of the restricted, preliminary test, and shrinkage estimators are investigated. The performances of the estimators are compared based on the criteria of unbiasedness and mean square error in order to search for a ‘best’ estimator. Both analytical and graphical methods are explored. There is no superior estimator that uniformly dominates the others. However, if the non-sample information regarding the value of the mean is close to its true value, the shrinkage estimator over performs the rest of the estimators. Received: June 19, 1999; revised version: March 23, 2000     

Nonparametric regression estimators in complex surveys

In this article, we extend smoothing splines to model the regression mean structure when data are sampled through a complex survey. Smoothing splines are evaluated both with and without sample weights, and are compared with local linear estimator. Simulation studies find that nonparametric estimators perform better when sample weights are incorporated, rather than being treated as if iid. They also find that smoothing splines perform better than local linear estimator through completely data-driven bandwidth selection methods.     

Efficient estimation for time series following generalized linear models

In this paper, we consider James–Stein shrinkage and pretest estimation methods for time series following generalized linear models when it is conjectured that some of the regression parameters may be restricted to a subspace. Efficient estimation strategies are developed when there are many covariates in the model and some of them are not statistically significant. Statistical properties of the pretest and shrinkage estimation methods including asymptotic distributional bias and risk are developed. We investigate the relative performances of shrinkage and pretest estimators with respect to the unrestricted maximum partial likelihood estimator (MPLE). We show that the shrinkage estimators have a lower relative mean squared error as compared to the unrestricted MPLE when the number of significant covariates exceeds two. Monte Carlo simulation experiments were conducted for different combinations of inactive covariates and the performance of each estimator was evaluated in terms of its mean squared error. The practical benefits of the proposed methods are illustrated using two real data sets.     

Comparison of preliminary test estimators based on generalized order statistics from proportional hazard family using Pitman measure of closeness

In this article, based on generalized order statistics from a family of proportional hazard rate model, we use a statistical test to generate a class of preliminary test estimators and shrinkage preliminary test estimators for the proportionality parameter. These estimators are compared under Pitman measure of closeness (PMC) as well as MSE criteria. Although the PMC suffers from non transitivity, in the first class of estimators, it has the transitivity property and we obtain the Pitman-closest estimator. Analytical and graphical methods are used to show the range of parameter in which preliminary test and shrinkage preliminary test estimators perform better than their competitor estimators. Results reveal that when the prior information is not too far from its real value, the proposed estimators are superior based on both mentioned criteria.     

SHRINKAGE, PRETEST AND ABSOLUTE PENALTY ESTIMATORS IN PARTIALLY LINEAR MODELS

We consider a partially linear model in which the vector of coefficients β in the linear part can be partitioned as ( β ₁, β ₂) , where β ₁ is the coefficient vector for main effects (e.g. treatment effect, genetic effects) and β ₂ is a vector for ‘nuisance’ effects (e.g. age, laboratory). In this situation, inference about β ₁ may benefit from moving the least squares estimate for the full model in the direction of the least squares estimate without the nuisance variables (Steinian shrinkage), or from dropping the nuisance variables if there is evidence that they do not provide useful information (pretesting). We investigate the asymptotic properties of Stein‐type and pretest semiparametric estimators under quadratic loss and show that, under general conditions, a Stein‐type semiparametric estimator improves on the full model conventional semiparametric least squares estimator. The relative performance of the estimators is examined using asymptotic analysis of quadratic risk functions and it is found that the Stein‐type estimator outperforms the full model estimator uniformly. By contrast, the pretest estimator dominates the least squares estimator only in a small part of the parameter space, which is consistent with the theory. We also consider an absolute penalty‐type estimator for partially linear models and give a Monte Carlo simulation comparison of shrinkage, pretest and the absolute penalty‐type estimators. The comparison shows that the shrinkage method performs better than the absolute penalty‐type estimation method when the dimension of the β ₂ parameter space is large.