Choosing shrinkage estimators for regression problems

A Bayesian formulation of the canonical form of the standard regression model is used to compare various Stein-type estimators and the ridge estimator of regression coefficients, A particular (“constant prior”) Stein-type estimator having the same pattern of shrinkage as the ridge estimator is recommended for use.     

Model selection and parameter estimation of a multinomial logistic regression model

In the multinomial regression model, we consider the methodology for simultaneous model selection and parameter estimation by using the shrinkage and LASSO (least absolute shrinkage and selection operation) [R. Tibshirani, Regression shrinkage and selection via the LASSO, J. R. Statist. Soc. Ser. B 58 (1996), pp. 267–288] strategies. The shrinkage estimators (SEs) provide significant improvement over their classical counterparts in the case where some of the predictors may or may not be active for the response of interest. The asymptotic properties of the SEs are developed using the notion of asymptotic distributional risk. We then compare the relative performance of the LASSO estimator with two SEs in terms of simulated relative efficiency. A simulation study shows that the shrinkage and LASSO estimators dominate the full model estimator. Further, both SEs perform better than the LASSO estimators when there are many inactive predictors in the model. A real-life data set is used to illustrate the suggested shrinkage and LASSO estimators.     

Combining coordinates in simultaneous estimation of normal means

The problem of combining coordinates in Stein-type estimators, when simultaneously estimating normal means, is considered. The question of deciding whether to use all coordinates in one combined shrinkage estimator or to separate into groups and use separate shrinkage estimators on each group is considered. A Bayesian viewpoint is (of necessity) taken, and it is shown that the ‘combined’ estimator is, somewhat surprisingly, often superior.     

On the choice of co-ordinates in simultaneous estimation of normal means under misspecification of normal priors

The problem of choice of coordinates in Stein-type estimators,when simultaneously estimating normal means, is considered. The question of deciding whether to use all coordinates in one combined shrinkage estimators or to separate into groups and use separate shrinkage estimators on each group is considered in the situation in which part of the prior information may be " misspecified". It is observed that the amount of misspecification determines whether to use the combined shrinkage estimator the separate shrinkage estimator.     

Application of shrinkage estimation in linear regression models with autoregressive errors

In this paper, we consider the shrinkage and penalty estimation procedures in the linear regression model with autoregressive errors of order p when it is conjectured that some of the regression parameters are inactive. We develop the statistical properties of the shrinkage estimation method including asymptotic distributional biases and risks. We show that the shrinkage estimators have a significantly higher relative efficiency than the classical estimator. Furthermore, we consider the two penalty estimators: least absolute shrinkage and selection operator (LASSO) and adaptive LASSO estimators, and numerically compare their relative performance with that of the shrinkage estimators. A Monte Carlo simulation experiment is conducted for different combinations of inactive predictors and the performance of each estimator is evaluated in terms of the simulated mean-squared error. This study shows that the shrinkage estimators are comparable to the penalty estimators when the number of inactive predictors in the model is relatively large. The shrinkage and penalty methods are applied to a real data set to illustrate the usefulness of the procedures in practice.     

Adaptive LASSO estimation for ARDL models with GARCH innovations

ABSTRACT

In this paper, we show the validity of the adaptive least absolute shrinkage and selection operator (LASSO) procedure in estimating stationary autoregressive distributed lag(p,q) models with innovations in a broad class of conditionally heteroskedastic models. We show that the adaptive LASSO selects the relevant variables with probability converging to one and that the estimator is oracle efficient, meaning that its distribution converges to the same distribution of the oracle-assisted least squares, i.e., the least square estimator calculated as if we knew the set of relevant variables beforehand. Finally, we show that the LASSO estimator can be used to construct the initial weights. The performance of the method in finite samples is illustrated using Monte Carlo simulation.     

Risk behavior of a pre-test estimator for normal variance with the Stein-type estimator

In this paper, we examine the risk behavior of a pre-test estimator for normal variance with the Stein-type estimator. The one-sided pre-test is conducted for the null hypothesis that the population variance is equal to a specific value, and the Stein-type estimator is used if the null hypothesis is rejected. A sufficient condition for the pre-test estimator to dominate the Stein-type estimator is shown.     

Non-penalty shrinkage estimation of random effect models for longitudinal data with AR(1) errors

In this paper, we consider the non-penalty shrinkage estimation method of random effect models with autoregressive errors for longitudinal data when there are many covariates and some of them may not be active for the response variable. In observational studies, subjects are followed over equally or unequally spaced visits to determine the continuous response and whether the response is associated with the risk factors/covariates. Measurements from the same subject are usually more similar to each other and thus are correlated with each other but not with observations of other subjects. To analyse this data, we consider a linear model that contains both random effects across subjects and within-subject errors that follows autoregressive structure of order 1 (AR(1)). Considering the subject-specific random effect as a nuisance parameter, we use two competing models, one includes all the covariates and the other restricts the coefficients based on the auxiliary information. We consider the non-penalty shrinkage estimation strategy that shrinks the unrestricted estimator in the direction of the restricted estimator. We discuss the asymptotic properties of the shrinkage estimators using the notion of asymptotic biases and risks. A Monte Carlo simulation study is conducted to examine the relative performance of the shrinkage estimators with the unrestricted estimator when the shrinkage dimension exceeds two. We also numerically compare the performance of the shrinkage estimators to that of the LASSO estimator. A longitudinal CD4 cell count data set will be used to illustrate the usefulness of shrinkage and LASSO estimators.     

Penalized and Shrinkage Estimation in the Cox Proportional Hazards Model

This article considers the shrinkage estimation procedure in the Cox's proportional hazards regression model when it is suspected that some of the parameters may be restricted to a subspace. We have developed the statistical properties of the shrinkage estimators including asymptotic distributional biases and risks. The shrinkage estimators have much higher relative efficiency than the classical estimator, furthermore, we consider two penalty estimators—the LASSO and adaptive LASSO—and compare their relative performance with that of the shrinkage estimators numerically. A Monte Carlo simulation experiment is conducted for different combinations of irrelevant predictors and the performance of each estimator is evaluated in terms of simulated mean squared error. Simulation study shows that the shrinkage estimators are comparable to the penalty estimators when the number of irrelevant predictors in the model is relatively large. The shrinkage and penalty methods are applied to two real data sets to illustrate the usefulness of the procedures in practice.     

A note on Stein-type shrinkage estimator in partial linear models

In this paper, an exact sufficient condition for the dominance of the Stein-type shrinkage estimator over the usual unbiased estimator in a partial linear model is exhibited. Comparison result is then done under the balanced loss function. It is assumed that the vector of disturbances is typically distributed according to the law belonging to the sub-class of elliptically contoured models. It is also shown that the dominance condition is robust. Furthermore, a nonparametric estimation after estimation of the linear part is added for detecting the efficiency of the obtained results.     

Positive-rule stein-type almost unbiased ridge estimator in linear regression model

Abstract

In this article, when it is suspected that regression coefficients may be restricted to a subspace, we discuss the parameter estimation of regression coefficients in a multiple regression model. Then, in order to improve the preliminary test almost ridge estimator, we study the positive-rule Stein-type almost unbiased ridge estimator based on the positive-rule stein-type shrinkage estimator and almost unbiased ridge estimator. After that, quadratic bias and quadratic risk values of the new estimator are derived and compared with some relative estimators. And we also discuss the option of parameter k. Finally, we perform a real data example and a Monte Carlo study to illustrate theoretical results.     

Meta-analysis,pretest, and shrinkage estimation of kurtosis parameters

An asymptotic theory for the improved estimation of kurtosis parameter vector is developed for multi-sample case using uncertain prior information (UPI) that several kurtosis parameters are the same. Meta-analysis is performed to obtain pooled estimator, as it is a statistical methodology for pooling quantitative evidence. Pooled estimator is a good choice when assumption of homogeneity holds but it becomes inconsistent as assumption violates, therefore pretest and Stein-type shrinkage estimators are proposed as they combine sample and nonsample information in a superior way. Asymptotic properties of suggested estimators are discussed and their risk comparisons are also mentioned.     

Improved estimation of the mean vector for student-t model

Improved James-Stein type estimation of the mean vector μ of a multovaroate Student-t population of dimension p with ν degrees of freedom is considered. In addition to the sample data, uncertain prior information on the value of the mean vector, in the form of a null hypothesis, is used for the estiamtion. The usual maximum liklihood estimator((mle) of μ is obtained and a test statistic for testing H₀:μ=μ₀ is derived. Based on the mle of μ and the tes statistic the preliminary test estimator (PTE), Stein-type shrinkage estimator (SE) and positive-rule shrinkage esiimator (PRSE) are defined. The bias and the quadratic risk of the estimators are evaiuated. The relative performances of the estimators are mvestigated by analyzing the risks under different condltlons It is observed that the FRSE dommates over he other three estimators, regardless of the vaiidity of the null hypothesis and the value ν.     

The exact distribution and density functions of the stein-type estimator for normal variance

In this paper, we derive the exact distribution and density functions of the Stein-type estimator for the normal variance. It is shown by numerical evaluation that the density function of the Stein-type estimator is unimodal and concentrates around the mode more than that of the usual estimator.     

Stein-type estimation in logistic regression models based on minimum ϕ-divergence estimators

In this paper we present a study of Stein-type estimators for the unknown parameters in logistic regression models when it is suspected that the parameters may be restricted to a subspace of the parameter space. The Stein-type estimators studied are based on the minimum phi-divergence estimator instead on the maximum likelihood estimator as well as on phi-divergence test statistics.     

Performance of the stein-type two-parameter estimator in multiple linear regression model

In this article, we consider the Stein-type approach to the estimation of the regression parameter in a multiple regression model under a multicollinearity situation. The Stein-type two-parameter estimator is proposed when it is suspected that the regression parameter may be restricted to a subspace. The bias and the quadratic risk of the proposed estimator are derived and compared with the two-parameter estimator (TPE), the restricted TPE and the preliminary test TPE. The conditions of superiority of the proposed estimator are obtained. Finally, a real data example is provided to illustrate some of the theoretical results.     

Model selection by LASSO methods in a change-point model

The paper considers a linear regression model with multiple change-points occurring at unknown times. The LASSO technique is very interesting since it allows simultaneously the parametric estimation, including the change-points estimation, and the automatic variable selection. The asymptotic properties of the LASSO-type (which has as particular case the LASSO estimator) and of the adaptive LASSO estimators are studied. For this last estimator the Oracle properties are proved. In both cases, a model selection criterion is proposed. Numerical examples are provided showing the performances of the adaptive LASSO estimator compared to the least squares estimator.     

Shrinkage and LASSO strategies in high-dimensional heteroscedastic models

ABSTRACT

In this paper, we consider the estimation problem of the parameter vector in the linear regression model with heteroscedastic errors. First, under heteroscedastic errors, we study the performance of shrinkage-type estimators and their performance as compared to theunrestricted and restricted least squares estimators. In order to accommodate the heteroscedastic structure, we generalize an identity which is useful in deriving the risk function. Thanks to the established identity, we prove that shrinkage estimators dominate the unrestricted estimator. Finally, we explore the performance of high-dimensional heteroscedastic regression estimator as compared to classical LASSO and shrinkage estimators.     

Merging data from multiple sources: pretest and shrinkage perspectives

In this article, we have developed asymptotic theory for the simultaneous estimation of the k means of arbitrary populations under the common mean hypothesis and further assuming that corresponding population variances are unknown and unequal. The unrestricted estimator, the Graybill-Deal-type restricted estimator, the preliminary test, and the Stein-type shrinkage estimators are suggested. A large sample test statistic is also proposed as a pretest for testing the common mean hypothesis. Under the sequence of local alternatives and squared error loss, we have compared the asymptotic properties of the estimators by means of asymptotic distributional quadratic bias and risk. Comprehensive Monte-Carlo simulation experiments were conducted to study the relative risk performance of the estimators with reference to the unrestricted estimator in finite samples. Two real-data examples are also furnished to illustrate the application of the suggested estimation strategies.