Shrinkage and LASSO strategies in high-dimensional heteroscedastic models

ABSTRACT

In this paper, we consider the estimation problem of the parameter vector in the linear regression model with heteroscedastic errors. First, under heteroscedastic errors, we study the performance of shrinkage-type estimators and their performance as compared to theunrestricted and restricted least squares estimators. In order to accommodate the heteroscedastic structure, we generalize an identity which is useful in deriving the risk function. Thanks to the established identity, we prove that shrinkage estimators dominate the unrestricted estimator. Finally, we explore the performance of high-dimensional heteroscedastic regression estimator as compared to classical LASSO and shrinkage estimators.     

Statistical Analysis of Non Linear Least Squares Estimation for Harmonic Signals in Multiplicative and Additive Noise

Abstract

This article concerns the stochastically constrained linear model under a biased assumption. We propose a quasi-stochastically constrained least squares estimator. Furthermore, we provide the expectation of this estimator, demonstrate its consistency and asymptotic normality. In the end of the article, the simulation study of the new estimator shows that it is superior to the least squares estimator, ridge estimator, and the linear constrained estimators under certain conditions by comparing the mean squared errors of these estimators.     

Estimating the Intercept in an Orthogonally Blocked Experiment when the Block Effects are Random

Abstract

For an orthogonally blocked experiment, Khuri [Khuri, A. I. (1992). Response surface models with random block effects. Technometrics 34:26–37] has shown that the ordinary least squares estimator, the generalized least squares estimator and the intra-block estimator of the factor effects in a response surface model with random block effects coincide. The ordinary least squares estimator ignores the blocks, whereas the generalized least squares and the intra-block estimators treat the block effects as random and fixed, respectively. As shown in this paper, the equivalence does not hold for the estimation of the intercept when the block sizes are heterogeneous. Practical examples are given to illustrate the theoretical results.     

On a class of shrinkage estimators of vector of regression coefficients the

This paper studies a class of shrinkage estimators of the vector of regression coefficients. The small disturbance approximations for the bias and the mean squared error matrix of the estimator are derived. In the sense of mean squared error, these estimators dominate the least squares estimator and the generalized Stein estimator developed by Hosmane (1988).     

SHRINKAGE, PRETEST AND ABSOLUTE PENALTY ESTIMATORS IN PARTIALLY LINEAR MODELS

We consider a partially linear model in which the vector of coefficients β in the linear part can be partitioned as ( β ₁, β ₂) , where β ₁ is the coefficient vector for main effects (e.g. treatment effect, genetic effects) and β ₂ is a vector for ‘nuisance’ effects (e.g. age, laboratory). In this situation, inference about β ₁ may benefit from moving the least squares estimate for the full model in the direction of the least squares estimate without the nuisance variables (Steinian shrinkage), or from dropping the nuisance variables if there is evidence that they do not provide useful information (pretesting). We investigate the asymptotic properties of Stein‐type and pretest semiparametric estimators under quadratic loss and show that, under general conditions, a Stein‐type semiparametric estimator improves on the full model conventional semiparametric least squares estimator. The relative performance of the estimators is examined using asymptotic analysis of quadratic risk functions and it is found that the Stein‐type estimator outperforms the full model estimator uniformly. By contrast, the pretest estimator dominates the least squares estimator only in a small part of the parameter space, which is consistent with the theory. We also consider an absolute penalty‐type estimator for partially linear models and give a Monte Carlo simulation comparison of shrinkage, pretest and the absolute penalty‐type estimators. The comparison shows that the shrinkage method performs better than the absolute penalty‐type estimation method when the dimension of the β ₂ parameter space is large.     

Confidence regions based on inefficient estimators

We consider an unbiased estimator of a function of mean value parameters, which is not efficient. This inefficient estimator is correlated with a residual vector. Thus, if a unit dispersion is unknown, it is impossible to determine the correct confidence region for a function of mean value parameters via a standard estimator of an unknown dispersion with the exception of the case when the ordinary least squares (OLS) estimator is considered in a model with a special covariance structure such that the OLS and the generalized least squares (GLS) estimator are the same, that is the OLS estimator is efficient. Two different estimators of a unit dispersion independent of an inefficient estimator are derived in a singular linear statistical model. Their quality was verified by simulations for several types of experimental designs. Two new estimators of the unit dispersion were compared with the standard estimators based on the GLS and the OLS estimators of the function of the mean value parameters. The OLS estimator was considered in the incorrect model with a different covariance matrix such that the originally inefficient estimator became efficient. The numerical examples led to a slightly surprising result which seems to be due to data behaviour. An example from geodetic practice is presented in the paper.     

A Comparison of Partially Adaptive and Reweighted Least Squares Estimation

The small sample performance of least median of squares, reweighted least squares, least squares, least absolute deviations, and three partially adaptive estimators are compared using Monte Carlo simulations. Two data problems are addressed in the paper: (1) data generated from non-normal error distributions and (2) contaminated data. Breakdown plots are used to investigate the sensitivity of partially adaptive estimators to data contamination relative to RLS. One partially adaptive estimator performs especially well when the errors are skewed, while another partially adaptive estimator and RLS perform particularly well when the errors are extremely leptokur-totic. In comparison with RLS, partially adaptive estimators are only moderately effective in resisting data contamination; however, they outperform least squares and least absolute deviation estimators.     

Estimating the Optimum of a Stochastic System using Simulation

We incorporate new techniques for obtaining unbiased estimators of gradients from single simulations of stochastic systems in optimization procedures. We develop an "enhanced" least squares estimator of the optimum which incorporates information about both the function and its gradient and improves substantially on techniques which use only the function. We also propose a sequential design to use with the enhanced least squares estimator to optimize a regression function when it is evaluated by simulation.     

Two Kinds of Restricted Estimators in Singular Linear Regression

In this article, the parameter estimators in singular linear model with linear equality restrictions are considered. The restricted root estimator and the generalized restricted root estimator are proposed and some properties of the estimators are also studied. Furthermore, we compare them with the restricted unified least squares estimator and show their sufficient conditions under which their superior over the restricted unified least squares estimator in terms of mean squares error, and discuss the choice of the unknown parameters of the generalized restricted root estimator.     

An evaluation of biased estimators of regression coefficients—A simulation study

Different versions of generalized and ordinary ridge estimators and shrinkage estimators of regression coefficients are studied in comparison with least squares estimators using simulations. The results show that some of the biased estimators considered are better than the least squares estimator in general and the improvement is substantial in some cases.     

Empirical study of robust estimation methods for PAR models with application to the air quality area

Abstract

This paper compares three estimators for periodic autoregressive (PAR) models. The first is the classical periodic Yule-Walker estimator (YWE). The second is a robust version of YWE (RYWE) which uses the robust autocovariance function in the periodic Yule-Walker equations, and the third is the robust least squares estimator (RLSE) based on iterative least squares with robust versions of the original time series. The daily mean particulate matter concentration (PM10) data is used to illustrate the methodologies in a real application, that is, in the Air Quality area.     

A test of the mean square error criterion for linear admissible estimators

A test for choosing between a linear admissible estimator and the least squares estimator (LSE) is developed. A characterization of linear admissible estimators useful for comparing estimators is presented and necessary and sufficient conditions for superiority of a linear admissible estimator over the LS estimetor is derived for the test. The test is based on the MSE matrix superiority, but also new resl?!ts concerning covariance matrix comparisons of linear estimators are derived. Further,shown that the test of Toro - Vizcarrondo and Wailace applies iioi only the restricted least squares estimators but also to certain estimators outside this class.     

Modified maximum likelihood estimator under the Jones and Faddy's skew t-error distribution for censored regression model

It is well-known that classical Tobit estimator of the parameters of the censored regression (CR) model is inefficient in case of non-normal error terms. In this paper, we propose to use the modified maximum likelihood (MML) estimator under the Jones and Faddy''s skew t-error distribution, which covers a wide range of skew and symmetric distributions, for the CR model. The MML estimators, providing an alternative to the Tobit estimator, are explicitly expressed and they are asymptotically equivalent to the maximum likelihood estimator. A simulation study is conducted to compare the efficiencies of the MML estimators with the classical estimators such as the ordinary least squares, Tobit, censored least absolute deviations and symmetrically trimmed least squares estimators. The results of the simulation study show that the MML estimators work well among the others with respect to the root mean square error criterion for the CR model. A real life example is also provided to show the suitability of the MML methodology.     

Estimation of parameters in a generalized GMANOVA model based on an outer product analogy and least squares

This paper investigates estimation of parameters in a combination of the multivariate linear model and growth curve model, called a generalized GMANOVA model. Making analogy between the outer product of data vectors and covariance yields an approach to directly do least squares to covariance. An outer product least squares estimator of covariance (COPLS estimator) is obtained and its distribution is presented if a normal assumption is imposed on the error matrix. Based on the COPLS estimator, two-stage generalized least squares estimators of the regression coefficients are derived. In addition, asymptotic normalities of these estimators are investigated. Simulation studies have shown that the COPLS estimator and two-stage GLS estimators are alternative competitors with more efficiency in the sense of sample mean, standard deviations and mean of the variance estimates to the existing ML estimator in finite samples. An example of application is also illustrated.     

Measuring the degree of severity of heteroskedasticity and the choice between the ols estimator and the 2sae

This paper dwells on the choice between the ordinary least squares and the estimated generalized least squares estimators when the presence of heteroskedasticity is suspected. Since the estimated generalized least squares estimator does not dominate the ordinary least squares estimator completely over the whole parameter space, it is of interest to the researcher to know in advance whether the degree of severity of heteroskedasticity is such that OLS estimator outperforms the estimated generalized least squares (or 2SAE). Casting the problem in the non-spherical error mold and exploiting the principle underlying the Bayesian pretest estimator, an intuitive non-mathematical procedure is proposed to serve as an aid to the researcher in deciding when to use either the ordinary least squares (OLS) or the estimated generalized least squares (2SAE) estimators.     

Adaptive Estimation Using Weighted Least Squares

This paper proposes an adaptive estimator that is more precise than the ordinary least squares estimator if the distribution of random errors is skewed or has long tails. The adaptive estimates are computed using a weighted least squares approach with weights based on the lengths of the tails of the distribution of residuals. Smaller weights are assigned to those observations that have residuals in the tails of long-tailed distributions and larger weights are assigned to observations having residuals in the tails of short-tailed distributions. Monte Carlo methods are used to compare the performance of the proposed estimator and the performance of the ordinary least squares estimator. The estimates that were studied in this simulation include the difference between the means of two populations, the mean of a symmetric distribution, and the slope of a regression line. The adaptive estimators are shown to have lower mean squared errors than those for the ordinary least squares estimators for short-tailed, long-tailed, and skewed distributions, provided the sample size is at least 20. The ordinary least squares estimator has slightly lower mean squared error for normally distributed errors. The adaptive estimator is recommended for general use for studies having sample sizes of at least 20 observations unless the random errors are known to be normally distributed.     

Robust Sparse Regression with High-Breakdown Value

Penalized least squares estimators are sensitive to the influence of outliers like the ordinary least squares estimator. We propose a sparse regression estimator for robust variable selection and estimation based on a robust initial estimator. It is proven that our estimator has at least the same breakdown value as the initial estimator. Numerical examples are presented to illustrate our method.     

Estimation of the linear-plateau segmented regression model in the presence of measurement error

It is well known that when the true values of the independent variable are unobservable due to measurement error, the least squares estimator for a regression model is biased and inconsistent. When repeated observations on each x_i are taken, consistent estimators for the linear-plateau model can be formed. The repeated observations are required to classify each observation to the appropriate line segment. Two cases of repeated observations are treated in detail. First, when a single value of y_i is observed with the repeated observations of x_i the least squares estimator using the mean of the repeated x_i observations is consistent and asymptotically normal. Second, when repeated observations on the pair (x_i, y_i ) are taken the least squares estimator is inconsistent, but two consistent estimators are proposed: one that consistently estimates the bias of the least squares estimator and adjusts accordingly; the second is the least squares estimator using the mean of the repeated observations on each pair.     

A note on minimum average risk estimators for coefficients in linear models

In this note, we have derived a set of necessary and sufficient conditions for the biased estimators analyzed by Swamy and Mehta (1976) to be better than the generalized least squares estimator of the coefficient vector in a standard linear regression model.     

Efficiency of least squares estimators in the presence of spatial autocorrelation

The effect of spatial autocorrelation on inferences made using ordinary least squares estimation is considered. It is found, in some cases, that ordinary least squares estimators provide a reasonable alternative to the estimated generalized least squares estimators recommended in the spatial statistics literature. One of the most serious problems in using ordinary least squares is that the usual variance estimators are severely biased when the errors are correlated. An alternative variance estimator that adjusts for any observed correlation is proposed. The need to take autocorrelation into account in variance estimation negates much of the advantage that ordinary least squares estimation has in terms of computational simplicity