MORE ON THE PRE-TEST ESTIMATOR IN RIDGE REGRESSION

ABSTRACT

The problem of estimation of the regression coefficients in a multiple regression model is considered under a multicollinearity situation when it is suspected that the regression coefficients may be restricted to a subspace. The objective of this paper is to compare the usual preliminary test estimator and the preliminary test ridge regression estimator in the sense of the dispersion matrix of one dominating that of the other. In particular we proved two results giving necessary and sufficient conditions for the superiority of the preliminary test ridge regression estimator over the preliminary test estimator associated with the δ = 0 (or Δ = 0) and δ ≠ 0 (or Δ ≠ 0).     

THE DISTRIBUTION OF STOCHASTIC SHRINKAGE PARAMETERS IN RIDGE REGRESSION

ABSTRACT

In this article we derive the density and distribution functions of the stochastic shrinkage parameters of three well known operational Ridge Regression (RR) estimators by assuming normality. The stochastic behavior of these parameters is likely to affect the properties of the resulting RR estimator, therefore such knowledge can be useful in the selection of the shrinkage rule. Some numerical calculations are carried out to illustrate the behavior of these distributions, throwing light on the performance of the different RR estimators.     

A new biased estimator based on ridge estimation

In this paper we introduce a new biased estimator for the vector of parameters in a linear regression model and discuss its properties. We show that our new biased estimator is superior, in the mean square error(mse) sense, to the ordinary least squares (OLS) estimator, the ordinary ridge regression (ORR) estimator and the Liu estimator. We also compare the performance of our new biased estimator with two other special Liu-type estimators proposed in Liu (2003). We illustrate our findings with a numerical example based on the widely analysed dataset on Portland cement.     

ON THE USE OF THE STEIN VARIANCE ESTIMATOR IN THE DOUBLE k-CLASS ESTIMATOR IN REGRESSION

This paper investigates the predictive mean squared error performance of a modified double k-class estimator by incorporating the Stein variance estimator. Recent studies show that the performance of the Stein rule estimator can be improved by using the Stein variance estimator. However, as we demonstrate below, this conclusion does not hold in general for all members of the double k-class estimators. On the other hand, an estimator is found to have smaller predictive mean squared error than the Stein variance-Stein rule estimator, over quite large parts of the parameter space.     

THE EXAMINATION AND ANALYSIS OF RESIDUALS FOR SOME BIASED ESTIMATORS IN LINEAR REGRESSION

In the presence of collinearity certain biased estimation procedures like ridge regression, generalized inverse estimator, principal component regression, Liu estimator, or improved ridge and Liu estimators are used to improve the ordinary least squares (OLS) estimates in the linear regression model. In this paper new biased estimator (Liu estimator), almost unbiased (improved) Liu estimator and their residuals will be analyzed and compared with OLS residuals in terms of mean-squared error.     

THE SMALL SAMPLE PROPERTIES OF THE PRINCIPAL COMPONENTS ESTIMATOR FOR REGRESSION COEFFICIENTS

ABSTRACT

Under the mean square error matrix (MSEM) criterion and Pitman closeness (PC) criterion, the principal components estimator (PRCE) is compared with the least square estimator (LSE) and the superiority of PRCE over LSE is achieved respectively. Finally, we examine the superiority of PRCE predictor over the LSE predictor based on these two criteria.     

THE EXACT BIAS OF THE LOG-PERIODOGRAM REGRESSION ESTIMATOR

The paper makes two contributions. First, we provide a formula for the exact distribution of the periodogram evaluated at any arbitrary frequency, when the sample is taken from any zero-mean stationary Gaussian process. The inadequacy of the asymptotic distribution is demonstrated through an example in which the observations are generated by a fractional Gaussian noise process. The results are then applied in deriving the exact bias of the log-periodogram regression estimator (Geweke and Porter-Hudak (1983), Robinson (1995)). The formula is computable. Practical bounds on this bias are developed and their arithmetic mean is shown to be accurate and useful.     

THE SAMPLING PERFORMANCE OF INEQUALITY RESTRICTED AND PRE-TEST ESTIMATORS IN A MIS-SPECIFIED LINEAR MODEL

The literature on the sampling properties of the inequality restricted and pre-test estimators typically assumes a properly specified model and focuses on the estimation of the regression coefficient vector. In this paper, we derive and evaluate the risk functions of these estimators for both the prediction vector and the disturbance variance in a model which is mis-specified through the exclusion of relevant regressors. The results suggest that unrestricted estimation is generally preferable to pre-testing or naively imposing restrictions.     

SEPARATE VERSUS SYSTEM METHODS OF STEIN-RULE ESTIMATION IN SEEMINGLY UNRELATED REGRESSION MODELS

ABSTRACT

Despite the sizeable literature associated with the seemingly unrelated regression models, not much is known about the use of Stein-rule estimators in these models. This gap is remedied in this paper, in which two families of Stein-rule estimators in seemingly unrelated regression equations are presented and their large sample asymptotic properties explored and evaluated. One family of estimators uses a shrinkage factor obtained solely from the equation under study while the other has a shrinkage factor based on all the equations of the model. Using a quadratic loss measure and Monte-Carlo sampling experiments, the finite sample risk performance of these estimators is also evaluated and compared with the traditional feasible generalized least squares estimator.     

A New Two-Parameter Estimator in Linear Regression

This article is concerned with the parameter estimation in linear regression model. To overcome the multicollinearity problem, a new two-parameter estimator is proposed. This new estimator is a general estimator which includes the ordinary least squares (OLS) estimator, the ridge regression (RR) estimator, and the Liu estimator as special cases. Necessary and sufficient conditions for the superiority of the new estimator over the OLS, RR, Liu estimators, and the two-parameter estimator proposed by Ozkale and Kaciranlar (2007 Ozkale , M. R. , Kaciranlar , S. ( 2007 ). The restricted and unrestricted two-parameter estimators . Commun. Statist. Theor. Meth. 36 : 2707 – 2725 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) in the mean squared error matrix (MSEM) sense are derived. Furthermore, we obtain the estimators of the biasing parameters and give a numerical example to illustrate some of the theoretical results.     

Performance of some new preliminary test ridge regression estimators and their properties

The problem of estimation of the regression coefficients in a multiple regression model is considered under multicollinearity situation when it is suspected that the regression coefficients may be restricted to a subspace. We present the estimators of the regression coefficients combining the idea of preliminary test and ridge regression methodology. Accordingly, we consider three estimators, namely, the unrestricted ridge regression estimator (URRE), the restricted ridge regression estimator (RRRE), and finally, the preliminary test ridge regression estimator (PTRRE). The biases, variancematrices and mean square errors (mse) of the estimators are derived and compared with the usual estimators. Regions of optimality of the estimators are determined by studying the mse criterion. The conditions of superiority of the estimators over the traditional estimators as in Saleh and Han (1990) and Ali and Saleh (1991) have also been discussed.     

THE OPTIMAL CRITICAL VALUE OF A PRE-TEST FOR AN INEQUALITY RESTRICTION IN A MIS-SPECIFIED REGRESSION MODEL

Optimal critical values are derived for a pre-test of an inequality restriction in a model where relevant regressors are unwittingly omitted. The criterion adopted is either that of the minimum average relative risk, or that of the mini-max regret. The latter approach yields an optimal critical value which is sensitive to the degree of model mis-specification, while the former criterion always leads to the choice of the unrestricted estimator.     

A pre-test like estimator dominating the least-squares method

We develop a pre-test type estimator of a deterministic parameter vector β$\mathit{\beta }$ in a linear Gaussian regression model. In contrast to conventional pre-test strategies, that do not dominate the least-squares (LS) method in terms of mean-squared error (MSE), our technique is shown to dominate LS when the effective dimension is greater than or equal to 4. Our estimator is based on a simple and intuitive approach in which we first determine the linear minimum MSE (MMSE) estimate that minimizes the MSE. Since the unknown vector β$\mathit{\beta }$ is deterministic, the MSE, and consequently the MMSE solution, will depend in general on β$\mathit{\beta }$ and therefore cannot be implemented. Instead, we propose applying the linear MMSE strategy with the LS substituted for the true value of β$\mathit{\beta }$ to obtain a new estimate. We then use the current estimate in conjunction with the linear MMSE solution to generate another estimate and continue iterating until convergence. As we show, the limit is a pre-test type method which is zero when the norm of the data is small, and is otherwise a non-linear shrinkage of LS.     

Estimation in linear models using gradient descent with early stopping

A new shrinkage estimator of the coefficients of a linear model is derived. The estimator is motivated by the gradient-descent algorithm used to minimize the sum of squared errors and results from early stopping of the algorithm. The statistical properties of the estimator are examined and compared with other well-established methods such as least squares and ridge regression, both analytically and through a simulation study. An important result is that the new estimator is shown to be comparable to other shrinkage estimators in terms of mean squared error of parameters and of predictions, and superior under certain circumstances.Supported by the Greek State Scholarships Foundation     

KERNEL DENSITY ESTIMATION WHEN THE BANDWIDTH IS LARGE

The performance of kernel density estimation, in terms of mean integrated squared error, is investigated in the opposite of the usual situation, namely when the bandwidth is large. This affords noteworthy insights including the special role taken by the normal density function as kernel and a tie-in with ‘semiparametric’ approaches to density estimation.     

ON ASYMPTOTIC PROPERTIES OF A TWO DIMENSIONAL FREQUENCY ESTIMATOR

Estimating parameters of a two dimensional frequency model is an important problem in statistical signal processing. In this paper, we consider the two-dimensional frequency model in presence of an additive stationary noise. We consider two different estimators and obtain their asymptotic properties. The asymptotic properties can be used to construct confidence intervals of the unknown parameters and for testing purposes also. The small sample performances of these estimators are observed using numerical simulations.     

A sufficient condition for the MSE dominance of the positive-part shrinkage estimator when each individual regression coefficient is estimated in a misspecified linear regression model

In this paper, assuming that there exist omitted explanatory variables in the specified model, we derive the exact formula for the mean squared error (MSE) of a general family of shrinkage estimators for each individual regression coefficient. It is shown analytically that when our concern is to estimate each individual regression coefficient, the positive-part shrinkage estimators have smaller MSE than the original shrinkage estimators under some conditions even when the relevant regressors are omitted. Also, by numerical evaluations, we showed the effects of our theorem for several specific cases. It is shown that the positive-part shrinkage estimators have smaller MSE than the original shrinkage estimators for wide region of parameter space even when there exist omitted variables in the specified model.     

A family of shrinkage estimators for the square of mean in normal distribution

This paper is devoted to the problem of estimating the square of population mean (μ²) in normal distribution when a prior estimate or guessed value σ₀ ² of the population variance σ² is available. We have suggested a family of shrinkage estimators , say, for μ² with its mean squared error formula. A condition is obtained in which the suggested estimator is more efficient than Srivastava et al’s (1980) estimator T_min. Numerical illustrations have been carried out to demonstrate the merits of the constructed estimator over T_min. It is observed that some of these estimators offer improvements over T_min particularly when the population is heterogeneous and σ² is in the vicinity of σ₀ ².     

RANKED SET SAMPLING FROM LOCATION-SCALE FAMILIES OF SYMMETRIC DISTRIBUTIONS

Statistical inference based on ranked set sampling has primarily been motivated by nonparametric problems. However, the sampling procedure can provide an improved estimator of the population mean when the population is partially known. In this article, we consider estimation of the population mean and variance for the location-scale families of distributions. We derive and compare different unbiased estimators of these parameters based on rindependent replications of a ranked set sample of size n.Large sample properties, along with asymptotic relative efficiencies, help identify which estimators are best suited for different location-scale distributions.     

Small Area Estimation for Zero-Inflated Data

The commonly used method of small area estimation (SAE) under a linear mixed model may not be efficient if data contain substantial proportion of zeros than would be expected under standard model assumptions (hereafter zero-inflated data). The authors discuss the SAE for zero-inflated data under a two-part random effects model that account for excess zeros in the data. Empirical results show that proposed method for SAE works well and produces an efficient set of small area estimates. An application to real survey data from the National Sample Survey Office of India demonstrates the satisfactory performance of the method. The authors describe a parametric bootstrap method to estimate the mean squared error (MSE) of the proposed estimator of small areas. The bootstrap estimates of the MSE are compared to the true MSE in simulation study.