On the minimum mean squared error estimators in a regression model

In this paper we analyze the properties of two estimators oroposed by Farebrother (1975) for linear regression models.     

Comment on hoerl and kennard's ridge regression simulation methodology

Ridge regression has received strong support in several Monte carlo studies, leading some authors to advocate its general use. It is argued, however, that these studies were strongly biased in favor of ridge regression by simulating regression coefficient vectors centered at the origin; a condition well suited for a shrinkage technique. Studies which modeled some non-zero regression coefficients and which showed only qualified support for ridge regression are cited in support of this argument. It is concluded that only to the extent that ridge regression type coefficient vectors actually underlie real data sets -a poorly understood phenomenon - will ridge regression be of use.     

Generalized mean squared error properties of regression estimators

Theobald (1974) compares Ordinary Least Squares and Ridge Regression estimators of regression parameters using a generalized mean squared error criterion. This paper presents the generalized mean squared error of a Principal Components Regression estimator and comparisons are made with each of the above estimators. In general the choice of which estimator to use depends on the magnitude and the orientation of the unknown parameter vector.     

A note on the moments of stochastic shrinkage parameters in ridge regression

The purpose of this note is to gain insight on the performance of two well known operational Ridge Regression estimators by deriving the moments of their stochastic shrinkage parameters. We also show that, under certain conditions, one of them has bounded moments.     

On prediction and mean squared error

Practical questions motivate the search for predictors either of an as yet unobserved random vector, or of a random function of a parameter. An extension of the classical UMVUE theory is presented to cover such situations. In includes a Rao-Blackwell-type theorem, a Cramer-Rao-type inequality, and necessary and sufficient conditions for a predictor to minimize the mean squared error uniformly in the parameter. Applications are considered to the problem of selected means, the species problem, and the examination of some u-v estimates of Robbins (1988).     

Minimum mean squared error estimation of each individual coefficient in a linear regression model

In this paper, we derive the exact mean squared error (MSE) of the minimum MSE estimator for each individual coefficient in a linear regression model, and show a sufficient condition for the minimum MSE estimator for each individual coefficient to dominate the OLS estimator. Numerical results show that when the number of independent variables is 2 and 3, the minimum MSE estimator for each individual coefficient can be a good alternative to the OLS and Stein-rule estimators.     

Bounds on the biasing parameter in ridge regression

One of the problems with the ridge estimator is the appropriate value for the unknown biasing parameter k. In theory k can take a range of values, though some values are preferable to others. In this note we consider the optimum value for k and give bounds on the expected value of k.     

Some properties of the relative squared error approach to linear regression analysis

In order to obtain optimal estimators in a generalized linear regression model we apply the minimax principle to the relative squared error. It turns out that this approach is equivalent to the application of the minimax principle to the absolute squared error when an ellipsoidal prior information set is given. We discuss the admissibility of these minimax estimators. Furthermore, a close relation to a Bayesian approach is derived.     

Exact small sample properties of an operational variant of the minimum mean squared error estimator

In this paper, we show a sufficient condition for an operational variant of the minimum mean squared error estimator (simply, the minimum MSE estimator) to dominate the ordinary least squares (OLS) estimator. It is also shown numerically that the minimum MSE estimator dominates the OLS estimator if the number of regression coefficients is larger than or equal to three, even if the sufficient condition is not satisfied. When the number of regression coefficients is smaller than three, our numerical results show that the gain in MSE of using the minimum MSE estimator is larger than the loss.     

On the pointwise mean squared error of a multidimensional term-by-term thresholding wavelet estimator

In this paper we provide a theoretical contribution to the pointwise mean squared error of an adaptive multidimensional term-by-term thresholding wavelet estimator. A general result exhibiting fast rates of convergence under mild assumptions on the model is proved. It can be applied for a wide range of non parametric models including possible dependent observations. We give applications of this result for the non parametric regression function estimation problem (with random design) and the conditional density estimation problem.     

Asymptotic behaviour of the mean integrated squared error of kernel density estimators for dependent observations

Let X₁, X₂, … be a strictly stationary sequence of observations, and g be the joint density of (X₁, …, X_d) for some fixed d ? 1. We consider kernel estimators of the density g. The asymptotic behaviour of the mean integrated squared error of the kernel estimators is obtained under an assumption of weak dependence between the observations.     

Reducing the mean squared error in kernel density estimation

In this article, we propose a version of a kernel density estimator which reduces the mean squared error of the existing kernel density estimator by combining bias reduction and variance reduction techniques. Its theoretical properties are investigated, and a Monte Carlo simulation study supporting theoretical results on the proposed estimator is given.     

Bootstrap mean squared error of a small-area EBLUP

Concerning the estimation of linear parameters in small areas, a nested-error regression model is assumed for the values of the target variable in the units of a finite population. Then, a bootstrap procedure is proposed for estimating the mean squared error (MSE) of the EBLUP under the finite population setup. The consistency of the bootstrap procedure is studied, and a simulation experiment is carried out in order to compare the performance of two different bootstrap estimators with the approximation given by Prasad and Rao [Prasad, N.G.N. and Rao, J.N.K., 1990, The estimation of the mean squared error of small-area estimators. Journal of the American Statistical Association, 85, 163–171.]. In the numerical results, one of the bootstrap estimators shows a better bias behavior than the Prasad–Rao approximation for some of the small areas and not much worse in any case. Further, it shows less MSE in situations of moderate heteroscedasticity and under mispecification of the error distribution as normal when the true distribution is logistic or Gumbel. The proposed bootstrap method can be applied to more general types of parameters (linear of not) and predictors.     

A comparison of mean squared error approximations for a small estimated state space model*

Summary: This paper investigates mean squared errors for unobserved states in state space models when estimation uncertainty of hyperparameters is taken into account. Three alternative approximations to mean squared errors with estimation uncertainty are compared in a Monte Carlo experiment, where the random walk with noise model serves as DGP: A naive method which neglects estimation uncertainty completely, an approximation based on an expansion around the true state with respect to the estimated parameters, and a bootstrap approach. Overall, the bootstrap method performs best in the simulations. However, the gains are not systematic, and the computationally burden of this method is relatively high.*This paper represents the authors personal opinions and does not necessarily reflect the views of the Deutsche Bundesbank. I am grateful to Malte Knüppel, Jeong-Ryeol Kurz-Kim, Karl-Heinz Tödter and a referee for helpful comments. The computer programs for this paper were written in Ox and SsfPack, see Doornik (1998) and Koopman et al. (1999). The used SsfPack version is 2.2.     

Optimal minimax squared error risk estimation of the mean of a multivariate normal distribution

Minimax squared error risk estimators of the mean of a multivariate normal distribution are characterized which have smallest Bayes risk with respect to a spherically symmetric prior distribution for (i) squared error loss, and (ii) zero-one loss depending on whether or not estimates are consistent with the hypothesis that the mean is null. In (i), the optimal estimators are the usual Bayes estimators for prior distributions with special structure. In (ii), preliminary test estimators are optimal. The results are obtained by applying the theory of minimax-Bayes-compromise decision problems.     

Note on the minimum mean integrated squared error of kernel estimates of a distribution function and its derivatives

An exact expiession for the minimum integrated squared error associated with the kernel distribution function and its derivatives is given. Furthermore, the virtual optimality of the Fourier integral estimate in density estimation, shown by Davis (1977), is extended to estimation of a distibution function and its derivatives.     

A note on non-negative mean square error estimation of regression estimators in randomized response surveys

Generalized regression estimators are considered for the survey population total of a quantitative sensitive variable based on randomized responses. Formulae are presented for ‘non-negative’ estimators of approximate mean square errors of these biased estimators when population and sample sizes are large.     

Bayes minimax ridge regression estimators

The problem of estimating of the vector β of the linear regression model y = Aβ + ? with ? ～ N_p(0, σ²I_p) under quadratic loss function is considered when common variance σ² is unknown. We first find a class of minimax estimators for this problem which extends a class given by Maruyama and Strawderman (2005 Maruyama, Y., and W. E. Strawderman. 2005. A new class of generalized Bayes minimax ridge regression estimators. Annals of Statistics 33:1753–70.[Crossref], [Web of Science ®] , [Google Scholar]) and using these estimators, we obtain a large class of (proper and generalized) Bayes minimax estimators and show that the result of Maruyama and Strawderman (2005 Maruyama, Y., and W. E. Strawderman. 2005. A new class of generalized Bayes minimax ridge regression estimators. Annals of Statistics 33:1753–70.[Crossref], [Web of Science ®] , [Google Scholar]) is a special case of our result. We also show that under certain conditions, these generalized Bayes minimax estimators have greater numerical stability (i.e., smaller condition number) than the least-squares estimator.     

Relative squared error prediction in the generalized linear regression model

In linear regression models, predictors based on least squares or on generalized least squares estimators are usually applied which, however, fail in case of multicollinearity. As an alternative biased estimators like ridge estimators, Kuks-Olman estimators, Bayes or minimax estimators are sometimes suggested. In our analysis the relative instead of the generally used absolute squared error enters the objective function. An explicit minimax solution is derived which, in an important special case, can be viewed as a predictor based on a Kuks-Olman estimator.     

Two methods of evaluating hoerl and kennard's ridge regression

We formulate a modified version of the Hoerl-Kennard ridge regression method to solve the problem of estimating coefficients in economic relationships. We investigate two approaches for determining the biasing parameter One approach utilizes prior information in choosing jr, the other approach estimates y from the sample data. Monte Carlo experiments are used to evaluate the relative efficiencies of alternative ridge estimators.