A Note on Prediction Error with Collinearity

Abstract

Least squares (LS) estimator is the best linear unbiased estimator for linear models. It is well known that LS performs poorly in estimation when collinearity is present among regressors. However, it is not fully understood and is even controversial whether LS performs well in prediction. To address this controversy, we study the mean and variance of the prediction squared error (PSE) of LS estimator, and conclude theoretically that although the mean PSE remains invariant regardless of the collinearity, the variance of PSE increases with the collinearity. Thus the prediction error is sensitive to the location in the feature space.     

A gm estimation of the location parameters in a spatial linear model

Universal kriging is a form of interpolation that takes into account the local trends in data when minimizing the error associated with the estimator. Under multivariate normality assumptions, the given predictor is the best linear unbiased predictor. but if the underlying distribution is not normal, the estimator will not be unbiased and will be vulnerable to outliers. With spatial data, it is not only the presence of outliers that may spoil the predictions, but also the boundary sites. usually corners, that tend to have high leverage. As an alternative, a weighted one-step generalized M estimator of the location parameters in a spatial linear model is proposed. It is especially recommended in the case of irregularly spaced data.     

Optimum linear unbiased estimation of the scale parameter by absolute values of order statistics in the double exponential and the double weibull distributions

The first two moments and product moments of absolute values of order statistics are obtained for the double exponential and the double Weibull distributions. In both of the distributions an optimum linear unbiased estimator of the scale parameter, by absolute values of the order statistics, is obtained from complete and censored samples of size n=3(1)10. It is found that the new estimator is generally more efficient than the best linear unbiased estimator (BLUE) of the scale parameter by order statistcs in both of the distributions.     

Estimation in Singular Linear Models with Stochastic Linear Restrictions

This article generalizes the ordinary mixed estimator (OME) in theory, and obtains the estimator of the unknown regression parameters in singular linear models with stochastic linear restrictions: singular mixed estimator (SME). We also give some properties of SME obtained in this article, and prove that it is superior to unrestricted least squared estimator (LSE) in singular linear models in the sense of the covariance matrix and generalized mean square error (GMSE). After that, we also have a discussion about the two-stage estimator of SME. The result we give in this article could be regarded as generalizations of both OME and unrestricted LSE at the same time.     

Moments of scale mixtures of skew-normal distributions and their quadratic forms

We obtain the first four moments of scale mixtures of skew-normal distributions allowing for scale parameters. The first two moments of their quadratic forms are obtained using those moments. Previous studies derived the moments, but all relevant results do not allow for scale parameters. In particular, it is shown that the mean squared error becomes an unbiased estimator of σ² with skewed and heavy-tailed errors. Two measures of multivariate skewness are calculated.     

Optimality of quasi-score in the multivariate mean–variance model with an application to the zero-inflated Poisson model with measurement errors

In a multivariate mean–variance model, the class of linear score (LS) estimators based on an unbiased linear estimating function is introduced. A special member of this class is the (extended) quasi-score (QS) estimator. It is ‘extended’ in the sense that it comprises the parameters describing the distribution of the regressor variables. It is shown that QS is (asymptotically) most efficient within the class of LS estimators. An application is the multivariate measurement error model, where the parameters describing the regressor distribution are nuisance parameters. A special case is the zero-inflated Poisson model with measurement errors, which can be treated within this framework.     

A MORE GENERAL CRITERION FOR SUBSET SELECTION IN MULTIPLE LINEAR REGRESSION

ABSTRACT

In this article, we propose a more general criterion called S_p -criterion, for subset selection in the multiple linear regression Model. Many subset selection methods are based on the Least Squares (LS) estimator of β, but whenever the data contain an influential observation or the distribution of the error variable deviates from normality, the LS estimator performs ‘poorly’ and hence a method based on this estimator (for example, Mallows’ C_p -criterion) tends to select a ‘wrong’ subset. The proposed method overcomes this drawback and its main feature is that it can be used with any type of estimator (either the LS estimator or any robust estimator) of β without any need for modification of the proposed criterion. Moreover, this technique is operationally simple to implement as compared to other existing criteria. The method is illustrated with examples.     

Stein rule prediction of the composite target function in a general linear regression model

This paper considers the problem of simultaneous prediction of the actual and average values of the dependent variable in a general linear regression model. Utilizing the philosophy of Stein rule procedure, a family of improved predictors for a linear function of the actual and expected value of the dependent variable for the forecast period has been proposed. An unbiased estimator for the mean squared error (MSE) matrix of the proposed family of predictors has been obtained and dominance of the family of Stein rule predictors over the best linear unbiased predictor (BLUP) has been established under a quadratic loss function.     

Benchmarked linear shrinkage prediction in the Fay–Herriot small area model

The empirical best linear unbiased predictor (EBLUP) is a linear shrinkage of the direct estimate toward the regression estimate and useful for the small area estimation in the sense of increasing precision of estimation of small area means. However, one potential difficulty of EBLUP is that the overall estimate for a larger geographical area based on a sum of EBLUP is not necessarily identical to the corresponding direct estimate like the overall sample mean. To fix this problem, the paper suggests a new method for benchmarking EBLUP in the Fay–Herriot model without assuming normality of random effects and sampling errors. The resulting benchmarked empirical linear shrinkage (BELS) predictor has novelty in the sense that coefficients for benchmarking are adjusted based on the data from each area. To measure the uncertainty of BELS, the second-order unbiased estimator of the mean squared error is derived.     

Linear approximate ML estimation in scaled Type I generalized logistic distribution based on Type-II censored samples

The scaled (two-parameter) Type I generalized logistic distribution (GLD) is considered with the known shape parameter. The ML method does not yield an explicit estimator for the scale parameter even in complete samples. In this article, we therefore construct a new linear estimator for scale parameter, based on complete and doubly Type-II censored samples, by making linear approximations to the intractable terms of the likelihood equation using least-squares (LS) method, a new approach of linearization. We call this as linear approximate maximum likelihood estimator (LAMLE). We also construct LAMLE based on Taylor series method of linear approximation and found that this estimator is slightly biased than that based on the LS method. A Monte Carlo simulation is used to investigate the performance of LAMLE and found that it is almost as efficient as MLE, though biased than MLE. We also compare unbiased LAMLE with BLUE based on the exact variances of the estimators and interestingly this new unbiased LAMLE is found just as efficient as the BLUE in both complete and Type-II censored samples. Since MLE is known as asymptotically unbiased, in large samples we compare unbiased LAMLE with MLE and found that this estimator is almost as efficient as MLE. We have also discussed interval estimation of the scale parameter from complete and Type-II censored samples. Finally, we present some numerical examples to illustrate the construction of the new estimators developed here.     

Shrinkage estimator of regression model under asymmetric loss

This article investigates the performance of the shrinkage estimator (SE) of the parameters of a simple linear regression model under the LINEX loss criterion. The risk function of the estimator under the asymmetric LINEX loss is derived and analyzed. The moment-generating functions and the first two moments of the estimators are also obtained. The risks of the SE have been compared numerically with that of pre-test and least-square estimators (LSEs) under the LINEX loss criterion. The numerical comparison reveals that under certain conditions the LSE is inadmissible, and the SE is the best among the three estimators.     

Optimal and Robust Designs for Estimating the Concentration Curve and the AUC

The problem of interest is to estimate the concentration curve and the area under the curve (AUC) by estimating the parameters of a linear regression model with an autocorrelated error process. We introduce a simple linear unbiased estimator of the concentration curve and the AUC. We show that this estimator constructed from a sampling design generated by an appropriate density is asymptotically optimal in the sense that it has exactly the same asymptotic performance as the best linear unbiased estimator. Moreover, we prove that the optimal design is robust with respect to a minimax criterion. When repeated observations are available, this estimator is consistent and has an asymptotic normal distribution. Finally, a simulated annealing algorithm is applied to a pharmacokinetic model with correlated errors.     

A test of the mean square error criterion for linear admissible estimators

A test for choosing between a linear admissible estimator and the least squares estimator (LSE) is developed. A characterization of linear admissible estimators useful for comparing estimators is presented and necessary and sufficient conditions for superiority of a linear admissible estimator over the LS estimetor is derived for the test. The test is based on the MSE matrix superiority, but also new resl?!ts concerning covariance matrix comparisons of linear estimators are derived. Further,shown that the test of Toro - Vizcarrondo and Wailace applies iioi only the restricted least squares estimators but also to certain estimators outside this class.     

Recovery of inter-block information: extensions in a two variance component model

For a two variance component mixed linear model, it is shown that under suitable conditions there exists a nonlinear unbiased estimator that is better than a best linear unbiased estimator defined with respect to a given singular covariance matrix. It is also shown how this result applies to improving on intra-block estimators and on estimators like the unweighted means estimator in a random one-way model.     

On the restricted almost unbiased estimators in linear regression

In this paper, the restricted almost unbiased ridge regression estimator and restricted almost unbiased Liu estimator are introduced for the vector of parameters in a multiple linear regression model with linear restrictions. The bias, variance matrices and mean square error (MSE) of the proposed estimators are derived and compared. It is shown that the proposed estimators will have smaller quadratic bias but larger variance than the corresponding competitors in literatures. However, they will respectively outperform the latter according to the MSE criterion under certain conditions. Finally, a simulation study and a numerical example are given to illustrate some of the theoretical results.     

On Bayes Linear Unbiased Estimator Under the Balanced Loss Function

In this paper, the Bayes linear unbiased estimator (Bayes LUE) is derived under the balanced loss function. Moreover, the superiority of Bayes LUE over ordinary least square estimator is studied under the mean square error matrix criterion and Pitman closeness criterion. Furthermore, we compare Bayes LUE under the balanced loss function with Bayes LUE under the quadratic loss function.     

Computational comparison of the general weighted moments estimators of the scale parameter of a Pareto distribution with a known shape parameter with other estimators based on a multiply type II-censored sample

Wu et al. [Computational comparison for weighted moments estimators and BLUE of the scale parameter of a Pareto distribution with known shape parameter under type II multiply censored sample, Appl. Math. Comput. 181 (2006), pp. 1462–1470] proposed the weighted moments estimators (WMEs) of the scale parameter of a Pareto distribution with known shape parameter on a multiply type II-censored sample. They claimed that some WMEs are better than the best linear unbiased estimator (BLUE) based on the exact mean-squared error (MSE). In this paper, the general WME (GWME) is proposed and the computational comparison of the proposed estimator with the WMEs and BLUE is done on the basis of the exact MSE for given sample sizes and different censoring schemes. As a result, the GWME is performing better than the best estimator among 12 WMEs and BLUE for all cases. Therefore, GWME is recommended for use. At last, one example is given to demonstrate the proposed GWME.     

On the Stochastic Restricted Almost Unbiased Estimators in Linear Regression Model

In this article, the stochastic restricted almost unbiased ridge regression estimator and stochastic restricted almost unbiased Liu estimator are proposed to overcome the well-known multicollinearity problem in linear regression model. The quadratic bias and mean square error matrix of the proposed estimators are derived and compared. Furthermore, a numerical example and a Monte Carlo simulation are given to illustrate some of the theoretical results.     

An evaluation of ridge estimator in linear mixed models: an example from kidney failure data

This paper is concerned with the ridge estimation of fixed and random effects in the context of Henderson's mixed model equations in the linear mixed model. For this purpose, a penalized likelihood method is proposed. A linear combination of ridge estimator for fixed and random effects is compared to a linear combination of best linear unbiased estimator for fixed and random effects under the mean-square error (MSE) matrix criterion. Additionally, for choosing the biasing parameter, a method of MSE under the ridge estimator is given. A real data analysis is provided to illustrate the theoretical results and a simulation study is conducted to characterize the performance of ridge and best linear unbiased estimators approach in the linear mixed model.     

On the restricted almost unbiased two-parameter estimator in linear regression model

Özkale and Kaçiranlar introduced the restricted two-parameter estimator (RTPE) to deal with the well-known multicollinearity problem in linear regression model. In this paper, the restricted almost unbiased two-parameter estimator (RAUTPE) based on the RTPE is presented. The quadratic bias and mean-squared error of the proposed estimator is discussed and compared with the corresponding competitors in literatures. Furthermore, a numerical example and a Monte Carlo simulation study are given to explain some of the theoretical results.