Properties of the ordinary least squares and stein-rule predictions in linear regression models with proxy variables

This article considers a linear regression model in which misspecification relates to the use of a stochastic proxy variable. The analysis indicates the decline in efficiency of the predictions arising from the ordinary least squares and the Stein-rule estimation procedures when a proxy variable is used in the place of an unobservable variable. However, the performance of the Stein-rule predictions is still found to be better than the ordinary least squares predictions over a broad range of k, the characterizing scalar of the Stein-rule estimator.     

Selecting important independent variables in linear regression models

A large body of literature exists on the techniques for selecting the important variables in linear regression analysis. Many of these techniques are ad hoc in nature and have not been studied from a theoretical viewpoint. In this paper we discuss some of the more commonly used techniques and propose a selection procedure based on the statistical selection and ranking approach. This procedure is easy to compute and apply. The procedure depends on the goodness of fit of the model and the total error associated with it.     

A linear regression with unobserved dependent variables

Let (?,X) be a random vector such that E(X|?) = ? and Var(x|?) a + b? + c?² for some known constants a, b and c. Assume X₁,…,X_n are independent observations which have the same distribution as X. Let t(X) be the linear regression of ? on X. The linear empirical Bayes estimator is used to approximate the linear regression function. It is shown that under appropriate conditions, the linear empirical Bayes estimator approximates the linear regression well in the sense of mean squared error.     

Multiple linear regression model with stochastic design variables

In a simple multiple linear regression model, the design variables have traditionally been assumed to be non-stochastic. In numerous real-life situations, however, they are stochastic and non-normal. Estimators of parameters applicable to such situations are developed. It is shown that these estimators are efficient and robust. A real-life example is given.     

The exact distribution and density functions of a pre-test estimator of the error variance in a linear regression model with proxy variables

We consider a linear regression model when some independent variables are unobservable, but proxy variables are available instead of them. We derive the distribution and density functions of a pre-test estimator of the error variance after a pre-test for the null hypothesis that the coefficients for the unobservable variables are zeros. Based on the density function, we show that when the critical value of the pre-test is unity, the coverage probability in the interval estimation of the error variance is maximum.     

PMSE dominance of the positive-part shrinkage estimator in a regression model with proxy variables

Consider a linear regression model with some relevant regressors are unobservable. In such a situation, we estimate the model by using the proxy variables as regressors or by simply omitting the relevant regressors. In this paper, we derive the explicit formula of predictive mean squared error (PMSE) of a general family of shrinkage estimators of regression coefficients. It is shown analytically that the positive-part shrinkage estimator dominates the ordinary shrinkage estimator even when proxy variables are used in place of the unobserved variables. Also, as an example, our result is applied to the double k-class estimator proposed by Ullah and Ullah (Double k-class estimators of coefficients in linear regression. Econometrica. 1978;46:705–722). Our numerical results show that the positive-part double k-class estimator with proxy variables has preferable PMSE performance.     

Stein-rule estimator under inclusion of superfluous variables in linear regression models

Stein-rule estimation is a well-known method to improve the unbiased OLSE in the sense of smaller Mean-Square-Error. The paper is investigating the behaviour of this efficiency relation in case of misspecification of the linear model caused by inclusion of superfluous variables     

Some sampling properties of the two-stage test in a linear regression with a proxy variable

In thia paper, we examine the sampling properties of the two-stage test which consists of a pre-test for disturbance variance followed by a main-test a regression coefficient of an unobservable variable when a proxy variable is used instead of the unobservable variable. It is shown that the pre-test works effectively if the significance level of the pre-test is appropriately chosen.     

Inclusion of binary proxy variables in logistic regression improves treatment effect estimation in observational studies in the presence of binary unmeasured confounding variables

We present a simulation study and application that shows inclusion of binary proxy variables related to binary unmeasured confounders improves the estimate of a related treatment effect in binary logistic regression. The simulation study included 60,000 randomly generated parameter scenarios of sample size 10,000 across six different simulation structures. We assessed bias by comparing the probability of finding the expected treatment effect relative to the modeled treatment effect with and without the proxy variable. Inclusion of a proxy variable in the logistic regression model significantly reduced the bias of the treatment or exposure effect when compared to logistic regression without the proxy variable. Including proxy variables in the logistic regression model improves the estimation of the treatment effect at weak, moderate, and strong association with unmeasured confounders and the outcome, treatment, or proxy variables. Comparative advantages held for weakly and strongly collapsible situations, as the number of unmeasured confounders increased, and as the number of proxy variables adjusted for increased.     

Statistical inference for fixed-effects partially linear regression models with errors in variables

Fixed-effects partially linear regression models are useful tools to analyze data from economic, genetic and other fields. In this paper, we consider estimation and inference procedures when some of the covariates are measured with errors. The previously proposed estimations, including difference-based series estimation (Baltagi and Li in Ann Econ Finan 3:103--116, 2002) and profile least squares estimation (Fan et al. in J Am Stat Assoc 100:781--813, 2005) are no longer consistent because of the attenuation. We propose a new estimation by taking the measurement errors into account. Our proposed estimators are shown to be consistent and asymptotically normal. Consistent estimations of the error variance are also developed. In addition, we propose a variable-selection procedure to variable selection in the parametric part. The procedure is an extension of the nonconcave penalized likelihood (Fan and Li in J Am Stat Assoc 85:1348--1360, 2001), which simultaneously selects the important variables and estimates the unknown parameters. The resulting estimate is shown to possess an oracle property. Extensive simulation studies are conducted to illustrate the finite sample performance of the proposed procedures.     

Joint impact of multiple observations on a subset of variables in multiple linear regression

In multiple linear regression analysis, each observation affects the fitted regression equation differently and has varying influences on the regression coefficients of the different variables. Chatterjee & Hadi (1988) have proposed some measures such as DSSE_ij (Impact on Residual Sum of Squares of simultaneously omitting the ith observation and the jth variable), F_j (Partial F-test for the jth variable) and F_j(i) (Partial F-test for the jth variable omitting the ith observation) to show the joint impact and the interrelationship that exists among a variable and an observation. In this paper we have proposed more extended form of those measures DSSE_IJ, F_J and F_J(I) to deal with the interrelationships that exist among the multiple observations and a subset of variables by monitoring the effects of the simultaneous omission of multiple variables and multiple observations.     

Latent root regression: a biased regression methodology for use with collinear predictor variables

Many different biased regression techniques have been proposed for estimating parameters of a multiple linear regression model when the predictor variables are collinear. One particular alternative, latent root regression analysis, is a technique based on analyzing the latent roots and latent vectors of the correlation matrix of both the response and the predictor variables. It is the purpose of this paper to review the latent root regression estimator and to re-examine some of its properties and applications. It is shown that the latent root estimator is a member of a wider class of estimators for linear models     

Constant width simultaneous confidence bands in multiple linear regression with predictor variables constrained in intervals

In the last fifty years, a great deal of research effort has been made on the construction of simultaneous confidence bands for a linear regression function. Two most frequently quoted confidence bands in the statistics literature are the Scheffé type and constant width bands over a given rectangular region of the predictor variables. For the constant width bands, a method is given by Gafarian [Gafarian, A.V., 1964, Confidence bands in straight line regression. Journal of the American Statistical Association, 59, 182–213.] for the calculation of critical constants only for the special case of one predictor variable. In this article, a method is proposed to construct constant width bands when there are any number of predictor variables. A new criterion for assessing a confidence band is also proposed; it is the probability that a confidence band excludes a false regression function and can be viewed as the power function of a test associated, naturally, with a confidence band. Under this criterion, a numerical comparison between the Scheffé type and constant width bands is then carried out. It emerges from this comparison that the constant width bands can be better than the Scheffé type bands for certain designs.     

Smoothed empirical likelihood analysis of partially linear quantile regression models with missing response variables

In this paper, we consider the estimation and inference of the parameters and the nonparametric part in partially linear quantile regression models with responses that are missing at random. First, we extend the normal approximation (NA)-based methods of Sun (2005) to the missing data case. However, the asymptotic covariance matrices of NA-based methods are difficult to estimate, which complicates inference. To overcome this problem, alternatively, we propose the smoothed empirical likelihood (SEL)-based methods. We define SEL statistics for the parameters and the nonparametric part and demonstrate that the limiting distributions of the statistics are Chi-squared distributions. Accordingly, confidence regions can be obtained without the estimation of the asymptotic covariance matrices. Monte Carlo simulations are conducted to evaluate the performance of the proposed method. Finally, the NA- and SEL-based methods are applied to real data.     

Characterization of minimax linear estimators in linear regression

Two characterization theorems of the minimax linear estimator (Mile) are proven for the case, where the regression parameter varies only in an arbitrary ellipsoid. Furthermore, the existence, uniqueness and admissibility of Mile are shown. The explicit determination of Mile is carried out for a special case.     

On the relationship between the sample size and the number of variables in a linear regression model

The problem of determining the number of variables to be included in the linear regression model is considered under the assumption that the dependent and independent variables have a joint normal distribution. It is shown that for a given sample size n there exists an optimal number k₀ (0 ≤ k₀ < n-2) of variables among all independent variables in the model, such that the expectation of the mean squared error corresponding to the prediction equation with k₀ variables is minimal.Application of this result to ustepwise procedures is discussed.     

On contractions in linear regression

In this paper we demonstrate how the concept of a contractive matrix plays its role in linear regression. We review some well-known facts on the outperformance of the ordinary least-squares estimator and combine these with some new results on admissibility of estimators. Moreover, results on linear sufficiency and linear completeness are given.     

Collinearity in generalized linear regression

The problem of ill-conditioning in generalized linear regression is investigated. Besides collinearity among the explanatory variables, we define another type of ill-conditioning, namely ML-collinearity, which has similar detrimental effects on the covariance matrix, e.g. inflation of some of the estimated standard errors of the regression coefficients. For either situation there is collinearity among the columns of the matrix of the weighted variables. We present both methods to detect, as well as practical examples to illustrate, the difference between these two types of ill-conditioning. Also the applicability of alternative regression methods will be reviewed.     

Bootstrap in functional linear regression

We have considered the functional linear model with scalar response and functional explanatory variable. One of the most popular methodologies for estimating the model parameter is based on functional principal components analysis (FPCA). In recent literature, weak convergence for a wide class of FPCA-type estimates has been proved, and consequently asymptotic confidence sets can be built. In this paper, we have proposed an alternative approach in order to obtain pointwise confidence intervals by means of a bootstrap procedure, for which we have obtained its asymptotic validity. Besides, a simulation study allows us to compare the practical behaviour of asymptotic and bootstrap confidence intervals in terms of coverage rates for different sample sizes.     

Polynomial regression with errors in the variables

A polynomial functional relationship with errors in both variables can be consistently estimated by constructing an ordinary least squares estimator for the regression coefficients, assuming hypothetically the latent true regressor variable to be known, and then adjusting for the errors. If normality of the error variables can be assumed, the estimator can be simplified considerably. Only the variance of the errors in the regressor variable and its covariance with the errors of the response variable need to be known. If the variance of the errors in the dependent variable is also known, another estimator can be constructed.