Improper use of the ordinary least squares estimator in the switching regression model

The paper considers the consequences of incorrectly using the ordinary least squares estimator, when the true but unknown model is a switching regression. Bias and mean square error express ons are given for slope and residual variance estimators. Except for in very specialized cases the estimators are biased. A numerical exarnple illustrates some of the issues raised and provides a conpelison between the ordinary least squares and maximum likelihood estimators.     

EXACT FINITE-SAMPLE PROPERTIES OF A PRE-TEST ESTIMATOR IN RIDGE REGRESSION

This paper discusses a pre-test regression estimator which uses the least squares estimate when it is “large” and a ridge regression estimate for “small” regression coefficients, where the preliminary test is applied separately to each regression coefficient in turn to determine whether it is “large” or “small.” For orthogonal regressors, the exact finite-sample bias and mean squared error of the pre-test estimator are derived. The latter is less biased than a ridge estimator, and over much of the parameter space the pre-test estimator has smaller mean squared error than least squares. A ridge estimator is found to be inferior to the pre-test estimator in terms of mean squared error in many situations, and at worst the latter estimator is only slightly less efficient than the former at commonly used significance levels.     

Estimation Bias in the First-Order Autoregressive Model and Its Impact on Predictions and Prediction Intervals

The least squares estimate of the autoregressive coefficient in the AR(1) model is known to be biased towards zero, especially for parameters close to the stationarity boundary. Several methods for correcting the autoregressive parameter estimate for the bias have been suggested. Using simulations, we study the bias and the mean square error of the least squares estimate and the bias-corrections proposed by Kendall and Quenouille.

We also study the mean square forecast error and the coverage of the 95% prediction interval when using the biased least squares estimate or one of its bias-corrected versions. We find that the estimation bias matters little for point forecasts, but that it affects the coverage of the prediction intervals. Prediction intervals for forecasts more than one step ahead, when calculated with the biased least squares estimate, are too narrow.     

A new biased estimator based on ridge estimation

In this paper we introduce a new biased estimator for the vector of parameters in a linear regression model and discuss its properties. We show that our new biased estimator is superior, in the mean square error(mse) sense, to the ordinary least squares (OLS) estimator, the ordinary ridge regression (ORR) estimator and the Liu estimator. We also compare the performance of our new biased estimator with two other special Liu-type estimators proposed in Liu (2003). We illustrate our findings with a numerical example based on the widely analysed dataset on Portland cement.     

A Monte Carlo simulation study on partially adaptive estimators of linear regression models

This paper presents a comprehensive comparison of well-known partially adaptive estimators (PAEs) in terms of efficiency in estimating regression parameters. The aim is to identify the best estimators of regression parameters when error terms follow from normal, Laplace, Student's t, normal mixture, lognormal and gamma distribution via the Monte Carlo simulation. In the results of the simulation, efficient PAEs are determined in the case of symmetric leptokurtic and skewed leptokurtic regression error data. Additionally, these estimators are also compared in terms of regression applications. Regarding these applications, using certain standard error estimators, it is shown that PAEs can reduce the standard error of the slope parameter estimate relative to ordinary least squares.     

Robust ridge and robust Liu estimator for regression based on the LTS estimator

In the multiple linear regression analysis, the ridge regression estimator and the Liu estimator are often used to address multicollinearity. Besides multicollinearity, outliers are also a problem in the multiple linear regression analysis. We propose new biased estimators based on the least trimmed squares (LTS) ridge estimator and the LTS Liu estimator in the case of the presence of both outliers and multicollinearity. For this purpose, a simulation study is conducted in order to see the difference between the robust ridge estimator and the robust Liu estimator in terms of their effectiveness; the mean square error. In our simulations, the behavior of the new biased estimators is examined for types of outliers: X-space outlier, Y-space outlier, and X-and Y-space outlier. The results for a number of different illustrative cases are presented. This paper also provides the results for the robust ridge regression and robust Liu estimators based on a real-life data set combining the problem of multicollinearity and outliers.     

Ridge and related estimation procedures: theory and practice

For the classical linear regression problem, a number of estimators alternative to least squares have been proposed for situations in which multicollinearity is a problem. There is, however, relatively little known about how these estimators behave in practice. This paper investigates mean square error properties for a number of biased regression estimators, and discusses some practical implications of the use of such estimators, A conclusion is that certain types of ridge estimatorsappear to have good mean square error properties, and this may be useful in situations in which mean square error is important     

Bias in Small-Sample Inference With Count-Data Models

Abstract

Both Poisson and negative binomial regression can provide quasi-likelihood estimates for coefficients in exponential-mean models that are consistent in the presence of distributional misspecification. It has generally been recommended, however, that inference be carried out using asymptotically robust estimators for the parameter covariance matrix. As with linear models, such robust inference tends to lead to over-rejection of null hypotheses in small samples. Alternative methods for estimating coefficient estimator variances are considered. No one approach seems to remove all test bias, but the results do suggest that the use of the jackknife with Poisson regression tends to be least biased for inference.     

Moment properties of estimators for a type 1 extreme-value regression model

A regression model is considered in which the response variable has a type 1 extreme-value distribution for smallest values. Bias approximations for the maximum likelihood estimators are pivm and a bias reduction estimator for the scale parameter is proposed. The small sample moment properties of the maximum likelihood estimators are compared with the properties of the ordinary least squares estimators and the best linear unbiased estimators based on order statistics for grouped data.     

Approaches to regression analysis with multiple measurements from individual sampling units

Application of ordinary least-squares regression to data sets which contain multiple measurements from individual sampling units produces an unbiased estimator of the parameters but a biased estimator of the covariance matrix of the parameter estimates. The present work considers a random coefficient, linear model to deal with such data sets: this model permits many senses in which multiple measurements are taken from a sampling unit, not just when it is measured at several times. Three procedures to estimate the covariance matrix of the error term of the model are considered. Given these, three procedures to estimate the parameters of the model and their covariance matrix are considered; these are ordinary least-squares, generalized least-squares, and an adjusted ordinary least-squares procedure which produces an unbiased estimator of the covariance matrix of the parameters with small samples. These various procedures are compared in simulation studies using three examples from the biological literature. The possibility of testing hypotheses about the vector of parameters is also considered. It is found that all three procedures for regression estimation produce estimators of the parameters with bias of no practical consequence, Both generalized least-squares and adjusted ordinary least-squares generally produce estimators of the covariance matrix of the parameter estimates with bias of no practical consequence, while ordinary least-squares produces a negatively biased estimator. Neither ordinary nor generalized least-squares provide satisfactory hypothesis tests of the vector of parameter estimates. It is concluded that adjusted ordinary least-squares, when applied with either of two of the procedures used to estimate the error coveriance matrix, shows promise for practical application with data sets of the nature considered here.     

Fractional principal components regression: a general approach to biased estimators

Several biased estimators have been proposed as alternatives to the least squares estimator when multicollinearity is present in the multiple linear regression model. The ridge estimator and the principal components estimator are two techniques that have been proposed for such problems. In this paper the class of fractional principal component estimators is developed for the multiple linear regression model. This class contains many of the biased estimators commonly used to combat multicollinearity. In the fractional principal components framework, two new estimation techniques are introduced. The theoretical performances of the new estimators are evaluated and their small sample properties are compared via simulation with the ridge, generalized ridge and principal components estimators     

Estimation of the linear-plateau segmented regression model in the presence of measurement error

It is well known that when the true values of the independent variable are unobservable due to measurement error, the least squares estimator for a regression model is biased and inconsistent. When repeated observations on each x_i are taken, consistent estimators for the linear-plateau model can be formed. The repeated observations are required to classify each observation to the appropriate line segment. Two cases of repeated observations are treated in detail. First, when a single value of y_i is observed with the repeated observations of x_i the least squares estimator using the mean of the repeated x_i observations is consistent and asymptotically normal. Second, when repeated observations on the pair (x_i, y_i ) are taken the least squares estimator is inconsistent, but two consistent estimators are proposed: one that consistently estimates the bias of the least squares estimator and adjusts accordingly; the second is the least squares estimator using the mean of the repeated observations on each pair.     

Note on a linear model with errors in the variables and with trend

The least squares estimate of the slope parameter of a simple linear model with errors in the variables is typically biased. However the bias vanishes asymptotically for increasing sample size if the regressor variable follows a linear trend. For this case asymptotic expansion formulas for bias and variance of the least squares estimator are derived from exact expressions presented by Richardson and Wu (1970) and certain bounds to these expressions given by Friedmann (1990).     

Partially Adaptive Estimation of the Censored Regression Model

Data censoring causes ordinary least squares estimates of linear models to be biased and inconsistent. Tobit, semiparametric, and partially adaptive estimators have been considered as possible solutions. This paper proposes several new partially adaptive estimators that cover a wide range of distributional characteristics. A simulation study is used to investigate the estimators’ relative efficiency in these settings. The partially adaptive censored regression estimators have little efficiency loss for censored normal errors and may outperform Tobit and semiparametric estimators considered for non-normal distributions. An empirical example of out-of-pocket expenditures for a health insurance plan provides an example, which supports these results.     

Parameter Estimation Through Weighted Least-Squares Rank Regression with Specific Reference to the Weibull and Gumbel Distributions

Probability plots are often used to estimate the parameters of distributions. Using large sample properties of the empirical distribution function and order statistics, weights to stabilize the variance in order to perform weighted least squares regression are derived. Weighted least squares regression is then applied to the estimation of the parameters of the Weibull, and the Gumbel distribution. The weights are independent of the parameters of the distributions considered. Monte Carlo simulation shows that the weighted least-squares estimators outperform the usual least-squares estimators totally, especially in small samples.     

A note on millers's empirical weights for heteroscedastic linear regression

For heteroscedastic simple linear regression when the variances are proportional to a power of the mean of the response variable, Miller (1986) recommends the following procedure: do a weighted least squares regression with the weights (empirical weights) estimated by the inverse of the appropriate power of the response variable. The practical appeal of this approach is its simplicity.

In this article some of the consequences of this simple procedure are considered. Specifically, the effect of this procedure on the bias of the point estimators of the regression coefficients and on the coverage probabilities of their corresponding confidence intervals is examined. It is found that the performance of the process of employing empirical weights in a weighted least squares regression depends on : (1) the particular regression parameter (slope or intercept) of interest, (2) the appropriate power of the mean of the response variable involved, and (3) the amount of variation in the data about the true regression line.     

An evaluation of biased estimators of regression coefficients—A simulation study

Different versions of generalized and ordinary ridge estimators and shrinkage estimators of regression coefficients are studied in comparison with least squares estimators using simulations. The results show that some of the biased estimators considered are better than the least squares estimator in general and the improvement is substantial in some cases.     

Monotone Nonparametric Regression and Confidence Intervals

Several variations of monotone nonparametric regression have been developed over the past 30 years. One approach is to first apply nonparametric regression to data and then monotone smooth the initial estimates to “iron out” violations to the assumed order. Here, such estimators are considered, where local polynomial regression is first used, followed by either least squares isotonic regression or a monotone method using simple averages. The primary focus of this work is to evaluate different types of confidence intervals for these monotone nonparametric regression estimators through Monte Carlo simulation. Most of the confidence intervals use bootstrap or jackknife procedures. Estimation of a response variable as a function of two continuous predictor variables is considered, where the estimation is performed at the observed values of the predictors (instead of on a grid). The methods are then applied to data involving subjects that worked at plants that use beryllium metal who have developed chronic beryllium disease.     

Consistency of non parametric estimators of the area under the ROC curve

ABSTRACT

The area under the receiver operating characteristic (ROC) curve is a popular summary index that measures the accuracy of a continuous-scale diagnostic test to measure its accuracy. Under certain conditions on estimators of distribution functions, we prove a theorem on strong consistency of the non parametric “plugin” estimators of the area under the ROC curve. Based on this theorem, we construct some new “plugin” consistent estimators. The performance of the non parametric estimators considered is illustrated numerically and the estimators are compared in terms of bias, variance, and mean square error.     

Statistical inference in partially time-varying coefficient models

A partially time-varying coefficient time series model is introduced to characterize the nonlinearity and trending phenomenon. To estimate the regression parameter and the nonlinear coefficient function, the profile least squares approach is applied with the help of local linear approximation. The asymptotic distributions of the proposed estimators are established under mild conditions. Meanwhile, the generalized likelihood ratio test is studied and the test statistics are demonstrated to follow asymptotic χ²-distribution under the null hypothesis. Furthermore, some extensions of the proposed model are discussed and several numerical examples are provided to illustrate the finite sample behavior of the proposed methods.