On adaptive linear regression

Ordinary least squares (OLS) is omnipresent in regression modeling. Occasionally, least absolute deviations (LAD) or other methods are used as an alternative when there are outliers. Although some data adaptive estimators have been proposed, they are typically difficult to implement. In this paper, we propose an easy to compute adaptive estimator which is simply a linear combination of OLS and LAD. We demonstrate large sample normality of our estimator and show that its performance is close to best for both light-tailed (e.g. normal and uniform) and heavy-tailed (e.g. double exponential and t ₃) error distributions. We demonstrate this through three simulation studies and illustrate our method on state public expenditures and lutenizing hormone data sets. We conclude that our method is general and easy to use, which gives good efficiency across a wide range of error distributions.     

Estimation of regression equation with cauchy disturbances

In this paper we present two methods of estimating a linear regression equation with Cauchy disturbances. The first method uses the maximum likelihood principle and therefore the estimators obtained are consistent. The asymptotic covariance is derived which provides with the necessary statistics for the purpose of making inference in large samples. The second method is the method of least lines which minimizes the sum of absolute errors (MSAE) from the fitted regression. Then these two methods are compared through a Monte Carlo study. The maximum likelihood method emerges superior over the MSAE method. However, the MSAE procedure which does not depend on the distribution of the error term appears to be a close competitor to the maximum likelihood estimator.     

A Novel Regression Approach: Least Squares Ratio

Regression Analysis (RA) is one of the frequently used tool for forecasting. The Ordinary Least Squares (OLS) Technique is the basic instrument of RA and there are many regression techniques based on OLS. This paper includes a new regression approach, called Least Squares Ratio (LSR), and comparison of OLS and LSR according to mean square errors of estimation of theoretical regression parameters (mse ß) and dependent value (mse y).     

On Ridge Parameters in Logistic Regression

This article applies and investigates a number of logistic ridge regression (RR) parameters that are estimable by using the maximum likelihood (ML) method. By conducting an extensive Monte Carlo study, the performances of ML and logistic RR are investigated in the presence of multicollinearity and under different conditions. The simulation study evaluates a number of methods of estimating the RR parameter k that has recently been developed for use in linear regression analysis. The results from the simulation study show that there is at least one RR estimator that has a lower mean squared error (MSE) than the ML method for all the different evaluated situations.     

Feasible robust estimator in restricted semiparametric regression models based on the LTS approach

Under some nonstochastic linear restrictions based on either additional information or prior knowledge in a semiparametric regression model, a family of feasible generalized robust estimators for the regression parameter is proposed. The least trimmed squares (LTS) method proposed by Rousseeuw as a highly robust regression estimator is a statistical technique for fitting a regression model based on the subset of h observations (out of n) whose least-square fit possesses the smallest sum of squared residuals. The coverage h may be set between n/2 and n. The LTS estimator involves computing the hyperplane that minimizes the sum of the smallest h squared residuals. For practical purpose, it is assumed that the covariance matrix of the error term is unknown and thus feasible estimators are replaced. Then, we develop an algorithm for the LTS estimator based on feasible methods. Through the Monte Carlo simulation studies and a real data example, performance of the feasible type of robust estimators is compared with the classical ones in restricted semiparametric regression models.     

A comparison of linear regression techniques in method comparison studies

In this study, the performances of linear regression techniques, which are especially used in clinical chemistry in method comparison studies, are compared via the Monte-Carlo simulation. The regression techniques that take the measurement errors of both dependent and independent variables into account are called Type II regression techniques. In this study, we also compare the performances of Type II and Type I (classical regression techniques that do not take the measurement errors of the independent variable into account) regression techniques for different sample sizes and different shape parameters of the Weibull distribution. The mean square error is used as a performance criterion of each technique. MATLAB 7.02 software is used in the simulation study. As a result, in all conditions, the ordinary least-square (OLS)-bisector regression technique, which bisects the OLS(Y | X) and the OLS(X | Y), shows the best performance.     

On the almost unbiased ridge regression estimator

The purpose of this paper is two-fold. One is to compare the almost unbiased generalized ridge regression (AUGRR) estimator proposed by Singh, Chaubey and Dwivedi (1986) with the generalized ridge regression (GRR) estimator and with the ordinary least squares (OLS) estimator in terms of the mean squared error criterion. Second is to examine small sample properties of the operational almost unbiased ordinary ridge regression (AUORR) estimator by Monte Carlo experiments.     

Using Liu-Type Estimator to Combat Collinearity

Abstract

Linear regression model and least squares method are widely used in many fields of natural and social sciences. In the presence of collinearity, the least squares estimator is unstable and often gives misleading information. Ridge regression is the most common method to overcome this problem. We find that when there exists severe collinearity, the shrinkage parameter selected by existing methods for ridge regression may not fully address the ill conditioning problem. To solve this problem, we propose a new two-parameter estimator. We show using both theoretic results and simulation that our new estimator has two advantages over ridge regression. First, our estimator has less mean squared error (MSE). Second, our estimator can fully address the ill conditioning problem. A numerical example from literature is used to illustrate the results.     

Performance of Kibria's Method for the Heteroscedastic Ridge Regression Model: Some Monte Carlo Evidence

It is common for a linear regression model that the error terms display some form of heteroscedasticity and at the same time, the regressors are also linearly correlated. Both of these problems have serious impact on the ordinary least squares (OLS) estimates. In the presence of heteroscedasticity, the OLS estimator becomes inefficient and the similar adverse impact can also be found on the ridge regression estimator that is alternatively used to cope with the problem of multicollinearity. In the available literature, the adaptive estimator has been established to be more efficient than the OLS estimator when there is heteroscedasticity of unknown form. The present article proposes the similar adaptation for the ridge regression setting with an attempt to have more efficient estimator. Our numerical results, based on the Monte Carlo simulations, provide very attractive performance of the proposed estimator in terms of efficiency. Three different existing methods have been used for the selection of biasing parameter. Moreover, three different distributions of the error term have been studied to evaluate the proposed estimator and these are normal, Student's t and F distribution.     

State space modeling of multiple time series: a comment

This paper provides guidance in choosing k₁ andk₂ of the double k-class (KK) estimator such that it will improve upon both the ordinary least squares (OLS) and Stein-rule (SR) estimators in predictive mean squared error (PMSE). Asymptotic bias and mean squared error (MSE) results are derived for nonnormal and other cases. A simulation compares the KK estimator with the OLS and SR estimators.     

Comparisons of the <Emphasis Type="Italic">r</Emphasis> − <Emphasis Type="Italic">k</Emphasis> class estimator to the ordinary least squares estimator under the Pitman’s closeness criterion

In the presence of multicollinearity, the r − k class estimator is proposed as an alternative to the ordinary least squares (OLS) estimator which is a general estimator including the ordinary ridge regression (ORR), the principal components regression (PCR) and the OLS estimators. Comparison of competing estimators of a parameter in the sense of mean square error (MSE) criterion is of central interest. An alternative criterion to the MSE criterion is the Pitman’s (1937) closeness (PC) criterion. In this paper, we compare the r − k class estimator to the OLS estimator in terms of PC criterion so that we can get the comparison of the ORR estimator to the OLS estimator under the PC criterion which was done by Mason et al. (1990) and also the comparison of the PCR estimator to the OLS estimator by means of the PC criterion which was done by Lin and Wei (2002).     

Finite sample properties of an HPT estimator when each individual regression coefficient is estimated in a misspecified linear regression model

ABSTRACT

In this paper, assuming that there exist omitted variables in the specified model, we analytically derive the exact formula for the mean squared error (MSE) of a heterogeneous pre-test (HPT) estimator whose components are the ordinary least squares (OLS) and feasible ridge regression (FRR) estimators. Since we cannot examine the MSE performance analytically, we execute numerical evaluations to investigate small sample properties of the HPT estimator, and compare the MSE performance of the HPT estimator with those of the FRR estimator and the usual OLS estimator. Our numerical results show that (1) the HPT estimator is more efficient when the model misspecification is severe; (2) the HPT estimator with the optimal critical value obtained under the correctly specified model can be safely used even when there exist omitted variables in the specified model.     

MSE performance of the weighted average estimators consisting of shrinkage estimators when each individual regression coefficient is estimated

In this paper, we analytically derive the exact formula for the mean squared error (MSE) of two weighted average (WA) estimators for each individual regression coefficient. Further, we execute numerical evaluations to investigate small sample properties of the WA estimators, and compare the MSE performance of the WA estimators with the other shrinkage estimators and the usual OLS estimator. Our numerical results show that (1) the WA estimators have smaller MSE than the other shrinkage estimators and the OLS estimator over a wide region of parameter space; (2) the range where the relative MSE of the WA estimator is smaller than that of the OLS estimator gets narrower as the number of explanatory variables k increases.     

Estimation in linear models using gradient descent with early stopping

A new shrinkage estimator of the coefficients of a linear model is derived. The estimator is motivated by the gradient-descent algorithm used to minimize the sum of squared errors and results from early stopping of the algorithm. The statistical properties of the estimator are examined and compared with other well-established methods such as least squares and ridge regression, both analytically and through a simulation study. An important result is that the new estimator is shown to be comparable to other shrinkage estimators in terms of mean squared error of parameters and of predictions, and superior under certain circumstances.Supported by the Greek State Scholarships Foundation     

Support vector regression based residual control charts

Control charts for residuals, based on the regression model, require a robust fitting technique for minimizing the error resulting from the fitted model. However, in the multivariate case, when the number of variables is high and data become complex, traditional fitting techniques, such as ordinary least squares (OLS), lose efficiency. In this paper, support vector regression (SVR) is used to construct robust control charts for residuals, called SVR-chart. This choice is based on the fact that the SVR is designed to minimize the structural error whereas other techniques minimize the empirical error. An application shows that SVR methods gives competitive results in comparison with the OLS and the partial least squares method, in terms of standard deviation of the error prediction and the standard error of performance. A sensitivity study is conducted to evaluate the SVR-chart performance based on the average run length (ARL) and showed that the SVR-chart has the best ARL behaviour in comparison with the other residuals control charts.     

A New Robust Regression Method Based on Particle Swarm Optimization

Regression analysis is one of methods widely used in prediction problems. Although there are many methods used for parameter estimation in regression analysis, ordinary least squares (OLS) technique is the most commonly used one among them. However, this technique is highly sensitive to outlier observation. Therefore, in literature, robust techniques are suggested when data set includes outlier observation. Besides, in prediction a problem, using the techniques that reduce the effectiveness of outlier and using the median as a target function rather than an error mean will be more successful in modeling these kinds of data. In this study, a new parameter estimation method using the median of absolute rate obtained by division of the difference between observation values and predicted values by the observation value and based on particle swarm optimization was proposed. The performance of the proposed method was evaluated with a simulation study by comparing it with OLS and some other robust methods in the literature.     

Developing a restricted two-parameter Liu-type estimator: A comparison of restricted estimators in the binary logistic regression model

In the context of estimating regression coefficients of an ill-conditioned binary logistic regression model, we develop a new biased estimator having two parameters for estimating the regression vector parameter β when it is subjected to lie in the linear subspace restriction Hβ = h. The matrix mean squared error and mean squared error (MSE) functions of these newly defined estimators are derived. Moreover, a method to choose the two parameters is proposed. Then, the performance of the proposed estimator is compared to that of the restricted maximum likelihood estimator and some other existing estimators in the sense of MSE via a Monte Carlo simulation study. According to the simulation results, the performance of the estimators depends on the sample size, number of explanatory variables, and degree of correlation. The superiority region of our proposed estimator is identified based on the biasing parameters, numerically. It is concluded that the new estimator is superior to the others in most of the situations considered and it is recommended to the researchers.     

Modeling and predicting IBNR reserve: extended chain ladder and heteroscedastic regression analysis

This work deals with two methodologies for predicting incurred but not reported (IBNR) actuarial reserves. The first is the traditional chain ladder, which is extended for dealing with the calendar year IBNR reserve. The second is based on heteroscedastic regression models suitable to deal with the tail effect of the runoff triangle – and to forecast calendar year IBNR reserves as well. Theoretical results regarding closed expressions for IBNR predictors and mean squared errors are established – for the case of the second methodology, a Monte Carlo study is designed and implemented for accessing finite sample performances of feasible mean squared error formulae. Finally, the methods are implemented with two real data sets. The main conclusions: (i) considering tail effects does not imply theoretical and/or computational problems; and (ii) both methodologies are interesting to design softwares for IBNR reserve prediction.     

Ridge estimation in regression problems with autocorrelated errors: A monte carlo study

This paper presents the results of a Monte Carlo study of OLS and GLS based adaptive ridge estimators for regression problems in which the independent variables are collinear and the errors are autocorrelated. It studies the effects of degree of collinearity, magnitude of error variance, orientation of the parameter vector and serial correlation of the independent variables on the mean squared error performance of these estimators. Results suggest that such estimators produce greatly improved performance in favorable portions of the parameter space. The GLS based methods are best when the independent variables are also serially correlated.     

Fitting linear regression models to censored data by least squares and maximum likelihood methods

Several approaches have been suggested for fitting linear regression models to censored data. These include Cox's propor­tional hazard models based on quasi-likelihoods. Methods of fitting based on least squares and maximum likelihoods have also been proposed. The methods proposed so far all require special purpose optimization routines. We describe an approach here which requires only a modified standard least squares routine.

We present methods for fitting a linear regression model to censored data by least squares and method of maximum likelihood. In the least squares method, the censored values are replaced by their expectations, and the residual sum of squares is minimized. Several variants are suggested in the ways in which the expect­ation is calculated. A parametric (assuming a normal error model) and two non-parametric approaches are described. We also present a method for solving the maximum likelihood equations in the estimation of the regression parameters in the censored regression situation. It is shown that the solutions can be obtained by a recursive algorithm which needs only a least squares routine for optimization. The suggested procesures gain considerably in computational officiency. The Stanford Heart Transplant data is used to illustrate the various methods.