Least Absolute Regression Revisited

We describe a method for fitting a least absolute residual (LAR) line through a set of two–dimensional points. The algorithm is based on a labeling technique derived from linear programming. It is suited for interactive data analysis and can be carried out with graph paper and a programmable hand calculator. Tests conducted with a Pascal program indicate that the algorithm is computationally efficient.     

Algorithms for restricted least absolute value estimation

We present a concise summary of recent progress in developing algorithms for restricted least absolute value (LAV) estimation (i. e. ?₁ approximation subject to linear constraints). The emphasis is on our own new algorithm, and we provide some numerical results obtained with it.     

Least absolute value estimation in regression models: an annotated bibliography

This paper presents a comprehensive listing of articles on least absolute value (LAV) estimation as applied to linear and non-linear regression models and in systems of equations. References to the LAV method as applied in approximation theory are also included. Annotations describing the content of each article follow each reference.     

L1 for the simple linear regression model

By modifying the direct method to solve the overdetermined linear system we are able to present an algorithm for L₁ estimation which appears to be superior computationally to any other known algorithm for the simple linear regression problem.     

On least absolute values estimation

The resistance of least absolute values (L₁) estimators to outliers and their robustness to heavy-tailed distributions make these estimators useful alternatives to the usual least squares estimators. The recent development of efficient algorithms for L₁ estimation in linear models has permitted their use in practical data analysis. Although in general the L₁ estimators are not unique, there are a number of properties they all share. The set of all L₁ estimators for a given model and data set can be characterized as the convex hull of some extreme estimators. Properties of the extreme estimators and of the L₁-estimate set are considered.     

LARS-type algorithm for group lasso

The least absolute shrinkage and selection operator (lasso) has been widely used in regression analysis. Based on the piecewise linear property of the solution path, least angle regression provides an efficient algorithm for computing the solution paths of lasso. Group lasso is an important generalization of lasso that can be applied to regression with grouped variables. However, the solution path of group lasso is not piecewise linear and hence cannot be obtained by least angle regression. By transforming the problem into a system of differential equations, we develop an algorithm for efficient computation of group lasso solution paths. Simulation studies are conducted for comparing the proposed algorithm to the best existing algorithm: the groupwise-majorization-descent algorithm.     

Robust Coordinate Descent Algorithm Robust Solution Path for High-dimensional Sparse Regression Modeling

The L₁-type regularization provides a useful tool for variable selection in high-dimensional regression modeling. Various algorithms have been proposed to solve optimization problems for L₁-type regularization. Especially the coordinate descent algorithm has been shown to be effective in sparse regression modeling. Although the algorithm shows a remarkable performance to solve optimization problems for L₁-type regularization, it suffers from outliers, since the procedure is based on the inner product of predictor variables and partial residuals obtained from a non-robust manner. To overcome this drawback, we propose a robust coordinate descent algorithm, especially focusing on the high-dimensional regression modeling based on the principal components space. We show that the proposed robust algorithm converges to the minimum value of its objective function. Monte Carlo experiments and real data analysis are conducted to examine the efficiency of the proposed robust algorithm. We observe that our robust coordinate descent algorithm effectively performs for the high-dimensional regression modeling even in the presence of outliers.     

Obtaining least absolute value estimates for a two-way classification model

The importance of the two-way classification model is well known, but the standard method of analysis is least squares. Often, the data of the model calls for a more robust estimation technique. This paper demonstrates the equivalence between the problem of obtaining least absolute value estimates for the two-way classification model and a capacitated transportation problem. A special purpose primal algorithm is developed to provide the least absolute value estimates. A computational comparison is made between an implementation of this specialized algorithm and a standard capacitated transportation code.     

The optimal Lp norm estimator in linear regression models

The least squares estimator is usually applied when estimating the parameters in linear regression models. As this estimator is sensitive to departures from normality in the residual distribution, several alternatives have been proposed. The L_p norm estimators is one class of such alternatives. It has been proposed that the kurtosis of the residual distribution be taken into account when a choice of estimator in the L_p norm class is made (i.e. the choice of p). In this paper, the asymtotic variance of the estimators is used as the criterion in the choice of p. It is shown that when this criterion is applied, other characteristics of the residual distribution than the kurtosis (namely moments of order p-2 and 2p-2) are important.     

Comparison of computer programs for simple linear L 1 regression

A number of efficient computer codes are available for the simple linear L ₁ regression problem. However, a number of these codes can be made more efficient by utilizing the least squares solution. In fact, a couple of available computer programs already do so.

We report the results of a computational study comparing several openly available computer programs for solving the simple linear L ₁ regression problem with and without computing and utilizing a least squares solution.     

Bayesian regression under combinations of constraints

A Bayesian method for regression under several types of constraints is proposed. The constraints can be range-restricted and include shape restrictions, constraints on the value of the regression function, smoothness conditions and combinations of these types of constraints. The support of the prior distribution is included in the set of piecewise linear functions. It is shown that the proposed prior can be arbitrarily close to the distribution induced by the addition of a polynomial plus an (m−1)-fold integrated Brownian motion. Hence, despite its piecewise linearity, the regression function behaves (approximately) like an m−1 times continuously differentiable random function. Furthermore, thanks to the piecewise linear property, many combinations of constraints can easily be considered. The regression function is estimated by the posterior mode computed by a simulated annealing algorithm. The constraints on the shape and the values of the regression function are taken into account thanks to the proposal distribution, while the smoothness condition is handled by the acceptation step. Simulations from the posterior distribution are obtained by a Gibbs sampling algorithm.     

DASSO: connections between the Dantzig selector and lasso

Summary.  We propose a new algorithm, DASSO, for fitting the entire coefficient path of the Dantzig selector with a similar computational cost to the least angle regression algorithm that is used to compute the lasso. DASSO efficiently constructs a piecewise linear path through a sequential simplex-like algorithm, which is remarkably similar to the least angle regression algorithm. Comparison of the two algorithms sheds new light on the question of how the lasso and Dantzig selector are related. In addition, we provide theoretical conditions on the design matrix X under which the lasso and Dantzig selector coefficient estimates will be identical for certain tuning parameters. As a consequence, in many instances, we can extend the powerful non-asymptotic bounds that have been developed for the Dantzig selector to the lasso. Finally, through empirical studies of simulated and real world data sets we show that in practice, when the bounds hold for the Dantzig selector, they almost always also hold for the lasso.     

A special purpose linear programming algorithm for obtaining least absolute value estimators in a linear model with dummy variables

Dummy (0, 1) variables are frequently used in statistical modeling to represent the effect of certain extraneous factors. This paper presents a special purpose linear programming algorithm for obtaining least-absolute-value estimators in a linear model with dummy variables. The algorithm employs a compact basis inverse procedure and incorporates the advanced basis exchange techniques available in specialized algorithms for the general linear least-absolute-value problem. Computational results with a computer code version of the algorithm are given.     

Standard and robust orthogonal regression

A fast routine for converting regression algorithms into corresponding orthogonal regression (OR) algorithms was introduced in Ammann and Van Ness (1988). The present paper discusses the properties of various ordinary and robust OR procedures created using this routine. OR minimizes the sum of the orthogonal distances from the regression plane to the data points. OR has three types of applications. First, L ₂ OR is the maximum likelihood solution of the Gaussian errors-in-variables (EV) regression problem. This L ₂ solution is unstable, thus the robust OR algorithms created from robust regression algorithms should prove very useful. Secondly, OR is intimately related to principal components analysis. Therefore, the routine can also be used to create L ₁, robust, etc. principal components algorithms. Thirdly, OR treats the x and y variables symmetrically which is important in many modeling problems. Using Monte Carlo studies this paper compares the performance of standard regression, robust regression, OR, and robust OR on Gaussian EV data, contaminated Gaussian EV data, heavy-tailed EV data, and contaminated heavy-tailed EV data.     

Least absolute deviations estimation via the EM algorithm

This paper derives EM and generalized EM (GEM) algorithms for calculating least absolute deviations (LAD) estimates of the parameters of linear and nonlinear regression models. It shows that Schlossmacher's iterative reweighted least squares algorithm for calculating LAD estimates (E.J. Schlossmacher, Journal of the American Statistical Association 68: 857–859, 1973) is an EM algorithm. A GEM algorithm for computing LAD estimates of the parameters of nonlinear regression models is also provided and is applied in some examples.     

A fast algorithm for the coordinate-wise minimum distance estimation

Application of the minimum distance (MD) estimation method to the linear regression model for estimating regression parameters is a difficult and time-consuming process due to the complexity of its distance function, and hence, it is computationally expensive. To deal with the computational cost, this paper proposes a fast algorithm which makes the best use of coordinate-wise minimization technique in order to obtain the MD estimator. R package (KoulMde) based on the proposed algorithm and written in Rcpp is available online.     

Total least squares solution for compositional data using linear models

The restrictive properties of compositional data, that is multivariate data with positive parts that carry only relative information in their components, call for special care to be taken while performing standard statistical methods, for example, regression analysis. Among the special methods suitable for handling this problem is the total least squares procedure (TLS, orthogonal regression, regression with errors in variables, calibration problem), performed after an appropriate log-ratio transformation. The difficulty or even impossibility of deeper statistical analysis (confidence regions, hypotheses testing) using the standard TLS techniques can be overcome by calibration solution based on linear regression. This approach can be combined with standard statistical inference, for example, confidence and prediction regions and bounds, hypotheses testing, etc., suitable for interpretation of results. Here, we deal with the simplest TLS problem where we assume a linear relationship between two errorless measurements of the same object (substance, quantity). We propose an iterative algorithm for estimating the calibration line and also give confidence ellipses for the location of unknown errorless results of measurement. Moreover, illustrative examples from the fields of geology, geochemistry and medicine are included. It is shown that the iterative algorithm converges to the same values as those obtained using the standard TLS techniques. Fitted lines and confidence regions are presented for both original and transformed compositional data. The paper contains basic principles of linear models and addresses many related problems.     

Empirical Comparison of Nonparametric Regression Estimates on Real Data

The performance of nine different nonparametric regression estimates is empirically compared on ten different real datasets. The number of data points in the real datasets varies between 7, 900 and 18, 000, where each real dataset contains between 5 and 20 variables. The nonparametric regression estimates include kernel, partitioning, nearest neighbor, additive spline, neural network, penalized smoothing splines, local linear kernel, regression trees, and random forests estimates. The main result is a table containing the empirical L₂ risks of all nine nonparametric regression estimates on the evaluation part of the different datasets. The neural networks and random forests are the two estimates performing best. The datasets are publicly available, so that any new regression estimate can be easily compared with all nine estimates considered in this article by just applying it to the publicly available data and by computing its empirical L₂ risks on the evaluation part of the datasets.     

On sign-based regression quantiles

A sign-based (SB) approach suggests an alternative criterion for quantile regression fit. The SB criterion is a piecewise constant function, which often leads to a non-unique solution. We compare the mid-point of this SB solution with the least absolute deviations (LAD) method and describe asymptotic properties of SB estimators under a weaker set of assumptions as compared with the assumptions often used with the generalized method of moments. Asymptotic properties of LAD and SB estimators are equivalent; however, there are finite sample differences as we show in simulation studies. At small to moderate sample sizes, the SB procedure for modelling quantiles at longer tails demonstrates a substantially lower bias, variance, and mean-squared error when compared with the LAD. In the illustrative example, we model a 0.8-level quantile of hospital charges and highlight finite sample advantage of the SB versus LAD.     

Rank-based ridge estimation in multiple linear regression

Multicollinearity and model misspecification are frequently encountered problems in practice that produce undesirable effects on classical ordinary least squares (OLS) regression estimator. The ridge regression estimator is an important tool to reduce the effects of multicollinearity, but it is still sensitive to a model misspecification of error distribution. Although rank-based statistical inference has desirable robustness properties compared to the OLS procedures, it can be unstable in the presence of multicollinearity. This paper introduces a rank regression estimator for regression parameters and develops tests for general linear hypotheses in a multiple linear regression model. The proposed estimator and the tests have desirable robustness features against the multicollinearity and model misspecification of error distribution. Asymptotic behaviours of the proposed estimator and the test statistics are investigated. Real and simulated data sets are used to demonstrate the feasibility and the performance of the estimator and the tests.