Least absolute deviations estimation via the EM algorithm

This paper derives EM and generalized EM (GEM) algorithms for calculating least absolute deviations (LAD) estimates of the parameters of linear and nonlinear regression models. It shows that Schlossmacher's iterative reweighted least squares algorithm for calculating LAD estimates (E.J. Schlossmacher, Journal of the American Statistical Association 68: 857–859, 1973) is an EM algorithm. A GEM algorithm for computing LAD estimates of the parameters of nonlinear regression models is also provided and is applied in some examples.     

THE WEIGHTED MEDIAN AND MULTIPLE REGRESSION1

This paper reviews the iterative use of the weighted median to estimate the parameter vector in the classical linear model when the fitting criterion is (i) least absolute deviation sum (LAD); and (ii) the Cauchy criterion. The implications of the Cauchy criterion, little developed hitherto, are compared and contrasted with results for the better-known LAD procedure. Since the weighted median is essentially an estimation technique for the simplest regression model, its use in these contexts illustrates the central role in statistical theory that is played by regression analysis, a focal area of the work of E. J. Williams (1959).     

Developing ridge estimation method for median regression

In this paper, the ridge estimation method is generalized to the median regression. Though the least absolute deviation (LAD) estimation method is robust in the presence of non-Gaussian or asymmetric error terms, it can still deteriorate into a severe multicollinearity problem when non-orthogonal explanatory variables are involved. The proposed method increases the efficiency of the LAD estimators by reducing the variance inflation and giving more room for the bias to get a smaller mean squared error of the LAD estimators. This paper includes an application of the new methodology and a simulation study as well.     

A fast wavelet approach for recovering damaged images

A wavelet method is proposed for recovering damaged images. The proposed method combines wavelet shrinkage with preprocessing based on a binning process and an imputation procedure that is designed to extend the scope of wavelet shrinkage to data with missing values and perturbed locations. The proposed algorithm, termed as the BTW algorithm is simple to implement and efficient for recovering an image. Furthermore, this algorithm can be easily applied to wavelet regression for one-dimensional (1-D) signal estimation with irregularly spaced data. Simulation studies and real examples show that the proposed method can produce substantially effective results.     

On sign-based regression quantiles

A sign-based (SB) approach suggests an alternative criterion for quantile regression fit. The SB criterion is a piecewise constant function, which often leads to a non-unique solution. We compare the mid-point of this SB solution with the least absolute deviations (LAD) method and describe asymptotic properties of SB estimators under a weaker set of assumptions as compared with the assumptions often used with the generalized method of moments. Asymptotic properties of LAD and SB estimators are equivalent; however, there are finite sample differences as we show in simulation studies. At small to moderate sample sizes, the SB procedure for modelling quantiles at longer tails demonstrates a substantially lower bias, variance, and mean-squared error when compared with the LAD. In the illustrative example, we model a 0.8-level quantile of hospital charges and highlight finite sample advantage of the SB versus LAD.     

LAD-Lasso variable selection for doubly censored median regression models

ABSTRACT

A variable selection procedure based on least absolute deviation (LAD) estimation and adaptive lasso (LAD-Lasso for short) is proposed for median regression models with doubly censored data. The proposed procedure can select significant variables and estimate the parameters simultaneously, and the resulting estimators enjoy the oracle property. Simulation results show that the proposed method works well.     

A nonparametric confidence interval for slope based on spearman's rho

An algorithm is presented for computing an exact nonparametric interval estimate of the slope parameter in a simple linear regression model. The confidence interval is obtained by inverting the hypothesis test for slope that uses Spearman's rho. This method is compared to an exact procedure based on Kendall's tau. The Spearman rho procedure will generally give exact levels of confidence closer to desired levels, especially in small samples. Monte carlo results comparing these two methods with the parametric procedure are given     

Extension of the pool-adjacent-violators algorithm

The pool-adjacent-violators algorithm (PAVA) is an efficient algorithm which converges in a finite number of steps. However, it has been applicable so far only in isotonic regression with the simple order. This report extends its applicability to other quadratic programming problems, including certain one-sided multivariate testing problems and concave regression problems.     

A generalization of the Solis-Wets method

In this paper we focus on the application of global stochastic optimization methods to extremum estimators. We propose a general stochastic method—the master method—which includes several stochastic optimization algorithms as a particular case. The proposed method is sufficiently general to include the Solis-Wets method, the improving hit-and-run algorithm, and a stochastic version of the zigzag algorithm. A matrix formulation of the master method is presented and some specific results are given for the stochastic zigzag algorithm. Convergence of the proposed method is established under a mild set of conditions, and a simple regression model is used to illustrate the method.     

Nonparametric M-regression with free knot splines

Many problems of practical interest can be formulated as the nonparametric estimation of a certain function such as a regression function, logistic or other generalized regression function, density function, conditional density function, hazard function, or conditional hazard function. Extended linear modeling provides a convenient theoretical framework for using polynomial splines and their selected tensor products in such function estimation problems and especially for obtaining rates of convergence of the resulting estimates in a unified manner. For a long time the theoretical results were restricted to fixed knot splines and to log-likelihood functions that were twice continuously differentiable. Recently, Stone and Huang extended the theory to handle free knot splines. In the present paper, the theory is further extended to handle contexts in which the log-likelihood function may not be differentiable. Specifically, we establish rates of convergence for estimation based on free knot splines in the context of nonparametric regression corresponding to M-estimates, which includes least absolute deviations (LAD) regression, quantile regression, and robust regression as special cases.     

Modelling beyond regression functions: an application of multimodal regression to speed–flow data

Summary.  For speed–flow data, which are intensively discussed in transportation science, common nonparametric regression models of the type y = m ( x )+noise turn out to be inadequate since simple functional models cannot capture the essential relationship between the predictor and response. Instead a more general setting is required, allowing for multifunctions rather than functions. The tool proposed is conditional modes estimation which, in the form of local modes, yields several branches that correspond to the local modes. A simple algorithm for computing the branches is derived. This is based on a conditional mean shift algorithm and is shown to work well in the application that is considered.     

The Bayesian elastic net regression

A Bayesian elastic net approach is presented for variable selection and coefficient estimation in linear regression models. A simple Gibbs sampling algorithm was developed for posterior inference using a location-scale mixture representation of the Bayesian elastic net prior for the regression coefficients. The penalty parameters are chosen through an empirical method that maximizes the data marginal likelihood. Both simulated and real data examples show that the proposed method performs well in comparison to the other approaches.     

Bayesian bridge quantile regression

Regularization methods for simultaneous variable selection and coefficient estimation have been shown to be effective in quantile regression in improving the prediction accuracy. In this article, we propose the Bayesian bridge for variable selection and coefficient estimation in quantile regression. A simple and efficient Gibbs sampling algorithm was developed for posterior inference using a scale mixture of uniform representation of the Bayesian bridge prior. This is the first work to discuss regularized quantile regression with the bridge penalty. Both simulated and real data examples show that the proposed method often outperforms quantile regression without regularization, lasso quantile regression, and Bayesian lasso quantile regression.     

M-estimators with non-standard rates of convergence and weakly dependent data

This paper analyzes M-estimators over general objective functions. We do not assume convexity and differentiability of the functions. A new result regarding M-estimators is derived. Unlike most of the former econometric literature, the rate of convergence is not square root n. The rate of convergence is non-standard and depends on the moment bounds of the objective function analyzed. We can actually connect the rate of convergence to the smoothness of the objective function in certain class of functions as described in van der Vaart and Wellner (Weak Convergence and Empirical Processes, Springer, Berlin, 1996). We also simplify this rate of convergence idea and extend to weakly dependent data from iid case. This rate is simple and usable in econometrics literature. We illustrate the techniques by deriving the rate of convergence for LAD estimator for censored regression and maximum score estimator with weakly dependent data.     

Multivariate applications of the ash in regression

A simple algorithm for estimating the regression function over the United States is introduced. The approach allows for data obtained from a complicated sampling design, as well as for the inclusion of a few additional covariates. The regression estimates are obtained from an associated probability density estimate, namely the averaged shifted histogram. The algorithm has proven especially successful over a large mesh, say 300 by 200 nodes, in a data rich setting, even on a 486 computer running Splus. We currently run much higher resolution meshes on a Pentium. Commonly available alternative codes including kriging failed to produce useful estimates in this setting.     

Convergence Analysis of MCMC Algorithms for Bayesian Multivariate Linear Regression with Non‐Gaussian Errors

When Gaussian errors are inappropriate in a multivariate linear regression setting, it is often assumed that the errors are iid from a distribution that is a scale mixture of multivariate normals. Combining this robust regression model with a default prior on the unknown parameters results in a highly intractable posterior density. Fortunately, there is a simple data augmentation (DA) algorithm and a corresponding Haar PX‐DA algorithm that can be used to explore this posterior. This paper provides conditions (on the mixing density) for geometric ergodicity of the Markov chains underlying these Markov chain Monte Carlo algorithms. Letting d denote the dimension of the response, the main result shows that the DA and Haar PX‐DA Markov chains are geometrically ergodic whenever the mixing density is generalized inverse Gaussian, log‐normal, inverted Gamma (with shape parameter larger than d /2) or Fréchet (with shape parameter larger than d /2). The results also apply to certain subsets of the Gamma, F and Weibull families.     

A Proof of the Asymptotic Equivalence of Two-Tail Probability Approximations

A modified efficient jump algorithm is proposed for the Markov Chain Monte Carlo draws of the exponential power distribution. Bayesian inference based on the exponential power error term and that on the normal error term are compared. Unbiasedness of the LAD estimator is proven.     

Estimation and Inference Based on Neumann Series Approximation to Locally Efficient Score in Missing Data Problems

Abstract.  Theory on semi-parametric efficient estimation in missing data problems has been systematically developed by Robins and his coauthors. Except in relatively simple problems, semi-parametric efficient scores cannot be expressed in closed forms. Instead, the efficient scores are often expressed as solutions to integral equations. Neumann series was proposed in the form of successive approximation to the efficient scores in those situations. Statistical properties of the estimator based on the Neumann series approximation are difficult to obtain and as a result, have not been clearly studied. In this paper, we reformulate the successive approximation in a simple iterative form and study the statistical properties of the estimator based on the reformulation. We show that a doubly robust locally efficient estimator can be obtained following the algorithm in robustifying the likelihood score. The results can be applied to, among others, parametric regression, marginal regression and Cox regression when data are subject to missing values and the data are missing at random. A simulation study is conducted to evaluate the performance of the approach and a real data example is analysed to demonstrate the use of the approach.     

Assessing case influence on confidence intervals in nonlinear regression

Diagnostic techniques are proposed for assessing the influence of individual cases on confidence intervals in nonlinear regression. The technique proposed uses the method of profile t-plots applied to the case-deletion model. The effect of the geometry of the statistical model on the influence measures is assessed, and an algorithm for computing case-deleted confidence intervals is described. This algorithm provides a direct method for constructing a simple diagnostic measure based on the ratio of the lengths of confidence intervals. The generalization of these methods to multiresponse models is discussed.     

On adaptive linear regression

Ordinary least squares (OLS) is omnipresent in regression modeling. Occasionally, least absolute deviations (LAD) or other methods are used as an alternative when there are outliers. Although some data adaptive estimators have been proposed, they are typically difficult to implement. In this paper, we propose an easy to compute adaptive estimator which is simply a linear combination of OLS and LAD. We demonstrate large sample normality of our estimator and show that its performance is close to best for both light-tailed (e.g. normal and uniform) and heavy-tailed (e.g. double exponential and t ₃) error distributions. We demonstrate this through three simulation studies and illustrate our method on state public expenditures and lutenizing hormone data sets. We conclude that our method is general and easy to use, which gives good efficiency across a wide range of error distributions.