Techniques for nonlinear least squares and robust regression

Recently, the authors and others have made considerable progress in developing algorithms for solving certain large-residual nonlinear least-squares problems where Gauss-Newton (GN) methods can be expected to perform poorly. These methods take account of the term in the Hessian ignored by the GN methods and use quasi-Newton procedures to update this term explicitly. This paper reviews these new approaches and discusses how they can be modified to give good performance on nonlinear models with robust loss functions where lack of scale invariance causes several new problems to arise.