Diagnostics for non-linear regression

Sensitivity analysis in regression is concerned with assessing the sensitivity of the results of a regression model (e.g., the objective function, the regression parameters, and the fitted values) to changes in the data. Sensitivity analysis in least squares linear regression has seen a great surge of research activities over the last three decades. By contrast, sensitivity analysis in non-linear regression has received very little attention. This paper deals with the problem of local sensitivity analysis in non-linear regression. Closed-form general formulas are provided for the sensitivities of three standard methods for the estimation of the parameters of a non-linear regression model based on a set of data. These methods are the least squares, the minimax, and the least absolute value methods. The effectiveness of the proposed measures is illustrated by application to several non-linear models including the ultrasonic data and the onion yield data. The proposed sensitivity measures are shown to deal effectively with the detection of influential observations in non-linear regression models.     

Bootstrap versus traditional hypothesis testing procedures for coefficients in least absolute value regression

Several approaches to hypothesis testing for coefficients in least absolute value regression are compared using a Monte Carlo simulation: likelihood ratio test, Lagrange multiplier test, and three versions of the bootstrap hypothesis test. Factors considered that might influence test performance include the disturbance distribution, the type of independent variable, and the sample size. Overall, the likelihood ratio and the bootstrap tests perform best, with the likelihood ratio test being marginally more powerful. Least absolute value tests are also compared to the standard t test and three versions of the bootstrapped t test for least squares regression.     

Nonuniqueness of least absolute values regression

The question of nonuniqueness of the least absolute values (L₁) regression is discussed, and examples are given of situations where the L₁-regression is unique and where it has 2, 3 and 4 limiting positions. A method is proposed for finding a compromise regression line when the L₁ regression is not unique. Suggestions are made for further research.     

On finite-sample properties of adaptive least squares regression estimates

This paper is mainly concerned with deriving finite-sample properties of least squares estimators for the regression function in a nonparametric regression situation under some simplifying assumptions such as normally distributed errors with a common known variance. The selection of basis functions to be used for the construction of an estimator may be regarded as a smoothing problem, and will usually be done in a data-dependent way, A straightforward application of a result by P. J. Kernpthorne yields that, under a squared error loss, all selection procedures are admissible. Furthermore, the minimax approach provides an interpolating estimator, which is often impractical, Therefore, within a certain class of selection procedures an optimal one is determined using the minimax regret principle. It can be seen to behave similarly to the procedure minimizing either an unbiased risk estimator or, equivalently, the C_p-criterion.     

A general method of finding a minimax estimator of a distribution function when no equalizer rule is available

We propose a feasible general method for finding a minimax estimator of an unknown distribution function F in the nonparametric problem. As an application, some minimax estimators are proposed. Furthermore, some minimax binomial parametric problems are studied.     

Minimax prediction in the linear model with a relative squared error

Arnold and Stahlecker (Stat Pap 44:107–115, 2003) considered the prediction of future values of the dependent variable in the linear regression model with a relative squared error and deterministic disturbances. They found an explicit form for a minimax linear affine solution d* of that problem. In the paper we generalize this result proving that the decision rule d* is also minimax when the class D{\mathcal{D}} of possible predictors of the dependent variable is unrestricted. Then we show that d* remains minimax in D{\mathcal{D}} when the disturbances are random with the mean vector zero and the known positive definite covariance matrix.     

Sensitivity analysis in ordered and restricted parameter models

This paper deals with the problem of local sensitivity analysis in ordered parameter models. In addition to order restrictions, some constraints imposed on the parameters by the model and/or the data are considered. Measures for assessing how much a change in the data modifies the results and conclusions of a statistical analysis of these models are presented. The sensitivity measures are derived using recent results in mathematical programming. The estimation problem is formulated as a primal nonlinear programming problem, and the sensitivities of the parameter estimates as well as the objective function sensitivities with respect to data are obtained. They are very effective in revealing the influential observations in this type of models and in evaluating the changes due to changes in data values. The methods are illustrated by their application to a wide variety of examples of order-restricted models including ordered exponential family parameters, ordered multinomial parameters, ordered linear model parameters, ordered and data constrained parameters, and ordered functions of parameters.     

Examining rounding error in least absolute values regression computations

Two techniques for detecting inaccuracies in least absolute values (LAV) regression computations are presented and discussed. Examples of the use of the methods are given. The techniques are shown to apply to the more general case of M-estimation.     

Restricted minimax estimation in semiparametric linear models

Abstract

In this article we develop the minimax estimation approach of general linear models to the semiparametric linear models when the parameters are simultaneously constrained by an ellipsoid and linear restrictions. Combining sample information and prior constraints the minimax estimator is obtained and compared with partially least square estimator by theoretical and simulation methods.     

Designs for weighted least squares regression, with estimated weights

We study designs, optimal up to and including terms that are O(n ^?1), for weighted least squares regression, when the weights are intended to be inversely proportional to the variances but are estimated with random error. We take a finite, but arbitrarily large, design space from which the support points are to be chosen, and obtain the optimal proportions of observations to be assigned to each point. Specific examples of D- and I-optimal design for polynomial responses are studied. In some cases the same designs that are optimal under homoscedasticity remain so for a range of variance functions; in others there tend to be more support points than are required in the homoscedastic case. We also exhibit minimax designs, that minimize the maximum, over finite classes of variance functions, value of the loss. These also tend to have more support points, often resulting from the breaking down of replicates into clusters.     

Obtaining efficient estimates of park use and testing for the structural adequacy of models

This paper provides an examination of the problem of heteroscedasticity as it relates to estimating park use, although the results can also be applied to a wide variety of flow problems involving traffic, people or commodities. The major issue is that estimates of flows obtained using ordinary least squares, OLS, often yield statistically significant results while still giving rise to large differences between observed and predicted flows (residuals). The paper presents results which show that for the flow estimation problem of concern, more accurate use estimates may be obtained by using generalized least squares, GLS, rather than using OLS. Weights to use in GLS regression are developed taking into account the variance to be expected in origin-destination flows. It is shown that deriving the correct weights, estimates of variances, to use in a regression analysis results in an ‘absolute’ test for the structural appropriateness of the regression model. Tests related to the ‘absolute’ adequacy test are introduced and their use to identify specific structural problems with a model is illustrated.     

Bayes minimax estimation of a bernoulli p in a restricted parameter space

In many estimation problems the parameter of interest is known,a priori, to belong to a proper subspace of the natural parameter space. Although useful in practice this type of additional information can lead to surprising theoretical difficulties. In this paper the problem of minimax estimation of a Bernoulli pwhen pis restricted to a symmetric subinterval of the natural parameter space is considered. For the sample sizes n = 1,2,3, and 4 least favorable priors with finite support are provided and the corresponding Bayes estimators are shown to be minimax. For n = 5 and 6 the usual constant risk minimax estimator is shown to be the Bayes minimax estimator corresponding to a least favorable prior with finite support, provided the restriction on the parameter space is not too tight.     

On least absolute values estimation

The resistance of least absolute values (L₁) estimators to outliers and their robustness to heavy-tailed distributions make these estimators useful alternatives to the usual least squares estimators. The recent development of efficient algorithms for L₁ estimation in linear models has permitted their use in practical data analysis. Although in general the L₁ estimators are not unique, there are a number of properties they all share. The set of all L₁ estimators for a given model and data set can be characterized as the convex hull of some extreme estimators. Properties of the extreme estimators and of the L₁-estimate set are considered.     

Fitting linear regression models to censored data by least squares and maximum likelihood methods

Several approaches have been suggested for fitting linear regression models to censored data. These include Cox's propor­tional hazard models based on quasi-likelihoods. Methods of fitting based on least squares and maximum likelihoods have also been proposed. The methods proposed so far all require special purpose optimization routines. We describe an approach here which requires only a modified standard least squares routine.

We present methods for fitting a linear regression model to censored data by least squares and method of maximum likelihood. In the least squares method, the censored values are replaced by their expectations, and the residual sum of squares is minimized. Several variants are suggested in the ways in which the expect­ation is calculated. A parametric (assuming a normal error model) and two non-parametric approaches are described. We also present a method for solving the maximum likelihood equations in the estimation of the regression parameters in the censored regression situation. It is shown that the solutions can be obtained by a recursive algorithm which needs only a least squares routine for optimization. The suggested procesures gain considerably in computational officiency. The Stanford Heart Transplant data is used to illustrate the various methods.     

Improving parameter estimation using constrained optimization methods

This paper presents a methodology based on transforming estimation methods in optimization problems in order to incorporate in a natural way some constraints that contain extra information not considered by standard estimation methods, with the aim of improving the quality of the parameter estimates. We include here three types of such information: bounds for the cumulative distribution function, bounds for the quantiles, and any restrictions on the parameters such as those imposed by the support of the random variable under consideration. The method is quite general and can be applied to many estimation methods such as the maximum likelihood (ML), the method of moments (MOM), the least squares, the least absolute values, and the minimax methods. The performances of the obtained estimates from several families of distributions are investigated for the ML and the MOM, using simulations. The simulation results show that for small sample sizes important gains can be achieved with respect to the case where the above information is ignored. In addition, we discuss sensitivity analysis methods for assessing the influence of observations on the proposed estimators. The method applies to both univariate and multivariate data.     

Obtaining least absolute value estimates for a two-way classification model

The importance of the two-way classification model is well known, but the standard method of analysis is least squares. Often, the data of the model calls for a more robust estimation technique. This paper demonstrates the equivalence between the problem of obtaining least absolute value estimates for the two-way classification model and a capacitated transportation problem. A special purpose primal algorithm is developed to provide the least absolute value estimates. A computational comparison is made between an implementation of this specialized algorithm and a standard capacitated transportation code.     

Analysis of least squares regression estimates in case of additional errors in the variables

Estimation of a regression function from independent and identical distributed data is considered. The L₂ error with integration with respect to the design measure is used as error criterion. Upper bounds on the L₂ error of least squares regression estimates are presented, which bound the error of the estimate in case that in the sample given to the estimate the values of the independent and the dependent variables are pertubated by some arbitrary procedure. The bounds are applied to analyze regression-based Monte Carlo methods for pricing American options in case of errors in modelling the price process.     

A dual problem of calibration of design weights

In this note, a dual problem to the calibration of design weights of the Deville and Särndal [Calibration estimators in survey sampling, J. Amer. Statist. Assoc. 87 (1992), pp. 376–382] method has been considered. We conclude that the chi-squared distance between the design weights and the calibrated weights equals the square of the standardized Z-score obtained by the difference between the known population total of the auxiliary variable and its corresponding Horvitz and Thompson [A generalization of sampling without replacement from a finite universe, J. Amer. Statist. Assoc. 47 (1952), pp. 663–685] estimator divided by the sample standard deviation of the auxiliary variable to obtain the linear regression estimator in survey sampling.     

Total least squares solution for compositional data using linear models

The restrictive properties of compositional data, that is multivariate data with positive parts that carry only relative information in their components, call for special care to be taken while performing standard statistical methods, for example, regression analysis. Among the special methods suitable for handling this problem is the total least squares procedure (TLS, orthogonal regression, regression with errors in variables, calibration problem), performed after an appropriate log-ratio transformation. The difficulty or even impossibility of deeper statistical analysis (confidence regions, hypotheses testing) using the standard TLS techniques can be overcome by calibration solution based on linear regression. This approach can be combined with standard statistical inference, for example, confidence and prediction regions and bounds, hypotheses testing, etc., suitable for interpretation of results. Here, we deal with the simplest TLS problem where we assume a linear relationship between two errorless measurements of the same object (substance, quantity). We propose an iterative algorithm for estimating the calibration line and also give confidence ellipses for the location of unknown errorless results of measurement. Moreover, illustrative examples from the fields of geology, geochemistry and medicine are included. It is shown that the iterative algorithm converges to the same values as those obtained using the standard TLS techniques. Fitted lines and confidence regions are presented for both original and transformed compositional data. The paper contains basic principles of linear models and addresses many related problems.