Outlier Resistant Estimation in Difference-Based Semiparametric Partially Linear Models

Partially linear models are extensions of linear models that include a nonparametric function of some covariate allowing an adequate and more flexible handling of explanatory variables than in linear models. The difference-based estimation in partially linear models is an approach designed to estimate parametric component by using the ordinary least squares estimator after removing the nonparametric component from the model by differencing. However, it is known that least squares estimates do not provide useful information for the majority of data when the error distribution is not normal, particularly when the errors are heavy-tailed and when outliers are present in the dataset. This paper aims to find an outlier-resistant fit that represents the information in the majority of the data by robustly estimating the parametric and the nonparametric components of the partially linear model. Simulations and a real data example are used to illustrate the feasibility of the proposed methodology and to compare it with the classical difference-based estimator when outliers exist.     

Robust difference-based outlier detection

Abstract

In this paper, we propose an outlier-detection approach that uses the properties of an intercept estimator in a difference-based regression model (DBRM) that we first introduce. This DBRM uses multiple linear regression, and invented it to detect outliers in a multiple linear regression. Our outlier-detection approach uses only the intercept; it does not require estimates for the other parameters in the DBRM. In this paper, we first employed a difference-based intercept estimator to study the outlier-detection problem in a multiple regression model. We compared our approach with several existing methods in a simulation study and the results suggest that our approach outperformed the others. We also demonstrated the advantage of our approach using a real data application. Our approach can extend to nonparametric regression models for outliers detection.     

Outlier detection using difference-based variance estimators in multiple regression

In this article, we propose an outlier detection approach in a multiple regression model using the properties of a difference-based variance estimator. This type of a difference-based variance estimator was originally used to estimate error variance in a non parametric regression model without estimating a non parametric function. This article first employed a difference-based error variance estimator to study the outlier detection problem in a multiple regression model. Our approach uses the leave-one-out type method based on difference-based error variance. The existing outlier detection approaches using the leave-one-out approach are highly affected by other outliers, while ours is not because our approach does not use the regression coefficient estimator. We compared our approach with several existing methods using a simulation study, suggesting the outperformance of our approach. The advantages of our approach are demonstrated using a real data application. Our approach can be extended to the non parametric regression model for outlier detection.     

Using difference-based methods for inference in nonparametric regression with time series errors

Summary. We show that difference-based methods can be used to construct simple and explicit estimators of error covariance and autoregressive parameters in nonparametric regression with time series errors. When the error process is Gaussian our estimators are efficient, but they are available well beyond the Gaussian case. As an illustration of their usefulness we show that difference-based estimators can be used to produce a simplified version of time series cross-validation. This new approach produces a bandwidth selector that is equivalent, to both first and second orders, to that given by the full time series cross-validation algorithm. Other applications of difference-based methods are to variance estimation and construction of confidence bands in nonparametric regression.     

Estimating the error variance in nonparametric regression by a covariate-matched u-statistic

For nonparametric regression models with fixed and random design, two classes of estimators for the error variance have been introduced: second sample moments based on residuals from a nonparametric fit, and difference-based estimators. The former are asymptotically optimal but require estimating the regression function; the latter are simple but have larger asymptotic variance. For nonparametric regression models with random covariates, we introduce a class of estimators for the error variance that are related to difference-based estimators: covariate-matched U-statistics. We give conditions on the random weights involved that lead to asymptotically optimal estimators of the error variance. Our explicit construction of the weights uses a kernel estimator for the covariate density.     

Variance estimation in nonparametric regression with jump discontinuities

Variance estimation is an important topic in nonparametric regression. In this paper, we propose a pairwise regression method for estimating the residual variance. Specifically, we regress the squared difference between observations on the squared distance between design points, and then estimate the residual variance as the intercept. Unlike most existing difference-based estimators that require a smooth regression function, our method applies to regression models with jump discontinuities. Our method also applies to the situations where the design points are unequally spaced. Finally, we conduct extensive simulation studies to evaluate the finite-sample performance of the proposed method and compare it with some existing competitors.     

Difference-based estimation and model identification for panel data semiparametric models with cross-section dependence

Abstract

In this article, we consider a panel data partially linear regression model with fixed effect and non parametric time trend function. The data can be dependent cross individuals through linear regressor and error components. Unlike the methods using non parametric smoothing technique, a difference-based method is proposed to estimate linear regression coefficients of the model to avoid bandwidth selection. Here the difference technique is employed to eliminate the non parametric function effect, not the fixed effects, on linear regressor coefficient estimation totally. Therefore, a more efficient estimator for parametric part is anticipated, which is shown to be true by the simulation results. For the non parametric component, the polynomial spline technique is implemented. The asymptotic properties of estimators for parametric and non parametric parts are presented. We also show how to select informative ones from a number of covariates in the linear part by using smoothly clipped absolute deviation-penalized estimators on a difference-based least-squares objective function, and the resulting estimators perform asymptotically as well as the oracle procedure in terms of selecting the correct model.     

Note on a linear model with errors in the variables and with trend

The least squares estimate of the slope parameter of a simple linear model with errors in the variables is typically biased. However the bias vanishes asymptotically for increasing sample size if the regressor variable follows a linear trend. For this case asymptotic expansion formulas for bias and variance of the least squares estimator are derived from exact expressions presented by Richardson and Wu (1970) and certain bounds to these expressions given by Friedmann (1990).     

Reducing errors-in-variables bias in linear regression using compact genetic algorithms

A new technique is devised to mitigate the errors-in-variables bias in linear regression. The procedure mimics a 2-stage least squares procedure where an auxiliary regression which generates a better behaved predictor variable is derived. The generated variable is then used as a substitute for the error-prone variable in the first-stage model. The performance of the algorithm is tested by simulation and regression analyses. Simulations suggest the algorithm efficiently captures the additive error term used to contaminate the artificial variables. Regressions provide further credit to the simulations as they clearly show that the compact genetic algorithm-based estimate of the true but unobserved regressor yields considerably better results. These conclusions are robust across different sample sizes and different variance structures imposed on both the measurement error and regression disturbances.     

Bias and mean squared error of the slope estimator in a regression with not necessarily normal errors in both variables

The least squares estimation of the slope parameter of a simple linear regression is biased if the regressor variable is measured with random errors. This bias as well as the mean squared error is computed up to the order of 1/T without assuming normality for the error variable. They depend on the fourth moment of the error variable.     

Smoothing the difference-based estimates of variance using variational mode decomposition

We propose a variational mode decomposition approach to estimate the variance function in a nonparametric heteroscedastic fixed design regression model. A data-driven estimator is constructed by applying variational mode decomposition technique to the difference-based initial estimates. The numerical results show that the proposed estimator performs better than the existing variance estimation procedures in the mean square sense.     

Local Linear Least Squares Kernel Regression for Linear and Circular Predictors

We extend nonparametric regression models with local linear least squares fitting using kernel weights to the case of linear and circular predictors. We derive the asymptotic properties of the conditional bias and variance of bivariate local linear least squares kernel estimators. A small simulation study and a real experiment are given.     

Testing discontinuities in nonparametric regression

In nonparametric regression, it is often needed to detect whether there are jump discontinuities in the mean function. In this paper, we revisit the difference-based method in [13] and propose to further improve it. To achieve the goal, we first reveal that their method is less efficient due to the inappropriate choice of the response variable in their linear regression model. We then propose a new regression model for estimating the residual variance and the total amount of discontinuities simultaneously. In both theory and simulation, we show that the proposed variance estimator has a smaller mean-squared error compared to the existing estimator, whereas the estimation efficiency for the total amount of discontinuities remains unchanged. Finally, we construct a new test procedure for detection of discontinuities using the proposed method; and via simulation studies, we demonstrate that our new test procedure outperforms the existing one in most settings.     

Errors-in-variables estimation with wavelets

This paper proposes a wavelet (spectral) approach to estimate the parameters of a linear regression model where the regressand and the regressors are persistent processes and contain a measurement error. We propose a wavelet filtering approach which does not require instruments and yields unbiased estimates for the intercept and the slope parameters. Our Monte Carlo results also show that the wavelet approach is particularly effective when measurement errors for the regressand and the regressor are serially correlated. With this paper, we hope to bring a fresh perspective and stimulate further theoretical research in this area.     

Kernel Density-Based Linear Regression Estimate

For linear regression models with non normally distributed errors, the least squares estimate (LSE) will lose some efficiency compared to the maximum likelihood estimate (MLE). In this article, we propose a kernel density-based regression estimate (KDRE) that is adaptive to the unknown error distribution. The key idea is to approximate the likelihood function by using a nonparametric kernel density estimate of the error density based on some initial parameter estimate. The proposed estimate is shown to be asymptotically as efficient as the oracle MLE which assumes the error density were known. In addition, we propose an EM type algorithm to maximize the estimated likelihood function and show that the KDRE can be considered as an iterated weighted least squares estimate, which provides us some insights on the adaptiveness of KDRE to the unknown error distribution. Our Monte Carlo simulation studies show that, while comparable to the traditional LSE for normal errors, the proposed estimation procedure can have substantial efficiency gain for non normal errors. Moreover, the efficiency gain can be achieved even for a small sample size.     

Statistical inference for fixed-effects partially linear regression models with errors in variables

Fixed-effects partially linear regression models are useful tools to analyze data from economic, genetic and other fields. In this paper, we consider estimation and inference procedures when some of the covariates are measured with errors. The previously proposed estimations, including difference-based series estimation (Baltagi and Li in Ann Econ Finan 3:103--116, 2002) and profile least squares estimation (Fan et al. in J Am Stat Assoc 100:781--813, 2005) are no longer consistent because of the attenuation. We propose a new estimation by taking the measurement errors into account. Our proposed estimators are shown to be consistent and asymptotically normal. Consistent estimations of the error variance are also developed. In addition, we propose a variable-selection procedure to variable selection in the parametric part. The procedure is an extension of the nonconcave penalized likelihood (Fan and Li in J Am Stat Assoc 85:1348--1360, 2001), which simultaneously selects the important variables and estimates the unknown parameters. The resulting estimate is shown to possess an oracle property. Extensive simulation studies are conducted to illustrate the finite sample performance of the proposed procedures.     

AN EXAMINATION OF ESTIMATED RESIDUALS IN A REGRESSION WITH AN INFINITE ORDER PARAMETRIC MODEL

We consider a linear regression with the error term that obeys an autoregressive model of infinite order and estimate parameters of the models. The parameters of the autoregressive model should be estimated based on estimated residuals obtained by means of the method of ordinary least squares, because the errors are unobservable. The consistency of the coefficients, variance and spectral density of the model obeyed by the error term is shown. Further, we estimate the coefficients of the linear regression by means of the method of estimated generalized least squares. We also show the consistency of the estimator.

     

Specification of household engel curves by nonparametric regression

This paper demonstrates the usefulness of nonparametric regression analysis for functional specfication of houshold Engel curves.

After a brief review in section 2 of the literature on demand functions and equivalence scales and the functional specifications used, we first discuss in section 3 the issues of using income versus total expenditure, the origin and nature of the error terms in the light of utility theroy, and the interpretation of empirical demand functions. we shall reach the unorthodox view that household demand functions should be interpreted as conditional expectations relative to prices, household composition and either income or the conditional expectation of total expenditure (rather that total expenditure itself), where the latter conditional expectation is taken relative to income, prices and household composition. these two forms appear to be equivalent. this result also solves the simultaneity problem: the error variance matrix is no longer singular. Moreover, the errors are in general heteroskedastic.

In section 4 we discuss the model and the data, and in section 5 we review the nonparametric kernal regression approach.

In section 6 we derive the functional form of our household engel curves from nonparametric regression results, using the 1980 budget survey for the netherlands, in order to avoid model misspecification. thus the modl is derived directly from the data, without restricting its functional form. the nonparametric regression results are then translated to suitable parametric functional specifications, i.e., we choose parametric functional forms in accordance with the nanparametric regression results. these parametric specification are estimated by least squares, and various parameter restrictions are tested in order to simplify the models. this yields very simple final specifications of the household engel curves involved, namely linear functions of income and the number of children in two age groups.     

Specification of household engel curves by nonparametric regression

This paper demonstrates the usefulness of nonparametric regression analysis for functional specfication of houshold Engel curves.

After a brief review in section 2 of the literature on demand functions and equivalence scales and the functional specifications used, we first discuss in section 3 the issues of using income versus total expenditure, the origin and nature of the error terms in the light of utility theroy, and the interpretation of empirical demand functions. we shall reach the unorthodox view that household demand functions should be interpreted as conditional expectations relative to prices, household composition and either income or the conditional expectation of total expenditure (rather that total expenditure itself), where the latter conditional expectation is taken relative to income, prices and household composition. these two forms appear to be equivalent. this result also solves the simultaneity problem: the error variance matrix is no longer singular. Moreover, the errors are in general heteroskedastic.

In section 4 we discuss the model and the data, and in section 5 we review the nonparametric kernal regression approach.

In section 6 we derive the functional form of our household engel curves from nonparametric regression results, using the 1980 budget survey for the netherlands, in order to avoid model misspecification. thus the modl is derived directly from the data, without restricting its functional form. the nonparametric regression results are then translated to suitable parametric functional specifications, i.e., we choose parametric functional forms in accordance with the nanparametric regression results. these parametric specification are estimated by least squares, and various parameter restrictions are tested in order to simplify the models. this yields very simple final specifications of the household engel curves involved, namely linear functions of income and the number of children in two age groups.     

Estimating the variance in nonparametric regression—what is a reasonable choice?

The exact mean-squared error (MSE) of estimators of the variance in nonparametric regression based on quadratic forms is investigated. In particular, two classes of estimators are compared: Hall, Kay and Titterington's optimal difference-based estimators and a class of ordinary difference-based estimators which generalize methods proposed by Rice and Gasser, Sroka and Jennen-Steinmetz. For small sample sizes the MSE of the first estimator is essentially increased by the magnitude of the integrated first two squared derivatives of the regression function. It is shown that in many situations ordinary difference-based estimators are more appropriate for estimating the variance, because they control the bias much better and hence have a much better overall performance. It is also demonstrated that Rice's estimator does not always behave well. Data-driven guidelines are given to select the estimator with the smallest MSE.