Automatic bandwidth selection in robust nonparametric regression

Nonparametric regression techniques have been studied extensively in the literature in recent years due to their flexibility.In addition robust versions of these techniques have become popular and have been incorporated into some of the standard statistical analysis packages.With new techniques available comes the responsibility of using them properly and in appropriate situations. Often, as in the case presented here, model-fitting diagnostics, such as cross-validation statistics,are not available as tools to determine if the smoothing parameter value being used is preferable to some other arbitrarily chosen value.     

Bandwidth selection for local linear regression smoothers

Summary. The paper presents a general strategy for selecting the bandwidth of nonparametric regression estimators and specializes it to local linear regression smoothers. The procedure requires the sample to be divided into a training sample and a testing sample. Using the training sample we first compute a family of regression smoothers indexed by their bandwidths. Next we select the bandwidth by minimizing the empirical quadratic prediction error on the testing sample. The resulting bandwidth satisfies a finite sample oracle inequality which holds for all bounded regression functions. This permits asymptotically optimal estimation for nearly any regression function. The practical performance of the method is illustrated by a simulation study which shows good finite sample behaviour of our method compared with other bandwidth selection procedures.     

More efficient local polynomial regression with random-effects panel data models

We propose a modification on the local polynomial estimation procedure to account for the “within-subject” correlation presented in panel data. The proposed procedure is rather simple to compute and has a closed-form expression. We study the asymptotic bias and variance of the proposed procedure and show that it outperforms the working independence estimator uniformly up to the first order. Simulation study shows that the gains in efficiency with the proposed method in the presence of “within-subject” correlation can be significant in small samples. For illustration purposes, the procedure is applied to explore the impact of market concentration on airfare.     

Bootstrap inference in local polynomial regression of time series

In this paper we consider the inferential aspect of the nonparametric estimation of a conditional function , where X _t,m represents the vector containing the m conditioning lagged values of the series. Here is an arbitrary measurable function. The local polynomial estimator of order p is used for the estimation of the function g, and of its partial derivatives up to a total order p. We consider α-mixing processes, and we propose the use of a particular resampling method, the local polynomial bootstrap, for the approximation of the sampling distribution of the estimator. After analyzing the consistency of the proposed method, we present a simulation study which gives evidence of its finite sample behaviour.     

On variable selection in generalized linear and related regression models

This paper is concerned with selection of explanatory variables in generalized linear models (GLM). The class of GLM's is quite large and contains e.g. the ordinary linear regression, the binary logistic regression, the probit model and Poisson regression with linear or log-linear parameter structure. We show that, through an approximation of the log likelihood and a certain data transformation, the variable selection problem in a GLM can be converted into variable selection in an ordinary (unweighted) linear regression model. As a consequence no specific computer software for variable selection in GLM's is needed. Instead, some suitable variable selection program for linear regression can be used. We also present a simulation study which shows that the log likelihood approximation is very good in many practical situations. Finally, we mention briefly possible extensions to regression models outside the class of GLM's.     

Bayesian variable selection approach to a Bernstein polynomial regression model with stochastic constraints

This paper provides a Bayesian estimation procedure for monotone regression models incorporating the monotone trend constraint subject to uncertainty. For monotone regression modeling with stochastic restrictions, we propose a Bayesian Bernstein polynomial regression model using two-stage hierarchical prior distributions based on a family of rectangle-screened multivariate Gaussian distributions extended from the work of Gurtis and Ghosh [7 S.M. Curtis and S.K. Ghosh, A variable selection approach to monotonic regression with Bernstein polynomials, J. Appl. Stat. 38 (2011), pp. 961–976. doi: 10.1080/02664761003692423[Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]]. This approach reflects the uncertainty about the prior constraint, and thus proposes a regression model subject to monotone restriction with uncertainty. Based on the proposed model, we derive the posterior distributions for unknown parameters and present numerical schemes to generate posterior samples. We show the empirical performance of the proposed model based on synthetic data and real data applications and compare the performance to the Bernstein polynomial regression model of Curtis and Ghosh [7 S.M. Curtis and S.K. Ghosh, A variable selection approach to monotonic regression with Bernstein polynomials, J. Appl. Stat. 38 (2011), pp. 961–976. doi: 10.1080/02664761003692423[Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]] for the shape restriction with certainty. We illustrate the effectiveness of our proposed method that incorporates the uncertainty of the monotone trend and automatically adapts the regression function to the monotonicity, through empirical analysis with synthetic data and real data applications.     

On the local linear estimate for functional regression: Uniform in bandwidth consistency

We consider the problem of local linear estimation of the regression function when the regressor is functional. The main result of this paper is to prove the strong convergence (with rates), uniformly in bandwidth parameters (UIB), of the considered estimator. The main interest of this result is the possibility to derive the asymptotic properties of our estimate even if the bandwidth parameter is a random variable.     

Multivariate bandwidth selection for local linear regression

The existence and properties of optimal bandwidths for multivariate local linear regression are established, using either a scalar bandwidth for all regressors or a diagonal bandwidth vector that has a different bandwidth for each regressor. Both involve functionals of the derivatives of the unknown multivariate regression function. Estimating these functionals is difficult primarily because they contain multivariate derivatives. In this paper, an estimator of the multivariate second derivative is obtained via local cubic regression with most cross-terms left out. This estimator has the optimal rate of convergence but is simpler and uses much less computing time than the full local estimator. Using this as a pilot estimator, we obtain plug-in formulae for the optimal bandwidth, both scalar and diagonal, for multivariate local linear regression. As a simpler alternative, we also provide rule-of-thumb bandwidth selectors. All these bandwidths have satisfactory performance in our simulation study.     

A computational framework for variable selection in multivariate regression

Stepwise variable selection procedures are computationally inexpensive methods for constructing useful regression models for a single dependent variable. At each step a variable is entered into or deleted from the current model, based on the criterion of minimizing the error sum of squares (SSE). When there is more than one dependent variable, the situation is more complex. In this article we propose variable selection criteria for multivariate regression which generalize the univariate SSE criterion. Specifically, we suggest minimizing some function of the estimated error covariance matrix: the trace, the determinant, or the largest eigenvalue. The computations associated with these criteria may be burdensome. We develop a computational framework based on the use of the SWEEP operator which greatly reduces these calculations for stepwise variable selection in multivariate regression.     

One-step local quasi-likelihood estimation

Local quasi-likelihood estimation is a useful extension of local least squares methods, but its computational cost and algorithmic convergence problems make the procedure less appealing, particularly when it is iteratively used in methods such as the back-fitting algorithm, cross-validation and bootstrapping. A one-step local quasi-likelihood estimator is introduced to overcome the computational drawbacks of the local quasi-likelihood method. We demonstrate that as long as the initial estimators are reasonably good, the one-step estimator has the same asymptotic behaviour as the local quasi-likelihood method. Our simulation shows that the one-step estimator performs at least as well as the local quasi-likelihood method for a wide range of choices of bandwidths. A data-driven bandwidth selector is proposed for the one-step estimator based on the pre-asymptotic substitution method of Fan and Gijbels. It is then demonstrated that the data-driven one-step local quasi-likelihood estimator performs as well as the maximum local quasi-likelihood estimator by using the ideal optimal bandwidth.     

Double-smoothing for bias reduction in local linear regression

Local linear regression involves fitting a straight line segment over a small region whose midpoint is the target point x, and the local linear estimate at x   is the estimated intercept of that straight line segment, with an asymptotic bias of order h²$h^2$ and variance of order (nh)^-1$(\mathit{nh}{)}^{-1}$ (h is the bandwidth). In this paper, we propose a new estimator, the double-smoothing local linear estimator, which is constructed by integrally combining all fitted values at x   of local lines in its neighborhood with another round of smoothing. The proposed estimator attempts to make use of all information obtained from fitting local lines. Without changing the order of variance, the new estimator can reduce the bias to an order of h⁴$h^4$. The proposed estimator has better performance than local linear regression in situations with considerable bias effects; it also has less variability and more easily overcomes the sparse data problem than local cubic regression. At boundary points, the proposed estimator is comparable to local linear regression. Simulation studies are conducted and an ethanol example is used to compare the new approach with other competitive methods.     

Coefficients of determinations for variable selection in the msae regression

Our objective is to modify a robust coefficient of determination for the minimum sum of absolute errors MSAE regression proposed by McKean and Sievers (1987) so that it satisfies all the desirable properties. We also propose an adjusted coefficient of determination that is appropriate for comparing several models with different number of variables. Further, it has the property that if it decreases with the addition of predictor variables to the model, then the contribution of these variables is statistically non-significant. We illustrate the results with an example.     

Absolute error criteria for bandwidth selection in density estimation from censored data

In Kernel density estimation, a criticism of bandwidth selection techniques which minimize squared error expressions is that they perform poorly when estimating tails of probability density functions. Techniques minimizing absolute error expressions are thought to result in more uniform performance and be potentially superior. An asympotic mean absolute error expression for nonparametric kernel density estimators from right-censored data is developed here. This expression is used to obtain local and global bandwidths that are optimal in the sense that they minimize asymptotic mean absolute error and integrated asymptotic mean absolute error, respectively. These estimators are illustrated fro eight data sets from known distributions. Computer simulation results are discussed, comparing the estimation methods with squared-error-based bandwidth selection for right-censored data.     

Local polynomial regression and simulation–extrapolation

Summary.  The paper introduces a new local polynomial estimator and develops supporting asymptotic theory for nonparametric regression in the presence of covariate measurement error. We address the measurement error with Cook and Stefanski's simulation–extrapolation (SIMEX) algorithm. Our method improves on previous local polynomial estimators for this problem by using a bandwidth selection procedure that addresses SIMEX's particular estimation method and considers higher degree local polynomial estimators. We illustrate the accuracy of our asymptotic expressions with a Monte Carlo study, compare our method with other estimators with a second set of Monte Carlo simulations and apply our method to a data set from nutritional epidemiology. SIMEX was originally developed for parametric models. Although SIMEX is, in principle, applicable to nonparametric models, a serious problem arises with SIMEX in nonparametric situations. The problem is that smoothing parameter selectors that are developed for data without measurement error are no longer appropriate and can result in considerable undersmoothing. We believe that this is the first paper to address this difficulty.     

Linex discrepancy for bandwidth selection

A bandwidth selection based on Linex discrepancy is proposed for kernel smoothing of periodogram. The selection minimizes Linex discrepancy between the smoothed and true spectrums. Two estimators are introduced for Linex discrepancy. The bandwidth choice outperforms some common bandwidth choices.     

On some stepdown procedures with application to consistent variable selection in linear regression

The paper considers the problem of consistent variable selection with the use of stepdown procedures in the classical linear regression model and for the model with dependent errors. The stated results complete the results obtained by Bunea et al. [Consistent variable selection in high dimensional regression via multiple testing. J Stat Plann Inference. 2006;136(12):4349–4364].     

Shrinkage inverse regression estimation for model-free variable selection

Summary.  The family of inverse regression estimators that was recently proposed by Cook and Ni has proven effective in dimension reduction by transforming the high dimensional predictor vector to its low dimensional projections. We propose a general shrinkage estimation strategy for the entire inverse regression estimation family that is capable of simultaneous dimension reduction and variable selection. We demonstrate that the new estimators achieve consistency in variable selection without requiring any traditional model, meanwhile retaining the root n estimation consistency of the dimension reduction basis. We also show the effectiveness of the new estimators through both simulation and real data analysis.     

Adaptive elastic net-penalized quantile regression for variable selection

Abstract

There has been much attention on the high-dimensional linear regression models, which means the number of observations is much less than that of covariates. Considering the fact that the high dimensionality often induces the collinearity problem, in this article, we study the penalized quantile regression with the elastic net (EnetQR) that combines the strengths of the quadratic regularization and the lasso shrinkage. We investigate the weak oracle property of the EnetQR under mild conditions in the high dimensional setting. Moreover, we propose a two-step procedure, called adaptive elastic net quantile regression (AEnetQR), in which the weight vector in the second step is constructed from the EnetQR estimate in the first step. This two-step procedure is justified theoretically to possess the weak oracle property. The finite sample properties are performed through the Monte Carlo simulation and a real-data analysis.