A model-averaging approach for smoothing spline regression

ABSTRACT

This article considers nonparametric regression problems and develops a model-averaging procedure for smoothing spline regression problems. Unlike most smoothing parameter selection studies determining an optimum smoothing parameter, our focus here is on the prediction accuracy for the true conditional mean of Y given a predictor X. Our method consists of two steps. The first step is to construct a class of smoothing spline regression models based on nonparametric bootstrap samples, each with an appropriate smoothing parameter. The second step is to average bootstrap smoothing spline estimates of different smoothness to form a final improved estimate. To minimize the prediction error, we estimate the model weights using a delete-one-out cross-validation procedure. A simulation study has been performed by using a program written in R. The simulation study provides a comparison of the most well known cross-validation (CV), generalized cross-validation (GCV), and the proposed method. This new method is straightforward to implement, and gives reliable performances in simulations.     

Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion

Many different methods have been proposed to construct nonparametric estimates of a smooth regression function, including local polynomial, (convolution) kernel and smoothing spline estimators. Each of these estimators uses a smoothing parameter to control the amount of smoothing performed on a given data set. In this paper an improved version of a criterion based on the Akaike information criterion (AIC), termed AIC_C, is derived and examined as a way to choose the smoothing parameter. Unlike plug-in methods, AIC_C can be used to choose smoothing parameters for any linear smoother, including local quadratic and smoothing spline estimators. The use of AIC_C avoids the large variability and tendency to undersmooth (compared with the actual minimizer of average squared error) seen when other 'classical' approaches (such as generalized cross-validation (GCV) or the AIC) are used to choose the smoothing parameter. Monte Carlo simulations demonstrate that the AIC_C-based smoothing parameter is competitive with a plug-in method (assuming that one exists) when the plug-in method works well but also performs well when the plug-in approach fails or is unavailable.     

S-estimator in partially linear regression models

In this paper, a robust estimator is proposed for partially linear regression models. We first estimate the nonparametric component using the penalized regression spline, then we construct an estimator of parametric component by using robust S-estimator. We propose an iterative algorithm to solve the proposed optimization problem, and introduce a robust generalized cross-validation to select the penalized parameter. Simulation studies and a real data analysis illustrate that the our proposed method is robust against outliers in the dataset or errors with heavy tails.     

A method for choosing the smoothing parameter in a semi-parametric model for detecting change-points in blood flow

In a smoothing spline model with unknown change-points, the choice of the smoothing parameter strongly influences the estimation of the change-point locations and the function at the change-points. In a tumor biology example, where change-points in blood flow in response to treatment were of interest, choosing the smoothing parameter based on minimizing generalized cross-validation (GCV) gave unsatisfactory estimates of the change-points. We propose a new method, aGCV, that re-weights the residual sum of squares and generalized degrees of freedom terms from GCV. The weight is chosen to maximize the decrease in the generalized degrees of freedom as a function of the weight value, while simultaneously minimizing aGCV as a function of the smoothing parameter and the change-points. Compared with GCV, simulation studies suggest that the aGCV method yields improved estimates of the change-point and the value of the function at the change-point.     

Performance of Robust GCV and Modified GCV for Spline Smoothing

Abstract. While it is a popular selection criterion for spline smoothing, generalized cross‐validation (GCV) occasionally yields severely undersmoothed estimates. Two extensions of GCV called robust GCV (RGCV) and modified GCV have been proposed as more stable criteria. Each involves a parameter that must be chosen, but the only guidance has come from simulation results. We investigate the performance of the criteria analytically. In most studies, the mean square prediction error is the only loss function considered. Here, we use both the prediction error and a stronger Sobolev norm error, which provides a better measure of the quality of the estimate. A geometric approach is used to analyse the superior small‐sample stability of RGCV compared to GCV. In addition, by deriving the asymptotic inefficiency for both the prediction error and the Sobolev error, we find intervals for the parameters of RGCV and modified GCV for which the criteria have optimal performance.     

Free-knot polynomial splines with confidence intervals

Summary.  We construct approximate confidence intervals for a nonparametric regression function, using polynomial splines with free-knot locations. The number of knots is determined by generalized cross-validation. The estimates of knot locations and coefficients are obtained through a non-linear least squares solution that corresponds to the maximum likelihood estimate. Confidence intervals are then constructed based on the asymptotic distribution of the maximum likelihood estimator. Average coverage probabilities and the accuracy of the estimate are examined via simulation. This includes comparisons between our method and some existing methods such as smoothing spline and variable knots selection as well as a Bayesian version of the variable knots method. Simulation results indicate that our method works well for smooth underlying functions and also reasonably well for discontinuous functions. It also performs well for fairly small sample sizes.     

ON SPLINE SMOOTHING WITH AUTOCORRELATED ERRORS

The generalised cross-validation criterion for choosing the degree of smoothing in spline regression is extended to accommodate an autocorrelated error sequence. It is demonstrated via simulation that the minimum generalised cross-validation smoothing spline is an inconsistent estimator in the presence of autocorrelated errors and that ignoring even moderate autocorrelation structure can seriously affect the performance of the cross-validated smoothing spline. The method of penalised maximum likelihood is used to develop an efficient algorithm for the case in which the autocorrelation decays exponentially. An application of the method to a published data-set is described. The method does not require the data to be equally spaced in time.     

Composite quantile regression for varying-coefficient single-index models

ABSTRACT

The varying-coefficient single-index model (VCSIM) is a very general and flexible tool for exploring the relationship between a response variable and a set of predictors. Popular special cases include single-index models and varying-coefficient models. In order to estimate the index-coefficient and the non parametric varying-coefficients in the VCSIM, we propose a two-stage composite quantile regression estimation procedure, which integrates the local linear smoothing method and the information of quantile regressions at a number of conditional quantiles of the response variable. We establish the asymptotic properties of the proposed estimators for the index-coefficient and varying-coefficients when the error is heterogeneous. When compared with the existing mean-regression-based estimation method, our simulation results indicate that our proposed method has comparable performance for normal error and is more robust for error with outliers or heavy tail. We illustrate our methodologies with a real example.     

Spatially-adaptive Penalties for Spline Fitting

The paper studies spline fitting with a roughness penalty that adapts to spatial heterogeneity in the regression function. The estimates are p th degree piecewise polynomials with p − 1 continuous derivatives. A large and fixed number of knots is used and smoothing is achieved by putting a quadratic penalty on the jumps of the p th derivative at the knots. To be spatially adaptive, the logarithm of the penalty is itself a linear spline but with relatively few knots and with values at the knots chosen to minimize the generalized cross validation (GCV) criterion. This locally-adaptive spline estimator is compared with other spline estimators in the literature such as cubic smoothing splines and knot-selection techniques for least squares regression. Our estimator can be interpreted as an empirical Bayes estimate for a prior allowing spatial heterogeneity. In cases of spatially heterogeneous regression functions, empirical Bayes confidence intervals using this prior achieve better pointwise coverage probabilities than confidence intervals based on a global-penalty parameter. The method is developed first for univariate models and then extended to additive models.     

A non-iterative optimization method for smoothness in penalized spline regression

Typically, an optimal smoothing parameter in a penalized spline regression is determined by minimizing an information criterion, such as one of the C _p , CV and GCV criteria. Since an explicit solution to the minimization problem for an information criterion cannot be obtained, it is necessary to carry out an iterative procedure to search for the optimal smoothing parameter. In order to avoid such extra calculation, a non-iterative optimization method for smoothness in penalized spline regression is proposed using the formulation of generalized ridge regression. By conducting numerical simulations, we verify that our method has better performance than other methods which optimize the number of basis functions and the single smoothing parameter by means of the CV or GCV criteria.     

Making robust the cross-validatory choice of smoothing parameter in spline smoothing regression

Cross-validation as a means of choosing the smoothing parameter in spline regression has achieved a wide popularity. Its appeal comprises of an automatic method based on an attractive criterion and along with many other methods it has been shown to minimize predictive mean square error asymptotically. However, in practice there may be a substantial proportion of applications where a cross-validation style choice may lead to drastic undersmoothing often as far as interpolation. Furthermore, because the criterion is so appealing the user may be misled by an inappropriate, automatically-chosen value. In this paper we investigate the nature of cross-validatory methods in spline smoothing regression and suggest variants which provide small sample protection against undersmoothing.     

Thin plate regression splines

Summary. I discuss the production of low rank smoothers for d  ≥ 1 dimensional data, which can be fitted by regression or penalized regression methods. The smoothers are constructed by a simple transformation and truncation of the basis that arises from the solution of the thin plate spline smoothing problem and are optimal in the sense that the truncation is designed to result in the minimum possible perturbation of the thin plate spline smoothing problem given the dimension of the basis used to construct the smoother. By making use of Lanczos iteration the basis change and truncation are computationally efficient. The smoothers allow the use of approximate thin plate spline models with large data sets, avoid the problems that are associated with 'knot placement' that usually complicate modelling with regression splines or penalized regression splines, provide a sensible way of modelling interaction terms in generalized additive models, provide low rank approximations to generalized smoothing spline models, appropriate for use with large data sets, provide a means for incorporating smooth functions of more than one variable into non-linear models and improve the computational efficiency of penalized likelihood models incorporating thin plate splines. Given that the approach produces spline-like models with a sparse basis, it also provides a natural way of incorporating unpenalized spline-like terms in linear and generalized linear models, and these can be treated just like any other model terms from the point of view of model selection, inference and diagnostics.     

ON CONFIDENCE INTERVALS FOR GENERALIZED ADDITIVE MODELS BASED ON PENALIZED REGRESSION SPLINES

Generalized additive models represented using low rank penalized regression splines, estimated by penalized likelihood maximisation and with smoothness selected by generalized cross validation or similar criteria, provide a computationally efficient general framework for practical smooth modelling. Various authors have proposed approximate Bayesian interval estimates for such models, based on extensions of the work of Wahba, G. (1983) [Bayesian confidence intervals for the cross validated smoothing spline. J. R. Statist. Soc. B 45 , 133–150] and Silverman, B.W. (1985) [Some aspects of the spline smoothing approach to nonparametric regression curve fitting. J. R. Statist. Soc. B 47 , 1–52] on smoothing spline models of Gaussian data, but testing of such intervals has been rather limited and there is little supporting theory for the approximations used in the generalized case. This paper aims to improve this situation by providing simulation tests and obtaining asymptotic results supporting the approximations employed for the generalized case. The simulation results suggest that while across‐the‐model performance is good, component‐wise coverage probabilities are not as reliable. Since this is likely to result from the neglect of smoothing parameter variability, a simple and efficient simulation method is proposed to account for smoothing parameter uncertainty: this is demonstrated to substantially improve the performance of component‐wise intervals.     

Comparison of spline estimator at various levels of autocorrelation in smoothing spline non parametric regression for longitudinal data

The purpose of this research are: (1) to obtain spline function estimation in non parametric regression for longitudinal data with and without considering the autocorrelation between data of observation within subject, (2) to develop the algorithm that generates simulation data with certain autocorrelation level based on size of sample (N) and error variance (EV), and (3) to establish shape of spline estimator in non parametric regression for longitudinal data to simulation with various level of autocorrelation, as well as compare DM and TM approaches in predicting spline estimator in the data simulation with different of autocorrelation observational data on within subject. The results of the application are as follows: (a) implementation of smoothing spline with penalized weighted least square (PWLS) approach with or without consideration of autocorrelation in general (in all sizes and all error variances levels) provides significantly different spline estimator when the autocorrelation level >0.8; (b) based on size comparison, spline estimator in non parametric regression smoothing spline with PLS approach with (DM), or without (DM) consideration of autocorrelation showed significantly different result in level of autocorrelation > 0.8 (in overall size, moderate and large sample size), and > 0.7 (in small sample size); (c) based on level of variance, spline estimator in non parametric regression smoothing spline with PLS approach with (DM), or without (DM) consideration of autocorrelation showed significantly different result in level of autocorrelation > 0.8 (in overall level of variance, moderate and large variance), and > 0.7 (in small variance).     

Spline smoothing over difficult regions

Summary. It is occasionally necessary to smooth data over domains in R ² with complex irregular boundaries or interior holes. Traditional methods of smoothing which rely on the Euclidean metric or which measure smoothness over the entire real plane may then be inappropriate. This paper introduces a bivariate spline smoothing function defined as the minimizer of a penalized sum-of-squares functional. The roughness penalty is based on a partial differential operator and is integrated only over the problem domain by using finite element analysis. The method is motivated by and applied to two sample smoothing problems and is compared with the thin plate spline.     

Kernel spline regression

The authors propose «kernel spline regression,» a method of combining spline regression and kernel smoothing by replacing the polynomial approximation for local polynomial kernel regression with the spline basis. The new approach retains the local weighting scheme and the use of a bandwidth to control the size of local neighborhood. The authors compute the bias and variance of the kernel linear spline estimator, which they compare with local linear regression. They show that kernel spline estimators can succeed in capturing the main features of the underlying curve more effectively than local polynomial regression when the curvature changes rapidly. They also show through simulation that kernel spline regression often performs better than ordinary spline regression and local polynomial regression.     

Robust estimation for finite populations based on a working model

A common scenario in finite population inference is that it is possible to find a working superpopulation model which explains the main features of the population but which may not capture all the fine details. In addition, there are often outliers in the population which do not follow the assumed superpopulation model. In situations like these, it is still advantageous to make use of the working model to estimate finite population quantities, provided that we do it in a robust manner. The approach that we suggest is first to fit the working model to the sample and then to fine-tune for departures from the model assumed by estimating the conditional distribution of the residuals as a function of the auxiliary variable. This is a more direct approach to handling outliers and model misspecification than the Huber approach that is currently being used. Two simple methods, stratification and nearest neighbour smoothing, are used to estimate the conditional distributions of the residuals, which result in two modifications to the standard model-based estimator of the population distribution function. The estimators suggested perform very well in simulation studies involving two types of model departure and have small variances due to their model-based construction as well as acceptable bias. The potential advantage of the proposed robustified model-based approach over direct nonparametric regression is also demonstrated.     

Variance Reduction in Smoothing Splines

Abstract.  We develop a variance reduction method for smoothing splines. For a given point of estimation, we define a variance-reduced spline estimate as a linear combination of classical spline estimates at three nearby points. We first develop a variance reduction method for spline estimators in univariate regression models. We then develop an analogous variance reduction method for spline estimators in clustered/longitudinal models. Simulation studies are performed which demonstrate the efficacy of our variance reduction methods in finite sample settings. Finally, a real data analysis with the motorcycle data set is performed. Here we consider variance estimation and generate 95% pointwise confidence intervals for the unknown regression function.