Approximate regression models and splines

The literature pertaining to splines in regression analysis is reviewed. Spline regression is motivated as a simple extension of the basic polynomial regression model. Using this framework, the concepts of fixed and variable knot spline regression are developed and corresponding inferential procedures are considered. Smoothing splines are also seen to be an extension of polynomial regression and various optimality properties, as well as inferential and diagnostic methods, for these types of splines are discussed.     

Automatic Bayesian quantile regression curve fitting

Quantile regression, including median regression, as a more completed statistical model than mean regression, is now well known with its wide spread applications. Bayesian inference on quantile regression or Bayesian quantile regression has attracted much interest recently. Most of the existing researches in Bayesian quantile regression focus on parametric quantile regression, though there are discussions on different ways of modeling the model error by a parametric distribution named asymmetric Laplace distribution or by a nonparametric alternative named scale mixture asymmetric Laplace distribution. This paper discusses Bayesian inference for nonparametric quantile regression. This general approach fits quantile regression curves using piecewise polynomial functions with an unknown number of knots at unknown locations, all treated as parameters to be inferred through reversible jump Markov chain Monte Carlo (RJMCMC) of Green (Biometrika 82:711–732, 1995). Instead of drawing samples from the posterior, we use regression quantiles to create Markov chains for the estimation of the quantile curves. We also use approximate Bayesian factor in the inference. This method extends the work in automatic Bayesian mean curve fitting to quantile regression. Numerical results show that this Bayesian quantile smoothing technique is competitive with quantile regression/smoothing splines of He and Ng (Comput. Stat. 14:315–337, 1999) and P-splines (penalized splines) of Eilers and de Menezes (Bioinformatics 21(7):1146–1153, 2005).     

ON SEMIPARAMETRIC REGRESSION WITH O'SULLIVAN PENALIZED SPLINES

An exposition on the use of O'Sullivan penalized splines in contemporary semiparametric regression, including mixed model and Bayesian formulations, is presented. O'Sullivan penalized splines are similar to P-splines, but have the advantage of being a direct generalization of smoothing splines. Exact expressions for the O'Sullivan penalty matrix are obtained. Comparisons between the two types of splines reveal that O'Sullivan penalized splines more closely mimic the natural boundary behaviour of smoothing splines. Implementation in modern computing environments such as Matlab , r and bugs is discussed.     

Asymptotics of bivariate penalised splines

We study the class of bivariate penalised splines that use tensor product splines and a smoothness penalty. Similar to Claeskens, G., Krivobokova, T., and Opsomer, J.D. [(2009), ‘Asymptotic Properties of Penalised Spline Estimators’, Biometrika, 96(3), 529–544] for the univariate penalised splines, we show that, depending on the number of knots and penalty, the global asymptotic convergence rate of bivariate penalised splines is either similar to that of tensor product regression splines or to that of thin plate splines. In each scenario, the bivariate penalised splines are found rate optimal in the sense of Stone, C.J. [(12, 1982), ‘Optimal Global Rates of Convergence for Nonparametric Regression’, The Annals of Statistics, 10(4), 1040–1053] for a corresponding class of functions with appropriate smoothness. For the scenario where a small number of knots is used, we obtain expressions for the local asymptotic bias and variance and derive the point-wise and uniform asymptotic normality. The theoretical results are applicable to tensor product regression splines.     

Monotone fitting for developmental variables

In order to study developmental variables, for example, neuromotor development of children and adolescents, monotone fitting is typically needed. Most methods, to estimate a monotone regression function non-parametrically, however, are not straightforward to implement, a difficult issue being the choice of smoothing parameters. In this paper, a convenient implementation of the monotone B-spline estimates of Ramsay [Monotone regression splines in action (with discussion), Stat. Sci. 3 (1988), pp. 425–461] and Kelly and Rice [Montone smoothing with application to dose-response curves and the assessment of synergism, Biometrics 46 (1990), pp. 1071–1085] is proposed and applied to neuromotor data. Knots are selected adaptively using ideas found in Friedman and Silverman [Flexible parsimonous smoothing and additive modelling (with discussion), Technometrics 31 (1989), pp. 3–39] yielding a flexible algorithm to automatically and accurately estimate a monotone regression function. Using splines also simultaneously allows to include other aspects in the estimation problem, such as modeling a constant difference between two groups or a known jump in the regression function. Finally, an estimate which is not only monotone but also has a ‘levelling-off’ (i.e. becomes constant after some point) is derived. This is useful when the developmental variable is known to attain a maximum/minimum within the interval of observation.     

Quantile splines with several covariates

We extend univariate regression quantile splines to problems with several covariates. We adopt an ANOVA-type decomposition approach with main effects captured by linear splines and second-order ‘interactions’ modeled by bi-linear tensor-product splines. Both univariate linear splines and bi-linear tensor-product splines are optimal when fidelity to data are balanced by a roughness penalty on the fitted function. The problem of sub-model selection and asymptotic justification for using a smaller sub-space of the spline functions in the approximation are discussed. Two examples are considered to illustrate the empirical performance of the proposed methods.     

A Comparative Study of Semiparametric Estimation in Partially Linear Single-index Models

We consider a semiparametric method based on partial splines for estimating the unknown function and partially linear regression parameters in partially linear single-index models. Three methods—project pursuit regression (PPR), average derivative estimation (ADE), and a boosting method—are considered for estimating the single-index parameters. Simulations revealed that PPR with partial splines was superior in estimating single-index parameters, while the boosting method with partial splines performed no better than PPR and ADE. All three methods performed similarly in estimating the partially linear regression parameters. The relative performances of the methods are also illustrated using a real-world data example.     

BAYESIAN BETA REGRESSION: APPLICATIONS TO HOUSEHOLD EXPENDITURE DATA AND GENETIC DISTANCE BETWEEN FOOT-AND-MOUTH DISEASE VIRUSES

There is considerable interest in understanding how factors such as time and geographic distance between isolates might influence the evolutionary direction of foot‐and‐mouth disease. Genetic differences between viruses can be measured as the proportion of nucleotides that differ for a given sequence or gene. We present a Bayesian hierarchical regression model for the statistical analysis of continuous data with sample space restricted to the interval (0, 1). The data are modelled using beta distributions with means that depend on covariates through a link function. We discuss methodology for: (i) the incorporation of informative prior information into an analysis; (ii) fitting the model using Markov chain Monte Carlo sampling; (iii) model selection using Bayes factors; and (iv) semiparametric beta regression using penalized splines. The model was applied to two different datasets.     

Nonparametric M-regression with free knot splines

Many problems of practical interest can be formulated as the nonparametric estimation of a certain function such as a regression function, logistic or other generalized regression function, density function, conditional density function, hazard function, or conditional hazard function. Extended linear modeling provides a convenient theoretical framework for using polynomial splines and their selected tensor products in such function estimation problems and especially for obtaining rates of convergence of the resulting estimates in a unified manner. For a long time the theoretical results were restricted to fixed knot splines and to log-likelihood functions that were twice continuously differentiable. Recently, Stone and Huang extended the theory to handle free knot splines. In the present paper, the theory is further extended to handle contexts in which the log-likelihood function may not be differentiable. Specifically, we establish rates of convergence for estimation based on free knot splines in the context of nonparametric regression corresponding to M-estimates, which includes least absolute deviations (LAD) regression, quantile regression, and robust regression as special cases.     

Nonparametric regression estimators in complex surveys

In this article, we extend smoothing splines to model the regression mean structure when data are sampled through a complex survey. Smoothing splines are evaluated both with and without sample weights, and are compared with local linear estimator. Simulation studies find that nonparametric estimators perform better when sample weights are incorporated, rather than being treated as if iid. They also find that smoothing splines perform better than local linear estimator through completely data-driven bandwidth selection methods.     

Thin plate regression splines

Summary. I discuss the production of low rank smoothers for d  ≥ 1 dimensional data, which can be fitted by regression or penalized regression methods. The smoothers are constructed by a simple transformation and truncation of the basis that arises from the solution of the thin plate spline smoothing problem and are optimal in the sense that the truncation is designed to result in the minimum possible perturbation of the thin plate spline smoothing problem given the dimension of the basis used to construct the smoother. By making use of Lanczos iteration the basis change and truncation are computationally efficient. The smoothers allow the use of approximate thin plate spline models with large data sets, avoid the problems that are associated with 'knot placement' that usually complicate modelling with regression splines or penalized regression splines, provide a sensible way of modelling interaction terms in generalized additive models, provide low rank approximations to generalized smoothing spline models, appropriate for use with large data sets, provide a means for incorporating smooth functions of more than one variable into non-linear models and improve the computational efficiency of penalized likelihood models incorporating thin plate splines. Given that the approach produces spline-like models with a sparse basis, it also provides a natural way of incorporating unpenalized spline-like terms in linear and generalized linear models, and these can be treated just like any other model terms from the point of view of model selection, inference and diagnostics.     

Semiparametric inference for open populations using the Jolly–Seber model: a penalized spline approach

When there are frequent capture occasions, both semiparametric and nonparametric estimators for the size of an open population have been proposed using kernel smoothing methods. While kernel smoothing methods are mathematically tractable, fitting them to data is computationally intensive. Here, we use smoothing splines in the form of P-splines to provide an alternate less computationally intensive method of fitting these models to capture–recapture data from open populations with frequent capture occasions. We fit the model to capture data collected over 64 occasions and model the population size as a function of time, seasonal effects and an environmental covariate. A small simulation study is also conducted to examine the performance of the estimators and their standard errors.     

Inference in generalized additive mixed modelsby using smoothing splines

Generalized additive mixed models are proposed for overdispersed and correlated data, which arise frequently in studies involving clustered, hierarchical and spatial designs. This class of models allows flexible functional dependence of an outcome variable on covariates by using nonparametric regression, while accounting for correlation between observations by using random effects. We estimate nonparametric functions by using smoothing splines and jointly estimate smoothing parameters and variance components by using marginal quasi-likelihood. Because numerical integration is often required by maximizing the objective functions, double penalized quasi-likelihood is proposed to make approximate inference. Frequentist and Bayesian inferences are compared. A key feature of the method proposed is that it allows us to make systematic inference on all model components within a unified parametric mixed model framework and can be easily implemented by fitting a working generalized linear mixed model by using existing statistical software. A bias correction procedure is also proposed to improve the performance of double penalized quasi-likelihood for sparse data. We illustrate the method with an application to infectious disease data and we evaluate its performance through simulation.     

Smoothing spline Gaussian regression: more scalable computation via efficient approximation

Summary.  Smoothing splines via the penalized least squares method provide versatile and effective nonparametric models for regression with Gaussian responses. The computation of smoothing splines is generally of the order O ( n ³), n being the sample size, which severely limits its practical applicability. We study more scalable computation of smoothing spline regression via certain low dimensional approximations that are asymptotically as efficient. A simple algorithm is presented and the Bayes model that is associated with the approximations is derived, with the latter guiding the porting of Bayesian confidence intervals. The practical choice of the dimension of the approximating space is determined through simulation studies, and empirical comparisons of the approximations with the exact solution are presented. Also evaluated is a simple modification of the generalized cross-validation method for smoothing parameter selection, which to a large extent fixes the occasional undersmoothing problem that is suffered by generalized cross-validation.     

Testing the monotonicity or convexity of a function using regression splines

Methods for smoothed isotonic or convex regression are useful in many applications. Sometimes the shape assumptions constitute a priori knowledge about the regression function, but often the shape is part of the research question. The authors propose tests for monotonicity and convexity using constrained and unconstrained regression splines. The tests have good large‐sample properties and the small‐sample behaviour is illustrated through simulations. Extensions to the partial linear model and the generalized regression model are presented. The Canadian Journal of Statistics 39: 89–107; 2011 © 2011 Statistical Society of Canada     

A Stochastic Estimator of the Trace of the Influence Matrix for Laplacian Smoothing Splines

An unbiased stochastic estimator of tr(I–A), where A is the influence matrix associated with the calculation of Laplacian smoothing splines, is described. The estimator is similar to one recently developed by Girard but satisfies a minimum variance criterion and does not require the simulation of a standard normal variable. It uses instead simulations of the discrete random variable which takes the values 1, -1 each with probability 1/2. Bounds on the variance of the estimator, similar to those established by Girard, are obtained using elementary methods. The estimator can be used to approximately minimize generalised cross validation (GCV) when using discretized iterative methods for fitting Laplacian smoothing splines to very large data sets. Simulated examples show that the estimated trace values, using either the estimator presented here or the estimator of Girard, perform almost as well as the exact values when applied to the minimization of GCV for n as small as a few hundred, where n is the number of data points.     

Analysis of Deviance for Hypothesis Testing in Generalized Partially Linear Models

In this study, we develop nonparametric analysis of deviance tools for generalized partially linear models based on local polynomial fitting. Assuming a canonical link, we propose expressions for both local and global analysis of deviance, which admit an additivity property that reduces to analysis of variance decompositions in the Gaussian case. Chi-square tests based on integrated likelihood functions are proposed to formally test whether the nonparametric term is significant. Simulation results are shown to illustrate the proposed chi-square tests and to compare them with an existing procedure based on penalized splines. The methodology is applied to German Bundesbank Federal Reserve data.     

Confidence intervals for nonparametric quantile regression: an emphasis on smoothing splines approach

In this paper we consider the problem of constructing confidence intervals for nonparametric quantile regression with an emphasis on smoothing splines. The mean‐based approaches for smoothing splines of Wahba (1983) and Nychka (1988) may not be efficient for constructing confidence intervals for the underlying function when the observed data are non‐Gaussian distributed, for instance if they are skewed or heavy‐tailed. This paper proposes a method of constructing confidence intervals for the unknown τth quantile function (0<τ<1) based on smoothing splines. In this paper we investigate the extent to which the proposed estimator provides the desired coverage probability. In addition, an improvement based on a local smoothing parameter that provides more uniform pointwise coverage is developed. The results from numerical studies including a simulation study and real data analysis demonstrate the promising empirical properties of the proposed approach.     

ADDITIVE MODELS WITH PREDICTORS SUBJECT TO MEASUREMENT ERROR

This paper develops a likelihood‐based method for fitting additive models in the presence of measurement error. It formulates the additive model using the linear mixed model representation of penalized splines. In the presence of a structural measurement error model, the resulting likelihood involves intractable integrals, and a Monte Carlo expectation maximization strategy is developed for obtaining estimates. The method's performance is illustrated with a simulation study.     

Asymptotic design and estimation using linear splines

The estimation of a regression function g using linear splines is considered. The integrated mean square error is minimized using choice of estimator, allocation of observations and displacement of knots.