A note on smoothing parameter selection for penalized spline smoothing

In nonparametric regression the smoothing parameter can be selected by minimizing a Mean Squared Error (MSE) based criterion. For spline smoothing one can also rewrite the smooth estimation as a Linear Mixed Model where the smoothing parameter appears as the a priori variance of spline basis coefficients. This allows to employ Maximum Likelihood (ML) theory to estimate the smoothing parameter as variance component. In this paper the relation between the two approaches is illuminated for penalized spline smoothing (P-spline) as suggested in Eilers and Marx Statist. Sci. 11(2) (1996) 89. Theoretical and empirical arguments are given showing that the ML approach is biased towards undersmoothing, i.e. it chooses a too complex model compared to the MSE. The result is in line with classical spline smoothing, even though the asymptotic arguments are different. This is because in P-spline smoothing a finite dimensional basis is employed while in classical spline smoothing the basis grows with the sample size.     

A penalized spline estimator for fixed effects panel data models

Estimating nonlinear effects of continuous covariates by penalized splines is well established for regressions with cross-sectional data as well as for panel data regressions with random effects. Penalized splines are particularly advantageous since they enable both the estimation of unknown nonlinear covariate effects and inferential statements about these effects. The latter are based, for example, on simultaneous confidence bands that provide a simultaneous uncertainty assessment for the whole estimated functions. In this paper, we consider fixed effects panel data models instead of random effects specifications and develop a first-difference approach for the inclusion of penalized splines in this case. We take the resulting dependence structure into account and adapt the construction of simultaneous confidence bands accordingly. In addition, the penalized spline estimates as well as the confidence bands are also made available for derivatives of the estimated effects which are of considerable interest in many application areas. As an empirical illustration, we analyze the dynamics of life satisfaction over the life span based on data from the German Socio-Economic Panel. An open-source software implementation of our methods is available in the R package pamfe.     

A non-iterative posterior sampling algorithm for Laplace linear regression model

In this article, a non-iterative sampling algorithm is developed to obtain an independently and identically distributed samples approximately from the posterior distribution of parameters in Laplace linear regression model. By combining the inverse Bayes formulae, sampling/importance resampling, and expectation maximum algorithm, the algorithm eliminates the diagnosis of convergence in the iterative Gibbs sampling and the samples generated from it can be used for inferences immediately. Simulations are conducted to illustrate the robustness and effectiveness of the algorithm. Finally, real data are studied to show the usefulness of the proposed methodology.     

A non-iterative posterior sampling algorithm for linear quantile regression model

In this article, a non-iterative posterior sampling algorithm for linear quantile regression model based on the asymmetric Laplace distribution is proposed. The algorithm combines the inverse Bayes formulae, sampling/importance resampling, and the expectation maximization algorithm to obtain independently and identically distributed samples approximately from the observed posterior distribution, which eliminates the convergence problems in the iterative Gibbs sampling and overcomes the difficulty in evaluating the standard deviance in the EM algorithm. The numeric results in simulations and application to the classical Engel data show that the non-iterative sampling algorithm is more effective than the Gibbs sampling and EM algorithm.     

A model-averaging approach for smoothing spline regression

ABSTRACT

This article considers nonparametric regression problems and develops a model-averaging procedure for smoothing spline regression problems. Unlike most smoothing parameter selection studies determining an optimum smoothing parameter, our focus here is on the prediction accuracy for the true conditional mean of Y given a predictor X. Our method consists of two steps. The first step is to construct a class of smoothing spline regression models based on nonparametric bootstrap samples, each with an appropriate smoothing parameter. The second step is to average bootstrap smoothing spline estimates of different smoothness to form a final improved estimate. To minimize the prediction error, we estimate the model weights using a delete-one-out cross-validation procedure. A simulation study has been performed by using a program written in R. The simulation study provides a comparison of the most well known cross-validation (CV), generalized cross-validation (GCV), and the proposed method. This new method is straightforward to implement, and gives reliable performances in simulations.     

A non-iterative Bayesian sampling algorithm for censored Student-t linear regression models

ABSTRACT

In this paper, we consider an effective Bayesian inference for censored Student-t linear regression model, which is a robust alternative to the usual censored Normal linear regression model. Based on the mixture representation of the Student-t distribution, we propose a non-iterative Bayesian sampling procedure to obtain independently and identically distributed samples approximately from the observed posterior distributions, which is different from the iterative Markov Chain Monte Carlo algorithm. We conduct model selection and influential analysis using the posterior samples to choose the best fitted model and to detect latent outliers. We illustrate the performance of the procedure through simulation studies, and finally, we apply the procedure to two real data sets, one is the insulation life data with right censoring and the other is the wage rates data with left censoring, and we get some interesting results.     

A coordinate descent algorithm for computing penalized smooth quantile regression

The computation of penalized quantile regression estimates is often computationally intensive in high dimensions. In this paper we propose a coordinate descent algorithm for computing the penalized smooth quantile regression (cdaSQR) with convex and nonconvex penalties. The cdaSQR approach is based on the approximation of the objective check function, which is not differentiable at zero, by a modified check function which is differentiable at zero. Then, using the maximization-minimization trick of the gcdnet algorithm (Yang and Zou in, J Comput Graph Stat 22(2):396–415, 2013), we update each coefficient simply and efficiently. In our implementation, we consider the convex penalties \(\ell _1+\ell _2\) and the nonconvex penalties SCAD (or MCP) \(+ \ell _2\). We establishe the convergence property of the csdSQR with \(\ell _1+\ell _2\) penalty. The numerical results show that our implementation is an order of magnitude faster than its competitors. Using simulations we compare the speed of our algorithm to its competitors. Finally, the performance of our algorithm is illustrated on three real data sets from diabetes, leukemia and Bardet–Bidel syndrome gene expression studies.     

A note on penalized minimum distance estimation in nonparametric regression

The authors introduce a penalized minimum distance regression estimator. They show the estimator to balance, among a sequence of nested models of increasing complexity, the L¹ ‐approximation error of each model class and a penalty term which reflects the richness of each model and serves as a upper bound for the estimation error.     

Adaptive penalized quantile regression for high dimensional data

We propose a new adaptive L₁ penalized quantile regression estimator for high-dimensional sparse regression models with heterogeneous error sequences. We show that under weaker conditions compared with alternative procedures, the adaptive L₁ quantile regression selects the true underlying model with probability converging to one, and the unique estimates of nonzero coefficients it provides have the same asymptotic normal distribution as the quantile estimator which uses only the covariates with non-zero impact on the response. Thus, the adaptive L₁ quantile regression enjoys oracle properties. We propose a completely data driven choice of the penalty level _λn${\lambda }_n$, which ensures good performance of the adaptive L₁ quantile regression. Extensive Monte Carlo simulation studies have been conducted to demonstrate the finite sample performance of the proposed method.     

Wavelet kernel penalized estimation for non-equispaced design regression

The paper considers regression problems with univariate design points. The design points are irregular and no assumptions on their distribution are imposed. The regression function is retrieved by a wavelet based reproducing kernel Hilbert space (RKHS) technique with the penalty equal to the sum of blockwise RKHS norms. In order to simplify numerical optimization, the problem is replaced by an equivalent quadratic minimization problem with an additional penalty term. The computational algorithm is described in detail and is implemented with both the sets of simulated and real data. Comparison with existing methods showed that the technique suggested in the paper does not oversmooth the function and is superior in terms of the mean squared error. It is also demonstrated that under additional assumptions on design points the method achieves asymptotic optimality in a wide range of Besov spaces.     

Sparse optimization for nonconvex group penalized estimation

We consider a linear regression model where there are group structures in covariates. The group LASSO has been proposed for group variable selections. Many nonconvex penalties such as smoothly clipped absolute deviation and minimax concave penalty were extended to group variable selection problems. The group coordinate descent (GCD) algorithm is used popularly for fitting these models. However, the GCD algorithms are hard to be applied to nonconvex group penalties due to computational complexity unless the design matrix is orthogonal. In this paper, we propose an efficient optimization algorithm for nonconvex group penalties by combining the concave convex procedure and the group LASSO algorithm. We also extend the proposed algorithm for generalized linear models. We evaluate numerical efficiency of the proposed algorithm compared to existing GCD algorithms through simulated data and real data sets.     

A lower bound based smoothed quasi-Newton algorithm for group bridge penalized regression

In this paper, we propose a lower bound based smoothed quasi-Newton algorithm for computing the solution paths of the group bridge estimator in linear regression models. Our method is based on the quasi-Newton algorithm with a smoothed group bridge penalty in combination with a novel data-driven thresholding rule for the regression coefficients. This rule is derived based on a necessary KKT condition of the group bridge optimization problem. It is easy to implement and can be used to eliminate groups with zero coefficients. Thus, it reduces the dimension of the optimization problem. The proposed algorithm removes the restriction of groupwise orthogonal condition needed in coordinate descent and LARS algorithms for group variable selection. Numerical results show that the proposed algorithm outperforms the coordinate descent based algorithms in both efficiency and accuracy.     

Copula regression spline models for binary outcomes

We introduce a framework for estimating the effect that a binary treatment has on a binary outcome in the presence of unobserved confounding. The methodology is applied to a case study which uses data from the Medical Expenditure Panel Survey and whose aim is to estimate the effect of private health insurance on health care utilization. Unobserved confounding arises when variables which are associated with both treatment and outcome are not available (in economics this issue is known as endogeneity). Also, treatment and outcome may exhibit a dependence which cannot be modeled using a linear measure of association, and observed confounders may have a non-linear impact on the treatment and outcome variables. The problem of unobserved confounding is addressed using a two-equation structural latent variable framework, where one equation essentially describes a binary outcome as a function of a binary treatment whereas the other equation determines whether the treatment is received. Non-linear dependence between treatment and outcome is dealt using copula functions, whereas covariate-response relationships are flexibly modeled using a spline approach. Related model fitting and inferential procedures are developed, and asymptotic arguments presented.     

Model selection in spline nonparametric regression

A Bayesian approach is presented for model selection in nonparametric regression with Gaussian errors and in binary nonparametric regression. A smoothness prior is assumed for each component of the model and the posterior probabilities of the candidate models are approximated using the Bayesian information criterion. We study the model selection method by simulation and show that it has excellent frequentist properties and gives improved estimates of the regression surface. All the computations are carried out efficiently using the Gibbs sampler.     

A plug-in the number of knots selector for polynomial spline regression

A plug-in the number of interior knots (NIKs) selector is proposed for polynomial spline estimation in nonparametric regression. The existence and properties of the optimal NIKs for spline regression are established by minimising the weighted mean integrated squared error. We obtain plug-in formulae for the optimal NIKs based on the theoretical results of asymptotic optimality, and develop strategies for choosing the NIKs of the spline estimator. The proposed NIKs selection method is tested on our simulated data with quite satisfactory performance, and is illustrated by analysing a fossil data set.     

Variable selection in spatial regression via penalized least squares

We consider variable selection in linear regression of geostatistical data that arise often in environmental and ecological studies. A penalized least squares procedure is studied for simultaneous variable selection and parameter estimation. Various penalty functions are considered including smoothly clipped absolute deviation. Asymptotic properties of penalized least squares estimates, particularly the oracle properties, are established, under suitable regularity conditions imposed on a random field model for the error process. Moreover, computationally feasible algorithms are proposed for estimating regression coefficients and their standard errors. Finite‐sample properties of the proposed methods are investigated in a simulation study and comparison is made among different penalty functions. The methods are illustrated by an ecological dataset of landcover in Wisconsin. The Canadian Journal of Statistics 37: 607–624; 2009 © 2009 Statistical Society of Canada     

A note on all-bias designs with applications in spline regression models

If a model is fitted to empirical data, bias can arise from terms which are not incorporated in the model assumptions. As a consequence the commonly used optimality criteria based on the generalized variance of the estimator of the model parameters may not lead to efficient designs for the statistical analysis. In this note some general aspects of all-bias designs are presented, which were introduced in this context by Box and Draper (1959). Using an interesting correspondence between the points of all-bias designs and the knots of quadrature formulas we establish sufficient conditions such that a given design is an all-bias design. The results are illustrated in the special case of spline regression models. In particular our results generalize recent findings of Woods and Lewis (2006).     

A non-iterative sampling Bayesian method for linear mixed models with normal independent distributions

In this paper, we utilize normal/independent (NI) distributions as a tool for robust modeling of linear mixed models (LMM) under a Bayesian paradigm. The purpose is to develop a non-iterative sampling method to obtain i.i.d. samples approximately from the observed posterior distribution by combining the inverse Bayes formulae, sampling/importance resampling and posterior mode estimates from the expectation maximization algorithm to LMMs with NI distributions, as suggested by Tan et al. [33 Tan, M., Tian, G. and Ng, K. 2003. A noniterative sampling method for computing posteriors in the structure of EM-type algorithms. Statist. Sinica, 13(3): 625–640. [Web of Science ®] , [Google Scholar]]. The proposed algorithm provides a novel alternative to perfect sampling and eliminates the convergence problems of Markov chain Monte Carlo methods. In order to examine the robust aspects of the NI class, against outlying and influential observations, we present a Bayesian case deletion influence diagnostics based on the Kullback–Leibler divergence. Further, some discussions on model selection criteria are given. The new methodologies are exemplified through a real data set, illustrating the usefulness of the proposed methodology.     

On algorithms for ordinary least squares regression spline fitting: A comparative study

Regression spline smoothing is a popular approach for conducting nonparametric regression. An important issue associated with it is the choice of a "theoretically best" set of knots. Different statistical model selection methods, such as Akaike's information criterion and generalized cross-validation, have been applied to derive different "theoretically best" sets of knots. Typically these best knot sets are defined implicitly as the optimizers of some objective functions. Hence another equally important issue concerning regression spline smoothing is how to optimize such objective functions. In this article different numerical algorithms that are designed for carrying out such optimization problems are compared by means of a simulation study. Both the univariate and bivariate smoothing settings will be considered. Based on the simulation results, recommendations for choosing a suitable optimization algorithm under various settings will be provided.