Tuning Parameter Estimation in Penalized Least Squares Methodology

The efficiency of the penalized methods (Fan and Li, 2001 Fan , J. , Li , R. ( 2001 ). Variable selection via nonconcave penalized likelihood and its oracle properties . Journal of the American Statistical Association 96 : 1348 – 1360 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) depends strongly on a tuning parameter due to the fact that it controls the extent of penalization. Therefore, it is important to select it appropriately. In general, tuning parameters are chosen by data-driven approaches, such as the commonly used generalized cross validation. In this article, we propose an alternative method for the derivation of the tuning parameter selector in penalized least squares framework, which can lead to an ameliorated estimate. Simulation studies are presented to support theoretical findings and a comparison of the Type I and Type II error rates, considering the L ₁, the hard thresholding and the Smoothly Clipped Absolute Deviation penalty functions, is performed. The results are given in tables and discussion follows.     

Tuning Parameter Selection in Penalized Frailty Models

The penalized likelihood approach of Fan and Li (2001 Fan, J., Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Association 96:1348–1360.[Taylor &; Francis Online], [Web of Science ®] , [Google Scholar], 2002 Fan, J., Li, R. (2002). Variable selection for Cox’s proportional hazards model and frailty model. The Annals of Statistics 30:74–99.[Crossref], [Web of Science ®] , [Google Scholar]) differs from the traditional variable selection procedures in that it deletes the non-significant variables by estimating their coefficients as zero. Nevertheless, the desirable performance of this shrinkage methodology relies heavily on an appropriate selection of the tuning parameter which is involved in the penalty functions. In this work, new estimates of the norm of the error are firstly proposed through the use of Kantorovich inequalities and, subsequently, applied to the frailty models framework. These estimates are used in order to derive a tuning parameter selection procedure for penalized frailty models and clustered data. In contrast with the standard methods, the proposed approach does not depend on resampling and therefore results in a considerable gain in computational time. Moreover, it produces improved results. Simulation studies are presented to support theoretical findings and two real medical data sets are analyzed.     

Smoothing Spline Estimation for Partially Linear Single-index Models

In this article, the partially linear single-index models are discussed based on smoothing spline and average derivative estimation method. This proposed technique consists of two stages: one is to estimate the vector parameter in the linear part using the smoothing cubic spline method, simultaneously, obtaining the estimator of unknown single-index function; the other is to estimate the single-index coefficients in the single-index part by the using average derivative estimator procedure. Some simulated and real examples are presented to illustrate the performance of this method.     

Schwarz Method for Penalized Quasi-Likelihood in Generalized Additive Models

The topic is penalized quasi-maximum likelihood estimation in generalized additive models by an approximation using a sequence of sub-models, here called blocks. The Schwarz method uses a sequence of sub-models, The technique might be useful to model comparison, where the fitted values from a sub-model are used as starting values for a larger model.

We show that the algorithm method converges in canonical link of generalized additive models, and a theorem about bound condition of sub-models convergence with uncanonical link.     

广义部分线性单指数模型的惩罚样条估计

 本文讨论了指数族广义部分线性单指数模型(Generalized Partially Linear Single Index Models, GPLSIM) 的惩罚样条迭代估计,提出了基于惩罚似然和一组预先取定的单指数参数向量 的初始估计的迭代估计算法。另外本文还通过一组模拟数据的分析对所提出的迭代算法进行了验证。     

A simple approach for varying-coefficient model selection

In varying-coefficient models, an important question is to determine whether some of the varying coefficients are actually invariant coefficients. This article proposes a penalized likelihood method in the framework of the smoothing spline ANOVA models, with a penalty designed toward the goal of automatically distinguishing varying coefficients and those which are not varying. Unlike the stepwise procedure, the method simultaneously quantifies and estimates the coefficients. An efficient algorithm is given and ways of choosing the smoothing parameters are discussed. Simulation results and an analysis on the Boston housing data illustrate the usefulness of the method. The proposed approach is further extended to longitudinal data analysis.     

Robust Model-Based Inference for Incomplete Data via Penalized Spline Propensity Prediction

Parametric model-based regression imputation is commonly applied to missing-data problems, but is sensitive to misspecification of the imputation model. Little and An (2004 Little , R. J. A. , An , H. ( 2004 ). Robust likelihood-based analysis of multivariate data with missing values . Statistica Sinica 14 : 949 – 968 .[Web of Science ®] , [Google Scholar]) proposed a semiparametric approach called penalized spline propensity prediction (PSPP), where the variable with missing values is modeled by a penalized spline (P-Spline) of the response propensity score, which is logit of the estimated probability of being missing given the observed variables. Variables other than the response propensity are included parametrically in the imputation model. However they only considered point estimation based on single imputation with PSPP. We consider here three approaches to standard errors estimation incorporating the uncertainty due to non response: (a) standard errors based on the asymptotic variance of the PSPP estimator, ignoring sampling error in estimating the response propensity; (b) standard errors based on the bootstrap method; and (c) multiple imputation-based standard errors using draws from the joint posterior predictive distribution of missing values under the PSPP model. Simulation studies suggest that the bootstrap and multiple imputation approaches yield good inferences under a range of simulation conditions, with multiple imputation showing some evidence of closer to nominal confidence interval coverage when the sample size is small.     

Finite Mixture of Generalized Semiparametric Models: Variable Selection via Penalized Estimation

Selection of the important variables is one of the most important model selection problems in statistical applications. In this article, we address variable selection in finite mixture of generalized semiparametric models. To overcome computational burden, we introduce a class of variable selection procedures for finite mixture of generalized semiparametric models using penalized approach for variable selection. Estimation of nonparametric component will be done via multivariate kernel regression. It is shown that the new method is consistent for variable selection and the performance of proposed method will be assessed via simulation.     

A model-averaging approach for smoothing spline regression

ABSTRACT

This article considers nonparametric regression problems and develops a model-averaging procedure for smoothing spline regression problems. Unlike most smoothing parameter selection studies determining an optimum smoothing parameter, our focus here is on the prediction accuracy for the true conditional mean of Y given a predictor X. Our method consists of two steps. The first step is to construct a class of smoothing spline regression models based on nonparametric bootstrap samples, each with an appropriate smoothing parameter. The second step is to average bootstrap smoothing spline estimates of different smoothness to form a final improved estimate. To minimize the prediction error, we estimate the model weights using a delete-one-out cross-validation procedure. A simulation study has been performed by using a program written in R. The simulation study provides a comparison of the most well known cross-validation (CV), generalized cross-validation (GCV), and the proposed method. This new method is straightforward to implement, and gives reliable performances in simulations.     

Polynomial Spline Estimation for a Generalized Additive Coefficient Model

Abstract.  We study a semiparametric generalized additive coefficient model (GACM), in which linear predictors in the conventional generalized linear models are generalized to unknown functions depending on certain covariates, and approximate the non-parametric functions by using polynomial spline. The asymptotic expansion with optimal rates of convergence for the estimators of the non-parametric part is established. Semiparametric generalized likelihood ratio test is also proposed to check if a non-parametric coefficient can be simplified as a parametric one. A conditional bootstrap version is suggested to approximate the distribution of the test under the null hypothesis. Extensive Monte Carlo simulation studies are conducted to examine the finite sample performance of the proposed methods. We further apply the proposed model and methods to a data set from a human visceral Leishmaniasis study conducted in Brazil from 1994 to 1997. Numerical results outperform the traditional generalized linear model and the proposed GACM is preferable.     

A Numerical Study of PQL Estimation Biases in Generalized Linear Mixed Models Under Heterogeneity of Random Effects

The penalized quasi-likelihood (PQL) approach is the most common estimation procedure for the generalized linear mixed model (GLMM). However, it has been noticed that the PQL tends to underestimate variance components as well as regression coefficients in the previous literature. In this article, we numerically show that the biases of variance component estimates by PQL are systematically related to the biases of regression coefficient estimates by PQL, and also show that the biases of variance component estimates by PQL increase as random effects become more heterogeneous.     

Automatic Smoothing for Poisson Regression

Abstract

Adaptive choice of smoothing parameters for nonparametric Poisson regression (O'Sullivan et al., 1986 O'Sullivan , F. , Yandell , B. S. , Raynor , W. J., Jr. ( 1986 ). Automatic smoothing of regression functions in generalized linear models . J. Amer. Statist. Assoc. 81 : 96 – 103 . [CSA] [Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) is considered in this article. A computable approximation of the unbiased risk estimate (AUBR) for Poisson regression is introduced. This approximation can be used to automatically tune the smoothing parameter for the penalized likelihood estimator. An alternative choice is the generalized approximate cross validation (GACV) proposed by Xiang and Wahba (1996 Xiang , D. , Wahba , G. ( 1996 ). A generalized approximate cross validation for smoothing splines with non-Gaussian data . Statist. Sinica 6 (3): 675 – 692 .[Web of Science ®] , [Google Scholar]). Although GACV enjoys a great success in practice when applying for nonparametric logisitic regression, its performance for Poisson regression is not clear. Numerical simulations have been conducted to evaluate the GACV and AUBR based tuning methods. We found that GACV has a tendency to oversmooth the data when the intensity function is small. As a consequence, we suggest tuning the smoothing parameter using AUBR in practice.     

半参数纵向模型的惩罚二次推断函数估计

针对纵向数据半参数模型E(y|x,t)=XTβ+f(t),采用惩罚二次推断函数方法同时估计模型中的回归参数β和未知光滑函数f(t)。首先利用截断幂函数基对未知光滑函数进行基函数展开近似,然后利用惩罚样条的思想构造关于回归参数和基函数系数的惩罚二次推断函数,最小化惩罚二次推断函数便可得到回归参数和基函数系数的惩罚二次推断函数估计。理论结果显示,估计结果具有相合性和渐近正态性,通过数值方法也得到了较好的模拟结果。     

Semiparametric Estimation with Profile Algorithm for Longitudinal Binary Data

This article considers analyzing longitudinal binary data semiparametrically and proposing GEE-Smoothing spline in the estimation of parametric and nonparametric components. The method is an extension of the parametric generalized estimating equation to semiparametric. The nonparametric component is estimated by smoothing spline approach, i.e., natural cubic spline. We use profile algorithm in the estimation of both parametric and nonparametric components. Properties of the estimators are evaluated by simulation.     

Performance of Robust GCV and Modified GCV for Spline Smoothing

Abstract. While it is a popular selection criterion for spline smoothing, generalized cross‐validation (GCV) occasionally yields severely undersmoothed estimates. Two extensions of GCV called robust GCV (RGCV) and modified GCV have been proposed as more stable criteria. Each involves a parameter that must be chosen, but the only guidance has come from simulation results. We investigate the performance of the criteria analytically. In most studies, the mean square prediction error is the only loss function considered. Here, we use both the prediction error and a stronger Sobolev norm error, which provides a better measure of the quality of the estimate. A geometric approach is used to analyse the superior small‐sample stability of RGCV compared to GCV. In addition, by deriving the asymptotic inefficiency for both the prediction error and the Sobolev error, we find intervals for the parameters of RGCV and modified GCV for which the criteria have optimal performance.     

Reference priors for linear models with general covariance structures

We develop a new class of reference priors for linear models with general covariance structures. A general Markov chain Monte Carlo algorithm is also proposed for implementing the computation. We present several examples to demonstrate the results: Bayesian penalized spline smoothing, a Bayesian approach to bivariate smoothing for a spatial model, and prior specification for structural equation models.     

ON CONFIDENCE INTERVALS FOR GENERALIZED ADDITIVE MODELS BASED ON PENALIZED REGRESSION SPLINES

Generalized additive models represented using low rank penalized regression splines, estimated by penalized likelihood maximisation and with smoothness selected by generalized cross validation or similar criteria, provide a computationally efficient general framework for practical smooth modelling. Various authors have proposed approximate Bayesian interval estimates for such models, based on extensions of the work of Wahba, G. (1983) [Bayesian confidence intervals for the cross validated smoothing spline. J. R. Statist. Soc. B 45 , 133–150] and Silverman, B.W. (1985) [Some aspects of the spline smoothing approach to nonparametric regression curve fitting. J. R. Statist. Soc. B 47 , 1–52] on smoothing spline models of Gaussian data, but testing of such intervals has been rather limited and there is little supporting theory for the approximations used in the generalized case. This paper aims to improve this situation by providing simulation tests and obtaining asymptotic results supporting the approximations employed for the generalized case. The simulation results suggest that while across‐the‐model performance is good, component‐wise coverage probabilities are not as reliable. Since this is likely to result from the neglect of smoothing parameter variability, a simple and efficient simulation method is proposed to account for smoothing parameter uncertainty: this is demonstrated to substantially improve the performance of component‐wise intervals.     

Parameter Estimation for Hidden Markov Models with Intractable Likelihoods

Approximate Bayesian computation (ABC) is a popular technique for analysing data for complex models where the likelihood function is intractable. It involves using simulation from the model to approximate the likelihood, with this approximate likelihood then being used to construct an approximate posterior. In this paper, we consider methods that estimate the parameters by maximizing the approximate likelihood used in ABC. We give a theoretical analysis of the asymptotic properties of the resulting estimator. In particular, we derive results analogous to those of consistency and asymptotic normality for standard maximum likelihood estimation. We also discuss how sequential Monte Carlo methods provide a natural method for implementing our likelihood‐based ABC procedures.     

Weak Convergence of the Regularization Path in Penalized M‐Estimation

Abstract. We consider a function defined as the pointwise minimization of a doubly index random process. We are interested in the weak convergence of the minimizer in the space of bounded functions. Such convergence results can be applied in the context of penalized M‐estimation, that is, when the random process to minimize is expressed as a goodness‐of‐fit term plus a penalty term multiplied by a penalty weight. This weight is called the regularization parameter and the minimizing function the regularization path. The regularization path can be seen as a collection of estimators indexed by the regularization parameter. We obtain a consistency result and a central limit theorem for the regularization path in a functional sense. Various examples are provided, including the ?¹‐regularization path for general linear models, the ?¹‐ or ?²‐regularization path of the least absolute deviation regression and the Akaike information criterion.     

Enhancing Estimation for Interest Rate Diffusion Models With Bond Prices

We consider improving estimating parameters of diffusion processes for interest rates by incorporating information in bond prices. This is designed to improve the estimation of the drift parameters, which are known to be subject to large estimation errors. It is shown that having the bond prices together with the short rates leads to more efficient estimation of all parameters for the interest rate models. It enhances the estimation efficiency of the maximum likelihood estimation based on the interest rate dynamics alone. The combined estimation based on the bond prices and the interest rate dynamics can also provide inference to the risk premium parameter. Simulation experiments were conducted to confirm the theoretical properties of the estimators concerned. We analyze the overnight Fed fund rates together with the U.S. Treasury bond prices. Supplementary materials for this article are available online.