Principal components regression estimator of the parameters in partially linear models

ABSTRACT

As a compromise between parametric regression and non-parametric regression models, partially linear models are frequently used in statistical modelling. This paper is concerned with the estimation of partially linear regression model in the presence of multicollinearity. Based on the profile least-squares approach, we propose a novel principal components regression (PCR) estimator for the parametric component. When some additional linear restrictions on the parametric component are available, we construct a corresponding restricted PCR estimator. Some simulations are conducted to examine the performance of our proposed estimators and the results are satisfactory. Finally, a real data example is analysed.     

Linear or Nonlinear? Automatic Structure Discovery for Partially Linear Models

Partially linear models provide a useful class of tools for modeling complex data by naturally incorporating a combination of linear and nonlinear effects within one framework. One key question in partially linear models is the choice of model structure, that is, how to decide which covariates are linear and which are nonlinear. This is a fundamental, yet largely unsolved problem for partially linear models. In practice, one often assumes that the model structure is given or known and then makes estimation and inference based on that structure. Alternatively, there are two methods in common use for tackling the problem: hypotheses testing and visual screening based on the marginal fits. Both methods are quite useful in practice but have their drawbacks. First, it is difficult to construct a powerful procedure for testing multiple hypotheses of linear against nonlinear fits. Second, the screening procedure based on the scatterplots of individual covariate fits may provide an educated guess on the regression function form, but the procedure is ad hoc and lacks theoretical justifications. In this article, we propose a new approach to structure selection for partially linear models, called the LAND (Linear And Nonlinear Discoverer). The procedure is developed in an elegant mathematical framework and possesses desired theoretical and computational properties. Under certain regularity conditions, we show that the LAND estimator is able to identify the underlying true model structure correctly and at the same time estimate the multivariate regression function consistently. The convergence rate of the new estimator is established as well. We further propose an iterative algorithm to implement the procedure and illustrate its performance by simulated and real examples. Supplementary materials for this article are available online.     

Bayesian spectral analysis models for quantile regression with Dirichlet process mixtures

This paper presents a Bayesian analysis of partially linear additive models for quantile regression. We develop a semiparametric Bayesian approach to quantile regression models using a spectral representation of the nonparametric regression functions and the Dirichlet process (DP) mixture for error distribution. We also consider Bayesian variable selection procedures for both parametric and nonparametric components in a partially linear additive model structure based on the Bayesian shrinkage priors via a stochastic search algorithm. Based on the proposed Bayesian semiparametric additive quantile regression model referred to as BSAQ, the Bayesian inference is considered for estimation and model selection. For the posterior computation, we design a simple and efficient Gibbs sampler based on a location-scale mixture of exponential and normal distributions for an asymmetric Laplace distribution, which facilitates the commonly used collapsed Gibbs sampling algorithms for the DP mixture models. Additionally, we discuss the asymptotic property of the sempiparametric quantile regression model in terms of consistency of posterior distribution. Simulation studies and real data application examples illustrate the proposed method and compare it with Bayesian quantile regression methods in the literature.     

A Comparative Study of Semiparametric Estimation in Partially Linear Single-index Models

We consider a semiparametric method based on partial splines for estimating the unknown function and partially linear regression parameters in partially linear single-index models. Three methods—project pursuit regression (PPR), average derivative estimation (ADE), and a boosting method—are considered for estimating the single-index parameters. Simulations revealed that PPR with partial splines was superior in estimating single-index parameters, while the boosting method with partial splines performed no better than PPR and ADE. All three methods performed similarly in estimating the partially linear regression parameters. The relative performances of the methods are also illustrated using a real-world data example.     

Model selection and model averaging for semiparametric partially linear models with missing data

We study model selection and model averaging in semiparametric partially linear models with missing responses. An imputation method is used to estimate the linear regression coefficients and the nonparametric function. We show that the corresponding estimators of the linear regression coefficients are asymptotically normal. Then a focused information criterion and frequentist model average estimators are proposed and their theoretical properties are established. Simulation studies are performed to demonstrate the superiority of the proposed methods over the existing strategies in terms of mean squared error and coverage probability. Finally, the approach is applied to a real data case.     

Partially adaptive quantile estimators

This paper contrasts two approaches to estimating quantile regression models: traditional semi-parametric methods and partially adaptive estimators using flexible probability density functions (pdfs). While more general pdfs could have been used, the skewed Laplace was selected for pedagogical purposes. Monte Carlo simulations are used to compare the behavior of the semi-parametric and partially adaptive quantile estimators in the presence of possibly skewed and heteroskedastic data. Both approaches accommodate skewness and heteroskedasticity which are consistent with linear quantiles; however, the partially adaptive estimator considered allows for non linear quantiles and also provides simple tests for symmetry and heteroskedasticity. The methods are applied to the problem of estimating conditional quantile functions for wages corresponding to different levels of education.     

Semiparametric Regression Models with Applications to Scoring: A Review

Semiparametric regression models have been proposed in the econometric literature as a trade-off between the simple but easily implementable and interpretable parametric models and the flexible but structure free smoothing techniques. Some semiparametric models for binary response with possible application to scoring data are reviewed: single-index models, generalized partially linear models, generalized partially linear single-index models, and multiple-index models. All these models are extensions of the classical logistic regression.     

Automatic structure discovery for varying-coefficient partially linear models

Varying-coefficient partially linear models provide a useful tools for modeling of covariate effects on the response variable in regression. One key question in varying-coefficient partially linear models is the choice of model structure, that is, how to decide which covariates have linear effect and which have non linear effect. In this article, we propose a profile method for identifying the covariates with linear effect or non linear effect. Our proposed method is a penalized regression approach based on group minimax concave penalty. Under suitable conditions, we show that the proposed method can correctly determine which covariates have a linear effect and which do not with high probability. The convergence rate of the linear estimator is established as well as the asymptotical normality. The performance of the proposed method is evaluated through a simulation study which supports our theoretical results.     

Model averaging procedure for varying-coefficient partially linear models with missing responses

This paper is concerned with model averaging procedure for varying-coefficient partially linear models with missing responses. The profile least-squares estimation process and inverse probability weighted method are employed to estimate regression coefficients of the partially restricted models, in which the propensity score is estimated by the covariate balancing propensity score method. The estimators of the linear parameters are shown to be asymptotically normal. Then we develop the focused information criterion, formulate the frequentist model averaging estimators and construct the corresponding confidence intervals. Some simulation studies are conducted to examine the finite sample performance of the proposed methods. We find that the covariate balancing propensity score improves the performance of the inverse probability weighted estimator. We also demonstrate the superiority of the proposed model averaging estimators over those of existing strategies in terms of mean squared error and coverage probability. Finally, our approach is further applied to a real data example.     

Parameterization and inference for nonparametric regression problems

We consider local likelihood or local estimating equations, in which a multivariate function () is estimated but a derived function () of () is of interest. In many applications, when most naturally formulated the derived function is a non-linear function of (). In trying to understand whether the derived non-linear function is constant or linear, a problem arises with this approach: when the function is actually constant or linear, the expectation of the function estimate need not be constant or linear, at least to second order. In such circumstances, the simplest standard methods in nonparametric regression for testing whether a function is constant or linear cannot be applied. We develop a simple general solution which is applicable to nonparametric regression, varying-coefficient models, nonparametric generalized linear models, etc. We show that, in local linear kernel regression, inference about the derived function () is facilitated without a loss of power by reparameterization so that () is itself a component of (). Our approach is in contrast with the standard practice of choosing () for convenience and allowing ()> to be a non-linear function of (). The methods are applied to an important data set in nutritional epidemiology.     

Restricted minimax estimation in semiparametric linear models

Abstract

In this article we develop the minimax estimation approach of general linear models to the semiparametric linear models when the parameters are simultaneously constrained by an ellipsoid and linear restrictions. Combining sample information and prior constraints the minimax estimator is obtained and compared with partially least square estimator by theoretical and simulation methods.     

Diagnostics for repeated measurements in generalized linear mixed effects models

ABSTRACT

As there is an extensive body of research on diagnostics in regression models, various outlier detection methods have been developed. These methods have been extended to mixed effects models and generalized linear models, but there exist intrinsic drawbacks and limitations. This paper presents two-dimensional plots to identify discordant subjects and observations in generalized linear mixed effects models, displaying discordance in two directions. The sTudentized Residual Sum of Squares is not an extension of any regression tools but a new approach designed to efficiently reflect the characteristics of repeated measures. And this noteworthy clustering of outliers is identified in the plot. Applications to real-life examples are presented to illustrate the favorable/beneficial performance of the new tool.     

Fitting linear regression models to censored data by least squares and maximum likelihood methods

Several approaches have been suggested for fitting linear regression models to censored data. These include Cox's propor­tional hazard models based on quasi-likelihoods. Methods of fitting based on least squares and maximum likelihoods have also been proposed. The methods proposed so far all require special purpose optimization routines. We describe an approach here which requires only a modified standard least squares routine.

We present methods for fitting a linear regression model to censored data by least squares and method of maximum likelihood. In the least squares method, the censored values are replaced by their expectations, and the residual sum of squares is minimized. Several variants are suggested in the ways in which the expect­ation is calculated. A parametric (assuming a normal error model) and two non-parametric approaches are described. We also present a method for solving the maximum likelihood equations in the estimation of the regression parameters in the censored regression situation. It is shown that the solutions can be obtained by a recursive algorithm which needs only a least squares routine for optimization. The suggested procesures gain considerably in computational officiency. The Stanford Heart Transplant data is used to illustrate the various methods.     

The Dantzig Selector in Cox's Proportional Hazards Model

Abstract. The Dantzig selector (DS) is a recent approach of estimation in high‐dimensional linear regression models with a large number of explanatory variables and a relatively small number of observations. As in the least absolute shrinkage and selection operator (LASSO), this approach sets certain regression coefficients exactly to zero, thus performing variable selection. However, such a framework, contrary to the LASSO, has never been used in regression models for survival data with censoring. A key motivation of this article is to study the estimation problem for Cox's proportional hazards (PH) function regression models using a framework that extends the theory, the computational advantages and the optimal asymptotic rate properties of the DS to the class of Cox's PH under appropriate sparsity scenarios. We perform a detailed simulation study to compare our approach with other methods and illustrate it on a well‐known microarray gene expression data set for predicting survival from gene expressions.     

B-spline estimation for partially linear varying coefficient composite quantile regression models

Abstract

In this article, a new composite quantile regression estimation (CQR) approach is proposed for partially linear varying coefficient models (PLVCM) under composite quantile loss function with B-spline approximations. The major advantage of the proposed procedures over the existing ones is easy to implement using existing software, and it requires no specification of the error distributions. Under the regularity conditions, the consistency and asymptotic normality of the estimators are also derived. Finally, a simulation study and a real data application are undertaken to assess the finite sample performance of the proposed estimation procedure.     

ESTIMATION AND TESTING FOR PARTIALLY LINEAR SINGLE-INDEX MODELS

In partially linear single-index models, we obtain the semiparametrically efficient profile least-squares estimators of regression coefficients. We also employ the smoothly clipped absolute deviation penalty (SCAD) approach to simultaneously select variables and estimate regression coefficients. We show that the resulting SCAD estimators are consistent and possess the oracle property. Subsequently, we demonstrate that a proposed tuning parameter selector, BIC, identifies the true model consistently. Finally, we develop a linear hypothesis test for the parametric coefficients and a goodness-of-fit test for the nonparametric component, respectively. Monte Carlo studies are also presented.     

A class of partially linear single‐index survival models

The authors define a class of “partially linear single‐index” survival models that are more flexible than the classical proportional hazards regression models in their treatment of covariates. The latter enter the proposed model either via a parametric linear form or a nonparametric single‐index form. It is then possible to model both linear and functional effects of covariates on the logarithm of the hazard function and if necessary, to reduce the dimensionality of multiple covariates via the single‐index component. The partially linear hazards model and the single‐index hazards model are special cases of the proposed model. The authors develop a likelihood‐based inference to estimate the model components via an iterative algorithm. They establish an asymptotic distribution theory for the proposed estimators, examine their finite‐sample behaviour through simulation, and use a set of real data to illustrate their approach.     

Sample size planning for testing significance of curves

Smoothing methods for curve estimation have received considerable attention in statistics with a wide range of applications. However, to our knowledge, sample size planning for testing significance of curves has not been discussed in the literature. This paper focuses on sample size calculations for nonparametric regression and partially linear models based on local linear estimators. We describe explicit procedures for sample size calculations based on non- and semi-parametric F-tests. Data examples are provided to demonstrate the use of the procedures.     

Joint Mean-Covariance Models with Applications to Longitudinal Data in Partially Linear Model

Semiparametric regression models and estimating covariance functions are very useful in longitudinal study. Unfortunately, challenges arise in estimating the covariance function of longitudinal data collected at irregular time points. In this article, for mean term, a partially linear model is introduced and for covariance structure, a modified Cholesky decomposition approach is proposed to heed the positive-definiteness constraint. We estimate the regression function by using the local linear technique and propose quasi-likelihood estimating equations for both the mean and covariance structures. Moreover, asymptotic normality of the resulting estimators is established. Finally, simulation study and real data analysis are used to illustrate the proposed approach.     

WLAD-LASSO method for robust estimation and variable selection in partially linear models

This paper focuses on robust estimation and variable selection for partially linear models. We combine the weighted least absolute deviation (WLAD) regression with the adaptive least absolute shrinkage and selection operator (LASSO) to achieve simultaneous robust estimation and variable selection for partially linear models. Compared with the LAD-LASSO method, the WLAD-LASSO method will resist to the heavy-tailed errors and outliers in the parametric components. In addition, we estimate the unknown smooth function by a robust local linear regression. Under some regular conditions, the theoretical properties of the proposed estimators are established. We further examine finite-sample performance of the proposed procedure by simulation studies and a real data example.