Bayesian Semiparametric Modelling in Quantile Regression

Abstract. We propose a Bayesian semiparametric methodology for quantile regression modelling. In particular, working with parametric quantile regression functions, we develop Dirichlet process mixture models for the error distribution in an additive quantile regression formulation. The proposed non‐parametric prior probability models allow the shape of the error density to adapt to the data and thus provide more reliable predictive inference than models based on parametric error distributions. We consider extensions to quantile regression for data sets that include censored observations. Moreover, we employ dependent Dirichlet processes to develop quantile regression models that allow the error distribution to change non‐parametrically with the covariates. Posterior inference is implemented using Markov chain Monte Carlo methods. We assess and compare the performance of our models using both simulated and real data sets.     

Statistical Inference and Applications of Mixture of Varying Coefficient Models

In this paper, we consider a new mixture of varying coefficient models, in which each mixture component follows a varying coefficient model and the mixing proportions and dispersion parameters are also allowed to be unknown smooth functions. We systematically study the identifiability, estimation and inference for the new mixture model. The proposed new mixture model is rather general, encompassing many mixture models as its special cases such as mixtures of linear regression models, mixtures of generalized linear models, mixtures of partially linear models and mixtures of generalized additive models, some of which are new mixture models by themselves and have not been investigated before. The new mixture of varying coefficient model is shown to be identifiable under mild conditions. We develop a local likelihood procedure and a modified expectation–maximization algorithm for the estimation of the unknown non‐parametric functions. Asymptotic normality is established for the proposed estimator. A generalized likelihood ratio test is further developed for testing whether some of the unknown functions are constants. We derive the asymptotic distribution of the proposed generalized likelihood ratio test statistics and prove that the Wilks phenomenon holds. The proposed methodology is illustrated by Monte Carlo simulations and an analysis of a CO₂‐GDP data set.     

Bayesian spectral analysis models for quantile regression with Dirichlet process mixtures

This paper presents a Bayesian analysis of partially linear additive models for quantile regression. We develop a semiparametric Bayesian approach to quantile regression models using a spectral representation of the nonparametric regression functions and the Dirichlet process (DP) mixture for error distribution. We also consider Bayesian variable selection procedures for both parametric and nonparametric components in a partially linear additive model structure based on the Bayesian shrinkage priors via a stochastic search algorithm. Based on the proposed Bayesian semiparametric additive quantile regression model referred to as BSAQ, the Bayesian inference is considered for estimation and model selection. For the posterior computation, we design a simple and efficient Gibbs sampler based on a location-scale mixture of exponential and normal distributions for an asymmetric Laplace distribution, which facilitates the commonly used collapsed Gibbs sampling algorithms for the DP mixture models. Additionally, we discuss the asymptotic property of the sempiparametric quantile regression model in terms of consistency of posterior distribution. Simulation studies and real data application examples illustrate the proposed method and compare it with Bayesian quantile regression methods in the literature.     

Focused information criterion and model averaging based on weighted composite quantile regression

We study the focused information criterion and frequentist model averaging and their application to post‐model‐selection inference for weighted composite quantile regression (WCQR) in the context of the additive partial linear models. With the non‐parametric functions approximated by polynomial splines, we show that, under certain conditions, the asymptotic distribution of the frequentist model averaging WCQR‐estimator of a focused parameter is a non‐linear mixture of normal distributions. This asymptotic distribution is used to construct confidence intervals that achieve the nominal coverage probability. With properly chosen weights, the focused information criterion based WCQR estimators are not only robust to outliers and non‐normal residuals but also can achieve efficiency close to the maximum likelihood estimator, without assuming the true error distribution. Simulation studies and a real data analysis are used to illustrate the effectiveness of the proposed procedure.     

Partially Linear Additive Models with Unknown Link Functions

In this paper, we consider partially linear additive models with an unknown link function, which include single‐index models and additive models as special cases. We use polynomial spline method for estimating the unknown link function as well as the component functions in the additive part. We establish that convergence rates for all nonparametric functions are the same as in one‐dimensional nonparametric regression. For a faster rate of the parametric part, we need to define appropriate ‘projection’ that is more complicated than that defined previously for partially linear additive models. Compared to previous approaches, a distinct advantage of our estimation approach in implementation is that estimation directly reduces estimation in the single‐index model and can thus deal with much larger dimensional problems than previous approaches for additive models with unknown link functions. Simulations and a real dataset are used to illustrate the proposed model.     

Bayesian composite quantile regression for linear mixed-effects models

Longitudinal data are commonly modeled with the normal mixed-effects models. Most modeling methods are based on traditional mean regression, which results in non robust estimation when suffering extreme values or outliers. Median regression is also not a best choice to estimation especially for non normal errors. Compared to conventional modeling methods, composite quantile regression can provide robust estimation results even for non normal errors. In this paper, based on a so-called pseudo composite asymmetric Laplace distribution (PCALD), we develop a Bayesian treatment to composite quantile regression for mixed-effects models. Furthermore, with the location-scale mixture representation of the PCALD, we establish a Bayesian hierarchical model and achieve the posterior inference of all unknown parameters and latent variables using Markov Chain Monte Carlo (MCMC) method. Finally, this newly developed procedure is illustrated by some Monte Carlo simulations and a case analysis of HIV/AIDS clinical data set.     

Estimation on semi-functional linear errors-in-variables models

Abstract

Semi-functional linear regression models are important in practice. In this paper, their estimation is discussed when function-valued and real-valued random variables are all measured with additive error. By means of functional principal component analysis and kernel smoothing techniques, the estimators of the slope function and the non parametric component are obtained. To account for errors in variables, deconvolution is involved in the construction of a new class of kernel estimators. The convergence rates of the estimators of the unknown slope function and non parametric component are established under suitable norm and conditions. Simulation studies are conducted to illustrate the finite sample performance of our method.     

Automatic structure discovery for varying-coefficient partially linear models

Varying-coefficient partially linear models provide a useful tools for modeling of covariate effects on the response variable in regression. One key question in varying-coefficient partially linear models is the choice of model structure, that is, how to decide which covariates have linear effect and which have non linear effect. In this article, we propose a profile method for identifying the covariates with linear effect or non linear effect. Our proposed method is a penalized regression approach based on group minimax concave penalty. Under suitable conditions, we show that the proposed method can correctly determine which covariates have a linear effect and which do not with high probability. The convergence rate of the linear estimator is established as well as the asymptotical normality. The performance of the proposed method is evaluated through a simulation study which supports our theoretical results.     

Sparse additive models

Summary.  We present a new class of methods for high dimensional non-parametric regression and classification called sparse additive models. Our methods combine ideas from sparse linear modelling and additive non-parametric regression. We derive an algorithm for fitting the models that is practical and effective even when the number of covariates is larger than the sample size. Sparse additive models are essentially a functional version of the grouped lasso of Yuan and Lin. They are also closely related to the COSSO model of Lin and Zhang but decouple smoothing and sparsity, enabling the use of arbitrary non-parametric smoothers. We give an analysis of the theoretical properties of sparse additive models and present empirical results on synthetic and real data, showing that they can be effective in fitting sparse non-parametric models in high dimensional data.     

Mixtures of Linear Regression with Measurement Errors

Existing research on mixtures of regression models are limited to directly observed predictors. The estimation of mixtures of regression for measurement error data imposes challenges for statisticians. For linear regression models with measurement error data, the naive ordinary least squares method, which directly substitutes the observed surrogates for the unobserved error-prone variables, yields an inconsistent estimate for the regression coefficients. The same inconsistency also happens to the naive mixtures of regression estimate, which is based on the traditional maximum likelihood estimator and simply ignores the measurement error. To solve this inconsistency, we propose to use the deconvolution method to estimate the mixture likelihood of the observed surrogates. Then our proposed estimate is found by maximizing the estimated mixture likelihood. In addition, a generalized EM algorithm is also developed to find the estimate. The simulation results demonstrate that the proposed estimation procedures work well and perform much better than the naive estimates.     

Restricted Estimation in Partially Linear Errors-in-Variables Models

As a compromise between parametric regression and nonparametric regression, partially linear models are frequently used in statistical modelling. This article considers statistical inference for this semiparametric model when the linear covariate is measured with additive error and some additional linear restrictions on the parametric component are assumed to hold. We propose a restricted corrected profile least-squares estimator for the parametric component, and study the asymptotic normality of the estimator. To test hypothesis on the parametric component, we construct a Wald test statistic and obtain its limiting distribution. Some simulation studies are conducted to illustrate our approaches.     

Non parametric mixture of strictly monotone regression models

In this article, we propose the non parametric mixture of strictly monotone regression models. For implementation, a two-step procedure is derived. We further establish the asymptotic normality of the resultant estimator and demonstrate its good performance through numerical examples.     

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

Summary.  Structured additive regression models are perhaps the most commonly used class of models in statistical applications. It includes, among others, (generalized) linear models, (generalized) additive models, smoothing spline models, state space models, semiparametric regression, spatial and spatiotemporal models, log-Gaussian Cox processes and geostatistical and geoadditive models. We consider approximate Bayesian inference in a popular subset of structured additive regression models, latent Gaussian models , where the latent field is Gaussian, controlled by a few hyperparameters and with non-Gaussian response variables. The posterior marginals are not available in closed form owing to the non-Gaussian response variables. For such models, Markov chain Monte Carlo methods can be implemented, but they are not without problems, in terms of both convergence and computational time. In some practical applications, the extent of these problems is such that Markov chain Monte Carlo sampling is simply not an appropriate tool for routine analysis. We show that, by using an integrated nested Laplace approximation and its simplified version, we can directly compute very accurate approximations to the posterior marginals. The main benefit of these approximations is computational: where Markov chain Monte Carlo algorithms need hours or days to run, our approximations provide more precise estimates in seconds or minutes. Another advantage with our approach is its generality, which makes it possible to perform Bayesian analysis in an automatic, streamlined way, and to compute model comparison criteria and various predictive measures so that models can be compared and the model under study can be challenged.     

Weighted composite quantile regression for partially linear varying coefficient models

Partially linear varying coefficient models (PLVCMs) with heteroscedasticity are considered in this article. Based on composite quantile regression, we develop a weighted composite quantile regression (WCQR) to estimate the non parametric varying coefficient functions and the parametric regression coefficients. The WCQR is augmented using a data-driven weighting scheme. Moreover, the asymptotic normality of proposed estimators for both the parametric and non parametric parts are studied explicitly. In addition, by comparing the asymptotic relative efficiency theoretically and numerically, WCQR method all outperforms the CQR method and some other estimate methods. To achieve sparsity with high-dimensional covariates, we develop a variable selection procedure to select significant parametric components for the PLVCM and prove the method possessing the oracle property. Both simulations and data analysis are conducted to illustrate the finite-sample performance of the proposed methods.     

A flexible semiparametric regression model for bimodal,asymmetric and censored data

In this paper, we propose a new semiparametric heteroscedastic regression model allowing for positive and negative skewness and bimodal shapes using the B-spline basis for nonlinear effects. The proposed distribution is based on the generalized additive models for location, scale and shape framework in order to model any or all parameters of the distribution using parametric linear and/or nonparametric smooth functions of explanatory variables. We motivate the new model by means of Monte Carlo simulations, thus ignoring the skewness and bimodality of the random errors in semiparametric regression models, which may introduce biases on the parameter estimates and/or on the estimation of the associated variability measures. An iterative estimation process and some diagnostic methods are investigated. Applications to two real data sets are presented and the method is compared to the usual regression methods.     

A SEMIPARAMETRIC HIERARCHICAL METHOD FOR A REGRESSION MODEL WITH AN INTERVAL-CENSORED COVARIATE

A Bayesian framework is proposed for analysing regression models in which one of the covariates is interval‐censored. Such a situation was encountered in an AIDS clinical trial in which the goal was to examine the association between delays in initiating a new treatment after Indinavir failure and the subsequent viral load level of patients at the time of enrolment into the new treatment. The new method uses a mixture of Dirichlet processes allowing all the components in the model to be specified parametrically, except for the distribution of the interval‐censored covariate, which is treated non‐parametrically. The paper explains the proposed method for the linear regression model in detail. The performance of the method is assessed by simulations and illustrated using the AIDS clinical trial.     

On additive and multiplicative damage models and the characterizations of linear and logarithmic exponential families

Rao (1963) has formulated a damage model which we call an additive damage model. A suitable damage model, which we call a multiplicative damage model, has been considered by Krishnaji (1970) for income-related problems. In these models, an original observation is subjected to damage, e.g., death or under-reporting, according to a specified probability law. Within the framework of an additive damage model, with a special form of damage, characterizations of the linear and logarithmic exponential families are formulated using regression properties of the damaged part on the undamaged part. The characterizations of the gamma and Pareto distributions that have been found of some use in the theory of income distributions, are obtained as special cases. Similar results are investigated within the framework of the multiplicative damage model.     

Semiparametric distribution forecasting

Given m time series regression models, linear or not, with additive noise components, it is shown how to estimate semiparametrically the predictive probability distribution of one of the time series conditional on past random covariate data. This is done by assuming that the distributions of the residual components associated with the regression models are tilted versions of a reference distribution.     

On asymptotic properties of Bayesian partially linear models

In this paper, we present large sample properties of a partially linear model from the Bayesian perspective, in which responses are explained by the semiparametric regression model with the additive form of the linear component and the nonparametric component. For this purpose, we investigate asymptotic behaviors of posterior distributions in terms of consistency. Specifically, we deal with a specific Bayesian partially linear regression model with additive noises in which the nonparametric component is modeled using Gaussian process priors. Under the Bayesian partially linear model using Gaussian process priors, we focus on consistency of posterior distribution and consistency of the Bayes factor, and extend these results to generalized additive regression models and study their asymptotic properties. In addition we illustrate the asymptotic properties based on empirical analysis through simulation studies.     

Generalized additive mixed models

Following the extension from linear mixed models to additive mixed models, extension from generalized linear mixed models to generalized additive mixed models is made, Algorithms are developed to compute the MLE's of the nonlinear effects and the covariance structures based on the penalized marginal likelihood. Convergence of the algorithms and selection of the smooth param¬eters are discussed.