Kernel spline regression

The authors propose «kernel spline regression,» a method of combining spline regression and kernel smoothing by replacing the polynomial approximation for local polynomial kernel regression with the spline basis. The new approach retains the local weighting scheme and the use of a bandwidth to control the size of local neighborhood. The authors compute the bias and variance of the kernel linear spline estimator, which they compare with local linear regression. They show that kernel spline estimators can succeed in capturing the main features of the underlying curve more effectively than local polynomial regression when the curvature changes rapidly. They also show through simulation that kernel spline regression often performs better than ordinary spline regression and local polynomial regression.     

Estimation and variable selection in nonparametric heteroscedastic regression

The article considers a Gaussian model with the mean and the variance modeled flexibly as functions of the independent variables. The estimation is carried out using a Bayesian approach that allows the identification of significant variables in the variance function, as well as averaging over all possible models in both the mean and the variance functions. The computation is carried out by a simulation method that is carefully constructed to ensure that it converges quickly and produces iterates from the posterior distribution that have low correlation. Real and simulated examples demonstrate that the proposed method works well. The method in this paper is important because (a) it produces more realistic prediction intervals than nonparametric regression estimators that assume a constant variance; (b) variable selection identifies the variables in the variance function that are important; (c) variable selection and model averaging produce more efficient prediction intervals than those obtained by regular nonparametric regression.     

Automatic bandwidth selection in robust nonparametric regression

Nonparametric regression techniques have been studied extensively in the literature in recent years due to their flexibility.In addition robust versions of these techniques have become popular and have been incorporated into some of the standard statistical analysis packages.With new techniques available comes the responsibility of using them properly and in appropriate situations. Often, as in the case presented here, model-fitting diagnostics, such as cross-validation statistics,are not available as tools to determine if the smoothing parameter value being used is preferable to some other arbitrarily chosen value.     

Bayesian variable selection and estimation in maximum entropy quantile regression

Quantile regression has gained increasing popularity as it provides richer information than the regular mean regression, and variable selection plays an important role in the quantile regression model building process, as it improves the prediction accuracy by choosing an appropriate subset of regression predictors. Unlike the traditional quantile regression, we consider the quantile as an unknown parameter and estimate it jointly with other regression coefficients. In particular, we adopt the Bayesian adaptive Lasso for the maximum entropy quantile regression. A flat prior is chosen for the quantile parameter due to the lack of information on it. The proposed method not only addresses the problem about which quantile would be the most probable one among all the candidates, but also reflects the inner relationship of the data through the estimated quantile. We develop an efficient Gibbs sampler algorithm and show that the performance of our proposed method is superior than the Bayesian adaptive Lasso and Bayesian Lasso through simulation studies and a real data analysis.     

Bayesian quantile regression and variable selection for partial linear single-index model: Using free knot spline

Partial linear single-index model (PLSIM) has both the flexibility of nonparametric treatment and interpretability of linear term, yet existing literatures about it mainly focused on mean regression, and quantile regression analysis is scarce. Based on free knot spline approximation, we apply asymmetric Laplace distribution to implement Bayesian quantile regression, and perform variable selection in linear term and index vector via binary indicators. Our approach is exempt from regularity conditions in frequentist method, and could execute variable selection and quantile regression under mutual posterior correction, which is also the first work to implement them jointly for PLSIM in fully Bayesian framework. The numerical simulation manifests the superiority of our approach to previous methods, which embodied in better efficiency of variable selection, index vector estimates and link function approximation with different error distributions. For illustration of its application, we build a power consumption model of A₂/O process in wastewater treatment and emphatically analyze the impact of water quality factors.     

Variable selection in quantile regression via Gibbs sampling

Due to computational challenges and non-availability of conjugate prior distributions, Bayesian variable selection in quantile regression models is often a difficult task. In this paper, we address these two issues for quantile regression models. In particular, we develop an informative stochastic search variable selection (ISSVS) for quantile regression models that introduces an informative prior distribution. We adopt prior structures which incorporate historical data into the current data by quantifying them with a suitable prior distribution on the model parameters. This allows ISSVS to search more efficiently in the model space and choose the more likely models. In addition, a Gibbs sampler is derived to facilitate the computation of the posterior probabilities. A major advantage of ISSVS is that it avoids instability in the posterior estimates for the Gibbs sampler as well as convergence problems that may arise from choosing vague priors. Finally, the proposed methods are illustrated with both simulation and real data.     

Model selection in regression based on pre-smoothing

In this paper, we investigate the effect of pre-smoothing on model selection. Christóbal et al 6 Christóbal Christóbal, J. A., Faraldo Roca, P. and González Manteiga, W. 1987. A class of linear regression parameter estimators constructed by nonparametric estimation. Ann. Statist.,, 15: 603–609. [Crossref], [Web of Science ®] [Google Scholar] showed the beneficial effect of pre-smoothing on estimating the parameters in a linear regression model. Here, in a regression setting, we show that smoothing the response data prior to model selection by Akaike's information criterion can lead to an improved selection procedure. The bootstrap is used to control the magnitude of the random error structure in the smoothed data. The effect of pre-smoothing on model selection is shown in simulations. The method is illustrated in a variety of settings, including the selection of the best fractional polynomial in a generalized linear model.     

Bayesian variable selection in Poisson change-point regression analysis

In this article, we develop a Bayesian variable selection method that concerns selection of covariates in the Poisson change-point regression model with both discrete and continuous candidate covariates. Ranging from a null model with no selected covariates to a full model including all covariates, the Bayesian variable selection method searches the entire model space, estimates posterior inclusion probabilities of covariates, and obtains model averaged estimates on coefficients to covariates, while simultaneously estimating a time-varying baseline rate due to change-points. For posterior computation, the Metropolis-Hastings within partially collapsed Gibbs sampler is developed to efficiently fit the Poisson change-point regression model with variable selection. We illustrate the proposed method using simulated and real datasets.     

Bayesian elastic net single index quantile regression

Single index model conditional quantile regression is proposed in order to overcome the dimensionality problem in nonparametric quantile regression. In the proposed method, the Bayesian elastic net is suggested for single index quantile regression for estimation and variables selection. The Gaussian process prior is considered for unknown link function and a Gibbs sampler algorithm is adopted for posterior inference. The results of the simulation studies and numerical example indicate that our propose method, BENSIQReg, offers substantial improvements over two existing methods, SIQReg and BSIQReg. The BENSIQReg has consistently show a good convergent property, has the least value of median of mean absolute deviations and smallest standard deviations, compared to the other two methods.     

Consistent model identification of varying coefficient quantile regression with BIC tuning parameter selection

Quantile regression provides a flexible platform for evaluating covariate effects on different segments of the conditional distribution of response. As the effects of covariates may change with quantile level, contemporaneously examining a spectrum of quantiles is expected to have a better capacity to identify variables with either partial or full effects on the response distribution, as compared to focusing on a single quantile. Under this motivation, we study a general adaptively weighted LASSO penalization strategy in the quantile regression setting, where a continuum of quantile index is considered and coefficients are allowed to vary with quantile index. We establish the oracle properties of the resulting estimator of coefficient function. Furthermore, we formally investigate a Bayesian information criterion (BIC)-type uniform tuning parameter selector and show that it can ensure consistent model selection. Our numerical studies confirm the theoretical findings and illustrate an application of the new variable selection procedure.     

Model selection in quantile regression models

Lasso methods are regularisation and shrinkage methods widely used for subset selection and estimation in regression problems. From a Bayesian perspective, the Lasso-type estimate can be viewed as a Bayesian posterior mode when specifying independent Laplace prior distributions for the coefficients of independent variables [32 T. Park, G. Casella, The Bayesian Lasso, J. Amer. Statist. Assoc. 103 (2008), pp. 681–686. doi: 10.1198/016214508000000337[Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]]. A scale mixture of normal priors can also provide an adaptive regularisation method and represents an alternative model to the Bayesian Lasso-type model. In this paper, we assign a normal prior with mean zero and unknown variance for each quantile coefficient of independent variable. Then, a simple Markov Chain Monte Carlo-based computation technique is developed for quantile regression (QReg) models, including continuous, binary and left-censored outcomes. Based on the proposed prior, we propose a criterion for model selection in QReg models. The proposed criterion can be applied to classical least-squares, classical QReg, classical Tobit QReg and many others. For example, the proposed criterion can be applied to rq(), lm() and crq() which is available in an R package called Brq. Through simulation studies and analysis of a prostate cancer data set, we assess the performance of the proposed methods. The simulation studies and the prostate cancer data set analysis confirm that our methods perform well, compared with other approaches.     

Subset selection in quantile regression analysis via alternative Bayesian information criteria and heuristic optimization

Subset selection is an extensively studied problem in statistical learning. Especially it becomes popular for regression analysis. This problem has considerable attention for generalized linear models as well as other types of regression methods. Quantile regression is one of the most used types of regression method. In this article, we consider subset selection problem for quantile regression analysis with adopting some recent Bayesian information criteria. We also utilized heuristic optimization during selection process. Simulation and real data application results demonstrate the capability of the mentioned information criteria. According to results, these information criteria can determine the true models effectively in quantile regression models.     

Bayesian Tobit quantile regression using g-prior distribution with ridge parameter

A Bayesian approach is proposed for coefficient estimation in the Tobit quantile regression model. The proposed approach is based on placing a g-prior distribution depends on the quantile level on the regression coefficients. The prior is generalized by introducing a ridge parameter to address important challenges that may arise with censored data, such as multicollinearity and overfitting problems. Then, a stochastic search variable selection approach is proposed for Tobit quantile regression model based on g-prior. An expression for the hyperparameter g is proposed to calibrate the modified g-prior with a ridge parameter to the corresponding g-prior. Some possible extensions of the proposed approach are discussed, including the continuous and binary responses in quantile regression. The methods are illustrated using several simulation studies and a microarray study. The simulation studies and the microarray study indicate that the proposed approach performs well.     

Particle swarm optimization-based variable selection in Poisson regression analysis via information complexity-type criteria

Modeling of count responses is widely performed via Poisson regression models. This paper covers the problem of variable selection in Poisson regression analysis. The basic emphasis of this paper is to present the usefulness of information complexity-based criteria for Poisson regression. Particle swarm optimization (PSO) algorithm was adopted to minimize the information criteria. A real dataset example and two simulation studies were conducted for highly collinear and lowly correlated datasets. Results demonstrate the capability of information complexity-type criteria. According to the results, information complexity-type criteria can be effectively used instead of classical criteria in count data modeling via the PSO algorithm.     

Variable selection in gamma regression models via artificial bee colony algorithm

Variable selection is an important task in regression analysis. Performance of the statistical model highly depends on the determination of the subset of predictors. There are several methods to select most relevant variables to construct a good model. However in practice, the dependent variable may have positive continuous values and not normally distributed. In such situations, gamma distribution is more suitable than normal for building a regression model. This paper introduces an heuristic approach to perform variable selection using artificial bee colony optimization for gamma regression models. We evaluated the proposed method against with classical selection methods such as backward and stepwise. Both simulation studies and real data set examples proved the accuracy of our selection procedure.     

Performance of Variable Selection Methods in Regression Using Variations of the Bayesian Information Criterion

The Bayesian information criterion (BIC) is widely used for variable selection. We focus on the regression setting for which variations of the BIC have been proposed. A version that includes the Fisher Information matrix of the predictor variables performed best in one published study. In this article, we extend the evaluation, introduce a performance measure involving how closely posterior probabilities are approximated, and conclude that the version that includes the Fisher Information often favors regression models having more predictors, depending on the scale and correlation structure of the predictor matrix. In the image analysis application that we describe, we therefore prefer the standard BIC approximation because of its relative simplicity and competitive performance at approximating the true posterior probabilities.     

Bayesian variable selection in logistic regression: predicting company earnings direction

This paper presents a Bayesian technique for the estimation of a logistic regression model including variable selection. As in Ou & Penman (1989), the model is used to predict the direction of company earnings, one year ahead, from a large set of accounting variables from financial statements. To estimate the model, the paper presents a Markov chain Monte Carlo sampling scheme that includes the variable selection technique of Smith & Kohn (1996) and the non-Gaussian estimation method of Mira & Tierney (2001). The technique is applied to data for companies in the United States and Australia. The results obtained compare favourably to the technique used by Ou & Penman (1989) for both regions.     

An efficient cross-validation algorithm for window width selection for nonparametric kernel regression

This paper presents an approach to cross-validated window width choice which greatly reduces computation time, which can be used regardless of the nature of the kernel function, and which avoids the use of the Fast Fourier Transform. This approach is developed for window width selection in the context of kernel estimation of an unknown conditional mean.