Bayesian regularisation in structured additive regression: a unifying perspective on shrinkage, smoothing and predictor selection

This paper surveys various shrinkage, smoothing and selection priors from a unifying perspective and shows how to combine them for Bayesian regularisation in the general class of structured additive regression models. As a common feature, all regularisation priors are conditionally Gaussian, given further parameters regularising model complexity. Hyperpriors for these parameters encourage shrinkage, smoothness or selection. It is shown that these regularisation (log-) priors can be interpreted as Bayesian analogues of several well-known frequentist penalty terms. Inference can be carried out with unified and computationally efficient MCMC schemes, estimating regularised regression coefficients and basis function coefficients simultaneously with complexity parameters and measuring uncertainty via corresponding marginal posteriors. For variable and function selection we discuss several variants of spike and slab priors which can also be cast into the framework of conditionally Gaussian priors. The performance of the Bayesian regularisation approaches is demonstrated in a hazard regression model and a high-dimensional geoadditive regression model.     

Copula, marginal distributions and model selection: a Bayesian note

Copula functions and marginal distributions are combined to produce multivariate distributions. We show advantages of estimating all parameters of these models using the Bayesian approach, which can be done with standard Markov chain Monte Carlo algorithms. Deviance-based model selection criteria are also discussed when applied to copula models since they are invariant under monotone increasing transformations of the marginals. We focus on the deviance information criterion. The joint estimation takes into account all dependence structure of the parameters’ posterior distributions in our chosen model selection criteria. Two Monte Carlo studies are conducted to show that model identification improves when the model parameters are jointly estimated. We study the Bayesian estimation of all unknown quantities at once considering bivariate copula functions and three known marginal distributions.     

Factor screening in nonregular two-level designs based on projection-based variable selection

In this paper, we focus on the problem of factor screening in nonregular two-level designs through gradually reducing the number of possible sets of active factors. We are particularly concerned with situations when three or four factors are active. Our proposed method works through examining fits of projection models, where variable selection techniques are used to reduce the number of terms. To examine the reliability of the methods in combination with such techniques, a panel of models consisting of three or four active factors with data generated from the 12-run and the 20-run Plackett–Burman (PB) design is used. The dependence of the procedure on the amount of noise, the number of active factors and the number of experimental factors is also investigated. For designs with few runs such as the 12-run PB design, variable selection should be done with care and default procedures in computer software may not be reliable to which we suggest improvements. A real example is included to show how we propose factor screening can be done in practice.     

Functional variable selection via Gram–Schmidt orthogonalization for multiple functional linear regression

ABSTRACT

Functional linear model is of great practical importance, as exemplified by applications in high-throughput studies such as meteorological and biomedical research. In this paper, we propose a new functional variable selection procedure, called functional variable selection via Gram–Schmidt (FGS) orthogonalization, for a functional linear model with a scalar response and multiple functional predictors. Instead of the regularization methods, FGS takes into account the similarity between the functional predictors in a data-driven way and utilizes the technique of Gram–Schmidt orthogonalization to remove the irrelevant predictors. FGS can successfully discriminate between the relevant and the irrelevant functional predictors to achieve a high true positive ratio without including many irrelevant predictors, and yield explainable models, which offers a new perspective for the variable selection method in the functional linear model. Simulation studies are carried out to evaluate the finite sample performance of the proposed method, and also a weather data set is analysed.     

An autocorrelation criterion for bandwidth selection in nonparametric regression ∗

In this note, we propose a new method for selecting the bandwidth parameter in non-parametric regression. While standard criteria, such as cross-validation, are based on the true regression curve about which we know little, we propose a criterion which focuses on the true errors about which assumptions may be made. Our proposal is to choose the bandwidth for which the residuals are as uncorrelated as possible. We use the Box-Pierce statistic as the objective to be minimized. In doing so, the behaviour of our residuals will be close to that of the true errors under the hypothesis of independent errors. A simulation study shows that our method succeeds in capturing the main features of the regression curve, in the sense that the number of turning-points of the curve is correctly estimated most of the time.     

Cross-validation prior choice in Bayesian probit regression with?many covariates

This paper examines prior choice in probit regression through a predictive cross-validation criterion. In particular, we focus on situations where the number of potential covariates is far larger than the number of observations, such as in gene expression data. Cross-validation avoids the tendency of such models to fit perfectly. We choose the scale parameter c in the standard variable selection prior as the minimizer of the log predictive score. Naive evaluation of the log predictive score requires substantial computational effort, and we investigate computationally cheaper methods using importance sampling. We find that K-fold importance densities perform best, in combination with either mixing over different values of c or with integrating over c through an auxiliary distribution.     

The Frisch–Waugh–Lovell theorem for the lasso and the ridge regression

The Frisch–Waugh–Lovell (FWL) (partitioned regression) theorem is essential in regression analysis. This is partly because it is quite useful to derive theoretical results. The lasso regression and the ridge regression, both of which are penalized least-squares regressions, have become popular statistical techniques. This article describes that the FWL theorem remains valid for these penalized least-squares regressions. More precisely, we demonstrate that the covariates corresponding to unpenalized regression parameters in these penalized least-squares regression can be projected out. Some other results related to the FWL theorem in such penalized least-squares regressions are also presented.     

Variable selection via penalized minimum φ-divergence estimation in logistic regression

We propose penalized minimum φ-divergence estimator for parameter estimation and variable selection in logistic regression. Using an appropriate penalty function, we show that penalized φ-divergence estimator has oracle property. With probability tending to 1, penalized φ-divergence estimator identifies the true model and estimates nonzero coefficients as efficiently as if the sparsity of the true model was known in advance. The advantage of penalized φ-divergence estimator is that it produces estimates of nonzero parameters efficiently than penalized maximum likelihood estimator when sample size is small and is equivalent to it for large one. Numerical simulations confirm our findings.     

Robust variable selection for the varying coefficient model based on composite L 1–L 2 regression

The varying coefficient model (VCM) is an important generalization of the linear regression model and many existing estimation procedures for VCM were built on L ₂ loss, which is popular for its mathematical beauty but is not robust to non-normal errors and outliers. In this paper, we address the problem of both robustness and efficiency of estimation and variable selection procedure based on the convex combined loss of L ₁ and L ₂ instead of only quadratic loss for VCM. By using local linear modeling method, the asymptotic normality of estimation is driven and a useful selection method is proposed for the weight of composite L ₁ and L ₂. Then the variable selection procedure is given by combining local kernel smoothing with adaptive group LASSO. With appropriate selection of tuning parameters by Bayesian information criterion (BIC) the theoretical properties of the new procedure, including consistency in variable selection and the oracle property in estimation, are established. The finite sample performance of the new method is investigated through simulation studies and the analysis of body fat data. Numerical studies show that the new method is better than or at least as well as the least square-based method in terms of both robustness and efficiency for variable selection.     

Bayesian P-Values for Testing Independence in 2 × 2 Contingency Tables

This article considers Bayesian p-values for testing independence in 2 × 2 contingency tables with cell counts observed from the two independent binomial sampling scheme and the multinomial sampling scheme. From the frequentist perspective, Fisher's p-value (p _F ) is the most commonly used p-value but it can be conservative for small to moderate sample sizes. On the other hand, from the Bayesian perspective, Bayarri and Berger (2000 Bayarri , M. J. , Berger , J. O. ( 2000 ). P-values for composite null models (with discussion) . J. Amer. Statist. Assoc. 95 : 1127 – 1170 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) first proposed the partial posterior predictive p-value (p _PPOST ), which can avoid the double use of the data that occurs in another Bayesian p-value proposed by Guttman (1967 Guttman , I. ( 1967 ). The use of the concept of a future observation in goodness-of-fit problems . J. Roy. Statist. Soc. Ser. B 29 : 83 – 100 . [Google Scholar]) and Rubin (1984 Rubin , D. B. ( 1984 ). Bayesianly justifiable and relevant frequency calculations for the applied statistician . Ann. Statist. 12 : 1151 – 1172 .[Crossref], [Web of Science ®] , [Google Scholar]), called the posterior predictive p-value (p _POST ). The subjective and objective Bayesian p-values in terms of p _POST and p _PPOST are derived under the beta prior and the (noninformative) Jeffreys prior, respectively. Numerical comparisons among p _F , p _POST , and p _PPOST reveal that p _PPOST performs much better than p _F and p _POST for small to moderate sample sizes from the frequentist perspective.     

Penalized full likelihood approach to variable selection for Cox’s regression model under nested case–control sampling

Lifetime Data Analysis - Assuming Cox’s regression model, we consider penalized full likelihood approach to conduct variable selection under nested case–control (NCC) sampling....     

Asymptotic properties for LS estimators in EV regression model with dependent errors

In this paper, we establish the strong consistency and asymptotic normality for the least square (LS) estimators in simple linear errors-in-variables (EV) regression models when the errors form a stationary α-mixing sequence of random variables. The quadratic-mean consistency is also considered.     

Conditional Score Tests for Heteroscedasticity in the Two-Way Error Components Model

This paper extends the one-way heteroskedasticity score test of Holly and Gardiol (2000, In: Krishnakumar, J, Ronchetti, E (Eds.), Panel Data Econometrics: Future Directions, North-Holland, Amsterdam, pp. 199–211) to two conditional Lagrange Multiplier (LM) tests of heteroskedasticity under contiguous alternatives within the two-way error components model framework. In each case, the derivation of Rao's efficient score statistics for testing heteroskedasticity is first obtained. Then, based on a specific set of assumptions, the asymptotic distribution of the score under contiguous alternatives is established. Finally, the expression for the score test statistic in the presence of heteroskedasticity and related asymptotic local powers of these score test statistics are derived and discussed.     

Bayesian estimation and stochastic model specification search for dynamic survival models

Dynamic survival models are a useful extension of the popular Cox model as the effects of explanatory variables are allowed to change over time. In this paper a new auxiliary mixture sampler for Bayesian estimation of the model parameters is introduced. This sampler forms the basis of a model space MCMC method for stochastic model specification search in dynamic survival models, which involves selection of covariates to include in the model as well as specification of effects as time-varying or constant. The method is applied to two well-known data sets from the literature.     

A Comparison of Bayesian Models of Heteroscedasticity in Nested Normal Data

We consider the fitting of a Bayesian model to grouped data in which observations are assumed normally distributed around group means that are themselves normally distributed, and consider several alternatives for accommodating the possibility of heteroscedasticity within the data. We consider the case where the underlying distribution of the variances is unknown, and investigate several candidate prior distributions for those variances. In each case, the parameters of the candidate priors (the hyperparameters) are themselves given uninformative priors (hyperpriors). The most mathematically convenient model for the group variances is to assign them inverse gamma distributed priors, the inverse gamma distribution being the conjugate prior distribution for the unknown variance of a normal population. We demonstrate that for a wide class of underlying distributions of the group variances, a model that assigns the variances an inverse gamma-distributed prior displays favorable goodness-of-fit properties relative to other candidate priors, and hence may be used as standard for modeling such data. This allows us to take advantage of the elegant mathematical property of prior conjugacy in a wide variety of contexts without compromising model fitness. We test our findings on nine real world publicly available datasets from different domains, and on a wide range of artificially generated datasets.     

Bayesian Estimation of the Log–linear Exponential Regression Model with Censorship and Collinearity

In this work, a simulation study is conducted to evaluate the performance of Bayesian estimators for the log–linear exponential regression model under different levels of censoring and degrees of collinearity for two covariates. The diffuse normal, independent Student-t and multivariate Student-t distributions are considered as prior distributions and to draw from the posterior distributions, the Metropolis algorithm is implemented. Also, the results are compared with the maximum likelihood estimators in terms of the mean squared error, coverages and length of the credibility and confidence intervals.     

Influence analysis in skew-Birnbaum–Saunders regression models and applications

In this paper, we propose a method to assess influence in skew-Birnbaum–Saunders regression models, which are an extension based on the skew-normal distribution of the usual Birnbaum–Saunders (BS) regression model. An interesting characteristic that the new regression model has is the capacity of predicting extreme percentiles, which is not possible with the BS model. In addition, since the observed likelihood function associated with the new regression model is more complex than that from the usual model, we facilitate the parameter estimation using a type-EM algorithm. Moreover, we employ influence diagnostic tools that considers this algorithm. Finally, a numerical illustration includes a brief simulation study and an analysis of real data in order to show the proposed methodology.     

Marginal-sum and maximum-likelihood estimation in a multiplicative tariff

It is well known that, for a multiplicative tariff with independent Poisson distributed claim numbers in the different tariff cells, the maximum-likelihood estimators of the parameters satisfy the marginal-sum equations. In the present paper we show that this is also true under the more general assumption that the claim numbers of the different cells arise from the decomposition of a collective model for the whole portfolio of risks. In this general setting, the claim numbers of the different cells need not be independent and need not be Poisson distributed.