Bayesian variable selection with related predictors

In data sets with many predictors, algorithms for identifying a good subset of predictors are often used. Most such algorithms do not allow for any relationships between predictors. For example, stepwise regression might select a model containing an interaction AB but neither main effect A or B. This paper develops mathematical representations of this and other relations between predictors, which may then be incorporated in a model selection procedure. A Bayesian approach that goes beyond the standard independence prior for variable selection is adopted, and preference for certain models is interpreted as prior information. Priors relevant to arbitrary interactions and polynomials, dummy variables for categorical factors, competing predictors, and restrictions on the size of the models are developed. Since the relations developed are for priors, they may be incorporated in any Bayesian variable selection algorithm for any type of linear model. The application of the methods here is illustrated via the stochastic search variable selection algorithm of George and McCulloch (1993), which is modified to utilize the new priors. The performance of the approach is illustrated with two constructed examples and a computer performance dataset.     

Semiparametric inference for survival models with step process covariates

The authors consider Bayesian methods for fitting three semiparametric survival models, incorporating time‐dependent covariates that are step functions. In particular, these are models due to Cox [Cox ( 1972 ) Journal of the Royal Statistical Society, Series B, 34, 187–208], Prentice & Kalbfleisch and Cox & Oakes [Cox & Oakes ( 1984 ) Analysis of Survival Data, Chapman and Hall, London]. The model due to Prentice & Kalbfleisch [Prentice & Kalbfleisch ( 1979 ) Biometrics, 35, 25–39], which has seen very limited use, is given particular consideration. The prior for the baseline distribution in each model is taken to be a mixture of Polya trees and posterior inference is obtained through standard Markov chain Monte Carlo methods. They demonstrate the implementation and comparison of these three models on the celebrated Stanford heart transplant data and the study of the timing of cerebral edema diagnosis during emergency room treatment of diabetic ketoacidosis in children. An important feature of their overall discussion is the comparison of semi‐parametric families, and ultimate criterion based selection of a family within the context of a given data set. The Canadian Journal of Statistics 37: 60–79; © 2009 Statistical Society of Canada     

Cross-validation prior choice in Bayesian probit regression with?many covariates

This paper examines prior choice in probit regression through a predictive cross-validation criterion. In particular, we focus on situations where the number of potential covariates is far larger than the number of observations, such as in gene expression data. Cross-validation avoids the tendency of such models to fit perfectly. We choose the scale parameter c in the standard variable selection prior as the minimizer of the log predictive score. Naive evaluation of the log predictive score requires substantial computational effort, and we investigate computationally cheaper methods using importance sampling. We find that K-fold importance densities perform best, in combination with either mixing over different values of c or with integrating over c through an auxiliary distribution.     

A Note on the Lower Bounds on Maximum Number of Clear Two-Factor Interactions for (2 m−p III) and (2 m−p IV) Designs

ABSTRACT

In the reliability analysis of mechanical repairable equipment subjected to reliability deterioration with operating time, two forms of the non-homogeneous Poisson processes, namely the Power-Law (PL) and the Log-Linear (LL) model, have found general acceptance in the literature. Inferential procedures, conditioned on the assumption of the PL or LL model, underestimate the overall uncertainty about a quantity of interest because the PL and LL models can provide different estimates of the quantity of interest, even when both of them adequately fit the observed data. In this paper, a composite estimation procedure, which uses the PL and LL models as competing models, is proposed in the framework of Bayesian statistics, thus allowing the uncertainty involved in model selection to be considered. A model-free approach is then proposed for incorporating technical information on the failure mechanism into the inferential procedure. Such an approach, which is based on two model-free quantities defined irrespectively of the functional form of the failure model, prevents that the prior information on the failure mechanism can improperly introduce prior probabilities on the adequacy of each model to fit the observed data. Finally, numerical applications are provided to illustrate the proposed procedures.     

Semiparametric hierarchical selection models for bayesian meta analysis

Meta-analysis refers to quantitative methods for combining results from independent studies in order to draw overall conclusions. Hierarchical models including selection models are introduced and shown to be useful in such Bayesian meta-analysis. Semiparametric hierarchical models are proposed using the Dirichlet process prior. These rich class of models combine the information of independent studies, allowing investigation of variability both between and within studies, and weight function. Here we investigate sensitivity of results to unobserved studies by considering a hierarchical selection model with including unknown weight function and use Markov chain Monte Carlo methods to develop inference for the parameters of interest. Using Bayesian method, this model is used on a meta-analysis of twelve studies comparing the effectiveness of two different types of flouride, in preventing cavities. Clinical informative prior is assumed. Summaries and plots of model parameters are analyzed to address questions of interest.     

Optimal allocation of time points for the random effects model

In this article the problem of the optimal selection and allocation of time points in repeated measures experiments is considered. D‐ optimal designs for linear regression models with a random intercept and first order auto‐regressive serial correlations are computed numerically and compared with designs having equally spaced time points. When the order of the polynomial is known and the serial correlations are not too small, the comparison shows that for any fixed number of repeated measures, a design with equally spaced time points is almost as efficient as the D‐ optimal design. When, however, there is no prior knowledge about the order of the underlying polynomial, the best choice in terms of efficiency is a D‐ optimal design for the highest possible relevant order of the polynomial. A design with equally‐spaced time points is the second best choice     

Feature selection and machine learning with mass spectrometry data for distinguishing cancer and non-cancer samples

This is a comparative study of various clustering and classification algorithms as applied to differentiate cancer and non-cancer protein samples using mass spectrometry data. Our study demonstrates the usefulness of a feature selection step prior to applying a machine learning tool. A natural and common choice of a feature selection tool is the collection of marginal p-values obtained from t-tests for testing the intensity differences at each m/z ratio in the cancer versus non-cancer samples. We study the effect of selecting a cutoff in terms of the overall Type 1 error rate control on the performance of the clustering and classification algorithms using the significant features. For the classification problem, we also considered m/z selection using the importance measures computed by the Random Forest algorithm of Breiman. Using a data set of proteomic analysis of serum from ovarian cancer patients and serum from cancer-free individuals in the Food and Drug Administration and National Cancer Institute Clinical Proteomics Database, we undertake a comparative study of the net effect of the machine learning algorithm–feature selection tool–cutoff criteria combination on the performance as measured by an appropriate error rate measure.     

Choosing Prior Hyperparameters: With Applications to Time-Varying Parameter Models

Time-varying parameter models with stochastic volatility are widely used to study macroeconomic and financial data. These models are almost exclusively estimated using Bayesian methods. A common practice is to focus on prior distributions that themselves depend on relatively few hyperparameters such as the scaling factor for the prior covariance matrix of the residuals governing time variation in the parameters. The choice of these hyperparameters is crucial because their influence is sizeable for standard sample sizes. In this article, we treat the hyperparameters as part of a hierarchical model and propose a fast, tractable, easy-to-implement, and fully Bayesian approach to estimate those hyperparameters jointly with all other parameters in the model. We show via Monte Carlo simulations that, in this class of models, our approach can drastically improve on using fixed hyperparameters previously proposed in the literature. Supplementary materials for this article are available online.     

Bayesian variable selection using an adaptive powered correlation prior

The problem of selecting the correct subset of predictors within a linear model has received much attention in recent literature. Within the Bayesian framework, a popular choice of prior has been Zellner's g$g$-prior which is based on the inverse of empirical covariance matrix of the predictors. An extension of the Zellner's prior is proposed in this article which allow for a power parameter on the empirical covariance of the predictors. The power parameter helps control the degree to which correlated predictors are smoothed towards or away from one another. In addition, the empirical covariance of the predictors is used to obtain suitable priors over model space. In this manner, the power parameter also helps to determine whether models containing highly collinear predictors are preferred or avoided. The proposed power parameter can be chosen via an empirical Bayes method which leads to a data adaptive choice of prior. Simulation studies and a real data example are presented to show how the power parameter is well determined from the degree of cross-correlation within predictors. The proposed modification compares favorably to the standard use of Zellner's prior and an intrinsic prior in these examples.     

Model‐Averaged Confidence Intervals

We develop an approach to evaluating frequentist model averaging procedures by considering them in a simple situation in which there are two‐nested linear regression models over which we average. We introduce a general class of model averaged confidence intervals, obtain exact expressions for the coverage and the scaled expected length of the intervals, and use these to compute these quantities for the model averaged profile likelihood (MPI) and model‐averaged tail area confidence intervals proposed by D. Fletcher and D. Turek. We show that the MPI confidence intervals can perform more poorly than the standard confidence interval used after model selection but ignoring the model selection process. The model‐averaged tail area confidence intervals perform better than the MPI and postmodel‐selection confidence intervals but, for the examples that we consider, offer little over simply using the standard confidence interval for θ under the full model, with the same nominal coverage.     

Prior distributions for variance parameters in a sparse-event meta-analysis of a few small trials

In rare diseases, typically only a small number of patients are available for a randomized clinical trial. Nevertheless, it is not uncommon that more than one study is performed to evaluate a (new) treatment. Scarcity of available evidence makes it particularly valuable to pool the data in a meta-analysis. When the primary outcome is binary, the small sample sizes increase the chance of observing zero events. The frequentist random-effects model is known to induce bias and to result in improper interval estimation of the overall treatment effect in a meta-analysis with zero events. Bayesian hierarchical modeling could be a promising alternative. Bayesian models are known for being sensitive to the choice of prior distributions for between-study variance (heterogeneity) in sparse settings. In a rare disease setting, only limited data will be available to base the prior on, therefore, robustness of estimation is desirable. We performed an extensive and diverse simulation study, aiming to provide practitioners with advice on the choice of a sufficiently robust prior distribution shape for the heterogeneity parameter. Our results show that priors that place some concentrated mass on small τ values but do not restrict the density for example, the Uniform(−10, 10) heterogeneity prior on the log(τ²) scale, show robust 95% coverage combined with less overestimation of the overall treatment effect, across varying degrees of heterogeneity. We illustrate the results with meta-analyzes of a few small trials.     

A review of optimal designs in survey sampling

Results in five areas of survey sampling dealing with the choice of the sampling design are reviewed. In Section 2, the results and discussions surrounding the purposive selection methods suggested by linear regression superpopulation models are reviewed. In Section 3, similar models to those in the previous section are considered; however, random sampling designs are considered and attention is focused on the optimal choice of π_j. Then in Section 4, systematic sampling methods obtained under autocorrelated superpopulation models are reviewed. The next section examines minimax sampling designs. The work in the final section is based solely on the randomization. In Section 6 methods of sample selection which yield inclusion probabilities π_j = n/N and π_ij = n(n - 1)/N(N - 1), but for which there are fewer than _NC_n possible samples, are mentioned briefly.     

Bayesian fractional polynomials

This paper sets out to implement the Bayesian paradigm for fractional polynomial models under the assumption of normally distributed error terms. Fractional polynomials widen the class of ordinary polynomials and offer an additive and transportable modelling approach. The methodology is based on a Bayesian linear model with a quasi-default hyper-g prior and combines variable selection with parametric modelling of additive effects. A Markov chain Monte Carlo algorithm for the exploration of the model space is presented. This theoretically well-founded stochastic search constitutes a substantial improvement to ad hoc stepwise procedures for the fitting of fractional polynomial models. The method is applied to a data set on the relationship between ozone levels and meteorological parameters, previously analysed in the literature.     

Comparing Two-Stage Segmentation Methods for Choice Data with a One-Stage Latent Class Choice Analysis

Market segmentation is a key concept in marketing research. Identification of consumer segments helps in setting up and improving a marketing strategy. Hence, the need is to improve existing methods and to develop new segmentation methods. We introduce two new consumer indicators that can be used as segmentation basis in two-stage methods, the forces and the dfbetas. Both bases express a subject’s effect on the aggregate estimates of the parameters in a conditional logit model. Further, individual-level estimates, obtained by either estimating a conditional logit model for each individual separately with maximum likelihood or by hierarchical Bayes (HB) estimation of a mixed logit choice model, and the respondents’ raw choices are also used as segmentation basis. In the second stage of the methods the bases are classified into segments with cluster analysis or latent class models. All methods are applied to choice data because of the increasing popularity of choice experiments to analyze choice behavior. To verify whether two-stage segmentation methods can compete with a one-stage approach, a latent class choice model is estimated as well. A simulation study reveals the superiority of the two-stage method that clusters the HB estimates and the one-stage latent class choice model. Additionally, very good results are obtained for two-stage latent class cluster analysis of the choices as well as for the two-stage methods clustering the forces, the dfbetas and the choices.     

Remedying the Neyman–Scott phenomenon in model discrimination

The objective of this paper is to investigate through simulation the possible presence of the incidental parameters problem when performing frequentist model discrimination with stratified data. In this context, model discrimination amounts to considering a structural parameter taking values in a finite space, with k points, k≥2. This setting seems to have not yet been considered in the literature about the Neyman–Scott phenomenon. Here we provide Monte Carlo evidence of the severity of the incidental parameters problem also in the model discrimination setting and propose a remedy for a special class of models. In particular, we focus on models that are scale families in each stratum. We consider traditional model selection procedures, such as the Akaike and Takeuchi information criteria, together with the best frequentist selection procedure based on maximization of the marginal likelihood induced by the maximal invariant, or of its Laplace approximation. Results of two Monte Carlo experiments indicate that when the sample size in each stratum is fixed and the number of strata increases, correct selection probabilities for traditional model selection criteria may approach zero, unlike what happens for model discrimination based on exact or approximate marginal likelihoods. Finally, two examples with real data sets are given.     

Model identification using 27 runs three level orthogonal arrays

In this paper we examine all the combinatorial non-isomorphic OA(27, q, 3, t), with 3≤q≤13 three level quantitative factors, with respect to model identification, estimation capacity and efficiency. We use the popular D-efficiency criterion to evaluate the ability of each design considered in estimating the parameters of a second-order model with adequate efficiency. The prior selection of the ‘middle’ level of factors plays an important role in the results.     

Boosting techniques for nonlinear time series models

Many of the popular nonlinear time series models require a priori the choice of parametric functions which are assumed to be appropriate in specific applications. This approach is mainly used in financial applications, when sufficient knowledge is available about the nonlinear structure between the covariates and the response. One principal strategy to investigate a broader class on nonlinear time series is the Nonlinear Additive AutoRegressive (NAAR) model. The NAAR model estimates the lags of a time series as flexible functions in order to detect non-monotone relationships between current and past observations. We consider linear and additive models for identifying nonlinear relationships. A componentwise boosting algorithm is applied for simultaneous model fitting, variable selection, and model choice. Thus, with the application of boosting for fitting potentially nonlinear models we address the major issues in time series modelling: lag selection and nonlinearity. By means of simulation we compare boosting to alternative nonparametric methods. Boosting shows a strong overall performance in terms of precise estimations of highly nonlinear lag functions. The forecasting potential of boosting is examined on the German industrial production (IP); to improve the model’s forecasting quality we include additional exogenous variables. Thus we address the second major aspect in this paper which concerns the issue of high dimensionality in models. Allowing additional inputs in the model extends the NAAR model to a broader class of models, namely the NAARX model. We show that boosting can cope with large models which have many covariates compared to the number of observations.     

Choosing the link function and accounting for link uncertainty in generalized linear models using Bayes factors

One important component of model selection using generalized linear models (GLM) is the choice of a link function. We propose using approximate Bayes factors to assess the improvement in fit over a GLM with canonical link when a parametric link family is used. The approximate Bayes factors are calculated using the Laplace approximations given in [32], together with a reference set of prior distributions. This methodology can be used to differentiate between different parametric link families, as well as allowing one to jointly select the link family and the independent variables. This involves comparing nonnested models and so standard significance tests cannot be used. The approach also accounts explicitly for uncertainty about the link function. The methods are illustrated using parametric link families studied in [12] for two data sets involving binomial responses. The first author was supported by Sonderforschungsbereich 386 Statistische Analyse Diskreter Strukturen, and the second author by NIH Grant 1R01CA094212-01 and ONR Grant N00014-01-10745.     

The performance of model averaged tail area confidence intervals

We investigate the exact coverage and expected length properties of the model averaged tail area (MATA) confidence interval proposed by Turek and Fletcher, CSDA, 2012, in the context of two nested, normal linear regression models. The simpler model is obtained by applying a single linear constraint on the regression parameter vector of the full model. For given length of response vector and nominal coverage of the MATA confidence interval, we consider all possible models of this type and all possible true parameter values, together with a wide class of design matrices and parameters of interest. Our results show that, while not ideal, MATA confidence intervals perform surprisingly well in our regression scenario, provided that we use the minimum weight within the class of weights that we consider on the simpler model.