Bootstrap confidence intervals for smoothing splines and their comparison to bayesian confidence intervals

We construct bootstrap confidence intervals for smoothing spline estimates based on Gaussian data, and penalized likelihood smoothing spline estimates based on data from .exponential families. Several vari- ations of bootstrap confidence intervals are considered and compared. We find that the commonly used ootstrap percentile intervals are inferior to the T intervals and to intervals based on bootstrap estimation of mean squared errors. The best variations of the bootstrap confidence intervals behave similar to the well known Bayesian confidence intervals. These bootstrap confidence intervals have an average coverage probability across the function being estimated, as opposed to a pointwise property.     

Coverage Properties of Confidence Intervals for Generalized Additive Model Components

Abstract. We study the coverage properties of Bayesian confidence intervals for the smooth component functions of generalized additive models (GAMs) represented using any penalized regression spline approach. The intervals are the usual generalization of the intervals first proposed by Wahba and Silverman in 1983 and 1985, respectively, to the GAM component context. We present simulation evidence showing these intervals have close to nominal ‘across‐the‐function’ frequentist coverage probabilities, except when the truth is close to a straight line/plane function. We extend the argument introduced by Nychka in 1988 for univariate smoothing splines to explain these results. The theoretical argument suggests that close to nominal coverage probabilities can be achieved, provided that heavy oversmoothing is avoided, so that the bias is not too large a proportion of the sampling variability. The theoretical results allow us to derive alternative intervals from a purely frequentist point of view, and to explain the impact that the neglect of smoothing parameter variability has on confidence interval performance. They also suggest switching the target of inference for component‐wise intervals away from smooth components in the space of the GAM identifiability constraints.     

Generalized Point Estimation with Application to Small Response Estimation

Motivated by a number of drawbacks of classical methods of point estimation, we generalize the definitions of point estimation, and address such notions as unbiasedness and estimation under constraints. The utility of the extension is shown by deriving more reliable estimates for small coefficients of regression models, and for variance components and random effects of mixed models. The extension is in the spirit of generalized confidence intervals introduced by Weerahandi (1993 Weerahandi , S. ( 1993 ). Generalized confidence intervals . J. Amer. Statist. Assoc. 88 : 899 – 905 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) and should encourage much needed further research in point estimation in unbalanced models, multi-variate models, non normal models, and nonlinear models.     

An iterative Monte Carlo method for nonconjugate Bayesian analysis

The Gibbs sampler has been proposed as a general method for Bayesian calculation in Gelfand and Smith (1990). However, the predominance of experience to date resides in applications assuming conjugacy where implementation is reasonably straightforward. This paper describes a tailored approximate rejection method approach for implementation of the Gibbs sampler when nonconjugate structure is present. Several challenging applications are presented for illustration.     

Monotone Smoothing with Application to Dose-Response Curve

Motivated by problems that arise in dose-response curve estimation, we developed a new method to estimate a monotone curve. The resulting monotone estimator is obtained by combining techniques from smoothing splines with nonnegativity properties of cubic B-splines. Numerical experiments are given to exemplify the method.     

A hierarchical Bayesian model for predicting the functional consequences of amino-acid polymorphisms

Summary.  Genetic polymorphisms in deoxyribonucleic acid coding regions may have a phenotypic effect on the carrier, e.g. by influencing susceptibility to disease. Detection of deleterious mutations via association studies is hampered by the large number of candidate sites; therefore methods are needed to narrow down the search to the most promising sites. For this, a possible approach is to use structural and sequence-based information of the encoded protein to predict whether a mutation at a particular site is likely to disrupt the functionality of the protein itself. We propose a hierarchical Bayesian multivariate adaptive regression spline (BMARS) model for supervised learning in this context and assess its predictive performance by using data from mutagenesis experiments on lac repressor and lysozyme proteins. In these experiments, about 12 amino-acid substitutions were performed at each native amino-acid position and the effect on protein functionality was assessed. The training data thus consist of repeated observations at each position, which the hierarchical framework is needed to account for. The model is trained on the lac repressor data and tested on the lysozyme mutations and vice versa. In particular, we show that the hierarchical BMARS model, by allowing for the clustered nature of the data, yields lower out-of-sample misclassification rates compared with both a BMARS and a frequen-tist MARS model, a support vector machine classifier and an optimally pruned classification tree.     

Notes on likelihood intervals and profiling

In many applications, decisions are made on the basis of function of parameters g(θ). When the value of g(theta;) is calculated using estimated values for te parameters, its is important to have a measure of the uncertainty associated with that value of g(theta;). Likelihood ratio approaches to finding likelihood intervals for functions of parameters have been shown to be more reliable, in terms of coverage probability, than the linearization approach. Two approaches to the generalization of the profiling algorithm have been proposed in the literature to enable construction of likelihood intervals for a function of parameters (Chen and Jennrich, 1996; Bates and Watts, 1988). In this paper we show the equivalence of these two methods. We also provide and analysis of cases in which neither profiling algorithm is appropriate. For one of these cases an alternate approach is suggested Whereas generalized profiling is based on maximizing the likelihood function given a constraint on the value of g(θ), the alternative algorithm is based on optimizing g(θ) given a constraint on the value of the likelihood function.     

Efficient sampling schemes for Bayesian MARS models with many predictors

Multivariate adaptive regression spline fitting or MARS (Friedman 1991) provides a useful methodology for flexible adaptive regression with many predictors. The MARS methodology produces an estimate of the mean response that is a linear combination of adaptively chosen basis functions. Recently, a Bayesian version of MARS has been proposed (Denison, Mallick and Smith 1998a, Holmes and Denison, 2002) combining the MARS methodology with the benefits of Bayesian methods for accounting for model uncertainty to achieve improvements in predictive performance. In implementation of the Bayesian MARS approach, Markov chain Monte Carlo methods are used for computations, in which at each iteration of the algorithm it is proposed to change the current model by either (a) Adding a basis function (birth step) (b) Deleting a basis function (death step) or (c) Altering an existing basis function (change step). In the algorithm of Denison, Mallick and Smith (1998a), when a birth step is proposed, the type of basis function is determined by simulation from the prior. This works well in problems with a small number of predictors, is simple to program, and leads to a simple form for Metropolis-Hastings acceptance probabilities. However, in problems with very large numbers of predictors where many of the predictors are useless it may be difficult to find interesting interactions with such an approach. In the original MARS algorithm of Friedman (1991) a heuristic is used of building up higher order interactions from lower order ones, which greatly reduces the complexity of the search for good basis functions to add to the model. While we do not exactly follow the intuition of the original MARS algorithm in this paper, we nevertheless suggest a similar idea in which the Metropolis-Hastings proposals of Denison, Mallick and Smith (1998a) are altered to allow dependence on the current model. Our modification allows more rapid identification and exploration of important interactions, especially in problems with very large numbers of predictor variables and many useless predictors. Performance of the algorithms is compared in simulation studies.     

CoSmo: A Constrained Scatterplot Smoother for Estimating Convex,Monotonic Transformations

In many of the applied sciences, it is common that the forms of empirical relationships are almost completely unknown prior to study. Scatterplot smoothers used in nonparametric regression methods have considerable potential to ease the burden of model specification that a researcher would otherwise face in this situation. Occasionally the researcher will know the sign of the first or second derivatives, or both. This article develops a smoothing method that can incorporate this kind of information. I show that cubic regression splines with bounds on the coefficients offer a simple and effective approximation to monotonic, convex or concave transformations. I also discuss methods for testing whether the constraints should be imposed. Monte Carlo results indicate that this method, dubbed CoSmo, has a lower approximation error than either locally weighted regression or two other constrained smoothing methods. CoSmo has many potential applications and should be especially useful in applied econometrics. As an illustration, I apply CoSmo in a multivariate context to estimate a hedonic price function and to test for concavity in one of the variables.     

Robust Confidence Intervals for Regression Parameters

The paper considers the problem of finding accurate small sample confidence intervals for regression parameters. Its approach is to construct conditional intervals with good robustness characteristics. This robustness is obtained by the choice of the density under which the conditional interval is computed. Both bounded influence and S-estimate style intervals are given. The required tail area computations are carried out using the results of DiCiccio, Field & Fraser (1990).     

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

Summary.  Structured additive regression models are perhaps the most commonly used class of models in statistical applications. It includes, among others, (generalized) linear models, (generalized) additive models, smoothing spline models, state space models, semiparametric regression, spatial and spatiotemporal models, log-Gaussian Cox processes and geostatistical and geoadditive models. We consider approximate Bayesian inference in a popular subset of structured additive regression models, latent Gaussian models , where the latent field is Gaussian, controlled by a few hyperparameters and with non-Gaussian response variables. The posterior marginals are not available in closed form owing to the non-Gaussian response variables. For such models, Markov chain Monte Carlo methods can be implemented, but they are not without problems, in terms of both convergence and computational time. In some practical applications, the extent of these problems is such that Markov chain Monte Carlo sampling is simply not an appropriate tool for routine analysis. We show that, by using an integrated nested Laplace approximation and its simplified version, we can directly compute very accurate approximations to the posterior marginals. The main benefit of these approximations is computational: where Markov chain Monte Carlo algorithms need hours or days to run, our approximations provide more precise estimates in seconds or minutes. Another advantage with our approach is its generality, which makes it possible to perform Bayesian analysis in an automatic, streamlined way, and to compute model comparison criteria and various predictive measures so that models can be compared and the model under study can be challenged.     

Heteroscedastic Weibull-Normal Mixture Models: A Bayesian Approach

In this article, we propose Bayesian methodology to obtain parameter estimates of the mixture of distributions belonging to the normal and biparametric Weibull families, modeling the mean and the variance parameters. Simulated studies and applications show the performance of the proposed models.     

Using jackknife in nonlinear models - confidence regions for a function of the structural parameter

A method of constructing jackknife confidence regions for a function of the structural parameter of a nonlinear model is investigated. The method is available for nonlinear regression models as well as for models with errors in the variables. Properties are discussed in comparison with traditional methods. This is supported by a simulation study.     

Exact Likelihood Inference for k Exponential Populations Under Joint Progressive Type-II Censoring

Comparative lifetime experiments are of great importance when the interest is in ascertaining the relative merits of k competing products with regard to their reliability. In this paper, when a joint progressively Type-II censored sample arising from k independent exponential populations is available, the conditional MLEs of the k exponential mean parameters are derived. Their conditional moment generating functions and exact densities are obtained, using which exact confidence intervals are developed for the parameters. Moreover, approximate confidence intervals based on the asymptotic normality of the MLEs and credible confidence regions from a Bayesian viewpoint are discussed. An empirical evaluation of the exact, approximate, bootstrap, and Bayesian intervals is also made in terms of coverage probabilities and average widths. Finally, an example is presented in order to illustrate all the methods of inference developed here.     

Bayesian regression with multivariate linear splines

We present a Bayesian analysis of a piecewise linear model constructed by using basis functions which generalizes the univariate linear spline to higher dimensions. Prior distributions are adopted on both the number and the locations of the splines, which leads to a model averaging approach to prediction with predictive distributions that take into account model uncertainty. Conditioning on the data produces a Bayes local linear model with distributions on both predictions and local linear parameters. The method is spatially adaptive and covariate selection is achieved by using splines of lower dimension than the data.     

Frequentist and Bayesian Interval Estimators for the Normal Mean

It is well known that a Bayesian credible interval for a parameter of interest is derived from a prior distribution that appropriately describes the prior information. However, it is less well known that there exists a frequentist approach developed by Pratt (1961 Pratt , J. W. ( 1961 ). Length of confidence intervals . J. Amer. Statist. Assoc. 56 : 549 – 657 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) that also utilizes prior information in the construction of frequentist confidence intervals. This frequentist approach produces confidence intervals that have minimum weighted average expected length, averaged according to some weight function that appropriately describes the prior information. We begin with a simple model as a starting point in comparing these two distinct procedures in interval estimation. Consider X ₁,…, X _n that are independent and identically N(μ, σ²) distributed random variables, where σ² is known, and the parameter of interest is μ. Suppose also that previous experience with similar data sets and/or specific background and expert opinion suggest that μ = 0. Our aim is to: (a) develop two types of Bayesian 1 ? α credible intervals for μ, derived from an appropriate prior cumulative distribution function F(μ) more importantly; (b) compare these Bayesian 1 ? α credible intervals for μ to the frequentist 1 ? α confidence interval for μ derived from Pratt's frequentist approach, in which the weight function corresponds to the prior cumulative distribution function F(μ). We show that the endpoints of the Bayesian 1 ? α credible intervals for μ are very different to the endpoints of the frequentist 1 ? α confidence interval for μ, when the prior information strongly suggests that μ = 0 and the data supports the uncertain prior information about μ. In addition, we assess the performance of these intervals by analyzing their coverage probability properties and expected lengths.     

Mixture of measurement errors and their impact on parameter inferences

A mixture measurement error model built upon skew normal distributions and normal distributions is developed to evaluate various impacts of measurement errors to parameter inferences in logistic regressions. Data generated from survey questionnaires are usually error contaminated. We consider two types of errors: person-specific bias and random errors. Person-specific bias is modelled using skew normal distribution, and the distribution of random errors is described by a normal distribution. Intensive simulations are conducted to evaluate the contribution of each component in the mixture to outcomes of interest. The proposed method is then applied to a questionnaire data set generated from a neural tube defect study. Simulation results and real data application indicate that ignoring measurement errors or misspecifying measurement error components can both produce misleading results, especially when measurement errors are actually skew distributed. The inferred parameters can be attenuated or inflated depending on how the measurement error components are specified. We expect the findings will self-explain the importance of adjusting measurement errors and thus benefit future data collection effort.     

Efficient Bayesian Multivariate Surface Regression

Methods for choosing a fixed set of knot locations in additive spline models are fairly well established in the statistical literature. The curse of dimensionality makes it nontrivial to extend these methods to nonadditive surface models, especially when there are more than a couple of covariates. We propose a multivariate Gaussian surface regression model that combines both additive splines and interactive splines, and a highly efficient Markov chain Monte Carlo algorithm that updates all the knot locations jointly. We use shrinkage prior to avoid overfitting with different estimated shrinkage factors for the additive and surface part of the model, and also different shrinkage parameters for the different response variables. Simulated data and an application to firm leverage data show that the approach is computationally efficient, and that allowing for freely estimated knot locations can offer a substantial improvement in out‐of‐sample predictive performance.     

Asymptotic normality of the lengths of a class of nonparametric confidence intervals for a regression parameter

In the linear regression model, the asymptotic distributions of certain functions of confidence bounds of a class of confidence intervals for the regression parameter arc investigated. The class of confidence intervals we consider in this paper are based on the usual linear rank statistics (signed as well as unsigned). Under suitable assumptions, if the confidence intervals are based on the signed linear rank statistics, it is established that the lengths, properly normalized, of the confidence intervals converge in law to the standard normal distributions; if the confidence intervals arc based on the unsigned linear rank statistics, it is then proved that a linear function of the confidence bounds converges in law to a normal distribution.     

A note on simultaneous inference with rerandomization tests

Valid simultaneous confidence intervals based on rerandomization are provided for the first time. They are derived from joint confidence regions which are constructed by testing for all possible parametric values. A simple exampe illustrates these confidence intervals and compares inferences from them with other methods.