On Parametric Bootstrapping and Bayesian Prediction

Abstract.  We investigate bootstrapping and Bayesian methods for prediction. The observations and the variable being predicted are distributed according to different distributions. Many important problems can be formulated in this setting. This type of prediction problem appears when we deal with a Poisson process. Regression problems can also be formulated in this setting. First, we show that bootstrap predictive distributions are equivalent to Bayesian predictive distributions in the second-order expansion when some conditions are satisfied. Next, the performance of predictive distributions is compared with that of a plug-in distribution with an estimator. The accuracy of prediction is evaluated by using the Kullback–Leibler divergence. Finally, we give some examples.     

On predictive distributions and Bayesian networks

In this paper we are interested in discrete prediction problems for a decision-theoretic setting, where the task is to compute the predictive distribution for a finite set of possible alternatives. This question is first addressed in a general Bayesian framework, where we consider a set of probability distributions defined by some parametric model class. Given a prior distribution on the model parameters and a set of sample data, one possible approach for determining a predictive distribution is to fix the parameters to the instantiation with the maximum a posteriori probability. A more accurate predictive distribution can be obtained by computing the evidence (marginal likelihood), i.e., the integral over all the individual parameter instantiations. As an alternative to these two approaches, we demonstrate how to use Rissanen's new definition of stochastic complexity for determining predictive distributions, and show how the evidence predictive distribution with Jeffrey's prior approaches the new stochastic complexity predictive distribution in the limit with increasing amount of sample data. To compare the alternative approaches in practice, each of the predictive distributions discussed is instantiated in the Bayesian network model family case. In particular, to determine Jeffrey's prior for this model family, we show how to compute the (expected) Fisher information matrix for a fixed but arbitrary Bayesian network structure. In the empirical part of the paper the predictive distributions are compared by using the simple tree-structured Naive Bayes model, which is used in the experiments for computational reasons. The experimentation with several public domain classification datasets suggest that the evidence approach produces the most accurate predictions in the log-score sense. The evidence-based methods are also quite robust in the sense that they predict surprisingly well even when only a small fraction of the full training set is used.     

Interval estimators for the population mean for skewed distributions with a small sample size

In finite population sampling, it has long been known that, for small sample sizes, when sampling from a skewed population, the usual frequentist intervals for the population mean cover the true value less often than their stated frequency of coverage. Recently, a non-informative Bayesian approach to some problems in finite population sampling has been developed, which is based on the 'Polya posterior'. For large sample sizes, these methods often closely mimic standard frequentist methods. In this paper, a modification of the 'Polya posterior', which employs the weighted Polya distribution, is shown to give interval estimators with improved coverage properties for problems with skewed populations and small sample sizes. This approach also yields improved tests for hypotheses about the mean of a skewed distribution.     

A nonparametric plug-in rule for selecting optimal block lengths for block bootstrap methods

In this paper, we consider the problem of empirical choice of optimal block sizes for block bootstrap estimation of population parameters. We suggest a nonparametric plug-in principle that can be used for estimating ‘mean squared error’-optimal smoothing parameters in general curve estimation problems, and establish its validity for estimating optimal block sizes in various block bootstrap estimation problems. A key feature of the proposed plug-in rule is that it can be applied without explicit analytical expressions for the constants that appear in the leading terms of the optimal block lengths. Furthermore, we also discuss the computational efficacy of the method and explore its finite sample properties through a simulation study.     

Experience with a bayesian bootstrap method incorporating proper prior information

Although bootstrapping has become widely used in statistical analysis, there has been little reported concerning bootstrapped Bayesian analyses, especially when there is proper prior informa-tion concerning the parameter of interest. In this paper, we first propose an operationally implementable definition of a Bayesian bootstrap. Thereafter, in simulated studies of the estimation of means and variances, this Bayesian bootstrap is compared to various parametric procedures. It turns out that little information is lost in using the Bayesian bootstrap even when the sampling distribution is known. On the other hand, the parametric procedures are at times very sensitive to incorrectly specified sampling distributions, implying that the Bayesian bootstrap is a very robust procedure for determining the posterior distribution of the parameter.     

Bayesian neural tree models for nonparametric regression

Frequentist and Bayesian methods differ in many aspects but share some basic optimal properties. In real-life prediction problems, situations exist in which a model based on one of the above paradigms is preferable depending on some subjective criteria. Nonparametric classification and regression techniques, such as decision trees and neural networks, have both frequentist (classification and regression trees (CARTs) and artificial neural networks) as well as Bayesian counterparts (Bayesian CART and Bayesian neural networks) to learning from data. In this paper, we present two hybrid models combining the Bayesian and frequentist versions of CART and neural networks, which we call the Bayesian neural tree (BNT) models. BNT models can simultaneously perform feature selection and prediction, are highly flexible, and generalise well in settings with limited training observations. We study the statistical consistency of the proposed approaches and derive the optimal value of a vital model parameter. The excellent performance of the newly proposed BNT models is shown using simulation studies. We also provide some illustrative examples using a wide variety of standard regression datasets from a public available machine learning repository to show the superiority of the proposed models in comparison to popularly used Bayesian CART and Bayesian neural network models.     

Bayesian Prediction of the Median of the Burr Distribution with Fixed and Random Sample Sizes

sLingappaiah (1986) was the first to introduce the idea of Bayesian prediction in life testing when the size of the future sample is a random variable. In this paper we discuss the Bayesion prediction of the sample median when the parent distribution is a generalized Burr distribution (GBD), the old sample is censored type II and the size of the future sample is a random variable. A numerical illustration is provided.     

Quasi-random resampling for the bootstrap

Quasi-random sequences are known to give efficient numerical integration rules in many Bayesian statistical problems where the posterior distribution can be transformed into periodic functions on then-dimensional hypercube. From this idea we develop a quasi-random approach to the generation of resamples used for Monte Carlo approximations to bootstrap estimates of bias, variance and distribution functions. We demonstrate a major difference between quasi-random bootstrap resamples, which are generated by deterministic algorithms and have no true randomness, and the usual pseudo-random bootstrap resamples generated by the classical bootstrap approach. Various quasi-random approaches are considered and are shown via a simulation study to result in approximants that are competitive in terms of efficiency when compared with other bootstrap Monte Carlo procedures such as balanced and antithetic resampling.     

Bayesian prediction of order statistics with fixed and random sample sizes based on k-record values from Pareto distribution

In this paper, the two-parameter Pareto distribution is considered and the problem of prediction of order statistics from a future sample and that of its geometric mean are discussed. The Bayesian approach is applied to construct predictors based on observed k-record values for the cases when the future sample size is fixed and when it is random. Several Bayesian prediction intervals are derived. Finally, the results of a simulation study and a numerical example are presented for illustrating all the inferential procedures developed here.     

A Martingale-based bootstrap inference with censored data

In this paper, me shall investigate a bootstrap method hasd on a martingale representation of the relevant statistic for inference to a class of functionals of the survival distribution. The method is similar in spirit to Efron's (1981) bootstrap, and thus in the present paper will be referred to as “martingale-based bootstrap” The method was derived from Lin,Wei and Ying (1993), who appiied the method in checking the Cox model with cumulative sums of martingale-based residuals. It is shown that this martingale-based bootstrap gives a correct first-order asymptotic approximation to the distribution function of the corresponding functional of the Kaplan-Meier estimator. As a consequence, confidence intervals constructed by the martingale-based bootstrap have asymptotially correct coverage probability. Our simulation study indicats that the martingale-based bootst strap method for a small and moderate sample sizes can be uniformly better than the usual bootstrap method in estimating the sampling distribution for a mean function and a point probability in survival analysis.     

A simple bootstrap bandwidth selector for local polynomial fitting

A new, fully data-driven bandwidth selector with a double smoothing (DS) bias term and a data-driven variance estimator is developed following the bootstrap idea. The data-driven variance estimation does not involve any additional bandwidth selection. The proposed bandwidth selector convergences faster than a plug-in one due to the DS bias estimate, whereas the data-driven variance improves its finite sample performance clearly and makes it stable. Asymptotic results of the proposals are obtained. A comparative simulation study was done to show the overall gains and the gains obtained by improving either the bias term or the variance estimate, respectively. It is shown that the use of a good variance estimator is more important when the sample size is relatively small.     

Marginal maximum likelihood estimation methods for the tuning parameters of ridge,power ridge,and generalized ridge regression

This study introduces fast marginal maximum likelihood (MML) algorithms for estimating the tuning (shrinkage) parameter(s) of the ridge and power ridge regression models, and an automatic plug-in MML estimator for the generalized ridge regression model, in a Bayesian framework. These methods are applicable to multicollinear or singular covariate design matrices, including matrices where the number of covariates exceeds the sample size. According to analyses of many real and simulated datasets, these MML-based ridge methods tend to compare favorably to other tuning parameter selection methods, in terms of computation speed, prediction accuracy, and ability to detect relevant covariates.     

Weighted Bootstrap Confidence Intervals for Generalized Behrens-Fisher Problems

In this article, the weighted bootstrap difference between two-sample means for generalized Behrens-Fisher problems is investigated along with its strong consistency. Moreover, the one-order accurate weighted bootstrap approximation to the sample distribution of sample difference is also established and hence based on it the weighted bootstrap intervals for the population difference is constructed. Simulation studies show that the weighted bootstrap interval performs better than other intervals we considered in some cases.     

Prediction for future failures in Weibull distribution under hybrid censoring

In this paper, we consider the prediction of a future observation based on a type-I hybrid censored sample when the lifetime distribution of experimental units is assumed to be a Weibull random variable. Different classical and Bayesian point predictors are obtained. Bayesian predictors are obtained using squared error and linear-exponential loss functions. We also provide a simulation consistent method for computing Bayesian prediction intervals. Monte Carlo simulations are performed to compare the performances of the different methods, and one data analysis has been presented for illustrative purposes.     

Bootstrapping a consistent nonparametric goodness-of-fit test

In this paper, we employ the parametric bootstrap to approximate the finite sample distribution of a goodness-of-fit test statistic in Fan (1994). We show that the proposed bootstrap procedure works in that the bootstrap distribution conditional on the random sample tends to the asymptotic distribution of the test statistic in probability. A simulation study demonstrates that the bootstrap approximation works extremely well in small samples with only 25 observations and is very robust to the value of the smoothing parameter in the kernel density estimation.     

Finite-sample properties of the bootstrap estimator in a Markov-switching model

The size distortion problem is clearly indicative of the small-sample approximation in the Markov-switching regression model. This paper shows that the bootstrap procedure can relieve the effects that this problem has. Our Monte Carlo simulation results reveal that the bootstrap maximum likelihood asymptotic approximations to the distribution can often be good when the sample size is small.     

协整参数的自举推断

 针对完全修正最小二乘(full-modified ordinary least square,简称FMOLS）估计方法,给出一种协整参数的自举推断程序,证明零假设下自举统计量与检验统计量具有相同的渐近分布。关于检验功效的研究表明,虽然有约束自举的实际检验水平表现良好,但如果零假设不成立,自举统计量的分布是不确定的,因而其经验分布不能作为检验统计量精确分布的有效估计。实际应用中建议使用无约束自举,因为无论观测数据是否满足零假设,其自举统计量与零假设下检验统计量都具有相同的渐近分布。最后,利用蒙特卡洛模拟对自举推断和渐近推断的有限样本表现进行比较研究。     

A composite quantile function estimator with applications in bootstrapping

In this note we define a composite quantile function estimator in order to improve the accuracy of the classical bootstrap procedure in small sample setting. The composite quantile function estimator employs a parametric model for modelling the tails of the distribution and uses the simple linear interpolation quantile function estimator to estimate quantiles lying between 1/(n+1) and n/(n+1). The method is easily programmed using standard software packages and has general applicability. It is shown that the composite quantile function estimator improves the bootstrap percentile interval coverage for a variety of statistics and is robust to misspecification of the parametric component. Moreover, it is also shown that the composite quantile function based approach surprisingly outperforms the parametric bootstrap for a variety of small sample situations.     

Bootstrap approximations for Bayesian analysis of Geo/G/1 discrete-time queueing models

In this paper we consider a Bayesian nonparametric approach to the analysis of discrete-time queueing models. The main motivation consists in applications to telecommunications, and in particular to asynchronous transfer mode (ATM) systems. Attention is focused on the posterior distribution of the overflow rate. Since the exact distribution of such a quantity is not available in a closed form, an approximation based on “proper” Bayesian bootstrap is proposed, and its properties are studied. Some possible alternatives to proper Bayesian bootstrap are also discussed. Finally, an application to real data is provided.     

Bootstrap Testing for Changes in Persistence with Heavy-Tailed Innovations

This article considers the problem of detection for changes in persistence with heavy-tailed innovations. We adopt a ratio type test and derive its null asymptotic distribution which is dependent on the stable index. Then a residual-based bootstrap is proposed when the stable index is unknown. Our procedure requires drawing bootstrap samples of size m < T, T being the size of original sample. We establish the convergence in probability of the bootstrap distribution function assuming that m → ∞ and m/T → 0. A Monte Carlo study has shown that the bootstrap improve the finite sample size and power compared to the asymptotic test, especially for small stable index.