On the bootstrap and smoothed bootstrap

The standard bootstrap and two commonly used types of smoothed bootstrap are investigated. The saddlepoint approximations are used to evaluate the accuracy of the three bootstrap estimates of the density of a sample mean. The optimal choice for the smoothing parameter is obtained when smoothing is useful in reducing the mean squared error.     

Bootstrap confidence regions computed from autoregressions of arbitrary order

Given a linear time series, e.g. an autoregression of infinite order, we may construct a finite order approximation and use that as the basis for confidence regions. The sieve or autoregressive bootstrap, as this method is often called, is generally seen as a competitor with the better-understood block bootstrap approach. However, in the present paper we argue that, for linear time series, the sieve bootstrap has significantly better performance than blocking methods and offers a wider range of opportunities. In particular, since it does not corrupt second-order properties then it may be used in a double-bootstrap form, with the second bootstrap application being employed to calibrate a basic percentile method confidence interval. This approach confers second-order accuracy without the need to estimate variance. That offers substantial benefits, since variances of statistics based on time series can be difficult to estimate reliably, and—partly because of the relatively small amount of information contained in a dependent process—are notorious for causing problems when used to Studentize. Other advantages of the sieve bootstrap include considerably greater robustness against variations in the choice of the tuning parameter, here equal to the autoregressive order, and the fact that, in contradistinction to the case of the block bootstrap, the percentile t version of the sieve bootstrap may be based on the 'raw' estimator of standard error. In the process of establishing these properties we show that the sieve bootstrap is second order correct.     

A model-averaging approach for smoothing spline regression

ABSTRACT

This article considers nonparametric regression problems and develops a model-averaging procedure for smoothing spline regression problems. Unlike most smoothing parameter selection studies determining an optimum smoothing parameter, our focus here is on the prediction accuracy for the true conditional mean of Y given a predictor X. Our method consists of two steps. The first step is to construct a class of smoothing spline regression models based on nonparametric bootstrap samples, each with an appropriate smoothing parameter. The second step is to average bootstrap smoothing spline estimates of different smoothness to form a final improved estimate. To minimize the prediction error, we estimate the model weights using a delete-one-out cross-validation procedure. A simulation study has been performed by using a program written in R. The simulation study provides a comparison of the most well known cross-validation (CV), generalized cross-validation (GCV), and the proposed method. This new method is straightforward to implement, and gives reliable performances in simulations.     

The relative performances of improved ridge estimators and an empirical bayes estimator: some monte carlo results

The relative 'performances of improved ridge estimators and an empirical Bayes estimator are studied by means of Monte Carlo simulations. The empirical Bayes method is seen to perform consistently better in terms of smaller MSE and more accurate empirical coverage than any of the estimators considered here. A bootstrap method is proposed to obtain more reliable estimates of the MSE of ridge esimators. Some theorems on the bootstrap for the ridge estimators are also given and they are used to provide an analytical understanding of the proposed bootstrap procedure. Empirical coverages of the ridge estimators based on the proposed procedure are generally closer to the nominal coverage when compared to their earlier counterparts. In general, except for a few cases, these coverages are still less accurate than the empirical coverages of the empirical Bayes estimator.     

Ordinal ridge regression with categorical predictors

In multi-category response models, categories are often ordered. In the case of ordinal response models, the usual likelihood approach becomes unstable with ill-conditioned predictor space or when the number of parameters to be estimated is large relative to the sample size. The likelihood estimates do not exist when the number of observations is less than the number of parameters. The same problem arises if constraint on the order of intercept values is not met during the iterative procedure. Proportional odds models (POMs) are most commonly used for ordinal responses. In this paper, penalized likelihood with quadratic penalty is used to address these issues with a special focus on POMs. To avoid large differences between two parameter values corresponding to the consecutive categories of an ordinal predictor, the differences between the parameters of two adjacent categories should be penalized. The considered penalized-likelihood function penalizes the parameter estimates or differences between the parameter estimates according to the type of predictors. Mean-squared error for parameter estimates, deviance of fitted probabilities and prediction error for ridge regression are compared with usual likelihood estimates in a simulation study and an application.     

Bayesian bootstrap prediction

In this paper, bootstrap prediction is adapted to resolve some problems in small sample datasets. The bootstrap predictive distribution is obtained by applying Breiman's bagging to the plug-in distribution with the maximum likelihood estimator. The effectiveness of bootstrap prediction has previously been shown, but some problems may arise when bootstrap prediction is constructed in small sample datasets. In this paper, Bayesian bootstrap is used to resolve the problems. The effectiveness of Bayesian bootstrap prediction is confirmed by some examples. These days, analysis of small sample data is quite important in various fields. In this paper, some datasets are analyzed in such a situation. For real datasets, it is shown that plug-in prediction and bootstrap prediction provide very poor prediction when the sample size is close to the dimension of parameter while Bayesian bootstrap prediction provides stable prediction.     

Stochastically optimal bootstrap sample size for shrinkage-type statistics

In nonregular problems where the conventional \(n\) out of \(n\) bootstrap is inconsistent, the \(m\) out of \(n\) bootstrap provides a useful remedy to restore consistency. Conventionally, optimal choice of the bootstrap sample size \(m\) is taken to be the minimiser of a frequentist error measure, estimation of which has posed a major difficulty hindering practical application of the \(m\) out of \(n\) bootstrap method. Relatively little attention has been paid to a stronger, stochastic, version of the optimal bootstrap sample size, defined as the minimiser of an error measure calculated directly from the observed sample. Motivated by this stronger notion of optimality, we develop procedures for calculating the stochastically optimal value of \(m\). Our procedures are shown to work under special forms of Edgeworth-type expansions which are typically satisfied by statistics of the shrinkage type. Theoretical and empirical properties of our methods are illustrated with three examples, namely the James–Stein estimator, the ridge regression estimator and the post-model-selection regression estimator.     

Ridge estimation in generalized linear models and proportional hazards regressions

Conventionally, a ridge parameter is estimated as a function of regression parameters based on ordinary least squares. In this article, we proposed an iterative procedure instead of the one-step or conventional ridge method. Additionally, we construct an indicator that measures the potential degree of improvement in mean squared error when ridge estimates are employed. Simulations show that our methods are appropriate for a wide class of non linear models including generalized linear models and proportional hazards (PHs) regressions. The method is applied to a PH regression with highly collinear covariates in a cancer recurrence study.     

A weighted bootstrap approach to bootstrap iteration

The operation of resampling from a bootstrap resample, encountered in applications of the double bootstrap, maybe viewed as resampling directly from the sample but using probability weights that are proportional to the numbers of times that sample values appear in the resample. This suggests an approximate approach to double-bootstrap Monte Carlo simulation, where weighted bootstrap methods are used to circumvent much of the labour involved in compounded Monte Carlo approximation. In the case of distribution estimation or, equivalently, confidence interval calibration, the new method may be used to reduce the computational labour. Moreover, the method produces the same order of magnitude of coverage error for confidence intervals, or level error for hypothesis tests, as a full application of the double bootstrap.     

Selection of the Ridge Parameter Using Mathematical Programming

Ridge regression solves multicollinearity problems by introducing a biasing parameter that is called ridge parameter; it shrinks the estimates and their standard errors in order to reach acceptable results. Selection of the ridge parameter was done using several subjective and objective techniques that are concerned with certain criteria. In this study, selection of the ridge parameter depends on other important statistical measures to reach a better value of the ridge parameter. The proposed ridge parameter selection technique depends on a mathematical programming model and the results are evaluated using a simulation study. The performance of the proposed method is good when the error variance is greater than or equal to one; the sample consists of 20 observations, the number of explanatory variables in the model is 2, and there is a very strong correlation between the two explanatory variables.     

Bootstrap prediction regions for multivariate autoregressive processes

Two new methods for improving prediction regions in the context of vector autoregressive (VAR) models are proposed. These methods, which are based on the bootstrap technique, take into account the uncertainty associated with the estimation of the model order and parameters. In particular, by exploiting an independence property of the prediction error, we will introduce a bootstrap procedure that allows for better estimates of the forecasting distribution, in the sense that the variability of its quantile estimators is substantially reduced, without requiring additional bootstrap replications. The proposed methods have a good performance even if the disturbances distribution is not Gaussian. An application to a real data set is presented.     

A note about calibrated prediction regions and distributions

In this work we address the problem of the construction of prediction regions and distribution functions, with particular regard to the multidimensional setting. Firstly, we define a simple procedure for calculating the predictive distribution function which gives improved prediction limits. Secondly, with a multivariate generalization of a result presented in Ueki and Fueda (2007), we propose a method for correcting estimative prediction regions, to reduce their coverage error to the third-order accuracy. The improved prediction regions and the associated distribution functions are easy to calculate using a suitable bootstrap procedure. Examples of application are included, showing the good performance of the proposed method, even if we consider an approximated model for prediction purposes.     

Fast and robust bootstrap

In this paper we review recent developments on a bootstrap method for robust estimators which is computationally faster and more resistant to outliers than the classical bootstrap. This fast and robust bootstrap method is, under reasonable regularity conditions, asymptotically consistent. We describe the method in general and then consider its application to perform inference based on robust estimators for the linear regression and multivariate location-scatter models. In particular, we study confidence and prediction intervals and tests of hypotheses for linear regression models, inference for location-scatter parameters and principal components, and classification error estimation for discriminant analysis.     

A new linearized ridge Poisson estimator in the presence of multicollinearity

Poisson regression is a very commonly used technique for modeling the count data in applied sciences, in which the model parameters are usually estimated by the maximum likelihood method. However, the presence of multicollinearity inflates the variance of maximum likelihood (ML) estimator and the estimated parameters give unstable results. In this article, a new linearized ridge Poisson estimator is introduced to deal with the problem of multicollinearity. Based on the asymptotic properties of ML estimator, the bias, covariance and mean squared error of the proposed estimator are obtained and the optimal choice of shrinkage parameter is derived. The performance of the existing estimators and proposed estimator is evaluated through Monte Carlo simulations and two real data applications. The results clearly reveal that the proposed estimator outperforms the existing estimators in the mean squared error sense.KEYWORDS: Poisson regression, multicollinearity, ridge Poisson estimator, linearized ridge regression estimator, mean squared errorMathematics Subject Classifications: 62J07, 62F10     

A location-invariant non-positive moment-type estimator of the extreme value index

This paper investigates a class of location invariant non-positive moment-type estimators of extreme value index, which is highly flexible due to the tuning parameter involved. Its asymptotic expansions and its optimal sample fraction in terms of minimal asymptotic mean square error are derived. A small scale Monte Carlo simulation turns out that the new estimators, with a suitable choice of the tuning parameter driven by the data itself, perform well compared to the known ones. Finally, the proposed estimators with a bootstrap optimal sample fraction are applied to an environmental data set.     

Ideal bootstrap estimation of expected prediction error for k-nearest neighbor classifiers: Applications for classification and error assessment

Euclidean distance k-nearest neighbor (k-NN) classifiers are simple nonparametric classification rules. Bootstrap methods, widely used for estimating the expected prediction error of classification rules, are motivated by the objective of calculating the ideal bootstrap estimate of expected prediction error. In practice, bootstrap methods use Monte Carlo resampling to estimate the ideal bootstrap estimate because exact calculation is generally intractable. In this article, we present analytical formulae for exact calculation of the ideal bootstrap estimate of expected prediction error for k-NN classifiers and propose a new weighted k-NN classifier based on resampling ideas. The resampling-weighted k-NN classifier replaces the k-NN posterior probability estimates by their expectations under resampling and predicts an unclassified covariate as belonging to the group with the largest resampling expectation. A simulation study and an application involving remotely sensed data show that the resampling-weighted k-NN classifier compares favorably to unweighted and distance-weighted k-NN classifiers.     

Penalized angular regression for personalized predictions

Personalization is becoming an important aspect of many predictive applications. We introduce a penalized regression method which inherently implements personalization. Personalized angle (PAN) regression constructs regression coefficients that are specific to the covariate vector for which one is producing a prediction, thus personalizing the regression model itself. This is achieved by penalizing the normalized prediction for a given covariate vector. The method therefore penalizes the normalized regression coefficients, or the angles of the regression coefficients in a hyperspherical parametrization, introducing a new angle-based class of penalties. PAN hence combines two novel concepts: penalizing the normalized coefficients and personalization. For an orthogonal design matrix, we show that the PAN estimator is the solution to a low-dimensional eigenvector equation. Based on the hyperspherical parametrization, we construct an efficient algorithm to calculate the PAN estimator. We propose a parametric bootstrap procedure for selecting the tuning parameter, and simulations show that PAN regression can outperform ordinary least squares, ridge regression and other penalized regression methods in terms of prediction error. Finally, we demonstrate the method in a medical application.     

The Bootstrap and Kriging Prediction Intervals

Kriging is a method for spatial prediction that, given observations of a spatial process, gives the optimal linear predictor of the process at a new specified point. The kriging predictor may be used to define a prediction interval for the value of interest. The coverage of the prediction interval will, however, equal the nominal desired coverage only if it is constructed using the correct underlying covariance structure of the process. If this is unknown, it must be estimated from the data. We study the effect on the coverage accuracy of the prediction interval of substituting the true covariance parameters by estimators, and the effect of bootstrap calibration of coverage properties of the resulting 'plugin' interval. We demonstrate that plugin and bootstrap calibrated intervals are asymptotically accurate in some generality and that bootstrap calibration appears to have a significant effect in improving the rate of convergence of coverage error.     

Bootstrap Tests of Nonnested Hypotheses: Some Further Results

Nonnested models are sometimes tested using a simulated reference distribution for the uncentred log likelihood ratio statistic. This approach has been recommended for the specific problem of testing linear and logarithmic regression models. The general asymptotic validity of the reference distribution test under correct choice of error distributions is questioned. The asymptotic behaviour of the test under incorrect assumptions about error distributions is also examined. In order to complement these analyses, Monte Carlo results for the case of linear and logarithmic regression models are provided. The finite sample properties of several standard tests for testing these alternative functional forms are also studied, under normal and nonnormal error distributions. These regression-based variable-addition tests are implemented using asymptotic and bootstrap critical values.     

Iterated Bootstrap-t Confidence Intervals for Density Functions

Abstract.  Conventional bootstrap- t intervals for density functions based on kernel density estimators exhibit poor coverages due to failure of the bootstrap to estimate the bias correctly. The problem can be resolved by either estimating the bias explicitly or undersmoothing the kernel density estimate to undermine its bias asymptotically. The resulting bias-corrected intervals have an optimal coverage error of order arbitrarily close to second order for a sufficiently smooth density function. We investigated the effects on coverage error of both bias-corrected intervals when the nominal coverage level is calibrated by the iterated bootstrap. In either case, an asymptotic reduction of coverage error is possible provided that the bias terms are handled using an extra round of smoothed bootstrapping. Under appropriate smoothness conditions, the optimal coverage error of the iterated bootstrap- t intervals has order arbitrarily close to third order. Examples of both simulated and real data are reported to illustrate the iterated bootstrap procedures.