Assessing different uncertainty measures of EBLUP: a resampling-based approach

The empirical best linear unbiased prediction approach is a popular method for the estimation of small area parameters. However, the estimation of reliable mean squared prediction error (MSPE) of the estimated best linear unbiased predictors (EBLUP) is a complicated process. In this paper we study the use of resampling methods for MSPE estimation of the EBLUP. A cross-sectional and time-series stationary small area model is used to provide estimates in small areas. Under this model, a parametric bootstrap procedure and a weighted jackknife method are introduced. A Monte Carlo simulation study is conducted in order to compare the performance of different resampling-based measures of uncertainty of the EBLUP with the analytical approximation. Our empirical results show that the proposed resampling-based approaches performed better than the analytical approximation in several situations, although in some cases they tend to underestimate the true MSPE of the EBLUP in a higher number of small areas.     

Estimation of the Distribution Function of a Standardized Statistic

For estimating the distribution of a standardized statistic, the bootstrap estimate is known to be local asymptotic minimax. Various computational techniques have been developed to improve on the simulation efficiency of uniform resampling, the standard Monte Carlo approach to approximating the bootstrap estimate. Two new approaches are proposed which give accurate yet simple approximations to the bootstrap estimate. The second of the approaches even improves the convergence rate of the simulation error. A simulation study examines the performance of these two approaches in comparison with other modified bootstrap estimates.     

A weighted bootstrap approach to bootstrap iteration

The operation of resampling from a bootstrap resample, encountered in applications of the double bootstrap, maybe viewed as resampling directly from the sample but using probability weights that are proportional to the numbers of times that sample values appear in the resample. This suggests an approximate approach to double-bootstrap Monte Carlo simulation, where weighted bootstrap methods are used to circumvent much of the labour involved in compounded Monte Carlo approximation. In the case of distribution estimation or, equivalently, confidence interval calibration, the new method may be used to reduce the computational labour. Moreover, the method produces the same order of magnitude of coverage error for confidence intervals, or level error for hypothesis tests, as a full application of the double bootstrap.     

Adaptive resampling algorithms for estimating bootstrap distributions

Based on recent developments in the field of operations research, we propose two adaptive resampling algorithms for estimating bootstrap distributions. One algorithm applies the principle of the recently proposed cross-entropy (CE) method for rare event simulation, and does not require calculation of the resampling probability weights via numerical optimization methods (e.g., Newton's method), whereas the other algorithm can be viewed as a multi-stage extension of the classical two-step variance minimization approach. The two algorithms can be easily used as part of a general algorithm for Monte Carlo calculation of bootstrap confidence intervals and tests, and are especially useful in estimating rare event probabilities. We analyze theoretical properties of both algorithms in an idealized setting and carry out simulation studies to demonstrate their performance. Empirical results on both one-sample and two-sample problems as well as a real survival data set show that the proposed algorithms are not only superior to traditional approaches, but may also provide more than an order of magnitude of computational efficiency gains.     

On the Bootstrap and Monotone Likelihood in the Cox Proportional Hazards Regression Model

Recent literature has provided encouragement for using the bootstrap for inference on regression parameters in the Cox proportional hazards (PH) model. However, generating and performing the necessary partial likelihood computations on multitudinous bootstrap samples greatly increases the chances of incurring problems with monotone likelihood at some point in the analysis. The only symptom of monotone likelihood may be a failure to converge in the numerical maximization procedure, and so the problem might naively be dismissed by deleting the offending data set and replacing it with a new one. This strategy is shown to lead to potentially high selection biases in the subsequent summary statistics. This note discusses the importance of keeping track of these monotone likelihood cases and provides recommendations for their use in interpreting bootstrap findings, and for avoiding unwanted biases that may result from high rates of occurrence. In many cases, high monotone likelihood rates indicate that a more highly-specified model may be preferred. Special consideration is given to the problem of high monotone likelihood incidence in Monte Carlo studies of the bootstrap.     

Finite-sample properties of the bootstrap estimator in a Markov-switching model

The size distortion problem is clearly indicative of the small-sample approximation in the Markov-switching regression model. This paper shows that the bootstrap procedure can relieve the effects that this problem has. Our Monte Carlo simulation results reveal that the bootstrap maximum likelihood asymptotic approximations to the distribution can often be good when the sample size is small.     

Monte Carlo Approximation of Bootstrap Variances

It is widely believed that the number of resamples required for bootstrap variance estimation is relatively small An argument based on the unconditional coefficient of variation of the Monte Carlo approximation, suggests that as few as 25 resamples will give reasonable results. In this article we argue that the number of resamples should, in fact, be determined by the conditional coefficient of variation, involving only resampling variability. Our conditional analysis is founded on a belief that Monte Carlo error should not be allowed to determine the conclusions of a statistical analysis and indicates that approximately 800 resamples are required for this purpose. The argument can be generalized to the multivariate setting and a simple formula is given for determining a lower bound on the number of resamples required to approximate an m-dimensional bootstrap variance-covariance matrix.     

A Fast Iterated Bootstrap Procedure for Approximating the Small-Sample Bias

This article proposes a fast approximation for the small sample bias correction of the iterated bootstrap. The approximation adapts existing fast approximation techniques of the bootstrap p-value and quantile functions to the problem of estimating the bias function. We show an optimality result which holds under general conditions not requiring an asymptotic pivot. Monte Carlo evidence, from the linear instrumental variable model and the nonlinear GMM, suggest that in addition to its computational appeal and success in reducing the mean and median bias in identified models, the fast approximation provides scope for bias reduction in weakly identified configurations.     

Smoothed bootstrap bandwidth selection for nonparametric hazard rate estimation

A smoothed bootstrap method is presented for the purpose of bandwidth selection in nonparametric hazard rate estimation for iid data. In this context, two new bootstrap bandwidth selectors are established based on the exact expression of the bootstrap version of the mean integrated squared error of some approximations of the kernel hazard rate estimator. This is very useful since Monte Carlo approximation is no longer needed for the implementation of the two bootstrap selectors. A simulation study is carried out in order to show the empirical performance of the new bootstrap bandwidths and to compare them with other existing selectors. The methods are illustrated by applying them to a diabetes data set.     

Monte Carlo Approximations of the Quantiles of a Sample Statistic

We consider in this article the problem of numerically approximating the quantiles of a sample statistic for a given population, a problem of interest in many applications, such as bootstrap confidence intervals. The proposed Monte Carlo method can be routinely applied to handle complex problems that lack analytical results. Furthermore, the method yields estimates of the quantiles of a sample statistic of any sample size though Monte Carlo simulations for only two optimally selected sample sizes are needed. An analysis of the Monte Carlo design is performed to obtain the optimal choices of these two sample sizes and the number of simulated samples required for each sample size. Theoretical results are presented for the bias and variance of the numerical method proposed. The results developed are illustrated via simulation studies for the classical problem of estimating a bivariate linear structural relationship. It is seen that the size of the simulated samples used in the Monte Carlo method does not have to be very large and the method provides a better approximation to quantiles than those based on an asymptotic normal theory for skewed sampling distributions.     

Importance of interpolation when constructing double-bootstrap confidence intervals

We show that, in the context of double-bootstrap confidence intervals, linear interpolation at the second level of the double bootstrap can reduce the simulation error component of coverage error by an order of magnitude. Intervals that are indistinguishable in terms of coverage error with theoretical, infinite simulation, double-bootstrap confidence intervals may be obtained at substantially less computational expense than by using the standard Monte Carlo approximation method. The intervals retain the simplicity of uniform bootstrap sampling and require no special analysis or computational techniques. Interpolation at the first level of the double bootstrap is shown to have a relatively minor effect on the simulation error.     

Exact Bootstrap Variances of the Area Under ROC Curve

The area under the Receiver Operating Characteristic (ROC) curve (AUC) and related summary indices are widely used for assessment of accuracy of an individual and comparison of performances of several diagnostic systems in many areas including studies of human perception, decision making, and the regulatory approval process for new diagnostic technologies. Many investigators have suggested implementing the bootstrap approach to estimate variability of AUC-based indices. Corresponding bootstrap quantities are typically estimated by sampling a bootstrap distribution. Such a process, frequently termed Monte Carlo bootstrap, is often computationally burdensome and imposes an additional sampling error on the resulting estimates. In this article, we demonstrate that the exact or ideal (sampling error free) bootstrap variances of the nonparametric estimator of AUC can be computed directly, i.e., avoiding resampling of the original data, and we develop easy-to-use formulas to compute them. We derive the formulas for the variances of the AUC corresponding to a single given or random reader, and to the average over several given or randomly selected readers. The derived formulas provide an algorithm for computing the ideal bootstrap variances exactly and hence improve many bootstrap methods proposed earlier for analyzing AUCs by eliminating the sampling error and sometimes burdensome computations associated with a Monte Carlo (MC) approximation. In addition, the availability of closed-form solutions provides the potential for an analytical assessment of the properties of bootstrap variance estimators. Applications of the proposed method are shown on two experimentally ascertained datasets that illustrate settings commonly encountered in diagnostic imaging. In the context of the two examples we also demonstrate the magnitude of the effect of the sampling error of the MC estimators on the resulting inferences.     

Edgeworth Corrections for Realized Volatility

The quality of the asymptotic normality of realized volatility can be poor if sampling does not occur at very high frequencies. In this article we consider an alternative approximation to the finite sample distribution of realized volatility based on Edgeworth expansions. In particular, we show how confidence intervals for integrated volatility can be constructed using these Edgeworth expansions. The Monte Carlo study we conduct shows that the intervals based on the Edgeworth corrections have improved properties relatively to the conventional intervals based on the normal approximation. Contrary to the bootstrap, the Edgeworth approach is an analytical approach that is easily implemented, without requiring any resampling of one's data. A comparison between the bootstrap and the Edgeworth expansion shows that the bootstrap outperforms the Edgeworth corrected intervals. Thus, if we are willing to incur in the additional computational cost involved in computing bootstrap intervals, these are preferred over the Edgeworth intervals. Nevertheless, if we are not willing to incur in this additional cost, our results suggest that Edgeworth corrected intervals should replace the conventional intervals based on the first order normal approximation.     

Estimation Under Purposive Sampling

Purposive sampling is described as a random selection of sampling units within the segment of the population with the most information on the characteristic of interest. Nonparametric bootstrap is proposed in estimating location parameters and the corresponding variances. An estimate of bias and a measure of variance of the point estimate are computed using the Monte Carlo method. The bootstrap estimator of the population mean is efficient and consistent in the homogeneous, heterogeneous, and two-segment populations simulated. The design-unbiased approximation of the standard error estimate differs substantially from the bootstrap estimate in severely heterogeneous and positively skewed populations.     

The effect of Monte Carlo approximation on coverage error of double-bootstrap confidence intervals

A double-bootstrap confidence interval must usually be approximated by a Monte Carlo simulation, consisting of two nested levels of bootstrap sampling. We provide an analysis of the coverage accuracy of the interval which takes account of both the inherent bootstrap and Monte Carlo errors. The analysis shows that, by a suitable choice of the number of resamples drawn at the inner level of bootstrap sampling, we can reduce the order of coverage error. We consider also the effects of performing a finite Monte Carlo simulation on the mean length and variability of length of two-sided intervals. An adaptive procedure is presented for the choice of the number of inner level resamples. The effectiveness of the procedure is illustrated through a small simulation study.     

Analytical approximations for iterated bootstrap confidence intervals

Standard algorithms for the construction of iterated bootstrap confidence intervals are computationally very demanding, requiring nested levels of bootstrap resampling. We propose an alternative approach to constructing double bootstrap confidence intervals that involves replacing the inner level of resampling by an analytical approximation. This approximation is based on saddlepoint methods and a tail probability approximation of DiCiccio and Martin (1991). Our technique significantly reduces the computational expense of iterated bootstrap calculations. A formal algorithm for the construction of our approximate iterated bootstrap confidence intervals is presented, and some crucial practical issues arising in its implementation are discussed. Our procedure is illustrated in the case of constructing confidence intervals for ratios of means using both real and simulated data. We repeat an experiment of Schenker (1985) involving the construction of bootstrap confidence intervals for a variance and demonstrate that our technique makes feasible the construction of accurate bootstrap confidence intervals in that context. Finally, we investigate the use of our technique in a more complex setting, that of constructing confidence intervals for a correlation coefficient.     

Bootstrap LR tests of stationarity,common trends and cointegration

This paper considers the likelihood ratio (LR) tests of stationarity, common trends and cointegration for multivariate time series. As the distribution of these tests is not known, a bootstrap version is proposed via a state- space representation. The bootstrap samples are obtained from the Kalman filter innovations under the null hypothesis. Monte Carlo simulations for the Gaussian univariate random walk plus noise model show that the bootstrap LR test achieves higher power for medium-sized deviations from the null hypothesis than a locally optimal and one-sided Lagrange Multiplier (LM) test that has a known asymptotic distribution. The power gains of the bootstrap LR test are significantly larger for testing the hypothesis of common trends and cointegration in multivariate time series, as the alternative asymptotic procedure – obtained as an extension of the LM test of stationarity – does not possess properties of optimality. Finally, it is shown that the (pseudo-)LR tests maintain good size and power properties also for the non-Gaussian series. An empirical illustration is provided.     

On a new test for autocorrelation in regression models under nonnormality

A new test for autocorrelation in a general regression model under departures from the assumption of normality is derived by applying a beta distribution and bootstrap approximation, Critical values of the test, can be computed for each given design matrix, irrespective of the form of the underlying error distribution, Monte Carlo simulations are conducted in order to illustrate the performance of the test. Among others, it. is found that the suggested test is more robust and far more powerful than existing nonparametric tests.     

Bayesian and Classical Approaches for Hypotheses Testing in Trisomies

We introduce classical approaches for testing hypotheses on the meiosis I non disjunction fraction in trisomies, such as the likelihood-ratio, bootstrap, and Monte Carlo procedures. To calculate the p-values for the bootstrap and Monte Carlo procedures, different transformations in the data are considered. Bootstrap confidence intervals are also used as a tool to perform hypotheses tests. A Monte Carlo study is carried out to compare the proposed test procedures with two Bayesian ones: Jeffreys and Pereira-Stern tests. The results show that the likelihood-ratio and the Bayesian tests present the best performance. Down syndrome data are analyzed to illustrate the procedures.     

Using the Bootstrap to Test for Symmetry Under Unknown Dependence

This article considers tests for symmetry of the one-dimensional marginal distribution of fractionally integrated processes. The tests are implemented by using an autoregressive sieve bootstrap approximation to the null sampling distribution of the relevant test statistics. The sieve bootstrap allows inference on symmetry to be carried out without knowledge of either the memory parameter of the data or of the appropriate norming factor for the test statistic and its asymptotic distribution. The small-sample properties of the proposed method are examined by means of Monte Carlo experiments, and applications to real-world data are also presented.