Implementing Monte Carlo tests with p-value buckets

Software packages usually report the results of statistical tests using p-values. Users often interpret these values by comparing them with standard thresholds, for example, 0.1, 1, and 5%, which is sometimes reinforced by a star rating (***, **, and *, respectively). We consider an arbitrary statistical test whose p-value p is not available explicitly, but can be approximated by Monte Carlo samples, for example, by bootstrap or permutation tests. The standard implementation of such tests usually draws a fixed number of samples to approximate p. However, the probability that the exact and the approximated p-value lie on different sides of a threshold (the resampling risk) can be high, particularly for p-values close to a threshold. We present a method to overcome this. We consider a finite set of user-specified intervals that cover [0, 1] and that can be overlapping. We call these p-value buckets. We present algorithms that, with arbitrarily high probability, return a p-value bucket containing p. We prove that for both a bounded resampling risk and a finite runtime, overlapping buckets need to be employed, and that our methods both bound the resampling risk and guarantee a finite runtime for such overlapping buckets. To interpret decisions with overlapping buckets, we propose an extension of the star rating system. We demonstrate that our methods are suitable for use in standard software, including for low p-value thresholds occurring in multiple testing settings, and that they can be computationally more efficient than standard implementations.     

QuickMMCTest: quick multiple Monte Carlo testing

Multiple hypothesis testing is widely used to evaluate scientific studies involving statistical tests. However, for many of these tests, p values are not available and are thus often approximated using Monte Carlo tests such as permutation tests or bootstrap tests. This article presents a simple algorithm based on Thompson Sampling to test multiple hypotheses. It works with arbitrary multiple testing procedures, in particular with step-up and step-down procedures. Its main feature is to sequentially allocate Monte Carlo effort, generating more Monte Carlo samples for tests whose decisions are so far less certain. A simulation study demonstrates that for a low computational effort, the new approach yields a higher power and a higher degree of reproducibility of its results than previously suggested methods.     

Model checks for Cox-type regression models based on optimally weighted martingale residuals

We introduce directed goodness-of-fit tests for Cox-type regression models in survival analysis. “Directed” means that one may choose against which alternatives the tests are particularly powerful. The tests are based on sums of weighted martingale residuals and their asymptotic distributions. We derive optimal tests against certain competing models which include Cox-type regression models with different covariates and/or a different link function. We report results from several simulation studies and apply our test to a real dataset.     

An omnibus CUSUM chart for monitoring time to event data

A non-parametric method is proposed for monitoring time-to-event data. A cumulative sum chart is constructed that is able to detect an unknown out-of-control state. This method exploits the absolute differences between the Kaplan–Meier estimator and the in-control distribution over specific time intervals. The efficiency of the algorithm is studied via a simulation and a real data study. The new method is also tested via the simulation study against existing methods.     

A Framework for Monte Carlo based Multiple Testing

We are concerned with a situation in which we would like to test multiple hypotheses with tests whose p‐values cannot be computed explicitly but can be approximated using Monte Carlo simulation. This scenario occurs widely in practice. We are interested in obtaining the same rejections and non‐rejections as the ones obtained if the p‐values for all hypotheses had been available. The present article introduces a framework for this scenario by providing a generic algorithm for a general multiple testing procedure. We establish conditions that guarantee that the rejections and non‐rejections obtained through Monte Carlo simulations are identical to the ones obtained with the p‐values. Our framework is applicable to a general class of step‐up and step‐down procedures, which includes many established multiple testing corrections such as the ones of Bonferroni, Holm, Sidak, Hochberg or Benjamini–Hochberg. Moreover, we show how to use our framework to improve algorithms available in the literature in such a way as to yield theoretical guarantees on their results. These modifications can easily be implemented in practice and lead to a particular way of reporting multiple testing results as three sets together with an error bound on their correctness, demonstrated exemplarily using a real biological dataset.     

Public Opinion Surveys and the Formation of Privacy Policy

The laws that condition the boundaries that separate the public from the private spheres shape our expectations of privacy. Public opinion helps to shape the development and implementation of those laws. Commercial firms in the information-intensive industries have been the primary sponsors of public opinion surveys introduced into testimony as assessments of the public's will. Representatives of business and consumer organizations have relied upon the same industry-sponsored surveys to frame their arguments in support of or in opposition to specific privacy policies. In the past 25 years, references to public opinion have been used to frame the public as concerned, differentiated and, most recently, as willing to negotiate their privacy demands.     

Perceptions of geography students in the USA and Egypt on global issues

The study aimed to compare the perceptions of Egyptian and US students on global issues. The authors developed a survey of global issues and administered it to sample of 321 Geography students in the USA and Egypt. The survey tapped five issues: global citizenship, cultural diversity, global conflicts, nuclear arms race, and global warming. The results showed that there were significant differences in Egyptian and US students’ perception of each of the global issues.     

Checking a Semiparametric Additive Risk Model

McKeague and Sasieni [A partly parametric additive risk model. Biometrika 81 (1994) 501] propose a restriction of Aalen’s additive risk model by the additional hypothesis that some of the covariates have time-independent influence on the intensity of the observed counting process. We introduce goodness-of-fit tests for this semiparametric Aalen model. The asymptotic distribution properties of the test statistics are derived by means of martingale techniques. The tests can be adjusted to detect particular alternatives. As one of the most important alternatives we consider Cox’s proportional hazards model. We present simulation studies and an application to a real data set.     

MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests

Consider testing multiple hypotheses using tests that can only be evaluated by simulation, such as permutation tests or bootstrap tests. This article introduces MMCTest , a sequential algorithm that gives, with arbitrarily high probability, the same classification as a specific multiple testing procedure applied to ideal p‐values. The method can be used with a class of multiple testing procedures that include the Benjamini and Hochberg false discovery rate procedure and the Bonferroni correction controlling the familywise error rate. One of the key features of the algorithm is that it stops sampling for all the hypotheses that can already be decided as being rejected or non‐rejected. MMCTest can be interrupted at any stage and then returns three sets of hypotheses: the rejected, the non‐rejected and the undecided hypotheses. A simulation study motivated by actual biological data shows that MMCTest is usable in practice and that, despite the additional guarantee, it can be computationally more efficient than other methods.