Coherent Hypothesis Testing

Multiple hypothesis testing, an important quantitative tool to report the results of scientific inquiries, frequently leads to contradictory conclusions. For instance, in an analysis of variance (ANOVA) setting, the same dataset can lead one to reject the equality of two means, say μ₁ = μ₂, but at the same time to not reject the hypothesis that μ₁ = μ₂ = 0. These two conclusions violate the coherence principle introduced by Gabriel in 1969 Gabriel, K. R. (1969), “Simultaneous Test Procedures - Some Theory of Multiple Comparisons,” The Annals of Mathematical Statistics, 40, 224–250. Available at http://www.jstor.org/stable/2239213[Crossref] , [Google Scholar], and lead to results that are difficult to communicate, and, many times, embarrassing for practitioners of statistical methods. Although this situation is common in the daily life of statisticians, it is usually not discussed in courses of statistics. In this work, we enrich the teaching and discussion of this important topic by investigating through a few examples whether several standard test procedures are coherent or not. We also discuss the relationship between coherent tests and measures of support. Finally, we show how a Bayesian decision-theoretical framework can be used to build coherent tests. These approaches to coherence enlighten when such property is appealing in multiple testing and provide means of obtaining it.