Familywise robustness criteria for multiple-comparison procedures

A criticism of multiple-comparison procedures is that the family of inferences over which an error rate is controlled is often arbitrarily selected, yet the conclusion may depend heavily on the choice of the family. Such ambiguity is most likely in large exploratory studies requiring numerous simultaneous inferences. In ambiguous situations it is desirable that results of multiple-comparison procedures depend little on the chosen family. To assess this, we propose several familywise robustness criteria to evaluate such procedures, and we find some of their properties theoretically and by simulation. Procedures that control the false discovery rate seem to be familywise robust.