Information Criteria for the Paired-Comparisons Problem

We contrast comparisons of several treatments to control in a single experiment versus separate experiments in terms of Type I error rate and power. It is shown that if no Dunnett correction is applied in the single experiment case with relatively few treatments, the distribution of the number of Type I errors is not that different from what it would be in separate experiments with the same number of subjects in each treatment. The difference becomes more pronounced with a larger number of treatments. Extreme outcomes (either very few or very many rejections) are more likely when comparisons are made in a single experiment. When the total number of subjects is the same in a single versus separate experiments, power is generally higher in a single experiment even if a Dunnett adjustment is made.