Simple methods for evaluating and comparing binary experiments

We consider a confidence parametrization of binary information sources in terms of appropriate likelihood ratios. This parametrization is used for Bayesian belief updates and for the equivalent comparison of binary experiments. In contrast to the standard parametrization of a binary information source in terms of its specificity and its sensitivity, one of the two confidence parameters is sufficient for a Bayesian belief update conditional on a signal realization. We introduce a confidence-augmented receiver operating characteristic for comparisons of binary experiments for a class of “balanced” decision problems, relative to which the confidence order offers a higher resolution than Blackwell’s informativeness order.

Where observation is concerned, Chance favors only the prepared mind. —Louis Pasteur (1822–1895).