Assessing classification rules

Having constructed a rule for classifying objects into classes, one will need to evaluate the performance of that rule both in absolute terms (is it good enough?) and in relative terms (is it better than an alternative?). In this paper, we discuss such evaluation, focusing primarily on the first question, and covering discriminability (how effective the rule is in classifying new objects to the correct class) and reliability (how accurately it estimates probabilities of class membership). Measures based on percentages correct, measures based on probabilities of being correct and distance based measures are outlined, and attractive and problematic properties are discussed.