An extreme null distribution approach to the problem of paired comparisons

David (1963) and Davidson & Farquhar(1976) contain extensive bibliographies of proposed approaches to problems involving paired comparisons. However, each of the discussed methods that is based on a hypothesis test, relies heavily on the assumption that all paired comparisons are made independently. In this paper we eliminate this assumption and develop a new procedure based on an adaptation of a statistic considered by Kendall & Babington Smith (1940). We show that their original test procedure substantially underestimates the true significance level if the comparisons are not made independently. Our modification utilizes the approach developed in Costello & Wolfe (1985) for the problem of agreement between two groups of judges and relies heavily on computer-generated tables.