Comparisons for agreement
Let’s say we have data that is only rank order from two or more evaluators (people, algorithms, etc.) and we want to determine if the evaluators agree or not.
Agreement here meaning the results from one person or another are in agreement, or they are concordant. This is typically done with this non-parametric method for 3 or more evaluators. For a comparison of two evaluators consider using Cohen’s Kappa or Spearman’s correlation coefficient as they are more appropriate.
To use an example, let’s ask three people to rank order ten popular movies. 1 being the least favorite and 10 being the favorite of the list. Here’s the data from evaluator’s A, B, and C:
These three had perfect agreement we wouldn’t need to evaluate if they agreed. So, the question is, do they agree well enough to conclude they tend to like the same movies or not?
where, i is the individual items being ranked, in this case 1 through 10. Basically tally up the scores from each evaluator for each item.
m is the number of evaluators, in this case 3.
Therefore, we find
n is the number of items being ranked, in this case 10.
R̄ = 16.5.
Compute S, sum of squared deviations
S = 320.5
This is the tally of the squared differences between the the sum of the three scores for each movie and the overall average rank.
Compute Kendall’s coefficient of concordance, W
W is determined with
W will be between zero and one. Values close to zero imply no agreement and W values closer to one imply agreement.
Working out the example we find
W = 0.432
Compute the test statistic
The test statistic, T.S., comes from the data and is compared to the critical value to determine if there is concordance or not.
In the example, this turns out to be 11.654
Compute critical value
With 10 items, n = 10, and since degrees of freedom, df, is one less then n, we have df = n-1 and in this case df = 9.
We used a chi-squared, χ2 with confidence of 1 – α, and df = n – 1.
For n > 7 use the χ2 table, for n < 7 use the direct probability from table of critical values in your statistics book (one example is Siegel S and Castellan Jr. N.J. Nonparametric Statistics for the Behavioral Sciences (1988) International Edition. McGraw-Hill Book Company New York. ISBN 0-07-057357-3 Table T. Critical values for Kendall coefficient of concordance W p. 365)
In our example, χ20.10, 9 = 14.684.
Compare W to test statistic
Ho: There is no convincing evidence of agreement if T.S. < the critical value
Ha: There is convincing evidence of agreement if T.S. > the critical value
In our example, the T.S. = 11.654 and the critical value is 14.684, therefore the T.S. is less than the critical value and we conclude that while there appears to become agreement it is not sufficient to conclude the evaluates agree.