Effect size measure used with rank order judging

In general, effect sizes measure the number of standard deviation the true population value of interest falls from the value that would be true if chance alone were at work. The standard deviation used is for one subject, trial, etc., rather than being the standard error of the sample statistic used in the hypothesis test.

In rank-order judging, let R be the rank for one trial. If the number of possible choices is N, then we find:

E(R) = (N + 1)/2

and

Var(R) = (N² - 1)/12.

Therefore, when N = 5, we find E(Ri) = 3 and Var(Ri) = 2. The effect size is therefore:

Effect Size = (3.0 - Average Rank)/√2.