A research participant sees some sequence of dots. Later they are asked to say what order they think the dots appeared in. We want a meaningful way to say how accurate they are.
A sequence of numbers which has no repeats and includes all numbers in a range from 1 to n is called a permutation. It can be thought of as a rearrangement of the standard sequence (1,2,3,...,n).
A metric is a way of assigning distance. It is a function of two variables which returns a number. There are a few other axioms to require, and they ensure that the function gives a meaningful notion of "distance" between things.
We have a number of different ideas for what metrics make sense to use. In fact, we could use all of them, and give some kind of composite score.
We want to score permutations the same if they differ by a cyclic rearrangement. One easy way to achieve this is, given any permutation, "rotate" it until the first term in the sequence is 1. (Start with 1, and go in the order specified by the sequence. When you get to the end, start back at the beginning until you get to the term before 1.) This can be done uniquely for every permutation.
For example, (6,2,3,1,4,5,7) would rotate to (1,4,5,7,6,2,3).
Note: Suppose n = 4. Consider the permutation given by (2,4,1,3). This would be rotated to (1,3,2,4). In both cases, there is exactly 1 correct transition, from 4 back to 1. We should be sure to count those transitions in our ranking.
Another Note: If we want to do some counting over the set of all permutations, we can speed up by only considering those which are already rotated to start with 1, and multiplying by the number of rotations (n).
Another: Some weight functions are already independent of rotation in this sense. Permutation weight and transition weight, for example.
Another: If a weight function is not independent of rotation, it could happen that different rotation has smaller weight. For example, the Hamming weight of (1,7,2,3,4,5,6) is 6, but (7,2,3,4,5,6,1) has Hamming weight 2.
Here is my understanding of the metric that Chris described. I call it the "transition metric" because it counts how many correct transitions are in the sequence.
Given a permutation a = (a, a, ..., a[n]), count the number of i such that a[i+1] is not equal to a[i] + 1. For i = n, take i+1 = 1.
This is similar in spirit to the Kendall tau below, but not quite the same.
There are (at least) 3 interesting problem domains where this kind of question comes up. Each one has a couple of standard metrics, and a whole host of modified versions for variants of the basic problems. Since none of these exactly match our setup, I propose that we just start with these.