Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). In the event that there is less agreement between the raters than expected by chance, the metric may go below 0. The quadratic weighted kappa is calculated between the scores assigned by the human rater and the predicted scores.
Results have 4 possible ratings, 1,2,3,4.  Each search record is characterized by a tuple
(e
a
,
e
b
)
, which corresponds to its scores by
Rater A
(human) and
Rater B
(predicted).  The quadratic weighted kappa is calculated as follows. First, an N x N histogram matrix
O
is constructed, such that
O
i,j
corresponds to the number of search records that received a rating
i
by
A
and a rating
j
by
B
. An
N-by-N
matrix of weights,
w
, is calculated based on the difference between raters' scores:
$$w_{i,j} = \frac{\left(i-j\right)^2}{\left(N-1\right)^2}$$
An
N-by-N
histogram matrix of expected ratings,
E
, is calculated, assuming that there is no correlation between rating scores.  This is calculated as the outer product between each rater's histogram vector of ratings, normalized such that
E
and
O
have the same sum.
From these three matrices, the quadratic weighted kappa is calculated as:
$$\kappa=1-\frac{\sum_{i,j}w_{i,j}O_{i,j}}{\sum_{i,j}w_{i,j}E_{i,j}}.$$
Submission Format
You must submit a csv file with the product id and a predicted search relevance for each search record. The order of the rows does not matter. The file must have a header and should look like the following:
id,prediction
1,3
3,2
4,1
5,4
etc..