The evaluation metric for this competition is Root Mean Squared Error (RMSE), which is defined as:
\[ RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_{i} - \hat{y_{i}})^{2} }\]
where \(\hat{y_{i}}\) is the predicted relevance score for each product-query pair, and y is the actual relevance score assigned by human raters (i.e., the ground truth).
We want to minimize RMSE. Smaller values, towards 0, indicate better performance (because RMSE is always non-negative).
Submission Format
For each id (of the product-query pair) in the test set, you must predict a relevance score. .  The file should contain a header and have the following format:
A submission file should contain
two columns
: id (of the product-query pair) and the predicted relevance score.  Columns must be separated by a comma.  The file should must ontain a header line, and should have the following format:
id,relevance
1,1
4,2
5,3
...
The relevance scores must be a real number in the range [1,3].