### Evaluation 

We use **Weighted Categorization Accuracy** to evaluate your performance. Weighted categorization accuracy can be generally defined as:

$$ WeightedAccuracy(y, \hat{y}, \omega) = \frac{1}{n} \displaystyle{\sum_{i=1}^{n}} \frac{\omega_i(y_i=\hat{y}_i)}{\sum \omega_i} $$

where \\(y\\) are ground truths, \\(\hat{y}\\) are the predicted results, and \\(\omega_i\\) is the weight associated with the \\(i\\)th item in the dataset.

In our test set, we assign each testing item a weight according to its category. The weights of the three categories, agreed, disagreed and unrelated are \\(\frac{1}{15}\\), \\(\frac{1}{5}\\), \\(\frac{1}{16}\\), respectively. We set the weights in consideration of the imbalance of the data distribution to minimize the bias to your performance caused by the majority class (unrelated pairs accounts for approximately 70% of the dataset).