Evaluation Criteria
Evaluation Criteria
Metric
Systems will be evaluated based on their accuracy in predicting human preferences for CS text. This will be measured by comparing the system's ranking of generated sentences (Sent 1 vs. Sent 2) with human annotations in the CSPref dataset.

Dataset Structure
The CSPref dataset contains:

Original L1: English sentences
Original L2: Hindi, Tamil, or Malayalam sentences
Sent 1, Sent 2: Two different CS generations based on the original sentences
Chosen: Human annotation indicating the preferred sentence (Sent 1, Sent 2, or Tie)
Lang: Language pair
Evaluation Process
Public leaderboard evaluation based on English-Hindi, English-Tamil, and English-Malayalam language pairs
Private leaderboard evaluation on unseen language pairs (English-Indonesian, Indonesian-Javanese, Singlish) to assess generalization ability
Final ranking will be determined by the weighted average of performances across all language pairs