Each day, your team is able to submit up to 5 agents (bots) to the competition. Each submission will play episodes (games) against other bots on the ladder that have a similar skill rating. Over time, skill ratings will go up with wins or down with losses. Every bot submitted will continue to play games until the end of the competition. On the leaderboard only your best scoring bot will be shown, but you can track the progress of all of your submissions on your Submissions page.

Each Submission has an estimated Skill Rating which is modeled by a Gaussian N(μ,σ2) where μ is the estimated skill and σ represents our uncertainty of that estimate which will decrease over time.

When you upload a Submission, we first play a Validation Episode where that Submission plays against a copy of itself to make sure it works properly. If the Episode fails, the Submission is marked as Error. Otherwise, we initialize the Submission with μ0=600 and it joins the pool of All Submissions for ongoing evaluation.

We repeatedly run Episodes from the pool of All Submissions and try to pick Submissions with similar ratings for fair matches. We aim to run ~8 Episodes a day per Submission with an additional slight rate increase for the newest-submitted Episodes to give you feedback faster.

After an Episode finishes, we'll update the Rating estimate of both agents in that Episode. If one agent won, we'll increase its μ and decrease its opponent's μ -- if the result was a draw, then we'll move the two μ values closer towards their mean. The updates will have magnitude relative to the deviation from the expected result based on the previous μ values, and also relative to each Submission's uncertainty σ. We also reduce the σ terms relative to the amount of information gained by the result. The score by which your bot wins or loses an Episode does not affect the skill rating updates.

At the submission deadline, additional submissions will be locked. One additional week will be allotted to continue to run games. At the conclusion of this week, the leaderboard is final.