The aggregate [word error rate](https://en.wikipedia.org/wiki/Word_error_rate) metric will be used to evaluate decoding performance across the 1,450 held-out test sentences contained in this dataset (specifically, the `data_test.hdf5` files for each session). Word error rate is the "edit distance" between the decoded sentence and the actual sentence, computed over words (i.e., the number of substitutions, insertions and deletions of words needed to make the decoded sentence match the true sentence). The aggregate word error rate across all sentences is equal to the summed edit distance over all sentences, divided by the total number of words across all true transcriptions. "Words" are sequences of characters separated by spaces. To be counted as correct, all characters in the decoded word must match the true word (including apostrophes for contractions like "don't"). Punctuation marks (such as periods, question marks and commas) are not evaluated and should not be included in the decoded transcripts. 

The test set consists of 1,450 sentences. Not that all sessions do not include *val* or *test* partitions. 1/3 of the *test* sentences will be used for the public leaderboard and the remaining 2/3rds will be used for the private leaderboard. The public leaderboard will be visible throughout the competition, but the private leaderboard will only be revealed after the competition ends on December 31, 2025. Prizes will be based on the private leaderboard.

Your submission file should be a `.csv` file with an `id` and a `text` column. Sentences should appear in chronological order (by session date, block, and trial number) from the *test* partition, with `id` incrementing from 0 to 1449. For example, your `.csv` submission file should first have the held out *test* trials from session `t15.2023.08.13`, followed by `t15.2023.08.18`, then `t15.2023.08.20`, and so on.

See our [example evaluation script](https://github.com/Neuroprosthetics-Lab/nejm-brain-to-text/blob/main/model_training/evaluate_model.py) for how to load data, do inference with pretrained models, and create a correctly-formatted submission file. See our [example baseline submission file](https://github.com/Neuroprosthetics-Lab/nejm-brain-to-text/blob/main/model_training/rnn_baseline_submission_file_valsplit.csv) for what your decoded transcripts should look like (note that the linked example submission file corresponds to the *val* partition, not the *test* partition).

Please feel free to discuss decoding strategies and questions on the *Discussion* tab of this Kaggle competition.