Submissions are evaluated by a mean [**Word Error Rate**](https://en.wikipedia.org/wiki/Word_error_rate), proceeding as follows:
1. The WER is computed for each instance in the test set.
2. The WERs are averaged within domains, weighted by the number of words in the sentence.
3. The (unweighted) mean of the domain averages is the final score.

This Python code computes the metric:
```python
import jiwer  # you may need to install this library

def mean_wer(solution, submission):
    joined = solution.merge(submission.rename(columns={'sentence': 'predicted'}))
    domain_scores = joined.groupby('domain').apply(
        # note that jiwer.wer computes a weighted average wer by default when given lists of strings
        lambda df: jiwer.wer(df['sentence'].to_list(), df['predicted'].to_list()),
    )
    return domain_scores.mean()

assert (solution.columns == ['id', 'domain', 'sentence']).all()
assert (submission.columns == ['id',' sentence']).all()
```

## Submission Format

The submission files should contain two columns: `id` and `sentence`.  You will need to predict the sentence for each recording in the `test/` folder. 

The submission file should contain a header and have the following format:
```
id,sentence
0f3dac00655e,এছাড়াও নিউজিল্যান্ড এ ক্রিকেট দলের হয়েও খেলছেন তিনি।
a9395e01ad21,এছাড়াও নিউজিল্যান্ড এ ক্রিকেট দলের হয়েও খেলছেন তিনি।
bf36ea8b718d,এছাড়াও নিউজিল্যান্ড এ ক্রিকেট দলের হয়েও খেলছেন তিনি।
...
```