# 67th Place Solution - Bronze - LB 0.515

First of all, thank you to the organizers for arranging this exciting competition, and congratulations to the winners. I have learned a lot throughout this journey. The discussion section and public notebooks were quite helpful. I feel a bit regretful because I couldn't dedicate enough time due to my final exams.
My Approach
A few months back, I improved
[the public notebook](https://www.kaggle.com/code/nischaydnk/bengali-finetuning-baseline-wav2vec2-inference)
by
[@nischaydnk](https://www.kaggle.com/nischaydnk)
just by training my own
[n-gram language model](https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro)
. It improved both my public LB (0.445 -> 0.436) and private LB (0.534 -> 0.515).
Preparing the Data
I normalized the training texts using
[bnunicodenormalizer](https://github.com/mnansary/bnUnicodeNormalizer)
as well as the normalizer developed by
[CSE BUET NLP](https://github.com/csebuetnlp/normalizer)
. Some data could not be normalized because of the confusion which contained the character " ঃ" (
[বিসর্গ](https://bn.wikipedia.org/wiki/%E0%A6%AC%E0%A6%BF%E0%A6%B8%E0%A6%B0%E0%A7%8D%E0%A6%97)
) as a separate word. As it is mostly used in place of ":" (colon), I replaced them with colons instead.
![](assets/33867bdd90be47af9295c00d.png)
Training the N-gram Model
Before training the n-gram model, I removed the punctuation from the normalized text data because Wav2Vec2 does not predict any punctuation. I trained the n-gram model using
[pyctcdecode](https://github.com/kensho-technologies/pyctcdecode)
.
Solution Code
Data preparation:
[https://www.kaggle.com/code/umongsain/macro-normalization](https://www.kaggle.com/code/umongsain/macro-normalization)
Training code:
[https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro/notebook](https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro/notebook)
Inference code:
[https://www.kaggle.com/code/umongsain/67th-place-solution-inference-notebook](https://www.kaggle.com/code/umongsain/67th-place-solution-inference-notebook)
Things I Tried But Did Not Work
Later, I tried to fine-tune multiple XLS-R and MMS models from scratch. As I had limited resources, I focused on low-resource training, using only 30k samples from the Common Voice 13. Unfortunately, they could not beat my previous baseline. But what I found is that
[MMS 1B](https://huggingface.co/facebook/mms-1b)
works quite well for low-resource languages. Sadly, it is available only for non-commercial uses. The checkpoints can be found here:
[https://huggingface.co/Umong/wav2vec2-xls-r-300m-bengali](https://huggingface.co/Umong/wav2vec2-xls-r-300m-bengali)
[https://huggingface.co/Umong/wav2vec2-large-mms-1b-bengali](https://huggingface.co/Umong/wav2vec2-large-mms-1b-bengali)
Things That I Could Have Tried
I had plans to use QLoRa to fine-tune on a few out-of-domain (OOD) hand-labeled samples for domain adaptation after fine-tuning on the crowd-sourced data. I wish I had a bit more free time.
I am glad to have my 1st Bronze competition medal. Again, thanks to the organizers.