Our solution is a weighted ensemble of 8 distinct pipelines, 4 using Helsinki model and 4 using byT5 base model.
\nBecause we started this competition in the last 15 days. I couldn't train byT5 base models from scratch, and relied on finetuning best publicly available models. Because Helsinki was a relatively smaller model and was faster to train, I trained Helsinki models from scratch.
\nI knew relying only on train.csv was not feasible, so I spend my initial 3 days in processing the Kultepe Tablets, and extracting line by line translations along with extensive preprocessing. I have shared the detailed pipeline here.
\nFor me this submission was ONE-SHOT. We were ranking ~550 and this was the first and last ensemble submission we could make, because we had already exhausted 4 out of 5 submissions on the last day. It was a great feeling to have clutched nearly 400 ranks in the final hours of the competition and to remain stable in silver zone in private LB as well. Happy to be joining the Master Tier!
\nThe best part of this submission was how efficient it was. We were able to run 4 Helsinki in 8 minutes and 4 byT5 in 25 minutes, making the entire ensemble complete in under 33 minutes!
\nOur final submission combined:
\nHelsinki-NLP/opus-mt models (Weights: 25.0 - 27.0)
\nhelsinki_final (Trained on gold quality data)helsinki_5_ep (Early stopped version of helsinki_final model)helsinki_all_ext (Trained on all datasets combined external+train.csv)helsinki_mt_mul (This uses mt-mul-en base whereas the other 3 used ROMANCE-en base)Max Len = 256
\nGoogle/byt5 models (Weights: 27.0 - 35.0)
\nbyt5_34_4 (Finetuned version of public final-byt5 model on gold data, Highest weight: 35.0)byt5_ext_vocab (Finetuned on all external data for vocabulary)byt5_matia_orig (Public mbr v2 model)byt5_matia_base (Finetuned version of public mbr v2 model on gold data)Max Len = 512
\nFor each of the 8 models, inference is run using Beam Search with
\nnum_beams=4num_return_sequences=4. CUDA_VISIBLE_DEVICES=0 and 1) using bash background processing to save time.We apply Minimum Bayes Risk (MBR) decoding over all candidates.
\n<gap>, ..., x, etc.)I knew gold solutions would definitely be using LLM based approaches. So, I had finetuned Qwen2.5-7B using a simple prompt similar to byT5 with unsloth, but because we had not independently checked its performance on LB, I did not feel confident about adding it to the final ensemble.
\nThis was probably our best training idea. Instead of just training on Akkadian → English translation directly, I set up a multitask framework with multiple objectives:
Each task was prefixed so the model knew what to do (translate Akkadian to English:, denoise Akkadian:, etc.). The task ratios were roughly 80% translation, 10% denoising, 10% cross-lingual denoising.
The denoising objectives acted as regularization and made the models robust. One key observation was also that MTL (Multi Task Learning) not only raised the ceiling of the CV scores, but also led to lesser variance in the results, and the training was much more smoother.
\nI applied R-Drop (Regularized Dropout) during multitask training. I ran R-Drop alongside multitask learning (Phase 0:2), and then fine-tuned the model on the competition training data (Phase 3) as a part of curriculum learning mentioned in the next section. But that final fine-tuning killed most of the improvement from R-Drop.
\nIn hindsight, the Phase 1 checkpoint alone would've been much stronger. But it was my teammate's hypothesis that we should still stick to training data for final fine tuning to make sure our models are as good locally as on test data and especially to avoid any kind of shakeups. Also, with very few submissions left, we wanted concrete improvements (even if tiny) over any experiments, so we never submitted the Phase 1 model on its own.
\nWe were amazed to see an elegant implementation of R-Drop strategy in 6th place solution, we learned something new!
\nI used a 4-phase curriculum for training, going from easy/noisy data to hard/clean data:
\nCurriculum learning led to much smoother training with virtually no overfitting, until Phase 3. That final fine-tuning on competition data was completely unnecessary, as we now realize.
\nThat hypothesis of sticking to competition provided training data for final finetuning made curriculum learning the biggest blindspot. If instead I had skipped curriculum learning completely, and sticked to just single phase multi task learning, the actual improvements we were observing would have been easily reflected and visible on LB. But I had only submitted the after phase 3 versions in almost all the experiments.
\nBecause we had less time and submissions, we had to switch on to more CV and less LB feedback but this again amplified our mistakes, because we made cross validation too based on train.csv. On the other hand, there are many top solutions which mentioned that they never even touched the training data at all.
\nHad we had enough time (and submissions left) to notice LB scores after exploratory experiments, we too would have had discarded train.csv easily. On that regard, Kudos to my teammate Paritosh, because it was his work on extracting and cleaning as useful data as possible from all the sources available, and he did his job so well that train.csv was completely unnecessary for us, even though we weren't confident enough to see it.
\nOn the final day, me and Paritosh were discussing about fine-tuning an LLM like Qwen directly. He'd already given it a shot earlier, but I decided to try again. So, I fine-tuned Qwen 7B for around 5 hours on an RTX Pro 6000. Since my multitask training setup was well configured by this point, the model showed consistent improvement: ~34 geo mean after epoch 1, ~35 after epoch 2, ~36.7 after epoch 3 — and it was still improving when I had to stop.
\nWhy stop? Because it was already 6 PM. To make sure submissions got their full 9 hours of runtime before results were declared at 5:30 AM, we had to submit everything by 8:30 PM.
\nThe predictions were really good. Some sentences were looking near perfect, and the vocabulary was brilliant. But inference on a 7B model was too slow for the Kaggle time limit, and we had no time left to figure out optimization. I now saw some top solutions also utilized models as big as 70B (10x the size I tried) but they did the quantization properly, and configured the optimization nicely, especially using tools like unsloth. So I guess its another lesson we got here, inference is not only important when you have to ship the models, but also in such competitions too.
\nArjun initially worked on the implementation of Round-Trip RL with byT5 because it utilises the bi-directional capabilities of byT5 to the max. But RL training was very unstable and led to poor results. I do believe the reward implementation could be improved and this approach can give way better results. Paper here. But, life got busy for Arjun and he could not contribute much to the team.
", "messages": [ { "id": 3428005, "postDate": "2026-03-24T19:15:39.007Z", "content": "Our solution is a weighted ensemble of 8 distinct pipelines, 4 using Helsinki model and 4 using byT5 base model.
\nBecause we started this competition in the last 15 days. I couldn't train byT5 base models from scratch, and relied on finetuning best publicly available models. Because Helsinki was a relatively smaller model and was faster to train, I trained Helsinki models from scratch.
\nI knew relying only on train.csv was not feasible, so I spend my initial 3 days in processing the Kultepe Tablets, and extracting line by line translations along with extensive preprocessing. I have shared the detailed pipeline here.
\nFor me this submission was ONE-SHOT. We were ranking ~550 and this was the first and last ensemble submission we could make, because we had already exhausted 4 out of 5 submissions on the last day. It was a great feeling to have clutched nearly 400 ranks in the final hours of the competition and to remain stable in silver zone in private LB as well. Happy to be joining the Master Tier!
\nThe best part of this submission was how efficient it was. We were able to run 4 Helsinki in 8 minutes and 4 byT5 in 25 minutes, making the entire ensemble complete in under 33 minutes!
\nOur final submission combined:
\nHelsinki-NLP/opus-mt models (Weights: 25.0 - 27.0)
\nhelsinki_final (Trained on gold quality data)helsinki_5_ep (Early stopped version of helsinki_final model)helsinki_all_ext (Trained on all datasets combined external+train.csv)helsinki_mt_mul (This uses mt-mul-en base whereas the other 3 used ROMANCE-en base)Max Len = 256
\nGoogle/byt5 models (Weights: 27.0 - 35.0)
\nbyt5_34_4 (Finetuned version of public final-byt5 model on gold data, Highest weight: 35.0)byt5_ext_vocab (Finetuned on all external data for vocabulary)byt5_matia_orig (Public mbr v2 model)byt5_matia_base (Finetuned version of public mbr v2 model on gold data)Max Len = 512
\nFor each of the 8 models, inference is run using Beam Search with
\nnum_beams=4num_return_sequences=4. CUDA_VISIBLE_DEVICES=0 and 1) using bash background processing to save time.We apply Minimum Bayes Risk (MBR) decoding over all candidates.
\n<gap>, ..., x, etc.)I knew gold solutions would definitely be using LLM based approaches. So, I had finetuned Qwen2.5-7B using a simple prompt similar to byT5 with unsloth, but because we had not independently checked its performance on LB, I did not feel confident about adding it to the final ensemble.
\nThis was probably our best training idea. Instead of just training on Akkadian → English translation directly, I set up a multitask framework with multiple objectives:
Each task was prefixed so the model knew what to do (translate Akkadian to English:, denoise Akkadian:, etc.). The task ratios were roughly 80% translation, 10% denoising, 10% cross-lingual denoising.
The denoising objectives acted as regularization and made the models robust. One key observation was also that MTL (Multi Task Learning) not only raised the ceiling of the CV scores, but also led to lesser variance in the results, and the training was much more smoother.
\nI applied R-Drop (Regularized Dropout) during multitask training. I ran R-Drop alongside multitask learning (Phase 0:2), and then fine-tuned the model on the competition training data (Phase 3) as a part of curriculum learning mentioned in the next section. But that final fine-tuning killed most of the improvement from R-Drop.
\nIn hindsight, the Phase 1 checkpoint alone would've been much stronger. But it was my teammate's hypothesis that we should still stick to training data for final fine tuning to make sure our models are as good locally as on test data and especially to avoid any kind of shakeups. Also, with very few submissions left, we wanted concrete improvements (even if tiny) over any experiments, so we never submitted the Phase 1 model on its own.
\nWe were amazed to see an elegant implementation of R-Drop strategy in 6th place solution, we learned something new!
\nI used a 4-phase curriculum for training, going from easy/noisy data to hard/clean data:
\nCurriculum learning led to much smoother training with virtually no overfitting, until Phase 3. That final fine-tuning on competition data was completely unnecessary, as we now realize.
\nThat hypothesis of sticking to competition provided training data for final finetuning made curriculum learning the biggest blindspot. If instead I had skipped curriculum learning completely, and sticked to just single phase multi task learning, the actual improvements we were observing would have been easily reflected and visible on LB. But I had only submitted the after phase 3 versions in almost all the experiments.
\nBecause we had less time and submissions, we had to switch on to more CV and less LB feedback but this again amplified our mistakes, because we made cross validation too based on train.csv. On the other hand, there are many top solutions which mentioned that they never even touched the training data at all.
\nHad we had enough time (and submissions left) to notice LB scores after exploratory experiments, we too would have had discarded train.csv easily. On that regard, Kudos to my teammate Paritosh, because it was his work on extracting and cleaning as useful data as possible from all the sources available, and he did his job so well that train.csv was completely unnecessary for us, even though we weren't confident enough to see it.
\nOn the final day, me and Paritosh were discussing about fine-tuning an LLM like Qwen directly. He'd already given it a shot earlier, but I decided to try again. So, I fine-tuned Qwen 7B for around 5 hours on an RTX Pro 6000. Since my multitask training setup was well configured by this point, the model showed consistent improvement: ~34 geo mean after epoch 1, ~35 after epoch 2, ~36.7 after epoch 3 — and it was still improving when I had to stop.
\nWhy stop? Because it was already 6 PM. To make sure submissions got their full 9 hours of runtime before results were declared at 5:30 AM, we had to submit everything by 8:30 PM.
\nThe predictions were really good. Some sentences were looking near perfect, and the vocabulary was brilliant. But inference on a 7B model was too slow for the Kaggle time limit, and we had no time left to figure out optimization. I now saw some top solutions also utilized models as big as 70B (10x the size I tried) but they did the quantization properly, and configured the optimization nicely, especially using tools like unsloth. So I guess its another lesson we got here, inference is not only important when you have to ship the models, but also in such competitions too.
\nArjun initially worked on the implementation of Round-Trip RL with byT5 because it utilises the bi-directional capabilities of byT5 to the max. But RL training was very unstable and led to poor results. I do believe the reward implementation could be improved and this approach can give way better results. Paper here. But, life got busy for Arjun and he could not contribute much to the team.
", "rawMarkdown": "## 1. Final Solution (Paritosh's Part)\nOur solution is a weighted ensemble of 8 distinct pipelines, 4 using Helsinki model and 4 using byT5 base model.\n\nBecause we started this competition in the last 15 days. I couldn't train byT5 base models from scratch, and relied on finetuning best publicly available models. Because Helsinki was a relatively smaller model and was faster to train, I trained Helsinki models from scratch.\n\nI knew relying only on train.csv was not feasible, so I spend my initial 3 days in processing the Kultepe Tablets, and extracting line by line translations along with extensive preprocessing. I have shared the detailed pipeline [here.](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/684249)\n\nFor me this submission was ONE-SHOT. We were ranking ~550 and this was the first and last ensemble submission we could make, because we had already exhausted 4 out of 5 submissions on the last day. It was a great feeling to have clutched nearly 400 ranks in the final hours of the competition and to remain stable in silver zone in private LB as well. Happy to be joining the **Master** Tier!\n\nThe best part of this submission was how efficient it was. We were able to run 4 Helsinki in 8 minutes and 4 byT5 in 25 minutes, making the entire ensemble complete in under 33 minutes!\n\nOur final submission combined:\n\n**Helsinki-NLP/opus-mt models (Weights: 25.0 - 27.0)**\n* `helsinki_final` (Trained on gold quality data)\n* `helsinki_5_ep` (Early stopped version of helsinki_final model)\n* `helsinki_all_ext` (Trained on all datasets combined external+train.csv)\n* `helsinki_mt_mul` (This uses mt-mul-en base whereas the other 3 used ROMANCE-en base)\n\nMax Len = 256\n\n**Google/byt5 models (Weights: 27.0 - 35.0)**\n* `byt5_34_4` (Finetuned version of public final-byt5 model on gold data, Highest weight: 35.0)\n* `byt5_ext_vocab` (Finetuned on all external data for vocabulary)\n* `byt5_matia_orig` (Public mbr v2 model)\n* `byt5_matia_base` (Finetuned version of public mbr v2 model on gold data)\n\nMax Len = 512\n\n---\n\n## 2. Inference & Candidate Generation\nFor each of the 8 models, inference is run using **Beam Search** with\n- `num_beams=4`\n- `num_return_sequences=4`. \n* This generates exactly 4 translation candidates per model for every single input text.\n* **Total Pool:** 8 models × 4 candidates = 32 candidate translations per row.\n* **Hardware Optimization:** Inference scripts are run concurrently across 2 GPUs (`CUDA_VISIBLE_DEVICES=0` and `1`) using bash background processing to save time.\n\n---\n\n## 3. Final submission.csv generation\nWe apply **Minimum Bayes Risk (MBR)** decoding over all candidates.\n\n### Steps:\n\n1. **Normalize gap tokens**: Standardize missing-text markers (`