Our final solution is an ensemble of three sequence-to-sequence models — ByT5-Large, ByT5-XL, and MADLAD-400-3B-MT — all initialised from continual pretraining (CPT) checkpoints and further improved with pseudo-labeled data on published_texts.csv.
The overall pipeline consists of three stages:
\nThe final submission ensembles predictions from all three retrained models + MBR decoding.
\nTo supplement the official training data, we incorporated two external resources from Kaggle:
\nSource: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/668485
\nScanned tablet PDFs were processed with GLM-OCR (0.9B), with each page handled independently to extract Akkadian/English text pairs.
\nWe then used the Gemini, GPT, and Claude APIs for:
\nThe same alignment and cleaning pipeline was applied to the published_texts corpus before using it for pseudo-labeling.
To ensure a reliable evaluation setup, we reserved one held-out fold exclusively for validation and excluded it from all training stages.
\nWe experimented with several strong sequence-to-sequence backbones:
\nByT5 was particularly well-suited to this task: its byte-level tokenisation handles rare scripts and low-frequency transliteration patterns more robustly than standard subword tokenisation. MADLAD-400-3B-MT was included to add diversity to the final ensemble.
\nEach model was first trained for 3 epochs of continual pretraining on document-level Akkadian text, combining:
\nThis stage adapts the pretrained multilingual models to the Akkadian domain before introducing translation supervision.
\nStarting from CPT checkpoints, each model was fine-tuned at the sentence level on parallel translation data, again combining:
\nThe CPT → SFT pipeline consistently outperformed direct fine-tuning, likely because CPT first helps the model internalise the structure and distribution of the target language.
\nWe selected the best single model based on cross-validation and leaderboard performance, and used it to generate pseudo-labels for the published_texts corpus at the sentence level.
Three final models were retrained on the combined dataset — official training data, cleaned external Assyrian data, and pseudo-labeled published texts:
\n| Model | \nPublic | \nPrivate | \n
|---|---|---|
| ByT5-Large | \n37.4 | \n39.3 | \n
| ByT5-XL | \n37.6 | \n39.2 | \n
| MADLAD-400-3B-MT | \n37.3 | \n38.9 | \n
| Ensemble | \n38.5 | \n39.9 | \n
Our final solution is an ensemble of three sequence-to-sequence models — ByT5-Large, ByT5-XL, and MADLAD-400-3B-MT — all initialised from continual pretraining (CPT) checkpoints and further improved with pseudo-labeled data on published_texts.csv.
The overall pipeline consists of three stages:
\nThe final submission ensembles predictions from all three retrained models + MBR decoding.
\nTo supplement the official training data, we incorporated two external resources from Kaggle:
\nSource: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/668485
\nScanned tablet PDFs were processed with GLM-OCR (0.9B), with each page handled independently to extract Akkadian/English text pairs.
\nWe then used the Gemini, GPT, and Claude APIs for:
\nThe same alignment and cleaning pipeline was applied to the published_texts corpus before using it for pseudo-labeling.
To ensure a reliable evaluation setup, we reserved one held-out fold exclusively for validation and excluded it from all training stages.
\nWe experimented with several strong sequence-to-sequence backbones:
\nByT5 was particularly well-suited to this task: its byte-level tokenisation handles rare scripts and low-frequency transliteration patterns more robustly than standard subword tokenisation. MADLAD-400-3B-MT was included to add diversity to the final ensemble.
\nEach model was first trained for 3 epochs of continual pretraining on document-level Akkadian text, combining:
\nThis stage adapts the pretrained multilingual models to the Akkadian domain before introducing translation supervision.
\nStarting from CPT checkpoints, each model was fine-tuned at the sentence level on parallel translation data, again combining:
\nThe CPT → SFT pipeline consistently outperformed direct fine-tuning, likely because CPT first helps the model internalise the structure and distribution of the target language.
\nWe selected the best single model based on cross-validation and leaderboard performance, and used it to generate pseudo-labels for the published_texts corpus at the sentence level.
Three final models were retrained on the combined dataset — official training data, cleaned external Assyrian data, and pseudo-labeled published texts:
\n| Model | \nPublic | \nPrivate | \n
|---|---|---|
| ByT5-Large | \n37.4 | \n39.3 | \n
| ByT5-XL | \n37.6 | \n39.2 | \n
| MADLAD-400-3B-MT | \n37.3 | \n38.9 | \n
| Ensemble | \n38.5 | \n39.9 | \n
nice work @xyzdivergence .. can you please tell me how much time it took for you to byt5-xl and inference time and also the hardware you use for training ? did you use any lora setting to train byt5-xl?
", "rawMarkdown": "nice work @xyzdivergence .. can you please tell me how much time it took for you to byt5-xl and inference time and also the hardware you use for training ? did you use any lora setting to train byt5-xl?", "votes": null }, { "id": "3427748", "postDate": "03/24/2026 13:43:01", "content": "Model: byt5-xl (no lora)
\nnice work @xyzdivergence .. can you please tell me how much time it took for you to byt5-xl and inference time and also the hardware you use for training ? did you use any lora setting to train byt5-xl?
", "votes": null, "replies": [ { "id": 3427748, "author_name": "xyzdivergence", "author_url": "", "post_date": "03/24/2026 13:43:01", "content": "Model: byt5-xl (no lora)
\n