Thank you Kaggle for hosting the competition. Special thanks to our host @deeppast for proactively clarifying our queries and helping in the release of revised higher quality datasets, which made a real difference. Lastly, I would like to thank my teammate @aman1391 for forcefully inviting me to join this competition (please continue doing so 😆).
\nIf we're being honest, the first few weeks were humbling. We experimented with focal loss, annotation tricks, and various architectural changes; yet the LB barely moved. The breakthrough came when we revisited the host's discussion on data quality, and it crystallized something we had been overlooking: this competition was never about clever models. It was about DATA. The moment we shifted our entire focus to building robust OCR pipelines, curating clean parallel text from academic publications, and generating high-quality pseudo-labels — everything changed. Sometimes the most impactful insight is the one hiding in plain sight.
\nWe trained ByT5 and Qwen3 models on competition data, PDF data, and pseudo-labelled (PL) data created by a Gemma3-27B model trained on competition and PDF data. We created PDF data iteratively and trained our models. Our final submission consisted of an MBR ensemble of ByT5 Small, ByT5-Base, Qwen3-14B, and Qwen3-4B-Instruct models using sacrebleu.metrics.CHRF(word_order=2).sentence_score as MBR metric. We used beam search for ByT5s and temperature sampling for Qwen models.
We leveraged zai-org/GLM-OCR to extract text from the PDF collection shared by the host. We ran OCR in two ways:
We then leverage models hosted on Groq to generate aligned Akkadian-English data. We align the text data to belong to the full letter and use Kimi K2 to get the parallel data. Example prompt:
# Context\nYou are given a Akkadian-Engligh texts in 2 formats.\n\n## Format 1: Interleaved Akkadian and English\n- Akkadian and English interleaved line-by-line\n- Extracted from cropping each line of text using Tesseract\n- Useful for understanding alignment\n- Some of the Akkadian Text might be noisy, refer to the Format 2 to understand the rendering\n\n## Format 2: \n- Akkadian text is clustered, followed by English\n- Extracted by OCRing the entire page of text\n- Akkadian text is better rendered, though alignment of Akkadian and English is not present\n\n# Role\n\nYour task is to extract Akkadian-English parallel data, understand alignment from `Format 1` and text rendering from `Format 2`.\nThe portion of Akkadian Text in `Format 1` must be followed in the parallel data as well i.e. **Do Not** merge consecutive Akkadian lines of text in `Format 1` into a single piece of text\nThe English translations need not be formal, complete or grammatically correct.\n\n**Remember**\n\n- Don't hallucinate or generate your own translations, strictly pick text from the given context\n- Don't include XMLs in Akkadian text (like <sub>4</sub>), render them as characters\n- The English translations need not be full sentences\nWe used different prompts for different PDF depending on the letter layout. For example, some Turkish translation PDFs had translations with line markers at the end.
\n# Context\nYou are given a Akkadian-Turkish text.\n\n- Akkadian text is clustered, followed by it's translation in Turkish\n- Extracted by OCRing the entire page of text\n\n\n# Role\n\nYour task is to extract Akkadian-Turkish-English parallel data, Akkadian and Turkish are present in the text, the Turkish text needs to be translated to English\nThe Akkadian and Turkish text alignment in present, you need to detect it via line numbers (eg. \"1-5)\" -> Turkish of Akkdian text from lines 1-5 )\nThe English translations need not be formal, complete or grammatically correct.\n\n**Remember**\n\n- Strictly pick Akkadian and Turkish text from the given context and correctly translated Turkish into English\n- Don't include XMLs in Akkadian text (like <sub>4</sub>), render them as characters\n- The English translations need not be full sentences\n- Only extract cases where you are confident of the aligned translation\nAKT-6 series PDFs were not easy to extract, and the above approaches did not work. A multi-modal approach with meta-llama/llama-4-maverick-17b-128e-instruct did seem to work, but the model was deprecated before we could process the data (PDF page images were super-resolutioned using a light-weight model). Finally, we used our extracted OCR text with Claude to process them.
We held-out 10% of the competition data to track our cv.
\nWe used Claude extensively to prepare our data preprocessing code as per the guidelines shared by the host, building cleaning pipelines that handled everything from LaTeX artifacts, superscript/subscript normalization, OCR character fixes (İ→I, ḥ→h, ß→š), Sumerogram expansion (KÙ.B.→KÙ.BABBAR), gap synchronization between transliteration and translation, and filtering misaligned Turkish/English rows. We also built a name correction postprocessor using lookup dictionaries that standardized proper name spellings across model outputs (e.g., ensuring \"Šalim-Aššur\" was consistently rendered regardless of which model produced it). Beyond preprocessing, we used Claude to get quality assessments of every OCR-extracted PDF; checking charset compliance, keyword alignment percentages, and identifying which sources were clean enough to train on and which needed filtering.
\nOur initial focus was to maximize the performance of ByT5 and Qwen3-4B based on the competition data and our processing. Once our PDF extraction pipeline was in shape, we extracted all PDFs iteratively and trained our models periodically. We extracted ~31k samples from PDFs. We also found Gemma3-27B to learn well from (competition + PDF) data and used it to generate PL data for texts found in published_texts.csv. To increase PL data quality, we ensembled Gemma models using MBR. To include diversity in our submission, we also trained Qwen3-14B. All models were trained on the data mixture of competition, PDF, and PL. ByT5 models were trained with code adapted from public kernel, and Qwen models were trained using unsloth
We also trained a Gemma3-27B on all PDF data and used it to predict the competition data. The idea was simple: Gemma learned from Akkadian→English translation patterns and was used to generate \"teacher\" translations for the competition data. These Gemma translations served as soft labels for knowledge distillation into our ByT5 student models, where each training row appeared twice: once with the ground truth label and once with Gemma's translation at a reduced weight. Including this KD model provided diversity to our ensemble.
Our private submission consists of 5 ByT5 models, 1 Qwen3-14B (inferred with temperature 0.0 and 0.7), and 2 Qwen3-4B models. The generation from all models was ensembled using MBR with sacrebleu.metrics.CHRF(word_order=2).sentence_score. This was recommended by Claude; the competition metric did not seem to work well for us. Our submission runs for almost 9 hours.
Qwen3.5 models for generating PL, but were not able to train them successfullyThank you Kaggle for hosting the competition. Special thanks to our host @deeppast for proactively clarifying our queries and helping in the release of revised higher quality datasets, which made a real difference. Lastly, I would like to thank my teammate @aman1391 for forcefully inviting me to join this competition (please continue doing so 😆).
\nIf we're being honest, the first few weeks were humbling. We experimented with focal loss, annotation tricks, and various architectural changes; yet the LB barely moved. The breakthrough came when we revisited the host's discussion on data quality, and it crystallized something we had been overlooking: this competition was never about clever models. It was about DATA. The moment we shifted our entire focus to building robust OCR pipelines, curating clean parallel text from academic publications, and generating high-quality pseudo-labels — everything changed. Sometimes the most impactful insight is the one hiding in plain sight.
\nWe trained ByT5 and Qwen3 models on competition data, PDF data, and pseudo-labelled (PL) data created by a Gemma3-27B model trained on competition and PDF data. We created PDF data iteratively and trained our models. Our final submission consisted of an MBR ensemble of ByT5 Small, ByT5-Base, Qwen3-14B, and Qwen3-4B-Instruct models using sacrebleu.metrics.CHRF(word_order=2).sentence_score as MBR metric. We used beam search for ByT5s and temperature sampling for Qwen models.
We leveraged zai-org/GLM-OCR to extract text from the PDF collection shared by the host. We ran OCR in two ways:
We then leverage models hosted on Groq to generate aligned Akkadian-English data. We align the text data to belong to the full letter and use Kimi K2 to get the parallel data. Example prompt:
# Context\nYou are given a Akkadian-Engligh texts in 2 formats.\n\n## Format 1: Interleaved Akkadian and English\n- Akkadian and English interleaved line-by-line\n- Extracted from cropping each line of text using Tesseract\n- Useful for understanding alignment\n- Some of the Akkadian Text might be noisy, refer to the Format 2 to understand the rendering\n\n## Format 2: \n- Akkadian text is clustered, followed by English\n- Extracted by OCRing the entire page of text\n- Akkadian text is better rendered, though alignment of Akkadian and English is not present\n\n# Role\n\nYour task is to extract Akkadian-English parallel data, understand alignment from `Format 1` and text rendering from `Format 2`.\nThe portion of Akkadian Text in `Format 1` must be followed in the parallel data as well i.e. **Do Not** merge consecutive Akkadian lines of text in `Format 1` into a single piece of text\nThe English translations need not be formal, complete or grammatically correct.\n\n**Remember**\n\n- Don't hallucinate or generate your own translations, strictly pick text from the given context\n- Don't include XMLs in Akkadian text (like <sub>4</sub>), render them as characters\n- The English translations need not be full sentences\nWe used different prompts for different PDF depending on the letter layout. For example, some Turkish translation PDFs had translations with line markers at the end.
\n# Context\nYou are given a Akkadian-Turkish text.\n\n- Akkadian text is clustered, followed by it's translation in Turkish\n- Extracted by OCRing the entire page of text\n\n\n# Role\n\nYour task is to extract Akkadian-Turkish-English parallel data, Akkadian and Turkish are present in the text, the Turkish text needs to be translated to English\nThe Akkadian and Turkish text alignment in present, you need to detect it via line numbers (eg. \"1-5)\" -> Turkish of Akkdian text from lines 1-5 )\nThe English translations need not be formal, complete or grammatically correct.\n\n**Remember**\n\n- Strictly pick Akkadian and Turkish text from the given context and correctly translated Turkish into English\n- Don't include XMLs in Akkadian text (like <sub>4</sub>), render them as characters\n- The English translations need not be full sentences\n- Only extract cases where you are confident of the aligned translation\nAKT-6 series PDFs were not easy to extract, and the above approaches did not work. A multi-modal approach with meta-llama/llama-4-maverick-17b-128e-instruct did seem to work, but the model was deprecated before we could process the data (PDF page images were super-resolutioned using a light-weight model). Finally, we used our extracted OCR text with Claude to process them.
We held-out 10% of the competition data to track our cv.
\nWe used Claude extensively to prepare our data preprocessing code as per the guidelines shared by the host, building cleaning pipelines that handled everything from LaTeX artifacts, superscript/subscript normalization, OCR character fixes (İ→I, ḥ→h, ß→š), Sumerogram expansion (KÙ.B.→KÙ.BABBAR), gap synchronization between transliteration and translation, and filtering misaligned Turkish/English rows. We also built a name correction postprocessor using lookup dictionaries that standardized proper name spellings across model outputs (e.g., ensuring \"Šalim-Aššur\" was consistently rendered regardless of which model produced it). Beyond preprocessing, we used Claude to get quality assessments of every OCR-extracted PDF; checking charset compliance, keyword alignment percentages, and identifying which sources were clean enough to train on and which needed filtering.
\nOur initial focus was to maximize the performance of ByT5 and Qwen3-4B based on the competition data and our processing. Once our PDF extraction pipeline was in shape, we extracted all PDFs iteratively and trained our models periodically. We extracted ~31k samples from PDFs. We also found Gemma3-27B to learn well from (competition + PDF) data and used it to generate PL data for texts found in published_texts.csv. To increase PL data quality, we ensembled Gemma models using MBR. To include diversity in our submission, we also trained Qwen3-14B. All models were trained on the data mixture of competition, PDF, and PL. ByT5 models were trained with code adapted from public kernel, and Qwen models were trained using unsloth
We also trained a Gemma3-27B on all PDF data and used it to predict the competition data. The idea was simple: Gemma learned from Akkadian→English translation patterns and was used to generate \"teacher\" translations for the competition data. These Gemma translations served as soft labels for knowledge distillation into our ByT5 student models, where each training row appeared twice: once with the ground truth label and once with Gemma's translation at a reduced weight. Including this KD model provided diversity to our ensemble.
Our private submission consists of 5 ByT5 models, 1 Qwen3-14B (inferred with temperature 0.0 and 0.7), and 2 Qwen3-4B models. The generation from all models was ensembled using MBR with sacrebleu.metrics.CHRF(word_order=2).sentence_score. This was recommended by Claude; the competition metric did not seem to work well for us. Our submission runs for almost 9 hours.
Qwen3.5 models for generating PL, but were not able to train them successfullyTrust me,if you guys use models other than byt5-base,such as large and xl,you'll definitelly make it to the prize pool.Great work!
", "rawMarkdown": "Trust me,if you guys use models other than byt5-base,such as large and xl,you'll definitelly make it to the prize pool.Great work!", "votes": null }, { "id": "3429023", "postDate": "03/26/2026 08:38:24", "content": "Thanks @qifeihhh666
", "rawMarkdown": "Thanks @qifeihhh666", "votes": null } ], "comments": [ { "id": 3429014, "author_name": "qifeihhh666", "author_url": "", "post_date": "03/26/2026 08:25:36", "content": "Trust me,if you guys use models other than byt5-base,such as large and xl,you'll definitelly make it to the prize pool.Great work!
", "votes": null, "replies": [ { "id": 3429023, "author_name": "aman1391", "author_url": "", "post_date": "03/26/2026 08:38:24", "content": "Thanks @qifeihhh666
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3428946": "Thank you Kaggle for hosting the competition. Special thanks to our host @deeppast for proactively clarifying our queries and helping in the release of revised higher quality datasets, which made a real difference. Lastly, I would like to thank my teammate @aman1391 for forcefully inviting me to join this competition (please continue doing so 😆).\n\nIf we're being honest, the first few weeks were humbling. We experimented with focal loss, annotation tricks, and various architectural changes; yet the LB barely moved. The breakthrough came when we revisited the host's [discussion](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/678899) on data quality, and it crystallized something we had been overlooking: this competition was never about clever models. It was about DATA. The moment we shifted our entire focus to building robust OCR pipelines, curating clean parallel text from academic publications, and generating high-quality pseudo-labels — everything changed. Sometimes the most impactful insight is the one hiding in plain sight.\n\n## Summary\n\nWe trained `ByT5` and `Qwen3` models on competition data, PDF data, and pseudo-labelled (PL) data created by a Gemma3-27B model trained on competition and PDF data. We created PDF data iteratively and trained our models. Our final submission consisted of an MBR ensemble of `ByT5 Small`, `ByT5-Base`, `Qwen3-14B`, and `Qwen3-4B-Instruct` models using `sacrebleu.metrics.CHRF(word_order=2).sentence_score` as MBR metric. We used beam search for ByT5s and temperature sampling for Qwen models.\n\n\n## PDF Data Extraction\n\nWe leveraged `zai-org/GLM-OCR` to extract text from the PDF collection shared by the host. We ran OCR in two ways:\n\n1. Run OCR on the full page\n - higher text quality\n - renders characters appropriately\n - lacks structure with a blob of Akkadian text followed by a blob of English/other language text\n2. Run OCR on Tesseract-cropped portions\n - provides layout signals since text can be ordered \n\nWe then leverage models hosted on [Groq](https://console.groq.com/docs/models) to generate aligned Akkadian-English data. We align the text data to belong to the full letter and use `Kimi K2` to get the parallel data. Example prompt:\n\n```\n# Context\nYou are given a Akkadian-Engligh texts in 2 formats.\n\n## Format 1: Interleaved Akkadian and English\n- Akkadian and English interleaved line-by-line\n- Extracted from cropping each line of text using Tesseract\n- Useful for understanding alignment\n- Some of the Akkadian Text might be noisy, refer to the Format 2 to understand the rendering\n\n## Format 2: \n- Akkadian text is clustered, followed by English\n- Extracted by OCRing the entire page of text\n- Akkadian text is better rendered, though alignment of Akkadian and English is not present\n\n# Role\n\nYour task is to extract Akkadian-English parallel data, understand alignment from `Format 1` and text rendering from `Format 2`.\nThe portion of Akkadian Text in `Format 1` must be followed in the parallel data as well i.e. **Do Not** merge consecutive Akkadian lines of text in `Format 1` into a single piece of text\nThe English translations need not be formal, complete or grammatically correct.\n\n**Remember**\n\n- Don't hallucinate or generate your own translations, strictly pick text from the given context\n- Don't include XMLs in Akkadian text (like 4), render them as characters\n- The English translations need not be full sentences\n\n```\n\nWe used different prompts for different PDF depending on the letter layout. For example, some Turkish translation PDFs had translations with line markers at the end.\n\n```\n# Context\nYou are given a Akkadian-Turkish text.\n\n- Akkadian text is clustered, followed by it's translation in Turkish\n- Extracted by OCRing the entire page of text\n\n\n# Role\n\nYour task is to extract Akkadian-Turkish-English parallel data, Akkadian and Turkish are present in the text, the Turkish text needs to be translated to English\nThe Akkadian and Turkish text alignment in present, you need to detect it via line numbers (eg. \"1-5)\" -> Turkish of Akkdian text from lines 1-5 )\nThe English translations need not be formal, complete or grammatically correct.\n\n**Remember**\n\n- Strictly pick Akkadian and Turkish text from the given context and correctly translated Turkish into English\n- Don't include XMLs in Akkadian text (like 4), render them as characters\n- The English translations need not be full sentences\n- Only extract cases where you are confident of the aligned translation\n\n```\n\nAKT-6 series PDFs were not easy to extract, and the above approaches did not work. A multi-modal approach with `meta-llama/llama-4-maverick-17b-128e-instruct` did seem to work, but the model was deprecated before we could process the data (PDF page images were super-resolutioned using a [light-weight model](https://huggingface.co/spaces/bookbot/Image-Upscaling-Playground)). Finally, we used our extracted OCR text with `Claude` to process them.\n\n\n## Validation\n\nWe held-out 10% of the competition data to track our cv.\n\n## Data Pre/Post Processing\n\nWe used Claude extensively to prepare our data preprocessing code as per the guidelines shared by the host, building cleaning pipelines that handled everything from LaTeX artifacts, superscript/subscript normalization, OCR character fixes (İ→I, ḥ→h, ß→š), Sumerogram expansion (KÙ.B.→KÙ.BABBAR), gap synchronization between transliteration and translation, and filtering misaligned Turkish/English rows. We also built a name correction postprocessor using lookup dictionaries that standardized proper name spellings across model outputs (e.g., ensuring \"Šalim-Aššur\" was consistently rendered regardless of which model produced it). Beyond preprocessing, we used Claude to get quality assessments of every OCR-extracted PDF; checking charset compliance, keyword alignment percentages, and identifying which sources were clean enough to train on and which needed filtering.\n\n## Training\n\nOur initial focus was to maximize the performance of `ByT5` and `Qwen3-4B` based on the competition data and our processing. Once our PDF extraction pipeline was in shape, we extracted all PDFs iteratively and trained our models periodically. We extracted ~31k samples from PDFs. We also found `Gemma3-27B` to learn well from (competition + PDF) data and used it to generate PL data for texts found in `published_texts.csv`. To increase PL data quality, we ensembled Gemma models using MBR. To include diversity in our submission, we also trained `Qwen3-14B`. All models were trained on the data mixture of competition, PDF, and PL. ByT5 models were trained with code adapted from public kernel, and Qwen models were trained using [unsloth](https://unsloth.ai/)\n\n### Knowledge Distillation\n\nWe also trained a `Gemma3-27B` on all PDF data and used it to predict the competition data. The idea was simple: Gemma learned from Akkadian→English translation patterns and was used to generate \"teacher\" translations for the competition data. These Gemma translations served as soft labels for knowledge distillation into our ByT5 student models, where each training row appeared twice: once with the ground truth label and once with Gemma's translation at a reduced weight. Including this KD model provided diversity to our ensemble.\n\n## Ensemble\n\nOur private submission consists of 5 ByT5 models, 1 Qwen3-14B (inferred with temperature 0.0 and 0.7), and 2 Qwen3-4B models. The generation from all models was ensembled using MBR with `sacrebleu.metrics.CHRF(word_order=2).sentence_score`. This was recommended by `Claude`; the competition metric did not seem to work well for us. Our submission runs for almost 9 hours. \n\n\n## Failed Experiments\n\n* ByT5 Large or XL did not work with our setup\n* We had an idea similar to the CPT approach adopted by other teams. Our approach was to train a CPT and SFT model and then model-soup them together to get the benefits, which did not work (model soups in general did not work for us). We should have just trained in a staged manner\n* We tried training `Qwen3.5` models for generating PL, but were not able to train them successfully\n* Our experiment with normalizing PN/GN form with placeholders did not work", "3429014": "Trust me,if you guys use models other than byt5-base,such as large and xl,you'll definitelly make it to the prize pool.Great work!", "3429023": "Thanks @qifeihhh666" }, "source": "existing_discussion", "complete": true, "competition_id": 121150, "competition_slug": "deep-past-initiative-machine-translation", "team_id": 14983163, "solution_url": "/competitions/deep-past-initiative-machine-translation/writeups/15th-place-gold-solution" }