![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2107527%2F052376d9af1148d7d798be1eacba868d%2F2026-03-31%209.34.29.png?generation=1774917414062115&alt=media\")
Thank you to the organizers for this fascinating competition! Translating 4,000-year-old Akkadian texts was a truly unique NLP challenge.
\nLike most competitors, ByT5 was the clear winner for this task. Akkadian transliteration is full of special diacritics (š, ṣ, ṭ, ḫ) and subscript digits that subword tokenizers handle poorly. ByT5's byte-level tokenization avoids OOV issues entirely.
\nEarly experiments with mBART-50 (LB 21.3) and NLLB-200 (~18 CV) hit a ceiling around BLEU 10-11. Switching to ByT5-base immediately jumped to LB 26-28.
\nAn important design choice: I trained on document-level pairs rather than splitting into sentence-level pairs. The official train.csv contains paragraph/document-level transliteration-translation pairs, and I kept this granularity. Most top teams split documents into sentence-level pairs using LLMs (DeepSeek, Gemini) for alignment, since the test data is evaluated at sentence level. In my experiments, document-level actually outperformed sentence-level — but this may have been because my own sentence splitting was low quality rather than document-level being inherently better. Top teams with high-quality LLM-based sentence alignment clearly benefited from sentence-level training.
The official train.csv (1,561 pairs) was far too small. I expanded the training data through three sources:
a) OARE Published Texts (~5,900 pairs)\nExtracted sentence-level pairs from the OARE database's published_texts.csv using chunk matching with length distribution alignment and overlap-aware deduplication.
b) LLM-Generated Translations (6,281 pairs)\nUsed GPT-5.2 to translate ~6,400 Akkadian transliterations from published_texts.csv that lacked English translations. 5-shot prompting with genre hints, conservative temperature (0.3), and quality filtering (gap ratio, refusal detection, length ratio). ~$38 total cost.
c) Cherry-Picked High-Confidence Pairs\nAdditional curated pairs from category D (new extractions), filtered for quality.
\nTotal: ~10,700 pairs used in final models (after filtering out low-quality samples).
\nDifferent data sources have different quality levels. I used distinct task prefixes per source (e.g., \"translate original Akkadian to English: \" vs \"translate llm Akkadian to English: \") to let the model implicitly learn source reliability. At inference time, all inputs use the highest-quality prefix.
Built Streamlit apps to visually inspect data quality at every stage — browsing augmented data, checking prediction diffs (word-level highlighting vs ground truth), and reviewing transliteration/translation statistics. Being able to quickly eyeball hundreds of examples caught quality issues that aggregate metrics alone would have missed, and directly informed which data to keep or discard.\n
The final submission used 3 ByT5-base models — 2 own models trained on different max source lengths and 1 public model trained on different data. Each model generates beam search candidates, and MBR selects the best translation by weighted consensus (chrF++, BLEU, Jaccard). Cross-family diversity (models trained on different data distributions) was key to MBR effectiveness.
\n[SRC] + [DRAFT] → [REFINED] using 3-fold CV drafts. Interesting but didn't improve LB.VLM-based PDF extraction pipeline: The #1 differentiator between Gold and Silver. Top teams used Gemini Pro / Qwen-VL to extract 20k-60k sentence pairs from academic PDFs (AKT series, etc.). I relied on OARE + LLM translation, which was lower quality.
Scale up to ByT5-Large/XL: With enough clean data, larger models clearly win. CTranslate2 quantization (int8) makes XL feasible on T4 within the 9-hour kernel limit.
Continual Pre-Training (CPT): Domain-adaptive pre-training with span corruption on raw Akkadian text before SFT. Almost all Gold teams did this; I skipped it entirely.
Loss weighting by data quality: 1st place weighted high-quality data at 1.3-1.4x and LLM data at 0.5-0.7x. My source-prefix approach was a rough proxy but not as effective.
Creative synthetic data: 3rd place generated 300k+ \"vocabulary/grammar drills\" from dictionaries. 5th place used back-translation. My LLM augmentation was straightforward translation — I should have been more creative about teaching the model Akkadian structure.
Thank you to the organizers for this fascinating competition! Translating 4,000-year-old Akkadian texts was a truly unique NLP challenge.
\nLike most competitors, ByT5 was the clear winner for this task. Akkadian transliteration is full of special diacritics (š, ṣ, ṭ, ḫ) and subscript digits that subword tokenizers handle poorly. ByT5's byte-level tokenization avoids OOV issues entirely.
\nEarly experiments with mBART-50 (LB 21.3) and NLLB-200 (~18 CV) hit a ceiling around BLEU 10-11. Switching to ByT5-base immediately jumped to LB 26-28.
\nAn important design choice: I trained on document-level pairs rather than splitting into sentence-level pairs. The official train.csv contains paragraph/document-level transliteration-translation pairs, and I kept this granularity. Most top teams split documents into sentence-level pairs using LLMs (DeepSeek, Gemini) for alignment, since the test data is evaluated at sentence level. In my experiments, document-level actually outperformed sentence-level — but this may have been because my own sentence splitting was low quality rather than document-level being inherently better. Top teams with high-quality LLM-based sentence alignment clearly benefited from sentence-level training.
The official train.csv (1,561 pairs) was far too small. I expanded the training data through three sources:
a) OARE Published Texts (~5,900 pairs)\nExtracted sentence-level pairs from the OARE database's published_texts.csv using chunk matching with length distribution alignment and overlap-aware deduplication.
b) LLM-Generated Translations (6,281 pairs)\nUsed GPT-5.2 to translate ~6,400 Akkadian transliterations from published_texts.csv that lacked English translations. 5-shot prompting with genre hints, conservative temperature (0.3), and quality filtering (gap ratio, refusal detection, length ratio). ~$38 total cost.
c) Cherry-Picked High-Confidence Pairs\nAdditional curated pairs from category D (new extractions), filtered for quality.
\nTotal: ~10,700 pairs used in final models (after filtering out low-quality samples).
\nDifferent data sources have different quality levels. I used distinct task prefixes per source (e.g., \"translate original Akkadian to English: \" vs \"translate llm Akkadian to English: \") to let the model implicitly learn source reliability. At inference time, all inputs use the highest-quality prefix.
Built Streamlit apps to visually inspect data quality at every stage — browsing augmented data, checking prediction diffs (word-level highlighting vs ground truth), and reviewing transliteration/translation statistics. Being able to quickly eyeball hundreds of examples caught quality issues that aggregate metrics alone would have missed, and directly informed which data to keep or discard.\n
The final submission used 3 ByT5-base models — 2 own models trained on different max source lengths and 1 public model trained on different data. Each model generates beam search candidates, and MBR selects the best translation by weighted consensus (chrF++, BLEU, Jaccard). Cross-family diversity (models trained on different data distributions) was key to MBR effectiveness.
\n[SRC] + [DRAFT] → [REFINED] using 3-fold CV drafts. Interesting but didn't improve LB.VLM-based PDF extraction pipeline: The #1 differentiator between Gold and Silver. Top teams used Gemini Pro / Qwen-VL to extract 20k-60k sentence pairs from academic PDFs (AKT series, etc.). I relied on OARE + LLM translation, which was lower quality.
Scale up to ByT5-Large/XL: With enough clean data, larger models clearly win. CTranslate2 quantization (int8) makes XL feasible on T4 within the 9-hour kernel limit.
Continual Pre-Training (CPT): Domain-adaptive pre-training with span corruption on raw Akkadian text before SFT. Almost all Gold teams did this; I skipped it entirely.
Loss weighting by data quality: 1st place weighted high-quality data at 1.3-1.4x and LLM data at 0.5-0.7x. My source-prefix approach was a rough proxy but not as effective.
Creative synthetic data: 3rd place generated 300k+ \"vocabulary/grammar drills\" from dictionaries. 5th place used back-translation. My LLM augmentation was straightforward translation — I should have been more creative about teaching the model Akkadian structure.