![\"Our](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15995218%2F115e202893dac9c41f3c42eaad9f7aa5%2FDataBoom_presentation.png?generation=1775076541760501&alt=media\")
Our final solution was a single ByT5-large model. Most of the gain came from external data, preprocessing, source repair, and decoding.
\nThe main ingredients were:
\nIn short:
\na single ByT5-large model + external data + preprocessing + test-time source repair + MBR
A large part of this competition was not just fitting a larger model. The real challenge was handling noise and making good use of external data.
\nThe data had:
\nBecause of that, the best gains came from making the data cleaner and the decoding more reliable, rather than from model scaling alone.
\nOur final pipeline had four parts:
\nWe used ByT5-large for the final solution.
\nSince this competition was very character-sensitive and most strong solutions were based on ByT5, the real difference came less from the backbone itself and more from the surrounding pipeline: external data, preprocessing, source repair, and inference.
\nWe used two stages of data.
\nThis was used for final supervised fine-tuning.
\nWe merged several public Akkadian-English datasets with an OCR-derived parallel corpus extracted from PDF publications. The OCR data was noisy, but it added useful extra coverage.
\nBefore pretraining, we applied light cleanup:
\nOne simple filter that helped was removing pairs with a very large source-target length mismatch:
\ndf[\"len_diff\"] = (\n df[\"translation\"].astype(str).apply(len)\n - df[\"transliteration\"].astype(str).apply(len)\n).abs()\ndf = df[df[\"len_diff\"] < 70].copy()\nThis removed many pathological alignments at almost no cost.
\nA large part of the solution was reducing unnecessary variation across sources.
\nOn the Akkadian side, we normalized recurring differences such as:
\nOn the English side, we normalized recurring formatting differences such as:
\nThe goal was simple: make the train data more consistent and reduce avoidable train-test mismatch.
\nThis was especially important because the evaluation combined BLEU and chrF++, so even small surface differences could matter.
\nOne of the most useful parts of the pipeline was a test-time source repair module.
\nIts role was not just to normalize formatting, but to repair corrupted transliteration forms and move noisy test inputs closer to the training distribution before decoding.
\nBesides regex-based cleanup, the module also used mappings derived from clean training data to restore recurring broken forms more conservatively. This mattered because many corrupted symbols were ambiguous, so naive global replacement could easily damage already-correct transliterations.
\nIn practice, the module combined:
\nSo the source pipeline was:
\nraw test transliteration\n→ test-time source repair\n→ model input
\nThis step was small, but consistently useful because even minor source corruption could push the model toward a bad translation.
\nWe first pretrained ByT5-large on the merged external corpus.
\nThe goal of pretraining was not to fit the external data perfectly, but to expose the model to more transliteration variants, more vocabulary, and more translation patterns before final supervised training.
\nAfter pretraining, we fine-tuned on the organizer train set.
\nWe kept the same overall preprocessing philosophy:
\nWe tried different training variants, but the most important thing was the full pipeline rather than any single training trick.
\nThe final submission used a single-model MBR decoding pipeline.
\nFor each input, we first applied test-time source repair. Then we generated a diverse candidate pool using a mix of:
\nInstead of taking the top beam directly, we applied Minimum Bayes Risk decoding with sentence-level chrF++ and selected the candidate with the highest average agreement with the rest of the pool.
\nThis worked well for the competition because MBR often chose safer outputs than raw beam search.
\nA compact summary of representative runs is below.
\n| Model / setup | \nPretraining data | \nDecoding | \nPrivate LB | \nPublic LB | \n
|---|---|---|---|---|
| ByT5-base | \nHF + OCR | \nbeam | \n38.3467 | \n37.1513 | \n
| ByT5-large | \nHF + OCR | \nbeam | \n37.8768 | \n37.3950 | \n
| ByT5-XL | \nHF + OCR | \nbeam | \n39.5868 | \n38.0995 | \n
| ByT5-XL + ByT5-large | \nHF + OCR | \nbeam + sampling + MBR | \n39.6216 | \n38.2817 | \n
| ByT5-large | \nHF + OCR | \nbeam + sampling + MBR (0.75 / 0.85) | \n39.6175 | \n38.4482 | \n
| ByT5-large | \nOCR only | \nbeam + sampling + MBR (0.7 / 0.85) | \n39.6652 | \n38.3060 | \n
Even though we tried multiple model sizes, the main takeaway stayed the same: the biggest improvements came from data and inference quality, not from scaling alone.
\nSome directions were less useful than expected:
\nIn short, many ideas that made the system more complex were less valuable than simply making the inputs cleaner and the final decoding more robust.
\nThe main lesson for us was straightforward:
\n\n\nin low-resource translation with noisy historical text, better data, better preprocessing, and better decoding can matter more than a larger model
\n
The biggest gains came from making the data more consistent, reducing the train-test mismatch, and choosing better outputs at inference time.
", "messages": [ { "id": 3433479, "postDate": "2026-04-01T18:29:15.260Z", "content": "
Our final solution was a single ByT5-large model. Most of the gain came from external data, preprocessing, source repair, and decoding.
\nThe main ingredients were:
\nIn short:
\na single ByT5-large model + external data + preprocessing + test-time source repair + MBR
A large part of this competition was not just fitting a larger model. The real challenge was handling noise and making good use of external data.
\nThe data had:
\nBecause of that, the best gains came from making the data cleaner and the decoding more reliable, rather than from model scaling alone.
\nOur final pipeline had four parts:
\nWe used ByT5-large for the final solution.
\nSince this competition was very character-sensitive and most strong solutions were based on ByT5, the real difference came less from the backbone itself and more from the surrounding pipeline: external data, preprocessing, source repair, and inference.
\nWe used two stages of data.
\nThis was used for final supervised fine-tuning.
\nWe merged several public Akkadian-English datasets with an OCR-derived parallel corpus extracted from PDF publications. The OCR data was noisy, but it added useful extra coverage.
\nBefore pretraining, we applied light cleanup:
\nOne simple filter that helped was removing pairs with a very large source-target length mismatch:
\ndf[\"len_diff\"] = (\n df[\"translation\"].astype(str).apply(len)\n - df[\"transliteration\"].astype(str).apply(len)\n).abs()\ndf = df[df[\"len_diff\"] < 70].copy()\nThis removed many pathological alignments at almost no cost.
\nA large part of the solution was reducing unnecessary variation across sources.
\nOn the Akkadian side, we normalized recurring differences such as:
\nOn the English side, we normalized recurring formatting differences such as:
\nThe goal was simple: make the train data more consistent and reduce avoidable train-test mismatch.
\nThis was especially important because the evaluation combined BLEU and chrF++, so even small surface differences could matter.
\nOne of the most useful parts of the pipeline was a test-time source repair module.
\nIts role was not just to normalize formatting, but to repair corrupted transliteration forms and move noisy test inputs closer to the training distribution before decoding.
\nBesides regex-based cleanup, the module also used mappings derived from clean training data to restore recurring broken forms more conservatively. This mattered because many corrupted symbols were ambiguous, so naive global replacement could easily damage already-correct transliterations.
\nIn practice, the module combined:
\nSo the source pipeline was:
\nraw test transliteration\n→ test-time source repair\n→ model input
\nThis step was small, but consistently useful because even minor source corruption could push the model toward a bad translation.
\nWe first pretrained ByT5-large on the merged external corpus.
\nThe goal of pretraining was not to fit the external data perfectly, but to expose the model to more transliteration variants, more vocabulary, and more translation patterns before final supervised training.
\nAfter pretraining, we fine-tuned on the organizer train set.
\nWe kept the same overall preprocessing philosophy:
\nWe tried different training variants, but the most important thing was the full pipeline rather than any single training trick.
\nThe final submission used a single-model MBR decoding pipeline.
\nFor each input, we first applied test-time source repair. Then we generated a diverse candidate pool using a mix of:
\nInstead of taking the top beam directly, we applied Minimum Bayes Risk decoding with sentence-level chrF++ and selected the candidate with the highest average agreement with the rest of the pool.
\nThis worked well for the competition because MBR often chose safer outputs than raw beam search.
\nA compact summary of representative runs is below.
\n| Model / setup | \nPretraining data | \nDecoding | \nPrivate LB | \nPublic LB | \n
|---|---|---|---|---|
| ByT5-base | \nHF + OCR | \nbeam | \n38.3467 | \n37.1513 | \n
| ByT5-large | \nHF + OCR | \nbeam | \n37.8768 | \n37.3950 | \n
| ByT5-XL | \nHF + OCR | \nbeam | \n39.5868 | \n38.0995 | \n
| ByT5-XL + ByT5-large | \nHF + OCR | \nbeam + sampling + MBR | \n39.6216 | \n38.2817 | \n
| ByT5-large | \nHF + OCR | \nbeam + sampling + MBR (0.75 / 0.85) | \n39.6175 | \n38.4482 | \n
| ByT5-large | \nOCR only | \nbeam + sampling + MBR (0.7 / 0.85) | \n39.6652 | \n38.3060 | \n
Even though we tried multiple model sizes, the main takeaway stayed the same: the biggest improvements came from data and inference quality, not from scaling alone.
\nSome directions were less useful than expected:
\nIn short, many ideas that made the system more complex were less valuable than simply making the inputs cleaner and the final decoding more robust.
\nThe main lesson for us was straightforward:
\n\n\nin low-resource translation with noisy historical text, better data, better preprocessing, and better decoding can matter more than a larger model
\n
The biggest gains came from making the data more consistent, reducing the train-test mismatch, and choosing better outputs at inference time.
", "rawMarkdown": "# Deep Past Challenge: ByT5-large + External Data + Test-Time Source Repair + MBR\n\n## TL;DR\n\nOur final solution was a **single ByT5-large model**. Most of the gain came from **external data, preprocessing, source repair, and decoding**.\n\nThe main ingredients were:\n\n- **external pretraining** on public Akkadian-English data\n- additional **OCR-derived parallel data** extracted from PDF publications\n- **preprocessing** on both source and target\n- **test-time source repair** to fix broken transliteration symbols in test data and bring them closer to the train distribution\n- **MBR reranking** over beam and sampled candidates using **chrF++**\n\nIn short: \n**a single ByT5-large model + external data + preprocessing + test-time source repair + MBR**\n\n\n## Overview\n\nA large part of this competition was not just fitting a larger model. The real challenge was handling **noise** and making good use of **external data**.\n\nThe data had:\n\n- inconsistent transliteration conventions\n- OCR corruption\n- mixed Unicode / ASCII forms\n- formatting variation on both source and target sides\n- limited clean parallel supervision\n\nBecause of that, the best gains came from making the data cleaner and the decoding more reliable, rather than from model scaling alone.\n\nOur final pipeline had four parts:\n\n1. **pretrain ByT5-large on external Akkadian-English data**\n2. **fine-tune on the organizer train set**\n3. **repair and normalize test transliterations with test-time source repair**\n4. **generate multiple candidates and select with chrF++-based MBR**\n\n\n## Model\n\nWe used **ByT5-large** for the final solution.\n\nSince this competition was very character-sensitive and most strong solutions were based on ByT5, the real difference came less from the backbone itself and more from the surrounding pipeline: external data, preprocessing, source repair, and inference.\n\n## Data\n\nWe used two stages of data.\n\n### 1. Organizer train set\n\nThis was used for final supervised fine-tuning.\n\n### 2. External pretraining data\n\nWe merged several public Akkadian-English datasets with an **OCR-derived parallel corpus extracted from PDF publications**. The OCR data was noisy, but it added useful extra coverage.\n\nBefore pretraining, we applied light cleanup:\n\n- merge all sources into a common schema\n- remove duplicates\n- filter obviously broken pairs\n- apply simple sentence-level cleanup where possible\n\nOne simple filter that helped was removing pairs with a very large source-target length mismatch:\n\n```python\ndf[\"len_diff\"] = (\n df[\"translation\"].astype(str).apply(len)\n - df[\"transliteration\"].astype(str).apply(len)\n).abs()\ndf = df[df[\"len_diff\"] < 70].copy()\n```\n\nThis removed many pathological alignments at almost no cost.\n\n\n## Preprocessing\n\nA large part of the solution was reducing unnecessary variation across sources.\n\nOn the **Akkadian side**, we normalized recurring differences such as:\n\n* alternative diacritic encodings\n* subscript forms\n* determinatives\n* broken-text markers\n* some numeric variants\n\nOn the **English side**, we normalized recurring formatting differences such as:\n\n* gap markers\n* month and number expressions\n* editorial notes\n* a few domain-specific surface forms\n\nThe goal was simple: make the train data more consistent and reduce avoidable train-test mismatch.\n\nThis was especially important because the evaluation combined **BLEU** and **chrF++**, so even small surface differences could matter.\n\n\n## Test-Time Source Repair\n\nOne of the most useful parts of the pipeline was a **test-time source repair module**.\n\nIts role was not just to normalize formatting, but to **repair corrupted transliteration forms** and move noisy test inputs closer to the training distribution before decoding.\n\nBesides regex-based cleanup, the module also used **mappings derived from clean training data** to restore recurring broken forms more conservatively. This mattered because many corrupted symbols were **ambiguous**, so naive global replacement could easily damage already-correct transliterations.\n\nIn practice, the module combined:\n* normalization of recurring Unicode / ASCII variants\n* cleanup of malformed gap and broken-text patterns\n* repair of recurring corrupted transliteration forms\n* a small set of **precision-focused context-sensitive fixes**\n\nSo the source pipeline was:\n\n**raw test transliteration**\n→ **test-time source repair**\n→ **model input**\n\nThis step was small, but consistently useful because even minor source corruption could push the model toward a bad translation.\n\n\n## Pretraining\n\nWe first pretrained **ByT5-large** on the merged external corpus.\n\nThe goal of pretraining was not to fit the external data perfectly, but to expose the model to more transliteration variants, more vocabulary, and more translation patterns before final supervised training.\n\n\n## Fine-tuning\n\nAfter pretraining, we fine-tuned on the organizer train set.\n\nWe kept the same overall preprocessing philosophy:\n\n* normalize transliterations\n* normalize target formatting\n* reduce avoidable surface variation\n\nWe tried different training variants, but the most important thing was the full pipeline rather than any single training trick.\n\n\n## Inference\n\nThe final submission used a **single-model MBR decoding pipeline**.\n\nFor each input, we first applied **test-time source repair**. Then we generated a diverse candidate pool using a mix of:\n\n* beam search\n* sampling with multiple temperatures\n\nInstead of taking the top beam directly, we applied **Minimum Bayes Risk decoding** with sentence-level **chrF++** and selected the candidate with the highest average agreement with the rest of the pool.\n\nThis worked well for the competition because MBR often chose safer outputs than raw beam search.\n\n\n## Results\n\nA compact summary of representative runs is below.\n\n| Model / setup | Pretraining data | Decoding | Private LB | Public LB |\n| ------------- | ---------------- | ------------------------------------- | ----------: | --------: |\n| ByT5-base | HF + OCR | beam | 38.3467 | 37.1513 |\n| ByT5-large | HF + OCR | beam | 37.8768 | 37.3950 |\n| ByT5-XL | HF + OCR | beam | 39.5868 | 38.0995 |\n| ByT5-XL + ByT5-large | HF + OCR | beam + sampling + MBR | 39.6216 | 38.2817 |\n| ByT5-large | HF + OCR | beam + sampling + MBR (`0.75 / 0.85`) | 39.6175 | 38.4482 |\n| ByT5-large | OCR only | beam + sampling + MBR (`0.7 / 0.85`) | **39.6652** | 38.3060 |\n\nEven though we tried multiple model sizes, the main takeaway stayed the same: the biggest improvements came from **data and inference quality**, not from scaling alone.\n\n\n## What did not work\n\nSome directions were less useful than expected:\n\n* **model scaling alone** was not enough\n* **more complicated training tricks** did not help as much as better data and preprocessing\n* **raw OCR data without enough cleanup** was much less effective\n* **taking the top beam directly** was weaker than using a more diverse candidate pool with MBR\n\nIn short, many ideas that made the system more complex were less valuable than simply making the inputs cleaner and the final decoding more robust.\n\n\n## Main lesson\n\nThe main lesson for us was straightforward:\n\n> in low-resource translation with noisy historical text, better data, better preprocessing, and better decoding can matter more than a larger model\n\nThe biggest gains came from making the data more consistent, reducing the train-test mismatch, and choosing better outputs at inference time.", "votes": 7 } ], "comments": [], "raw_markdown_by_id": { "3433479": "# Deep Past Challenge: ByT5-large + External Data + Test-Time Source Repair + MBR\n\n## TL;DR\n\nOur final solution was a **single ByT5-large model**. Most of the gain came from **external data, preprocessing, source repair, and decoding**.\n\nThe main ingredients were:\n\n- **external pretraining** on public Akkadian-English data\n- additional **OCR-derived parallel data** extracted from PDF publications\n- **preprocessing** on both source and target\n- **test-time source repair** to fix broken transliteration symbols in test data and bring them closer to the train distribution\n- **MBR reranking** over beam and sampled candidates using **chrF++**\n\nIn short: \n**a single ByT5-large model + external data + preprocessing + test-time source repair + MBR**\n\n\n## Overview\n\nA large part of this competition was not just fitting a larger model. The real challenge was handling **noise** and making good use of **external data**.\n\nThe data had:\n\n- inconsistent transliteration conventions\n- OCR corruption\n- mixed Unicode / ASCII forms\n- formatting variation on both source and target sides\n- limited clean parallel supervision\n\nBecause of that, the best gains came from making the data cleaner and the decoding more reliable, rather than from model scaling alone.\n\nOur final pipeline had four parts:\n\n1. **pretrain ByT5-large on external Akkadian-English data**\n2. **fine-tune on the organizer train set**\n3. **repair and normalize test transliterations with test-time source repair**\n4. **generate multiple candidates and select with chrF++-based MBR**\n\n\n## Model\n\nWe used **ByT5-large** for the final solution.\n\nSince this competition was very character-sensitive and most strong solutions were based on ByT5, the real difference came less from the backbone itself and more from the surrounding pipeline: external data, preprocessing, source repair, and inference.\n\n## Data\n\nWe used two stages of data.\n\n### 1. Organizer train set\n\nThis was used for final supervised fine-tuning.\n\n### 2. External pretraining data\n\nWe merged several public Akkadian-English datasets with an **OCR-derived parallel corpus extracted from PDF publications**. The OCR data was noisy, but it added useful extra coverage.\n\nBefore pretraining, we applied light cleanup:\n\n- merge all sources into a common schema\n- remove duplicates\n- filter obviously broken pairs\n- apply simple sentence-level cleanup where possible\n\nOne simple filter that helped was removing pairs with a very large source-target length mismatch:\n\n```python\ndf[\"len_diff\"] = (\n df[\"translation\"].astype(str).apply(len)\n - df[\"transliteration\"].astype(str).apply(len)\n).abs()\ndf = df[df[\"len_diff\"] < 70].copy()\n```\n\nThis removed many pathological alignments at almost no cost.\n\n\n## Preprocessing\n\nA large part of the solution was reducing unnecessary variation across sources.\n\nOn the **Akkadian side**, we normalized recurring differences such as:\n\n* alternative diacritic encodings\n* subscript forms\n* determinatives\n* broken-text markers\n* some numeric variants\n\nOn the **English side**, we normalized recurring formatting differences such as:\n\n* gap markers\n* month and number expressions\n* editorial notes\n* a few domain-specific surface forms\n\nThe goal was simple: make the train data more consistent and reduce avoidable train-test mismatch.\n\nThis was especially important because the evaluation combined **BLEU** and **chrF++**, so even small surface differences could matter.\n\n\n## Test-Time Source Repair\n\nOne of the most useful parts of the pipeline was a **test-time source repair module**.\n\nIts role was not just to normalize formatting, but to **repair corrupted transliteration forms** and move noisy test inputs closer to the training distribution before decoding.\n\nBesides regex-based cleanup, the module also used **mappings derived from clean training data** to restore recurring broken forms more conservatively. This mattered because many corrupted symbols were **ambiguous**, so naive global replacement could easily damage already-correct transliterations.\n\nIn practice, the module combined:\n* normalization of recurring Unicode / ASCII variants\n* cleanup of malformed gap and broken-text patterns\n* repair of recurring corrupted transliteration forms\n* a small set of **precision-focused context-sensitive fixes**\n\nSo the source pipeline was:\n\n**raw test transliteration**\n→ **test-time source repair**\n→ **model input**\n\nThis step was small, but consistently useful because even minor source corruption could push the model toward a bad translation.\n\n\n## Pretraining\n\nWe first pretrained **ByT5-large** on the merged external corpus.\n\nThe goal of pretraining was not to fit the external data perfectly, but to expose the model to more transliteration variants, more vocabulary, and more translation patterns before final supervised training.\n\n\n## Fine-tuning\n\nAfter pretraining, we fine-tuned on the organizer train set.\n\nWe kept the same overall preprocessing philosophy:\n\n* normalize transliterations\n* normalize target formatting\n* reduce avoidable surface variation\n\nWe tried different training variants, but the most important thing was the full pipeline rather than any single training trick.\n\n\n## Inference\n\nThe final submission used a **single-model MBR decoding pipeline**.\n\nFor each input, we first applied **test-time source repair**. Then we generated a diverse candidate pool using a mix of:\n\n* beam search\n* sampling with multiple temperatures\n\nInstead of taking the top beam directly, we applied **Minimum Bayes Risk decoding** with sentence-level **chrF++** and selected the candidate with the highest average agreement with the rest of the pool.\n\nThis worked well for the competition because MBR often chose safer outputs than raw beam search.\n\n\n## Results\n\nA compact summary of representative runs is below.\n\n| Model / setup | Pretraining data | Decoding | Private LB | Public LB |\n| ------------- | ---------------- | ------------------------------------- | ----------: | --------: |\n| ByT5-base | HF + OCR | beam | 38.3467 | 37.1513 |\n| ByT5-large | HF + OCR | beam | 37.8768 | 37.3950 |\n| ByT5-XL | HF + OCR | beam | 39.5868 | 38.0995 |\n| ByT5-XL + ByT5-large | HF + OCR | beam + sampling + MBR | 39.6216 | 38.2817 |\n| ByT5-large | HF + OCR | beam + sampling + MBR (`0.75 / 0.85`) | 39.6175 | 38.4482 |\n| ByT5-large | OCR only | beam + sampling + MBR (`0.7 / 0.85`) | **39.6652** | 38.3060 |\n\nEven though we tried multiple model sizes, the main takeaway stayed the same: the biggest improvements came from **data and inference quality**, not from scaling alone.\n\n\n## What did not work\n\nSome directions were less useful than expected:\n\n* **model scaling alone** was not enough\n* **more complicated training tricks** did not help as much as better data and preprocessing\n* **raw OCR data without enough cleanup** was much less effective\n* **taking the top beam directly** was weaker than using a more diverse candidate pool with MBR\n\nIn short, many ideas that made the system more complex were less valuable than simply making the inputs cleaner and the final decoding more robust.\n\n\n## Main lesson\n\nThe main lesson for us was straightforward:\n\n> in low-resource translation with noisy historical text, better data, better preprocessing, and better decoding can matter more than a larger model\n\nThe biggest gains came from making the data more consistent, reducing the train-test mismatch, and choosing better outputs at inference time." } }