Score: 41.9 (Public) / 41.0 (Private) | Team: wukeneth | Model: google/byt5-large
\nA fully data-centric solution. The model is vanilla ByT5-large with no architectural changes. Every point of improvement came from building a larger, cleaner training corpus through:
\nThe competition provides three key files:
\nTraining directly on full-document pairs (Akkadian full text → full English translation) works but leaves significant value on the table. The model needs clean sentence-level pairs to learn tight correspondences. The core challenge was building those pairs at scale.
\nI partitioned all published texts into three non-overlapping sets and applied a different LLM strategy to each. The three sets are defined by strict set arithmetic on the official data files:
\n| Set | \nDefinition | \nLLM Strategy | \n
|---|---|---|
| Breaker | \n(U_train ∪ U_pub) − S_meta | \nSegment full translation into sentence-level pairs | \n
| Fixer | \nS_meta | \nRe-anchor corrupt fragments using OARE API word indices | \n
| Generator | \nU_pub − (U_train ∪ S_meta) | \nSynthesize translation from morphological metadata | \n
Input: Akkadian transliteration + full English translation from train.csv + OARE API word indices and physical line numbers.
Method: For each document, the OARE ID is used to look up per-word dictionary entries (normalized form, lemma) from the OARE word table. An LLM is then prompted with: the full Akkadian text, the per-word dictionary context, and the full English translation. It splits the English translation into sentence-sized chunks and assigns a word_range (start/end word index) to each chunk using the physical word coordinates as anchors. Because the source texts are clean (no pre-existing corrupt segmentation), the output is reliable.
Result: 1,558 documents → 9,378 sentence pairs
\nInput: Corrupt S_meta fragments + OARE API\normalized_form, word_index and line_number fields for each word.
\nMethod: Use the existing fragments as a weak reference only. Re-anchor each fragment by matching normalized word forms from the API to the physical tablet structure. The LLM reconstructs correct sentence boundaries from scratch, using the S_meta entries only as coarse hints.
\nResult: 1,416 documents → 11,302 sentence pairs
\nAttempted to generate pseudo-translations using only morphological metadata (G-Stem, 3ms, Preterite, dictionary entries). The outputs were too noisy and inconsistent. Including them in training hurt the score — dropped entirely.
\nThis was the single biggest score improvement. In addition to the competition-provided PDF corpus (AKT series, large monographs), I independently located, downloaded, OCR-processed, and extracted dozens of additional Old Assyrian cuneiform text editions from academic journals and open-access repositories.
\nPDF-extracted sources already carry sentence-level structure (each tablet entry in a publication presents transliteration and translation line-by-line or sentence-by-sentence). The LLM extraction pipeline preserves this structure automatically — no separate sentence-breaking step is needed for external data.
\n| Category | \nKey sources | \n
|---|---|
| AKT series | \nAKT 1–4, 7A/B, 9A/B, 10, 11A/B, 12 (AKT 5/6A-6E/8 already in official train.csv) | \n
| ArAn (Archivum Anatolicum) | \n~15 articles (albayrak, cecen, erol, kuzuoglu, etc.) | \n
| Belleten | \n~10 articles (bayram, donbaz, gunbatti, sever, etc.) | \n
| DTCF Dergisi | \n~8 articles | \n
| ATDD / TAD | \n~6 articles | \n
| HAL open access | \nFrench institutionally-archived papers | \n
| International journals | \nJCS, JNES, Anatolica, JEOL, AoF, RA, NABU, etc. | \n
| Large monographs | \nMichel LAPO19, Veenhof PIHANS 111, Barjamovic PIHANS 120, OIP27, etc. | \n
| Misc | \nWomen of Assur and Kanesh, ATHE Kienast, Assur-Nada 2002, etc. | \n
~60,654 external sentence pairs from 149 distinct sources after cleaning.
\n| Language | \nSentence pairs | \n
|---|---|
| Turkish (TR) | \n27,089 | \n
| English (EN) | \n21,683 | \n
| French (FR) | \n6,436 | \n
| German (DE) | \n5,413 | \n
| Dataset is in the attachment below. | \n\n |
Many Turkish- and German-language papers cover the same Akkadian tablets as English publications. During extraction, the LLM translated non-English text directly into English, so all training pairs remain Akkadian → English. This adds more Akkadian surface forms to training without introducing a multi-target problem.
When the same physical tablet (identified by IDs like Kt 88/k 71, CCT 4 12a) appears in both an English and a Turkish/German publication, only the English version is kept (Prefer-EN dedup), preventing conflicting translations for the same source.
Turkish-language sources are already amplified up to 3× through multi-version extraction (main + two backup runs), while fresh English-language academic papers have no backup copies — only a single extraction version. To compensate for this asymmetry, English-language academic papers are explicitly boosted 2× so they remain competitive in training exposure alongside the naturally-amplified Turkish sources.
\nThe LLM-based extraction pipeline was executed three separate times on each non-English source (Turkish, German, French, etc.), at different points in time. Each run independently translated the source text (TR/DE/FR → EN) and stored its outputs in a separate database directory (main, Backup_20260227, Backup_manual). Because LLM outputs vary across runs, the three passes yield slightly different English renditions for the same Akkadian input.
The dataset generator includes all three versions for any source that has backup rows (is_backup=1), providing implicit data augmentation through natural translation variance across extraction runs. Sources with no backup rows (single extraction only) use just the main version.
Sources already covered by the Sentence Breaker / Fixer pipeline (AKT 5, 6A–6E, 8 — English-language and forming the core of train.csv) were not given backup extraction copies. Running additional passes on these would produce near-duplicates of the breaker/fixer sentence pairs already in the dataset, adding no new information.
Akkadian transliteration orthography is highly inconsistent across publications. Without normalization, the same cuneiform sign appears in dozens of surface variants.
\nKey rules (all applied to source side):
\n| Issue | \nRule | \n
|---|---|
| sz / s, + consonant | \n→ š | \n
| , + consonant | \n→ ṭ | \n
| Subscript 2 after vowel (a2, e2…) | \nacute accent: a2→á, e2→é, i2→í, u2→ú | \n
| Subscript 3 after vowel (a3, e3…) | \ngrave accent: a3→à, e3→è, i3→ì, u3→ù | \n
| Higher subscripts (₄₅₆…) | \n→ plain ASCII digits | \n
| Hamza variants (ʾ, ʿ, ʼ) | \n→ deleted | \n
| ḫ / Ḫ | \n→ h / H (competition convention) | \n
| Determinatives (d), (ki), etc. | \n→ {d}, {ki} (cuneiform standard) | \n
| Other edge cases | \nadditional rules in text_normalization.py | \n
| The script of data cleaning is attached in the links below. | \n\n |
In addition to sentence-level pairs, consecutive sentences are concatenated into document-level training examples up to a 768-byte (UTF-8) ceiling. This:
\n| Parameter | \nValue | \n
|---|---|
| Model | \ngoogle/byt5-large | \n
| Max sequence length | \n768 bytes | \n
| Learning rate | \n7e-5 | \n
| Optimizer | \nAdafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) | \n
| Effective batch size | \n32 × 4 grad accumulation = 128 | \n
| Epochs | \n16 | \n
| Warmup steps | \n800 | \n
| LR scheduler | \nLinear | \n
| Weight decay | \n0.01 | \n
| Max grad norm | \n10.0 | \n
| Group by length | \nYes (sorts by input length to minimize padding waste) | \n
| Save epochs | \n14, 15, 16 (epoch 15 selected — highest score) | \n
| Parameter | \nValue | \n
|---|---|
| Model | \ngoogle/byt5-large | \n
| Max sequence length | \n768 bytes | \n
| Learning rate | \n2e-4 | \n
| Optimizer | \nAdafactor (β₁=0.9, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) | \n
| Effective batch size | \n32 × 8 grad accumulation = 256 | \n
| Epochs | \n8 (target ~5,500 steps) | \n
| Warmup ratio | \n1.1 epoch | \n
| LR scheduler | \nLinear | \n
| Weight decay | \n0.01 | \n
| Max grad norm | \n1.0 | \n
| Group by length | \nYes | \n
| Parameter | \nValue | \n
|---|---|
| Model | \ngoogle/byt5-large | \n
| Max sequence length | \n768 bytes | \n
| Learning rate | \n7e-5 | \n
| Optimizer | \nAdafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) | \n
| Effective batch size | \n32 × 4 grad accumulation = 128 | \n
| Epochs | \n14 (target ~11,000 steps) | \n
| Warmup ratio | \n1.1 epoch | \n
| LR scheduler | \nLinear | \n
| Weight decay | \n0.01 | \n
| Max grad norm | \n10.0 | \n
| Group by length | \nYes | \n
group_by_length=True batches samples of similar input length together, significantly reducing padding waste and speeding up training. However, it introduces a side effect: consecutive batches within the same length bucket have very similar sequence lengths, which creates non-uniform gradient magnitudes across steps — batches of long sequences produce much larger raw gradients than batches of short ones, causing instability in the loss curve.
The fix was setting β₁=0.9 in Adafactor. By default, Adafactor uses no first-moment estimate (β₁=None). Adding β₁=0.9 introduces a momentum term that carries a weighted average of recent gradients, providing inertia that dampens the sharp per-step variance introduced by length-grouped batching. This stabilized training without sacrificing the throughput gains from group_by_length.
| Dataset version | \nTraining pairs | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| Run 1 — Submitted | \n~90,522 | \n41.8–41.9 | \n40.9–41.0 | \n
| Run 2 — Same dataset, alt config | \n~90,522 | \n42.1 | \n40.9 | \n
| Run 3 — Best config (more data) | \n~100,143 | \n42.4 | \n40.4 | \n
Run 2 used the same dataset as Run 1 but with a different hyperparameter configuration, achieving a higher public score (42.1 vs 41.9). Run 3 added more data (~100k pairs) and reached the highest public score (42.4), though with a larger public/private gap (40.4 private). Run 1 was selected as the final submission based on its more stable public/private alignment.
\n| Technique | \nImpact | \n
|---|---|
| LLM-based sentence breaker/fixer pipeline | \nHigh — core training data | \n
| External academic PDF mining | \nHigh — biggest data expansion | \n
| Text normalization (sz/s,/ṭ, subscripts, ḫ) | \nMedium — removes surface noise | \n
| 768-byte document augmentation | \nMedium — contextual signal | \n
| English prioritization (prefer-EN dedup) | \nSmall positive | \n
| Multi-language (TR/DE) augmentation | \nSmall positive | \n
| English academic papers 2× boost | \nSmall positive | \n
translate Akkadian word to English: awilum → man, person pairs as training samples showed no benefit.This competition is a data bottleneck problem. The Akkadian→English dataset is tiny by NMT standards. My solution treats data construction as the primary engineering challenge: a systematic LLM pipeline for sentence alignment across ~3,000 documents, combined with large-scale external data mining from the academic literature on Old Assyrian cuneiform (~60 publications). The model itself is standard ByT5-large with no structural modifications.
\nThe key unrealized opportunity: the public/private LB gap indicated I should have kept collecting more external data rather than tuning the model. The private leaderboard reward clean, diverse data much more than architectural tricks.
", "messages": [ { "id": "3427798", "postDate": "03/24/2026 15:03:06", "content": "Score: 41.9 (Public) / 41.0 (Private) | Team: wukeneth | Model: google/byt5-large
\nA fully data-centric solution. The model is vanilla ByT5-large with no architectural changes. Every point of improvement came from building a larger, cleaner training corpus through:
\nThe competition provides three key files:
\nTraining directly on full-document pairs (Akkadian full text → full English translation) works but leaves significant value on the table. The model needs clean sentence-level pairs to learn tight correspondences. The core challenge was building those pairs at scale.
\nI partitioned all published texts into three non-overlapping sets and applied a different LLM strategy to each. The three sets are defined by strict set arithmetic on the official data files:
\n| Set | \nDefinition | \nLLM Strategy | \n
|---|---|---|
| Breaker | \n(U_train ∪ U_pub) − S_meta | \nSegment full translation into sentence-level pairs | \n
| Fixer | \nS_meta | \nRe-anchor corrupt fragments using OARE API word indices | \n
| Generator | \nU_pub − (U_train ∪ S_meta) | \nSynthesize translation from morphological metadata | \n
Input: Akkadian transliteration + full English translation from train.csv + OARE API word indices and physical line numbers.
Method: For each document, the OARE ID is used to look up per-word dictionary entries (normalized form, lemma) from the OARE word table. An LLM is then prompted with: the full Akkadian text, the per-word dictionary context, and the full English translation. It splits the English translation into sentence-sized chunks and assigns a word_range (start/end word index) to each chunk using the physical word coordinates as anchors. Because the source texts are clean (no pre-existing corrupt segmentation), the output is reliable.
Result: 1,558 documents → 9,378 sentence pairs
\nInput: Corrupt S_meta fragments + OARE API\normalized_form, word_index and line_number fields for each word.
\nMethod: Use the existing fragments as a weak reference only. Re-anchor each fragment by matching normalized word forms from the API to the physical tablet structure. The LLM reconstructs correct sentence boundaries from scratch, using the S_meta entries only as coarse hints.
\nResult: 1,416 documents → 11,302 sentence pairs
\nAttempted to generate pseudo-translations using only morphological metadata (G-Stem, 3ms, Preterite, dictionary entries). The outputs were too noisy and inconsistent. Including them in training hurt the score — dropped entirely.
\nThis was the single biggest score improvement. In addition to the competition-provided PDF corpus (AKT series, large monographs), I independently located, downloaded, OCR-processed, and extracted dozens of additional Old Assyrian cuneiform text editions from academic journals and open-access repositories.
\nPDF-extracted sources already carry sentence-level structure (each tablet entry in a publication presents transliteration and translation line-by-line or sentence-by-sentence). The LLM extraction pipeline preserves this structure automatically — no separate sentence-breaking step is needed for external data.
\n| Category | \nKey sources | \n
|---|---|
| AKT series | \nAKT 1–4, 7A/B, 9A/B, 10, 11A/B, 12 (AKT 5/6A-6E/8 already in official train.csv) | \n
| ArAn (Archivum Anatolicum) | \n~15 articles (albayrak, cecen, erol, kuzuoglu, etc.) | \n
| Belleten | \n~10 articles (bayram, donbaz, gunbatti, sever, etc.) | \n
| DTCF Dergisi | \n~8 articles | \n
| ATDD / TAD | \n~6 articles | \n
| HAL open access | \nFrench institutionally-archived papers | \n
| International journals | \nJCS, JNES, Anatolica, JEOL, AoF, RA, NABU, etc. | \n
| Large monographs | \nMichel LAPO19, Veenhof PIHANS 111, Barjamovic PIHANS 120, OIP27, etc. | \n
| Misc | \nWomen of Assur and Kanesh, ATHE Kienast, Assur-Nada 2002, etc. | \n
~60,654 external sentence pairs from 149 distinct sources after cleaning.
\n| Language | \nSentence pairs | \n
|---|---|
| Turkish (TR) | \n27,089 | \n
| English (EN) | \n21,683 | \n
| French (FR) | \n6,436 | \n
| German (DE) | \n5,413 | \n
| Dataset is in the attachment below. | \n\n |
Many Turkish- and German-language papers cover the same Akkadian tablets as English publications. During extraction, the LLM translated non-English text directly into English, so all training pairs remain Akkadian → English. This adds more Akkadian surface forms to training without introducing a multi-target problem.
When the same physical tablet (identified by IDs like Kt 88/k 71, CCT 4 12a) appears in both an English and a Turkish/German publication, only the English version is kept (Prefer-EN dedup), preventing conflicting translations for the same source.
Turkish-language sources are already amplified up to 3× through multi-version extraction (main + two backup runs), while fresh English-language academic papers have no backup copies — only a single extraction version. To compensate for this asymmetry, English-language academic papers are explicitly boosted 2× so they remain competitive in training exposure alongside the naturally-amplified Turkish sources.
\nThe LLM-based extraction pipeline was executed three separate times on each non-English source (Turkish, German, French, etc.), at different points in time. Each run independently translated the source text (TR/DE/FR → EN) and stored its outputs in a separate database directory (main, Backup_20260227, Backup_manual). Because LLM outputs vary across runs, the three passes yield slightly different English renditions for the same Akkadian input.
The dataset generator includes all three versions for any source that has backup rows (is_backup=1), providing implicit data augmentation through natural translation variance across extraction runs. Sources with no backup rows (single extraction only) use just the main version.
Sources already covered by the Sentence Breaker / Fixer pipeline (AKT 5, 6A–6E, 8 — English-language and forming the core of train.csv) were not given backup extraction copies. Running additional passes on these would produce near-duplicates of the breaker/fixer sentence pairs already in the dataset, adding no new information.
Akkadian transliteration orthography is highly inconsistent across publications. Without normalization, the same cuneiform sign appears in dozens of surface variants.
\nKey rules (all applied to source side):
\n| Issue | \nRule | \n
|---|---|
| sz / s, + consonant | \n→ š | \n
| , + consonant | \n→ ṭ | \n
| Subscript 2 after vowel (a2, e2…) | \nacute accent: a2→á, e2→é, i2→í, u2→ú | \n
| Subscript 3 after vowel (a3, e3…) | \ngrave accent: a3→à, e3→è, i3→ì, u3→ù | \n
| Higher subscripts (₄₅₆…) | \n→ plain ASCII digits | \n
| Hamza variants (ʾ, ʿ, ʼ) | \n→ deleted | \n
| ḫ / Ḫ | \n→ h / H (competition convention) | \n
| Determinatives (d), (ki), etc. | \n→ {d}, {ki} (cuneiform standard) | \n
| Other edge cases | \nadditional rules in text_normalization.py | \n
| The script of data cleaning is attached in the links below. | \n\n |
In addition to sentence-level pairs, consecutive sentences are concatenated into document-level training examples up to a 768-byte (UTF-8) ceiling. This:
\n| Parameter | \nValue | \n
|---|---|
| Model | \ngoogle/byt5-large | \n
| Max sequence length | \n768 bytes | \n
| Learning rate | \n7e-5 | \n
| Optimizer | \nAdafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) | \n
| Effective batch size | \n32 × 4 grad accumulation = 128 | \n
| Epochs | \n16 | \n
| Warmup steps | \n800 | \n
| LR scheduler | \nLinear | \n
| Weight decay | \n0.01 | \n
| Max grad norm | \n10.0 | \n
| Group by length | \nYes (sorts by input length to minimize padding waste) | \n
| Save epochs | \n14, 15, 16 (epoch 15 selected — highest score) | \n
| Parameter | \nValue | \n
|---|---|
| Model | \ngoogle/byt5-large | \n
| Max sequence length | \n768 bytes | \n
| Learning rate | \n2e-4 | \n
| Optimizer | \nAdafactor (β₁=0.9, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) | \n
| Effective batch size | \n32 × 8 grad accumulation = 256 | \n
| Epochs | \n8 (target ~5,500 steps) | \n
| Warmup ratio | \n1.1 epoch | \n
| LR scheduler | \nLinear | \n
| Weight decay | \n0.01 | \n
| Max grad norm | \n1.0 | \n
| Group by length | \nYes | \n
| Parameter | \nValue | \n
|---|---|
| Model | \ngoogle/byt5-large | \n
| Max sequence length | \n768 bytes | \n
| Learning rate | \n7e-5 | \n
| Optimizer | \nAdafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) | \n
| Effective batch size | \n32 × 4 grad accumulation = 128 | \n
| Epochs | \n14 (target ~11,000 steps) | \n
| Warmup ratio | \n1.1 epoch | \n
| LR scheduler | \nLinear | \n
| Weight decay | \n0.01 | \n
| Max grad norm | \n10.0 | \n
| Group by length | \nYes | \n
group_by_length=True batches samples of similar input length together, significantly reducing padding waste and speeding up training. However, it introduces a side effect: consecutive batches within the same length bucket have very similar sequence lengths, which creates non-uniform gradient magnitudes across steps — batches of long sequences produce much larger raw gradients than batches of short ones, causing instability in the loss curve.
The fix was setting β₁=0.9 in Adafactor. By default, Adafactor uses no first-moment estimate (β₁=None). Adding β₁=0.9 introduces a momentum term that carries a weighted average of recent gradients, providing inertia that dampens the sharp per-step variance introduced by length-grouped batching. This stabilized training without sacrificing the throughput gains from group_by_length.
| Dataset version | \nTraining pairs | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| Run 1 — Submitted | \n~90,522 | \n41.8–41.9 | \n40.9–41.0 | \n
| Run 2 — Same dataset, alt config | \n~90,522 | \n42.1 | \n40.9 | \n
| Run 3 — Best config (more data) | \n~100,143 | \n42.4 | \n40.4 | \n
Run 2 used the same dataset as Run 1 but with a different hyperparameter configuration, achieving a higher public score (42.1 vs 41.9). Run 3 added more data (~100k pairs) and reached the highest public score (42.4), though with a larger public/private gap (40.4 private). Run 1 was selected as the final submission based on its more stable public/private alignment.
\n| Technique | \nImpact | \n
|---|---|
| LLM-based sentence breaker/fixer pipeline | \nHigh — core training data | \n
| External academic PDF mining | \nHigh — biggest data expansion | \n
| Text normalization (sz/s,/ṭ, subscripts, ḫ) | \nMedium — removes surface noise | \n
| 768-byte document augmentation | \nMedium — contextual signal | \n
| English prioritization (prefer-EN dedup) | \nSmall positive | \n
| Multi-language (TR/DE) augmentation | \nSmall positive | \n
| English academic papers 2× boost | \nSmall positive | \n
translate Akkadian word to English: awilum → man, person pairs as training samples showed no benefit.This competition is a data bottleneck problem. The Akkadian→English dataset is tiny by NMT standards. My solution treats data construction as the primary engineering challenge: a systematic LLM pipeline for sentence alignment across ~3,000 documents, combined with large-scale external data mining from the academic literature on Old Assyrian cuneiform (~60 publications). The model itself is standard ByT5-large with no structural modifications.
\nThe key unrealized opportunity: the public/private LB gap indicated I should have kept collecting more external data rather than tuning the model. The private leaderboard reward clean, diverse data much more than architectural tricks.
", "rawMarkdown": "# [2nd Place] Data-Centric Akkadian NMT: LLM-Assisted Sentence Alignment + Large-Scale Academic PDF Mining\n\n**Score: 41.9 (Public) / 41.0 (Private) | Team: wukeneth | Model: google/byt5-large**\n\n---\n\n## TL;DR\n\nA fully data-centric solution. The model is vanilla **ByT5-large** with no architectural changes. Every point of improvement came from building a larger, cleaner training corpus through:\n\n1. A three-stage LLM pipeline to produce sentence-level alignments from the official data\n2. Manually sourcing and OCR-extracting ~60 academic publications on Old Assyrian cuneiform not covered by the official dataset\n3. Text normalization that unifies the highly variable orthography across publications\n\n---\n\n## The Problem\n\nThe competition provides three key files:\n\n- **train.csv** — ~6,500 documents with full English translations, but **no sentence-level alignment**\n- **published_texts.csv** — ~13,456 published texts, many untranslated\n- **Sentences_Oare_FirstWord_LinNum.csv (S_meta)** — pre-existing sentence fragments for ~1,700 documents, but the word indices and translations are **frequently misaligned or corrupt**\n\nTraining directly on full-document pairs (Akkadian full text → full English translation) works but leaves significant value on the table. The model needs clean **sentence-level pairs** to learn tight correspondences. The core challenge was building those pairs at scale.\n\n---\n\n## Data Pipeline: Three-Stage LLM Segmentation\n\nI partitioned all published texts into three non-overlapping sets and applied a different LLM strategy to each. The three sets are defined by strict set arithmetic on the official data files:\n\n| Set | Definition | LLM Strategy |\n| ------------- | -------------------------- | ------------------------------------------------------- |\n| **Breaker** | (U_train ∪ U_pub) − S_meta | Segment full translation into sentence-level pairs |\n| **Fixer** | S_meta | Re-anchor corrupt fragments using OARE API word indices |\n| **Generator** | U_pub − (U_train ∪ S_meta) | Synthesize translation from morphological metadata |\n\n### Set 1 — Sentence Breaker (clean full-text alignment)\n\n**Input**: Akkadian transliteration + full English translation from `train.csv` + OARE API word indices and physical line numbers.\n\n**Method**: For each document, the OARE ID is used to look up per-word dictionary entries (normalized form, lemma) from the OARE word table. An LLM is then prompted with: the full Akkadian text, the per-word dictionary context, and the full English translation. It splits the English translation into sentence-sized chunks and assigns a `word_range` (start/end word index) to each chunk using the physical word coordinates as anchors. Because the source texts are clean (no pre-existing corrupt segmentation), the output is reliable.\n\n**Result**: 1,558 documents → **9,378 sentence pairs**\n\n### Set 2 — Translation Fixer (misalignment correction)\n\n**Input**: Corrupt S_meta fragments + OARE API\normalized_form, word_index and line_number fields for each word.\n\n**Method**: Use the existing fragments as a _weak reference only_. Re-anchor each fragment by matching normalized word forms from the API to the physical tablet structure. The LLM reconstructs correct sentence boundaries from scratch, using the S_meta entries only as coarse hints.\n\n**Result**: 1,416 documents → **11,302 sentence pairs**\n\n### Set 3 — Translation Generator (attempted, did not help)\n\nAttempted to generate pseudo-translations using only morphological metadata (G-Stem, 3ms, Preterite, dictionary entries). The outputs were too noisy and inconsistent. Including them in training **hurt the score** — dropped entirely.\n\n---\n\n## External Data: Official PDFs + Independent Literature Mining\n\nThis was the single biggest score improvement. In addition to the competition-provided PDF corpus (AKT series, large monographs), I independently located, downloaded, OCR-processed, and extracted dozens of additional Old Assyrian cuneiform text editions from academic journals and open-access repositories.\n\nPDF-extracted sources already carry sentence-level structure (each tablet entry in a publication presents transliteration and translation line-by-line or sentence-by-sentence). The LLM extraction pipeline preserves this structure automatically — no separate sentence-breaking step is needed for external data.\n\n### Sources covered\n\n| Category | Key sources |\n| -------------------------- | -------------------------------------------------------------------------------- |\n| AKT series | AKT 1–4, 7A/B, 9A/B, 10, 11A/B, 12 (AKT 5/6A-6E/8 already in official train.csv) |\n| ArAn (Archivum Anatolicum) | ~15 articles (albayrak, cecen, erol, kuzuoglu, etc.) |\n| Belleten | ~10 articles (bayram, donbaz, gunbatti, sever, etc.) |\n| DTCF Dergisi | ~8 articles |\n| ATDD / TAD | ~6 articles |\n| HAL open access | French institutionally-archived papers |\n| International journals | JCS, JNES, Anatolica, JEOL, AoF, RA, NABU, etc. |\n| Large monographs | Michel LAPO19, Veenhof PIHANS 111, Barjamovic PIHANS 120, OIP27, etc. |\n| Misc | Women of Assur and Kanesh, ATHE Kienast, Assur-Nada 2002, etc. |\n\n**~60,654 external sentence pairs** from **149 distinct sources** after cleaning.\n\n### Language distribution and non-English handling\n\n| Language | Sentence pairs |\n| ------------ | -------------- |\n| Turkish (TR) | 27,089 |\n| English (EN) | 21,683 |\n| French (FR) | 6,436 |\n| German (DE) | 5,413 |\nDataset is in the attachment below.\n\nMany Turkish- and German-language papers cover the same Akkadian tablets as English publications. During extraction, the LLM **translated non-English text directly into English**, so all training pairs remain `Akkadian → English`. This adds more Akkadian surface forms to training without introducing a multi-target problem.\n\nWhen the same physical tablet (identified by IDs like `Kt 88/k 71`, `CCT 4 12a`) appears in both an English and a Turkish/German publication, only the English version is kept (_Prefer-EN dedup_), preventing conflicting translations for the same source.\n\nTurkish-language sources are already amplified up to **3×** through multi-version extraction (main + two backup runs), while fresh English-language academic papers have no backup copies — only a single extraction version. To compensate for this asymmetry, English-language academic papers are explicitly **boosted 2×** so they remain competitive in training exposure alongside the naturally-amplified Turkish sources.\n\n### Multi-version extraction\n\nThe LLM-based extraction pipeline was executed **three separate times** on each non-English source (Turkish, German, French, etc.), at different points in time. Each run independently translated the source text (TR/DE/FR → EN) and stored its outputs in a separate database directory (`main`, `Backup_20260227`, `Backup_manual`). Because LLM outputs vary across runs, the three passes yield slightly different English renditions for the same Akkadian input.\n\nThe dataset generator includes **all three versions** for any source that has backup rows (`is_backup=1`), providing implicit data augmentation through natural translation variance across extraction runs. Sources with no backup rows (single extraction only) use just the main version.\n\nSources already covered by the Sentence Breaker / Fixer pipeline (AKT 5, 6A–6E, 8 — English-language and forming the core of `train.csv`) were not given backup extraction copies. Running additional passes on these would produce near-duplicates of the breaker/fixer sentence pairs already in the dataset, adding no new information.\n\n---\n\n## Text Normalization\n\nAkkadian transliteration orthography is highly inconsistent across publications. Without normalization, the same cuneiform sign appears in dozens of surface variants.\n\nKey rules (all applied to source side):\n\n| Issue | Rule |\n| --------------------------------- | -------------------------------- |\n| sz / s, + consonant | → š |\n| , + consonant | → ṭ |\n| Subscript 2 after vowel (a2, e2...) | acute accent: a2→á, e2→é, i2→í, u2→ú |\n| Subscript 3 after vowel (a3, e3...) | grave accent: a3→à, e3→è, i3→ì, u3→ù |\n| Higher subscripts (₄₅₆...) | → plain ASCII digits |\n| Hamza variants (ʾ, ʿ, ʼ) | → deleted |\n| ḫ / Ḫ | → h / H (competition convention) |\n| Determinatives (d), (ki), etc. | → {d}, {ki} (cuneiform standard) |\n| Other edge cases | additional rules in `text_normalization.py` |\nThe script of data cleaning is attached in the links below.\n\n---\n\n## Document-Level Augmentation (768-byte chunks)\n\nIn addition to sentence-level pairs, consecutive sentences are concatenated into **document-level training examples** up to a 768-byte (UTF-8) ceiling. This:\n\n- Provides inter-sentence context during training\n- Creates implicit augmentation via variable chunking boundaries\n- Matches max_source_length = 768 of ByT5-large exactly (no truncation)\n\n---\n\n## Training Configuration\n\n### Run 1 — Submitted (Public 41.8 / Private 41.0, ~90,522 pairs)\n\n| Parameter | Value |\n| -------------------- | ----------------------------------------------------------------- |\n| Model | `google/byt5-large` |\n| Max sequence length | 768 bytes |\n| Learning rate | 7e-5 |\n| Optimizer | Adafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) |\n| Effective batch size | 32 × 4 grad accumulation = 128 |\n| Epochs | 16 |\n| Warmup steps | 800 |\n| LR scheduler | Linear |\n| Weight decay | 0.01 |\n| Max grad norm | 10.0 |\n| Group by length | Yes (sorts by input length to minimize padding waste) |\n| Save epochs | 14, 15, 16 (epoch 15 selected — highest score) |\n\n### Run 2 — Same dataset, alternative config (Public 42.1 / Private 40.9, ~90,522 pairs)\n\n| Parameter | Value |\n| -------------------- | ---------------------------------------------------------------- |\n| Model | `google/byt5-large` |\n| Max sequence length | 768 bytes |\n| Learning rate | 2e-4 |\n| Optimizer | Adafactor (β₁=0.9, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) |\n| Effective batch size | 32 × 8 grad accumulation = 256 |\n| Epochs | 8 (target ~5,500 steps) |\n| Warmup ratio | 1.1 epoch |\n| LR scheduler | Linear |\n| Weight decay | 0.01 |\n| Max grad norm | 1.0 |\n| Group by length | Yes |\n\n### Run 3 — Best config (Public 42.4 / Private 40.4, ~100,143 pairs)\n\n| Parameter | Value |\n| -------------------- | ----------------------------------------------------------------- |\n| Model | `google/byt5-large` |\n| Max sequence length | 768 bytes |\n| Learning rate | 7e-5 |\n| Optimizer | Adafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) |\n| Effective batch size | 32 × 4 grad accumulation = 128 |\n| Epochs | 14 (target ~11,000 steps) |\n| Warmup ratio | 1.1 epoch |\n| LR scheduler | Linear |\n| Weight decay | 0.01 |\n| Max grad norm | 10.0 |\n| Group by length | Yes |\n\n### Note on group_by_length and gradient stability\n\n`group_by_length=True` batches samples of similar input length together, significantly reducing padding waste and speeding up training. However, it introduces a side effect: consecutive batches within the same length bucket have very similar sequence lengths, which creates **non-uniform gradient magnitudes** across steps — batches of long sequences produce much larger raw gradients than batches of short ones, causing instability in the loss curve.\n\nThe fix was setting `β₁=0.9` in Adafactor. By default, Adafactor uses no first-moment estimate (β₁=None). Adding β₁=0.9 introduces a momentum term that carries a weighted average of recent gradients, providing inertia that dampens the sharp per-step variance introduced by length-grouped batching. This stabilized training without sacrificing the throughput gains from `group_by_length`.\n\n| Dataset version | Training pairs | Public LB | Private LB |\n| -------------------------------- | -------------- | --------- | ---------- |\n| Run 1 — Submitted | ~90,522 | 41.8–41.9 | 40.9–41.0 |\n| Run 2 — Same dataset, alt config | ~90,522 | 42.1 | 40.9 |\n| Run 3 — Best config (more data) | ~100,143 | 42.4 | 40.4 |\n\nRun 2 used the same dataset as Run 1 but with a different hyperparameter configuration, achieving a higher public score (42.1 vs 41.9). Run 3 added more data (~100k pairs) and reached the highest public score (42.4), though with a larger public/private gap (40.4 private). Run 1 was selected as the final submission based on its more stable public/private alignment.\n\n---\n\n## What Worked\n\n| Technique | Impact |\n| ------------------------------------------- | ---------------------------------- |\n| LLM-based sentence breaker/fixer pipeline | **High** — core training data |\n| External academic PDF mining | **High** — biggest data expansion |\n| Text normalization (sz/s,/ṭ, subscripts, ḫ) | **Medium** — removes surface noise |\n| 768-byte document augmentation | **Medium** — contextual signal |\n| English prioritization (prefer-EN dedup) | **Small positive** |\n| Multi-language (TR/DE) augmentation | **Small positive** |\n| English academic papers 2× boost | **Small positive** |\n\n## What Didn't Work\n\n- **Translation Generator**: LLM translations from morphological metadata alone (no reference English) were too noisy; including them hurt scores.\n- **Word-level dictionary augmentation**: Adding `translate Akkadian word to English: awilum → man, person` pairs as training samples showed no benefit.\n- **PN/GN post-processing**: Built a fuzzy-matching post-processor (OARE eBL dictionary, ~18k entries) to replace hallucinated name translations with correct transliterated forms. No meaningful improvement — likely because the test set comes from diverse sources with inconsistent name conventions, so no single normalization scheme aligns well; the matching logic was also complex enough to introduce its own errors. In hindsight, if the source OCR is correct, the model already learns the proper name forms from training data and post-hoc correction is unnecessary.\n\n---\n\n## Summary\n\nThis competition is a **data bottleneck problem**. The Akkadian→English dataset is tiny by NMT standards. My solution treats data construction as the primary engineering challenge: a systematic LLM pipeline for sentence alignment across ~3,000 documents, combined with large-scale external data mining from the academic literature on Old Assyrian cuneiform (~60 publications). The model itself is standard ByT5-large with no structural modifications.\n\nThe key unrealized opportunity: the public/private LB gap indicated I should have kept collecting more external data rather than tuning the model. The private leaderboard reward clean, diverse data much more than architectural tricks.", "votes": null }, { "id": "3427820", "postDate": "03/24/2026 15:43:39", "content": "Great solution! This was also my approach, but you executed it so much better.
", "rawMarkdown": "Great solution! This was also my approach, but you executed it so much better.", "votes": null }, { "id": "3427847", "postDate": "03/24/2026 16:07:23", "content": "Congratulations on 2nd place! This is a really impressive solo effort. Thanks for sharing such a clear and helpful write-up.
", "rawMarkdown": "Congratulations on 2nd place! This is a really impressive solo effort. Thanks for sharing such a clear and helpful write-up.", "votes": null }, { "id": "3427901", "postDate": "03/24/2026 16:55:44", "content": "Thanks !!!
", "rawMarkdown": "Thanks !!!", "votes": null }, { "id": "3427911", "postDate": "03/24/2026 17:05:21", "content": "Thanks! Good minds think alike
", "rawMarkdown": "Thanks! Good minds think alike", "votes": null }, { "id": "3428043", "postDate": "03/24/2026 21:07:52", "content": "Very well done! Do you have a link to you data?
", "rawMarkdown": "Very well done! Do you have a link to you data?", "votes": null }, { "id": "3428119", "postDate": "03/25/2026 01:11:10", "content": "Congratulations on achieving such a strong placement! I used a somewhat similar approach as well, but the amount of data I collected and the extent of my data processing were far behind yours.
", "rawMarkdown": "Congratulations on achieving such a strong placement! I used a somewhat similar approach as well, but the amount of data I collected and the extent of my data processing were far behind yours.", "votes": null }, { "id": "3428130", "postDate": "03/25/2026 02:02:51", "content": "in Three-Stage LLM Segmentation,Does the LLM use a commercial model or a local model, such as Qwen?
", "rawMarkdown": "in Three-Stage LLM Segmentation,Does the LLM use a commercial model or a local model, such as Qwen?", "votes": null }, { "id": "3428288", "postDate": "03/25/2026 08:04:30", "content": "https://www.kaggle.com/datasets/wukeneth/akkadian-v9-experi-dataset This's it.
", "rawMarkdown": "https://www.kaggle.com/datasets/wukeneth/akkadian-v9-experi-dataset This's it.", "votes": null }, { "id": "3428297", "postDate": "03/25/2026 08:21:17", "content": "No, I use the commercial model such as gpt-5.4.I try 2 small models, GLM-ocr and qwen-vl . The winners are aways the commercial models and almost the same. Perhaps commercial models have seen more of these parallel corpora in pre-training.
", "rawMarkdown": "No, I use the commercial model such as gpt-5.4.I try 2 small models, GLM-ocr and qwen-vl . The winners are aways the commercial models and almost the same. Perhaps commercial models have seen more of these parallel corpora in pre-training.", "votes": null }, { "id": "3428327", "postDate": "03/25/2026 09:32:45", "content": "Thanks KEWU!
\nI remember you had a significant jump 39+ and short after 40+. What was that?
", "rawMarkdown": "Thanks KEWU!\n\nI remember you had a significant jump 39+ and short after 40+. What was that?", "votes": null }, { "id": "3428378", "postDate": "03/25/2026 11:54:46", "content": "I think it might be the progress from only extracting pdfs(host provided) to some sources I found by ai externally such as Women_of_Assur_and_Kanesh.pdf. That's a lot of open books or articles or papers in the interne, so I don't list them all.But basically the key I find is to add more data and prevent overfitting.
", "rawMarkdown": "I think it might be the progress from only extracting pdfs(host provided) to some sources I found by ai externally such as Women_of_Assur_and_Kanesh.pdf. That's a lot of open books or articles or papers in the interne, so I don't list them all.But basically the key I find is to add more data and prevent overfitting.", "votes": null }, { "id": "3429359", "postDate": "03/26/2026 16:52:03", "content": "Congratulations on getting 2nd position. Thanks for sharing your valuable insights.
", "rawMarkdown": "Congratulations on getting 2nd position. Thanks for sharing your valuable insights.", "votes": null }, { "id": "3440083", "postDate": "04/11/2026 21:40:49", "content": "amazing work!\nI was taking a look at your inference code to try it out on some of my own data, and saw that some of the datasets in the notebook (https://www.kaggle.com/code/wukeneth/submission-word-dict) are private.\nI was wondering whether you could make them public / share them with me / share working inference code so i can try it out\nThanks again and congrats,\nshahar
", "rawMarkdown": "amazing work!\nI was taking a look at your inference code to try it out on some of my own data, and saw that some of the datasets in the notebook (https://www.kaggle.com/code/wukeneth/submission-word-dict) are private.\nI was wondering whether you could make them public / share them with me / share working inference code so i can try it out\nThanks again and congrats,\nshahar", "votes": null }, { "id": "3440476", "postDate": "04/12/2026 12:34:18", "content": "It is already public
", "rawMarkdown": "It is already public", "votes": null }, { "id": "3443627", "postDate": "04/16/2026 18:54:42", "content": "Thank you!!!
", "rawMarkdown": "Thank you!!!", "votes": null }, { "id": "3511023", "postDate": "08/10/2026 06:42:50", "content": "Hey, \nI see that text-normalization is indeed public, but the other dataset is not. Any chance you could make it public?\nThanks!
", "rawMarkdown": "Hey, \nI see that text-normalization is indeed public, but the other dataset is not. Any chance you could make it public?\nThanks!", "votes": null } ], "comments": [ { "id": 3427820, "author_name": "psilogram", "author_url": "", "post_date": "03/24/2026 15:43:39", "content": "Great solution! This was also my approach, but you executed it so much better.
", "votes": null, "replies": [ { "id": 3427911, "author_name": "wukeneth", "author_url": "", "post_date": "03/24/2026 17:05:21", "content": "Thanks! Good minds think alike
", "votes": null, "replies": [] } ] }, { "id": 3427847, "author_name": "musapeker", "author_url": "", "post_date": "03/24/2026 16:07:23", "content": "Congratulations on 2nd place! This is a really impressive solo effort. Thanks for sharing such a clear and helpful write-up.
", "votes": null, "replies": [ { "id": 3427901, "author_name": "wukeneth", "author_url": "", "post_date": "03/24/2026 16:55:44", "content": "Thanks !!!
", "votes": null, "replies": [] } ] }, { "id": 3428043, "author_name": "leedrake", "author_url": "", "post_date": "03/24/2026 21:07:52", "content": "Very well done! Do you have a link to you data?
", "votes": null, "replies": [ { "id": 3428288, "author_name": "wukeneth", "author_url": "", "post_date": "03/25/2026 08:04:30", "content": "https://www.kaggle.com/datasets/wukeneth/akkadian-v9-experi-dataset This's it.
", "votes": null, "replies": [ { "id": 3443627, "author_name": "leedrake", "author_url": "", "post_date": "04/16/2026 18:54:42", "content": "Thank you!!!
", "votes": null, "replies": [] } ] } ] }, { "id": 3428119, "author_name": "guoyihao", "author_url": "", "post_date": "03/25/2026 01:11:10", "content": "Congratulations on achieving such a strong placement! I used a somewhat similar approach as well, but the amount of data I collected and the extent of my data processing were far behind yours.
", "votes": null, "replies": [] }, { "id": 3428130, "author_name": "aiwody", "author_url": "", "post_date": "03/25/2026 02:02:51", "content": "in Three-Stage LLM Segmentation,Does the LLM use a commercial model or a local model, such as Qwen?
", "votes": null, "replies": [ { "id": 3428297, "author_name": "wukeneth", "author_url": "", "post_date": "03/25/2026 08:21:17", "content": "No, I use the commercial model such as gpt-5.4.I try 2 small models, GLM-ocr and qwen-vl . The winners are aways the commercial models and almost the same. Perhaps commercial models have seen more of these parallel corpora in pre-training.
", "votes": null, "replies": [] } ] }, { "id": 3428327, "author_name": "samson8", "author_url": "", "post_date": "03/25/2026 09:32:45", "content": "Thanks KEWU!
\nI remember you had a significant jump 39+ and short after 40+. What was that?
", "votes": null, "replies": [ { "id": 3428378, "author_name": "wukeneth", "author_url": "", "post_date": "03/25/2026 11:54:46", "content": "I think it might be the progress from only extracting pdfs(host provided) to some sources I found by ai externally such as Women_of_Assur_and_Kanesh.pdf. That's a lot of open books or articles or papers in the interne, so I don't list them all.But basically the key I find is to add more data and prevent overfitting.
", "votes": null, "replies": [] } ] }, { "id": 3429359, "author_name": "haseebkn", "author_url": "", "post_date": "03/26/2026 16:52:03", "content": "Congratulations on getting 2nd position. Thanks for sharing your valuable insights.
", "votes": null, "replies": [] }, { "id": 3440083, "author_name": "shaharspencer", "author_url": "", "post_date": "04/11/2026 21:40:49", "content": "amazing work!\nI was taking a look at your inference code to try it out on some of my own data, and saw that some of the datasets in the notebook (https://www.kaggle.com/code/wukeneth/submission-word-dict) are private.\nI was wondering whether you could make them public / share them with me / share working inference code so i can try it out\nThanks again and congrats,\nshahar
", "votes": null, "replies": [ { "id": 3440476, "author_name": "wukeneth", "author_url": "", "post_date": "04/12/2026 12:34:18", "content": "It is already public
", "votes": null, "replies": [ { "id": 3511023, "author_name": "shaharspencer", "author_url": "", "post_date": "08/10/2026 06:42:50", "content": "Hey, \nI see that text-normalization is indeed public, but the other dataset is not. Any chance you could make it public?\nThanks!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3427798": "# [2nd Place] Data-Centric Akkadian NMT: LLM-Assisted Sentence Alignment + Large-Scale Academic PDF Mining\n\n**Score: 41.9 (Public) / 41.0 (Private) | Team: wukeneth | Model: google/byt5-large**\n\n---\n\n## TL;DR\n\nA fully data-centric solution. The model is vanilla **ByT5-large** with no architectural changes. Every point of improvement came from building a larger, cleaner training corpus through:\n\n1. A three-stage LLM pipeline to produce sentence-level alignments from the official data\n2. Manually sourcing and OCR-extracting ~60 academic publications on Old Assyrian cuneiform not covered by the official dataset\n3. Text normalization that unifies the highly variable orthography across publications\n\n---\n\n## The Problem\n\nThe competition provides three key files:\n\n- **train.csv** — ~6,500 documents with full English translations, but **no sentence-level alignment**\n- **published_texts.csv** — ~13,456 published texts, many untranslated\n- **Sentences_Oare_FirstWord_LinNum.csv (S_meta)** — pre-existing sentence fragments for ~1,700 documents, but the word indices and translations are **frequently misaligned or corrupt**\n\nTraining directly on full-document pairs (Akkadian full text → full English translation) works but leaves significant value on the table. The model needs clean **sentence-level pairs** to learn tight correspondences. The core challenge was building those pairs at scale.\n\n---\n\n## Data Pipeline: Three-Stage LLM Segmentation\n\nI partitioned all published texts into three non-overlapping sets and applied a different LLM strategy to each. The three sets are defined by strict set arithmetic on the official data files:\n\n| Set | Definition | LLM Strategy |\n| ------------- | -------------------------- | ------------------------------------------------------- |\n| **Breaker** | (U_train ∪ U_pub) − S_meta | Segment full translation into sentence-level pairs |\n| **Fixer** | S_meta | Re-anchor corrupt fragments using OARE API word indices |\n| **Generator** | U_pub − (U_train ∪ S_meta) | Synthesize translation from morphological metadata |\n\n### Set 1 — Sentence Breaker (clean full-text alignment)\n\n**Input**: Akkadian transliteration + full English translation from `train.csv` + OARE API word indices and physical line numbers.\n\n**Method**: For each document, the OARE ID is used to look up per-word dictionary entries (normalized form, lemma) from the OARE word table. An LLM is then prompted with: the full Akkadian text, the per-word dictionary context, and the full English translation. It splits the English translation into sentence-sized chunks and assigns a `word_range` (start/end word index) to each chunk using the physical word coordinates as anchors. Because the source texts are clean (no pre-existing corrupt segmentation), the output is reliable.\n\n**Result**: 1,558 documents → **9,378 sentence pairs**\n\n### Set 2 — Translation Fixer (misalignment correction)\n\n**Input**: Corrupt S_meta fragments + OARE API\normalized_form, word_index and line_number fields for each word.\n\n**Method**: Use the existing fragments as a _weak reference only_. Re-anchor each fragment by matching normalized word forms from the API to the physical tablet structure. The LLM reconstructs correct sentence boundaries from scratch, using the S_meta entries only as coarse hints.\n\n**Result**: 1,416 documents → **11,302 sentence pairs**\n\n### Set 3 — Translation Generator (attempted, did not help)\n\nAttempted to generate pseudo-translations using only morphological metadata (G-Stem, 3ms, Preterite, dictionary entries). The outputs were too noisy and inconsistent. Including them in training **hurt the score** — dropped entirely.\n\n---\n\n## External Data: Official PDFs + Independent Literature Mining\n\nThis was the single biggest score improvement. In addition to the competition-provided PDF corpus (AKT series, large monographs), I independently located, downloaded, OCR-processed, and extracted dozens of additional Old Assyrian cuneiform text editions from academic journals and open-access repositories.\n\nPDF-extracted sources already carry sentence-level structure (each tablet entry in a publication presents transliteration and translation line-by-line or sentence-by-sentence). The LLM extraction pipeline preserves this structure automatically — no separate sentence-breaking step is needed for external data.\n\n### Sources covered\n\n| Category | Key sources |\n| -------------------------- | -------------------------------------------------------------------------------- |\n| AKT series | AKT 1–4, 7A/B, 9A/B, 10, 11A/B, 12 (AKT 5/6A-6E/8 already in official train.csv) |\n| ArAn (Archivum Anatolicum) | ~15 articles (albayrak, cecen, erol, kuzuoglu, etc.) |\n| Belleten | ~10 articles (bayram, donbaz, gunbatti, sever, etc.) |\n| DTCF Dergisi | ~8 articles |\n| ATDD / TAD | ~6 articles |\n| HAL open access | French institutionally-archived papers |\n| International journals | JCS, JNES, Anatolica, JEOL, AoF, RA, NABU, etc. |\n| Large monographs | Michel LAPO19, Veenhof PIHANS 111, Barjamovic PIHANS 120, OIP27, etc. |\n| Misc | Women of Assur and Kanesh, ATHE Kienast, Assur-Nada 2002, etc. |\n\n**~60,654 external sentence pairs** from **149 distinct sources** after cleaning.\n\n### Language distribution and non-English handling\n\n| Language | Sentence pairs |\n| ------------ | -------------- |\n| Turkish (TR) | 27,089 |\n| English (EN) | 21,683 |\n| French (FR) | 6,436 |\n| German (DE) | 5,413 |\nDataset is in the attachment below.\n\nMany Turkish- and German-language papers cover the same Akkadian tablets as English publications. During extraction, the LLM **translated non-English text directly into English**, so all training pairs remain `Akkadian → English`. This adds more Akkadian surface forms to training without introducing a multi-target problem.\n\nWhen the same physical tablet (identified by IDs like `Kt 88/k 71`, `CCT 4 12a`) appears in both an English and a Turkish/German publication, only the English version is kept (_Prefer-EN dedup_), preventing conflicting translations for the same source.\n\nTurkish-language sources are already amplified up to **3×** through multi-version extraction (main + two backup runs), while fresh English-language academic papers have no backup copies — only a single extraction version. To compensate for this asymmetry, English-language academic papers are explicitly **boosted 2×** so they remain competitive in training exposure alongside the naturally-amplified Turkish sources.\n\n### Multi-version extraction\n\nThe LLM-based extraction pipeline was executed **three separate times** on each non-English source (Turkish, German, French, etc.), at different points in time. Each run independently translated the source text (TR/DE/FR → EN) and stored its outputs in a separate database directory (`main`, `Backup_20260227`, `Backup_manual`). Because LLM outputs vary across runs, the three passes yield slightly different English renditions for the same Akkadian input.\n\nThe dataset generator includes **all three versions** for any source that has backup rows (`is_backup=1`), providing implicit data augmentation through natural translation variance across extraction runs. Sources with no backup rows (single extraction only) use just the main version.\n\nSources already covered by the Sentence Breaker / Fixer pipeline (AKT 5, 6A–6E, 8 — English-language and forming the core of `train.csv`) were not given backup extraction copies. Running additional passes on these would produce near-duplicates of the breaker/fixer sentence pairs already in the dataset, adding no new information.\n\n---\n\n## Text Normalization\n\nAkkadian transliteration orthography is highly inconsistent across publications. Without normalization, the same cuneiform sign appears in dozens of surface variants.\n\nKey rules (all applied to source side):\n\n| Issue | Rule |\n| --------------------------------- | -------------------------------- |\n| sz / s, + consonant | → š |\n| , + consonant | → ṭ |\n| Subscript 2 after vowel (a2, e2...) | acute accent: a2→á, e2→é, i2→í, u2→ú |\n| Subscript 3 after vowel (a3, e3...) | grave accent: a3→à, e3→è, i3→ì, u3→ù |\n| Higher subscripts (₄₅₆...) | → plain ASCII digits |\n| Hamza variants (ʾ, ʿ, ʼ) | → deleted |\n| ḫ / Ḫ | → h / H (competition convention) |\n| Determinatives (d), (ki), etc. | → {d}, {ki} (cuneiform standard) |\n| Other edge cases | additional rules in `text_normalization.py` |\nThe script of data cleaning is attached in the links below.\n\n---\n\n## Document-Level Augmentation (768-byte chunks)\n\nIn addition to sentence-level pairs, consecutive sentences are concatenated into **document-level training examples** up to a 768-byte (UTF-8) ceiling. This:\n\n- Provides inter-sentence context during training\n- Creates implicit augmentation via variable chunking boundaries\n- Matches max_source_length = 768 of ByT5-large exactly (no truncation)\n\n---\n\n## Training Configuration\n\n### Run 1 — Submitted (Public 41.8 / Private 41.0, ~90,522 pairs)\n\n| Parameter | Value |\n| -------------------- | ----------------------------------------------------------------- |\n| Model | `google/byt5-large` |\n| Max sequence length | 768 bytes |\n| Learning rate | 7e-5 |\n| Optimizer | Adafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) |\n| Effective batch size | 32 × 4 grad accumulation = 128 |\n| Epochs | 16 |\n| Warmup steps | 800 |\n| LR scheduler | Linear |\n| Weight decay | 0.01 |\n| Max grad norm | 10.0 |\n| Group by length | Yes (sorts by input length to minimize padding waste) |\n| Save epochs | 14, 15, 16 (epoch 15 selected — highest score) |\n\n### Run 2 — Same dataset, alternative config (Public 42.1 / Private 40.9, ~90,522 pairs)\n\n| Parameter | Value |\n| -------------------- | ---------------------------------------------------------------- |\n| Model | `google/byt5-large` |\n| Max sequence length | 768 bytes |\n| Learning rate | 2e-4 |\n| Optimizer | Adafactor (β₁=0.9, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) |\n| Effective batch size | 32 × 8 grad accumulation = 256 |\n| Epochs | 8 (target ~5,500 steps) |\n| Warmup ratio | 1.1 epoch |\n| LR scheduler | Linear |\n| Weight decay | 0.01 |\n| Max grad norm | 1.0 |\n| Group by length | Yes |\n\n### Run 3 — Best config (Public 42.4 / Private 40.4, ~100,143 pairs)\n\n| Parameter | Value |\n| -------------------- | ----------------------------------------------------------------- |\n| Model | `google/byt5-large` |\n| Max sequence length | 768 bytes |\n| Learning rate | 7e-5 |\n| Optimizer | Adafactor (β₁=None, decay_rate=-0.8, clip=1.0, eps=(1e-30, 1e-3)) |\n| Effective batch size | 32 × 4 grad accumulation = 128 |\n| Epochs | 14 (target ~11,000 steps) |\n| Warmup ratio | 1.1 epoch |\n| LR scheduler | Linear |\n| Weight decay | 0.01 |\n| Max grad norm | 10.0 |\n| Group by length | Yes |\n\n### Note on group_by_length and gradient stability\n\n`group_by_length=True` batches samples of similar input length together, significantly reducing padding waste and speeding up training. However, it introduces a side effect: consecutive batches within the same length bucket have very similar sequence lengths, which creates **non-uniform gradient magnitudes** across steps — batches of long sequences produce much larger raw gradients than batches of short ones, causing instability in the loss curve.\n\nThe fix was setting `β₁=0.9` in Adafactor. By default, Adafactor uses no first-moment estimate (β₁=None). Adding β₁=0.9 introduces a momentum term that carries a weighted average of recent gradients, providing inertia that dampens the sharp per-step variance introduced by length-grouped batching. This stabilized training without sacrificing the throughput gains from `group_by_length`.\n\n| Dataset version | Training pairs | Public LB | Private LB |\n| -------------------------------- | -------------- | --------- | ---------- |\n| Run 1 — Submitted | ~90,522 | 41.8–41.9 | 40.9–41.0 |\n| Run 2 — Same dataset, alt config | ~90,522 | 42.1 | 40.9 |\n| Run 3 — Best config (more data) | ~100,143 | 42.4 | 40.4 |\n\nRun 2 used the same dataset as Run 1 but with a different hyperparameter configuration, achieving a higher public score (42.1 vs 41.9). Run 3 added more data (~100k pairs) and reached the highest public score (42.4), though with a larger public/private gap (40.4 private). Run 1 was selected as the final submission based on its more stable public/private alignment.\n\n---\n\n## What Worked\n\n| Technique | Impact |\n| ------------------------------------------- | ---------------------------------- |\n| LLM-based sentence breaker/fixer pipeline | **High** — core training data |\n| External academic PDF mining | **High** — biggest data expansion |\n| Text normalization (sz/s,/ṭ, subscripts, ḫ) | **Medium** — removes surface noise |\n| 768-byte document augmentation | **Medium** — contextual signal |\n| English prioritization (prefer-EN dedup) | **Small positive** |\n| Multi-language (TR/DE) augmentation | **Small positive** |\n| English academic papers 2× boost | **Small positive** |\n\n## What Didn't Work\n\n- **Translation Generator**: LLM translations from morphological metadata alone (no reference English) were too noisy; including them hurt scores.\n- **Word-level dictionary augmentation**: Adding `translate Akkadian word to English: awilum → man, person` pairs as training samples showed no benefit.\n- **PN/GN post-processing**: Built a fuzzy-matching post-processor (OARE eBL dictionary, ~18k entries) to replace hallucinated name translations with correct transliterated forms. No meaningful improvement — likely because the test set comes from diverse sources with inconsistent name conventions, so no single normalization scheme aligns well; the matching logic was also complex enough to introduce its own errors. In hindsight, if the source OCR is correct, the model already learns the proper name forms from training data and post-hoc correction is unnecessary.\n\n---\n\n## Summary\n\nThis competition is a **data bottleneck problem**. The Akkadian→English dataset is tiny by NMT standards. My solution treats data construction as the primary engineering challenge: a systematic LLM pipeline for sentence alignment across ~3,000 documents, combined with large-scale external data mining from the academic literature on Old Assyrian cuneiform (~60 publications). The model itself is standard ByT5-large with no structural modifications.\n\nThe key unrealized opportunity: the public/private LB gap indicated I should have kept collecting more external data rather than tuning the model. The private leaderboard reward clean, diverse data much more than architectural tricks.", "3427820": "Great solution! This was also my approach, but you executed it so much better.", "3427847": "Congratulations on 2nd place! This is a really impressive solo effort. Thanks for sharing such a clear and helpful write-up.", "3427901": "Thanks !!!", "3427911": "Thanks! Good minds think alike", "3428043": "Very well done! Do you have a link to you data?", "3428119": "Congratulations on achieving such a strong placement! I used a somewhat similar approach as well, but the amount of data I collected and the extent of my data processing were far behind yours.", "3428130": "in Three-Stage LLM Segmentation,Does the LLM use a commercial model or a local model, such as Qwen?", "3428288": "https://www.kaggle.com/datasets/wukeneth/akkadian-v9-experi-dataset This's it.", "3428297": "No, I use the commercial model such as gpt-5.4.I try 2 small models, GLM-ocr and qwen-vl . The winners are aways the commercial models and almost the same. Perhaps commercial models have seen more of these parallel corpora in pre-training.", "3428327": "Thanks KEWU!\n\nI remember you had a significant jump 39+ and short after 40+. What was that?", "3428378": "I think it might be the progress from only extracting pdfs(host provided) to some sources I found by ai externally such as Women_of_Assur_and_Kanesh.pdf. That's a lot of open books or articles or papers in the interne, so I don't list them all.But basically the key I find is to add more data and prevent overfitting.", "3429359": "Congratulations on getting 2nd position. Thanks for sharing your valuable insights.", "3440083": "amazing work!\nI was taking a look at your inference code to try it out on some of my own data, and saw that some of the datasets in the notebook (https://www.kaggle.com/code/wukeneth/submission-word-dict) are private.\nI was wondering whether you could make them public / share them with me / share working inference code so i can try it out\nThanks again and congrats,\nshahar", "3440476": "It is already public", "3443627": "Thank you!!!", "3511023": "Hey, \nI see that text-normalization is indeed public, but the other dataset is not. Any chance you could make it public?\nThanks!" }, "source": "meta" }