{
  "id": 684203,
  "title": "Multi-phase curriculum training ",
  "url": "/competitions/deep-past-initiative-machine-translation/writeups/multi-phase-curriculum-training",
  "author_name": "",
  "post_date": "2026-03-24T02:01:50.840Z",
  "votes": 5,
  "comment_count": 1,
  "views": 0,
  "content": "<p><strong>Team members:</strong> <a href=\"https://www.kaggle.com/arnavmishra02\" target=\"_blank\">@arnavmishra02</a></p>\n<p>Huge thanks to <a href=\"https://www.kaggle.com/takamichitoda\" target=\"_blank\">@takamichitoda</a> for the professional base code that served as the foundation for this solution, and to <a href=\"https://www.kaggle.com/phucthaiv02\" target=\"_blank\">@phucthaiv02</a> for the PDF-extracted Akkadian translation dataset 🙏</p>\n<p><strong>Model used:</strong> <code>google/byt5-base</code></p>\n<p>Why ByT5? Akkadian transliterations are full of diacritics (š, ṣ, ṭ, ḫ), subscript numbers, and Unicode fractions (⅓, ⅔). Subword tokenizers choke on these. ByT5 operates at the byte level — every character is a token — so it handles all of this natively without vocabulary issues.</p>\n<h2>Datasets</h2>\n<p>We used <strong>7 data sources</strong> of increasing quality, trained in curriculum order:</p>\n<table>\n<thead>\n<tr>\n<th>Phase</th>\n<th>Source</th>\n<th>What it is</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>OA Lexicon + eBL Dictionary + Onomasticon</td>\n<td>Word-level pairs (form → English definition, name spellings → canonical names)</td>\n</tr>\n<tr>\n<td>0.5</td>\n<td>Gemini back-translations</td>\n<td>Synthetic sentence pairs from back-translating English into Akkadian</td>\n</tr>\n<tr>\n<td>0.9</td>\n<td>HuggingFace <code>phucthaiv02/akkadian-translation</code></td>\n<td>Translation pairs extracted from academic PDFs</td>\n</tr>\n<tr>\n<td>1</td>\n<td>Gemini pseudo-labels (OARE corpus)</td>\n<td>LLM-generated translations of the full OARE transliteration corpus</td>\n</tr>\n<tr>\n<td>1.5</td>\n<td>CDLI 10K + Gemini pseudo-labels</td>\n<td>10K CDLI texts translated by Gemini</td>\n</tr>\n<tr>\n<td>1.9</td>\n<td>Published OARE translations</td>\n<td>Translations from scholarly publications</td>\n</tr>\n<tr>\n<td>2</td>\n<td>Competition <code>train.csv</code></td>\n<td>Human gold-standard annotations</td>\n</tr>\n</tbody>\n</table>\n<p>All phases were augmented with <strong>Joint Dropout (JD)</strong> — perturbed copies of training pairs where tokens are randomly dropped from both source and target. This is huge for cuneiform because tablets are almost always damaged/fragmentary, so the model needs to handle missing text gracefully. Adding JD consistently boosted the LB by <strong>~0.2 per dataset</strong> — small individually, but it compounds across all 7 phases.</p>\n<p>Anti-leak filtering was applied rigorously: JD pairs matching validation set IDs or translations were removed before training in every phase.</p>\n<h2>Preprocessing</h2>\n<p>Mostly based on the host's discussion post, plus:</p>\n<ul>\n<li>All break/gap markers (large break, N broken lines, <code>…</code>, <code>[x]</code>, lone <code>x</code>) → single <code>&lt;gap&gt;</code> token; consecutive gaps collapsed</li>\n<li>Determinatives preserved: <code>(d)</code> → <code>{d}</code>, <code>(ki)</code> → <code>{ki}</code></li>\n<li><code>Ḫ</code>/<code>ḫ</code> → <code>H</code>/<code>h</code>, subscript digits → regular digits</li>\n<li>Decimal fractions → Unicode fractions (<code>0.3333</code> → <code>⅓</code>)</li>\n<li><code>KÙ.B.</code> → <code>KÙ.BABBAR</code> (silver abbreviation fix)</li>\n<li>Target side: <code>-gold</code> → <code>pašallum gold</code>, <code>-tax</code> → <code>šadduātum tax</code>, <code>-textiles</code> → <code>kutānum textiles</code></li>\n<li>Shekel fraction arithmetic: <code>1/12 (shekel)</code> → <code>15 grains</code>, <code>5/12 shekel</code> → <code>⅓ shekel 15 grains</code>, etc.</li>\n<li>Roman numeral months → Arabic: <code>month XII</code> → <code>month 12</code></li>\n<li><code>PN</code> → <code>&lt;gap&gt;</code>, removed linguistic annotations (<code>fem.</code>, <code>sing.</code>, <code>pl.</code>, <code>(?)</code>)</li>\n</ul>\n<p>A simple sentence aligner splits multi-sentence documents into 1:1 line pairs when source lines and target sentences match in count.</p>\n<h2>Training</h2>\n<p><strong>7-phase curriculum</strong> — each phase loads the previous checkpoint and continues training. The idea: start noisy and broad, finish clean and precise.</p>\n<table>\n<thead>\n<tr>\n<th>Phase</th>\n<th>Epochs</th>\n<th>LR</th>\n<th>Key detail</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0 (Dictionary)</td>\n<td>10</td>\n<td>2e-4</td>\n<td>Teaches word-level mappings</td>\n</tr>\n<tr>\n<td>0.5 (Back-trans)</td>\n<td>10</td>\n<td>1e-4</td>\n<td>Synthetic fluency</td>\n</tr>\n<tr>\n<td>0.9 (PDF)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Academic style exposure</td>\n</tr>\n<tr>\n<td>1 (Gemini)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Broad OARE coverage</td>\n</tr>\n<tr>\n<td>1.5 (CDLI)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Broader cuneiform genres</td>\n</tr>\n<tr>\n<td>1.9 (Published)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Scholarly quality</td>\n</tr>\n<tr>\n<td>2 (Gold)</td>\n<td>9</td>\n<td>1e-5</td>\n<td>Early stopping (patience=3), best checkpoint by eval_loss</td>\n</tr>\n</tbody>\n</table>\n<p>All phases: batch size 16, gradient accumulation 2 (effective 32), label smoothing 0.1, weight decay 0.01, cosine scheduler.</p>\n<p>Phase 2 uses a <strong>10× lower LR</strong> (1e-5) to prevent catastrophic forgetting of all the pretraining knowledge. Early stopping + <code>load_best_model_at_end</code> ensures we pick the best checkpoint.</p>\n<p>Prefix for all inputs: <code>\"translate Akkadian to English: \"</code></p>\n<h2>Validation</h2>\n<p>Train/val split: 90/10 random split (seed=42) on each phase's data independently. Phase 2 evaluates with <code>eval_loss</code> for checkpoint selection, plus BLEU, chrF++, and geometric mean <code>√(BLEU × chrF++)</code> for reporting.</p>\n<h2>What Worked</h2>\n<ul>\n<li><strong>ByT5 byte-level approach</strong> — no tokenizer headaches with Akkadian special characters</li>\n<li><strong>Curriculum learning</strong> — noisy-to-clean progression was key; without the pretraining phases, gold-only training underperforms significantly</li>\n<li><strong>Joint Dropout augmentation</strong> — essential for robustness to fragmentary tablets</li>\n<li><strong>Domain-specific preprocessing</strong> — fraction normalization, gap standardization, and determinative handling made a real difference</li>\n<li><strong>Low LR for final fine-tuning</strong> — 1e-5 with early stopping was the sweet spot for not destroying pretrained knowledge</li>\n</ul>\n<h2>Failed / Not Tried</h2>\n<ul>\n<li>ByT5-Large or bigger (not investigated in depth)</li>\n<li>Ensemble decoding across phase checkpoints</li>\n<li>Retrieval-augmented generation with the dictionary at inference time</li>\n<li>Using a separate LLM for post-processing / deduplication</li>\n</ul>",
  "messages": [
    {
      "id": "3427360",
      "postDate": "03/24/2026 02:01:30",
      "content": "<p><strong>Team members:</strong> <a href=\"https://www.kaggle.com/arnavmishra02\" target=\"_blank\">@arnavmishra02</a></p>\n<p>Huge thanks to <a href=\"https://www.kaggle.com/takamichitoda\" target=\"_blank\">@takamichitoda</a> for the professional base code that served as the foundation for this solution, and to <a href=\"https://www.kaggle.com/phucthaiv02\" target=\"_blank\">@phucthaiv02</a> for the PDF-extracted Akkadian translation dataset 🙏</p>\n<p><strong>Model used:</strong> <code>google/byt5-base</code></p>\n<p>Why ByT5? Akkadian transliterations are full of diacritics (š, ṣ, ṭ, ḫ), subscript numbers, and Unicode fractions (⅓, ⅔). Subword tokenizers choke on these. ByT5 operates at the byte level — every character is a token — so it handles all of this natively without vocabulary issues.</p>\n<h2>Datasets</h2>\n<p>We used <strong>7 data sources</strong> of increasing quality, trained in curriculum order:</p>\n<table>\n<thead>\n<tr>\n<th>Phase</th>\n<th>Source</th>\n<th>What it is</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>OA Lexicon + eBL Dictionary + Onomasticon</td>\n<td>Word-level pairs (form → English definition, name spellings → canonical names)</td>\n</tr>\n<tr>\n<td>0.5</td>\n<td>Gemini back-translations</td>\n<td>Synthetic sentence pairs from back-translating English into Akkadian</td>\n</tr>\n<tr>\n<td>0.9</td>\n<td>HuggingFace <code>phucthaiv02/akkadian-translation</code></td>\n<td>Translation pairs extracted from academic PDFs</td>\n</tr>\n<tr>\n<td>1</td>\n<td>Gemini pseudo-labels (OARE corpus)</td>\n<td>LLM-generated translations of the full OARE transliteration corpus</td>\n</tr>\n<tr>\n<td>1.5</td>\n<td>CDLI 10K + Gemini pseudo-labels</td>\n<td>10K CDLI texts translated by Gemini</td>\n</tr>\n<tr>\n<td>1.9</td>\n<td>Published OARE translations</td>\n<td>Translations from scholarly publications</td>\n</tr>\n<tr>\n<td>2</td>\n<td>Competition <code>train.csv</code></td>\n<td>Human gold-standard annotations</td>\n</tr>\n</tbody>\n</table>\n<p>All phases were augmented with <strong>Joint Dropout (JD)</strong> — perturbed copies of training pairs where tokens are randomly dropped from both source and target. This is huge for cuneiform because tablets are almost always damaged/fragmentary, so the model needs to handle missing text gracefully. Adding JD consistently boosted the LB by <strong>~0.2 per dataset</strong> — small individually, but it compounds across all 7 phases.</p>\n<p>Anti-leak filtering was applied rigorously: JD pairs matching validation set IDs or translations were removed before training in every phase.</p>\n<h2>Preprocessing</h2>\n<p>Mostly based on the host's discussion post, plus:</p>\n<ul>\n<li>All break/gap markers (large break, N broken lines, <code>…</code>, <code>[x]</code>, lone <code>x</code>) → single <code>&lt;gap&gt;</code> token; consecutive gaps collapsed</li>\n<li>Determinatives preserved: <code>(d)</code> → <code>{d}</code>, <code>(ki)</code> → <code>{ki}</code></li>\n<li><code>Ḫ</code>/<code>ḫ</code> → <code>H</code>/<code>h</code>, subscript digits → regular digits</li>\n<li>Decimal fractions → Unicode fractions (<code>0.3333</code> → <code>⅓</code>)</li>\n<li><code>KÙ.B.</code> → <code>KÙ.BABBAR</code> (silver abbreviation fix)</li>\n<li>Target side: <code>-gold</code> → <code>pašallum gold</code>, <code>-tax</code> → <code>šadduātum tax</code>, <code>-textiles</code> → <code>kutānum textiles</code></li>\n<li>Shekel fraction arithmetic: <code>1/12 (shekel)</code> → <code>15 grains</code>, <code>5/12 shekel</code> → <code>⅓ shekel 15 grains</code>, etc.</li>\n<li>Roman numeral months → Arabic: <code>month XII</code> → <code>month 12</code></li>\n<li><code>PN</code> → <code>&lt;gap&gt;</code>, removed linguistic annotations (<code>fem.</code>, <code>sing.</code>, <code>pl.</code>, <code>(?)</code>)</li>\n</ul>\n<p>A simple sentence aligner splits multi-sentence documents into 1:1 line pairs when source lines and target sentences match in count.</p>\n<h2>Training</h2>\n<p><strong>7-phase curriculum</strong> — each phase loads the previous checkpoint and continues training. The idea: start noisy and broad, finish clean and precise.</p>\n<table>\n<thead>\n<tr>\n<th>Phase</th>\n<th>Epochs</th>\n<th>LR</th>\n<th>Key detail</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0 (Dictionary)</td>\n<td>10</td>\n<td>2e-4</td>\n<td>Teaches word-level mappings</td>\n</tr>\n<tr>\n<td>0.5 (Back-trans)</td>\n<td>10</td>\n<td>1e-4</td>\n<td>Synthetic fluency</td>\n</tr>\n<tr>\n<td>0.9 (PDF)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Academic style exposure</td>\n</tr>\n<tr>\n<td>1 (Gemini)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Broad OARE coverage</td>\n</tr>\n<tr>\n<td>1.5 (CDLI)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Broader cuneiform genres</td>\n</tr>\n<tr>\n<td>1.9 (Published)</td>\n<td>5</td>\n<td>1e-4</td>\n<td>Scholarly quality</td>\n</tr>\n<tr>\n<td>2 (Gold)</td>\n<td>9</td>\n<td>1e-5</td>\n<td>Early stopping (patience=3), best checkpoint by eval_loss</td>\n</tr>\n</tbody>\n</table>\n<p>All phases: batch size 16, gradient accumulation 2 (effective 32), label smoothing 0.1, weight decay 0.01, cosine scheduler.</p>\n<p>Phase 2 uses a <strong>10× lower LR</strong> (1e-5) to prevent catastrophic forgetting of all the pretraining knowledge. Early stopping + <code>load_best_model_at_end</code> ensures we pick the best checkpoint.</p>\n<p>Prefix for all inputs: <code>\"translate Akkadian to English: \"</code></p>\n<h2>Validation</h2>\n<p>Train/val split: 90/10 random split (seed=42) on each phase's data independently. Phase 2 evaluates with <code>eval_loss</code> for checkpoint selection, plus BLEU, chrF++, and geometric mean <code>√(BLEU × chrF++)</code> for reporting.</p>\n<h2>What Worked</h2>\n<ul>\n<li><strong>ByT5 byte-level approach</strong> — no tokenizer headaches with Akkadian special characters</li>\n<li><strong>Curriculum learning</strong> — noisy-to-clean progression was key; without the pretraining phases, gold-only training underperforms significantly</li>\n<li><strong>Joint Dropout augmentation</strong> — essential for robustness to fragmentary tablets</li>\n<li><strong>Domain-specific preprocessing</strong> — fraction normalization, gap standardization, and determinative handling made a real difference</li>\n<li><strong>Low LR for final fine-tuning</strong> — 1e-5 with early stopping was the sweet spot for not destroying pretrained knowledge</li>\n</ul>\n<h2>Failed / Not Tried</h2>\n<ul>\n<li>ByT5-Large or bigger (not investigated in depth)</li>\n<li>Ensemble decoding across phase checkpoints</li>\n<li>Retrieval-augmented generation with the dictionary at inference time</li>\n<li>Using a separate LLM for post-processing / deduplication</li>\n</ul>",
      "rawMarkdown": "**Team members:** @arnavmishra02\n \nHuge thanks to @takamichitoda for the professional base code that served as the foundation for this solution, and to @phucthaiv02 for the PDF-extracted Akkadian translation dataset 🙏\n \n**Model used:** `google/byt5-base`\n \nWhy ByT5? Akkadian transliterations are full of diacritics (š, ṣ, ṭ, ḫ), subscript numbers, and Unicode fractions (⅓, ⅔). Subword tokenizers choke on these. ByT5 operates at the byte level — every character is a token — so it handles all of this natively without vocabulary issues.\n \n## Datasets\n \nWe used **7 data sources** of increasing quality, trained in curriculum order:\n \n| Phase | Source | What it is |\n|---|---|---|\n| 0 | OA Lexicon + eBL Dictionary + Onomasticon | Word-level pairs (form → English definition, name spellings → canonical names) |\n| 0.5 | Gemini back-translations | Synthetic sentence pairs from back-translating English into Akkadian |\n| 0.9 | HuggingFace `phucthaiv02/akkadian-translation` | Translation pairs extracted from academic PDFs |\n| 1 | Gemini pseudo-labels (OARE corpus) | LLM-generated translations of the full OARE transliteration corpus |\n| 1.5 | CDLI 10K + Gemini pseudo-labels | 10K CDLI texts translated by Gemini |\n| 1.9 | Published OARE translations | Translations from scholarly publications |\n| 2 | Competition `train.csv` | Human gold-standard annotations |\n \nAll phases were augmented with **Joint Dropout (JD)** — perturbed copies of training pairs where tokens are randomly dropped from both source and target. This is huge for cuneiform because tablets are almost always damaged/fragmentary, so the model needs to handle missing text gracefully. Adding JD consistently boosted the LB by **~0.2 per dataset** — small individually, but it compounds across all 7 phases.\n \nAnti-leak filtering was applied rigorously: JD pairs matching validation set IDs or translations were removed before training in every phase.\n \n## Preprocessing\n \nMostly based on the host's discussion post, plus:\n \n- All break/gap markers (large break, N broken lines, `…`, `[x]`, lone `x`) → single `<gap>` token; consecutive gaps collapsed\n- Determinatives preserved: `(d)` → `{d}`, `(ki)` → `{ki}`\n- `Ḫ`/`ḫ` → `H`/`h`, subscript digits → regular digits\n- Decimal fractions → Unicode fractions (`0.3333` → `⅓`)\n- `KÙ.B.` → `KÙ.BABBAR` (silver abbreviation fix)\n- Target side: `-gold` → `pašallum gold`, `-tax` → `šadduātum tax`, `-textiles` → `kutānum textiles`\n- Shekel fraction arithmetic: `1/12 (shekel)` → `15 grains`, `5/12 shekel` → `⅓ shekel 15 grains`, etc.\n- Roman numeral months → Arabic: `month XII` → `month 12`\n- `PN` → `<gap>`, removed linguistic annotations (`fem.`, `sing.`, `pl.`, `(?)`)\n \nA simple sentence aligner splits multi-sentence documents into 1:1 line pairs when source lines and target sentences match in count.\n \n## Training\n \n**7-phase curriculum** — each phase loads the previous checkpoint and continues training. The idea: start noisy and broad, finish clean and precise.\n \n| Phase | Epochs | LR | Key detail |\n|---|---|---|---|\n| 0 (Dictionary) | 10 | 2e-4 | Teaches word-level mappings |\n| 0.5 (Back-trans) | 10 | 1e-4 | Synthetic fluency |\n| 0.9 (PDF) | 5 | 1e-4 | Academic style exposure |\n| 1 (Gemini) | 5 | 1e-4 | Broad OARE coverage |\n| 1.5 (CDLI) | 5 | 1e-4 | Broader cuneiform genres |\n| 1.9 (Published) | 5 | 1e-4 | Scholarly quality |\n| 2 (Gold) | 9 | 1e-5 | Early stopping (patience=3), best checkpoint by eval_loss |\n \nAll phases: batch size 16, gradient accumulation 2 (effective 32), label smoothing 0.1, weight decay 0.01, cosine scheduler.\n \nPhase 2 uses a **10× lower LR** (1e-5) to prevent catastrophic forgetting of all the pretraining knowledge. Early stopping + `load_best_model_at_end` ensures we pick the best checkpoint.\n \nPrefix for all inputs: `\"translate Akkadian to English: \"`\n \n## Validation\n \nTrain/val split: 90/10 random split (seed=42) on each phase's data independently. Phase 2 evaluates with `eval_loss` for checkpoint selection, plus BLEU, chrF++, and geometric mean `√(BLEU × chrF++)` for reporting.\n \n## What Worked\n \n- **ByT5 byte-level approach** — no tokenizer headaches with Akkadian special characters\n- **Curriculum learning** — noisy-to-clean progression was key; without the pretraining phases, gold-only training underperforms significantly\n- **Joint Dropout augmentation** — essential for robustness to fragmentary tablets\n- **Domain-specific preprocessing** — fraction normalization, gap standardization, and determinative handling made a real difference\n- **Low LR for final fine-tuning** — 1e-5 with early stopping was the sweet spot for not destroying pretrained knowledge\n \n## Failed / Not Tried\n \n- ByT5-Large or bigger (not investigated in depth)\n- Ensemble decoding across phase checkpoints\n- Retrieval-augmented generation with the dictionary at inference time\n- Using a separate LLM for post-processing / deduplication",
      "votes": null
    },
    {
      "id": "3427530",
      "postDate": "03/24/2026 07:44:52",
      "content": "<p>That's amazing work. I'm very proud that my datasets are helpful to you.</p>",
      "rawMarkdown": "That's amazing work. I'm very proud that my datasets are helpful to you.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3427530,
      "author_name": "phucthaiv02",
      "author_url": "",
      "post_date": "03/24/2026 07:44:52",
      "content": "<p>That's amazing work. I'm very proud that my datasets are helpful to you.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3427360": "**Team members:** @arnavmishra02\n \nHuge thanks to @takamichitoda for the professional base code that served as the foundation for this solution, and to @phucthaiv02 for the PDF-extracted Akkadian translation dataset 🙏\n \n**Model used:** `google/byt5-base`\n \nWhy ByT5? Akkadian transliterations are full of diacritics (š, ṣ, ṭ, ḫ), subscript numbers, and Unicode fractions (⅓, ⅔). Subword tokenizers choke on these. ByT5 operates at the byte level — every character is a token — so it handles all of this natively without vocabulary issues.\n \n## Datasets\n \nWe used **7 data sources** of increasing quality, trained in curriculum order:\n \n| Phase | Source | What it is |\n|---|---|---|\n| 0 | OA Lexicon + eBL Dictionary + Onomasticon | Word-level pairs (form → English definition, name spellings → canonical names) |\n| 0.5 | Gemini back-translations | Synthetic sentence pairs from back-translating English into Akkadian |\n| 0.9 | HuggingFace `phucthaiv02/akkadian-translation` | Translation pairs extracted from academic PDFs |\n| 1 | Gemini pseudo-labels (OARE corpus) | LLM-generated translations of the full OARE transliteration corpus |\n| 1.5 | CDLI 10K + Gemini pseudo-labels | 10K CDLI texts translated by Gemini |\n| 1.9 | Published OARE translations | Translations from scholarly publications |\n| 2 | Competition `train.csv` | Human gold-standard annotations |\n \nAll phases were augmented with **Joint Dropout (JD)** — perturbed copies of training pairs where tokens are randomly dropped from both source and target. This is huge for cuneiform because tablets are almost always damaged/fragmentary, so the model needs to handle missing text gracefully. Adding JD consistently boosted the LB by **~0.2 per dataset** — small individually, but it compounds across all 7 phases.\n \nAnti-leak filtering was applied rigorously: JD pairs matching validation set IDs or translations were removed before training in every phase.\n \n## Preprocessing\n \nMostly based on the host's discussion post, plus:\n \n- All break/gap markers (large break, N broken lines, `…`, `[x]`, lone `x`) → single `<gap>` token; consecutive gaps collapsed\n- Determinatives preserved: `(d)` → `{d}`, `(ki)` → `{ki}`\n- `Ḫ`/`ḫ` → `H`/`h`, subscript digits → regular digits\n- Decimal fractions → Unicode fractions (`0.3333` → `⅓`)\n- `KÙ.B.` → `KÙ.BABBAR` (silver abbreviation fix)\n- Target side: `-gold` → `pašallum gold`, `-tax` → `šadduātum tax`, `-textiles` → `kutānum textiles`\n- Shekel fraction arithmetic: `1/12 (shekel)` → `15 grains`, `5/12 shekel` → `⅓ shekel 15 grains`, etc.\n- Roman numeral months → Arabic: `month XII` → `month 12`\n- `PN` → `<gap>`, removed linguistic annotations (`fem.`, `sing.`, `pl.`, `(?)`)\n \nA simple sentence aligner splits multi-sentence documents into 1:1 line pairs when source lines and target sentences match in count.\n \n## Training\n \n**7-phase curriculum** — each phase loads the previous checkpoint and continues training. The idea: start noisy and broad, finish clean and precise.\n \n| Phase | Epochs | LR | Key detail |\n|---|---|---|---|\n| 0 (Dictionary) | 10 | 2e-4 | Teaches word-level mappings |\n| 0.5 (Back-trans) | 10 | 1e-4 | Synthetic fluency |\n| 0.9 (PDF) | 5 | 1e-4 | Academic style exposure |\n| 1 (Gemini) | 5 | 1e-4 | Broad OARE coverage |\n| 1.5 (CDLI) | 5 | 1e-4 | Broader cuneiform genres |\n| 1.9 (Published) | 5 | 1e-4 | Scholarly quality |\n| 2 (Gold) | 9 | 1e-5 | Early stopping (patience=3), best checkpoint by eval_loss |\n \nAll phases: batch size 16, gradient accumulation 2 (effective 32), label smoothing 0.1, weight decay 0.01, cosine scheduler.\n \nPhase 2 uses a **10× lower LR** (1e-5) to prevent catastrophic forgetting of all the pretraining knowledge. Early stopping + `load_best_model_at_end` ensures we pick the best checkpoint.\n \nPrefix for all inputs: `\"translate Akkadian to English: \"`\n \n## Validation\n \nTrain/val split: 90/10 random split (seed=42) on each phase's data independently. Phase 2 evaluates with `eval_loss` for checkpoint selection, plus BLEU, chrF++, and geometric mean `√(BLEU × chrF++)` for reporting.\n \n## What Worked\n \n- **ByT5 byte-level approach** — no tokenizer headaches with Akkadian special characters\n- **Curriculum learning** — noisy-to-clean progression was key; without the pretraining phases, gold-only training underperforms significantly\n- **Joint Dropout augmentation** — essential for robustness to fragmentary tablets\n- **Domain-specific preprocessing** — fraction normalization, gap standardization, and determinative handling made a real difference\n- **Low LR for final fine-tuning** — 1e-5 with early stopping was the sweet spot for not destroying pretrained knowledge\n \n## Failed / Not Tried\n \n- ByT5-Large or bigger (not investigated in depth)\n- Ensemble decoding across phase checkpoints\n- Retrieval-augmented generation with the dictionary at inference time\n- Using a separate LLM for post-processing / deduplication",
    "3427530": "That's amazing work. I'm very proud that my datasets are helpful to you."
  },
  "source": "meta"
}