{
  "id": 684419,
  "title": "ByT5-Large with word-level DAPT",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/684419",
  "author_name": "Luke Li",
  "post_date": "2026-03-24T19:08:36.116000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<h2>1. Model Choice</h2>\n<ul>\n<li><strong>Architecture:</strong> Google ByT5-Large - byte-level sequence-to-sequence Transformer as many others have chosen, aviod the messy tokenization with Akkadian</li>\n<li><strong>Base vs Large:</strong> ByT5-base and ByT5-large performed similarly on holdout and public LB (same score for base &amp; large), but large edged out on private</li>\n<li><strong>Precision:</strong> Always trained in FP32 (BF16 training caused degradation). Inference also in FP32 on the final submission, but FP16 inference probably could have worked</li>\n</ul>\n<h2>2. Data Sources &amp; Preprocessing</h2>\n<p><strong>Data sources (~510K clean parallel pairs after filtering):</strong></p>\n<p>All based on materials provided by the competition host. PDFs were first processed with DeepSeek-OCR 2, then used LLM for separating meaningful corpus pairs. Assigned quality tiers of data for later selective training exposure, based on length, number of , unique word ratio etc.</p>\n<table>\n<thead>\n<tr>\n<th>Source</th>\n<th>Entries</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Official competition data</td>\n<td>1,561</td>\n<td>Provided by the host as starting point</td>\n</tr>\n<tr>\n<td>Chicago Assyrian Dictionary (CAD)</td>\n<td>280K</td>\n<td>All word level data, but pretty messy</td>\n</tr>\n<tr>\n<td>Publications (415 papers)</td>\n<td>76K</td>\n<td>Majority source of sentence level data</td>\n</tr>\n<tr>\n<td>Old Assyrian Lexicon (eBL)</td>\n<td>165K</td>\n<td>Word-level transliteration/definition pairs</td>\n</tr>\n<tr>\n<td>AKT batch</td>\n<td>16K</td>\n<td>Commercial/legal letter translations</td>\n</tr>\n<tr>\n<td>Hecker ICK4 letters</td>\n<td>1.4K</td>\n<td>Complete letter document translations</td>\n</tr>\n<tr>\n<td>Kouwenberg grammars</td>\n<td>10K</td>\n<td>Grammar examples from 2 textbooks</td>\n</tr>\n<tr>\n<td>Other (eSAD, onomasticon list from host,etc.)</td>\n<td>~17K</td>\n<td>Supplementary dictionaries and glossaries</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Preprocessing pipeline (9 stages):</strong></p>\n<ol>\n<li><strong>Unicode normalization:</strong> NFC, transliteration variant mapping (ḫ→h, subscripts→digits, diacritics normalized)</li>\n<li><strong>Source cleaning:</strong> Determinative types normalized (<code>{ki}</code>, <code>{d}</code>), editorial markers stripped, English leakage removed, name placeholders (PN/GN/DN) → <code>&lt;gap&gt;</code></li>\n<li><strong>Target cleaning:</strong> Shekel fraction conversion, LaTeX removal, citation stripping, grammatical gloss removal, commodity abbreviation expansion, scholarly apparatus removal</li>\n<li><strong>Gap unification:</strong> All damage markers (<code>[...]</code>, <code>[x x x]</code>, <code>(break)</code>, half-brackets, ellipses, etc.) → canonical <code>&lt;gap&gt;</code> token. Adjacent gaps collapsed. This was a major contributor to score improvement.</li>\n<li><strong>Quality filtering:</strong> Removed ~80K entries (empty, high damage ratio, scholarly annotations, self-referential, metalanguage)</li>\n<li><strong>Deduplication:</strong> ~13K exact duplicates removed (priority-based: higher-quality source kept)</li>\n<li><strong>Long entry splitting:</strong> Entries &gt;512B split via LLM processing at clause boundaries</li>\n<li><strong>Back-translation:</strong> Back translated some English translation to Akkadian using a trained ByT5 base model (English→Akkadian), use as synthetic data to broaden the spectrum</li>\n</ol>\n<h2>3. Training Method &amp; Curriculum</h2>\n<p><strong>Training curriculum:</strong></p>\n<table>\n<thead>\n<tr>\n<th>Stage</th>\n<th>Data</th>\n<th>LR</th>\n<th>Aug</th>\n<th>Purpose</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>DAPT</td>\n<td>114K word-level (unique texts)</td>\n<td>5e-4</td>\n<td>none</td>\n<td>Domain adaptation pre-training</td>\n</tr>\n<tr>\n<td>S1</td>\n<td>221K word-level glosses</td>\n<td>5e-4</td>\n<td>none</td>\n<td>Vocabulary grounding</td>\n</tr>\n<tr>\n<td>S2</td>\n<td>282K all sentences</td>\n<td>3e-4</td>\n<td>none</td>\n<td>Sentence-level learning</td>\n</tr>\n<tr>\n<td>S3–S8</td>\n<td>Sentences &lt;512B</td>\n<td>2e-4 → 3e-5</td>\n<td>0.2 → 0.3</td>\n<td>Progressive fine-tuning with augmentation escalation</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Synthetic augmentation / selection after S2 with increasing probability:</strong></p>\n<ul>\n<li>A: Diacritical simplification (š→s, ṣ→s)</li>\n<li>B: Hyphenation merge/split/move</li>\n<li>C: Determinative perturbation</li>\n<li>D: Sign reading number swap (subscript indices)</li>\n<li>E: Spacing variation</li>\n<li>F: Gap augmentation (character/word spans)</li>\n</ul>\n<p><strong>Key training findings:</strong></p>\n<ul>\n<li>Low–moderate augmentation (0.2–0.4) optimal; high (0.5+) actively hurts</li>\n<li>Hard Example Mining (HEM) not helping: aggressive → reduced score; gentle → nearly neutral</li>\n</ul>\n<h2>4. Inference Settings &amp; Selection Strategy</h2>\n<p><strong>Best submission:</strong></p>\n<ul>\n<li><code>num_beams=12, length_penalty=1.1, repetition_penalty=1.2</code></li>\n<li><code>max_new_tokens=512, do_sample=False</code></li>\n<li><strong>Simple beam top-1 selection</strong> — no MBR, no reranking</li>\n<li>FP32 inference on dual T4 GPUs (workload split round-robin)</li>\n<li>Dynamic batch sizing (base=6 per GPU, scaled inversely by source byte length)</li>\n<li>OOM fallback to batch_size=1</li>\n<li>Source chunking: Split &gt; 512-byte chunks at word boundaries and reassemble with de-dup</li>\n</ul>\n<p><strong>Other approaches explored but did not improve LB:</strong></p>\n<p><em>MBR reranking:</em></p>\n<ul>\n<li>chrF++ MBR: Better vs. beam top-1 (36% of oracle gap) for my own eval set, but no impact on LB</li>\n<li>Greedy MBR+features: chrF++ (80%) + UWR (20%) → +FWR (10%) → +Period (5%). same, better onmy own eval set but not on LB</li>\n</ul>\n<p><em>Other things:</em></p>\n<ul>\n<li>LogProb reranking</li>\n<li>TTA (test-time augmentation): no impact on my own eval set</li>\n<li>kNN-MT decoding: implemented (adaptive λ, k=16, GPU-based L2 distance) not working</li>\n<li>Per-bucket length penalty: marginal improvement and heavy overhead, dropped</li>\n</ul>\n<h2>5. Postprocessing</h2>\n<p>Applied after generation:</p>\n<ol>\n<li><strong>normalize_gaps</strong> — all gap variants → canonical <code>&lt;gap&gt;</code>, adjacent gaps collapsed</li>\n<li><strong>bracket removal</strong> — <code>[text]</code> → <code>text</code>, preserving <code>&lt;gap&gt;</code></li>\n<li><strong>gap collapse</strong> — <code>&lt;gap&gt; &lt;gap&gt;</code> → <code>&lt;gap&gt;</code></li>\n<li><strong>empty fallback</strong> — empty translations → <code>&lt;gap&gt;.</code></li>\n</ol>\n<p>Additional postprocessing developed during the project (used in MBR variants):</p>\n<ul>\n<li>fix_capitalization (ByT5 byte-confusion mid-word case)</li>\n<li>fix_common_artifacts, fix_typos (data-driven corrections)</li>\n<li>clean_repetitions + remove_repeated_ngrams (beam search phrase loops)</li>\n<li>remove_scribal_notations, transliterate_akkadian_chars</li>\n<li>convert_decimals_to_fractions, normalize_subscript_digits</li>\n</ul>\n<h2>6. Other Explorations</h2>\n<p><strong>Cross-model ensemble:</strong></p>\n<ul>\n<li>Tried Large + Base ensemble with merged candidate pools → MBR</li>\n<li>Also tried multiple Base ensembles (different ByT5-base checkpoints, curriculms,. data exposure)</li>\n<li>Result: very modest gain on my local eval and all slightly worse in LB, did not end up using</li>\n</ul>",
  "messages": [
    {
      "id": 3428000,
      "postDate": "2026-03-24T19:08:36.117Z",
      "content": "<h2>1. Model Choice</h2>\n<ul>\n<li><strong>Architecture:</strong> Google ByT5-Large - byte-level sequence-to-sequence Transformer as many others have chosen, aviod the messy tokenization with Akkadian</li>\n<li><strong>Base vs Large:</strong> ByT5-base and ByT5-large performed similarly on holdout and public LB (same score for base &amp; large), but large edged out on private</li>\n<li><strong>Precision:</strong> Always trained in FP32 (BF16 training caused degradation). Inference also in FP32 on the final submission, but FP16 inference probably could have worked</li>\n</ul>\n<h2>2. Data Sources &amp; Preprocessing</h2>\n<p><strong>Data sources (~510K clean parallel pairs after filtering):</strong></p>\n<p>All based on materials provided by the competition host. PDFs were first processed with DeepSeek-OCR 2, then used LLM for separating meaningful corpus pairs. Assigned quality tiers of data for later selective training exposure, based on length, number of , unique word ratio etc.</p>\n<table>\n<thead>\n<tr>\n<th>Source</th>\n<th>Entries</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Official competition data</td>\n<td>1,561</td>\n<td>Provided by the host as starting point</td>\n</tr>\n<tr>\n<td>Chicago Assyrian Dictionary (CAD)</td>\n<td>280K</td>\n<td>All word level data, but pretty messy</td>\n</tr>\n<tr>\n<td>Publications (415 papers)</td>\n<td>76K</td>\n<td>Majority source of sentence level data</td>\n</tr>\n<tr>\n<td>Old Assyrian Lexicon (eBL)</td>\n<td>165K</td>\n<td>Word-level transliteration/definition pairs</td>\n</tr>\n<tr>\n<td>AKT batch</td>\n<td>16K</td>\n<td>Commercial/legal letter translations</td>\n</tr>\n<tr>\n<td>Hecker ICK4 letters</td>\n<td>1.4K</td>\n<td>Complete letter document translations</td>\n</tr>\n<tr>\n<td>Kouwenberg grammars</td>\n<td>10K</td>\n<td>Grammar examples from 2 textbooks</td>\n</tr>\n<tr>\n<td>Other (eSAD, onomasticon list from host,etc.)</td>\n<td>~17K</td>\n<td>Supplementary dictionaries and glossaries</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Preprocessing pipeline (9 stages):</strong></p>\n<ol>\n<li><strong>Unicode normalization:</strong> NFC, transliteration variant mapping (ḫ→h, subscripts→digits, diacritics normalized)</li>\n<li><strong>Source cleaning:</strong> Determinative types normalized (<code>{ki}</code>, <code>{d}</code>), editorial markers stripped, English leakage removed, name placeholders (PN/GN/DN) → <code>&lt;gap&gt;</code></li>\n<li><strong>Target cleaning:</strong> Shekel fraction conversion, LaTeX removal, citation stripping, grammatical gloss removal, commodity abbreviation expansion, scholarly apparatus removal</li>\n<li><strong>Gap unification:</strong> All damage markers (<code>[...]</code>, <code>[x x x]</code>, <code>(break)</code>, half-brackets, ellipses, etc.) → canonical <code>&lt;gap&gt;</code> token. Adjacent gaps collapsed. This was a major contributor to score improvement.</li>\n<li><strong>Quality filtering:</strong> Removed ~80K entries (empty, high damage ratio, scholarly annotations, self-referential, metalanguage)</li>\n<li><strong>Deduplication:</strong> ~13K exact duplicates removed (priority-based: higher-quality source kept)</li>\n<li><strong>Long entry splitting:</strong> Entries &gt;512B split via LLM processing at clause boundaries</li>\n<li><strong>Back-translation:</strong> Back translated some English translation to Akkadian using a trained ByT5 base model (English→Akkadian), use as synthetic data to broaden the spectrum</li>\n</ol>\n<h2>3. Training Method &amp; Curriculum</h2>\n<p><strong>Training curriculum:</strong></p>\n<table>\n<thead>\n<tr>\n<th>Stage</th>\n<th>Data</th>\n<th>LR</th>\n<th>Aug</th>\n<th>Purpose</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>DAPT</td>\n<td>114K word-level (unique texts)</td>\n<td>5e-4</td>\n<td>none</td>\n<td>Domain adaptation pre-training</td>\n</tr>\n<tr>\n<td>S1</td>\n<td>221K word-level glosses</td>\n<td>5e-4</td>\n<td>none</td>\n<td>Vocabulary grounding</td>\n</tr>\n<tr>\n<td>S2</td>\n<td>282K all sentences</td>\n<td>3e-4</td>\n<td>none</td>\n<td>Sentence-level learning</td>\n</tr>\n<tr>\n<td>S3–S8</td>\n<td>Sentences &lt;512B</td>\n<td>2e-4 → 3e-5</td>\n<td>0.2 → 0.3</td>\n<td>Progressive fine-tuning with augmentation escalation</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Synthetic augmentation / selection after S2 with increasing probability:</strong></p>\n<ul>\n<li>A: Diacritical simplification (š→s, ṣ→s)</li>\n<li>B: Hyphenation merge/split/move</li>\n<li>C: Determinative perturbation</li>\n<li>D: Sign reading number swap (subscript indices)</li>\n<li>E: Spacing variation</li>\n<li>F: Gap augmentation (character/word spans)</li>\n</ul>\n<p><strong>Key training findings:</strong></p>\n<ul>\n<li>Low–moderate augmentation (0.2–0.4) optimal; high (0.5+) actively hurts</li>\n<li>Hard Example Mining (HEM) not helping: aggressive → reduced score; gentle → nearly neutral</li>\n</ul>\n<h2>4. Inference Settings &amp; Selection Strategy</h2>\n<p><strong>Best submission:</strong></p>\n<ul>\n<li><code>num_beams=12, length_penalty=1.1, repetition_penalty=1.2</code></li>\n<li><code>max_new_tokens=512, do_sample=False</code></li>\n<li><strong>Simple beam top-1 selection</strong> — no MBR, no reranking</li>\n<li>FP32 inference on dual T4 GPUs (workload split round-robin)</li>\n<li>Dynamic batch sizing (base=6 per GPU, scaled inversely by source byte length)</li>\n<li>OOM fallback to batch_size=1</li>\n<li>Source chunking: Split &gt; 512-byte chunks at word boundaries and reassemble with de-dup</li>\n</ul>\n<p><strong>Other approaches explored but did not improve LB:</strong></p>\n<p><em>MBR reranking:</em></p>\n<ul>\n<li>chrF++ MBR: Better vs. beam top-1 (36% of oracle gap) for my own eval set, but no impact on LB</li>\n<li>Greedy MBR+features: chrF++ (80%) + UWR (20%) → +FWR (10%) → +Period (5%). same, better onmy own eval set but not on LB</li>\n</ul>\n<p><em>Other things:</em></p>\n<ul>\n<li>LogProb reranking</li>\n<li>TTA (test-time augmentation): no impact on my own eval set</li>\n<li>kNN-MT decoding: implemented (adaptive λ, k=16, GPU-based L2 distance) not working</li>\n<li>Per-bucket length penalty: marginal improvement and heavy overhead, dropped</li>\n</ul>\n<h2>5. Postprocessing</h2>\n<p>Applied after generation:</p>\n<ol>\n<li><strong>normalize_gaps</strong> — all gap variants → canonical <code>&lt;gap&gt;</code>, adjacent gaps collapsed</li>\n<li><strong>bracket removal</strong> — <code>[text]</code> → <code>text</code>, preserving <code>&lt;gap&gt;</code></li>\n<li><strong>gap collapse</strong> — <code>&lt;gap&gt; &lt;gap&gt;</code> → <code>&lt;gap&gt;</code></li>\n<li><strong>empty fallback</strong> — empty translations → <code>&lt;gap&gt;.</code></li>\n</ol>\n<p>Additional postprocessing developed during the project (used in MBR variants):</p>\n<ul>\n<li>fix_capitalization (ByT5 byte-confusion mid-word case)</li>\n<li>fix_common_artifacts, fix_typos (data-driven corrections)</li>\n<li>clean_repetitions + remove_repeated_ngrams (beam search phrase loops)</li>\n<li>remove_scribal_notations, transliterate_akkadian_chars</li>\n<li>convert_decimals_to_fractions, normalize_subscript_digits</li>\n</ul>\n<h2>6. Other Explorations</h2>\n<p><strong>Cross-model ensemble:</strong></p>\n<ul>\n<li>Tried Large + Base ensemble with merged candidate pools → MBR</li>\n<li>Also tried multiple Base ensembles (different ByT5-base checkpoints, curriculms,. data exposure)</li>\n<li>Result: very modest gain on my local eval and all slightly worse in LB, did not end up using</li>\n</ul>",
      "rawMarkdown": "## 1. Model Choice\n\n- **Architecture:** Google ByT5-Large - byte-level sequence-to-sequence Transformer as many others have chosen, aviod the messy tokenization with Akkadian\n- **Base vs Large:** ByT5-base and ByT5-large performed similarly on holdout and public LB (same score for base & large), but large edged out on private\n- **Precision:** Always trained in FP32 (BF16 training caused degradation). Inference also in FP32 on the final submission, but FP16 inference probably could have worked\n\n\n## 2. Data Sources & Preprocessing\n\n**Data sources (~510K clean parallel pairs after filtering):**\n\nAll based on materials provided by the competition host. PDFs were first processed with DeepSeek-OCR 2, then used LLM for separating meaningful corpus pairs. Assigned quality tiers of data for later selective training exposure, based on length, number of <gap>, unique word ratio etc.\n\n| Source | Entries | Description |\n|--------|---------|-------------|\n| Official competition data | 1,561 | Provided by the host as starting point |\n| Chicago Assyrian Dictionary (CAD) | 280K | All word level data, but pretty messy |\n| Publications (415 papers) | 76K | Majority source of sentence level data |\n| Old Assyrian Lexicon (eBL) | 165K | Word-level transliteration/definition pairs |\n| AKT batch | 16K | Commercial/legal letter translations |\n| Hecker ICK4 letters | 1.4K | Complete letter document translations |\n| Kouwenberg grammars | 10K | Grammar examples from 2 textbooks |\n| Other (eSAD, onomasticon list from host,etc.) | ~17K | Supplementary dictionaries and glossaries |\n\n\n**Preprocessing pipeline (9 stages):**\n1. **Unicode normalization:** NFC, transliteration variant mapping (ḫ→h, subscripts→digits, diacritics normalized)\n2. **Source cleaning:** Determinative types normalized (`{ki}`, `{d}`), editorial markers stripped, English leakage removed, name placeholders (PN/GN/DN) → `<gap>`\n3. **Target cleaning:** Shekel fraction conversion, LaTeX removal, citation stripping, grammatical gloss removal, commodity abbreviation expansion, scholarly apparatus removal\n4. **Gap unification:** All damage markers (`[...]`, `[x x x]`, `(break)`, half-brackets, ellipses, etc.) → canonical `<gap>` token. Adjacent gaps collapsed. This was a major contributor to score improvement.\n5. **Quality filtering:** Removed ~80K entries (empty, high damage ratio, scholarly annotations, self-referential, metalanguage)\n6. **Deduplication:** ~13K exact duplicates removed (priority-based: higher-quality source kept)\n7. **Long entry splitting:** Entries >512B split via LLM processing at clause boundaries\n8. **Back-translation:** Back translated some English translation to Akkadian using a trained ByT5 base model (English→Akkadian), use as synthetic data to broaden the spectrum\n\n\n## 3. Training Method & Curriculum\n\n**Training curriculum:**\n\n| Stage | Data | LR | Aug | Purpose |\n|-------|------|-----|-----|---------|\n| DAPT | 114K word-level (unique texts) | 5e-4 | none | Domain adaptation pre-training |\n| S1 | 221K word-level glosses | 5e-4 | none | Vocabulary grounding |\n| S2 | 282K all sentences | 3e-4 | none | Sentence-level learning |\n| S3–S8 | Sentences <512B | 2e-4 → 3e-5 | 0.2 → 0.3 | Progressive fine-tuning with augmentation escalation |\n\n**Synthetic augmentation / selection after S2 with increasing probability:**\n\n- A: Diacritical simplification (š→s, ṣ→s)\n- B: Hyphenation merge/split/move\n- C: Determinative perturbation\n- D: Sign reading number swap (subscript indices)\n- E: Spacing variation\n- F: Gap augmentation (character/word spans)\n\n**Key training findings:**\n\n- Low–moderate augmentation (0.2–0.4) optimal; high (0.5+) actively hurts\n- Hard Example Mining (HEM) not helping: aggressive → reduced score; gentle → nearly neutral\n\n\n## 4. Inference Settings & Selection Strategy\n\n**Best submission:**\n\n- `num_beams=12, length_penalty=1.1, repetition_penalty=1.2`\n- `max_new_tokens=512, do_sample=False`\n- **Simple beam top-1 selection** — no MBR, no reranking\n- FP32 inference on dual T4 GPUs (workload split round-robin)\n- Dynamic batch sizing (base=6 per GPU, scaled inversely by source byte length)\n- OOM fallback to batch_size=1\n- Source chunking: Split > 512-byte chunks at word boundaries and reassemble with de-dup\n\n**Other approaches explored but did not improve LB:**\n\n*MBR reranking:*\n\n- chrF++ MBR: Better vs. beam top-1 (36% of oracle gap) for my own eval set, but no impact on LB\n- Greedy MBR+features: chrF++ (80%) + UWR (20%) → +FWR (10%) → +Period (5%). same, better onmy own eval set but not on LB\n\n*Other things:*\n\n- LogProb reranking\n- TTA (test-time augmentation): no impact on my own eval set\n- kNN-MT decoding: implemented (adaptive λ, k=16, GPU-based L2 distance) not working\n- Per-bucket length penalty: marginal improvement and heavy overhead, dropped\n\n\n## 5. Postprocessing\n\nApplied after generation:\n\n1. **normalize_gaps** — all gap variants → canonical `<gap>`, adjacent gaps collapsed\n2. **bracket removal** — `[text]` → `text`, preserving `<gap>`\n3. **gap collapse** — `<gap> <gap>` → `<gap>`\n4. **empty fallback** — empty translations → `<gap>.`\n\nAdditional postprocessing developed during the project (used in MBR variants):\n\n- fix_capitalization (ByT5 byte-confusion mid-word case)\n- fix_common_artifacts, fix_typos (data-driven corrections)\n- clean_repetitions + remove_repeated_ngrams (beam search phrase loops)\n- remove_scribal_notations, transliterate_akkadian_chars\n- convert_decimals_to_fractions, normalize_subscript_digits\n\n\n## 6. Other Explorations\n\n**Cross-model ensemble:**\n\n- Tried Large + Base ensemble with merged candidate pools → MBR\n- Also tried multiple Base ensembles (different ByT5-base checkpoints, curriculms,. data exposure)\n- Result: very modest gain on my local eval and all slightly worse in LB, did not end up using",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3428000": "## 1. Model Choice\n\n- **Architecture:** Google ByT5-Large - byte-level sequence-to-sequence Transformer as many others have chosen, aviod the messy tokenization with Akkadian\n- **Base vs Large:** ByT5-base and ByT5-large performed similarly on holdout and public LB (same score for base & large), but large edged out on private\n- **Precision:** Always trained in FP32 (BF16 training caused degradation). Inference also in FP32 on the final submission, but FP16 inference probably could have worked\n\n\n## 2. Data Sources & Preprocessing\n\n**Data sources (~510K clean parallel pairs after filtering):**\n\nAll based on materials provided by the competition host. PDFs were first processed with DeepSeek-OCR 2, then used LLM for separating meaningful corpus pairs. Assigned quality tiers of data for later selective training exposure, based on length, number of <gap>, unique word ratio etc.\n\n| Source | Entries | Description |\n|--------|---------|-------------|\n| Official competition data | 1,561 | Provided by the host as starting point |\n| Chicago Assyrian Dictionary (CAD) | 280K | All word level data, but pretty messy |\n| Publications (415 papers) | 76K | Majority source of sentence level data |\n| Old Assyrian Lexicon (eBL) | 165K | Word-level transliteration/definition pairs |\n| AKT batch | 16K | Commercial/legal letter translations |\n| Hecker ICK4 letters | 1.4K | Complete letter document translations |\n| Kouwenberg grammars | 10K | Grammar examples from 2 textbooks |\n| Other (eSAD, onomasticon list from host,etc.) | ~17K | Supplementary dictionaries and glossaries |\n\n\n**Preprocessing pipeline (9 stages):**\n1. **Unicode normalization:** NFC, transliteration variant mapping (ḫ→h, subscripts→digits, diacritics normalized)\n2. **Source cleaning:** Determinative types normalized (`{ki}`, `{d}`), editorial markers stripped, English leakage removed, name placeholders (PN/GN/DN) → `<gap>`\n3. **Target cleaning:** Shekel fraction conversion, LaTeX removal, citation stripping, grammatical gloss removal, commodity abbreviation expansion, scholarly apparatus removal\n4. **Gap unification:** All damage markers (`[...]`, `[x x x]`, `(break)`, half-brackets, ellipses, etc.) → canonical `<gap>` token. Adjacent gaps collapsed. This was a major contributor to score improvement.\n5. **Quality filtering:** Removed ~80K entries (empty, high damage ratio, scholarly annotations, self-referential, metalanguage)\n6. **Deduplication:** ~13K exact duplicates removed (priority-based: higher-quality source kept)\n7. **Long entry splitting:** Entries >512B split via LLM processing at clause boundaries\n8. **Back-translation:** Back translated some English translation to Akkadian using a trained ByT5 base model (English→Akkadian), use as synthetic data to broaden the spectrum\n\n\n## 3. Training Method & Curriculum\n\n**Training curriculum:**\n\n| Stage | Data | LR | Aug | Purpose |\n|-------|------|-----|-----|---------|\n| DAPT | 114K word-level (unique texts) | 5e-4 | none | Domain adaptation pre-training |\n| S1 | 221K word-level glosses | 5e-4 | none | Vocabulary grounding |\n| S2 | 282K all sentences | 3e-4 | none | Sentence-level learning |\n| S3–S8 | Sentences <512B | 2e-4 → 3e-5 | 0.2 → 0.3 | Progressive fine-tuning with augmentation escalation |\n\n**Synthetic augmentation / selection after S2 with increasing probability:**\n\n- A: Diacritical simplification (š→s, ṣ→s)\n- B: Hyphenation merge/split/move\n- C: Determinative perturbation\n- D: Sign reading number swap (subscript indices)\n- E: Spacing variation\n- F: Gap augmentation (character/word spans)\n\n**Key training findings:**\n\n- Low–moderate augmentation (0.2–0.4) optimal; high (0.5+) actively hurts\n- Hard Example Mining (HEM) not helping: aggressive → reduced score; gentle → nearly neutral\n\n\n## 4. Inference Settings & Selection Strategy\n\n**Best submission:**\n\n- `num_beams=12, length_penalty=1.1, repetition_penalty=1.2`\n- `max_new_tokens=512, do_sample=False`\n- **Simple beam top-1 selection** — no MBR, no reranking\n- FP32 inference on dual T4 GPUs (workload split round-robin)\n- Dynamic batch sizing (base=6 per GPU, scaled inversely by source byte length)\n- OOM fallback to batch_size=1\n- Source chunking: Split > 512-byte chunks at word boundaries and reassemble with de-dup\n\n**Other approaches explored but did not improve LB:**\n\n*MBR reranking:*\n\n- chrF++ MBR: Better vs. beam top-1 (36% of oracle gap) for my own eval set, but no impact on LB\n- Greedy MBR+features: chrF++ (80%) + UWR (20%) → +FWR (10%) → +Period (5%). same, better onmy own eval set but not on LB\n\n*Other things:*\n\n- LogProb reranking\n- TTA (test-time augmentation): no impact on my own eval set\n- kNN-MT decoding: implemented (adaptive λ, k=16, GPU-based L2 distance) not working\n- Per-bucket length penalty: marginal improvement and heavy overhead, dropped\n\n\n## 5. Postprocessing\n\nApplied after generation:\n\n1. **normalize_gaps** — all gap variants → canonical `<gap>`, adjacent gaps collapsed\n2. **bracket removal** — `[text]` → `text`, preserving `<gap>`\n3. **gap collapse** — `<gap> <gap>` → `<gap>`\n4. **empty fallback** — empty translations → `<gap>.`\n\nAdditional postprocessing developed during the project (used in MBR variants):\n\n- fix_capitalization (ByT5 byte-confusion mid-word case)\n- fix_common_artifacts, fix_typos (data-driven corrections)\n- clean_repetitions + remove_repeated_ngrams (beam search phrase loops)\n- remove_scribal_notations, transliterate_akkadian_chars\n- convert_decimals_to_fractions, normalize_subscript_digits\n\n\n## 6. Other Explorations\n\n**Cross-model ensemble:**\n\n- Tried Large + Base ensemble with merged candidate pools → MBR\n- Also tried multiple Base ensembles (different ByT5-base checkpoints, curriculms,. data exposure)\n- Result: very modest gain on my local eval and all slightly worse in LB, did not end up using"
  }
}