{
  "id": 684187,
  "title": "Make the most of all available resources",
  "url": "/competitions/deep-past-initiative-machine-translation/writeups/make-the-most-of-all-available-resources",
  "author_name": "",
  "post_date": "2026-03-24T00:41:40.777Z",
  "votes": 13,
  "comment_count": 1,
  "views": 0,
  "content": "<h1>Old Assyrian Machine Translation Experiments Summary</h1>\n<h2>1. Baseline Model</h2>\n<ul>\n<li><strong>Task</strong>: Translating Old Assyrian transliteration to English (Kaggle Deep Past Challenge).</li>\n<li><strong>Model Architecture</strong>: ByT5 (Byte-level T5), <code>MAX_LENGTH=512</code>.</li>\n<li><strong>Base Dataset (A)</strong>: <code>train.csv</code> (1,561 rows) + <code>test.csv</code> (4 rows).</li>\n<li><strong>Validation Strategy</strong>: Fixed validation set randomly sampled from the original <code>train.csv</code> to prevent data leakage.</li>\n<li><strong>Baseline Score</strong>: ~33.1 (A).</li>\n</ul>\n<hr>\n<h2>2. Experiments &amp; Findings</h2>\n<h3>Experiment 1: Synthetic Data via Sentence Recombination (Step 1)</h3>\n<ul>\n<li><strong>Method</strong>: Joined <code>Sentences_Oare_FirstWord_LinNum.csv</code> with <code>published_texts.csv</code> on <code>text_uuid</code>. Reconstructed document-level translations by concatenating sentences and filtering by length ratio (<code>RATIO_MIN=0.33</code>, <code>RATIO_MAX=2.69</code>).</li>\n<li><strong>Data Yield</strong>: 990 \"clean\" pairs.</li>\n<li><strong>Result</strong>: <strong>Score Decreased (Negative impact).</strong></li>\n<li><strong>Analysis</strong>: Recombined texts likely suffered from unnatural sentence boundaries and contextual misalignment.</li>\n</ul>\n<h3>Experiment 2: LLM Translation of Unlabeled Texts (Step 2)</h3>\n<ul>\n<li><strong>Method</strong>: Used LLM (Claude) to translate <code>published_texts.csv</code> (which had transliteration but no translation). Configured with 3 few-shot examples and dictionary assistance (<code>OA_Lexicon</code> + <code>eBL_Dictionary</code>).</li>\n<li><strong>Data Yield</strong>: ~3,000+ pairs generated.</li>\n<li><strong>Result</strong>: <strong>Score Decreased (A+E = 32.7 vs Baseline 33.1).</strong></li>\n<li><strong>Analysis</strong>:<ul>\n<li><em>Validation Leakage</em>: Initially, synthetic data leaked into the validation set, causing artificially high validation scores (46.88) while actual inference scores dropped.</li>\n<li><em>Length Truncation</em>: Many generated translations exceeded ByT5's 512-byte limit and were silently truncated.</li>\n<li><em>Domain Drift/Noise</em>: LLM-generated translations lacked the stylistic consistency of human experts, introducing noise that outweighed the benefits of more data.</li></ul></li>\n</ul>\n<h3>Experiment 3: Extracting Expert Translations from Academic PDFs (Step 3)</h3>\n<ul>\n<li><strong>Raw Data Source</strong>: <code>publications.csv</code> (~9,890 pages of OCR text from Assyriology academic publications, including EL/Lewy, AKT, CMK, NABU, CCT series).</li>\n<li><strong>Method</strong>: First apply rule-based filters to narrow 9,890 pages to high-quality bilingual pages, then use LLM to extract (transliteration, translation) pairs from each page.</li>\n<li><strong>V1 Filter Rules</strong>:<ol>\n<li><code>has_akkadian == true</code> (pre-labeled field in the CSV)</li>\n<li>Must contain Akkadian special characters (e.g., <code>š, ṭ, ṣ, ā, ē, ī, ū, ḫ</code>)</li>\n<li>Must contain core Akkadian vocabulary (e.g., <code>um-ma</code>, <code>qí-bi-ma</code>, <code>ma-na</code>, <code>KÙ.BABBAR</code>, <code>AN.NA</code>)</li>\n<li>Must contain German or English keywords (e.g., <code>von, der, Silber</code> or <code>silver, mina, shekel</code>)</li>\n<li>Must contain standard line number format (regex <code>\\d{1,2}[\\s\\.\\)]</code>)</li></ol></li>\n<li><strong>V1 LLM Prompt (EXTRACT_PROMPT) Key Points</strong>:<ul>\n<li>Role: Expert in Old Assyrian cuneiform texts and Assyriology</li>\n<li>Task: Extract all (transliteration, translation) pairs</li>\n<li>If translation is in German, translate it to English</li>\n<li>Keep proper names and technical terms untranslated (Puzur-Aššur, kārum, ḫamuštum, etc.)</li>\n<li>Mark broken/missing text as <code>&lt;gap&gt;</code></li>\n<li>Only extract pairs where transliteration is 10–500 characters</li>\n<li>Includes 4 few-shot examples (English, German, Seal/contract format)</li>\n<li>Return pure JSON array; return <code>[]</code> if no valid pairs found</li></ul></li>\n<li><strong>Data Yield (V1)</strong>: 556 valid pages found. A sample of 300 pairs was extracted.</li>\n<li><strong>Result</strong>: <strong>Score Increased (+0.6 improvement).</strong></li>\n<li><strong>Analysis</strong>: Human expert translations from academic sources perfectly matched the style and quality of the training set.</li>\n</ul>\n<h3>Experiment 4: Unlocking Direct Sentence Alignments (Step 4)</h3>\n<ul>\n<li><strong>Method</strong>: Discovered 8,564 unused expert-translated sentences in <code>Sentences_Oare</code>. Wrote a script to align them directly with document transliterations without concatenating them.</li>\n<li><strong>Data Yield</strong>: 589 pairs (after removing duplicates found in <code>train.csv</code>).</li>\n<li><strong>Result</strong>: <strong>Score Decreased.</strong></li>\n<li><strong>Analysis</strong>: 412 rows contained \"float artifacts\" (e.g., the fraction <code>2 5/6</code> was stored as <code>2.83333</code> in the DB), which confused the byte-level tokenizer.</li>\n<li><strong>Fix Applied</strong>: Added strict regex filtering to remove float artifacts and German translations, resulting in <strong>401 clean pairs</strong> (pending re-test).</li>\n</ul>\n<h3>Experiment 5: Cleaning the Michel Letters Corpus (Step 5)</h3>\n<ul>\n<li><strong>Method</strong>: Processed the newly acquired <code>Michel Old Assyrian Letters Corpus.csv</code> (264 rows).</li>\n<li><strong>Data Yield</strong>:<ul>\n<li><code>augmented_step5_michel.csv</code>: 74 clean pairs (≤500 chars).</li>\n<li><code>augmented_step5_michel_full.csv</code>: 474 pairs (long texts split by paragraph line numbers).</li></ul></li>\n<li><strong>Result</strong>: <strong>Pending test.</strong> (Expected to be highly positive as it is a premium human-translated source).</li>\n</ul>\n<h3>Experiment 6: Optimizing PDF Extraction Filter (Step 3 V2)</h3>\n<ul>\n<li><strong>Method</strong>: Analyzed the 9,334 pages blocked by the V1 filter. Discovered that 900+ pages of high-quality French translations (CMK 2001) were blocked, alongside many core texts (AKT 7a, EL 1/2) that used non-standard line numbers (<code>obv. 1</code>, <code>1'</code>).</li>\n<li><strong>V2 Filter Changes</strong>:<ol>\n<li><strong>Added French keyword detection</strong> (<code>mines, sicles, argent, cuivre, étain, tissus, doit, reçu, envoyé...</code>)</li>\n<li><strong>Extended line number regex</strong>: <code>\\d{1,2}</code> → <code>\\d{1,3}</code> (supports 3-digit numbers), added fragment format <code>\\d{1,3}'</code>, added archaeological format <code>obv/rev/col</code></li>\n<li><strong>Added PDF blacklist (BLACKLIST_PDFS)</strong> to exclude: AKT 7a/7b (Turkish commentary), Barjamovic (academic prose, 148 pages), Grayson RIMA (Neo-Assyrian dialect), Dercksen trade analysis, Larsen 2002, etc.</li></ol></li>\n<li><strong>V2 LLM Prompt Changes</strong>:<ul>\n<li>Updated language description to: <code>\"in German, English, or French\"</code></li>\n<li>Explicitly instructs LLM to translate French to English as well</li>\n<li>Added French few-shot example (CMK format): <code>\"Ainsi (parle) Aššur-bāni: dis à Pūšu-kēn...\"</code> → English</li>\n<li>Total: 5 few-shot examples (English × 1, German × 1, French × 1, Seal × 1, plus original English example)</li></ul></li>\n<li><strong>Data Yield</strong>: Expanded valid pages from 556 to <strong>536 highly concentrated, noise-free pages</strong> (yielding an estimated 400-600 premium pairs).</li>\n</ul>\n<hr>\n<h2>3. Key Conclusions</h2>\n<ol>\n<li><strong>Quality over Quantity</strong>: ByT5 is extremely sensitive to data quality. LLM-generated translations act as noise and degrade performance. <strong>Only human/expert translations improve the score.</strong></li>\n<li><strong>Data Cleanliness is Critical</strong>: Hidden database artifacts (like <code>2.83333</code> instead of fractions) or unexpected languages (German/French) will immediately drop the score.</li>\n<li><strong>Training Strategy</strong>: Unidirectional training (Akkadian → English) is required for augmented data to prevent reverse-direction interference.</li>\n<li><strong>Perhaps The Best Combination</strong>: Train on <code>train.csv</code> + <code>Step 3 (V2 Publications)</code> + <code>Step 4 (Cleaned Sentences)</code> + <code>Step 5 (Michel Corpus)</code>.\nFor various reasons, I have not been able to make full use of all the data sets available.\nThese are some datasets I have compiled myself; due to time constraints, I have only used two of them:augmented_step3v2_publications.csv</li>\n</ol>\n<p>tip:As I don’t have access to high-spec machine learning hardware and I’m short on time (I’m working hard to make a living), many of my experiments have to rely on free resources or Kaggle’s free online GPUs and data allowances. I’d recommend that you use Kaggle’s built-in benchmarking tools to test your models for free. Thank you to Kaggle for providing us with so many high-quality, free resources.</p>",
  "messages": [
    {
      "id": "3427322",
      "postDate": "03/24/2026 00:40:36",
      "content": "<h1>Old Assyrian Machine Translation Experiments Summary</h1>\n<h2>1. Baseline Model</h2>\n<ul>\n<li><strong>Task</strong>: Translating Old Assyrian transliteration to English (Kaggle Deep Past Challenge).</li>\n<li><strong>Model Architecture</strong>: ByT5 (Byte-level T5), <code>MAX_LENGTH=512</code>.</li>\n<li><strong>Base Dataset (A)</strong>: <code>train.csv</code> (1,561 rows) + <code>test.csv</code> (4 rows).</li>\n<li><strong>Validation Strategy</strong>: Fixed validation set randomly sampled from the original <code>train.csv</code> to prevent data leakage.</li>\n<li><strong>Baseline Score</strong>: ~33.1 (A).</li>\n</ul>\n<hr>\n<h2>2. Experiments &amp; Findings</h2>\n<h3>Experiment 1: Synthetic Data via Sentence Recombination (Step 1)</h3>\n<ul>\n<li><strong>Method</strong>: Joined <code>Sentences_Oare_FirstWord_LinNum.csv</code> with <code>published_texts.csv</code> on <code>text_uuid</code>. Reconstructed document-level translations by concatenating sentences and filtering by length ratio (<code>RATIO_MIN=0.33</code>, <code>RATIO_MAX=2.69</code>).</li>\n<li><strong>Data Yield</strong>: 990 \"clean\" pairs.</li>\n<li><strong>Result</strong>: <strong>Score Decreased (Negative impact).</strong></li>\n<li><strong>Analysis</strong>: Recombined texts likely suffered from unnatural sentence boundaries and contextual misalignment.</li>\n</ul>\n<h3>Experiment 2: LLM Translation of Unlabeled Texts (Step 2)</h3>\n<ul>\n<li><strong>Method</strong>: Used LLM (Claude) to translate <code>published_texts.csv</code> (which had transliteration but no translation). Configured with 3 few-shot examples and dictionary assistance (<code>OA_Lexicon</code> + <code>eBL_Dictionary</code>).</li>\n<li><strong>Data Yield</strong>: ~3,000+ pairs generated.</li>\n<li><strong>Result</strong>: <strong>Score Decreased (A+E = 32.7 vs Baseline 33.1).</strong></li>\n<li><strong>Analysis</strong>:<ul>\n<li><em>Validation Leakage</em>: Initially, synthetic data leaked into the validation set, causing artificially high validation scores (46.88) while actual inference scores dropped.</li>\n<li><em>Length Truncation</em>: Many generated translations exceeded ByT5's 512-byte limit and were silently truncated.</li>\n<li><em>Domain Drift/Noise</em>: LLM-generated translations lacked the stylistic consistency of human experts, introducing noise that outweighed the benefits of more data.</li></ul></li>\n</ul>\n<h3>Experiment 3: Extracting Expert Translations from Academic PDFs (Step 3)</h3>\n<ul>\n<li><strong>Raw Data Source</strong>: <code>publications.csv</code> (~9,890 pages of OCR text from Assyriology academic publications, including EL/Lewy, AKT, CMK, NABU, CCT series).</li>\n<li><strong>Method</strong>: First apply rule-based filters to narrow 9,890 pages to high-quality bilingual pages, then use LLM to extract (transliteration, translation) pairs from each page.</li>\n<li><strong>V1 Filter Rules</strong>:<ol>\n<li><code>has_akkadian == true</code> (pre-labeled field in the CSV)</li>\n<li>Must contain Akkadian special characters (e.g., <code>š, ṭ, ṣ, ā, ē, ī, ū, ḫ</code>)</li>\n<li>Must contain core Akkadian vocabulary (e.g., <code>um-ma</code>, <code>qí-bi-ma</code>, <code>ma-na</code>, <code>KÙ.BABBAR</code>, <code>AN.NA</code>)</li>\n<li>Must contain German or English keywords (e.g., <code>von, der, Silber</code> or <code>silver, mina, shekel</code>)</li>\n<li>Must contain standard line number format (regex <code>\\d{1,2}[\\s\\.\\)]</code>)</li></ol></li>\n<li><strong>V1 LLM Prompt (EXTRACT_PROMPT) Key Points</strong>:<ul>\n<li>Role: Expert in Old Assyrian cuneiform texts and Assyriology</li>\n<li>Task: Extract all (transliteration, translation) pairs</li>\n<li>If translation is in German, translate it to English</li>\n<li>Keep proper names and technical terms untranslated (Puzur-Aššur, kārum, ḫamuštum, etc.)</li>\n<li>Mark broken/missing text as <code>&lt;gap&gt;</code></li>\n<li>Only extract pairs where transliteration is 10–500 characters</li>\n<li>Includes 4 few-shot examples (English, German, Seal/contract format)</li>\n<li>Return pure JSON array; return <code>[]</code> if no valid pairs found</li></ul></li>\n<li><strong>Data Yield (V1)</strong>: 556 valid pages found. A sample of 300 pairs was extracted.</li>\n<li><strong>Result</strong>: <strong>Score Increased (+0.6 improvement).</strong></li>\n<li><strong>Analysis</strong>: Human expert translations from academic sources perfectly matched the style and quality of the training set.</li>\n</ul>\n<h3>Experiment 4: Unlocking Direct Sentence Alignments (Step 4)</h3>\n<ul>\n<li><strong>Method</strong>: Discovered 8,564 unused expert-translated sentences in <code>Sentences_Oare</code>. Wrote a script to align them directly with document transliterations without concatenating them.</li>\n<li><strong>Data Yield</strong>: 589 pairs (after removing duplicates found in <code>train.csv</code>).</li>\n<li><strong>Result</strong>: <strong>Score Decreased.</strong></li>\n<li><strong>Analysis</strong>: 412 rows contained \"float artifacts\" (e.g., the fraction <code>2 5/6</code> was stored as <code>2.83333</code> in the DB), which confused the byte-level tokenizer.</li>\n<li><strong>Fix Applied</strong>: Added strict regex filtering to remove float artifacts and German translations, resulting in <strong>401 clean pairs</strong> (pending re-test).</li>\n</ul>\n<h3>Experiment 5: Cleaning the Michel Letters Corpus (Step 5)</h3>\n<ul>\n<li><strong>Method</strong>: Processed the newly acquired <code>Michel Old Assyrian Letters Corpus.csv</code> (264 rows).</li>\n<li><strong>Data Yield</strong>:<ul>\n<li><code>augmented_step5_michel.csv</code>: 74 clean pairs (≤500 chars).</li>\n<li><code>augmented_step5_michel_full.csv</code>: 474 pairs (long texts split by paragraph line numbers).</li></ul></li>\n<li><strong>Result</strong>: <strong>Pending test.</strong> (Expected to be highly positive as it is a premium human-translated source).</li>\n</ul>\n<h3>Experiment 6: Optimizing PDF Extraction Filter (Step 3 V2)</h3>\n<ul>\n<li><strong>Method</strong>: Analyzed the 9,334 pages blocked by the V1 filter. Discovered that 900+ pages of high-quality French translations (CMK 2001) were blocked, alongside many core texts (AKT 7a, EL 1/2) that used non-standard line numbers (<code>obv. 1</code>, <code>1'</code>).</li>\n<li><strong>V2 Filter Changes</strong>:<ol>\n<li><strong>Added French keyword detection</strong> (<code>mines, sicles, argent, cuivre, étain, tissus, doit, reçu, envoyé...</code>)</li>\n<li><strong>Extended line number regex</strong>: <code>\\d{1,2}</code> → <code>\\d{1,3}</code> (supports 3-digit numbers), added fragment format <code>\\d{1,3}'</code>, added archaeological format <code>obv/rev/col</code></li>\n<li><strong>Added PDF blacklist (BLACKLIST_PDFS)</strong> to exclude: AKT 7a/7b (Turkish commentary), Barjamovic (academic prose, 148 pages), Grayson RIMA (Neo-Assyrian dialect), Dercksen trade analysis, Larsen 2002, etc.</li></ol></li>\n<li><strong>V2 LLM Prompt Changes</strong>:<ul>\n<li>Updated language description to: <code>\"in German, English, or French\"</code></li>\n<li>Explicitly instructs LLM to translate French to English as well</li>\n<li>Added French few-shot example (CMK format): <code>\"Ainsi (parle) Aššur-bāni: dis à Pūšu-kēn...\"</code> → English</li>\n<li>Total: 5 few-shot examples (English × 1, German × 1, French × 1, Seal × 1, plus original English example)</li></ul></li>\n<li><strong>Data Yield</strong>: Expanded valid pages from 556 to <strong>536 highly concentrated, noise-free pages</strong> (yielding an estimated 400-600 premium pairs).</li>\n</ul>\n<hr>\n<h2>3. Key Conclusions</h2>\n<ol>\n<li><strong>Quality over Quantity</strong>: ByT5 is extremely sensitive to data quality. LLM-generated translations act as noise and degrade performance. <strong>Only human/expert translations improve the score.</strong></li>\n<li><strong>Data Cleanliness is Critical</strong>: Hidden database artifacts (like <code>2.83333</code> instead of fractions) or unexpected languages (German/French) will immediately drop the score.</li>\n<li><strong>Training Strategy</strong>: Unidirectional training (Akkadian → English) is required for augmented data to prevent reverse-direction interference.</li>\n<li><strong>Perhaps The Best Combination</strong>: Train on <code>train.csv</code> + <code>Step 3 (V2 Publications)</code> + <code>Step 4 (Cleaned Sentences)</code> + <code>Step 5 (Michel Corpus)</code>.\nFor various reasons, I have not been able to make full use of all the data sets available.\nThese are some datasets I have compiled myself; due to time constraints, I have only used two of them:augmented_step3v2_publications.csv</li>\n</ol>\n<p>tip:As I don’t have access to high-spec machine learning hardware and I’m short on time (I’m working hard to make a living), many of my experiments have to rely on free resources or Kaggle’s free online GPUs and data allowances. I’d recommend that you use Kaggle’s built-in benchmarking tools to test your models for free. Thank you to Kaggle for providing us with so many high-quality, free resources.</p>",
      "rawMarkdown": "# Old Assyrian Machine Translation Experiments Summary\n\n## 1. Baseline Model\n- **Task**: Translating Old Assyrian transliteration to English (Kaggle Deep Past Challenge).\n- **Model Architecture**: ByT5 (Byte-level T5), `MAX_LENGTH=512`.\n- **Base Dataset (A)**: `train.csv` (1,561 rows) + `test.csv` (4 rows).\n- **Validation Strategy**: Fixed validation set randomly sampled from the original `train.csv` to prevent data leakage.\n- **Baseline Score**: ~33.1 (A).\n\n---\n\n## 2. Experiments & Findings\n\n### Experiment 1: Synthetic Data via Sentence Recombination (Step 1)\n- **Method**: Joined `Sentences_Oare_FirstWord_LinNum.csv` with `published_texts.csv` on `text_uuid`. Reconstructed document-level translations by concatenating sentences and filtering by length ratio (`RATIO_MIN=0.33`, `RATIO_MAX=2.69`).\n- **Data Yield**: 990 \"clean\" pairs.\n- **Result**: **Score Decreased (Negative impact).**\n- **Analysis**: Recombined texts likely suffered from unnatural sentence boundaries and contextual misalignment.\n\n### Experiment 2: LLM Translation of Unlabeled Texts (Step 2)\n- **Method**: Used LLM (Claude) to translate `published_texts.csv` (which had transliteration but no translation). Configured with 3 few-shot examples and dictionary assistance (`OA_Lexicon` + `eBL_Dictionary`).\n- **Data Yield**: ~3,000+ pairs generated.\n- **Result**: **Score Decreased (A+E = 32.7 vs Baseline 33.1).**\n- **Analysis**:\n  - *Validation Leakage*: Initially, synthetic data leaked into the validation set, causing artificially high validation scores (46.88) while actual inference scores dropped.\n  - *Length Truncation*: Many generated translations exceeded ByT5's 512-byte limit and were silently truncated.\n  - *Domain Drift/Noise*: LLM-generated translations lacked the stylistic consistency of human experts, introducing noise that outweighed the benefits of more data.\n\n### Experiment 3: Extracting Expert Translations from Academic PDFs (Step 3)\n- **Raw Data Source**: `publications.csv` (~9,890 pages of OCR text from Assyriology academic publications, including EL/Lewy, AKT, CMK, NABU, CCT series).\n- **Method**: First apply rule-based filters to narrow 9,890 pages to high-quality bilingual pages, then use LLM to extract (transliteration, translation) pairs from each page.\n- **V1 Filter Rules**:\n  1. `has_akkadian == true` (pre-labeled field in the CSV)\n  2. Must contain Akkadian special characters (e.g., `š, ṭ, ṣ, ā, ē, ī, ū, ḫ`)\n  3. Must contain core Akkadian vocabulary (e.g., `um-ma`, `qí-bi-ma`, `ma-na`, `KÙ.BABBAR`, `AN.NA`)\n  4. Must contain German or English keywords (e.g., `von, der, Silber` or `silver, mina, shekel`)\n  5. Must contain standard line number format (regex `\\d{1,2}[\\s\\.\\)]`)\n- **V1 LLM Prompt (EXTRACT_PROMPT) Key Points**:\n  - Role: Expert in Old Assyrian cuneiform texts and Assyriology\n  - Task: Extract all (transliteration, translation) pairs\n  - If translation is in German, translate it to English\n  - Keep proper names and technical terms untranslated (Puzur-Aššur, kārum, ḫamuštum, etc.)\n  - Mark broken/missing text as `<gap>`\n  - Only extract pairs where transliteration is 10–500 characters\n  - Includes 4 few-shot examples (English, German, Seal/contract format)\n  - Return pure JSON array; return `[]` if no valid pairs found\n- **Data Yield (V1)**: 556 valid pages found. A sample of 300 pairs was extracted.\n- **Result**: **Score Increased (+0.6 improvement).**\n- **Analysis**: Human expert translations from academic sources perfectly matched the style and quality of the training set.\n\n### Experiment 4: Unlocking Direct Sentence Alignments (Step 4)\n- **Method**: Discovered 8,564 unused expert-translated sentences in `Sentences_Oare`. Wrote a script to align them directly with document transliterations without concatenating them.\n- **Data Yield**: 589 pairs (after removing duplicates found in `train.csv`).\n- **Result**: **Score Decreased.**\n- **Analysis**: 412 rows contained \"float artifacts\" (e.g., the fraction `2 5/6` was stored as `2.83333` in the DB), which confused the byte-level tokenizer.\n- **Fix Applied**: Added strict regex filtering to remove float artifacts and German translations, resulting in **401 clean pairs** (pending re-test).\n\n### Experiment 5: Cleaning the Michel Letters Corpus (Step 5)\n- **Method**: Processed the newly acquired `Michel Old Assyrian Letters Corpus.csv` (264 rows).\n- **Data Yield**:\n  - `augmented_step5_michel.csv`: 74 clean pairs (≤500 chars).\n  - `augmented_step5_michel_full.csv`: 474 pairs (long texts split by paragraph line numbers).\n- **Result**: **Pending test.** (Expected to be highly positive as it is a premium human-translated source).\n\n### Experiment 6: Optimizing PDF Extraction Filter (Step 3 V2)\n- **Method**: Analyzed the 9,334 pages blocked by the V1 filter. Discovered that 900+ pages of high-quality French translations (CMK 2001) were blocked, alongside many core texts (AKT 7a, EL 1/2) that used non-standard line numbers (`obv. 1`, `1'`).\n- **V2 Filter Changes**:\n  1. **Added French keyword detection** (`mines, sicles, argent, cuivre, étain, tissus, doit, reçu, envoyé...`)\n  2. **Extended line number regex**: `\\d{1,2}` → `\\d{1,3}` (supports 3-digit numbers), added fragment format `\\d{1,3}'`, added archaeological format `obv/rev/col`\n  3. **Added PDF blacklist (BLACKLIST_PDFS)** to exclude: AKT 7a/7b (Turkish commentary), Barjamovic (academic prose, 148 pages), Grayson RIMA (Neo-Assyrian dialect), Dercksen trade analysis, Larsen 2002, etc.\n- **V2 LLM Prompt Changes**:\n  - Updated language description to: `\"in German, English, or French\"`\n  - Explicitly instructs LLM to translate French to English as well\n  - Added French few-shot example (CMK format): `\"Ainsi (parle) Aššur-bāni: dis à Pūšu-kēn...\"` → English\n  - Total: 5 few-shot examples (English × 1, German × 1, French × 1, Seal × 1, plus original English example)\n- **Data Yield**: Expanded valid pages from 556 to **536 highly concentrated, noise-free pages** (yielding an estimated 400-600 premium pairs).\n\n---\n\n## 3. Key Conclusions\n1. **Quality over Quantity**: ByT5 is extremely sensitive to data quality. LLM-generated translations act as noise and degrade performance. **Only human/expert translations improve the score.**\n2. **Data Cleanliness is Critical**: Hidden database artifacts (like `2.83333` instead of fractions) or unexpected languages (German/French) will immediately drop the score.\n3. **Training Strategy**: Unidirectional training (Akkadian → English) is required for augmented data to prevent reverse-direction interference.\n4. **Perhaps The Best Combination**: Train on `train.csv` + `Step 3 (V2 Publications)` + `Step 4 (Cleaned Sentences)` + `Step 5 (Michel Corpus)`.\nFor various reasons, I have not been able to make full use of all the data sets available.\nThese are some datasets I have compiled myself; due to time constraints, I have only used two of them:augmented_step3v2_publications.csv\n\n\ntip:As I don’t have access to high-spec machine learning hardware and I’m short on time (I’m working hard to make a living), many of my experiments have to rely on free resources or Kaggle’s free online GPUs and data allowances. I’d recommend that you use Kaggle’s built-in benchmarking tools to test your models for free. Thank you to Kaggle for providing us with so many high-quality, free resources.",
      "votes": null
    },
    {
      "id": "3427450",
      "postDate": "03/24/2026 04:50:25",
      "content": "<p>how do you train ByT5 on kaggle? i kept getting ram errors.</p>",
      "rawMarkdown": "how do you train ByT5 on kaggle? i kept getting ram errors.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3427450,
      "author_name": "konohayui",
      "author_url": "",
      "post_date": "03/24/2026 04:50:25",
      "content": "<p>how do you train ByT5 on kaggle? i kept getting ram errors.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3427322": "# Old Assyrian Machine Translation Experiments Summary\n\n## 1. Baseline Model\n- **Task**: Translating Old Assyrian transliteration to English (Kaggle Deep Past Challenge).\n- **Model Architecture**: ByT5 (Byte-level T5), `MAX_LENGTH=512`.\n- **Base Dataset (A)**: `train.csv` (1,561 rows) + `test.csv` (4 rows).\n- **Validation Strategy**: Fixed validation set randomly sampled from the original `train.csv` to prevent data leakage.\n- **Baseline Score**: ~33.1 (A).\n\n---\n\n## 2. Experiments & Findings\n\n### Experiment 1: Synthetic Data via Sentence Recombination (Step 1)\n- **Method**: Joined `Sentences_Oare_FirstWord_LinNum.csv` with `published_texts.csv` on `text_uuid`. Reconstructed document-level translations by concatenating sentences and filtering by length ratio (`RATIO_MIN=0.33`, `RATIO_MAX=2.69`).\n- **Data Yield**: 990 \"clean\" pairs.\n- **Result**: **Score Decreased (Negative impact).**\n- **Analysis**: Recombined texts likely suffered from unnatural sentence boundaries and contextual misalignment.\n\n### Experiment 2: LLM Translation of Unlabeled Texts (Step 2)\n- **Method**: Used LLM (Claude) to translate `published_texts.csv` (which had transliteration but no translation). Configured with 3 few-shot examples and dictionary assistance (`OA_Lexicon` + `eBL_Dictionary`).\n- **Data Yield**: ~3,000+ pairs generated.\n- **Result**: **Score Decreased (A+E = 32.7 vs Baseline 33.1).**\n- **Analysis**:\n  - *Validation Leakage*: Initially, synthetic data leaked into the validation set, causing artificially high validation scores (46.88) while actual inference scores dropped.\n  - *Length Truncation*: Many generated translations exceeded ByT5's 512-byte limit and were silently truncated.\n  - *Domain Drift/Noise*: LLM-generated translations lacked the stylistic consistency of human experts, introducing noise that outweighed the benefits of more data.\n\n### Experiment 3: Extracting Expert Translations from Academic PDFs (Step 3)\n- **Raw Data Source**: `publications.csv` (~9,890 pages of OCR text from Assyriology academic publications, including EL/Lewy, AKT, CMK, NABU, CCT series).\n- **Method**: First apply rule-based filters to narrow 9,890 pages to high-quality bilingual pages, then use LLM to extract (transliteration, translation) pairs from each page.\n- **V1 Filter Rules**:\n  1. `has_akkadian == true` (pre-labeled field in the CSV)\n  2. Must contain Akkadian special characters (e.g., `š, ṭ, ṣ, ā, ē, ī, ū, ḫ`)\n  3. Must contain core Akkadian vocabulary (e.g., `um-ma`, `qí-bi-ma`, `ma-na`, `KÙ.BABBAR`, `AN.NA`)\n  4. Must contain German or English keywords (e.g., `von, der, Silber` or `silver, mina, shekel`)\n  5. Must contain standard line number format (regex `\\d{1,2}[\\s\\.\\)]`)\n- **V1 LLM Prompt (EXTRACT_PROMPT) Key Points**:\n  - Role: Expert in Old Assyrian cuneiform texts and Assyriology\n  - Task: Extract all (transliteration, translation) pairs\n  - If translation is in German, translate it to English\n  - Keep proper names and technical terms untranslated (Puzur-Aššur, kārum, ḫamuštum, etc.)\n  - Mark broken/missing text as `<gap>`\n  - Only extract pairs where transliteration is 10–500 characters\n  - Includes 4 few-shot examples (English, German, Seal/contract format)\n  - Return pure JSON array; return `[]` if no valid pairs found\n- **Data Yield (V1)**: 556 valid pages found. A sample of 300 pairs was extracted.\n- **Result**: **Score Increased (+0.6 improvement).**\n- **Analysis**: Human expert translations from academic sources perfectly matched the style and quality of the training set.\n\n### Experiment 4: Unlocking Direct Sentence Alignments (Step 4)\n- **Method**: Discovered 8,564 unused expert-translated sentences in `Sentences_Oare`. Wrote a script to align them directly with document transliterations without concatenating them.\n- **Data Yield**: 589 pairs (after removing duplicates found in `train.csv`).\n- **Result**: **Score Decreased.**\n- **Analysis**: 412 rows contained \"float artifacts\" (e.g., the fraction `2 5/6` was stored as `2.83333` in the DB), which confused the byte-level tokenizer.\n- **Fix Applied**: Added strict regex filtering to remove float artifacts and German translations, resulting in **401 clean pairs** (pending re-test).\n\n### Experiment 5: Cleaning the Michel Letters Corpus (Step 5)\n- **Method**: Processed the newly acquired `Michel Old Assyrian Letters Corpus.csv` (264 rows).\n- **Data Yield**:\n  - `augmented_step5_michel.csv`: 74 clean pairs (≤500 chars).\n  - `augmented_step5_michel_full.csv`: 474 pairs (long texts split by paragraph line numbers).\n- **Result**: **Pending test.** (Expected to be highly positive as it is a premium human-translated source).\n\n### Experiment 6: Optimizing PDF Extraction Filter (Step 3 V2)\n- **Method**: Analyzed the 9,334 pages blocked by the V1 filter. Discovered that 900+ pages of high-quality French translations (CMK 2001) were blocked, alongside many core texts (AKT 7a, EL 1/2) that used non-standard line numbers (`obv. 1`, `1'`).\n- **V2 Filter Changes**:\n  1. **Added French keyword detection** (`mines, sicles, argent, cuivre, étain, tissus, doit, reçu, envoyé...`)\n  2. **Extended line number regex**: `\\d{1,2}` → `\\d{1,3}` (supports 3-digit numbers), added fragment format `\\d{1,3}'`, added archaeological format `obv/rev/col`\n  3. **Added PDF blacklist (BLACKLIST_PDFS)** to exclude: AKT 7a/7b (Turkish commentary), Barjamovic (academic prose, 148 pages), Grayson RIMA (Neo-Assyrian dialect), Dercksen trade analysis, Larsen 2002, etc.\n- **V2 LLM Prompt Changes**:\n  - Updated language description to: `\"in German, English, or French\"`\n  - Explicitly instructs LLM to translate French to English as well\n  - Added French few-shot example (CMK format): `\"Ainsi (parle) Aššur-bāni: dis à Pūšu-kēn...\"` → English\n  - Total: 5 few-shot examples (English × 1, German × 1, French × 1, Seal × 1, plus original English example)\n- **Data Yield**: Expanded valid pages from 556 to **536 highly concentrated, noise-free pages** (yielding an estimated 400-600 premium pairs).\n\n---\n\n## 3. Key Conclusions\n1. **Quality over Quantity**: ByT5 is extremely sensitive to data quality. LLM-generated translations act as noise and degrade performance. **Only human/expert translations improve the score.**\n2. **Data Cleanliness is Critical**: Hidden database artifacts (like `2.83333` instead of fractions) or unexpected languages (German/French) will immediately drop the score.\n3. **Training Strategy**: Unidirectional training (Akkadian → English) is required for augmented data to prevent reverse-direction interference.\n4. **Perhaps The Best Combination**: Train on `train.csv` + `Step 3 (V2 Publications)` + `Step 4 (Cleaned Sentences)` + `Step 5 (Michel Corpus)`.\nFor various reasons, I have not been able to make full use of all the data sets available.\nThese are some datasets I have compiled myself; due to time constraints, I have only used two of them:augmented_step3v2_publications.csv\n\n\ntip:As I don’t have access to high-spec machine learning hardware and I’m short on time (I’m working hard to make a living), many of my experiments have to rely on free resources or Kaggle’s free online GPUs and data allowances. I’d recommend that you use Kaggle’s built-in benchmarking tools to test your models for free. Thank you to Kaggle for providing us with so many high-quality, free resources.",
    "3427450": "how do you train ByT5 on kaggle? i kept getting ram errors."
  },
  "source": "meta"
}