{
  "id": 671458,
  "title": "How are you handling Normalization and Sentence Alignment?",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/671458",
  "author_name": "hikaggler",
  "post_date": "2026-02-02T03:49:40.448000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>I'm currently working on the Akkadian-to-English translation task using ByT5-Large. I’ve encountered several technical challenges regarding data preprocessing and would love to hear how others are approaching them.</p>\n<h4>1. Normalization Strategy</h4>\n<p>Akkadian transliterations are complex. I’m currently implementing the following normalization steps. Do you find these helpful, or is it better to keep the raw Unicode strings for character-based models?</p>\n<pre><code>import re\n\ndef normalize_transliteration(text):\n    # 1. Handling ASCII substitutes to Diacritics\n    # (e.g., standardizing \"sz\" -&gt; \"š\", \"s,\" -&gt; \"ṣ\", \"t,\" -&gt; \"ṭ\")\n    text = text.replace(\"sz\", \"š\").replace(\"s,\", \"ṣ\").replace(\"t,\", \"ṭ\")\n\n    # 2. Handling numeric readings to accent marks (a2 -&gt; á, e3 -&gt; è, etc.)\n    accent_map = {\"a2\": \"á\", \"a3\": \"à\", \"e2\": \"é\", \"e3\": \"è\", ...}\n    for old, new in accent_map.items():\n        text = text.replace(old, new)\n\n    # 3. Gap handling: converting \"...\" or \"x x x\" to special tags\n    text = re.sub(r'\\.{3,}|…', '&lt;big_gap&gt;', text)\n    text = re.sub(r'\\bx x x+\\b', '&lt;big_gap&gt;', text)\n\n    # 4. Canonicalizing diacritics (e.g., ḫ -&gt; h)\n    # Note: Test data seems to use 'h' exclusively. Do you convert all 'ḫ' to 'h'?\n    text = text.replace(\"ḫ\", \"h\")\n\n    return text.strip()\n</code></pre>\n<ul>\n<li><strong>Question</strong>: Are you stripping brackets <code>[]</code> and other philological notations?</li>\n<li><strong>Question</strong>: Is converting <code>ḫ</code> to <code>h</code> throughout the dataset recommended for consistency with the test set?</li>\n</ul>\n<h4>2. Sentence-level Segmentation and Alignment</h4>\n<p>The <code>train.csv</code> gives us document-level text, but the <code>test.csv</code> consists of only 4 short sentences. </p>\n<ul>\n<li>How are you splitting the documents into sentences while maintaining the alignment between Akkadian and English? </li>\n<li>Simple punctuation splitting (by <code>.</code>) often results in different segment counts. Are you using any specific alignment tools (e.g., hunalign) or relying on document-level training?</li>\n</ul>\n<h4>3. Data Source Utilization (published_texts.csv)</h4>\n<p>I noticed that <code>published_texts.csv</code> has ~8k rows but the <code>AICC_translation</code> column contains URLs instead of text.</p>\n<ul>\n<li>Has anyone successfully integrated this data into their <strong>translation pipeline</strong>, or are you using it strictly for <strong>unsupervised pre-training</strong> (like Span Corruption)?</li>\n<li>Similarly, how are you handling the large <code>publications.csv</code> (OCR text)? It’s quite noisy with multiple languages.</li>\n</ul>\n<h4>4. Training Stability</h4>\n<p>I've noticed that our model sometimes collapses into \"template-like\" outputs (repetitive letter headers like \"To Ali-ahum from Aššur-malik\") during fine-tuning. </p>\n<ul>\n<li>What learning rates or regularization techniques (e.g., Label Smoothing) have you found effective to keep the model from overfitting to frequent document structures?</li>\n</ul>\n<p>I look forward to hearing about your strategies!</p>",
  "messages": [
    {
      "id": 3400681,
      "postDate": "2026-02-02T03:49:40.450Z",
      "content": "<p>Hi everyone,</p>\n<p>I'm currently working on the Akkadian-to-English translation task using ByT5-Large. I’ve encountered several technical challenges regarding data preprocessing and would love to hear how others are approaching them.</p>\n<h4>1. Normalization Strategy</h4>\n<p>Akkadian transliterations are complex. I’m currently implementing the following normalization steps. Do you find these helpful, or is it better to keep the raw Unicode strings for character-based models?</p>\n<pre><code>import re\n\ndef normalize_transliteration(text):\n    # 1. Handling ASCII substitutes to Diacritics\n    # (e.g., standardizing \"sz\" -&gt; \"š\", \"s,\" -&gt; \"ṣ\", \"t,\" -&gt; \"ṭ\")\n    text = text.replace(\"sz\", \"š\").replace(\"s,\", \"ṣ\").replace(\"t,\", \"ṭ\")\n\n    # 2. Handling numeric readings to accent marks (a2 -&gt; á, e3 -&gt; è, etc.)\n    accent_map = {\"a2\": \"á\", \"a3\": \"à\", \"e2\": \"é\", \"e3\": \"è\", ...}\n    for old, new in accent_map.items():\n        text = text.replace(old, new)\n\n    # 3. Gap handling: converting \"...\" or \"x x x\" to special tags\n    text = re.sub(r'\\.{3,}|…', '&lt;big_gap&gt;', text)\n    text = re.sub(r'\\bx x x+\\b', '&lt;big_gap&gt;', text)\n\n    # 4. Canonicalizing diacritics (e.g., ḫ -&gt; h)\n    # Note: Test data seems to use 'h' exclusively. Do you convert all 'ḫ' to 'h'?\n    text = text.replace(\"ḫ\", \"h\")\n\n    return text.strip()\n</code></pre>\n<ul>\n<li><strong>Question</strong>: Are you stripping brackets <code>[]</code> and other philological notations?</li>\n<li><strong>Question</strong>: Is converting <code>ḫ</code> to <code>h</code> throughout the dataset recommended for consistency with the test set?</li>\n</ul>\n<h4>2. Sentence-level Segmentation and Alignment</h4>\n<p>The <code>train.csv</code> gives us document-level text, but the <code>test.csv</code> consists of only 4 short sentences. </p>\n<ul>\n<li>How are you splitting the documents into sentences while maintaining the alignment between Akkadian and English? </li>\n<li>Simple punctuation splitting (by <code>.</code>) often results in different segment counts. Are you using any specific alignment tools (e.g., hunalign) or relying on document-level training?</li>\n</ul>\n<h4>3. Data Source Utilization (published_texts.csv)</h4>\n<p>I noticed that <code>published_texts.csv</code> has ~8k rows but the <code>AICC_translation</code> column contains URLs instead of text.</p>\n<ul>\n<li>Has anyone successfully integrated this data into their <strong>translation pipeline</strong>, or are you using it strictly for <strong>unsupervised pre-training</strong> (like Span Corruption)?</li>\n<li>Similarly, how are you handling the large <code>publications.csv</code> (OCR text)? It’s quite noisy with multiple languages.</li>\n</ul>\n<h4>4. Training Stability</h4>\n<p>I've noticed that our model sometimes collapses into \"template-like\" outputs (repetitive letter headers like \"To Ali-ahum from Aššur-malik\") during fine-tuning. </p>\n<ul>\n<li>What learning rates or regularization techniques (e.g., Label Smoothing) have you found effective to keep the model from overfitting to frequent document structures?</li>\n</ul>\n<p>I look forward to hearing about your strategies!</p>",
      "rawMarkdown": "Hi everyone,\n\nI'm currently working on the Akkadian-to-English translation task using ByT5-Large. I’ve encountered several technical challenges regarding data preprocessing and would love to hear how others are approaching them.\n\n#### 1. Normalization Strategy\nAkkadian transliterations are complex. I’m currently implementing the following normalization steps. Do you find these helpful, or is it better to keep the raw Unicode strings for character-based models?\n\n```python\nimport re\n\ndef normalize_transliteration(text):\n    # 1. Handling ASCII substitutes to Diacritics\n    # (e.g., standardizing \"sz\" -> \"š\", \"s,\" -> \"ṣ\", \"t,\" -> \"ṭ\")\n    text = text.replace(\"sz\", \"š\").replace(\"s,\", \"ṣ\").replace(\"t,\", \"ṭ\")\n\n    # 2. Handling numeric readings to accent marks (a2 -> á, e3 -> è, etc.)\n    accent_map = {\"a2\": \"á\", \"a3\": \"à\", \"e2\": \"é\", \"e3\": \"è\", ...}\n    for old, new in accent_map.items():\n        text = text.replace(old, new)\n\n    # 3. Gap handling: converting \"...\" or \"x x x\" to special tags\n    text = re.sub(r'\\.{3,}|…', '<big_gap>', text)\n    text = re.sub(r'\\bx x x+\\b', '<big_gap>', text)\n\n    # 4. Canonicalizing diacritics (e.g., ḫ -> h)\n    # Note: Test data seems to use 'h' exclusively. Do you convert all 'ḫ' to 'h'?\n    text = text.replace(\"ḫ\", \"h\")\n    \n    return text.strip()\n```\n\n- **Question**: Are you stripping brackets `[]` and other philological notations?\n- **Question**: Is converting `ḫ` to `h` throughout the dataset recommended for consistency with the test set?\n\n#### 2. Sentence-level Segmentation and Alignment\nThe `train.csv` gives us document-level text, but the `test.csv` consists of only 4 short sentences. \n\n- How are you splitting the documents into sentences while maintaining the alignment between Akkadian and English? \n- Simple punctuation splitting (by `.`) often results in different segment counts. Are you using any specific alignment tools (e.g., hunalign) or relying on document-level training?\n\n#### 3. Data Source Utilization (published_texts.csv)\nI noticed that `published_texts.csv` has ~8k rows but the `AICC_translation` column contains URLs instead of text.\n\n- Has anyone successfully integrated this data into their **translation pipeline**, or are you using it strictly for **unsupervised pre-training** (like Span Corruption)?\n- Similarly, how are you handling the large `publications.csv` (OCR text)? It’s quite noisy with multiple languages.\n\n#### 4. Training Stability\nI've noticed that our model sometimes collapses into \"template-like\" outputs (repetitive letter headers like \"To Ali-ahum from Aššur-malik\") during fine-tuning. \n\n- What learning rates or regularization techniques (e.g., Label Smoothing) have you found effective to keep the model from overfitting to frequent document structures?\n\nI look forward to hearing about your strategies!\n",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3400681": "Hi everyone,\n\nI'm currently working on the Akkadian-to-English translation task using ByT5-Large. I’ve encountered several technical challenges regarding data preprocessing and would love to hear how others are approaching them.\n\n#### 1. Normalization Strategy\nAkkadian transliterations are complex. I’m currently implementing the following normalization steps. Do you find these helpful, or is it better to keep the raw Unicode strings for character-based models?\n\n```python\nimport re\n\ndef normalize_transliteration(text):\n    # 1. Handling ASCII substitutes to Diacritics\n    # (e.g., standardizing \"sz\" -> \"š\", \"s,\" -> \"ṣ\", \"t,\" -> \"ṭ\")\n    text = text.replace(\"sz\", \"š\").replace(\"s,\", \"ṣ\").replace(\"t,\", \"ṭ\")\n\n    # 2. Handling numeric readings to accent marks (a2 -> á, e3 -> è, etc.)\n    accent_map = {\"a2\": \"á\", \"a3\": \"à\", \"e2\": \"é\", \"e3\": \"è\", ...}\n    for old, new in accent_map.items():\n        text = text.replace(old, new)\n\n    # 3. Gap handling: converting \"...\" or \"x x x\" to special tags\n    text = re.sub(r'\\.{3,}|…', '<big_gap>', text)\n    text = re.sub(r'\\bx x x+\\b', '<big_gap>', text)\n\n    # 4. Canonicalizing diacritics (e.g., ḫ -> h)\n    # Note: Test data seems to use 'h' exclusively. Do you convert all 'ḫ' to 'h'?\n    text = text.replace(\"ḫ\", \"h\")\n    \n    return text.strip()\n```\n\n- **Question**: Are you stripping brackets `[]` and other philological notations?\n- **Question**: Is converting `ḫ` to `h` throughout the dataset recommended for consistency with the test set?\n\n#### 2. Sentence-level Segmentation and Alignment\nThe `train.csv` gives us document-level text, but the `test.csv` consists of only 4 short sentences. \n\n- How are you splitting the documents into sentences while maintaining the alignment between Akkadian and English? \n- Simple punctuation splitting (by `.`) often results in different segment counts. Are you using any specific alignment tools (e.g., hunalign) or relying on document-level training?\n\n#### 3. Data Source Utilization (published_texts.csv)\nI noticed that `published_texts.csv` has ~8k rows but the `AICC_translation` column contains URLs instead of text.\n\n- Has anyone successfully integrated this data into their **translation pipeline**, or are you using it strictly for **unsupervised pre-training** (like Span Corruption)?\n- Similarly, how are you handling the large `publications.csv` (OCR text)? It’s quite noisy with multiple languages.\n\n#### 4. Training Stability\nI've noticed that our model sometimes collapses into \"template-like\" outputs (repetitive letter headers like \"To Ali-ahum from Aššur-malik\") during fine-tuning. \n\n- What learning rates or regularization techniques (e.g., Label Smoothing) have you found effective to keep the model from overfitting to frequent document structures?\n\nI look forward to hearing about your strategies!\n"
  }
}