{
  "id": 665209,
  "title": "Two practical stumbling blocks in Akkadian → English MT (and how to address them)",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/665209",
  "author_name": "Adam Anderson",
  "post_date": "2025-12-30T19:17:06.085000",
  "votes": 75,
  "comment_count": 54,
  "views": 0,
  "content": "<p>Based on recent experiments and discussions with several participants, I want to share a set of observations about what appear to be the <em>primary bottlenecks</em> in this competition. These are not model-architecture issues per se, but data- and representation-level issues that strongly affect downstream training, evaluation, and reinforcement learning.</p>\n<p>After multiple iterations, debugging sessions, and failed-but-informative experiments, two main stumbling blocks consistently emerge:</p>\n<p><strong>(1) Named entities (personal names, place names, divine names)</strong>\n<strong>(2) Inconsistent ASCII / transliteration formats across Akkadian datasets</strong></p>\n<p>Both issues disproportionately affect tokenization, alignment, and reward stability, and in practice they limit performance more than model size or optimizer choice.</p>\n<hr>\n<h3>1. Named entities are a dominant source of error</h3>\n<p>Personal names, geographic names, and divine names behave very differently from ordinary lexical items:</p>\n<ul>\n<li>They are often transliterated inconsistently across editions.</li>\n<li>They frequently preserve older orthographic conventions.</li>\n<li>They are semantically opaque to the model unless explicitly supported.</li>\n</ul>\n<p>In experiments, many otherwise reasonable translations fail almost entirely because a name is mangled, dropped, or hallucinated. This affects not only final accuracy but also reward-based methods, where small orthographic deviations can collapse a sentence-level score.</p>\n<p>To address this, I have prepared an <strong>onomasticon</strong> (a curated list of names and attested spellings), which I will share as supplemental data. Participants may find it useful to:</p>\n<ul>\n<li>add it as a lookup or constraint layer,</li>\n<li>bias decoding for known names,</li>\n<li>or use it for post-generation repair.</li>\n</ul>\n<p>Even partial normalization of named entities tends to improve both perceived translation quality and metric stability.</p>\n<hr>\n<h3>2. Transliteration format normalization is not optional</h3>\n<p>A second, and often underestimated, issue is <strong>ASCII-format variation in Akkadian transliteration</strong>. Different corpora encode the same underlying text using different conventions, many of which are <em>not interchangeable</em> without loss.</p>\n<p>A concrete example that came up in a private discussion illustrates the problem. One approach converted diacritics into ASCII sequences (e.g., <code>š → sz</code>, <code>ú → u2</code>, etc.) before training. This is a reasonable instinct, but in this case it was done in the wrong direction: the <em>evaluation data already contains diacritics</em>, and reducing the alphabet removed distinctions that are semantically meaningful in Akkadian.</p>\n<p>For example, the evaluation data expects forms like:</p>\n<ul>\n<li><code>i-ṣí-ba-at</code> rather than <code>i-Si₂-ba-at</code></li>\n<li><code>KÙ-pì-a</code> rather than <code>KU₃-pi₃-a</code></li>\n<li><code>KIŠIB</code> rather than <code>KISZIB</code></li>\n</ul>\n<p>The key takeaway is that <strong>the competition data uses an extended alphabet by design</strong>, and collapsing it into ASCII can degrade both meaning and alignment. While this increases tokenization difficulty, preserving distinctions such as <code>s / ṣ / š</code> and <code>t / ṭ</code> is preferable to losing them.</p>\n<p>The recommended strategy is therefore:</p>\n<ul>\n<li>keep diacritics,</li>\n<li>convert ASCII substitutes (e.g., <code>sz</code>) <em>into</em> diacritics,</li>\n<li>and normalize everything toward the format used in the training and evaluation sets.</li>\n</ul>\n<hr>\n<h3>Gaps, damage markers, and parallel alignment [update: 2/18/26]</h3>\n<p>Another recurring source of confusion concerns damaged text and gap markers in the data:</p>\n<ul>\n<li><code>x</code> represents a single broken sign,</li>\n<li>sequences like <code>x x x x</code> or <code>...</code> represent a larger lacuna.</li>\n</ul>\n<p>For modeling purposes we reduced all breaks to a single marker: <code>&lt;gap&gt;</code></p>\n<ul>\n<li>we removed the tag for <code>&lt;big_gap&gt;</code> from the train and test (and other transliterations). We also deduplicated instances multiple sequential gaps (e.g. <code>&lt;gap&gt; &lt;gap</code>, <code>&lt;gap&gt;-&lt;gap&gt;</code>, <code>&lt;gap&gt; &lt;gap&gt;</code>, <code>&lt;gap&gt;. &lt;gap&gt;</code>, etc.</li>\n</ul>\n<p>Ideally, these gap markers would be <strong>parallelized between transliteration and translation whenever possible</strong>. However, this was not completely accomplished, as the translation we had access to did not pay strict attention to the gaps. This will be an aspect of the challenge in which a significant advantage will remain for those who have controlled for this data. If a large gap appears in one side but not the other, the model is forced to learn misalignment rather than translation.</p>\n<p>This also applies to edge cases such as <code>&lt;gap&gt;</code> attached to a word (e.g., <code>&lt;gap&gt;-A-šùr</code>), which should be preserved rather than blindly removed.</p>\n<hr>\n<h3>Why this matters for training and RL-based methods</h3>\n<p>Many participants have observed that standard supervised fine-tuning produces reasonable loss curves, while reinforcement or preference-based methods fail to improve or show no reward signal. In nearly all cases examined so far, this traces back to <strong>output non-conformance</strong> caused by the issues above.</p>\n<p>If the model is penalized for:</p>\n<ul>\n<li>orthographic mismatches,</li>\n<li>bracket artifacts,</li>\n<li>inconsistent gap handling,</li>\n<li>or malformed named entities,</li>\n</ul>\n<p>then reward functions cannot reliably distinguish “closer” from “farther” outputs. Addressing normalization and alignment first makes rewards smoother and learning signals usable.</p>\n<hr>\n<h3>Closing note</h3>\n<p>None of this is meant as criticism of existing approaches; these are difficult problems, and much of the complexity comes from the philological nature of the data itself. The hope in sharing this publicly is to reduce duplicated effort and make preprocessing choices more transparent across submissions.</p>\n<p>I will continue to share supplemental resources (including the onomasticon) as they are finalized, and I’m happy to discuss normalization or alignment strategies further if helpful to others.</p>",
  "messages": [
    {
      "id": 3383756,
      "postDate": "2025-12-30T19:17:06.087Z",
      "content": "<p>Based on recent experiments and discussions with several participants, I want to share a set of observations about what appear to be the <em>primary bottlenecks</em> in this competition. These are not model-architecture issues per se, but data- and representation-level issues that strongly affect downstream training, evaluation, and reinforcement learning.</p>\n<p>After multiple iterations, debugging sessions, and failed-but-informative experiments, two main stumbling blocks consistently emerge:</p>\n<p><strong>(1) Named entities (personal names, place names, divine names)</strong>\n<strong>(2) Inconsistent ASCII / transliteration formats across Akkadian datasets</strong></p>\n<p>Both issues disproportionately affect tokenization, alignment, and reward stability, and in practice they limit performance more than model size or optimizer choice.</p>\n<hr>\n<h3>1. Named entities are a dominant source of error</h3>\n<p>Personal names, geographic names, and divine names behave very differently from ordinary lexical items:</p>\n<ul>\n<li>They are often transliterated inconsistently across editions.</li>\n<li>They frequently preserve older orthographic conventions.</li>\n<li>They are semantically opaque to the model unless explicitly supported.</li>\n</ul>\n<p>In experiments, many otherwise reasonable translations fail almost entirely because a name is mangled, dropped, or hallucinated. This affects not only final accuracy but also reward-based methods, where small orthographic deviations can collapse a sentence-level score.</p>\n<p>To address this, I have prepared an <strong>onomasticon</strong> (a curated list of names and attested spellings), which I will share as supplemental data. Participants may find it useful to:</p>\n<ul>\n<li>add it as a lookup or constraint layer,</li>\n<li>bias decoding for known names,</li>\n<li>or use it for post-generation repair.</li>\n</ul>\n<p>Even partial normalization of named entities tends to improve both perceived translation quality and metric stability.</p>\n<hr>\n<h3>2. Transliteration format normalization is not optional</h3>\n<p>A second, and often underestimated, issue is <strong>ASCII-format variation in Akkadian transliteration</strong>. Different corpora encode the same underlying text using different conventions, many of which are <em>not interchangeable</em> without loss.</p>\n<p>A concrete example that came up in a private discussion illustrates the problem. One approach converted diacritics into ASCII sequences (e.g., <code>š → sz</code>, <code>ú → u2</code>, etc.) before training. This is a reasonable instinct, but in this case it was done in the wrong direction: the <em>evaluation data already contains diacritics</em>, and reducing the alphabet removed distinctions that are semantically meaningful in Akkadian.</p>\n<p>For example, the evaluation data expects forms like:</p>\n<ul>\n<li><code>i-ṣí-ba-at</code> rather than <code>i-Si₂-ba-at</code></li>\n<li><code>KÙ-pì-a</code> rather than <code>KU₃-pi₃-a</code></li>\n<li><code>KIŠIB</code> rather than <code>KISZIB</code></li>\n</ul>\n<p>The key takeaway is that <strong>the competition data uses an extended alphabet by design</strong>, and collapsing it into ASCII can degrade both meaning and alignment. While this increases tokenization difficulty, preserving distinctions such as <code>s / ṣ / š</code> and <code>t / ṭ</code> is preferable to losing them.</p>\n<p>The recommended strategy is therefore:</p>\n<ul>\n<li>keep diacritics,</li>\n<li>convert ASCII substitutes (e.g., <code>sz</code>) <em>into</em> diacritics,</li>\n<li>and normalize everything toward the format used in the training and evaluation sets.</li>\n</ul>\n<hr>\n<h3>Gaps, damage markers, and parallel alignment [update: 2/18/26]</h3>\n<p>Another recurring source of confusion concerns damaged text and gap markers in the data:</p>\n<ul>\n<li><code>x</code> represents a single broken sign,</li>\n<li>sequences like <code>x x x x</code> or <code>...</code> represent a larger lacuna.</li>\n</ul>\n<p>For modeling purposes we reduced all breaks to a single marker: <code>&lt;gap&gt;</code></p>\n<ul>\n<li>we removed the tag for <code>&lt;big_gap&gt;</code> from the train and test (and other transliterations). We also deduplicated instances multiple sequential gaps (e.g. <code>&lt;gap&gt; &lt;gap</code>, <code>&lt;gap&gt;-&lt;gap&gt;</code>, <code>&lt;gap&gt; &lt;gap&gt;</code>, <code>&lt;gap&gt;. &lt;gap&gt;</code>, etc.</li>\n</ul>\n<p>Ideally, these gap markers would be <strong>parallelized between transliteration and translation whenever possible</strong>. However, this was not completely accomplished, as the translation we had access to did not pay strict attention to the gaps. This will be an aspect of the challenge in which a significant advantage will remain for those who have controlled for this data. If a large gap appears in one side but not the other, the model is forced to learn misalignment rather than translation.</p>\n<p>This also applies to edge cases such as <code>&lt;gap&gt;</code> attached to a word (e.g., <code>&lt;gap&gt;-A-šùr</code>), which should be preserved rather than blindly removed.</p>\n<hr>\n<h3>Why this matters for training and RL-based methods</h3>\n<p>Many participants have observed that standard supervised fine-tuning produces reasonable loss curves, while reinforcement or preference-based methods fail to improve or show no reward signal. In nearly all cases examined so far, this traces back to <strong>output non-conformance</strong> caused by the issues above.</p>\n<p>If the model is penalized for:</p>\n<ul>\n<li>orthographic mismatches,</li>\n<li>bracket artifacts,</li>\n<li>inconsistent gap handling,</li>\n<li>or malformed named entities,</li>\n</ul>\n<p>then reward functions cannot reliably distinguish “closer” from “farther” outputs. Addressing normalization and alignment first makes rewards smoother and learning signals usable.</p>\n<hr>\n<h3>Closing note</h3>\n<p>None of this is meant as criticism of existing approaches; these are difficult problems, and much of the complexity comes from the philological nature of the data itself. The hope in sharing this publicly is to reduce duplicated effort and make preprocessing choices more transparent across submissions.</p>\n<p>I will continue to share supplemental resources (including the onomasticon) as they are finalized, and I’m happy to discuss normalization or alignment strategies further if helpful to others.</p>",
      "rawMarkdown": "Based on recent experiments and discussions with several participants, I want to share a set of observations about what appear to be the *primary bottlenecks* in this competition. These are not model-architecture issues per se, but data- and representation-level issues that strongly affect downstream training, evaluation, and reinforcement learning.\n\nAfter multiple iterations, debugging sessions, and failed-but-informative experiments, two main stumbling blocks consistently emerge:\n\n**(1) Named entities (personal names, place names, divine names)**\n**(2) Inconsistent ASCII / transliteration formats across Akkadian datasets**\n\nBoth issues disproportionately affect tokenization, alignment, and reward stability, and in practice they limit performance more than model size or optimizer choice.\n\n---\n\n### 1. Named entities are a dominant source of error\n\nPersonal names, geographic names, and divine names behave very differently from ordinary lexical items:\n\n* They are often transliterated inconsistently across editions.\n* They frequently preserve older orthographic conventions.\n* They are semantically opaque to the model unless explicitly supported.\n\nIn experiments, many otherwise reasonable translations fail almost entirely because a name is mangled, dropped, or hallucinated. This affects not only final accuracy but also reward-based methods, where small orthographic deviations can collapse a sentence-level score.\n\nTo address this, I have prepared an **onomasticon** (a curated list of names and attested spellings), which I will share as supplemental data. Participants may find it useful to:\n\n* add it as a lookup or constraint layer,\n* bias decoding for known names,\n* or use it for post-generation repair.\n\nEven partial normalization of named entities tends to improve both perceived translation quality and metric stability.\n\n---\n\n### 2. Transliteration format normalization is not optional\n\nA second, and often underestimated, issue is **ASCII-format variation in Akkadian transliteration**. Different corpora encode the same underlying text using different conventions, many of which are *not interchangeable* without loss.\n\nA concrete example that came up in a private discussion illustrates the problem. One approach converted diacritics into ASCII sequences (e.g., `š → sz`, `ú → u2`, etc.) before training. This is a reasonable instinct, but in this case it was done in the wrong direction: the *evaluation data already contains diacritics*, and reducing the alphabet removed distinctions that are semantically meaningful in Akkadian.\n\nFor example, the evaluation data expects forms like:\n\n* `i-ṣí-ba-at` rather than `i-Si₂-ba-at`\n* `KÙ-pì-a` rather than `KU₃-pi₃-a`\n* `KIŠIB` rather than `KISZIB`\n\nThe key takeaway is that **the competition data uses an extended alphabet by design**, and collapsing it into ASCII can degrade both meaning and alignment. While this increases tokenization difficulty, preserving distinctions such as `s / ṣ / š` and `t / ṭ` is preferable to losing them.\n\nThe recommended strategy is therefore:\n\n* keep diacritics,\n* convert ASCII substitutes (e.g., `sz`) *into* diacritics,\n* and normalize everything toward the format used in the training and evaluation sets.\n\n---\n\n### Gaps, damage markers, and parallel alignment [update: 2/18/26]\n\nAnother recurring source of confusion concerns damaged text and gap markers in the data:\n\n* `x` represents a single broken sign,\n* sequences like `x x x x` or `...` represent a larger lacuna.\n\nFor modeling purposes we reduced all breaks to a single marker: `<gap>`\n\n* we removed the tag for `<big_gap>` from the train and test (and other transliterations). We also deduplicated instances multiple sequential gaps (e.g. `<gap> <gap`, `<gap>-<gap>`, `<gap> <gap>`, `<gap>. <gap>`, etc.\n\nIdeally, these gap markers would be **parallelized between transliteration and translation whenever possible**. However, this was not completely accomplished, as the translation we had access to did not pay strict attention to the gaps. This will be an aspect of the challenge in which a significant advantage will remain for those who have controlled for this data. If a large gap appears in one side but not the other, the model is forced to learn misalignment rather than translation.\n\nThis also applies to edge cases such as `<gap>` attached to a word (e.g., `<gap>-A-šùr`), which should be preserved rather than blindly removed.\n\n---\n\n### Why this matters for training and RL-based methods\n\nMany participants have observed that standard supervised fine-tuning produces reasonable loss curves, while reinforcement or preference-based methods fail to improve or show no reward signal. In nearly all cases examined so far, this traces back to **output non-conformance** caused by the issues above.\n\nIf the model is penalized for:\n\n* orthographic mismatches,\n* bracket artifacts,\n* inconsistent gap handling,\n* or malformed named entities,\n\nthen reward functions cannot reliably distinguish “closer” from “farther” outputs. Addressing normalization and alignment first makes rewards smoother and learning signals usable.\n\n---\n\n### Closing note\n\nNone of this is meant as criticism of existing approaches; these are difficult problems, and much of the complexity comes from the philological nature of the data itself. The hope in sharing this publicly is to reduce duplicated effort and make preprocessing choices more transparent across submissions.\n\nI will continue to share supplemental resources (including the onomasticon) as they are finalized, and I’m happy to discuss normalization or alignment strategies further if helpful to others.\n",
      "votes": 75
    },
    {
      "id": 3402145,
      "postDate": "2026-02-05T10:30:13.190Z",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Regarding normalization, could you clarify which of the following is true for the test set?</p>\n<ol>\n<li>Both transliterations and translations are already normalized (i.e., tokens like  and  are already present in both columns).</li>\n<li>Only the translations are normalized. Participants must implement their own logic to convert <code>x</code> or <code>...</code> into <code>&lt;gap&gt;</code> or <code>&lt;big_gap&gt;</code> for the test transliterations.</li>\n</ol>\n<p>If Case 2 is true, the final scores will be highly sensitive to specific preprocessing choices. This could turn the competition into a \"preprocessing lottery\" and encourage over-fitting to the Public LB rather than improving the actual MT model.</p>",
      "rawMarkdown": "@deeppast Regarding normalization, could you clarify which of the following is true for the test set?\n\n1. Both transliterations and translations are already normalized (i.e., tokens like <gap> and <big_gap> are already present in both columns).\n2. Only the translations are normalized. Participants must implement their own logic to convert `x` or `...` into `<gap>` or `<big_gap>` for the test transliterations.\n\nIf Case 2 is true, the final scores will be highly sensitive to specific preprocessing choices. This could turn the competition into a \"preprocessing lottery\" and encourage over-fitting to the Public LB rather than improving the actual MT model.",
      "votes": 12
    },
    {
      "id": 3417055,
      "postDate": "2026-03-04T13:29:24.590Z",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> , while going through your onomasticon and the train labels, i noticed that there are some differences in the naming convention.</p>\n<p>Example 1 — Different name entirely:</p>\n<p>Akkadian token: šu-{d}EN.LÍL ,\nOnomasticon canonical form: Šu-Enlil ,\nActual training label: Šu-Illil</p>\n<p>Example 2 — Geminate consonant:</p>\n<p>Akkadian token: šu-ku-tum ,\nOnomasticon canonical form: Šukatum ,\nActual training label: Šukkutum</p>\n<p>(The onomasticon drops the geminate kk that the training labels preserve.)</p>\n<p>Example 3 — Long vowel:</p>\n<p>Akkadian token: ṣí-lu-lu ,\nOnomasticon canonical form: Ṣilulu ,\nActual training label: Ṣilūlu</p>\n<p>(The onomasticon omits the macron on the ū that the training labels include.)</p>\n<p>these are some examples pointed out by claude. \nwhich convention do the test labels follow — the onomasticon's or the training data's?</p>",
      "rawMarkdown": "@deeppast , while going through your onomasticon and the train labels, i noticed that there are some differences in the naming convention.\n\nExample 1 — Different name entirely:\n\nAkkadian token: šu-{d}EN.LÍL ,\nOnomasticon canonical form: Šu-Enlil ,\nActual training label: Šu-Illil\n\nExample 2 — Geminate consonant:\n\nAkkadian token: šu-ku-tum ,\nOnomasticon canonical form: Šukatum ,\nActual training label: Šukkutum\n\n(The onomasticon drops the geminate kk that the training labels preserve.)\n\nExample 3 — Long vowel:\n\nAkkadian token: ṣí-lu-lu ,\nOnomasticon canonical form: Ṣilulu ,\nActual training label: Ṣilūlu\n\n(The onomasticon omits the macron on the ū that the training labels include.)\n\nthese are some examples pointed out by claude. \nwhich convention do the test labels follow — the onomasticon's or the training data's?\n",
      "votes": 5
    },
    {
      "id": 3417798,
      "postDate": "2026-03-06T08:58:57.160Z",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Clarification needed.</p>\n<p>I'm going through the glossaries and find that in Summerian the nasal \"ŋ\" is used somewhat often. This appears, e.g., in the determinative for wood <code>{ŋeš}</code>, which for this competition is written as <code>{geš}</code>. Can we safely assume <code>ŋ -&gt; g</code> substitution as valid?</p>\n<p>Another question is about treating Summerian words from the ORACC glossary, e.g., <code>{na₄}za-gin₃-ŋu₁₀</code>. Intuitively I would change that into: <code>NA₄.ZA.GÌN-gu₁₀</code>, i.e.:</p>\n<ol>\n<li>treat the base as a summerogram -&gt; uppercase, no <code>{}</code>, <code>.</code>-joined</li>\n<li>treat the rest as a <code>-</code>-joined suffix</li>\n</ol>\n<p>What would be the approach to best match the data in <code>train.csv</code> and other sources?</p>",
      "rawMarkdown": "@deeppast Clarification needed.\n\nI'm going through the glossaries and find that in Summerian the nasal \"ŋ\" is used somewhat often. This appears, e.g., in the determinative for wood `{ŋeš}`, which for this competition is written as `{geš}`. Can we safely assume `ŋ -> g` substitution as valid?\n\nAnother question is about treating Summerian words from the ORACC glossary, e.g., `{na₄}za-gin₃-ŋu₁₀`. Intuitively I would change that into: `NA₄.ZA.GÌN-gu₁₀`, i.e.:\n1. treat the base as a summerogram -> uppercase, no `{}`, `.`-joined\n2. treat the rest as a `-`-joined suffix\n\nWhat would be the approach to best match the data in `train.csv` and other sources?",
      "votes": 1
    },
    {
      "id": 3384015,
      "postDate": "2025-12-31T10:01:27.653Z",
      "content": "<p>I want to be sure on these cases:</p>\n<ul>\n<li><code>&lt;big_gap&gt; &lt;gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>&lt;gap&gt; &lt;big_gap&gt; &lt;gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>&lt;big_gap&gt; &lt;big_gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>&lt;gap&gt; &lt;gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>someword-&lt;gap&gt; &lt;gap&gt;-someword</code> → <code>someword-&lt;gap&gt; &lt;gap&gt;-someword</code> OR <code>someword-&lt;big_gap&gt;-someword</code> ?</li>\n<li><code>&lt;big_gap&gt; &lt;gap&gt;-A-šùr</code> → <code>&lt;big_gap&gt; &lt;gap&gt;-A-šùr</code></li>\n<li><code>xxxx-kam</code> (train.csv line 43)  → <code>&lt;big_gap&gt;-kam</code> OR <code>&lt;big_gap&gt; &lt;gap&gt;-kam</code> ?</li>\n</ul>\n<p>It would be good to take feedback on these from you.</p>\n<p>I think that having to figure out the normalization logic you are using prevents us from focusing on the actual goal of the competition. There is some processing happening in the background that also affects the original “ground truth” translations you have, so instead of focusing on the machine learning part we end up spending our effort on reverse-engineering your normalization logic. Can't you share the transliteration-translation preprocessors you're using?</p>",
      "rawMarkdown": "I want to be sure on these cases:\n\n- `<big_gap> <gap>` → `<big_gap>`\n- `<gap> <big_gap> <gap>` → `<big_gap>`\n- `<big_gap> <big_gap>` → `<big_gap>`\n- `<gap> <gap>` → `<big_gap>`\n- `someword-<gap> <gap>-someword` → `someword-<gap> <gap>-someword` OR `someword-<big_gap>-someword` ?\n- `<big_gap> <gap>-A-šùr` → `<big_gap> <gap>-A-šùr`\n- `xxxx-kam` (train.csv line 43)  → `<big_gap>-kam` OR `<big_gap> <gap>-kam` ?\n\nIt would be good to take feedback on these from you.\n\nI think that having to figure out the normalization logic you are using prevents us from focusing on the actual goal of the competition. There is some processing happening in the background that also affects the original “ground truth” translations you have, so instead of focusing on the machine learning part we end up spending our effort on reverse-engineering your normalization logic. Can't you share the transliteration-translation preprocessors you're using?",
      "votes": 8,
      "replies": [
        {
          "id": 3384059,
          "postDate": "2025-12-31T12:02:06.863Z",
          "content": "<p>I'm wondering how to interpret this section that is unclear to me about \"?\", \"!\", and \":\" for <strong>translations</strong> :</p>\n<p><a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions</a></p>\n<p><strong>Formatting Suggestions for Transliterations and Translations</strong>:</p>\n<p>Remove (modern scribal notations):</p>\n<ul>\n<li>! (<em>certain reading</em>)</li>\n<li>? (<em>questionable reading</em>)</li>\n<li>/ (line divider) - Now we now there is none of these  according to <a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </li>\n<li>: OR . (word divider)</li>\n</ul>\n<p>We've a lot of ?!: in training set translations. But do we have the same in test set? </p>\n<p>Should we apply the suggestion above =&gt; Remove all !?: from translations in training set. Or only if they are enclosed with parenthesis.</p>\n<p>Some make sense to me with \":\" usage for a rate and \"?\" for a question in translation. It does not look <em>questionable reading</em> for the question marks below.</p>\n<blockquote>\n  <p>ff9442fd-9e7d-449c-a2d6-0cc35921cd65:  Šalim-Aššur answered: \"And what if I have tin and from my goods remaining in his possession? Will he pay me at the rate 6:1 for my tin and 15 shekels per in silver?\"</p>\n</blockquote>\n<p>Here the \":\" make sense too. If I remove \":\" then I should remove \";\" too here.</p>\n<blockquote>\n  <p>00f0d841-eb7a-46f8-86fc-bf9fd7d52cbf: From Šu-Tammuzī, Elaya, Ennam-Aššur and Lamassī to Ennam-Aššur and Ali-ahum: In accordance with your missive we have hired a attorney for you; Abiya son of Bebe is our attorney. </p>\n</blockquote>\n<p>I would just remove things like:  <code>\"(?)\",  \"(!)\",  \"(fem. plur.)\", \"(fem.)\",  \"(plur.)\", \"(pl.)\", \"(sing.)\", \"(plural)\"</code></p>\n<p>Can't you share the transliteration and translation list of characters you're using / allowing?</p>",
          "rawMarkdown": "I'm wondering how to interpret this section that is unclear to me about \"?\", \"!\", and \":\" for **translations** :\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\n\n**Formatting Suggestions for Transliterations and Translations**:\n\nRemove (modern scribal notations):\n- ! (*certain reading*)\n- ? (*questionable reading*)\n- / (line divider) - Now we now there is none of these  according to @deeppast \n- : OR . (word divider)\n\nWe've a lot of ?!: in training set translations. But do we have the same in test set? \n\nShould we apply the suggestion above => Remove all !?: from translations in training set. Or only if they are enclosed with parenthesis.\n\nSome make sense to me with \":\" usage for a rate and \"?\" for a question in translation. It does not look *questionable reading* for the question marks below.\n>ff9442fd-9e7d-449c-a2d6-0cc35921cd65:  Šalim-Aššur answered: \"And what if I have tin and from my goods remaining in his possession? Will he pay me at the rate 6:1 for my tin and 15 shekels per in silver?\"\n\nHere the \":\" make sense too. If I remove \":\" then I should remove \";\" too here.\n>00f0d841-eb7a-46f8-86fc-bf9fd7d52cbf: From Šu-Tammuzī, Elaya, Ennam-Aššur and Lamassī to Ennam-Aššur and Ali-ahum: In accordance with your missive we have hired a attorney for you; Abiya son of Bebe is our attorney. \n\nI would just remove things like:  `\"(?)\",  \"(!)\",  \"(fem. plur.)\", \"(fem.)\",  \"(plur.)\", \"(pl.)\", \"(sing.)\", \"(plural)\"`\n    \n\nCan't you share the transliteration and translation list of characters you're using / allowing?",
          "votes": 1,
          "replies": [
            {
              "id": 3384200,
              "postDate": "2025-12-31T17:32:34.607Z",
              "content": "<p>Good questions, here's the character list for transliterations:</p>\n<p><strong>Transliterations Characters</strong></p>\n<ul>\n<li>-</li>\n<li>a</li>\n<li>A</li>\n<li>i</li>\n<li>I</li>\n<li>u</li>\n<li>U</li>\n<li>m</li>\n<li>M</li>\n<li>š</li>\n<li>Š</li>\n<li>n</li>\n<li>N</li>\n<li>b</li>\n<li>B</li>\n<li>r</li>\n<li>R</li>\n<li>t</li>\n<li>T</li>\n<li>l</li>\n<li>L</li>\n<li>k</li>\n<li>K</li>\n<li>G</li>\n<li>g</li>\n<li>í</li>\n<li>Í</li>\n<li>D</li>\n<li>d</li>\n<li>Ù</li>\n<li>ù</li>\n<li>á</li>\n<li>Á</li>\n<li>.</li>\n<li>ú</li>\n<li>Ú</li>\n<li>p</li>\n<li>P</li>\n<li>e</li>\n<li>E</li>\n<li>h</li>\n<li>H</li>\n<li>q</li>\n<li>Q</li>\n<li>1</li>\n<li>ṣ</li>\n<li>Ṣ</li>\n<li>é</li>\n<li>É</li>\n<li>&lt;</li>\n<li>&gt;</li>\n<li>à</li>\n<li>À</li>\n<li>4</li>\n<li>z</li>\n<li>Z</li>\n<li>s</li>\n<li>S</li>\n<li>ì</li>\n<li>Ì</li>\n<li>5</li>\n<li>_</li>\n<li>2</li>\n<li>0</li>\n<li>½</li>\n<li>w</li>\n<li>W</li>\n<li>3</li>\n<li>{</li>\n<li>}</li>\n<li>ṭ</li>\n<li>Ṭ</li>\n<li>6</li>\n<li>⅓</li>\n<li>8</li>\n<li>⅔</li>\n<li>7</li>\n<li>è</li>\n<li>È</li>\n<li>⅚</li>\n<li>9</li>\n<li>¼</li>\n<li>!</li>\n<li>+</li>\n<li>⅙</li>\n<li>ı</li>\n<li>…</li>\n<li>ş</li>\n<li>İ</li>\n<li>:</li>\n</ul>\n<p><strong>Translations Characters</strong></p>\n<ul>\n<li>'</li>\n<li>?</li>\n<li>e</li>\n<li>E</li>\n<li>a</li>\n<li>A</li>\n<li>i</li>\n<li>I</li>\n<li>t</li>\n<li>T</li>\n<li>n</li>\n<li>N</li>\n<li>s</li>\n<li>S</li>\n<li>o</li>\n<li>O</li>\n<li>r</li>\n<li>R</li>\n<li>l</li>\n<li>L</li>\n<li>h</li>\n<li>H</li>\n<li>u</li>\n<li>U</li>\n<li>m</li>\n<li>M</li>\n<li>d</li>\n<li>D</li>\n<li>F</li>\n<li>f</li>\n<li>š</li>\n<li>Š</li>\n<li>-</li>\n<li>p</li>\n<li>P</li>\n<li>w</li>\n<li>W</li>\n<li>b</li>\n<li>B</li>\n<li>g</li>\n<li>G</li>\n<li>y</li>\n<li>Y</li>\n<li>.</li>\n<li>K</li>\n<li>k</li>\n<li>,</li>\n<li>C</li>\n<li>c</li>\n<li>v</li>\n<li>ā</li>\n<li>1</li>\n<li>)</li>\n<li>(</li>\n<li>&lt;</li>\n<li>&gt;</li>\n<li>z</li>\n<li>Z</li>\n<li>_</li>\n<li>Q</li>\n<li>q</li>\n<li>2</li>\n<li>ī</li>\n<li>ṭ</li>\n<li>Ṭ</li>\n<li>0</li>\n<li>:</li>\n<li>3</li>\n<li>½</li>\n<li>5</li>\n<li>;</li>\n<li>x</li>\n<li>ē</li>\n<li>4</li>\n<li>ū</li>\n<li>6</li>\n<li>ṣ</li>\n<li>Ṣ</li>\n<li>⅓</li>\n<li>8</li>\n<li>’</li>\n<li>!</li>\n<li>7</li>\n<li>j</li>\n<li>J</li>\n<li>⅔</li>\n<li>“</li>\n<li>”</li>\n<li>9</li>\n<li>–</li>\n<li>⅚</li>\n<li>¼</li>\n<li>⅙</li>\n<li>\"</li>\n<li>‘</li>\n<li>ı</li>\n<li>—</li>\n<li>[</li>\n<li>]</li>\n<li>ğ</li>\n<li>â</li>\n<li>+</li>\n<li>à</li>\n<li>ş</li>\n</ul>",
              "rawMarkdown": "Good questions, here's the character list for transliterations:\n\n**Transliterations Characters**\n\t\t\t\n-\t-\n-\ta\n-\tA\n-\ti\n-\tI\n-\tu\n-\tU\n-\tm\n-\tM\n-\tš\n-\tŠ\n-\tn\n-\tN\n-\tb\n-\tB\n-\tr\n-\tR\n-\tt\n-\tT\n-\tl\n-\tL\n-\tk\n-\tK\n-\tG\n-\tg\n-\tí\n-\tÍ\n-\tD\n-\td\n-\tÙ\n-\tù\n-\tá\n-\tÁ\n-\t.\n-\tú\n-\tÚ\n-\tp\n-\tP\n-\te\n-\tE\n-\th\n-\tH\n-\tq\n-\tQ\n-\t1\n-\tṣ\n-\tṢ\n-\té\n-\tÉ\n-\t<\n-\t>\n-\tà\n-\tÀ\n-\t4\n-\tz\n-\tZ\n-\ts\n-\tS\n-\tì\n-\tÌ\n-\t5\n-\t_\n-\t2\n-\t0\n-\t½\n-\tw\n-\tW\n-\t3\n-\t{\n-\t}\n-\tṭ\n-\tṬ\n-\t6\n-\t⅓\n-\t8\n-\t⅔\n-\t7\n-\tè\n-\tÈ\n-\t⅚\n-\t9\n-\t¼\n-\t!\n-\t+\n-\t⅙\n-\tı\n-\t…\n-\tş\n-\tİ\n-\t:\n\n**Translations Characters**\n\n-\t'\n-\t?\n-\te\n-\tE\n-\ta\n-\tA\n-\ti\n-\tI\n-\tt\n-\tT\n-\tn\n-\tN\n-\ts\n-\tS\n-\to\n-\tO\n-\tr\n-\tR\n-\tl\n-\tL\n-\th\n-\tH\n-\tu\n-\tU\n-\tm\n-\tM\n-\td\n-\tD\n-\tF\n-\tf\n-\tš\n-\tŠ\n-\t-\n-\tp\n-\tP\n-\tw\n-\tW\n-\tb\n-\tB\n-\tg\n-\tG\n-\ty\n-\tY\n-\t.\n-\tK\n-\tk\n-\t,\n-\tC\n-\tc\n-\tv\n-\tā\n-\t1\n-\t)\n-\t(\n-\t<\n-\t>\n-\tz\n-\tZ\n-\t_\n-\tQ\n-\tq\n-\t2\n-\tī\n-\tṭ\n-\tṬ\n-\t0\n-\t:\n-\t3\n-\t½\n-\t5\n-\t;\n-\tx\n-\tē\n-\t4\n-\tū\n-\t6\n-\tṣ\n-\tṢ\n-\t⅓\n-\t8\n-\t’\n-\t!\n-\t7\n-\tj\n-\tJ\n-\t⅔\n-\t“\n-\t”\n-\t9\n-\t–\n-\t⅚\n-\t¼\n-\t⅙\n-\t\"\n-\t‘\n-\tı\n-\t—\n-\t[\n-\t]\n-\tğ\n-\tâ\n-\t+\n-\tà\n-\tş",
              "votes": 5
            },
            {
              "id": 3384450,
              "postDate": "2026-01-01T09:11:08.517Z",
              "content": "<p>did you actually publish the test set in plain chars ? 🧐</p>",
              "rawMarkdown": "did you actually publish the test set in plain chars ? 🧐",
              "votes": 2
            },
            {
              "id": 3384470,
              "postDate": "2026-01-01T10:04:46.220Z",
              "content": "<p>The frequency counts are the same for upper and lower case -- the counter looks like it's case-insensitive. Is it intentional?</p>",
              "rawMarkdown": "The frequency counts are the same for upper and lower case -- the counter looks like it's case-insensitive. Is it intentional?",
              "votes": 1
            },
            {
              "id": 3384479,
              "postDate": "2026-01-01T10:16:06.637Z",
              "content": "<p>I have the same confusion, but I’ve been submitting consistently to check if the LB score improves. I can now say it helps – after fixing a lot of formatting issues, my LB score has gone up to 32.3.Happy new year!</p>",
              "rawMarkdown": "I have the same confusion, but I’ve been submitting consistently to check if the LB score improves. I can now say it helps – after fixing a lot of formatting issues, my LB score has gone up to 32.3.Happy new year!",
              "votes": 1
            },
            {
              "id": 3384643,
              "postDate": "2026-01-01T16:41:59.637Z",
              "content": "<p>Thanks, it helps a lot <a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>\n<p>The only thing that looks like a bit weird is the frequency count:</p>\n<ul>\n<li>Top # 2 does not seem possible. ' =&gt; 1651390 ? =&gt; 181474. It means most of the text is a quote.</li>\n<li>Upper/lower case have the same count</li>\n</ul>\n<p>I believe your top #2 is something else and ? is not a real question mark but one or more characters that has been lost during counting or copy/paste or an intermediate sum/index.</p>",
              "rawMarkdown": "Thanks, it helps a lot @deeppast \n\nThe only thing that looks like a bit weird is the frequency count:\n- Top # 2 does not seem possible. ' => 1651390 ? => 181474. It means most of the text is a quote.\n- Upper/lower case have the same count\n\nI believe your top #2 is something else and ? is not a real question mark but one or more characters that has been lost during counting or copy/paste or an intermediate sum/index.",
              "votes": -1
            },
            {
              "id": 3384712,
              "postDate": "2026-01-01T18:57:25.390Z",
              "content": "<p>Thanks, yes those numbers were way off, the formula I used was wrong. I'll see if I can get the right counts, but at least the character set should help.</p>",
              "rawMarkdown": "Thanks, yes those numbers were way off, the formula I used was wrong. I'll see if I can get the right counts, but at least the character set should help.",
              "votes": 1
            },
            {
              "id": 3384718,
              "postDate": "2026-01-01T19:04:28.110Z",
              "content": "<p>these are all unicode characters (UTF-8)</p>",
              "rawMarkdown": "these are all unicode characters (UTF-8)",
              "votes": 3
            },
            {
              "id": 3387378,
              "postDate": "2026-01-06T20:38:12.507Z",
              "content": "<p>Does the translation character list apply to the target test translations, or it only applies to the train translations? Specifically, do you expect our generated translations to use unicode fractions (e.g., ¼) or normal texts (e.g., 1/4)?</p>",
              "rawMarkdown": "Does the translation character list apply to the target test translations, or it only applies to the train translations? Specifically, do you expect our generated translations to use unicode fractions (e.g., ¼) or normal texts (e.g., 1/4)?"
            },
            {
              "id": 3387879,
              "postDate": "2026-01-07T18:20:17.790Z",
              "content": "<p>I suppose this is the hidden test set, right?\nAre there missing characters because the data source for it is different, or it just happened to be this way because of the random split got it this way (I doubt it)?\nIf the data source is different, are there other such crucial characteristic differences that we should know of?\nIs public set a subset of the hidden test set?</p>",
              "rawMarkdown": "I suppose this is the hidden test set, right?\nAre there missing characters because the data source for it is different, or it just happened to be this way because of the random split got it this way (I doubt it)?\nIf the data source is different, are there other such crucial characteristic differences that we should know of?\nIs public set a subset of the hidden test set?"
            },
            {
              "id": 3387926,
              "postDate": "2026-01-07T20:11:19.960Z",
              "content": "<p>Yes, that's correct. The test set wasn't in an online database for starters. I realize these formatting issues are frustrating, but our long-term hope is that the resulting MT model will be able to handle these different formatting differences, and not just overfit on the test set. For this reason the 8k published texts were provided to give a larger sample of these different formats. </p>\n<p>It's worth mentioning here that the other databases out there (e.g. CDLI, ORACC, eBL, Archibab) also have different transliteration formats. It is perhaps too ambitious to hope that all of these formats can be harmonized at this time / for this challenge, but it is a long-term goal within my research.</p>",
              "rawMarkdown": "Yes, that's correct. The test set wasn't in an online database for starters. I realize these formatting issues are frustrating, but our long-term hope is that the resulting MT model will be able to handle these different formatting differences, and not just overfit on the test set. For this reason the 8k published texts were provided to give a larger sample of these different formats. \n\nIt's worth mentioning here that the other databases out there (e.g. CDLI, ORACC, eBL, Archibab) also have different transliteration formats. It is perhaps too ambitious to hope that all of these formats can be harmonized at this time / for this challenge, but it is a long-term goal within my research.",
              "votes": 3
            },
            {
              "id": 3387940,
              "postDate": "2026-01-07T20:37:59.560Z",
              "content": "<p>Yeah it is annoying but I also understand that there is no hard standardized format..\nThank you a lot for your response!</p>",
              "rawMarkdown": "Yeah it is annoying but I also understand that there is no hard standardized format..\nThank you a lot for your response!"
            },
            {
              "id": 3388619,
              "postDate": "2026-01-09T08:55:12.310Z",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> In the list of allowed characters above, in the transliterations part, there is a symbol \"…\" at the very end. Shouldn't this be converted to ?</p>",
              "rawMarkdown": "@deeppast In the list of allowed characters above, in the transliterations part, there is a symbol \"…\" at the very end. Shouldn't this be converted to <big_gap>?",
              "votes": 1
            },
            {
              "id": 3394612,
              "postDate": "2026-01-21T11:02:02.560Z",
              "content": "<h4><strong>Transliterations Characters</strong></h4>\n<p><code>!+-.0123456789:&lt;&gt;ABDEGHIKLMNPQRSTUWZ_abdeghiklmnpqrstuwz{}¼½ÀÁÈÉÌÍÙÚàáèéìíùúİışŠšṢṣṬṭ…⅓⅔⅙⅚</code></p>\n<h4><strong>Translations Characters</strong></h4>\n<p><code>!\"\\'()+,-.0123456789:;&lt;&gt;?ABCDEFGHIJKLMNOPQRSTUWYZ[]_abcdefghijklmnopqrstuvwxyz¼½àâāēğīışŠšūṢṣṬṭ–—‘’“”⅓⅔⅙⅚</code></p>\n<p>(<strong>Characters sorted for brevity and a better view</strong>)</p>\n<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>\n<p>I wanted to come back to this as new question appeared regarding the character set.</p>\n<p>Several of those characters do not appear in the <code>train.csv</code> or are extremely rare and I'd like to know more on how to handle those cases, as it appears the normalization steps needed may affect scoring.</p>\n<h2>Unknown special characters (<code>şİıâ</code>)</h2>\n<p>Conversing with Gemini gave me the following clues:</p>\n<ol>\n<li>ş = ṣ (used in older publications)</li>\n<li>İ = I (Uppercase <strong>i</strong> written with dot - typical for Turkish sources)</li>\n<li>ı = i (Lowercase <strong>i</strong> but dotless - typical for Turkish sources)</li>\n</ol>\n<h2>Not normalized characters (<code>…:!{}</code>)</h2>\n<p>As per <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\" target=\"_blank\">Formatting suggestions for Transliterations and Translations</a> we should treat these as artifacts from the normalization process:</p>\n<ol>\n<li><code>…</code> - replace with <code>&lt;big_gap&gt;</code></li>\n<li><code>:</code> - replace with a dot (<strong>.</strong>)</li>\n<li><code>!</code> - remove (scribal note for certain reading of a sign)</li>\n<li><code>{}</code> - replaces <code>()</code> determinatives (predominant use in <code>train.csv</code> and <code>OA_lexicon.eBL.csv</code>)</li>\n</ol>\n<h2>Same fractions (<code>¼½⅓⅔⅙⅚</code>) in transliterations/translations</h2>\n<p>This strongly suggests that at least part of the translations do not recalculate the fractions, i.e., <code>1.83333 ma-na 3.5 GÍN</code> becomes <code>1⅚ ma-na 3½ GÍN</code> in test but becomes <code>1⅚ mina 3½ shekels</code> while in <code>train.csv</code> it is recalculated to <code>1 mina 56.5 shekels</code> (<code>1 mina 56½ shekels</code> with Unicode fraction notation).</p>\n<p>Should we retain the fractions as they are in the transliterations or should we recalculate them?</p>\n<h2>Translation characters <code>àâğ\\'()‘’“”</code></h2>\n<ol>\n<li>à = ā ?</li>\n<li>â = ā ?</li>\n<li>ğ = g ?</li>\n<li>\\ - a line divider (remove), alternative reading?</li>\n<li>' - glottal stop?</li>\n<li>() - additional information, e.g. <code>(fem. plur.)</code></li>\n<li>‘’ - should be standardized to <code>''</code>?</li>\n<li>“” - should be standardized to <code>\"\"</code>?</li>\n</ol>",
              "rawMarkdown": "#### **Transliterations Characters**\n`!+-.0123456789:<>ABDEGHIKLMNPQRSTUWZ_abdeghiklmnpqrstuwz{}¼½ÀÁÈÉÌÍÙÚàáèéìíùúİışŠšṢṣṬṭ…⅓⅔⅙⅚`\n\n\n#### **Translations Characters**\n`!\"\\'()+,-.0123456789:;<>?ABCDEFGHIJKLMNOPQRSTUWYZ[]_abcdefghijklmnopqrstuvwxyz¼½àâāēğīışŠšūṢṣṬṭ–—‘’“”⅓⅔⅙⅚`\n\n(**Characters sorted for brevity and a better view**)\n\n@deeppast \n\nI wanted to come back to this as new question appeared regarding the character set.\n\nSeveral of those characters do not appear in the `train.csv` or are extremely rare and I'd like to know more on how to handle those cases, as it appears the normalization steps needed may affect scoring.\n\n## Unknown special characters (`şİıâ`)\nConversing with Gemini gave me the following clues:\n1. ş = ṣ (used in older publications)\n2. İ = I (Uppercase **i** written with dot - typical for Turkish sources)\n3. ı = i (Lowercase **i** but dotless - typical for Turkish sources)\n\n## Not normalized characters (`…:!{}`)\nAs per [Formatting suggestions for Transliterations and Translations](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions) we should treat these as artifacts from the normalization process:\n1. `…` - replace with `<big_gap>`\n2. `:` - replace with a dot (**.**)\n3. `!` - remove (scribal note for certain reading of a sign)\n4. `{}` - replaces `()` determinatives (predominant use in `train.csv` and `OA_lexicon.eBL.csv`)\n\n## Same fractions (`¼½⅓⅔⅙⅚`) in transliterations/translations\nThis strongly suggests that at least part of the translations do not recalculate the fractions, i.e., `1.83333 ma-na 3.5 GÍN` becomes `1⅚ ma-na 3½ GÍN` in test but becomes `1⅚ mina 3½ shekels` while in `train.csv` it is recalculated to `1 mina 56.5 shekels` (`1 mina 56½ shekels` with Unicode fraction notation).\n\nShould we retain the fractions as they are in the transliterations or should we recalculate them?\n\n## Translation characters `àâğ\\'()‘’“”`\n1. à = ā ?\n2. â = ā ?\n3. ğ = g ?\n4. \\ - a line divider (remove), alternative reading?\n5. ' - glottal stop?\n6. () - additional information, e.g. `(fem. plur.)`\n7. ‘’ - should be standardized to `''`?\n8. “” - should be standardized to `\"\"`?",
              "votes": 5
            },
            {
              "id": 3401159,
              "postDate": "2026-02-03T03:41:45.523Z",
              "content": "<p>1.'[' in translation need to be removed?  e.g:[I paid … … … … … …]\n2.is there &lt; gap &gt; or &lt; big-gap &gt; in the translation column of pulibc and private test sets?</p>",
              "rawMarkdown": "1.'[' in translation need to be removed?  e.g:[I paid ... ... ... ... ... ...]\n2.is there < gap > or < big-gap > in the translation column of pulibc and private test sets?"
            }
          ]
        },
        {
          "id": 3384067,
          "postDate": "2025-12-31T12:10:15.940Z",
          "content": "<p>From my LB results, not merging <code>&lt;gap&gt;</code> <code>&lt;big_gap&gt;</code>performs a bit better.</p>",
          "rawMarkdown": "From my LB results, not merging `<gap>` `<big_gap>`performs a bit better.",
          "votes": 3,
          "replies": [
            {
              "id": 3384210,
              "postDate": "2025-12-31T18:02:05.483Z",
              "content": "<p>Yes, sorry for the confusion. Here's what I have found in terms of co-occurrances of gaps with frequency counts.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F28748094%2F10acb49069b39d449b16628d790292b8%2FGaps_no_EXIF.png?generation=1767204595136708&amp;alt=media\" alt=\"\">\nP.S. This is not an exhaustive list, but a representative one.</p>",
              "rawMarkdown": "Yes, sorry for the confusion. Here's what I have found in terms of co-occurrances of gaps with frequency counts.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F28748094%2F10acb49069b39d449b16628d790292b8%2FGaps_no_EXIF.png?generation=1767204595136708&alt=media)\nP.S. This is not an exhaustive list, but a representative one.",
              "votes": 9
            },
            {
              "id": 3384436,
              "postDate": "2026-01-01T08:49:05.640Z",
              "content": "<p>You said </p>\n<p><code>In practice, collapsing combinations such as &lt;gap&gt; &lt;big_gap&gt; or &lt;big_gap&gt; &lt;gap&gt; into a single &lt;big_gap&gt; on both sides improves sentence-level alignment and reduces noise.</code></p>\n<p>but the list you shared includes non-merged combinations.</p>",
              "rawMarkdown": "You said \n\n`In practice, collapsing combinations such as <gap> <big_gap> or <big_gap> <gap> into a single <big_gap> on both sides improves sentence-level alignment and reduces noise.`\n\nbut the list you shared includes non-merged combinations.",
              "votes": 3
            },
            {
              "id": 3384476,
              "postDate": "2026-01-01T10:13:20.043Z",
              "content": "<p>While analysing the sentence mismatch <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664177#3384304\" target=\"_blank\">discussed in a parallel thread</a> I found that in some transliterations <code>(2 broken lines)</code> would be space-split and treated as <code>x x x</code> causing a shift.</p>\n<p>Targetting <code>&lt;gap&gt;/&lt;big_gap&gt;</code> combinations without the connected ones was a problem of its own and I ended up with this capture pattern from Perplexity:</p>\n<pre><code>(?&lt;!\\S)&lt;(?:gap|big_gap)&gt;(?!\\s?-|\\S)(?:\\s&lt;(?:gap|big_gap)&gt;(?!\\s?-|\\S))+\n</code></pre>\n<p>It worked for me while working on <code>Sentences_Oare_FirstWord_LinNum.csv</code>. I haven't tried it on corrected <code>train.csv</code> yet.</p>\n<p>edit: the capture pattern was modified (should have ended with <code>+</code> instead of <code>*</code>)</p>",
              "rawMarkdown": "While analysing the sentence mismatch [discussed in a parallel thread](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664177#3384304) I found that in some transliterations `(2 broken lines)` would be space-split and treated as `x x x` causing a shift.\n\nTargetting `<gap>/<big_gap>` combinations without the connected ones was a problem of its own and I ended up with this capture pattern from Perplexity:\n```python\n(?<!\\S)<(?:gap|big_gap)>(?!\\s?-|\\S)(?:\\s<(?:gap|big_gap)>(?!\\s?-|\\S))+\n```\nIt worked for me while working on `Sentences_Oare_FirstWord_LinNum.csv`. I haven't tried it on corrected `train.csv` yet.\n\nedit: the capture pattern was modified (should have ended with `+` instead of `*`)"
            },
            {
              "id": 3393353,
              "postDate": "2026-01-18T21:18:48.023Z",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> so is your statement regarding collapsing gaps (quoted below, from the topic post) still valid?</p>\n<blockquote>\n  <p>multiple consecutive x tokens should be collapsed into a single &lt;big_gap&gt;.</p>\n</blockquote>",
              "rawMarkdown": "@deeppast so is your statement regarding collapsing gaps (quoted below, from the topic post) still valid?\n>multiple consecutive x tokens should be collapsed into a single \\<big_gap>."
            },
            {
              "id": 3393362,
              "postDate": "2026-01-18T21:29:47.253Z",
              "content": "<blockquote>\n  <p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> so is your statement regarding collapsing gaps (quoted below, from the topic post) still valid?</p>\n  <blockquote>\n    <p>multiple consecutive x tokens should be collapsed into a single .</p>\n  </blockquote>\n</blockquote>\n<p>Anything in angle brackets becomes invisible when formatted. Better wrap it in single backticks, e.g., <code>&lt;big_gap&gt;</code></p>",
              "rawMarkdown": "> @deeppast so is your statement regarding collapsing gaps (quoted below, from the topic post) still valid?\n> >multiple consecutive x tokens should be collapsed into a single <big_gap>.\n\nAnything in angle brackets becomes invisible when formatted. Better wrap it in single backticks, e.g., `<big_gap>`"
            },
            {
              "id": 3402088,
              "postDate": "2026-02-05T07:54:20.480Z",
              "content": "<blockquote>\n  <p>You said </p>\n  <p><code>In practice, collapsing combinations such as &lt;gap&gt; &lt;big_gap&gt; or &lt;big_gap&gt; &lt;gap&gt; into a single &lt;big_gap&gt; on both sides improves sentence-level alignment and reduces noise.</code></p>\n  <p>but the list you shared includes non-merged combinations.</p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> could you please provide a clarification regarding this comment? </p>",
              "rawMarkdown": "> You said \n> \n> `In practice, collapsing combinations such as <gap> <big_gap> or <big_gap> <gap> into a single <big_gap> on both sides improves sentence-level alignment and reduces noise.`\n> \n> but the list you shared includes non-merged combinations.\n \n@deeppast could you please provide a clarification regarding this comment? \n"
            }
          ]
        },
        {
          "id": 3385136,
          "postDate": "2026-01-02T15:52:14.287Z",
          "content": "<p>As for the two questions you posed here:</p>\n<ul>\n<li>Both of these can be found: someword-&lt; gap &gt; &lt; gap &gt;-someword AND someword-&lt; big_gap &gt;-someword</li>\n<li>xxxx-kam → &lt; big_gap &gt;-kam</li>\n<li>x-kam → &lt; gap &gt;-kam</li>\n</ul>",
          "rawMarkdown": "As for the two questions you posed here:\n- Both of these can be found: someword-< gap > < gap >-someword AND someword-< big_gap >-someword\n- xxxx-kam → < big_gap >-kam\n- x-kam → < gap >-kam",
          "votes": 2,
          "replies": [
            {
              "id": 3385223,
              "postDate": "2026-01-02T18:33:18.763Z",
              "content": "<p>The <code>&lt;gap&gt;</code> and <code>&lt;big_gap&gt;</code> are missing. Is it the markdown formatting?</p>\n<p>edit: it is. It shows up after clicking <code>Quote</code>.</p>",
              "rawMarkdown": "The `<gap>` and `<big_gap>` are missing. Is it the markdown formatting?\n\nedit: it is. It shows up after clicking `Quote`."
            },
            {
              "id": 3387938,
              "postDate": "2026-01-07T20:34:55.447Z",
              "content": "<p>yes, sorry, fixed it</p>",
              "rawMarkdown": "yes, sorry, fixed it",
              "votes": 1
            }
          ]
        },
        {
          "id": 3386031,
          "postDate": "2026-01-04T12:35:15.483Z",
          "content": "<p><a href=\"https://www.kaggle.com/nlztrk\" target=\"_blank\">@nlztrk</a> Did it work for you to collapse gaps? CV is stable but LB is worse here with simple gap/big_gap merge:</p>\n<pre><code>def collapse_gaps(text):\n    \"\"\"\n    Collapse consecutive &lt;gap&gt;/&lt;big_gap&gt; into a single &lt;big_gap&gt;.\n    Keep single isolated &lt;gap&gt; as-is.\n    \"\"\"\n    tokens = text.split()\n\n    collapsed = []\n    i = 0\n    n = len(tokens)\n\n    while i &lt; n:\n        if tokens[i] in {\"&lt;gap&gt;\", \"&lt;big_gap&gt;\"}:\n            j = i\n            while j &lt; n and tokens[j] in {\"&lt;gap&gt;\", \"&lt;big_gap&gt;\"}:\n                j += 1\n\n            run_len = j - i\n\n            # If more than one gap marker =&gt; big gap\n            if run_len &gt;= 2:\n                collapsed.append(\"&lt;big_gap&gt;\")\n            else:\n                collapsed.append(tokens[i])\n\n            i = j\n        else:\n            collapsed.append(tokens[i])\n            i += 1\n\n    return \" \".join(collapsed) \n</code></pre>",
          "rawMarkdown": "@nlztrk Did it work for you to collapse gaps? CV is stable but LB is worse here with simple gap/big_gap merge:\n\n```python\ndef collapse_gaps(text):\n    \"\"\"\n    Collapse consecutive <gap>/<big_gap> into a single <big_gap>.\n    Keep single isolated <gap> as-is.\n    \"\"\"\n    tokens = text.split()\n\n    collapsed = []\n    i = 0\n    n = len(tokens)\n\n    while i < n:\n        if tokens[i] in {\"<gap>\", \"<big_gap>\"}:\n            j = i\n            while j < n and tokens[j] in {\"<gap>\", \"<big_gap>\"}:\n                j += 1\n\n            run_len = j - i\n\n            # If more than one gap marker => big gap\n            if run_len >= 2:\n                collapsed.append(\"<big_gap>\")\n            else:\n                collapsed.append(tokens[i])\n\n            i = j\n        else:\n            collapsed.append(tokens[i])\n            i += 1\n\n    return \" \".join(collapsed) \n```",
          "votes": 1,
          "replies": [
            {
              "id": 3386115,
              "postDate": "2026-01-04T16:30:35.077Z",
              "content": "<p>Im collapsing all big gaps into one big gap and all adjacent big gaps and gaps into a big gap. For now it works as the best. But the examples of the organizator violates this logic, we still don't know theirs.</p>",
              "rawMarkdown": "Im collapsing all big gaps into one big gap and all adjacent big gaps and gaps into a big gap. For now it works as the best. But the examples of the organizator violates this logic, we still don't know theirs.",
              "votes": 3
            }
          ]
        }
      ]
    },
    {
      "id": 3385720,
      "postDate": "2026-01-03T18:55:02.610Z",
      "content": "<p>This is my main worry about participating in this contest. I've been working on cuneiform LLMS for the past two years, and a good text pipeline has to standardize the presentation from multiple script types (on my end I'm also working with Elamite, Sumerian, and Hittite). Getting it to align with ASCII helps learnability tremendously if one wants to leverage pre-trained models like T5 or NLLB. But I don't know how we prepare the unknown text - can we write an input pipeline (and prompt structure) to take in the unknown data with a submission?  </p>",
      "rawMarkdown": "This is my main worry about participating in this contest. I've been working on cuneiform LLMS for the past two years, and a good text pipeline has to standardize the presentation from multiple script types (on my end I'm also working with Elamite, Sumerian, and Hittite). Getting it to align with ASCII helps learnability tremendously if one wants to leverage pre-trained models like T5 or NLLB. But I don't know how we prepare the unknown text - can we write an input pipeline (and prompt structure) to take in the unknown data with a submission?  ",
      "votes": 4
    },
    {
      "id": 3385029,
      "postDate": "2026-01-02T12:56:09.057Z",
      "content": "<p>For the subscripts used in the determinatives, do they also need to be transformed to either diacritics or plain numbers?</p>\n<p>e.g. should it be {tug₂} or {tug2} ? </p>",
      "rawMarkdown": "For the subscripts used in the determinatives, do they also need to be transformed to either diacritics or plain numbers?\n\ne.g. should it be {tug₂} or {tug2} ? ",
      "votes": 2,
      "replies": [
        {
          "id": 3385102,
          "postDate": "2026-01-02T14:50:33.080Z",
          "content": "<p>On second thought, since the determinatives appear only in the transliteration text, I can pick either scheme. Then, during inference, I can detect the determinatives in the test set input (should be fairly easy) and transform them according to the scheme I pick.  </p>",
          "rawMarkdown": "On second thought, since the determinatives appear only in the transliteration text, I can pick either scheme. Then, during inference, I can detect the determinatives in the test set input (should be fairly easy) and transform them according to the scheme I pick.  ",
          "votes": 1
        },
        {
          "id": 3385134,
          "postDate": "2026-01-02T15:48:51.327Z",
          "content": "<p>The expected format for {tug₂} or {tug2} is actually {túg}</p>\n<p>For background, in ASCII ATF (CDLI) they don't use diacritics for ú = u2, and ù = u3, and so on. Further, in Oracc these numbers become subscripts: ú = u₂, etc. That second comment was intended to indicate that for the readings with 2 or 3 this dataset used diacritics instead of numbers or subscripted numbers (i.e. á à é è í ì ú ù; NOT a2, a3, etc.; NOT a₂, etc.). However, sign values with 4, 5, 6, …, 10, 12, 13, etc. are found in the dataset (e.g., DU10). So aside from sign values with these readings á à é è í ì ú ù, this dataset uses integer numbers, not subscripted numbers.</p>",
          "rawMarkdown": "The expected format for {tug₂} or {tug2} is actually {túg}\n\nFor background, in ASCII ATF (CDLI) they don't use diacritics for ú = u2, and ù = u3, and so on. Further, in Oracc these numbers become subscripts: ú = u₂, etc. That second comment was intended to indicate that for the readings with 2 or 3 this dataset used diacritics instead of numbers or subscripted numbers (i.e. á à é è í ì ú ù; NOT a2, a3, etc.; NOT a₂, etc.). However, sign values with 4, 5, 6, …, 10, 12, 13, etc. are found in the dataset (e.g., DU10). So aside from sign values with these readings á à é è í ì ú ù, this dataset uses integer numbers, not subscripted numbers.",
          "votes": 6,
          "replies": [
            {
              "id": 3385744,
              "postDate": "2026-01-03T20:35:02.630Z",
              "content": "<p>{túg} or {TÚG}? there is no lowercased {túg} in any guide you've shared. Please just give a reproducible normalization script or a very clear and strict conversion table. Most of your statements contradict each other.</p>",
              "rawMarkdown": "{túg} or {TÚG}? there is no lowercased {túg} in any guide you've shared. Please just give a reproducible normalization script or a very clear and strict conversion table. Most of your statements contradict each other.",
              "votes": 4
            },
            {
              "id": 3385749,
              "postDate": "2026-01-03T20:59:28.867Z",
              "content": "<p>Uppercase.</p>",
              "rawMarkdown": "Uppercase.",
              "votes": 1
            },
            {
              "id": 3385751,
              "postDate": "2026-01-03T21:10:14.390Z",
              "rawMarkdown": "",
              "votes": 1,
              "isDeleted": true
            },
            {
              "id": 3385756,
              "postDate": "2026-01-03T21:33:34.757Z",
              "content": "<p>Check the overview page and the data provided</p>",
              "rawMarkdown": "Check the overview page and the data provided",
              "votes": 1
            },
            {
              "id": 3387872,
              "postDate": "2026-01-07T18:12:17.423Z",
              "content": "<p>TÚG, without brackets (in the test data), but {tug2} in the CDLI data and {tug₂} in the Oracc data, in case you choose to use those.</p>",
              "rawMarkdown": "TÚG, without brackets (in the test data), but {tug2} in the CDLI data and {tug₂} in the Oracc data, in case you choose to use those.",
              "votes": 1
            },
            {
              "id": 3387880,
              "postDate": "2026-01-07T18:23:38.160Z",
              "content": "<p>In the overview tab you are saying:</p>\n<p>{tug₂} = (TÚG) preceding textiles and other woven objects</p>\n<p>TÚG is being used with parentheses there but you are now saying <strong>without brackets</strong>. I think you need to clarify the <strong>exact</strong> form for that list. What should we expect for these in the test set:</p>\n<ul>\n<li>{d} = dingir ‘god, deity’ — d preceding non-human divine actors</li>\n<li>{mul} = ‘stars’ — MUL preceding astronomical bodies and constellations</li>\n<li>{ki} = ‘earth’ — KI following a geographical place name or location</li>\n<li>{lu₂} = LÚ preceding people and professions</li>\n<li>{e₂} = {É} preceding buildings and institutions, such as temples and palaces</li>\n<li>{uru} = (URU) preceding names of settlements, such as villages, towns and cities</li>\n<li>{kur} = (KUR) preceding lands and territories as well as mountains</li>\n<li>{mi} = munus (f) preceding feminine personal names</li>\n<li>{m} = (1 or m) preceding masculine personal names</li>\n<li>{geš} / {ĝeš) = (GIŠ) preceding trees and things made of wood</li>\n<li>{tug₂} = (TÚG) preceding textiles and other woven objects</li>\n<li>{dub} = (DUB) preceding clay tablets, and by extension, documents and legal records</li>\n<li>{id₂} = (ÍD) (a ligature of A and ENGUR, transliterated: A.ENGUR) preceding names of canals or rivers or when written on its own referring to the divine river</li>\n<li>{mušen} = (MUŠEN) preceding birds</li>\n<li>{na₄} = (na4) preceding stone</li>\n<li>{kuš} = (kuš) preceding (animal) skin, fleece, hides</li>\n<li>{u₂} = (Ú) preceding plants</li>\n</ul>",
              "rawMarkdown": "In the overview tab you are saying:\n\n{tug₂} = (TÚG) preceding textiles and other woven objects\n\nTÚG is being used with parentheses there but you are now saying **without brackets**. I think you need to clarify the **exact** form for that list. What should we expect for these in the test set:\n\n- {d} = dingir ‘god, deity’ — d preceding non-human divine actors\n- {mul} = ‘stars’ — MUL preceding astronomical bodies and constellations\n- {ki} = ‘earth’ — KI following a geographical place name or location\n- {lu₂} = LÚ preceding people and professions\n- {e₂} = {É} preceding buildings and institutions, such as temples and palaces\n- {uru} = (URU) preceding names of settlements, such as villages, towns and cities\n- {kur} = (KUR) preceding lands and territories as well as mountains\n- {mi} = munus (f) preceding feminine personal names\n- {m} = (1 or m) preceding masculine personal names\n- {geš} / {ĝeš) = (GIŠ) preceding trees and things made of wood\n- {tug₂} = (TÚG) preceding textiles and other woven objects\n- {dub} = (DUB) preceding clay tablets, and by extension, documents and legal records\n- {id₂} = (ÍD) (a ligature of A and ENGUR, transliterated: A.ENGUR) preceding names of canals or rivers or when written on its own referring to the divine river\n- {mušen} = (MUŠEN) preceding birds\n- {na₄} = (na4) preceding stone\n- {kuš} = (kuš) preceding (animal) skin, fleece, hides\n- {u₂} = (Ú) preceding plants\n",
              "votes": 5
            }
          ]
        }
      ]
    },
    {
      "id": 3384225,
      "postDate": "2025-12-31T18:28:13.033Z",
      "content": "<p>Deep Past Challenge - Translate Akkadian to English</p>",
      "rawMarkdown": "Deep Past Challenge - Translate Akkadian to English",
      "votes": -10
    },
    {
      "id": 3408964,
      "postDate": "2026-02-21T18:30:40.723Z",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Hi, I was wondering where I can get the onomasticon (a curated list of names and attested spellings). Could you help me with that please?</p>",
      "rawMarkdown": "@deeppast Hi, I was wondering where I can get the onomasticon (a curated list of names and attested spellings). Could you help me with that please?\n"
    },
    {
      "id": 3383782,
      "postDate": "2025-12-30T20:27:28.243Z",
      "content": "<p>do not convert diacritics to ASCII (information loss!)\ndo not remove subscripts (semantic meaning!)\nEvaluation data contains diacritics - be compatible with this</p>\n<p>am i right?</p>",
      "rawMarkdown": "do not convert diacritics to ASCII (information loss!)\ndo not remove subscripts (semantic meaning!)\nEvaluation data contains diacritics - be compatible with this\n\nam i right?",
      "replies": [
        {
          "id": 3383787,
          "postDate": "2025-12-30T20:40:36.527Z",
          "content": "<p>Close, but let me be clear:</p>\n<ul>\n<li>(1) with the exception of ḫ, the test data has diacritics (š ṣ ṭ ā á à ī í ì ū ú ù - and upper case versions). If you're obtaining data from CDLI or ORACC, you will need to convert their formatted transliterations to this diacritic style (conversions are listed in the overview) — note ā ī ū are in translations, not transliteration text.</li>\n<li>(2) there are no subscripted numbers in the test data, so expect A-šùr-DU10 for example.</li>\n<li>(3) keep diacritics for transliterations, but normalize them for translations of named entities, for example: A-šùr-DU10 (transliteration) --&gt; Aššur-ṭāb (translation)</li>\n</ul>",
          "rawMarkdown": "Close, but let me be clear:\n\n- (1) with the exception of ḫ, the test data has diacritics (š ṣ ṭ ā á à ī í ì ū ú ù - and upper case versions). If you're obtaining data from CDLI or ORACC, you will need to convert their formatted transliterations to this diacritic style (conversions are listed in the overview) — note ā ī ū are in translations, not transliteration text.\n- (2) there are no subscripted numbers in the test data, so expect A-šùr-DU10 for example.\n- (3) keep diacritics for transliterations, but normalize them for translations of named entities, for example: A-šùr-DU10 (transliteration) --> Aššur-ṭāb (translation)",
          "votes": 4,
          "replies": [
            {
              "id": 3383830,
              "postDate": "2025-12-30T22:29:48.533Z",
              "rawMarkdown": "",
              "isDeleted": true
            },
            {
              "id": 3383953,
              "postDate": "2025-12-31T07:19:41.317Z",
              "content": "<p>Hello, why is your CHRF metric so high? What's the difference between it and the CHRF from the public notebook? Public notebook link: <a href=\"https://www.kaggle.com/code/takamichitoda/dpc-starter-train\" target=\"_blank\">https://www.kaggle.com/code/takamichitoda/dpc-starter-train</a></p>",
              "rawMarkdown": "Hello, why is your CHRF metric so high? What's the difference between it and the CHRF from the public notebook? Public notebook link: https://www.kaggle.com/code/takamichitoda/dpc-starter-train"
            },
            {
              "id": 3383967,
              "postDate": "2025-12-31T07:58:10.150Z",
              "content": "<p>My guess is that if the sentences are the same, then you most likely did sentence alignment.</p>",
              "rawMarkdown": "My guess is that if the sentences are the same, then you most likely did sentence alignment."
            },
            {
              "id": 3384009,
              "postDate": "2025-12-31T09:50:19.640Z",
              "content": "<p>This public notebook does not use competition metrics.</p>\n<p>Try: <a href=\"https://www.kaggle.com/code/metric/dpi-bleu-chrf\" target=\"_blank\">https://www.kaggle.com/code/metric/dpi-bleu-chrf</a></p>",
              "rawMarkdown": "This public notebook does not use competition metrics.\n\nTry: https://www.kaggle.com/code/metric/dpi-bleu-chrf"
            },
            {
              "id": 3384017,
              "postDate": "2025-12-31T10:04:33.200Z",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>\n<blockquote>\n  <p>(2) there are no subscripted numbers in the test data, so expect A-šùr-DU10 for example.</p>\n</blockquote>\n<p>You said something a bit different in another post:</p>\n<blockquote>\n  <p>Subscript digits are converted to normal digits, except for 2 and 3, which are represented by diacritic marks over the vowels (see overview for more details)</p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664518#3382901\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664518#3382901</a></p>\n<p>Do I miss something?</p>",
              "rawMarkdown": "@deeppast \n>(2) there are no subscripted numbers in the test data, so expect A-šùr-DU10 for example.\n\nYou said something a bit different in another post:\n>Subscript digits are converted to normal digits, except for 2 and 3, which are represented by diacritic marks over the vowels (see overview for more details)\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664518#3382901\n\nDo I miss something?",
              "votes": 4
            },
            {
              "id": 3384671,
              "postDate": "2026-01-01T17:10:00.527Z",
              "content": "<p>Answer to myself: Now we know that first assertion it True and second is False.</p>",
              "rawMarkdown": "Answer to myself: Now we know that first assertion it True and second is False.",
              "votes": 1
            },
            {
              "id": 3384721,
              "postDate": "2026-01-01T19:17:16.883Z",
              "content": "<p>Yes, right, the ack of specificity in that second statement is problematic. For background, in ASCII ATF (CDLI) they don't use diacritics for ú = u2, and ù = u3, and so on. Further, in Oracc these numbers become subscripts: ú = u₂, etc.\nThat second comment was intended to indicate that for the readings with 2 or 3 this dataset used diacritics instead of numbers or subscripted numbers (i.e. á à é è í ì ú ù; NOT a2, a3, etc.; NOT a₂, etc.).\nHowever, sign values with 4, 5, 6, …, 10, 12, 13, etc. are found in the dataset (e.g., DU10). \nSo aside from sign values with these readings á à é è í ì ú ù, this dataset uses integer numbers, not subscripted numbers.</p>",
              "rawMarkdown": "Yes, right, the ack of specificity in that second statement is problematic. For background, in ASCII ATF (CDLI) they don't use diacritics for ú = u2, and ù = u3, and so on. Further, in Oracc these numbers become subscripts: ú = u₂, etc.\nThat second comment was intended to indicate that for the readings with 2 or 3 this dataset used diacritics instead of numbers or subscripted numbers (i.e. á à é è í ì ú ù; NOT a2, a3, etc.; NOT a₂, etc.).\nHowever, sign values with 4, 5, 6, ..., 10, 12, 13, etc. are found in the dataset (e.g., DU10). \nSo aside from sign values with these readings á à é è í ì ú ù, this dataset uses integer numbers, not subscripted numbers.",
              "votes": 4
            },
            {
              "id": 3419547,
              "postDate": "2026-03-11T00:43:26.457Z",
              "content": "<p>We joined this a bit late but training models atm, first I wanted to say how cool it is people competing for prizes share data to help others. A very cool aspect I did not consider. And I did not have a lot of time to read all this so please forgive me if the question has already been addressed, but you mentioned names and other areas of confusion, are there any specific datasets I can collect that have these things? Currently training on about 113k pairs. Very cool post and very cool you posted this. I joined this more as an experience to learn how to do things like this and its been a bit frustrating as I have never rented an RTX 6000 before but finally got models training with it. Awesome post! </p>",
              "rawMarkdown": "We joined this a bit late but training models atm, first I wanted to say how cool it is people competing for prizes share data to help others. A very cool aspect I did not consider. And I did not have a lot of time to read all this so please forgive me if the question has already been addressed, but you mentioned names and other areas of confusion, are there any specific datasets I can collect that have these things? Currently training on about 113k pairs. Very cool post and very cool you posted this. I joined this more as an experience to learn how to do things like this and its been a bit frustrating as I have never rented an RTX 6000 before but finally got models training with it. Awesome post! ",
              "votes": 1,
              "isDeleted": true
            },
            {
              "id": 3419555,
              "postDate": "2026-03-11T01:54:47.680Z",
              "content": "<p>Here's a dataset which contains an onomasticon.csv for most of the names in this dataset: <a href=\"https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources/data\" target=\"_blank\">https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources/data</a></p>",
              "rawMarkdown": "Here's a dataset which contains an onomasticon.csv for most of the names in this dataset: https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources/data"
            }
          ]
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 3402145,
      "author_name": "Bilzard",
      "author_url": "",
      "post_date": "2026-02-05T10:30:13.190000",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Regarding normalization, could you clarify which of the following is true for the test set?</p>\n<ol>\n<li>Both transliterations and translations are already normalized (i.e., tokens like  and  are already present in both columns).</li>\n<li>Only the translations are normalized. Participants must implement their own logic to convert <code>x</code> or <code>...</code> into <code>&lt;gap&gt;</code> or <code>&lt;big_gap&gt;</code> for the test transliterations.</li>\n</ol>\n<p>If Case 2 is true, the final scores will be highly sensitive to specific preprocessing choices. This could turn the competition into a \"preprocessing lottery\" and encourage over-fitting to the Public LB rather than improving the actual MT model.</p>",
      "votes": 12,
      "replies": []
    },
    {
      "id": 3417055,
      "author_name": "Souhardya",
      "author_url": "",
      "post_date": "2026-03-04T13:29:24.590000",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> , while going through your onomasticon and the train labels, i noticed that there are some differences in the naming convention.</p>\n<p>Example 1 — Different name entirely:</p>\n<p>Akkadian token: šu-{d}EN.LÍL ,\nOnomasticon canonical form: Šu-Enlil ,\nActual training label: Šu-Illil</p>\n<p>Example 2 — Geminate consonant:</p>\n<p>Akkadian token: šu-ku-tum ,\nOnomasticon canonical form: Šukatum ,\nActual training label: Šukkutum</p>\n<p>(The onomasticon drops the geminate kk that the training labels preserve.)</p>\n<p>Example 3 — Long vowel:</p>\n<p>Akkadian token: ṣí-lu-lu ,\nOnomasticon canonical form: Ṣilulu ,\nActual training label: Ṣilūlu</p>\n<p>(The onomasticon omits the macron on the ū that the training labels include.)</p>\n<p>these are some examples pointed out by claude. \nwhich convention do the test labels follow — the onomasticon's or the training data's?</p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 3417798,
      "author_name": "Angantyr",
      "author_url": "",
      "post_date": "2026-03-06T08:58:57.160000",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Clarification needed.</p>\n<p>I'm going through the glossaries and find that in Summerian the nasal \"ŋ\" is used somewhat often. This appears, e.g., in the determinative for wood <code>{ŋeš}</code>, which for this competition is written as <code>{geš}</code>. Can we safely assume <code>ŋ -&gt; g</code> substitution as valid?</p>\n<p>Another question is about treating Summerian words from the ORACC glossary, e.g., <code>{na₄}za-gin₃-ŋu₁₀</code>. Intuitively I would change that into: <code>NA₄.ZA.GÌN-gu₁₀</code>, i.e.:</p>\n<ol>\n<li>treat the base as a summerogram -&gt; uppercase, no <code>{}</code>, <code>.</code>-joined</li>\n<li>treat the rest as a <code>-</code>-joined suffix</li>\n</ol>\n<p>What would be the approach to best match the data in <code>train.csv</code> and other sources?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3384015,
      "author_name": "Anil Ozturk",
      "author_url": "",
      "post_date": "2025-12-31T10:01:27.653000",
      "content": "<p>I want to be sure on these cases:</p>\n<ul>\n<li><code>&lt;big_gap&gt; &lt;gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>&lt;gap&gt; &lt;big_gap&gt; &lt;gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>&lt;big_gap&gt; &lt;big_gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>&lt;gap&gt; &lt;gap&gt;</code> → <code>&lt;big_gap&gt;</code></li>\n<li><code>someword-&lt;gap&gt; &lt;gap&gt;-someword</code> → <code>someword-&lt;gap&gt; &lt;gap&gt;-someword</code> OR <code>someword-&lt;big_gap&gt;-someword</code> ?</li>\n<li><code>&lt;big_gap&gt; &lt;gap&gt;-A-šùr</code> → <code>&lt;big_gap&gt; &lt;gap&gt;-A-šùr</code></li>\n<li><code>xxxx-kam</code> (train.csv line 43)  → <code>&lt;big_gap&gt;-kam</code> OR <code>&lt;big_gap&gt; &lt;gap&gt;-kam</code> ?</li>\n</ul>\n<p>It would be good to take feedback on these from you.</p>\n<p>I think that having to figure out the normalization logic you are using prevents us from focusing on the actual goal of the competition. There is some processing happening in the background that also affects the original “ground truth” translations you have, so instead of focusing on the machine learning part we end up spending our effort on reverse-engineering your normalization logic. Can't you share the transliteration-translation preprocessors you're using?</p>",
      "votes": 8,
      "replies": [
        {
          "id": 3384059,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2025-12-31T12:02:06.863000",
          "content": "<p>I'm wondering how to interpret this section that is unclear to me about \"?\", \"!\", and \":\" for <strong>translations</strong> :</p>\n<p><a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions</a></p>\n<p><strong>Formatting Suggestions for Transliterations and Translations</strong>:</p>\n<p>Remove (modern scribal notations):</p>\n<ul>\n<li>! (<em>certain reading</em>)</li>\n<li>? (<em>questionable reading</em>)</li>\n<li>/ (line divider) - Now we now there is none of these  according to <a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </li>\n<li>: OR . (word divider)</li>\n</ul>\n<p>We've a lot of ?!: in training set translations. But do we have the same in test set? </p>\n<p>Should we apply the suggestion above =&gt; Remove all !?: from translations in training set. Or only if they are enclosed with parenthesis.</p>\n<p>Some make sense to me with \":\" usage for a rate and \"?\" for a question in translation. It does not look <em>questionable reading</em> for the question marks below.</p>\n<blockquote>\n  <p>ff9442fd-9e7d-449c-a2d6-0cc35921cd65:  Šalim-Aššur answered: \"And what if I have tin and from my goods remaining in his possession? Will he pay me at the rate 6:1 for my tin and 15 shekels per in silver?\"</p>\n</blockquote>\n<p>Here the \":\" make sense too. If I remove \":\" then I should remove \";\" too here.</p>\n<blockquote>\n  <p>00f0d841-eb7a-46f8-86fc-bf9fd7d52cbf: From Šu-Tammuzī, Elaya, Ennam-Aššur and Lamassī to Ennam-Aššur and Ali-ahum: In accordance with your missive we have hired a attorney for you; Abiya son of Bebe is our attorney. </p>\n</blockquote>\n<p>I would just remove things like:  <code>\"(?)\",  \"(!)\",  \"(fem. plur.)\", \"(fem.)\",  \"(plur.)\", \"(pl.)\", \"(sing.)\", \"(plural)\"</code></p>\n<p>Can't you share the transliteration and translation list of characters you're using / allowing?</p>",
          "votes": 1,
          "replies": [
            {
              "id": 3384200,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2025-12-31T17:32:34.607000",
              "content": "<p>Good questions, here's the character list for transliterations:</p>\n<p><strong>Transliterations Characters</strong></p>\n<ul>\n<li>-</li>\n<li>a</li>\n<li>A</li>\n<li>i</li>\n<li>I</li>\n<li>u</li>\n<li>U</li>\n<li>m</li>\n<li>M</li>\n<li>š</li>\n<li>Š</li>\n<li>n</li>\n<li>N</li>\n<li>b</li>\n<li>B</li>\n<li>r</li>\n<li>R</li>\n<li>t</li>\n<li>T</li>\n<li>l</li>\n<li>L</li>\n<li>k</li>\n<li>K</li>\n<li>G</li>\n<li>g</li>\n<li>í</li>\n<li>Í</li>\n<li>D</li>\n<li>d</li>\n<li>Ù</li>\n<li>ù</li>\n<li>á</li>\n<li>Á</li>\n<li>.</li>\n<li>ú</li>\n<li>Ú</li>\n<li>p</li>\n<li>P</li>\n<li>e</li>\n<li>E</li>\n<li>h</li>\n<li>H</li>\n<li>q</li>\n<li>Q</li>\n<li>1</li>\n<li>ṣ</li>\n<li>Ṣ</li>\n<li>é</li>\n<li>É</li>\n<li>&lt;</li>\n<li>&gt;</li>\n<li>à</li>\n<li>À</li>\n<li>4</li>\n<li>z</li>\n<li>Z</li>\n<li>s</li>\n<li>S</li>\n<li>ì</li>\n<li>Ì</li>\n<li>5</li>\n<li>_</li>\n<li>2</li>\n<li>0</li>\n<li>½</li>\n<li>w</li>\n<li>W</li>\n<li>3</li>\n<li>{</li>\n<li>}</li>\n<li>ṭ</li>\n<li>Ṭ</li>\n<li>6</li>\n<li>⅓</li>\n<li>8</li>\n<li>⅔</li>\n<li>7</li>\n<li>è</li>\n<li>È</li>\n<li>⅚</li>\n<li>9</li>\n<li>¼</li>\n<li>!</li>\n<li>+</li>\n<li>⅙</li>\n<li>ı</li>\n<li>…</li>\n<li>ş</li>\n<li>İ</li>\n<li>:</li>\n</ul>\n<p><strong>Translations Characters</strong></p>\n<ul>\n<li>'</li>\n<li>?</li>\n<li>e</li>\n<li>E</li>\n<li>a</li>\n<li>A</li>\n<li>i</li>\n<li>I</li>\n<li>t</li>\n<li>T</li>\n<li>n</li>\n<li>N</li>\n<li>s</li>\n<li>S</li>\n<li>o</li>\n<li>O</li>\n<li>r</li>\n<li>R</li>\n<li>l</li>\n<li>L</li>\n<li>h</li>\n<li>H</li>\n<li>u</li>\n<li>U</li>\n<li>m</li>\n<li>M</li>\n<li>d</li>\n<li>D</li>\n<li>F</li>\n<li>f</li>\n<li>š</li>\n<li>Š</li>\n<li>-</li>\n<li>p</li>\n<li>P</li>\n<li>w</li>\n<li>W</li>\n<li>b</li>\n<li>B</li>\n<li>g</li>\n<li>G</li>\n<li>y</li>\n<li>Y</li>\n<li>.</li>\n<li>K</li>\n<li>k</li>\n<li>,</li>\n<li>C</li>\n<li>c</li>\n<li>v</li>\n<li>ā</li>\n<li>1</li>\n<li>)</li>\n<li>(</li>\n<li>&lt;</li>\n<li>&gt;</li>\n<li>z</li>\n<li>Z</li>\n<li>_</li>\n<li>Q</li>\n<li>q</li>\n<li>2</li>\n<li>ī</li>\n<li>ṭ</li>\n<li>Ṭ</li>\n<li>0</li>\n<li>:</li>\n<li>3</li>\n<li>½</li>\n<li>5</li>\n<li>;</li>\n<li>x</li>\n<li>ē</li>\n<li>4</li>\n<li>ū</li>\n<li>6</li>\n<li>ṣ</li>\n<li>Ṣ</li>\n<li>⅓</li>\n<li>8</li>\n<li>’</li>\n<li>!</li>\n<li>7</li>\n<li>j</li>\n<li>J</li>\n<li>⅔</li>\n<li>“</li>\n<li>”</li>\n<li>9</li>\n<li>–</li>\n<li>⅚</li>\n<li>¼</li>\n<li>⅙</li>\n<li>\"</li>\n<li>‘</li>\n<li>ı</li>\n<li>—</li>\n<li>[</li>\n<li>]</li>\n<li>ğ</li>\n<li>â</li>\n<li>+</li>\n<li>à</li>\n<li>ş</li>\n</ul>",
              "votes": 5,
              "replies": []
            },
            {
              "id": 3384450,
              "author_name": "steubk",
              "author_url": "",
              "post_date": "2026-01-01T09:11:08.517000",
              "content": "<p>did you actually publish the test set in plain chars ? 🧐</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3384470,
              "author_name": "Angantyr",
              "author_url": "",
              "post_date": "2026-01-01T10:04:46.220000",
              "content": "<p>The frequency counts are the same for upper and lower case -- the counter looks like it's case-insensitive. Is it intentional?</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3384479,
              "author_name": "耶✌",
              "author_url": "",
              "post_date": "2026-01-01T10:16:06.637000",
              "content": "<p>I have the same confusion, but I’ve been submitting consistently to check if the LB score improves. I can now say it helps – after fixing a lot of formatting issues, my LB score has gone up to 32.3.Happy new year!</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3384643,
              "author_name": "MPWARE",
              "author_url": "",
              "post_date": "2026-01-01T16:41:59.637000",
              "content": "<p>Thanks, it helps a lot <a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>\n<p>The only thing that looks like a bit weird is the frequency count:</p>\n<ul>\n<li>Top # 2 does not seem possible. ' =&gt; 1651390 ? =&gt; 181474. It means most of the text is a quote.</li>\n<li>Upper/lower case have the same count</li>\n</ul>\n<p>I believe your top #2 is something else and ? is not a real question mark but one or more characters that has been lost during counting or copy/paste or an intermediate sum/index.</p>",
              "votes": -1,
              "replies": []
            },
            {
              "id": 3384712,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-01-01T18:57:25.390000",
              "content": "<p>Thanks, yes those numbers were way off, the formula I used was wrong. I'll see if I can get the right counts, but at least the character set should help.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3384718,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-01-01T19:04:28.110000",
              "content": "<p>these are all unicode characters (UTF-8)</p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 3387378,
              "author_name": "Wisdom Aduah",
              "author_url": "",
              "post_date": "2026-01-06T20:38:12.507000",
              "content": "<p>Does the translation character list apply to the target test translations, or it only applies to the train translations? Specifically, do you expect our generated translations to use unicode fractions (e.g., ¼) or normal texts (e.g., 1/4)?</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3387879,
              "author_name": "nofreewill42",
              "author_url": "",
              "post_date": "2026-01-07T18:20:17.790000",
              "content": "<p>I suppose this is the hidden test set, right?\nAre there missing characters because the data source for it is different, or it just happened to be this way because of the random split got it this way (I doubt it)?\nIf the data source is different, are there other such crucial characteristic differences that we should know of?\nIs public set a subset of the hidden test set?</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3387926,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-01-07T20:11:19.960000",
              "content": "<p>Yes, that's correct. The test set wasn't in an online database for starters. I realize these formatting issues are frustrating, but our long-term hope is that the resulting MT model will be able to handle these different formatting differences, and not just overfit on the test set. For this reason the 8k published texts were provided to give a larger sample of these different formats. </p>\n<p>It's worth mentioning here that the other databases out there (e.g. CDLI, ORACC, eBL, Archibab) also have different transliteration formats. It is perhaps too ambitious to hope that all of these formats can be harmonized at this time / for this challenge, but it is a long-term goal within my research.</p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 3387940,
              "author_name": "nofreewill42",
              "author_url": "",
              "post_date": "2026-01-07T20:37:59.560000",
              "content": "<p>Yeah it is annoying but I also understand that there is no hard standardized format..\nThank you a lot for your response!</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3388619,
              "author_name": "Jovan Drenovcanin",
              "author_url": "",
              "post_date": "2026-01-09T08:55:12.310000",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> In the list of allowed characters above, in the transliterations part, there is a symbol \"…\" at the very end. Shouldn't this be converted to ?</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3394612,
              "author_name": "Angantyr",
              "author_url": "",
              "post_date": "2026-01-21T11:02:02.560000",
              "content": "<h4><strong>Transliterations Characters</strong></h4>\n<p><code>!+-.0123456789:&lt;&gt;ABDEGHIKLMNPQRSTUWZ_abdeghiklmnpqrstuwz{}¼½ÀÁÈÉÌÍÙÚàáèéìíùúİışŠšṢṣṬṭ…⅓⅔⅙⅚</code></p>\n<h4><strong>Translations Characters</strong></h4>\n<p><code>!\"\\'()+,-.0123456789:;&lt;&gt;?ABCDEFGHIJKLMNOPQRSTUWYZ[]_abcdefghijklmnopqrstuvwxyz¼½àâāēğīışŠšūṢṣṬṭ–—‘’“”⅓⅔⅙⅚</code></p>\n<p>(<strong>Characters sorted for brevity and a better view</strong>)</p>\n<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>\n<p>I wanted to come back to this as new question appeared regarding the character set.</p>\n<p>Several of those characters do not appear in the <code>train.csv</code> or are extremely rare and I'd like to know more on how to handle those cases, as it appears the normalization steps needed may affect scoring.</p>\n<h2>Unknown special characters (<code>şİıâ</code>)</h2>\n<p>Conversing with Gemini gave me the following clues:</p>\n<ol>\n<li>ş = ṣ (used in older publications)</li>\n<li>İ = I (Uppercase <strong>i</strong> written with dot - typical for Turkish sources)</li>\n<li>ı = i (Lowercase <strong>i</strong> but dotless - typical for Turkish sources)</li>\n</ol>\n<h2>Not normalized characters (<code>…:!{}</code>)</h2>\n<p>As per <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\" target=\"_blank\">Formatting suggestions for Transliterations and Translations</a> we should treat these as artifacts from the normalization process:</p>\n<ol>\n<li><code>…</code> - replace with <code>&lt;big_gap&gt;</code></li>\n<li><code>:</code> - replace with a dot (<strong>.</strong>)</li>\n<li><code>!</code> - remove (scribal note for certain reading of a sign)</li>\n<li><code>{}</code> - replaces <code>()</code> determinatives (predominant use in <code>train.csv</code> and <code>OA_lexicon.eBL.csv</code>)</li>\n</ol>\n<h2>Same fractions (<code>¼½⅓⅔⅙⅚</code>) in transliterations/translations</h2>\n<p>This strongly suggests that at least part of the translations do not recalculate the fractions, i.e., <code>1.83333 ma-na 3.5 GÍN</code> becomes <code>1⅚ ma-na 3½ GÍN</code> in test but becomes <code>1⅚ mina 3½ shekels</code> while in <code>train.csv</code> it is recalculated to <code>1 mina 56.5 shekels</code> (<code>1 mina 56½ shekels</code> with Unicode fraction notation).</p>\n<p>Should we retain the fractions as they are in the transliterations or should we recalculate them?</p>\n<h2>Translation characters <code>àâğ\\'()‘’“”</code></h2>\n<ol>\n<li>à = ā ?</li>\n<li>â = ā ?</li>\n<li>ğ = g ?</li>\n<li>\\ - a line divider (remove), alternative reading?</li>\n<li>' - glottal stop?</li>\n<li>() - additional information, e.g. <code>(fem. plur.)</code></li>\n<li>‘’ - should be standardized to <code>''</code>?</li>\n<li>“” - should be standardized to <code>\"\"</code>?</li>\n</ol>",
              "votes": 5,
              "replies": []
            },
            {
              "id": 3401159,
              "author_name": "",
              "author_url": "",
              "post_date": "2026-02-03T03:41:45.523000",
              "content": "<p>1.'[' in translation need to be removed?  e.g:[I paid … … … … … …]\n2.is there &lt; gap &gt; or &lt; big-gap &gt; in the translation column of pulibc and private test sets?</p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 3384067,
          "author_name": "耶✌",
          "author_url": "",
          "post_date": "2025-12-31T12:10:15.940000",
          "content": "<p>From my LB results, not merging <code>&lt;gap&gt;</code> <code>&lt;big_gap&gt;</code>performs a bit better.</p>",
          "votes": 3,
          "replies": [
            {
              "id": 3384210,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2025-12-31T18:02:05.483000",
              "content": "<p>Yes, sorry for the confusion. Here's what I have found in terms of co-occurrances of gaps with frequency counts.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F28748094%2F10acb49069b39d449b16628d790292b8%2FGaps_no_EXIF.png?generation=1767204595136708&amp;alt=media\" alt=\"\">\nP.S. This is not an exhaustive list, but a representative one.</p>",
              "votes": 9,
              "replies": []
            },
            {
              "id": 3384436,
              "author_name": "Anil Ozturk",
              "author_url": "",
              "post_date": "2026-01-01T08:49:05.640000",
              "content": "<p>You said </p>\n<p><code>In practice, collapsing combinations such as &lt;gap&gt; &lt;big_gap&gt; or &lt;big_gap&gt; &lt;gap&gt; into a single &lt;big_gap&gt; on both sides improves sentence-level alignment and reduces noise.</code></p>\n<p>but the list you shared includes non-merged combinations.</p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 3384476,
              "author_name": "Angantyr",
              "author_url": "",
              "post_date": "2026-01-01T10:13:20.043000",
              "content": "<p>While analysing the sentence mismatch <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664177#3384304\" target=\"_blank\">discussed in a parallel thread</a> I found that in some transliterations <code>(2 broken lines)</code> would be space-split and treated as <code>x x x</code> causing a shift.</p>\n<p>Targetting <code>&lt;gap&gt;/&lt;big_gap&gt;</code> combinations without the connected ones was a problem of its own and I ended up with this capture pattern from Perplexity:</p>\n<pre><code>(?&lt;!\\S)&lt;(?:gap|big_gap)&gt;(?!\\s?-|\\S)(?:\\s&lt;(?:gap|big_gap)&gt;(?!\\s?-|\\S))+\n</code></pre>\n<p>It worked for me while working on <code>Sentences_Oare_FirstWord_LinNum.csv</code>. I haven't tried it on corrected <code>train.csv</code> yet.</p>\n<p>edit: the capture pattern was modified (should have ended with <code>+</code> instead of <code>*</code>)</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3393353,
              "author_name": "einherjer",
              "author_url": "",
              "post_date": "2026-01-18T21:18:48.023000",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> so is your statement regarding collapsing gaps (quoted below, from the topic post) still valid?</p>\n<blockquote>\n  <p>multiple consecutive x tokens should be collapsed into a single &lt;big_gap&gt;.</p>\n</blockquote>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3393362,
              "author_name": "Angantyr",
              "author_url": "",
              "post_date": "2026-01-18T21:29:47.253000",
              "content": "<blockquote>\n  <p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> so is your statement regarding collapsing gaps (quoted below, from the topic post) still valid?</p>\n  <blockquote>\n    <p>multiple consecutive x tokens should be collapsed into a single .</p>\n  </blockquote>\n</blockquote>\n<p>Anything in angle brackets becomes invisible when formatted. Better wrap it in single backticks, e.g., <code>&lt;big_gap&gt;</code></p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3402088,
              "author_name": "einherjer",
              "author_url": "",
              "post_date": "2026-02-05T07:54:20.480000",
              "content": "<blockquote>\n  <p>You said </p>\n  <p><code>In practice, collapsing combinations such as &lt;gap&gt; &lt;big_gap&gt; or &lt;big_gap&gt; &lt;gap&gt; into a single &lt;big_gap&gt; on both sides improves sentence-level alignment and reduces noise.</code></p>\n  <p>but the list you shared includes non-merged combinations.</p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> could you please provide a clarification regarding this comment? </p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 3385136,
          "author_name": "Adam Anderson",
          "author_url": "",
          "post_date": "2026-01-02T15:52:14.287000",
          "content": "<p>As for the two questions you posed here:</p>\n<ul>\n<li>Both of these can be found: someword-&lt; gap &gt; &lt; gap &gt;-someword AND someword-&lt; big_gap &gt;-someword</li>\n<li>xxxx-kam → &lt; big_gap &gt;-kam</li>\n<li>x-kam → &lt; gap &gt;-kam</li>\n</ul>",
          "votes": 2,
          "replies": [
            {
              "id": 3385223,
              "author_name": "Angantyr",
              "author_url": "",
              "post_date": "2026-01-02T18:33:18.763000",
              "content": "<p>The <code>&lt;gap&gt;</code> and <code>&lt;big_gap&gt;</code> are missing. Is it the markdown formatting?</p>\n<p>edit: it is. It shows up after clicking <code>Quote</code>.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3387938,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-01-07T20:34:55.447000",
              "content": "<p>yes, sorry, fixed it</p>",
              "votes": 1,
              "replies": []
            }
          ]
        },
        {
          "id": 3386031,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2026-01-04T12:35:15.483000",
          "content": "<p><a href=\"https://www.kaggle.com/nlztrk\" target=\"_blank\">@nlztrk</a> Did it work for you to collapse gaps? CV is stable but LB is worse here with simple gap/big_gap merge:</p>\n<pre><code>def collapse_gaps(text):\n    \"\"\"\n    Collapse consecutive &lt;gap&gt;/&lt;big_gap&gt; into a single &lt;big_gap&gt;.\n    Keep single isolated &lt;gap&gt; as-is.\n    \"\"\"\n    tokens = text.split()\n\n    collapsed = []\n    i = 0\n    n = len(tokens)\n\n    while i &lt; n:\n        if tokens[i] in {\"&lt;gap&gt;\", \"&lt;big_gap&gt;\"}:\n            j = i\n            while j &lt; n and tokens[j] in {\"&lt;gap&gt;\", \"&lt;big_gap&gt;\"}:\n                j += 1\n\n            run_len = j - i\n\n            # If more than one gap marker =&gt; big gap\n            if run_len &gt;= 2:\n                collapsed.append(\"&lt;big_gap&gt;\")\n            else:\n                collapsed.append(tokens[i])\n\n            i = j\n        else:\n            collapsed.append(tokens[i])\n            i += 1\n\n    return \" \".join(collapsed) \n</code></pre>",
          "votes": 1,
          "replies": [
            {
              "id": 3386115,
              "author_name": "Anil Ozturk",
              "author_url": "",
              "post_date": "2026-01-04T16:30:35.077000",
              "content": "<p>Im collapsing all big gaps into one big gap and all adjacent big gaps and gaps into a big gap. For now it works as the best. But the examples of the organizator violates this logic, we still don't know theirs.</p>",
              "votes": 3,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3385720,
      "author_name": "Lee Drake",
      "author_url": "",
      "post_date": "2026-01-03T18:55:02.610000",
      "content": "<p>This is my main worry about participating in this contest. I've been working on cuneiform LLMS for the past two years, and a good text pipeline has to standardize the presentation from multiple script types (on my end I'm also working with Elamite, Sumerian, and Hittite). Getting it to align with ASCII helps learnability tremendously if one wants to leverage pre-trained models like T5 or NLLB. But I don't know how we prepare the unknown text - can we write an input pipeline (and prompt structure) to take in the unknown data with a submission?  </p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 3385029,
      "author_name": "Yicong XIAO",
      "author_url": "",
      "post_date": "2026-01-02T12:56:09.057000",
      "content": "<p>For the subscripts used in the determinatives, do they also need to be transformed to either diacritics or plain numbers?</p>\n<p>e.g. should it be {tug₂} or {tug2} ? </p>",
      "votes": 2,
      "replies": [
        {
          "id": 3385102,
          "author_name": "Yicong XIAO",
          "author_url": "",
          "post_date": "2026-01-02T14:50:33.080000",
          "content": "<p>On second thought, since the determinatives appear only in the transliteration text, I can pick either scheme. Then, during inference, I can detect the determinatives in the test set input (should be fairly easy) and transform them according to the scheme I pick.  </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 3385134,
          "author_name": "Adam Anderson",
          "author_url": "",
          "post_date": "2026-01-02T15:48:51.327000",
          "content": "<p>The expected format for {tug₂} or {tug2} is actually {túg}</p>\n<p>For background, in ASCII ATF (CDLI) they don't use diacritics for ú = u2, and ù = u3, and so on. Further, in Oracc these numbers become subscripts: ú = u₂, etc. That second comment was intended to indicate that for the readings with 2 or 3 this dataset used diacritics instead of numbers or subscripted numbers (i.e. á à é è í ì ú ù; NOT a2, a3, etc.; NOT a₂, etc.). However, sign values with 4, 5, 6, …, 10, 12, 13, etc. are found in the dataset (e.g., DU10). So aside from sign values with these readings á à é è í ì ú ù, this dataset uses integer numbers, not subscripted numbers.</p>",
          "votes": 6,
          "replies": [
            {
              "id": 3385744,
              "author_name": "Anil Ozturk",
              "author_url": "",
              "post_date": "2026-01-03T20:35:02.630000",
              "content": "<p>{túg} or {TÚG}? there is no lowercased {túg} in any guide you've shared. Please just give a reproducible normalization script or a very clear and strict conversion table. Most of your statements contradict each other.</p>",
              "votes": 4,
              "replies": []
            },
            {
              "id": 3385749,
              "author_name": "Jack",
              "author_url": "",
              "post_date": "2026-01-03T20:59:28.867000",
              "content": "<p>Uppercase.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3385751,
              "author_name": "",
              "author_url": "",
              "post_date": "2026-01-03T21:10:14.390000",
              "content": "",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3385756,
              "author_name": "Jack",
              "author_url": "",
              "post_date": "2026-01-03T21:33:34.757000",
              "content": "<p>Check the overview page and the data provided</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3387872,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-01-07T18:12:17.423000",
              "content": "<p>TÚG, without brackets (in the test data), but {tug2} in the CDLI data and {tug₂} in the Oracc data, in case you choose to use those.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3387880,
              "author_name": "Anil Ozturk",
              "author_url": "",
              "post_date": "2026-01-07T18:23:38.160000",
              "content": "<p>In the overview tab you are saying:</p>\n<p>{tug₂} = (TÚG) preceding textiles and other woven objects</p>\n<p>TÚG is being used with parentheses there but you are now saying <strong>without brackets</strong>. I think you need to clarify the <strong>exact</strong> form for that list. What should we expect for these in the test set:</p>\n<ul>\n<li>{d} = dingir ‘god, deity’ — d preceding non-human divine actors</li>\n<li>{mul} = ‘stars’ — MUL preceding astronomical bodies and constellations</li>\n<li>{ki} = ‘earth’ — KI following a geographical place name or location</li>\n<li>{lu₂} = LÚ preceding people and professions</li>\n<li>{e₂} = {É} preceding buildings and institutions, such as temples and palaces</li>\n<li>{uru} = (URU) preceding names of settlements, such as villages, towns and cities</li>\n<li>{kur} = (KUR) preceding lands and territories as well as mountains</li>\n<li>{mi} = munus (f) preceding feminine personal names</li>\n<li>{m} = (1 or m) preceding masculine personal names</li>\n<li>{geš} / {ĝeš) = (GIŠ) preceding trees and things made of wood</li>\n<li>{tug₂} = (TÚG) preceding textiles and other woven objects</li>\n<li>{dub} = (DUB) preceding clay tablets, and by extension, documents and legal records</li>\n<li>{id₂} = (ÍD) (a ligature of A and ENGUR, transliterated: A.ENGUR) preceding names of canals or rivers or when written on its own referring to the divine river</li>\n<li>{mušen} = (MUŠEN) preceding birds</li>\n<li>{na₄} = (na4) preceding stone</li>\n<li>{kuš} = (kuš) preceding (animal) skin, fleece, hides</li>\n<li>{u₂} = (Ú) preceding plants</li>\n</ul>",
              "votes": 5,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3384225,
      "author_name": "Khushi Yadav",
      "author_url": "",
      "post_date": "2025-12-31T18:28:13.033000",
      "content": "<p>Deep Past Challenge - Translate Akkadian to English</p>",
      "votes": -10,
      "replies": []
    },
    {
      "id": 3408964,
      "author_name": "Oscar Guarnizo",
      "author_url": "",
      "post_date": "2026-02-21T18:30:40.723000",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Hi, I was wondering where I can get the onomasticon (a curated list of names and attested spellings). Could you help me with that please?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3383782,
      "author_name": "ulasdesouza",
      "author_url": "",
      "post_date": "2025-12-30T20:27:28.243000",
      "content": "<p>do not convert diacritics to ASCII (information loss!)\ndo not remove subscripts (semantic meaning!)\nEvaluation data contains diacritics - be compatible with this</p>\n<p>am i right?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3383787,
          "author_name": "Adam Anderson",
          "author_url": "",
          "post_date": "2025-12-30T20:40:36.527000",
          "content": "<p>Close, but let me be clear:</p>\n<ul>\n<li>(1) with the exception of ḫ, the test data has diacritics (š ṣ ṭ ā á à ī í ì ū ú ù - and upper case versions). If you're obtaining data from CDLI or ORACC, you will need to convert their formatted transliterations to this diacritic style (conversions are listed in the overview) — note ā ī ū are in translations, not transliteration text.</li>\n<li>(2) there are no subscripted numbers in the test data, so expect A-šùr-DU10 for example.</li>\n<li>(3) keep diacritics for transliterations, but normalize them for translations of named entities, for example: A-šùr-DU10 (transliteration) --&gt; Aššur-ṭāb (translation)</li>\n</ul>",
          "votes": 4,
          "replies": [
            {
              "id": 3383830,
              "author_name": "",
              "author_url": "",
              "post_date": "2025-12-30T22:29:48.533000",
              "content": "",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3383953,
              "author_name": "QianYuu",
              "author_url": "",
              "post_date": "2025-12-31T07:19:41.317000",
              "content": "<p>Hello, why is your CHRF metric so high? What's the difference between it and the CHRF from the public notebook? Public notebook link: <a href=\"https://www.kaggle.com/code/takamichitoda/dpc-starter-train\" target=\"_blank\">https://www.kaggle.com/code/takamichitoda/dpc-starter-train</a></p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3383967,
              "author_name": "QianYuu",
              "author_url": "",
              "post_date": "2025-12-31T07:58:10.150000",
              "content": "<p>My guess is that if the sentences are the same, then you most likely did sentence alignment.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3384009,
              "author_name": "heng",
              "author_url": "",
              "post_date": "2025-12-31T09:50:19.640000",
              "content": "<p>This public notebook does not use competition metrics.</p>\n<p>Try: <a href=\"https://www.kaggle.com/code/metric/dpi-bleu-chrf\" target=\"_blank\">https://www.kaggle.com/code/metric/dpi-bleu-chrf</a></p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3384017,
              "author_name": "MPWARE",
              "author_url": "",
              "post_date": "2025-12-31T10:04:33.200000",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>\n<blockquote>\n  <p>(2) there are no subscripted numbers in the test data, so expect A-šùr-DU10 for example.</p>\n</blockquote>\n<p>You said something a bit different in another post:</p>\n<blockquote>\n  <p>Subscript digits are converted to normal digits, except for 2 and 3, which are represented by diacritic marks over the vowels (see overview for more details)</p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664518#3382901\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664518#3382901</a></p>\n<p>Do I miss something?</p>",
              "votes": 4,
              "replies": []
            },
            {
              "id": 3384671,
              "author_name": "MPWARE",
              "author_url": "",
              "post_date": "2026-01-01T17:10:00.527000",
              "content": "<p>Answer to myself: Now we know that first assertion it True and second is False.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3384721,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-01-01T19:17:16.883000",
              "content": "<p>Yes, right, the ack of specificity in that second statement is problematic. For background, in ASCII ATF (CDLI) they don't use diacritics for ú = u2, and ù = u3, and so on. Further, in Oracc these numbers become subscripts: ú = u₂, etc.\nThat second comment was intended to indicate that for the readings with 2 or 3 this dataset used diacritics instead of numbers or subscripted numbers (i.e. á à é è í ì ú ù; NOT a2, a3, etc.; NOT a₂, etc.).\nHowever, sign values with 4, 5, 6, …, 10, 12, 13, etc. are found in the dataset (e.g., DU10). \nSo aside from sign values with these readings á à é è í ì ú ù, this dataset uses integer numbers, not subscripted numbers.</p>",
              "votes": 4,
              "replies": []
            },
            {
              "id": 3419547,
              "author_name": "",
              "author_url": "",
              "post_date": "2026-03-11T00:43:26.457000",
              "content": "<p>We joined this a bit late but training models atm, first I wanted to say how cool it is people competing for prizes share data to help others. A very cool aspect I did not consider. And I did not have a lot of time to read all this so please forgive me if the question has already been addressed, but you mentioned names and other areas of confusion, are there any specific datasets I can collect that have these things? Currently training on about 113k pairs. Very cool post and very cool you posted this. I joined this more as an experience to learn how to do things like this and its been a bit frustrating as I have never rented an RTX 6000 before but finally got models training with it. Awesome post! </p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3419555,
              "author_name": "Adam Anderson",
              "author_url": "",
              "post_date": "2026-03-11T01:54:47.680000",
              "content": "<p>Here's a dataset which contains an onomasticon.csv for most of the names in this dataset: <a href=\"https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources/data\" target=\"_blank\">https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources/data</a></p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3383756": "Based on recent experiments and discussions with several participants, I want to share a set of observations about what appear to be the *primary bottlenecks* in this competition. These are not model-architecture issues per se, but data- and representation-level issues that strongly affect downstream training, evaluation, and reinforcement learning.\n\nAfter multiple iterations, debugging sessions, and failed-but-informative experiments, two main stumbling blocks consistently emerge:\n\n**(1) Named entities (personal names, place names, divine names)**\n**(2) Inconsistent ASCII / transliteration formats across Akkadian datasets**\n\nBoth issues disproportionately affect tokenization, alignment, and reward stability, and in practice they limit performance more than model size or optimizer choice.\n\n---\n\n### 1. Named entities are a dominant source of error\n\nPersonal names, geographic names, and divine names behave very differently from ordinary lexical items:\n\n* They are often transliterated inconsistently across editions.\n* They frequently preserve older orthographic conventions.\n* They are semantically opaque to the model unless explicitly supported.\n\nIn experiments, many otherwise reasonable translations fail almost entirely because a name is mangled, dropped, or hallucinated. This affects not only final accuracy but also reward-based methods, where small orthographic deviations can collapse a sentence-level score.\n\nTo address this, I have prepared an **onomasticon** (a curated list of names and attested spellings), which I will share as supplemental data. Participants may find it useful to:\n\n* add it as a lookup or constraint layer,\n* bias decoding for known names,\n* or use it for post-generation repair.\n\nEven partial normalization of named entities tends to improve both perceived translation quality and metric stability.\n\n---\n\n### 2. Transliteration format normalization is not optional\n\nA second, and often underestimated, issue is **ASCII-format variation in Akkadian transliteration**. Different corpora encode the same underlying text using different conventions, many of which are *not interchangeable* without loss.\n\nA concrete example that came up in a private discussion illustrates the problem. One approach converted diacritics into ASCII sequences (e.g., `š → sz`, `ú → u2`, etc.) before training. This is a reasonable instinct, but in this case it was done in the wrong direction: the *evaluation data already contains diacritics*, and reducing the alphabet removed distinctions that are semantically meaningful in Akkadian.\n\nFor example, the evaluation data expects forms like:\n\n* `i-ṣí-ba-at` rather than `i-Si₂-ba-at`\n* `KÙ-pì-a` rather than `KU₃-pi₃-a`\n* `KIŠIB` rather than `KISZIB`\n\nThe key takeaway is that **the competition data uses an extended alphabet by design**, and collapsing it into ASCII can degrade both meaning and alignment. While this increases tokenization difficulty, preserving distinctions such as `s / ṣ / š` and `t / ṭ` is preferable to losing them.\n\nThe recommended strategy is therefore:\n\n* keep diacritics,\n* convert ASCII substitutes (e.g., `sz`) *into* diacritics,\n* and normalize everything toward the format used in the training and evaluation sets.\n\n---\n\n### Gaps, damage markers, and parallel alignment [update: 2/18/26]\n\nAnother recurring source of confusion concerns damaged text and gap markers in the data:\n\n* `x` represents a single broken sign,\n* sequences like `x x x x` or `...` represent a larger lacuna.\n\nFor modeling purposes we reduced all breaks to a single marker: `<gap>`\n\n* we removed the tag for `<big_gap>` from the train and test (and other transliterations). We also deduplicated instances multiple sequential gaps (e.g. `<gap> <gap`, `<gap>-<gap>`, `<gap> <gap>`, `<gap>. <gap>`, etc.\n\nIdeally, these gap markers would be **parallelized between transliteration and translation whenever possible**. However, this was not completely accomplished, as the translation we had access to did not pay strict attention to the gaps. This will be an aspect of the challenge in which a significant advantage will remain for those who have controlled for this data. If a large gap appears in one side but not the other, the model is forced to learn misalignment rather than translation.\n\nThis also applies to edge cases such as `<gap>` attached to a word (e.g., `<gap>-A-šùr`), which should be preserved rather than blindly removed.\n\n---\n\n### Why this matters for training and RL-based methods\n\nMany participants have observed that standard supervised fine-tuning produces reasonable loss curves, while reinforcement or preference-based methods fail to improve or show no reward signal. In nearly all cases examined so far, this traces back to **output non-conformance** caused by the issues above.\n\nIf the model is penalized for:\n\n* orthographic mismatches,\n* bracket artifacts,\n* inconsistent gap handling,\n* or malformed named entities,\n\nthen reward functions cannot reliably distinguish “closer” from “farther” outputs. Addressing normalization and alignment first makes rewards smoother and learning signals usable.\n\n---\n\n### Closing note\n\nNone of this is meant as criticism of existing approaches; these are difficult problems, and much of the complexity comes from the philological nature of the data itself. The hope in sharing this publicly is to reduce duplicated effort and make preprocessing choices more transparent across submissions.\n\nI will continue to share supplemental resources (including the onomasticon) as they are finalized, and I’m happy to discuss normalization or alignment strategies further if helpful to others.\n",
    "3402145": "@deeppast Regarding normalization, could you clarify which of the following is true for the test set?\n\n1. Both transliterations and translations are already normalized (i.e., tokens like <gap> and <big_gap> are already present in both columns).\n2. Only the translations are normalized. Participants must implement their own logic to convert `x` or `...` into `<gap>` or `<big_gap>` for the test transliterations.\n\nIf Case 2 is true, the final scores will be highly sensitive to specific preprocessing choices. This could turn the competition into a \"preprocessing lottery\" and encourage over-fitting to the Public LB rather than improving the actual MT model.",
    "3417055": "@deeppast , while going through your onomasticon and the train labels, i noticed that there are some differences in the naming convention.\n\nExample 1 — Different name entirely:\n\nAkkadian token: šu-{d}EN.LÍL ,\nOnomasticon canonical form: Šu-Enlil ,\nActual training label: Šu-Illil\n\nExample 2 — Geminate consonant:\n\nAkkadian token: šu-ku-tum ,\nOnomasticon canonical form: Šukatum ,\nActual training label: Šukkutum\n\n(The onomasticon drops the geminate kk that the training labels preserve.)\n\nExample 3 — Long vowel:\n\nAkkadian token: ṣí-lu-lu ,\nOnomasticon canonical form: Ṣilulu ,\nActual training label: Ṣilūlu\n\n(The onomasticon omits the macron on the ū that the training labels include.)\n\nthese are some examples pointed out by claude. \nwhich convention do the test labels follow — the onomasticon's or the training data's?\n",
    "3417798": "@deeppast Clarification needed.\n\nI'm going through the glossaries and find that in Summerian the nasal \"ŋ\" is used somewhat often. This appears, e.g., in the determinative for wood `{ŋeš}`, which for this competition is written as `{geš}`. Can we safely assume `ŋ -> g` substitution as valid?\n\nAnother question is about treating Summerian words from the ORACC glossary, e.g., `{na₄}za-gin₃-ŋu₁₀`. Intuitively I would change that into: `NA₄.ZA.GÌN-gu₁₀`, i.e.:\n1. treat the base as a summerogram -> uppercase, no `{}`, `.`-joined\n2. treat the rest as a `-`-joined suffix\n\nWhat would be the approach to best match the data in `train.csv` and other sources?",
    "3384015": "I want to be sure on these cases:\n\n- `<big_gap> <gap>` → `<big_gap>`\n- `<gap> <big_gap> <gap>` → `<big_gap>`\n- `<big_gap> <big_gap>` → `<big_gap>`\n- `<gap> <gap>` → `<big_gap>`\n- `someword-<gap> <gap>-someword` → `someword-<gap> <gap>-someword` OR `someword-<big_gap>-someword` ?\n- `<big_gap> <gap>-A-šùr` → `<big_gap> <gap>-A-šùr`\n- `xxxx-kam` (train.csv line 43)  → `<big_gap>-kam` OR `<big_gap> <gap>-kam` ?\n\nIt would be good to take feedback on these from you.\n\nI think that having to figure out the normalization logic you are using prevents us from focusing on the actual goal of the competition. There is some processing happening in the background that also affects the original “ground truth” translations you have, so instead of focusing on the machine learning part we end up spending our effort on reverse-engineering your normalization logic. Can't you share the transliteration-translation preprocessors you're using?",
    "3385720": "This is my main worry about participating in this contest. I've been working on cuneiform LLMS for the past two years, and a good text pipeline has to standardize the presentation from multiple script types (on my end I'm also working with Elamite, Sumerian, and Hittite). Getting it to align with ASCII helps learnability tremendously if one wants to leverage pre-trained models like T5 or NLLB. But I don't know how we prepare the unknown text - can we write an input pipeline (and prompt structure) to take in the unknown data with a submission?  ",
    "3385029": "For the subscripts used in the determinatives, do they also need to be transformed to either diacritics or plain numbers?\n\ne.g. should it be {tug₂} or {tug2} ? ",
    "3384225": "Deep Past Challenge - Translate Akkadian to English",
    "3408964": "@deeppast Hi, I was wondering where I can get the onomasticon (a curated list of names and attested spellings). Could you help me with that please?\n",
    "3383782": "do not convert diacritics to ASCII (information loss!)\ndo not remove subscripts (semantic meaning!)\nEvaluation data contains diacritics - be compatible with this\n\nam i right?"
  }
}