{
  "id": 684278,
  "title": "89th Place - Solution",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/684278",
  "author_name": "xavier fernando",
  "post_date": "2026-03-24T09:53:05.571000",
  "votes": 4,
  "comment_count": 0,
  "views": 0,
  "content": "<h3>Stage 1: Domain-Specific Pretraining</h3>\n<ul>\n<li><p>I performed pretraining on Akkadian text using a span masking objective, similar to masked language modeling but better suited for structured and partially missing sequences.</p></li>\n<li><p>The goal was to adapt the model to the Akkadian language domain, improving its ability to understand context, symbols, and linguistic patterns specific to the dataset.</p>\n<ul>\n<li><p>Tokens like <strong><code>&lt;extra_id_0&gt;</code>, <code>&lt;extra_id_1&gt;</code>, <code>&lt;extra_id_2&gt;</code></strong> are treated as <strong>special symbols</strong> during training.</p></li>\n<li><p>These are <strong>not natural language words</strong>, but <strong>placeholders used for span masking</strong>.</p></li>\n<li><p>The model learns to <strong>predict missing spans</strong>, making it robust to noisy and incomplete sequences.</p></li></ul></li>\n</ul>\n<p>'a-na 8 ha-am-ša-t **<code>&lt;extra_id_0&gt;</code> i-ša-qal šu-ma i-na u4-me-šu m <code>&lt;extra_id_1&gt;</code> -al-ú-tim lá iš-qú-ul  -ma a-wa-at kà-ri-im ú  GI en-nam-a-šur'</p>\n<p>**<code>&lt;extra_id_0&gt;</code>šar-dí-ú<code>&lt;extra_id_1&gt;</code> -qá-a-šu a-na a-wa-  u-um kà-ni-ì  ša a-šur ší-'</p>\n<h3>Stage 2: Data Preparation and Cleaning</h3>\n<p>The provided sentence data had significant alignment issues. Many lines were not properly segmented, which negatively affected training.</p>\n<p>To address this:</p>\n<ul>\n<li>I manually inspected and split the publish,sentence csv files.Created a clean and well-aligned training set.Focused on improving sequence consistency and reducing noise.</li>\n</ul>\n<h3>Stage 3: Fine-tuning Strategy</h3>\n<p>For training, I used the competition data and dropped the incomplete translations</p>\n<ul>\n<li>Differential learning rates across model layers</li>\n<li>Weight decay tuning for better generalization</li>\n<li>Maximum token length of 3048</li>\n<li>Training on an A100 GPU for efficient large-sequence handling</li>\n</ul>\n<p>These optimizations helped the model converge better and capture long-context dependencies.</p>\n<p>The final prediction was generated using an ensemble of the top three models.ensemble from public notebook.</p>\n<h4>Approaches that did not work:</h4>\n<ul>\n<li>Pseudo-labeling, </li>\n<li>training a ByT5 model from scratch, </li>\n<li>adding LSTM/GRU layers as a custom head on top of T5.</li>\n</ul>",
  "messages": [
    {
      "id": 3427614,
      "postDate": "2026-03-24T09:53:05.570Z",
      "content": "<h3>Stage 1: Domain-Specific Pretraining</h3>\n<ul>\n<li><p>I performed pretraining on Akkadian text using a span masking objective, similar to masked language modeling but better suited for structured and partially missing sequences.</p></li>\n<li><p>The goal was to adapt the model to the Akkadian language domain, improving its ability to understand context, symbols, and linguistic patterns specific to the dataset.</p>\n<ul>\n<li><p>Tokens like <strong><code>&lt;extra_id_0&gt;</code>, <code>&lt;extra_id_1&gt;</code>, <code>&lt;extra_id_2&gt;</code></strong> are treated as <strong>special symbols</strong> during training.</p></li>\n<li><p>These are <strong>not natural language words</strong>, but <strong>placeholders used for span masking</strong>.</p></li>\n<li><p>The model learns to <strong>predict missing spans</strong>, making it robust to noisy and incomplete sequences.</p></li></ul></li>\n</ul>\n<p>'a-na 8 ha-am-ša-t **<code>&lt;extra_id_0&gt;</code> i-ša-qal šu-ma i-na u4-me-šu m <code>&lt;extra_id_1&gt;</code> -al-ú-tim lá iš-qú-ul  -ma a-wa-at kà-ri-im ú  GI en-nam-a-šur'</p>\n<p>**<code>&lt;extra_id_0&gt;</code>šar-dí-ú<code>&lt;extra_id_1&gt;</code> -qá-a-šu a-na a-wa-  u-um kà-ni-ì  ša a-šur ší-'</p>\n<h3>Stage 2: Data Preparation and Cleaning</h3>\n<p>The provided sentence data had significant alignment issues. Many lines were not properly segmented, which negatively affected training.</p>\n<p>To address this:</p>\n<ul>\n<li>I manually inspected and split the publish,sentence csv files.Created a clean and well-aligned training set.Focused on improving sequence consistency and reducing noise.</li>\n</ul>\n<h3>Stage 3: Fine-tuning Strategy</h3>\n<p>For training, I used the competition data and dropped the incomplete translations</p>\n<ul>\n<li>Differential learning rates across model layers</li>\n<li>Weight decay tuning for better generalization</li>\n<li>Maximum token length of 3048</li>\n<li>Training on an A100 GPU for efficient large-sequence handling</li>\n</ul>\n<p>These optimizations helped the model converge better and capture long-context dependencies.</p>\n<p>The final prediction was generated using an ensemble of the top three models.ensemble from public notebook.</p>\n<h4>Approaches that did not work:</h4>\n<ul>\n<li>Pseudo-labeling, </li>\n<li>training a ByT5 model from scratch, </li>\n<li>adding LSTM/GRU layers as a custom head on top of T5.</li>\n</ul>",
      "rawMarkdown": "### Stage 1: Domain-Specific Pretraining\n\n - I performed pretraining on Akkadian text using a span masking objective, similar to masked language modeling but better suited for structured and partially missing sequences.\n - The goal was to adapt the model to the Akkadian language domain, improving its ability to understand context, symbols, and linguistic patterns specific to the dataset.\n\n- Tokens like **`<extra_id_0>`, `<extra_id_1>`, `<extra_id_2>`** are treated as **special symbols** during training.\n- These are **not natural language words**, but **placeholders used for span masking**.\n\n- The model learns to **predict missing spans**, making it robust to noisy and incomplete sequences.\n\n'a-na 8 ha-am-ša-t **`<extra_id_0>` i-ša-qal šu-ma i-na u4-me-šu m `<extra_id_1>` -al-ú-tim lá iš-qú-ul <extra_id_2> -ma a-wa-at kà-ri-im ú <extra_id_3> GI en-nam-a-šur'\n\n**`<extra_id_0>`šar-dí-ú`<extra_id_1>` -qá-a-šu a-na a-wa- <extra_id_2> u-um kà-ni-ì <extra_id_3> ša a-šur ší-'\n\n\n### Stage 2: Data Preparation and Cleaning\n\nThe provided sentence data had significant alignment issues. Many lines were not properly segmented, which negatively affected training.\n\nTo address this:\n - I manually inspected and split the publish,sentence csv files.Created a clean and well-aligned training set.Focused on improving sequence consistency and reducing noise.\n\n### Stage 3: Fine-tuning Strategy\n\nFor training, I used the competition data and dropped the incomplete translations\n\n - Differential learning rates across model layers\n - Weight decay tuning for better generalization\n - Maximum token length of 3048\n - Training on an A100 GPU for efficient large-sequence handling\n\nThese optimizations helped the model converge better and capture long-context dependencies.\n\nThe final prediction was generated using an ensemble of the top three models.ensemble from public notebook.\n\n#### Approaches that did not work: \n - Pseudo-labeling, \n - training a ByT5 model from scratch, \n - adding LSTM/GRU layers as a custom head on top of T5.\n\n",
      "votes": 4
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3427614": "### Stage 1: Domain-Specific Pretraining\n\n - I performed pretraining on Akkadian text using a span masking objective, similar to masked language modeling but better suited for structured and partially missing sequences.\n - The goal was to adapt the model to the Akkadian language domain, improving its ability to understand context, symbols, and linguistic patterns specific to the dataset.\n\n- Tokens like **`<extra_id_0>`, `<extra_id_1>`, `<extra_id_2>`** are treated as **special symbols** during training.\n- These are **not natural language words**, but **placeholders used for span masking**.\n\n- The model learns to **predict missing spans**, making it robust to noisy and incomplete sequences.\n\n'a-na 8 ha-am-ša-t **`<extra_id_0>` i-ša-qal šu-ma i-na u4-me-šu m `<extra_id_1>` -al-ú-tim lá iš-qú-ul <extra_id_2> -ma a-wa-at kà-ri-im ú <extra_id_3> GI en-nam-a-šur'\n\n**`<extra_id_0>`šar-dí-ú`<extra_id_1>` -qá-a-šu a-na a-wa- <extra_id_2> u-um kà-ni-ì <extra_id_3> ša a-šur ší-'\n\n\n### Stage 2: Data Preparation and Cleaning\n\nThe provided sentence data had significant alignment issues. Many lines were not properly segmented, which negatively affected training.\n\nTo address this:\n - I manually inspected and split the publish,sentence csv files.Created a clean and well-aligned training set.Focused on improving sequence consistency and reducing noise.\n\n### Stage 3: Fine-tuning Strategy\n\nFor training, I used the competition data and dropped the incomplete translations\n\n - Differential learning rates across model layers\n - Weight decay tuning for better generalization\n - Maximum token length of 3048\n - Training on an A100 GPU for efficient large-sequence handling\n\nThese optimizations helped the model converge better and capture long-context dependencies.\n\nThe final prediction was generated using an ensemble of the top three models.ensemble from public notebook.\n\n#### Approaches that did not work: \n - Pseudo-labeling, \n - training a ByT5 model from scratch, \n - adding LSTM/GRU layers as a custom head on top of T5.\n\n"
  },
  "complete": true,
  "source": "existing_discussion",
  "competition_id": 121150,
  "competition_slug": "deep-past-initiative-machine-translation",
  "team_id": 15297584,
  "solution_url": "/competitions/deep-past-initiative-machine-translation/writeups/97th-place"
}