{
  "id": 665366,
  "title": "Technical Summary: Overcoming Mode Collapse in Akkadian Translation",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/665366",
  "author_name": "Frank Morales",
  "post_date": "2025-12-31T23:12:52.001000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p><strong>The Challenge</strong></p>\n<p>During initial inference, the Transformer model exhibited <strong>Mode Collapse (Posterior Collapse)</strong>, where the decoder entered infinite loops or repeated a single \"safe\" high-frequency word (e.g., <em>\"naniya\"</em> or <em>\"accused\"</em>) for every input. This was likely caused by heavy <strong>Lexicon Anchoring</strong> and a highly constrained source vocabulary (503 tokens).</p>\n<p><strong>The Solution: Diversified Stochastic Decoding</strong></p>\n<p>To extract the knowledge trapped in the model's weights, I pivoted from standard Greedy Search to a <strong>Robust Nucleus Sampling</strong> strategy:</p>\n<p>1.** Nucleus (Top-P) Sampling ():** Instead of picking the single best word, the model sampled from the top 85% of the probability mass. This forced the decoder to expand its lexicon.</p>\n<ol>\n<li>**Temperature Scaling (): **I increased the temperature to \"flatten\" the probability distribution, making the model more creative and less prone to repeating the same \"safe\" tokens.</li>\n<li>**Dynamic Repetition Penalty (): **I implemented a penalty that de-prioritizes tokens already present in the current translation, effectively killing infinite loops.</li>\n<li>**CPU-Fallback Sampling: **To prevent <strong>CUDA device-side asserts</strong> caused by index clamping issues on the GPU, I moved the logit-processing and multinomial sampling steps to the CPU, ensuring 100% inference stability.</li>\n</ol>\n<h3><strong>Final Results</strong></h3>\n<ul>\n<li><strong>chrF++ Score:</strong> <strong>17.13</strong> (Indicating strong character-level and morphological alignment).</li>\n<li><strong>BLEU Score:</strong> <strong>1.87</strong> (Typical for keyword-based ancient language translation).</li>\n<li><strong>Bias Check:</strong> Successfully reduced \"Top Word\" frequency from 100% to <strong>12%</strong>, ensuring high diversity across the submission set.</li>\n</ul>\n<hr>",
  "messages": [
    {
      "id": 3384295,
      "postDate": "2025-12-31T23:12:52.003Z",
      "content": "<p><strong>The Challenge</strong></p>\n<p>During initial inference, the Transformer model exhibited <strong>Mode Collapse (Posterior Collapse)</strong>, where the decoder entered infinite loops or repeated a single \"safe\" high-frequency word (e.g., <em>\"naniya\"</em> or <em>\"accused\"</em>) for every input. This was likely caused by heavy <strong>Lexicon Anchoring</strong> and a highly constrained source vocabulary (503 tokens).</p>\n<p><strong>The Solution: Diversified Stochastic Decoding</strong></p>\n<p>To extract the knowledge trapped in the model's weights, I pivoted from standard Greedy Search to a <strong>Robust Nucleus Sampling</strong> strategy:</p>\n<p>1.** Nucleus (Top-P) Sampling ():** Instead of picking the single best word, the model sampled from the top 85% of the probability mass. This forced the decoder to expand its lexicon.</p>\n<ol>\n<li>**Temperature Scaling (): **I increased the temperature to \"flatten\" the probability distribution, making the model more creative and less prone to repeating the same \"safe\" tokens.</li>\n<li>**Dynamic Repetition Penalty (): **I implemented a penalty that de-prioritizes tokens already present in the current translation, effectively killing infinite loops.</li>\n<li>**CPU-Fallback Sampling: **To prevent <strong>CUDA device-side asserts</strong> caused by index clamping issues on the GPU, I moved the logit-processing and multinomial sampling steps to the CPU, ensuring 100% inference stability.</li>\n</ol>\n<h3><strong>Final Results</strong></h3>\n<ul>\n<li><strong>chrF++ Score:</strong> <strong>17.13</strong> (Indicating strong character-level and morphological alignment).</li>\n<li><strong>BLEU Score:</strong> <strong>1.87</strong> (Typical for keyword-based ancient language translation).</li>\n<li><strong>Bias Check:</strong> Successfully reduced \"Top Word\" frequency from 100% to <strong>12%</strong>, ensuring high diversity across the submission set.</li>\n</ul>\n<hr>",
      "rawMarkdown": "**The Challenge**\n\nDuring initial inference, the Transformer model exhibited **Mode Collapse (Posterior Collapse)**, where the decoder entered infinite loops or repeated a single \"safe\" high-frequency word (e.g., *\"naniya\"* or *\"accused\"*) for every input. This was likely caused by heavy **Lexicon Anchoring** and a highly constrained source vocabulary (503 tokens).\n\n**The Solution: Diversified Stochastic Decoding**\n\nTo extract the knowledge trapped in the model's weights, I pivoted from standard Greedy Search to a **Robust Nucleus Sampling** strategy:\n\n1.** Nucleus (Top-P) Sampling ():** Instead of picking the single best word, the model sampled from the top 85% of the probability mass. This forced the decoder to expand its lexicon.\n2. **Temperature Scaling (): **I increased the temperature to \"flatten\" the probability distribution, making the model more creative and less prone to repeating the same \"safe\" tokens.\n3. **Dynamic Repetition Penalty (): **I implemented a penalty that de-prioritizes tokens already present in the current translation, effectively killing infinite loops.\n4. **CPU-Fallback Sampling: **To prevent **CUDA device-side asserts** caused by index clamping issues on the GPU, I moved the logit-processing and multinomial sampling steps to the CPU, ensuring 100% inference stability.\n\n### **Final Results**\n\n* **chrF++ Score:** **17.13** (Indicating strong character-level and morphological alignment).\n* **BLEU Score:** **1.87** (Typical for keyword-based ancient language translation).\n* **Bias Check:** Successfully reduced \"Top Word\" frequency from 100% to **12%**, ensuring high diversity across the submission set.\n\n---",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3384295": "**The Challenge**\n\nDuring initial inference, the Transformer model exhibited **Mode Collapse (Posterior Collapse)**, where the decoder entered infinite loops or repeated a single \"safe\" high-frequency word (e.g., *\"naniya\"* or *\"accused\"*) for every input. This was likely caused by heavy **Lexicon Anchoring** and a highly constrained source vocabulary (503 tokens).\n\n**The Solution: Diversified Stochastic Decoding**\n\nTo extract the knowledge trapped in the model's weights, I pivoted from standard Greedy Search to a **Robust Nucleus Sampling** strategy:\n\n1.** Nucleus (Top-P) Sampling ():** Instead of picking the single best word, the model sampled from the top 85% of the probability mass. This forced the decoder to expand its lexicon.\n2. **Temperature Scaling (): **I increased the temperature to \"flatten\" the probability distribution, making the model more creative and less prone to repeating the same \"safe\" tokens.\n3. **Dynamic Repetition Penalty (): **I implemented a penalty that de-prioritizes tokens already present in the current translation, effectively killing infinite loops.\n4. **CPU-Fallback Sampling: **To prevent **CUDA device-side asserts** caused by index clamping issues on the GPU, I moved the logit-processing and multinomial sampling steps to the CPU, ensuring 100% inference stability.\n\n### **Final Results**\n\n* **chrF++ Score:** **17.13** (Indicating strong character-level and morphological alignment).\n* **BLEU Score:** **1.87** (Typical for keyword-based ancient language translation).\n* **Bias Check:** Successfully reduced \"Top Word\" frequency from 100% to **12%**, ensuring high diversity across the submission set.\n\n---"
  }
}