{
  "id": 681734,
  "title": "dictionary-based pre-training",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/681734",
  "author_name": "WOOSUNG YOON",
  "post_date": "2026-03-16T21:11:57.709000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4808143%2Ff054ddd564050dda34b4ea560ceb7525%2Fabbb.png?generation=1773696248416522&amp;alt=media\" alt=\"\"></p>\n<ol>\n<li>BDLM Paper Summary (arXiv:2103.07040, 2021)</li>\n</ol>\n<h3>Title</h3>\n<p>Bilingual Dictionary-based Language Model Pretraining for Neural Machine Translation (BDLM)</p>\n<h3>Core Idea</h3>\n<p>Instead of expensive parallel corpora, pre-train a language model using only monolingual text + bilingual dictionary.</p>\n<p>The model learns translation knowledge during pre-training and is then fine-tuned for NMT.</p>\n<h3>Key Method (BDLM)</h3>\n<p>Uses one Transformer with 3 pre-training objectives:</p>\n<ul>\n<li>MLM (Masked LM)</li>\n<li>RLM (Replaced LM – replace word with its dictionary translation)</li>\n<li>IPLM (predict translation / POS / synonyms / definition / NE)</li>\n</ul>\n<p>No parallel sentences needed.</p>\n<h3>Main Results</h3>\n<ul>\n<li>Chinese-English (WMT News’19): 55.0 BLEU (+8.4 over vanilla Transformer)</li>\n<li>Faster convergence</li>\n<li>Much better at rare words and dictionary-covered terms</li>\n<li>Strong gains on low-resource Romanian-English</li>\n</ul>\n<h3>Why It Matters for Akkadian</h3>\n<ul>\n<li>Perfect for low-resource + specialized vocabulary tasks.</li>\n<li>Dictionary (e.g. CAD PDF) can be turned into high-quality pre-training data to improve name handling, commodity terms, and rare expressions.</li>\n</ul>",
  "messages": [
    {
      "id": 3422074,
      "postDate": "2026-03-16T21:11:57.710Z",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4808143%2Ff054ddd564050dda34b4ea560ceb7525%2Fabbb.png?generation=1773696248416522&amp;alt=media\" alt=\"\"></p>\n<ol>\n<li>BDLM Paper Summary (arXiv:2103.07040, 2021)</li>\n</ol>\n<h3>Title</h3>\n<p>Bilingual Dictionary-based Language Model Pretraining for Neural Machine Translation (BDLM)</p>\n<h3>Core Idea</h3>\n<p>Instead of expensive parallel corpora, pre-train a language model using only monolingual text + bilingual dictionary.</p>\n<p>The model learns translation knowledge during pre-training and is then fine-tuned for NMT.</p>\n<h3>Key Method (BDLM)</h3>\n<p>Uses one Transformer with 3 pre-training objectives:</p>\n<ul>\n<li>MLM (Masked LM)</li>\n<li>RLM (Replaced LM – replace word with its dictionary translation)</li>\n<li>IPLM (predict translation / POS / synonyms / definition / NE)</li>\n</ul>\n<p>No parallel sentences needed.</p>\n<h3>Main Results</h3>\n<ul>\n<li>Chinese-English (WMT News’19): 55.0 BLEU (+8.4 over vanilla Transformer)</li>\n<li>Faster convergence</li>\n<li>Much better at rare words and dictionary-covered terms</li>\n<li>Strong gains on low-resource Romanian-English</li>\n</ul>\n<h3>Why It Matters for Akkadian</h3>\n<ul>\n<li>Perfect for low-resource + specialized vocabulary tasks.</li>\n<li>Dictionary (e.g. CAD PDF) can be turned into high-quality pre-training data to improve name handling, commodity terms, and rare expressions.</li>\n</ul>",
      "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4808143%2Ff054ddd564050dda34b4ea560ceb7525%2Fabbb.png?generation=1773696248416522&alt=media)\n\n1. BDLM Paper Summary (arXiv:2103.07040, 2021)\n\n### Title   \nBilingual Dictionary-based Language Model Pretraining for Neural Machine Translation (BDLM)\n\n### Core Idea  \nInstead of expensive parallel corpora, pre-train a language model using only monolingual text + bilingual dictionary.\n\nThe model learns translation knowledge during pre-training and is then fine-tuned for NMT.\n\n### Key Method (BDLM)\nUses one Transformer with 3 pre-training objectives:\n\n- MLM (Masked LM)\n- RLM (Replaced LM – replace word with its dictionary translation)\n- IPLM (predict translation / POS / synonyms / definition / NE)\n\nNo parallel sentences needed.\n\n### Main Results\n\n- Chinese-English (WMT News’19): 55.0 BLEU (+8.4 over vanilla Transformer)\n- Faster convergence\n- Much better at rare words and dictionary-covered terms\n- Strong gains on low-resource Romanian-English\n\n\n### Why It Matters for Akkadian\n- Perfect for low-resource + specialized vocabulary tasks.\n- Dictionary (e.g. CAD PDF) can be turned into high-quality pre-training data to improve name handling, commodity terms, and rare expressions.\n\n",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3422074": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4808143%2Ff054ddd564050dda34b4ea560ceb7525%2Fabbb.png?generation=1773696248416522&alt=media)\n\n1. BDLM Paper Summary (arXiv:2103.07040, 2021)\n\n### Title   \nBilingual Dictionary-based Language Model Pretraining for Neural Machine Translation (BDLM)\n\n### Core Idea  \nInstead of expensive parallel corpora, pre-train a language model using only monolingual text + bilingual dictionary.\n\nThe model learns translation knowledge during pre-training and is then fine-tuned for NMT.\n\n### Key Method (BDLM)\nUses one Transformer with 3 pre-training objectives:\n\n- MLM (Masked LM)\n- RLM (Replaced LM – replace word with its dictionary translation)\n- IPLM (predict translation / POS / synonyms / definition / NE)\n\nNo parallel sentences needed.\n\n### Main Results\n\n- Chinese-English (WMT News’19): 55.0 BLEU (+8.4 over vanilla Transformer)\n- Faster convergence\n- Much better at rare words and dictionary-covered terms\n- Strong gains on low-resource Romanian-English\n\n\n### Why It Matters for Akkadian\n- Perfect for low-resource + specialized vocabulary tasks.\n- Dictionary (e.g. CAD PDF) can be turned into high-quality pre-training data to improve name handling, commodity terms, and rare expressions.\n\n"
  }
}