{
  "id": 680041,
  "title": "0.78% Train Transliterations Found in eBL_Dictionary?",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/680041",
  "author_name": "",
  "post_date": "2026-03-05T13:33:33.083639700Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I imagine I am doing something obviously wrong here, but I just don't know what:</p>\n<p>Of 11,583 unique transliterations in the train, only 9 are in <code>eBL_Dictionary.csv</code>.  </p>\n<pre><code>#transliterations in Train (with frequency counts)\nword_counts = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/train.csv')\\\n    ['transliteration'].str.split(expand=True).stack().reset_index()[0].value_counts()\\\n    .reset_index().rename(columns={0:'transliteration'}) \n\n#cleaned words from eBL_Dictionary\ndct = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/eBL_Dictionary.csv')\ndct['word2'] = dct['word'].str.replace(r'\\s+I$', '', regex=True) #remove the ' |' at the end of the word\n\n#matching\nword_counts['in_dictionary'] = word_counts['transliteration'].isin(dct['word2'].values)\nprint(word_counts.in_dictionary.mean())\n&gt;&gt; 0.000777\n</code></pre>\n<p>Below are the words that appear in dictionary from train, along with their frequency counts:</p>\n<pre><code>transliteration    count   \n    ša  3441    \n    i   453 \n    la  56  \n    šu  34  \n    iš  10  \n    wa  4   \n    u   2   \n    ul  2   \n    mu  2   \n</code></pre>",
  "messages": [
    {
      "id": "3417473",
      "postDate": "03/05/2026 13:33:33",
      "content": "<p>I imagine I am doing something obviously wrong here, but I just don't know what:</p>\n<p>Of 11,583 unique transliterations in the train, only 9 are in <code>eBL_Dictionary.csv</code>.  </p>\n<pre><code>#transliterations in Train (with frequency counts)\nword_counts = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/train.csv')\\\n    ['transliteration'].str.split(expand=True).stack().reset_index()[0].value_counts()\\\n    .reset_index().rename(columns={0:'transliteration'}) \n\n#cleaned words from eBL_Dictionary\ndct = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/eBL_Dictionary.csv')\ndct['word2'] = dct['word'].str.replace(r'\\s+I$', '', regex=True) #remove the ' |' at the end of the word\n\n#matching\nword_counts['in_dictionary'] = word_counts['transliteration'].isin(dct['word2'].values)\nprint(word_counts.in_dictionary.mean())\n&gt;&gt; 0.000777\n</code></pre>\n<p>Below are the words that appear in dictionary from train, along with their frequency counts:</p>\n<pre><code>transliteration    count   \n    ša  3441    \n    i   453 \n    la  56  \n    šu  34  \n    iš  10  \n    wa  4   \n    u   2   \n    ul  2   \n    mu  2   \n</code></pre>",
      "rawMarkdown": "I imagine I am doing something obviously wrong here, but I just don't know what:\n\nOf 11,583 unique transliterations in the train, only 9 are in `eBL_Dictionary.csv`.  \n\n```\n#transliterations in Train (with frequency counts)\nword_counts = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/train.csv')\\\n    ['transliteration'].str.split(expand=True).stack().reset_index()[0].value_counts()\\\n    .reset_index().rename(columns={0:'transliteration'}) \n\n#cleaned words from eBL_Dictionary\ndct = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/eBL_Dictionary.csv')\ndct['word2'] = dct['word'].str.replace(r'\\s+I$', '', regex=True) #remove the ' |' at the end of the word\n\n#matching\nword_counts['in_dictionary'] = word_counts['transliteration'].isin(dct['word2'].values)\nprint(word_counts.in_dictionary.mean())\n>> 0.000777\n```\n\nBelow are the words that appear in dictionary from train, along with their frequency counts:\n```\n\ntransliteration\tcount\t\n\tša\t3441\t\n\ti\t453\t\n\tla\t56\t\n\tšu\t34\t\n\tiš\t10\t\n\twa\t4\t\n\tu\t2\t\n\tul\t2\t\n\tmu\t2\t\n```",
      "votes": null
    },
    {
      "id": "3417501",
      "postDate": "03/05/2026 14:12:50",
      "content": "<p>Hi,\nI think we should narrow the data down to records that include OA, since vocabulary from different periods is getting mixed in.\nAlso, regardless of whether a lemmatizer is actually effective for this competition, I don’t think the terms will match the dictionary unless we lemmatize them.</p>",
      "rawMarkdown": "Hi,\nI think we should narrow the data down to records that include OA, since vocabulary from different periods is getting mixed in.\nAlso, regardless of whether a lemmatizer is actually effective for this competition, I don’t think the terms will match the dictionary unless we lemmatize them.",
      "votes": null
    },
    {
      "id": "3417658",
      "postDate": "03/05/2026 23:08:52",
      "content": "<p>I think they just use different transliteration systems, and also the words can take different forms depending on the grammatical context. I can usually find the words when I'm looking for translations. </p>",
      "rawMarkdown": "I think they just use different transliteration systems, and also the words can take different forms depending on the grammatical context. I can usually find the words when I'm looking for translations.",
      "votes": null
    },
    {
      "id": "3417995",
      "postDate": "03/06/2026 19:36:55",
      "content": "<p>I use a tool to find matches: <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/680256\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/680256</a></p>",
      "rawMarkdown": "I use a tool to find matches: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/680256",
      "votes": null
    },
    {
      "id": "3421591",
      "postDate": "03/15/2026 22:07:53",
      "content": "<p>You need to use the dictionary \"via\" the Lexicon. You can't just lookup the words in the transliteration directly in the dictionary.</p>",
      "rawMarkdown": "You need to use the dictionary \"via\" the Lexicon. You can't just lookup the words in the transliteration directly in the dictionary.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3417501,
      "author_name": "fx6300",
      "author_url": "",
      "post_date": "03/05/2026 14:12:50",
      "content": "<p>Hi,\nI think we should narrow the data down to records that include OA, since vocabulary from different periods is getting mixed in.\nAlso, regardless of whether a lemmatizer is actually effective for this competition, I don’t think the terms will match the dictionary unless we lemmatize them.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3417658,
      "author_name": "tennogh",
      "author_url": "",
      "post_date": "03/05/2026 23:08:52",
      "content": "<p>I think they just use different transliteration systems, and also the words can take different forms depending on the grammatical context. I can usually find the words when I'm looking for translations. </p>",
      "votes": null,
      "replies": [
        {
          "id": 3417995,
          "author_name": "davidochoacorrales",
          "author_url": "",
          "post_date": "03/06/2026 19:36:55",
          "content": "<p>I use a tool to find matches: <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/680256\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/680256</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3421591,
      "author_name": "einherjer",
      "author_url": "",
      "post_date": "03/15/2026 22:07:53",
      "content": "<p>You need to use the dictionary \"via\" the Lexicon. You can't just lookup the words in the transliteration directly in the dictionary.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3417473": "I imagine I am doing something obviously wrong here, but I just don't know what:\n\nOf 11,583 unique transliterations in the train, only 9 are in `eBL_Dictionary.csv`.  \n\n```\n#transliterations in Train (with frequency counts)\nword_counts = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/train.csv')\\\n    ['transliteration'].str.split(expand=True).stack().reset_index()[0].value_counts()\\\n    .reset_index().rename(columns={0:'transliteration'}) \n\n#cleaned words from eBL_Dictionary\ndct = pd.read_csv('/kaggle/input/competitions/deep-past-initiative-machine-translation/eBL_Dictionary.csv')\ndct['word2'] = dct['word'].str.replace(r'\\s+I$', '', regex=True) #remove the ' |' at the end of the word\n\n#matching\nword_counts['in_dictionary'] = word_counts['transliteration'].isin(dct['word2'].values)\nprint(word_counts.in_dictionary.mean())\n>> 0.000777\n```\n\nBelow are the words that appear in dictionary from train, along with their frequency counts:\n```\n\ntransliteration\tcount\t\n\tša\t3441\t\n\ti\t453\t\n\tla\t56\t\n\tšu\t34\t\n\tiš\t10\t\n\twa\t4\t\n\tu\t2\t\n\tul\t2\t\n\tmu\t2\t\n```",
    "3417501": "Hi,\nI think we should narrow the data down to records that include OA, since vocabulary from different periods is getting mixed in.\nAlso, regardless of whether a lemmatizer is actually effective for this competition, I don’t think the terms will match the dictionary unless we lemmatize them.",
    "3417658": "I think they just use different transliteration systems, and also the words can take different forms depending on the grammatical context. I can usually find the words when I'm looking for translations.",
    "3417995": "I use a tool to find matches: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/680256",
    "3421591": "You need to use the dictionary \"via\" the Lexicon. You can't just lookup the words in the transliteration directly in the dictionary."
  },
  "source": "meta"
}