{
  "id": 666431,
  "title": "Use Sentences_Oare_FirstWord_LinNum to extract the sentence alignment",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/666431",
  "author_name": "MJeremy",
  "post_date": "2026-01-07T05:22:05.073000",
  "votes": 10,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I am trying to use the <code>Sentences_Oare_FirstWord_LinNum</code> to get the sentence pairs in format <code>(translit_sentence, translation_sent)</code> for training.</p>\n<p>Here is the dataset and code I used</p>\n<p><a href=\"https://www.kaggle.com/code/zhangyue199/dpc-sentence-alignment-oare-firstword/output?scriptVersionId=290466968\" target=\"_blank\">https://www.kaggle.com/code/zhangyue199/dpc-sentence-alignment-oare-firstword/output?scriptVersionId=290466968</a></p>\n<h1>The Idea</h1>\n<p>The goal is to get <code>(translit_sentence, translation_sent)</code> pairs. In the extra provided data set, it has <code>first_word_number</code>, which corresponds to the first word of the transliteration sentence, and one <code>oare_id</code> text can spans into multiple sentences, so I took</p>\n<pre><code>start_index = current_row[first_word_number - 1]\nend_index = next_row[first_word_number-1]\ntransliteration_sent = transliteration[start_index:end_index]\n</code></pre>\n<p>I realise there's some mismatchs in the word or index caused by <code>&lt;gap&gt;</code> or other matters, but I assume <strong>these matches are not too far off and would not affect training too much</strong>. (This might be false assumption)</p>\n<p>For translation, I found the translation provided by the new data is editorial and has abbreviations that doe not match the original English translations, for example, in row 1, the translation is</p>\n<pre><code>Seals of Mannum-bālum-Aššur s. ?, Šū-Enlil s. ?, Puzur-Aššur s. Attaya \n</code></pre>\n<p>But the actual correponding translation should be </p>\n<pre><code>Seal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.\n</code></pre>\n<p>There's no great way to match the translation, so I used <code>CHRF</code> score to</p>\n<ul>\n<li>Split the original translation into smaller chunks.</li>\n<li>Compare each chunk with the provided translation using the <code>CHRF</code> score.</li>\n<li>Pick the one with the highest score</li>\n<li>Consider expand the chunk to the next if expansion gives a higher score.</li>\n</ul>\n<p>Not sure my assumptions are correct, so feel free to use and point it out for improvement.</p>",
  "messages": [
    {
      "id": 3387479,
      "postDate": "2026-01-07T05:22:05.073Z",
      "content": "<p>I am trying to use the <code>Sentences_Oare_FirstWord_LinNum</code> to get the sentence pairs in format <code>(translit_sentence, translation_sent)</code> for training.</p>\n<p>Here is the dataset and code I used</p>\n<p><a href=\"https://www.kaggle.com/code/zhangyue199/dpc-sentence-alignment-oare-firstword/output?scriptVersionId=290466968\" target=\"_blank\">https://www.kaggle.com/code/zhangyue199/dpc-sentence-alignment-oare-firstword/output?scriptVersionId=290466968</a></p>\n<h1>The Idea</h1>\n<p>The goal is to get <code>(translit_sentence, translation_sent)</code> pairs. In the extra provided data set, it has <code>first_word_number</code>, which corresponds to the first word of the transliteration sentence, and one <code>oare_id</code> text can spans into multiple sentences, so I took</p>\n<pre><code>start_index = current_row[first_word_number - 1]\nend_index = next_row[first_word_number-1]\ntransliteration_sent = transliteration[start_index:end_index]\n</code></pre>\n<p>I realise there's some mismatchs in the word or index caused by <code>&lt;gap&gt;</code> or other matters, but I assume <strong>these matches are not too far off and would not affect training too much</strong>. (This might be false assumption)</p>\n<p>For translation, I found the translation provided by the new data is editorial and has abbreviations that doe not match the original English translations, for example, in row 1, the translation is</p>\n<pre><code>Seals of Mannum-bālum-Aššur s. ?, Šū-Enlil s. ?, Puzur-Aššur s. Attaya \n</code></pre>\n<p>But the actual correponding translation should be </p>\n<pre><code>Seal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.\n</code></pre>\n<p>There's no great way to match the translation, so I used <code>CHRF</code> score to</p>\n<ul>\n<li>Split the original translation into smaller chunks.</li>\n<li>Compare each chunk with the provided translation using the <code>CHRF</code> score.</li>\n<li>Pick the one with the highest score</li>\n<li>Consider expand the chunk to the next if expansion gives a higher score.</li>\n</ul>\n<p>Not sure my assumptions are correct, so feel free to use and point it out for improvement.</p>",
      "rawMarkdown": "I am trying to use the `Sentences_Oare_FirstWord_LinNum` to get the sentence pairs in format `(translit_sentence, translation_sent)` for training.\n\nHere is the dataset and code I used\n\nhttps://www.kaggle.com/code/zhangyue199/dpc-sentence-alignment-oare-firstword/output?scriptVersionId=290466968\n\n# The Idea\nThe goal is to get `(translit_sentence, translation_sent)` pairs. In the extra provided data set, it has `first_word_number`, which corresponds to the first word of the transliteration sentence, and one `oare_id` text can spans into multiple sentences, so I took\n\n```\nstart_index = current_row[first_word_number - 1]\nend_index = next_row[first_word_number-1]\ntransliteration_sent = transliteration[start_index:end_index]\n```\n\nI realise there's some mismatchs in the word or index caused by `<gap>` or other matters, but I assume __these matches are not too far off and would not affect training too much__. (This might be false assumption)\n\nFor translation, I found the translation provided by the new data is editorial and has abbreviations that doe not match the original English translations, for example, in row 1, the translation is\n```\nSeals of Mannum-bālum-Aššur s. ?, Šū-Enlil s. ?, Puzur-Aššur s. Attaya \n```\nBut the actual correponding translation should be \n```\nSeal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.\n```\n\nThere's no great way to match the translation, so I used `CHRF` score to\n\n- Split the original translation into smaller chunks.\n- Compare each chunk with the provided translation using the `CHRF` score.\n- Pick the one with the highest score\n- Consider expand the chunk to the next if expansion gives a higher score.\n\nNot sure my assumptions are correct, so feel free to use and point it out for improvement.",
      "votes": 10
    },
    {
      "id": 3387831,
      "postDate": "2026-01-07T17:24:21.150Z",
      "content": "<p>Yes, your assumptions appear to me to be correct, and similar to what those in this discussion on the same topic here: <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664948#3384626\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664948#3384626</a></p>\n<p>Exact alignment might be difficult because of the numbers. Some of that would need supervision. Additionally, some of the translations have these abbreviations, as you note in the editorializing, and your correction of the translation is wise. Soon we hope to release the PDFs from which these translations were taken / typed into a database (by students). Hopefully having the original text translations will help resolve these details.</p>",
      "rawMarkdown": "Yes, your assumptions appear to me to be correct, and similar to what those in this discussion on the same topic here: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664948#3384626\n\nExact alignment might be difficult because of the numbers. Some of that would need supervision. Additionally, some of the translations have these abbreviations, as you note in the editorializing, and your correction of the translation is wise. Soon we hope to release the PDFs from which these translations were taken / typed into a database (by students). Hopefully having the original text translations will help resolve these details.",
      "votes": 2,
      "replies": [
        {
          "id": 3388041,
          "postDate": "2026-01-08T03:04:40.560Z",
          "content": "<p>got it thanks for the affirmation.</p>",
          "rawMarkdown": "got it thanks for the affirmation."
        },
        {
          "id": 3390694,
          "postDate": "2026-01-13T18:41:40.873Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 3404174,
      "postDate": "2026-02-10T01:49:39.290Z",
      "content": "<p>There is a problem with the translation you sent:\nSome of the samples contain only one word.</p>",
      "rawMarkdown": "There is a problem with the translation you sent:\nSome of the samples contain only one word.",
      "replies": [
        {
          "id": 3404300,
          "postDate": "2026-02-10T08:35:16.530Z",
          "content": "<p>Some <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663849\" target=\"_blank\">translations/transliterations are incomplete</a>. There have been several more discussions on the topic.</p>",
          "rawMarkdown": "Some [translations/transliterations are incomplete](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663849). There have been several more discussions on the topic."
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 3387831,
      "author_name": "Adam Anderson",
      "author_url": "",
      "post_date": "2026-01-07T17:24:21.150000",
      "content": "<p>Yes, your assumptions appear to me to be correct, and similar to what those in this discussion on the same topic here: <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664948#3384626\" target=\"_blank\">https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664948#3384626</a></p>\n<p>Exact alignment might be difficult because of the numbers. Some of that would need supervision. Additionally, some of the translations have these abbreviations, as you note in the editorializing, and your correction of the translation is wise. Soon we hope to release the PDFs from which these translations were taken / typed into a database (by students). Hopefully having the original text translations will help resolve these details.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 3388041,
          "author_name": "MJeremy",
          "author_url": "",
          "post_date": "2026-01-08T03:04:40.560000",
          "content": "<p>got it thanks for the affirmation.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 3390694,
          "author_name": "",
          "author_url": "",
          "post_date": "2026-01-13T18:41:40.873000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3404174,
      "author_name": "water joe",
      "author_url": "",
      "post_date": "2026-02-10T01:49:39.290000",
      "content": "<p>There is a problem with the translation you sent:\nSome of the samples contain only one word.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3404300,
          "author_name": "Angantyr",
          "author_url": "",
          "post_date": "2026-02-10T08:35:16.530000",
          "content": "<p>Some <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663849\" target=\"_blank\">translations/transliterations are incomplete</a>. There have been several more discussions on the topic.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3387479": "I am trying to use the `Sentences_Oare_FirstWord_LinNum` to get the sentence pairs in format `(translit_sentence, translation_sent)` for training.\n\nHere is the dataset and code I used\n\nhttps://www.kaggle.com/code/zhangyue199/dpc-sentence-alignment-oare-firstword/output?scriptVersionId=290466968\n\n# The Idea\nThe goal is to get `(translit_sentence, translation_sent)` pairs. In the extra provided data set, it has `first_word_number`, which corresponds to the first word of the transliteration sentence, and one `oare_id` text can spans into multiple sentences, so I took\n\n```\nstart_index = current_row[first_word_number - 1]\nend_index = next_row[first_word_number-1]\ntransliteration_sent = transliteration[start_index:end_index]\n```\n\nI realise there's some mismatchs in the word or index caused by `<gap>` or other matters, but I assume __these matches are not too far off and would not affect training too much__. (This might be false assumption)\n\nFor translation, I found the translation provided by the new data is editorial and has abbreviations that doe not match the original English translations, for example, in row 1, the translation is\n```\nSeals of Mannum-bālum-Aššur s. ?, Šū-Enlil s. ?, Puzur-Aššur s. Attaya \n```\nBut the actual correponding translation should be \n```\nSeal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.\n```\n\nThere's no great way to match the translation, so I used `CHRF` score to\n\n- Split the original translation into smaller chunks.\n- Compare each chunk with the provided translation using the `CHRF` score.\n- Pick the one with the highest score\n- Consider expand the chunk to the next if expansion gives a higher score.\n\nNot sure my assumptions are correct, so feel free to use and point it out for improvement.",
    "3387831": "Yes, your assumptions appear to me to be correct, and similar to what those in this discussion on the same topic here: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/664948#3384626\n\nExact alignment might be difficult because of the numbers. Some of that would need supervision. Additionally, some of the translations have these abbreviations, as you note in the editorializing, and your correction of the translation is wise. Soon we hope to release the PDFs from which these translations were taken / typed into a database (by students). Hopefully having the original text translations will help resolve these details.",
    "3404174": "There is a problem with the translation you sent:\nSome of the samples contain only one word."
  }
}