{
  "id": 665949,
  "title": "How to handle dot h/H?",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/665949",
  "author_name": "",
  "post_date": "2026-01-04T16:15:49.115610100Z",
  "votes": 9,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hi all,</p>\n<p>Transliteration normalization is a challenge both train set and test set. I'm entering an endless loop with <strong>ḫ</strong> and <strong>Ḫ</strong> 😁</p>\n<ul>\n<li>We know that test set contains h and H but not ḫ nor Ḫ</li>\n<li>Our train set (train.csv, same for published_text.csv) contains zero h nor H but many ḫ and Ḫ</li>\n</ul>\n<p>How to manage this headache?</p>\n<ol>\n<li>Convert all ḫ / Ḫ into h / H in train set, no changes on test set.</li>\n<li>Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set.</li>\n<li>Try also variants reported <a href=\"https://www.kaggle.com/code/manwithacat/byt5-improved-v1-0-0-host-guidance\" target=\"_blank\">here</a> below in test set:</li>\n</ol>\n<pre><code># H variants\n\"h,\": \"ḫ\",\n\"H,\": \"Ḫ\",\n\".h\": \"ḫ\",\n\".H\": \"Ḫ\",\n\"hh\": \"ḫ\",\n\"HH\": \"Ḫ\",\n</code></pre>\n<p>Any feedback on these approaches or other are welcome.</p>\n<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Does your normalization function convert any ḫ or Ḫ to h / H? Is it possible in Akkadian transliteration to have h / H that are not ḫ / Ḫ?</p>",
  "messages": [
    {
      "id": "3386108",
      "postDate": "01/04/2026 16:15:49",
      "content": "<p>Hi all,</p>\n<p>Transliteration normalization is a challenge both train set and test set. I'm entering an endless loop with <strong>ḫ</strong> and <strong>Ḫ</strong> 😁</p>\n<ul>\n<li>We know that test set contains h and H but not ḫ nor Ḫ</li>\n<li>Our train set (train.csv, same for published_text.csv) contains zero h nor H but many ḫ and Ḫ</li>\n</ul>\n<p>How to manage this headache?</p>\n<ol>\n<li>Convert all ḫ / Ḫ into h / H in train set, no changes on test set.</li>\n<li>Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set.</li>\n<li>Try also variants reported <a href=\"https://www.kaggle.com/code/manwithacat/byt5-improved-v1-0-0-host-guidance\" target=\"_blank\">here</a> below in test set:</li>\n</ol>\n<pre><code># H variants\n\"h,\": \"ḫ\",\n\"H,\": \"Ḫ\",\n\".h\": \"ḫ\",\n\".H\": \"Ḫ\",\n\"hh\": \"ḫ\",\n\"HH\": \"Ḫ\",\n</code></pre>\n<p>Any feedback on these approaches or other are welcome.</p>\n<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Does your normalization function convert any ḫ or Ḫ to h / H? Is it possible in Akkadian transliteration to have h / H that are not ḫ / Ḫ?</p>",
      "rawMarkdown": "Hi all,\n\nTransliteration normalization is a challenge both train set and test set. I'm entering an endless loop with **ḫ** and **Ḫ** 😁\n\n- We know that test set contains h and H but not ḫ nor Ḫ\n- Our train set (train.csv, same for published_text.csv) contains zero h nor H but many ḫ and Ḫ\n\nHow to manage this headache?\n\n1. Convert all ḫ / Ḫ into h / H in train set, no changes on test set.\n2. Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set.\n3. Try also variants reported [here](https://www.kaggle.com/code/manwithacat/byt5-improved-v1-0-0-host-guidance) below in test set:\n\n```python\n# H variants\n\"h,\": \"ḫ\",\n\"H,\": \"Ḫ\",\n\".h\": \"ḫ\",\n\".H\": \"Ḫ\",\n\"hh\": \"ḫ\",\n\"HH\": \"Ḫ\",\n```\n\nAny feedback on these approaches or other are welcome.\n\n@deeppast Does your normalization function convert any ḫ or Ḫ to h / H? Is it possible in Akkadian transliteration to have h / H that are not ḫ / Ḫ?",
      "votes": null
    },
    {
      "id": "3386127",
      "postDate": "01/04/2026 16:52:31",
      "content": "<p>well well well 😄 have you tried any of these options?</p>",
      "rawMarkdown": "well well well 😄 have you tried any of these options?",
      "votes": null
    },
    {
      "id": "3386149",
      "postDate": "01/04/2026 17:26:58",
      "content": "<p>I'm trying all these but no real changes:</p>\n<ol>\n<li>Baseline: Keep ḫ / Ḫ in train set, no change on test set: LB=34.0</li>\n<li>Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) on test set: LB=34.0+ </li>\n<li>Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) + convert remaining h / H to ḫ / Ḫ in test set: LB=34.0-</li>\n<li>Baseline: Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set: LB=34.0--</li>\n<li>Retrain: Convert all ḫ / Ḫ into h / H in train set, no changes on test set: LB=33.5</li>\n</ol>\n<p>Last one sounds interesting but I believe it looses information because LB drops a bit. I think there is no \"plain h\" in the Akkadian language. Experiment#2 with the H variant alone looks better but we would need to know if the transliteration normalization converts <strong>Ḫ</strong> into <strong>.H</strong> or <strong>H,</strong> or <strong>H</strong> <a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> ? </p>",
      "rawMarkdown": "I'm trying all these but no real changes:\n1. Baseline: Keep ḫ / Ḫ in train set, no change on test set: LB=34.0\n2. Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) on test set: LB=34.0+ \n3. Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) + convert remaining h / H to ḫ / Ḫ in test set: LB=34.0-\n4. Baseline: Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set: LB=34.0--\n5. Retrain: Convert all ḫ / Ḫ into h / H in train set, no changes on test set: LB=33.5\n\nLast one sounds interesting but I believe it looses information because LB drops a bit. I think there is no \"plain h\" in the Akkadian language. Experiment#2 with the H variant alone looks better but we would need to know if the transliteration normalization converts **Ḫ** into **.H** or **H,** or **H** @deeppast ?",
      "votes": null
    },
    {
      "id": "3387334",
      "postDate": "01/06/2026 18:41:28",
      "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Any information about your dot h/H transliteration normalization would help. Let's us know if you can provide it. Thanks a lot.</p>",
      "rawMarkdown": "deeppast Any information about your dot h/H transliteration normalization would help. Let's us know if you can provide it. Thanks a lot.",
      "votes": null
    },
    {
      "id": "3387348",
      "postDate": "01/06/2026 19:16:16",
      "content": "<p>Is this saying that keeping the ḫ / Ḫ makes your score 0.5 better?</p>",
      "rawMarkdown": "Is this saying that keeping the ḫ / Ḫ makes your score 0.5 better?",
      "votes": null
    },
    {
      "id": "3387363",
      "postDate": "01/06/2026 19:54:59",
      "content": "<p>Currently I keep ḫ / Ḫ as is in the training set knowing that they do not exist in test set and it looks better than replacing them by h / H (but it was on a single submission).</p>",
      "rawMarkdown": "Currently I keep ḫ / Ḫ as is in the training set knowing that they do not exist in test set and it looks better than replacing them by h / H (but it was on a single submission).",
      "votes": null
    },
    {
      "id": "3387452",
      "postDate": "01/07/2026 03:23:58",
      "content": "<p>Are you leaving anything else? I've spent countless hours going through this data and reading the competition recommendations only to make changes and see reduced performances.. I'd be curious to know how you're handling punctuation, fractions, and the text within angled brackets/parenthesis for the train.csv data. My recent experiments are definitely not giving outcomes I'd expect</p>",
      "rawMarkdown": "Are you leaving anything else? I've spent countless hours going through this data and reading the competition recommendations only to make changes and see reduced performances.. I'd be curious to know how you're handling punctuation, fractions, and the text within angled brackets/parenthesis for the train.csv data. My recent experiments are definitely not giving outcomes I'd expect",
      "votes": null
    },
    {
      "id": "3387816",
      "postDate": "01/07/2026 16:59:39",
      "content": "<p>\"We know that test set contains h and H but not ḫ nor Ḫ\" wait what? is this true?</p>",
      "rawMarkdown": "\"We know that test set contains h and H but not ḫ nor Ḫ\" wait what? is this true?",
      "votes": null
    },
    {
      "id": "3387824",
      "postDate": "01/07/2026 17:10:48",
      "content": "<p>Maybe,At least the characters published by the organizer do not show ḫ and Ḫ.  <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209#3384200\" target=\"_blank\">there</a></p>",
      "rawMarkdown": "Maybe,At least the characters published by the organizer do not show ḫ and Ḫ.  [there](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209#3384200)",
      "votes": null
    },
    {
      "id": "3387917",
      "postDate": "01/07/2026 19:45:05",
      "content": "<p>I hope, the allowed chars list has been provided by organizer.</p>",
      "rawMarkdown": "I hope, the allowed chars list has been provided by organizer.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3386127,
      "author_name": "nlztrk",
      "author_url": "",
      "post_date": "01/04/2026 16:52:31",
      "content": "<p>well well well 😄 have you tried any of these options?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3386149,
          "author_name": "mpware",
          "author_url": "",
          "post_date": "01/04/2026 17:26:58",
          "content": "<p>I'm trying all these but no real changes:</p>\n<ol>\n<li>Baseline: Keep ḫ / Ḫ in train set, no change on test set: LB=34.0</li>\n<li>Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) on test set: LB=34.0+ </li>\n<li>Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) + convert remaining h / H to ḫ / Ḫ in test set: LB=34.0-</li>\n<li>Baseline: Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set: LB=34.0--</li>\n<li>Retrain: Convert all ḫ / Ḫ into h / H in train set, no changes on test set: LB=33.5</li>\n</ol>\n<p>Last one sounds interesting but I believe it looses information because LB drops a bit. I think there is no \"plain h\" in the Akkadian language. Experiment#2 with the H variant alone looks better but we would need to know if the transliteration normalization converts <strong>Ḫ</strong> into <strong>.H</strong> or <strong>H,</strong> or <strong>H</strong> <a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> ? </p>",
          "votes": null,
          "replies": [
            {
              "id": 3387334,
              "author_name": "mpware",
              "author_url": "",
              "post_date": "01/06/2026 18:41:28",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> Any information about your dot h/H transliteration normalization would help. Let's us know if you can provide it. Thanks a lot.</p>",
              "votes": null,
              "replies": []
            },
            {
              "id": 3387348,
              "author_name": "jackvd",
              "author_url": "",
              "post_date": "01/06/2026 19:16:16",
              "content": "<p>Is this saying that keeping the ḫ / Ḫ makes your score 0.5 better?</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3387363,
                  "author_name": "mpware",
                  "author_url": "",
                  "post_date": "01/06/2026 19:54:59",
                  "content": "<p>Currently I keep ḫ / Ḫ as is in the training set knowing that they do not exist in test set and it looks better than replacing them by h / H (but it was on a single submission).</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3387452,
                      "author_name": "jackvd",
                      "author_url": "",
                      "post_date": "01/07/2026 03:23:58",
                      "content": "<p>Are you leaving anything else? I've spent countless hours going through this data and reading the competition recommendations only to make changes and see reduced performances.. I'd be curious to know how you're handling punctuation, fractions, and the text within angled brackets/parenthesis for the train.csv data. My recent experiments are definitely not giving outcomes I'd expect</p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 3387816,
      "author_name": "nofreewill",
      "author_url": "",
      "post_date": "01/07/2026 16:59:39",
      "content": "<p>\"We know that test set contains h and H but not ḫ nor Ḫ\" wait what? is this true?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3387824,
          "author_name": "qifeihhh666",
          "author_url": "",
          "post_date": "01/07/2026 17:10:48",
          "content": "<p>Maybe,At least the characters published by the organizer do not show ḫ and Ḫ.  <a href=\"https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209#3384200\" target=\"_blank\">there</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3387917,
          "author_name": "mpware",
          "author_url": "",
          "post_date": "01/07/2026 19:45:05",
          "content": "<p>I hope, the allowed chars list has been provided by organizer.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3386108": "Hi all,\n\nTransliteration normalization is a challenge both train set and test set. I'm entering an endless loop with **ḫ** and **Ḫ** 😁\n\n- We know that test set contains h and H but not ḫ nor Ḫ\n- Our train set (train.csv, same for published_text.csv) contains zero h nor H but many ḫ and Ḫ\n\nHow to manage this headache?\n\n1. Convert all ḫ / Ḫ into h / H in train set, no changes on test set.\n2. Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set.\n3. Try also variants reported [here](https://www.kaggle.com/code/manwithacat/byt5-improved-v1-0-0-host-guidance) below in test set:\n\n```python\n# H variants\n\"h,\": \"ḫ\",\n\"H,\": \"Ḫ\",\n\".h\": \"ḫ\",\n\".H\": \"Ḫ\",\n\"hh\": \"ḫ\",\n\"HH\": \"Ḫ\",\n```\n\nAny feedback on these approaches or other are welcome.\n\n@deeppast Does your normalization function convert any ḫ or Ḫ to h / H? Is it possible in Akkadian transliteration to have h / H that are not ḫ / Ḫ?",
    "3386127": "well well well 😄 have you tried any of these options?",
    "3386149": "I'm trying all these but no real changes:\n1. Baseline: Keep ḫ / Ḫ in train set, no change on test set: LB=34.0\n2. Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) on test set: LB=34.0+ \n3. Baseline: Keep ḫ / Ḫ in train set, option #3 (H variants) + convert remaining h / H to ḫ / Ḫ in test set: LB=34.0-\n4. Baseline: Keep ḫ / Ḫ in train set, convert h / H to ḫ / Ḫ in test set: LB=34.0--\n5. Retrain: Convert all ḫ / Ḫ into h / H in train set, no changes on test set: LB=33.5\n\nLast one sounds interesting but I believe it looses information because LB drops a bit. I think there is no \"plain h\" in the Akkadian language. Experiment#2 with the H variant alone looks better but we would need to know if the transliteration normalization converts **Ḫ** into **.H** or **H,** or **H** @deeppast ?",
    "3387334": "deeppast Any information about your dot h/H transliteration normalization would help. Let's us know if you can provide it. Thanks a lot.",
    "3387348": "Is this saying that keeping the ḫ / Ḫ makes your score 0.5 better?",
    "3387363": "Currently I keep ḫ / Ḫ as is in the training set knowing that they do not exist in test set and it looks better than replacing them by h / H (but it was on a single submission).",
    "3387452": "Are you leaving anything else? I've spent countless hours going through this data and reading the competition recommendations only to make changes and see reduced performances.. I'd be curious to know how you're handling punctuation, fractions, and the text within angled brackets/parenthesis for the train.csv data. My recent experiments are definitely not giving outcomes I'd expect",
    "3387816": "\"We know that test set contains h and H but not ḫ nor Ḫ\" wait what? is this true?",
    "3387824": "Maybe,At least the characters published by the organizer do not show ḫ and Ḫ.  [there](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209#3384200)",
    "3387917": "I hope, the allowed chars list has been provided by organizer."
  },
  "source": "meta"
}