{
  "id": 679766,
  "title": "Old Assyrian Tokens",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/679766",
  "author_name": "",
  "post_date": "2026-03-03T15:38:59.393688500Z",
  "votes": 7,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I made this short list: it contains all the syllables that appear in the OARE database. This can be useful in token problems; I use it with the unicodedata library in Python, simply normalizing it.</p>\n<p><a href=\"https://www.kaggle.com/datasets/davidochoacorrales/oare-tokens\" target=\"_blank\">https://www.kaggle.com/datasets/davidochoacorrales/oare-tokens</a></p>",
  "messages": [
    {
      "id": "3416734",
      "postDate": "03/03/2026 15:38:59",
      "content": "<p>I made this short list: it contains all the syllables that appear in the OARE database. This can be useful in token problems; I use it with the unicodedata library in Python, simply normalizing it.</p>\n<p><a href=\"https://www.kaggle.com/datasets/davidochoacorrales/oare-tokens\" target=\"_blank\">https://www.kaggle.com/datasets/davidochoacorrales/oare-tokens</a></p>",
      "rawMarkdown": "I made this short list: it contains all the syllables that appear in the OARE database. This can be useful in token problems; I use it with the unicodedata library in Python, simply normalizing it.\n\nhttps://www.kaggle.com/datasets/davidochoacorrales/oare-tokens",
      "votes": null
    },
    {
      "id": "3416735",
      "postDate": "03/03/2026 15:40:29",
      "content": "<h1>Unicode Normalization</h1>\n<pre><code>def canonical(text):\n    return unicodedata.normalize(\"NFC\", text)\n</code></pre>",
      "rawMarkdown": "# Unicode Normalization\n```python\ndef canonical(text):\n    return unicodedata.normalize(\"NFC\", text)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3416735,
      "author_name": "davidochoacorrales",
      "author_url": "",
      "post_date": "03/03/2026 15:40:29",
      "content": "<h1>Unicode Normalization</h1>\n<pre><code>def canonical(text):\n    return unicodedata.normalize(\"NFC\", text)\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3416734": "I made this short list: it contains all the syllables that appear in the OARE database. This can be useful in token problems; I use it with the unicodedata library in Python, simply normalizing it.\n\nhttps://www.kaggle.com/datasets/davidochoacorrales/oare-tokens",
    "3416735": "# Unicode Normalization\n```python\ndef canonical(text):\n    return unicodedata.normalize(\"NFC\", text)\n```"
  },
  "source": "meta"
}