{
  "id": 674708,
  "title": "How should we tokenize?",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/674708",
  "author_name": "",
  "post_date": "2026-02-21T17:57:09.586108900Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I’m not participating in this competition,<br>\nbut it feels like the real challenge begins when we tokenize Akkadian text. </p>\n<p>I think experimenting with tokenizers and transformer architectures<br>\nmight be the easier direction for this competition.</p>\n<p>What’s the best way to get better at it?  </p>",
  "messages": [
    {
      "id": "3408953",
      "postDate": "02/21/2026 17:57:09",
      "content": "<p>I’m not participating in this competition,<br>\nbut it feels like the real challenge begins when we tokenize Akkadian text. </p>\n<p>I think experimenting with tokenizers and transformer architectures<br>\nmight be the easier direction for this competition.</p>\n<p>What’s the best way to get better at it?  </p>",
      "rawMarkdown": "I’m not participating in this competition,   \nbut it feels like the real challenge begins when we tokenize Akkadian text. \n\nI think experimenting with tokenizers and transformer architectures   \nmight be the easier direction for this competition.\n\nWhat’s the best way to get better at it?",
      "votes": null
    },
    {
      "id": "3409133",
      "postDate": "02/22/2026 07:42:34",
      "content": "<p>best LB public notebooks use byteT5 model that operates directly on UTF-8 bytes without use of a explicit tokenizer</p>",
      "rawMarkdown": "best LB public notebooks use byteT5 model that operates directly on UTF-8 bytes without use of a explicit tokenizer",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3409133,
      "author_name": "steubk",
      "author_url": "",
      "post_date": "02/22/2026 07:42:34",
      "content": "<p>best LB public notebooks use byteT5 model that operates directly on UTF-8 bytes without use of a explicit tokenizer</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3408953": "I’m not participating in this competition,   \nbut it feels like the real challenge begins when we tokenize Akkadian text. \n\nI think experimenting with tokenizers and transformer architectures   \nmight be the easier direction for this competition.\n\nWhat’s the best way to get better at it?",
    "3409133": "best LB public notebooks use byteT5 model that operates directly on UTF-8 bytes without use of a explicit tokenizer"
  },
  "source": "meta"
}