{
  "id": 684242,
  "title": "Faster decoding",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/684242",
  "author_name": "",
  "post_date": "2026-03-24T06:57:18.166331800Z",
  "votes": 15,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I don't have much to share except this maybe. I noticed that decoding with the byt5 tokenizer was quite slow. There is a much faster way, given encoding is just adding 3 to each byte:</p>\n<pre><code>def decode(output):\n    bstring = bytes((o - 3) for o in output if o &gt; 3)\n    try:\n        return bstring.decode('utf-8', errors='ignore').strip()\n    except:\n        return ''\n</code></pre>\n<p>I put the try for safety, not sure it is required. Error handling is require though, as the generated strings may contain illegal utf-8 sequences.</p>\n<p>Regarding the comp, I joined after the last train update and had little time to process additional data. The one thing I tried was to scrap all the text mentioned in the sequence file from <a href=\"https://oare.byu.edu/epigraphies/\" target=\"_blank\">https://oare.byu.edu/epigraphies/</a> but adding these did not help. </p>",
  "messages": [
    {
      "id": "3427505",
      "postDate": "03/24/2026 06:57:18",
      "content": "<p>I don't have much to share except this maybe. I noticed that decoding with the byt5 tokenizer was quite slow. There is a much faster way, given encoding is just adding 3 to each byte:</p>\n<pre><code>def decode(output):\n    bstring = bytes((o - 3) for o in output if o &gt; 3)\n    try:\n        return bstring.decode('utf-8', errors='ignore').strip()\n    except:\n        return ''\n</code></pre>\n<p>I put the try for safety, not sure it is required. Error handling is require though, as the generated strings may contain illegal utf-8 sequences.</p>\n<p>Regarding the comp, I joined after the last train update and had little time to process additional data. The one thing I tried was to scrap all the text mentioned in the sequence file from <a href=\"https://oare.byu.edu/epigraphies/\" target=\"_blank\">https://oare.byu.edu/epigraphies/</a> but adding these did not help. </p>",
      "rawMarkdown": "I don't have much to share except this maybe. I noticed that decoding with the byt5 tokenizer was quite slow. There is a much faster way, given encoding is just adding 3 to each byte:\n\n```python\ndef decode(output):\n    bstring = bytes((o - 3) for o in output if o > 3)\n    try:\n        return bstring.decode('utf-8', errors='ignore').strip()\n    except:\n        return ''\n\n```\nI put the try for safety, not sure it is required. Error handling is require though, as the generated strings may contain illegal utf-8 sequences.\n\nRegarding the comp, I joined after the last train update and had little time to process additional data. The one thing I tried was to scrap all the text mentioned in the sequence file from https://oare.byu.edu/epigraphies/ but adding these did not help.",
      "votes": null
    },
    {
      "id": "3427521",
      "postDate": "03/24/2026 07:32:07",
      "content": "<p>That's a neat insight. How big of an increase in inference speed did you find?</p>",
      "rawMarkdown": "That's a neat insight. How big of an increase in inference speed did you find?",
      "votes": null
    },
    {
      "id": "3427522",
      "postDate": "03/24/2026 07:34:18",
      "content": "<p>At least 25%</p>",
      "rawMarkdown": "At least 25%",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3427521,
      "author_name": "angantyr",
      "author_url": "",
      "post_date": "03/24/2026 07:32:07",
      "content": "<p>That's a neat insight. How big of an increase in inference speed did you find?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3427522,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "03/24/2026 07:34:18",
          "content": "<p>At least 25%</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3427505": "I don't have much to share except this maybe. I noticed that decoding with the byt5 tokenizer was quite slow. There is a much faster way, given encoding is just adding 3 to each byte:\n\n```python\ndef decode(output):\n    bstring = bytes((o - 3) for o in output if o > 3)\n    try:\n        return bstring.decode('utf-8', errors='ignore').strip()\n    except:\n        return ''\n\n```\nI put the try for safety, not sure it is required. Error handling is require though, as the generated strings may contain illegal utf-8 sequences.\n\nRegarding the comp, I joined after the last train update and had little time to process additional data. The one thing I tried was to scrap all the text mentioned in the sequence file from https://oare.byu.edu/epigraphies/ but adding these did not help.",
    "3427521": "That's a neat insight. How big of an increase in inference speed did you find?",
    "3427522": "At least 25%"
  },
  "source": "meta"
}