{
  "id": 668621,
  "title": "Translate Gemma",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/668621",
  "author_name": "",
  "post_date": "2026-01-17T20:32:50.382213Z",
  "votes": 8,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Translate Gemma model was released yesterday - <a href=\"https://huggingface.co/google/translategemma-4b-it\" target=\"_blank\">https://huggingface.co/google/translategemma-4b-it</a> </p>\n<p>Paper - <a href=\"https://arxiv.org/abs/2601.09012\" target=\"_blank\">https://arxiv.org/abs/2601.09012</a> </p>\n<p>This uses BPE tokenizer like Gemma unlike character based tokenizer for ByT5. </p>\n<p>Wondering if this would be useful for fine tuning here, thoughts ? </p>",
  "messages": [
    {
      "id": "3392932",
      "postDate": "01/17/2026 20:32:50",
      "content": "<p>Translate Gemma model was released yesterday - <a href=\"https://huggingface.co/google/translategemma-4b-it\" target=\"_blank\">https://huggingface.co/google/translategemma-4b-it</a> </p>\n<p>Paper - <a href=\"https://arxiv.org/abs/2601.09012\" target=\"_blank\">https://arxiv.org/abs/2601.09012</a> </p>\n<p>This uses BPE tokenizer like Gemma unlike character based tokenizer for ByT5. </p>\n<p>Wondering if this would be useful for fine tuning here, thoughts ? </p>",
      "rawMarkdown": "Translate Gemma model was released yesterday - https://huggingface.co/google/translategemma-4b-it \n\nPaper - https://arxiv.org/abs/2601.09012 \n\nThis uses BPE tokenizer like Gemma unlike character based tokenizer for ByT5. \n\nWondering if this would be useful for fine tuning here, thoughts ?",
      "votes": null
    },
    {
      "id": "3392947",
      "postDate": "01/17/2026 22:08:25",
      "content": "<p>I saw this too it’s worth a try I think the two biggest risks are the model size which often means slower fine tuning and the tokenizer even though it’s bpe might apply noisy merges to the Akkadian transliterations since it wasn’t trained on the language I’m not sure if there are enough data examples to improve the tokenizer. If you do try it let us know how it goes.</p>",
      "rawMarkdown": "I saw this too it’s worth a try I think the two biggest risks are the model size which often means slower fine tuning and the tokenizer even though it’s bpe might apply noisy merges to the Akkadian transliterations since it wasn’t trained on the language I’m not sure if there are enough data examples to improve the tokenizer. If you do try it let us know how it goes.",
      "votes": null
    },
    {
      "id": "3394255",
      "postDate": "01/20/2026 18:21:10",
      "content": "<p>I thought that as well, tried it, but it was painstakingly slow. Quantized it and trained a LoRa-Adapter, but could not make data parallel training on the T4s work. It was supposed to take around a full day and then I retreated.</p>\n<p>Previously trained such models on A100s, but I am still to cheap to buy more GPU power, and don’t know whether that would even be permitted for the competition.</p>",
      "rawMarkdown": "I thought that as well, tried it, but it was painstakingly slow. Quantized it and trained a LoRa-Adapter, but could not make data parallel training on the T4s work. It was supposed to take around a full day and then I retreated.\n\nPreviously trained such models on A100s, but I am still to cheap to buy more GPU power, and don’t know whether that would even be permitted for the competition.",
      "votes": null
    },
    {
      "id": "3394411",
      "postDate": "01/21/2026 01:57:05",
      "content": "<p>Its certainly permitted for training - you can use whatever compute you want. </p>\n<p>That said, this training didn’t work well for me. Generation was quite slow when using 4 beams, and even then the results weren’t very good .. so far.</p>",
      "rawMarkdown": "Its certainly permitted for training - you can use whatever compute you want. \n\nThat said, this training didn’t work well for me. Generation was quite slow when using 4 beams, and even then the results weren’t very good .. so far.",
      "votes": null
    },
    {
      "id": "3418145",
      "postDate": "03/07/2026 09:03:16",
      "content": "<p>Thanks for sharing your initial thoughts on TranslateGemma!\nSince we are in the final stretch of the competition, I was wondering if you or your team eventually found a way to make Decoder-only models competitive?</p>",
      "rawMarkdown": "Thanks for sharing your initial thoughts on TranslateGemma!\nSince we are in the final stretch of the competition, I was wondering if you or your team eventually found a way to make Decoder-only models competitive?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3392947,
      "author_name": "aaronbornstein",
      "author_url": "",
      "post_date": "01/17/2026 22:08:25",
      "content": "<p>I saw this too it’s worth a try I think the two biggest risks are the model size which often means slower fine tuning and the tokenizer even though it’s bpe might apply noisy merges to the Akkadian transliterations since it wasn’t trained on the language I’m not sure if there are enough data examples to improve the tokenizer. If you do try it let us know how it goes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3394255,
      "author_name": "danielpleus",
      "author_url": "",
      "post_date": "01/20/2026 18:21:10",
      "content": "<p>I thought that as well, tried it, but it was painstakingly slow. Quantized it and trained a LoRa-Adapter, but could not make data parallel training on the T4s work. It was supposed to take around a full day and then I retreated.</p>\n<p>Previously trained such models on A100s, but I am still to cheap to buy more GPU power, and don’t know whether that would even be permitted for the competition.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3394411,
          "author_name": "rashmibanthia",
          "author_url": "",
          "post_date": "01/21/2026 01:57:05",
          "content": "<p>Its certainly permitted for training - you can use whatever compute you want. </p>\n<p>That said, this training didn’t work well for me. Generation was quite slow when using 4 beams, and even then the results weren’t very good .. so far.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3418145,
      "author_name": "daylighth",
      "author_url": "",
      "post_date": "03/07/2026 09:03:16",
      "content": "<p>Thanks for sharing your initial thoughts on TranslateGemma!\nSince we are in the final stretch of the competition, I was wondering if you or your team eventually found a way to make Decoder-only models competitive?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3392932": "Translate Gemma model was released yesterday - https://huggingface.co/google/translategemma-4b-it \n\nPaper - https://arxiv.org/abs/2601.09012 \n\nThis uses BPE tokenizer like Gemma unlike character based tokenizer for ByT5. \n\nWondering if this would be useful for fine tuning here, thoughts ?",
    "3392947": "I saw this too it’s worth a try I think the two biggest risks are the model size which often means slower fine tuning and the tokenizer even though it’s bpe might apply noisy merges to the Akkadian transliterations since it wasn’t trained on the language I’m not sure if there are enough data examples to improve the tokenizer. If you do try it let us know how it goes.",
    "3394255": "I thought that as well, tried it, but it was painstakingly slow. Quantized it and trained a LoRa-Adapter, but could not make data parallel training on the T4s work. It was supposed to take around a full day and then I retreated.\n\nPreviously trained such models on A100s, but I am still to cheap to buy more GPU power, and don’t know whether that would even be permitted for the competition.",
    "3394411": "Its certainly permitted for training - you can use whatever compute you want. \n\nThat said, this training didn’t work well for me. Generation was quite slow when using 4 beams, and even then the results weren’t very good .. so far.",
    "3418145": "Thanks for sharing your initial thoughts on TranslateGemma!\nSince we are in the final stretch of the competition, I was wondering if you or your team eventually found a way to make Decoder-only models competitive?"
  },
  "source": "meta"
}