{
  "id": 674348,
  "title": "Cuneiform base model",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/674348",
  "author_name": "Lee Drake",
  "post_date": "2026-02-20T00:21:06.358000",
  "votes": 14,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi all, </p>\n<p>I've been working the past couple years on training cuneiform models - I think a couple folks here are using some. Just pushed a new one: <a href=\"https://huggingface.co/Thalesian/cuneiformBase-400m\" target=\"_blank\">https://huggingface.co/Thalesian/cuneiformBase-400m</a>. It includes Sumerian, Akkadian (Oracc + CDLI), Hittite, Linear B, and a little bit of Elamite. </p>\n<p>This one is built from UMT5-base, but ties embeddings to keep it small. I got decent performance for Akkadian. The Hittite performance I suspect is due to data leakage from a new dataset I added for English, but the German performance is consistent with past models. </p>\n<p>This model was trained on four flavors of input: </p>\n<ul>\n<li>Cuneiform glyph -&gt; English</li>\n<li>Transliteration (CDLI notation) -&gt; English</li>\n<li>Complex transliteration (diacritics) -&gt; English</li>\n<li>\"Simple\" transliteration (hyphens removed, syllables joined to form words) -&gt; English</li>\n</ul>\n<p>Feel free to use, I checked and none of the old Assyrian data present in the training documents here was used to train this model. It is prompt-based like all T5-style models. The functions I use to prepare text can be found in <a href=\"https://github.com/leedrake5/cuneiform\" target=\"_blank\">text_pipeline.py</a>. </p>\n<p>Another tool which might help you is <a href=\"https://github.com/leedrake5/tokenpack-trainer\" target=\"_blank\">TokenpackTrainer</a> - I developed this to train the base model. It is designed to adjust batch size based on the length of inputs and make training more computationally efficient without padding a bunch of 0s on short sequences. It should calibrate the right number of tokens per step based on your hardware, when it OOMs it just goes to a fallback setting then ramps back up until it finds the right limits. It integrates with transformers and requires no new training arguments. I find it really useful for cuneiform datasets which can have extremely variable document lengths.</p>",
  "messages": [
    {
      "id": 3408172,
      "postDate": "2026-02-20T00:21:06.357Z",
      "content": "<p>Hi all, </p>\n<p>I've been working the past couple years on training cuneiform models - I think a couple folks here are using some. Just pushed a new one: <a href=\"https://huggingface.co/Thalesian/cuneiformBase-400m\" target=\"_blank\">https://huggingface.co/Thalesian/cuneiformBase-400m</a>. It includes Sumerian, Akkadian (Oracc + CDLI), Hittite, Linear B, and a little bit of Elamite. </p>\n<p>This one is built from UMT5-base, but ties embeddings to keep it small. I got decent performance for Akkadian. The Hittite performance I suspect is due to data leakage from a new dataset I added for English, but the German performance is consistent with past models. </p>\n<p>This model was trained on four flavors of input: </p>\n<ul>\n<li>Cuneiform glyph -&gt; English</li>\n<li>Transliteration (CDLI notation) -&gt; English</li>\n<li>Complex transliteration (diacritics) -&gt; English</li>\n<li>\"Simple\" transliteration (hyphens removed, syllables joined to form words) -&gt; English</li>\n</ul>\n<p>Feel free to use, I checked and none of the old Assyrian data present in the training documents here was used to train this model. It is prompt-based like all T5-style models. The functions I use to prepare text can be found in <a href=\"https://github.com/leedrake5/cuneiform\" target=\"_blank\">text_pipeline.py</a>. </p>\n<p>Another tool which might help you is <a href=\"https://github.com/leedrake5/tokenpack-trainer\" target=\"_blank\">TokenpackTrainer</a> - I developed this to train the base model. It is designed to adjust batch size based on the length of inputs and make training more computationally efficient without padding a bunch of 0s on short sequences. It should calibrate the right number of tokens per step based on your hardware, when it OOMs it just goes to a fallback setting then ramps back up until it finds the right limits. It integrates with transformers and requires no new training arguments. I find it really useful for cuneiform datasets which can have extremely variable document lengths.</p>",
      "rawMarkdown": "Hi all, \n\nI've been working the past couple years on training cuneiform models - I think a couple folks here are using some. Just pushed a new one: https://huggingface.co/Thalesian/cuneiformBase-400m. It includes Sumerian, Akkadian (Oracc + CDLI), Hittite, Linear B, and a little bit of Elamite. \n\nThis one is built from UMT5-base, but ties embeddings to keep it small. I got decent performance for Akkadian. The Hittite performance I suspect is due to data leakage from a new dataset I added for English, but the German performance is consistent with past models. \n\nThis model was trained on four flavors of input: \n\n- Cuneiform glyph -> English\n- Transliteration (CDLI notation) -> English\n- Complex transliteration (diacritics) -> English\n- \"Simple\" transliteration (hyphens removed, syllables joined to form words) -> English\n\nFeel free to use, I checked and none of the old Assyrian data present in the training documents here was used to train this model. It is prompt-based like all T5-style models. The functions I use to prepare text can be found in [text_pipeline.py](https://github.com/leedrake5/cuneiform). \n\nAnother tool which might help you is [TokenpackTrainer](https://github.com/leedrake5/tokenpack-trainer) - I developed this to train the base model. It is designed to adjust batch size based on the length of inputs and make training more computationally efficient without padding a bunch of 0s on short sequences. It should calibrate the right number of tokens per step based on your hardware, when it OOMs it just goes to a fallback setting then ramps back up until it finds the right limits. It integrates with transformers and requires no new training arguments. I find it really useful for cuneiform datasets which can have extremely variable document lengths.",
      "votes": 14
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3408172": "Hi all, \n\nI've been working the past couple years on training cuneiform models - I think a couple folks here are using some. Just pushed a new one: https://huggingface.co/Thalesian/cuneiformBase-400m. It includes Sumerian, Akkadian (Oracc + CDLI), Hittite, Linear B, and a little bit of Elamite. \n\nThis one is built from UMT5-base, but ties embeddings to keep it small. I got decent performance for Akkadian. The Hittite performance I suspect is due to data leakage from a new dataset I added for English, but the German performance is consistent with past models. \n\nThis model was trained on four flavors of input: \n\n- Cuneiform glyph -> English\n- Transliteration (CDLI notation) -> English\n- Complex transliteration (diacritics) -> English\n- \"Simple\" transliteration (hyphens removed, syllables joined to form words) -> English\n\nFeel free to use, I checked and none of the old Assyrian data present in the training documents here was used to train this model. It is prompt-based like all T5-style models. The functions I use to prepare text can be found in [text_pipeline.py](https://github.com/leedrake5/cuneiform). \n\nAnother tool which might help you is [TokenpackTrainer](https://github.com/leedrake5/tokenpack-trainer) - I developed this to train the base model. It is designed to adjust batch size based on the length of inputs and make training more computationally efficient without padding a bunch of 0s on short sequences. It should calibrate the right number of tokens per step based on your hardware, when it OOMs it just goes to a fallback setting then ramps back up until it finds the right limits. It integrates with transformers and requires no new training arguments. I find it really useful for cuneiform datasets which can have extremely variable document lengths."
  }
}