Since the corpus of Akkadian to English is so different from most of the corpus out there, I think it might make sense to rebuild the model for training.
\nThe model didn't learn well, so open for advice for improvement.
\n", "messages": [ { "id": "3393904", "postDate": "01/20/2026 05:45:41", "content": "Since the corpus of Akkadian to English is so different from most of the corpus out there, I think it might make sense to rebuild the model for training.
\nThe model didn't learn well, so open for advice for improvement.
\n", "rawMarkdown": "Since the corpus of Akkadian to English is so different from most of the corpus out there, I think it might make sense to rebuild the model for training.\n\nThe model didn't learn well, so open for advice for improvement.\n\nhttps://www.kaggle.com/code/zhangyue199/dpc-byt5-retrain-tokenizer/notebook?scriptVersionId=292842377", "votes": null }, { "id": "3395451", "postDate": "01/23/2026 02:26:30", "content": "This is a really impressive pipeline. Nice job building the custom inference class and training loop!
\nI noticed one small thing that might explain why your predictions are repeating
\nIt looks like you are accidentally training from scratch instead of fine-tuning. In your model section, you used: model = T5ForConditionalGeneration(config) That creates a blank model with random weights. Since our dataset is so small (2.6k rows), the model can't learn a language from scratch, so it just memorizes the most common phrase. For a while, I was trying to do this, and running into the same issue.
\nMy suggested fix: If you want to use ByT5 (one of the best models for this data), try this:
\nSkip the SentencePiece training. ByT5 doesn't need a tokenizer! It reads raw bytes/characters.
\nLoad the pre-trained weights:
\nPython\ntokenizer = AutoTokenizer.from_pretrained(\"google/byt5-small\")\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-small\")\nThis will let you start with a model that already 'knows' how to read other languages, so it just needs to learn Akkadian. Hope this helps!
", "rawMarkdown": "This is a really impressive pipeline. Nice job building the custom inference class and training loop!\n\nI noticed one small thing that might explain why your predictions are repeating\n\nIt looks like you are accidentally training from scratch instead of fine-tuning. In your model section, you used: model = T5ForConditionalGeneration(config) That creates a blank model with random weights. Since our dataset is so small (2.6k rows), the model can't learn a language from scratch, so it just memorizes the most common phrase. For a while, I was trying to do this, and running into the same issue. \n\nMy suggested fix: If you want to use ByT5 (one of the best models for this data), try this:\n\nSkip the SentencePiece training. ByT5 doesn't need a tokenizer! It reads raw bytes/characters.\n\nLoad the pre-trained weights:\n\nPython\ntokenizer = AutoTokenizer.from_pretrained(\"google/byt5-small\")\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-small\")\nThis will let you start with a model that already 'knows' how to read other languages, so it just needs to learn Akkadian. Hope this helps!", "votes": null } ], "comments": [ { "id": 3395451, "author_name": "leowithrow", "author_url": "", "post_date": "01/23/2026 02:26:30", "content": "This is a really impressive pipeline. Nice job building the custom inference class and training loop!
\nI noticed one small thing that might explain why your predictions are repeating
\nIt looks like you are accidentally training from scratch instead of fine-tuning. In your model section, you used: model = T5ForConditionalGeneration(config) That creates a blank model with random weights. Since our dataset is so small (2.6k rows), the model can't learn a language from scratch, so it just memorizes the most common phrase. For a while, I was trying to do this, and running into the same issue.
\nMy suggested fix: If you want to use ByT5 (one of the best models for this data), try this:
\nSkip the SentencePiece training. ByT5 doesn't need a tokenizer! It reads raw bytes/characters.
\nLoad the pre-trained weights:
\nPython\ntokenizer = AutoTokenizer.from_pretrained(\"google/byt5-small\")\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-small\")\nThis will let you start with a model that already 'knows' how to read other languages, so it just needs to learn Akkadian. Hope this helps!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3393904": "Since the corpus of Akkadian to English is so different from most of the corpus out there, I think it might make sense to rebuild the model for training.\n\nThe model didn't learn well, so open for advice for improvement.\n\nhttps://www.kaggle.com/code/zhangyue199/dpc-byt5-retrain-tokenizer/notebook?scriptVersionId=292842377", "3395451": "This is a really impressive pipeline. Nice job building the custom inference class and training loop!\n\nI noticed one small thing that might explain why your predictions are repeating\n\nIt looks like you are accidentally training from scratch instead of fine-tuning. In your model section, you used: model = T5ForConditionalGeneration(config) That creates a blank model with random weights. Since our dataset is so small (2.6k rows), the model can't learn a language from scratch, so it just memorizes the most common phrase. For a while, I was trying to do this, and running into the same issue. \n\nMy suggested fix: If you want to use ByT5 (one of the best models for this data), try this:\n\nSkip the SentencePiece training. ByT5 doesn't need a tokenizer! It reads raw bytes/characters.\n\nLoad the pre-trained weights:\n\nPython\ntokenizer = AutoTokenizer.from_pretrained(\"google/byt5-small\")\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-small\")\nThis will let you start with a model that already 'knows' how to read other languages, so it just needs to learn Akkadian. Hope this helps!" }, "source": "meta" }