Thirty days ago I had never trained a model, rented GPUs, or built AI agents. My only background was hobby work collecting ancient language datasets from university archives—mostly cuneiform, though I actually understand Maya epigraphy better.\nI decided to try the Akkadian translation competition as a way to learn.\nMy first attempt was training T5-small on CPU, which took about 36 hours per epoch, so I moved to renting GPUs. Over the course of the project I gradually moved up the hardware ladder:\nRTX 5090\nRTX A6000\nH200\nB200 (for ByT5)\nI built my own dataset rather than starting with the Kaggle dataset. I used scrapers and data-mining agents to collect about 145k pairs, then cleaned and refined it down to roughly 123k pairs.\nUsing PyTorch, I trained several models. With ByT5, the results looked promising around epoch 2, where I was getting reasonable Akkadian → English translations. But by epochs 4–6 the model collapsed, producing junk outputs and sometimes defaulting to French or Akkadian. I reverted to earlier checkpoints and tried fine-tuning with a refined Kaggle dataset, but performance kept degrading.\nI still don’t fully understand why that happened.\nThe goal of this competition wasn’t to win—it was to learn how to build and train models. In that sense it was successful. I learned about:\nGPU rental infrastructure\ntraining pipelines\ndataset cleaning and augmentation\nPyTorch workflows\ntransformer model behavior\nOver the last month I also built a number of automation tools including agent swarms, scrapers, and a fully automated YouTube pipeline.\nWhat I realized during this process is that the ancient language model I truly want to build is for Classic Maya.\nMy long-term idea is an ensemble system, similar to what I attempted with Akkadian:\nForward model: Maya → English\nCandidate generation (beam search or secondary model)\nReverse models: English → Maya\nReranking based on round-trip consistency\nBut extended to include:\nlogograms\ncalendar glyphs\nnames and titles\nSpanish and English translations\nI’m still very new to training models, but I’m extremely determined to build something meaningful for Maya epigraphy.\nIf anyone here has more experience with model training, vision models, or low-resource language pipelines and would be interested in collaborating on something like this, I would genuinely appreciate connecting.\nFor now I wiped my previous pipeline so I could step away and reset, but after a few days of rest I plan to start building a Maya dataset and a new training pipeline from scratch.\nEven though this competition didn’t produce a final model, I’m glad I attempted it—I learned a tremendous amount in a very short time.
", "messages": [ { "id": "3421588", "postDate": "03/15/2026 21:56:32", "content": "Thirty days ago I had never trained a model, rented GPUs, or built AI agents. My only background was hobby work collecting ancient language datasets from university archives—mostly cuneiform, though I actually understand Maya epigraphy better.\nI decided to try the Akkadian translation competition as a way to learn.\nMy first attempt was training T5-small on CPU, which took about 36 hours per epoch, so I moved to renting GPUs. Over the course of the project I gradually moved up the hardware ladder:\nRTX 5090\nRTX A6000\nH200\nB200 (for ByT5)\nI built my own dataset rather than starting with the Kaggle dataset. I used scrapers and data-mining agents to collect about 145k pairs, then cleaned and refined it down to roughly 123k pairs.\nUsing PyTorch, I trained several models. With ByT5, the results looked promising around epoch 2, where I was getting reasonable Akkadian → English translations. But by epochs 4–6 the model collapsed, producing junk outputs and sometimes defaulting to French or Akkadian. I reverted to earlier checkpoints and tried fine-tuning with a refined Kaggle dataset, but performance kept degrading.\nI still don’t fully understand why that happened.\nThe goal of this competition wasn’t to win—it was to learn how to build and train models. In that sense it was successful. I learned about:\nGPU rental infrastructure\ntraining pipelines\ndataset cleaning and augmentation\nPyTorch workflows\ntransformer model behavior\nOver the last month I also built a number of automation tools including agent swarms, scrapers, and a fully automated YouTube pipeline.\nWhat I realized during this process is that the ancient language model I truly want to build is for Classic Maya.\nMy long-term idea is an ensemble system, similar to what I attempted with Akkadian:\nForward model: Maya → English\nCandidate generation (beam search or secondary model)\nReverse models: English → Maya\nReranking based on round-trip consistency\nBut extended to include:\nlogograms\ncalendar glyphs\nnames and titles\nSpanish and English translations\nI’m still very new to training models, but I’m extremely determined to build something meaningful for Maya epigraphy.\nIf anyone here has more experience with model training, vision models, or low-resource language pipelines and would be interested in collaborating on something like this, I would genuinely appreciate connecting.\nFor now I wiped my previous pipeline so I could step away and reset, but after a few days of rest I plan to start building a Maya dataset and a new training pipeline from scratch.\nEven though this competition didn’t produce a final model, I’m glad I attempted it—I learned a tremendous amount in a very short time.
", "rawMarkdown": "Thirty days ago I had never trained a model, rented GPUs, or built AI agents. My only background was hobby work collecting ancient language datasets from university archives—mostly cuneiform, though I actually understand Maya epigraphy better.\n\nI decided to try the Akkadian translation competition as a way to learn.\n\nMy first attempt was training T5-small on CPU, which took about 36 hours per epoch, so I moved to renting GPUs. Over the course of the project I gradually moved up the hardware ladder:\n\nRTX 5090\n\nRTX A6000\n\nH200\n\nB200 (for ByT5)\n\nI built my own dataset rather than starting with the Kaggle dataset. I used scrapers and data-mining agents to collect about 145k pairs, then cleaned and refined it down to roughly 123k pairs.\n\nUsing PyTorch, I trained several models. With ByT5, the results looked promising around epoch 2, where I was getting reasonable Akkadian → English translations. But by epochs 4–6 the model collapsed, producing junk outputs and sometimes defaulting to French or Akkadian. I reverted to earlier checkpoints and tried fine-tuning with a refined Kaggle dataset, but performance kept degrading.\n\nI still don’t fully understand why that happened.\n\nThe goal of this competition wasn’t to win—it was to learn how to build and train models. In that sense it was successful. I learned about:\n\nGPU rental infrastructure\n\ntraining pipelines\n\ndataset cleaning and augmentation\n\nPyTorch workflows\n\ntransformer model behavior\n\nOver the last month I also built a number of automation tools including agent swarms, scrapers, and a fully automated YouTube pipeline.\n\nWhat I realized during this process is that the ancient language model I truly want to build is for Classic Maya.\n\nMy long-term idea is an ensemble system, similar to what I attempted with Akkadian:\n\nForward model: Maya → English\nCandidate generation (beam search or secondary model)\nReverse models: English → Maya\nReranking based on round-trip consistency\n\nBut extended to include:\n\nlogograms\n\ncalendar glyphs\n\nnames and titles\n\nSpanish and English translations\n\nI’m still very new to training models, but I’m extremely determined to build something meaningful for Maya epigraphy.\n\nIf anyone here has more experience with model training, vision models, or low-resource language pipelines and would be interested in collaborating on something like this, I would genuinely appreciate connecting.\n\nFor now I wiped my previous pipeline so I could step away and reset, but after a few days of rest I plan to start building a Maya dataset and a new training pipeline from scratch.\n\nEven though this competition didn’t produce a final model, I’m glad I attempted it—I learned a tremendous amount in a very short time.", "votes": null }, { "id": "3421597", "postDate": "03/15/2026 22:34:25", "content": "If you don't have previous experience with language models, playing around with the public notebooks using Kaggle GPU quotas is probably a more gentle way to get into it. You can also look at the top solutions after this competition is over (although some of their choices will have more to do with the quirks of this particular competition than what you're interested in).
", "rawMarkdown": "If you don't have previous experience with language models, playing around with the public notebooks using Kaggle GPU quotas is probably a more gentle way to get into it. You can also look at the top solutions after this competition is over (although some of their choices will have more to do with the quirks of this particular competition than what you're interested in).", "votes": null }, { "id": "3422245", "postDate": "03/17/2026 09:48:32", "content": "Continuing @tennogh's comment: Here is a specific set of notebooks you can easily play with (training and inference). No external GPU needed. It's click & play.
\nTraining: https://www.kaggle.com/code/takamichitoda/dpc-starter-train
\nInference: https://www.kaggle.com/code/qifeihhh666/dpc-starter-infer-add-sentencealign
\nI am in somewhat similar shoes to you. I have been a human translator (German/English) for years and have gotten interested in machine learning since about 2014. Progress has come in spurts, but this competition has helped me over a big hump (especially learning Pytorch, Hugging Face).
\nIf you want to dig more, I suggest this 3-course series. I found it fantastic. A lot of the projects involve images, but NLP is also covered, including a translation model in course 3: https://www.coursera.org/professional-certificates/pytorch-for-deep-learning
", "rawMarkdown": "Continuing @tennogh's comment: Here is a specific set of notebooks you can easily play with (training and inference). No external GPU needed. It's click & play.\n\nTraining: https://www.kaggle.com/code/takamichitoda/dpc-starter-train\n\nInference: https://www.kaggle.com/code/qifeihhh666/dpc-starter-infer-add-sentencealign\n\nI am in somewhat similar shoes to you. I have been a human translator (German/English) for years and have gotten interested in machine learning since about 2014. Progress has come in spurts, but this competition has helped me over a big hump (especially learning Pytorch, Hugging Face).\n\nIf you want to dig more, I suggest this 3-course series. I found it fantastic. A lot of the projects involve images, but NLP is also covered, including a translation model in course 3: https://www.coursera.org/professional-certificates/pytorch-for-deep-learning", "votes": null } ], "comments": [ { "id": 3421597, "author_name": "tennogh", "author_url": "", "post_date": "03/15/2026 22:34:25", "content": "If you don't have previous experience with language models, playing around with the public notebooks using Kaggle GPU quotas is probably a more gentle way to get into it. You can also look at the top solutions after this competition is over (although some of their choices will have more to do with the quirks of this particular competition than what you're interested in).
", "votes": null, "replies": [ { "id": 3422245, "author_name": "garbo999", "author_url": "", "post_date": "03/17/2026 09:48:32", "content": "Continuing @tennogh's comment: Here is a specific set of notebooks you can easily play with (training and inference). No external GPU needed. It's click & play.
\nTraining: https://www.kaggle.com/code/takamichitoda/dpc-starter-train
\nInference: https://www.kaggle.com/code/qifeihhh666/dpc-starter-infer-add-sentencealign
\nI am in somewhat similar shoes to you. I have been a human translator (German/English) for years and have gotten interested in machine learning since about 2014. Progress has come in spurts, but this competition has helped me over a big hump (especially learning Pytorch, Hugging Face).
\nIf you want to dig more, I suggest this 3-course series. I found it fantastic. A lot of the projects involve images, but NLP is also covered, including a translation model in course 3: https://www.coursera.org/professional-certificates/pytorch-for-deep-learning
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3421588": "Thirty days ago I had never trained a model, rented GPUs, or built AI agents. My only background was hobby work collecting ancient language datasets from university archives—mostly cuneiform, though I actually understand Maya epigraphy better.\n\nI decided to try the Akkadian translation competition as a way to learn.\n\nMy first attempt was training T5-small on CPU, which took about 36 hours per epoch, so I moved to renting GPUs. Over the course of the project I gradually moved up the hardware ladder:\n\nRTX 5090\n\nRTX A6000\n\nH200\n\nB200 (for ByT5)\n\nI built my own dataset rather than starting with the Kaggle dataset. I used scrapers and data-mining agents to collect about 145k pairs, then cleaned and refined it down to roughly 123k pairs.\n\nUsing PyTorch, I trained several models. With ByT5, the results looked promising around epoch 2, where I was getting reasonable Akkadian → English translations. But by epochs 4–6 the model collapsed, producing junk outputs and sometimes defaulting to French or Akkadian. I reverted to earlier checkpoints and tried fine-tuning with a refined Kaggle dataset, but performance kept degrading.\n\nI still don’t fully understand why that happened.\n\nThe goal of this competition wasn’t to win—it was to learn how to build and train models. In that sense it was successful. I learned about:\n\nGPU rental infrastructure\n\ntraining pipelines\n\ndataset cleaning and augmentation\n\nPyTorch workflows\n\ntransformer model behavior\n\nOver the last month I also built a number of automation tools including agent swarms, scrapers, and a fully automated YouTube pipeline.\n\nWhat I realized during this process is that the ancient language model I truly want to build is for Classic Maya.\n\nMy long-term idea is an ensemble system, similar to what I attempted with Akkadian:\n\nForward model: Maya → English\nCandidate generation (beam search or secondary model)\nReverse models: English → Maya\nReranking based on round-trip consistency\n\nBut extended to include:\n\nlogograms\n\ncalendar glyphs\n\nnames and titles\n\nSpanish and English translations\n\nI’m still very new to training models, but I’m extremely determined to build something meaningful for Maya epigraphy.\n\nIf anyone here has more experience with model training, vision models, or low-resource language pipelines and would be interested in collaborating on something like this, I would genuinely appreciate connecting.\n\nFor now I wiped my previous pipeline so I could step away and reset, but after a few days of rest I plan to start building a Maya dataset and a new training pipeline from scratch.\n\nEven though this competition didn’t produce a final model, I’m glad I attempted it—I learned a tremendous amount in a very short time.", "3421597": "If you don't have previous experience with language models, playing around with the public notebooks using Kaggle GPU quotas is probably a more gentle way to get into it. You can also look at the top solutions after this competition is over (although some of their choices will have more to do with the quirks of this particular competition than what you're interested in).", "3422245": "Continuing @tennogh's comment: Here is a specific set of notebooks you can easily play with (training and inference). No external GPU needed. It's click & play.\n\nTraining: https://www.kaggle.com/code/takamichitoda/dpc-starter-train\n\nInference: https://www.kaggle.com/code/qifeihhh666/dpc-starter-infer-add-sentencealign\n\nI am in somewhat similar shoes to you. I have been a human translator (German/English) for years and have gotten interested in machine learning since about 2014. Progress has come in spurts, but this competition has helped me over a big hump (especially learning Pytorch, Hugging Face).\n\nIf you want to dig more, I suggest this 3-course series. I found it fantastic. A lot of the projects involve images, but NLP is also covered, including a translation model in course 3: https://www.coursera.org/professional-certificates/pytorch-for-deep-learning" }, "source": "meta" }