![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13977249%2F8da40d268aac6825443cd6885249295c%2Fphoto_2026-03-24_16-59-08.jpg?generation=1774346414282992&alt=media\")
Hi Kaggle!
\nThanks to the hosts and Kaggle staff for a great competition and full commitment. Congratulations to all the winners and participants!
\nBad finish for me, but I just want to make a quick note about the best and worst experiments of mine without going into details.
\nData
\nI spent a lot of hours at the start manually aligning sentences, but the impact was rather low (but I can read and understand Akkadian quite well by now!).
\nIn the end, I expanded initial train up to 4x of its initial size.
\nInnaya_v2.pdf, Barjamovic, Gojko - A Historical Geography of Ancient Anatolia in the Assyrian Colony Period. CNI 38, 2011.pdf, Stratford, Edward - Agents, Archives, and Risk. A Micronarrative Account of Old Assyrian Trade Through Salim-ahum's Activities in 1890 B.C. Diss Chicago 2010.pdf published_texts.csv against previous data, split the rest into ~30-word strings and pseudolabeled those -> additional 18k samples.My final training set had ~36k samples of various lengths.
\nModels
\nMy final model is a byt5-xl trained for 3 epochs and I mostly was experimenting with byt5-large just to iterate faster
", "messages": [ { "id": "3427415", "postDate": "03/24/2026 03:53:25", "content": "Hi Kaggle!
\nThanks to the hosts and Kaggle staff for a great competition and full commitment. Congratulations to all the winners and participants!
\nBad finish for me, but I just want to make a quick note about the best and worst experiments of mine without going into details.
\nData
\nI spent a lot of hours at the start manually aligning sentences, but the impact was rather low (but I can read and understand Akkadian quite well by now!).
\nIn the end, I expanded initial train up to 4x of its initial size.
\nInnaya_v2.pdf, Barjamovic, Gojko - A Historical Geography of Ancient Anatolia in the Assyrian Colony Period. CNI 38, 2011.pdf, Stratford, Edward - Agents, Archives, and Risk. A Micronarrative Account of Old Assyrian Trade Through Salim-ahum's Activities in 1890 B.C. Diss Chicago 2010.pdf published_texts.csv against previous data, split the rest into ~30-word strings and pseudolabeled those -> additional 18k samples.My final training set had ~36k samples of various lengths.
\nModels
\nMy final model is a byt5-xl trained for 3 epochs and I mostly was experimenting with byt5-large just to iterate faster
", "rawMarkdown": "Hi Kaggle!\n\nThanks to the hosts and Kaggle staff for a great competition and full commitment. Congratulations to all the winners and participants!\n\nBad finish for me, but I just want to make a quick note about the best and worst experiments of mine without going into details.\n\n**Data**\n\nI spent a lot of hours at the start manually aligning sentences, but the impact was rather low (but I can read and understand Akkadian quite well by now!).\n\nIn the end, I expanded initial train up to 4x of its initial size.\n\n- Additional data sources:\n 1. Translate Turkish->English for shared non-english AKTs -> additional ~6k samples\n 2. Retrieve transliterations and translations from other pdfs like (sorted by size) \n - `Innaya_v2.pdf`, \n - `Barjamovic, Gojko - A Historical Geography of Ancient Anatolia in the Assyrian Colony Period. CNI 38, 2011.pdf`, \n - `Stratford, Edward - Agents, Archives, and Risk. A Micronarrative Account of Old Assyrian Trade Through Salim-ahum's Activities in 1890 B.C. Diss Chicago 2010.pdf` \n - etc (16 more sources) -> additional ~6k samples\n- Deduplicated `published_texts.csv` against previous data, split the rest into ~30-word strings and pseudolabeled those -> additional 18k samples.\n\nMy final training set had ~36k samples of various lengths.\n\n**Models**\n\n- Size matters. byt5-small << byt5-base << byt5-large ~< byt5-xl\n- Solving a problem as a CasualLM and not Seq2Seq. Any LLM (Mistral and Qwen) fine-tuning was worse comparing to byt5 fine-tunning, but decent enough for a gold zone \n- Other T5 variants were much worse. Recently T5-Gemma was released. I didn’t have enough time to try this model.\n\nMy final model is a byt5-xl trained for 3 epochs and I mostly was experimenting with byt5-large just to iterate faster", "votes": null }, { "id": "3427509", "postDate": "03/24/2026 07:12:54", "content": "Congrats on getting so far!
\nWhat were your typical inference speeds?
", "rawMarkdown": "Congrats on getting so far!\n\nWhat were your typical inference speeds?", "votes": null }, { "id": "3427516", "postDate": "03/24/2026 07:21:43", "content": "Thank you.
\nfp32 byt5-xl is inferenced for ~2 hours with bs=1 and max_length=900 on a single P100. I didn't spend a sub optimising it.
", "rawMarkdown": "Thank you.\n\nfp32 byt5-xl is inferenced for ~2 hours with bs=1 and max_length=900 on a single P100. I didn't spend a sub optimising it.", "votes": null }, { "id": "3427592", "postDate": "03/24/2026 09:22:17", "content": "Great work!
\nWhat was the process you used for expanding the dataset -- lots of elbow grease for sentence alignment/pair cleanup (there were a few pairs in the given training set where either the translit or translat were too short for the other …), or was this automatable?
\nWith the competition over, do you intend to share the dataset?
", "rawMarkdown": "Great work!\n\nWhat was the process you used for expanding the dataset -- lots of elbow grease for sentence alignment/pair cleanup (there were a few pairs in the given training set where either the translit or translat were too short for the other ...), or was this automatable?\n\nWith the competition over, do you intend to share the dataset?", "votes": null }, { "id": "3427598", "postDate": "03/24/2026 09:27:28", "content": "Interesting, did you have to do optimization or conversion to fit XL on a P100? I had to split it on two T4s but didn't look too much into it.
\nI also spent a lot of time manually aligning the train set, in the end it seemed almost useless but it was weirdly fun.
", "rawMarkdown": "Interesting, did you have to do optimization or conversion to fit XL on a P100? I had to split it on two T4s but didn't look too much into it.\n\nI also spent a lot of time manually aligning the train set, in the end it seemed almost useless but it was weirdly fun.", "votes": null }, { "id": "3427615", "postDate": "03/24/2026 09:53:09", "content": "Not at all. It doesnt fit on a T4 (15GB RAM is not enough), but just enough to fit on a P100 (16GB is enough). And yea, 100% agree on the aligning part.
", "rawMarkdown": "Not at all. It doesnt fit on a T4 (15GB RAM is not enough), but just enough to fit on a P100 (16GB is enough). And yea, 100% agree on the aligning part.", "votes": null }, { "id": "3427621", "postDate": "03/24/2026 10:03:57", "content": "There was no need to do it manually. You can always train a sentece splitter model. But I did some calculations and understood that I can do close to a perfect sentence alignment in ~2-2.5 weeks for a good part of a train.
\nBasically I just created a google sheet and made an alignment. It was hard in the start, but much easier as you progress through the dataset. Akkadian is an interesting language, I noticed that mostly sentences end with a verb. And some popular words like um-ma, su-ma, ma-na, KISIB, IGI etc. make it easy to align.
\nI stopped when my LB scores stopped improving for few rounds of a sentence alignment.
\n
I guess I will share a dataset if top teams won't
", "rawMarkdown": "There was no need to do it manually. You can always train a sentece splitter model. But I did some calculations and understood that I can do close to a perfect sentence alignment in ~2-2.5 weeks for a good part of a train.\n\nBasically I just created a google sheet and made an alignment. It was hard in the start, but much easier as you progress through the dataset. Akkadian is an interesting language, I noticed that mostly sentences end with a verb. And some popular words like um-ma, su-ma, ma-na, KISIB, IGI etc. make it easy to align.\n\nI stopped when my LB scores stopped improving for few rounds of a sentence alignment.\n\n\n\nI guess I will share a dataset if top teams won't", "votes": null }, { "id": "3427688", "postDate": "03/24/2026 12:15:21", "content": "You must have worked really hard for this competition.
\nCongratulations on your great result!
\n\n… I can read and understand Akkadian quite well by now!
\n
It’s really impressive that you’re able to understand and read this language. 😄
", "rawMarkdown": "You must have worked really hard for this competition. \nCongratulations on your great result!\n\n>... I can read and understand Akkadian quite well by now!\n\nIt’s really impressive that you’re able to understand and read this language. 😄", "votes": null }, { "id": "3427755", "postDate": "03/24/2026 13:48:22", "content": "I think someone with actual assyrian knowledge could clean the train dataset pretty fast. Which is a bit of a shame for the competition but looks like it didn't matter in the end.
", "rawMarkdown": "I think someone with actual assyrian knowledge could clean the train dataset pretty fast. Which is a bit of a shame for the competition but looks like it didn't matter in the end.", "votes": null } ], "comments": [ { "id": 3427509, "author_name": "angantyr", "author_url": "", "post_date": "03/24/2026 07:12:54", "content": "Congrats on getting so far!
\nWhat were your typical inference speeds?
", "votes": null, "replies": [ { "id": 3427516, "author_name": "samson8", "author_url": "", "post_date": "03/24/2026 07:21:43", "content": "Thank you.
\nfp32 byt5-xl is inferenced for ~2 hours with bs=1 and max_length=900 on a single P100. I didn't spend a sub optimising it.
", "votes": null, "replies": [ { "id": 3427598, "author_name": "tennogh", "author_url": "", "post_date": "03/24/2026 09:27:28", "content": "Interesting, did you have to do optimization or conversion to fit XL on a P100? I had to split it on two T4s but didn't look too much into it.
\nI also spent a lot of time manually aligning the train set, in the end it seemed almost useless but it was weirdly fun.
", "votes": null, "replies": [ { "id": 3427615, "author_name": "samson8", "author_url": "", "post_date": "03/24/2026 09:53:09", "content": "Not at all. It doesnt fit on a T4 (15GB RAM is not enough), but just enough to fit on a P100 (16GB is enough). And yea, 100% agree on the aligning part.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3427592, "author_name": "blandcorporatio", "author_url": "", "post_date": "03/24/2026 09:22:17", "content": "Great work!
\nWhat was the process you used for expanding the dataset -- lots of elbow grease for sentence alignment/pair cleanup (there were a few pairs in the given training set where either the translit or translat were too short for the other …), or was this automatable?
\nWith the competition over, do you intend to share the dataset?
", "votes": null, "replies": [ { "id": 3427621, "author_name": "samson8", "author_url": "", "post_date": "03/24/2026 10:03:57", "content": "There was no need to do it manually. You can always train a sentece splitter model. But I did some calculations and understood that I can do close to a perfect sentence alignment in ~2-2.5 weeks for a good part of a train.
\nBasically I just created a google sheet and made an alignment. It was hard in the start, but much easier as you progress through the dataset. Akkadian is an interesting language, I noticed that mostly sentences end with a verb. And some popular words like um-ma, su-ma, ma-na, KISIB, IGI etc. make it easy to align.
\nI stopped when my LB scores stopped improving for few rounds of a sentence alignment.
\n
I guess I will share a dataset if top teams won't
", "votes": null, "replies": [ { "id": 3427755, "author_name": "tennogh", "author_url": "", "post_date": "03/24/2026 13:48:22", "content": "I think someone with actual assyrian knowledge could clean the train dataset pretty fast. Which is a bit of a shame for the competition but looks like it didn't matter in the end.
", "votes": null, "replies": [] } ] } ] }, { "id": 3427688, "author_name": "woosungyoon", "author_url": "", "post_date": "03/24/2026 12:15:21", "content": "You must have worked really hard for this competition.
\nCongratulations on your great result!
\n\n… I can read and understand Akkadian quite well by now!
\n
It’s really impressive that you’re able to understand and read this language. 😄
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3427415": "Hi Kaggle!\n\nThanks to the hosts and Kaggle staff for a great competition and full commitment. Congratulations to all the winners and participants!\n\nBad finish for me, but I just want to make a quick note about the best and worst experiments of mine without going into details.\n\n**Data**\n\nI spent a lot of hours at the start manually aligning sentences, but the impact was rather low (but I can read and understand Akkadian quite well by now!).\n\nIn the end, I expanded initial train up to 4x of its initial size.\n\n- Additional data sources:\n 1. Translate Turkish->English for shared non-english AKTs -> additional ~6k samples\n 2. Retrieve transliterations and translations from other pdfs like (sorted by size) \n - `Innaya_v2.pdf`, \n - `Barjamovic, Gojko - A Historical Geography of Ancient Anatolia in the Assyrian Colony Period. CNI 38, 2011.pdf`, \n - `Stratford, Edward - Agents, Archives, and Risk. A Micronarrative Account of Old Assyrian Trade Through Salim-ahum's Activities in 1890 B.C. Diss Chicago 2010.pdf` \n - etc (16 more sources) -> additional ~6k samples\n- Deduplicated `published_texts.csv` against previous data, split the rest into ~30-word strings and pseudolabeled those -> additional 18k samples.\n\nMy final training set had ~36k samples of various lengths.\n\n**Models**\n\n- Size matters. byt5-small << byt5-base << byt5-large ~< byt5-xl\n- Solving a problem as a CasualLM and not Seq2Seq. Any LLM (Mistral and Qwen) fine-tuning was worse comparing to byt5 fine-tunning, but decent enough for a gold zone \n- Other T5 variants were much worse. Recently T5-Gemma was released. I didn’t have enough time to try this model.\n\nMy final model is a byt5-xl trained for 3 epochs and I mostly was experimenting with byt5-large just to iterate faster", "3427509": "Congrats on getting so far!\n\nWhat were your typical inference speeds?", "3427516": "Thank you.\n\nfp32 byt5-xl is inferenced for ~2 hours with bs=1 and max_length=900 on a single P100. I didn't spend a sub optimising it.", "3427592": "Great work!\n\nWhat was the process you used for expanding the dataset -- lots of elbow grease for sentence alignment/pair cleanup (there were a few pairs in the given training set where either the translit or translat were too short for the other ...), or was this automatable?\n\nWith the competition over, do you intend to share the dataset?", "3427598": "Interesting, did you have to do optimization or conversion to fit XL on a P100? I had to split it on two T4s but didn't look too much into it.\n\nI also spent a lot of time manually aligning the train set, in the end it seemed almost useless but it was weirdly fun.", "3427615": "Not at all. It doesnt fit on a T4 (15GB RAM is not enough), but just enough to fit on a P100 (16GB is enough). And yea, 100% agree on the aligning part.", "3427621": "There was no need to do it manually. You can always train a sentece splitter model. But I did some calculations and understood that I can do close to a perfect sentence alignment in ~2-2.5 weeks for a good part of a train.\n\nBasically I just created a google sheet and made an alignment. It was hard in the start, but much easier as you progress through the dataset. Akkadian is an interesting language, I noticed that mostly sentences end with a verb. And some popular words like um-ma, su-ma, ma-na, KISIB, IGI etc. make it easy to align.\n\nI stopped when my LB scores stopped improving for few rounds of a sentence alignment.\n\n\n\nI guess I will share a dataset if top teams won't", "3427688": "You must have worked really hard for this competition. \nCongratulations on your great result!\n\n>... I can read and understand Akkadian quite well by now!\n\nIt’s really impressive that you’re able to understand and read this language. 😄", "3427755": "I think someone with actual assyrian knowledge could clean the train dataset pretty fast. Which is a bit of a shame for the competition but looks like it didn't matter in the end." }, "source": "meta" }