![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18445905%2F5b2fb78dc3f239ddec9d9c1c33a280f8%2FScreenshot%202026-01-27%20at%2012-38-39%20byt-ensemble%20Kaggle.png?generation=1769497754971029&alt=media\")
We can try to make a semantically akkadian rich LLM but the ground truth labels are just like a bag of words without proper structure. So no gains would be found even if the predictions actually make sense. And usage of BLEU will plateau the leader board score to not more than 50. Not a hypothesis but if you extrapolate with a few sentences and make some hand made outputs this is the best you can get with the current akkadian data. I am not even talking about the training data, we already have and we can get clean pairwise data. But the test set ? The sentences for sure are not like a proper sentence. At least that's how they are in the training data provided.
\nFor example, in the code I have with some models I trained (not the 34.5 baselines)\nMakes no sense whatsoever. Neater sentences should score more. But here this reveals that the test set itself is ill formatted. So better solutions hurt the score? Should we make bad outputs to increase the score? How would this even address the problem statement of \"Akkadian translations\" if this about hacking the leader board? The top teams are doing a great job but idk, I lost interest.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18445905%2Fe22ad438440cacc364dba43044570f8d%2FScreenshot%202026-01-27%20at%2012-38-22%20Deep%20Past%20Challenge%20-%20Translate%20Akkadian%20to%20English%20Kaggle.png?generation=1769497775219513&alt=media\"){kind=spacer}
We can try to make a semantically akkadian rich LLM but the ground truth labels are just like a bag of words without proper structure. So no gains would be found even if the predictions actually make sense. And usage of BLEU will plateau the leader board score to not more than 50. Not a hypothesis but if you extrapolate with a few sentences and make some hand made outputs this is the best you can get with the current akkadian data. I am not even talking about the training data, we already have and we can get clean pairwise data. But the test set ? The sentences for sure are not like a proper sentence. At least that's how they are in the training data provided.
\nFor example, in the code I have with some models I trained (not the 34.5 baselines)\nMakes no sense whatsoever. Neater sentences should score more. But here this reveals that the test set itself is ill formatted. So better solutions hurt the score? Should we make bad outputs to increase the score? How would this even address the problem statement of \"Akkadian translations\" if this about hacking the leader board? The top teams are doing a great job but idk, I lost interest.
\n
Yes, this challenge seems poorly prepared. One confusing issue is: the host could polish the transliteration in published_texts.csv into the test-like form, but still hesitate to give us a normalized train.csv and test.csv, or share their preprocessing notebook. Don't know why they refuse to do so.
", "rawMarkdown": "Yes, this challenge seems poorly prepared. One confusing issue is: the host could polish the transliteration in published_texts.csv into the test-like form, but still hesitate to give us a normalized train.csv and test.csv, or share their preprocessing notebook. Don't know why they refuse to do so.", "votes": 5, "replies": [ { "id": 3397474, "postDate": "2026-01-27T10:37:42.343Z", "content": "I suspect that the test set might have been manually processed according to a guideline similar to that in the challenge overview. And it is hard to transform that manual work perfectly into preprocessing functions. The transliteration transformation in published_texts.csv could be just an approximation.
", "rawMarkdown": "I suspect that the test set might have been manually processed according to a guideline similar to that in the challenge overview. And it is hard to transform that manual work perfectly into preprocessing functions. The transliteration transformation in published_texts.csv could be just an approximation.", "replies": [ { "id": 3397481, "postDate": "2026-01-27T10:58:48.200Z", "content": "Either way it's not optimal. At the start I thought gathering training data was issue. Now it's the complete opposite. That image I shared above is supporting evidence that atleast the test set has to be clean. Doesn't even have to be 4000 entries, a few cleaned and strict entries would have been better. I mean why do we need to train models to predict tokens like <gap> and so on, what's the point? I get that it weirdly gives boosts of 3-4 on score but it just doesn't seem right given what this competition is targeting
I am facing the same exact issues. My theoretically worse models are getting better LB score. I cross checked a lot of examples from my test set with Gemini and they seem to be very good translations, but somehow the model performs like garbage on the LB.
\nFor example my model was trained in a way that using num_beams would harm the cross validation score, but then on the LB it significantly increases my score by 4-5 points, so this is utterly confusing.
\nThe LB using BLEU instead of a better more modern metric that checks for actual meaning is also not helpful since unless you match the same exact word and not a synonym, you get a bad score.
", "rawMarkdown": "I am facing the same exact issues. My theoretically worse models are getting better LB score. I cross checked a lot of examples from my test set with Gemini and they seem to be very good translations, but somehow the model performs like garbage on the LB.\n\nFor example my model was trained in a way that using num_beams would harm the cross validation score, but then on the LB it significantly increases my score by 4-5 points, so this is utterly confusing.\n\nThe LB using BLEU instead of a better more modern metric that checks for actual meaning is also not helpful since unless you match the same exact word and not a synonym, you get a bad score.", "votes": 4 }, { "id": 3397540, "postDate": "2026-01-27T14:41:28.067Z", "content": "in my case normalize one ill-formated single or double quote character ' can get my score down by 1.0 on LB, maybe i am missing something here but it feels like a waste of time
", "rawMarkdown": "in my case normalize one ill-formated single or double quote character ' can get my score down by 1.0 on LB, maybe i am missing something here but it feels like a waste of time", "votes": 1, "replies": [ { "id": 3397555, "postDate": "2026-01-27T15:19:33.537Z", "content": "@honganzhu \nYou can see that I gained +3 just by keeping the faulty tokens. You're not missing anything, it's like good outputs are not what's needed to win or something like that. Weird stuff.
", "rawMarkdown": "@honganzhu \nYou can see that I gained +3 just by keeping the faulty tokens. You're not missing anything, it's like good outputs are not what's needed to win or something like that. Weird stuff. ", "replies": [ { "id": 3397580, "postDate": "2026-01-27T16:13:29.143Z", "content": "I agree. But in the example you showed, you shouldn't clean gap and big gap in translation because they are expected (transliteration also have them) so the proper translation should also have gap and big gap! I think that's why you are seeing the +3 improvement.
", "rawMarkdown": "I agree. But in the example you showed, you shouldn't clean gap and big gap in translation because they are expected (transliteration also have them) so the proper translation should also have gap and big gap! I think that's why you are seeing the +3 improvement." }, { "id": 3397594, "postDate": "2026-01-27T16:26:52.450Z", "content": "Yess. I am aware. I wanted to show how much of a difference those make. This shows that the number of occurrences of these characters in the test set must be pretty huge.
", "rawMarkdown": "Yess. I am aware. I wanted to show how much of a difference those make. This shows that the number of occurrences of these characters in the test set must be pretty huge. " }, { "id": 3397628, "postDate": "2026-01-27T17:41:27.137Z", "content": "\"it's like good outputs are not what's needed to win or something like that\" - person with bad outputs
", "rawMarkdown": "*\"it's like good outputs are not what's needed to win or something like that\"* - person with bad outputs", "votes": 1 }, { "id": 3397669, "postDate": "2026-01-27T18:39:12.240Z", "content": "Yes, this challenge seems poorly prepared. One confusing issue is: the host could polish the transliteration in published_texts.csv into the test-like form, but still hesitate to give us a normalized train.csv and test.csv, or share their preprocessing notebook. Don't know why they refuse to do so.
", "votes": 5, "replies": [ { "id": 3397474, "author_name": "Tomorin", "author_url": "", "post_date": "2026-01-27T10:37:42.343000", "content": "I suspect that the test set might have been manually processed according to a guideline similar to that in the challenge overview. And it is hard to transform that manual work perfectly into preprocessing functions. The transliteration transformation in published_texts.csv could be just an approximation.
", "votes": 0, "replies": [ { "id": 3397481, "author_name": "", "author_url": "", "post_date": "2026-01-27T10:58:48.200000", "content": "Either way it's not optimal. At the start I thought gathering training data was issue. Now it's the complete opposite. That image I shared above is supporting evidence that atleast the test set has to be clean. Doesn't even have to be 4000 entries, a few cleaned and strict entries would have been better. I mean why do we need to train models to predict tokens like <gap> and so on, what's the point? I get that it weirdly gives boosts of 3-4 on score but it just doesn't seem right given what this competition is targeting
I am facing the same exact issues. My theoretically worse models are getting better LB score. I cross checked a lot of examples from my test set with Gemini and they seem to be very good translations, but somehow the model performs like garbage on the LB.
\nFor example my model was trained in a way that using num_beams would harm the cross validation score, but then on the LB it significantly increases my score by 4-5 points, so this is utterly confusing.
\nThe LB using BLEU instead of a better more modern metric that checks for actual meaning is also not helpful since unless you match the same exact word and not a synonym, you get a bad score.
", "votes": 4, "replies": [] }, { "id": 3397540, "author_name": "hongan", "author_url": "", "post_date": "2026-01-27T14:41:28.067000", "content": "in my case normalize one ill-formated single or double quote character ' can get my score down by 1.0 on LB, maybe i am missing something here but it feels like a waste of time
", "votes": 1, "replies": [ { "id": 3397555, "author_name": "", "author_url": "", "post_date": "2026-01-27T15:19:33.537000", "content": "@honganzhu \nYou can see that I gained +3 just by keeping the faulty tokens. You're not missing anything, it's like good outputs are not what's needed to win or something like that. Weird stuff.
", "votes": 0, "replies": [ { "id": 3397580, "author_name": "hongan", "author_url": "", "post_date": "2026-01-27T16:13:29.143000", "content": "I agree. But in the example you showed, you shouldn't clean gap and big gap in translation because they are expected (transliteration also have them) so the proper translation should also have gap and big gap! I think that's why you are seeing the +3 improvement.
", "votes": 0, "replies": [] }, { "id": 3397594, "author_name": "", "author_url": "", "post_date": "2026-01-27T16:26:52.450000", "content": "Yess. I am aware. I wanted to show how much of a difference those make. This shows that the number of occurrences of these characters in the test set must be pretty huge.
", "votes": 0, "replies": [] }, { "id": 3397628, "author_name": "Jack", "author_url": "", "post_date": "2026-01-27T17:41:27.137000", "content": "\"it's like good outputs are not what's needed to win or something like that\" - person with bad outputs
", "votes": 1, "replies": [] }, { "id": 3397669, "author_name": "", "author_url": "", "post_date": "2026-01-27T18:39:12.240000", "content": "