Hi everyone,\nI discovered a significant domain shift between the train and test sets that may explain why models plateau around 31-32 points. I'd like to share my findings and hear your thoughts.
\nVocabulary Coverage: Only 57.1% (27 out of 63 test tokens are OOV)\nThe test set uses a COMPLETELY DIFFERENT transliteration system\nCritical characters š, ḫ, ṣ, ṭ appear 28,654 times in train but 0 times in test\nTest set has a unique character „ (German quotation mark) not in train
\nTHE CORE PROBLEM: Different Transliteration Systems\nCharacter Usage Comparison:\nš (shin): Train = 20,354× | Test = 0× ⚠️\nḫ (het): Train = 4,552× | Test = 0× ⚠️\nṣ (tsade): Train = 2,533× | Test = 0× ⚠️\nṭ (tet): Train = 1,215× | Test = 0× ⚠️\n„ (quote): Train = 0× | Test = 6× ⚠️
CHARACTER SUBSTITUTION PATTERN\nTest → Train (Frequency) - Meaning\naa → ša (6,291×) - \"of/that\"\nau-mì → šu-mì (193×) - \"name\"\nmup-pu-um → ṭup-pu-um (116×) - \"tablet\"\naé-bi„-lá → šé-bi-la (41×) - \"send\"\nqí-bi„-ma → qí-bí-ma (18×) - \"speak\"\nKey Pattern: š → a, ṭ → m in the test set
VOCABULARY STATISTICS\nTrain vocabulary: 11,471 unique tokens\nTest vocabulary: 63 unique tokens\nOverlap: 36 tokens (57.1%)\nOOV: 27 tokens (42.9%)
TEXT LENGTH\nTrain Mean: 426.5 chars\nTest Mean: 168.8 chars (~40% of train length)
POTENTIAL SOLUTIONS
Preprocessing: Normalize test set to train conventions\nData Augmentation: Add character substitution noise (š↔a, ṭ↔m)\nTrain with both transliteration conventions
\nQUESTIONS
\nIs this intentional to test robustness?\nHas anyone found the source for text_id: 332fda50?
\nWould love to hear your thoughts! 🙏
", "messages": [ { "id": "3385399", "postDate": "01/03/2026 08:22:17", "content": "Hi everyone,\nI discovered a significant domain shift between the train and test sets that may explain why models plateau around 31-32 points. I'd like to share my findings and hear your thoughts.
\nVocabulary Coverage: Only 57.1% (27 out of 63 test tokens are OOV)\nThe test set uses a COMPLETELY DIFFERENT transliteration system\nCritical characters š, ḫ, ṣ, ṭ appear 28,654 times in train but 0 times in test\nTest set has a unique character „ (German quotation mark) not in train
\nTHE CORE PROBLEM: Different Transliteration Systems\nCharacter Usage Comparison:\nš (shin): Train = 20,354× | Test = 0× ⚠️\nḫ (het): Train = 4,552× | Test = 0× ⚠️\nṣ (tsade): Train = 2,533× | Test = 0× ⚠️\nṭ (tet): Train = 1,215× | Test = 0× ⚠️\n„ (quote): Train = 0× | Test = 6× ⚠️
CHARACTER SUBSTITUTION PATTERN\nTest → Train (Frequency) - Meaning\naa → ša (6,291×) - \"of/that\"\nau-mì → šu-mì (193×) - \"name\"\nmup-pu-um → ṭup-pu-um (116×) - \"tablet\"\naé-bi„-lá → šé-bi-la (41×) - \"send\"\nqí-bi„-ma → qí-bí-ma (18×) - \"speak\"\nKey Pattern: š → a, ṭ → m in the test set
VOCABULARY STATISTICS\nTrain vocabulary: 11,471 unique tokens\nTest vocabulary: 63 unique tokens\nOverlap: 36 tokens (57.1%)\nOOV: 27 tokens (42.9%)
TEXT LENGTH\nTrain Mean: 426.5 chars\nTest Mean: 168.8 chars (~40% of train length)
POTENTIAL SOLUTIONS
Preprocessing: Normalize test set to train conventions\nData Augmentation: Add character substitution noise (š↔a, ṭ↔m)\nTrain with both transliteration conventions
\nQUESTIONS
\nIs this intentional to test robustness?\nHas anyone found the source for text_id: 332fda50?
\nWould love to hear your thoughts! 🙏
", "rawMarkdown": "Hi everyone,\nI discovered a significant domain shift between the train and test sets that may explain why models plateau around 31-32 points. I'd like to share my findings and hear your thoughts.\n\nVocabulary Coverage: Only 57.1% (27 out of 63 test tokens are OOV)\nThe test set uses a COMPLETELY DIFFERENT transliteration system\nCritical characters š, ḫ, ṣ, ṭ appear 28,654 times in train but 0 times in test\nTest set has a unique character „ (German quotation mark) not in train\n\n1. THE CORE PROBLEM: Different Transliteration Systems\nCharacter Usage Comparison:\nš (shin): Train = 20,354× | Test = 0× ⚠️\nḫ (het): Train = 4,552× | Test = 0× ⚠️\nṣ (tsade): Train = 2,533× | Test = 0× ⚠️\nṭ (tet): Train = 1,215× | Test = 0× ⚠️\n„ (quote): Train = 0× | Test = 6× ⚠️\n\n2. CHARACTER SUBSTITUTION PATTERN\nTest → Train (Frequency) - Meaning\naa → ša (6,291×) - \"of/that\"\nau-mì → šu-mì (193×) - \"name\"\nmup-pu-um → ṭup-pu-um (116×) - \"tablet\"\naé-bi„-lá → šé-bi-la (41×) - \"send\"\nqí-bi„-ma → qí-bí-ma (18×) - \"speak\"\nKey Pattern: š → a, ṭ → m in the test set\n\n3. VOCABULARY STATISTICS\nTrain vocabulary: 11,471 unique tokens\nTest vocabulary: 63 unique tokens\nOverlap: 36 tokens (57.1%)\nOOV: 27 tokens (42.9%)\n\n4. TEXT LENGTH\nTrain Mean: 426.5 chars\nTest Mean: 168.8 chars (~40% of train length)\n\n5. POTENTIAL SOLUTIONS\n\nPreprocessing: Normalize test set to train conventions\nData Augmentation: Add character substitution noise (š↔a, ṭ↔m)\nTrain with both transliteration conventions\n\nQUESTIONS\n\nIs this intentional to test robustness?\nHas anyone found the source for text_id: 332fda50?\n\nWould love to hear your thoughts! 🙏", "votes": null }, { "id": "3385405", "postDate": "01/03/2026 08:45:13", "content": "\n\nPlease note that this is a Code Competition. The data in
\ntest.csvis only dummy data to help you author your solutions. When your submission is scored, this example test data will be replaced with the full test set.
So the test.csv you analyzed is just placeholder data for debugging purposes — it's not the actual hidden test set used for scoring.
\nThe host has confirmed in multiple discussion threads that the hidden test set follows the same formatting conventions as described in the Dataset Instructions, including normalized gaps (, ), proper subscript handling, and Unicode fractions.
\nSee this thread for the host's confirmation.
", "rawMarkdown": ">Please note that this is a [Code Competition](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/code-requirements). The data in `test.csv` is only dummy data to help you author your solutions. When your submission is scored, this example test data will be replaced with the full test set.\n\nSo the test.csv you analyzed is just placeholder data for debugging purposes — it's not the actual hidden test set used for scoring.\n\nThe host has confirmed in multiple discussion threads that the hidden test set follows the same formatting conventions as described in the Dataset Instructions, including normalized gaps (The test.csv does look like it has broken transliterations. Would be good to hear from the Hosts to make sure if the issue is true only for the test.csv and not for the hidden test as well.
Considering the length differences, aside from what @daylighth posted about <gap>/<big_gap> normalization, it's worth comparing the data to the \"Sentences_Oare_FirstWord_LinNum.csv\" as they are split into smaller chunks, similar to how the hidden test looks like.
你的发现完全是对的，但是我把所有test出现可能是问题的 a „ mup 替换后，其实没有得到任何增加，至于test的答案应该是train里的3e87aad8-cd8c-4f8d-96e9-c68e062d4a1a，而且你可以发现，在Overview里，主办方给的submission答案，就是这个，lol
", "rawMarkdown": "你的发现完全是对的，但是我把所有test出现可能是问题的 a „ mup 替换后，其实没有得到任何增加，至于test的答案应该是train里的3e87aad8-cd8c-4f8d-96e9-c68e062d4a1a，而且你可以发现，在Overview里，主办方给的submission答案，就是这个，lol", "votes": null }, { "id": "3385650", "postDate": "01/03/2026 16:53:42", "content": "OK, thank you for your reply. However, after conducting some post-processing on the training data, I discovered that certain post-processing techniques could improve the score during validation, but the same character processing had a negative impact on the test set. This might require the official to release the actual dictionary.
", "rawMarkdown": "OK, thank you for your reply. However, after conducting some post-processing on the training data, I discovered that certain post-processing techniques could improve the score during validation, but the same character processing had a negative impact on the test set. This might require the official to release the actual dictionary.", "votes": null }, { "id": "3385652", "postDate": "01/03/2026 16:54:59", "content": "OK, thank you for your reply. After the gap was handled according to the official procedure, the score actually dropped. Has this happened to me alone, or is it a common occurrence?
", "rawMarkdown": "OK, thank you for your reply. After the gap was handled according to the official procedure, the score actually dropped. Has this happened to me alone, or is it a common occurrence?", "votes": null }, { "id": "3385653", "postDate": "01/03/2026 16:55:57", "content": "我觉得train和真正的test很有可能也有较大的分布偏差。并且不仅仅局限在官方所说的句子长度问题
\nI think there is also likely to be a significant distributional deviation in the train and the actual test sets. And this problem is not limited to the sentence length issue as stated by the official.
", "rawMarkdown": "我觉得train和真正的test很有可能也有较大的分布偏差。并且不仅仅局限在官方所说的句子长度问题\n\nI think there is also likely to be a significant distributional deviation in the train and the actual test sets. And this problem is not limited to the sentence length issue as stated by the official.", "votes": null }, { "id": "3385673", "postDate": "01/03/2026 17:26:15", "content": "The characters seen in the test set were released in another discussion.
", "rawMarkdown": "The characters seen in the test set were released in another discussion.", "votes": null }, { "id": "3385696", "postDate": "01/03/2026 18:10:25", "content": "Thanks. But where did you get the test set for analysis? test.csv is only a dummy file with no real information so I was wondering how did you analyze the test set…
", "rawMarkdown": "Thanks. But where did you get the test set for analysis? test.csv is only a dummy file with no real information so I was wondering how did you analyze the test set...", "votes": null }, { "id": "3385700", "postDate": "01/03/2026 18:15:58", "content": "Not you alone. I’ve worked through it.. 95% of my score is from my preprocessing pipeline.
", "rawMarkdown": "Not you alone. I’ve worked through it.. 95% of my score is from my preprocessing pipeline.", "votes": null }, { "id": "3385918", "postDate": "01/04/2026 08:43:59", "content": "About #1, here is the list of official transliteration characters provided by host here:\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209#3384200
\nš (shin), ṣ (tsade), ṭ (tet) are available in test set.\nḫ and Ḫ are available in train.csv and missing in test set.
\nnew_transliteration_list = [\n '-', 'a', 'A', 'i', 'I', 'u', 'U', 'm', 'M', 'š', 'Š', 'n', 'N', 'b', 'B', \n 'r', 'R', 't', 'T', 'l', 'L', 'k', 'K', 'G', 'g', 'í', 'Í', 'D', 'd', 'Ù', \n 'ù', 'á', 'Á', '.', 'ú', 'Ú', 'p', 'P', 'e', 'E', 'h', 'H', 'q', 'Q', '1', \n 'ṣ', 'Ṣ', 'é', 'É', '<', '>', 'à', 'À', '4', 'z', 'Z', 's', 'S', 'ì', 'Ì', \n '5', '_', '2', '0', '½', 'w', 'W', '3', '{', '}', 'ṭ', 'Ṭ', '6', '⅓', '8', \n '⅔', '7', 'è', 'È', '⅚', '9', '¼', '!', '+', '⅙', 'ı', '…', 'ş', 'İ', ':'\n]\nStandard Alpha a, A, b, B, e, E...\nSpecial Transliteration š, Š, ṣ, Ṣ, ṭ, Ṭ, ı, ş, İ\nAccented Vowels í, Í, Ù, ù, á, Á, ú, Ú, é, É, à, À, ì, Ì, è, È\nNumbers & Fractions 0-9, ½, ⅓, ⅔, ⅚, ¼, ⅙\nPunctuation & Symbols -, ., <, >, _, {, }, !, +, …, :\n\n\nPlease note that this is a Code Competition. The data in
\ntest.csvis only dummy data to help you author your solutions. When your submission is scored, this example test data will be replaced with the full test set.
So the test.csv you analyzed is just placeholder data for debugging purposes — it's not the actual hidden test set used for scoring.
\nThe host has confirmed in multiple discussion threads that the hidden test set follows the same formatting conventions as described in the Dataset Instructions, including normalized gaps (, ), proper subscript handling, and Unicode fractions.
\nSee this thread for the host's confirmation.
", "votes": null, "replies": [ { "id": 3385652, "author_name": "ruichardliu", "author_url": "", "post_date": "01/03/2026 16:54:59", "content": "OK, thank you for your reply. After the gap was handled according to the official procedure, the score actually dropped. Has this happened to me alone, or is it a common occurrence?
", "votes": null, "replies": [ { "id": 3385700, "author_name": "jackvd", "author_url": "", "post_date": "01/03/2026 18:15:58", "content": "Not you alone. I’ve worked through it.. 95% of my score is from my preprocessing pipeline.
", "votes": null, "replies": [] } ] } ] }, { "id": 3385431, "author_name": "angantyr", "author_url": "", "post_date": "01/03/2026 09:31:23", "content": "\nThe test.csv does look like it has broken transliterations. Would be good to hear from the Hosts to make sure if the issue is true only for the test.csv and not for the hidden test as well.
Considering the length differences, aside from what @daylighth posted about <gap>/<big_gap> normalization, it's worth comparing the data to the \"Sentences_Oare_FirstWord_LinNum.csv\" as they are split into smaller chunks, similar to how the hidden test looks like.
OK, thank you for your reply. However, after conducting some post-processing on the training data, I discovered that certain post-processing techniques could improve the score during validation, but the same character processing had a negative impact on the test set. This might require the official to release the actual dictionary.
", "votes": null, "replies": [ { "id": 3385673, "author_name": "jackvd", "author_url": "", "post_date": "01/03/2026 17:26:15", "content": "The characters seen in the test set were released in another discussion.
", "votes": null, "replies": [] } ] } ] }, { "id": 3385467, "author_name": "qifeihhh666", "author_url": "", "post_date": "01/03/2026 11:14:41", "content": "你的发现完全是对的，但是我把所有test出现可能是问题的 a „ mup 替换后，其实没有得到任何增加，至于test的答案应该是train里的3e87aad8-cd8c-4f8d-96e9-c68e062d4a1a，而且你可以发现，在Overview里，主办方给的submission答案，就是这个，lol
", "votes": null, "replies": [ { "id": 3385653, "author_name": "ruichardliu", "author_url": "", "post_date": "01/03/2026 16:55:57", "content": "我觉得train和真正的test很有可能也有较大的分布偏差。并且不仅仅局限在官方所说的句子长度问题
\nI think there is also likely to be a significant distributional deviation in the train and the actual test sets. And this problem is not limited to the sentence length issue as stated by the official.
", "votes": null, "replies": [] } ] }, { "id": 3385696, "author_name": "honganzhu", "author_url": "", "post_date": "01/03/2026 18:10:25", "content": "Thanks. But where did you get the test set for analysis? test.csv is only a dummy file with no real information so I was wondering how did you analyze the test set…
", "votes": null, "replies": [] }, { "id": 3385918, "author_name": "mpware", "author_url": "", "post_date": "01/04/2026 08:43:59", "content": "About #1, here is the list of official transliteration characters provided by host here:\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209#3384200
\nš (shin), ṣ (tsade), ṭ (tet) are available in test set.\nḫ and Ḫ are available in train.csv and missing in test set.
\nnew_transliteration_list = [\n '-', 'a', 'A', 'i', 'I', 'u', 'U', 'm', 'M', 'š', 'Š', 'n', 'N', 'b', 'B', \n 'r', 'R', 't', 'T', 'l', 'L', 'k', 'K', 'G', 'g', 'í', 'Í', 'D', 'd', 'Ù', \n 'ù', 'á', 'Á', '.', 'ú', 'Ú', 'p', 'P', 'e', 'E', 'h', 'H', 'q', 'Q', '1', \n 'ṣ', 'Ṣ', 'é', 'É', '<', '>', 'à', 'À', '4', 'z', 'Z', 's', 'S', 'ì', 'Ì', \n '5', '_', '2', '0', '½', 'w', 'W', '3', '{', '}', 'ṭ', 'Ṭ', '6', '⅓', '8', \n '⅔', '7', 'è', 'È', '⅚', '9', '¼', '!', '+', '⅙', 'ı', '…', 'ş', 'İ', ':'\n]\nStandard Alpha a, A, b, B, e, E...\nSpecial Transliteration š, Š, ṣ, Ṣ, ṭ, Ṭ, ı, ş, İ\nAccented Vowels í, Í, Ù, ù, á, Á, ú, Ú, é, É, à, À, ì, Ì, è, È\nNumbers & Fractions 0-9, ½, ⅓, ⅔, ⅚, ¼, ⅙\nPunctuation & Symbols -, ., <, >, _, {, }, !, +, …, :\n