{ "id": 670467, "title": "PDF Dataset", "url": "/competitions/deep-past-initiative-machine-translation/discussion/670467", "author_name": "", "post_date": "2026-01-28T06:31:49.114851100Z", "votes": 5, "comment_count": 1, "views": 0, "content": "

Hi guys,\nI am extracting part of a dataset from PDF publications and translating it into fluent English.

\n

Before:

\n

1.

\n

Can the host add data augmentation?

\n
id,text_id,line_start,line_end,transliteration\n0,332fda50,1,7,um-ma kà-ru-um kà-ni-ia-ma a-na aa-qí-il… da-tim aí-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi„-ma mup-pu-um aa a-lim(ki) i-li-kam\n
\n

transliteration in traning csv:

\n
um-ma kà-ru-um kà-ni-iš-ma a-na ša-qí-il₅ da-tim ší-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi₄-ma ṭup-pu-um ša a-lim(ki) i-li-kam \n
\n

translation in traning csv:

\n
\"Thus  Kanesh, say to the -payers, our messenger, every single colony, and the trading stations...\n
\n
\n

I found test-set augmentation in this mapping:

\n
map = {\n    'š': 'a', 'Š': 'A',\n    'ṭ': 'm', 'Ṭ': 'M',\n    'ḫ': '+', 'Ḫ': '+',\n    '₄': '„', '₅': '…',\n    '₆': '6', '₀': '0', '₁': '1', '₂': '2', '₃': '3'\n}\n
\n

ša-qí-il₅ → aa-qí-il… ??\nYou’re not kidding me, right?

\n
\n

2.

\n

I will extract a dataset from the top 200 PDFs where the flag is_akkadin == True.

\n

Dataset link:

\n
https://www.kaggle.com/datasets/sangrampatil5150/extra-pdf-dataset?select=translation_pdf_200.csv\n
\n

If you like it, please upvote so you can also see other competitors.

\n
\n

3.

\n

When I see this kind of test set, I get confused about whether the test or hidden labels are really high quality for evaluation.

\n

In the official data, we are given 1,561 samples.\nIf I remove samples based on the ratio condition:

\n
lower_limit = 0.4\nupper_limit = 2.0\n\ntrain_df = train_df[\n    (train_df['ratio'] >= lower_limit) & \n    (train_df['ratio'] <= upper_limit)\n].copy()\n
\n

we are left with only 1,490 samples. That means 71 samples are either over-translated or under-translated.

\n

This doesn’t feel correct. I doubt whether the hidden public and private test sets also contain the same ratio distribution.

\n
\n

4.

\n
This leaderboard is calculated using approximately 34% of the test data.\nThe final results will be based on the remaining 66%, so the final standings may differ.\n
\n

No matter how much we generalize the model, if the ground truth is not perfect, the evaluation is not reliable.

\n

My most generalized model achieved 30.9 on the public leaderboard (34%).\nIf the remaining 66% follows the same pattern, this setup creates real uncertainty.

", "messages": [ { "id": "3397853", "postDate": "01/28/2026 06:31:49", "content": "

Hi guys,\nI am extracting part of a dataset from PDF publications and translating it into fluent English.

\n

Before:

\n

1.

\n

Can the host add data augmentation?

\n
id,text_id,line_start,line_end,transliteration\n0,332fda50,1,7,um-ma kà-ru-um kà-ni-ia-ma a-na aa-qí-il… da-tim aí-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi„-ma mup-pu-um aa a-lim(ki) i-li-kam\n
\n

transliteration in traning csv:

\n
um-ma kà-ru-um kà-ni-iš-ma a-na ša-qí-il₅ da-tim ší-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi₄-ma ṭup-pu-um ša a-lim(ki) i-li-kam \n
\n

translation in traning csv:

\n
\"Thus  Kanesh, say to the -payers, our messenger, every single colony, and the trading stations...\n
\n
\n

I found test-set augmentation in this mapping:

\n
map = {\n    'š': 'a', 'Š': 'A',\n    'ṭ': 'm', 'Ṭ': 'M',\n    'ḫ': '+', 'Ḫ': '+',\n    '₄': '„', '₅': '…',\n    '₆': '6', '₀': '0', '₁': '1', '₂': '2', '₃': '3'\n}\n
\n

ša-qí-il₅ → aa-qí-il… ??\nYou’re not kidding me, right?

\n
\n

2.

\n

I will extract a dataset from the top 200 PDFs where the flag is_akkadin == True.

\n

Dataset link:

\n
https://www.kaggle.com/datasets/sangrampatil5150/extra-pdf-dataset?select=translation_pdf_200.csv\n
\n

If you like it, please upvote so you can also see other competitors.

\n
\n

3.

\n

When I see this kind of test set, I get confused about whether the test or hidden labels are really high quality for evaluation.

\n

In the official data, we are given 1,561 samples.\nIf I remove samples based on the ratio condition:

\n
lower_limit = 0.4\nupper_limit = 2.0\n\ntrain_df = train_df[\n    (train_df['ratio'] >= lower_limit) & \n    (train_df['ratio'] <= upper_limit)\n].copy()\n
\n

we are left with only 1,490 samples. That means 71 samples are either over-translated or under-translated.

\n

This doesn’t feel correct. I doubt whether the hidden public and private test sets also contain the same ratio distribution.

\n
\n

4.

\n
This leaderboard is calculated using approximately 34% of the test data.\nThe final results will be based on the remaining 66%, so the final standings may differ.\n
\n

No matter how much we generalize the model, if the ground truth is not perfect, the evaluation is not reliable.

\n

My most generalized model achieved 30.9 on the public leaderboard (34%).\nIf the remaining 66% follows the same pattern, this setup creates real uncertainty.

", "rawMarkdown": "Hi guys,\nI am extracting part of a dataset from PDF publications and translating it into fluent English.\n\n### Before:\n\n### 1.\n\nCan the host add data augmentation?\n```\nid,text_id,line_start,line_end,transliteration\n0,332fda50,1,7,um-ma kà-ru-um kà-ni-ia-ma a-na aa-qí-il… da-tim aí-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi„-ma mup-pu-um aa a-lim(ki) i-li-kam\n```\n\n\ntransliteration in traning csv:\n```\num-ma kà-ru-um kà-ni-iš-ma a-na ša-qí-il₅ da-tim ší-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi₄-ma ṭup-pu-um ša a-lim(ki) i-li-kam \n```\n\n\ntranslation in traning csv:\n```\n\"Thus Kanesh, say to the -payers, our messenger, every single colony, and the trading stations...\n```\n\n---\n\nI found test-set augmentation in this mapping:\n\n```python\nmap = {\n 'š': 'a', 'Š': 'A',\n 'ṭ': 'm', 'Ṭ': 'M',\n 'ḫ': '+', 'Ḫ': '+',\n '₄': '„', '₅': '…',\n '₆': '6', '₀': '0', '₁': '1', '₂': '2', '₃': '3'\n}\n```\n\n`ša-qí-il₅` → `aa-qí-il…` ??\nYou’re not kidding me, right?\n\n---\n\n### 2.\n\nI will extract a dataset from the top 200 PDFs where the flag `is_akkadin == True`.\n\nDataset link:\n\n```\nhttps://www.kaggle.com/datasets/sangrampatil5150/extra-pdf-dataset?select=translation_pdf_200.csv\n```\n\n*If you like it, please upvote so you can also see other competitors.*\n\n---\n\n### 3.\n\nWhen I see this kind of test set, I get confused about whether the test or hidden labels are really high quality for evaluation.\n\nIn the official data, we are given 1,561 samples.\nIf I remove samples based on the ratio condition:\n\n```python\nlower_limit = 0.4\nupper_limit = 2.0\n\ntrain_df = train_df[\n (train_df['ratio'] >= lower_limit) & \n (train_df['ratio'] <= upper_limit)\n].copy()\n```\n\nwe are left with only 1,490 samples. That means 71 samples are either over-translated or under-translated.\n\nThis doesn’t feel correct. I doubt whether the hidden public and private test sets also contain the same ratio distribution.\n\n---\n\n### 4.\n\n```\nThis leaderboard is calculated using approximately 34% of the test data.\nThe final results will be based on the remaining 66%, so the final standings may differ.\n```\n\nNo matter how much we generalize the model, if the ground truth is not perfect, the evaluation is not reliable.\n\nMy most generalized model achieved **30.9** on the public leaderboard (34%).\nIf the remaining 66% follows the same pattern, this setup creates real uncertainty.", "votes": null }, { "id": "3397930", "postDate": "01/28/2026 09:53:45", "content": "

I wouldn't follow the character mapping. This is likely an issue with the test.csv not the entire Hidden Test Set.

", "rawMarkdown": "I wouldn't follow the character mapping. This is likely an issue with the `test.csv` not the entire Hidden Test Set.", "votes": null } ], "comments": [ { "id": 3397930, "author_name": "angantyr", "author_url": "", "post_date": "01/28/2026 09:53:45", "content": "

I wouldn't follow the character mapping. This is likely an issue with the test.csv not the entire Hidden Test Set.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3397853": "Hi guys,\nI am extracting part of a dataset from PDF publications and translating it into fluent English.\n\n### Before:\n\n### 1.\n\nCan the host add data augmentation?\n```\nid,text_id,line_start,line_end,transliteration\n0,332fda50,1,7,um-ma kà-ru-um kà-ni-ia-ma a-na aa-qí-il… da-tim aí-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi„-ma mup-pu-um aa a-lim(ki) i-li-kam\n```\n\n\ntransliteration in traning csv:\n```\num-ma kà-ru-um kà-ni-iš-ma a-na ša-qí-il₅ da-tim ší-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi₄-ma ṭup-pu-um ša a-lim(ki) i-li-kam \n```\n\n\ntranslation in traning csv:\n```\n\"Thus Kanesh, say to the -payers, our messenger, every single colony, and the trading stations...\n```\n\n---\n\nI found test-set augmentation in this mapping:\n\n```python\nmap = {\n 'š': 'a', 'Š': 'A',\n 'ṭ': 'm', 'Ṭ': 'M',\n 'ḫ': '+', 'Ḫ': '+',\n '₄': '„', '₅': '…',\n '₆': '6', '₀': '0', '₁': '1', '₂': '2', '₃': '3'\n}\n```\n\n`ša-qí-il₅` → `aa-qí-il…` ??\nYou’re not kidding me, right?\n\n---\n\n### 2.\n\nI will extract a dataset from the top 200 PDFs where the flag `is_akkadin == True`.\n\nDataset link:\n\n```\nhttps://www.kaggle.com/datasets/sangrampatil5150/extra-pdf-dataset?select=translation_pdf_200.csv\n```\n\n*If you like it, please upvote so you can also see other competitors.*\n\n---\n\n### 3.\n\nWhen I see this kind of test set, I get confused about whether the test or hidden labels are really high quality for evaluation.\n\nIn the official data, we are given 1,561 samples.\nIf I remove samples based on the ratio condition:\n\n```python\nlower_limit = 0.4\nupper_limit = 2.0\n\ntrain_df = train_df[\n (train_df['ratio'] >= lower_limit) & \n (train_df['ratio'] <= upper_limit)\n].copy()\n```\n\nwe are left with only 1,490 samples. That means 71 samples are either over-translated or under-translated.\n\nThis doesn’t feel correct. I doubt whether the hidden public and private test sets also contain the same ratio distribution.\n\n---\n\n### 4.\n\n```\nThis leaderboard is calculated using approximately 34% of the test data.\nThe final results will be based on the remaining 66%, so the final standings may differ.\n```\n\nNo matter how much we generalize the model, if the ground truth is not perfect, the evaluation is not reliable.\n\nMy most generalized model achieved **30.9** on the public leaderboard (34%).\nIf the remaining 66% follows the same pattern, this setup creates real uncertainty.", "3397930": "I wouldn't follow the character mapping. This is likely an issue with the `test.csv` not the entire Hidden Test Set." }, "source": "meta" }