![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2064220%2F491e90cf7b492d7e0676a56ab0ca084f%2Fakt12_page043_crop.png?generation=1774512354416002&alt=media\")
Thank you to the organizers for hosting this competition. The two key elements of our approach were extracting ~20K transliteration–translation pairs from published scholarly PDFs using a VLM-based pipeline, and generating synthetic data via pseudo-labeling and back-translation.
\nInitially, we fine-tuned in two stages — first on EvaCun (ORACC Akkadian Parallel Corpus), then on PDF-extracted data — so we describe the preparation for each below.
\nCompetition train.csv has only 1,561 Akkadian–English pairs. We built ~20,251 pairs by extracting transliteration–translation pairs directly from the AKT series and other published scholarly PDFs (20 volumes total) using a VLM-based 4-phase pipeline: Detect excavation numbers → Link references → Extract pairs → Translate to English. train.csv was only used as reference within the pipeline (linking and few-shot).
| Volume | \nLanguage | \nPairs | \n
|---|---|---|
| AKT 1–4 | \nTurkish / German | \n1,924 | \n
| AKT 5, 6A–6E, 8, 12 | \nEnglish | \n9,202 | \n
| AKT 7A–7B, 9A, 10, 11A–11B | \nTurkish | \n6,517 | \n
| Larsen 2002 (PIHANS 96) | \nEnglish | \n1,199 | \n
| ICK 4 (1998) | \nGerman | \n1,415 | \n
| Total | \n\n | ~20,251 | \n
PDF pages are rendered to PNG images and sent to Qwen 3.5 Plus (397B) to detect excavation numbers (tablet headings like \"Kt. 86/k 39\", \"I 427\") and their page numbers.
Detected excavation numbers are matched against the published_texts.csv to retrieve known transliterations and against train.csv to retrieve known English translations. Both are provided to the VLM as reference in Phase 3. Matching handles notation variants by trying multiple alias formats (e.g., \"Kt n/k 594\" ↔ \"Kt. n/k 594\", museum numbers, pipe-separated aliases).
For each excavation number, the VLM is given the page image(s) where it appears along with the linked references as context, and extracts transliteration line-by-line and translation by the grouping defined in the PDF (e.g., lines 1–3 grouped together, line 4 alone). We chose this granularity because we expected the test data to be segmented at these same group boundaries.
\nThe PDFs have two distinct layouts, so we prepared separate extraction prompts for each:
\n
{\n \"transliteration_lines\": {\n \"1\": \"[ ]-ši-nim\",\n \"2\": \"Šu-Sú-in : lu i-na GÁ[N]\",\n \"3\": \"lu i-na a-lim\",\n \"5\": \"i-ta-áb-ši : ša A-[mur]-A-š[ùr]\",\n \"6\": \"E-mu-a ù-lá ṭá-hu\",\n \"7\": \"A-mur-A-šùr ù E-mu-a\",\n \"8\": \"zi-zu-ú-[ma] : a-hu-um\",\n \"9\": \"a-na a-hi-im a-n[a KÙ.B]\",\n \"10\": \"ù-lá i-tù-[ar]\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1\": \"If silver of\",\n \"2-4\": \"Šū-Suen proves to be either in the countryside (Anatolia) or in the City, is belongs to Amur-Aššur,\",\n \"5-10\": \"Emua is not entitled to it. Amur-Aššur and Emua have divided (the inheritance) and they will not raise claims against each other for silver.\"\n \"...\": \"...\"\n }\n}\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2064220%2Fba4676b27f78bf466909751a68cd7a72%2Fakt9a_page157_crop.png?generation=1774512324980388&alt=media\")
{\n \"transliteration_lines\": {\n \"1\": \"15 GÍN KÙ.BABBAR\",\n \"2\": \"i-ṣé-er\",\n \"3\": \"Ḫa-šu-i-li\",\n \"4\": \"Šu-Ištar i-šu a-na\",\n \"5\": \"ku-bu-ur ú-ṭí-tim\",\n \"6\": \"i-ša-qal\",\n \"7\": \"šu-ma lá iš-qúl\",\n \"8\": \"2/3 GÍN 15 ŠE.TA\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1-4\": \"Šu-Ištar'ın, Ḫaššuli'nin üzerinde 15 šeḳel gümüšü vardır.\",\n \"4-6\": \"(Ḫaššuli) hububatın olgunlaşmasında ödeyecek.\",\n \"...\": \"...\"\n }\n}\nIn practice, tablet records often span multiple pages. To handle this, the target page and subsequent context pages are stitched into a single image before sending to the VLM. Some source PDFs had misordered pages, so we manually fixed the page order before running the extraction pipeline. For example, in AKT 1 we observed page sequences such as 73, 76, 77, 78, 79, 78, 79, 74, suggesting page ordering and duplication issues in the source PDF.
\nAfter extraction, a validation step checks for inconsistencies (e.g., transliteration present but translation missing, uncovered line ranges). Failed records are re-sent to the VLM with a description of what went wrong, prompting it to correct the extraction.
\nTurkish/German volumes are translated to English using Gemini 3.1 Flash Lite with similar few-shot examples retrieved by Levenshtein distance on transliterations from train.csv.
In addition to the standard preprocessing used for the competition dataset, we applied an extra normalization step to the transliteration side of EvaCun: immediately repeated consecutive words were collapsed into a single token. This was motivated by the fact that the transliterations occasionally contain local repetition artifacts that are unlikely to be meaningful for translation and can introduce unnecessary noise during training.
\nFor example, the transliteration
\nša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te É-hi-il-la-na-te ma-a im-ma-te im-ma-te ú-šá-ra-qu
was normalized to
\nša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te ma-a im-ma-te ú-šá-ra-qu
Training on ByT5 variants (best submission used byt5-xl):
\nStage 1: Build base models
\npublished_texts.csv.A reverse model (English → Akkadian) is also trained with the same pipeline by swapping transliteration and translation.
\nStage 2: Synthetic Data Generation
\nUsing the reverse model from Stage 1, we generate additional training pairs:
\npublished_texts.csv transliterations to English with the forward model using beam search.train.csv and the PDF-extracted data as few-shot examples, and then back-translate them into Akkadian with the reverse model.The generated dataset is here
\nWe additionally tested scaling this back-translation data to 100,000 examples. Relative to training with 10,000 examples, this resulted in a -0.1 change on the public leaderboard but a +0.3 improvement on the private leaderboard.
\nStage 3: Final training
\nStarting from the Stage 1 EvaCun checkpoint:
\nprompt format:
\n[context] <prev_line_1> [sep] <prev_line_2> [/context] <current_input>\nWe convert trained models to CTranslate2 format, achieving ~5x speedup over HuggingFace inference. CTranslate2 also enables ByT5-xxl inference on Kaggle's dual T4 GPUs via int8_float32 quantization and tensor parallelism (we patched CTranslate2's source to fix a bug in its TP implementation). Note that int8_float32 quantization degrades LB score by ~-0.1. xxl was not used in our best submission due to insufficient training time (We changed the training dataset in the last day of the competition).
We average weights from multiple CV fold models into a single model before inference. At inference time, we use line_start from test.csv to prepend context from previous lines of the same tablet, matching the training format. Decoding is done with beam search.
We used 25% of train.csv as a hold-out set. The split was stratified by the transliteration/translation length ratio to preserve the distribution of examples with different source–target length characteristics.
In practice, validation loss (token-level cross entropy loss) was reasonably well correlated with LB score, so we treated it as the main offline metric throughout development. As a result, we prioritized changes that reliably improved validation loss, and used the hold-out set for checkpoint selection and ablation comparison.
\nAll models in the ablation study are ByT5-xl trained with the same parameters (e.g., learning rate, batch size, number of steps)
\n| Model | \nCPT | \nEvaCun | \nSynthetic data | \nTablet context | \nPrivate LB | \nPublic LB | \n
|---|---|---|---|---|---|---|
| PDF data only | \n\n | \n | \n | \n | 35.4 | \n35.1 | \n
| + CPT | \n✓ | \n\n | \n | \n | 38.9 | \n38.0 | \n
| + EvaCun | \n✓ | \n✓ | \n\n | \n | 39.7 | \n38.7 | \n
| + tablet context | \n✓ | \n✓ | \n\n | ✓ | \n39.8 | \n39.2 | \n
| + synthetic data | \n✓ | \n✓ | \n✓ | \n✓ | \n40.7 | \n40.2 | \n
| + 4 fold weight average | \n✓ | \n✓ | \n✓ | \n✓ | \n40.8 | \n40.1 | \n
Thank you to the organizers for hosting this competition. The two key elements of our approach were extracting ~20K transliteration–translation pairs from published scholarly PDFs using a VLM-based pipeline, and generating synthetic data via pseudo-labeling and back-translation.
\nInitially, we fine-tuned in two stages — first on EvaCun (ORACC Akkadian Parallel Corpus), then on PDF-extracted data — so we describe the preparation for each below.
\nCompetition train.csv has only 1,561 Akkadian–English pairs. We built ~20,251 pairs by extracting transliteration–translation pairs directly from the AKT series and other published scholarly PDFs (20 volumes total) using a VLM-based 4-phase pipeline: Detect excavation numbers → Link references → Extract pairs → Translate to English. train.csv was only used as reference within the pipeline (linking and few-shot).
| Volume | \nLanguage | \nPairs | \n
|---|---|---|
| AKT 1–4 | \nTurkish / German | \n1,924 | \n
| AKT 5, 6A–6E, 8, 12 | \nEnglish | \n9,202 | \n
| AKT 7A–7B, 9A, 10, 11A–11B | \nTurkish | \n6,517 | \n
| Larsen 2002 (PIHANS 96) | \nEnglish | \n1,199 | \n
| ICK 4 (1998) | \nGerman | \n1,415 | \n
| Total | \n\n | ~20,251 | \n
PDF pages are rendered to PNG images and sent to Qwen 3.5 Plus (397B) to detect excavation numbers (tablet headings like \"Kt. 86/k 39\", \"I 427\") and their page numbers.
Detected excavation numbers are matched against the published_texts.csv to retrieve known transliterations and against train.csv to retrieve known English translations. Both are provided to the VLM as reference in Phase 3. Matching handles notation variants by trying multiple alias formats (e.g., \"Kt n/k 594\" ↔ \"Kt. n/k 594\", museum numbers, pipe-separated aliases).
For each excavation number, the VLM is given the page image(s) where it appears along with the linked references as context, and extracts transliteration line-by-line and translation by the grouping defined in the PDF (e.g., lines 1–3 grouped together, line 4 alone). We chose this granularity because we expected the test data to be segmented at these same group boundaries.
\nThe PDFs have two distinct layouts, so we prepared separate extraction prompts for each:
\n
{\n \"transliteration_lines\": {\n \"1\": \"[ ]-ši-nim\",\n \"2\": \"Šu-Sú-in : lu i-na GÁ[N]\",\n \"3\": \"lu i-na a-lim\",\n \"5\": \"i-ta-áb-ši : ša A-[mur]-A-š[ùr]\",\n \"6\": \"E-mu-a ù-lá ṭá-hu\",\n \"7\": \"A-mur-A-šùr ù E-mu-a\",\n \"8\": \"zi-zu-ú-[ma] : a-hu-um\",\n \"9\": \"a-na a-hi-im a-n[a KÙ.B]\",\n \"10\": \"ù-lá i-tù-[ar]\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1\": \"If silver of\",\n \"2-4\": \"Šū-Suen proves to be either in the countryside (Anatolia) or in the City, is belongs to Amur-Aššur,\",\n \"5-10\": \"Emua is not entitled to it. Amur-Aššur and Emua have divided (the inheritance) and they will not raise claims against each other for silver.\"\n \"...\": \"...\"\n }\n}\n
{\n \"transliteration_lines\": {\n \"1\": \"15 GÍN KÙ.BABBAR\",\n \"2\": \"i-ṣé-er\",\n \"3\": \"Ḫa-šu-i-li\",\n \"4\": \"Šu-Ištar i-šu a-na\",\n \"5\": \"ku-bu-ur ú-ṭí-tim\",\n \"6\": \"i-ša-qal\",\n \"7\": \"šu-ma lá iš-qúl\",\n \"8\": \"2/3 GÍN 15 ŠE.TA\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1-4\": \"Šu-Ištar'ın, Ḫaššuli'nin üzerinde 15 šeḳel gümüšü vardır.\",\n \"4-6\": \"(Ḫaššuli) hububatın olgunlaşmasında ödeyecek.\",\n \"...\": \"...\"\n }\n}\nIn practice, tablet records often span multiple pages. To handle this, the target page and subsequent context pages are stitched into a single image before sending to the VLM. Some source PDFs had misordered pages, so we manually fixed the page order before running the extraction pipeline. For example, in AKT 1 we observed page sequences such as 73, 76, 77, 78, 79, 78, 79, 74, suggesting page ordering and duplication issues in the source PDF.
\nAfter extraction, a validation step checks for inconsistencies (e.g., transliteration present but translation missing, uncovered line ranges). Failed records are re-sent to the VLM with a description of what went wrong, prompting it to correct the extraction.
\nTurkish/German volumes are translated to English using Gemini 3.1 Flash Lite with similar few-shot examples retrieved by Levenshtein distance on transliterations from train.csv.
In addition to the standard preprocessing used for the competition dataset, we applied an extra normalization step to the transliteration side of EvaCun: immediately repeated consecutive words were collapsed into a single token. This was motivated by the fact that the transliterations occasionally contain local repetition artifacts that are unlikely to be meaningful for translation and can introduce unnecessary noise during training.
\nFor example, the transliteration
\nša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te É-hi-il-la-na-te ma-a im-ma-te im-ma-te ú-šá-ra-qu
was normalized to
\nša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te ma-a im-ma-te ú-šá-ra-qu
Training on ByT5 variants (best submission used byt5-xl):
\nStage 1: Build base models
\npublished_texts.csv.A reverse model (English → Akkadian) is also trained with the same pipeline by swapping transliteration and translation.
\nStage 2: Synthetic Data Generation
\nUsing the reverse model from Stage 1, we generate additional training pairs:
\npublished_texts.csv transliterations to English with the forward model using beam search.train.csv and the PDF-extracted data as few-shot examples, and then back-translate them into Akkadian with the reverse model.The generated dataset is here
\nWe additionally tested scaling this back-translation data to 100,000 examples. Relative to training with 10,000 examples, this resulted in a -0.1 change on the public leaderboard but a +0.3 improvement on the private leaderboard.
\nStage 3: Final training
\nStarting from the Stage 1 EvaCun checkpoint:
\nprompt format:
\n[context] <prev_line_1> [sep] <prev_line_2> [/context] <current_input>\nWe convert trained models to CTranslate2 format, achieving ~5x speedup over HuggingFace inference. CTranslate2 also enables ByT5-xxl inference on Kaggle's dual T4 GPUs via int8_float32 quantization and tensor parallelism (we patched CTranslate2's source to fix a bug in its TP implementation). Note that int8_float32 quantization degrades LB score by ~-0.1. xxl was not used in our best submission due to insufficient training time (We changed the training dataset in the last day of the competition).
We average weights from multiple CV fold models into a single model before inference. At inference time, we use line_start from test.csv to prepend context from previous lines of the same tablet, matching the training format. Decoding is done with beam search.
We used 25% of train.csv as a hold-out set. The split was stratified by the transliteration/translation length ratio to preserve the distribution of examples with different source–target length characteristics.
In practice, validation loss (token-level cross entropy loss) was reasonably well correlated with LB score, so we treated it as the main offline metric throughout development. As a result, we prioritized changes that reliably improved validation loss, and used the hold-out set for checkpoint selection and ablation comparison.
\nAll models in the ablation study are ByT5-xl trained with the same parameters (e.g., learning rate, batch size, number of steps)
\n| Model | \nCPT | \nEvaCun | \nSynthetic data | \nTablet context | \nPrivate LB | \nPublic LB | \n
|---|---|---|---|---|---|---|
| PDF data only | \n\n | \n | \n | \n | 35.4 | \n35.1 | \n
| + CPT | \n✓ | \n\n | \n | \n | 38.9 | \n38.0 | \n
| + EvaCun | \n✓ | \n✓ | \n\n | \n | 39.7 | \n38.7 | \n
| + tablet context | \n✓ | \n✓ | \n\n | ✓ | \n39.8 | \n39.2 | \n
| + synthetic data | \n✓ | \n✓ | \n✓ | \n✓ | \n40.7 | \n40.2 | \n
| + 4 fold weight average | \n✓ | \n✓ | \n✓ | \n✓ | \n40.8 | \n40.1 | \n
\n\n```json\n{\n \"transliteration_lines\": {\n \"1\": \"[ ]-ši-nim\",\n \"2\": \"Šu-Sú-in : lu i-na GÁ[N]\",\n \"3\": \"lu i-na a-lim\",\n \"5\": \"i-ta-áb-ši : ša A-[mur]-A-š[ùr]\",\n \"6\": \"E-mu-a ù-lá ṭá-hu\",\n \"7\": \"A-mur-A-šùr ù E-mu-a\",\n \"8\": \"zi-zu-ú-[ma] : a-hu-um\",\n \"9\": \"a-na a-hi-im a-n[a KÙ.B]\",\n \"10\": \"ù-lá i-tù-[ar]\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1\": \"If silver of\",\n \"2-4\": \"Šū-Suen proves to be either in the countryside (Anatolia) or in the City, is belongs to Amur-Aššur,\",\n \"5-10\": \"Emua is not entitled to it. Amur-Aššur and Emua have divided (the inheritance) and they will not raise claims against each other for silver.\"\n \"...\": \"...\"\n }\n}\n```\n\n### Example: AKT 9A, Kt. c/k 1249 (sequential)\n\n\n\n```json\n{\n \"transliteration_lines\": {\n \"1\": \"15 GÍN KÙ.BABBAR\",\n \"2\": \"i-ṣé-er\",\n \"3\": \"Ḫa-šu-i-li\",\n \"4\": \"Šu-Ištar i-šu a-na\",\n \"5\": \"ku-bu-ur ú-ṭí-tim\",\n \"6\": \"i-ša-qal\",\n \"7\": \"šu-ma lá iš-qúl\",\n \"8\": \"2/3 GÍN 15 ŠE.TA\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1-4\": \"Šu-Ištar'ın, Ḫaššuli'nin üzerinde 15 šeḳel gümüšü vardır.\",\n \"4-6\": \"(Ḫaššuli) hububatın olgunlaşmasında ödeyecek.\",\n \"...\": \"...\"\n }\n}\n```\n\nIn practice, tablet records often span multiple pages. To handle this, the target page and subsequent context pages are stitched into a single image before sending to the VLM. Some source PDFs had misordered pages, so we manually fixed the page order before running the extraction pipeline. For example, in AKT 1 we observed page sequences such as 73, 76, 77, 78, 79, 78, 79, 74, suggesting page ordering and duplication issues in the source PDF.\n\nAfter extraction, a validation step checks for inconsistencies (e.g., transliteration present but translation missing, uncovered line ranges). Failed records are re-sent to the VLM with a description of what went wrong, prompting it to correct the extraction.\n\n### Phase 4: Translate to English\n\nTurkish/German volumes are translated to English using Gemini 3.1 Flash Lite with similar few-shot examples retrieved by Levenshtein distance on transliterations from `train.csv`.\n\n### Postprocessing\n- Quality filter: remove pairs with translation/transliteration length ratio outside [0.3, 4.0]\n\n### API Cost\n- Approximately 15$ for the entire extraction process, OpenRouter is used as the API gateway.\n\n## EvaCun(ORACC Akkadian Parallel Corpus) Preparation\n\n### Used Files\n- 45k transliteration-translation pairs from ranscription_train.txt, transcription_val.txt, english_train.txt, english_val.txt.\n\n### Preprocessing\n\nIn addition to the standard preprocessing used for the competition dataset, we applied an extra normalization step to the **transliteration side** of EvaCun: immediately repeated consecutive words were collapsed into a single token. This was motivated by the fact that the transliterations occasionally contain local repetition artifacts that are unlikely to be meaningful for translation and can introduce unnecessary noise during training.\n\nFor example, the transliteration\n\n`ša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te É-hi-il-la-na-te ma-a im-ma-te im-ma-te ú-šá-ra-qu`\n\nwas normalized to\n\n`ša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te ma-a im-ma-te ú-šá-ra-qu`\n\n# Training Pipeline\n\nTraining on ByT5 variants (best submission used byt5-xl):\n\n**Stage 1: Build base models**\n\n1. Continued pre-training — T5 span corruption on transliterations from `published_texts.csv`.\n2. Fine-tuning on EvaCun. \n3. Fine-tuning on PDF data.\n\nA reverse model (English → Akkadian) is also trained with the same pipeline by swapping transliteration and translation.\n\n**Stage 2: Synthetic Data Generation**\n\nUsing the reverse model from Stage 1, we generate additional training pairs:\n\n- Pseudo-label: translate `published_texts.csv` transliterations to English with the forward model using beam search.\n- Back-translation: first generate 10,000 translation-like English sentences with Qwen3.5-27B, using translations from both `train.csv` and the PDF-extracted data as few-shot examples, and then back-translate them into Akkadian with the reverse model.\n\nThe generated dataset is [here](https://www.kaggle.com/datasets/yukiokumura1/deep-past-challenge-back-translated-synthetic-data)\n\nWe additionally tested scaling this back-translation data to 100,000 examples. Relative to training with 10,000 examples, this resulted in a -0.1 change on the public leaderboard but a +0.3 improvement on the private leaderboard.\n\n**Stage 3: Final training**\n\nStarting from the Stage 1 EvaCun checkpoint:\n\n1. Fine-tuning on Stage 2 data.\n2. Fine-tuning on PDF data with context — previous lines of the same tablet are prepended to the input(max 2 previous lines):\n\nprompt format:\n```\n[context] Congratulations @ebinan92 and @yukiokumura1 on becoming Kaggle Competitions GMs!
", "rawMarkdown": "Congratulations @ebinan92 and @yukiokumura1 on becoming Kaggle Competitions GMs!", "votes": null }, { "id": "3440077", "postDate": "04/11/2026 21:14:59", "content": "Hey, the solution looks great! \nwould you mind sharing working inference code? I'm not managing to run the inference code right now, the one here, because of private dataset dependencies -
\nhttps://www.kaggle.com/code/ebinan92/5th-solution/notebook?scriptVersionId=305950408
\nThanks and congrats!
", "rawMarkdown": "Hey, the solution looks great! \nwould you mind sharing working inference code? I'm not managing to run the inference code right now, the one here, because of private dataset dependencies -\n\nhttps://www.kaggle.com/code/ebinan92/5th-solution/notebook?scriptVersionId=305950408\n\nThanks and congrats!", "votes": null }, { "id": "3440836", "postDate": "04/13/2026 05:06:30", "content": "@shaharspencer \nHey, thanks for reaching out! I believe the issue has been resolved in the latest version, so could you check it out?
", "rawMarkdown": "shaharspencer \nHey, thanks for reaching out! I believe the issue has been resolved in the latest version, so could you check it out?", "votes": null }, { "id": "3457905", "postDate": "05/14/2026 15:34:02", "content": "Hey Ebi, \nSorry for the late reply, I missed your reply. \nThank you so much, it does work now. The solution works really well, well done 🥳 and thanks again
", "rawMarkdown": "Hey Ebi, \nSorry for the late reply, I missed your reply. \nThank you so much, it does work now. The solution works really well, well done 🥳 and thanks again", "votes": null } ], "comments": [ { "id": 3429026, "author_name": "samson8", "author_url": "", "post_date": "03/26/2026 08:43:22", "content": "Congratulations @ebinan92 and @yukiokumura1 on becoming Kaggle Competitions GMs!
", "votes": null, "replies": [] }, { "id": 3440077, "author_name": "shaharspencer", "author_url": "", "post_date": "04/11/2026 21:14:59", "content": "Hey, the solution looks great! \nwould you mind sharing working inference code? I'm not managing to run the inference code right now, the one here, because of private dataset dependencies -
\nhttps://www.kaggle.com/code/ebinan92/5th-solution/notebook?scriptVersionId=305950408
\nThanks and congrats!
", "votes": null, "replies": [ { "id": 3440836, "author_name": "ebinan92", "author_url": "", "post_date": "04/13/2026 05:06:30", "content": "@shaharspencer \nHey, thanks for reaching out! I believe the issue has been resolved in the latest version, so could you check it out?
", "votes": null, "replies": [ { "id": 3457905, "author_name": "shaharspencer", "author_url": "", "post_date": "05/14/2026 15:34:02", "content": "Hey Ebi, \nSorry for the late reply, I missed your reply. \nThank you so much, it does work now. The solution works really well, well done 🥳 and thanks again
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3429009": "# Comment\nThank you to the organizers for hosting this competition. The two key elements of our approach were extracting ~20K transliteration–translation pairs from published scholarly PDFs using a VLM-based pipeline, and generating synthetic data via pseudo-labeling and back-translation.\n\n# Solution summary\n- Model: ByT5 variants (best submission: ByT5-xl)\n- Data: \n - PDF data: ~20K transliteration–translation pairs extracted from AKT series and other scholarly PDFs with a VLM-based pipeline\n - EvaCun (ORACC Akkadian Parallel Corpus)\n - Synthetic data generated via pseudo-labeling and back-translation\n- Training pipeline:\n - stage1: CPT on transliterations from published_test.csv → FT on EvaCun → FT on PDF data\n - stage2: Synthetic data generation via pseudo-labeling and back-translation\n - stage3: FT on synthetic data from EvaCun checkpoint → Final FT on PDF data with tablet context\n- Inference: CTranslate2 with weight-averaged fold models and beam search (no ensemble in test time)\n\n# Data Preparation\n\nInitially, we fine-tuned in two stages — first on [EvaCun (ORACC Akkadian Parallel Corpus)](https://zenodo.org/records/17220688), then on PDF-extracted data — so we describe the preparation for each below. \n\n## PDF Extraction Pipeline\n\nCompetition `train.csv` has only 1,561 Akkadian–English pairs. We built ~20,251 pairs by extracting transliteration–translation pairs directly from the AKT series and other published scholarly PDFs (20 volumes total) using a VLM-based 4-phase pipeline: Detect excavation numbers → Link references → Extract pairs → Translate to English. `train.csv` was only used as reference within the pipeline (linking and few-shot).\n\n| Volume | Language | Pairs |\n| --- | --- | --- |\n| AKT 1–4 | Turkish / German | 1,924 |\n| AKT 5, 6A–6E, 8, 12 | English | 9,202 |\n| AKT 7A–7B, 9A, 10, 11A–11B | Turkish | 6,517 |\n| Larsen 2002 (PIHANS 96) | English | 1,199 |\n| ICK 4 (1998) | German | 1,415 |\n| **Total** | | **~20,251** |\n\n### Phase 1: Detect excavation numbers\n\nPDF pages are rendered to PNG images and sent to Qwen 3.5 Plus (397B) to detect excavation numbers (tablet headings like `\"Kt. 86/k 39\"`, `\"I 427\"`) and their page numbers.\n\n### Phase 2: Link references\n\nDetected excavation numbers are matched against the `published_texts.csv` to retrieve known transliterations and against `train.csv` to retrieve known English translations. Both are provided to the VLM as reference in Phase 3. Matching handles notation variants by trying multiple alias formats (e.g., `\"Kt n/k 594\"` ↔ `\"Kt. n/k 594\"`, museum numbers, pipe-separated aliases).\n\n### Phase 3: Extract pairs\n\nFor each excavation number, the VLM is given the page image(s) where it appears along with the linked references as context, and extracts transliteration line-by-line and translation by the grouping defined in the PDF (e.g., lines 1–3 grouped together, line 4 alone). We chose this granularity because we expected the test data to be segmented at these same group boundaries.\n\nThe PDFs have two distinct layouts, so we prepared separate extraction prompts for each:\n\n- Side-by-side: transliteration and translation appear in parallel columns on the same page (English volumes: AKT 5, 6, 8, 12, Larsen)\n- Sequential: transliteration block followed by translation block below (Turkish/German volumes: AKT 1–4, 7, 9–11, ICK 4)\n\n### Example: AKT 12, Kt. 86/k 39 (side-by-side)\n\n\n\n```json\n{\n \"transliteration_lines\": {\n \"1\": \"[ ]-ši-nim\",\n \"2\": \"Šu-Sú-in : lu i-na GÁ[N]\",\n \"3\": \"lu i-na a-lim\",\n \"5\": \"i-ta-áb-ši : ša A-[mur]-A-š[ùr]\",\n \"6\": \"E-mu-a ù-lá ṭá-hu\",\n \"7\": \"A-mur-A-šùr ù E-mu-a\",\n \"8\": \"zi-zu-ú-[ma] : a-hu-um\",\n \"9\": \"a-na a-hi-im a-n[a KÙ.B]\",\n \"10\": \"ù-lá i-tù-[ar]\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1\": \"If silver of\",\n \"2-4\": \"Šū-Suen proves to be either in the countryside (Anatolia) or in the City, is belongs to Amur-Aššur,\",\n \"5-10\": \"Emua is not entitled to it. Amur-Aššur and Emua have divided (the inheritance) and they will not raise claims against each other for silver.\"\n \"...\": \"...\"\n }\n}\n```\n\n### Example: AKT 9A, Kt. c/k 1249 (sequential)\n\n\n\n```json\n{\n \"transliteration_lines\": {\n \"1\": \"15 GÍN KÙ.BABBAR\",\n \"2\": \"i-ṣé-er\",\n \"3\": \"Ḫa-šu-i-li\",\n \"4\": \"Šu-Ištar i-šu a-na\",\n \"5\": \"ku-bu-ur ú-ṭí-tim\",\n \"6\": \"i-ša-qal\",\n \"7\": \"šu-ma lá iš-qúl\",\n \"8\": \"2/3 GÍN 15 ŠE.TA\",\n \"...\": \"...\"\n },\n \"translation_ranges\": {\n \"1-4\": \"Šu-Ištar'ın, Ḫaššuli'nin üzerinde 15 šeḳel gümüšü vardır.\",\n \"4-6\": \"(Ḫaššuli) hububatın olgunlaşmasında ödeyecek.\",\n \"...\": \"...\"\n }\n}\n```\n\nIn practice, tablet records often span multiple pages. To handle this, the target page and subsequent context pages are stitched into a single image before sending to the VLM. Some source PDFs had misordered pages, so we manually fixed the page order before running the extraction pipeline. For example, in AKT 1 we observed page sequences such as 73, 76, 77, 78, 79, 78, 79, 74, suggesting page ordering and duplication issues in the source PDF.\n\nAfter extraction, a validation step checks for inconsistencies (e.g., transliteration present but translation missing, uncovered line ranges). Failed records are re-sent to the VLM with a description of what went wrong, prompting it to correct the extraction.\n\n### Phase 4: Translate to English\n\nTurkish/German volumes are translated to English using Gemini 3.1 Flash Lite with similar few-shot examples retrieved by Levenshtein distance on transliterations from `train.csv`.\n\n### Postprocessing\n- Quality filter: remove pairs with translation/transliteration length ratio outside [0.3, 4.0]\n\n### API Cost\n- Approximately 15$ for the entire extraction process, OpenRouter is used as the API gateway.\n\n## EvaCun(ORACC Akkadian Parallel Corpus) Preparation\n\n### Used Files\n- 45k transliteration-translation pairs from ranscription_train.txt, transcription_val.txt, english_train.txt, english_val.txt.\n\n### Preprocessing\n\nIn addition to the standard preprocessing used for the competition dataset, we applied an extra normalization step to the **transliteration side** of EvaCun: immediately repeated consecutive words were collapsed into a single token. This was motivated by the fact that the transliterations occasionally contain local repetition artifacts that are unlikely to be meaningful for translation and can introduce unnecessary noise during training.\n\nFor example, the transliteration\n\n`ša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te É-hi-il-la-na-te ma-a im-ma-te im-ma-te ú-šá-ra-qu`\n\nwas normalized to\n\n`ša LUGAL be-lí iš-pur-an-ni ma-a gul-la-a-te ša KÁ ša šap-la tim-me ša É-hi-il-la-na-te ma-a im-ma-te ú-šá-ra-qu`\n\n# Training Pipeline\n\nTraining on ByT5 variants (best submission used byt5-xl):\n\n**Stage 1: Build base models**\n\n1. Continued pre-training — T5 span corruption on transliterations from `published_texts.csv`.\n2. Fine-tuning on EvaCun. \n3. Fine-tuning on PDF data.\n\nA reverse model (English → Akkadian) is also trained with the same pipeline by swapping transliteration and translation.\n\n**Stage 2: Synthetic Data Generation**\n\nUsing the reverse model from Stage 1, we generate additional training pairs:\n\n- Pseudo-label: translate `published_texts.csv` transliterations to English with the forward model using beam search.\n- Back-translation: first generate 10,000 translation-like English sentences with Qwen3.5-27B, using translations from both `train.csv` and the PDF-extracted data as few-shot examples, and then back-translate them into Akkadian with the reverse model.\n\nThe generated dataset is [here](https://www.kaggle.com/datasets/yukiokumura1/deep-past-challenge-back-translated-synthetic-data)\n\nWe additionally tested scaling this back-translation data to 100,000 examples. Relative to training with 10,000 examples, this resulted in a -0.1 change on the public leaderboard but a +0.3 improvement on the private leaderboard.\n\n**Stage 3: Final training**\n\nStarting from the Stage 1 EvaCun checkpoint:\n\n1. Fine-tuning on Stage 2 data.\n2. Fine-tuning on PDF data with context — previous lines of the same tablet are prepended to the input(max 2 previous lines):\n\nprompt format:\n```\n[context]