![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2125251%2F12fe7dbc01db543f5892ac91f0e979ea%2Fensemble_diagram.png?generation=1774372330220882&alt=media\")
Thank you Kaggle for hosting this interesting competition. Special shout-out to @deeppast for active engagement throughout. It was great to see performance improving right up to the final hour! At a broader level, we think some of the techniques here would generalize to teaching an LLM a new skill (e.g. a new programming language) or encoding new knowledge (e.g. post cutoff recent events).
\nWe ensembled a ByT5-Large and a ByT5-XL model, using a fine-tuned Qwen3-8B pairwise Reward Model to select the better translation per sentence. Each model was trained in two stages:
CPT stage teaches Old Assyrian (OA) language fundamentals (Vocab, Grammar, Morphology etc.) using synthetic data, while having a light exposure to the actual translation task. FT stage was aimed at teaching the translation style of OA scholars using only high-quality scholar-translated OA (transliteration, translation) pairs.
\n
We used Claude Code extensively throughout this competition — for data extraction, quality auditing, training pipeline development, and deep research. A separate post on key learnings from using agentic tools (Claude Code skills, MCP integrations, parallel subagents) will follow.
\nThis competition was heavily data-driven. We spent the majority of our time preparing high-quality training data: extracting parallel pairs from publication PDFs, OCR texts, and generating synthetic drills to teach OA fundamentals.
\nWe used Claude 4.5 Sonnet to generate diverse synthetic tasks to teach OA fundamentals to ByT5 models.
We extracted ~1k OA lemmas (headword) from the CAD and eSAD dictionaries: Old Assyrian Grammars and other resources. Each lemma had one or more attested senses (glosses) with example (transliteration, translation) pairs. For each (lemma, gloss) pair (~3k), we asked an LLM to generate a list of 32 synthetic sentence pairs, resulting in ~90k examples.
\nFor polysemous lemmas, we sampled and concatenated up to 4 examples from different senses of the same lemma into a single training pair. This forces the model to learn sense disambiguation from context. This gave us ~60k additional training pairs by repurposing the earlier generated examples.
\nWe extracted ~500 grammar rules from the Kouwenberg 2017 GOA book. Rules had the following structure:
\n {\n \"title\": \"G-stem as passive of factitive D-stem (anti-factitive)\",\n \"rule\": \"In certain intransitive verbs with a frequently used factitive D-stem...\",\n \"transformation\": \"For seeds containing D-stem forms of abātum, kabātum, lamādum, nasāḫum, aqāpum, or nasārum: change D → G for passive, or G → D to add an agent. D-stem prt pattern: ú-PaRRiS (e.g., ú-bu-ú, ú-ka-bi-it, ú-na-si-iḫ). G-stem stative: PaRiS (e.g., a-bi, ka-bi-it, na-si-iḫ).\",\n \"examples\": [\n {\n \"transliteration\": \"ki-ma ú-bu-ú\",\n \"translation\": \"As he cleared (the debt).\",\n \"grammar\": \"D prt 3ms of abātum\"\n },\n {\n \"transliteration\": \"ḫu-bu-la-tum ib-bi-a\",\n \"translation\": \"The debts have been cleared.\",\n \"grammar\": \"G stv 3fp of abātum (anti-factitive)\"\n },\n...\n ]\n },\nWe then use LLM to apply the rules on seed sentences (sampled from the competition training tablets) to transform them by making minimal changes e.g. swap tense, person, mood, or clause structure. We generated ~50k examples following this approach.
\nExtracted 600+ templates with typed slots (PN, GN, amount, commodity, deadline, month, occupation etc). Example:
\n{\n \"transliteration\": \"{AMT_TL} KÙ.BABBAR ṣa-ru-pá-am i-ṣé-er {PN1_TL} {PN2_TL} i-šu\",\n \"translation\": \"{PN1_TR} owes {AMT_TR} of refined silver to {PN2_TR}.\"\n}\nand filled them programmatically from curated candidates for the typed slots (e.g. names sampled from host provided onomasticon.csv). We created ~150K pairs this way.
\nCreated a small seed dataset with around 60 transliteration forms which are ambiguous i.e. they can be a personal name or a common word depending on the context. This seed dataset included few shot pairs from real tablets showing the same form used as a NAME in one sentence and as a WORD in another. Then we used LLM to generate more such examples for the same forms.
\nThe competition's train.csv (1,561 tablets) and Sentences_Oare CSV were combined and deduplicated by oare_id to produce ~2,700 tablet-level (transliteration, translation) pairs. These are full-tablet translations, so we used Gemini 3 Pro with CoT to split each tablet into sentence-level pairs, aligning OA segments to their corresponding English translation fragments. A LLM judge then scored each pair on translation quality (high/medium/low) evaluating faithfulness to the OA. This produced ~17k sentence pairs with 85% high, 7% medium, 8% low quality distribution.
We independently extracted parallel pairs from the Kültepe Tablets PDFs using the multimodal capabilities of Gemini-3.0-Pro. PDF pages were rendered at 576 DPI, critical for correctly recognizing Akkadian diacritics (š, ṣ, ṭ, ḫ) and subscript digits (₄, ₆) and sent as images in 10-page overlapping chunks. A demo notebook shows the full pipeline. The prompt went through 24+ iterations. Key design choices:
We extracted parallel (transliteration, translation) pairs from the PDFs shared in Old Assyrian Kültepe Tablets in PDF. 3.3k tablets producing around 36k sentence pairs.
\nPublications not available as PDFs were extracted from the OCR data in the competition's publications.csv. It provided ~20k sentence pairs with scholar translation.
\nThe competition's published_texts.csv contains ~8k tablets with transliterations. We used Gemini to generate translations and split them into sentence-level pairs. The prompt inlined the host's dataset Instructions covering name normalization, Sumerogram conventions, gap marker parallelism (<gap> in transliteration → <gap> in translation at the same semantic positions), standard letter/debt/witness formulas, and unit handling.
We provided 80 dynamically sampled expert few-shot examples to anchor the translation style. In a second round, we replaced random sampling with embedding-based retrieval (NVIDIA NV-Embed) to find the 8 most semantically similar expert pairs per sentence. This improved self-reported high-quality rate from 61% to 82%. Produced ~35k sentence pairs across after deduplication against the Competition Data.
We used OA publication PDFs from Hecker's HPM catalog (9.9k transliteration-only segments), Michel legal case publications (HAL/CNRS open access), Dergipark Turkish academic journals (ArAn, Belleten, DTCF — Turkey's national open-access platform), and assorted open-access dissertations. These PDFs were extracted using the same Gemini multimodal recipe as the Kültepe tablets. All external sources are either open-access by institutional mandate (Dergipark/TÜBİTAK, HAL/CNRS), competition-provided, or public domain (CAD/Oriental Institute).
\nWe used our fine-tuned ByT5 models to generate translations for unlabelled transliterations in the published_texts.csv and publications.csv.
For CPT, we used all of the synthetic drill examples. In addition, we used the examples from the competition dataset, PDF extraction pipeline, extracted pairs from OCR texts and synthetic translations. We upsampled high quality subsets (e.g. competition Data, PDF Extraction data) by 2x.
\nFor FT, we used
\npublications.csv)Both transliteration and translation required deterministic normalization to match the test format. The normalization details can be found in our inference notebook.
\n| Config | \nWhat's Added | \nPublic | \nPrivate | \n
|---|---|---|---|
| Baseline | \nNo CPT, direct train on only scholar data | \n39.9 | \n40.3 | \n
| CPT only | \nCPT checkpoint, no FT stage | \n39.6 | \n40.4 | \n
| B0 | \nCPT -> FT on scholar data | \n40.6 | \n41.0 | \n
| B1 | \nB0 + carry over synthetic drills (5k sentences) + upsampling competition tablets | \n40.8 | \n41.4 | \n
| B2 | \nB1 + pseudo-labeled data (27.7K sentences) | \n41.2 | \n41.4 | \n
| B2_ns | \nB2 + name swap augmentation | \n41.3 | \n41.2 | \n
| M2 | \nB2_ns + More PL data + more pretrain drill carry | \n41.0 | \n41.3 | \n
Starting from the CPT checkpoint at step 14K, we fine-tuned on the high-quality FT datamix described above. For this stage, we used
\nWe used beam search with 8 beams. Our inference notebook is available here.
", "messages": [ { "id": "3428014", "postDate": "03/24/2026 19:35:32", "content": "Thank you Kaggle for hosting this interesting competition. Special shout-out to @deeppast for active engagement throughout. It was great to see performance improving right up to the final hour! At a broader level, we think some of the techniques here would generalize to teaching an LLM a new skill (e.g. a new programming language) or encoding new knowledge (e.g. post cutoff recent events).
\nWe ensembled a ByT5-Large and a ByT5-XL model, using a fine-tuned Qwen3-8B pairwise Reward Model to select the better translation per sentence. Each model was trained in two stages:
CPT stage teaches Old Assyrian (OA) language fundamentals (Vocab, Grammar, Morphology etc.) using synthetic data, while having a light exposure to the actual translation task. FT stage was aimed at teaching the translation style of OA scholars using only high-quality scholar-translated OA (transliteration, translation) pairs.
\n
We used Claude Code extensively throughout this competition — for data extraction, quality auditing, training pipeline development, and deep research. A separate post on key learnings from using agentic tools (Claude Code skills, MCP integrations, parallel subagents) will follow.
\nThis competition was heavily data-driven. We spent the majority of our time preparing high-quality training data: extracting parallel pairs from publication PDFs, OCR texts, and generating synthetic drills to teach OA fundamentals.
\nWe used Claude 4.5 Sonnet to generate diverse synthetic tasks to teach OA fundamentals to ByT5 models.
We extracted ~1k OA lemmas (headword) from the CAD and eSAD dictionaries: Old Assyrian Grammars and other resources. Each lemma had one or more attested senses (glosses) with example (transliteration, translation) pairs. For each (lemma, gloss) pair (~3k), we asked an LLM to generate a list of 32 synthetic sentence pairs, resulting in ~90k examples.
\nFor polysemous lemmas, we sampled and concatenated up to 4 examples from different senses of the same lemma into a single training pair. This forces the model to learn sense disambiguation from context. This gave us ~60k additional training pairs by repurposing the earlier generated examples.
\nWe extracted ~500 grammar rules from the Kouwenberg 2017 GOA book. Rules had the following structure:
\n {\n \"title\": \"G-stem as passive of factitive D-stem (anti-factitive)\",\n \"rule\": \"In certain intransitive verbs with a frequently used factitive D-stem...\",\n \"transformation\": \"For seeds containing D-stem forms of abātum, kabātum, lamādum, nasāḫum, aqāpum, or nasārum: change D → G for passive, or G → D to add an agent. D-stem prt pattern: ú-PaRRiS (e.g., ú-bu-ú, ú-ka-bi-it, ú-na-si-iḫ). G-stem stative: PaRiS (e.g., a-bi, ka-bi-it, na-si-iḫ).\",\n \"examples\": [\n {\n \"transliteration\": \"ki-ma ú-bu-ú\",\n \"translation\": \"As he cleared (the debt).\",\n \"grammar\": \"D prt 3ms of abātum\"\n },\n {\n \"transliteration\": \"ḫu-bu-la-tum ib-bi-a\",\n \"translation\": \"The debts have been cleared.\",\n \"grammar\": \"G stv 3fp of abātum (anti-factitive)\"\n },\n...\n ]\n },\nWe then use LLM to apply the rules on seed sentences (sampled from the competition training tablets) to transform them by making minimal changes e.g. swap tense, person, mood, or clause structure. We generated ~50k examples following this approach.
\nExtracted 600+ templates with typed slots (PN, GN, amount, commodity, deadline, month, occupation etc). Example:
\n{\n \"transliteration\": \"{AMT_TL} KÙ.BABBAR ṣa-ru-pá-am i-ṣé-er {PN1_TL} {PN2_TL} i-šu\",\n \"translation\": \"{PN1_TR} owes {AMT_TR} of refined silver to {PN2_TR}.\"\n}\nand filled them programmatically from curated candidates for the typed slots (e.g. names sampled from host provided onomasticon.csv). We created ~150K pairs this way.
\nCreated a small seed dataset with around 60 transliteration forms which are ambiguous i.e. they can be a personal name or a common word depending on the context. This seed dataset included few shot pairs from real tablets showing the same form used as a NAME in one sentence and as a WORD in another. Then we used LLM to generate more such examples for the same forms.
\nThe competition's train.csv (1,561 tablets) and Sentences_Oare CSV were combined and deduplicated by oare_id to produce ~2,700 tablet-level (transliteration, translation) pairs. These are full-tablet translations, so we used Gemini 3 Pro with CoT to split each tablet into sentence-level pairs, aligning OA segments to their corresponding English translation fragments. A LLM judge then scored each pair on translation quality (high/medium/low) evaluating faithfulness to the OA. This produced ~17k sentence pairs with 85% high, 7% medium, 8% low quality distribution.
We independently extracted parallel pairs from the Kültepe Tablets PDFs using the multimodal capabilities of Gemini-3.0-Pro. PDF pages were rendered at 576 DPI, critical for correctly recognizing Akkadian diacritics (š, ṣ, ṭ, ḫ) and subscript digits (₄, ₆) and sent as images in 10-page overlapping chunks. A demo notebook shows the full pipeline. The prompt went through 24+ iterations. Key design choices:
We extracted parallel (transliteration, translation) pairs from the PDFs shared in Old Assyrian Kültepe Tablets in PDF. 3.3k tablets producing around 36k sentence pairs.
\nPublications not available as PDFs were extracted from the OCR data in the competition's publications.csv. It provided ~20k sentence pairs with scholar translation.
\nThe competition's published_texts.csv contains ~8k tablets with transliterations. We used Gemini to generate translations and split them into sentence-level pairs. The prompt inlined the host's dataset Instructions covering name normalization, Sumerogram conventions, gap marker parallelism (<gap> in transliteration → <gap> in translation at the same semantic positions), standard letter/debt/witness formulas, and unit handling.
We provided 80 dynamically sampled expert few-shot examples to anchor the translation style. In a second round, we replaced random sampling with embedding-based retrieval (NVIDIA NV-Embed) to find the 8 most semantically similar expert pairs per sentence. This improved self-reported high-quality rate from 61% to 82%. Produced ~35k sentence pairs across after deduplication against the Competition Data.
We used OA publication PDFs from Hecker's HPM catalog (9.9k transliteration-only segments), Michel legal case publications (HAL/CNRS open access), Dergipark Turkish academic journals (ArAn, Belleten, DTCF — Turkey's national open-access platform), and assorted open-access dissertations. These PDFs were extracted using the same Gemini multimodal recipe as the Kültepe tablets. All external sources are either open-access by institutional mandate (Dergipark/TÜBİTAK, HAL/CNRS), competition-provided, or public domain (CAD/Oriental Institute).
\nWe used our fine-tuned ByT5 models to generate translations for unlabelled transliterations in the published_texts.csv and publications.csv.
For CPT, we used all of the synthetic drill examples. In addition, we used the examples from the competition dataset, PDF extraction pipeline, extracted pairs from OCR texts and synthetic translations. We upsampled high quality subsets (e.g. competition Data, PDF Extraction data) by 2x.
\nFor FT, we used
\npublications.csv)Both transliteration and translation required deterministic normalization to match the test format. The normalization details can be found in our inference notebook.
\n| Config | \nWhat's Added | \nPublic | \nPrivate | \n
|---|---|---|---|
| Baseline | \nNo CPT, direct train on only scholar data | \n39.9 | \n40.3 | \n
| CPT only | \nCPT checkpoint, no FT stage | \n39.6 | \n40.4 | \n
| B0 | \nCPT -> FT on scholar data | \n40.6 | \n41.0 | \n
| B1 | \nB0 + carry over synthetic drills (5k sentences) + upsampling competition tablets | \n40.8 | \n41.4 | \n
| B2 | \nB1 + pseudo-labeled data (27.7K sentences) | \n41.2 | \n41.4 | \n
| B2_ns | \nB2 + name swap augmentation | \n41.3 | \n41.2 | \n
| M2 | \nB2_ns + More PL data + more pretrain drill carry | \n41.0 | \n41.3 | \n
Starting from the CPT checkpoint at step 14K, we fine-tuned on the high-quality FT datamix described above. For this stage, we used
\nWe used beam search with 8 beams. Our inference notebook is available here.
", "rawMarkdown": "Thank you Kaggle for hosting this interesting competition. Special shout-out to @deeppast for active engagement throughout. It was great to see performance improving right up to the final hour! At a broader level, we think some of the techniques here would generalize to teaching an LLM a new skill (e.g. a new programming language) or encoding new knowledge (e.g. post cutoff recent events).\n\n# TL;DR\n\nWe ensembled a `ByT5-Large` and a `ByT5-XL` model, using a fine-tuned `Qwen3-8B` pairwise Reward Model to select the better translation per sentence. Each model was trained in two stages: \n- Continued seq2seq pre-training (CPT)\n- Fine-tuning (FT)\n\nCPT stage teaches Old Assyrian (OA) language fundamentals (Vocab, Grammar, Morphology etc.) using synthetic data, while having a light exposure to the actual translation task. FT stage was aimed at teaching the translation style of OA scholars using only high-quality scholar-translated OA (transliteration, translation) pairs.\n\n\n\n## Agentic Tooling\n\nWe used Claude Code extensively throughout this competition — for data extraction, quality auditing, training pipeline development, and deep research. A separate post on key learnings from using agentic tools (Claude Code skills, MCP integrations, parallel subagents) will follow.\n\n# Data Sources\n\nThis competition was heavily data-driven. We spent the majority of our time preparing high-quality training data: extracting parallel pairs from publication PDFs, OCR texts, and generating synthetic drills to teach OA fundamentals.\n\n## Synthetic Data\nWe used `Claude 4.5 Sonnet` to generate diverse synthetic tasks to teach OA fundamentals to ByT5 models.\n\n### Vocab Drills\n\nWe extracted ~1k OA lemmas (headword) from the CAD and eSAD dictionaries: [Old Assyrian Grammars and other resources](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources). Each lemma had one or more attested senses (glosses) with example (transliteration, translation) pairs. For each (lemma, gloss) pair (~3k), we asked an LLM to generate a list of 32 synthetic sentence pairs, resulting in ~90k examples.\n\nFor polysemous lemmas, we sampled and concatenated up to 4 examples from different senses of the same lemma into a single training pair. This forces the model to learn sense disambiguation from context. This gave us ~60k additional training pairs by repurposing the earlier generated examples.\n\n### Grammar Drills\n\nWe extracted ~500 grammar rules from the [Kouwenberg 2017 GOA book](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources?select=Kouwenberg_2017_GOA_HdO118.pdf). Rules had the following structure:\n\n```\n {\n \"title\": \"G-stem as passive of factitive D-stem (anti-factitive)\",\n \"rule\": \"In certain intransitive verbs with a frequently used factitive D-stem...\",\n \"transformation\": \"For seeds containing D-stem forms of abātum, kabātum, lamādum, nasāḫum, aqāpum, or nasārum: change D → G for passive, or G → D to add an agent. D-stem prt pattern: ú-PaRRiS (e.g., ú-bu-ú, ú-ka-bi-it, ú-na-si-iḫ). G-stem stative: PaRiS (e.g., a-bi, ka-bi-it, na-si-iḫ).\",\n \"examples\": [\n {\n \"transliteration\": \"ki-ma ú-bu-ú\",\n \"translation\": \"As he cleared (the debt).\",\n \"grammar\": \"D prt 3ms of abātum\"\n },\n {\n \"transliteration\": \"ḫu-bu-la-tum ib-bi-a\",\n \"translation\": \"The debts have been cleared.\",\n \"grammar\": \"G stv 3fp of abātum (anti-factitive)\"\n },\n...\n ]\n },\n```\nWe then use LLM to apply the rules on seed sentences (sampled from the competition training tablets) to transform them by making minimal changes e.g. swap tense, person, mood, or clause structure. We generated ~50k examples following this approach.\n\n### Slot-Fill Templates \n\nExtracted 600+ templates with typed slots (PN, GN, amount, commodity, deadline, month, occupation etc). Example:\n```\n{\n \"transliteration\": \"{AMT_TL} KÙ.BABBAR ṣa-ru-pá-am i-ṣé-er {PN1_TL} {PN2_TL} i-šu\",\n \"translation\": \"{PN1_TR} owes {AMT_TR} of refined silver to {PN2_TR}.\"\n}\n```\nand filled them programmatically from curated candidates for the typed slots (e.g. names sampled from host provided [onomasticon.csv](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources?select=onomasticon.csv)). We created ~150K pairs this way.\n\n### PN vs Word Contrastive\n\nCreated a small seed dataset with around 60 transliteration forms which are ambiguous i.e. they can be a personal name or a common word depending on the context. This seed dataset included few shot pairs from real tablets showing the same form used as a NAME in one sentence and as a WORD in another. Then we used LLM to generate more such examples for the same forms.\n\n### Other Synthetic data\n- Weight Conversion Drills\n- Synthetic Tablets: coherent multi-sentence tablets generated via augmenting real tables e.g via clause extension, name substitution, interpolation across real tablets.\n\n## Real Tablets\n\n### Competition Data\nThe competition's `train.csv` (1,561 tablets) and `Sentences_Oare` CSV were combined and deduplicated by `oare_id` to produce ~2,700 tablet-level `(transliteration, translation)` pairs. These are full-tablet translations, so we used Gemini 3 Pro with CoT to split each tablet into sentence-level pairs, aligning OA segments to their corresponding English translation fragments. A LLM judge then scored each pair on translation quality (high/medium/low) evaluating faithfulness to the OA. This produced ~17k sentence pairs with 85% high, 7% medium, 8% low quality distribution.\n\n### PDF Extraction Pipeline\nWe independently extracted parallel pairs from the [Kültepe Tablets PDFs](https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf) using the multimodal capabilities of `Gemini-3.0-Pro`. PDF pages were rendered at 576 DPI, critical for correctly recognizing Akkadian diacritics (`š, ṣ, ṭ, ḫ`) and subscript digits (`₄, ₆`) and sent as images in 10-page overlapping chunks. A [demo notebook](https://www.kaggle.com/code/darraghdog/k-ltepe-tablets-with-gemini-flash) shows the full pipeline. The prompt went through 24+ iterations. Key design choices:\n - Visual layout: Identifies transliteration blocks by visual structure (margin line counts, Vs./Rs. markers, indentation) rather than text parsing.\n - Marker alignment: Uses superscript line ranges in the translation (e.g., 1-2), 3-6)) as anchors for semantic alignment.\n - Trust the scholar: Not to alter the published translation, entities (names, commodities). This prevents hallucinating names not in the cuneiform.\n - Complete extraction: Explicit \"include ALL lines\" instruction. Without it, the gemini silently skipped damaged tablets, losing ~30% of pairs.\n - In-line translation: For German/Turkish sources, a domain vocabulary supplement enables English translation in the same API call.\n\nWe extracted parallel (transliteration, translation) pairs from the PDFs shared in [Old Assyrian Kültepe Tablets in PDF](https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf). 3.3k tablets producing around 36k sentence pairs.\n\n### OCR Texts\nPublications not available as PDFs were extracted from the OCR data in the competition's [publications.csv](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/data?select=publications.csv). It provided ~20k sentence pairs with scholar translation.\n\n### Synthetic Translations\nThe competition's `published_texts.csv` contains ~8k tablets with transliterations. We used Gemini to generate translations and split them into sentence-level pairs. The prompt inlined the host's dataset Instructions covering name normalization, Sumerogram conventions, gap marker parallelism (`Congrats and thank you for the detailed write up. It's a lot of interesting information to digest.
", "rawMarkdown": "Congrats and thank you for the detailed write up. It's a lot of interesting information to digest.", "votes": null }, { "id": "3428412", "postDate": "03/25/2026 13:21:48", "content": "Congrats on your 3d place! Could you explain the logic behind some parameter decisions of the CPT training? For example, 3 epochs, long warmup, max norm? Or were they determined empirically through different experiments?
", "rawMarkdown": "Congrats on your 3d place! Could you explain the logic behind some parameter decisions of the CPT training? For example, 3 epochs, long warmup, max norm? Or were they determined empirically through different experiments?", "votes": null }, { "id": "3428716", "postDate": "03/25/2026 20:58:17", "content": "Thank you for the detailed write up
", "rawMarkdown": "Thank you for the detailed write up", "votes": null }, { "id": "3428928", "postDate": "03/26/2026 04:55:37", "content": "Hi @alejopaullier Thanks for the questions! On training param selection for CPT, main focus was on training stability and qualitative assessment of OOF translations (by Claude Code).
\nHere is the head to head comparison (first 5k steps) of two runs with same dataset
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F3d2dea48bd6325c7018f3302cac6d197%2Fh2h-monitor.png?generation=1774499958003834&alt=media\")
Both runs had high gradient norm at the start when byt5 encoder was first exposed to the OA transliterations. But, I think run2 with aggressive gradient clip + long warmup absorbed the shock better (e.g. looking at the activation norm evolution at layer 11 and 23).
\nAs for 3 epochs, idea was to use warmup + linear LR schedule to generate many checkpoints at regular intervals. Then pick one based on the qualitative analysis done by Claude Code:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2Fa700e7e72970af03c47e54dc2afc64be%2Fquality-compare.png?generation=1774500627341562&alt=media\")
We wanted to select a slightly \"under-trained\" checkpoint where the model was still learning and didn't show much regression on any quality dimensions. CC analysis found these learning stages:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F1a86448888cbb5d4066c2ae0bc17de6e%2Fs1.png?generation=1774500811122898&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F1f01a922895f69c96fee9aded192bd33%2Fs2.png?generation=1774500821639872&alt=media\")
The idea of using warmup + constant LR schedule was inspired by this paper: Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
\n\n", "rawMarkdown": "Hi @alejopaullier Thanks for the questions! On training param selection for CPT, main focus was on training stability and qualitative assessment of OOF translations (by Claude Code).\n\nHere is the head to head comparison (first 5k steps) of two runs with same dataset\n- Run 1: short warm up, max_gradient_norm = 1000, no AWP\n- Run 2: long warm up, max_gradient_norm = 0.3, AWP\n\n\n\nBoth runs had high gradient norm at the start when byt5 encoder was first exposed to the OA transliterations. But, I think run2 with aggressive gradient clip + long warmup absorbed the shock better (e.g. looking at the activation norm evolution at layer 11 and 23).\n\nAs for 3 epochs, idea was to use warmup + linear LR schedule to generate many checkpoints at regular intervals. Then pick one based on the qualitative analysis done by Claude Code:\n\n\n\nWe wanted to select a slightly \"under-trained\" checkpoint where the model was still learning and didn't show much regression on any quality dimensions. CC analysis found these learning stages:\n\n\n\n\nThe idea of using warmup + constant LR schedule was inspired by this paper: [Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective](https://arxiv.org/abs/2410.05192)\n\n>Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can in principle continue indefinitely without a pre-specified compute budget. Then, given any compute budget, one can branch out from the main branch at a proper time with a rapidly decaying learning rate to produce a strong model.", "votes": null }, { "id": "3429212", "postDate": "03/26/2026 13:36:32", "content": "Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can in principle continue indefinitely without a pre-specified compute budget. Then, given any compute budget, one can branch out from the main branch at a proper time with a rapidly decaying learning rate to produce a strong model.
\n
@conjuring92 Thanks for the detailed answer! One last short question, for the qualitative analysis by Claude Code did you pass all the OOF translations as context? I ask this because I imagine these are quite a few thousand and I wondered if Claude Code can still perform a reliable analysis given such a large context (LLMs sometimes degrade as context size grows). I personally would have been a bit skeptical about Claude's analysis 😅
", "rawMarkdown": "conjuring92 Thanks for the detailed answer! One last short question, for the qualitative analysis by Claude Code did you pass **all** the OOF translations as context? I ask this because I imagine these are quite a few thousand and I wondered if Claude Code can still perform a reliable analysis given such a large context (LLMs sometimes degrade as context size grows). I personally would have been a bit skeptical about Claude's analysis 😅", "votes": null }, { "id": "3429511", "postDate": "03/26/2026 20:27:23", "content": "Can you explain this post https://x.com/gonedarragh/status/2036390508885316060 which states, \n\"Deep Past Challenge was my first kaggle competition without coding, and with @raja_biswas\n we came 3rd 🥉 \nIt was fun to use agentic tools to learn about life and settling disputes 4000 years ago. Thanks @kaggle\n for hosting\"\nDoes this mean there was literally no human coding in this entire project and everything was done via natural language (English) against various LLMs (Claude, Gemini, Codex, etc.)? Or is this a hyperbole? I ask because there's a huge difference between purely natural language based interaction and that with scripting (Python, Bash, etc) or inputting prior code (possibly human generated) into the LLM.
", "rawMarkdown": "Can you explain this post https://x.com/gonedarragh/status/2036390508885316060 which states, \n\"Deep Past Challenge was my first kaggle competition without coding, and with @raja_biswas\n we came 3rd 🥉 \nIt was fun to use agentic tools to learn about life and settling disputes 4000 years ago. Thanks @kaggle\n for hosting\"\nDoes this mean there was literally no human coding in this entire project and everything was done via natural language (English) against various LLMs (Claude, Gemini, Codex, etc.)? Or is this a hyperbole? I ask because there's a huge difference between purely natural language based interaction and that with scripting (Python, Bash, etc) or inputting prior code (possibly human generated) into the LLM.", "votes": null }, { "id": "3429597", "postDate": "03/27/2026 00:30:57", "content": "Thanks for the question Mike, and I hope I get your question correctly, but yes - I wrote no code this entire project, only directed Claude Code what was needed, and sometimes what to base the code on - for example, use Use Nemo/RL to train a Qwen3-4B based on this data..., when I moved to byte5, I downloaded a kaggle notebook with a training pipeline and asked claude to reproduce the numbers, using my data on the connected server.
\nSo re bash, cat my_new_file.jsonl | tail | jq was done by hand plenty of times on command line, but no writing bash files.
\nRaja wrote very little. As this gained some interest I'm publishing a blog with some numbers and the learnings.
But the Kaggle notebook is populated with Python code (perhaps not written by you) not written by your Claude?
", "rawMarkdown": "But the Kaggle notebook is populated with Python code (perhaps not written by you) not written by your Claude?", "votes": null }, { "id": "3429612", "postDate": "03/27/2026 01:19:24", "content": "cool Datamix @darraghdog & @conjuring92
", "rawMarkdown": "cool Datamix @darraghdog & @conjuring92", "votes": null }, { "id": "3429632", "postDate": "03/27/2026 02:25:51", "content": "I asked CC to use 8 sub agents to manage the context better. Yes, capabilities degrade after certain % of the max context window is filled. With CC, I have seen no degradation up to at least 500k (opus 1m model) tokens, in fact it gets better as it ingests more project specific contexts.
", "rawMarkdown": "I asked CC to use 8 sub agents to manage the context better. Yes, capabilities degrade after certain % of the max context window is filled. With CC, I have seen no degradation up to at least 500k (opus 1m model) tokens, in fact it gets better as it ingests more project specific contexts.", "votes": null }, { "id": "3429727", "postDate": "03/27/2026 06:55:36", "content": "Exactly—there was human code used as a template, but I meant it was my first project without me manually writing the code. Code was written.
", "rawMarkdown": "Exactly—there was human code used as a template, but I meant it was my first project without me manually writing the code. Code was written.", "votes": null } ], "comments": [ { "id": 3428061, "author_name": "tennogh", "author_url": "", "post_date": "03/24/2026 22:04:30", "content": "Congrats and thank you for the detailed write up. It's a lot of interesting information to digest.
", "votes": null, "replies": [] }, { "id": 3428412, "author_name": "alejopaullier", "author_url": "", "post_date": "03/25/2026 13:21:48", "content": "Congrats on your 3d place! Could you explain the logic behind some parameter decisions of the CPT training? For example, 3 epochs, long warmup, max norm? Or were they determined empirically through different experiments?
", "votes": null, "replies": [ { "id": 3428928, "author_name": "conjuring92", "author_url": "", "post_date": "03/26/2026 04:55:37", "content": "Hi @alejopaullier Thanks for the questions! On training param selection for CPT, main focus was on training stability and qualitative assessment of OOF translations (by Claude Code).
\nHere is the head to head comparison (first 5k steps) of two runs with same dataset
\n
Both runs had high gradient norm at the start when byt5 encoder was first exposed to the OA transliterations. But, I think run2 with aggressive gradient clip + long warmup absorbed the shock better (e.g. looking at the activation norm evolution at layer 11 and 23).
\nAs for 3 epochs, idea was to use warmup + linear LR schedule to generate many checkpoints at regular intervals. Then pick one based on the qualitative analysis done by Claude Code:
\n
We wanted to select a slightly \"under-trained\" checkpoint where the model was still learning and didn't show much regression on any quality dimensions. CC analysis found these learning stages:\n
The idea of using warmup + constant LR schedule was inspired by this paper: Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
\n\n", "votes": null, "replies": [ { "id": 3429212, "author_name": "alejopaullier", "author_url": "", "post_date": "03/26/2026 13:36:32", "content": "Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can in principle continue indefinitely without a pre-specified compute budget. Then, given any compute budget, one can branch out from the main branch at a proper time with a rapidly decaying learning rate to produce a strong model.
\n
@conjuring92 Thanks for the detailed answer! One last short question, for the qualitative analysis by Claude Code did you pass all the OOF translations as context? I ask this because I imagine these are quite a few thousand and I wondered if Claude Code can still perform a reliable analysis given such a large context (LLMs sometimes degrade as context size grows). I personally would have been a bit skeptical about Claude's analysis 😅
", "votes": null, "replies": [ { "id": 3429632, "author_name": "conjuring92", "author_url": "", "post_date": "03/27/2026 02:25:51", "content": "I asked CC to use 8 sub agents to manage the context better. Yes, capabilities degrade after certain % of the max context window is filled. With CC, I have seen no degradation up to at least 500k (opus 1m model) tokens, in fact it gets better as it ingests more project specific contexts.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3428716, "author_name": "durgakumari12", "author_url": "", "post_date": "03/25/2026 20:58:17", "content": "Thank you for the detailed write up
", "votes": null, "replies": [] }, { "id": 3429511, "author_name": "mikeskim", "author_url": "", "post_date": "03/26/2026 20:27:23", "content": "Can you explain this post https://x.com/gonedarragh/status/2036390508885316060 which states, \n\"Deep Past Challenge was my first kaggle competition without coding, and with @raja_biswas\n we came 3rd 🥉 \nIt was fun to use agentic tools to learn about life and settling disputes 4000 years ago. Thanks @kaggle\n for hosting\"\nDoes this mean there was literally no human coding in this entire project and everything was done via natural language (English) against various LLMs (Claude, Gemini, Codex, etc.)? Or is this a hyperbole? I ask because there's a huge difference between purely natural language based interaction and that with scripting (Python, Bash, etc) or inputting prior code (possibly human generated) into the LLM.
", "votes": null, "replies": [ { "id": 3429597, "author_name": "darraghdog", "author_url": "", "post_date": "03/27/2026 00:30:57", "content": "Thanks for the question Mike, and I hope I get your question correctly, but yes - I wrote no code this entire project, only directed Claude Code what was needed, and sometimes what to base the code on - for example, use Use Nemo/RL to train a Qwen3-4B based on this data..., when I moved to byte5, I downloaded a kaggle notebook with a training pipeline and asked claude to reproduce the numbers, using my data on the connected server.
\nSo re bash, cat my_new_file.jsonl | tail | jq was done by hand plenty of times on command line, but no writing bash files.
\nRaja wrote very little. As this gained some interest I'm publishing a blog with some numbers and the learnings.
But the Kaggle notebook is populated with Python code (perhaps not written by you) not written by your Claude?
", "votes": null, "replies": [ { "id": 3429727, "author_name": "darraghdog", "author_url": "", "post_date": "03/27/2026 06:55:36", "content": "Exactly—there was human code used as a template, but I meant it was my first project without me manually writing the code. Code was written.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3429612, "author_name": "navneetbende", "author_url": "", "post_date": "03/27/2026 01:19:24", "content": "cool Datamix @darraghdog & @conjuring92
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3428014": "Thank you Kaggle for hosting this interesting competition. Special shout-out to @deeppast for active engagement throughout. It was great to see performance improving right up to the final hour! At a broader level, we think some of the techniques here would generalize to teaching an LLM a new skill (e.g. a new programming language) or encoding new knowledge (e.g. post cutoff recent events).\n\n# TL;DR\n\nWe ensembled a `ByT5-Large` and a `ByT5-XL` model, using a fine-tuned `Qwen3-8B` pairwise Reward Model to select the better translation per sentence. Each model was trained in two stages: \n- Continued seq2seq pre-training (CPT)\n- Fine-tuning (FT)\n\nCPT stage teaches Old Assyrian (OA) language fundamentals (Vocab, Grammar, Morphology etc.) using synthetic data, while having a light exposure to the actual translation task. FT stage was aimed at teaching the translation style of OA scholars using only high-quality scholar-translated OA (transliteration, translation) pairs.\n\n\n\n## Agentic Tooling\n\nWe used Claude Code extensively throughout this competition — for data extraction, quality auditing, training pipeline development, and deep research. A separate post on key learnings from using agentic tools (Claude Code skills, MCP integrations, parallel subagents) will follow.\n\n# Data Sources\n\nThis competition was heavily data-driven. We spent the majority of our time preparing high-quality training data: extracting parallel pairs from publication PDFs, OCR texts, and generating synthetic drills to teach OA fundamentals.\n\n## Synthetic Data\nWe used `Claude 4.5 Sonnet` to generate diverse synthetic tasks to teach OA fundamentals to ByT5 models.\n\n### Vocab Drills\n\nWe extracted ~1k OA lemmas (headword) from the CAD and eSAD dictionaries: [Old Assyrian Grammars and other resources](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources). Each lemma had one or more attested senses (glosses) with example (transliteration, translation) pairs. For each (lemma, gloss) pair (~3k), we asked an LLM to generate a list of 32 synthetic sentence pairs, resulting in ~90k examples.\n\nFor polysemous lemmas, we sampled and concatenated up to 4 examples from different senses of the same lemma into a single training pair. This forces the model to learn sense disambiguation from context. This gave us ~60k additional training pairs by repurposing the earlier generated examples.\n\n### Grammar Drills\n\nWe extracted ~500 grammar rules from the [Kouwenberg 2017 GOA book](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources?select=Kouwenberg_2017_GOA_HdO118.pdf). Rules had the following structure:\n\n```\n {\n \"title\": \"G-stem as passive of factitive D-stem (anti-factitive)\",\n \"rule\": \"In certain intransitive verbs with a frequently used factitive D-stem...\",\n \"transformation\": \"For seeds containing D-stem forms of abātum, kabātum, lamādum, nasāḫum, aqāpum, or nasārum: change D → G for passive, or G → D to add an agent. D-stem prt pattern: ú-PaRRiS (e.g., ú-bu-ú, ú-ka-bi-it, ú-na-si-iḫ). G-stem stative: PaRiS (e.g., a-bi, ka-bi-it, na-si-iḫ).\",\n \"examples\": [\n {\n \"transliteration\": \"ki-ma ú-bu-ú\",\n \"translation\": \"As he cleared (the debt).\",\n \"grammar\": \"D prt 3ms of abātum\"\n },\n {\n \"transliteration\": \"ḫu-bu-la-tum ib-bi-a\",\n \"translation\": \"The debts have been cleared.\",\n \"grammar\": \"G stv 3fp of abātum (anti-factitive)\"\n },\n...\n ]\n },\n```\nWe then use LLM to apply the rules on seed sentences (sampled from the competition training tablets) to transform them by making minimal changes e.g. swap tense, person, mood, or clause structure. We generated ~50k examples following this approach.\n\n### Slot-Fill Templates \n\nExtracted 600+ templates with typed slots (PN, GN, amount, commodity, deadline, month, occupation etc). Example:\n```\n{\n \"transliteration\": \"{AMT_TL} KÙ.BABBAR ṣa-ru-pá-am i-ṣé-er {PN1_TL} {PN2_TL} i-šu\",\n \"translation\": \"{PN1_TR} owes {AMT_TR} of refined silver to {PN2_TR}.\"\n}\n```\nand filled them programmatically from curated candidates for the typed slots (e.g. names sampled from host provided [onomasticon.csv](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources?select=onomasticon.csv)). We created ~150K pairs this way.\n\n### PN vs Word Contrastive\n\nCreated a small seed dataset with around 60 transliteration forms which are ambiguous i.e. they can be a personal name or a common word depending on the context. This seed dataset included few shot pairs from real tablets showing the same form used as a NAME in one sentence and as a WORD in another. Then we used LLM to generate more such examples for the same forms.\n\n### Other Synthetic data\n- Weight Conversion Drills\n- Synthetic Tablets: coherent multi-sentence tablets generated via augmenting real tables e.g via clause extension, name substitution, interpolation across real tablets.\n\n## Real Tablets\n\n### Competition Data\nThe competition's `train.csv` (1,561 tablets) and `Sentences_Oare` CSV were combined and deduplicated by `oare_id` to produce ~2,700 tablet-level `(transliteration, translation)` pairs. These are full-tablet translations, so we used Gemini 3 Pro with CoT to split each tablet into sentence-level pairs, aligning OA segments to their corresponding English translation fragments. A LLM judge then scored each pair on translation quality (high/medium/low) evaluating faithfulness to the OA. This produced ~17k sentence pairs with 85% high, 7% medium, 8% low quality distribution.\n\n### PDF Extraction Pipeline\nWe independently extracted parallel pairs from the [Kültepe Tablets PDFs](https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf) using the multimodal capabilities of `Gemini-3.0-Pro`. PDF pages were rendered at 576 DPI, critical for correctly recognizing Akkadian diacritics (`š, ṣ, ṭ, ḫ`) and subscript digits (`₄, ₆`) and sent as images in 10-page overlapping chunks. A [demo notebook](https://www.kaggle.com/code/darraghdog/k-ltepe-tablets-with-gemini-flash) shows the full pipeline. The prompt went through 24+ iterations. Key design choices:\n - Visual layout: Identifies transliteration blocks by visual structure (margin line counts, Vs./Rs. markers, indentation) rather than text parsing.\n - Marker alignment: Uses superscript line ranges in the translation (e.g., 1-2), 3-6)) as anchors for semantic alignment.\n - Trust the scholar: Not to alter the published translation, entities (names, commodities). This prevents hallucinating names not in the cuneiform.\n - Complete extraction: Explicit \"include ALL lines\" instruction. Without it, the gemini silently skipped damaged tablets, losing ~30% of pairs.\n - In-line translation: For German/Turkish sources, a domain vocabulary supplement enables English translation in the same API call.\n\nWe extracted parallel (transliteration, translation) pairs from the PDFs shared in [Old Assyrian Kültepe Tablets in PDF](https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf). 3.3k tablets producing around 36k sentence pairs.\n\n### OCR Texts\nPublications not available as PDFs were extracted from the OCR data in the competition's [publications.csv](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/data?select=publications.csv). It provided ~20k sentence pairs with scholar translation.\n\n### Synthetic Translations\nThe competition's `published_texts.csv` contains ~8k tablets with transliterations. We used Gemini to generate translations and split them into sentence-level pairs. The prompt inlined the host's dataset Instructions covering name normalization, Sumerogram conventions, gap marker parallelism (`