![\"histogram_of_languages\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2590026%2F4aff36dccd07fa82ba719f2d8b7752d6%2Fhist_of_languages.png?generation=1774382397183303&alt=media\")
First of all, I'd like to thank the Deep Past Challenge team at Kaggle for hosting this fascinating competition. I also found many publicly available notebooks and discussions invaluable throughout the process, and I came away having learned a great deal.
\npublications.csv to help the model understand Akkadian vocabularies and grammartrain.csv to align it with the competition format and domain knowledgeI found that page_text in publications.csv contains many translations of Akkadian transliterations into multiple languages, including English, German, and French. I used an LLM to extract these pairs from the raw text, converting the unstructured PDF-extracted content into a clean, structured JSON format. I allowed the LLM to output null for cases where no corresponding translation exists, which I believe improved the overall quality of the dataset.
### Output Format (Strict JSON)\nReturn ONLY a JSON array of objects with the following structure. Do not include markdown headers or extra text.\n{\n \"items\": [\n {\n \"akkadian_transliteration\": \"string\",\n \"translation\": \"string or null\",\n \"language\": \"string or null\",\n }\n ]\n}\nThe extracted data covers many languages (with some potential hallucinations from the LLM). The histogram below shows the language distribution in my dataset. I decided to use only English, French, German, and Turkish for pre-training, as these had sufficient data volume.
\n
Training uses Seq2SeqTrainer and takes approximately 20 hours on a single A100 GPU in Colab.\nI preprocessed the data following the dataset instructions and valuable discussions.
Hyperparameters:
\n| Parameter | \nValue | \n
|---|---|
| epochs | \n3 | \n
| batch size | \n8 | \n
| learning rate | \n1e-4 | \n
| max length | \n512 | \n
| label smoothing factor | \n0.1 | \n
| EMA decay | \n0.999 | \n
The training data contains translations aligned at the document level, whereas the test data is aligned at the sentence level. To bridge this gap, I used an LLM to segment the Akkadian transliterations and their English translations into sentence-by-sentence pairs.
\nSince I was unsure about the typical length of test examples, I analyzed test.csv and found that transliteration lengths range from 129 to 267 characters. Based on this, I applied a length constraint of 100–400 characters per segment. The key instructions I used are as follows:
Maintain Sequence: Do NOT change the original order of sentences in either language.\nLength Constraints: Each Akkadian segment must be between 100 and 400 characters long.\n - If a segment is **too short** (< 100 chars): merge it with the adjacent clause.\n - If a segment is **too long** (> 400 chars): split it at a natural boundary such as a new clause, conjunction (ù, -ma), or topic shift.\nI also noticed mismatching translation pairs in train.csv. The histogram below shows the distribution of translation-to-transliteration length ratios.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2590026%2Ff5b62424f92cb3c44fbc372dafcf0ba9%2Fhist_of_ratio.png?generation=1774382431554487&alt=media\")
I replaced outliers with LLM-generated translations, using 0.66 and 2.0 as the lower and upper thresholds. I provided few-shot examples to guide the correction:
# Task\nImprove and correct the following English translation of an Akkadian text, using the original translation as a reference base.\n\n...\n\n# Examples\nAkkadian: 0.3333 ma-na 2.5 GÍN KÙ.BABBAR 20 NINDA i-ṣé-er tù-wa-ra-a-aḫ-šu a-lá-ḫu-um i-šu\nOriginal: Ali-ahum has 0.3333 mina 2.5 shekels of silver and 20 loaves of bread from Tuwar-ahšu.\nCorrected: Tuwar-ahšu owes 0.3333 mina 2.5 shekels of silver (and) 20 loaves of bread to Ali-ahum.\n\n...\n\n# Input\nAkkadian: {ak_input}\nOriginal: {original_translation}\nCorrected:\nI also generated translations for transliterations in publication_texts.csv using a similar prompt for use in pre-training.
Finally, I fine-tuned the pre-trained ByT5-large on the refined train.csv using the same training code as pre-training, with epochs=2 and lr=1e-5.
I ensembled three ByT5-large models fine-tuned on different pre-training datasets. To evaluate the impact of MBR decoding, I ran experiments on the 100 hardest validation samples (selected by val score) to keep the process manageable. I compared four candidate generation methods: normal beam search, group beam search, nucleus sampling, and epsilon sampling. The best result came from using normal beam search for candidate generation and group beam search for the candidate pool. I used the competition metric directly as the utility function for MBR decoding. Applying a soft margin penalty based on the translation/transliteration ratio, along with PN/GN lemmatization, provided a small additional boost.
\ndef calculate_soft_margin_penalty(ratio, low=0.9, high=2.0, sigma=0.2):\n distance = max(0, low - ratio, ratio - high)\n return np.exp(-(distance**2) / (2 * (sigma**2)))\nThe val scores were not well-correlated with the public leaderboard scores, which was puzzling. I believe the main cause is that my validation set was constructed by segmenting train.csv with an LLM and replacing mismatched pairs with LLM-generated translations. This made the val set share the same distribution and style as the training data, inflating the val scores and causing them to diverge from the actual test set distribution. That said, I'm relieved the final submission turned out reasonably well.
D1: published_texts.csv D2: publications.csv D3: train.csv
| pre-training dataset | \npre-training steps | \nval (w/o ft) | \nval | \nval hard | \npublic | \nprivate | \n
|---|---|---|---|---|---|---|
D1: 37910 D2: 79806 | \n42000 | \n36.2 | \n41.7 | \n23.8 | \n39.0 | \n38.9 | \n
D1: 37910 D2: 126439 D3: 3987 | \n38000 | \n45.8 | \n47.9 | \n29.6 | \n38.0 | \n38.3 | \n
D1: 37910 D2: 18834 | \n30000 | \n45.5 | \n46.6 | \n28.6 | \n38.2 | \n37.8 | \n
| (final submission) | \n- | \n- | \n- | \n30.14 | \n39.4 | \n39.8 | \n
OA_Lexicon_eBL.csveBL_Dictionary.csvFirst of all, I'd like to thank the Deep Past Challenge team at Kaggle for hosting this fascinating competition. I also found many publicly available notebooks and discussions invaluable throughout the process, and I came away having learned a great deal.
\npublications.csv to help the model understand Akkadian vocabularies and grammartrain.csv to align it with the competition format and domain knowledgeI found that page_text in publications.csv contains many translations of Akkadian transliterations into multiple languages, including English, German, and French. I used an LLM to extract these pairs from the raw text, converting the unstructured PDF-extracted content into a clean, structured JSON format. I allowed the LLM to output null for cases where no corresponding translation exists, which I believe improved the overall quality of the dataset.
### Output Format (Strict JSON)\nReturn ONLY a JSON array of objects with the following structure. Do not include markdown headers or extra text.\n{\n \"items\": [\n {\n \"akkadian_transliteration\": \"string\",\n \"translation\": \"string or null\",\n \"language\": \"string or null\",\n }\n ]\n}\nThe extracted data covers many languages (with some potential hallucinations from the LLM). The histogram below shows the language distribution in my dataset. I decided to use only English, French, German, and Turkish for pre-training, as these had sufficient data volume.
\n
Training uses Seq2SeqTrainer and takes approximately 20 hours on a single A100 GPU in Colab.\nI preprocessed the data following the dataset instructions and valuable discussions.
Hyperparameters:
\n| Parameter | \nValue | \n
|---|---|
| epochs | \n3 | \n
| batch size | \n8 | \n
| learning rate | \n1e-4 | \n
| max length | \n512 | \n
| label smoothing factor | \n0.1 | \n
| EMA decay | \n0.999 | \n
The training data contains translations aligned at the document level, whereas the test data is aligned at the sentence level. To bridge this gap, I used an LLM to segment the Akkadian transliterations and their English translations into sentence-by-sentence pairs.
\nSince I was unsure about the typical length of test examples, I analyzed test.csv and found that transliteration lengths range from 129 to 267 characters. Based on this, I applied a length constraint of 100–400 characters per segment. The key instructions I used are as follows:
Maintain Sequence: Do NOT change the original order of sentences in either language.\nLength Constraints: Each Akkadian segment must be between 100 and 400 characters long.\n - If a segment is **too short** (< 100 chars): merge it with the adjacent clause.\n - If a segment is **too long** (> 400 chars): split it at a natural boundary such as a new clause, conjunction (ù, -ma), or topic shift.\nI also noticed mismatching translation pairs in train.csv. The histogram below shows the distribution of translation-to-transliteration length ratios.
I replaced outliers with LLM-generated translations, using 0.66 and 2.0 as the lower and upper thresholds. I provided few-shot examples to guide the correction:
# Task\nImprove and correct the following English translation of an Akkadian text, using the original translation as a reference base.\n\n...\n\n# Examples\nAkkadian: 0.3333 ma-na 2.5 GÍN KÙ.BABBAR 20 NINDA i-ṣé-er tù-wa-ra-a-aḫ-šu a-lá-ḫu-um i-šu\nOriginal: Ali-ahum has 0.3333 mina 2.5 shekels of silver and 20 loaves of bread from Tuwar-ahšu.\nCorrected: Tuwar-ahšu owes 0.3333 mina 2.5 shekels of silver (and) 20 loaves of bread to Ali-ahum.\n\n...\n\n# Input\nAkkadian: {ak_input}\nOriginal: {original_translation}\nCorrected:\nI also generated translations for transliterations in publication_texts.csv using a similar prompt for use in pre-training.
Finally, I fine-tuned the pre-trained ByT5-large on the refined train.csv using the same training code as pre-training, with epochs=2 and lr=1e-5.
I ensembled three ByT5-large models fine-tuned on different pre-training datasets. To evaluate the impact of MBR decoding, I ran experiments on the 100 hardest validation samples (selected by val score) to keep the process manageable. I compared four candidate generation methods: normal beam search, group beam search, nucleus sampling, and epsilon sampling. The best result came from using normal beam search for candidate generation and group beam search for the candidate pool. I used the competition metric directly as the utility function for MBR decoding. Applying a soft margin penalty based on the translation/transliteration ratio, along with PN/GN lemmatization, provided a small additional boost.
\ndef calculate_soft_margin_penalty(ratio, low=0.9, high=2.0, sigma=0.2):\n distance = max(0, low - ratio, ratio - high)\n return np.exp(-(distance**2) / (2 * (sigma**2)))\nThe val scores were not well-correlated with the public leaderboard scores, which was puzzling. I believe the main cause is that my validation set was constructed by segmenting train.csv with an LLM and replacing mismatched pairs with LLM-generated translations. This made the val set share the same distribution and style as the training data, inflating the val scores and causing them to diverge from the actual test set distribution. That said, I'm relieved the final submission turned out reasonably well.
D1: published_texts.csv D2: publications.csv D3: train.csv
| pre-training dataset | \npre-training steps | \nval (w/o ft) | \nval | \nval hard | \npublic | \nprivate | \n
|---|---|---|---|---|---|---|
D1: 37910 D2: 79806 | \n42000 | \n36.2 | \n41.7 | \n23.8 | \n39.0 | \n38.9 | \n
D1: 37910 D2: 126439 D3: 3987 | \n38000 | \n45.8 | \n47.9 | \n29.6 | \n38.0 | \n38.3 | \n
D1: 37910 D2: 18834 | \n30000 | \n45.5 | \n46.6 | \n28.6 | \n38.2 | \n37.8 | \n
| (final submission) | \n- | \n- | \n- | \n30.14 | \n39.4 | \n39.8 | \n
OA_Lexicon_eBL.csveBL_Dictionary.csv