A huge thank you to the competition organizers and the Deep Past Initiative for hosting such an unique challenge - never heard about Akkadian language before :)
\nOur approach focused heavily on data curation (including synthetic translations via Gemini), a 3-phase training pipeline using byt5-base, and model souping to create a robust final ensemble without increasing inference time.
\nHere is a breakdown of our pipeline.
\nWe started with the provided train.csv, but then thought the difference will be made by having more data (reading the other write-ups not so sure…).
\nWe managed to scale up to ~80K pairs (average quality):
\nSentence Segmentation: Segmented the provided competition data into cleaner, sentence-level pairs.
\nGemini extraction & synthetic translation: Used Gemini to extract and align Akkadian-English parallel data from OCR'd books and academic papers (AKTs plus most of the titles mentioned by the others - lost track at some point…). This created a large, mixed-quality dataset.
\nGold Data: We curated a smaller, highly refined dataset (\"gold data\") consisting of the cleanest alignments for late-stage fine-tuning (~10K). Focused on quality of English and allowed noisy Akkadian (except fractions and other markers) thinking would make the model immune to mistakes.
\nExtensive Normalization: Because ByT5 works at the byte/character level, consistent text representation is crucial. We built a normalization pipeline that standardized gaps, converted decimal fractions to Unicode fractions (e.g., 0.5 -> ½, 0.33 -> ⅓), and standardized transliteration characters (e.g., sz -> š, Sa -> ṣa).
\nWent for google/byt5-base stock. Later tested with byt5-large and looking at the final scoring we should have not abandoned it. Looks like our validation data was not ideal.
\nAdded special tokens: < gap >.
\nBefore teaching the model to translate, we taught it the structure of Akkadian by concatenating all the available transliteration texts (parallel, extended, and extra datasets) and ran an unsupervised span corruption (Masked Language Modeling) task.
\nParameters: noise_density = 0.15, mean_noise_span_length = 15.
\nThis helped the model learn the byte-level morphology of Akkadian transliterations.
\nUsing the ~80K mixed-quality dataset, fine-tuned the denoised ByT5 model to translate Akkadian to English.
\nTo ensure stability and diversity, we trained two separate models on this dataset using two different random seeds (famous 42 and the current 2026).
\nHyperparameters: Adafactor optimizer, learning_rate=5e-5, bf16=True, gradient accumulation steps=4, and label smoothing (0.1).
\nTook the model and ran a final, shorter fine-tuning phase (5 epochs) exclusively on the highly refined \"Gold Data\". This acts as a curriculum learning step, teaching the model to prioritize high-quality academic phrasing and formatting over the noisier OCR data.
\nFor the sake of simplicity and saving compute we used Weight Averaging.
\nIntermediate Soup (50/50): Averaged the weights of the two Phase 2 models (Seed 42 and Seed 2026) at a 1:1 ratio.
\nFinal Soup (75/25): Took the Intermediate Soup and averaged it with the Phase 3 (Gold Data) model (obviously we first tried running a 2-3 epochs at lower learning rate as a polish, but failed).
\nFinal_Weights = (0.75 * Intermediate_Soup) + (0.25 * Phase_3_Model)
\nThis combination grounded the model in the vast vocabulary of the 80K dataset while gently shifting the distribution toward the high-quality gold data.
\nFor inference, sequences longer than 512 were dynamically split at the nearest space boundary to avoid truncation, decoded separately, and re-stitched. This increased a bit the score: +0.02ish
\nGeneration Parameters (we know… strange numbers):
\nnum_beams = 12\nno_repeat_ngram_size = 20\nrepetition_penalty = 1.6
\nCapitalization: regex fixing internal word capitalization errors.
\nTerminology Standardization: Hardcoded fixes for common domain terms (e.g., forcing \"pašallu-gold\" to \"pašallum gold\", \"transport tariff\" to \"šadduātum tax\").
\nOnomasticon Fuzzy Matching: Used difflib.get_close_matches against the onomasticon for composed names containing hypen. If the model generated a composed name but wasn't quite right, the script fuzzy-matched it against the onomasticon (with a 0.7 similarity cutoff) and snapped it to the historically accurate spelling.
\nSpent a few :) hours but worth every minute.
", "messages": [ { "id": "3427891", "postDate": "03/24/2026 16:46:18", "content": "A huge thank you to the competition organizers and the Deep Past Initiative for hosting such an unique challenge - never heard about Akkadian language before :)
\nOur approach focused heavily on data curation (including synthetic translations via Gemini), a 3-phase training pipeline using byt5-base, and model souping to create a robust final ensemble without increasing inference time.
\nHere is a breakdown of our pipeline.
\nWe started with the provided train.csv, but then thought the difference will be made by having more data (reading the other write-ups not so sure…).
\nWe managed to scale up to ~80K pairs (average quality):
\nSentence Segmentation: Segmented the provided competition data into cleaner, sentence-level pairs.
\nGemini extraction & synthetic translation: Used Gemini to extract and align Akkadian-English parallel data from OCR'd books and academic papers (AKTs plus most of the titles mentioned by the others - lost track at some point…). This created a large, mixed-quality dataset.
\nGold Data: We curated a smaller, highly refined dataset (\"gold data\") consisting of the cleanest alignments for late-stage fine-tuning (~10K). Focused on quality of English and allowed noisy Akkadian (except fractions and other markers) thinking would make the model immune to mistakes.
\nExtensive Normalization: Because ByT5 works at the byte/character level, consistent text representation is crucial. We built a normalization pipeline that standardized gaps, converted decimal fractions to Unicode fractions (e.g., 0.5 -> ½, 0.33 -> ⅓), and standardized transliteration characters (e.g., sz -> š, Sa -> ṣa).
\nWent for google/byt5-base stock. Later tested with byt5-large and looking at the final scoring we should have not abandoned it. Looks like our validation data was not ideal.
\nAdded special tokens: < gap >.
\nBefore teaching the model to translate, we taught it the structure of Akkadian by concatenating all the available transliteration texts (parallel, extended, and extra datasets) and ran an unsupervised span corruption (Masked Language Modeling) task.
\nParameters: noise_density = 0.15, mean_noise_span_length = 15.
\nThis helped the model learn the byte-level morphology of Akkadian transliterations.
\nUsing the ~80K mixed-quality dataset, fine-tuned the denoised ByT5 model to translate Akkadian to English.
\nTo ensure stability and diversity, we trained two separate models on this dataset using two different random seeds (famous 42 and the current 2026).
\nHyperparameters: Adafactor optimizer, learning_rate=5e-5, bf16=True, gradient accumulation steps=4, and label smoothing (0.1).
\nTook the model and ran a final, shorter fine-tuning phase (5 epochs) exclusively on the highly refined \"Gold Data\". This acts as a curriculum learning step, teaching the model to prioritize high-quality academic phrasing and formatting over the noisier OCR data.
\nFor the sake of simplicity and saving compute we used Weight Averaging.
\nIntermediate Soup (50/50): Averaged the weights of the two Phase 2 models (Seed 42 and Seed 2026) at a 1:1 ratio.
\nFinal Soup (75/25): Took the Intermediate Soup and averaged it with the Phase 3 (Gold Data) model (obviously we first tried running a 2-3 epochs at lower learning rate as a polish, but failed).
\nFinal_Weights = (0.75 * Intermediate_Soup) + (0.25 * Phase_3_Model)
\nThis combination grounded the model in the vast vocabulary of the 80K dataset while gently shifting the distribution toward the high-quality gold data.
\nFor inference, sequences longer than 512 were dynamically split at the nearest space boundary to avoid truncation, decoded separately, and re-stitched. This increased a bit the score: +0.02ish
\nGeneration Parameters (we know… strange numbers):
\nnum_beams = 12\nno_repeat_ngram_size = 20\nrepetition_penalty = 1.6
\nCapitalization: regex fixing internal word capitalization errors.
\nTerminology Standardization: Hardcoded fixes for common domain terms (e.g., forcing \"pašallu-gold\" to \"pašallum gold\", \"transport tariff\" to \"šadduātum tax\").
\nOnomasticon Fuzzy Matching: Used difflib.get_close_matches against the onomasticon for composed names containing hypen. If the model generated a composed name but wasn't quite right, the script fuzzy-matched it against the onomasticon (with a 0.7 similarity cutoff) and snapped it to the historically accurate spelling.
\nSpent a few :) hours but worth every minute.
", "rawMarkdown": "A huge thank you to the competition organizers and the Deep Past Initiative for hosting such an unique challenge - never heard about Akkadian language before :)\n\nOur approach focused heavily on data curation (including synthetic translations via Gemini), a 3-phase training pipeline using byt5-base, and model souping to create a robust final ensemble without increasing inference time.\n\nHere is a breakdown of our pipeline.\n\n### **1. Data Preparation & Augmentation**\nWe started with the provided train.csv, but then thought the difference will be made by having more data (reading the other write-ups not so sure...). \n\nWe managed to scale up to ~80K pairs (average quality):\n\n*Sentence Segmentation:* Segmented the provided competition data into cleaner, sentence-level pairs.\n\n*Gemini extraction & synthetic translation:* Used Gemini to extract and align Akkadian-English parallel data from OCR'd books and academic papers (AKTs plus most of the titles mentioned by the others - lost track at some point...). This created a large, mixed-quality dataset.\n\n*Gold Data:* We curated a smaller, highly refined dataset (\"gold data\") consisting of the cleanest alignments for late-stage fine-tuning (~10K). Focused on quality of English and allowed noisy Akkadian (except fractions and other markers) thinking would make the model immune to mistakes.\n\n*Extensive Normalization:* Because ByT5 works at the byte/character level, consistent text representation is crucial. We built a normalization pipeline that standardized gaps, converted decimal fractions to Unicode fractions (e.g., 0.5 -> ½, 0.33 -> ⅓), and standardized transliteration characters (e.g., sz -> š, Sa -> ṣa).\n\n###**2. Training Pipeline (3 Phases)**\nWent for google/byt5-base stock. Later tested with byt5-large and looking at the final scoring we should have not abandoned it. Looks like our validation data was not ideal.\n\nAdded special tokens: < gap >.\n\n##*Phase 1: Unsupervised Denoising (Span Corruption)*\nBefore teaching the model to translate, we taught it the structure of Akkadian by concatenating all the available transliteration texts (parallel, extended, and extra datasets) and ran an unsupervised span corruption (Masked Language Modeling) task.\n\nParameters: noise_density = 0.15, mean_noise_span_length = 15.\n\nThis helped the model learn the byte-level morphology of Akkadian transliterations.\n\n##*Phase 2: Supervised Fine-Tuning (Mixed Data)*\nUsing the ~80K mixed-quality dataset, fine-tuned the denoised ByT5 model to translate Akkadian to English.\n\nTo ensure stability and diversity, we trained two separate models on this dataset using two different random seeds (famous 42 and the current 2026).\n\nHyperparameters: Adafactor optimizer, learning_rate=5e-5, bf16=True, gradient accumulation steps=4, and label smoothing (0.1).\n\n##*Phase 3: High-Quality Fine-Tuning (Gold Data)*\nTook the model and ran a final, shorter fine-tuning phase (5 epochs) exclusively on the highly refined \"Gold Data\". This acts as a curriculum learning step, teaching the model to prioritize high-quality academic phrasing and formatting over the noisier OCR data.\n\n###*3. Making the soup 🥣*\nFor the sake of simplicity and saving compute we used Weight Averaging.\n\nIntermediate Soup (50/50): Averaged the weights of the two Phase 2 models (Seed 42 and Seed 2026) at a 1:1 ratio.\n\nFinal Soup (75/25): Took the Intermediate Soup and averaged it with the Phase 3 (Gold Data) model (obviously we first tried running a 2-3 epochs at lower learning rate as a polish, but failed).\n\nFinal_Weights = (0.75 * Intermediate_Soup) + (0.25 * Phase_3_Model)\n\nThis combination grounded the model in the vast vocabulary of the 80K dataset while gently shifting the distribution toward the high-quality gold data.\n\n###*4. Inference & Post-Processing*\nFor inference, sequences longer than 512 were dynamically split at the nearest space boundary to avoid truncation, decoded separately, and re-stitched. This increased a bit the score: +0.02ish\n\nGeneration Parameters (we know... strange numbers):\n\nnum_beams = 12\nno_repeat_ngram_size = 20\nrepetition_penalty = 1.6\n\n####+ some post-processing:\n\n*Capitalization:* regex fixing internal word capitalization errors.\n\n*Terminology Standardization:* Hardcoded fixes for common domain terms (e.g., forcing \"pašallu-gold\" to \"pašallum gold\", \"transport tariff\" to \"šadduātum tax\"). \n\n*Onomasticon Fuzzy Matching:* Used difflib.get_close_matches against the onomasticon for composed names containing hypen. If the model generated a composed name but wasn't quite right, the script fuzzy-matched it against the onomasticon (with a 0.7 similarity cutoff) and snapped it to the historically accurate spelling.\n\nSpent a few :) hours but worth every minute.", "votes": null }, { "id": "3517315", "postDate": "08/26/2026 18:34:20", "content": "Hey Vin and Eva,\nWell done on placing so well on the competition!\nI am doing some exploration of the competition, and was wondering if you could provide me with your final solution as i could not find a publci rerunnable script. I would like to run it on some texts of my own\nThanks in advance!
", "rawMarkdown": "Hey Vin and Eva,\nWell done on placing so well on the competition!\nI am doing some exploration of the competition, and was wondering if you could provide me with your final solution as i could not find a publci rerunnable script. I would like to run it on some texts of my own\nThanks in advance!", "votes": null } ], "comments": [ { "id": 3517315, "author_name": "shaharspencer", "author_url": "", "post_date": "08/26/2026 18:34:20", "content": "Hey Vin and Eva,\nWell done on placing so well on the competition!\nI am doing some exploration of the competition, and was wondering if you could provide me with your final solution as i could not find a publci rerunnable script. I would like to run it on some texts of my own\nThanks in advance!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3427891": "A huge thank you to the competition organizers and the Deep Past Initiative for hosting such an unique challenge - never heard about Akkadian language before :)\n\nOur approach focused heavily on data curation (including synthetic translations via Gemini), a 3-phase training pipeline using byt5-base, and model souping to create a robust final ensemble without increasing inference time.\n\nHere is a breakdown of our pipeline.\n\n### **1. Data Preparation & Augmentation**\nWe started with the provided train.csv, but then thought the difference will be made by having more data (reading the other write-ups not so sure...). \n\nWe managed to scale up to ~80K pairs (average quality):\n\n*Sentence Segmentation:* Segmented the provided competition data into cleaner, sentence-level pairs.\n\n*Gemini extraction & synthetic translation:* Used Gemini to extract and align Akkadian-English parallel data from OCR'd books and academic papers (AKTs plus most of the titles mentioned by the others - lost track at some point...). This created a large, mixed-quality dataset.\n\n*Gold Data:* We curated a smaller, highly refined dataset (\"gold data\") consisting of the cleanest alignments for late-stage fine-tuning (~10K). Focused on quality of English and allowed noisy Akkadian (except fractions and other markers) thinking would make the model immune to mistakes.\n\n*Extensive Normalization:* Because ByT5 works at the byte/character level, consistent text representation is crucial. We built a normalization pipeline that standardized gaps, converted decimal fractions to Unicode fractions (e.g., 0.5 -> ½, 0.33 -> ⅓), and standardized transliteration characters (e.g., sz -> š, Sa -> ṣa).\n\n###**2. Training Pipeline (3 Phases)**\nWent for google/byt5-base stock. Later tested with byt5-large and looking at the final scoring we should have not abandoned it. Looks like our validation data was not ideal.\n\nAdded special tokens: < gap >.\n\n##*Phase 1: Unsupervised Denoising (Span Corruption)*\nBefore teaching the model to translate, we taught it the structure of Akkadian by concatenating all the available transliteration texts (parallel, extended, and extra datasets) and ran an unsupervised span corruption (Masked Language Modeling) task.\n\nParameters: noise_density = 0.15, mean_noise_span_length = 15.\n\nThis helped the model learn the byte-level morphology of Akkadian transliterations.\n\n##*Phase 2: Supervised Fine-Tuning (Mixed Data)*\nUsing the ~80K mixed-quality dataset, fine-tuned the denoised ByT5 model to translate Akkadian to English.\n\nTo ensure stability and diversity, we trained two separate models on this dataset using two different random seeds (famous 42 and the current 2026).\n\nHyperparameters: Adafactor optimizer, learning_rate=5e-5, bf16=True, gradient accumulation steps=4, and label smoothing (0.1).\n\n##*Phase 3: High-Quality Fine-Tuning (Gold Data)*\nTook the model and ran a final, shorter fine-tuning phase (5 epochs) exclusively on the highly refined \"Gold Data\". This acts as a curriculum learning step, teaching the model to prioritize high-quality academic phrasing and formatting over the noisier OCR data.\n\n###*3. Making the soup 🥣*\nFor the sake of simplicity and saving compute we used Weight Averaging.\n\nIntermediate Soup (50/50): Averaged the weights of the two Phase 2 models (Seed 42 and Seed 2026) at a 1:1 ratio.\n\nFinal Soup (75/25): Took the Intermediate Soup and averaged it with the Phase 3 (Gold Data) model (obviously we first tried running a 2-3 epochs at lower learning rate as a polish, but failed).\n\nFinal_Weights = (0.75 * Intermediate_Soup) + (0.25 * Phase_3_Model)\n\nThis combination grounded the model in the vast vocabulary of the 80K dataset while gently shifting the distribution toward the high-quality gold data.\n\n###*4. Inference & Post-Processing*\nFor inference, sequences longer than 512 were dynamically split at the nearest space boundary to avoid truncation, decoded separately, and re-stitched. This increased a bit the score: +0.02ish\n\nGeneration Parameters (we know... strange numbers):\n\nnum_beams = 12\nno_repeat_ngram_size = 20\nrepetition_penalty = 1.6\n\n####+ some post-processing:\n\n*Capitalization:* regex fixing internal word capitalization errors.\n\n*Terminology Standardization:* Hardcoded fixes for common domain terms (e.g., forcing \"pašallu-gold\" to \"pašallum gold\", \"transport tariff\" to \"šadduātum tax\"). \n\n*Onomasticon Fuzzy Matching:* Used difflib.get_close_matches against the onomasticon for composed names containing hypen. If the model generated a composed name but wasn't quite right, the script fuzzy-matched it against the onomasticon (with a 0.7 similarity cutoff) and snapped it to the historically accurate spelling.\n\nSpent a few :) hours but worth every minute.", "3517315": "Hey Vin and Eva,\nWell done on placing so well on the competition!\nI am doing some exploration of the competition, and was wondering if you could provide me with your final solution as i could not find a publci rerunnable script. I would like to run it on some texts of my own\nThanks in advance!" }, "source": "meta" }