Rank: 32 / 2674 teams | Public: 37.6 | Private: 38.1
\nNo Gemini, no synthetic data, no large-scale PDF extraction. Our strategy was simpler: start from a strong base model, train carefully, iterate continuously, and use the leaderboard as the real validation signal.
\nThis writeup is for participants who want to understand what is achievable with only public data and a disciplined iterative process.
\nStage 1: Akkademia Pre-training
\nByT5-base was first fine-tuned on ~69k sentence pairs from the Akkademia corpus. Akkademia is not Old Assyrian — the relationship is similar to Portuguese vs Spanish. The goal was not to learn the competition domain directly, but to give the model familiarity with Akkadian grammar and morphology before seeing competition data. Without this stage, the model treats Old Assyrian as completely foreign.
\nStage 2: Iterative Fine-tuning
\nDue to Kaggle runtime limits (~12h/session), we adopted a progressive fine-tuning loop in cycles of 3 epochs:
\nWe repeated this loop over nearly 3 months, reaching 21 total epochs. Each iteration taught us which normalization rules helped, which hurt, and what decoding parameters worked best. The leaderboard became our continuous evaluation framework.
\nPublic data only:
\n| Source | \nRows | \n
|---|---|
train.csv | \n1,561 documents | \n
published_texts.csv | \n~199 recovered transliterations | \n
Sentences_Oare_FirstWord_LinNum.csv | \n841 gold sentence pairs | \n
akkademia_train_CLEAN.csv | \n~69k pre-training pairs | \n
After splitting long documents: ~2,800 sentence-level training samples
\nThe Sentences_Oare_FirstWord_LinNum.csv file was a critical late discovery ; it contains gold sentence boundaries using first_word_spelling as anchor, which closely matches how the hidden test set was constructed.
early_stopping=True causes repetition artifacts → hurts scorelength_penalty=0.91 consistently outperforms lower valuesApplied to training data following host recommendations:
\nḫ/Ḫ → h/H (confirmed absent from test set)0.3333 → ⅓)<gap><gap> / <big_gap> → single <gap>₄ → 4)fem., pl., (?) annotationsPN → <gap>Model: akk-v24-ep21 on Kaggle
\ngoogle/byt5-basenum_beams=6, length_penalty=0.91, early_stopping=FalseOur final inference is intentionally minimal:
\nout = model.generate(\n **enc,\n max_new_tokens=300,\n num_beams=6,\n length_penalty=0.91,\n # NO early_stopping\n)\npredictions = tokenizer.batch_decode(out, skip_special_tokens=True)\n# NO postprocessing applied\nNo preprocessing of input. No postprocessing of output.
\nThis was a late but important discovery. We spent months building a careful postprocessing pipeline; normalizing fractions, removing annotations, fixing gaps. Every rule was based on host-confirmed guidelines.
\nYet when we tested simple inference against our full pipeline, simple inference scored the same or higher.
\nWhy? The model was trained on already-cleaned data. It learned to output correct fractions, correct quotes, correct gaps directly. Our postprocessing was correcting things that were already correct; and occasionally breaking things that were fine.
\nThe lesson: preprocess training data carefully, postprocess model output as little as possible. Interestingly, the 1st place team said exactly the same thing in their writeup.
\nThe 1st place writeup says it best: \"Data quality dictates everything.\" Top teams had 30k+ clean sentence pairs. We had ~2,800. Despite this gap, the results show that careful iteration and data alignment can compensate for limited resources.
\n37.6 public → 38.1 private. Public data only. ByT5-base.
\nRank: 32 / 2674 teams | Public: 37.6 | Private: 38.1
\nNo Gemini, no synthetic data, no large-scale PDF extraction. Our strategy was simpler: start from a strong base model, train carefully, iterate continuously, and use the leaderboard as the real validation signal.
\nThis writeup is for participants who want to understand what is achievable with only public data and a disciplined iterative process.
\nStage 1: Akkademia Pre-training
\nByT5-base was first fine-tuned on ~69k sentence pairs from the Akkademia corpus. Akkademia is not Old Assyrian — the relationship is similar to Portuguese vs Spanish. The goal was not to learn the competition domain directly, but to give the model familiarity with Akkadian grammar and morphology before seeing competition data. Without this stage, the model treats Old Assyrian as completely foreign.
\nStage 2: Iterative Fine-tuning
\nDue to Kaggle runtime limits (~12h/session), we adopted a progressive fine-tuning loop in cycles of 3 epochs:
\nWe repeated this loop over nearly 3 months, reaching 21 total epochs. Each iteration taught us which normalization rules helped, which hurt, and what decoding parameters worked best. The leaderboard became our continuous evaluation framework.
\nPublic data only:
\n| Source | \nRows | \n
|---|---|
train.csv | \n1,561 documents | \n
published_texts.csv | \n~199 recovered transliterations | \n
Sentences_Oare_FirstWord_LinNum.csv | \n841 gold sentence pairs | \n
akkademia_train_CLEAN.csv | \n~69k pre-training pairs | \n
After splitting long documents: ~2,800 sentence-level training samples
\nThe Sentences_Oare_FirstWord_LinNum.csv file was a critical late discovery ; it contains gold sentence boundaries using first_word_spelling as anchor, which closely matches how the hidden test set was constructed.
early_stopping=True causes repetition artifacts → hurts scorelength_penalty=0.91 consistently outperforms lower valuesApplied to training data following host recommendations:
\nḫ/Ḫ → h/H (confirmed absent from test set)0.3333 → ⅓)<gap><gap> / <big_gap> → single <gap>₄ → 4)fem., pl., (?) annotationsPN → <gap>Model: akk-v24-ep21 on Kaggle
\ngoogle/byt5-basenum_beams=6, length_penalty=0.91, early_stopping=FalseOur final inference is intentionally minimal:
\nout = model.generate(\n **enc,\n max_new_tokens=300,\n num_beams=6,\n length_penalty=0.91,\n # NO early_stopping\n)\npredictions = tokenizer.batch_decode(out, skip_special_tokens=True)\n# NO postprocessing applied\nNo preprocessing of input. No postprocessing of output.
\nThis was a late but important discovery. We spent months building a careful postprocessing pipeline; normalizing fractions, removing annotations, fixing gaps. Every rule was based on host-confirmed guidelines.
\nYet when we tested simple inference against our full pipeline, simple inference scored the same or higher.
\nWhy? The model was trained on already-cleaned data. It learned to output correct fractions, correct quotes, correct gaps directly. Our postprocessing was correcting things that were already correct; and occasionally breaking things that were fine.
\nThe lesson: preprocess training data carefully, postprocess model output as little as possible. Interestingly, the 1st place team said exactly the same thing in their writeup.
\nThe 1st place writeup says it best: \"Data quality dictates everything.\" Top teams had 30k+ clean sentence pairs. We had ~2,800. Despite this gap, the results show that careful iteration and data alignment can compensate for limited resources.
\n37.6 public → 38.1 private. Public data only. ByT5-base.
\nFantastic work team! 👏 Your disciplined iterative approach and careful data preprocessing clearly paid off, especially given the limited resources. I’m impressed by how you leveraged public data effectively and turned each leaderboard submission into a learning opportunity. This writeup is a great example of how strategy and consistency can sometimes outweigh sheer scale. Excited to see how we can apply these insights to our next challenge!
", "rawMarkdown": "Fantastic work team! 👏 Your disciplined iterative approach and careful data preprocessing clearly paid off, especially given the limited resources. I’m impressed by how you leveraged public data effectively and turned each leaderboard submission into a learning opportunity. This writeup is a great example of how strategy and consistency can sometimes outweigh sheer scale. Excited to see how we can apply these insights to our next challenge!", "votes": null }, { "id": "3429255", "postDate": "03/26/2026 14:23:43", "content": "Fantastic work.
", "rawMarkdown": "Fantastic work.", "votes": null }, { "id": "3429286", "postDate": "03/26/2026 15:21:37", "content": "Happy with our result and the journey! This competition pushed us to explore new ideas and improve our approach.
", "rawMarkdown": "Happy with our result and the journey! This competition pushed us to explore new ideas and improve our approach.", "votes": null }, { "id": "3443570", "postDate": "04/16/2026 17:54:36", "content": "This solution is fascinating - I am very impressed by your ability to get such a high score with such a low data input and lower computational cost.
", "rawMarkdown": "This solution is fascinating - I am very impressed by your ability to get such a high score with such a low data input and lower computational cost.", "votes": null }, { "id": "3452351", "postDate": "05/03/2026 06:15:44", "content": "This is impressive work. I genuinely appreciate the effort and clarity.
", "rawMarkdown": "This is impressive work. I genuinely appreciate the effort and clarity.", "votes": null } ], "comments": [ { "id": 3429161, "author_name": "", "author_url": "", "post_date": "03/26/2026 13:03:55", "content": "Fantastic work team! 👏 Your disciplined iterative approach and careful data preprocessing clearly paid off, especially given the limited resources. I’m impressed by how you leveraged public data effectively and turned each leaderboard submission into a learning opportunity. This writeup is a great example of how strategy and consistency can sometimes outweigh sheer scale. Excited to see how we can apply these insights to our next challenge!
", "votes": null, "replies": [] }, { "id": 3429255, "author_name": "", "author_url": "", "post_date": "03/26/2026 14:23:43", "content": "Fantastic work.
", "votes": null, "replies": [] }, { "id": 3429286, "author_name": "", "author_url": "", "post_date": "03/26/2026 15:21:37", "content": "Happy with our result and the journey! This competition pushed us to explore new ideas and improve our approach.
", "votes": null, "replies": [] }, { "id": 3443570, "author_name": "leedrake", "author_url": "", "post_date": "04/16/2026 17:54:36", "content": "This solution is fascinating - I am very impressed by your ability to get such a high score with such a low data input and lower computational cost.
", "votes": null, "replies": [] }, { "id": 3452351, "author_name": "", "author_url": "", "post_date": "05/03/2026 06:15:44", "content": "This is impressive work. I genuinely appreciate the effort and clarity.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3429158": "# [32nd] ByT5-base Fine-tuning | No LLMs, No Synthetic Data, No Ensembles\n\n**Rank:** 32 / 2674 teams | **Public:** 37.6 | **Private:** 38.1\n\n---\n\n## Overview\n\nNo Gemini, no synthetic data, no large-scale PDF extraction. Our strategy was simpler: start from a strong base model, train carefully, iterate continuously, and use the leaderboard as the real validation signal.\n\nThis writeup is for participants who want to understand what is achievable with only public data and a disciplined iterative process.\n\n---\n\n## Two-Stage Training\n\n**Stage 1: Akkademia Pre-training**\n\nByT5-base was first fine-tuned on ~69k sentence pairs from the Akkademia corpus. Akkademia is not Old Assyrian — the relationship is similar to Portuguese vs Spanish. The goal was not to learn the competition domain directly, but to give the model familiarity with Akkadian grammar and morphology before seeing competition data. Without this stage, the model treats Old Assyrian as completely foreign.\n\n**Stage 2: Iterative Fine-tuning**\n\nDue to Kaggle runtime limits (~12h/session), we adopted a progressive fine-tuning loop in cycles of 3 epochs:\n\n1. Train 3 epochs from best checkpoint\n2. Save → create Kaggle dataset\n3. Run inference → submit\n4. Analyze score\n5. Continue or adjust\n\nWe repeated this loop over nearly 3 months, reaching **21 total epochs**. Each iteration taught us which normalization rules helped, which hurt, and what decoding parameters worked best. The leaderboard became our continuous evaluation framework.\n\n---\n\n## Data\n\nPublic data only:\n\n| Source | Rows |\n|---|---|\n| `train.csv` | 1,561 documents |\n| `published_texts.csv` | ~199 recovered transliterations |\n| `Sentences_Oare_FirstWord_LinNum.csv` | 841 gold sentence pairs |\n| `akkademia_train_CLEAN.csv` | ~69k pre-training pairs |\n\n---\nAfter splitting long documents: **~2,800 sentence-level training samples**\n\nThe `Sentences_Oare_FirstWord_LinNum.csv` file was a critical late discovery ; it contains gold sentence boundaries using `first_word_spelling` as anchor, which closely matches how the hidden test set was constructed.\n\n---\n\n## Key Findings from Leaderboard Submissions\n\n- `early_stopping=True` causes repetition artifacts → hurts score\n- `length_penalty=0.91` consistently outperforms lower values\n- Complex postprocessing provides no benefit; the model already internalized normalizations during training\n- Splitting documents to sentence level improves alignment with test distribution\n\n---\n\n## Normalization\n\nApplied to training data following host recommendations:\n\n- `ḫ/Ḫ → h/H` (confirmed absent from test set)\n- Decimals → unicode fractions (`0.3333 → ⅓`)\n- `