{ "id": 684353, "title": "[DPC 1st] Data Quality Dictates Everything", "url": "/competitions/deep-past-initiative-machine-translation/writeups/dpc-1st-data-quality-dictates-everything", "author_name": "", "post_date": "2026-03-25T00:15:44.660Z", "votes": 94, "comment_count": 28, "views": 0, "content": "

Acknowledgments

\n

Thanks to the hosts and Kaggle staffs, especially Adam Anderson. In the early stages of the competition, there were significant data issues that caused a lot of confusion, but Anderson worked diligently to resolve the most critical problems, which greatly reduced the difficulty of optimizing the models.

\n

Thanks to my teammates mutian, zhengru, and yueting. Mutian and I have participated in many competitions on other platforms, and finally, in the year before starting our master's degrees, we had the time to compete on Kaggle, which is a much tougher challenge.

\n

Datatech Club is a student club founded by mutian at his university (ShanghaiTech University), and zhengru and yueting are both members. The club received computing resource support from the School of Information Science and Technology at ShanghaiTech University, and we are extremely grateful for their strong support.

\n

TL;DR

\n

The best submission (which we did not select) had a public score of 41.5 and a private score of 43.2. You can find the code and models in the end of this article.

\n

\"\"

\n

The best submission is an ensemble of 11 byt5-xl models. Each model generates 10 candidate translations using beam search and sampling at different temperatures, and finally, the best translation is selected via a community-contributed MBR selector.

\n

These 11 byt5 models were trained on different combinations of datasets. 10 of them used the basic Seq2SeqTrainer, bs=64, trained for 3 epochs; 1 model used a custom loss weighted by data quality, bs=48, trained for 4 epochs.

\n

Single-model inference for these models can easily reach the gold medal zone, and the best ckpt could have directly won 1st place as a single model. Below are some single-model inference submission scores (which do not fully comprise the final models used).

\n

\"\"

\n

The byt5-xl models were pre-compiled and quantized to \"int8-float32\"—this feature is implemented by the ctranslate2 library. The script performed data-parallel inference on 2*T5 GPUs, and we almost used up the entire 9-hour time limit.

\n

My thoughts on pre-processing and post-processing: Pre-process the training data as carefully as possible, and post-process the model output as little as possible. This is because we can always ensure the transliteration format is uniform, but relying on the Public LB for post-processing brings the risk of overfitting and shake-up.

\n

One of our selected submissions added a cleaning step for repeated n-grams (n=2-7) on top of the best submission. Its public score was 41.6, and private score was 42.8. (I always believed post-processing only increases overfitting risk, so I didn't conduct any post-processing experiments. But a day before the deadline, I had no other ways to continue improving the score, so I ultimately chose this submission which gave me a higher public LB rank. The fact that it wasn't the best submission exactly proves my judgment was correct 😀)

\n

\"\"

\n

Another selected submission did not use ct2 or quantization. It was a conservative ensemble of the top 4 ckpts on the LB. Its public score was 41.2, private score was 42.4.

\n

\"\"

\n

Data is the Most Important

\n

1. OCR Data: Extracting Transliteration-Translation from Books

\n

We extracted training data from PDFs provided by the hosts and collected by ourselves (we completely discarded train.csv).

\n

For well-structured books (with fixed chapter formats and clear transliteration-translation alignments), such as AKT series, ICK_4, Larsen_2002, Michel_2020, POAT, we extracted data in batches using a structured approach.

\n\n

For poorly-structured books, especially some short open-access texts scattered across the internet, we used a frontend created via vibe coding for extraction. The frontend allowed us to manually take screenshots, paste them into the window, select pre-set prompts, create queries, and call the API in batches.

\n

\"\"

\n

Our prompts were written in Chinese, and here are some key points translated in English by LLM:

\n

1) Segmenting by Logical Meaning

\n
1. Strip away \"physical line breaks\" and pursue \"logical sentence meaning\"\n- No punctuation in ancient times: Akkadian transliterations on tablets strictly follow the physical lines of the clay tablet (words at the end of a line are broken directly, e.g., A-šùr-ta-ak-lá-/ku).\n- Use English translation as an anchor: English translations by modern scholars have already sorted out the logical syntax and added modern punctuation. Therefore, when segmenting sentences, you must never map them line-by-line. Instead, you must first understand the complete, long English sentence, and then look for the corresponding start and end points in the transliteration.\n
\n

2) Correctly Understanding the Tablet

\n
2. Beware of cross-page and cross-section continuations caused by \"3D writing\"\n- A clay tablet is a three-dimensional writing medium. After filling the obverse, scribes would continue to the lower edge (l.e.), then flip to the reverse (rev.), and finally might write on the upper edge (u.e.) and left edge (l.e.).\n- Difficulty: A complete clause often spans across these areas. During alignment, if you find the second half of a sentence is missing, you must immediately look for the connecting point in the following lines.\n
\n

3) Understanding Grammar

\n
3. Adapt to the \"spatiotemporal dislocation\" of grammar (SOV vs SVO)\n- Akkadian is SOV (Subject-Object-Verb): Verbs are usually placed at the very end of the sentence (e.g., I - silver - gave).\n- English is SVO (Subject-Verb-Object): When translating, the predicate verb is moved forward. This means that a verb at the beginning of an English sentence might only be found in the last few lines of that large transliteration paragraph on the left. Sentence segmentation requires a \"global view\" rather than fixating on individual words.\n
\n

4) Locating Special Vocabulary

\n
4. Make good use of \"anchor\" words for precise positioning\n- In the dense transliteration, several types of words make excellent \"anchor points\":\n- Sumerian logograms (remember to write them in uppercase letters in your returned results): e.g., KÙ.BABBAR (silver), AN.NA (tin), TÚG (textile/garment), GÚ (talent), GÍN (shekel).\n- Numbers: Ancient merchants' accounts were extremely precise. Finding numbers (e.g., 1/2, 10, 15) is the fastest way to match bilingual texts.\n- Personal and Divine Names: e.g., A-šùr (Ashur), Šu-Nu-nu (Shu-Nunu). These proper nouns usually have highly consistent spellings in both the transliteration and translation.\n
\n

2. LLM Labeled Data: Generating Translations for Untranslated Transliterations using LLMs

\n

The official OARE website provides a great database containing a massive amount of untranslated data. Based on tablet names and multi-sequence matching, we excluded the data that had already been OCR'd. Combining dictionary lookups and similar transliteration-translation example queries, we called Gemini to generate translations.

\n

3. LLM Synthetic Data: Generating Completely Fictional Transliterations and Translations using LLMs

\n

1) Dictionary vocabulary synthetic data (synth1). This module first conducted a coverage gap analysis between the dictionary and the training set to filter out words truly missing from the training set. Then, it used the LLM to generate transliteration-translation pairs containing these words. The LLM input included 100 example sentences and 5 dictionary words, outputting 5-10 data pairs. Personal names and words lacking translations were excluded from the dictionary words.

\n

2) Free synthetic data (synth2). Based on 100 randomly selected sentences from the training set, it generated 3-7 sentence pairs.

\n

Dataset usage for model training

\n

Next, we introduce the datasets used for training:

\n

data1 (OCR data): Completed on March 8, containing 29,908 sentence-level data pairs (4,472 tablets). All sourced from LLM-extracted book data, without any manual cleaning.

\n

\n \n  \n \n

\n

data2 (OCR data): Completed on March 14, containing 30,931 sentence-level data pairs (4,476 tablets). Based on data1, we trained models using 4-fold CV, and each fold's model inferred on its respective training fold. Since the training set should be well-fitted, if the inferred translation was far from the training translation, the sample was considered an error/hard sample. Filtering by a cross-fold average geo_metric < 20, we identified 740 tablets/chapters containing such sentence-level samples. We re-extracted these tablets using the LLM and supplemented with manual verification.

\n

\n \n  \n \n

\n

data3 (OCR data): Completed on March 19, containing 34,146 sentence-level data pairs (4,837 tablets). We further supplemented some extraction results on top of data2. Additionally, transliteration-translation length mismatches and overly long transliterations can cause model misunderstandings. Filtering by a character length difference > 100 and transliteration character length > 500, we identified 131 tablets containing such samples. We re-extracted these tablets using the LLM and supplemented with manual verification.

\n

\n \n  \n \n

\n

llmlabel (LLM Labeled Data): Completed on March 20, containing 21,759 sentence-level data pairs (3,946 tablets). Training with only this dataset yielded a public score of 35.0 and a private score of 37.1.

\n

\n \n  \n \n

\n

synth1 (LLM Synthetic Data): Completed on March 8, containing 9,685 sentence-level data pairs.

\n

\n \n  \n \n

\n

synth2 (LLM Synthetic Data): Completed on March 20, containing 4,972 sentence-level data pairs.

\n

\n \n  \n \n

\n

The final ensemble of 11 models is basically a combination of the data above:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
model namedata1data2data3llmlabelsynth1synth2all/splitnotePublic ScorePrivate Score
A✅all39.640.4
B✅✅all40.141.2
C✅all39.740.9
D✅✅all40.341.3
E✅all40.140.6
F✅✅all40.141.6
G✅✅✅✅all40.141.3
H✅✅✅✅all40.040.6
I✅split9:1 train valid split40.041.2
J✅split40.441.1
K✅splitweighted loss by data quality39.140.0
\n

Note: these socres are tested on ct2 int8_float32 model

\n

For comparision, some float16 transformers models results are as follows:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
model namedata1data2data3llmlabelsynth1synth2all/splitnotePublic ScorePrivate Score
byt5-xl-0317-ocr_akka2en/1308✅split40.541.0
byt5-xl-0319-ocr_akka2en/1563✅all40.140.9
byt5-xl-0321-data2_llmlabel/split/2238✅✅split40.541.4
byt5-xl-0321-data2_llmlabel_synth2/all/2703✅✅✅all40.442.0
byt5-xl-0322-data3_llmlabel/split/2364✅✅split40.741.0
\n

Regarding Model Training and Inference

\n
    \n

  1. Beware of the model over-memorizing specific formats

    \n

    Use eval_loss for evaluation: Since there is inevitably a distribution and style inconsistency between the local validation set and the hidden test set, continuous improvements in eval_bleu/chrf after 3 epochs are untrustworthy. Therefore, we selected the 3-epoch checkpoints based on the validation set's eval_loss rather than eval_bleu/chrf.

    \n

    \"\"

    2. \n

  2. Larger models are better

    \n

    About 10 days before the competition ended, we started switching from byt5-base to byt5-large, and then further to byt5-xl. The score increase on the public LB was significant.

    \n

    However, the prerequisite for this conclusion might be: Having sufficient and clean training data, so that larger models do not fall into overfitting.

    3. \n

  3. MBR

    \n

    Referring to open-source MBR strategies, during the inference stage, beam search (beam_size=8) selected 4 candidates, and sampling at three temperatures [0.60, 0.80, 1.05] yielded 2 candidates each. After unified post-processing, we combined chrF++, BLEU, Jaccard, and length rewards to use weighted MBR to select the final translation that had the highest overall consistency with other candidates.

    4. \n

  4. Some necessary inference details

    \n
    5. \n

  5. Things we tried but eventually abandoned (also listing related community writeups to compensate for our insufficient experiments)

    \n
    6. \n
\n

Computing Resources Used

\n

In the early stages of training the byt5-base models, we used 1 x A100 for training.

\n

In the later stages of training the byt5-xl models, we used 8 x H20 for training.

", "messages": [ { "id": "3427819", "postDate": "03/24/2026 15:43:33", "content": "

Acknowledgments

\n

Thanks to the hosts and Kaggle staffs, especially Adam Anderson. In the early stages of the competition, there were significant data issues that caused a lot of confusion, but Anderson worked diligently to resolve the most critical problems, which greatly reduced the difficulty of optimizing the models.

\n

Thanks to my teammates mutian, zhengru, and yueting. Mutian and I have participated in many competitions on other platforms, and finally, in the year before starting our master's degrees, we had the time to compete on Kaggle, which is a much tougher challenge.

\n

Datatech Club is a student club founded by mutian at his university (ShanghaiTech University), and zhengru and yueting are both members. The club received computing resource support from the School of Information Science and Technology at ShanghaiTech University, and we are extremely grateful for their strong support.

\n

TL;DR

\n

The best submission (which we did not select) had a public score of 41.5 and a private score of 43.2. You can find the code and models in the end of this article.

\n

\"\"

\n

The best submission is an ensemble of 11 byt5-xl models. Each model generates 10 candidate translations using beam search and sampling at different temperatures, and finally, the best translation is selected via a community-contributed MBR selector.

\n

These 11 byt5 models were trained on different combinations of datasets. 10 of them used the basic Seq2SeqTrainer, bs=64, trained for 3 epochs; 1 model used a custom loss weighted by data quality, bs=48, trained for 4 epochs.

\n

Single-model inference for these models can easily reach the gold medal zone, and the best ckpt could have directly won 1st place as a single model. Below are some single-model inference submission scores (which do not fully comprise the final models used).

\n

\"\"

\n

The byt5-xl models were pre-compiled and quantized to \"int8-float32\"—this feature is implemented by the ctranslate2 library. The script performed data-parallel inference on 2*T5 GPUs, and we almost used up the entire 9-hour time limit.

\n

My thoughts on pre-processing and post-processing: Pre-process the training data as carefully as possible, and post-process the model output as little as possible. This is because we can always ensure the transliteration format is uniform, but relying on the Public LB for post-processing brings the risk of overfitting and shake-up.

\n

One of our selected submissions added a cleaning step for repeated n-grams (n=2-7) on top of the best submission. Its public score was 41.6, and private score was 42.8. (I always believed post-processing only increases overfitting risk, so I didn't conduct any post-processing experiments. But a day before the deadline, I had no other ways to continue improving the score, so I ultimately chose this submission which gave me a higher public LB rank. The fact that it wasn't the best submission exactly proves my judgment was correct 😀)

\n

\"\"

\n

Another selected submission did not use ct2 or quantization. It was a conservative ensemble of the top 4 ckpts on the LB. Its public score was 41.2, private score was 42.4.

\n

\"\"

\n

Data is the Most Important

\n

1. OCR Data: Extracting Transliteration-Translation from Books

\n

We extracted training data from PDFs provided by the hosts and collected by ourselves (we completely discarded train.csv).

\n

For well-structured books (with fixed chapter formats and clear transliteration-translation alignments), such as AKT series, ICK_4, Larsen_2002, Michel_2020, POAT, we extracted data in batches using a structured approach.

\n\n

For poorly-structured books, especially some short open-access texts scattered across the internet, we used a frontend created via vibe coding for extraction. The frontend allowed us to manually take screenshots, paste them into the window, select pre-set prompts, create queries, and call the API in batches.

\n

\"\"

\n

Our prompts were written in Chinese, and here are some key points translated in English by LLM:

\n

1) Segmenting by Logical Meaning

\n
1. Strip away \"physical line breaks\" and pursue \"logical sentence meaning\"\n- No punctuation in ancient times: Akkadian transliterations on tablets strictly follow the physical lines of the clay tablet (words at the end of a line are broken directly, e.g., A-šùr-ta-ak-lá-/ku).\n- Use English translation as an anchor: English translations by modern scholars have already sorted out the logical syntax and added modern punctuation. Therefore, when segmenting sentences, you must never map them line-by-line. Instead, you must first understand the complete, long English sentence, and then look for the corresponding start and end points in the transliteration.\n
\n

2) Correctly Understanding the Tablet

\n
2. Beware of cross-page and cross-section continuations caused by \"3D writing\"\n- A clay tablet is a three-dimensional writing medium. After filling the obverse, scribes would continue to the lower edge (l.e.), then flip to the reverse (rev.), and finally might write on the upper edge (u.e.) and left edge (l.e.).\n- Difficulty: A complete clause often spans across these areas. During alignment, if you find the second half of a sentence is missing, you must immediately look for the connecting point in the following lines.\n
\n

3) Understanding Grammar

\n
3. Adapt to the \"spatiotemporal dislocation\" of grammar (SOV vs SVO)\n- Akkadian is SOV (Subject-Object-Verb): Verbs are usually placed at the very end of the sentence (e.g., I - silver - gave).\n- English is SVO (Subject-Verb-Object): When translating, the predicate verb is moved forward. This means that a verb at the beginning of an English sentence might only be found in the last few lines of that large transliteration paragraph on the left. Sentence segmentation requires a \"global view\" rather than fixating on individual words.\n
\n

4) Locating Special Vocabulary

\n
4. Make good use of \"anchor\" words for precise positioning\n- In the dense transliteration, several types of words make excellent \"anchor points\":\n- Sumerian logograms (remember to write them in uppercase letters in your returned results): e.g., KÙ.BABBAR (silver), AN.NA (tin), TÚG (textile/garment), GÚ (talent), GÍN (shekel).\n- Numbers: Ancient merchants' accounts were extremely precise. Finding numbers (e.g., 1/2, 10, 15) is the fastest way to match bilingual texts.\n- Personal and Divine Names: e.g., A-šùr (Ashur), Šu-Nu-nu (Shu-Nunu). These proper nouns usually have highly consistent spellings in both the transliteration and translation.\n
\n

2. LLM Labeled Data: Generating Translations for Untranslated Transliterations using LLMs

\n

The official OARE website provides a great database containing a massive amount of untranslated data. Based on tablet names and multi-sequence matching, we excluded the data that had already been OCR'd. Combining dictionary lookups and similar transliteration-translation example queries, we called Gemini to generate translations.

\n

3. LLM Synthetic Data: Generating Completely Fictional Transliterations and Translations using LLMs

\n

1) Dictionary vocabulary synthetic data (synth1). This module first conducted a coverage gap analysis between the dictionary and the training set to filter out words truly missing from the training set. Then, it used the LLM to generate transliteration-translation pairs containing these words. The LLM input included 100 example sentences and 5 dictionary words, outputting 5-10 data pairs. Personal names and words lacking translations were excluded from the dictionary words.

\n

2) Free synthetic data (synth2). Based on 100 randomly selected sentences from the training set, it generated 3-7 sentence pairs.

\n

Dataset usage for model training

\n

Next, we introduce the datasets used for training:

\n

data1 (OCR data): Completed on March 8, containing 29,908 sentence-level data pairs (4,472 tablets). All sourced from LLM-extracted book data, without any manual cleaning.

\n

\n \n  \n \n

\n

data2 (OCR data): Completed on March 14, containing 30,931 sentence-level data pairs (4,476 tablets). Based on data1, we trained models using 4-fold CV, and each fold's model inferred on its respective training fold. Since the training set should be well-fitted, if the inferred translation was far from the training translation, the sample was considered an error/hard sample. Filtering by a cross-fold average geo_metric < 20, we identified 740 tablets/chapters containing such sentence-level samples. We re-extracted these tablets using the LLM and supplemented with manual verification.

\n

\n \n  \n \n

\n

data3 (OCR data): Completed on March 19, containing 34,146 sentence-level data pairs (4,837 tablets). We further supplemented some extraction results on top of data2. Additionally, transliteration-translation length mismatches and overly long transliterations can cause model misunderstandings. Filtering by a character length difference > 100 and transliteration character length > 500, we identified 131 tablets containing such samples. We re-extracted these tablets using the LLM and supplemented with manual verification.

\n

\n \n  \n \n

\n

llmlabel (LLM Labeled Data): Completed on March 20, containing 21,759 sentence-level data pairs (3,946 tablets). Training with only this dataset yielded a public score of 35.0 and a private score of 37.1.

\n

\n \n  \n \n

\n

synth1 (LLM Synthetic Data): Completed on March 8, containing 9,685 sentence-level data pairs.

\n

\n \n  \n \n

\n

synth2 (LLM Synthetic Data): Completed on March 20, containing 4,972 sentence-level data pairs.

\n

\n \n  \n \n

\n

The final ensemble of 11 models is basically a combination of the data above:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
model namedata1data2data3llmlabelsynth1synth2all/splitnotePublic ScorePrivate Score
A✅all39.640.4
B✅✅all40.141.2
C✅all39.740.9
D✅✅all40.341.3
E✅all40.140.6
F✅✅all40.141.6
G✅✅✅✅all40.141.3
H✅✅✅✅all40.040.6
I✅split9:1 train valid split40.041.2
J✅split40.441.1
K✅splitweighted loss by data quality39.140.0
\n

Note: these socres are tested on ct2 int8_float32 model

\n

For comparision, some float16 transformers models results are as follows:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
model namedata1data2data3llmlabelsynth1synth2all/splitnotePublic ScorePrivate Score
byt5-xl-0317-ocr_akka2en/1308✅split40.541.0
byt5-xl-0319-ocr_akka2en/1563✅all40.140.9
byt5-xl-0321-data2_llmlabel/split/2238✅✅split40.541.4
byt5-xl-0321-data2_llmlabel_synth2/all/2703✅✅✅all40.442.0
byt5-xl-0322-data3_llmlabel/split/2364✅✅split40.741.0
\n

Regarding Model Training and Inference

\n
    \n

  1. Beware of the model over-memorizing specific formats

    \n

    Use eval_loss for evaluation: Since there is inevitably a distribution and style inconsistency between the local validation set and the hidden test set, continuous improvements in eval_bleu/chrf after 3 epochs are untrustworthy. Therefore, we selected the 3-epoch checkpoints based on the validation set's eval_loss rather than eval_bleu/chrf.

    \n

    \"\"

    2. \n

  2. Larger models are better

    \n

    About 10 days before the competition ended, we started switching from byt5-base to byt5-large, and then further to byt5-xl. The score increase on the public LB was significant.

    \n

    However, the prerequisite for this conclusion might be: Having sufficient and clean training data, so that larger models do not fall into overfitting.

    3. \n

  3. MBR

    \n

    Referring to open-source MBR strategies, during the inference stage, beam search (beam_size=8) selected 4 candidates, and sampling at three temperatures [0.60, 0.80, 1.05] yielded 2 candidates each. After unified post-processing, we combined chrF++, BLEU, Jaccard, and length rewards to use weighted MBR to select the final translation that had the highest overall consistency with other candidates.

    4. \n

  4. Some necessary inference details

    \n
    5. \n

  5. Things we tried but eventually abandoned (also listing related community writeups to compensate for our insufficient experiments)

    \n
    6. \n
\n

Computing Resources Used

\n

In the early stages of training the byt5-base models, we used 1 x A100 for training.

\n

In the later stages of training the byt5-xl models, we used 8 x H20 for training.

", "rawMarkdown": "# Acknowledgments\n\nThanks to the hosts and Kaggle staffs, especially Adam Anderson. In the early stages of the competition, there were significant data issues that caused a lot of confusion, but Anderson worked diligently to resolve the most critical problems, which greatly reduced the difficulty of optimizing the models.\n\nThanks to my teammates mutian, zhengru, and yueting. Mutian and I have participated in many competitions on other platforms, and finally, in the year before starting our master's degrees, we had the time to compete on Kaggle, which is a much tougher challenge.\n\nDatatech Club is a student club founded by mutian at his university (ShanghaiTech University), and zhengru and yueting are both members. The club received computing resource support from the School of Information Science and Technology at ShanghaiTech University, and we are extremely grateful for their strong support.\n\n# TL;DR\n\nThe best submission (which we did not select) had a public score of 41.5 and a private score of 43.2. You can find the code and models in the end of this article.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F0f300b3db457c2297055bb2d4c6a0c61%2Fimage.png?generation=1774365796542368&alt=media)\n\nThe best submission is an ensemble of 11 `byt5-xl` models. Each model generates 10 candidate translations using beam search and sampling at different temperatures, and finally, the best translation is selected via a community-contributed MBR selector.\n\nThese 11 `byt5` models were trained on different combinations of datasets. 10 of them used the basic `Seq2SeqTrainer`, `bs=64`, trained for 3 epochs; 1 model used a custom loss weighted by data quality, `bs=48`, trained for 4 epochs.\n\nSingle-model inference for these models can easily reach the gold medal zone, and the best ckpt could have directly won 1st place as a single model. Below are some single-model inference submission scores (which do not fully comprise the final models used).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F05d334739d44918a559f37f596da9e15%2Fimage%201.png?generation=1774365851192641&alt=media)\n\nThe `byt5-xl` models were pre-compiled and quantized to \"int8-float32\"—this feature is implemented by the `ctranslate2` library. The script performed data-parallel inference on 2*T5 GPUs, and we almost used up the entire 9-hour time limit.\n\nMy thoughts on pre-processing and post-processing: Pre-process the training data as carefully as possible, and post-process the model output as little as possible. This is because we can always ensure the transliteration format is uniform, but relying on the Public LB for post-processing brings the risk of overfitting and shake-up.\n\nOne of our selected submissions added a cleaning step for repeated n-grams (n=2-7) on top of the best submission. Its public score was 41.6, and private score was 42.8. (I always believed post-processing only increases overfitting risk, so I didn't conduct any post-processing experiments. But a day before the deadline, I had no other ways to continue improving the score, so I ultimately chose this submission which gave me a higher public LB rank. The fact that it wasn't the best submission exactly proves my judgment was correct 😀)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F41735c0d61eebe5127f00599de21569b%2Fimage%202.png?generation=1774365898703066&alt=media)\n\nAnother selected submission did not use `ct2` or quantization. It was a conservative ensemble of the top 4 ckpts on the LB. Its public score was 41.2, private score was 42.4.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2Fd25f6a267bb163cbf24a88316024800e%2Fimage%203.png?generation=1774365918321903&alt=media)\n\n# Data is the Most Important\n\n## 1. OCR Data: Extracting Transliteration-Translation from Books\n\nWe extracted training data from PDFs provided by the hosts and collected by ourselves (we completely discarded `train.csv`).\n\nFor well-structured books (with fixed chapter formats and clear transliteration-translation alignments), such as AKT series, ICK_4, Larsen_2002, Michel_2020, POAT, we extracted data in batches using a structured approach.\n\n- Step 1: Using GLM-OCR for layout_analysis, supplemented by regex for chapter names, we extracted a `chapter_dict` (tablet/chapter name -> page range) for each book. Using the tablet rather than the page as the basic unit preserves information more completely.\n- Step 2: Utilizing the multimodal capabilities of Gemini-3.0-pro-preview, we sent the prompt, tablet name, and page images of the tablet to Gemini to extract sentence-level transliteration-translation pairs. For German/Turkish translations, we further asked Gemini to generate English translations.\n\nFor poorly-structured books, especially some short open-access texts scattered across the internet, we used a frontend created via vibe coding for extraction. The frontend allowed us to manually take screenshots, paste them into the window, select pre-set prompts, create queries, and call the API in batches.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F935ce1e86c5702c0d7f4fe10503d18ee%2Fimage%204.png?generation=1774365939653736&alt=media)\n\nOur prompts were written in Chinese, and here are some key points translated in English by LLM:\n\n1) Segmenting by Logical Meaning\n\n```text\n1. Strip away \"physical line breaks\" and pursue \"logical sentence meaning\"\n- No punctuation in ancient times: Akkadian transliterations on tablets strictly follow the physical lines of the clay tablet (words at the end of a line are broken directly, e.g., A-šùr-ta-ak-lá-/ku).\n- Use English translation as an anchor: English translations by modern scholars have already sorted out the logical syntax and added modern punctuation. Therefore, when segmenting sentences, you must never map them line-by-line. Instead, you must first understand the complete, long English sentence, and then look for the corresponding start and end points in the transliteration.\n```\n\n2) Correctly Understanding the Tablet\n\n```text\n2. Beware of cross-page and cross-section continuations caused by \"3D writing\"\n- A clay tablet is a three-dimensional writing medium. After filling the obverse, scribes would continue to the lower edge (l.e.), then flip to the reverse (rev.), and finally might write on the upper edge (u.e.) and left edge (l.e.).\n- Difficulty: A complete clause often spans across these areas. During alignment, if you find the second half of a sentence is missing, you must immediately look for the connecting point in the following lines.\n```\n\n3) Understanding Grammar\n\n```text\n3. Adapt to the \"spatiotemporal dislocation\" of grammar (SOV vs SVO)\n- Akkadian is SOV (Subject-Object-Verb): Verbs are usually placed at the very end of the sentence (e.g., I - silver - gave).\n- English is SVO (Subject-Verb-Object): When translating, the predicate verb is moved forward. This means that a verb at the beginning of an English sentence might only be found in the last few lines of that large transliteration paragraph on the left. Sentence segmentation requires a \"global view\" rather than fixating on individual words.\n```\n\n4) Locating Special Vocabulary\n\n```text\n4. Make good use of \"anchor\" words for precise positioning\n- In the dense transliteration, several types of words make excellent \"anchor points\":\n- Sumerian logograms (remember to write them in uppercase letters in your returned results): e.g., KÙ.BABBAR (silver), AN.NA (tin), TÚG (textile/garment), GÚ (talent), GÍN (shekel).\n- Numbers: Ancient merchants' accounts were extremely precise. Finding numbers (e.g., 1/2, 10, 15) is the fastest way to match bilingual texts.\n- Personal and Divine Names: e.g., A-šùr (Ashur), Šu-Nu-nu (Shu-Nunu). These proper nouns usually have highly consistent spellings in both the transliteration and translation.\n```\n\n## 2. LLM Labeled Data: Generating Translations for Untranslated Transliterations using LLMs\n\nThe official OARE website provides a great database containing a massive amount of untranslated data. Based on tablet names and multi-sequence matching, we excluded the data that had already been OCR'd. Combining dictionary lookups and similar transliteration-translation example queries, we called Gemini to generate translations.\n\n## 3. LLM Synthetic Data: Generating Completely Fictional Transliterations and Translations using LLMs\n \n1) Dictionary vocabulary synthetic data (synth1). This module first conducted a coverage gap analysis between the dictionary and the training set to filter out words truly missing from the training set. Then, it used the LLM to generate transliteration-translation pairs containing these words. The LLM input included 100 example sentences and 5 dictionary words, outputting 5-10 data pairs. Personal names and words lacking translations were excluded from the dictionary words.\n \n2) Free synthetic data (synth2). Based on 100 randomly selected sentences from the training set, it generated 3-7 sentence pairs.\n\n## Dataset usage for model training\n\nNext, we introduce the datasets used for training:\n\n**data1 (OCR data)**: Completed on March 8, containing 29,908 sentence-level data pairs (4,472 tablets). All sourced from LLM-extracted book data, without any manual cleaning.\n\n

\n \n  \n \n

\n\n**data2 (OCR data)**: Completed on March 14, containing 30,931 sentence-level data pairs (4,476 tablets). Based on data1, we trained models using 4-fold CV, and each fold's model inferred on its respective training fold. Since the training set should be well-fitted, if the inferred translation was far from the training translation, the sample was considered an error/hard sample. Filtering by a cross-fold average `geo_metric < 20`, we identified 740 tablets/chapters containing such sentence-level samples. We re-extracted these tablets using the LLM and supplemented with manual verification.\n\n

\n \n  \n \n

\n\n**data3 (OCR data)**: Completed on March 19, containing 34,146 sentence-level data pairs (4,837 tablets). We further supplemented some extraction results on top of data2. Additionally, transliteration-translation length mismatches and overly long transliterations can cause model misunderstandings. Filtering by a character length difference > 100 and transliteration character length > 500, we identified 131 tablets containing such samples. We re-extracted these tablets using the LLM and supplemented with manual verification.\n\n

\n \n  \n \n

\n\n**llmlabel (LLM Labeled Data)**: Completed on March 20, containing 21,759 sentence-level data pairs (3,946 tablets). Training with only this dataset yielded a public score of 35.0 and a private score of 37.1.\n\n

\n \n  \n \n

\n\n**synth1 (LLM Synthetic Data)**: Completed on March 8, containing 9,685 sentence-level data pairs.\n\n

\n \n  \n \n

\n\n**synth2 (LLM Synthetic Data)**: Completed on March 20, containing 4,972 sentence-level data pairs.\n\n

\n \n  \n \n

\n\nThe final ensemble of 11 models is basically a combination of the data above:\n\n| model name | data1 | data2 | data3 | llmlabel | synth1 | synth2 | all/split | note | Public Score | Private Score |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| A | ✅ | | | | | | all | | 39.6 | 40.4 |\n| B | ✅ | | | ✅ | | | all | | 40.1 | 41.2 |\n| C | | ✅ | | | | | all | | 39.7 | 40.9 |\n| D | | ✅ | | ✅ | | | all | | 40.3 | 41.3 |\n| E | | | ✅ | | | | all | | 40.1 | 40.6 |\n| F | | | ✅ | ✅ | | | all | | 40.1 | 41.6 |\n| G | | ✅ | | ✅ | ✅ | ✅ | all | | 40.1 | 41.3 |\n| H | | | ✅ | ✅ | ✅ | ✅ | all | | 40.0 | 40.6 |\n| I | | ✅ | | | | | split | 9:1 train valid split | 40.0 | 41.2 |\n| J | | | ✅ | | | | split | | 40.4 | 41.1 |\n| K | | ✅ | | | | | split | weighted loss by data quality | 39.1 | 40.0 |\n\n*Note: these socres are tested on ct2 int8_float32 model*\n\nFor comparision, some float16 transformers models results are as follows:\n\n| model name | data1 | data2 | data3 | llmlabel | synth1 | synth2 | all/split | note | Public Score | Private Score |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| byt5-xl-0317-ocr_akka2en/1308 | | ✅ | | | | | split | | 40.5 | 41.0 |\n| byt5-xl-0319-ocr_akka2en/1563 | | | ✅ | | | | all | | 40.1 | 40.9 |\n| byt5-xl-0321-data2_llmlabel/split/2238 | | ✅ | | ✅ | | | split | | 40.5 | 41.4 |\n| byt5-xl-0321-data2_llmlabel_synth2/all/2703 | | ✅ | | ✅ | | ✅ | all | | 40.4 | 42.0 |\n| byt5-xl-0322-data3_llmlabel/split/2364 | | | ✅ | ✅ | | | split | | 40.7 | 41.0 |\n\n# Regarding Model Training and Inference\n\n1. Beware of the model over-memorizing specific formats\n\n Use `eval_loss` for evaluation: Since there is inevitably a distribution and style inconsistency between the local validation set and the hidden test set, continuous improvements in `eval_bleu`/`chrf` after 3 epochs are untrustworthy. Therefore, we selected the 3-epoch checkpoints based on the validation set's `eval_loss` rather than `eval_bleu`/`chrf`.\n\n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F2af2c67ac78707e612c20d0cbc5928e2%2Fimage%2017.png?generation=1774366753099944&alt=media)\n\n2. Larger models are better\n\n About 10 days before the competition ended, we started switching from `byt5-base` to `byt5-large`, and then further to `byt5-xl`. The score increase on the public LB was significant.\n\n However, the prerequisite for this conclusion might be: Having sufficient and clean training data, so that larger models do not fall into overfitting.\n\n3. MBR\n\n Referring to [open-source MBR strategies](https://www.kaggle.com/code/ngyzly/lb-35-5-better-candidate-diversity-on-public-model), during the inference stage, beam search (`beam_size=8`) selected 4 candidates, and sampling at three temperatures `[0.60, 0.80, 1.05]` yielded 2 candidates each. After unified post-processing, we combined chrF++, BLEU, Jaccard, and length rewards to use weighted MBR to select the final translation that had the highest overall consistency with other candidates.\n\n4. Some necessary inference details\n - Sorted `test_df` in descending order by `char_length`.\n - More quantized models are better than fewer full-precision models.\n - Pre-compiling `ct2` models saved 30 mins; otherwise, an 11-model ensemble would have timed out.\n - Fully utilized 4 CPU cores, performing MBR in parallel, which also saved time.\n\n5. Things we tried but eventually abandoned (also listing related community writeups to compensate for our insufficient experiments)\n - Using decoder-only models, such as Qwen, translate-gemma. [This 25th solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/25th-post-training-qwen2-5-32b-and-72b-with-gemi) provided proof that fine-tuning Qwen can achieve great results.\n - Using CPT (Continual Pre-Training). I found the quality of the pre-training corpus to be quite poor and hard to clean. [This 10th solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/10th-place-solution-seq2seq-cpt-pseudo-label) and [23th solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/23rd-place-cpt-grpo-no-sft) illustrated feasible paths for CPT.\n - Multilingual training, distinguishing akka->en/de/tr using prefixes.\n - Using context. [This solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/684270) mentioned context could be an effective approach.\n - Test Time Augmentation (TTA), using `byt5` outputs on the test set as data for further fine-tuning. [This solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/llm-online-learning-is-all-you-need) pointed out that heterogeneous models should be used for augmentation.\n - In pre-processing: Standardizing Sumerian spelling, standardizing determinative formats, stripping supplementary text enclosed in parentheses in translations.\n - In post-processing: Standardizing names and places according to `onomasticon.csv`, cleaning repeated n-grams.\n\n### Computing Resources Used\n\nIn the early stages of training the `byt5-base` models, we used 1 x A100 for training.\n\nIn the later stages of training the `byt5-xl` models, we used 8 x H20 for training.", "votes": null }, { "id": "3427889", "postDate": "03/24/2026 16:42:59", "content": "

Really great write-up, and a very well-deserved win. The amount of effort behind rebuilding, cleaning, and improving the training data is super impressive, and it clearly paid off in the final result. It was very interesting to read.

\n

One thing I’m curious about: do you happen to know roughly how much the ensemble improved over the best single model, especially on the private LB?

", "rawMarkdown": "Really great write-up, and a very well-deserved win. The amount of effort behind rebuilding, cleaning, and improving the training data is super impressive, and it clearly paid off in the final result. It was very interesting to read.\n\nOne thing I’m curious about: do you happen to know roughly how much the ensemble improved over the best single model, especially on the private LB?", "votes": null }, { "id": "3427944", "postDate": "03/24/2026 17:48:03", "content": "

Great writeup and congratulations on the win! Could you expand a bit on this?

\n
\n

1 model used a custom loss weighted by data quality

\n
", "rawMarkdown": "Great writeup and congratulations on the win! Could you expand a bit on this?\n> 1 model used a custom loss weighted by data quality", "votes": null }, { "id": "3428006", "postDate": "03/24/2026 19:16:14", "content": "

Great work. Thanks for sharing your approach. Congratulation!

", "rawMarkdown": "Great work. Thanks for sharing your approach. Congratulation!", "votes": null }, { "id": "3428026", "postDate": "03/24/2026 20:17:59", "content": "

Can you explain what you meant here \"Utilizing the multimodal capabilities of Gemini-1.5-pro-preview, we sent the prompt, tablet name, and page images of the tablet to Gemini to extract sentence-level transliteration-translation pairs\". Confused why Gemini 1.5 pro preview was used. How would it be possible for Gemini-1.5-pro to extract transliteration-translation pairs? Why not use Gemini 3.1 Pro?

", "rawMarkdown": "Can you explain what you meant here \"Utilizing the multimodal capabilities of Gemini-1.5-pro-preview, we sent the prompt, tablet name, and page images of the tablet to Gemini to extract sentence-level transliteration-translation pairs\". Confused why Gemini 1.5 pro preview was used. How would it be possible for Gemini-1.5-pro to extract transliteration-translation pairs? Why not use Gemini 3.1 Pro?", "votes": null }, { "id": "3428041", "postDate": "03/24/2026 21:06:35", "content": "

Can you provide a link to your data as well? Very well done.

", "rawMarkdown": "Can you provide a link to your data as well? Very well done.", "votes": null }, { "id": "3428054", "postDate": "03/24/2026 21:32:16", "content": "

joining the request!

", "rawMarkdown": "joining the request!", "votes": null }, { "id": "3428067", "postDate": "03/24/2026 22:57:46", "content": "

Congrats and thanks for a great write up!

", "rawMarkdown": "Congrats and thanks for a great write up!", "votes": null }, { "id": "3428101", "postDate": "03/25/2026 00:17:06", "content": "

sorry, this is a typo, we actually used gemini-3.0-pro

", "rawMarkdown": "sorry, this is a typo, we actually used gemini-3.0-pro", "votes": null }, { "id": "3428183", "postDate": "03/25/2026 03:27:24", "content": "

Sure! I've updated each single model's public/private score in the table, both int8 model and folat16 model.

", "rawMarkdown": "Sure! I've updated each single model's public/private score in the table, both int8 model and folat16 model.", "votes": null }, { "id": "3428197", "postDate": "03/25/2026 04:17:27", "content": "

We overrode the compute_loss function in the Seq2SeqTrainer. Instead of using the default behavior, which averages the loss over the entire batch, we first calculated the average token loss for each individual sample, multiplied it by a predefined sample weight, and then computed the final weighted average across the batch.

\n

The weight allocation logic (roughly ordered by data quality from highest to lowest) is as follows:

\n\n

In single-model testing, we did not actually observe any direct performance improvement from this weighting strategy (the score of the corresponding single model was actually slightly lower). However, we ultimately included this weighted model into our 11-model ensemble, reasoning that this distinct loss bias would introduce valuable prediction diversity to the ensemble.

\n

Due to the extreme time constraints towards the end of the competition, we unfortunately ran out of time to conduct an ablation study on the specific performance gain of this weighted model within the ensemble, but this generally outlines our underlying rationale and approach.

", "rawMarkdown": "We overrode the compute_loss function in the Seq2SeqTrainer. Instead of using the default behavior, which averages the loss over the entire batch, we first calculated the average token loss for each individual sample, multiplied it by a predefined sample weight, and then computed the final weighted average across the batch.\n\nThe weight allocation logic (roughly ordered by data quality from highest to lowest) is as follows:\n\n- 1.3 ~ 1.4: High-quality OCR data (e.g., publications like the AKT series and ICK_4).\n- 1.0: Other English publication OCR data.\n- 0.8: Turkish/German publication OCR data.\n- 0.5 ~ 0.7: Semi-supervised data annotated by LLMs, as well as fully LLM-synthesized data.\n- 0.45: Low-quality OCR data that tended to cause performance degradation (score drops) in our historical experiments.\n\nIn single-model testing, we did not actually observe any direct performance improvement from this weighting strategy (the score of the corresponding single model was actually slightly lower). However, we ultimately included this weighted model into our 11-model ensemble, reasoning that this distinct loss bias would introduce valuable prediction diversity to the ensemble.\n\nDue to the extreme time constraints towards the end of the competition, we unfortunately ran out of time to conduct an ablation study on the specific performance gain of this weighted model within the ensemble, but this generally outlines our underlying rationale and approach.", "votes": null }, { "id": "3428264", "postDate": "03/25/2026 07:04:23", "content": "

Thank you so much for sharing this incredibly detailed writeup! Your solution taught me a lot about the importance of data quality over model tricks.\nA few questions if you don't mind:

\n", "rawMarkdown": "Thank you so much for sharing this incredibly detailed writeup! Your solution taught me a lot about the importance of data quality over model tricks.\nA few questions if you don't mind:\n- Checkpoint selection in \"all\" mode: When training on full data without a validation split, how did you select the final checkpoint? Did you calibrate the optimal number of epochs (3) from split-mode experiments first, then simply take the last checkpoint when training on all data?\n- Completely discarding train.csv: Did you truly use zero rows from the official train.csv in any of your 11 models, or was it partially mixed in? What gave you the confidence that your self-extracted data was strictly better?\n- Iterative data cleaning cost: The data1→data2→data3 pipeline (train model → find errors → re-extract with LLM → manual verification) is very impressive. Roughly how much time and API cost did each iteration take?\nCongratulations again on a well-deserved 1st place!", "votes": null }, { "id": "3428283", "postDate": "03/25/2026 07:49:15", "content": "

Thank you for your encouragement!

\n", "rawMarkdown": "Thank you for your encouragement!\n\n- We first performed a 9:1 train/validation split on all the datasets, and even applied 4-fold cross-validation (CV) on some of them. All of them reached their lowest eval loss at 3 epochs.\n\n- You can check the oare_ids in the training set—they are all included in the published_text.csv file. If you merge them, you will get the source of the training data and find that they all come from the AKT_ book series. Therefore, once we were confident that our extraction pipeline was more accurate, we simply discarded train.csv.\n\n- We did indeed invest a considerable amount of time. First of all, thanks to my teammates, we were able to quickly extract a large amount of data via the frontend. As for myself, I've been spending 6 hours or more on the competition almost every day since mid-February. The API cost actually wasn't that high because our pipeline optimized this by: 1) using regex to anchor the text scope, and 2) using manual screenshots to define the extraction areas. Both of these methods significantly reduced the number of tokens the LLM needed to read and process. I estimate it only cost us around $100–$200, or possibly even less.", "votes": null }, { "id": "3428336", "postDate": "03/25/2026 09:46:47", "content": "

Excellent work Your explanation was clear, insightful, and exceptionally well delivered. Congratu\nlations.

", "rawMarkdown": "Excellent work Your explanation was clear, insightful, and exceptionally well delivered. Congratu\nlations.", "votes": null }, { "id": "3428338", "postDate": "03/25/2026 09:55:16", "content": "

Congratulations for getting on top and thanks for sharing detailed insights

", "rawMarkdown": "Congratulations for getting on top and thanks for sharing detailed insights", "votes": null }, { "id": "3428366", "postDate": "03/25/2026 11:15:01", "content": "

Amazing work. The focus on data quality over complex modeling really stands out \ngreat reminder that better data often beats everything else.

", "rawMarkdown": "Amazing work. The focus on data quality over complex modeling really stands out \ngreat reminder that better data often beats everything else.", "votes": null }, { "id": "3428367", "postDate": "03/25/2026 11:15:50", "content": "

Incredible solution. The way you improved data step by step is really insightful \nthis shows how much impact clean data can have on performance.

", "rawMarkdown": "Incredible solution. The way you improved data step by step is really insightful \nthis shows how much impact clean data can have on performance.", "votes": null }, { "id": "3428626", "postDate": "03/25/2026 18:25:56", "content": "

Congratulations! Well done!

\n

I think it may have already been mentioned, but if possible, please share your training data! Like many of these write-ups have been saying, data-mining seems to be the primary factor that separated the high scorers from the low scorers. I was a bit too late to the competition to explore data-scraping in depth, but I do have some ideas about improvements regarding the model itself that I think could benefit the quality of the translations. I really love the mission of this competition, so I'd love to continue exploring ways to improve these translations! Thank you for your hard work!

", "rawMarkdown": "Congratulations! Well done! \n\nI think it may have already been mentioned, but if possible, please share your training data! Like many of these write-ups have been saying, data-mining seems to be the primary factor that separated the high scorers from the low scorers. I was a bit too late to the competition to explore data-scraping in depth, but I do have some ideas about improvements regarding the model itself that I think could benefit the quality of the translations. I really love the mission of this competition, so I'd love to continue exploring ways to improve these translations! Thank you for your hard work!", "votes": null }, { "id": "3428854", "postDate": "03/26/2026 01:53:43", "content": "

Ablation study done. Without the weighted model, the score is:\nprivate score: 43.0\npublic score: 41.4

\n

and our best model is (the best one, not selected)\nprivate score: 43.2\npublic score: 41.6

\n

So it did brought some diversity to the ensemble.

", "rawMarkdown": "Ablation study done. Without the weighted model, the score is:\nprivate score: 43.0\npublic score: 41.4\n\nand our best model is (the best one, not selected)\nprivate score: 43.2\npublic score: 41.6\n\nSo it did brought some diversity to the ensemble.", "votes": null }, { "id": "3428933", "postDate": "03/26/2026 05:13:15", "content": "

Congo & well done brother.

", "rawMarkdown": "Congo & well done brother.", "votes": null }, { "id": "3428960", "postDate": "03/26/2026 06:01:52", "content": "

Great 👍 congratulations 👏

", "rawMarkdown": "Great 👍 congratulations 👏", "votes": null }, { "id": "3428986", "postDate": "03/26/2026 06:59:06", "content": "

Sorry for the late response! To ensure full compliance, we've sent the full data to the host first. Following their suggestion, please reach out to me via Linkedin or email (you can find in my profile) to obtain the anonymous data for your research use.

\n

Also, stay tuned for updates to the OARE database! I've been informed that they're planning to expand it—including adding translations—as an open-source project. If that sounds like something you'd like to be part of, feel free to reach out to the host to get involved. 👏

", "rawMarkdown": "Sorry for the late response! To ensure full compliance, we've sent the full data to the host first. Following their suggestion, please reach out to me via Linkedin or email (you can find in my profile) to obtain the anonymous data for your research use.\n\nAlso, stay tuned for updates to the OARE database! I've been informed that they're planning to expand it—including adding translations—as an open-source project. If that sounds like something you'd like to be part of, feel free to reach out to the host to get involved. 👏", "votes": null }, { "id": "3429203", "postDate": "03/26/2026 13:30:10", "content": "

Congratulations excellent work!! \nThank you for sharing this amazing writeup!\nWell deserved First place

", "rawMarkdown": "Congratulations excellent work!! \nThank you for sharing this amazing writeup!\nWell deserved First place", "votes": null }, { "id": "3429326", "postDate": "03/26/2026 16:11:35", "content": "

No worries! I'm sure you are very busy! Thank you - I will!

", "rawMarkdown": "No worries! I'm sure you are very busy! Thank you - I will!", "votes": null }, { "id": "3429628", "postDate": "03/27/2026 02:17:41", "content": "

It has been sent! Hope it can be helpful, good luck!

", "rawMarkdown": "It has been sent! Hope it can be helpful, good luck!", "votes": null }, { "id": "3431823", "postDate": "03/30/2026 18:03:56", "content": "

Great example of how data quality dominates model gains - very insightful pipeline, especially the iterative OCR refinement and filtering.

", "rawMarkdown": "Great example of how data quality dominates model gains - very insightful pipeline, especially the iterative OCR refinement and filtering.", "votes": null }, { "id": "3437765", "postDate": "04/08/2026 03:54:37", "content": "

I have a question. You mentioned using 5-fold cross-validation. Why were there 11 models during the final inference?

", "rawMarkdown": "I have a question. You mentioned using 5-fold cross-validation. Why were there 11 models during the final inference?", "votes": null }, { "id": "3456014", "postDate": "05/11/2026 02:04:08", "content": "

Congrats on the win! This is a very helpful write-up. I was especially impressed by the variations of the datasets you experimented with. Using the LLM-produced synthetic data approach is really cool, too. I would like to do further research, trying to improve Akkadian translation for my DL class, so I was wondering if you guys could let me use your curated datasets. I would really appreciate it!

", "rawMarkdown": "Congrats on the win! This is a very helpful write-up. I was especially impressed by the variations of the datasets you experimented with. Using the LLM-produced synthetic data approach is really cool, too. I would like to do further research, trying to improve Akkadian translation for my DL class, so I was wondering if you guys could let me use your curated datasets. I would really appreciate it!", "votes": null }, { "id": "3457143", "postDate": "05/13/2026 07:44:54", "content": "

You can contact us by sending an email to Guoqin, you can find his email in his kaggle profile.

", "rawMarkdown": "You can contact us by sending an email to Guoqin, you can find his email in his kaggle profile.", "votes": null } ], "comments": [ { "id": 3427889, "author_name": "tonylica", "author_url": "", "post_date": "03/24/2026 16:42:59", "content": "

Really great write-up, and a very well-deserved win. The amount of effort behind rebuilding, cleaning, and improving the training data is super impressive, and it clearly paid off in the final result. It was very interesting to read.

\n

One thing I’m curious about: do you happen to know roughly how much the ensemble improved over the best single model, especially on the private LB?

", "votes": null, "replies": [ { "id": 3428183, "author_name": "guoqingu", "author_url": "", "post_date": "03/25/2026 03:27:24", "content": "

Sure! I've updated each single model's public/private score in the table, both int8 model and folat16 model.

", "votes": null, "replies": [] } ] }, { "id": 3427944, "author_name": "alejopaullier", "author_url": "", "post_date": "03/24/2026 17:48:03", "content": "

Great writeup and congratulations on the win! Could you expand a bit on this?

\n
\n

1 model used a custom loss weighted by data quality

\n
", "votes": null, "replies": [ { "id": 3428197, "author_name": "hongori", "author_url": "", "post_date": "03/25/2026 04:17:27", "content": "

We overrode the compute_loss function in the Seq2SeqTrainer. Instead of using the default behavior, which averages the loss over the entire batch, we first calculated the average token loss for each individual sample, multiplied it by a predefined sample weight, and then computed the final weighted average across the batch.

\n

The weight allocation logic (roughly ordered by data quality from highest to lowest) is as follows:

\n\n

In single-model testing, we did not actually observe any direct performance improvement from this weighting strategy (the score of the corresponding single model was actually slightly lower). However, we ultimately included this weighted model into our 11-model ensemble, reasoning that this distinct loss bias would introduce valuable prediction diversity to the ensemble.

\n

Due to the extreme time constraints towards the end of the competition, we unfortunately ran out of time to conduct an ablation study on the specific performance gain of this weighted model within the ensemble, but this generally outlines our underlying rationale and approach.

", "votes": null, "replies": [ { "id": 3428854, "author_name": "hongori", "author_url": "", "post_date": "03/26/2026 01:53:43", "content": "

Ablation study done. Without the weighted model, the score is:\nprivate score: 43.0\npublic score: 41.4

\n

and our best model is (the best one, not selected)\nprivate score: 43.2\npublic score: 41.6

\n

So it did brought some diversity to the ensemble.

", "votes": null, "replies": [] } ] } ] }, { "id": 3428006, "author_name": "satyaprakash889", "author_url": "", "post_date": "03/24/2026 19:16:14", "content": "

Great work. Thanks for sharing your approach. Congratulation!

", "votes": null, "replies": [] }, { "id": 3428026, "author_name": "andrevma18", "author_url": "", "post_date": "03/24/2026 20:17:59", "content": "

Can you explain what you meant here \"Utilizing the multimodal capabilities of Gemini-1.5-pro-preview, we sent the prompt, tablet name, and page images of the tablet to Gemini to extract sentence-level transliteration-translation pairs\". Confused why Gemini 1.5 pro preview was used. How would it be possible for Gemini-1.5-pro to extract transliteration-translation pairs? Why not use Gemini 3.1 Pro?

", "votes": null, "replies": [ { "id": 3428101, "author_name": "guoqingu", "author_url": "", "post_date": "03/25/2026 00:17:06", "content": "

sorry, this is a typo, we actually used gemini-3.0-pro

", "votes": null, "replies": [] } ] }, { "id": 3428041, "author_name": "leedrake", "author_url": "", "post_date": "03/24/2026 21:06:35", "content": "

Can you provide a link to your data as well? Very well done.

", "votes": null, "replies": [ { "id": 3428054, "author_name": "shaharspencer", "author_url": "", "post_date": "03/24/2026 21:32:16", "content": "

joining the request!

", "votes": null, "replies": [] } ] }, { "id": 3428067, "author_name": "tennogh", "author_url": "", "post_date": "03/24/2026 22:57:46", "content": "

Congrats and thanks for a great write up!

", "votes": null, "replies": [] }, { "id": 3428264, "author_name": "kingkong153", "author_url": "", "post_date": "03/25/2026 07:04:23", "content": "

Thank you so much for sharing this incredibly detailed writeup! Your solution taught me a lot about the importance of data quality over model tricks.\nA few questions if you don't mind:

\n", "votes": null, "replies": [ { "id": 3428283, "author_name": "guoqingu", "author_url": "", "post_date": "03/25/2026 07:49:15", "content": "

Thank you for your encouragement!

\n", "votes": null, "replies": [] } ] }, { "id": 3428336, "author_name": "ismail89", "author_url": "", "post_date": "03/25/2026 09:46:47", "content": "

Excellent work Your explanation was clear, insightful, and exceptionally well delivered. Congratu\nlations.

", "votes": null, "replies": [] }, { "id": 3428338, "author_name": "haseebkn", "author_url": "", "post_date": "03/25/2026 09:55:16", "content": "

Congratulations for getting on top and thanks for sharing detailed insights

", "votes": null, "replies": [] }, { "id": 3428366, "author_name": "aman990", "author_url": "", "post_date": "03/25/2026 11:15:01", "content": "

Amazing work. The focus on data quality over complex modeling really stands out \ngreat reminder that better data often beats everything else.

", "votes": null, "replies": [] }, { "id": 3428367, "author_name": "aman990", "author_url": "", "post_date": "03/25/2026 11:15:50", "content": "

Incredible solution. The way you improved data step by step is really insightful \nthis shows how much impact clean data can have on performance.

", "votes": null, "replies": [] }, { "id": 3428626, "author_name": "eliclowry", "author_url": "", "post_date": "03/25/2026 18:25:56", "content": "

Congratulations! Well done!

\n

I think it may have already been mentioned, but if possible, please share your training data! Like many of these write-ups have been saying, data-mining seems to be the primary factor that separated the high scorers from the low scorers. I was a bit too late to the competition to explore data-scraping in depth, but I do have some ideas about improvements regarding the model itself that I think could benefit the quality of the translations. I really love the mission of this competition, so I'd love to continue exploring ways to improve these translations! Thank you for your hard work!

", "votes": null, "replies": [ { "id": 3428986, "author_name": "guoqingu", "author_url": "", "post_date": "03/26/2026 06:59:06", "content": "

Sorry for the late response! To ensure full compliance, we've sent the full data to the host first. Following their suggestion, please reach out to me via Linkedin or email (you can find in my profile) to obtain the anonymous data for your research use.

\n

Also, stay tuned for updates to the OARE database! I've been informed that they're planning to expand it—including adding translations—as an open-source project. If that sounds like something you'd like to be part of, feel free to reach out to the host to get involved. 👏

", "votes": null, "replies": [ { "id": 3429326, "author_name": "eliclowry", "author_url": "", "post_date": "03/26/2026 16:11:35", "content": "

No worries! I'm sure you are very busy! Thank you - I will!

", "votes": null, "replies": [ { "id": 3429628, "author_name": "guoqingu", "author_url": "", "post_date": "03/27/2026 02:17:41", "content": "

It has been sent! Hope it can be helpful, good luck!

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3428933, "author_name": "koushikrudra", "author_url": "", "post_date": "03/26/2026 05:13:15", "content": "

Congo & well done brother.

", "votes": null, "replies": [] }, { "id": 3428960, "author_name": "junaid512", "author_url": "", "post_date": "03/26/2026 06:01:52", "content": "

Great 👍 congratulations 👏

", "votes": null, "replies": [] }, { "id": 3429203, "author_name": "sakotas", "author_url": "", "post_date": "03/26/2026 13:30:10", "content": "

Congratulations excellent work!! \nThank you for sharing this amazing writeup!\nWell deserved First place

", "votes": null, "replies": [] }, { "id": 3431823, "author_name": "durgakumari12", "author_url": "", "post_date": "03/30/2026 18:03:56", "content": "

Great example of how data quality dominates model gains - very insightful pipeline, especially the iterative OCR refinement and filtering.

", "votes": null, "replies": [] }, { "id": 3437765, "author_name": "bianhongfei", "author_url": "", "post_date": "04/08/2026 03:54:37", "content": "

I have a question. You mentioned using 5-fold cross-validation. Why were there 11 models during the final inference?

", "votes": null, "replies": [] }, { "id": 3456014, "author_name": "zeynepesennur", "author_url": "", "post_date": "05/11/2026 02:04:08", "content": "

Congrats on the win! This is a very helpful write-up. I was especially impressed by the variations of the datasets you experimented with. Using the LLM-produced synthetic data approach is really cool, too. I would like to do further research, trying to improve Akkadian translation for my DL class, so I was wondering if you guys could let me use your curated datasets. I would really appreciate it!

", "votes": null, "replies": [ { "id": 3457143, "author_name": "hongori", "author_url": "", "post_date": "05/13/2026 07:44:54", "content": "

You can contact us by sending an email to Guoqin, you can find his email in his kaggle profile.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3427819": "# Acknowledgments\n\nThanks to the hosts and Kaggle staffs, especially Adam Anderson. In the early stages of the competition, there were significant data issues that caused a lot of confusion, but Anderson worked diligently to resolve the most critical problems, which greatly reduced the difficulty of optimizing the models.\n\nThanks to my teammates mutian, zhengru, and yueting. Mutian and I have participated in many competitions on other platforms, and finally, in the year before starting our master's degrees, we had the time to compete on Kaggle, which is a much tougher challenge.\n\nDatatech Club is a student club founded by mutian at his university (ShanghaiTech University), and zhengru and yueting are both members. The club received computing resource support from the School of Information Science and Technology at ShanghaiTech University, and we are extremely grateful for their strong support.\n\n# TL;DR\n\nThe best submission (which we did not select) had a public score of 41.5 and a private score of 43.2. You can find the code and models in the end of this article.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F0f300b3db457c2297055bb2d4c6a0c61%2Fimage.png?generation=1774365796542368&alt=media)\n\nThe best submission is an ensemble of 11 `byt5-xl` models. Each model generates 10 candidate translations using beam search and sampling at different temperatures, and finally, the best translation is selected via a community-contributed MBR selector.\n\nThese 11 `byt5` models were trained on different combinations of datasets. 10 of them used the basic `Seq2SeqTrainer`, `bs=64`, trained for 3 epochs; 1 model used a custom loss weighted by data quality, `bs=48`, trained for 4 epochs.\n\nSingle-model inference for these models can easily reach the gold medal zone, and the best ckpt could have directly won 1st place as a single model. Below are some single-model inference submission scores (which do not fully comprise the final models used).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F05d334739d44918a559f37f596da9e15%2Fimage%201.png?generation=1774365851192641&alt=media)\n\nThe `byt5-xl` models were pre-compiled and quantized to \"int8-float32\"—this feature is implemented by the `ctranslate2` library. The script performed data-parallel inference on 2*T5 GPUs, and we almost used up the entire 9-hour time limit.\n\nMy thoughts on pre-processing and post-processing: Pre-process the training data as carefully as possible, and post-process the model output as little as possible. This is because we can always ensure the transliteration format is uniform, but relying on the Public LB for post-processing brings the risk of overfitting and shake-up.\n\nOne of our selected submissions added a cleaning step for repeated n-grams (n=2-7) on top of the best submission. Its public score was 41.6, and private score was 42.8. (I always believed post-processing only increases overfitting risk, so I didn't conduct any post-processing experiments. But a day before the deadline, I had no other ways to continue improving the score, so I ultimately chose this submission which gave me a higher public LB rank. The fact that it wasn't the best submission exactly proves my judgment was correct 😀)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F41735c0d61eebe5127f00599de21569b%2Fimage%202.png?generation=1774365898703066&alt=media)\n\nAnother selected submission did not use `ct2` or quantization. It was a conservative ensemble of the top 4 ckpts on the LB. Its public score was 41.2, private score was 42.4.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2Fd25f6a267bb163cbf24a88316024800e%2Fimage%203.png?generation=1774365918321903&alt=media)\n\n# Data is the Most Important\n\n## 1. OCR Data: Extracting Transliteration-Translation from Books\n\nWe extracted training data from PDFs provided by the hosts and collected by ourselves (we completely discarded `train.csv`).\n\nFor well-structured books (with fixed chapter formats and clear transliteration-translation alignments), such as AKT series, ICK_4, Larsen_2002, Michel_2020, POAT, we extracted data in batches using a structured approach.\n\n- Step 1: Using GLM-OCR for layout_analysis, supplemented by regex for chapter names, we extracted a `chapter_dict` (tablet/chapter name -> page range) for each book. Using the tablet rather than the page as the basic unit preserves information more completely.\n- Step 2: Utilizing the multimodal capabilities of Gemini-3.0-pro-preview, we sent the prompt, tablet name, and page images of the tablet to Gemini to extract sentence-level transliteration-translation pairs. For German/Turkish translations, we further asked Gemini to generate English translations.\n\nFor poorly-structured books, especially some short open-access texts scattered across the internet, we used a frontend created via vibe coding for extraction. The frontend allowed us to manually take screenshots, paste them into the window, select pre-set prompts, create queries, and call the API in batches.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F935ce1e86c5702c0d7f4fe10503d18ee%2Fimage%204.png?generation=1774365939653736&alt=media)\n\nOur prompts were written in Chinese, and here are some key points translated in English by LLM:\n\n1) Segmenting by Logical Meaning\n\n```text\n1. Strip away \"physical line breaks\" and pursue \"logical sentence meaning\"\n- No punctuation in ancient times: Akkadian transliterations on tablets strictly follow the physical lines of the clay tablet (words at the end of a line are broken directly, e.g., A-šùr-ta-ak-lá-/ku).\n- Use English translation as an anchor: English translations by modern scholars have already sorted out the logical syntax and added modern punctuation. Therefore, when segmenting sentences, you must never map them line-by-line. Instead, you must first understand the complete, long English sentence, and then look for the corresponding start and end points in the transliteration.\n```\n\n2) Correctly Understanding the Tablet\n\n```text\n2. Beware of cross-page and cross-section continuations caused by \"3D writing\"\n- A clay tablet is a three-dimensional writing medium. After filling the obverse, scribes would continue to the lower edge (l.e.), then flip to the reverse (rev.), and finally might write on the upper edge (u.e.) and left edge (l.e.).\n- Difficulty: A complete clause often spans across these areas. During alignment, if you find the second half of a sentence is missing, you must immediately look for the connecting point in the following lines.\n```\n\n3) Understanding Grammar\n\n```text\n3. Adapt to the \"spatiotemporal dislocation\" of grammar (SOV vs SVO)\n- Akkadian is SOV (Subject-Object-Verb): Verbs are usually placed at the very end of the sentence (e.g., I - silver - gave).\n- English is SVO (Subject-Verb-Object): When translating, the predicate verb is moved forward. This means that a verb at the beginning of an English sentence might only be found in the last few lines of that large transliteration paragraph on the left. Sentence segmentation requires a \"global view\" rather than fixating on individual words.\n```\n\n4) Locating Special Vocabulary\n\n```text\n4. Make good use of \"anchor\" words for precise positioning\n- In the dense transliteration, several types of words make excellent \"anchor points\":\n- Sumerian logograms (remember to write them in uppercase letters in your returned results): e.g., KÙ.BABBAR (silver), AN.NA (tin), TÚG (textile/garment), GÚ (talent), GÍN (shekel).\n- Numbers: Ancient merchants' accounts were extremely precise. Finding numbers (e.g., 1/2, 10, 15) is the fastest way to match bilingual texts.\n- Personal and Divine Names: e.g., A-šùr (Ashur), Šu-Nu-nu (Shu-Nunu). These proper nouns usually have highly consistent spellings in both the transliteration and translation.\n```\n\n## 2. LLM Labeled Data: Generating Translations for Untranslated Transliterations using LLMs\n\nThe official OARE website provides a great database containing a massive amount of untranslated data. Based on tablet names and multi-sequence matching, we excluded the data that had already been OCR'd. Combining dictionary lookups and similar transliteration-translation example queries, we called Gemini to generate translations.\n\n## 3. LLM Synthetic Data: Generating Completely Fictional Transliterations and Translations using LLMs\n \n1) Dictionary vocabulary synthetic data (synth1). This module first conducted a coverage gap analysis between the dictionary and the training set to filter out words truly missing from the training set. Then, it used the LLM to generate transliteration-translation pairs containing these words. The LLM input included 100 example sentences and 5 dictionary words, outputting 5-10 data pairs. Personal names and words lacking translations were excluded from the dictionary words.\n \n2) Free synthetic data (synth2). Based on 100 randomly selected sentences from the training set, it generated 3-7 sentence pairs.\n\n## Dataset usage for model training\n\nNext, we introduce the datasets used for training:\n\n**data1 (OCR data)**: Completed on March 8, containing 29,908 sentence-level data pairs (4,472 tablets). All sourced from LLM-extracted book data, without any manual cleaning.\n\n

\n \n  \n \n

\n\n**data2 (OCR data)**: Completed on March 14, containing 30,931 sentence-level data pairs (4,476 tablets). Based on data1, we trained models using 4-fold CV, and each fold's model inferred on its respective training fold. Since the training set should be well-fitted, if the inferred translation was far from the training translation, the sample was considered an error/hard sample. Filtering by a cross-fold average `geo_metric < 20`, we identified 740 tablets/chapters containing such sentence-level samples. We re-extracted these tablets using the LLM and supplemented with manual verification.\n\n

\n \n  \n \n

\n\n**data3 (OCR data)**: Completed on March 19, containing 34,146 sentence-level data pairs (4,837 tablets). We further supplemented some extraction results on top of data2. Additionally, transliteration-translation length mismatches and overly long transliterations can cause model misunderstandings. Filtering by a character length difference > 100 and transliteration character length > 500, we identified 131 tablets containing such samples. We re-extracted these tablets using the LLM and supplemented with manual verification.\n\n

\n \n  \n \n

\n\n**llmlabel (LLM Labeled Data)**: Completed on March 20, containing 21,759 sentence-level data pairs (3,946 tablets). Training with only this dataset yielded a public score of 35.0 and a private score of 37.1.\n\n

\n \n  \n \n

\n\n**synth1 (LLM Synthetic Data)**: Completed on March 8, containing 9,685 sentence-level data pairs.\n\n

\n \n  \n \n

\n\n**synth2 (LLM Synthetic Data)**: Completed on March 20, containing 4,972 sentence-level data pairs.\n\n

\n \n  \n \n

\n\nThe final ensemble of 11 models is basically a combination of the data above:\n\n| model name | data1 | data2 | data3 | llmlabel | synth1 | synth2 | all/split | note | Public Score | Private Score |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| A | ✅ | | | | | | all | | 39.6 | 40.4 |\n| B | ✅ | | | ✅ | | | all | | 40.1 | 41.2 |\n| C | | ✅ | | | | | all | | 39.7 | 40.9 |\n| D | | ✅ | | ✅ | | | all | | 40.3 | 41.3 |\n| E | | | ✅ | | | | all | | 40.1 | 40.6 |\n| F | | | ✅ | ✅ | | | all | | 40.1 | 41.6 |\n| G | | ✅ | | ✅ | ✅ | ✅ | all | | 40.1 | 41.3 |\n| H | | | ✅ | ✅ | ✅ | ✅ | all | | 40.0 | 40.6 |\n| I | | ✅ | | | | | split | 9:1 train valid split | 40.0 | 41.2 |\n| J | | | ✅ | | | | split | | 40.4 | 41.1 |\n| K | | ✅ | | | | | split | weighted loss by data quality | 39.1 | 40.0 |\n\n*Note: these socres are tested on ct2 int8_float32 model*\n\nFor comparision, some float16 transformers models results are as follows:\n\n| model name | data1 | data2 | data3 | llmlabel | synth1 | synth2 | all/split | note | Public Score | Private Score |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| byt5-xl-0317-ocr_akka2en/1308 | | ✅ | | | | | split | | 40.5 | 41.0 |\n| byt5-xl-0319-ocr_akka2en/1563 | | | ✅ | | | | all | | 40.1 | 40.9 |\n| byt5-xl-0321-data2_llmlabel/split/2238 | | ✅ | | ✅ | | | split | | 40.5 | 41.4 |\n| byt5-xl-0321-data2_llmlabel_synth2/all/2703 | | ✅ | | ✅ | | ✅ | all | | 40.4 | 42.0 |\n| byt5-xl-0322-data3_llmlabel/split/2364 | | | ✅ | ✅ | | | split | | 40.7 | 41.0 |\n\n# Regarding Model Training and Inference\n\n1. Beware of the model over-memorizing specific formats\n\n Use `eval_loss` for evaluation: Since there is inevitably a distribution and style inconsistency between the local validation set and the hidden test set, continuous improvements in `eval_bleu`/`chrf` after 3 epochs are untrustworthy. Therefore, we selected the 3-epoch checkpoints based on the validation set's `eval_loss` rather than `eval_bleu`/`chrf`.\n\n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12523104%2F2af2c67ac78707e612c20d0cbc5928e2%2Fimage%2017.png?generation=1774366753099944&alt=media)\n\n2. Larger models are better\n\n About 10 days before the competition ended, we started switching from `byt5-base` to `byt5-large`, and then further to `byt5-xl`. The score increase on the public LB was significant.\n\n However, the prerequisite for this conclusion might be: Having sufficient and clean training data, so that larger models do not fall into overfitting.\n\n3. MBR\n\n Referring to [open-source MBR strategies](https://www.kaggle.com/code/ngyzly/lb-35-5-better-candidate-diversity-on-public-model), during the inference stage, beam search (`beam_size=8`) selected 4 candidates, and sampling at three temperatures `[0.60, 0.80, 1.05]` yielded 2 candidates each. After unified post-processing, we combined chrF++, BLEU, Jaccard, and length rewards to use weighted MBR to select the final translation that had the highest overall consistency with other candidates.\n\n4. Some necessary inference details\n - Sorted `test_df` in descending order by `char_length`.\n - More quantized models are better than fewer full-precision models.\n - Pre-compiling `ct2` models saved 30 mins; otherwise, an 11-model ensemble would have timed out.\n - Fully utilized 4 CPU cores, performing MBR in parallel, which also saved time.\n\n5. Things we tried but eventually abandoned (also listing related community writeups to compensate for our insufficient experiments)\n - Using decoder-only models, such as Qwen, translate-gemma. [This 25th solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/25th-post-training-qwen2-5-32b-and-72b-with-gemi) provided proof that fine-tuning Qwen can achieve great results.\n - Using CPT (Continual Pre-Training). I found the quality of the pre-training corpus to be quite poor and hard to clean. [This 10th solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/10th-place-solution-seq2seq-cpt-pseudo-label) and [23th solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/23rd-place-cpt-grpo-no-sft) illustrated feasible paths for CPT.\n - Multilingual training, distinguishing akka->en/de/tr using prefixes.\n - Using context. [This solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/684270) mentioned context could be an effective approach.\n - Test Time Augmentation (TTA), using `byt5` outputs on the test set as data for further fine-tuning. [This solution](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/writeups/llm-online-learning-is-all-you-need) pointed out that heterogeneous models should be used for augmentation.\n - In pre-processing: Standardizing Sumerian spelling, standardizing determinative formats, stripping supplementary text enclosed in parentheses in translations.\n - In post-processing: Standardizing names and places according to `onomasticon.csv`, cleaning repeated n-grams.\n\n### Computing Resources Used\n\nIn the early stages of training the `byt5-base` models, we used 1 x A100 for training.\n\nIn the later stages of training the `byt5-xl` models, we used 8 x H20 for training.", "3427889": "Really great write-up, and a very well-deserved win. The amount of effort behind rebuilding, cleaning, and improving the training data is super impressive, and it clearly paid off in the final result. It was very interesting to read.\n\nOne thing I’m curious about: do you happen to know roughly how much the ensemble improved over the best single model, especially on the private LB?", "3427944": "Great writeup and congratulations on the win! Could you expand a bit on this?\n> 1 model used a custom loss weighted by data quality", "3428006": "Great work. Thanks for sharing your approach. Congratulation!", "3428026": "Can you explain what you meant here \"Utilizing the multimodal capabilities of Gemini-1.5-pro-preview, we sent the prompt, tablet name, and page images of the tablet to Gemini to extract sentence-level transliteration-translation pairs\". Confused why Gemini 1.5 pro preview was used. How would it be possible for Gemini-1.5-pro to extract transliteration-translation pairs? Why not use Gemini 3.1 Pro?", "3428041": "Can you provide a link to your data as well? Very well done.", "3428054": "joining the request!", "3428067": "Congrats and thanks for a great write up!", "3428101": "sorry, this is a typo, we actually used gemini-3.0-pro", "3428183": "Sure! I've updated each single model's public/private score in the table, both int8 model and folat16 model.", "3428197": "We overrode the compute_loss function in the Seq2SeqTrainer. Instead of using the default behavior, which averages the loss over the entire batch, we first calculated the average token loss for each individual sample, multiplied it by a predefined sample weight, and then computed the final weighted average across the batch.\n\nThe weight allocation logic (roughly ordered by data quality from highest to lowest) is as follows:\n\n- 1.3 ~ 1.4: High-quality OCR data (e.g., publications like the AKT series and ICK_4).\n- 1.0: Other English publication OCR data.\n- 0.8: Turkish/German publication OCR data.\n- 0.5 ~ 0.7: Semi-supervised data annotated by LLMs, as well as fully LLM-synthesized data.\n- 0.45: Low-quality OCR data that tended to cause performance degradation (score drops) in our historical experiments.\n\nIn single-model testing, we did not actually observe any direct performance improvement from this weighting strategy (the score of the corresponding single model was actually slightly lower). However, we ultimately included this weighted model into our 11-model ensemble, reasoning that this distinct loss bias would introduce valuable prediction diversity to the ensemble.\n\nDue to the extreme time constraints towards the end of the competition, we unfortunately ran out of time to conduct an ablation study on the specific performance gain of this weighted model within the ensemble, but this generally outlines our underlying rationale and approach.", "3428264": "Thank you so much for sharing this incredibly detailed writeup! Your solution taught me a lot about the importance of data quality over model tricks.\nA few questions if you don't mind:\n- Checkpoint selection in \"all\" mode: When training on full data without a validation split, how did you select the final checkpoint? Did you calibrate the optimal number of epochs (3) from split-mode experiments first, then simply take the last checkpoint when training on all data?\n- Completely discarding train.csv: Did you truly use zero rows from the official train.csv in any of your 11 models, or was it partially mixed in? What gave you the confidence that your self-extracted data was strictly better?\n- Iterative data cleaning cost: The data1→data2→data3 pipeline (train model → find errors → re-extract with LLM → manual verification) is very impressive. Roughly how much time and API cost did each iteration take?\nCongratulations again on a well-deserved 1st place!", "3428283": "Thank you for your encouragement!\n\n- We first performed a 9:1 train/validation split on all the datasets, and even applied 4-fold cross-validation (CV) on some of them. All of them reached their lowest eval loss at 3 epochs.\n\n- You can check the oare_ids in the training set—they are all included in the published_text.csv file. If you merge them, you will get the source of the training data and find that they all come from the AKT_ book series. Therefore, once we were confident that our extraction pipeline was more accurate, we simply discarded train.csv.\n\n- We did indeed invest a considerable amount of time. First of all, thanks to my teammates, we were able to quickly extract a large amount of data via the frontend. As for myself, I've been spending 6 hours or more on the competition almost every day since mid-February. The API cost actually wasn't that high because our pipeline optimized this by: 1) using regex to anchor the text scope, and 2) using manual screenshots to define the extraction areas. Both of these methods significantly reduced the number of tokens the LLM needed to read and process. I estimate it only cost us around $100–$200, or possibly even less.", "3428336": "Excellent work Your explanation was clear, insightful, and exceptionally well delivered. Congratu\nlations.", "3428338": "Congratulations for getting on top and thanks for sharing detailed insights", "3428366": "Amazing work. The focus on data quality over complex modeling really stands out \ngreat reminder that better data often beats everything else.", "3428367": "Incredible solution. The way you improved data step by step is really insightful \nthis shows how much impact clean data can have on performance.", "3428626": "Congratulations! Well done! \n\nI think it may have already been mentioned, but if possible, please share your training data! Like many of these write-ups have been saying, data-mining seems to be the primary factor that separated the high scorers from the low scorers. I was a bit too late to the competition to explore data-scraping in depth, but I do have some ideas about improvements regarding the model itself that I think could benefit the quality of the translations. I really love the mission of this competition, so I'd love to continue exploring ways to improve these translations! Thank you for your hard work!", "3428854": "Ablation study done. Without the weighted model, the score is:\nprivate score: 43.0\npublic score: 41.4\n\nand our best model is (the best one, not selected)\nprivate score: 43.2\npublic score: 41.6\n\nSo it did brought some diversity to the ensemble.", "3428933": "Congo & well done brother.", "3428960": "Great 👍 congratulations 👏", "3428986": "Sorry for the late response! To ensure full compliance, we've sent the full data to the host first. Following their suggestion, please reach out to me via Linkedin or email (you can find in my profile) to obtain the anonymous data for your research use.\n\nAlso, stay tuned for updates to the OARE database! I've been informed that they're planning to expand it—including adding translations—as an open-source project. If that sounds like something you'd like to be part of, feel free to reach out to the host to get involved. 👏", "3429203": "Congratulations excellent work!! \nThank you for sharing this amazing writeup!\nWell deserved First place", "3429326": "No worries! I'm sure you are very busy! Thank you - I will!", "3429628": "It has been sent! Hope it can be helpful, good luck!", "3431823": "Great example of how data quality dominates model gains - very insightful pipeline, especially the iterative OCR refinement and filtering.", "3437765": "I have a question. You mentioned using 5-fold cross-validation. Why were there 11 models during the final inference?", "3456014": "Congrats on the win! This is a very helpful write-up. I was especially impressed by the variations of the datasets you experimented with. Using the LLM-produced synthetic data approach is really cool, too. I would like to do further research, trying to improve Akkadian translation for my DL class, so I was wondering if you guys could let me use your curated datasets. I would really appreciate it!", "3457143": "You can contact us by sending an email to Guoqin, you can find his email in his kaggle profile." }, "source": "meta" }