# [50th] Solution: ByT5-base + TAPT + Ensemble

Acknowledgements
We would like to thank Deep Past Initiative and Kaggle for organizing this
Akkadian Translation Competition
.Also,we express our gratitude to
[@ruchiragar](https://www.kaggle.com/ruchiragar)
[@deeppast](https://www.kaggle.com/deeppast)
the organizers of the Deep Past Initiative, and the Kaggle staff for their patient explanations and good work.
gratitude to
[@takamichitoda](https://www.kaggle.com/takamichitoda)
[@vitorhugobarbedo](https://www.kaggle.com/vitorhugobarbedo)
for publishing outstanding notebooks and
[@phucthaiv02](https://www.kaggle.com/phucthaiv02)
for publishing akkadian_english_sentences_alignment_2 dataset in the hugging face.
My teammates:
[@shreyasadhari123](https://www.kaggle.com/shreyasadhari123)
[@aaronbornstein](https://www.kaggle.com/aaronbornstein)
[@kueipo](https://www.kaggle.com/kueipo)
[@llkh0a](https://www.kaggle.com/llkh0a)
[@qifeihhh666](https://www.kaggle.com/qifeihhh666)
Overview
Our final solution is an ensemble of four ByT5-base models — trained on
different fold data
and
extra aligned data
.
Three steps to train the end ByT5-base models:
Task-Adaptive Pre-training (TAPT)
on published_texts.csv
Supervised train the TAPT model on different fold data and name augument
Supervised fine-tuning the different fold model on extra aligned data
ensemble notebook use MBR decoding.
ensemble+ridge regression+Onomasticon+ MBR decoding.
Datasets
different fold data: 9k sentence + 2K passages + 6K pseudo-label sentence
We use AI agent to aligned train.csv get 9k sentence  data and generate pseudo-label sentence get 6k pseudo-label sentence data
Stratified 5-fold cross-validation by splitting data based on the log_ratio (translation/transliteration length ratio) to ensure balanced data distribution across folds. It assigns fold IDs (0–4) to main data and marks extra pseudo-labeled data as fold=-1, producing stable folds for training 4 ByT5-base ensemble models.(except fold 1)
extra aligned data: 404 sentence + 2277 sentence + 7742 hugging face data
I manually aligned and segmented a portion of the data from train.csv, resulting in 404 aligned sentences.
I then defined custom truncation tokens to split the document-level data and merged sentences that were too short or had incorrect length ratios, yielding 2,277 sentences.
Next, I filtered out sentences with invalid length ratios from the Hugging Face dataset, obtaining 7,742 sentences.
akk_names_with_count.csv
We use AI agent to automatically extract all person names and their corresponding translations from the dataset, then build a frequency table. During training, we randomly replace the translated person names in the translation column with a small probability, which yields a 0.1 score improvement.
Data Processing
We use decimal notation for training, not unicode_fractions.
transliteration data:
text = text.replace(r'{TÚG}', 'TÚG ')
    text = text.replace(r'+', ' ')
    text = text.replace(r'/', '')
    text = text.replace(r'(', '')
    text = text.replace(r')', '')
    text = text.replace(r'!', '')
    text = text.replace(r'ı', '')
    text = text.replace(r'ş', '')
    text = text.replace(r'İ', '')
    text = text.replace(r':', '')
    text = text.replace(r'  ', ' ')
translation data:
Only keep ' " : . -
Model Params
TAPT(Task-Adaptive Pre-training):
use TPU,around 10-15 epoch
Supervised train ByT5-base model:
LEARNING_RATE = 2e-4
MAX_LENGTH = 512
EPOCHS = 10

label_smoothing_factor=0.1,
lr_scheduler_type="linear"
warmup_ratio=0.05,
Supervised fine-tuning the different fold model:
Akkadian→English and English→Akkadian
LEARNING_RATE = 1e-4
MAX_LENGTH = 640
EPOCHS = 2
Results
Model Name
LB
PB
pseudo-labelled
35.7
mlm-fold0
35.5
35.5
mlm-fold2
35.3
35.5
mlm-fold3
35.3
35.2
mlm-fold4
35.0
35.3
ensemble
35.5
36.1
fine-tuning-mlm-fold0
35.9
36.3
fine-tuning-mlm-fold2
36.3
36.7
fine-tuning-mlm-fold3
36.2
36.2
fine-tuning-mlm-fold4
36.5
36.5
ensemble
36.9
37.3
ensemble+pseudo-labelled
36.4
37.5
Fail | Not Tried
Moe, byt5-large, byt5-XL, LLM, Hybrid-Reranker, Bidirection(Akkadian→English )