{ "id": 684243, "title": "[50th] Solution: ByT5-base + TAPT + Ensemble", "url": "/competitions/deep-past-initiative-machine-translation/discussion/684243", "author_name": "耶✌", "post_date": "2026-03-24T07:00:43.252000", "votes": 7, "comment_count": 0, "views": 0, "content": "

Acknowledgements

\n

We would like to thank Deep Past Initiative and Kaggle for organizing this Akkadian Translation Competition.Also,we express our gratitude to @ruchiragar @deeppast the organizers of the Deep Past Initiative, and the Kaggle staff for their patient explanations and good work.

\n

gratitude to @takamichitoda @vitorhugobarbedo for publishing outstanding notebooks and @phucthaiv02 for publishing akkadian_english_sentences_alignment_2 dataset in the hugging face.

\n

My teammates: @shreyasadhari123 @aaronbornstein @kueipo @llkh0a @qifeihhh666

\n

Overview

\n

Our final solution is an ensemble of four ByT5-base models — trained on different fold data and extra aligned data.

\n

Three steps to train the end ByT5-base models:

\n
    \n
  1. Task-Adaptive Pre-training (TAPT) on published_texts.csv
    2. \n
  2. Supervised train the TAPT model on different fold data and name augument
    3. \n
  3. Supervised fine-tuning the different fold model on extra aligned data
    4. \n
\n

ensemble notebook use MBR decoding.

\n

ensemble+ridge regression+Onomasticon+ MBR decoding.

\n

Datasets

\n

different fold data: 9k sentence + 2K passages + 6K pseudo-label sentence

\n
    \n
  1. We use AI agent to aligned train.csv get 9k sentence data and generate pseudo-label sentence get 6k pseudo-label sentence data
    2. \n
  2. Stratified 5-fold cross-validation by splitting data based on the log_ratio (translation/transliteration length ratio) to ensure balanced data distribution across folds. It assigns fold IDs (0–4) to main data and marks extra pseudo-labeled data as fold=-1, producing stable folds for training 4 ByT5-base ensemble models.(except fold 1)
    3. \n
\n

extra aligned data: 404 sentence + 2277 sentence + 7742 hugging face data

\n
    \n
  1. I manually aligned and segmented a portion of the data from train.csv, resulting in 404 aligned sentences.
    2. \n
  2. I then defined custom truncation tokens to split the document-level data and merged sentences that were too short or had incorrect length ratios, yielding 2,277 sentences.
    3. \n
  3. Next, I filtered out sentences with invalid length ratios from the Hugging Face dataset, obtaining 7,742 sentences.
    4. \n
\n

akk_names_with_count.csv

\n

We use AI agent to automatically extract all person names and their corresponding translations from the dataset, then build a frequency table. During training, we randomly replace the translated person names in the translation column with a small probability, which yields a 0.1 score improvement.

\n

Data Processing

\n

We use decimal notation for training, not unicode_fractions.

\n

transliteration data:

\n
    text = text.replace(r'{TÚG}', 'TÚG ')\n    text = text.replace(r'+', ' ')\n    text = text.replace(r'/', '')\n    text = text.replace(r'(', '')\n    text = text.replace(r')', '')\n    text = text.replace(r'!', '')\n    text = text.replace(r'ı', '')\n    text = text.replace(r'ş', '')\n    text = text.replace(r'İ', '')\n    text = text.replace(r':', '')\n    text = text.replace(r'  ', ' ')\n
\n

translation data:

\n

Only keep ' \" : . -

\n

Model Params

\n

TAPT(Task-Adaptive Pre-training):

\n

use TPU,around 10-15 epoch

\n

Supervised train ByT5-base model:

\n
LEARNING_RATE = 2e-4\nMAX_LENGTH = 512\nEPOCHS = 10\n\nlabel_smoothing_factor=0.1,\nlr_scheduler_type=\"linear\"\nwarmup_ratio=0.05,\n
\n

Supervised fine-tuning the different fold model:

\n

Akkadian→English and English→Akkadian

\n
LEARNING_RATE = 1e-4\nMAX_LENGTH = 640\nEPOCHS = 2\n
\n

Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model NameLBPB
pseudo-labelled35.7
mlm-fold035.535.5
mlm-fold235.335.5
mlm-fold335.335.2
mlm-fold435.035.3
ensemble35.536.1
fine-tuning-mlm-fold035.936.3
fine-tuning-mlm-fold236.336.7
fine-tuning-mlm-fold336.236.2
fine-tuning-mlm-fold436.536.5
ensemble36.937.3
ensemble+pseudo-labelled36.437.5
\n

Fail | Not Tried

\n

Moe, byt5-large, byt5-XL, LLM, Hybrid-Reranker, Bidirection(Akkadian→English )

", "messages": [ { "id": 3427506, "postDate": "2026-03-24T07:00:43.253Z", "content": "

Acknowledgements

\n

We would like to thank Deep Past Initiative and Kaggle for organizing this Akkadian Translation Competition.Also,we express our gratitude to @ruchiragar @deeppast the organizers of the Deep Past Initiative, and the Kaggle staff for their patient explanations and good work.

\n

gratitude to @takamichitoda @vitorhugobarbedo for publishing outstanding notebooks and @phucthaiv02 for publishing akkadian_english_sentences_alignment_2 dataset in the hugging face.

\n

My teammates: @shreyasadhari123 @aaronbornstein @kueipo @llkh0a @qifeihhh666

\n

Overview

\n

Our final solution is an ensemble of four ByT5-base models — trained on different fold data and extra aligned data.

\n

Three steps to train the end ByT5-base models:

\n
    \n
  1. Task-Adaptive Pre-training (TAPT) on published_texts.csv
    2. \n
  2. Supervised train the TAPT model on different fold data and name augument
    3. \n
  3. Supervised fine-tuning the different fold model on extra aligned data
    4. \n
\n

ensemble notebook use MBR decoding.

\n

ensemble+ridge regression+Onomasticon+ MBR decoding.

\n

Datasets

\n

different fold data: 9k sentence + 2K passages + 6K pseudo-label sentence

\n
    \n
  1. We use AI agent to aligned train.csv get 9k sentence data and generate pseudo-label sentence get 6k pseudo-label sentence data
    2. \n
  2. Stratified 5-fold cross-validation by splitting data based on the log_ratio (translation/transliteration length ratio) to ensure balanced data distribution across folds. It assigns fold IDs (0–4) to main data and marks extra pseudo-labeled data as fold=-1, producing stable folds for training 4 ByT5-base ensemble models.(except fold 1)
    3. \n
\n

extra aligned data: 404 sentence + 2277 sentence + 7742 hugging face data

\n
    \n
  1. I manually aligned and segmented a portion of the data from train.csv, resulting in 404 aligned sentences.
    2. \n
  2. I then defined custom truncation tokens to split the document-level data and merged sentences that were too short or had incorrect length ratios, yielding 2,277 sentences.
    3. \n
  3. Next, I filtered out sentences with invalid length ratios from the Hugging Face dataset, obtaining 7,742 sentences.
    4. \n
\n

akk_names_with_count.csv

\n

We use AI agent to automatically extract all person names and their corresponding translations from the dataset, then build a frequency table. During training, we randomly replace the translated person names in the translation column with a small probability, which yields a 0.1 score improvement.

\n

Data Processing

\n

We use decimal notation for training, not unicode_fractions.

\n

transliteration data:

\n
    text = text.replace(r'{TÚG}', 'TÚG ')\n    text = text.replace(r'+', ' ')\n    text = text.replace(r'/', '')\n    text = text.replace(r'(', '')\n    text = text.replace(r')', '')\n    text = text.replace(r'!', '')\n    text = text.replace(r'ı', '')\n    text = text.replace(r'ş', '')\n    text = text.replace(r'İ', '')\n    text = text.replace(r':', '')\n    text = text.replace(r'  ', ' ')\n
\n

translation data:

\n

Only keep ' \" : . -

\n

Model Params

\n

TAPT(Task-Adaptive Pre-training):

\n

use TPU,around 10-15 epoch

\n

Supervised train ByT5-base model:

\n
LEARNING_RATE = 2e-4\nMAX_LENGTH = 512\nEPOCHS = 10\n\nlabel_smoothing_factor=0.1,\nlr_scheduler_type=\"linear\"\nwarmup_ratio=0.05,\n
\n

Supervised fine-tuning the different fold model:

\n

Akkadian→English and English→Akkadian

\n
LEARNING_RATE = 1e-4\nMAX_LENGTH = 640\nEPOCHS = 2\n
\n

Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model NameLBPB
pseudo-labelled35.7
mlm-fold035.535.5
mlm-fold235.335.5
mlm-fold335.335.2
mlm-fold435.035.3
ensemble35.536.1
fine-tuning-mlm-fold035.936.3
fine-tuning-mlm-fold236.336.7
fine-tuning-mlm-fold336.236.2
fine-tuning-mlm-fold436.536.5
ensemble36.937.3
ensemble+pseudo-labelled36.437.5
\n

Fail | Not Tried

\n

Moe, byt5-large, byt5-XL, LLM, Hybrid-Reranker, Bidirection(Akkadian→English )

", "rawMarkdown": "# **Acknowledgements**\nWe would like to thank Deep Past Initiative and Kaggle for organizing this **Akkadian Translation Competition**.Also,we express our gratitude to @ruchiragar @deeppast the organizers of the Deep Past Initiative, and the Kaggle staff for their patient explanations and good work.\n\ngratitude to @takamichitoda @vitorhugobarbedo for publishing outstanding notebooks and @phucthaiv02 for publishing akkadian_english_sentences_alignment_2 dataset in the hugging face.\n\nMy teammates: @shreyasadhari123 @aaronbornstein @kueipo @llkh0a @qifeihhh666 \n\n# **Overview**\nOur final solution is an ensemble of four ByT5-base models — trained on **different fold data** and **extra aligned data**.\n\nThree steps to train the end ByT5-base models:\n1. **Task-Adaptive Pre-training (TAPT)** on published_texts.csv\n2. Supervised train the TAPT model on different fold data and name augument\n3. Supervised fine-tuning the different fold model on extra aligned data\n\nensemble notebook use MBR decoding.\n\nensemble+ridge regression+Onomasticon+ MBR decoding.\n\n# **Datasets**\n### different fold data: 9k sentence + 2K passages + 6K pseudo-label sentence\n1. We use AI agent to aligned train.csv get 9k sentence data and generate pseudo-label sentence get 6k pseudo-label sentence data\n2. Stratified 5-fold cross-validation by splitting data based on the log_ratio (translation/transliteration length ratio) to ensure balanced data distribution across folds. It assigns fold IDs (0–4) to main data and marks extra pseudo-labeled data as fold=-1, producing stable folds for training 4 ByT5-base ensemble models.(except fold 1)\n\n### extra aligned data: 404 sentence + 2277 sentence + 7742 hugging face data\n1. I manually aligned and segmented a portion of the data from train.csv, resulting in 404 aligned sentences.\n2. I then defined custom truncation tokens to split the document-level data and merged sentences that were too short or had incorrect length ratios, yielding 2,277 sentences.\n3. Next, I filtered out sentences with invalid length ratios from the Hugging Face dataset, obtaining 7,742 sentences.\n\n### akk_names_with_count.csv\nWe use AI agent to automatically extract all person names and their corresponding translations from the dataset, then build a frequency table. During training, we randomly replace the translated person names in the translation column with a small probability, which yields a 0.1 score improvement.\n\n\n# **Data Processing**\nWe use decimal notation for training, not unicode_fractions.\n### transliteration data: \n```python\n text = text.replace(r'{TÚG}', 'TÚG ')\n text = text.replace(r'+', ' ')\n text = text.replace(r'/', '')\n text = text.replace(r'(', '')\n text = text.replace(r')', '')\n text = text.replace(r'!', '')\n text = text.replace(r'ı', '')\n text = text.replace(r'ş', '')\n text = text.replace(r'İ', '')\n text = text.replace(r':', '')\n text = text.replace(r' ', ' ')\n```\n\n### translation data:\nOnly keep ' \" : . -\n\n# **Model Params**\n####TAPT(Task-Adaptive Pre-training):\nuse TPU,around 10-15 epoch\n\n####Supervised train ByT5-base model:\n```python\nLEARNING_RATE = 2e-4\nMAX_LENGTH = 512\nEPOCHS = 10\n\nlabel_smoothing_factor=0.1,\nlr_scheduler_type=\"linear\"\nwarmup_ratio=0.05,\n```\n\n####Supervised fine-tuning the different fold model:\nAkkadian→English and English→Akkadian\n```python\nLEARNING_RATE = 1e-4\nMAX_LENGTH = 640\nEPOCHS = 2\n```\n\n\n\n# **Results**\n\n| Model Name | LB | PB | \n|------------------|----------|\n| pseudo-labelled | 35.7 | | \n| mlm-fold0 | 35.5 | 35.5 | \n| mlm-fold2 | 35.3 | 35.5 | \n| mlm-fold3 | 35.3 | 35.2 | \n| mlm-fold4 | 35.0 | 35.3 | \n| ensemble | 35.5 | 36.1 | \n| fine-tuning-mlm-fold0 | 35.9 | 36.3 | \n| fine-tuning-mlm-fold2 | 36.3 | 36.7 |\n| fine-tuning-mlm-fold3 | 36.2 | 36.2 |\n| fine-tuning-mlm-fold4 | 36.5 | 36.5 | \n| ensemble | 36.9 | 37.3 | \n| ensemble+pseudo-labelled | 36.4 | 37.5 | \n\n\n\n\n\n\n# **Fail | Not Tried**\nMoe, byt5-large, byt5-XL, LLM, Hybrid-Reranker, Bidirection(Akkadian→English )", "votes": 7 } ], "comments": [], "raw_markdown_by_id": { "3427506": "# **Acknowledgements**\nWe would like to thank Deep Past Initiative and Kaggle for organizing this **Akkadian Translation Competition**.Also,we express our gratitude to @ruchiragar @deeppast the organizers of the Deep Past Initiative, and the Kaggle staff for their patient explanations and good work.\n\ngratitude to @takamichitoda @vitorhugobarbedo for publishing outstanding notebooks and @phucthaiv02 for publishing akkadian_english_sentences_alignment_2 dataset in the hugging face.\n\nMy teammates: @shreyasadhari123 @aaronbornstein @kueipo @llkh0a @qifeihhh666 \n\n# **Overview**\nOur final solution is an ensemble of four ByT5-base models — trained on **different fold data** and **extra aligned data**.\n\nThree steps to train the end ByT5-base models:\n1. **Task-Adaptive Pre-training (TAPT)** on published_texts.csv\n2. Supervised train the TAPT model on different fold data and name augument\n3. Supervised fine-tuning the different fold model on extra aligned data\n\nensemble notebook use MBR decoding.\n\nensemble+ridge regression+Onomasticon+ MBR decoding.\n\n# **Datasets**\n### different fold data: 9k sentence + 2K passages + 6K pseudo-label sentence\n1. We use AI agent to aligned train.csv get 9k sentence data and generate pseudo-label sentence get 6k pseudo-label sentence data\n2. Stratified 5-fold cross-validation by splitting data based on the log_ratio (translation/transliteration length ratio) to ensure balanced data distribution across folds. It assigns fold IDs (0–4) to main data and marks extra pseudo-labeled data as fold=-1, producing stable folds for training 4 ByT5-base ensemble models.(except fold 1)\n\n### extra aligned data: 404 sentence + 2277 sentence + 7742 hugging face data\n1. I manually aligned and segmented a portion of the data from train.csv, resulting in 404 aligned sentences.\n2. I then defined custom truncation tokens to split the document-level data and merged sentences that were too short or had incorrect length ratios, yielding 2,277 sentences.\n3. Next, I filtered out sentences with invalid length ratios from the Hugging Face dataset, obtaining 7,742 sentences.\n\n### akk_names_with_count.csv\nWe use AI agent to automatically extract all person names and their corresponding translations from the dataset, then build a frequency table. During training, we randomly replace the translated person names in the translation column with a small probability, which yields a 0.1 score improvement.\n\n\n# **Data Processing**\nWe use decimal notation for training, not unicode_fractions.\n### transliteration data: \n```python\n text = text.replace(r'{TÚG}', 'TÚG ')\n text = text.replace(r'+', ' ')\n text = text.replace(r'/', '')\n text = text.replace(r'(', '')\n text = text.replace(r')', '')\n text = text.replace(r'!', '')\n text = text.replace(r'ı', '')\n text = text.replace(r'ş', '')\n text = text.replace(r'İ', '')\n text = text.replace(r':', '')\n text = text.replace(r' ', ' ')\n```\n\n### translation data:\nOnly keep ' \" : . -\n\n# **Model Params**\n####TAPT(Task-Adaptive Pre-training):\nuse TPU,around 10-15 epoch\n\n####Supervised train ByT5-base model:\n```python\nLEARNING_RATE = 2e-4\nMAX_LENGTH = 512\nEPOCHS = 10\n\nlabel_smoothing_factor=0.1,\nlr_scheduler_type=\"linear\"\nwarmup_ratio=0.05,\n```\n\n####Supervised fine-tuning the different fold model:\nAkkadian→English and English→Akkadian\n```python\nLEARNING_RATE = 1e-4\nMAX_LENGTH = 640\nEPOCHS = 2\n```\n\n\n\n# **Results**\n\n| Model Name | LB | PB | \n|------------------|----------|\n| pseudo-labelled | 35.7 | | \n| mlm-fold0 | 35.5 | 35.5 | \n| mlm-fold2 | 35.3 | 35.5 | \n| mlm-fold3 | 35.3 | 35.2 | \n| mlm-fold4 | 35.0 | 35.3 | \n| ensemble | 35.5 | 36.1 | \n| fine-tuning-mlm-fold0 | 35.9 | 36.3 | \n| fine-tuning-mlm-fold2 | 36.3 | 36.7 |\n| fine-tuning-mlm-fold3 | 36.2 | 36.2 |\n| fine-tuning-mlm-fold4 | 36.5 | 36.5 | \n| ensemble | 36.9 | 37.3 | \n| ensemble+pseudo-labelled | 36.4 | 37.5 | \n\n\n\n\n\n\n# **Fail | Not Tried**\nMoe, byt5-large, byt5-XL, LLM, Hybrid-Reranker, Bidirection(Akkadian→English )" }, "complete": true, "source": "existing_discussion", "competition_id": 121150, "competition_slug": "deep-past-initiative-machine-translation", "team_id": 14985138, "solution_url": "/competitions/deep-past-initiative-machine-translation/writeups/56th-solution-byt5-base-tapt-ensemble" }