{
  "id": 684265,
  "title": "36th Place Solution - LLM-Based Sentence Alignment Strategy + MBR Model Ensembli",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/684265",
  "author_name": "Yihao Guo",
  "post_date": "2026-03-24T08:33:24.203000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Many thanks to the organizers and the Kaggle team for putting together such a wonderful competition! First of all, congratulations to all the winners and participants! Even though I invested nearly a month into this competition, the final result was not quite what I had hoped for (although for me personally, a gold medal is still something I need to keep striving toward), so I decided to document my final submission approach here.</p>\n<p>For this competition, I mainly used two models, byt5-base and byt5-large, and combined them through MBR ensembling to achieve my final score. Data was extremely important for me. In addition to the 1.5k+ samples provided by the organizers, I also obtained nearly <a href=\"https://huggingface.co/datasets/phucthaiv02/akkadian-translation\" target=\"_blank\">80k</a> additional samples from Hugging Face and applied a variety of preprocessing strategies to them.</p>\n<h1>Data Preparation</h1>\n<p>Since the official train.csv consists of paragraph-level translations, often accompanied by very long context, while the test set samples are provided at the sentence level, it was essential to realign the original training set at the sentence level. By carefully designing prompts, I used an LLM to perform sentence alignment at different levels of granularity.\nThe final submission used three datasets, processed as follows:</p>\n<ul>\n<li>I used prompts written with GPT-4o to split and align the organizer-provided data, resulting in about <strong>5k</strong> samples.</li>\n<li>I used another set of GPT-4o prompts to split and align the organizer-provided data, resulting in about <strong>3k</strong> samples. (This version distinguishes whether a sentence is complete, and the prompt emphasis differs from the one above.)</li>\n<li>I cleaned the additional 80k dataset by removing short sentences, resulting in <strong>50k+</strong> samples.</li>\n</ul>\n<h1>Training Setup</h1>\n<ul>\n<li>The first byt5-base model was trained on the first 5k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.</li>\n<li>The second byt5-base model was trained on the second 3k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.</li>\n<li>The first byt5-large model was trained on the cleaned 80k+5k dataset (where the two datasets were split proportionally and then recombined into training and validation sets), with a learning rate of 2e-4, using a single A100 GPU locally.</li>\n</ul>\n<h1>Inference results</h1>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Public Leaderboard</th>\n<th>Private Leaderboard</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>byt-base-5k</td>\n<td>35.4</td>\n<td>35.7</td>\n</tr>\n<tr>\n<td>byt-base-3k</td>\n<td>34.6</td>\n<td>34.9</td>\n</tr>\n<tr>\n<td>byt-large</td>\n<td>33.5</td>\n<td>34.9</td>\n</tr>\n<tr>\n<td>Ensemble</td>\n<td>37.4</td>\n<td>38.1</td>\n</tr>\n</tbody>\n</table>\n<p>In this competition, the main role of MBR was to take advantage of the complementarity between models trained from different data sources. Compared with a single model’s output, MBR can select the translation with the lowest overall risk and more stable style from multiple candidates. This is especially valuable in a task like Akkadian translation, which is low-resource, formulaic, and often contains incomplete text. Different training data sources—such as the official split data, external public data, and training sets constructed at different granularities—lead models to develop different preferences, and MBR is able to turn those differences into gains in final performance. Based on this idea, although none of my single models scored above 36, my ensemble was able to reach 38.1!</p>\n<p>Multiple rounds of experiments supported the following conclusion: simply adding more similar models does not necessarily lead to further improvement. What truly works is:</p>\n<ul>\n<li>keeping strong, high-quality core models</li>\n<li>adding auxiliary models with different data sources and stylistic tendencies</li>\n</ul>\n<p>Only then can MBR really show its value, rather than just repeatedly voting among a group of highly similar candidates.</p>\n<h1>Reflections</h1>\n<ul>\n<li>I competed solo, which limited my perspective, and I was unable to make any major breakthroughs in the later stages.</li>\n<li>I did not make full use of the official data, such as Published_texts.csv, which indirectly kept my single-model scores from improving much.</li>\n<li>Most of my gains came from multi-model ensembling, while the capability of each individual model remained limited.</li>\n<li>……</li>\n</ul>",
  "messages": [
    {
      "id": 3427553,
      "postDate": "2026-03-24T08:33:24.203Z",
      "content": "<p>Many thanks to the organizers and the Kaggle team for putting together such a wonderful competition! First of all, congratulations to all the winners and participants! Even though I invested nearly a month into this competition, the final result was not quite what I had hoped for (although for me personally, a gold medal is still something I need to keep striving toward), so I decided to document my final submission approach here.</p>\n<p>For this competition, I mainly used two models, byt5-base and byt5-large, and combined them through MBR ensembling to achieve my final score. Data was extremely important for me. In addition to the 1.5k+ samples provided by the organizers, I also obtained nearly <a href=\"https://huggingface.co/datasets/phucthaiv02/akkadian-translation\" target=\"_blank\">80k</a> additional samples from Hugging Face and applied a variety of preprocessing strategies to them.</p>\n<h1>Data Preparation</h1>\n<p>Since the official train.csv consists of paragraph-level translations, often accompanied by very long context, while the test set samples are provided at the sentence level, it was essential to realign the original training set at the sentence level. By carefully designing prompts, I used an LLM to perform sentence alignment at different levels of granularity.\nThe final submission used three datasets, processed as follows:</p>\n<ul>\n<li>I used prompts written with GPT-4o to split and align the organizer-provided data, resulting in about <strong>5k</strong> samples.</li>\n<li>I used another set of GPT-4o prompts to split and align the organizer-provided data, resulting in about <strong>3k</strong> samples. (This version distinguishes whether a sentence is complete, and the prompt emphasis differs from the one above.)</li>\n<li>I cleaned the additional 80k dataset by removing short sentences, resulting in <strong>50k+</strong> samples.</li>\n</ul>\n<h1>Training Setup</h1>\n<ul>\n<li>The first byt5-base model was trained on the first 5k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.</li>\n<li>The second byt5-base model was trained on the second 3k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.</li>\n<li>The first byt5-large model was trained on the cleaned 80k+5k dataset (where the two datasets were split proportionally and then recombined into training and validation sets), with a learning rate of 2e-4, using a single A100 GPU locally.</li>\n</ul>\n<h1>Inference results</h1>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Public Leaderboard</th>\n<th>Private Leaderboard</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>byt-base-5k</td>\n<td>35.4</td>\n<td>35.7</td>\n</tr>\n<tr>\n<td>byt-base-3k</td>\n<td>34.6</td>\n<td>34.9</td>\n</tr>\n<tr>\n<td>byt-large</td>\n<td>33.5</td>\n<td>34.9</td>\n</tr>\n<tr>\n<td>Ensemble</td>\n<td>37.4</td>\n<td>38.1</td>\n</tr>\n</tbody>\n</table>\n<p>In this competition, the main role of MBR was to take advantage of the complementarity between models trained from different data sources. Compared with a single model’s output, MBR can select the translation with the lowest overall risk and more stable style from multiple candidates. This is especially valuable in a task like Akkadian translation, which is low-resource, formulaic, and often contains incomplete text. Different training data sources—such as the official split data, external public data, and training sets constructed at different granularities—lead models to develop different preferences, and MBR is able to turn those differences into gains in final performance. Based on this idea, although none of my single models scored above 36, my ensemble was able to reach 38.1!</p>\n<p>Multiple rounds of experiments supported the following conclusion: simply adding more similar models does not necessarily lead to further improvement. What truly works is:</p>\n<ul>\n<li>keeping strong, high-quality core models</li>\n<li>adding auxiliary models with different data sources and stylistic tendencies</li>\n</ul>\n<p>Only then can MBR really show its value, rather than just repeatedly voting among a group of highly similar candidates.</p>\n<h1>Reflections</h1>\n<ul>\n<li>I competed solo, which limited my perspective, and I was unable to make any major breakthroughs in the later stages.</li>\n<li>I did not make full use of the official data, such as Published_texts.csv, which indirectly kept my single-model scores from improving much.</li>\n<li>Most of my gains came from multi-model ensembling, while the capability of each individual model remained limited.</li>\n<li>……</li>\n</ul>",
      "rawMarkdown": "Many thanks to the organizers and the Kaggle team for putting together such a wonderful competition! First of all, congratulations to all the winners and participants! Even though I invested nearly a month into this competition, the final result was not quite what I had hoped for (although for me personally, a gold medal is still something I need to keep striving toward), so I decided to document my final submission approach here.\n\nFor this competition, I mainly used two models, byt5-base and byt5-large, and combined them through MBR ensembling to achieve my final score. Data was extremely important for me. In addition to the 1.5k+ samples provided by the organizers, I also obtained nearly [80k](https://huggingface.co/datasets/phucthaiv02/akkadian-translation) additional samples from Hugging Face and applied a variety of preprocessing strategies to them.\n\n# Data Preparation\nSince the official train.csv consists of paragraph-level translations, often accompanied by very long context, while the test set samples are provided at the sentence level, it was essential to realign the original training set at the sentence level. By carefully designing prompts, I used an LLM to perform sentence alignment at different levels of granularity.\nThe final submission used three datasets, processed as follows:\n- I used prompts written with GPT-4o to split and align the organizer-provided data, resulting in about **5k** samples.\n- I used another set of GPT-4o prompts to split and align the organizer-provided data, resulting in about **3k** samples. (This version distinguishes whether a sentence is complete, and the prompt emphasis differs from the one above.)\n- I cleaned the additional 80k dataset by removing short sentences, resulting in **50k+** samples.\n\n# Training Setup\n- The first byt5-base model was trained on the first 5k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.\n- The second byt5-base model was trained on the second 3k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.\n- The first byt5-large model was trained on the cleaned 80k+5k dataset (where the two datasets were split proportionally and then recombined into training and validation sets), with a learning rate of 2e-4, using a single A100 GPU locally.\n\n# Inference results\n| Model | Public Leaderboard | Private Leaderboard | \n| --- | --- | --- |\n| byt-base-5k | 35.4 | 35.7 |\n| byt-base-3k | 34.6 | 34.9 | \n| byt-large | 33.5 | 34.9 |\n| Ensemble | 37.4 | 38.1 |\n\n\n\nIn this competition, the main role of MBR was to take advantage of the complementarity between models trained from different data sources. Compared with a single model’s output, MBR can select the translation with the lowest overall risk and more stable style from multiple candidates. This is especially valuable in a task like Akkadian translation, which is low-resource, formulaic, and often contains incomplete text. Different training data sources—such as the official split data, external public data, and training sets constructed at different granularities—lead models to develop different preferences, and MBR is able to turn those differences into gains in final performance. Based on this idea, although none of my single models scored above 36, my ensemble was able to reach 38.1!\n\nMultiple rounds of experiments supported the following conclusion: simply adding more similar models does not necessarily lead to further improvement. What truly works is:\n- keeping strong, high-quality core models\n- adding auxiliary models with different data sources and stylistic tendencies\n\nOnly then can MBR really show its value, rather than just repeatedly voting among a group of highly similar candidates.\n\n# Reflections\n- I competed solo, which limited my perspective, and I was unable to make any major breakthroughs in the later stages.\n- I did not make full use of the official data, such as Published_texts.csv, which indirectly kept my single-model scores from improving much.\n- Most of my gains came from multi-model ensembling, while the capability of each individual model remained limited.\n- ......",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3427553": "Many thanks to the organizers and the Kaggle team for putting together such a wonderful competition! First of all, congratulations to all the winners and participants! Even though I invested nearly a month into this competition, the final result was not quite what I had hoped for (although for me personally, a gold medal is still something I need to keep striving toward), so I decided to document my final submission approach here.\n\nFor this competition, I mainly used two models, byt5-base and byt5-large, and combined them through MBR ensembling to achieve my final score. Data was extremely important for me. In addition to the 1.5k+ samples provided by the organizers, I also obtained nearly [80k](https://huggingface.co/datasets/phucthaiv02/akkadian-translation) additional samples from Hugging Face and applied a variety of preprocessing strategies to them.\n\n# Data Preparation\nSince the official train.csv consists of paragraph-level translations, often accompanied by very long context, while the test set samples are provided at the sentence level, it was essential to realign the original training set at the sentence level. By carefully designing prompts, I used an LLM to perform sentence alignment at different levels of granularity.\nThe final submission used three datasets, processed as follows:\n- I used prompts written with GPT-4o to split and align the organizer-provided data, resulting in about **5k** samples.\n- I used another set of GPT-4o prompts to split and align the organizer-provided data, resulting in about **3k** samples. (This version distinguishes whether a sentence is complete, and the prompt emphasis differs from the one above.)\n- I cleaned the additional 80k dataset by removing short sentences, resulting in **50k+** samples.\n\n# Training Setup\n- The first byt5-base model was trained on the first 5k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.\n- The second byt5-base model was trained on the second 3k dataset, with a learning rate of 5e-4, using a single A100 GPU locally.\n- The first byt5-large model was trained on the cleaned 80k+5k dataset (where the two datasets were split proportionally and then recombined into training and validation sets), with a learning rate of 2e-4, using a single A100 GPU locally.\n\n# Inference results\n| Model | Public Leaderboard | Private Leaderboard | \n| --- | --- | --- |\n| byt-base-5k | 35.4 | 35.7 |\n| byt-base-3k | 34.6 | 34.9 | \n| byt-large | 33.5 | 34.9 |\n| Ensemble | 37.4 | 38.1 |\n\n\n\nIn this competition, the main role of MBR was to take advantage of the complementarity between models trained from different data sources. Compared with a single model’s output, MBR can select the translation with the lowest overall risk and more stable style from multiple candidates. This is especially valuable in a task like Akkadian translation, which is low-resource, formulaic, and often contains incomplete text. Different training data sources—such as the official split data, external public data, and training sets constructed at different granularities—lead models to develop different preferences, and MBR is able to turn those differences into gains in final performance. Based on this idea, although none of my single models scored above 36, my ensemble was able to reach 38.1!\n\nMultiple rounds of experiments supported the following conclusion: simply adding more similar models does not necessarily lead to further improvement. What truly works is:\n- keeping strong, high-quality core models\n- adding auxiliary models with different data sources and stylistic tendencies\n\nOnly then can MBR really show its value, rather than just repeatedly voting among a group of highly similar candidates.\n\n# Reflections\n- I competed solo, which limited my perspective, and I was unable to make any major breakthroughs in the later stages.\n- I did not make full use of the official data, such as Published_texts.csv, which indirectly kept my single-model scores from improving much.\n- Most of my gains came from multi-model ensembling, while the capability of each individual model remained limited.\n- ......"
  }
}