{ "id": 684189, "title": "[24th] Post-training Qwen2.5 32B and 72B with Gemini OCR/published texts pairs", "url": "/competitions/deep-past-initiative-machine-translation/writeups/25th-post-training-qwen2-5-32b-and-72b-with-gemi", "author_name": "", "post_date": "2026-03-24T00:57:55.877Z", "votes": 36, "comment_count": 7, "views": 0, "content": "

Solution Overview

\n

Notebook: https://www.kaggle.com/code/yeoyunsianggeremie/deep-past-inference

\n

Introduction

\n

Team members: @yeoyunsianggeremie @andrevma18

\n

We're both grinding for our startup, working 70-75 hour weeks, which left us almost no time to compete seriously. As such, we did the high-level thinking and solution-design ourselves (how to clean, how to generate data, etc) then distilled it into structured Markdown files for Qgentic-AI which is an autonomous Kaggle agent I have been building for the past 6 months. Qgentic-AI would run experiments iteratively throughout the week. We only spent 6 days grinding, which is very low given how unique and difficult this competition is.

\n

I almost said \"no\" to joining this competition - as I felt I wouldn't have enough time. But I realized, working under this intense constraint is something I am trying to actively get better at, and this would be a good practice!

\n

Models used

\n

Qwen2.5-32B, Qwen2.5-72B and ByT5-Base

\n

Datasets

\n

Published texts: Asked Gemini to split the data into sentences and generated translations for each of them\nPDFs: Fed each page into Gemini to extract transliteration/translation pairs

\n

Yup, you heard it right, we did not use the competition train data or attempt to clean it (I caught Qgentic-AI running in circles to decide whether to include it or not, and decided it was confusing so went against it)

\n

In total, we had about 55,000 transliteration/translation pairs after doing some manual cleaning and LLM-as-a-judge

\n

Training

\n

We post-trained Qwen2.5-32B and Qwen2.5-72B for 3 epochs on 8x B200 with LoRA rank 64 and alpha 128\nByT5 was trained for 15 epochs at a learning rate of 3e-4

\n

The parameters were tuned by Qgentic-AI (such a huge time saver 😁)

\n

We trained our models on ENG -> AKK and AKK -> ENG (so a total of ~110,000 training pairs)

\n

Validation

\n

We used a pertubed version of the train.csv removing all bad samples where the training and validation lengths are way off. There is still data leakage so we do not base 100% of our selections based on that. The CV is around 43 for our best models.

\n

Preprocessing

\n

Mostly based on the host's discussion - fixed a few bugs like 5/12 shekels -> 2/3 shekel 15 grains

\n

Postprocessing

\n

Removed repetitions using the base Qwen2.5-72B model using the prompt

\n
DEDUP_SYSTEM = \"You are a text editor. The user will give you an English translation that may contain repeated phrases or sentences. Remove any obvious repetitions while preserving the rest of the text exactly as-is. Do NOT rephrase, summarize, or add anything. Return ONLY the cleaned text.\"\n
\n

Failed Ideas

\n\n

Code for Preprocessing and Postprocessing

\n
import re\nimport pandas as pd\n\n\ndef replace_roman_months(text):\n    if pd.isna(text): return \"\"\n    text = str(text)\n    romans = {\n        \"XII\": \"12\", \"XI\": \"11\", \"IX\": \"9\", \"IV\": \"4\",\n        \"VIII\": \"8\", \"VII\": \"7\", \"III\": \"3\", \"VI\": \"6\",\n        \"X\": \"10\", \"V\": \"5\", \"II\": \"2\", \"I\": \"1\"\n    }\n    for r, num in romans.items():\n        text = re.sub(r'(?i)\\bmonth\\s+' + r + r'\\b', 'month ' + num, text)\n    return text\n\n_FRAC_MAP = {\n    0.5: \"½\", 0.3333: \"⅓\", 0.6666: \"⅔\", 0.6667: \"⅔\",\n    0.25: \"¼\", 0.75: \"¾\",\n    0.1666: \"⅙\", 0.1667: \"⅙\", 0.8333: \"⅚\",\n    0.125: \"⅛\", 0.375: \"⅜\", 0.625: \"⅝\", 0.875: \"⅞\",\n    0.2: \"⅕\", 0.4: \"⅖\", 0.6: \"⅗\", 0.8: \"⅘\",\n    0.333: \"⅓\",\n}\n\n_ASCII_FRAC_MAP = {\n    \"1/2\": \"½\", \"1/3\": \"⅓\", \"2/3\": \"⅔\",\n    \"1/4\": \"¼\", \"3/4\": \"¾\",\n    \"1/5\": \"⅕\", \"2/5\": \"⅖\", \"3/5\": \"⅗\", \"4/5\": \"⅘\",\n    \"1/6\": \"⅙\", \"5/6\": \"⅚\",\n    \"1/8\": \"⅛\", \"3/8\": \"⅜\", \"5/8\": \"⅝\", \"7/8\": \"⅞\",\n}\n\ndef _decimals_to_unicode(text):\n    def _replace(m):\n        val = float(m.group())\n        integer = int(val)\n        frac_part = round(val - integer, 4)\n        if frac_part == 0:\n            return m.group()\n        unicode_frac = _FRAC_MAP.get(frac_part)\n        if unicode_frac is None:\n            return m.group()\n        if integer == 0:\n            return unicode_frac\n        return f\"{integer} {unicode_frac}\"\n    return re.sub(r'(?<!\\d)\\d+\\.\\d+(?!\\d)', _replace, text)\n\ndef _ascii_fracs_to_unicode(text):\n    def _replace(m):\n        key = f\"{m.group(1)}/{m.group(2)}\"\n        return _ASCII_FRAC_MAP.get(key, m.group())\n    return re.sub(r'\\b(\\d+)\\s*/\\s*(\\d+)\\b', _replace, text)\n\ndef _convert_twelfth_shekels(text):\n    def _replace_11_12(m):\n        n = int(m.group(1))\n        return f\"{n + 1} shekels less 15 grains\"\n    text = re.sub(r'(\\d+)\\s+11\\s*/\\s*12\\s+shekels?', _replace_11_12, text)\n    twelfth_map = [\n        (\"7 / 12\",  \"½ shekel 15 grains\"),\n        (\"7/12\",    \"½ shekel 15 grains\"),\n        (\"5 / 12\",  \"⅓ shekel 15 grains\"),\n        (\"5/12\",    \"⅓ shekel 15 grains\"),\n        (\"1 / 12\",  \"15 grains\"),\n        (\"1/12\",    \"15 grains\"),\n    ]\n    for num, replacement in twelfth_map:\n        text = re.sub(\n            re.escape(num) + r'\\s*(?:\\(shekel\\)|shekels?)',\n            replacement,\n            text\n        )\n    return text\n\ndef clean_transliteration(text):\n    if pd.isna(text): return \"\"\n    text = str(text)\n    subscript_map = str.maketrans(\"₀₁₂₃₄₅₆₇₈₉\", \"0123456789\")\n    text = text.translate(subscript_map)\n    text = text.replace(\"Ḫ\", \"H\").replace(\"ḫ\", \"h\")\n    text = text.replace(\"KÙ.B.\", \"KÙ.BABBAR\")\n    for var in [r'\\[x\\]', r'\\[\\.\\.\\.\\.\\.\\.\\]', r'\\[\\.\\.\\. \\.\\.\\.\\]', r'\\.\\.\\.', r'\\(break\\)', r'\\(large break\\)', r'\\(n broken lines\\)', r'xₓ']:\n        text = re.sub(var, ' <gap> ', text)\n    text = _decimals_to_unicode(text)\n    text = _ascii_fracs_to_unicode(text)\n    text = replace_roman_months(text)\n    text = re.sub(r'(<\\s*gap\\s*>\\s*)+', '<gap> ', text)\n    return \" \".join(text.split())\n\ndef clean_translation(text):\n    if pd.isna(text): return \"\"\n    text = str(text)\n    for var in [r'\\[x\\]', r'\\[\\.\\.\\.\\.\\.\\.\\]', r'\\[\\.\\.\\. \\.\\.\\.\\]', r'\\.\\.\\.', r'\\(break\\)', r'\\(large break\\)', r'\\(n broken lines\\)', r'xₓ']:\n        text = re.sub(var, ' <gap> ', text)\n    for r_mark in [r'\\bfem\\.', r'\\bsing\\.', r'\\bpl\\.', r'\\bplural\\b', r'\\(\\?\\)', r'\\.\\.,', r'\\bxx\\b', r'\\bx\\b', r'<<\\s*>>', r'<\\s*>', r'\\[\\s*\\]', r'˹\\s*˺']:\n        text = re.sub(r_mark, '', text)\n    text = text.replace(\"PN\", \"<gap>\")\n    text = re.sub(r'(?<=\\s)-gold\\b', 'pašallum gold', text)\n    text = re.sub(r'(?<=\\s)-tax\\b', 'šadduātum tax', text)\n    text = re.sub(r'(?<=\\s)-textiles\\b', 'kutānum textiles', text)\n    text = _convert_twelfth_shekels(text)\n    text = _decimals_to_unicode(text)\n    text = _ascii_fracs_to_unicode(text)\n    text = replace_roman_months(text)\n    text = re.sub(r'(<\\s*gap\\s*>\\s*)+', '<gap> ', text)\n    return \" \".join(text.split())\n\ndef postprocess(text, transliteration=\"\"):\n    text = clean_translation(text)\n    text = re.sub(r'(\\w)šadduātum tax', r'\\1-tax', text)\n    text = re.sub(r'(\\w)pašallum gold', r'\\1-gold', text)\n    text = re.sub(r'(\\w)kutānum textiles', r'\\1-textiles', text)\n    text = text.replace('…', '<gap>')\n    text = re.sub(r'\\[gap\\]', '<gap>', text)\n    text = re.sub(r'(<\\s*gap\\s*>\\s*)+', '<gap> ', text)\n    if '<gap>' not in transliteration:\n        text = text.replace('<gap>', '')\n    text = \" \".join(text.split())\n    if len(text) == 0:\n        text = \"<gap>\"\n    return text\n
", "messages": [ { "id": "3427328", "postDate": "03/24/2026 00:57:19", "content": "

Solution Overview

\n

Notebook: https://www.kaggle.com/code/yeoyunsianggeremie/deep-past-inference

\n

Introduction

\n

Team members: @yeoyunsianggeremie @andrevma18

\n

We're both grinding for our startup, working 70-75 hour weeks, which left us almost no time to compete seriously. As such, we did the high-level thinking and solution-design ourselves (how to clean, how to generate data, etc) then distilled it into structured Markdown files for Qgentic-AI which is an autonomous Kaggle agent I have been building for the past 6 months. Qgentic-AI would run experiments iteratively throughout the week. We only spent 6 days grinding, which is very low given how unique and difficult this competition is.

\n

I almost said \"no\" to joining this competition - as I felt I wouldn't have enough time. But I realized, working under this intense constraint is something I am trying to actively get better at, and this would be a good practice!

\n

Models used

\n

Qwen2.5-32B, Qwen2.5-72B and ByT5-Base

\n

Datasets

\n

Published texts: Asked Gemini to split the data into sentences and generated translations for each of them\nPDFs: Fed each page into Gemini to extract transliteration/translation pairs

\n

Yup, you heard it right, we did not use the competition train data or attempt to clean it (I caught Qgentic-AI running in circles to decide whether to include it or not, and decided it was confusing so went against it)

\n

In total, we had about 55,000 transliteration/translation pairs after doing some manual cleaning and LLM-as-a-judge

\n

Training

\n

We post-trained Qwen2.5-32B and Qwen2.5-72B for 3 epochs on 8x B200 with LoRA rank 64 and alpha 128\nByT5 was trained for 15 epochs at a learning rate of 3e-4

\n

The parameters were tuned by Qgentic-AI (such a huge time saver 😁)

\n

We trained our models on ENG -> AKK and AKK -> ENG (so a total of ~110,000 training pairs)

\n

Validation

\n

We used a pertubed version of the train.csv removing all bad samples where the training and validation lengths are way off. There is still data leakage so we do not base 100% of our selections based on that. The CV is around 43 for our best models.

\n

Preprocessing

\n

Mostly based on the host's discussion - fixed a few bugs like 5/12 shekels -> 2/3 shekel 15 grains

\n

Postprocessing

\n

Removed repetitions using the base Qwen2.5-72B model using the prompt

\n
DEDUP_SYSTEM = \"You are a text editor. The user will give you an English translation that may contain repeated phrases or sentences. Remove any obvious repetitions while preserving the rest of the text exactly as-is. Do NOT rephrase, summarize, or add anything. Return ONLY the cleaned text.\"\n
\n

Failed Ideas

\n\n

Code for Preprocessing and Postprocessing

\n
import re\nimport pandas as pd\n\n\ndef replace_roman_months(text):\n    if pd.isna(text): return \"\"\n    text = str(text)\n    romans = {\n        \"XII\": \"12\", \"XI\": \"11\", \"IX\": \"9\", \"IV\": \"4\",\n        \"VIII\": \"8\", \"VII\": \"7\", \"III\": \"3\", \"VI\": \"6\",\n        \"X\": \"10\", \"V\": \"5\", \"II\": \"2\", \"I\": \"1\"\n    }\n    for r, num in romans.items():\n        text = re.sub(r'(?i)\\bmonth\\s+' + r + r'\\b', 'month ' + num, text)\n    return text\n\n_FRAC_MAP = {\n    0.5: \"½\", 0.3333: \"⅓\", 0.6666: \"⅔\", 0.6667: \"⅔\",\n    0.25: \"¼\", 0.75: \"¾\",\n    0.1666: \"⅙\", 0.1667: \"⅙\", 0.8333: \"⅚\",\n    0.125: \"⅛\", 0.375: \"⅜\", 0.625: \"⅝\", 0.875: \"⅞\",\n    0.2: \"⅕\", 0.4: \"⅖\", 0.6: \"⅗\", 0.8: \"⅘\",\n    0.333: \"⅓\",\n}\n\n_ASCII_FRAC_MAP = {\n    \"1/2\": \"½\", \"1/3\": \"⅓\", \"2/3\": \"⅔\",\n    \"1/4\": \"¼\", \"3/4\": \"¾\",\n    \"1/5\": \"⅕\", \"2/5\": \"⅖\", \"3/5\": \"⅗\", \"4/5\": \"⅘\",\n    \"1/6\": \"⅙\", \"5/6\": \"⅚\",\n    \"1/8\": \"⅛\", \"3/8\": \"⅜\", \"5/8\": \"⅝\", \"7/8\": \"⅞\",\n}\n\ndef _decimals_to_unicode(text):\n    def _replace(m):\n        val = float(m.group())\n        integer = int(val)\n        frac_part = round(val - integer, 4)\n        if frac_part == 0:\n            return m.group()\n        unicode_frac = _FRAC_MAP.get(frac_part)\n        if unicode_frac is None:\n            return m.group()\n        if integer == 0:\n            return unicode_frac\n        return f\"{integer} {unicode_frac}\"\n    return re.sub(r'(?<!\\d)\\d+\\.\\d+(?!\\d)', _replace, text)\n\ndef _ascii_fracs_to_unicode(text):\n    def _replace(m):\n        key = f\"{m.group(1)}/{m.group(2)}\"\n        return _ASCII_FRAC_MAP.get(key, m.group())\n    return re.sub(r'\\b(\\d+)\\s*/\\s*(\\d+)\\b', _replace, text)\n\ndef _convert_twelfth_shekels(text):\n    def _replace_11_12(m):\n        n = int(m.group(1))\n        return f\"{n + 1} shekels less 15 grains\"\n    text = re.sub(r'(\\d+)\\s+11\\s*/\\s*12\\s+shekels?', _replace_11_12, text)\n    twelfth_map = [\n        (\"7 / 12\",  \"½ shekel 15 grains\"),\n        (\"7/12\",    \"½ shekel 15 grains\"),\n        (\"5 / 12\",  \"⅓ shekel 15 grains\"),\n        (\"5/12\",    \"⅓ shekel 15 grains\"),\n        (\"1 / 12\",  \"15 grains\"),\n        (\"1/12\",    \"15 grains\"),\n    ]\n    for num, replacement in twelfth_map:\n        text = re.sub(\n            re.escape(num) + r'\\s*(?:\\(shekel\\)|shekels?)',\n            replacement,\n            text\n        )\n    return text\n\ndef clean_transliteration(text):\n    if pd.isna(text): return \"\"\n    text = str(text)\n    subscript_map = str.maketrans(\"₀₁₂₃₄₅₆₇₈₉\", \"0123456789\")\n    text = text.translate(subscript_map)\n    text = text.replace(\"Ḫ\", \"H\").replace(\"ḫ\", \"h\")\n    text = text.replace(\"KÙ.B.\", \"KÙ.BABBAR\")\n    for var in [r'\\[x\\]', r'\\[\\.\\.\\.\\.\\.\\.\\]', r'\\[\\.\\.\\. \\.\\.\\.\\]', r'\\.\\.\\.', r'\\(break\\)', r'\\(large break\\)', r'\\(n broken lines\\)', r'xₓ']:\n        text = re.sub(var, ' <gap> ', text)\n    text = _decimals_to_unicode(text)\n    text = _ascii_fracs_to_unicode(text)\n    text = replace_roman_months(text)\n    text = re.sub(r'(<\\s*gap\\s*>\\s*)+', '<gap> ', text)\n    return \" \".join(text.split())\n\ndef clean_translation(text):\n    if pd.isna(text): return \"\"\n    text = str(text)\n    for var in [r'\\[x\\]', r'\\[\\.\\.\\.\\.\\.\\.\\]', r'\\[\\.\\.\\. \\.\\.\\.\\]', r'\\.\\.\\.', r'\\(break\\)', r'\\(large break\\)', r'\\(n broken lines\\)', r'xₓ']:\n        text = re.sub(var, ' <gap> ', text)\n    for r_mark in [r'\\bfem\\.', r'\\bsing\\.', r'\\bpl\\.', r'\\bplural\\b', r'\\(\\?\\)', r'\\.\\.,', r'\\bxx\\b', r'\\bx\\b', r'<<\\s*>>', r'<\\s*>', r'\\[\\s*\\]', r'˹\\s*˺']:\n        text = re.sub(r_mark, '', text)\n    text = text.replace(\"PN\", \"<gap>\")\n    text = re.sub(r'(?<=\\s)-gold\\b', 'pašallum gold', text)\n    text = re.sub(r'(?<=\\s)-tax\\b', 'šadduātum tax', text)\n    text = re.sub(r'(?<=\\s)-textiles\\b', 'kutānum textiles', text)\n    text = _convert_twelfth_shekels(text)\n    text = _decimals_to_unicode(text)\n    text = _ascii_fracs_to_unicode(text)\n    text = replace_roman_months(text)\n    text = re.sub(r'(<\\s*gap\\s*>\\s*)+', '<gap> ', text)\n    return \" \".join(text.split())\n\ndef postprocess(text, transliteration=\"\"):\n    text = clean_translation(text)\n    text = re.sub(r'(\\w)šadduātum tax', r'\\1-tax', text)\n    text = re.sub(r'(\\w)pašallum gold', r'\\1-gold', text)\n    text = re.sub(r'(\\w)kutānum textiles', r'\\1-textiles', text)\n    text = text.replace('…', '<gap>')\n    text = re.sub(r'\\[gap\\]', '<gap>', text)\n    text = re.sub(r'(<\\s*gap\\s*>\\s*)+', '<gap> ', text)\n    if '<gap>' not in transliteration:\n        text = text.replace('<gap>', '')\n    text = \" \".join(text.split())\n    if len(text) == 0:\n        text = \"<gap>\"\n    return text\n
", "rawMarkdown": "# Solution Overview\n\nNotebook: https://www.kaggle.com/code/yeoyunsianggeremie/deep-past-inference\n\n## Introduction\n\nTeam members: @yeoyunsianggeremie @andrevma18\n\nWe're both grinding for our startup, working 70-75 hour weeks, which left us almost no time to compete seriously. As such, we did the high-level thinking and solution-design ourselves (how to clean, how to generate data, etc) then distilled it into structured Markdown files for [Qgentic-AI](https://github.com/bogoconic1/Qgentic-AI) which is an autonomous Kaggle agent I have been building for the past 6 months. Qgentic-AI would run experiments iteratively throughout the week. We only spent 6 days grinding, which is very low given how unique and difficult this competition is.\n\nI almost said \"no\" to joining this competition - as I felt I wouldn't have enough time. But I realized, working under this intense constraint is something I am trying to actively get better at, and this would be a good practice!\n\n\n## Models used\n\nQwen2.5-32B, Qwen2.5-72B and ByT5-Base\n\n## Datasets\n\nPublished texts: Asked Gemini to split the data into sentences and generated translations for each of them\nPDFs: Fed each page into Gemini to extract transliteration/translation pairs \n\nYup, you heard it right, we did **not use** the competition train data or attempt to clean it (I caught Qgentic-AI running in circles to decide whether to include it or not, and decided it was confusing so went against it)\n\nIn total, we had about 55,000 transliteration/translation pairs after doing some manual cleaning and LLM-as-a-judge\n\n## Training \n\nWe post-trained Qwen2.5-32B and Qwen2.5-72B for 3 epochs on 8x B200 with LoRA rank 64 and alpha 128\nByT5 was trained for 15 epochs at a learning rate of 3e-4\n\nThe parameters were tuned by Qgentic-AI (such a huge time saver 😁)\n\nWe trained our models on ENG -> AKK and AKK -> ENG (so a total of ~110,000 training pairs)\n\n## Validation\n\nWe used a pertubed version of the train.csv removing all bad samples where the training and validation lengths are way off. There is still data leakage so we do not base 100% of our selections based on that. The CV is around 43 for our best models.\n\n## Preprocessing\n\nMostly based on the host's [discussion](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/678899) - fixed a few bugs like 5/12 shekels -> 2/3 shekel 15 grains\n\n## Postprocessing\n\nRemoved repetitions using the base Qwen2.5-72B model using the prompt\n```\nDEDUP_SYSTEM = \"You are a text editor. The user will give you an English translation that may contain repeated phrases or sentences. Remove any obvious repetitions while preserving the rest of the text exactly as-is. Do NOT rephrase, summarize, or add anything. Return ONLY the cleaned text.\"\n```\n\n## Failed Ideas\n\n- Any ByT5 model >= Large. We haven't investigated why.\n- MBR\n- Using competition training data\n- Translating Akkadian -> Arabic -> English\n- Training a judge model to grade the quality of the translations\n\n## Code for Preprocessing and Postprocessing\n\n```\nimport re\nimport pandas as pd\n\n\ndef replace_roman_months(text):\n if pd.isna(text): return \"\"\n text = str(text)\n romans = {\n \"XII\": \"12\", \"XI\": \"11\", \"IX\": \"9\", \"IV\": \"4\",\n \"VIII\": \"8\", \"VII\": \"7\", \"III\": \"3\", \"VI\": \"6\",\n \"X\": \"10\", \"V\": \"5\", \"II\": \"2\", \"I\": \"1\"\n }\n for r, num in romans.items():\n text = re.sub(r'(?i)\\bmonth\\s+' + r + r'\\b', 'month ' + num, text)\n return text\n\n_FRAC_MAP = {\n 0.5: \"½\", 0.3333: \"⅓\", 0.6666: \"⅔\", 0.6667: \"⅔\",\n 0.25: \"¼\", 0.75: \"¾\",\n 0.1666: \"⅙\", 0.1667: \"⅙\", 0.8333: \"⅚\",\n 0.125: \"⅛\", 0.375: \"⅜\", 0.625: \"⅝\", 0.875: \"⅞\",\n 0.2: \"⅕\", 0.4: \"⅖\", 0.6: \"⅗\", 0.8: \"⅘\",\n 0.333: \"⅓\",\n}\n\n_ASCII_FRAC_MAP = {\n \"1/2\": \"½\", \"1/3\": \"⅓\", \"2/3\": \"⅔\",\n \"1/4\": \"¼\", \"3/4\": \"¾\",\n \"1/5\": \"⅕\", \"2/5\": \"⅖\", \"3/5\": \"⅗\", \"4/5\": \"⅘\",\n \"1/6\": \"⅙\", \"5/6\": \"⅚\",\n \"1/8\": \"⅛\", \"3/8\": \"⅜\", \"5/8\": \"⅝\", \"7/8\": \"⅞\",\n}\n\ndef _decimals_to_unicode(text):\n def _replace(m):\n val = float(m.group())\n integer = int(val)\n frac_part = round(val - integer, 4)\n if frac_part == 0:\n return m.group()\n unicode_frac = _FRAC_MAP.get(frac_part)\n if unicode_frac is None:\n return m.group()\n if integer == 0:\n return unicode_frac\n return f\"{integer} {unicode_frac}\"\n return re.sub(r'(? ', text)\n text = _decimals_to_unicode(text)\n text = _ascii_fracs_to_unicode(text)\n text = replace_roman_months(text)\n text = re.sub(r'(<\\s*gap\\s*>\\s*)+', ' ', text)\n return \" \".join(text.split())\n\ndef clean_translation(text):\n if pd.isna(text): return \"\"\n text = str(text)\n for var in [r'\\[x\\]', r'\\[\\.\\.\\.\\.\\.\\.\\]', r'\\[\\.\\.\\. \\.\\.\\.\\]', r'\\.\\.\\.', r'\\(break\\)', r'\\(large break\\)', r'\\(n broken lines\\)', r'xₓ']:\n text = re.sub(var, ' ', text)\n for r_mark in [r'\\bfem\\.', r'\\bsing\\.', r'\\bpl\\.', r'\\bplural\\b', r'\\(\\?\\)', r'\\.\\.,', r'\\bxx\\b', r'\\bx\\b', r'<<\\s*>>', r'<\\s*>', r'\\[\\s*\\]', r'˹\\s*˺']:\n text = re.sub(r_mark, '', text)\n text = text.replace(\"PN\", \"\")\n text = re.sub(r'(?<=\\s)-gold\\b', 'pašallum gold', text)\n text = re.sub(r'(?<=\\s)-tax\\b', 'šadduātum tax', text)\n text = re.sub(r'(?<=\\s)-textiles\\b', 'kutānum textiles', text)\n text = _convert_twelfth_shekels(text)\n text = _decimals_to_unicode(text)\n text = _ascii_fracs_to_unicode(text)\n text = replace_roman_months(text)\n text = re.sub(r'(<\\s*gap\\s*>\\s*)+', ' ', text)\n return \" \".join(text.split())\n\ndef postprocess(text, transliteration=\"\"):\n text = clean_translation(text)\n text = re.sub(r'(\\w)šadduātum tax', r'\\1-tax', text)\n text = re.sub(r'(\\w)pašallum gold', r'\\1-gold', text)\n text = re.sub(r'(\\w)kutānum textiles', r'\\1-textiles', text)\n text = text.replace('…', '')\n text = re.sub(r'\\[gap\\]', '', text)\n text = re.sub(r'(<\\s*gap\\s*>\\s*)+', ' ', text)\n if '' not in transliteration:\n text = text.replace('', '')\n text = \" \".join(text.split())\n if len(text) == 0:\n text = \"\"\n return text\n```", "votes": null }, { "id": "3427330", "postDate": "03/24/2026 01:05:19", "content": "

Quick Question : how much time it took for you to train Qwen 2.5 32/72b

", "rawMarkdown": "Quick Question : how much time it took for you to train Qwen 2.5 32/72b", "votes": null }, { "id": "3427337", "postDate": "03/24/2026 01:16:27", "content": "

Congratulations! Feels like it's time to start embracing LLM-powered, semi-autonomous agents in Kaggle conpetitions!

", "rawMarkdown": "Congratulations! Feels like it's time to start embracing LLM-powered, semi-autonomous agents in Kaggle conpetitions!", "votes": null }, { "id": "3427347", "postDate": "03/24/2026 01:30:18", "content": "

32B:

\n
    \n
  • single H100: about 15 hours
    • \n
  • 8x B200: 2 hours
    • \n
\n

72B:

\n
    \n
  • I did not manage to train it on 1x H100 even with 4-bit QLoRA
    • \n
  • 8x B200: 4 hours
    • \n
", "rawMarkdown": "32B:\n- single H100: about 15 hours\n- 8x B200: 2 hours\n\n72B:\n- I did not manage to train it on 1x H100 even with 4-bit QLoRA\n- 8x B200: 4 hours", "votes": null }, { "id": "3427393", "postDate": "03/24/2026 03:10:42", "content": "

one question, how do you make Qwen 2.5 32/72b to do infer in kaggle env with P100/T4/TPU? is the GPU MEM able to load model of such size?

", "rawMarkdown": "one question, how do you make Qwen 2.5 32/72b to do infer in kaggle env with P100/T4/TPU? is the GPU MEM able to load model of such size?", "votes": null }, { "id": "3427396", "postDate": "03/24/2026 03:18:08", "content": "

I didnt use Kaggle for training. I used a single H100 for most of my experiments\n(same for my teammate)

", "rawMarkdown": "I didnt use Kaggle for training. I used a single H100 for most of my experiments\n(same for my teammate)", "votes": null }, { "id": "3427403", "postDate": "03/24/2026 03:33:43", "content": "

yeah, I got that. I'm wondering the inference part, the submitted notebook, how to make Qwen 32/72b work on the limited GPU MEM

", "rawMarkdown": "yeah, I got that. I'm wondering the inference part, the submitted notebook, how to make Qwen 32/72b work on the limited GPU MEM", "votes": null }, { "id": "3427427", "postDate": "03/24/2026 04:10:52", "content": "

Grinding startups and Kaggle at the same time is a wild mindset! Good luck with both!

", "rawMarkdown": "Grinding startups and Kaggle at the same time is a wild mindset! Good luck with both!", "votes": null } ], "comments": [ { "id": 3427330, "author_name": "aman1391", "author_url": "", "post_date": "03/24/2026 01:05:19", "content": "

Quick Question : how much time it took for you to train Qwen 2.5 32/72b

", "votes": null, "replies": [ { "id": 3427347, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "03/24/2026 01:30:18", "content": "

32B:

\n
    \n
  • single H100: about 15 hours
    • \n
  • 8x B200: 2 hours
    • \n
\n

72B:

\n
    \n
  • I did not manage to train it on 1x H100 even with 4-bit QLoRA
    • \n
  • 8x B200: 4 hours
    • \n
", "votes": null, "replies": [] } ] }, { "id": 3427337, "author_name": "ducnh279", "author_url": "", "post_date": "03/24/2026 01:16:27", "content": "

Congratulations! Feels like it's time to start embracing LLM-powered, semi-autonomous agents in Kaggle conpetitions!

", "votes": null, "replies": [] }, { "id": 3427393, "author_name": "dreamweaver", "author_url": "", "post_date": "03/24/2026 03:10:42", "content": "

one question, how do you make Qwen 2.5 32/72b to do infer in kaggle env with P100/T4/TPU? is the GPU MEM able to load model of such size?

", "votes": null, "replies": [ { "id": 3427396, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "03/24/2026 03:18:08", "content": "

I didnt use Kaggle for training. I used a single H100 for most of my experiments\n(same for my teammate)

", "votes": null, "replies": [ { "id": 3427403, "author_name": "dreamweaver", "author_url": "", "post_date": "03/24/2026 03:33:43", "content": "

yeah, I got that. I'm wondering the inference part, the submitted notebook, how to make Qwen 32/72b work on the limited GPU MEM

", "votes": null, "replies": [] } ] } ] }, { "id": 3427427, "author_name": "samson8", "author_url": "", "post_date": "03/24/2026 04:10:52", "content": "

Grinding startups and Kaggle at the same time is a wild mindset! Good luck with both!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3427328": "# Solution Overview\n\nNotebook: https://www.kaggle.com/code/yeoyunsianggeremie/deep-past-inference\n\n## Introduction\n\nTeam members: @yeoyunsianggeremie @andrevma18\n\nWe're both grinding for our startup, working 70-75 hour weeks, which left us almost no time to compete seriously. As such, we did the high-level thinking and solution-design ourselves (how to clean, how to generate data, etc) then distilled it into structured Markdown files for [Qgentic-AI](https://github.com/bogoconic1/Qgentic-AI) which is an autonomous Kaggle agent I have been building for the past 6 months. Qgentic-AI would run experiments iteratively throughout the week. We only spent 6 days grinding, which is very low given how unique and difficult this competition is.\n\nI almost said \"no\" to joining this competition - as I felt I wouldn't have enough time. But I realized, working under this intense constraint is something I am trying to actively get better at, and this would be a good practice!\n\n\n## Models used\n\nQwen2.5-32B, Qwen2.5-72B and ByT5-Base\n\n## Datasets\n\nPublished texts: Asked Gemini to split the data into sentences and generated translations for each of them\nPDFs: Fed each page into Gemini to extract transliteration/translation pairs \n\nYup, you heard it right, we did **not use** the competition train data or attempt to clean it (I caught Qgentic-AI running in circles to decide whether to include it or not, and decided it was confusing so went against it)\n\nIn total, we had about 55,000 transliteration/translation pairs after doing some manual cleaning and LLM-as-a-judge\n\n## Training \n\nWe post-trained Qwen2.5-32B and Qwen2.5-72B for 3 epochs on 8x B200 with LoRA rank 64 and alpha 128\nByT5 was trained for 15 epochs at a learning rate of 3e-4\n\nThe parameters were tuned by Qgentic-AI (such a huge time saver 😁)\n\nWe trained our models on ENG -> AKK and AKK -> ENG (so a total of ~110,000 training pairs)\n\n## Validation\n\nWe used a pertubed version of the train.csv removing all bad samples where the training and validation lengths are way off. There is still data leakage so we do not base 100% of our selections based on that. The CV is around 43 for our best models.\n\n## Preprocessing\n\nMostly based on the host's [discussion](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/678899) - fixed a few bugs like 5/12 shekels -> 2/3 shekel 15 grains\n\n## Postprocessing\n\nRemoved repetitions using the base Qwen2.5-72B model using the prompt\n```\nDEDUP_SYSTEM = \"You are a text editor. The user will give you an English translation that may contain repeated phrases or sentences. Remove any obvious repetitions while preserving the rest of the text exactly as-is. Do NOT rephrase, summarize, or add anything. Return ONLY the cleaned text.\"\n```\n\n## Failed Ideas\n\n- Any ByT5 model >= Large. We haven't investigated why.\n- MBR\n- Using competition training data\n- Translating Akkadian -> Arabic -> English\n- Training a judge model to grade the quality of the translations\n\n## Code for Preprocessing and Postprocessing\n\n```\nimport re\nimport pandas as pd\n\n\ndef replace_roman_months(text):\n if pd.isna(text): return \"\"\n text = str(text)\n romans = {\n \"XII\": \"12\", \"XI\": \"11\", \"IX\": \"9\", \"IV\": \"4\",\n \"VIII\": \"8\", \"VII\": \"7\", \"III\": \"3\", \"VI\": \"6\",\n \"X\": \"10\", \"V\": \"5\", \"II\": \"2\", \"I\": \"1\"\n }\n for r, num in romans.items():\n text = re.sub(r'(?i)\\bmonth\\s+' + r + r'\\b', 'month ' + num, text)\n return text\n\n_FRAC_MAP = {\n 0.5: \"½\", 0.3333: \"⅓\", 0.6666: \"⅔\", 0.6667: \"⅔\",\n 0.25: \"¼\", 0.75: \"¾\",\n 0.1666: \"⅙\", 0.1667: \"⅙\", 0.8333: \"⅚\",\n 0.125: \"⅛\", 0.375: \"⅜\", 0.625: \"⅝\", 0.875: \"⅞\",\n 0.2: \"⅕\", 0.4: \"⅖\", 0.6: \"⅗\", 0.8: \"⅘\",\n 0.333: \"⅓\",\n}\n\n_ASCII_FRAC_MAP = {\n \"1/2\": \"½\", \"1/3\": \"⅓\", \"2/3\": \"⅔\",\n \"1/4\": \"¼\", \"3/4\": \"¾\",\n \"1/5\": \"⅕\", \"2/5\": \"⅖\", \"3/5\": \"⅗\", \"4/5\": \"⅘\",\n \"1/6\": \"⅙\", \"5/6\": \"⅚\",\n \"1/8\": \"⅛\", \"3/8\": \"⅜\", \"5/8\": \"⅝\", \"7/8\": \"⅞\",\n}\n\ndef _decimals_to_unicode(text):\n def _replace(m):\n val = float(m.group())\n integer = int(val)\n frac_part = round(val - integer, 4)\n if frac_part == 0:\n return m.group()\n unicode_frac = _FRAC_MAP.get(frac_part)\n if unicode_frac is None:\n return m.group()\n if integer == 0:\n return unicode_frac\n return f\"{integer} {unicode_frac}\"\n return re.sub(r'(? ', text)\n text = _decimals_to_unicode(text)\n text = _ascii_fracs_to_unicode(text)\n text = replace_roman_months(text)\n text = re.sub(r'(<\\s*gap\\s*>\\s*)+', ' ', text)\n return \" \".join(text.split())\n\ndef clean_translation(text):\n if pd.isna(text): return \"\"\n text = str(text)\n for var in [r'\\[x\\]', r'\\[\\.\\.\\.\\.\\.\\.\\]', r'\\[\\.\\.\\. \\.\\.\\.\\]', r'\\.\\.\\.', r'\\(break\\)', r'\\(large break\\)', r'\\(n broken lines\\)', r'xₓ']:\n text = re.sub(var, ' ', text)\n for r_mark in [r'\\bfem\\.', r'\\bsing\\.', r'\\bpl\\.', r'\\bplural\\b', r'\\(\\?\\)', r'\\.\\.,', r'\\bxx\\b', r'\\bx\\b', r'<<\\s*>>', r'<\\s*>', r'\\[\\s*\\]', r'˹\\s*˺']:\n text = re.sub(r_mark, '', text)\n text = text.replace(\"PN\", \"\")\n text = re.sub(r'(?<=\\s)-gold\\b', 'pašallum gold', text)\n text = re.sub(r'(?<=\\s)-tax\\b', 'šadduātum tax', text)\n text = re.sub(r'(?<=\\s)-textiles\\b', 'kutānum textiles', text)\n text = _convert_twelfth_shekels(text)\n text = _decimals_to_unicode(text)\n text = _ascii_fracs_to_unicode(text)\n text = replace_roman_months(text)\n text = re.sub(r'(<\\s*gap\\s*>\\s*)+', ' ', text)\n return \" \".join(text.split())\n\ndef postprocess(text, transliteration=\"\"):\n text = clean_translation(text)\n text = re.sub(r'(\\w)šadduātum tax', r'\\1-tax', text)\n text = re.sub(r'(\\w)pašallum gold', r'\\1-gold', text)\n text = re.sub(r'(\\w)kutānum textiles', r'\\1-textiles', text)\n text = text.replace('…', '')\n text = re.sub(r'\\[gap\\]', '', text)\n text = re.sub(r'(<\\s*gap\\s*>\\s*)+', ' ', text)\n if '' not in transliteration:\n text = text.replace('', '')\n text = \" \".join(text.split())\n if len(text) == 0:\n text = \"\"\n return text\n```", "3427330": "Quick Question : how much time it took for you to train Qwen 2.5 32/72b", "3427337": "Congratulations! Feels like it's time to start embracing LLM-powered, semi-autonomous agents in Kaggle conpetitions!", "3427347": "32B:\n- single H100: about 15 hours\n- 8x B200: 2 hours\n\n72B:\n- I did not manage to train it on 1x H100 even with 4-bit QLoRA\n- 8x B200: 4 hours", "3427393": "one question, how do you make Qwen 2.5 32/72b to do infer in kaggle env with P100/T4/TPU? is the GPU MEM able to load model of such size?", "3427396": "I didnt use Kaggle for training. I used a single H100 for most of my experiments\n(same for my teammate)", "3427403": "yeah, I got that. I'm wondering the inference part, the submitted notebook, how to make Qwen 32/72b work on the limited GPU MEM", "3427427": "Grinding startups and Kaggle at the same time is a wild mindset! Good luck with both!" }, "source": "existing_discussion", "complete": true, "competition_id": 121150, "competition_slug": "deep-past-initiative-machine-translation", "team_id": 15319166, "solution_url": "/competitions/deep-past-initiative-machine-translation/writeups/25th-post-training-qwen2-5-32b-and-72b-with-gemi" }