{
  "id": 684274,
  "title": "68th Place Solution🥈",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/684274",
  "author_name": "kyotoman",
  "post_date": "2026-03-24T09:25:04.733000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I joined this competition late, and up until three days before the submission deadline I was mostly focused on cleaning up and curating <code>train.csv</code>.\nBecause of that, I didn’t have enough time to properly verify local CV, and in the end I had to rely on LB score and gut feeling, which is something I regret.</p>\n<h1>Cleaning up <code>train.csv</code></h1>\n<p>The original <code>train.csv</code> had 1,561 records, but after doing the following work I expanded it to 2,600 records in the end:</p>\n<ul>\n<li>I visually inspected the PDF page by page and added transliteration/translation pairs that were missing from <code>train.csv</code> (I found 27 missing items).</li>\n<li>I split texts that were too long to fit into the ByT5 input window, while paying attention to the alignment between the transliteration and the translation (I did this after fixing the issues in <code>train.csv</code> described below).</li>\n</ul>\n<p>I also fixed issues like the following in <code>train.csv</code>:</p>\n<ul>\n<li>PDF OCR mistakes (cases where either the transliteration or the translation was completely wrong)</li>\n<li>I restored records where the transliteration text was truncated in the host-provided <code>train.csv</code> (this was relatively easy to recover by comparing with <code>published_texts.csv</code>).</li>\n<li>I excluded some long records with so many <code>&lt;gap&gt;</code> tokens that splitting the transliteration and translation was not feasible, since they were likely harmful for model training (surprisingly, I found only 3 such records).</li>\n</ul>\n<p>I also found issues like the following in many places, but since the volume was too large and I suspected that similar issues existed in <code>test.csv</code> used for the final evaluation, I intentionally left them as-is:</p>\n<ul>\n<li>Italic text in the PDF was not captured by OCR</li>\n</ul>\n<p>For fixing <code>train.csv</code> using the PDF, I used a very primitive method: taking screenshots of the relevant parts and having ChatGPT transcribe them.</p>\n<p><em>For transcription</em></p>\n<pre><code>&gt; NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\nThe **screenshot images** I’m going to send contain **Akkadian transliteration** on the left (or top) and the corresponding **English translation** on the right (or bottom).  \nPlease **transcribe both accurately** and return them in a format that is **easy to copy &amp; paste into a spreadsheet**.\n\nIn some cases, a single screenshot might not be enough and the content may be split across multiple screenshots. In that case, please infer the correct order of the screenshots and transcribe accordingly.\n\n---\n\n## Response format (copy/paste first)\nIn principle, please reply every time with the following 2-line set (no extra explanation needed):\n\ntransliteration (Akkadian text) &lt;newline&gt;\n&lt;separator&gt;\ntranslation (English text)\n\n- `&lt;newline&gt;` means a line break in the ChatGPT UI; `&lt;separator&gt;` also means a line break in the ChatGPT UI.\n- Keep the Akkadian text and the English text to **one line each**.\n\n---\n\n## Handling body text vs. notes (important)\n- The image may contain lines other than the “body text” (e.g., notes, headings, layout notes, page numbers, line numbers), such as `Seal A`, `rev.`, `obv.`, `(remainder obv. missing)`, `(3.5 cm un-inscribed)`, etc.\n- Please **extract only the body text** and **exclude note-like lines** from the transcription output.\n\nExample output (when NOTE-like lines are present):\nAkkadian: transcribed transliteration (body) &lt;newline&gt;\n&lt;newline&gt;\nEnglish: transcribed translation (body)\n\n---\n\n## Transcription rules (important)\n### 1) Normalization / replacements (required)\nFor expressions indicating missing/damaged/illegible text, if the meaning is the same, **unify everything to `&lt;gap&gt;`**.\n\n- `x` → `&lt;gap&gt;`\n- `[x]` → `&lt;gap&gt;`\n- `…` (ellipsis) → `&lt;gap&gt;`\n- `(break)` → `&lt;gap&gt;`\n- `(large break)` → `&lt;gap&gt;`\n- `(n broken lines)` → `&lt;gap&gt;` (it’s OK to use `&lt;gap&gt;` even when `n` is a number)\n- If `&lt;gap&gt;` appears multiple times consecutively, merge into one; i.e., `&lt;gap&gt; &lt;gap&gt;` → `&lt;gap&gt;`\n- **Any other “missing/broken/unreadable” indicators** → `&lt;gap&gt;`\n\nFor determinatives, normalize as follows:\n\n- `(d)` → `{d}`\n- `(ki)` → `{ki}`\n\nFor parenthesized words:\n\n- `(TÚG)` → `TÚG` (remove the parentheses)\n\n---\n\n## 2) Things to keep as-is\n- Keep uppercase/lowercase, apostrophes, hyphens, subscript numerals (e.g., `₂`), special characters (e.g., `š ṣ ṭ ḫ`), accents, etc. **exactly as shown**\n- Keep symbols/separators (like `-` and `.`) **as in the image**, except where the replacement rules apply\n- `&lt;gap&gt;` must always be exactly `&lt;gap&gt;` in **lowercase**\n\n---\n\n## 3) Uncertainty\n- If there is a character/word you truly cannot identify, you may replace that part with `&lt;gap&gt;`\n- However, preserve whatever you can read (don’t turn everything into `&lt;gap&gt;`)\n\n---\n\n## Minimal competition context (for clarity)\n- Input: Akkadian (cuneiform) transliteration = transliteration\n- Output: its English translation = translation\n- The goal is to create transliteration → English pairs, so please extract them as **two columns of text**\n\n---\n\n## Requests to make the work faster and more accurate\n- If possible, extract the “transliteration part” and the “English translation part” separately so they **don’t get mixed**\n- Since punctuation and missing closing parentheses are common errors, do a quick check that the left/right (or top/bottom) alignment hasn’t drifted before you output\n- The output is intended to be pasted into a spreadsheet, so please **avoid extra decoration** (bullets, block quotes, etc.)\n</code></pre>\n<p><em>For splitting long texts with alignment preserved</em></p>\n<pre><code>&gt; NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\n\nThe text and screenshot images I’m going to send are candidate training data for the Kaggle competition **Deep Past Challenge - Translate Akkadian to English**.  \nFor a single long record, please split it into **a small number (around 2–5) of independent records**, while being careful not to break the alignment between the **Akkadian transliteration** and the **English translation**.\n\nThe goal is to make each record easier to handle with a ByT5 model by keeping it **under 1024 bytes**.  \nHowever, if splitting would likely break the alignment, it is better to return it as “cannot split” than to over-split.\n\n---\n\n## Most important policy (v2: don’t over-split)\n\n- Target number of splits: **2–5** (prefer **2–4** if possible)\n- Split only when necessary\n  - Add extra split boundaries only when they are truly needed to keep chunks under 1024 bytes or to preserve alignment\n- Avoid producing many very short chunks\n  - If you can’t count bytes exactly, err on the safe side, but **don’t mass-produce tiny chunks**\n  - As a rough guide, aim for **large chunks (roughly 700–1000 bytes)** when possible\n- If it still won’t fit within 5 splits or alignment would break, return `needs_manual_review` (do not force 6+ splits)\n\n---\n\n## Background (minimal)\n\n- In this competition, we train a model to generate English translations from Akkadian transliterations.\n- ByT5 is byte-level, so **UTF-8 byte length** matters more than character count.\n- On the source side, we usually add the prefix `translate Akkadian to English: `.\n- In practice, we want both source and target to be **under 1024 bytes** per record.\n- You don’t need to compute byte length perfectly every time. If unsure, make it a bit shorter for safety.\n\n---\n\n## What I want you to do\n\n1. Read the given `transliteration` and `translation`\n2. Refer to the screenshot images if needed\n3. Find only a small number of boundaries where the **semantic alignment is clear** (don’t add too many)\n4. Split the record into **2–5** `transliteration_chunk` / `translation_chunk` pairs (choose the minimum split that works)\n5. If splitting is risky, return `needs_manual_review`\n\n---\n\n## Absolute rules\n\n### 1) This is “splitting”, not “editing”\n\n- For the body text, use the input text as-is and, in principle, only take **substrings** from the provided `transliteration` / `translation`\n- Do not fill in missing text, paraphrase, summarize, or re-translate\n- Even if you notice OCR mistakes or unnatural English, **do not fix them in this task**\n\n### 2) Preserve the order\n\n- When you stack the chunks from top to bottom, they must preserve the original order of the `transliteration` / `translation`\n\n### 3) Only cut at boundaries visible on both sides\n\n- Sentence/clause boundaries on the English side\n- Boundaries between itemized lists\n- Breaks in letter templates / formulaic phrases\n- Breaks between imperative/conditional sections\n- Clear line-group boundaries on the transliteration side\n\nOnly cut at places where a coherent “unit” is visible on both sides.\n\n### 4) Do not cut in the middle of these\n\n- Numbers, fractions, decimals, tax rates\n- Amount expressions like shekel / mina groupings\n- Personal names, divine names, place names\n- Phrases like `son of ...` or `belonging to ...`\n- Inside quotation marks\n- Inside an item name\n\n### 5) If it’s not safe, hold it\n\nIn the following cases, do not force a split; return `needs_manual_review`:\n\n- Even with the images, the aligned boundary cannot be determined\n- The English translation in train looks broken in the middle\n- The source/target are misaligned and 1:1 correspondence is unclear\n- Keeping it within 2–5 splits would inevitably break numeric/proper-noun sequences\n\n---\n\n## How to use the images\n\nUse the screenshots to decide **where to cut so that transliteration and translation stay aligned**.\n\nImportant:\n\n- The Akkadian and English text in the PDF may not exactly match the text provided in the chat.\n- This may be because the chat text has already been normalized (e.g., `&lt;gap&gt;` replacements, symbol cleanup, etc.).\n- Therefore, use the PDF as a reference to find alignment boundaries, but for the final output body text, **prefer the chat-provided text**.\n- Do not rewrite the body text to match the PDF’s visual appearance.\n\nPriority when using images:\n\n1. Verify alignment boundaries between `transliteration` and `translation`\n2. Resolve ambiguous boundaries that are hard to see from text alone\n3. Decide what spans form a coherent aligned unit\n\nDo not create new translations even if you use the images.\n\n---\n\n## Boundary heuristics (for a small number of splits)\n\n### Good candidates (high priority)\n\n- After an introductory formula like `a-na ... qí-bi-ma um-ma ...` (but don’t split into a tiny chunk just for the intro)\n- English expressions that indicate a section change, e.g. `all this ...`, `thereof ...`, `therefore ...`, `if ... then ...`\n- Places where a large block of an item list changes (avoid splitting each small item)\n- Where repetition of the same pattern ends as a block\n\n### Tempting but usually forbidden\n\n- Splitting every English sentence (often creates too many chunks)\n- Splitting every small list item (avoid)\n\n---\n\n## Output format (strict)\n\nReturn **only a JSON array**. No extra explanation text.\n\n### When splitting is possible (2–5 chunks)\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__01\",\n    \"status\": \"ok\",\n    \"transliteration_chunk\": \"split transliteration chunk (must be a substring of the input)\",\n    \"translation_chunk\": \"corresponding translation chunk (must be a substring of the input)\",\n    \"boundary_reason\": \"one sentence describing why you cut here (alignment rationale)\",\n    \"confidence\": \"high\"\n  }\n]\n```\n\n### When splitting is risky\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__00\",\n    \"status\": \"needs_manual_review\",\n    \"transliteration_chunk\": \"\",\n    \"translation_chunk\": \"\",\n    \"boundary_reason\": \"cannot cut into 2–5 chunks without breaking alignment\",\n    \"confidence\": \"low\"\n  }\n]\n```\n\n---\n\n## `confidence` guide\n\n- `high`: boundaries are very clear from both images and text\n- `medium`: mostly reasonable, but alignment is slightly weak on one side\n- `low`: the original train text may be missing/broken/misaligned\n\n---\n\n## When you receive an actual request, always follow these\n\n- Return only a JSON array\n- Strongly prefer 2–5 splits and do not add too many boundaries\n- Remember this is splitting, not editing\n- Do not add words that are not in the provided body text, even if you look at images\n- If unsure, return `needs_manual_review`\n</code></pre>\n<p>At first I tried to clean up <code>train.csv</code> using Codex agents and the skills feature, but I couldn’t control it well, and in the end I did it manually.</p>\n<h1>Final submission pipeline</h1>\n<p>I started from the public notebook “<a href=\"https://www.kaggle.com/code/waterjoe/lb-35-9-ensembling-post-processing-baseline\" target=\"_blank\">LB 35.9 Ensembling &amp; Post Processing Baseline</a>”, then increased the number of models and made small changes to preprocessing and postprocessing.\nThe last two submissions differed only in the number of training epochs; despite concerns about overfitting, the one trained for more epochs achieved a higher Private LB score.</p>\n<h2>Preprocessing / Postprocessing</h2>\n<h3>Preprocessing</h3>\n<p>Both preprocessing and postprocessing are differences from “LB 35.9 Ensembling &amp; Post Processing Baseline” (only the differences are listed here).</p>\n<ul>\n<li>Expanded fraction/decimal normalization (replacing baseline <code>_EXACT_FRAC_RE</code> + <code>_canon_decimal(float)</code>):<ul>\n<li>Convert fraction expressions like <code>1/2</code> and <code>2 1/2</code> into Unicode vulgar fractions (e.g., <code>½</code>)</li>\n<li>For decimals, map them to Unicode fractions using a “4-digit truncated representation” as the key (e.g., <code>0.16666...</code> → <code>⅙</code>)</li>\n<li>Keep <code>.5</code> values like <code>2.5</code> as-is (by design)</li>\n<li>Shorten long floats by truncating to 4 digits after the decimal point (truncate, not round)</li></ul></li>\n<li>Expanded the set of supported Unicode vulgar fractions (more than baseline)</li>\n</ul>\n<h3>Postprocessing</h3>\n<p>Postprocessing and the final submission formatting are also differences from “LB 35.9 Ensembling &amp; Post Processing Baseline” (only the differences are listed here).</p>\n<ul>\n<li>Fixed the replacement for <code>5/12 shekel(s)</code>: baseline <code>⅔ shekel 15 grains</code> → <code>⅓ shekel 15 grains</code></li>\n<li>Unified fraction/decimal normalization with the same logic as preprocessing (<code>1/2</code> / <code>2 1/2</code> / decimals → Unicode fractions)</li>\n<li>Improved removal of slash-based alternatives:<ul>\n<li>For patterns like <code>you / she</code>, keep the left side (<code>you</code>) and drop the right side (<code>she</code>)</li>\n<li>Changed the regex to avoid breaking fraction expressions like <code>1/2</code></li></ul></li>\n<li>Do not delete curly quotes (“ ” ‘ ’); normalize them to straight quotes (<code>\"</code> / <code>'</code>) instead (baseline deletes them)</li>\n<li>Adjusted the forbidden-character deletion target (unlike baseline, do not delete all <code>()</code> in bulk)</li>\n<li>Removed additional stray marks (<code>..</code> / <code>xx...</code>, etc.), and as a safeguard normalized <code>ḫ/Ḫ</code> to <code>h/H</code></li>\n<li>Safety checks (right before submission):<ul>\n<li>Force empty/None generations to <code>&lt;gap&gt;</code></li>\n<li>Use <code>validate_submission</code> to check columns/row count/empty translations/duplicate ids (baseline mainly logs the number of empty outputs)</li></ul></li>\n</ul>\n<h2>Models used</h2>\n<ul>\n<li>ByT5-small (downloaded from Hugging Face and trained on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>ByT5-base (downloaded from Hugging Face and trained on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>ByT5-large (downloaded from Hugging Face and trained on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>Public model: byt5 akkadian mbr v2 (input length 512, output length 384)</li>\n<li>Public model: final-byt5 (input length 512, output length 384)</li>\n<li>Public model: byt5 akkadian mbr v2 (additional fine-tuning on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>Public model: final-byt5 (additional fine-tuning on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n</ul>",
  "messages": [
    {
      "id": 3427596,
      "postDate": "2026-03-24T09:25:04.733Z",
      "content": "<p>I joined this competition late, and up until three days before the submission deadline I was mostly focused on cleaning up and curating <code>train.csv</code>.\nBecause of that, I didn’t have enough time to properly verify local CV, and in the end I had to rely on LB score and gut feeling, which is something I regret.</p>\n<h1>Cleaning up <code>train.csv</code></h1>\n<p>The original <code>train.csv</code> had 1,561 records, but after doing the following work I expanded it to 2,600 records in the end:</p>\n<ul>\n<li>I visually inspected the PDF page by page and added transliteration/translation pairs that were missing from <code>train.csv</code> (I found 27 missing items).</li>\n<li>I split texts that were too long to fit into the ByT5 input window, while paying attention to the alignment between the transliteration and the translation (I did this after fixing the issues in <code>train.csv</code> described below).</li>\n</ul>\n<p>I also fixed issues like the following in <code>train.csv</code>:</p>\n<ul>\n<li>PDF OCR mistakes (cases where either the transliteration or the translation was completely wrong)</li>\n<li>I restored records where the transliteration text was truncated in the host-provided <code>train.csv</code> (this was relatively easy to recover by comparing with <code>published_texts.csv</code>).</li>\n<li>I excluded some long records with so many <code>&lt;gap&gt;</code> tokens that splitting the transliteration and translation was not feasible, since they were likely harmful for model training (surprisingly, I found only 3 such records).</li>\n</ul>\n<p>I also found issues like the following in many places, but since the volume was too large and I suspected that similar issues existed in <code>test.csv</code> used for the final evaluation, I intentionally left them as-is:</p>\n<ul>\n<li>Italic text in the PDF was not captured by OCR</li>\n</ul>\n<p>For fixing <code>train.csv</code> using the PDF, I used a very primitive method: taking screenshots of the relevant parts and having ChatGPT transcribe them.</p>\n<p><em>For transcription</em></p>\n<pre><code>&gt; NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\nThe **screenshot images** I’m going to send contain **Akkadian transliteration** on the left (or top) and the corresponding **English translation** on the right (or bottom).  \nPlease **transcribe both accurately** and return them in a format that is **easy to copy &amp; paste into a spreadsheet**.\n\nIn some cases, a single screenshot might not be enough and the content may be split across multiple screenshots. In that case, please infer the correct order of the screenshots and transcribe accordingly.\n\n---\n\n## Response format (copy/paste first)\nIn principle, please reply every time with the following 2-line set (no extra explanation needed):\n\ntransliteration (Akkadian text) &lt;newline&gt;\n&lt;separator&gt;\ntranslation (English text)\n\n- `&lt;newline&gt;` means a line break in the ChatGPT UI; `&lt;separator&gt;` also means a line break in the ChatGPT UI.\n- Keep the Akkadian text and the English text to **one line each**.\n\n---\n\n## Handling body text vs. notes (important)\n- The image may contain lines other than the “body text” (e.g., notes, headings, layout notes, page numbers, line numbers), such as `Seal A`, `rev.`, `obv.`, `(remainder obv. missing)`, `(3.5 cm un-inscribed)`, etc.\n- Please **extract only the body text** and **exclude note-like lines** from the transcription output.\n\nExample output (when NOTE-like lines are present):\nAkkadian: transcribed transliteration (body) &lt;newline&gt;\n&lt;newline&gt;\nEnglish: transcribed translation (body)\n\n---\n\n## Transcription rules (important)\n### 1) Normalization / replacements (required)\nFor expressions indicating missing/damaged/illegible text, if the meaning is the same, **unify everything to `&lt;gap&gt;`**.\n\n- `x` → `&lt;gap&gt;`\n- `[x]` → `&lt;gap&gt;`\n- `…` (ellipsis) → `&lt;gap&gt;`\n- `(break)` → `&lt;gap&gt;`\n- `(large break)` → `&lt;gap&gt;`\n- `(n broken lines)` → `&lt;gap&gt;` (it’s OK to use `&lt;gap&gt;` even when `n` is a number)\n- If `&lt;gap&gt;` appears multiple times consecutively, merge into one; i.e., `&lt;gap&gt; &lt;gap&gt;` → `&lt;gap&gt;`\n- **Any other “missing/broken/unreadable” indicators** → `&lt;gap&gt;`\n\nFor determinatives, normalize as follows:\n\n- `(d)` → `{d}`\n- `(ki)` → `{ki}`\n\nFor parenthesized words:\n\n- `(TÚG)` → `TÚG` (remove the parentheses)\n\n---\n\n## 2) Things to keep as-is\n- Keep uppercase/lowercase, apostrophes, hyphens, subscript numerals (e.g., `₂`), special characters (e.g., `š ṣ ṭ ḫ`), accents, etc. **exactly as shown**\n- Keep symbols/separators (like `-` and `.`) **as in the image**, except where the replacement rules apply\n- `&lt;gap&gt;` must always be exactly `&lt;gap&gt;` in **lowercase**\n\n---\n\n## 3) Uncertainty\n- If there is a character/word you truly cannot identify, you may replace that part with `&lt;gap&gt;`\n- However, preserve whatever you can read (don’t turn everything into `&lt;gap&gt;`)\n\n---\n\n## Minimal competition context (for clarity)\n- Input: Akkadian (cuneiform) transliteration = transliteration\n- Output: its English translation = translation\n- The goal is to create transliteration → English pairs, so please extract them as **two columns of text**\n\n---\n\n## Requests to make the work faster and more accurate\n- If possible, extract the “transliteration part” and the “English translation part” separately so they **don’t get mixed**\n- Since punctuation and missing closing parentheses are common errors, do a quick check that the left/right (or top/bottom) alignment hasn’t drifted before you output\n- The output is intended to be pasted into a spreadsheet, so please **avoid extra decoration** (bullets, block quotes, etc.)\n</code></pre>\n<p><em>For splitting long texts with alignment preserved</em></p>\n<pre><code>&gt; NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\n\nThe text and screenshot images I’m going to send are candidate training data for the Kaggle competition **Deep Past Challenge - Translate Akkadian to English**.  \nFor a single long record, please split it into **a small number (around 2–5) of independent records**, while being careful not to break the alignment between the **Akkadian transliteration** and the **English translation**.\n\nThe goal is to make each record easier to handle with a ByT5 model by keeping it **under 1024 bytes**.  \nHowever, if splitting would likely break the alignment, it is better to return it as “cannot split” than to over-split.\n\n---\n\n## Most important policy (v2: don’t over-split)\n\n- Target number of splits: **2–5** (prefer **2–4** if possible)\n- Split only when necessary\n  - Add extra split boundaries only when they are truly needed to keep chunks under 1024 bytes or to preserve alignment\n- Avoid producing many very short chunks\n  - If you can’t count bytes exactly, err on the safe side, but **don’t mass-produce tiny chunks**\n  - As a rough guide, aim for **large chunks (roughly 700–1000 bytes)** when possible\n- If it still won’t fit within 5 splits or alignment would break, return `needs_manual_review` (do not force 6+ splits)\n\n---\n\n## Background (minimal)\n\n- In this competition, we train a model to generate English translations from Akkadian transliterations.\n- ByT5 is byte-level, so **UTF-8 byte length** matters more than character count.\n- On the source side, we usually add the prefix `translate Akkadian to English: `.\n- In practice, we want both source and target to be **under 1024 bytes** per record.\n- You don’t need to compute byte length perfectly every time. If unsure, make it a bit shorter for safety.\n\n---\n\n## What I want you to do\n\n1. Read the given `transliteration` and `translation`\n2. Refer to the screenshot images if needed\n3. Find only a small number of boundaries where the **semantic alignment is clear** (don’t add too many)\n4. Split the record into **2–5** `transliteration_chunk` / `translation_chunk` pairs (choose the minimum split that works)\n5. If splitting is risky, return `needs_manual_review`\n\n---\n\n## Absolute rules\n\n### 1) This is “splitting”, not “editing”\n\n- For the body text, use the input text as-is and, in principle, only take **substrings** from the provided `transliteration` / `translation`\n- Do not fill in missing text, paraphrase, summarize, or re-translate\n- Even if you notice OCR mistakes or unnatural English, **do not fix them in this task**\n\n### 2) Preserve the order\n\n- When you stack the chunks from top to bottom, they must preserve the original order of the `transliteration` / `translation`\n\n### 3) Only cut at boundaries visible on both sides\n\n- Sentence/clause boundaries on the English side\n- Boundaries between itemized lists\n- Breaks in letter templates / formulaic phrases\n- Breaks between imperative/conditional sections\n- Clear line-group boundaries on the transliteration side\n\nOnly cut at places where a coherent “unit” is visible on both sides.\n\n### 4) Do not cut in the middle of these\n\n- Numbers, fractions, decimals, tax rates\n- Amount expressions like shekel / mina groupings\n- Personal names, divine names, place names\n- Phrases like `son of ...` or `belonging to ...`\n- Inside quotation marks\n- Inside an item name\n\n### 5) If it’s not safe, hold it\n\nIn the following cases, do not force a split; return `needs_manual_review`:\n\n- Even with the images, the aligned boundary cannot be determined\n- The English translation in train looks broken in the middle\n- The source/target are misaligned and 1:1 correspondence is unclear\n- Keeping it within 2–5 splits would inevitably break numeric/proper-noun sequences\n\n---\n\n## How to use the images\n\nUse the screenshots to decide **where to cut so that transliteration and translation stay aligned**.\n\nImportant:\n\n- The Akkadian and English text in the PDF may not exactly match the text provided in the chat.\n- This may be because the chat text has already been normalized (e.g., `&lt;gap&gt;` replacements, symbol cleanup, etc.).\n- Therefore, use the PDF as a reference to find alignment boundaries, but for the final output body text, **prefer the chat-provided text**.\n- Do not rewrite the body text to match the PDF’s visual appearance.\n\nPriority when using images:\n\n1. Verify alignment boundaries between `transliteration` and `translation`\n2. Resolve ambiguous boundaries that are hard to see from text alone\n3. Decide what spans form a coherent aligned unit\n\nDo not create new translations even if you use the images.\n\n---\n\n## Boundary heuristics (for a small number of splits)\n\n### Good candidates (high priority)\n\n- After an introductory formula like `a-na ... qí-bi-ma um-ma ...` (but don’t split into a tiny chunk just for the intro)\n- English expressions that indicate a section change, e.g. `all this ...`, `thereof ...`, `therefore ...`, `if ... then ...`\n- Places where a large block of an item list changes (avoid splitting each small item)\n- Where repetition of the same pattern ends as a block\n\n### Tempting but usually forbidden\n\n- Splitting every English sentence (often creates too many chunks)\n- Splitting every small list item (avoid)\n\n---\n\n## Output format (strict)\n\nReturn **only a JSON array**. No extra explanation text.\n\n### When splitting is possible (2–5 chunks)\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__01\",\n    \"status\": \"ok\",\n    \"transliteration_chunk\": \"split transliteration chunk (must be a substring of the input)\",\n    \"translation_chunk\": \"corresponding translation chunk (must be a substring of the input)\",\n    \"boundary_reason\": \"one sentence describing why you cut here (alignment rationale)\",\n    \"confidence\": \"high\"\n  }\n]\n```\n\n### When splitting is risky\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__00\",\n    \"status\": \"needs_manual_review\",\n    \"transliteration_chunk\": \"\",\n    \"translation_chunk\": \"\",\n    \"boundary_reason\": \"cannot cut into 2–5 chunks without breaking alignment\",\n    \"confidence\": \"low\"\n  }\n]\n```\n\n---\n\n## `confidence` guide\n\n- `high`: boundaries are very clear from both images and text\n- `medium`: mostly reasonable, but alignment is slightly weak on one side\n- `low`: the original train text may be missing/broken/misaligned\n\n---\n\n## When you receive an actual request, always follow these\n\n- Return only a JSON array\n- Strongly prefer 2–5 splits and do not add too many boundaries\n- Remember this is splitting, not editing\n- Do not add words that are not in the provided body text, even if you look at images\n- If unsure, return `needs_manual_review`\n</code></pre>\n<p>At first I tried to clean up <code>train.csv</code> using Codex agents and the skills feature, but I couldn’t control it well, and in the end I did it manually.</p>\n<h1>Final submission pipeline</h1>\n<p>I started from the public notebook “<a href=\"https://www.kaggle.com/code/waterjoe/lb-35-9-ensembling-post-processing-baseline\" target=\"_blank\">LB 35.9 Ensembling &amp; Post Processing Baseline</a>”, then increased the number of models and made small changes to preprocessing and postprocessing.\nThe last two submissions differed only in the number of training epochs; despite concerns about overfitting, the one trained for more epochs achieved a higher Private LB score.</p>\n<h2>Preprocessing / Postprocessing</h2>\n<h3>Preprocessing</h3>\n<p>Both preprocessing and postprocessing are differences from “LB 35.9 Ensembling &amp; Post Processing Baseline” (only the differences are listed here).</p>\n<ul>\n<li>Expanded fraction/decimal normalization (replacing baseline <code>_EXACT_FRAC_RE</code> + <code>_canon_decimal(float)</code>):<ul>\n<li>Convert fraction expressions like <code>1/2</code> and <code>2 1/2</code> into Unicode vulgar fractions (e.g., <code>½</code>)</li>\n<li>For decimals, map them to Unicode fractions using a “4-digit truncated representation” as the key (e.g., <code>0.16666...</code> → <code>⅙</code>)</li>\n<li>Keep <code>.5</code> values like <code>2.5</code> as-is (by design)</li>\n<li>Shorten long floats by truncating to 4 digits after the decimal point (truncate, not round)</li></ul></li>\n<li>Expanded the set of supported Unicode vulgar fractions (more than baseline)</li>\n</ul>\n<h3>Postprocessing</h3>\n<p>Postprocessing and the final submission formatting are also differences from “LB 35.9 Ensembling &amp; Post Processing Baseline” (only the differences are listed here).</p>\n<ul>\n<li>Fixed the replacement for <code>5/12 shekel(s)</code>: baseline <code>⅔ shekel 15 grains</code> → <code>⅓ shekel 15 grains</code></li>\n<li>Unified fraction/decimal normalization with the same logic as preprocessing (<code>1/2</code> / <code>2 1/2</code> / decimals → Unicode fractions)</li>\n<li>Improved removal of slash-based alternatives:<ul>\n<li>For patterns like <code>you / she</code>, keep the left side (<code>you</code>) and drop the right side (<code>she</code>)</li>\n<li>Changed the regex to avoid breaking fraction expressions like <code>1/2</code></li></ul></li>\n<li>Do not delete curly quotes (“ ” ‘ ’); normalize them to straight quotes (<code>\"</code> / <code>'</code>) instead (baseline deletes them)</li>\n<li>Adjusted the forbidden-character deletion target (unlike baseline, do not delete all <code>()</code> in bulk)</li>\n<li>Removed additional stray marks (<code>..</code> / <code>xx...</code>, etc.), and as a safeguard normalized <code>ḫ/Ḫ</code> to <code>h/H</code></li>\n<li>Safety checks (right before submission):<ul>\n<li>Force empty/None generations to <code>&lt;gap&gt;</code></li>\n<li>Use <code>validate_submission</code> to check columns/row count/empty translations/duplicate ids (baseline mainly logs the number of empty outputs)</li></ul></li>\n</ul>\n<h2>Models used</h2>\n<ul>\n<li>ByT5-small (downloaded from Hugging Face and trained on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>ByT5-base (downloaded from Hugging Face and trained on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>ByT5-large (downloaded from Hugging Face and trained on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>Public model: byt5 akkadian mbr v2 (input length 512, output length 384)</li>\n<li>Public model: final-byt5 (input length 512, output length 384)</li>\n<li>Public model: byt5 akkadian mbr v2 (additional fine-tuning on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n<li>Public model: final-byt5 (additional fine-tuning on the expanded <code>train.csv</code>; input length 1024, output length 1024)</li>\n</ul>",
      "rawMarkdown": "I joined this competition late, and up until three days before the submission deadline I was mostly focused on cleaning up and curating `train.csv`.\nBecause of that, I didn’t have enough time to properly verify local CV, and in the end I had to rely on LB score and gut feeling, which is something I regret.\n\n# Cleaning up `train.csv`\nThe original `train.csv` had 1,561 records, but after doing the following work I expanded it to 2,600 records in the end:\n\n- I visually inspected the PDF page by page and added transliteration/translation pairs that were missing from `train.csv` (I found 27 missing items).\n- I split texts that were too long to fit into the ByT5 input window, while paying attention to the alignment between the transliteration and the translation (I did this after fixing the issues in `train.csv` described below).\n\nI also fixed issues like the following in `train.csv`:\n\n- PDF OCR mistakes (cases where either the transliteration or the translation was completely wrong)\n- I restored records where the transliteration text was truncated in the host-provided `train.csv` (this was relatively easy to recover by comparing with `published_texts.csv`).\n- I excluded some long records with so many `<gap>` tokens that splitting the transliteration and translation was not feasible, since they were likely harmful for model training (surprisingly, I found only 3 such records).\n\nI also found issues like the following in many places, but since the volume was too large and I suspected that similar issues existed in `test.csv` used for the final evaluation, I intentionally left them as-is:\n\n- Italic text in the PDF was not captured by OCR\n\nFor fixing `train.csv` using the PDF, I used a very primitive method: taking screenshots of the relevant parts and having ChatGPT transcribe them.\n\n*For transcription*\n````markdown\n> NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\nThe **screenshot images** I’m going to send contain **Akkadian transliteration** on the left (or top) and the corresponding **English translation** on the right (or bottom).  \nPlease **transcribe both accurately** and return them in a format that is **easy to copy & paste into a spreadsheet**.\n\nIn some cases, a single screenshot might not be enough and the content may be split across multiple screenshots. In that case, please infer the correct order of the screenshots and transcribe accordingly.\n\n---\n\n## Response format (copy/paste first)\nIn principle, please reply every time with the following 2-line set (no extra explanation needed):\n\ntransliteration (Akkadian text) <newline>\n<separator>\ntranslation (English text)\n\n- `<newline>` means a line break in the ChatGPT UI; `<separator>` also means a line break in the ChatGPT UI.\n- Keep the Akkadian text and the English text to **one line each**.\n\n---\n\n## Handling body text vs. notes (important)\n- The image may contain lines other than the “body text” (e.g., notes, headings, layout notes, page numbers, line numbers), such as `Seal A`, `rev.`, `obv.`, `(remainder obv. missing)`, `(3.5 cm un-inscribed)`, etc.\n- Please **extract only the body text** and **exclude note-like lines** from the transcription output.\n\nExample output (when NOTE-like lines are present):\nAkkadian: transcribed transliteration (body) <newline>\n<newline>\nEnglish: transcribed translation (body)\n\n---\n\n## Transcription rules (important)\n### 1) Normalization / replacements (required)\nFor expressions indicating missing/damaged/illegible text, if the meaning is the same, **unify everything to `<gap>`**.\n\n- `x` → `<gap>`\n- `[x]` → `<gap>`\n- `…` (ellipsis) → `<gap>`\n- `(break)` → `<gap>`\n- `(large break)` → `<gap>`\n- `(n broken lines)` → `<gap>` (it’s OK to use `<gap>` even when `n` is a number)\n- If `<gap>` appears multiple times consecutively, merge into one; i.e., `<gap> <gap>` → `<gap>`\n- **Any other “missing/broken/unreadable” indicators** → `<gap>`\n\nFor determinatives, normalize as follows:\n\n- `(d)` → `{d}`\n- `(ki)` → `{ki}`\n\nFor parenthesized words:\n\n- `(TÚG)` → `TÚG` (remove the parentheses)\n\n---\n\n## 2) Things to keep as-is\n- Keep uppercase/lowercase, apostrophes, hyphens, subscript numerals (e.g., `₂`), special characters (e.g., `š ṣ ṭ ḫ`), accents, etc. **exactly as shown**\n- Keep symbols/separators (like `-` and `.`) **as in the image**, except where the replacement rules apply\n- `<gap>` must always be exactly `<gap>` in **lowercase**\n\n---\n\n## 3) Uncertainty\n- If there is a character/word you truly cannot identify, you may replace that part with `<gap>`\n- However, preserve whatever you can read (don’t turn everything into `<gap>`)\n\n---\n\n## Minimal competition context (for clarity)\n- Input: Akkadian (cuneiform) transliteration = transliteration\n- Output: its English translation = translation\n- The goal is to create transliteration → English pairs, so please extract them as **two columns of text**\n\n---\n\n## Requests to make the work faster and more accurate\n- If possible, extract the “transliteration part” and the “English translation part” separately so they **don’t get mixed**\n- Since punctuation and missing closing parentheses are common errors, do a quick check that the left/right (or top/bottom) alignment hasn’t drifted before you output\n- The output is intended to be pasted into a spreadsheet, so please **avoid extra decoration** (bullets, block quotes, etc.)\n````\n\n*For splitting long texts with alignment preserved*\n````markdown\n> NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\n\nThe text and screenshot images I’m going to send are candidate training data for the Kaggle competition **Deep Past Challenge - Translate Akkadian to English**.  \nFor a single long record, please split it into **a small number (around 2–5) of independent records**, while being careful not to break the alignment between the **Akkadian transliteration** and the **English translation**.\n\nThe goal is to make each record easier to handle with a ByT5 model by keeping it **under 1024 bytes**.  \nHowever, if splitting would likely break the alignment, it is better to return it as “cannot split” than to over-split.\n\n---\n\n## Most important policy (v2: don’t over-split)\n\n- Target number of splits: **2–5** (prefer **2–4** if possible)\n- Split only when necessary\n  - Add extra split boundaries only when they are truly needed to keep chunks under 1024 bytes or to preserve alignment\n- Avoid producing many very short chunks\n  - If you can’t count bytes exactly, err on the safe side, but **don’t mass-produce tiny chunks**\n  - As a rough guide, aim for **large chunks (roughly 700–1000 bytes)** when possible\n- If it still won’t fit within 5 splits or alignment would break, return `needs_manual_review` (do not force 6+ splits)\n\n---\n\n## Background (minimal)\n\n- In this competition, we train a model to generate English translations from Akkadian transliterations.\n- ByT5 is byte-level, so **UTF-8 byte length** matters more than character count.\n- On the source side, we usually add the prefix `translate Akkadian to English: `.\n- In practice, we want both source and target to be **under 1024 bytes** per record.\n- You don’t need to compute byte length perfectly every time. If unsure, make it a bit shorter for safety.\n\n---\n\n## What I want you to do\n\n1. Read the given `transliteration` and `translation`\n2. Refer to the screenshot images if needed\n3. Find only a small number of boundaries where the **semantic alignment is clear** (don’t add too many)\n4. Split the record into **2–5** `transliteration_chunk` / `translation_chunk` pairs (choose the minimum split that works)\n5. If splitting is risky, return `needs_manual_review`\n\n---\n\n## Absolute rules\n\n### 1) This is “splitting”, not “editing”\n\n- For the body text, use the input text as-is and, in principle, only take **substrings** from the provided `transliteration` / `translation`\n- Do not fill in missing text, paraphrase, summarize, or re-translate\n- Even if you notice OCR mistakes or unnatural English, **do not fix them in this task**\n\n### 2) Preserve the order\n\n- When you stack the chunks from top to bottom, they must preserve the original order of the `transliteration` / `translation`\n\n### 3) Only cut at boundaries visible on both sides\n\n- Sentence/clause boundaries on the English side\n- Boundaries between itemized lists\n- Breaks in letter templates / formulaic phrases\n- Breaks between imperative/conditional sections\n- Clear line-group boundaries on the transliteration side\n\nOnly cut at places where a coherent “unit” is visible on both sides.\n\n### 4) Do not cut in the middle of these\n\n- Numbers, fractions, decimals, tax rates\n- Amount expressions like shekel / mina groupings\n- Personal names, divine names, place names\n- Phrases like `son of ...` or `belonging to ...`\n- Inside quotation marks\n- Inside an item name\n\n### 5) If it’s not safe, hold it\n\nIn the following cases, do not force a split; return `needs_manual_review`:\n\n- Even with the images, the aligned boundary cannot be determined\n- The English translation in train looks broken in the middle\n- The source/target are misaligned and 1:1 correspondence is unclear\n- Keeping it within 2–5 splits would inevitably break numeric/proper-noun sequences\n\n---\n\n## How to use the images\n\nUse the screenshots to decide **where to cut so that transliteration and translation stay aligned**.\n\nImportant:\n\n- The Akkadian and English text in the PDF may not exactly match the text provided in the chat.\n- This may be because the chat text has already been normalized (e.g., `<gap>` replacements, symbol cleanup, etc.).\n- Therefore, use the PDF as a reference to find alignment boundaries, but for the final output body text, **prefer the chat-provided text**.\n- Do not rewrite the body text to match the PDF’s visual appearance.\n\nPriority when using images:\n\n1. Verify alignment boundaries between `transliteration` and `translation`\n2. Resolve ambiguous boundaries that are hard to see from text alone\n3. Decide what spans form a coherent aligned unit\n\nDo not create new translations even if you use the images.\n\n---\n\n## Boundary heuristics (for a small number of splits)\n\n### Good candidates (high priority)\n\n- After an introductory formula like `a-na ... qí-bi-ma um-ma ...` (but don’t split into a tiny chunk just for the intro)\n- English expressions that indicate a section change, e.g. `all this ...`, `thereof ...`, `therefore ...`, `if ... then ...`\n- Places where a large block of an item list changes (avoid splitting each small item)\n- Where repetition of the same pattern ends as a block\n\n### Tempting but usually forbidden\n\n- Splitting every English sentence (often creates too many chunks)\n- Splitting every small list item (avoid)\n\n---\n\n## Output format (strict)\n\nReturn **only a JSON array**. No extra explanation text.\n\n### When splitting is possible (2–5 chunks)\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__01\",\n    \"status\": \"ok\",\n    \"transliteration_chunk\": \"split transliteration chunk (must be a substring of the input)\",\n    \"translation_chunk\": \"corresponding translation chunk (must be a substring of the input)\",\n    \"boundary_reason\": \"one sentence describing why you cut here (alignment rationale)\",\n    \"confidence\": \"high\"\n  }\n]\n```\n\n### When splitting is risky\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__00\",\n    \"status\": \"needs_manual_review\",\n    \"transliteration_chunk\": \"\",\n    \"translation_chunk\": \"\",\n    \"boundary_reason\": \"cannot cut into 2–5 chunks without breaking alignment\",\n    \"confidence\": \"low\"\n  }\n]\n```\n\n---\n\n## `confidence` guide\n\n- `high`: boundaries are very clear from both images and text\n- `medium`: mostly reasonable, but alignment is slightly weak on one side\n- `low`: the original train text may be missing/broken/misaligned\n\n---\n\n## When you receive an actual request, always follow these\n\n- Return only a JSON array\n- Strongly prefer 2–5 splits and do not add too many boundaries\n- Remember this is splitting, not editing\n- Do not add words that are not in the provided body text, even if you look at images\n- If unsure, return `needs_manual_review`\n````\n\nAt first I tried to clean up `train.csv` using Codex agents and the skills feature, but I couldn’t control it well, and in the end I did it manually.\n\n# Final submission pipeline\nI started from the public notebook “[LB 35.9 Ensembling & Post Processing Baseline](https://www.kaggle.com/code/waterjoe/lb-35-9-ensembling-post-processing-baseline)”, then increased the number of models and made small changes to preprocessing and postprocessing.\nThe last two submissions differed only in the number of training epochs; despite concerns about overfitting, the one trained for more epochs achieved a higher Private LB score.\n\n## Preprocessing / Postprocessing\n### Preprocessing\nBoth preprocessing and postprocessing are differences from “LB 35.9 Ensembling & Post Processing Baseline” (only the differences are listed here).\n\n- Expanded fraction/decimal normalization (replacing baseline `_EXACT_FRAC_RE` + `_canon_decimal(float)`):\n  - Convert fraction expressions like `1/2` and `2 1/2` into Unicode vulgar fractions (e.g., `½`)\n  - For decimals, map them to Unicode fractions using a “4-digit truncated representation” as the key (e.g., `0.16666...` → `⅙`)\n  - Keep `.5` values like `2.5` as-is (by design)\n  - Shorten long floats by truncating to 4 digits after the decimal point (truncate, not round)\n- Expanded the set of supported Unicode vulgar fractions (more than baseline)\n\n### Postprocessing\nPostprocessing and the final submission formatting are also differences from “LB 35.9 Ensembling & Post Processing Baseline” (only the differences are listed here).\n\n- Fixed the replacement for `5/12 shekel(s)`: baseline `⅔ shekel 15 grains` → `⅓ shekel 15 grains`\n- Unified fraction/decimal normalization with the same logic as preprocessing (`1/2` / `2 1/2` / decimals → Unicode fractions)\n- Improved removal of slash-based alternatives:\n  - For patterns like `you / she`, keep the left side (`you`) and drop the right side (`she`)\n  - Changed the regex to avoid breaking fraction expressions like `1/2`\n- Do not delete curly quotes (“ ” ‘ ’); normalize them to straight quotes (`\"` / `'`) instead (baseline deletes them)\n- Adjusted the forbidden-character deletion target (unlike baseline, do not delete all `()` in bulk)\n- Removed additional stray marks (`..` / `xx...`, etc.), and as a safeguard normalized `ḫ/Ḫ` to `h/H`\n- Safety checks (right before submission):\n  - Force empty/None generations to `<gap>`\n  - Use `validate_submission` to check columns/row count/empty translations/duplicate ids (baseline mainly logs the number of empty outputs)\n\n## Models used\n- ByT5-small (downloaded from Hugging Face and trained on the expanded `train.csv`; input length 1024, output length 1024)\n- ByT5-base (downloaded from Hugging Face and trained on the expanded `train.csv`; input length 1024, output length 1024)\n- ByT5-large (downloaded from Hugging Face and trained on the expanded `train.csv`; input length 1024, output length 1024)\n- Public model: byt5 akkadian mbr v2 (input length 512, output length 384)\n- Public model: final-byt5 (input length 512, output length 384)\n- Public model: byt5 akkadian mbr v2 (additional fine-tuning on the expanded `train.csv`; input length 1024, output length 1024)\n- Public model: final-byt5 (additional fine-tuning on the expanded `train.csv`; input length 1024, output length 1024)\n",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3427596": "I joined this competition late, and up until three days before the submission deadline I was mostly focused on cleaning up and curating `train.csv`.\nBecause of that, I didn’t have enough time to properly verify local CV, and in the end I had to rely on LB score and gut feeling, which is something I regret.\n\n# Cleaning up `train.csv`\nThe original `train.csv` had 1,561 records, but after doing the following work I expanded it to 2,600 records in the end:\n\n- I visually inspected the PDF page by page and added transliteration/translation pairs that were missing from `train.csv` (I found 27 missing items).\n- I split texts that were too long to fit into the ByT5 input window, while paying attention to the alignment between the transliteration and the translation (I did this after fixing the issues in `train.csv` described below).\n\nI also fixed issues like the following in `train.csv`:\n\n- PDF OCR mistakes (cases where either the transliteration or the translation was completely wrong)\n- I restored records where the transliteration text was truncated in the host-provided `train.csv` (this was relatively easy to recover by comparing with `published_texts.csv`).\n- I excluded some long records with so many `<gap>` tokens that splitting the transliteration and translation was not feasible, since they were likely harmful for model training (surprisingly, I found only 3 such records).\n\nI also found issues like the following in many places, but since the volume was too large and I suspected that similar issues existed in `test.csv` used for the final evaluation, I intentionally left them as-is:\n\n- Italic text in the PDF was not captured by OCR\n\nFor fixing `train.csv` using the PDF, I used a very primitive method: taking screenshots of the relevant parts and having ChatGPT transcribe them.\n\n*For transcription*\n````markdown\n> NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\nThe **screenshot images** I’m going to send contain **Akkadian transliteration** on the left (or top) and the corresponding **English translation** on the right (or bottom).  \nPlease **transcribe both accurately** and return them in a format that is **easy to copy & paste into a spreadsheet**.\n\nIn some cases, a single screenshot might not be enough and the content may be split across multiple screenshots. In that case, please infer the correct order of the screenshots and transcribe accordingly.\n\n---\n\n## Response format (copy/paste first)\nIn principle, please reply every time with the following 2-line set (no extra explanation needed):\n\ntransliteration (Akkadian text) <newline>\n<separator>\ntranslation (English text)\n\n- `<newline>` means a line break in the ChatGPT UI; `<separator>` also means a line break in the ChatGPT UI.\n- Keep the Akkadian text and the English text to **one line each**.\n\n---\n\n## Handling body text vs. notes (important)\n- The image may contain lines other than the “body text” (e.g., notes, headings, layout notes, page numbers, line numbers), such as `Seal A`, `rev.`, `obv.`, `(remainder obv. missing)`, `(3.5 cm un-inscribed)`, etc.\n- Please **extract only the body text** and **exclude note-like lines** from the transcription output.\n\nExample output (when NOTE-like lines are present):\nAkkadian: transcribed transliteration (body) <newline>\n<newline>\nEnglish: transcribed translation (body)\n\n---\n\n## Transcription rules (important)\n### 1) Normalization / replacements (required)\nFor expressions indicating missing/damaged/illegible text, if the meaning is the same, **unify everything to `<gap>`**.\n\n- `x` → `<gap>`\n- `[x]` → `<gap>`\n- `…` (ellipsis) → `<gap>`\n- `(break)` → `<gap>`\n- `(large break)` → `<gap>`\n- `(n broken lines)` → `<gap>` (it’s OK to use `<gap>` even when `n` is a number)\n- If `<gap>` appears multiple times consecutively, merge into one; i.e., `<gap> <gap>` → `<gap>`\n- **Any other “missing/broken/unreadable” indicators** → `<gap>`\n\nFor determinatives, normalize as follows:\n\n- `(d)` → `{d}`\n- `(ki)` → `{ki}`\n\nFor parenthesized words:\n\n- `(TÚG)` → `TÚG` (remove the parentheses)\n\n---\n\n## 2) Things to keep as-is\n- Keep uppercase/lowercase, apostrophes, hyphens, subscript numerals (e.g., `₂`), special characters (e.g., `š ṣ ṭ ḫ`), accents, etc. **exactly as shown**\n- Keep symbols/separators (like `-` and `.`) **as in the image**, except where the replacement rules apply\n- `<gap>` must always be exactly `<gap>` in **lowercase**\n\n---\n\n## 3) Uncertainty\n- If there is a character/word you truly cannot identify, you may replace that part with `<gap>`\n- However, preserve whatever you can read (don’t turn everything into `<gap>`)\n\n---\n\n## Minimal competition context (for clarity)\n- Input: Akkadian (cuneiform) transliteration = transliteration\n- Output: its English translation = translation\n- The goal is to create transliteration → English pairs, so please extract them as **two columns of text**\n\n---\n\n## Requests to make the work faster and more accurate\n- If possible, extract the “transliteration part” and the “English translation part” separately so they **don’t get mixed**\n- Since punctuation and missing closing parentheses are common errors, do a quick check that the left/right (or top/bottom) alignment hasn’t drifted before you output\n- The output is intended to be pasted into a spreadsheet, so please **avoid extra decoration** (bullets, block quotes, etc.)\n````\n\n*For splitting long texts with alignment preserved*\n````markdown\n> NOTE: The original version of this prompt was written in Japanese. The text below is an English translation.\n\n# Purpose (request to you)\n\nThe text and screenshot images I’m going to send are candidate training data for the Kaggle competition **Deep Past Challenge - Translate Akkadian to English**.  \nFor a single long record, please split it into **a small number (around 2–5) of independent records**, while being careful not to break the alignment between the **Akkadian transliteration** and the **English translation**.\n\nThe goal is to make each record easier to handle with a ByT5 model by keeping it **under 1024 bytes**.  \nHowever, if splitting would likely break the alignment, it is better to return it as “cannot split” than to over-split.\n\n---\n\n## Most important policy (v2: don’t over-split)\n\n- Target number of splits: **2–5** (prefer **2–4** if possible)\n- Split only when necessary\n  - Add extra split boundaries only when they are truly needed to keep chunks under 1024 bytes or to preserve alignment\n- Avoid producing many very short chunks\n  - If you can’t count bytes exactly, err on the safe side, but **don’t mass-produce tiny chunks**\n  - As a rough guide, aim for **large chunks (roughly 700–1000 bytes)** when possible\n- If it still won’t fit within 5 splits or alignment would break, return `needs_manual_review` (do not force 6+ splits)\n\n---\n\n## Background (minimal)\n\n- In this competition, we train a model to generate English translations from Akkadian transliterations.\n- ByT5 is byte-level, so **UTF-8 byte length** matters more than character count.\n- On the source side, we usually add the prefix `translate Akkadian to English: `.\n- In practice, we want both source and target to be **under 1024 bytes** per record.\n- You don’t need to compute byte length perfectly every time. If unsure, make it a bit shorter for safety.\n\n---\n\n## What I want you to do\n\n1. Read the given `transliteration` and `translation`\n2. Refer to the screenshot images if needed\n3. Find only a small number of boundaries where the **semantic alignment is clear** (don’t add too many)\n4. Split the record into **2–5** `transliteration_chunk` / `translation_chunk` pairs (choose the minimum split that works)\n5. If splitting is risky, return `needs_manual_review`\n\n---\n\n## Absolute rules\n\n### 1) This is “splitting”, not “editing”\n\n- For the body text, use the input text as-is and, in principle, only take **substrings** from the provided `transliteration` / `translation`\n- Do not fill in missing text, paraphrase, summarize, or re-translate\n- Even if you notice OCR mistakes or unnatural English, **do not fix them in this task**\n\n### 2) Preserve the order\n\n- When you stack the chunks from top to bottom, they must preserve the original order of the `transliteration` / `translation`\n\n### 3) Only cut at boundaries visible on both sides\n\n- Sentence/clause boundaries on the English side\n- Boundaries between itemized lists\n- Breaks in letter templates / formulaic phrases\n- Breaks between imperative/conditional sections\n- Clear line-group boundaries on the transliteration side\n\nOnly cut at places where a coherent “unit” is visible on both sides.\n\n### 4) Do not cut in the middle of these\n\n- Numbers, fractions, decimals, tax rates\n- Amount expressions like shekel / mina groupings\n- Personal names, divine names, place names\n- Phrases like `son of ...` or `belonging to ...`\n- Inside quotation marks\n- Inside an item name\n\n### 5) If it’s not safe, hold it\n\nIn the following cases, do not force a split; return `needs_manual_review`:\n\n- Even with the images, the aligned boundary cannot be determined\n- The English translation in train looks broken in the middle\n- The source/target are misaligned and 1:1 correspondence is unclear\n- Keeping it within 2–5 splits would inevitably break numeric/proper-noun sequences\n\n---\n\n## How to use the images\n\nUse the screenshots to decide **where to cut so that transliteration and translation stay aligned**.\n\nImportant:\n\n- The Akkadian and English text in the PDF may not exactly match the text provided in the chat.\n- This may be because the chat text has already been normalized (e.g., `<gap>` replacements, symbol cleanup, etc.).\n- Therefore, use the PDF as a reference to find alignment boundaries, but for the final output body text, **prefer the chat-provided text**.\n- Do not rewrite the body text to match the PDF’s visual appearance.\n\nPriority when using images:\n\n1. Verify alignment boundaries between `transliteration` and `translation`\n2. Resolve ambiguous boundaries that are hard to see from text alone\n3. Decide what spans form a coherent aligned unit\n\nDo not create new translations even if you use the images.\n\n---\n\n## Boundary heuristics (for a small number of splits)\n\n### Good candidates (high priority)\n\n- After an introductory formula like `a-na ... qí-bi-ma um-ma ...` (but don’t split into a tiny chunk just for the intro)\n- English expressions that indicate a section change, e.g. `all this ...`, `thereof ...`, `therefore ...`, `if ... then ...`\n- Places where a large block of an item list changes (avoid splitting each small item)\n- Where repetition of the same pattern ends as a block\n\n### Tempting but usually forbidden\n\n- Splitting every English sentence (often creates too many chunks)\n- Splitting every small list item (avoid)\n\n---\n\n## Output format (strict)\n\nReturn **only a JSON array**. No extra explanation text.\n\n### When splitting is possible (2–5 chunks)\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__01\",\n    \"status\": \"ok\",\n    \"transliteration_chunk\": \"split transliteration chunk (must be a substring of the input)\",\n    \"translation_chunk\": \"corresponding translation chunk (must be a substring of the input)\",\n    \"boundary_reason\": \"one sentence describing why you cut here (alignment rationale)\",\n    \"confidence\": \"high\"\n  }\n]\n```\n\n### When splitting is risky\n\n```json\n[\n  {\n    \"record_id\": \"original record id\",\n    \"split_id\": \"original record id__00\",\n    \"status\": \"needs_manual_review\",\n    \"transliteration_chunk\": \"\",\n    \"translation_chunk\": \"\",\n    \"boundary_reason\": \"cannot cut into 2–5 chunks without breaking alignment\",\n    \"confidence\": \"low\"\n  }\n]\n```\n\n---\n\n## `confidence` guide\n\n- `high`: boundaries are very clear from both images and text\n- `medium`: mostly reasonable, but alignment is slightly weak on one side\n- `low`: the original train text may be missing/broken/misaligned\n\n---\n\n## When you receive an actual request, always follow these\n\n- Return only a JSON array\n- Strongly prefer 2–5 splits and do not add too many boundaries\n- Remember this is splitting, not editing\n- Do not add words that are not in the provided body text, even if you look at images\n- If unsure, return `needs_manual_review`\n````\n\nAt first I tried to clean up `train.csv` using Codex agents and the skills feature, but I couldn’t control it well, and in the end I did it manually.\n\n# Final submission pipeline\nI started from the public notebook “[LB 35.9 Ensembling & Post Processing Baseline](https://www.kaggle.com/code/waterjoe/lb-35-9-ensembling-post-processing-baseline)”, then increased the number of models and made small changes to preprocessing and postprocessing.\nThe last two submissions differed only in the number of training epochs; despite concerns about overfitting, the one trained for more epochs achieved a higher Private LB score.\n\n## Preprocessing / Postprocessing\n### Preprocessing\nBoth preprocessing and postprocessing are differences from “LB 35.9 Ensembling & Post Processing Baseline” (only the differences are listed here).\n\n- Expanded fraction/decimal normalization (replacing baseline `_EXACT_FRAC_RE` + `_canon_decimal(float)`):\n  - Convert fraction expressions like `1/2` and `2 1/2` into Unicode vulgar fractions (e.g., `½`)\n  - For decimals, map them to Unicode fractions using a “4-digit truncated representation” as the key (e.g., `0.16666...` → `⅙`)\n  - Keep `.5` values like `2.5` as-is (by design)\n  - Shorten long floats by truncating to 4 digits after the decimal point (truncate, not round)\n- Expanded the set of supported Unicode vulgar fractions (more than baseline)\n\n### Postprocessing\nPostprocessing and the final submission formatting are also differences from “LB 35.9 Ensembling & Post Processing Baseline” (only the differences are listed here).\n\n- Fixed the replacement for `5/12 shekel(s)`: baseline `⅔ shekel 15 grains` → `⅓ shekel 15 grains`\n- Unified fraction/decimal normalization with the same logic as preprocessing (`1/2` / `2 1/2` / decimals → Unicode fractions)\n- Improved removal of slash-based alternatives:\n  - For patterns like `you / she`, keep the left side (`you`) and drop the right side (`she`)\n  - Changed the regex to avoid breaking fraction expressions like `1/2`\n- Do not delete curly quotes (“ ” ‘ ’); normalize them to straight quotes (`\"` / `'`) instead (baseline deletes them)\n- Adjusted the forbidden-character deletion target (unlike baseline, do not delete all `()` in bulk)\n- Removed additional stray marks (`..` / `xx...`, etc.), and as a safeguard normalized `ḫ/Ḫ` to `h/H`\n- Safety checks (right before submission):\n  - Force empty/None generations to `<gap>`\n  - Use `validate_submission` to check columns/row count/empty translations/duplicate ids (baseline mainly logs the number of empty outputs)\n\n## Models used\n- ByT5-small (downloaded from Hugging Face and trained on the expanded `train.csv`; input length 1024, output length 1024)\n- ByT5-base (downloaded from Hugging Face and trained on the expanded `train.csv`; input length 1024, output length 1024)\n- ByT5-large (downloaded from Hugging Face and trained on the expanded `train.csv`; input length 1024, output length 1024)\n- Public model: byt5 akkadian mbr v2 (input length 512, output length 384)\n- Public model: final-byt5 (input length 512, output length 384)\n- Public model: byt5 akkadian mbr v2 (additional fine-tuning on the expanded `train.csv`; input length 1024, output length 1024)\n- Public model: final-byt5 (additional fine-tuning on the expanded `train.csv`; input length 1024, output length 1024)\n"
  },
  "complete": true,
  "source": "existing_discussion",
  "competition_id": 121150,
  "competition_slug": "deep-past-initiative-machine-translation",
  "team_id": 15302247,
  "solution_url": "/competitions/deep-past-initiative-machine-translation/writeups/75th-place-solution"
}