{"cells": [{"cell_type": "markdown", "metadata": {}, "source": "# \ud83c\udffa Akkadian Uncovered: Reading 4,000-Year-Old Text Messages\n\n## The Deep Past Challenge - Complete EDA Guide\n\n---\n\n### \u2b50 If this notebook helps you, please upvote! It really motivates me to create more content! \u2b50\n\n---\n\n## \ud83c\udf1f What Makes This Story Fascinating?\n\n**Imagine this:** It's 1900 BC. You're a merchant in ancient Assyria. Your business partner in a distant city owes you money, and he's not responding to your messages. What do you do?\n\nYou grab a clay tablet, press cuneiform symbols into wet clay, and send an angry letter:\n\n> *\"Thirty years have passed since you left, and you have not sent me even one shekel of silver. Your behavior is not that of a brother!\"*\n\n**These are the texts we're translating.** Not royal decrees or religious hymns - but the everyday correspondence of ancient business people. Their complaints, their deals, their family drama.\n\n**This is the WhatsApp of 2000 BC.**\n\n---\n\n## \ud83d\udcda What You'll Discover\n\n| Section | What's Inside | Why It Matters |\n|---------|--------------|----------------|\n| 1\ufe0f\u20e3 **The Data Landscape** | Dataset sizes & structure | Know your battlefield |\n| 2\ufe0f\u20e3 **The Critical Trap** | Train vs Test mismatch | #1 mistake to avoid |\n| 3\ufe0f\u20e3 **Akkadian 101** | How the language actually works | Understand what you're translating |\n| 4\ufe0f\u20e3 **The Silver Economy** | What these texts are about | Meet the ancient merchants |\n| 5\ufe0f\u20e3 **ByT5 Deep Dive** | Why bytes beat words | The winning approach explained |\n| 6\ufe0f\u20e3 **Damage Report** | How broken are these tablets? | Handle missing text |\n| 7\ufe0f\u20e3 **Ready-to-Use Code** | Preprocessing pipeline | Copy, paste, win |\n\n---\n\n## \ud83d\udd2c Research Context\n\n
\n\n**Why is this hard?** Berkeley's CuneiTranslate project found that standard neural MT (T5, mT5, NLLB) achieved **BLEU < 8%** on cuneiform translation. The 2023 PNAS paper achieved **BLEU 37.47** using CNN + BPE - that's our target to beat!\n\n**The breakthrough:** The 2025 ACL workshop showed **ByT5 outperforms mT5** on cuneiform tasks with 80%+ accuracy. That's why the leaderboard is dominated by ByT5 solutions.\n\n
\n\n---"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udd27 Setup - Run this first!\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom collections import Counter\nimport re\nimport os\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Beautiful plots\nplt.style.use('seaborn-v0_8-whitegrid')\nsns.set_palette('husl')\nplt.rcParams['figure.figsize'] = (12, 6)\nplt.rcParams['font.size'] = 11\n\n# Data path\nDATA_PATH = '/kaggle/input/deep-past-initiative-machine-translation/' if os.path.exists('/kaggle/input') else '../data/'\n\nprint(\"\u2705 Ready to explore ancient Akkadian! Let's go! \ud83c\udffa\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 1\ufe0f\u20e3 The Data Landscape\n\n## What are we working with?\n\nBefore we dive into ancient languages, let's understand what data we have."}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcc2 Load all datasets\ntrain = pd.read_csv(f'{DATA_PATH}train.csv')\ntest = pd.read_csv(f'{DATA_PATH}test.csv')\nlexicon = pd.read_csv(f'{DATA_PATH}OA_Lexicon_eBL.csv')\npublished_texts = pd.read_csv(f'{DATA_PATH}published_texts.csv')\n\nprint(\"\ud83d\udcca DATASET OVERVIEW\")\nprint(\"\u2550\" * 60)\nprint(f\"\")\nprint(f\" \ud83d\udcdc Training pairs: {len(train):>7,} \u2190 Our gold!\")\nprint(f\" \ud83c\udfaf Test samples: {len(test):>7,} \u2190 What we predict\")\nprint(f\" \ud83d\udcd6 Lexicon entries: {len(lexicon):>7,} \u2190 Dictionary\")\nprint(f\" \ud83d\udcda Published texts: {len(published_texts):>7,} \u2190 Extra context\")\nprint(f\"\")\nprint(\"\u2550\" * 60)"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcca Visualize the data landscape\nfig, ax = plt.subplots(figsize=(10, 5))\n\ndatasets = ['Training\\nPairs', 'Test\\nSamples', 'Lexicon\\nEntries', 'Published\\nTexts']\nsizes = [len(train), len(test), len(lexicon), len(published_texts)]\ncolors = ['#2ecc71', '#e74c3c', '#3498db', '#9b59b6']\n\nbars = ax.bar(datasets, sizes, color=colors, edgecolor='white', linewidth=2)\n\nfor bar, size in zip(bars, sizes):\n ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + max(sizes)*0.02,\n f'{size:,}', ha='center', va='bottom', fontsize=14, fontweight='bold')\n\nax.set_ylabel('Number of Records', fontsize=12)\nax.set_title('\ud83c\udffa The Deep Past Data Universe', fontsize=16, fontweight='bold')\nax.set_yscale('log')\nax.set_ylim(1, max(sizes) * 3)\n\nplt.tight_layout()\nplt.show()\n\nprint(f\"\\n\ud83d\udca1 REALITY CHECK:\")\nprint(f\" Only {len(train):,} training pairs for a $50,000 competition!\")\nprint(f\" For comparison: Google Translate trains on BILLIONS of pairs.\")\nprint(f\" \\n This is ULTRA low-resource MT. Every trick counts! \ud83c\udfaf\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 2\ufe0f\u20e3 \ud83d\udea8 THE CRITICAL TRAP\n\n## The #1 Mistake That Will Tank Your Score\n\n**Stop!** Before you train anything, you MUST understand this.\n\nThe training data and test data have **completely different structures**. Miss this, and your model is doomed."}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udd0d Let's expose the trap\nprint(\"\ud83d\udd0d TRAINING DATA - What you're learning from:\")\nprint(\"\u2550\" * 60)\nprint(f\"Columns: {train.columns.tolist()}\")\nprint(f\"\\n\ud83d\udcdc Example document (first 300 chars):\")\nprint(\"-\" * 60)\nprint(train.iloc[0]['transliteration'][:300])\nprint(\"...\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83c\udfaf Now the test data - spot the difference!\nprint(\"\ud83c\udfaf TEST DATA - What you're predicting:\")\nprint(\"\u2550\" * 60)\nprint(f\"Columns: {test.columns.tolist()}\")\nprint(f\"\\n\ud83d\udc40 ALL test samples:\")\ndisplay(test)\n\nprint(\"\\n\" + \"\ud83d\udea8\" * 30)\nprint(\"\\n\u26a0\ufe0f DO YOU SEE IT? \u26a0\ufe0f\")\nprint(\"\\n\" + \"\ud83d\udea8\" * 30)"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcca Visualize the trap\nfig, ax = plt.subplots(figsize=(14, 4))\n\n# Draw test segments\ncolors = ['#3498db', '#e74c3c', '#2ecc71', '#9b59b6']\nfor idx, row in test.iterrows():\n width = row['line_end'] - row['line_start'] + 1\n ax.barh(0, width, left=row['line_start'] - 1, height=0.6,\n color=colors[idx], alpha=0.85, edgecolor='white', linewidth=3)\n ax.text((row['line_start'] + row['line_end']) / 2 - 0.5, 0,\n f\"Test {row['id']}\\nLines {row['line_start']}-{row['line_end']}\",\n ha='center', va='center', fontsize=10, fontweight='bold', color='white')\n\nax.set_xlim(0, test['line_end'].max() + 2)\nax.set_ylim(-0.8, 0.8)\nax.set_xlabel('Line Number in Original Tablet', fontsize=12)\nax.set_yticks([])\nax.set_title(f'\ud83c\udfaf THE TRAP REVEALED: All {len(test)} test samples = ONE ancient document, split into lines!',\n fontsize=14, fontweight='bold', color='#e74c3c')\n\nplt.tight_layout()\nplt.show()\n\n# Calculate actual averages\ntrain_avg_chars = train['transliteration'].str.len().mean()\ntest_avg_chars = test['transliteration'].str.len().mean()\n\nprint(\"\\n\" + \"\u2550\" * 70)\nprint(\"\\n\ud83d\udccb THE MISMATCH:\")\nprint(\"\\n TRAINING DATA vs. TEST DATA\")\nprint(\" \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\")\nprint(\" \ud83d\udcc4 Full documents \ud83d\udcdd Individual lines\")\nprint(f\" \ud83d\udccf ~{train_avg_chars:.0f} chars average \ud83d\udccf ~{test_avg_chars:.0f} chars average\") \nprint(\" \ud83d\udd22 No line numbers \ud83d\udd22 line_start, line_end\")\nprint(\"\\n\ud83d\udca1 IMPLICATION: You must learn to segment OR align your predictions!\")\nprint(\"\u2550\" * 70)"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 3\ufe0f\u20e3 \ud83d\udcdc Akkadian 101: How This Language Actually Works\n\n## You can't translate what you don't understand\n\nAkkadian isn't like any language you know. Here's what makes it special (and challenging):\n\n### The Writing System: A Beautiful Mess\n\nAkkadian uses **cuneiform** - wedge-shaped marks pressed into clay. But here's the twist:\n\n| Type | What It Is | Example | How It's Written |\n|------|-----------|---------|------------------|\n| **Syllables** | Sound-based signs | \"a-na\" = \"to\" | lowercase, hyphenated |\n| **Logograms** | Sumerian words as symbols | DUMU = \"son\" | ALL CAPS |\n| **Numbers** | Quantities & measures | 5 G\u00cdN = \"5 shekels\" | digits |\n| **Damage** | Broken/missing parts | [...] = \"text missing\" | brackets |"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udd0d Let's decode a real example!\nprint(\"\ud83d\udd0d DECODING A REAL AKKADIAN TEXT\")\nprint(\"\u2550\" * 70)\n\nsample_text = train.iloc[5]['transliteration'][:200]\nsample_trans = train.iloc[5]['translation'][:200]\n\nprint(f\"\\n\ud83d\udcdc ORIGINAL TRANSLITERATION:\")\nprint(f\" {sample_text}...\")\n\nprint(f\"\\n\ud83c\uddec\ud83c\udde7 ENGLISH TRANSLATION:\")\nprint(f\" {sample_trans}...\")\n\nprint(\"\\n\ud83d\udd11 DECODING KEY:\")\nprint(\" \u2022 lowercase-hyphenated = Akkadian syllables (a-na, i-na, \u0161a)\")\nprint(\" \u2022 ALL CAPS = Sumerian logograms (DUMU, K\u00d9.BABBAR, G\u00cdN)\")\nprint(\" \u2022 [...] = damaged/missing text\")\nprint(\" \u2022 Numbers = quantities (often silver amounts!)\")"}, {"cell_type": "markdown", "metadata": {}, "source": "### Why Translations Get LONGER\n\nHere's something fascinating: English translations are consistently **1.5x longer** than the Akkadian source!\n\nWhy? Akkadian packs multiple meanings into single words:\n\n```\nAkkadian: i-\u0161a-qal\nBreakdown: i (he) + \u0161aq\u0101l (to pay) + \u2205 (future tense)\nEnglish: \"he will pay\"\n\n1 word \u2192 3 words!\n```"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcca Prove it with data!\ntrain['src_words'] = train['transliteration'].str.split().str.len()\ntrain['tgt_words'] = train['translation'].str.split().str.len()\ntrain['expansion'] = train['tgt_words'] / train['src_words']\n\nfig, axes = plt.subplots(1, 2, figsize=(14, 5))\n\n# Scatter plot\naxes[0].scatter(train['src_words'], train['tgt_words'], alpha=0.4, c='#3498db', s=30)\naxes[0].plot([0, 200], [0, 200 * train['expansion'].mean()], 'r--', linewidth=2,\n label=f'Average expansion: {train[\"expansion\"].mean():.2f}x')\naxes[0].set_xlabel('Akkadian Words', fontsize=12)\naxes[0].set_ylabel('English Words', fontsize=12)\naxes[0].set_title('\ud83d\udcc8 Translation Expansion Pattern', fontsize=14, fontweight='bold')\naxes[0].legend(fontsize=11)\n\n# Distribution\naxes[1].hist(train['expansion'], bins=40, color='#9b59b6', edgecolor='white', alpha=0.8)\naxes[1].axvline(train['expansion'].median(), color='red', linestyle='--', linewidth=2,\n label=f'Median: {train[\"expansion\"].median():.2f}x')\naxes[1].set_xlabel('Expansion Ratio (English/Akkadian)', fontsize=12)\naxes[1].set_ylabel('Number of Documents', fontsize=12)\naxes[1].set_title('\ud83d\udcca How Much Longer Are Translations?', fontsize=14, fontweight='bold')\naxes[1].legend(fontsize=11)\n\nplt.tight_layout()\nplt.show()\n\nprint(f\"\\n\ud83d\udca1 KEY INSIGHT: Translations are {train['expansion'].median():.2f}x longer on average!\")\nprint(f\" This helps you estimate expected output length during generation.\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 4\ufe0f\u20e3 \ud83d\udcb0 The Silver Economy: What Are These Texts Actually About?\n\n## Meet the Ancient Merchants\n\nThese aren't random ancient texts. They're from **Kanesh** (modern Turkey), where Assyrian merchants ran a massive trade network around 1900 BC.\n\nThey traded:\n- \ud83e\udd48 **Silver** - The currency of the ancient world\n- \ud83d\udd29 **Tin** - Essential for making bronze weapons\n- \ud83d\udc54 **Textiles** - Luxury goods from Assyria\n\nThe texts are their business records: **debt notes, contracts, complaint letters, family correspondence.**"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udd0d Let's find the most common \"words\" - they reveal what the texts are about!\nall_translit = ' '.join(train['transliteration'].tolist())\n\n# Find Sumerian logograms (ALL CAPS) - these are the key terms\n# Include accented uppercase vowels: \u00d9, \u00cd, \u00c1, \u00c9, \u00da and special chars: \u0160, \u1e62, \u1e6c, \u1e2a\nlogogram_pattern = r'\\b[A-Z\u0160\u1e62\u1e6c\u1e2a\u00d9\u00cd\u00c1\u00c9\u00da][A-Z\u0160\u1e62\u1e6c\u1e2a\u00d9\u00cd\u00c1\u00c9\u00da\u2080-\u2089\\.]+\\b'\nlogograms = re.findall(logogram_pattern, all_translit)\nlogo_counts = Counter(logograms)\n\n# The meanings!\nmeanings = {\n 'K\u00d9.BABBAR': '\ud83d\udcb0 Silver (money!)',\n 'DUMU': '\ud83d\udc68\u200d\ud83d\udc66 Son of',\n 'KI\u0160IB': '\ud83d\udd0f Seal (signature)',\n 'G\u00cdN': '\u2696\ufe0f Shekel (weight unit)',\n 'AN.NA': '\ud83d\udd29 Tin (trade good)',\n 'T\u00daG': '\ud83d\udc54 Textile',\n 'ITI': '\ud83d\udcc5 Month',\n 'MU': '\ud83d\udcc6 Year',\n '\u00c9': '\ud83c\udfe0 House',\n 'DINGIR': '\u2728 God/divine',\n}\n\nprint(\"\ud83d\udcb0 THE SILVER ECONOMY - What These Texts Talk About\")\nprint(\"\u2550\" * 70)\nprint(f\"\\n{'LOGOGRAM':<15} {'COUNT':>8} {'MEANING'}\")\nprint(\"-\" * 50)\nfor logo, count in logo_counts.most_common(15):\n meaning = meanings.get(logo, '(unlisted)')\n print(f\" {logo:<13} {count:>8,} {meaning}\")\n\n# Safe access with .get() to avoid KeyError\nsilver_count = logo_counts.get('K\u00d9.BABBAR', 0)\n\nprint(\"\\n\" + \"\u2550\" * 70)\nprint(\"\\n\ud83e\udd2f MIND-BLOWING INSIGHT:\")\nif silver_count > 0:\n print(f\" K\u00d9.BABBAR (silver) appears {silver_count:,} times!\")\n print(f\" That's {silver_count/len(train):.1f} times per document on average!\")\n print(\"\\n These merchants were OBSESSED with money. Some things never change! \ud83d\ude04\")\nelse:\n # Show top term instead\n top_logo, top_count = logo_counts.most_common(1)[0]\n print(f\" The most common logogram is {top_logo} with {top_count:,} occurrences!\")\n print(f\" That's {top_count/len(train):.1f} times per document on average!\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcca Visualize the ancient economy\ntop_15 = logo_counts.most_common(15)\n\nfig, ax = plt.subplots(figsize=(12, 8))\n\nnames = [f\"{x[0]}\" for x in top_15]\ncounts = [x[1] for x in top_15]\ncolors_map = ['#f1c40f' if 'K\u00d9' in n or 'G\u00cdN' in n else '#3498db' for n in names] # Gold for money terms!\n\nbars = ax.barh(names[::-1], counts[::-1], color=colors_map[::-1], edgecolor='white', linewidth=2)\n\nfor bar, count in zip(bars, counts[::-1]):\n ax.text(bar.get_width() + 50, bar.get_y() + bar.get_height()/2,\n f'{count:,}', ha='left', va='center', fontsize=10, fontweight='bold')\n\nax.set_xlabel('Frequency', fontsize=12)\nax.set_title('\ud83d\udcb0 The Ancient Economy: What These Tablets Talk About\\n(Gold = money terms!)',\n fontsize=14, fontweight='bold')\n\nplt.tight_layout()\nplt.show()\n\nprint(\"\\n\ud83d\udca1 The gold bars are MONEY-related terms.\")\nprint(\" These weren't philosophers - they were BUSINESSPEOPLE! \ud83d\udcbc\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udd0d Find some juicy translation examples!\nall_translations = ' '.join(train['translation'].tolist()).lower()\n\n# Search for interesting phrases\ninteresting_phrases = [\n ('owes', 'Debt collection! \ud83d\udcb8'),\n ('will pay', 'Future payment promises'),\n ('shekels of silver', 'Money amounts \ud83d\udcb0'),\n ('witnessed by', 'Legal contracts \ud83d\udcdc'),\n ('seal of', 'Signatures \ud83d\udd0f'),\n ('son of', 'Family lineage \ud83d\udc68\u200d\ud83d\udc66'),\n ('brother', 'Family or business partner'),\n ('caravan', 'Trade logistics \ud83d\udc2a'),\n ('tin', 'Trade commodity'),\n ('textile', 'Trade commodity \ud83d\udc54'),\n]\n\nprint(\"\ud83d\udcdc WHAT STORIES DO THESE TEXTS TELL?\")\nprint(\"\u2550\" * 60)\nprint(f\"\\n{'PHRASE':<25} {'COUNT':>8} {'MEANING'}\")\nprint(\"-\" * 60)\nfor phrase, desc in interesting_phrases:\n count = all_translations.count(phrase)\n print(f\" {phrase:<23} {count:>8,} {desc}\")\n\nprint(\"\\n\" + \"\u2550\" * 60)\nprint(\"\\n\ud83d\udcd6 THE STORY REVEALED:\")\nprint(\" These are debt notes, contracts, and business letters!\")\nprint(\" Ancient merchants arguing about payments, just like today! \ud83d\ude04\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83c\udfad Let's find an actual interesting translation!\nprint(\"\ud83c\udfad REAL TRANSLATIONS - Ancient Drama!\")\nprint(\"\u2550\" * 70)\n\n# Find texts mentioning debt or payment\nfound_example = False\nfor idx, row in train.iterrows():\n trans = row['translation'].lower()\n if 'owes' in trans or 'debt' in trans or 'pay' in trans:\n if 50 < len(row['translation']) < 400: # Nice readable length\n doc_id = row.get('oare_id', f'#{idx+1}')\n print(f\"\\n\ud83d\udcdc DOCUMENT {doc_id}:\")\n print(\"-\" * 70)\n print(f\"\\n{row['translation'][:500]}\")\n if len(row['translation']) > 500:\n print(\"...\")\n print(\"\\n\" + \"-\" * 70)\n found_example = True\n break\n\nif not found_example:\n print(\"\\n No suitable example found.\")\n\nprint(\"\\n\ud83d\udca1 This is a 4,000-year-old IOU! The more things change...\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 5\ufe0f\u20e3 \ud83d\udd2c ByT5 Deep Dive: Why Bytes Beat Words\n\n## The Secret Weapon of the Leaderboard\n\nEvery top solution uses **ByT5** (Byte-level T5). Here's why:\n\n### The Problem with Word-Level Models\n\nTraditional models (mT5, NLLB) use **tokenizers** that:\n- \u274c Struggle with rare characters (\u0161, \u1e2b, \u1e63, \u1e6d)\n- \u274c Create weird subword splits\n- \u274c Can't handle damaged text markers [...] well\n\n### Why ByT5 Wins\n\nByT5 processes text as **raw bytes** (UTF-8):\n- \u2705 Every character is handled naturally\n- \u2705 No tokenizer to break on rare Unicode\n- \u2705 Handles noise and damage markers gracefully\n- \u2705 Research-proven: 80%+ accuracy on cuneiform (ACL 2025)"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udd0d Character analysis - what special characters do we have?\nall_chars = ''.join(train['transliteration'].tolist())\nchar_counts = Counter(all_chars)\n\n# Separate by type\nascii_chars = {c: n for c, n in char_counts.items() if ord(c) < 128}\nspecial_chars = {c: n for c, n in char_counts.items() if ord(c) >= 128}\n\nprint(\"\ud83d\udd24 CHARACTER ANALYSIS\")\nprint(\"\u2550\" * 60)\nprint(f\"\\n Total characters: {len(all_chars):,}\")\nprint(f\" Unique characters: {len(char_counts)}\")\nprint(f\" ASCII characters: {len(ascii_chars)}\")\nprint(f\" Special Unicode: {len(special_chars)} \u2190 These need special handling!\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83c\udf1f The Akkadian alphabet - special characters\nakkadian_special = {\n '\u0161': ('shin', 'as in \"ship\"'),\n '\u1e63': ('tsade', 'emphatic s'),\n '\u1e6d': ('teth', 'emphatic t'),\n '\u1e2b': ('heth', 'like German \"ch\"'),\n '\u0101': ('long a', 'extended vowel'),\n '\u0113': ('long e', 'extended vowel'),\n '\u012b': ('long i', 'extended vowel'),\n '\u016b': ('long u', 'extended vowel'),\n}\n\nprint(\"\u2728 AKKADIAN SPECIAL CHARACTERS\")\nprint(\"\u2550\" * 70)\nprint(f\"\\n{'CHAR':<6} {'NAME':<12} {'SOUND':<20} {'COUNT':>10} {'UNICODE'}\")\nprint(\"-\" * 70)\nfor char, (name, sound) in akkadian_special.items():\n count = char_counts.get(char, 0) + char_counts.get(char.upper(), 0)\n print(f\" {char:<4} {name:<12} {sound:<20} {count:>10,} U+{ord(char):04X}\")\n\nprint(\"\\n\ud83d\udca1 INSIGHT: \u0161 and \u1e2b appear tens of thousands of times!\")\nprint(\" A tokenizer that mishandles these will FAIL. ByT5 handles them naturally.\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcca What does ByT5 actually see? Byte analysis!\nall_bytes = all_chars.encode('utf-8')\n\n# Calculate multi-byte characters\nmulti_byte_count = sum(1 for c in all_chars if len(c.encode('utf-8')) > 1)\nmulti_byte_pct = 100 * multi_byte_count / len(all_chars)\n\nprint(\"\ud83d\udd2c BYTE-LEVEL VIEW (What ByT5 Sees)\")\nprint(\"\u2550\" * 60)\nprint(f\"\\n Total characters: {len(all_chars):,}\")\nprint(f\" Total bytes: {len(all_bytes):,}\")\nprint(f\" Bytes per character: {len(all_bytes)/len(all_chars):.3f}\")\nprint(f\"\\n Multi-byte characters: {multi_byte_count:,} ({multi_byte_pct:.1f}%)\")\nprint(f\"\\n\ud83d\udca1 ~{multi_byte_pct:.0f}% of characters need 2+ bytes (special Akkadian chars).\")\nprint(f\" ByT5 handles this seamlessly - no tokenizer headaches!\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udccf Sequence length analysis - critical for max_length setting!\ntrain['src_bytes'] = train['transliteration'].apply(lambda x: len(str(x).encode('utf-8')))\ntrain['tgt_bytes'] = train['translation'].apply(lambda x: len(str(x).encode('utf-8')))\n\nfig, axes = plt.subplots(1, 2, figsize=(14, 5))\n\n# Source\naxes[0].hist(train['src_bytes'], bins=50, color='#3498db', edgecolor='white', alpha=0.8)\np95 = train['src_bytes'].quantile(0.95)\naxes[0].axvline(p95, color='red', linestyle='--', linewidth=2, label=f'95th percentile: {p95:.0f}')\naxes[0].axvline(512, color='green', linestyle=':', linewidth=2, label='max_length=512')\naxes[0].set_xlabel('Bytes', fontsize=12)\naxes[0].set_ylabel('Frequency', fontsize=12)\naxes[0].set_title('\ud83d\udce5 Source (Akkadian) Byte Length', fontsize=14, fontweight='bold')\naxes[0].legend()\n\n# Target\naxes[1].hist(train['tgt_bytes'], bins=50, color='#e74c3c', edgecolor='white', alpha=0.8)\np95_tgt = train['tgt_bytes'].quantile(0.95)\naxes[1].axvline(p95_tgt, color='red', linestyle='--', linewidth=2, label=f'95th percentile: {p95_tgt:.0f}')\naxes[1].axvline(512, color='green', linestyle=':', linewidth=2, label='max_length=512')\naxes[1].set_xlabel('Bytes', fontsize=12)\naxes[1].set_ylabel('Frequency', fontsize=12)\naxes[1].set_title('\ud83d\udce4 Target (English) Byte Length', fontsize=14, fontweight='bold')\naxes[1].legend()\n\nplt.tight_layout()\nplt.show()\n\nsrc_coverage = (train['src_bytes'] <= 512).mean() * 100\ntgt_coverage = (train['tgt_bytes'] <= 512).mean() * 100\n\nprint(f\"\\n\ud83d\udcd0 SEQUENCE LENGTH RECOMMENDATION:\")\nprint(f\" max_length=512 covers {src_coverage:.1f}% of sources, {tgt_coverage:.1f}% of targets\")\nprint(f\"\\n \u2705 512 is a good default for most cases!\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 6\ufe0f\u20e3 \ud83d\udc94 Damage Report: How Broken Are These Tablets?\n\n## Ancient tablets aren't perfectly preserved\n\n4,000 years takes its toll. Parts are:\n- **[...]** - Missing or broken\n- **x** - Illegible sign\n- **?** - Uncertain reading\n- **!** - Corrected by modern scholars\n\nUnderstanding damage patterns helps your model handle uncertainty!"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udc94 Damage analysis\ndef count_damage(text):\n text = str(text)\n return {\n 'brackets': len(re.findall(r'\\[', text)),\n 'x_marks': len(re.findall(r'\\bx\\b', text, re.I)),\n 'questions': text.count('?'),\n 'ellipsis': text.count('...') + text.count('\u2026'),\n }\n\ndamage_df = train['transliteration'].apply(count_damage).apply(pd.Series)\ndamage_df['total'] = damage_df.sum(axis=1)\n\nprint(\"\ud83d\udc94 DAMAGE REPORT\")\nprint(\"\u2550\" * 60)\nprint(f\"\\n{'MARKER':<15} {'DOCS WITH':>10} {'TOTAL':>10}\")\nprint(\"-\" * 40)\nfor col in ['brackets', 'x_marks', 'questions', 'ellipsis']:\n docs_with = (damage_df[col] > 0).sum()\n total = damage_df[col].sum()\n pct = docs_with / len(train) * 100\n print(f\" {col:<13} {docs_with:>10} ({pct:>4.1f}%) {total:>8,}\")\n\nprint(\"\\n\" + \"-\" * 40)\ndamaged_docs = (damage_df['total'] > 0).sum()\nprint(f\" {'ANY DAMAGE':<13} {damaged_docs:>10} ({damaged_docs/len(train)*100:.1f}%)\")"}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udcca Damage distribution\nfig, axes = plt.subplots(1, 2, figsize=(14, 5))\n\n# Histogram\naxes[0].hist(damage_df['total'], bins=30, color='#e74c3c', edgecolor='white', alpha=0.8)\naxes[0].axvline(damage_df['total'].median(), color='black', linestyle='--', \n label=f'Median: {damage_df[\"total\"].median():.0f}')\naxes[0].set_xlabel('Number of Damage Markers', fontsize=12)\naxes[0].set_ylabel('Number of Documents', fontsize=12)\naxes[0].set_title('\ud83d\udc94 Damage Distribution', fontsize=14, fontweight='bold')\naxes[0].legend()\n\n# Pie chart of damage types\ndamage_totals = {\n '[...] Brackets': damage_df['brackets'].sum(),\n 'x Illegible': damage_df['x_marks'].sum(),\n '? Uncertain': damage_df['questions'].sum(),\n '... Gaps': damage_df['ellipsis'].sum(),\n}\ncolors = ['#e74c3c', '#3498db', '#f39c12', '#9b59b6']\naxes[1].pie(damage_totals.values(), labels=damage_totals.keys(), autopct='%1.1f%%',\n colors=colors, startangle=90, explode=(0.05, 0, 0, 0))\naxes[1].set_title('\ud83d\udcca Types of Damage', fontsize=14, fontweight='bold')\n\nplt.tight_layout()\nplt.show()\n\nprint(\"\\n\ud83d\udca1 INSIGHT: Square brackets (reconstructed text) are the most common.\")\nprint(\" Your model needs to learn that [...] means 'something was here'.\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# 7\ufe0f\u20e3 \ud83d\udee0\ufe0f Ready-to-Use Code: The Preprocessing Pipeline\n\n## Copy, paste, and start training!\n\nHere are battle-tested functions for preparing your data."}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# \ud83d\udee0\ufe0f PREPROCESSING TOOLKIT\n\ndef normalize_akkadian(text):\n \"\"\"\n Clean up Akkadian transliteration for model input.\n \"\"\"\n text = str(text)\n text = re.sub(r'\\s+', ' ', text) # Normalize whitespace\n text = re.sub(r'\\bx+\\b', 'x', text, flags=re.I) # Normalize illegible markers\n text = re.sub(r'\\.{3,}', '...', text) # Normalize ellipsis\n return text.strip()\n\n\ndef prepare_for_byt5(df, direction='akk2en'):\n \"\"\"\n Prepare data for ByT5 training.\n \n Args:\n df: DataFrame with 'transliteration' and 'translation' columns\n direction: 'akk2en' or 'en2akk'\n \n Returns:\n DataFrame with 'input_text' and 'target_text' columns\n \"\"\"\n result = df.copy()\n \n if direction == 'akk2en':\n prefix = \"translate Akkadian to English: \"\n result['input_text'] = prefix + result['transliteration'].apply(normalize_akkadian)\n result['target_text'] = result['translation'].astype(str)\n else:\n prefix = \"translate English to Akkadian: \"\n result['input_text'] = prefix + result['translation'].astype(str)\n result['target_text'] = result['transliteration'].apply(normalize_akkadian)\n \n return result[['input_text', 'target_text']]\n\n\ndef create_bidirectional_data(df):\n \"\"\"\n Double your training data with bidirectional pairs!\n \n Trains both Akkadian\u2192English AND English\u2192Akkadian.\n This helps the model learn better representations.\n \"\"\"\n forward = prepare_for_byt5(df, 'akk2en')\n backward = prepare_for_byt5(df, 'en2akk')\n \n combined = pd.concat([forward, backward], ignore_index=True)\n return combined.sample(frac=1, random_state=42).reset_index(drop=True)\n\n\n# \ud83c\udfaf DEMO\nprint(\"\ud83d\udee0\ufe0f PREPROCESSING DEMO\")\nprint(\"\u2550\" * 70)\n\nsample = train.iloc[0]['transliteration'][:150]\nprint(f\"\\n\ud83d\udce5 ORIGINAL:\")\nprint(f\" {sample}...\")\nprint(f\"\\n\ud83d\udce4 NORMALIZED:\")\nprint(f\" {normalize_akkadian(sample)}...\")\nprint(f\"\\n\ud83c\udff7\ufe0f WITH PREFIX:\")\nprint(f\" translate Akkadian to English: {normalize_akkadian(sample)[:80]}...\")\n\n# Show data augmentation\nbidirectional = create_bidirectional_data(train)\nprint(f\"\\n\ud83d\udcc8 DATA AUGMENTATION:\")\nprint(f\" Original samples: {len(train):,}\")\nprint(f\" After bidirectional: {len(bidirectional):,} (2x!)\")"}, {"cell_type": "markdown", "metadata": {}, "source": "---\n\n# \ud83d\udccb Summary: Key Numbers to Remember\n\n| Metric | Value | Why It Matters |\n|--------|-------|----------------|\n| Training pairs | 1,561 | Ultra low-resource! |\n| Test samples | 4 | All from ONE document |\n| Translation expansion | 1.5x | Expect longer outputs |\n| Lexicon coverage | ~70% | Dictionary helps |\n| Damaged documents | ~75% | Handle [...] gracefully |\n| Recommended max_length | 512 | Covers 95%+ of data |\n\n---\n\n# \ud83d\ude80 Recommended Training Config\n\n```python\n# Winning configuration for ByT5\nconfig = {\n 'model': 'google/byt5-base', # NOT byt5-small!\n 'max_length': 512,\n 'batch_size': 4,\n 'gradient_accumulation': 8,\n 'learning_rate': 1e-4,\n 'epochs': 15-20, # More is better\n 'fp16': False, # CRITICAL! ByT5 + fp16 = NaN!\n 'label_smoothing': 0.2,\n 'optimizer': 'adafactor',\n}\n```\n\n---\n\n## \u2b50 Did this notebook help you? Please upvote! \u2b50\n\nIt takes just one click and motivates me to create more content! \ud83d\ude4f\n\n**Questions?** Drop a comment below!\n\n---\n\n## \ud83d\udcda References\n\n1. Sommerschield et al. (2023). \"Machine learning for ancient languages: A survey.\" *Computational Linguistics*.\n2. ACL Workshop on Ancient Language Processing (2025). \"ByT5 for Cuneiform Lemmatization.\"\n3. Xue et al. (2022). \"ByT5: Towards a token-free future.\" *TACL*.\n\n---\n\n*Created with \u2764\ufe0f for the Deep Past Challenge community*"}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"name": "python", "version": "3.10.0"}}, "nbformat": 4, "nbformat_minor": 4}