{"cells":[{"cell_type":"markdown","id":"md_title","metadata":{},"source":"# Deep Past Translation — Starter Guide\n\n> **Competition:** Deep Past Initiative: Machine Translation  \n> **Task:** Translate Old Assyrian cuneiform transliterations into English  \n> **Metric:** `sqrt(BLEU × chrF++)` — geometric mean of word-level and character-level scores\n\nThis notebook walks you through:\n1. Understanding the data and domain\n2. Key preprocessing challenges\n3. The evaluation metric in detail\n4. A TF-IDF retrieval baseline (~4.4 score)\n5. Pointers to the ByT5 approach that reaches 33–35+\n\n---\n## What is this competition about?\n\nThe tablets in this dataset are ~3,900-year-old merchant records written in **Old Assyrian** (a dialect of Akkadian cuneiform). They document loans, contracts, and trade routes between Mesopotamia and Anatolia. The transliterations use Latin characters to represent cuneiform sounds — your job is to render them in readable English.\n\n**Why is this hard?**\n- Training data is tiny: only 1,562 examples\n- The language is ancient, highly inflected, and uses scribal shorthand\n- Gaps (`[...]`, `x`) indicate damaged or illegible tablet sections\n- Subscript numbers (`₁`, `₂`) distinguish homophonous signs\n- There are only **4 test sentences** — every word matters enormously"},{"cell_type":"code","execution_count":null,"id":"cell_setup","metadata":{},"outputs":[],"source":"import os\nimport re\nimport math\nimport warnings\nfrom pathlib import Path\nfrom collections import Counter\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nwarnings.filterwarnings('ignore')\n\n# ── Paths (works both locally and on Kaggle) ──────────────────────────────────\nCOMP = 'deep-past-initiative-machine-translation'\nKAGGLE_DIR = Path(f'/kaggle/input/{COMP}')\nLOCAL_DIR  = Path('data/raw')          # adjust if running locally\n\nDATA_DIR = KAGGLE_DIR if KAGGLE_DIR.exists() else LOCAL_DIR\nprint(f'Data directory: {DATA_DIR}')\n\ntrain_df = pd.read_csv(DATA_DIR / 'train.csv')\ntest_df  = pd.read_csv(DATA_DIR / 'test.csv')\nprint(f'Train: {train_df.shape} | Test: {test_df.shape}')"},{"cell_type":"markdown","id":"md_eda","metadata":{},"source":"## 1. Exploring the Data"},{"cell_type":"code","execution_count":null,"id":"cell_eda","metadata":{},"outputs":[],"source":"print('=== TRAINING SET OVERVIEW ===')\nprint(f'Rows: {len(train_df):,}')\nprint(f'Columns: {train_df.columns.tolist()}')\n\n# Length distributions\ntrain_df['src_words'] = train_df['transliteration'].str.split().str.len()\ntrain_df['tgt_words'] = train_df['translation'].str.split().str.len()\n\nprint('\\nSource (transliteration) length in words:')\nprint(train_df['src_words'].describe().round(1).to_string())\n\nprint('\\nTarget (English translation) length in words:')\nprint(train_df['tgt_words'].describe().round(1).to_string())\n\nprint(f'\\nAvg expansion ratio (tgt/src): {(train_df.tgt_words/train_df.src_words).mean():.2f}x')\nprint(f'  → English is on average 1.5x longer than the Akkadian transliteration')"},{"cell_type":"code","execution_count":null,"id":"cell_examples","metadata":{},"outputs":[],"source":"print('=== EXAMPLE TRANSLATIONS ===')\nfor _, row in train_df.sample(3, random_state=42).iterrows():\n    print(f\"ID:  {row['oare_id']}\")\n    print(f\"SRC: {row['transliteration'][:120]}\")\n    print(f\"TGT: {row['translation'][:120]}\")\n    print()\n\nprint('=== THE 4 TEST SENTENCES ===')\nfor _, row in test_df.iterrows():\n    print(f\"ID: {row['id']} | Tablet: {row['text_id']} | Lines {row['line_start']}-{row['line_end']}\")\n    print(f\"SRC: {row['transliteration'][:120]}\")\n    print()"},{"cell_type":"markdown","id":"md_scribal","metadata":{},"source":"## 2. Scribal Notation Challenges\n\nOld Assyrian tablets use several notation conventions that require preprocessing:\n\n| Pattern | Meaning | Example | Our treatment |\n|---------|---------|---------|---------------|\n| `[...]`, `...` | Damaged / illegible text | `a-na [x x] libbi` | → `<big_gap>` |\n| `xx`, `x` | Single illegible sign | `a-na xx ma-tim` | → `<gap>` |\n| `₀₁₂₃₄₅₆₇₈₉` | Subscript sign differentiators | `qí-bi₄-ma` | → digits `0-9` |\n| `ḫ/Ḫ` | Pharyngeal fricative (not standard h) | `ḫa-bu-lum` | → `h/H` |\n| `!`, `?`, `˹˺`, `⌈⌋` | Scribal uncertainty markers | `an!-ni-tum` | remove |\n| `[x]`, square brackets | Tentative reading | `[a-na]` | remove brackets |\n\nThese normalizations help models that have seen standard ASCII text handle the unusual characters."},{"cell_type":"code","execution_count":null,"id":"cell_preprocess","metadata":{},"outputs":[],"source":"class AkkadianPreprocessor:\n    \"\"\"Normalize Old Assyrian transliterations for model input.\"\"\"\n\n    def __init__(self):\n        self.big_gap_re     = re.compile(r'(\\.{3,}|\\u2026+)')\n        self.small_gap_re   = re.compile(r'(xx+|\\s+x\\s+)')\n        self.h_trans        = str.maketrans('\\u1e2a\\u1e2b\\u1e24\\u1e25', 'HhHh')\n        self.subscript_map  = str.maketrans('\\u2080\\u2081\\u2082\\u2083\\u2084\\u2085\\u2086\\u2087\\u2088\\u2089\\u2093',\n                                            '0123456789x')\n        self.scribal_strip  = re.compile(r'[!?/\\u02f9\\u02fa\\u2308\\u2309\\u230a\\u230b]')\n        self.bracket_strip  = re.compile(r'[\\[\\]]')\n        self.whitespace_re  = re.compile(r'\\s+')\n\n    def clean(self, text: str) -> str:\n        if pd.isna(text):\n            return ''\n        text = str(text)\n        text = self.big_gap_re.sub('<big_gap>', text)\n        text = self.small_gap_re.sub('<gap>', text)\n        text = text.translate(self.h_trans)\n        text = text.translate(self.subscript_map)\n        text = self.scribal_strip.sub('', text)\n        text = self.bracket_strip.sub('', text)\n        return self.whitespace_re.sub(' ', text).strip()\n\n    def make_prompt(self, text: str) -> str:\n        \"\"\"Format for ByT5 seq2seq models.\"\"\"\n        return f'translate Akkadian to English: {self.clean(text)}'\n\n\npreprocessor = AkkadianPreprocessor()\n\n# Demo\nexamples = [\n    'a-na IŠTAR-lá-ma-sí ù ni-ta-aḫ-šu-šar qí-bi₄-ma',\n    '[x] a-na ... [libbi] ḫa-bu-lum !an-ni!',\n    'um-ma šu-ku-tum-ma xx ma-tim i-ša-ar₂-ru',\n]\nprint('Preprocessing examples:')\nfor ex in examples:\n    print(f'  IN:  {ex}')\n    print(f'  OUT: {preprocessor.clean(ex)}')\n    print()"},{"cell_type":"markdown","id":"md_metric","metadata":{},"source":"## 3. Understanding the Evaluation Metric\n\nThe competition scores with:\n\n$$\\text{score} = \\sqrt{\\text{BLEU} \\times \\text{chrF++}}$$\n\n- **BLEU** measures word-level n-gram overlap (1–4 grams) between your translation and the reference. Good at capturing correct word choices.\n- **chrF++** measures character-level n-gram overlap (1–6 chars). More forgiving of morphological variation and better for morphologically rich languages like Akkadian.\n- The **geometric mean** means *both* metrics must be high. A translation that nails word choice but butchers spelling (or vice versa) will score poorly.\n\n**Implication:** Your model must be good at both the vocabulary *and* the character-level form of each word."},{"cell_type":"code","execution_count":null,"id":"cell_metric","metadata":{},"outputs":[],"source":"def _char_ngrams(text: str, n: int) -> Counter:\n    return Counter(text[i:i+n] for i in range(len(text) - n + 1))\n\ndef _word_ngrams(tokens, n: int) -> Counter:\n    return Counter(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))\n\ndef chrf(hyp: str, ref: str, max_order: int = 6, beta: float = 2.0) -> float:\n    \"\"\"chrF++ score.\"\"\"\n    if not hyp or not ref:\n        return 0.0\n    tp = tr = valid = 0.0\n    for n in range(1, max_order + 1):\n        h = _char_ngrams(hyp, n); r = _char_ngrams(ref, n)\n        if not h or not r: continue\n        ov = sum((h & r).values())\n        tp += ov / sum(h.values())\n        tr += ov / sum(r.values())\n        valid += 1\n    if not valid: return 0.0\n    p, r = tp / valid, tr / valid\n    if p + r == 0: return 0.0\n    b2 = beta ** 2\n    return (1 + b2) * p * r / (b2 * p + r)\n\ndef bleu(hyp: str, ref: str, max_order: int = 4) -> float:\n    \"\"\"Sentence BLEU with add-1 smoothing.\"\"\"\n    h, r = hyp.lower().split(), ref.lower().split()\n    if not h or not r: return 0.0\n    bp = 1.0 if len(h) >= len(r) else math.exp(1 - len(r) / len(h))\n    log_p = 0.0\n    for n in range(1, max_order + 1):\n        hn, rn = _word_ngrams(h, n), _word_ngrams(r, n)\n        ov = sum((hn & rn).values())\n        log_p += math.log((ov + 1) / (max(sum(hn.values()), 1) + 1))\n    return bp * math.exp(log_p / max_order)\n\ndef competition_score(hyp: str, ref: str) -> float:\n    b, c = bleu(hyp, ref), chrf(hyp, ref)\n    return math.sqrt(b * c) if b > 0 and c > 0 else 0.0\n\n\n# Illustrate metric behaviour\nref = 'He gave ten shekels of silver to the merchant.'\ntests = [\n    ('Perfect match',  'He gave ten shekels of silver to the merchant.'),\n    ('Good word match, bad spelling', 'He gave ten shekl of silvr to the marchant.'),\n    ('Different words, right chars',  'He paid ten talents of copper at the harbor.'),\n    ('Completely wrong',              'Banana rainbow unicorn.'),\n]\nprint(f\"{'Label':<40} {'BLEU':>6} {'chrF':>6} {'Score':>6}\")\nprint('-' * 62)\nfor label, hyp in tests:\n    b, c = bleu(hyp, ref), chrf(hyp, ref)\n    s = competition_score(hyp, ref)\n    print(f\"{label:<40} {b:6.3f} {c:6.3f} {s:6.3f}\")"},{"cell_type":"markdown","id":"md_baseline","metadata":{},"source":"## 4. TF-IDF Retrieval Baseline (~4.4 score)\n\nThe simplest possible approach: for each test sentence, find the most similar training sentence (by character n-gram TF-IDF cosine similarity) and return its translation.\n\nThis works surprisingly well as a **safety net** — if the model fails completely, retrieval can save you from a 0 score. It also demonstrates how to handle the submission format."},{"cell_type":"code","execution_count":null,"id":"cell_retrieval","metadata":{},"outputs":[],"source":"# Clean all texts\ntrain_cleaned = train_df['transliteration'].apply(preprocessor.clean)\ntest_cleaned  = test_df['transliteration'].apply(preprocessor.clean)\n\n# Build TF-IDF index on character 3–5 n-grams\nvectorizer = TfidfVectorizer(\n    analyzer='char_wb',\n    ngram_range=(3, 5),\n    max_features=50_000,\n    sublinear_tf=True,\n)\ntrain_vecs = vectorizer.fit_transform(train_cleaned)\ntest_vecs  = vectorizer.transform(test_cleaned)\n\n# Retrieve nearest neighbor for each test sentence\nsims = cosine_similarity(test_vecs, train_vecs)  # (4, 1562)\nbest_idx = sims.argmax(axis=1)\nbest_sim  = sims.max(axis=1)\n\nretrieval_translations = train_df['translation'].iloc[best_idx].values\n\nprint('=== RETRIEVAL RESULTS ===')\nfor i, (row, trans, sim) in enumerate(\n    zip(test_df.itertuples(), retrieval_translations, best_sim)\n):\n    print(f'\\n[Test {i+1}] Similarity: {sim:.3f}')\n    print(f'  TEST:      {row.transliteration[:80]}')\n    print(f'  RETRIEVED: {trans[:80]}')"},{"cell_type":"code","execution_count":null,"id":"cell_retrieval_submit","metadata":{},"outputs":[],"source":"# Create submission\nsubmission = pd.DataFrame({\n    'id': test_df['id'],\n    'translation': retrieval_translations\n})\nprint('Submission preview:')\nprint(submission.to_string())\n\nsubmission.to_csv('/kaggle/working/submission.csv', index=False)\nprint('\\nSaved submission.csv')\nprint('Expected score: ~4.4 (retrieval baseline)')"},{"cell_type":"markdown","id":"md_byt5","metadata":{},"source":"## 5. Going Further: ByT5 + MBR Decoding\n\nThe retrieval baseline (~4.4) is far from competitive. The current state-of-the-art in this competition uses **ByT5** — a byte-level T5 variant that is excellent for morphologically rich languages because it operates at the character level.\n\n### Why ByT5?\n\n| Model | Tokenization | Strength for Akkadian |\n|-------|-------------|----------------------|\n| Standard T5 | Word-piece / BPE | Breaks Akkadian into meaningless subwords |\n| mT5 | SentencePiece | Better, but still struggles with cuneiform conventions |\n| **ByT5** | **Raw bytes (UTF-8)** | Handles every Unicode character natively; great for transliterated text |\n\n### Pipeline to reach 33–35+\n\n```\nAkkadian transliteration\n    ↓\nPreprocess (gaps, subscripts, scribal marks)\n    ↓\nByT5 fine-tuned on Akkadian data\n    ↓\nGenerate 20+ candidates (diverse beams + temperature sampling)\n    ↓\nMBR decoding: pick candidate maximizing sqrt(BLEU × chrF++)\n    ↓\nPostprocess (remove repeated phrases, capitalize, etc.)\n    ↓\nsubmission.csv\n```\n\n### Key techniques:\n\n1. **Pre-trained Akkadian ByT5** — use a model already fine-tuned on cuneiform data. Several are available on Kaggle datasets: search for `byt5-akkadian`. The best publicly available model reaches ~35 BLEU.\n\n2. **Minimum Bayes Risk (MBR) decoding** — instead of taking the single highest-probability output, generate 20+ diverse candidates and pick the one that best agrees with the others (measured by the competition metric). This consistently outperforms standard beam search by 1–2 BLEU points.\n\n3. **Diverse beam search** (`num_beam_groups=3`, `diversity_penalty=0.8`) — forces beam groups to explore different translation strategies, giving MBR more to work with.\n\n4. **Multi-temperature sampling** — sample at temperatures [0.6, 0.8, 1.0] to get candidates ranging from high-confidence (conservative) to high-diversity (creative).\n\n5. **Repetition penalty** (`repetition_penalty=1.3`) — prevents the model from repeating phrases, a common failure mode on long Akkadian sentences.\n\n### ⚠️ Pitfall: context prefix degrades scores\n\nPrepending the previous line's translation as context (`\"context: {prev} | translate Akkadian to English: {src}\"`) sounds appealing but **hurts performance** with pre-trained models — they were not trained on this prompt format and produce lower-quality outputs. Stick with the plain `\"translate Akkadian to English: {src}\"` prompt.\n\nSee the companion **advanced notebook** for the full implementation."},{"cell_type":"markdown","id":"md_resources","metadata":{},"source":"## 6. Resources\n\n### Datasets (attach to your notebook)\n- `assiaben/final-byt5` — contains `byt5-akkadian-optimized-34x`, the recommended base model\n- `assiaben/byt5-akkadian-mbr` — Mattia Angeli's 35.4 BLEU model (may require loading fixes)\n- Competition data: `deep-past-initiative-machine-translation`\n\n### Key packages\n```python\nfrom transformers import AutoTokenizer, AutoModelForSeq2SeqLM\nimport torch\n```\n\n### Published texts\nThe competition provides `published_texts.csv` (~11MB) — a reference library of known Assyrian tablet translations. Use it for:\n- Additional training signal (fine-tuning)\n- RAG-style retrieval augmentation\n- Vocabulary lookup\n\n### Leaderboard context\n| Approach | Expected score |\n|----------|---------------|\n| TF-IDF retrieval (this notebook) | ~4.4 |\n| ByT5 + standard beam search | ~28–30 |\n| ByT5 + MBR (chrF utility) | ~33–34 |\n| ByT5 + MBR (BLEU×chrF utility) + diverse beams | ~35+ |\n| Top of leaderboard | ~39–40 |\n\n---\n\nGood luck! If you found this helpful, please upvote — it helps others discover it.\n\n*The full MBR pipeline is available in the companion advanced notebook.*"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.0"}},"nbformat":4,"nbformat_minor":5}