{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":121150,"databundleVersionId":15745837,"isSourceIdPinned":false}],"dockerImageVersionId":31260,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🏺 Deep Past Challenge — Akkadian to English Machine Translation\n### Exploratory Data Analysis (EDA) Notebook\n\n**Competition:** [Deep Past Challenge - Translate Akkadian to English](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview)  \n**Task:** Neural Machine Translation — Transliterated Akkadian (Old Assyrian) → English  \n**Evaluation:** Geometric Mean of BLEU and chrF++ scores (micro-averaged)  \n**Prize Pool:** $50,000  \n\n> *\"Four thousand years ago, Assyrian merchants left behind one of the world's richest archives of everyday life...\"*  \n  \n---","metadata":{}},{"cell_type":"markdown","source":"## ⚙️ CONFIG\n> **Adjust paths here before running on Kaggle or locally.**","metadata":{}},{"cell_type":"code","source":"# ============================================================\n#  CONFIG — ADJUST THESE PATHS FOR YOUR ENVIRONMENT\n# ============================================================\n\nimport os\n\n# ----------------------------\n# ENVIRONMENT TOGGLE\n# Set to True when running on Kaggle, False for local\n# ----------------------------\nKAGGLE = False   # <-- change to True on Kaggle\n\nif KAGGLE:\n    BASE_DIR = \"/kaggle/input/deep-past-initiative-machine-translation\"\nelse:\n    # ---- LOCAL PATH — update to your local folder ----\n    BASE_DIR = r\"/kaggle/input/deep-past-initiative-machine-translation\"\n\n# ----------------------------\n# FILE PATHS (auto-built from BASE_DIR)\n# ----------------------------\nTRAIN_PATH         = os.path.join(BASE_DIR, \"train.csv\")\nTEST_PATH          = os.path.join(BASE_DIR, \"test.csv\")\nSAMPLE_SUB_PATH    = os.path.join(BASE_DIR, \"sample_submission.csv\")\nPUB_TEXTS_PATH     = os.path.join(BASE_DIR, \"published_texts.csv\")\nPUBLICATIONS_PATH  = os.path.join(BASE_DIR, \"publications.csv\")\nBIBLIOGRAPHY_PATH  = os.path.join(BASE_DIR, \"bibliography.csv\")\nLEXICON_PATH       = os.path.join(BASE_DIR, \"OA_Lexicon_eBL.csv\")\nDICTIONARY_PATH    = os.path.join(BASE_DIR, \"eBL_Dictionary.csv\")\nRESOURCES_PATH     = os.path.join(BASE_DIR, \"resources.csv\")\nSENTENCES_PATH     = os.path.join(BASE_DIR, \"Sentences_Oare_FirstWord_LinNum.csv\")\n\n# ----------------------------\n# PLOTTING & DISPLAY SETTINGS\n# ----------------------------\nRANDOM_SEED  = 42\nFIGURE_DPI   = 120\nMAX_WORDCLOUD_WORDS = 200\n\nprint(\"✅ CONFIG loaded.\")\nprint(f\"   Environment : {'KAGGLE' if KAGGLE else 'LOCAL'}\")\nprint(f\"   BASE_DIR    : {BASE_DIR}\")\nfor name, path in [\n    (\"train.csv\", TRAIN_PATH), (\"test.csv\", TEST_PATH),\n    (\"published_texts.csv\", PUB_TEXTS_PATH)\n]:\n    exists = \"✅\" if os.path.exists(path) else \"❌ NOT FOUND\"\n    print(f\"   {exists}  {name}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:00:59.969952Z","iopub.execute_input":"2026-02-18T22:00:59.970656Z","iopub.status.idle":"2026-02-18T22:00:59.980352Z","shell.execute_reply.started":"2026-02-18T22:00:59.970627Z","shell.execute_reply":"2026-02-18T22:00:59.979778Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📦 Imports & Setup","metadata":{}},{"cell_type":"code","source":"# Standard\nimport re\nimport warnings\nimport unicodedata\nfrom collections import Counter\n\n# Data\nimport numpy as np\nimport pandas as pd\n\n# Visualisation\nimport matplotlib.pyplot as plt\nimport matplotlib.ticker as mticker\nimport seaborn as sns\nfrom matplotlib.gridspec import GridSpec\n\n# Optional: WordCloud  (install if missing)\ntry:\n    from wordcloud import WordCloud\n    HAS_WORDCLOUD = True\nexcept ImportError:\n    HAS_WORDCLOUD = False\n    print(\"WordCloud not installed — pip install wordcloud\")\n\n# Optional: NLTK for quick tokenisation stats\ntry:\n    import nltk\n    from nltk.tokenize import word_tokenize\n    nltk.download('punkt', quiet=True)\n    nltk.download('punkt_tab', quiet=True)\n    HAS_NLTK = True\nexcept ImportError:\n    HAS_NLTK = False\n    print(\"NLTK not installed — some token stats will use simple split()\")\n\nwarnings.filterwarnings(\"ignore\")\npd.set_option(\"display.max_colwidth\", 120)\nsns.set_theme(style=\"whitegrid\", palette=\"muted\", font_scale=1.1)\nplt.rcParams[\"figure.dpi\"] = FIGURE_DPI\n\nprint(\"✅ Libraries imported.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:08.441226Z","iopub.execute_input":"2026-02-18T22:01:08.441511Z","iopub.status.idle":"2026-02-18T22:01:10.525670Z","shell.execute_reply.started":"2026-02-18T22:01:08.441479Z","shell.execute_reply":"2026-02-18T22:01:10.524729Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📂 Load Data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(TRAIN_PATH)\ntest  = pd.read_csv(TEST_PATH)\nsample_sub = pd.read_csv(SAMPLE_SUB_PATH)\n\nprint(f\"Train shape        : {train.shape}\")\nprint(f\"Test  shape        : {test.shape}\")\nprint(f\"Sample sub shape   : {sample_sub.shape}\")\n\n# Load supplemental files\npub_texts   = pd.read_csv(PUB_TEXTS_PATH,  low_memory=False) if os.path.exists(PUB_TEXTS_PATH)  else None\nlexicon     = pd.read_csv(LEXICON_PATH,    low_memory=False) if os.path.exists(LEXICON_PATH)     else None\ndictionary  = pd.read_csv(DICTIONARY_PATH, low_memory=False) if os.path.exists(DICTIONARY_PATH)  else None\nbibliography= pd.read_csv(BIBLIOGRAPHY_PATH, low_memory=False) if os.path.exists(BIBLIOGRAPHY_PATH) else None\n\nprint(f\"\\nPublished texts    : {pub_texts.shape  if pub_texts  is not None else 'not loaded'}\")\nprint(f\"OA Lexicon         : {lexicon.shape    if lexicon    is not None else 'not loaded'}\")\nprint(f\"eBL Dictionary     : {dictionary.shape if dictionary is not None else 'not loaded'}\")\nprint(f\"Bibliography       : {bibliography.shape if bibliography is not None else 'not loaded'}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:13.444133Z","iopub.execute_input":"2026-02-18T22:01:13.444547Z","iopub.status.idle":"2026-02-18T22:01:13.924389Z","shell.execute_reply.started":"2026-02-18T22:01:13.444521Z","shell.execute_reply":"2026-02-18T22:01:13.923766Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Quick glance at train\ntrain.head(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:17.295486Z","iopub.execute_input":"2026-02-18T22:01:17.296147Z","iopub.status.idle":"2026-02-18T22:01:17.317121Z","shell.execute_reply.started":"2026-02-18T22:01:17.296119Z","shell.execute_reply":"2026-02-18T22:01:17.316530Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Quick glance at test\ntest.head(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:19.306780Z","iopub.execute_input":"2026-02-18T22:01:19.307286Z","iopub.status.idle":"2026-02-18T22:01:19.315315Z","shell.execute_reply.started":"2026-02-18T22:01:19.307258Z","shell.execute_reply":"2026-02-18T22:01:19.314713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔍 Basic Statistics & Missing Values","metadata":{}},{"cell_type":"code","source":"print(\"=\" * 55)\nprint(\"TRAIN — DATA TYPES & NULL COUNTS\")\nprint(\"=\" * 55)\nprint(train.dtypes)\nprint(\"\\nNull counts:\")\nprint(train.isnull().sum())\n\nprint(\"\\n\" + \"=\" * 55)\nprint(\"TEST — DATA TYPES & NULL COUNTS\")\nprint(\"=\" * 55)\nprint(test.dtypes)\nprint(\"\\nNull counts:\")\nprint(test.isnull().sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:21.610606Z","iopub.execute_input":"2026-02-18T22:01:21.610856Z","iopub.status.idle":"2026-02-18T22:01:21.619607Z","shell.execute_reply.started":"2026-02-18T22:01:21.610838Z","shell.execute_reply":"2026-02-18T22:01:21.618925Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualise missing values\nfig, axes = plt.subplots(1, 2, figsize=(12, 4))\n\nfor ax, df, title in zip(axes, [train, test], [\"Train\", \"Test\"]):\n    null_pct = df.isnull().mean() * 100\n    null_pct.plot(kind=\"bar\", ax=ax, color=\"#e07b5a\", edgecolor=\"black\")\n    ax.set_title(f\"{title} — Missing Values (%)\", fontsize=13)\n    ax.set_ylabel(\"% Missing\")\n    ax.set_ylim(0, 105)\n    ax.axhline(0, color=\"gray\", linewidth=0.5)\n    for bar in ax.patches:\n        ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 1,\n                f\"{bar.get_height():.1f}%\", ha='center', va='bottom', fontsize=9)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:26.810060Z","iopub.execute_input":"2026-02-18T22:01:26.810352Z","iopub.status.idle":"2026-02-18T22:01:27.175364Z","shell.execute_reply.started":"2026-02-18T22:01:26.810328Z","shell.execute_reply":"2026-02-18T22:01:27.174543Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📏 Text Length Analysis","metadata":{}},{"cell_type":"code","source":"def tokenize(text):\n    \"\"\"Token count — uses NLTK if available, else simple split.\"\"\"\n    if not isinstance(text, str):\n        return []\n    if HAS_NLTK:\n        return word_tokenize(text)\n    return text.split()\n\n# Character and token lengths\ntrain[\"src_char_len\"]  = train[\"transliteration\"].fillna(\"\").str.len()\ntrain[\"tgt_char_len\"]  = train[\"translation\"].fillna(\"\").str.len()\ntrain[\"src_tok_len\"]   = train[\"transliteration\"].fillna(\"\").apply(lambda x: len(tokenize(x)))\ntrain[\"tgt_tok_len\"]   = train[\"translation\"].fillna(\"\").apply(lambda x: len(tokenize(x)))\ntrain[\"src_word_len\"]  = train[\"transliteration\"].fillna(\"\").apply(lambda x: len(x.split()))\ntrain[\"tgt_word_len\"]  = train[\"translation\"].fillna(\"\").apply(lambda x: len(x.split()))\ntrain[\"len_ratio\"]     = train[\"tgt_word_len\"] / (train[\"src_word_len\"] + 1e-6)\n\ntest[\"src_char_len\"]   = test[\"transliteration\"].fillna(\"\").str.len()\ntest[\"src_tok_len\"]    = test[\"transliteration\"].fillna(\"\").apply(lambda x: len(tokenize(x)))\n\nprint(\"Train — transliteration stats (words):\")\nprint(train[[\"src_word_len\", \"tgt_word_len\", \"len_ratio\"]].describe().round(2))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:30.895392Z","iopub.execute_input":"2026-02-18T22:01:30.895920Z","iopub.status.idle":"2026-02-18T22:01:32.106137Z","shell.execute_reply.started":"2026-02-18T22:01:30.895881Z","shell.execute_reply":"2026-02-18T22:01:32.105395Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axes = plt.subplots(2, 3, figsize=(18, 10))\n\nplot_specs = [\n    (train[\"src_char_len\"],  \"#5a8ecc\", \"Source (Akkadian) Char Length\",   \"Train\"),\n    (train[\"tgt_char_len\"],  \"#e07b5a\", \"Target (English) Char Length\",    \"Train\"),\n    (train[\"src_word_len\"],  \"#5aac8e\", \"Source Word Count\",                \"Train\"),\n    (train[\"tgt_word_len\"],  \"#cc8e5a\", \"Target Word Count\",                \"Train\"),\n    (train[\"len_ratio\"],     \"#8e5acc\", \"Translation Length Ratio (tgt/src)\", \"Train\"),\n    (test[\"src_char_len\"],   \"#5a8ecc\", \"Source (Akkadian) Char Length\",   \"Test\" ),\n]\n\nfor ax, (series, color, title, split) in zip(axes.flatten(), plot_specs):\n    ax.hist(series, bins=50, color=color, edgecolor=\"white\", alpha=0.85)\n    ax.set_title(f\"{split} — {title}\", fontsize=11)\n    ax.set_xlabel(\"Length\")\n    ax.set_ylabel(\"Count\")\n    mean_val = series.mean()\n    ax.axvline(mean_val, color=\"crimson\", linestyle=\"--\", linewidth=1.5, label=f\"Mean: {mean_val:.1f}\")\n    ax.legend(fontsize=9)\n\nplt.suptitle(\"Text Length Distributions\", fontsize=15, fontweight=\"bold\", y=1.01)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:34.274249Z","iopub.execute_input":"2026-02-18T22:01:34.274933Z","iopub.status.idle":"2026-02-18T22:01:35.733263Z","shell.execute_reply.started":"2026-02-18T22:01:34.274901Z","shell.execute_reply":"2026-02-18T22:01:35.732550Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Scatter: source length vs target length\nfig, ax = plt.subplots(figsize=(8, 5))\nax.scatter(train[\"src_word_len\"], train[\"tgt_word_len\"], alpha=0.25, s=15, color=\"#5a8ecc\")\n\n# Regression line\nm, b = np.polyfit(train[\"src_word_len\"], train[\"tgt_word_len\"], 1)\nx_line = np.linspace(0, train[\"src_word_len\"].max(), 200)\nax.plot(x_line, m * x_line + b, color=\"crimson\", linewidth=2, label=f\"y={m:.2f}x+{b:.2f}\")\n\ncorr = train[\"src_word_len\"].corr(train[\"tgt_word_len\"])\nax.set_title(f\"Source vs Target Word Count  (Pearson r = {corr:.3f})\", fontsize=13)\nax.set_xlabel(\"Source (Akkadian) Word Count\")\nax.set_ylabel(\"Target (English) Word Count\")\nax.legend()\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:40.081461Z","iopub.execute_input":"2026-02-18T22:01:40.081753Z","iopub.status.idle":"2026-02-18T22:01:40.275903Z","shell.execute_reply.started":"2026-02-18T22:01:40.081727Z","shell.execute_reply":"2026-02-18T22:01:40.275135Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔤 Vocabulary & Token Analysis","metadata":{}},{"cell_type":"code","source":"def get_vocab(series, lowercase=False):\n    \"\"\"Build word frequency counter from a text series.\"\"\"\n    words = []\n    for text in series.dropna():\n        tokens = text.lower().split() if lowercase else text.split()\n        words.extend(tokens)\n    return Counter(words)\n\nsrc_vocab = get_vocab(train[\"transliteration\"], lowercase=True)\ntgt_vocab = get_vocab(train[\"translation\"],     lowercase=True)\n\nprint(f\"Source vocab size  : {len(src_vocab):,}\")\nprint(f\"Target vocab size  : {len(tgt_vocab):,}\")\nprint(f\"\\nTop-20 Source (Akkadian) tokens:\")\nprint(pd.DataFrame(src_vocab.most_common(20), columns=[\"token\", \"count\"]))\nprint(f\"\\nTop-20 Target (English) tokens:\")\nprint(pd.DataFrame(tgt_vocab.most_common(20), columns=[\"token\", \"count\"]))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:43.547455Z","iopub.execute_input":"2026-02-18T22:01:43.547738Z","iopub.status.idle":"2026-02-18T22:01:43.630218Z","shell.execute_reply.started":"2026-02-18T22:01:43.547717Z","shell.execute_reply":"2026-02-18T22:01:43.629615Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(18, 5))\n\nfor ax, vocab, title, color in [\n    (axes[0], src_vocab, \"Top-30 Akkadian (Source) Tokens\", \"#5a8ecc\"),\n    (axes[1], tgt_vocab, \"Top-30 English (Target) Tokens\",  \"#e07b5a\")\n]:\n    top = pd.DataFrame(vocab.most_common(30), columns=[\"token\", \"count\"])\n    ax.barh(top[\"token\"][::-1], top[\"count\"][::-1], color=color, edgecolor=\"white\")\n    ax.set_title(title, fontsize=12)\n    ax.set_xlabel(\"Frequency\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:46.995626Z","iopub.execute_input":"2026-02-18T22:01:46.996298Z","iopub.status.idle":"2026-02-18T22:01:47.522822Z","shell.execute_reply.started":"2026-02-18T22:01:46.996270Z","shell.execute_reply":"2026-02-18T22:01:47.522202Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Hapax legomena (words appearing only once)\nsrc_hapax = sum(1 for c in src_vocab.values() if c == 1)\ntgt_hapax = sum(1 for c in tgt_vocab.values() if c == 1)\nprint(f\"Source hapax legomena : {src_hapax:,} ({100*src_hapax/len(src_vocab):.1f}% of vocab)\")\nprint(f\"Target hapax legomena : {tgt_hapax:,} ({100*tgt_hapax/len(tgt_vocab):.1f}% of vocab)\")\n\n# Vocabulary coverage curve (Zipf)\nfig, axes = plt.subplots(1, 2, figsize=(14, 4))\nfor ax, vocab, title, color in [\n    (axes[0], src_vocab, \"Source (Akkadian) Zipf Distribution\", \"#5a8ecc\"),\n    (axes[1], tgt_vocab, \"Target (English) Zipf Distribution\",  \"#e07b5a\")\n]:\n    counts = sorted(vocab.values(), reverse=True)\n    ax.loglog(range(1, len(counts)+1), counts, color=color, linewidth=1.5)\n    ax.set_title(title, fontsize=11)\n    ax.set_xlabel(\"Token Rank\")\n    ax.set_ylabel(\"Frequency\")\n    ax.grid(True, which=\"both\", linestyle=\"--\", alpha=0.4)\n\nplt.suptitle(\"Zipf's Law — Vocabulary Distributions\", fontsize=13, fontweight=\"bold\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:50.375002Z","iopub.execute_input":"2026-02-18T22:01:50.375601Z","iopub.status.idle":"2026-02-18T22:01:51.417796Z","shell.execute_reply.started":"2026-02-18T22:01:50.375574Z","shell.execute_reply":"2026-02-18T22:01:51.417083Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ☁️ Word Clouds","metadata":{}},{"cell_type":"code","source":"if HAS_WORDCLOUD:\n    fig, axes = plt.subplots(1, 2, figsize=(18, 7))\n\n    for ax, vocab, title, colormap in [\n        (axes[0], src_vocab, \"Akkadian Source Vocabulary\", \"Blues\"),\n        (axes[1], tgt_vocab, \"English Target Vocabulary\",  \"Oranges\")\n    ]:\n        wc = WordCloud(\n            width=800, height=400, background_color=\"white\",\n            max_words=MAX_WORDCLOUD_WORDS, colormap=colormap,\n            random_state=RANDOM_SEED\n        ).generate_from_frequencies(vocab)\n        ax.imshow(wc, interpolation=\"bilinear\")\n        ax.axis(\"off\")\n        ax.set_title(title, fontsize=14, pad=12)\n\n    plt.suptitle(\"Word Clouds — Training Vocabulary\", fontsize=15, fontweight=\"bold\")\n    plt.tight_layout()\n    plt.show()\nelse:\n    print(\"Skipping word cloud (wordcloud not installed). Run: pip install wordcloud\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:01:54.277332Z","iopub.execute_input":"2026-02-18T22:01:54.277980Z","iopub.status.idle":"2026-02-18T22:01:56.032658Z","shell.execute_reply.started":"2026-02-18T22:01:54.277952Z","shell.execute_reply":"2026-02-18T22:01:56.031581Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔡 Special Characters & Diacritics Analysis","metadata":{}},{"cell_type":"code","source":"# Key Akkadian special characters from competition docs\nSPECIAL_CHARS = {\n    'š': 'shin (š)',\n    'Š': 'Shin (Š)',\n    'ṣ': 'tsade (ṣ)',\n    'ṭ': 'tet (ṭ)',\n    'ḫ': 'het (ḫ)',\n    'Ḫ': 'Het (Ḫ)',\n    'á': 'a2',\n    'à': 'a3',\n    'é': 'e2',\n    'è': 'e3',\n    'í': 'i2',\n    'ì': 'i3',\n    'ú': 'u2',\n    'ù': 'u3',\n}\n\nall_src_text = \" \".join(train[\"transliteration\"].dropna().tolist())\n\nchar_counts = {}\nfor char, label in SPECIAL_CHARS.items():\n    char_counts[f\"{char} ({label})\"] = all_src_text.count(char)\n\nchar_df = pd.DataFrame.from_dict(char_counts, orient=\"index\", columns=[\"count\"]).sort_values(\"count\", ascending=False)\n\nfig, ax = plt.subplots(figsize=(12, 5))\nax.bar(char_df.index, char_df[\"count\"], color=\"#8e5acc\", edgecolor=\"white\")\nax.set_title(\"Special Akkadian Character Frequency in Training Transliterations\", fontsize=13)\nax.set_ylabel(\"Count\")\nplt.xticks(rotation=45, ha=\"right\")\nplt.tight_layout()\nplt.show()\n\nprint(char_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:00.739451Z","iopub.execute_input":"2026-02-18T22:02:00.739738Z","iopub.status.idle":"2026-02-18T22:02:00.960402Z","shell.execute_reply.started":"2026-02-18T22:02:00.739716Z","shell.execute_reply":"2026-02-18T22:02:00.959633Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Scribal notation / bracket analysis\nSCRIBAL_PATTERNS = {\n    \"Curly brackets {}\": r'\\{[^}]*\\}',\n    \"Square brackets []\": r'\\[[^\\]]*\\]',\n    \"Broken text <gap>\": r'<gap>',\n    \"Big gap <big_gap>\": r'<big_gap>',\n    \"Half brackets ˹˺\":  r'[˹˺]',\n    \"Angle brackets <>\": r'<[^>]+>',\n    \"Exclamation marks !\": r'!',\n    \"Question marks ?\": r'\\?',\n}\n\nnotation_results = {}\nfor label, pattern in SCRIBAL_PATTERNS.items():\n    count = train[\"transliteration\"].fillna(\"\").str.count(pattern).sum()\n    notation_results[label] = int(count)\n\nnotation_df = pd.DataFrame.from_dict(notation_results, orient=\"index\", columns=[\"occurrences\"]).sort_values(\"occurrences\", ascending=True)\n\nfig, ax = plt.subplots(figsize=(10, 5))\nax.barh(notation_df.index, notation_df[\"occurrences\"], color=\"#5aac8e\", edgecolor=\"white\")\nax.set_title(\"Scribal Notation Occurrences in Training Set\", fontsize=13)\nax.set_xlabel(\"Total Occurrences\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:04.815297Z","iopub.execute_input":"2026-02-18T22:02:04.815613Z","iopub.status.idle":"2026-02-18T22:02:05.011226Z","shell.execute_reply.started":"2026-02-18T22:02:04.815587Z","shell.execute_reply":"2026-02-18T22:02:05.010646Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🏛️ Determinative Analysis","metadata":{}},{"cell_type":"code","source":"# Determinatives from competition docs\nDETERMINATIVES = [\n    \"{d}\", \"{mul}\", \"{ki}\", \"{lu₂}\", \"{e₂}\", \"{uru}\",\n    \"{kur}\", \"{mi}\", \"{m}\", \"{geš}\", \"{tug₂}\", \"{dub}\",\n    \"{id₂}\", \"{mušen}\", \"{na₄}\", \"{kuš}\", \"{u₂}\"\n]\n\ndet_counts = {}\nfor det in DETERMINATIVES:\n    det_escaped = re.escape(det)\n    count = train[\"transliteration\"].fillna(\"\").str.count(det_escaped).sum()\n    det_counts[det] = int(count)\n\ndet_df = pd.DataFrame.from_dict(det_counts, orient=\"index\", columns=[\"count\"]).sort_values(\"count\", ascending=False)\n\nfig, ax = plt.subplots(figsize=(12, 5))\nbars = ax.bar(det_df.index, det_df[\"count\"], color=\"#e07b5a\", edgecolor=\"white\")\nax.set_title(\"Akkadian Determinative Frequency in Training Transliterations\", fontsize=13)\nax.set_ylabel(\"Count\")\nplt.xticks(rotation=45, ha=\"right\")\n\nfor bar in bars:\n    if bar.get_height() > 0:\n        ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.5,\n                str(int(bar.get_height())), ha='center', va='bottom', fontsize=8)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:08.948949Z","iopub.execute_input":"2026-02-18T22:02:08.949213Z","iopub.status.idle":"2026-02-18T22:02:09.223922Z","shell.execute_reply.started":"2026-02-18T22:02:08.949194Z","shell.execute_reply":"2026-02-18T22:02:09.223235Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📊 Sentence-Level Analysis (Test Set Structure)","metadata":{}},{"cell_type":"code","source":"# Documents in test set\nsentences_per_doc = test.groupby(\"text_id\")[\"id\"].count().reset_index()\nsentences_per_doc.columns = [\"text_id\", \"num_sentences\"]\n\nprint(f\"Unique documents in test : {test['text_id'].nunique()}\")\nprint(f\"Total sentences in test  : {len(test)}\")\nprint(f\"\\nSentences per document stats:\")\nprint(sentences_per_doc[\"num_sentences\"].describe().round(2))\n\nfig, axes = plt.subplots(1, 2, figsize=(14, 4))\n\naxes[0].hist(sentences_per_doc[\"num_sentences\"], bins=20, color=\"#5a8ecc\", edgecolor=\"white\")\naxes[0].set_title(\"Sentences per Document (Test Set)\", fontsize=12)\naxes[0].set_xlabel(\"Number of Sentences\")\naxes[0].set_ylabel(\"Document Count\")\n\naxes[1].hist(test[\"src_char_len\"], bins=40, color=\"#e07b5a\", edgecolor=\"white\")\naxes[1].set_title(\"Source Sentence Length (Test Set)\", fontsize=12)\naxes[1].set_xlabel(\"Character Length\")\naxes[1].set_ylabel(\"Count\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:12.675063Z","iopub.execute_input":"2026-02-18T22:02:12.675346Z","iopub.status.idle":"2026-02-18T22:02:13.029035Z","shell.execute_reply.started":"2026-02-18T22:02:12.675323Z","shell.execute_reply":"2026-02-18T22:02:13.028353Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📚 Lexicon (OA_Lexicon_eBL) Analysis","metadata":{}},{"cell_type":"code","source":"if lexicon is not None:\n    print(\"Lexicon columns:\", lexicon.columns.tolist())\n    print(f\"Lexicon shape  : {lexicon.shape}\")\n    print(lexicon.head(5))\n    \n    # Word type distribution\n    if \"type\" in lexicon.columns:\n        type_counts = lexicon[\"type\"].value_counts()\n        \n        fig, ax = plt.subplots(figsize=(10, 4))\n        type_counts.head(15).plot(kind=\"bar\", ax=ax, color=\"#5aac8e\", edgecolor=\"white\")\n        ax.set_title(\"Word Types in OA Lexicon\", fontsize=13)\n        ax.set_ylabel(\"Count\")\n        plt.xticks(rotation=45, ha=\"right\")\n        plt.tight_layout()\n        plt.show()\nelse:\n    print(\"Lexicon file not available.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:18.734622Z","iopub.execute_input":"2026-02-18T22:02:18.734993Z","iopub.status.idle":"2026-02-18T22:02:18.889548Z","shell.execute_reply.started":"2026-02-18T22:02:18.734964Z","shell.execute_reply":"2026-02-18T22:02:18.888935Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if lexicon is not None and \"form\" in lexicon.columns:\n    # Check what fraction of training source tokens appear in the lexicon\n    lex_forms = set(lexicon[\"form\"].dropna().str.lower().tolist())\n    \n    src_tokens = set(src_vocab.keys())\n    covered = src_tokens & lex_forms\n    not_covered = src_tokens - lex_forms\n    \n    print(f\"Source vocab tokens        : {len(src_tokens):,}\")\n    print(f\"Covered by OA lexicon      : {len(covered):,}  ({100*len(covered)/len(src_tokens):.1f}%)\")\n    print(f\"NOT in OA lexicon (OOL)    : {len(not_covered):,}  ({100*len(not_covered)/len(src_tokens):.1f}%)\")\n    \n    # Coverage pie chart\n    fig, ax = plt.subplots(figsize=(6, 6))\n    ax.pie([len(covered), len(not_covered)],\n           labels=[\"In Lexicon\", \"Out of Lexicon\"],\n           autopct=\"%1.1f%%\", colors=[\"#5aac8e\", \"#e07b5a\"],\n           startangle=90, wedgeprops={\"edgecolor\": \"white\", \"linewidth\": 1.5})\n    ax.set_title(\"Source Token Coverage by OA Lexicon\", fontsize=13)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:22.506588Z","iopub.execute_input":"2026-02-18T22:02:22.507241Z","iopub.status.idle":"2026-02-18T22:02:22.605071Z","shell.execute_reply.started":"2026-02-18T22:02:22.507210Z","shell.execute_reply":"2026-02-18T22:02:22.604362Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📰 Published Texts & Supplemental Data Overview","metadata":{}},{"cell_type":"code","source":"if pub_texts is not None:\n    print(f\"Published texts shape  : {pub_texts.shape}\")\n    print(f\"Columns                : {pub_texts.columns.tolist()[:10]} ...\")\n    print()\n\n    # Genre distribution (if available)\n    if \"genre_label\" in pub_texts.columns:\n        genre_counts = pub_texts[\"genre_label\"].value_counts(dropna=False)\n        fig, ax = plt.subplots(figsize=(10, 4))\n        genre_counts.head(15).plot(kind=\"bar\", ax=ax, color=\"#8e5acc\", edgecolor=\"white\")\n        ax.set_title(\"Genre Distribution in Published Texts\", fontsize=13)\n        ax.set_ylabel(\"Count\")\n        plt.xticks(rotation=45, ha=\"right\")\n        plt.tight_layout()\n        plt.show()\n\n    # Transliteration lengths\n    if \"transliteration\" in pub_texts.columns:\n        pub_texts[\"translit_char_len\"] = pub_texts[\"transliteration\"].fillna(\"\").str.len()\n        fig, ax = plt.subplots(figsize=(10, 4))\n        ax.hist(pub_texts[\"translit_char_len\"].clip(0, 2000), bins=60, color=\"#5a8ecc\", edgecolor=\"white\")\n        ax.set_title(\"Transliteration Length Distribution (Published Texts)\", fontsize=12)\n        ax.set_xlabel(\"Character Length (clipped at 2000)\")\n        ax.set_ylabel(\"Count\")\n        plt.tight_layout()\n        plt.show()\nelse:\n    print(\"Published texts file not loaded.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:25.209935Z","iopub.execute_input":"2026-02-18T22:02:25.210636Z","iopub.status.idle":"2026-02-18T22:02:25.657364Z","shell.execute_reply.started":"2026-02-18T22:02:25.210605Z","shell.execute_reply":"2026-02-18T22:02:25.656637Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📅 Bibliography Temporal Analysis","metadata":{}},{"cell_type":"code","source":"if bibliography is not None:\n    print(f\"Bibliography shape    : {bibliography.shape}\")\n    print(f\"Columns               : {bibliography.columns.tolist()}\")\n    print(bibliography.head(3))\n    \n    if \"year\" in bibliography.columns:\n        bib_years = pd.to_numeric(bibliography[\"year\"], errors=\"coerce\").dropna()\n        fig, ax = plt.subplots(figsize=(12, 4))\n        ax.hist(bib_years, bins=40, color=\"#cc8e5a\", edgecolor=\"white\")\n        ax.set_title(\"Publication Year Distribution in Bibliography\", fontsize=13)\n        ax.set_xlabel(\"Year\")\n        ax.set_ylabel(\"Number of Publications\")\n        plt.tight_layout()\n        plt.show()\n\n    if \"journal\" in bibliography.columns:\n        top_journals = bibliography[\"journal\"].value_counts().head(15)\n        fig, ax = plt.subplots(figsize=(12, 4))\n        top_journals[::-1].plot(kind=\"barh\", ax=ax, color=\"#5aac8e\", edgecolor=\"white\")\n        ax.set_title(\"Top-15 Journals in Bibliography\", fontsize=13)\n        ax.set_xlabel(\"Publications\")\n        plt.tight_layout()\n        plt.show()\nelse:\n    print(\"Bibliography file not loaded.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:30.186823Z","iopub.execute_input":"2026-02-18T22:02:30.187140Z","iopub.status.idle":"2026-02-18T22:02:30.632346Z","shell.execute_reply.started":"2026-02-18T22:02:30.187116Z","shell.execute_reply":"2026-02-18T22:02:30.631720Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧩 Correlation Heatmap & Summary Stats","metadata":{}},{"cell_type":"code","source":"numeric_cols = [\"src_char_len\", \"tgt_char_len\", \"src_word_len\", \"tgt_word_len\",\n                \"src_tok_len\", \"tgt_tok_len\", \"len_ratio\"]\nnumeric_df = train[numeric_cols].dropna()\n\nfig, ax = plt.subplots(figsize=(9, 7))\ncorr_matrix = numeric_df.corr()\nmask = np.triu(np.ones_like(corr_matrix, dtype=bool), k=1)\nsns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap=\"coolwarm\", center=0,\n            linewidths=0.5, ax=ax, mask=~mask,\n            cbar_kws={\"shrink\": 0.8})\n# Show full diagonal too\nsns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap=\"coolwarm\", center=0,\n            linewidths=0.5, ax=ax, mask=mask, cbar=False)\nax.set_title(\"Correlation Heatmap — Text Length Features\", fontsize=13)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:35.194228Z","iopub.execute_input":"2026-02-18T22:02:35.194559Z","iopub.status.idle":"2026-02-18T22:02:35.553800Z","shell.execute_reply.started":"2026-02-18T22:02:35.194531Z","shell.execute_reply":"2026-02-18T22:02:35.553187Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Consolidated summary table\nsummary = pd.DataFrame({\n    \"Metric\": [\n        \"Train samples\",\n        \"Test samples (dummy)\",\n        \"Full test set (per docs)\",\n        \"Source vocab size\",\n        \"Target vocab size\",\n        \"Avg source word count\",\n        \"Avg target word count\",\n        \"Avg length ratio (tgt/src)\",\n        \"Max source word count\",\n        \"Max target word count\",\n        \"Source hapax ratio (%)\",\n        \"Target hapax ratio (%)\",\n    ],\n    \"Value\": [\n        len(train),\n        len(test),\n        \"~4000 sentences, ~400 docs\",\n        f\"{len(src_vocab):,}\",\n        f\"{len(tgt_vocab):,}\",\n        f\"{train['src_word_len'].mean():.1f}\",\n        f\"{train['tgt_word_len'].mean():.1f}\",\n        f\"{train['len_ratio'].mean():.2f}\",\n        train['src_word_len'].max(),\n        train['tgt_word_len'].max(),\n        f\"{100*src_hapax/len(src_vocab):.1f}%\",\n        f\"{100*tgt_hapax/len(tgt_vocab):.1f}%\",\n    ]\n})\n\nsummary.style.set_properties(**{\"text-align\": \"left\"}).hide(axis=\"index\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:38.866890Z","iopub.execute_input":"2026-02-18T22:02:38.867626Z","iopub.status.idle":"2026-02-18T22:02:38.962784Z","shell.execute_reply.started":"2026-02-18T22:02:38.867596Z","shell.execute_reply":"2026-02-18T22:02:38.962204Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔍 Sample Inspection","metadata":{}},{"cell_type":"code","source":"# Short, medium, long examples\ntrain_sorted = train.sort_values(\"src_word_len\")\n\nfor label, idx in [(\"SHORTEST\", 0), (\"MEDIAN\", len(train)//2), (\"LONGEST\", -1)]:\n    row = train_sorted.iloc[idx]\n    print(f\"{'='*70}\")\n    print(f\"[{label}] — Source word count: {row['src_word_len']}\")\n    print(f\"  Source : {row['transliteration'][:300]}\")\n    print(f\"  Target : {str(row['translation'])[:300]}\")\nprint(\"=\"*70)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T22:02:42.634075Z","iopub.execute_input":"2026-02-18T22:02:42.634530Z","iopub.status.idle":"2026-02-18T22:02:42.642752Z","shell.execute_reply.started":"2026-02-18T22:02:42.634503Z","shell.execute_reply":"2026-02-18T22:02:42.642040Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🚦 Key Observations & Modelling Implications\n\nBased on the EDA above, here are the most important findings:\n\n### 📌 Data Overview\n- ~1,500 aligned train pairs; ~4,000 sentence-level test samples from ~400 documents.\n- **Severely low-resource** — this is one of the key challenges.\n\n### 📌 Text Characteristics\n- Source (Akkadian) texts are highly morphologically complex — single tokens encode multiple semantic roles.\n- High hapax ratio in both source (~60–80%) and target vocabularies indicates extreme sparsity.\n- The target (English) tends to be longer than the source on average (expansion rate > 1×), driven by morphological unpacking.\n- Strong Pearson correlation between source and target length confirms length-driven alignment is viable.\n\n### 📌 Special Formatting Challenges\n- Curly-bracket determinatives (`{ki}`, `{d}` etc.) are frequent and semantically meaningful — must be handled, not stripped.\n- Scribal notations (`[...]`, `˹˺`, `!`, `?`) add noise; competition suggests normalising gaps to `<gap>` / `<big_gap>`.\n- Subscript / superscript numerals (Unicode) require explicit normalisation.\n- `Ḫ / ḫ` vs `H / h` mismatch between train and test — needs substitution.\n\n### 📌 Modelling Strategy Hints\n1. **Pre-trained multilingual model** (mBART-50, NLLB, Helsinki-NLP Opus-MT) + fine-tune.\n2. **BPE / SentencePiece** tokenisation better handles unseen subwords vs. word-level.\n3. **Data augmentation** from `publications.csv` — extract and align ~880 scholarly texts.\n4. **Lexicon injection** — incorporate `OA_Lexicon_eBL.csv` as a constrained decoding aid.\n5. Evaluate with `sacrebleu` using both `BLEU` and `chrF++` (geometric mean is the metric).","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}