{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":121150,"databundleVersionId":14976537,"sourceType":"competition"}],"dockerImageVersionId":31234,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐪 Deep Past Initiative: Exploratory Data Analysis\n\nThis notebook explores the **Old Assyrian → English** machine translation dataset for the *Deep Past Initiative* Kaggle challenge.  \nThe goal here is to **understand every CSV**, inspect text formats, and prepare insights for later modeling.\n\nWe will:\n- Inspect all provided files and their schema  \n- Look closely at transliterations, translations, and lexicon info  \n- Visualize key statistics to guide model design  \n\n> This notebook focuses on **clean, visual EDA** before touching any modeling.","metadata":{}},{"cell_type":"code","source":"import os\nimport textwrap\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nsns.set(style=\"whitegrid\", context=\"notebook\")\nplt.rcParams[\"figure.figsize\"] = (10, 5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:00:22.442440Z","iopub.execute_input":"2025-12-17T11:00:22.442781Z","iopub.status.idle":"2025-12-17T11:00:25.894011Z","shell.execute_reply.started":"2025-12-17T11:00:22.442753Z","shell.execute_reply":"2025-12-17T11:00:25.892977Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"OA_Lexicon = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/OA_Lexicon_eBL.csv\")\nbibliography = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/bibliography.csv\")\npublications = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/publications.csv\")\npub_text = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/published_texts.csv\")\nsample_sub = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/sample_submission.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/test.csv\")\ntrain_df = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/train.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:02:42.498185Z","iopub.execute_input":"2025-12-17T11:02:42.498723Z","iopub.status.idle":"2025-12-17T11:02:50.874650Z","shell.execute_reply.started":"2025-12-17T11:02:42.498692Z","shell.execute_reply":"2025-12-17T11:02:50.873499Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. Dataset Overview\n\nWe start by inspecting the **shape and columns** of each main CSV to understand what information is available and how tables relate to each other.\n\nThis section answers:\n- How many rows does each file contain?\n- What are the key columns and dtypes?\n- Are there obvious missing values?","metadata":{}},{"cell_type":"code","source":"datasets = {\n    \"train_df\": train_df,\n    \"test_df\": test_df,\n    \"sample_sub\": sample_sub,\n    \"pub_text\": pub_text,\n    \"publications\": publications,\n    \"bibliography\": bibliography,\n    \"OA_Lexicon\": OA_Lexicon,\n}\n\nsummary_rows = []\nfor name, df in datasets.items():\n    summary_rows.append({\n        \"name\": name,\n        \"n_rows\": len(df),\n        \"n_cols\": df.shape[1],\n        \"columns\": \", \".join(df.columns[:8]) + (\"...\" if df.shape[1] > 8 else \"\"),\n    })\n\noverview_df = pd.DataFrame(summary_rows).sort_values(\"name\")\noverview_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:03:31.831462Z","iopub.execute_input":"2025-12-17T11:03:31.831971Z","iopub.status.idle":"2025-12-17T11:03:31.871777Z","shell.execute_reply.started":"2025-12-17T11:03:31.831940Z","shell.execute_reply":"2025-12-17T11:03:31.870597Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"=== train_df.info() ===\")\ntrain_df.info()\nprint(\"\\n=== test_df.info() ===\")\ntest_df.info()\nprint(\"\\n=== pub_text.info() ===\")\npub_text.info()\nprint(\"\\n=== OA_Lexicon.info() ===\")\nOA_Lexicon.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:03:46.309087Z","iopub.execute_input":"2025-12-17T11:03:46.309397Z","iopub.status.idle":"2025-12-17T11:03:46.369405Z","shell.execute_reply.started":"2025-12-17T11:03:46.309372Z","shell.execute_reply":"2025-12-17T11:03:46.368410Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Exploring train and test texts\n\nHere we inspect **lengths and sample rows** from `train_df` and `test_df` to understand:\n- How long the transliterations and translations are\n- How document-level `train_df` differs from sentence-level `test_df`","metadata":{}},{"cell_type":"code","source":"# Add basic length features\ntrain_df[\"src_len_char\"] = train_df[\"transliteration\"].str.len()\ntrain_df[\"tgt_len_char\"] = train_df[\"translation\"].str.len()\ntrain_df[\"src_len_tok\"] = train_df[\"transliteration\"].str.split().str.len()\ntrain_df[\"tgt_len_tok\"] = train_df[\"translation\"].str.split().str.len()\n\nlength_summary = train_df[[\"src_len_char\", \"tgt_len_char\", \"src_len_tok\", \"tgt_len_tok\"]].describe().T\nlength_summary","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:05:36.817704Z","iopub.execute_input":"2025-12-17T11:05:36.818077Z","iopub.status.idle":"2025-12-17T11:05:36.899375Z","shell.execute_reply.started":"2025-12-17T11:05:36.818050Z","shell.execute_reply":"2025-12-17T11:05:36.898565Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# few rows to see raw formatting\npd.set_option(\"display.max_colwidth\", 200)\n\nprint(\"=== Sample train_df rows ===\")\ndisplay(train_df.sample(3, random_state=42))\n\nprint(\"\\n=== test_df ===\")\ndisplay(test_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:05:55.793598Z","iopub.execute_input":"2025-12-17T11:05:55.794873Z","iopub.status.idle":"2025-12-17T11:05:55.814312Z","shell.execute_reply.started":"2025-12-17T11:05:55.794798Z","shell.execute_reply":"2025-12-17T11:05:55.813549Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n\nsns.histplot(train_df[\"src_len_tok\"], bins=40, ax=axes[0], color=\"tab:blue\")\naxes[0].set_title(\"Transliteration token length (train)\")\naxes[0].set_xlabel(\"Tokens\")\n\nsns.histplot(train_df[\"tgt_len_tok\"], bins=40, ax=axes[1], color=\"tab:green\")\naxes[1].set_title(\"Translation token length (train)\")\naxes[1].set_xlabel(\"Tokens\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:06:40.802244Z","iopub.execute_input":"2025-12-17T11:06:40.802611Z","iopub.status.idle":"2025-12-17T11:06:41.335209Z","shell.execute_reply.started":"2025-12-17T11:06:40.802585Z","shell.execute_reply":"2025-12-17T11:06:41.334175Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Anatomy of the transliterations\n\nOld Assyrian transliterations contain:\n- Hyphenated syllables, determinatives in `{ }`, and line/gap markers  \n- Diacritics and special letters (š, Ṣ, Ṭ, ḫ, etc.)\n\nHere we look at:\n- Most frequent tokens\n- How often special characters and brackets appear","metadata":{}},{"cell_type":"code","source":"import re\nfrom collections import Counter\n\ndef get_token_counts(series, top_n=30):\n    tokens = []\n    for text in series.dropna():\n        tokens.extend(text.split())\n    counter = Counter(tokens)\n    return pd.DataFrame(counter.most_common(top_n), columns=[\"token\", \"freq\"])\n\ntop_tokens_train = get_token_counts(train_df[\"transliteration\"], top_n=40)\ntop_tokens_train.head(20)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:07:59.207784Z","iopub.execute_input":"2025-12-17T11:07:59.208332Z","iopub.status.idle":"2025-12-17T11:07:59.251862Z","shell.execute_reply.started":"2025-12-17T11:07:59.208299Z","shell.execute_reply":"2025-12-17T11:07:59.250705Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Special-character and bracket statistics in transliteration\nspecial_chars = [\"š\", \"Š\", \"ṣ\", \"Ṣ\", \"ṭ\", \"Ṭ\", \"ḫ\", \"Ḫ\", \"{\", \"}\", \"<\", \">\", \"[\", \"]\", \":\", \"!\", \"?\", \"/\"]\n\nstats = []\nfor ch in special_chars:\n    count = train_df[\"transliteration\"].str.count(re.escape(ch)).sum()\n    docs = (train_df[\"transliteration\"].str.contains(re.escape(ch))).sum()\n    stats.append({\"char\": ch, \"total_occurrences\": int(count), \"docs_with_char\": int(docs)})\n\nspecial_df = pd.DataFrame(stats).sort_values(\"docs_with_char\", ascending=False)\nspecial_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:08:24.253705Z","iopub.execute_input":"2025-12-17T11:08:24.254368Z","iopub.status.idle":"2025-12-17T11:08:24.328942Z","shell.execute_reply.started":"2025-12-17T11:08:24.254331Z","shell.execute_reply":"2025-12-17T11:08:24.327964Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# top 25 tokens in transliteration\nplt.figure(figsize=(10, 6))\nsns.barplot(\n    data=top_tokens_train.head(25),\n    x=\"freq\",\n    y=\"token\",\n    color=\"tab:blue\"\n)\nplt.title(\"Most frequent transliteration tokens (train)\")\nplt.xlabel(\"Frequency\")\nplt.ylabel(\"Token\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:08:59.806181Z","iopub.execute_input":"2025-12-17T11:08:59.807220Z","iopub.status.idle":"2025-12-17T11:09:00.245183Z","shell.execute_reply.started":"2025-12-17T11:08:59.807186Z","shell.execute_reply":"2025-12-17T11:09:00.244063Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 3.1 Observations on transliteration tokens\n\n- The most frequent tokens include function words (e.g. `a-na`, `ša`, `i-na`) and numeric / measure tokens such as `KÙ.BABBAR`, `ma-na`, `GÍN`, and digits.  \n- Diacritic characters like `š`, `ḫ`, and `ṣ` occur in the vast majority of documents, confirming that **Unicode-aware tokenization** is necessary.  \n- Modern scribal symbols like `{}`, `<>`, `!`, `?`, `/`, and `:` barely appear in `train_df`, suggesting that many formatting suggestions are already applied in this cleaned split.","metadata":{}},{"cell_type":"markdown","source":"## 4. Linking training texts to published metadata\n\nHere we ask:\n- How many training documents have entries in `published_texts.csv`?\n- What extra fields (genre, description, AICC translation) can be leveraged later?","metadata":{}},{"cell_type":"code","source":"train_with_meta = train_df.merge(pub_text, on=\"oare_id\", how=\"left\", suffixes=(\"\", \"_pub\"))\n\nn_with_meta = train_with_meta[\"online transcript\"].notna().sum()\nn_total = len(train_df)\n\nprint(f\"Training docs with metadata in pub_text: {n_with_meta} / {n_total} \"\n      f\"({n_with_meta / n_total:.1%})\")\n\n# few examples with metadata present\ncols_to_show = [\n    \"oare_id\", \"transliteration\", \"translation\",\n    \"label\", \"genre_label\", \"description\", \"AICC_translation\", \"online transcript\"\n]\ndisplay(train_with_meta[train_with_meta[\"online transcript\"].notna()][cols_to_show].head(3))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:11:14.417347Z","iopub.execute_input":"2025-12-17T11:11:14.418370Z","iopub.status.idle":"2025-12-17T11:11:14.453725Z","shell.execute_reply.started":"2025-12-17T11:11:14.418334Z","shell.execute_reply":"2025-12-17T11:11:14.452919Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.1 Metadata coverage insights\n\n- Every training document links to a `published_texts` record, so **oare_id is a perfect bridge** between aligned translations and richer metadata.  \n- Fields like `genre_label` (e.g. *debt note, note, unknown*) and detailed `description` can support **domain-aware analysis** or later conditioning in the model.  \n- The `AICC_translation` URLs and `online transcript` links provide external, noisy machine translations and tablet images that could be mined or used for qualitative inspection.","metadata":{}},{"cell_type":"markdown","source":"## 5. Lexicon coverage of training vocabulary\n\nThe `OA_Lexicon_eBL.csv` file lists Old Assyrian word forms, their normalized shapes, and lexemes, plus type labels (e.g. `word`, `PN`, `GN`).  \nHere we measure how much of the **training vocabulary** is explained by the lexicon.","metadata":{}},{"cell_type":"code","source":"# Build training vocabulary (case-sensitive tokens)\ntrain_tokens = []\nfor text in train_df[\"transliteration\"]:\n    train_tokens.extend(text.split())\n\ntrain_vocab = pd.Series(train_tokens).value_counts()\nprint(f\"Unique tokens in train transliteration: {len(train_vocab)}\")\n\n# lexicon lookup on form\nlex_forms = set(OA_Lexicon[\"form\"].dropna().astype(str))\n\n# coverage by exact form match\nin_lex = train_vocab.index.isin(lex_forms)\ncoverage_form = in_lex.mean()\nprint(f\"Fraction of train vocab found in lexicon by exact `form`: {coverage_form:.2%}\")\n\n# token frequency coverage\nfreq_coverage_form = train_vocab[in_lex].sum() / train_vocab.sum()\nprint(f\"Fraction of total token occurrences covered by `form`: {freq_coverage_form:.2%}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:13:11.121020Z","iopub.execute_input":"2025-12-17T11:13:11.121587Z","iopub.status.idle":"2025-12-17T11:13:11.192989Z","shell.execute_reply.started":"2025-12-17T11:13:11.121555Z","shell.execute_reply":"2025-12-17T11:13:11.191558Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Joining vocab with lexicon to inspect types of covered tokens\nvocab_df = train_vocab.rename(\"freq\").reset_index().rename(columns={\"index\": \"token\"})\n\nlex_small = OA_Lexicon[[\"form\", \"type\", \"norm\", \"lexeme\"]].dropna(subset=[\"form\"])\nvocab_lex = vocab_df.merge(lex_small, left_on=\"token\", right_on=\"form\", how=\"left\")\n\n# How many tokens get at least one lexicon entry?\ncovered = vocab_lex[\"type\"].notna().mean()\nprint(f\"Vocab tokens with at least one lexicon row: {covered:.2%}\")\n\ntype_stats = (\n    vocab_lex[vocab_lex[\"type\"].notna()]\n    .groupby(\"type\")[\"freq\"]\n    .sum()\n    .sort_values(ascending=False)\n    .reset_index()\n)\ntype_stats.head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:13:41.241138Z","iopub.execute_input":"2025-12-17T11:13:41.241590Z","iopub.status.idle":"2025-12-17T11:13:41.293558Z","shell.execute_reply.started":"2025-12-17T11:13:41.241543Z","shell.execute_reply":"2025-12-17T11:13:41.292553Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Examples of unmapped but frequent tokens\nunmapped = vocab_lex[vocab_lex[\"type\"].isna()].head(20)\nunmapped","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:14:08.793985Z","iopub.execute_input":"2025-12-17T11:14:08.794790Z","iopub.status.idle":"2025-12-17T11:14:08.809094Z","shell.execute_reply.started":"2025-12-17T11:14:08.794754Z","shell.execute_reply":"2025-12-17T11:14:08.808195Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Lexicon coverage insights\n\nAbout **70% of unique train tokens** and over **80% of all token occurrences** have a direct `form` match in the lexicon, and most of that coverage is regular words plus a substantial block of personal names, while unmapped tokens are mainly numerals and gap markers that the lexicon does not attempt to encode.\n\n- Out of **11,761** unique transliteration tokens in `train_df`, **69.76%** appear in `OA_Lexicon.form`, covering **82.37%** of all token occurrences, so the lexicon explains the majority of the running text.  \n- When joining vocabulary to `OA_Lexicon`, **73.02%** of token types get at least one lexical entry, with frequencies dominated by `word` (**123,268** occurrences), followed by `PN` (personal names, **22,116**) and `GN` (geographic names, **842**), confirming that names are a large, distinct slice of the corpus.  \n- The most frequent **unmapped tokens** include `x`, `…`, numeric tokens like `1, 2, 5, 10, 0.5, 0.33333`, and markers such as `[...]` or `xx`, which correspond to gaps, uncertainties, and numerical quantities that are intentionally outside the lexicon’s scope.","metadata":{}},{"cell_type":"markdown","source":"## 6. Publications and bibliography overview\n\nThe `publications.csv` file contains OCR text from ~900 PDFs, and `bibliography.csv` adds metadata (title, author, year, journal).  \nHere we measure how many pages and PDFs contain Akkadian transliterations and how these sources are distributed over time and venues.","metadata":{}},{"cell_type":"code","source":"# Basic counts\nn_pages = len(publications)\nn_pages_akk = publications[\"has_akkadian\"].sum()\nn_pdfs = publications[\"pdf_name\"].nunique()\nn_pdfs_akk = publications.loc[publications[\"has_akkadian\"], \"pdf_name\"].nunique()\n\nprint(f\"Total pages in publications.csv: {n_pages}\")\nprint(f\"Pages with has_akkadian=True: {n_pages_akk} \"\n      f\"({n_pages_akk / n_pages:.1%})\")\nprint(f\"Unique PDFs: {n_pdfs}\")\nprint(f\"PDFs with any Akkadian pages: {n_pdfs_akk} \"\n      f\"({n_pdfs_akk / n_pdfs:.1%})\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:16:09.743265Z","iopub.execute_input":"2025-12-17T11:16:09.744212Z","iopub.status.idle":"2025-12-17T11:16:09.779436Z","shell.execute_reply.started":"2025-12-17T11:16:09.744180Z","shell.execute_reply":"2025-12-17T11:16:09.778245Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Merge with bibliography to get year and journal\npub_pages = publications.merge(bibliography, on=\"pdf_name\", how=\"left\")\n\nyear_counts = (\n    pub_pages.dropna(subset=[\"year\"])\n    .groupby(\"year\")[\"pdf_name\"]\n    .nunique()\n    .sort_index()\n)\n\njournal_counts = (\n    pub_pages.dropna(subset=[\"journal\"])\n    .groupby(\"journal\")[\"pdf_name\"]\n    .nunique()\n    .sort_values(ascending=False)\n    .head(15)\n)\n\nyear_counts.head(), journal_counts.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:16:26.468087Z","iopub.execute_input":"2025-12-17T11:16:26.468778Z","iopub.status.idle":"2025-12-17T11:16:26.731657Z","shell.execute_reply.started":"2025-12-17T11:16:26.468746Z","shell.execute_reply":"2025-12-17T11:16:26.730398Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#PDFs per year (Akkadian only vs all)\n\npdf_year_all = (\n    pub_pages.dropna(subset=[\"year\"])\n    .groupby(\"year\")[\"pdf_name\"]\n    .nunique()\n    .rename(\"all_pdfs\")\n)\n\npdf_year_akk = (\n    pub_pages[pub_pages[\"has_akkadian\"]]\n    .dropna(subset=[\"year\"])\n    .groupby(\"year\")[\"pdf_name\"]\n    .nunique()\n    .rename(\"akk_pdfs\")\n)\n\npdf_year = pd.concat([pdf_year_all, pdf_year_akk], axis=1).fillna(0)\n\nplt.figure(figsize=(12, 5))\npdf_year.plot(kind=\"bar\", ax=plt.gca())\nplt.title(\"Number of PDFs per year (all vs with Akkadian pages)\")\nplt.xlabel(\"Year\")\nplt.ylabel(\"Number of PDFs\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:16:57.011765Z","iopub.execute_input":"2025-12-17T11:16:57.012732Z","iopub.status.idle":"2025-12-17T11:16:58.329982Z","shell.execute_reply.started":"2025-12-17T11:16:57.012701Z","shell.execute_reply":"2025-12-17T11:16:58.328799Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Publications corpus insights\n\nThe OCR corpus in `publications.csv` is **huge**: 216,602 pages across 952 PDFs, but only **31,286 pages (14.4%)** are flagged with `has_akkadian=True`, so Akkadian content is relatively sparse in each volume.  \nStill, almost **half of all PDFs (435 / 952, 45.7%)** contain at least one Akkadian page, meaning a wide spread of potentially useful sources for mining additional parallel data.\n\nThe bibliography spans more than a century: early entries appear already in the **1880s–1900s**, and coverage grows strongly after mid‑20th century, with a visible peak in late‑20th / early‑21st‑century years in the bar plot.  \nA small set of journals contributes a large share of PDFs, led by *Journal of the British Institute of Archaeology at Ankara* (44 PDFs), *Iraq* (32), *Journal of Cuneiform Studies* (16), *Zeitschrift für Assyriologie* (15), and *JESHO* (15), which are natural starting points if manually inspecting OCR quality and alignment.","metadata":{}},{"cell_type":"markdown","source":"## 7. What do OCR pages look like?\n\nTo understand how hard it will be to mine extra training data, we inspect a few pages where `has_akkadian=True`.  \nThis reveals how Akkadian transliterations and their translations are mixed with running prose, headings, and references.","metadata":{}},{"cell_type":"code","source":"# Random sample of Akkadian-containing pages\nakk_pages = publications[publications[\"has_akkadian\"]].sample(5, random_state=42)\nakk_pages[[\"pdf_name\", \"page\"]]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:18:14.822051Z","iopub.execute_input":"2025-12-17T11:18:14.822448Z","iopub.status.idle":"2025-12-17T11:18:14.840569Z","shell.execute_reply.started":"2025-12-17T11:18:14.822419Z","shell.execute_reply":"2025-12-17T11:18:14.839723Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# snippets of the OCR text\npd.set_option(\"display.max_colwidth\", 400)\n\nfor i, row in akk_pages.iterrows():\n    print(\"=\" * 80)\n    print(f\"PDF: {row['pdf_name']} | page: {row['page']}\")\n    print(\"-\" * 80)\n    snippet = row[\"page_text\"]\n    # Take a manageable slice\n    print(snippet[:2000])\n    print(\"\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T11:18:37.654598Z","iopub.execute_input":"2025-12-17T11:18:37.655687Z","iopub.status.idle":"2025-12-17T11:18:37.662641Z","shell.execute_reply.started":"2025-12-17T11:18:37.655652Z","shell.execute_reply":"2025-12-17T11:18:37.661796Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 8. Design notes for mining extra training data\n\nThe **publications corpus is very heterogeneous**, so careful filtering is needed before using it as additional training data.\n\n- Many pages interleave **Akkadian (or related) transliterations** with immediate translations and long stretches of modern-language commentary.  \n- Some pages are almost entirely **German/French/English prose** about history or law with only a few cited lines, while others are **pure bibliography/reference lists** without usable transliteration–translation pairs.\n\nA practical extraction strategy will likely need:\n\n- **Akkadian line detection**: identify candidate transliteration lines by features like heavy use of diacritics (`š, ḫ, ṣ`), hyphenated syllables, and limited lowercase prose.  \n- **Neighbouring translation pairing**: for each detected transliteration block, search in the nearby lines/paragraphs for sentences in a modern language (initially English, maybe later via MT from French/German) to form noisy parallel pairs.  \n- **Page-level filtering**: skip pages that look like bibliographies (many author-year patterns, very short lines, no diacritics or hyphenated tokens) or general narrative text with almost no Akkadian markers.  \n\nThese heuristics can guide a later **semi-automatic mining pipeline** that augments the small clean `train_df` with additional, but noisier, sentence-level examples.","metadata":{}},{"cell_type":"markdown","source":"## 9. Summary & key takeaways to start building your model\n\n<div style=\"background:#0f172a; color:#e5e7eb; padding:18px 20px; border-radius:10px; border:1px solid #1f2937; font-size:14px;\">\n\n<table style=\"width:100%; border-collapse:collapse; table-layout:fixed;\">\n  <thead>\n    <tr>\n      <th style=\"border-bottom:1px solid #4b5563; padding:8px; text-align:left; width:22%;\">Aspect</th>\n      <th style=\"border-bottom:1px solid #4b5563; padding:8px; text-align:left;\">Findings</th>\n      <th style=\"border-bottom:1px solid #4b5563; padding:8px; text-align:left; width:26%;\">Implications</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">Train vs. test structure</td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Train has 1,561 <strong>document-level</strong> pairs with median ≈49 source vs. 68 target tokens and long tails up to 700+ target tokens.  \n        Test is <strong>sentence-level</strong> with line ranges per tablet.\n      </td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Need strategies to <strong>segment documents into sentences</strong> for training and to handle long sequences (chunking or long-context models).\n      </td>\n    </tr>\n    <tr>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">Transliteration anatomy</td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Frequent tokens mix function words (<code>a-na</code>, <code>ša</code>, <code>i-na</code>), logograms (<code>KÙ.BABBAR</code>, <code>DUMU</code>, <code>GÍN</code>), and numeric markers.  \n        Diacritics like <code>š</code>, <code>ḫ</code>, <code>ṣ</code> appear in most documents; modern scribal punctuation is already rare.\n      </td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Requires <strong>Unicode-aware tokenization</strong> and probably custom normalization rules; standard Latin-only tokenizers will underperform.\n      </td>\n    </tr>\n    <tr>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">Metadata linkage</td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Every train row has a matching entry in <code>published_texts.csv</code> via <code>oare_id</code>, with fields like <code>genre_label</code>, descriptions, and external links.\n      </td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Enables <strong>genre-aware analysis</strong>, curriculum ideas, and easy sampling of specific document types (e.g. debt notes vs. letters).\n      </td>\n    </tr>\n    <tr>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">Lexicon coverage</td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Lexicon covers ≈<strong>70% of unique tokens</strong> and ≈<strong>82% of token occurrences</strong>; mapped tokens are dominated by <code>word</code>, with substantial <code>PN</code> and <code>GN</code>.  \n        Unmapped tokens are mainly numerals, gaps (<code>x</code>, <code>…</code>, <code>[...]</code>), and placeholders.\n      </td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Lexicon is a strong prior for <strong>morphology, lemmatization, and proper-name handling</strong>; unmapped tokens can be normalized or treated as numeric / gap classes.\n      </td>\n    </tr>\n    <tr>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">Publications corpus scale</td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        216k pages across 952 PDFs; only 14.4% of pages and ~46% of PDFs contain any Akkadian.  \n        A handful of journals contribute a large fraction of relevant material.\n      </td>\n      <td style=\"border-bottom:1px solid #374151; padding:8px;\">\n        Mining extra data is feasible but requires <strong>aggressive filtering</strong> to avoid drowning in non-parallel text.\n      </td>\n    </tr>\n    <tr>\n      <td style=\"padding:8px;\">OCR page structure</td>\n      <td style=\"padding:8px;\">\n        Pages mix transliteration lines, inline translations, multi-language commentary, and pure bibliography.  \n        Layout conventions differ widely between works and decades.\n      </td>\n      <td style=\"padding:8px;\">\n        Any automatic mining pipeline should combine <strong>Akkadian-line detection</strong>, nearby-sentence pairing, and bibliographic-page filtering before using mined pairs for training.\n      </td>\n    </tr>\n  </tbody>\n</table>\n\n</div>\n","metadata":{}},{"cell_type":"markdown","source":"## 🙏 Thanks for checking this notebook!\n\n<div style=\"margin-top:10px; padding:18px 20px; border-radius:10px; border:1px solid #e5e7eb; background:linear-gradient(135deg,#0f172a,#111827); color:#f9fafb; text-align:center; font-size:15px;\">\n\n  <div style=\"font-size:20px; font-weight:600; margin-bottom:8px;\">\n    Thanks for exploring this EDA!\n  </div>\n\n  <div style=\"max-width:700px; margin:0 auto 10px auto; line-height:1.6;\">\n    If this notebook helped you understand the Deep Past dataset or prepare your\n    machine translation model, consider giving it an <strong>upvote</strong> on Kaggle.\n    Your support helps others discover it and motivates further updates 🚀\n  </div>\n\n  <div style=\"font-size:13px; color:#9ca3af;\">\n    Feel free to fork, extend the analyses, or adapt the code for your own experiments.\n  </div>\n\n</div>","metadata":{}}]}