{"metadata":{"kernelspec":{"display_name":".venv","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.12.5"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":121150,"databundleVersionId":15061024,"sourceType":"competition"}],"dockerImageVersionId":31260,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What Are the Old Assyrian Texts Actually About? — BERTopic Analysis\n\nThe competition provides training data with ~1,561 document-level English translations.\nUsing **BERTopic** (BERT-based topic modeling), this notebook maps the **semantic landscape**\nof the training data at two granularity levels:\n\n1. **Document-level** — What topics do the ~1,500 training documents cover?\n2. **Sentence-level** — Splitting train.csv translations into individual sentences, what finer-grained topics emerge?\n\n### Goals\n\n1. **Discover the topic structure** of Old Assyrian texts —\n   trade, legal, correspondence, religious, etc.\n2. **Compare document vs sentence granularity** — Are there topics only visible at sentence level?\n3. **Identify under-represented areas** — Topic gaps that may affect generalization.","metadata":{}},{"cell_type":"code","source":"!pip install -q bertopic sentence-transformers umap-learn hdbscan einops\n\nimport pandas as pd\nimport numpy as np\nfrom pathlib import Path\nfrom bertopic import BERTopic\nfrom sentence_transformers import SentenceTransformer\nfrom umap import UMAP\nfrom hdbscan import HDBSCAN\nfrom sklearn.feature_extraction.text import CountVectorizer\nimport re\nimport plotly.express as px\nimport plotly.io as pio\npio.renderers.default = \"iframe\"\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Fix random seeds for reproducibility\nimport random\nSEED = 42\nrandom.seed(SEED)\nnp.random.seed(SEED)\nimport os\nos.environ['PYTHONHASHSEED'] = str(SEED)\n\n# Print library versions for debugging\nimport importlib.metadata\nfor pkg in ['bertopic', 'sentence-transformers', 'umap-learn', 'hdbscan', 'numpy']:\n    try:\n        v = importlib.metadata.version(pkg)\n    except Exception:\n        v = 'unknown'\n    print(f'{pkg}={v}', end='  ')\nprint()\n\n# Data path\nDATA_DIR = Path(\"/kaggle/input/competitions/deep-past-initiative-machine-translation\")\nif not DATA_DIR.exists():\n    DATA_DIR = Path(\"../data/raw\")  # local fallback","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. Loading Training Data\n\n| Dataset | Granularity | Records | Text column | Median length |\n|---------|-------------|---------|-------------|---------------|\n| train.csv | Document | 1,561 | `translation` | ~68 words |\n| train.csv (split) | Sentence | (derived) | sentences from `translation` | varies |","metadata":{}},{"cell_type":"code","source":"# --- train.csv ---\ntrain = pd.read_csv(DATA_DIR / \"train.csv\")\ntrain[\"translation\"] = train[\"translation\"].fillna(\"\")\ntrain_docs = train[train[\"translation\"].str.split().str.len() >= 5][\"translation\"].tolist()\n\nprint(f\"train.csv: {len(train_docs)} documents (after filtering < 5 words)\")\nprint(f\"  Median words: {np.median([len(d.split()) for d in train_docs]):.0f}\")\nprint(f\"  Example: {train_docs[0][:120]}...\")","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Document-Level BERTopic\n\n[BERTopic](https://maartengr.github.io/BERTopic/) discovers topics through four steps:\n\n1. **Embed** documents with a sentence transformer ([nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5), 8192 token context)\n2. **Reduce** dimensionality with UMAP\n3. **Cluster** with HDBSCAN\n4. **Extract** topic keywords with class-based TF-IDF (c-TF-IDF)","metadata":{}},{"cell_type":"code","source":"# Generate embeddings using nomic-embed-text-v1.5 (8192 token context window)\nembedding_model = SentenceTransformer(\"nomic-ai/nomic-embed-text-v1.5\", trust_remote_code=True)\nprint(f\"Model: nomic-ai/nomic-embed-text-v1.5 (max_seq={embedding_model.max_seq_length})\")\nprint(f\"Embedding {len(train_docs)} documents...\")\n\n# nomic requires \"search_document: \" prefix for encoding\nprefixed_docs = [\"search_document: \" + d for d in train_docs]\nembeddings = embedding_model.encode(prefixed_docs, show_progress_bar=True, batch_size=8)\nprint(f\"Embeddings shape: {embeddings.shape}\")\n\n# Embedding quality check\nfrom numpy.linalg import norm\nnorms = norm(embeddings, axis=1)\nprint(f\"Embedding norms: mean={norms.mean():.3f}, std={norms.std():.3f}, min={norms.min():.3f}, max={norms.max():.3f}\")\n# Check if embeddings are degenerate (all same)\ncos_sim_sample = np.dot(embeddings[:5], embeddings[5:10].T) / (norms[:5, None] * norms[5:10][None, :])\nprint(f\"Sample cosine similarities (should vary): {cos_sim_sample.diagonal()}\")\n","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Build BERTopic model\numap_model = UMAP(n_components=5, n_neighbors=15, min_dist=0.0, metric=\"cosine\", random_state=42)\nhdbscan_model = HDBSCAN(min_cluster_size=10, min_samples=3, metric=\"euclidean\",\n                        cluster_selection_method=\"eom\", prediction_data=True)\nvectorizer_model = CountVectorizer(stop_words=\"english\", ngram_range=(1, 2), min_df=2)\n\ntopic_model = BERTopic(\n    embedding_model=embedding_model,\n    umap_model=umap_model,\n    hdbscan_model=hdbscan_model,\n    vectorizer_model=vectorizer_model,\n    top_n_words=10,\n    verbose=True\n)\n\ntopics, probs = topic_model.fit_transform(train_docs, embeddings)\nn_topics = len(set(topics)) - (1 if -1 in topics else 0)\nn_outliers = sum(1 for t in topics if t == -1)\nprint()\nprint(f\"Discovered {n_topics} topics (+ {n_outliers} outlier docs)\")\nprint(f\"Topic distribution: {dict(pd.Series(topics).value_counts().head(10))}\")\n\n# If too few topics, retry with lower min_cluster_size\nif n_topics < 5:\n    print()\n    print(\"WARNING: Very few topics detected. Retrying with min_cluster_size=5...\")\n    umap_model2 = UMAP(n_components=5, n_neighbors=15, min_dist=0.0, metric=\"cosine\", random_state=42)\n    hdbscan_model2 = HDBSCAN(min_cluster_size=5, min_samples=2, metric=\"euclidean\",\n                             cluster_selection_method=\"eom\", prediction_data=True)\n    topic_model = BERTopic(\n        embedding_model=embedding_model,\n        umap_model=umap_model2,\n        hdbscan_model=hdbscan_model2,\n        vectorizer_model=vectorizer_model,\n        top_n_words=10,\n        verbose=True\n    )\n    topics, probs = topic_model.fit_transform(train_docs, embeddings)\n    n_topics = len(set(topics)) - (1 if -1 in topics else 0)\n    n_outliers = sum(1 for t in topics if t == -1)\n    print(f\"Retry: Discovered {n_topics} topics (+ {n_outliers} outlier docs)\")\n    print(f\"Topic distribution: {dict(pd.Series(topics).value_counts().head(10))}\")\n","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### What Topics Did BERTopic Find?\n\nEach topic is represented by its top keywords (via c-TF-IDF)\nand the number of documents assigned to it.","metadata":{}},{"cell_type":"code","source":"# Topic overview table\ntopic_info = topic_model.get_topic_info()\nprint(f\"Total topics: {len(topic_info) - 1} (excluding outliers)\")\nprint()\ntopic_info.head(20)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"try:\n    topic_model.visualize_topics()\nexcept Exception as e:\n    print(f\"visualize_topics() skipped: {e}\")","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"try:\n    topic_model.visualize_hierarchy()\nexcept Exception as e:\n    print(f\"visualize_hierarchy() skipped: {e}\")","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Document Landscape\n\nEach dot represents a training document, projected into 2D via UMAP and colored by topic.\nThis gives a bird's-eye view of the corpus — where are the dense clusters?\nWhere are the sparse, under-represented areas?","metadata":{}},{"cell_type":"code","source":"# UMAP 2D for visualization\numap_2d = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, metric=\"cosine\", random_state=42)\nembeddings_2d = umap_2d.fit_transform(embeddings)\n\nviz_df = pd.DataFrame({\n    \"x\": embeddings_2d[:, 0],\n    \"y\": embeddings_2d[:, 1],\n    \"topic\": [str(t) for t in topics],\n    \"text_preview\": [d[:100] + \"...\" for d in train_docs]\n})\n\nfig = px.scatter(\n    viz_df, x=\"x\", y=\"y\", color=\"topic\",\n    hover_data=[\"text_preview\"],\n    title=\"Train Document Landscape (colored by topic)\",\n    width=900, height=700,\n    opacity=0.6\n)\nfig.update_layout(legend=dict(font=dict(size=9)))\nfig.show()","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Document-Level Topic Structure\n\nWith `min_cluster_size=5` (retry), BERTopic discovers **97 topics** from 1,554 documents,\nwith 408 outlier documents (26%).\n\n**Top topics by document count:**\n\n| Topic | Docs | Keywords |\n|-------|------|----------|\n| T0 | 68 | city, father, said, answered, šalim-aššur |\n| T1 | 39 | seal, month, son, eponymy |\n| T2 | 39 | copper, talents, good copper |\n| T3 | 35 | shekels, cost, donkeys |\n| T4 | 34 | tablet, verdict, city |\n| T5 | 33 | tariff, transport tariff, duty added |\n| T6 | 32 | aššur dagger, testimony, colony |\n| T7 | 32 | silver owed, shekels silver |\n| T8 | 23 | textiles, kuziziya, sell |\n\n**Thematic categories:**\n- **Correspondence** (T0, T9): city/father formulas, šalim-aššur dialogues\n- **Seals & dating** (T1, T13): seal attestations, month/eponymy references\n- **Metals** (T2, T14): copper talents, good copper trade\n- **Debts & payments** (T3, T7): shekels, silver owed, cost calculations\n- **Legal** (T4, T6): tablet/verdict, dagger testimony, colony disputes\n- **Transport** (T5): tariff, import duty, transport fees\n- **Textiles** (T8, T12): textiles, quality, dark textiles\n- **Commodities** (T11, T15): tin, black donkeys, minas\n- **Named individuals** (T10): arik, issu-arik, šalim-aššur cases\n\nThe training data has **rich internal structure** —\n97 distinct topics across trade, legal, correspondence, and commodity domains.","metadata":{}},{"cell_type":"markdown","source":"## 3. Sentence-Level Topic Analysis\n\nThe document-level analysis above uses translations with a median of 68 words.\nEach document may contain multiple sentences spanning different sub-topics.\n\nTo get a more granular view, we split the **English translations into individual sentences**\nusing regex-based sentence boundary detection (splitting on `.` `!` `?`).\nSentences with fewer than 3 words are filtered out.\n\nThis reveals the topic structure at the **individual sentence level**\nusing only the actual training data.","metadata":{}},{"cell_type":"code","source":"# Split train.csv translations into individual sentences\nall_sentences = []\nfor _, row in train.iterrows():\n    tgt = str(row['translation'])\n    sents = [s.strip() for s in re.split(r'(?<=[.!?])\\s+', tgt) if s.strip()]\n    all_sentences.extend(sents)\n\n# Filter to sentences with >= 3 words\nsent_docs = [s for s in all_sentences if len(s.split()) >= 3]\nprint(f'Split {len(train)} documents into {len(all_sentences)} raw sentences')\nprint(f'After filtering (>= 3 words): {len(sent_docs)} sentences')\nprint(f'Median words per sentence: {np.median([len(s.split()) for s in sent_docs]):.0f}')\n\n# Embed sentences (reuse the nomic model)\nprint(f'Embedding {len(sent_docs)} sentences...')\nsent_prefixed = ['search_document: ' + d for d in sent_docs]\nsent_embeddings = embedding_model.encode(sent_prefixed, show_progress_bar=True, batch_size=8)\nprint(f'Embeddings shape: {sent_embeddings.shape}')\n\n# BERTopic on sentences\nsent_umap = UMAP(n_components=5, n_neighbors=15, min_dist=0.0, metric='cosine', random_state=42)\nsent_hdbscan = HDBSCAN(min_cluster_size=10, min_samples=3, metric='euclidean',\n                        cluster_selection_method='eom', prediction_data=True)\nsent_vectorizer = CountVectorizer(stop_words='english', ngram_range=(1, 2), min_df=3)\n\nsent_topic_model = BERTopic(\n    embedding_model=embedding_model,\n    umap_model=sent_umap,\n    hdbscan_model=sent_hdbscan,\n    vectorizer_model=sent_vectorizer,\n    top_n_words=10,\n    verbose=True\n)\n\nsent_topics, sent_probs = sent_topic_model.fit_transform(sent_docs, sent_embeddings)\nsent_n_topics = len(set(sent_topics)) - (1 if -1 in sent_topics else 0)\nn_sent_outliers = sum(1 for t in sent_topics if t == -1)\nprint()\nprint(f'Sentence-level: {sent_n_topics} topics (+ {n_sent_outliers} outliers)')\n\n# If too few topics, retry with lower min_cluster_size\nif sent_n_topics < 5:\n    print()\n    print('WARNING: Few topics. Retrying with min_cluster_size=5, min_samples=2...')\n    sent_umap2 = UMAP(n_components=5, n_neighbors=15, min_dist=0.0, metric='cosine', random_state=42)\n    sent_hdbscan2 = HDBSCAN(min_cluster_size=5, min_samples=2, metric='euclidean',\n                             cluster_selection_method='eom', prediction_data=True)\n    sent_topic_model = BERTopic(\n        embedding_model=embedding_model,\n        umap_model=sent_umap2,\n        hdbscan_model=sent_hdbscan2,\n        vectorizer_model=sent_vectorizer,\n        top_n_words=10,\n        verbose=True\n    )\n    sent_topics, sent_probs = sent_topic_model.fit_transform(sent_docs, sent_embeddings)\n    sent_n_topics = len(set(sent_topics)) - (1 if -1 in sent_topics else 0)\n    n_sent_outliers = sum(1 for t in sent_topics if t == -1)\n    print(f'Retry: {sent_n_topics} topics (+ {n_sent_outliers} outliers)')\n\nsent_topic_info = sent_topic_model.get_topic_info()\nsent_topic_info.head(20)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Sentence-level UMAP 2D scatter\nsent_umap_2d = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, metric=\"cosine\", random_state=42)\nsent_emb_2d = sent_umap_2d.fit_transform(sent_embeddings)\n\nsent_viz = pd.DataFrame({\n    \"x\": sent_emb_2d[:, 0],\n    \"y\": sent_emb_2d[:, 1],\n    \"topic\": [str(t) for t in sent_topics],\n    \"text_preview\": [d[:100] + \"...\" for d in sent_docs]\n})\n\nfig = px.scatter(\n    sent_viz, x=\"x\", y=\"y\", color=\"topic\",\n    hover_data=[\"text_preview\"],\n    title=\"Sentence-Level Topic Landscape (train.csv aligned)\",\n    width=900, height=700,\n    opacity=0.6\n)\nfig.update_layout(legend=dict(font=dict(size=9)))\nfig.show()","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Sentence-Level Results\n\nSplitting 1,561 train documents into individual sentences yields\n**9,345 raw sentences** (8,077 after filtering >= 3 words, median 13 words).\n\nBERTopic discovers **215 topics** with 2,317 outlier sentences (28.7%) —\na dramatic increase from the 97 document-level topics.\n\n**Why more topics at sentence level?**\n- Documents mix multiple formulas (seal + debt + witness in one text).\n  At sentence level, each formula type becomes its own topic.\n- Commodity sub-types split further: different textile types,\n  tin vs copper vs silver transactions each form distinct clusters.\n- Correspondence openings/closings separate from content.\n- Named individuals who appear frequently form dedicated clusters.\n\n**Outlier rate (28.7%)** is similar to document-level (26%),\nindicating a consistent long tail of unique/formulaic fragments.","metadata":{}},{"cell_type":"markdown","source":"## 4. Key Takeaways\n\n### Document-Level: 97 Topics (408 outliers / 1,554 docs)\n\n| Category | Example topics | Approx docs |\n|----------|---------------|-------------|\n| **Correspondence** | T0: city, father, said, answered | 68 |\n| **Seals & dating** | T1: seal, month, eponymy | 39 |\n| **Copper trade** | T2: copper, talents, good copper | 39 |\n| **Costs & donkeys** | T3: shekels, cost, donkeys | 35 |\n| **Legal / tablets** | T4: tablet, verdict, city | 34 |\n| **Transport / tariff** | T5: tariff, transport, duty | 33 |\n| **Colony / testimony** | T6: dagger, testimony, colony | 32 |\n| **Silver debts** | T7: silver owed, shekels | 32 |\n| **Textiles** | T8: textiles, kuziziya, sell | 23 |\n\n26% of documents are outliers — short or idiosyncratic documents\nthat don’t cluster cleanly.\n\n### Sentence-Level: 215 Topics (2,317 outliers / 8,077 sentences)\n\nSplitting into individual sentences reveals **2x more topics**.\nFormulaic sentence types (witness clauses, seal attestations, payment terms)\nemerge as distinct topics — invisible at document level where\ndocuments mix multiple formulas.\n\n### Implications for Modeling\n\n1. **Rich internal structure** — 97 document topics and 215 sentence topics\n   show the training data covers diverse vocabulary patterns.\n2. **Sentence-level evaluation** — the test set is evaluated per-sentence.\n   The 215 sentence topics help predict what the model must handle.\n3. **Outlier rates (~27%)** — about a quarter of both documents and sentences\n   are outliers, indicating a long tail of rare expressions.\n4. **Sentence splitting** — splitting 1,561 docs into 8,077 sentences\n   (median 13 words) can augment training with sentence-level pairs.\n\n---\n\nIf you found this analysis useful, please upvote\\!\nFeedback and discussion welcome in the comments.","metadata":{}}]}