{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":121150,"databundleVersionId":15833391}],"dockerImageVersionId":31259,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n
\n
\n
\n

TEXT EXPLORATORY DATA ANALYSIS

\n

Akkadian to English translation challenge

\n
\n

\n This notebook documents a full exploratory text analysis of a parallel corpus — covering data quality checks, text length distributions, vocabulary & word frequency analysis, word clouds, sentiment analysis, lexical diversity & readability, TF-IDF, and topic modelling using LDA.\n

\n
\n 📅 February 2026\n 👤 J.\n 📁 Akkadian - English dataset\n
\n
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n 01\n

ABOUT THE CORPUS

\n
\n\n
\n
\n
1,561
\n
Segments
\n
\n
\n
3
\n
Columns
\n
\n
\n
EN
\n
Target Lang
\n
\n
\n
2
\n
Text Columns
\n
\n
\n
0%
\n
Missing
\n
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n 02\n

IMPORTS & SETUP

\n
\n\n
\n ⚠ī¸ LIBRARIES USED

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
LibraryPurpose in this notebook
nltkTokenisation, stopword removal, sentence splitting, n-gram generation
spacyLinguistic annotations — POS tagging, lemmatisation, named entity recognition
textblobSentiment polarity & subjectivity scoring on the English translation column
sklearnTF-IDF vectorisation (TfidfVectorizer), document-term matrix (CountVectorizer), and LDA topic modelling (LatentDirichletAllocation)
wordcloudWord frequency clouds for full vocabulary and exclusive token sets per column
pyLDAvisInteractive visualisation of LDA topic-term distributions and inter-topic distances
seabornStatistical plots — KDE distributions, heatmaps, bar charts
matplotlibBase plotting engine underlying seaborn and all custom figures
reRegex-based text cleaning — stripping punctuation, artifacts, and whitespace
stringPunctuation constants used in cleaning and lexical diversity calculations
textwrapFormatting long segments for readable display in notebook output cells
\n
","metadata":{}},{"cell_type":"code","source":"import re\nimport string\nimport nltk\nimport spacy\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\nimport seaborn as sns\n\nfrom collections import Counter\nfrom textblob import TextBlob\n\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import word_tokenize, sent_tokenize\nfrom nltk.util import ngrams\n\nfrom wordcloud import WordCloud\n\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.decomposition import LatentDirichletAllocation\n\nimport pyLDAvis\nimport pyLDAvis.lda_model\n\nfrom IPython.display import HTML, display\n\nnltk.download('stopwords', quiet=True)\nnltk.download('punkt', quiet=True)\nnlp = spacy.load('en_core_web_sm')\n\nSTOP_WORDS = set(stopwords.words('english'))\n\npd.set_option('display.max_colwidth', 120)\nsns.set_theme(style='whitegrid', palette='muted')\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\n\nimport os\nos.environ[\"PYTHONWARNINGS\"] = \"ignore\"\n\nprint('✅ All imports successful')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T12:02:35.688850Z","iopub.execute_input":"2026-02-18T12:02:35.689199Z","iopub.status.idle":"2026-02-18T12:02:36.820600Z","shell.execute_reply.started":"2026-02-18T12:02:35.689170Z","shell.execute_reply":"2026-02-18T12:02:36.819395Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/train.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T10:32:24.309982Z","iopub.execute_input":"2026-02-18T10:32:24.310564Z","iopub.status.idle":"2026-02-18T10:32:24.445756Z","shell.execute_reply.started":"2026-02-18T10:32:24.310525Z","shell.execute_reply":"2026-02-18T10:32:24.444698Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def quick_view(df, title=\"Dataset Preview\"):\n\n display(HTML(f\"\"\"\n
\n

🔍 {title}

\n
\n \"\"\"))\n\n display(HTML(f\"\"\"\n
\n
\n
{df.shape[0]:,}
\n
Rows
\n
\n
\n
{df.shape[1]}
\n
Columns
\n
\n
\n
{df.isnull().sum().sum()}
\n
Missing Cells
\n
\n
\n
{df.duplicated().sum()}
\n
Duplicates
\n
\n
\n
{round(df.memory_usage(deep=True).sum() / 1024, 1)} KB
\n
Memory
\n
\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📄 First & Last 3 Rows\n
\n \"\"\"))\n preview_df = pd.concat([df.head(3), df.tail(3)])\n display(preview_df.style.background_gradient(cmap='Purples', axis=0))\n\n display(HTML(\"\"\"\n
\n 📋 Column Info\n
\n \"\"\"))\n meta_df = pd.DataFrame({\n \"Data Type\": df.dtypes,\n \"Unique Values\": df.nunique(),\n \"Missing\": df.isnull().sum(),\n \"Missing (%)\": (df.isnull().sum() / len(df) * 100).round(2),\n \"Memory (KB)\": (df.memory_usage(deep=True, index=False) / 1024).round(2)\n })\n display(meta_df.style.background_gradient(cmap='RdPu', subset=['Unique Values', 'Missing (%)']))\n\n display(HTML(\"\"\"\n
\n 📊 Statistical Summary\n
\n \"\"\"))\n display(df.describe().T.style.background_gradient(cmap='Purples'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T10:55:27.175007Z","iopub.execute_input":"2026-02-18T10:55:27.176365Z","iopub.status.idle":"2026-02-18T10:55:27.187215Z","shell.execute_reply.started":"2026-02-18T10:55:27.176320Z","shell.execute_reply":"2026-02-18T10:55:27.186118Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"quick_view(df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T10:55:27.958618Z","iopub.execute_input":"2026-02-18T10:55:27.959075Z","iopub.status.idle":"2026-02-18T10:55:28.021713Z","shell.execute_reply.started":"2026-02-18T10:55:27.959022Z","shell.execute_reply":"2026-02-18T10:55:28.020515Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 03\n

DATA QUALITY CHECKS

\n
\n\n
\n ✅ WHAT WE CHECK
\n Missing values per column and exact duplicate rows.\n
","metadata":{}},{"cell_type":"code","source":"def check_data_quality(df, title):\n\n display(HTML(f\"\"\"\n
\n

📋 {title}

\n
\n \"\"\"))\n\n missing = df.isnull().sum()\n pct_missing = (missing / len(df)) * 100\n health_df = pd.DataFrame({'Missing Values': missing, 'Percentage (%)': pct_missing.round(2)})\n report = health_df[health_df['Missing Values'] > 0].sort_values(by='Percentage (%)', ascending=False)\n\n if report.empty:\n display(HTML(\"\"\"\n
\n ✅ MISSING VALUES
\n No missing values detected in any column.\n
\n \"\"\"))\n else:\n display(HTML(f\"\"\"\n
\n 🚨 MISSING VALUES
\n Missing values detected in {len(report)} column(s).\n
\n \"\"\"))\n display(\n report.style\n .background_gradient(cmap='RdPu', subset=['Percentage (%)'])\n .format({'Percentage (%)': '{:.2f}%'})\n )\n\n n_dupes = df.duplicated().sum()\n\n if n_dupes == 0:\n display(HTML(\"\"\"\n
\n ✅ DUPLICATES
\n No duplicate rows detected.\n
\n \"\"\"))\n else:\n pct_dupes = round((n_dupes / len(df)) * 100, 2)\n display(HTML(f\"\"\"\n
\n 🚨 DUPLICATES
\n {n_dupes} duplicate row(s) found — {pct_dupes}% of the dataset.\n
\n \"\"\"))\n\n return report","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T10:54:04.577244Z","iopub.execute_input":"2026-02-18T10:54:04.577641Z","iopub.status.idle":"2026-02-18T10:54:04.588569Z","shell.execute_reply.started":"2026-02-18T10:54:04.577609Z","shell.execute_reply":"2026-02-18T10:54:04.587282Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"check_data_quality(df, title=\"Quality check\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T10:54:05.745561Z","iopub.execute_input":"2026-02-18T10:54:05.746301Z","iopub.status.idle":"2026-02-18T10:54:05.770484Z","shell.execute_reply.started":"2026-02-18T10:54:05.746265Z","shell.execute_reply":"2026-02-18T10:54:05.769307Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 04\n

TEXT LENGTH DISTRIBUTIONS

\n
\n\n
\n ℹī¸ GOAL
\n Understand how long segments are in characters, words, and sentences — comparing distributions across the transliteration and English translation columns.\n
","metadata":{}},{"cell_type":"code","source":"def length_stats(series):\n return {\n \"char_count\": series.str.len(),\n \"word_count\": series.str.split().str.len(),\n \"sent_count\": series.str.count(r'[.!?]+')\n }\n\ndef stat_cards(char_s, word_s, sent_s, color):\n return f\"\"\"\n
\n
\n
{int(char_s.mean()):,}
\n
Avg. Chars
\n
\n
\n
{int(char_s.max()):,}
\n
Max Chars
\n
\n
\n
{int(word_s.mean()):,}
\n
Avg. Words
\n
\n
\n
{int(word_s.max()):,}
\n
Max Words
\n
\n
\n
{int(word_s.median()):,}
\n
Median Words
\n
\n
\n
{int(sent_s.mean()):,}
\n
Avg. Sentences
\n
\n
\n \"\"\"\n \ndef text_length_distribution(df, col1, col2, col1_name=\"Transliteration\", col2_name=\"English Translation\"):\n\n s1 = length_stats(df[col1])\n s2 = length_stats(df[col2])\n\n display(HTML(f\"\"\"\n
\n

📏 Text Length Distribution — {col1_name} vs {col2_name}

\n
\n \"\"\"))\n\n display(HTML(f\"\"\"\n
\n 🔤 {col1_name}\n
\n \"\"\"))\n display(HTML(stat_cards(s1[\"char_count\"], s1[\"word_count\"], s1[\"sent_count\"], \"#9b59b6\")))\n\n display(HTML(f\"\"\"\n
\n 🔤 {col2_name}\n
\n \"\"\"))\n display(HTML(stat_cards(s2[\"char_count\"], s2[\"word_count\"], s2[\"sent_count\"], \"#f64f59\")))\n\n display(HTML(\"\"\"\n
\n 📊 Distribution Plots\n
\n \"\"\"))\n\n metrics = [\"char_count\", \"word_count\", \"sent_count\"]\n metric_labels = [\"Character Count\", \"Word Count\", \"Sentence Count\"]\n colors= {\"col1\": \"#9b59b6\", \"col2\": \"#f64f59\"}\n\n fig = plt.figure(figsize=(18, 14))\n gs = gridspec.GridSpec(3, 3, figure=fig, hspace=0.45, wspace=0.35)\n\n for row, (metric, label) in enumerate(zip(metrics, metric_labels)):\n data1 = s1[metric].dropna()\n data2 = s2[metric].dropna()\n\n ax_kde = fig.add_subplot(gs[row, 0])\n sns.kdeplot(data1, ax=ax_kde, color=colors[\"col1\"], fill=True, alpha=0.35, linewidth=2, label=col1_name)\n sns.kdeplot(data2, ax=ax_kde, color=colors[\"col2\"], fill=True, alpha=0.35, linewidth=2, label=col2_name)\n ax_kde.set_title(f\"{label} — KDE\", fontsize=11, fontweight=\"bold\", pad=10)\n ax_kde.set_xlabel(label); ax_kde.set_ylabel(\"Density\")\n ax_kde.legend(fontsize=9)\n sns.despine(ax=ax_kde)\n\n ax_hist = fig.add_subplot(gs[row, 1])\n ax_hist.hist(data1, bins=40, color=colors[\"col1\"], alpha=0.6, label=col1_name, edgecolor=\"white\")\n ax_hist.hist(data2, bins=40, color=colors[\"col2\"], alpha=0.6, label=col2_name, edgecolor=\"white\")\n ax_hist.set_title(f\"{label} — Histogram\", fontsize=11, fontweight=\"bold\", pad=10)\n ax_hist.set_xlabel(label); ax_hist.set_ylabel(\"Frequency\")\n ax_hist.legend(fontsize=9)\n sns.despine(ax=ax_hist)\n\n ax_box = fig.add_subplot(gs[row, 2])\n box_data = [data1.values, data2.values]\n bp = ax_box.boxplot(box_data, patch_artist=True, widths=0.5,\n medianprops=dict(color=\"white\", linewidth=2))\n for patch, color in zip(bp[\"boxes\"], [colors[\"col1\"], colors[\"col2\"]]):\n patch.set_facecolor(color); patch.set_alpha(0.75)\n for whisker in bp[\"whiskers\"]: whisker.set(color=\"#555\", linewidth=1.2)\n for cap in bp[\"caps\"]: cap.set(color=\"#555\", linewidth=1.2)\n for flier in bp[\"fliers\"]: flier.set(marker=\"o\", color=\"#aaa\", alpha=0.4, markersize=3)\n ax_box.set_xticks([1, 2])\n ax_box.set_xticklabels([col1_name, col2_name], fontsize=9)\n ax_box.set_title(f\"{label} — Boxplot\", fontsize=11, fontweight=\"bold\", pad=10)\n ax_box.set_ylabel(label)\n sns.despine(ax=ax_box)\n\n fig.suptitle(f\"Text Length Analysis: {col1_name} vs {col2_name}\",\n fontsize=14, fontweight=\"bold\", y=1.01, color=\"#2d1b69\")\n plt.savefig(\"text_length_distribution.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T10:59:57.502128Z","iopub.execute_input":"2026-02-18T10:59:57.502470Z","iopub.status.idle":"2026-02-18T10:59:57.525165Z","shell.execute_reply.started":"2026-02-18T10:59:57.502440Z","shell.execute_reply":"2026-02-18T10:59:57.524162Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"text_length_distribution(df, col1='transliteration', col2='translation')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:00:44.450678Z","iopub.execute_input":"2026-02-18T11:00:44.451433Z","iopub.status.idle":"2026-02-18T11:00:47.972914Z","shell.execute_reply.started":"2026-02-18T11:00:44.451393Z","shell.execute_reply":"2026-02-18T11:00:47.971999Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 05\n

VOCABULARY & WORD FREQUENCY

\n
\n\n
\n ℹī¸ GOAL
\n Identify the most frequent tokens in each column, compare vocabulary size and lexical diversity, and examine vocabulary overlap between the transliteration and English translation.\n
","metadata":{}},{"cell_type":"code","source":"def tokenise(series):\n return series.dropna().str.lower().str.split()\n \ndef vocab_cards(vocab, freq, total_tokens, color, name):\n return f\"\"\"\n
\n
\n
{total_tokens:,}
\n
Total Tokens
\n
\n
\n
{len(vocab):,}
\n
Vocab Size
\n
\n
\n
{round(len(vocab) / total_tokens * 100, 1)}%
\n
Lexical Diversity
\n
\n
\n
{freq.most_common(1)[0][0]}
\n
Top Token
\n
\n
\n
{freq.most_common(1)[0][1]:,}
\n
Top Token Count
\n
\n
\n \"\"\"\n \ndef plot_vocab_word_frequency(df, col1, col2, col1_name=\"Transliteration\", col2_name=\"English Translation\", top_n=20):\n\n display(HTML(f\"\"\"\n
\n

📚 Vocabulary & Word Frequency — {col1_name} vs {col2_name}

\n
\n \"\"\"))\n\n\n tokens1 = [w for tokens in tokenise(df[col1]) for w in tokens]\n tokens2 = [w for tokens in tokenise(df[col2]) for w in tokens]\n\n freq1 = Counter(tokens1)\n freq2 = Counter(tokens2)\n\n vocab1 = set(freq1.keys())\n vocab2 = set(freq2.keys())\n\n shared_vocab = vocab1 & vocab2\n unique_vocab1 = vocab1 - vocab2\n unique_vocab2 = vocab2 - vocab1\n\n\n display(HTML(f\"\"\"\n
\n 🔤 {col1_name}\n
\n \"\"\"))\n display(HTML(vocab_cards(vocab1, freq1, len(tokens1), \"#9b59b6\", col1_name)))\n\n display(HTML(f\"\"\"\n
\n 🔤 {col2_name}\n
\n \"\"\"))\n display(HTML(vocab_cards(vocab2, freq2, len(tokens2), \"#f64f59\", col2_name)))\n\n display(HTML(\"\"\"\n
\n 🔗 Vocabulary Overlap\n
\n \"\"\"))\n display(HTML(f\"\"\"\n
\n
\n
{len(shared_vocab):,}
\n
Shared Tokens
\n
\n
\n
{len(unique_vocab1):,}
\n
Unique to {col1_name}
\n
\n
\n
{len(unique_vocab2):,}
\n
Unique to {col2_name}
\n
\n
\n
{round(len(shared_vocab) / len(vocab1 | vocab2) * 100, 1)}%
\n
Overlap Ratio
\n
\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📊 Frequency Plots\n
\n \"\"\"))\n\n top1 = freq1.most_common(top_n)\n top2 = freq2.most_common(top_n)\n\n words1, counts1 = zip(*top1)\n words2, counts2 = zip(*top2)\n\n fig, axes = plt.subplots(2, 2, figsize=(18, 14))\n fig.suptitle(f\"Vocabulary & Word Frequency: {col1_name} vs {col2_name}\",\n fontsize=14, fontweight=\"bold\", y=1.01, color=\"#2d1b69\")\n\n axes[0, 0].barh(words1[::-1], counts1[::-1], color=\"#9b59b6\", edgecolor=\"white\", alpha=0.85)\n axes[0, 0].set_title(f\"Top {top_n} Tokens — {col1_name}\", fontsize=11, fontweight=\"bold\")\n axes[0, 0].set_xlabel(\"Frequency\")\n sns.despine(ax=axes[0, 0])\n\n axes[0, 1].barh(words2[::-1], counts2[::-1], color=\"#f64f59\", edgecolor=\"white\", alpha=0.85)\n axes[0, 1].set_title(f\"Top {top_n} Tokens — {col2_name}\", fontsize=11, fontweight=\"bold\")\n axes[0, 1].set_xlabel(\"Frequency\")\n sns.despine(ax=axes[0, 1])\n\n for freq_obj, color, name in [(freq1, \"#9b59b6\", col1_name), (freq2, \"#f64f59\", col2_name)]:\n sorted_counts = sorted(freq_obj.values(), reverse=True)\n ranks = range(1, len(sorted_counts) + 1)\n axes[1, 0].loglog(ranks, sorted_counts, color=color, alpha=0.8, linewidth=2, label=name)\n axes[1, 0].set_title(\"Zipf's Law — Rank vs Frequency\", fontsize=11, fontweight=\"bold\")\n axes[1, 0].set_xlabel(\"Rank (log)\")\n axes[1, 0].set_ylabel(\"Frequency (log)\")\n axes[1, 0].legend(fontsize=9)\n sns.despine(ax=axes[1, 0])\n\n overlap_data = {\n f\"Unique to\\n{col1_name}\": len(unique_vocab1),\n \"Shared\": len(shared_vocab),\n f\"Unique to\\n{col2_name}\": len(unique_vocab2),\n }\n bar_colors = [\"#9b59b6\", \"#27ae60\", \"#f64f59\"]\n axes[1, 1].bar(overlap_data.keys(), overlap_data.values(), color=bar_colors, edgecolor=\"white\", alpha=0.85)\n axes[1, 1].set_title(\"Vocabulary Overlap\", fontsize=11, fontweight=\"bold\")\n axes[1, 1].set_ylabel(\"Token Count\")\n for i, (k, v) in enumerate(overlap_data.items()):\n axes[1, 1].text(i, v + max(overlap_data.values()) * 0.01, f\"{v:,}\",\n ha=\"center\", fontsize=10, fontweight=\"bold\", color=bar_colors[i])\n sns.despine(ax=axes[1, 1])\n\n plt.tight_layout()\n plt.show()","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:06:54.438553Z","iopub.execute_input":"2026-02-18T11:06:54.439565Z","iopub.status.idle":"2026-02-18T11:06:54.463124Z","shell.execute_reply.started":"2026-02-18T11:06:54.439524Z","shell.execute_reply":"2026-02-18T11:06:54.461824Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_vocab_word_frequency(df, col1=\"transliteration\", col2=\"translation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:07:02.259312Z","iopub.execute_input":"2026-02-18T11:07:02.259692Z","iopub.status.idle":"2026-02-18T11:07:03.871710Z","shell.execute_reply.started":"2026-02-18T11:07:02.259658Z","shell.execute_reply":"2026-02-18T11:07:03.870524Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 06\n

WORD CLOUDS

\n
\n\n
\n ✅ GOAL
\n Visually explore the most frequent vocabulary in each column — full vocabulary side by side, and tokens exclusive to each column to highlight what is unique to the transliteration vs. the English translation.\n
","metadata":{}},{"cell_type":"code","source":"def get_text(series):\n return \" \".join(series.dropna().str.lower().str.split().explode().tolist())\n\ndef make_wordcloud(text, colormap):\n return WordCloud(\n width=900,\n height=450,\n background_color=\"white\",\n colormap=colormap,\n max_words=150,\n collocations=False,\n contour_width=0,\n prefer_horizontal=0.9\n ).generate(text)\n\n\n\ndef plot_wordclouds(df, col1, col2, col1_name=\"Transliteration\", col2_name=\"English Translation\"):\n\n display(HTML(f\"\"\"\n
\n

☁ī¸ Word Clouds — {col1_name} vs {col2_name}

\n
\n \"\"\"))\n\n text1 = get_text(df[col1])\n text2 = get_text(df[col2])\n\n display(HTML(\"\"\"\n
\n ☁ī¸ Full Vocabulary — Both Columns\n
\n \"\"\"))\n\n fig, axes = plt.subplots(1, 2, figsize=(18, 6))\n fig.suptitle(f\"Word Clouds: {col1_name} vs {col2_name}\",\n fontsize=14, fontweight=\"bold\", y=1.02, color=\"#2d1b69\")\n\n for ax, text, name, cmap, color in [\n (axes[0], text1, col1_name, \"Purples\", \"#9b59b6\"),\n (axes[1], text2, col2_name, \"RdPu\", \"#f64f59\"),\n ]:\n wc = make_wordcloud(text, cmap)\n ax.imshow(wc, interpolation=\"bilinear\")\n ax.axis(\"off\")\n ax.set_title(name, fontsize=13, fontweight=\"bold\", color=color, pad=12)\n\n plt.tight_layout()\n plt.savefig(\"wordcloud_both.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:17:05.491090Z","iopub.execute_input":"2026-02-18T11:17:05.491434Z","iopub.status.idle":"2026-02-18T11:17:05.503150Z","shell.execute_reply.started":"2026-02-18T11:17:05.491404Z","shell.execute_reply":"2026-02-18T11:17:05.502091Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_wordclouds(df, col1=\"transliteration\", col2=\"translation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:17:06.471877Z","iopub.execute_input":"2026-02-18T11:17:06.472248Z","iopub.status.idle":"2026-02-18T11:17:09.587517Z","shell.execute_reply.started":"2026-02-18T11:17:06.472217Z","shell.execute_reply":"2026-02-18T11:17:09.586552Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 07\n

SENTIMENT ANALYSIS

\n
\n\n
\n ℹī¸ GOAL
\n Analyse the emotional tone of the English translation column using TextBlob — measuring polarity (negative to positive) and subjectivity (objective to subjective) per segment, and examining how sentiment distributes across the corpus.\n
","metadata":{}},{"cell_type":"code","source":"def plot_sentiment_analysis(df, col, col_name=\"English Translation\"):\n\n display(HTML(f\"\"\"\n
\n

🎭 Sentiment Analysis — {col_name}

\n
\n \"\"\"))\n\n sentiments = df[col].dropna().apply(lambda x: TextBlob(x))\n df = df.copy()\n df[\"polarity\"] = df[col].dropna().apply(lambda x: TextBlob(x).sentiment.polarity)\n df[\"subjectivity\"] = df[col].dropna().apply(lambda x: TextBlob(x).sentiment.subjectivity)\n\n df[\"sentiment_label\"] = df[\"polarity\"].apply(\n lambda x: \"Positive\" if x > 0 else (\"Negative\" if x < 0 else \"Neutral\")\n )\n\n counts = df[\"sentiment_label\"].value_counts()\n n_pos = counts.get(\"Positive\", 0)\n n_neu = counts.get(\"Neutral\", 0)\n n_neg = counts.get(\"Negative\", 0)\n avg_pol = df[\"polarity\"].mean().round(3)\n avg_sub = df[\"subjectivity\"].mean().round(3)\n\n display(HTML(f\"\"\"\n
\n
\n
{n_pos:,}
\n
Positive
\n
\n
\n
{n_neu:,}
\n
Neutral
\n
\n
\n
{n_neg:,}
\n
Negative
\n
\n
\n
{avg_pol}
\n
Avg. Polarity
\n
\n
\n
{avg_sub}
\n
Avg. Subjectivity
\n
\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📊 Sentiment Distribution Plots\n
\n \"\"\"))\n\n sentiment_colors = {\"Positive\": \"#27ae60\", \"Neutral\": \"#95a5a6\", \"Negative\": \"#e74c3c\"}\n\n fig, axes = plt.subplots(2, 2, figsize=(16, 12))\n fig.suptitle(f\"Sentiment Analysis: {col_name}\",\n fontsize=14, fontweight=\"bold\", y=1.01, color=\"#2d1b69\")\n\n sns.kdeplot(df[\"polarity\"], ax=axes[0, 0], fill=True, color=\"#9b59b6\", alpha=0.4, linewidth=2)\n axes[0, 0].axvline(0, color=\"#e74c3c\", linestyle=\"--\", linewidth=1.5, label=\"Neutral (0)\")\n axes[0, 0].axvline(avg_pol, color=\"#2d1b69\", linestyle=\"--\", linewidth=1.5, label=f\"Mean ({avg_pol})\")\n axes[0, 0].set_title(\"Polarity Distribution\", fontsize=11, fontweight=\"bold\")\n axes[0, 0].set_xlabel(\"Polarity (−1 Negative → +1 Positive)\")\n axes[0, 0].set_ylabel(\"Density\")\n axes[0, 0].legend(fontsize=9)\n sns.despine(ax=axes[0, 0])\n\n sns.kdeplot(df[\"subjectivity\"], ax=axes[0, 1], fill=True, color=\"#f64f59\", alpha=0.4, linewidth=2)\n axes[0, 1].axvline(0.5, color=\"#e74c3c\", linestyle=\"--\", linewidth=1.5, label=\"Mid (0.5)\")\n axes[0, 1].axvline(avg_sub, color=\"#2d1b69\", linestyle=\"--\", linewidth=1.5, label=f\"Mean ({avg_sub})\")\n axes[0, 1].set_title(\"Subjectivity Distribution\", fontsize=11, fontweight=\"bold\")\n axes[0, 1].set_xlabel(\"Subjectivity (0 Objective → 1 Subjective)\")\n axes[0, 1].set_ylabel(\"Density\")\n axes[0, 1].legend(fontsize=9)\n sns.despine(ax=axes[0, 1])\n\n label_order = [\"Positive\", \"Neutral\", \"Negative\"]\n label_counts = [counts.get(l, 0) for l in label_order]\n bar_colors = [sentiment_colors[l] for l in label_order]\n bars = axes[1, 0].bar(label_order, label_counts, color=bar_colors, edgecolor=\"white\", alpha=0.85)\n for bar, count in zip(bars, label_counts):\n axes[1, 0].text(bar.get_x() + bar.get_width() / 2,\n bar.get_height() + max(label_counts) * 0.01,\n f\"{count:,}\", ha=\"center\", fontsize=10, fontweight=\"bold\")\n axes[1, 0].set_title(\"Sentiment Label Counts\", fontsize=11, fontweight=\"bold\")\n axes[1, 0].set_ylabel(\"Count\")\n sns.despine(ax=axes[1, 0])\n\n scatter_colors = df[\"sentiment_label\"].map(sentiment_colors)\n axes[1, 1].scatter(df[\"polarity\"], df[\"subjectivity\"],\n c=scatter_colors, alpha=0.35, s=18, edgecolors=\"none\")\n axes[1, 1].axvline(0, color=\"#aaa\", linestyle=\"--\", linewidth=1)\n axes[1, 1].axhline(0.5, color=\"#aaa\", linestyle=\"--\", linewidth=1)\n axes[1, 1].set_title(\"Polarity vs Subjectivity\", fontsize=11, fontweight=\"bold\")\n axes[1, 1].set_xlabel(\"Polarity\")\n axes[1, 1].set_ylabel(\"Subjectivity\")\n\n for label, color in sentiment_colors.items():\n axes[1, 1].scatter([], [], c=color, label=label, s=40)\n axes[1, 1].legend(fontsize=9)\n sns.despine(ax=axes[1, 1])\n\n plt.tight_layout()\n plt.savefig(\"sentiment_analysis.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2026-02-18T11:17:16.651589Z","iopub.execute_input":"2026-02-18T11:17:16.651941Z","iopub.status.idle":"2026-02-18T11:17:16.674999Z","shell.execute_reply.started":"2026-02-18T11:17:16.651903Z","shell.execute_reply":"2026-02-18T11:17:16.673738Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_sentiment_analysis(df, col=\"translation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:17:49.475467Z","iopub.execute_input":"2026-02-18T11:17:49.476019Z","iopub.status.idle":"2026-02-18T11:17:53.736072Z","shell.execute_reply.started":"2026-02-18T11:17:49.475984Z","shell.execute_reply":"2026-02-18T11:17:53.735179Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 08\n

LEXICAL DIVERSITY & READABILITY

\n
\n\n
\n ℹī¸ GOAL
\n Compare stylometric properties across both columns — including Type-Token Ratio (TTR), average word length, average sentence length, punctuation density, and Flesch Reading Ease — to understand how linguistically rich and readable each column is relative to the other.\n
","metadata":{}},{"cell_type":"code","source":"def compute_metrics(series):\n results = []\n for text in series.dropna():\n words = text.lower().split()\n word_count = len(words)\n unique = set(words)\n sentences = re.split(r'[.!?]+', text.strip())\n sentences = [s for s in sentences if s.strip()]\n sent_count= max(len(sentences), 1)\n char_count = len(text.replace(\" \", \"\"))\n\n # Type-Token Ratio\n ttr = len(unique) / word_count if word_count > 0 else 0\n\n avg_word_len = np.mean([len(w) for w in words]) if words else 0\n\n punct_density = sum(1 for c in text if c in string.punctuation) / max(len(text), 1)\n\n caps_ratio = sum(1 for c in text if c.isupper()) / max(len(text), 1)\n\n avg_sent_len = word_count / sent_count\n\n syllable_count = sum(\n max(1, len(re.findall(r'[aeiouAEIOU]', w))) for w in words\n )\n if word_count > 0 and sent_count > 0:\n flesch = (\n 206.835\n - 1.015 * (word_count / sent_count)\n - 84.6 * (syllable_count / max(word_count, 1))\n )\n else:\n flesch = 0\n\n results.append({\n \"ttr\": ttr,\n \"avg_word_len\": avg_word_len,\n \"punct_density\": punct_density,\n \"caps_ratio\": caps_ratio,\n \"avg_sent_len\": avg_sent_len,\n \"flesch\": flesch,\n })\n return pd.DataFrame(results)\n\n\ndef stat_cards(m, color, name):\n return f\"\"\"\n
\n
\n
{m['ttr'].mean():.3f}
\n
Avg. TTR
\n
\n
\n
{m['avg_word_len'].mean():.2f}
\n
Avg. Word Length
\n
\n
\n
{m['avg_sent_len'].mean():.1f}
\n
Avg. Sent. Length
\n
\n
\n
{m['punct_density'].mean():.3f}
\n
Punct. Density
\n
\n
\n
{m['flesch'].mean():.1f}
\n
Flesch Score
\n
\n
\n \"\"\"\n\ndef plot_lexical_diversity_readability(df, col1, col2, col1_name=\"Transliteration\", col2_name=\"English Translation\"):\n\n display(HTML(f\"\"\"\n
\n

📖 Lexical Diversity & Readability — {col1_name} vs {col2_name}

\n
\n \"\"\"))\n\n\n metrics1 = compute_metrics(df[col1])\n metrics2 = compute_metrics(df[col2])\n\n \n display(HTML(f\"\"\"\n
\n 🔤 {col1_name}\n
\n \"\"\"))\n display(HTML(stat_cards(metrics1, \"#9b59b6\", col1_name)))\n\n display(HTML(f\"\"\"\n
\n 🔤 {col2_name}\n
\n \"\"\"))\n display(HTML(stat_cards(metrics2, \"#f64f59\", col2_name)))\n\n display(HTML(\"\"\"\n
\n ⚠ī¸ FLESCH READING EASE SCALE
\n 90–100 Very easy (5th grade)  | \n 70–90 Easy  | \n 60–70 Standard  | \n 50–60 Fairly difficult  | \n 30–50 Difficult  | \n 0–30 Very difficult (academic)
\n Note: Flesch score is designed for English — treat the transliteration score as a comparative signal only.\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📊 Distribution Plots\n
\n \"\"\"))\n\n plot_metrics = [\"ttr\", \"avg_word_len\", \"avg_sent_len\", \"punct_density\", \"flesch\"]\n plot_labels = [\n \"Type-Token Ratio (TTR)\",\n \"Avg. Word Length (chars)\",\n \"Avg. Sentence Length (words)\",\n \"Punctuation Density\",\n \"Flesch Reading Ease\"\n ]\n\n fig, axes = plt.subplots(2, 3, figsize=(18, 11))\n axes = axes.flatten()\n fig.suptitle(f\"Lexical Diversity & Readability: {col1_name} vs {col2_name}\",\n fontsize=14, fontweight=\"bold\", y=1.01, color=\"#2d1b69\")\n\n for i, (metric, label) in enumerate(zip(plot_metrics, plot_labels)):\n ax = axes[i]\n d1 = metrics1[metric].dropna()\n d2 = metrics2[metric].dropna()\n mean1 = d1.mean()\n mean2 = d2.mean()\n\n sns.kdeplot(d1, ax=ax, fill=True, color=\"#9b59b6\", alpha=0.35, linewidth=2, label=col1_name)\n sns.kdeplot(d2, ax=ax, fill=True, color=\"#f64f59\", alpha=0.35, linewidth=2, label=col2_name)\n ax.axvline(mean1, color=\"#9b59b6\", linestyle=\"--\", linewidth=1.5, alpha=0.8,\n label=f\"{col1_name} mean ({mean1:.2f})\")\n ax.axvline(mean2, color=\"#f64f59\", linestyle=\"--\", linewidth=1.5, alpha=0.8,\n label=f\"{col2_name} mean ({mean2:.2f})\")\n ax.set_title(label, fontsize=11, fontweight=\"bold\")\n ax.set_xlabel(label)\n ax.set_ylabel(\"Density\")\n ax.legend(fontsize=8)\n sns.despine(ax=ax)\n\n ax = axes[5]\n means1 = [metrics1[m].mean() for m in plot_metrics]\n means2 = [metrics2[m].mean() for m in plot_metrics]\n short_labels = [\"TTR\", \"Avg Word Len\", \"Avg Sent Len\", \"Punct Density\", \"Flesch\"]\n x = np.arange(len(short_labels))\n width = 0.35\n bars1 = ax.bar(x - width/2, means1, width, color=\"#9b59b6\", alpha=0.8,\n edgecolor=\"white\", label=col1_name)\n bars2 = ax.bar(x + width/2, means2, width, color=\"#f64f59\", alpha=0.8,\n edgecolor=\"white\", label=col2_name)\n ax.set_xticks(x)\n ax.set_xticklabels(short_labels, fontsize=8, rotation=15, ha=\"right\")\n ax.set_title(\"Metric Comparison Overview\", fontsize=11, fontweight=\"bold\")\n ax.set_ylabel(\"Mean Value\")\n ax.legend(fontsize=9)\n sns.despine(ax=ax)\n\n plt.tight_layout()\n plt.savefig(\"lexical_diversity_readability.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:22:02.854599Z","iopub.execute_input":"2026-02-18T11:22:02.854941Z","iopub.status.idle":"2026-02-18T11:22:02.881148Z","shell.execute_reply.started":"2026-02-18T11:22:02.854912Z","shell.execute_reply":"2026-02-18T11:22:02.880128Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_lexical_diversity_readability(df, col1=\"transliteration\", col2=\"translation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:22:10.452586Z","iopub.execute_input":"2026-02-18T11:22:10.452978Z","iopub.status.idle":"2026-02-18T11:22:14.164553Z","shell.execute_reply.started":"2026-02-18T11:22:10.452948Z","shell.execute_reply":"2026-02-18T11:22:14.163593Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n 09\n

TF-IDF & TOPIC MODELLING

\n
\n\n
\n ℹī¸ GOAL
\n Identify the most statistically important terms in the English translation column using TF-IDF, then uncover latent themes across the corpus using LDA topic modelling — with the optimal number of topics selected automatically based on perplexity score.\n
","metadata":{}},{"cell_type":"code","source":"def plot_tfidf(df, col, col_name=\"English Translation\", top_n=20):\n\n display(HTML(f\"\"\"\n
\n

đŸ”ĸ TF-IDF Analysis — {col_name}

\n
\n \"\"\"))\n\n corpus = df[col].dropna().tolist()\n\n vectorizer = TfidfVectorizer(\n max_features=5000,\n stop_words=\"english\",\n ngram_range=(1, 2),\n min_df=2,\n sublinear_tf=True\n )\n \n tfidf_matrix = vectorizer.fit_transform(corpus)\n feature_names = vectorizer.get_feature_names_out()\n\n mean_scores = np.asarray(tfidf_matrix.mean(axis=0)).flatten()\n top_indices = mean_scores.argsort()[::-1][:top_n]\n top_terms = [feature_names[i] for i in top_indices]\n top_scores = [mean_scores[i] for i in top_indices]\n\n uni_mask = [i for i, t in enumerate(feature_names) if \" \" not in t]\n uni_scores = mean_scores[uni_mask]\n uni_terms = feature_names[uni_mask]\n top_uni_idx = uni_scores.argsort()[::-1][:top_n]\n top_uni_terms = uni_terms[top_uni_idx]\n top_uni_scores = uni_scores[top_uni_idx]\n\n bi_mask = [i for i, t in enumerate(feature_names) if \" \" in t]\n bi_scores = mean_scores[bi_mask]\n bi_terms = feature_names[bi_mask]\n top_bi_idx = bi_scores.argsort()[::-1][:top_n]\n top_bi_terms = bi_terms[top_bi_idx]\n top_bi_scores = bi_scores[top_bi_idx]\n\n display(HTML(f\"\"\"\n
\n
\n
{len(corpus):,}
\n
Documents
\n
\n
\n
{len(feature_names):,}
\n
Features
\n
\n
\n
{sum(1 for t in feature_names if \" \" not in t):,}
\n
Unigrams
\n
\n
\n
{sum(1 for t in feature_names if \" \" in t):,}
\n
Bigrams
\n
\n
\n
{top_terms[0]}
\n
Top Term
\n
\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📊 TF-IDF Plots\n
\n \"\"\"))\n\n fig, axes = plt.subplots(1, 3, figsize=(20, 7))\n fig.suptitle(f\"TF-IDF Analysis: {col_name}\",\n fontsize=14, fontweight=\"bold\", y=1.01, color=\"#2d1b69\")\n\n axes[0].barh(top_terms[::-1], top_scores[::-1], color=\"#9b59b6\", edgecolor=\"white\", alpha=0.85)\n axes[0].set_title(f\"Top {top_n} Terms (Unigrams + Bigrams)\", fontsize=11, fontweight=\"bold\")\n axes[0].set_xlabel(\"Mean TF-IDF Score\")\n sns.despine(ax=axes[0])\n\n axes[1].barh(top_uni_terms[::-1], top_uni_scores[::-1], color=\"#f64f59\", edgecolor=\"white\", alpha=0.85)\n axes[1].set_title(f\"Top {top_n} Unigrams\", fontsize=11, fontweight=\"bold\")\n axes[1].set_xlabel(\"Mean TF-IDF Score\")\n sns.despine(ax=axes[1])\n\n axes[2].barh(top_bi_terms[::-1], top_bi_scores[::-1], color=\"#3498db\", edgecolor=\"white\", alpha=0.85)\n axes[2].set_title(f\"Top {top_n} Bigrams\", fontsize=11, fontweight=\"bold\")\n axes[2].set_xlabel(\"Mean TF-IDF Score\")\n sns.despine(ax=axes[2])\n\n plt.tight_layout()\n plt.show()\n\n count_vectorizer = CountVectorizer(\n max_features=5000,\n stop_words=\"english\",\n min_df=2,\n max_df=0.95\n )\n dtm = count_vectorizer.fit_transform(corpus)\n\n return tfidf_matrix, vectorizer, dtm, count_vectorizer","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:54:46.855731Z","iopub.execute_input":"2026-02-18T11:54:46.856104Z","iopub.status.idle":"2026-02-18T11:54:46.874968Z","shell.execute_reply.started":"2026-02-18T11:54:46.856073Z","shell.execute_reply":"2026-02-18T11:54:46.873760Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tfidf_matrix, tfidf_vectorizer, dtm, cv = plot_tfidf(df, col=\"translation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:54:47.501168Z","iopub.execute_input":"2026-02-18T11:54:47.501490Z","iopub.status.idle":"2026-02-18T11:54:48.632495Z","shell.execute_reply.started":"2026-02-18T11:54:47.501463Z","shell.execute_reply":"2026-02-18T11:54:48.631488Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_lda_topics(df, col, col_name=\"English Translation\", n_topics_range=(2, 11), random_state=42):\n\n display(HTML(f\"\"\"\n
\n

🗂ī¸ LDA Topic Modelling — {col_name}

\n
\n \"\"\"))\n\n corpus = df[col].dropna().tolist()\n\n count_vectorizer = CountVectorizer(\n max_features=5000,\n stop_words=\"english\",\n min_df=2,\n max_df=0.95\n )\n dtm = count_vectorizer.fit_transform(corpus)\n vocab = count_vectorizer.get_feature_names_out()\n\n display(HTML(\"\"\"\n
\n 🔍 Auto-tuning — Searching Best Number of Topics\n
\n \"\"\"))\n\n topic_range = range(n_topics_range[0], n_topics_range[1])\n perplexities = []\n log_likelihoods = []\n\n for n in topic_range:\n lda_tmp = LatentDirichletAllocation(\n n_components=n,\n random_state=random_state,\n max_iter=10,\n learning_method=\"online\"\n )\n lda_tmp.fit(dtm)\n perplexities.append(lda_tmp.perplexity(dtm))\n log_likelihoods.append(lda_tmp.score(dtm))\n\n best_n = list(topic_range)[np.argmin(perplexities)]\n\n display(HTML(f\"\"\"\n
\n ✅ BEST NUMBER OF TOPICS: {best_n}
\n Selected based on lowest perplexity score across n = {n_topics_range[0]} to {n_topics_range[1] - 1}.\n Lower perplexity = better generalisability of the model.\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📊 Tuning Curves\n
\n \"\"\"))\n\n fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n fig.suptitle(\"LDA Auto-tuning: Perplexity & Log-Likelihood vs Number of Topics\",\n fontsize=13, fontweight=\"bold\", y=1.02, color=\"#2d1b69\")\n\n axes[0].plot(list(topic_range), perplexities, marker=\"o\", color=\"#9b59b6\",\n linewidth=2, markersize=7)\n axes[0].axvline(best_n, color=\"#f64f59\", linestyle=\"--\", linewidth=1.5,\n label=f\"Best n = {best_n}\")\n axes[0].set_title(\"Perplexity (lower = better)\", fontsize=11, fontweight=\"bold\")\n axes[0].set_xlabel(\"Number of Topics\")\n axes[0].set_ylabel(\"Perplexity\")\n axes[0].legend(fontsize=9)\n sns.despine(ax=axes[0])\n\n axes[1].plot(list(topic_range), log_likelihoods, marker=\"o\", color=\"#f64f59\",\n linewidth=2, markersize=7)\n axes[1].axvline(best_n, color=\"#9b59b6\", linestyle=\"--\", linewidth=1.5,\n label=f\"Best n = {best_n}\")\n axes[1].set_title(\"Log-Likelihood (higher = better)\", fontsize=11, fontweight=\"bold\")\n axes[1].set_xlabel(\"Number of Topics\")\n axes[1].set_ylabel(\"Log-Likelihood\")\n axes[1].legend(fontsize=9)\n sns.despine(ax=axes[1])\n\n plt.tight_layout()\n plt.savefig(\"lda_tuning.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()\n\n lda = LatentDirichletAllocation(\n n_components=best_n,\n random_state=random_state,\n max_iter=20,\n learning_method=\"online\"\n )\n lda.fit(dtm)\n doc_topics = lda.transform(dtm)\n\n dominant_topics = doc_topics.argmax(axis=1)\n topic_counts = Counter(dominant_topics)\n\n display(HTML(f\"\"\"\n
\n
\n
{best_n}
\n
Topics Found
\n
\n
\n
{len(corpus):,}
\n
Documents
\n
\n
\n
{len(vocab):,}
\n
Vocab Size
\n
\n
\n
{topic_counts[max(topic_counts, key=topic_counts.get)]:,}
\n
Largest Topic
\n
\n
\n
{round(min(perplexities), 1)}
\n
Best Perplexity
\n
\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n 📊 Top Words per Topic\n
\n \"\"\"))\n\n n_top_words = 10\n topic_colors = plt.cm.RdPu(np.linspace(0.3, 0.9, best_n))\n\n ncols = min(best_n, 3)\n nrows = int(np.ceil(best_n / ncols))\n\n fig, axes = plt.subplots(nrows, ncols, figsize=(7 * ncols, 5 * nrows))\n axes = np.array(axes).flatten()\n fig.suptitle(f\"LDA Topic Word Distributions (n={best_n} topics)\",\n fontsize=14, fontweight=\"bold\", y=1.01, color=\"#2d1b69\")\n\n for i, (topic, ax) in enumerate(zip(lda.components_, axes)):\n top_idx = topic.argsort()[::-1][:n_top_words]\n top_words = [vocab[j] for j in top_idx]\n top_probs = [topic[j] for j in top_idx]\n\n ax.barh(top_words[::-1], top_probs[::-1],\n color=topic_colors[i], edgecolor=\"white\", alpha=0.85)\n ax.set_title(f\"Topic {i + 1}\", fontsize=11, fontweight=\"bold\")\n ax.set_xlabel(\"Word Weight\")\n sns.despine(ax=ax)\n\n for j in range(best_n, len(axes)):\n axes[j].set_visible(False)\n\n plt.tight_layout()\n plt.savefig(\"lda_topics.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()\n\n display(HTML(\"\"\"\n
\n 📊 Document-Topic Distribution\n
\n \"\"\"))\n\n fig, axes = plt.subplots(1, 2, figsize=(16, 5))\n fig.suptitle(\"Document-Topic Distribution\",\n fontsize=13, fontweight=\"bold\", y=1.02, color=\"#2d1b69\")\n\n dom_counts = [topic_counts.get(i, 0) for i in range(best_n)]\n axes[0].bar([f\"Topic {i+1}\" for i in range(best_n)], dom_counts,\n color=topic_colors, edgecolor=\"white\", alpha=0.85)\n for idx, count in enumerate(dom_counts):\n axes[0].text(idx, count + max(dom_counts) * 0.01,\n str(count), ha=\"center\", fontsize=9, fontweight=\"bold\")\n axes[0].set_title(\"Documents per Dominant Topic\", fontsize=11, fontweight=\"bold\")\n axes[0].set_ylabel(\"Document Count\")\n axes[0].tick_params(axis=\"x\", rotation=30)\n sns.despine(ax=axes[0])\n\n mean_weights = doc_topics.mean(axis=0).reshape(1, -1)\n sns.heatmap(\n mean_weights,\n ax=axes[1],\n annot=True,\n fmt=\".3f\",\n cmap=\"RdPu\",\n xticklabels=[f\"Topic {i+1}\" for i in range(best_n)],\n yticklabels=[\"Mean Weight\"],\n linewidths=0.5,\n cbar=False\n )\n axes[1].set_title(\"Mean Topic Weight Across All Documents\", fontsize=11, fontweight=\"bold\")\n\n plt.tight_layout()\n plt.savefig(\"lda_doc_topics.png\", dpi=150, bbox_inches=\"tight\")\n plt.show()\n\n return lda, doc_topics, count_vectorizer","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:31:23.249951Z","iopub.execute_input":"2026-02-18T11:31:23.250508Z","iopub.status.idle":"2026-02-18T11:31:23.280283Z","shell.execute_reply.started":"2026-02-18T11:31:23.250467Z","shell.execute_reply":"2026-02-18T11:31:23.279164Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lda_model, doc_topics, cv = plot_lda_topics(df, col=\"translation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:31:29.449432Z","iopub.execute_input":"2026-02-18T11:31:29.449776Z","iopub.status.idle":"2026-02-18T11:32:23.166298Z","shell.execute_reply.started":"2026-02-18T11:31:29.449747Z","shell.execute_reply":"2026-02-18T11:32:23.165048Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_lda_interactive(lda_model, dtm, vectorizer, col_name=\"English Translation\"):\n\n display(HTML(f\"\"\"\n
\n

🌐 Interactive LDA Visualisation — {col_name}

\n
\n \"\"\"))\n\n display(HTML(\"\"\"\n
\n ⚠ī¸ HOW TO READ THIS
\n Each bubble is a topic — bubble size reflects how prevalent the topic is across the corpus.\n Overlapping bubbles indicate shared vocabulary between topics.\n Select a topic to see its most relevant terms on the right, and adjust the\n Îģ (lambda) slider to balance term frequency vs. topic exclusivity\n (Îģ = 1 ranks by frequency, Îģ = 0 ranks by exclusivity to that topic).\n
\n \"\"\"))\n\n panel = pyLDAvis.lda_model.prepare(\n lda_model,\n dtm,\n vectorizer,\n mds=\"tsne\",\n sort_topics=False\n )\n\n pyLDAvis.enable_notebook()\n display(panel)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:57:23.613164Z","iopub.execute_input":"2026-02-18T11:57:23.613520Z","iopub.status.idle":"2026-02-18T11:57:23.620124Z","shell.execute_reply.started":"2026-02-18T11:57:23.613492Z","shell.execute_reply":"2026-02-18T11:57:23.619026Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_lda_interactive(lda_model, dtm, cv)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-18T11:57:23.939107Z","iopub.execute_input":"2026-02-18T11:57:23.939428Z","iopub.status.idle":"2026-02-18T11:57:24.748002Z","shell.execute_reply.started":"2026-02-18T11:57:23.939403Z","shell.execute_reply":"2026-02-18T11:57:24.747173Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n\n
\n
\n\n
\n\n
\n
🙏
\n

\n THANK YOU!\n

\n
\n

\n I hope this analysis was helpful. If you found it useful, an upvote means a lot!\n

\n\n\n
\n 👍 Upvote if you liked it!\n
\n\n
\n
","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}