{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.12.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":3004},{"sourceType":"datasetVersion","sourceId":1934358}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":178.179645,"end_time":"2026-08-26T10:00:34.651621+00:00","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2026-08-26T09:57:36.471976+00:00","version":"2.7.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"0fd75440","cell_type":"markdown","source":"
\n
\n ML Series ยท Episode 39\n
\n

๐Ÿ“Š Confusion Matrix & Classification Metrics

\n

Why accuracy alone lies to you โ€” mastering Precision, Recall, F1 Score, and the Confusion Matrix to build classifiers you can actually trust.

\n
\n
\n
2 Datasets
\n
Heart Disease + MNIST
\n
\n
\n
4 Metrics
\n
Accuracy ยท Precision ยท Recall ยท F1
\n
\n
\n
Binary + Multi
\n
Both classification types
\n
\n
\n
","metadata":{"papermill":{"duration":0.006108,"end_time":"2026-08-26T09:57:39.309143+00:00","exception":false,"start_time":"2026-08-26T09:57:39.303035+00:00","status":"completed"},"tags":[]}},{"id":"ab5ec288","cell_type":"markdown","source":"
\n

๐Ÿ“‹ Table of Contents

\n
    \n
  1. Imports & Setup
    2. \n
  2. Dataset โ€” Heart Disease (Binary Classification)
    3. \n
  3. Training โ€” Logistic Regression vs Decision Tree
    4. \n
  4. Accuracy Score โ€” And Why It Can Mislead
    5. \n
  5. Confusion Matrix โ€” Reading TP, FP, FN, TN
    6. \n
  6. Type 1 & Type 2 Errors
    7. \n
  7. Precision, Recall & F1 Score
    8. \n
  8. Averaging Strategies โ€” None, Macro, Weighted
    9. \n
  9. Multi-class Classification โ€” MNIST Digits
    10. \n
  10. Classification Report
    11. \n
  11. Summary & When to Use Each Metric
    12. \n
\n
","metadata":{"papermill":{"duration":0.004498,"end_time":"2026-08-26T09:57:39.318565+00:00","exception":false,"start_time":"2026-08-26T09:57:39.314067+00:00","status":"completed"},"tags":[]}},{"id":"f196e811","cell_type":"markdown","source":"\n
\n Section 1\n

Imports & Setup

\n
","metadata":{"papermill":{"duration":0.004464,"end_time":"2026-08-26T09:57:39.327633+00:00","exception":false,"start_time":"2026-08-26T09:57:39.323169+00:00","status":"completed"},"tags":[]}},{"id":"01beec0f","cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import (\n accuracy_score, confusion_matrix,\n precision_score, recall_score,\n f1_score, classification_report,\n ConfusionMatrixDisplay\n)","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:23.107355Z","iopub.execute_input":"2026-08-27T14:22:23.107760Z","iopub.status.idle":"2026-08-27T14:22:25.146011Z","shell.execute_reply.started":"2026-08-27T14:22:23.107725Z","shell.execute_reply":"2026-08-27T14:22:25.145172Z"},"papermill":{"duration":3.580255,"end_time":"2026-08-26T09:57:42.912468+00:00","exception":false,"start_time":"2026-08-26T09:57:39.332213+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"f867bdcf","cell_type":"markdown","source":"\n
\n Section 2\n

Dataset โ€” Heart Disease (Binary Classification)

\n
\n\nThe **UCI Heart Disease dataset** contains 303 patients with 13 clinical features. The target is binary โ€” `1` means heart disease present, `0` means absent. This is a real medical classification problem where **the cost of errors is asymmetric** โ€” missing a sick patient (false negative) is far more dangerous than a false alarm (false positive).\n\n
\n
\n
303
patients
\n
13
features
\n
2
classes (binary)
\n
Medical
real-world stakes
\n
\n
","metadata":{"papermill":{"duration":0.004797,"end_time":"2026-08-26T09:57:42.922294+00:00","exception":false,"start_time":"2026-08-26T09:57:42.917497+00:00","status":"completed"},"tags":[]}},{"id":"f48be66b","cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/datasets/mragpavank/heart-diseaseuci/heart.csv\")\n\nprint(f\"Shape : {df.shape}\")\nprint(f\"Nulls : {df.isnull().sum().sum()}\")\nprint(f\"\\nTarget distribution:\")\nprint(df['target'].value_counts().rename({0: 'No Disease', 1: 'Disease'}).to_string())\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.147780Z","iopub.execute_input":"2026-08-27T14:22:25.148259Z","iopub.status.idle":"2026-08-27T14:22:25.221332Z","shell.execute_reply.started":"2026-08-27T14:22:25.148227Z","shell.execute_reply":"2026-08-27T14:22:25.220428Z"},"papermill":{"duration":0.069664,"end_time":"2026-08-26T09:57:42.996752+00:00","exception":false,"start_time":"2026-08-26T09:57:42.927088+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"3f970955","cell_type":"code","source":"X = df.drop(columns=['target'])\ny = df['target']\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nprint(f\"Train : {X_train.shape} | Test : {X_test.shape}\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.222288Z","iopub.execute_input":"2026-08-27T14:22:25.222614Z","iopub.status.idle":"2026-08-27T14:22:25.233981Z","shell.execute_reply.started":"2026-08-27T14:22:25.222582Z","shell.execute_reply":"2026-08-27T14:22:25.233117Z"},"papermill":{"duration":0.019603,"end_time":"2026-08-26T09:57:43.021563+00:00","exception":false,"start_time":"2026-08-26T09:57:43.00196+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"892f4c82","cell_type":"markdown","source":"\n
\n Section 3\n

Training โ€” Logistic Regression vs Decision Tree

\n
\n\nWe train two classifiers to compare their performance across multiple metrics. Comparing models on a single metric (accuracy) is misleading โ€” we need the full picture.","metadata":{"papermill":{"duration":0.005119,"end_time":"2026-08-26T09:57:43.032903+00:00","exception":false,"start_time":"2026-08-26T09:57:43.027784+00:00","status":"completed"},"tags":[]}},{"id":"1593079a","cell_type":"code","source":"clf1 = LogisticRegression(max_iter=1000)\nclf2 = DecisionTreeClassifier(random_state=42)\n\nclf1.fit(X_train, y_train)\nclf2.fit(X_train, y_train)\n\ny_pred1 = clf1.predict(X_test) # Logistic Regression predictions\ny_pred2 = clf2.predict(X_test) # Decision Tree predictions\n\nprint(\"Both models trained successfully.\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.235859Z","iopub.execute_input":"2026-08-27T14:22:25.236204Z","iopub.status.idle":"2026-08-27T14:22:25.422981Z","shell.execute_reply.started":"2026-08-27T14:22:25.236172Z","shell.execute_reply":"2026-08-27T14:22:25.422226Z"},"papermill":{"duration":0.186075,"end_time":"2026-08-26T09:57:43.224442+00:00","exception":false,"start_time":"2026-08-26T09:57:43.038367+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"edb93314","cell_type":"markdown","source":"\n
\n Section 4\n

Accuracy Score โ€” And Why It Can Mislead

\n
\n\n**Accuracy** = (Correct predictions) / (Total predictions). It's the most intuitive metric โ€” and the most dangerous one to use alone.\n\n
\n
When Accuracy Fails โ€” Classic Example
\n
\n Imagine a cancer screening model on 1000 patients where only 10 have cancer.
\n A model that always predicts No Cancer gets 99% accuracy โ€” but it's completely useless.
\n It misses all 10 cancer patients. Accuracy hid a catastrophic failure.\n
\n
\n\n
\n
\n
โš ๏ธ Imbalanced classes
\n
Rare events (fraud, disease) make accuracy trivially high by always predicting the majority class
\n
\n
\n
โš ๏ธ Unequal error costs
\n
Missing a sick patient costs more than a false alarm โ€” accuracy treats both errors identically
\n
\n
\n
โœ… Solution
\n
Use Confusion Matrix, Precision, Recall, and F1 to understand which errors your model makes
\n
\n
","metadata":{"papermill":{"duration":0.008299,"end_time":"2026-08-26T09:57:43.241433+00:00","exception":false,"start_time":"2026-08-26T09:57:43.233134+00:00","status":"completed"},"tags":[]}},{"id":"92eb5a56","cell_type":"code","source":"acc1 = accuracy_score(y_test, y_pred1)\nacc2 = accuracy_score(y_test, y_pred2)\n\nprint(f\"Logistic Regression accuracy : {acc1:.4f} ({acc1*100:.1f}%)\")\nprint(f\"Decision Tree accuracy : {acc2:.4f} ({acc2*100:.1f}%)\")\nprint()\nprint(\"๐Ÿ‘† Logistic Regression looks clearly better โ€” but let's dig deeper.\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.424193Z","iopub.execute_input":"2026-08-27T14:22:25.424507Z","iopub.status.idle":"2026-08-27T14:22:25.434929Z","shell.execute_reply.started":"2026-08-27T14:22:25.424457Z","shell.execute_reply":"2026-08-27T14:22:25.433528Z"},"papermill":{"duration":0.02454,"end_time":"2026-08-26T09:57:43.274601+00:00","exception":false,"start_time":"2026-08-26T09:57:43.250061+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"0c599eb7","cell_type":"markdown","source":"\n
\n Section 5\n

Confusion Matrix โ€” Reading TP, FP, FN, TN

\n
\n\nA confusion matrix shows the **full breakdown of every prediction** โ€” not just whether the model was right or wrong, but which type of right and which type of wrong.\n\n
\n
Confusion Matrix โ€” 2ร—2 Layout (Binary)
\n
\n
\n
TP
\n
True Positive
Predicted 1, Actually 1 โœ…
\n
\n
\n
FP
\n
False Positive
Predicted 1, Actually 0 โŒ
Type I Error
\n
\n
\n
FN
\n
False Negative
Predicted 0, Actually 1 โŒ
Type II Error
\n
\n
\n
TN
\n
True Negative
Predicted 0, Actually 0 โœ…
\n
\n
\n
Layout: rows = actual labels, columns = predicted labels
\n
","metadata":{"papermill":{"duration":0.008318,"end_time":"2026-08-26T09:57:43.29187+00:00","exception":false,"start_time":"2026-08-26T09:57:43.283552+00:00","status":"completed"},"tags":[]}},{"id":"5b77fc60","cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(13, 5))\nfig.suptitle('Confusion Matrices โ€” Logistic Regression vs Decision Tree',\n fontsize=13, fontweight='bold')\n\nfor ax, y_pred, title in zip(\n axes,\n [y_pred1, y_pred2],\n ['Logistic Regression', 'Decision Tree']\n):\n cm = confusion_matrix(y_test, y_pred)\n disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=['No Disease (0)', 'Disease (1)'])\n disp.plot(ax=ax, colorbar=False, cmap='Purples')\n ax.set_title(f'{title}\\nAccuracy: {accuracy_score(y_test, y_pred):.3f}',\n fontsize=11, fontweight='bold')\n ax.set_xlabel('Predicted Label', fontsize=10)\n ax.set_ylabel('True Label', fontsize=10)\n\nplt.tight_layout()\nplt.show()\n\n# Also print as DataFrame for reference\nprint(\"Logistic Regression Confusion Matrix:\")\nprint(pd.DataFrame(confusion_matrix(y_test, y_pred1),\n index=['Actual 0', 'Actual 1'],\n columns=['Predicted 0', 'Predicted 1']), \"\\n\")\n\nprint(\"Decision Tree Confusion Matrix:\")\nprint(pd.DataFrame(confusion_matrix(y_test, y_pred2),\n index=['Actual 0', 'Actual 1'],\n columns=['Predicted 0', 'Predicted 1']))","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.435833Z","iopub.execute_input":"2026-08-27T14:22:25.436132Z","iopub.status.idle":"2026-08-27T14:22:25.759262Z","shell.execute_reply.started":"2026-08-27T14:22:25.436102Z","shell.execute_reply":"2026-08-27T14:22:25.758280Z"},"papermill":{"duration":0.323858,"end_time":"2026-08-26T09:57:43.624351+00:00","exception":false,"start_time":"2026-08-26T09:57:43.300493+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"6f9a6562","cell_type":"code","source":"# Side-by-side prediction sample\nresult = pd.DataFrame({\n 'Actual': y_test.values,\n 'Logistic Regression': y_pred1,\n 'Decision Tree': y_pred2\n}, index=y_test.index)\n\nresult['LR Correct?'] = result['Actual'] == result['Logistic Regression']\nresult['DT Correct?'] = result['Actual'] == result['Decision Tree']\n\nresult.sample(10, random_state=1)","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.760452Z","iopub.execute_input":"2026-08-27T14:22:25.760799Z","iopub.status.idle":"2026-08-27T14:22:25.774932Z","shell.execute_reply.started":"2026-08-27T14:22:25.760772Z","shell.execute_reply":"2026-08-27T14:22:25.773875Z"},"papermill":{"duration":0.023882,"end_time":"2026-08-26T09:57:43.654559+00:00","exception":false,"start_time":"2026-08-26T09:57:43.630677+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"c2006a5b","cell_type":"markdown","source":"\n
\n Section 6\n

Type I & Type II Errors โ€” Why Error Type Matters

\n
\n\nNot all mistakes are equal. The confusion matrix reveals **which kind of errors** your model makes โ€” and in medical, legal, or safety contexts, the two types have very different consequences.\n\n
\n
\n
๐Ÿ”ด Type I Error โ€” False Positive (FP)
\n
\n Model predicts positive when the truth is negative.

\n Heart disease: Telling a healthy patient they have heart disease โ†’ unnecessary stress, tests, treatment

\n Spam filter: Marking a legitimate email as spam โ†’ missing an important message

\n Metric that catches this: Precision\n
\n
\n
\n
๐ŸŸก Type II Error โ€” False Negative (FN)
\n
\n Model predicts negative when the truth is positive.

\n Heart disease: Missing a sick patient who actually has heart disease โ†’ delayed treatment, death risk

\n Fraud detection: Flagging a real fraud as legitimate โ†’ financial loss

\n Metric that catches this: Recall\n
\n
\n
\n\n
\n ๐Ÿ’ก In our heart disease problem: A False Negative (missing a sick patient) is far more dangerous than a False Positive (extra tests for a healthy patient). This means Recall is more important than Precision here โ€” and accuracy alone hides this entirely.\n
","metadata":{"papermill":{"duration":0.006125,"end_time":"2026-08-26T09:57:43.667159+00:00","exception":false,"start_time":"2026-08-26T09:57:43.661034+00:00","status":"completed"},"tags":[]}},{"id":"c63ce604","cell_type":"markdown","source":"\n
\n Section 7\n

Precision, Recall & F1 Score

\n
\n\nEach metric answers a different question about your model's behaviour:\n\n
\n
\n
\n
\n
Precision
\n
\n
\n
TP / (TP + FP)
\n
\"Of all the patients I predicted have disease, what fraction actually do?\" โ€” measures quality of positive predictions
\n
\n
\n
\n
\n
Recall
\n
\n
\n
TP / (TP + FN)
\n
\"Of all patients who actually have disease, how many did I catch?\" โ€” measures coverage of actual positives
\n
\n
\n
\n
\n
F1 Score
\n
\n
\n
2 ยท (Precision ยท Recall) / (Precision + Recall)
\n
Harmonic mean of Precision and Recall โ€” balances both. Use when you care about both false positives and false negatives
\n
\n
\n
\n
","metadata":{"papermill":{"duration":0.006341,"end_time":"2026-08-26T09:57:43.680837+00:00","exception":false,"start_time":"2026-08-26T09:57:43.674496+00:00","status":"completed"},"tags":[]}},{"id":"920e6c65","cell_type":"code","source":"# โ”€โ”€ Logistic Regression metrics โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\nprint(\"=\" * 50)\nprint(\" LOGISTIC REGRESSION\")\nprint(\"=\" * 50)\ncm1 = confusion_matrix(y_test, y_pred1)\ntn, fp, fn, tp = cm1.ravel()\nprint(f\" TP={tp} FP={fp} FN={fn} TN={tn}\")\nprint(f\" Accuracy : {accuracy_score(y_test, y_pred1):.4f}\")\nprint(f\" Precision : {precision_score(y_test, y_pred1):.4f}\")\nprint(f\" Recall : {recall_score(y_test, y_pred1):.4f}\")\nprint(f\" F1 Score : {f1_score(y_test, y_pred1):.4f}\")\n\nprint()\n\n# โ”€โ”€ Decision Tree metrics โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\nprint(\"=\" * 50)\nprint(\" DECISION TREE\")\nprint(\"=\" * 50)\ncm2 = confusion_matrix(y_test, y_pred2)\ntn2, fp2, fn2, tp2 = cm2.ravel()\nprint(f\" TP={tp2} FP={fp2} FN={fn2} TN={tn2}\")\nprint(f\" Accuracy : {accuracy_score(y_test, y_pred2):.4f}\")\nprint(f\" Precision : {precision_score(y_test, y_pred2):.4f}\")\nprint(f\" Recall : {recall_score(y_test, y_pred2):.4f}\")\nprint(f\" F1 Score : {f1_score(y_test, y_pred2):.4f}\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.776220Z","iopub.execute_input":"2026-08-27T14:22:25.776577Z","iopub.status.idle":"2026-08-27T14:22:25.815066Z","shell.execute_reply.started":"2026-08-27T14:22:25.776540Z","shell.execute_reply":"2026-08-27T14:22:25.814032Z"},"papermill":{"duration":0.042522,"end_time":"2026-08-26T09:57:43.729919+00:00","exception":false,"start_time":"2026-08-26T09:57:43.687397+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"55683a92","cell_type":"code","source":"# Visual comparison โ€” radar / bar chart of all metrics\nmetrics = ['Accuracy', 'Precision', 'Recall', 'F1 Score']\nlr_scores = [\n accuracy_score(y_test, y_pred1),\n precision_score(y_test, y_pred1),\n recall_score(y_test, y_pred1),\n f1_score(y_test, y_pred1)\n]\ndt_scores = [\n accuracy_score(y_test, y_pred2),\n precision_score(y_test, y_pred2),\n recall_score(y_test, y_pred2),\n f1_score(y_test, y_pred2)\n]\n\nx = np.arange(len(metrics))\nwidth = 0.35\n\nfig, ax = plt.subplots(figsize=(10, 5))\nbars1 = ax.bar(x - width/2, lr_scores, width, label='Logistic Regression',\n color='#7c3aed', alpha=0.85)\nbars2 = ax.bar(x + width/2, dt_scores, width, label='Decision Tree',\n color='#a78bfa', alpha=0.85)\n\nax.set_xticks(x)\nax.set_xticklabels(metrics, fontsize=11)\nax.set_ylabel('Score', fontsize=11)\nax.set_ylim(0, 1.1)\nax.set_title('All Metrics โ€” Logistic Regression vs Decision Tree', fontsize=13, fontweight='bold')\nax.legend(fontsize=10)\nax.bar_label(bars1, fmt='%.3f', fontsize=9, padding=2)\nax.bar_label(bars2, fmt='%.3f', fontsize=9, padding=2)\nax.grid(axis='y', alpha=0.3)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:25.816273Z","iopub.execute_input":"2026-08-27T14:22:25.816653Z","iopub.status.idle":"2026-08-27T14:22:26.059158Z","shell.execute_reply.started":"2026-08-27T14:22:25.816615Z","shell.execute_reply":"2026-08-27T14:22:26.058280Z"},"papermill":{"duration":0.261539,"end_time":"2026-08-26T09:57:43.998039+00:00","exception":false,"start_time":"2026-08-26T09:57:43.7365+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"1d72033e","cell_type":"markdown","source":"
\n โœ… Key takeaway: Logistic Regression wins on every metric โ€” especially Recall (0.927). This means it catches more actual heart disease cases (fewer dangerous misses). The Decision Tree's Recall is lower, meaning it misses more sick patients โ€” much riskier in a medical context.\n
","metadata":{"papermill":{"duration":0.006934,"end_time":"2026-08-26T09:57:44.012449+00:00","exception":false,"start_time":"2026-08-26T09:57:44.005515+00:00","status":"completed"},"tags":[]}},{"id":"b51ce77b","cell_type":"markdown","source":"\n
\n Section 8\n

Averaging Strategies โ€” None, Macro & Weighted

\n
\n\nWhen there are multiple classes, sklearn needs to know how to combine per-class scores into a single number. There are three strategies:\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
StrategyWhat it doesUse when
NoneReturns score for each class separately (array)You need per-class breakdown
macroSimple average โ€” treats all classes equallyBalanced classes, each class equally important
weightedAverage weighted by class support (sample count)Imbalanced classes, larger classes matter more
\n
","metadata":{"papermill":{"duration":0.006813,"end_time":"2026-08-26T09:57:44.026348+00:00","exception":false,"start_time":"2026-08-26T09:57:44.019535+00:00","status":"completed"},"tags":[]}},{"id":"567a81b6","cell_type":"code","source":"print(\"โ”€โ”€ Per-class Precision (average=None) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\")\nprint(f\" Logistic Regression : {precision_score(y_test, y_pred1, average=None).round(4)}\")\nprint(f\" Decision Tree : {precision_score(y_test, y_pred2, average=None).round(4)}\")\n\nprint(\"\\nโ”€โ”€ Per-class Recall (average=None) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\")\nprint(f\" Logistic Regression : {recall_score(y_test, y_pred1, average=None).round(4)}\")\nprint(f\" Decision Tree : {recall_score(y_test, y_pred2, average=None).round(4)}\")\n\nprint(\"\\nโ”€โ”€ Macro Average (equal class weight) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\")\nprint(f\" Precision LR : {precision_score(y_test, y_pred1, average='macro'):.4f}\")\nprint(f\" Precision DT : {precision_score(y_test, y_pred2, average='macro'):.4f}\")\nprint(f\" Recall LR : {recall_score(y_test, y_pred1, average='macro'):.4f}\")\nprint(f\" Recall DT : {recall_score(y_test, y_pred2, average='macro'):.4f}\")\n\nprint(\"\\nโ”€โ”€ Weighted Average (weighted by class size) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\")\nprint(f\" Precision LR : {precision_score(y_test, y_pred1, average='weighted'):.4f}\")\nprint(f\" Precision DT : {precision_score(y_test, y_pred2, average='weighted'):.4f}\")\nprint(f\" Recall LR : {recall_score(y_test, y_pred1, average='weighted'):.4f}\")\nprint(f\" Recall DT : {recall_score(y_test, y_pred2, average='weighted'):.4f}\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:26.061370Z","iopub.execute_input":"2026-08-27T14:22:26.061666Z","iopub.status.idle":"2026-08-27T14:22:26.106555Z","shell.execute_reply.started":"2026-08-27T14:22:26.061639Z","shell.execute_reply":"2026-08-27T14:22:26.105693Z"},"papermill":{"duration":0.054884,"end_time":"2026-08-26T09:57:44.088317+00:00","exception":false,"start_time":"2026-08-26T09:57:44.033433+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"40bce5a4","cell_type":"markdown","source":"\n
\n Section 9\n

Multi-class Classification โ€” MNIST Handwritten Digits

\n
\n\nEverything we've discussed for binary classification extends naturally to **10 classes**. The confusion matrix grows to 10ร—10 โ€” each row represents an actual digit, each column a predicted digit. Off-diagonal cells reveal which digits are most commonly confused with each other.\n\n
\n โš ๏ธ Convergence warning fix: The original code gets a ConvergenceWarning because LogisticRegression hits the default max_iter=100 limit on 784-dimensional image data. We fix this with max_iter=1000.\n
","metadata":{"papermill":{"duration":0.007018,"end_time":"2026-08-26T09:57:44.102571+00:00","exception":false,"start_time":"2026-08-26T09:57:44.095553+00:00","status":"completed"},"tags":[]}},{"id":"b1bc465b","cell_type":"code","source":"df1 = pd.read_csv('/kaggle/input/competitions/digit-recognizer/train.csv')\n\nprint(f\"Shape : {df1.shape}\")\nprint(f\"Labels : {sorted(df1['label'].unique())}\")\nprint(f\"\\nClass distribution:\")\nprint(df1['label'].value_counts().sort_index().to_string())\n\ndf1.head(3)","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:26.107612Z","iopub.execute_input":"2026-08-27T14:22:26.107935Z","iopub.status.idle":"2026-08-27T14:22:29.399311Z","shell.execute_reply.started":"2026-08-27T14:22:26.107908Z","shell.execute_reply":"2026-08-27T14:22:29.398531Z"},"papermill":{"duration":2.819688,"end_time":"2026-08-26T09:57:46.929346+00:00","exception":false,"start_time":"2026-08-26T09:57:44.109658+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"10081615","cell_type":"code","source":"X1 = df1.drop(columns=['label'])\ny1 = df1['label']\n\nX_train1, X_test1, y_train1, y_test1 = train_test_split(\n X1, y1, test_size=0.25, random_state=2\n)\n\nprint(f\"Train : {X_train1.shape} | Test : {X_test1.shape}\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:29.400355Z","iopub.execute_input":"2026-08-27T14:22:29.401082Z","iopub.status.idle":"2026-08-27T14:22:29.734269Z","shell.execute_reply.started":"2026-08-27T14:22:29.401050Z","shell.execute_reply":"2026-08-27T14:22:29.733491Z"},"papermill":{"duration":0.271245,"end_time":"2026-08-26T09:57:47.20843+00:00","exception":false,"start_time":"2026-08-26T09:57:46.937185+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"d5a3527e","cell_type":"code","source":"# Fix: max_iter=1000 prevents ConvergenceWarning on high-dim pixel data\nclf3 = LogisticRegression(max_iter=1000, random_state=42)\nclf4 = DecisionTreeClassifier(random_state=42)\n\nprint(\"Training Logistic Regression on 784 pixel features...\")\nclf3.fit(X_train1, y_train1)\nprint(\"Training Decision Tree...\")\nclf4.fit(X_train1, y_train1)\n\ny_pred3 = clf3.predict(X_test1)\ny_pred4 = clf4.predict(X_test1)\n\nprint(f\"\\nLogistic Regression accuracy : {accuracy_score(y_test1, y_pred3):.4f}\")\nprint(f\"Decision Tree accuracy : {accuracy_score(y_test1, y_pred4):.4f}\")","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:22:29.735321Z","iopub.execute_input":"2026-08-27T14:22:29.735653Z","iopub.status.idle":"2026-08-27T14:25:12.885482Z","shell.execute_reply.started":"2026-08-27T14:22:29.735623Z","shell.execute_reply":"2026-08-27T14:25:12.884500Z"},"papermill":{"duration":165.280929,"end_time":"2026-08-26T10:00:32.496846+00:00","exception":false,"start_time":"2026-08-26T09:57:47.215917+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"64a63204","cell_type":"code","source":"# Heatmap confusion matrices for both models\nfig, axes = plt.subplots(1, 2, figsize=(16, 7))\nfig.suptitle('10ร—10 Confusion Matrices โ€” MNIST Digits', fontsize=14, fontweight='bold')\n\nfor ax, y_pred, title, acc in zip(\n axes,\n [y_pred3, y_pred4],\n ['Logistic Regression', 'Decision Tree'],\n [accuracy_score(y_test1, y_pred3), accuracy_score(y_test1, y_pred4)]\n):\n cm = confusion_matrix(y_test1, y_pred)\n sns.heatmap(cm, annot=True, fmt='d', cmap='Purples', ax=ax,\n xticklabels=range(10), yticklabels=range(10),\n linewidths=0.5, linecolor='#1e1040')\n ax.set_title(f'{title} โ€” Accuracy: {acc:.3f}', fontsize=12, fontweight='bold')\n ax.set_xlabel('Predicted Digit', fontsize=10)\n ax.set_ylabel('True Digit', fontsize=10)\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:25:12.886659Z","iopub.execute_input":"2026-08-27T14:25:12.886994Z","iopub.status.idle":"2026-08-27T14:25:13.760267Z","shell.execute_reply.started":"2026-08-27T14:25:12.886953Z","shell.execute_reply":"2026-08-27T14:25:13.759391Z"},"papermill":{"duration":0.924739,"end_time":"2026-08-26T10:00:33.434219+00:00","exception":false,"start_time":"2026-08-26T10:00:32.50948+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"c0c0c1a8","cell_type":"markdown","source":"
\n ๐Ÿ” Reading the heatmap: Dark diagonal cells = correct predictions. Any bright off-diagonal cell is a common confusion. For example, 8 vs 9 and 4 vs 9 are typical digit pairs that get confused because they look similar visually.\n
","metadata":{"papermill":{"duration":0.010179,"end_time":"2026-08-26T10:00:33.455093+00:00","exception":false,"start_time":"2026-08-26T10:00:33.444914+00:00","status":"completed"},"tags":[]}},{"id":"693d35f5","cell_type":"markdown","source":"\n
\n Section 10\n

Classification Report โ€” All Metrics at Once

\n
\n\n`classification_report` prints Precision, Recall, and F1 for every class simultaneously โ€” the fastest way to get a complete picture of model performance.\n\n
\n ๐Ÿ“Œ Reading the report:\n \n
","metadata":{"papermill":{"duration":0.0105,"end_time":"2026-08-26T10:00:33.476029+00:00","exception":false,"start_time":"2026-08-26T10:00:33.465529+00:00","status":"completed"},"tags":[]}},{"id":"56b7b28f","cell_type":"code","source":"print(\"=\" * 60)\nprint(\" LOGISTIC REGRESSION โ€” Classification Report (MNIST)\")\nprint(\"=\" * 60)\nprint(classification_report(y_test1, y_pred3))\n\nprint(\"=\" * 60)\nprint(\" DECISION TREE โ€” Classification Report (MNIST)\")\nprint(\"=\" * 60)\nprint(classification_report(y_test1, y_pred4))","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:25:13.761428Z","iopub.execute_input":"2026-08-27T14:25:13.762189Z","iopub.status.idle":"2026-08-27T14:25:13.792123Z","shell.execute_reply.started":"2026-08-27T14:25:13.762155Z","shell.execute_reply":"2026-08-27T14:25:13.791164Z"},"papermill":{"duration":0.045992,"end_time":"2026-08-26T10:00:33.532621+00:00","exception":false,"start_time":"2026-08-26T10:00:33.486629+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"d0bf6964","cell_type":"code","source":"# Per-digit F1 bar chart โ€” which digits are hardest to classify?\nfrom sklearn.metrics import f1_score\n\nf1_lr = f1_score(y_test1, y_pred3, average=None)\nf1_dt = f1_score(y_test1, y_pred4, average=None)\n\nx = np.arange(10)\nwidth = 0.35\n\nfig, ax = plt.subplots(figsize=(12, 5))\nbars1 = ax.bar(x - width/2, f1_lr, width, label='Logistic Regression', color='#7c3aed', alpha=0.85)\nbars2 = ax.bar(x + width/2, f1_dt, width, label='Decision Tree', color='#a78bfa', alpha=0.85)\n\nax.set_xticks(x)\nax.set_xticklabels([f'Digit {i}' for i in range(10)], fontsize=10)\nax.set_ylabel('F1 Score', fontsize=11)\nax.set_ylim(0.6, 1.05)\nax.set_title('Per-Digit F1 Score โ€” Which Digits Are Hardest to Classify?',\n fontsize=13, fontweight='bold')\nax.legend(fontsize=10)\nax.bar_label(bars1, fmt='%.2f', fontsize=8, padding=2)\nax.bar_label(bars2, fmt='%.2f', fontsize=8, padding=2)\nax.grid(axis='y', alpha=0.3)\nax.axhline(f1_lr.mean(), color='#7c3aed', linestyle='--', linewidth=1, alpha=0.5, label='LR mean')\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2026-08-27T14:25:13.793328Z","iopub.execute_input":"2026-08-27T14:25:13.793850Z","iopub.status.idle":"2026-08-27T14:25:14.125252Z","shell.execute_reply.started":"2026-08-27T14:25:13.793812Z","shell.execute_reply":"2026-08-27T14:25:14.124394Z"},"papermill":{"duration":0.35276,"end_time":"2026-08-26T10:00:33.896188+00:00","exception":false,"start_time":"2026-08-26T10:00:33.543428+00:00","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"id":"5bdef3c1","cell_type":"markdown","source":"\n
\n

๐Ÿ“Œ Summary โ€” When to Use Each Metric

\n

Choose your metric based on what kind of error matters most in your domain.

\n
\n
\n
๐Ÿ“Š Accuracy
\n
Use only when classes are balanced and errors are equally costly. Default metric but often misleading. Never use alone on imbalanced data.
\n
\n
\n
๐ŸŽฏ Precision
\n
Use when False Positives are costly. Spam filter (don't block good emails), recommendation systems (don't recommend bad items).
\n
\n
\n
๐Ÿ” Recall
\n
Use when False Negatives are costly. Disease detection (don't miss sick patients), fraud detection (don't miss fraudulent transactions).
\n
\n
\n
โš–๏ธ F1 Score
\n
Use when you need to balance both Precision and Recall โ€” imbalanced datasets where both error types matter. Best single metric for most real-world tasks.
\n
\n
\n
\n ๐Ÿฅ This episode's domain (Heart Disease): Recall is king โ€” missing a sick patient is far worse than a false alarm. Logistic Regression's Recall of 0.927 vs Decision Tree's 0.756 is the decisive difference โ€” accuracy alone (89% vs 78%) understates how much better LR really is.\n
\n
\n Next up โ†’ ML 40: ROC Curve, AUC & Threshold Tuning โ€” visualising the Precision-Recall tradeoff across all decision thresholds.\n
\n
","metadata":{"papermill":{"duration":0.011495,"end_time":"2026-08-26T10:00:33.919182+00:00","exception":false,"start_time":"2026-08-26T10:00:33.907687+00:00","status":"completed"},"tags":[]}}]}