{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":28755,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n# 🤖 ML 27 — PCA on MNIST Using Scikit-learn\n\n### **From 784 Pixels to Principal Components**\n\n---\n\n","metadata":{}},{"cell_type":"markdown","source":"# 📌 Introduction\n\n> In this notebook, we will apply **Principal Component Analysis (PCA)** to the **MNIST dataset** using Scikit-learn. We will see how PCA can reduce the dimensionality of handwritten digit images while retaining most of the important information.\n\n# 🔢 What is MNIST?\n\n**MNIST (Modified National Institute of Standards and Technology)** is a popular dataset of handwritten digits from **0 to 9**.\n\nEach image is:\n\n* 🖼️ **28 × 28 pixels**\n* 🔢 **784 pixel features** (`28 × 28 = 784`)\n* 🏷️ Labeled from **0 to 9**\n* ⚫ Each pixel contains a grayscale intensity value\n\nA single image can therefore be represented as:\n\n```text\n28 × 28 image\n     ↓\n784 pixel features\n     ↓\n[0, 0, 12, 45,  ... , 0]\n```\n\nThis makes MNIST a great example for PCA because **784 dimensions** can be reduced to a much smaller number of components.\n\n## 🔄 PCA Workflow\n\n```text\nMNIST Images\n     ↓\n784 Pixel Features\n     ↓\nFeature Scaling\n     ↓\nApply PCA\n     ↓\nReduce Dimensions\n     ↓\nAnalyze Explained Variance\n     ↓\n2D Visualization\n```\n","metadata":{}},{"cell_type":"markdown","source":"---\n# 🛠️ Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:30.833062Z","iopub.execute_input":"2026-08-12T06:56:30.833310Z","iopub.status.idle":"2026-08-12T06:56:30.837766Z","shell.execute_reply.started":"2026-08-12T06:56:30.833291Z","shell.execute_reply":"2026-08-12T06:56:30.837222Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n# 📂 Load the MNIST Dataset\n\nWe’ll use the **MNIST Digit Recognizer dataset**, where each handwritten digit is represented as a **28 × 28 grayscale image**.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/competitions/digit-recognizer/train.csv\")\ndf","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:30.910728Z","iopub.execute_input":"2026-08-12T06:56:30.911051Z","iopub.status.idle":"2026-08-12T06:56:32.133436Z","shell.execute_reply.started":"2026-08-12T06:56:30.911024Z","shell.execute_reply":"2026-08-12T06:56:32.132820Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n# 📐 Check Dataset Shape\n","metadata":{}},{"cell_type":"code","source":"df.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.134986Z","iopub.execute_input":"2026-08-12T06:56:32.135170Z","iopub.status.idle":"2026-08-12T06:56:32.139654Z","shell.execute_reply.started":"2026-08-12T06:56:32.135154Z","shell.execute_reply":"2026-08-12T06:56:32.138999Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\nSo, we have 42,000 samples and 784 input features for each image.","metadata":{}},{"cell_type":"markdown","source":"---\n# 🔍 3. Inspect Random Samples\n\nInstead of looking only at the first few rows, we can randomly sample five observations:","metadata":{}},{"cell_type":"code","source":"df.sample(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.140381Z","iopub.execute_input":"2026-08-12T06:56:32.140630Z","iopub.status.idle":"2026-08-12T06:56:32.162509Z","shell.execute_reply.started":"2026-08-12T06:56:32.140610Z","shell.execute_reply":"2026-08-12T06:56:32.161931Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Each row represents one handwritten digit, with its label followed by 784 pixel intensity values.","metadata":{}},{"cell_type":"markdown","source":"---\n# 🖼️ Visualize a Digit\n\nThe 784 pixel values can be reshaped back into the original 28 × 28 image.","metadata":{}},{"cell_type":"code","source":"plt.imshow(df.iloc[33707, 1:].values.reshape(28, 28), cmap=\"gray\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.164194Z","iopub.execute_input":"2026-08-12T06:56:32.164452Z","iopub.status.idle":"2026-08-12T06:56:32.357454Z","shell.execute_reply.started":"2026-08-12T06:56:32.164426Z","shell.execute_reply":"2026-08-12T06:56:32.356769Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This helps us understand what the raw pixel features actually represent.","metadata":{}},{"cell_type":"markdown","source":"---\n# 🎯 Separate Features and Target\n\nThe `label` column is our target, while the 784 pixel columns are the input features.","metadata":{}},{"cell_type":"code","source":"X = df.drop(columns=[\"label\"])\ny = df.label","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.358424Z","iopub.execute_input":"2026-08-12T06:56:32.358715Z","iopub.status.idle":"2026-08-12T06:56:32.398214Z","shell.execute_reply.started":"2026-08-12T06:56:32.358684Z","shell.execute_reply":"2026-08-12T06:56:32.397377Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n# ✂️ Split the Dataset\n\nWe’ll keep **20% of the data for testing** and use the remaining **80% for training**.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.399357Z","iopub.execute_input":"2026-08-12T06:56:32.399646Z","iopub.status.idle":"2026-08-12T06:56:32.569408Z","shell.execute_reply.started":"2026-08-12T06:56:32.399617Z","shell.execute_reply":"2026-08-12T06:56:32.568845Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.570198Z","iopub.execute_input":"2026-08-12T06:56:32.570453Z","iopub.status.idle":"2026-08-12T06:56:32.575595Z","shell.execute_reply.started":"2026-08-12T06:56:32.570426Z","shell.execute_reply":"2026-08-12T06:56:32.574940Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n# 🤖 Baseline Model — KNN\n\nBefore applying PCA, let's train a **K-Nearest Neighbors (KNN)** classifier using all 784 features.","metadata":{}},{"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.metrics import accuracy_score\n\nknn = KNeighborsClassifier()\n\nknn.fit(X_train, y_train)\n\ny_pred = knn.predict(X_test)\n\naccuracy_score(y_test, y_pred)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:32.576378Z","iopub.execute_input":"2026-08-12T06:56:32.576591Z","iopub.status.idle":"2026-08-12T06:56:39.480208Z","shell.execute_reply.started":"2026-08-12T06:56:32.576572Z","shell.execute_reply":"2026-08-12T06:56:39.479784Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The baseline KNN model achieves approximately **96% accuracy**.\n\nNow let's see whether PCA can reduce the number of features while maintaining good performance.","metadata":{}},{"cell_type":"markdown","source":"---\n# ⚙️ Standardize the Features\n\nPCA is sensitive to feature scale, so we standardize the pixel values before applying it.","metadata":{}},{"cell_type":"code","source":"scale = StandardScaler()\n\nX_train = scale.fit_transform(X_train)\nX_test = scale.transform(X_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:39.480772Z","iopub.execute_input":"2026-08-12T06:56:39.480954Z","iopub.status.idle":"2026-08-12T06:56:39.732498Z","shell.execute_reply.started":"2026-08-12T06:56:39.480936Z","shell.execute_reply":"2026-08-12T06:56:39.731440Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> **Important:** The scaler is fitted only on the training data and then used to transform the test data.","metadata":{}},{"cell_type":"markdown","source":"---\n# 🧩 Apply PCA with 50 Components\n\nThe original dataset has **784 dimensions**. Let's reduce it to just **50 principal components**.","metadata":{}},{"cell_type":"code","source":"pca = PCA(n_components=50)\n\nX_train_pca = pca.fit_transform(X_train)\nX_test_pca = pca.transform(X_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:39.734723Z","iopub.execute_input":"2026-08-12T06:56:39.734980Z","iopub.status.idle":"2026-08-12T06:56:40.252100Z","shell.execute_reply.started":"2026-08-12T06:56:39.734960Z","shell.execute_reply":"2026-08-12T06:56:40.251644Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train_pca.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:40.252666Z","iopub.execute_input":"2026-08-12T06:56:40.252829Z","iopub.status.idle":"2026-08-12T06:56:40.257433Z","shell.execute_reply.started":"2026-08-12T06:56:40.252811Z","shell.execute_reply":"2026-08-12T06:56:40.256747Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"PCA has reduced our feature space from **784 → 50 dimensions**.","metadata":{}},{"cell_type":"markdown","source":"---\n# 🤖 KNN After PCA\n\nLet's train KNN using the reduced dataset.","metadata":{}},{"cell_type":"code","source":"knn_pca = KNeighborsClassifier()\n\nknn_pca.fit(X_train_pca, y_train)\n\ny_pred_pca = knn_pca.predict(X_test_pca)\n\naccuracy_score(y_test, y_pred_pca)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:40.258138Z","iopub.execute_input":"2026-08-12T06:56:40.258336Z","iopub.status.idle":"2026-08-12T06:56:41.088104Z","shell.execute_reply.started":"2026-08-12T06:56:40.258316Z","shell.execute_reply":"2026-08-12T06:56:41.087394Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"With only **50 components**, the model still achieves approximately **95.33% accuracy**.\n\nThis shows that PCA can significantly reduce dimensionality while retaining most of the useful information.","metadata":{}},{"cell_type":"markdown","source":"---\n# 📊 Test Different Numbers of Components\n\nThe number of PCA components can strongly affect model performance.\n\nWe'll test different numbers of components and observe how the KNN accuracy changes.","metadata":{}},{"cell_type":"code","source":"for i in range(1, 785, 100):\n    pca = PCA(n_components=i)\n    X_train_trf = pca.fit_transform(X_train)\n    X_test_trf = pca.transform(X_test)\n    \n    knn = KNeighborsClassifier()\n    \n    knn.fit(X_train_trf,y_train)\n    \n    y_pred = knn.predict(X_test_trf)\n    \n    print(accuracy_score(y_test,y_pred))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:56:41.089249Z","iopub.execute_input":"2026-08-12T06:56:41.089405Z","iopub.status.idle":"2026-08-12T06:57:09.326952Z","shell.execute_reply.started":"2026-08-12T06:56:41.089389Z","shell.execute_reply":"2026-08-12T06:57:09.326211Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The results show that increasing the number of components does **not necessarily improve accuracy**. PCA is therefore useful not only for compression, but also for finding a lower-dimensional representation that works well for the model.","metadata":{}},{"cell_type":"markdown","source":"---\n# 📉 PCA Visualization in 2D\n\nPCA can also project high-dimensional data into **2 dimensions**, making it possible to visualize the dataset.","metadata":{}},{"cell_type":"code","source":"# transforming to a 2D coordinate system\npca = PCA(n_components=2)\n\nX_train_trf = pca.fit_transform(X_train)\nX_test_trf = pca.transform(X_test)\n\nX_train_trf","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:09.327687Z","iopub.execute_input":"2026-08-12T06:57:09.327904Z","iopub.status.idle":"2026-08-12T06:57:09.773167Z","shell.execute_reply.started":"2026-08-12T06:57:09.327862Z","shell.execute_reply":"2026-08-12T06:57:09.771183Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now every 784-dimensional image is represented using only **two principal components**.","metadata":{}},{"cell_type":"markdown","source":"## 🎨 Interactive 2D Visualization","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\ny_train_trf = y_train.astype(str)\nfig = px.scatter(x=X_train_trf[:,0],\n                 y=X_train_trf[:,1],\n                 color=y_train_trf,\n                 color_discrete_sequence=px.colors.qualitative.G10\n                )\nfig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:09.773721Z","iopub.execute_input":"2026-08-12T06:57:09.773916Z","iopub.status.idle":"2026-08-12T06:57:14.052322Z","shell.execute_reply.started":"2026-08-12T06:57:09.773896Z","shell.execute_reply":"2026-08-12T06:57:14.051321Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# transforming in 3D\npca = PCA(n_components=3)\nX_train_trf = pca.fit_transform(X_train)\nX_test_trf = pca.transform(X_test)\n\nX_train_trf","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:14.053409Z","iopub.execute_input":"2026-08-12T06:57:14.053619Z","iopub.status.idle":"2026-08-12T06:57:14.487864Z","shell.execute_reply.started":"2026-08-12T06:57:14.053590Z","shell.execute_reply":"2026-08-12T06:57:14.487471Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import plotly.express as px\ny_train_trf = y_train.astype(str)\nfig = px.scatter_3d(df, x=X_train_trf[:,0], y=X_train_trf[:,1], z=X_train_trf[:,2],\n              color=y_train_trf)\nfig.update_layout(\n    margin=dict(l=20, r=20, t=20, b=20),\n    paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:14.488708Z","iopub.execute_input":"2026-08-12T06:57:14.489014Z","iopub.status.idle":"2026-08-12T06:57:14.629270Z","shell.execute_reply.started":"2026-08-12T06:57:14.488993Z","shell.execute_reply":"2026-08-12T06:57:14.628611Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Eigen values & vectors\npca.explained_variance_, pca.components_.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:14.630133Z","iopub.execute_input":"2026-08-12T06:57:14.630337Z","iopub.status.idle":"2026-08-12T06:57:14.635006Z","shell.execute_reply.started":"2026-08-12T06:57:14.630314Z","shell.execute_reply":"2026-08-12T06:57:14.634271Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pca.explained_variance_ratio_","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:14.635663Z","iopub.execute_input":"2026-08-12T06:57:14.635901Z","iopub.status.idle":"2026-08-12T06:57:14.651766Z","shell.execute_reply.started":"2026-08-12T06:57:14.635863Z","shell.execute_reply":"2026-08-12T06:57:14.651080Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pca = PCA(n_components=None)\n\nX_train_trf = pca.fit_transform(X_train)\nX_test_trf = pca.transform(X_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:14.652766Z","iopub.execute_input":"2026-08-12T06:57:14.653465Z","iopub.status.idle":"2026-08-12T06:57:15.696365Z","shell.execute_reply.started":"2026-08-12T06:57:14.653425Z","shell.execute_reply":"2026-08-12T06:57:15.695558Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pca.explained_variance_.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:15.697082Z","iopub.execute_input":"2026-08-12T06:57:15.697286Z","iopub.status.idle":"2026-08-12T06:57:15.701822Z","shell.execute_reply.started":"2026-08-12T06:57:15.697263Z","shell.execute_reply":"2026-08-12T06:57:15.701063Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pca.components_.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:15.702491Z","iopub.execute_input":"2026-08-12T06:57:15.704043Z","iopub.status.idle":"2026-08-12T06:57:15.720318Z","shell.execute_reply.started":"2026-08-12T06:57:15.703997Z","shell.execute_reply":"2026-08-12T06:57:15.719780Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"np.cumsum(pca.explained_variance_ratio_)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:15.720909Z","iopub.execute_input":"2026-08-12T06:57:15.721084Z","iopub.status.idle":"2026-08-12T06:57:15.743047Z","shell.execute_reply.started":"2026-08-12T06:57:15.721065Z","shell.execute_reply":"2026-08-12T06:57:15.742435Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.plot(np.cumsum(pca.explained_variance_ratio_))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-12T06:57:15.743980Z","iopub.execute_input":"2026-08-12T06:57:15.745099Z","iopub.status.idle":"2026-08-12T06:57:15.859306Z","shell.execute_reply.started":"2026-08-12T06:57:15.745076Z","shell.execute_reply":"2026-08-12T06:57:15.858700Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}