{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30587,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Phần 1: Nhập các thư viện cần thiết\n------","metadata":{"_uuid":"ebd546a2-326f-4a93-9042-31cb470d1cc4","_cell_guid":"fe85bd07-ce02-43ce-933f-2d24429c163b","trusted":true}},{"cell_type":"markdown","source":"
\nThe MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.

\nThe original black and white images were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels. The original MNIST database contains 60,000 training images and 10,000 testing images, but in this competition we are given 42,000 training images (70%) and 18,000 testing images (30%).

\nThe state-of-the-art techniques to approach any Image Classification problem typically rely on Convolutional Neural Networks. This kind of architecture are really good at extracting patterns from images. However, since the MNIST is a simple problem, in this notebook we will aplly simple Machine Learning Algorithms (and we will get a decent accuracy!).

\nFirst, we will perform a Principal Component Analysis to reduce the dimension of the data (from 784 pixel values to 10 principal components). Then, we will run our favourite lazy learner algorithm: K-Nearest Neighbors. This combinatino achieves a Test Accuracy of 0.9271, which is pretty remarkable !!!

\nTable of Contents:\n\n
","metadata":{"_uuid":"e00355b6-e203-4d64-9600-35f736d67626","_cell_guid":"6c36196c-42f6-4b0c-bfbd-a2d573e6a320","trusted":true}},{"cell_type":"code","source":"## IMPORT THE NECESSARY PYTHON LIBRARIES\n\n# Read & Explore Data\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import MinMaxScaler\n\n# Principal Component Analysis (PCA)\nfrom sklearn.decomposition import PCA\nfrom sklearn.pipeline import Pipeline\nimport plotly.express as px\n\n# K-Nearest Neighbor Classifier (KNN)\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import classification_report, \\\n confusion_matrix, ConfusionMatrixDisplay \n\n# VHT\nfrom scipy.ndimage import rotate","metadata":{"_uuid":"c9f6edfd-588d-4af2-97d8-1073e4883692","_cell_guid":"8d8d769c-d2a0-4558-b63f-86352b4bafb8","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:50.119272Z","iopub.execute_input":"2024-04-23T04:42:50.119693Z","iopub.status.idle":"2024-04-23T04:42:50.128015Z","shell.execute_reply.started":"2024-04-23T04:42:50.119660Z","shell.execute_reply":"2024-04-23T04:42:50.126687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def func_array1d_rotate2d(Narray1d,rotate_angle,sizePixel):\n\n N = Narray1d.shape[0]\n\n Narray1d_rotate = np.zeros(Narray1d.shape)\n \n for i in range(N):\n image = np.reshape(Narray1d[i,:],(sizePixel,sizePixel))\n rotated_image = rotate(image,rotate_angle, reshape=False)\n rotated_array1d = np.reshape(rotated_image,(1,-1))\n Narray1d_rotate[i,:] = rotated_array1d\n \n return Narray1d_rotate","metadata":{"_uuid":"76ecec72-7aaa-4e9c-9366-c53a6fb1d164","_cell_guid":"f51ca997-1fbb-4259-94ee-5b05b70a035a","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:50.130219Z","iopub.execute_input":"2024-04-23T04:42:50.130755Z","iopub.status.idle":"2024-04-23T04:42:50.146885Z","shell.execute_reply.started":"2024-04-23T04:42:50.130711Z","shell.execute_reply":"2024-04-23T04:42:50.145642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"----\n# Phần 2: Đọc dữ liệu và khai thác dữ liệu","metadata":{"_uuid":"0ff27da2-835c-4122-aca0-9e731f3dc7ea","_cell_guid":"6248d1ca-53a8-45d0-9ab6-9003187ef756","trusted":true}},{"cell_type":"markdown","source":"
\nIn this section, we will use the well-known Pandas Python library to read the data from .csv files.

\nWe will visualize the distribution of the pixel values (histogram) using the Seaborn and Matplotlib Python libraries. We will also check that there are no missing values, and that the target classes are balanced.

\nLastly, we will visualize the images by plotting one image from each class.\n\n
","metadata":{"_uuid":"08c1c024-9f04-4d45-80af-6f8b9ed71f30","_cell_guid":"e27a867c-ee7f-4c03-85fd-bec46a9f2fd2","trusted":true}},{"cell_type":"code","source":"# Read the data files using Pandas\ndf_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndf_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"_uuid":"078702a8-cfea-4f1c-bb0b-1018fea19c2c","_cell_guid":"8b4f5e4b-fc80-473b-bd7a-7e620b412c90","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:50.148425Z","iopub.execute_input":"2024-04-23T04:42:50.148929Z","iopub.status.idle":"2024-04-23T04:42:55.554192Z","shell.execute_reply.started":"2024-04-23T04:42:50.148891Z","shell.execute_reply":"2024-04-23T04:42:55.553062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the size of the train and test DataFrames\nprint(\"df_train shape:\", df_train.shape) # 70%\nprint(\"df_test shape: \", df_test.shape) # 30%","metadata":{"_uuid":"ca8aa9af-b761-403d-aa85-a3642ea77c5c","_cell_guid":"ba38193d-5d27-48f6-a7b3-f505b93dfe84","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:55.556983Z","iopub.execute_input":"2024-04-23T04:42:55.557421Z","iopub.status.idle":"2024-04-23T04:42:55.564221Z","shell.execute_reply.started":"2024-04-23T04:42:55.557374Z","shell.execute_reply":"2024-04-23T04:42:55.562781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fortunately there are no missing values in our data\nprint(\n \"Missing values in df_train: \", \\\n df_train.isna().sum().sum()\n)\nprint(\n \"Missing values in df_test: \", \\\n df_test.isna().sum().sum()\n)","metadata":{"_uuid":"a10fb707-86d3-4a6b-98f0-ace8a866768f","_cell_guid":"7cede701-9df4-4ac1-a358-2c567d72cd67","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:55.566124Z","iopub.execute_input":"2024-04-23T04:42:55.566678Z","iopub.status.idle":"2024-04-23T04:42:55.625678Z","shell.execute_reply.started":"2024-04-23T04:42:55.566632Z","shell.execute_reply":"2024-04-23T04:42:55.624505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's have a look at the first five training rows\ndf_train.head()","metadata":{"_uuid":"087da919-79ff-4a00-a357-fdb65381edda","_cell_guid":"cbfdc56f-8ade-4af2-b159-a2f526b273a5","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:55.627038Z","iopub.execute_input":"2024-04-23T04:42:55.627379Z","iopub.status.idle":"2024-04-23T04:42:55.648545Z","shell.execute_reply.started":"2024-04-23T04:42:55.627348Z","shell.execute_reply":"2024-04-23T04:42:55.647072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can see that the first colum is the label,\n# and the rest of the columns are the 784 pixels\n# that represent 28x28 gray-scale images.\n# Let's separate X and Y\n\nX_train = df_train.drop(columns=['label']).values\ny_train = df_train['label'].values","metadata":{"_uuid":"2aefa02a-2611-42d1-9764-6ab390b73c0b","_cell_guid":"ccfff756-d687-424c-877d-028bb3180a59","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:55.650081Z","iopub.execute_input":"2024-04-23T04:42:55.650414Z","iopub.status.idle":"2024-04-23T04:42:55.744109Z","shell.execute_reply.started":"2024-04-23T04:42:55.650386Z","shell.execute_reply":"2024-04-23T04:42:55.742593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's visualize the distribution of the pixel values\n\nax = sns.histplot(x=X_train.flatten(), color='#4895EE')\n_ = ax.set(title='Pixel Value Histogram', xlabel='Pixel Value')\n\n# As we can see, the 0 value represents the background\n# and the >0 values represent the digits in the images","metadata":{"_uuid":"7a976629-a208-4444-970a-6f990f01b730","_cell_guid":"f8a9174d-c11e-4467-8dcd-bba2e0d51ac3","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:42:55.745687Z","iopub.execute_input":"2024-04-23T04:42:55.746738Z","iopub.status.idle":"2024-04-23T04:43:13.949027Z","shell.execute_reply.started":"2024-04-23T04:42:55.746690Z","shell.execute_reply":"2024-04-23T04:43:13.947624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's check that the target is balanced.\n# It means that the ten categories in the target\n# have roughly the same representation in the data.\n\nax = sns.countplot(x=y_train, color='#4895EE')\nax.set(\n title='Target distribution is balanced? Yes!',\n xlabel='Label'\n)\n# The expected number of samples for a balanced \n# class distribution is 42000/10 = 4200\n_ = ax.axhline(y=4200, color='#12086F', linestyle='--')\n\n# Since the target classes are balances, we don't have\n# worry about sampling methods and biases in our models.","metadata":{"_uuid":"56cd7484-96d5-4052-a732-31184b3010be","_cell_guid":"31dd96d2-e9fc-4ec8-8027-6182b59bbb56","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:13.955050Z","iopub.execute_input":"2024-04-23T04:43:13.955454Z","iopub.status.idle":"2024-04-23T04:43:14.433641Z","shell.execute_reply.started":"2024-04-23T04:43:13.955420Z","shell.execute_reply":"2024-04-23T04:43:14.432423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Next, let's visualize one image from each class.\n# We will use the matplotlib 'imshow' function.\n\nfig, axs = plt.subplots(nrows=2, ncols=5, figsize=(10,5))\n\nfor i in range(10):\n idx = np.where(y_train==i)[0][2]\n img = axs[int(i >= 5), i%5].imshow(\n X=X_train[idx].reshape(28, 28),\n cmap='Blues'\n )\n img.axes.get_yaxis().set_visible(False)\n img.axes.get_xaxis().set_visible(False)\n img.axes.set_title(f'Label: {i}')","metadata":{"_uuid":"6e57247a-308f-4d8c-b860-3fcca06504e8","_cell_guid":"85fb1afc-211b-4d78-a121-385e208ece49","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:14.435102Z","iopub.execute_input":"2024-04-23T04:43:14.435438Z","iopub.status.idle":"2024-04-23T04:43:15.000065Z","shell.execute_reply.started":"2024-04-23T04:43:14.435410Z","shell.execute_reply":"2024-04-23T04:43:14.998873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# transform data","metadata":{"_uuid":"9d4cd46f-3405-4ee3-a2e3-758b928024b1","_cell_guid":"abe3ee94-93c7-4941-9358-150e526414ec","trusted":true}},{"cell_type":"code","source":"# -------------------------------------------------- declare\nnumRotate_prior = 10\n\nangleOffset_prior = 0\nangleRange_prior = 90\n\nX_train_pick2rotate = {\n 'index': np.array([0,1,2,3,4]),\n 'offset': np.array([\n [-5,35,-5,0,-15, \n 10,-35,0,20,25],\n [0,0,0,0,0, \n -5,-10,0,10,10],\n [10,5,0,0,0, \n 0,10,0,20,0],\n [45,10,-20,0,0, \n 0,-15,0,-35,0],\n [0,-5,0,0,10, \n 10,-25,0,20,0],\n ])\n}\n\nnumPickRotate = len(X_train_pick2rotate['index'])\n\nprint('numPickRotate = ', numPickRotate)\n\n# -------------------------------------------------- rotate\n\nX_sample_rotate = np.zeros((10,numRotate_prior,numPickRotate,X_train.shape[1]))\ny_sample_rotate = np.zeros((10,numRotate_prior,numPickRotate))\n\nfor k0 in range(10):\n for k1 in range(numRotate_prior):\n for k2 in range(numPickRotate):\n \n idxPick_X_train = X_train_pick2rotate['index'][k2]\n \n idx = np.where(y_train==k0)[0][idxPick_X_train]\n \n y_sample_rotate[k0,k1,k2] = k0\n \n angleRotate = (k1/numRotate_prior * angleRange_prior) + angleOffset_prior + X_train_pick2rotate['offset'][idxPick_X_train][k0]\n \n X_sample_rotate[k0,k1,k2,:] = func_array1d_rotate2d(X_train[idx][np.newaxis,:],angleRotate, 28)\n\n \nX_sample_rotate[X_sample_rotate < 10] = 0\n#X_sample_rotate[X_sample_rotate > 300] = 0\n\n# Let's normalize the pixel values to the interval [0,1]\n# Formula: MinMaxscale(X) = (X - X_min) / (X_max - X_min)\n\nprint(\"Before applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_sample_rotate.min())\nprint(\" - Maximum value:\", X_sample_rotate.max())\n\nminmax_scaler = MinMaxScaler()\n\nfor k0 in range(10):\n for k1 in range(numRotate_prior):\n X_sample_rotate[k0,k1,:,:] = minmax_scaler.fit_transform(X_sample_rotate[k0,k1,:,:]) * 255\n\nprint(\"\\nAfter applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_sample_rotate.min())\nprint(\" - Maximum value:\", X_sample_rotate.max())\n\n\n\n# Next, let's visualize one image from each class.\n# We will use the matplotlib 'imshow' function.\n\nfor k2 in range(numPickRotate):\n\n k1 = 0\n # k1 = numRotate_prior - 1\n\n fig, axs = plt.subplots(nrows=2, ncols=5, figsize=(10,5))\n\n for k0 in range(10):\n img = axs[int(k0 >= 5), k0%5].imshow(\n X=X_sample_rotate[k0,k1,k2,:].reshape(28, 28),\n cmap='Blues'\n )\n img.axes.get_yaxis().set_visible(False)\n img.axes.get_xaxis().set_visible(False)\n img.axes.set_title(f'Label: {k0}')\n \nX_train_rotate = np.reshape(X_sample_rotate, (-1,784))\ny_train_rotate = np.reshape(y_sample_rotate, (-1,1)).flatten()","metadata":{"_uuid":"e87363e6-d051-4009-be4f-a825679930e7","_cell_guid":"db89b71d-4901-4716-9507-e9497e268d74","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:15.001750Z","iopub.execute_input":"2024-04-23T04:43:15.002240Z","iopub.status.idle":"2024-04-23T04:43:18.571884Z","shell.execute_reply.started":"2024-04-23T04:43:15.002196Z","shell.execute_reply":"2024-04-23T04:43:18.570592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------\n# Phần 3: Phân tích thành phần chính","metadata":{"_uuid":"8794e86b-0a44-46a2-8d06-c6186454767d","_cell_guid":"d404fd37-ba51-4a9f-9de3-878c4bb0ad21","trusted":true}},{"cell_type":"markdown","source":"
\n Principal Comopnent Analysis (PCA) is a statistical technique for reducing the dimensionality of a dataset. This is accomplished by linearly transforming the data into a new coordinate system where (most of) the variation in the data can be described with fewer dimensions than the initial data.

\nThe first principal component can equivalently be defined as a direction that maximizes the variance of the projected data. The i-th principal component can be taken as a direction orthogonal to the first i-1 principal components that maximizes the variance of the projected data.

\nsklearn.decomposition.PCA: Linear dimensionality reduction using Singular Value Decomposition of the data to project it to a lower dimensional space.

\nHere we will use the top 10 principal components, which explain almost 50% of the variance in the data. This means that we can express 50% of the variance of the data with a space of features >98% smaller than the original one (10 features vs. 784 features)\n
","metadata":{"_uuid":"988c70b6-e9d9-4ae7-b127-921423ab0f0e","_cell_guid":"231a3cf2-1f44-475b-a745-a28f3eb35d1e","trusted":true}},{"cell_type":"code","source":"# Let's normalize the pixel values to the interval [0,1]\n# Formula: MinMaxscale(X) = (X - X_min) / (X_max - X_min)\n\nprint(\"Before applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_train.min())\nprint(\" - Maximum value:\", X_train.max())\n\nminmax_scaler = MinMaxScaler()\nX_train_rotate_scaled = minmax_scaler.fit_transform(X_train_rotate)\n\nprint(\"\\nAfter applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_train_rotate_scaled.min())\nprint(\" - Maximum value:\", X_train_rotate_scaled.max())","metadata":{"_uuid":"2f018749-1981-43d6-b1b8-b1716a72e3fb","_cell_guid":"7c68f163-b747-4392-9a1e-8792cb07ccde","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:18.573465Z","iopub.execute_input":"2024-04-23T04:43:18.573845Z","iopub.status.idle":"2024-04-23T04:43:18.633118Z","shell.execute_reply.started":"2024-04-23T04:43:18.573814Z","shell.execute_reply":"2024-04-23T04:43:18.631718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Actually, we will embed the MinMaxScaler and the PCA\n# modules into a Sklearn Pipeline.\n\npca = Pipeline(steps=[\n ('scaler', MinMaxScaler()),\n ('pca', PCA(n_components=10))\n])\n\nX_train_rotate_pca = pca.fit_transform(X=X_train_rotate)\n\nprint('X_train_pca shape:', X_train_rotate_pca.shape)","metadata":{"_uuid":"4bfc3273-16ce-4a37-93ae-9cf46373e2c4","_cell_guid":"2d2d1d92-8b2d-4b97-af06-40a19024b09e","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:18.634802Z","iopub.execute_input":"2024-04-23T04:43:18.635197Z","iopub.status.idle":"2024-04-23T04:43:18.798279Z","shell.execute_reply.started":"2024-04-23T04:43:18.635164Z","shell.execute_reply":"2024-04-23T04:43:18.797161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's extract the explained variance ratio\n# And let's compute their cumulative sum\n\npca_object = pca.named_steps['pca']\nvar_ratio = pd.Series(\n data=pca_object.explained_variance_ratio_,\n index=range(1, 11)\n)\n\ncumsum_var_ratio = var_ratio.cumsum()","metadata":{"_uuid":"e551b2e2-ce61-4b14-abca-472ab8f8fa6f","_cell_guid":"8fe76194-de6d-44ef-89b7-489a936a3a12","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:18.799799Z","iopub.execute_input":"2024-04-23T04:43:18.800524Z","iopub.status.idle":"2024-04-23T04:43:18.808682Z","shell.execute_reply.started":"2024-04-23T04:43:18.800483Z","shell.execute_reply":"2024-04-23T04:43:18.807381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the explained variance ratio for each component (left)\n# And plot their cumulative sum (right)\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))\n\nsns.barplot(\n x=var_ratio.index, \n y=var_ratio.values, \n color='#4895EE', \n ax=ax1\n)\nax1.set(\n title='Explained Variance Ratio PCA',\n xlabel='Component',\n ylabel='Var Ratio',\n)\nax1.set_ylim(0, 0.5)\n\nsns.barplot(\n x=cumsum_var_ratio.index, \n y=cumsum_var_ratio.values, \n color='#4895EE', \n ax=ax2\n)\nax2.set(\n title='Cumulative Explained Variance Ratio PCA',\n xlabel='Component',\n ylabel='Cumulative Var Ratio',\n)\n_ = ax2.set_ylim(0, 0.5)","metadata":{"_uuid":"34ab9680-e976-4c81-bf92-6e9d8a0236bb","_cell_guid":"5e8c9489-629b-45b3-a577-b855fe33869f","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:18.811024Z","iopub.execute_input":"2024-04-23T04:43:18.812153Z","iopub.status.idle":"2024-04-23T04:43:20.055054Z","shell.execute_reply.started":"2024-04-23T04:43:18.812108Z","shell.execute_reply":"2024-04-23T04:43:20.053471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's put the components in Pandas DataFrame format\n# This way we can use the seaborn and plotly libraries\n# to visualize the data.\n# (these packages work better with Pandas DataFrames)\n\ndf_train_pca = pd.DataFrame(\n data=X_train_rotate_pca[:,:3], \n columns=['x', 'y', 'z']\n)\n# Let's also add the target to set the colors\ndf_train_pca['target'] = y_train_rotate\n\nprint(df_train_pca.shape)\ndf_train_pca.head()","metadata":{"_uuid":"b779e0bc-6938-424a-9c08-3579404299da","_cell_guid":"8d64a105-5b85-4924-9d4c-d5bc95d474e8","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:20.056641Z","iopub.execute_input":"2024-04-23T04:43:20.057036Z","iopub.status.idle":"2024-04-23T04:43:20.074897Z","shell.execute_reply.started":"2024-04-23T04:43:20.057002Z","shell.execute_reply":"2024-04-23T04:43:20.073670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Two-dimensional visualization of the data\n# Here the Top2 principal components are plotted\n\nsns.set_style(\"dark\")\nax = sns.scatterplot(\n data=df_train_pca,\n x='x', \n y='y', \n hue='target', \n palette='Blues',\n)\nax.set(\n title='PCA Two-Dimensional Projection',\n xlabel='1st Comp.',\n ylabel='2nd Comp.'\n)\nax.axhline(y=0, linestyle='--', color='#FFBF00', zorder=-1)\nax.axvline(x=0, linestyle='--', color='#FFBF00', zorder=-1)","metadata":{"_uuid":"90d9d1e3-fbcb-400a-b8d5-68d2a5f4f646","_cell_guid":"c05ac3f3-ec1c-4fd0-8e88-cd0991afb146","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:20.076347Z","iopub.execute_input":"2024-04-23T04:43:20.076758Z","iopub.status.idle":"2024-04-23T04:43:21.090035Z","shell.execute_reply.started":"2024-04-23T04:43:20.076723Z","shell.execute_reply":"2024-04-23T04:43:21.088702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Two-dimensional visualization of the data\n# Here the Top3 principal components are plotted\n\nsns.set_style(\"white\")\nfig = px.scatter_3d(\n data_frame=df_train_pca,\n x='x',\n y='y',\n z='z',\n labels={\n 'x': '1st Comp.',\n 'y': '2nd Comp.',\n 'z': '3rd Comp.'\n },\n color='target',\n title='PCA Three-Dimensional Projection',\n color_continuous_scale='blues',\n template='ggplot2'\n)\n\nfig.show()","metadata":{"_uuid":"1fb19f51-6852-4f02-8660-90a8a537fa30","_cell_guid":"b76b9b11-2a7e-4cc9-983c-0ae311d92428","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:21.091787Z","iopub.execute_input":"2024-04-23T04:43:21.092168Z","iopub.status.idle":"2024-04-23T04:43:21.175440Z","shell.execute_reply.started":"2024-04-23T04:43:21.092134Z","shell.execute_reply":"2024-04-23T04:43:21.174251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, let's apply the inverse_transform to compute\n# the approximation of images based on the 10 principal\n# compontents.\n# We can see that with just 10 components we convey a large\n# part of the information of each image.\n\npca_object = pca.named_steps['pca']\n\nfig, axs = plt.subplots(nrows=10, ncols=2, figsize=(6,20))\n\nfor i in range(10):\n idx = np.where(y_train_rotate==i)[0][0]\n original_img = axs[i, 0].imshow(\n X=X_train_rotate[idx].reshape(28, 28),\n cmap='Blues'\n )\n original_img.axes.get_yaxis().set_visible(False)\n original_img.axes.get_xaxis().set_visible(False)\n original_img.axes.set_title(f'Original [{i}]:')\n \n approx_img_ = pca_object.inverse_transform(X=X_train_rotate_pca[idx])\n approx_img = axs[i, 1].imshow(\n X=approx_img_.reshape(28, 28),\n cmap='Blues'\n )\n approx_img.axes.get_yaxis().set_visible(False)\n approx_img.axes.get_xaxis().set_visible(False)\n approx_img.axes.set_title(f'Approximation [{i}]:')","metadata":{"_uuid":"0d814515-0bec-4c78-af57-09ed84b2dfc4","_cell_guid":"7e42f02f-c973-4898-8c10-80b0ca2d625b","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:21.177266Z","iopub.execute_input":"2024-04-23T04:43:21.177795Z","iopub.status.idle":"2024-04-23T04:43:22.672437Z","shell.execute_reply.started":"2024-04-23T04:43:21.177749Z","shell.execute_reply":"2024-04-23T04:43:22.671063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show that the PCA components are uncorrelated\n# (This statement is true by definition and construction\n# of the Principal Component Analysis algorithm)\n\ncorr = np.corrcoef(x=X_train_rotate_pca, rowvar=False)\ncorr = np.around(corr, decimals=2)\n\nax = sns.heatmap(\n data=corr, \n vmax=1, \n vmin=-1,\n square=True,\n linewidths=.5,\n annot=True,\n cmap='Blues',\n)\n\n_ = ax.set(\n title='Correlation PCA Components [Uncorrelated]',\n xlabel='Components',\n ylabel='Components'\n)","metadata":{"_uuid":"8a21533b-a961-49b1-b24c-d4e7050cc806","_cell_guid":"32e1e369-193f-4071-9914-b2f03cce2893","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:22.674123Z","iopub.execute_input":"2024-04-23T04:43:22.674595Z","iopub.status.idle":"2024-04-23T04:43:24.531686Z","shell.execute_reply.started":"2024-04-23T04:43:22.674533Z","shell.execute_reply":"2024-04-23T04:43:24.530426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"----\n# Phần 4: Sử dụng K-Nearest Neighbors Classifier phân loại 10 thành phần chính hàng đầu từ PCA.","metadata":{"_uuid":"370f38cc-e5fb-4b4e-8dd3-6b9e303ee060","_cell_guid":"43a48481-ea7f-470e-b826-8fd2fcc9e37f","trusted":true}},{"cell_type":"markdown","source":"
\nThe K-Nearest Neighbors algorithm is a non-parametric, supervised learning classifier, which uses proximity to make classifications or predictions about the grouping of an individual data point. Prediction: A class label is assigned on the basis of a majority vote. the label that is most frequently represented around a given data point.

\nIn this project, the input of the KNN Classifier will be the top 10 principal components from the PCA.

\nsklearn.neighbors.KNeighborsClassifier: Classifier implementing the k-nearest neighbors vote. We will run a GridSearchCV to optimize the n_neighbors hyperparameter.\n
","metadata":{"_uuid":"db9e5ad7-aa3d-4606-9848-526bc3dc048f","_cell_guid":"efff5a1e-7ac3-41a4-b464-77ed4de056d3","trusted":true}},{"cell_type":"markdown","source":"---\n# mô hình học máy trên tập dữ liệu Train","metadata":{"_uuid":"7c0a4822-eedd-4aa0-bac6-a2cf5a7b89bf","_cell_guid":"39e3a7f5-5363-4914-b78d-295251352529","trusted":true}},{"cell_type":"code","source":"knn_clf = Pipeline(steps=[\n ('scaler', MinMaxScaler()),\n ('pca', PCA(n_components=10)),\n ('knn', KNeighborsClassifier())\n])\n\nknn_clf_params = {\n 'knn__n_neighbors': [2, 4, 8, 16, 32]\n}\n\nknn_clf_gs = GridSearchCV(\n estimator=knn_clf, \n param_grid=knn_clf_params,\n scoring='accuracy',\n n_jobs=-1,\n cv=5,\n verbose=1,\n return_train_score=True\n)\n\nknn_clf_gs.fit(X=X_train_rotate_pca, y=y_train_rotate)\n\nprint('best PCA+KNN params:', knn_clf_gs.best_params_)\nprint('best PCA+KNN cv accuracy:', knn_clf_gs.best_score_)","metadata":{"_uuid":"2d7cd392-e51f-420a-aad2-0eb5c72ff742","_cell_guid":"797ae756-d6fc-4456-83e5-447d3cc034e2","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:24.533452Z","iopub.execute_input":"2024-04-23T04:43:24.534223Z","iopub.status.idle":"2024-04-23T04:43:25.521919Z","shell.execute_reply.started":"2024-04-23T04:43:24.534187Z","shell.execute_reply":"2024-04-23T04:43:25.520315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the Training and Validation Accuracies\n# for each Hyperparamter value tested during the\n# Grid Search.\n# This plot allows us to check for overfitting.\n\nfig, ax = plt.subplots()\n\nax.plot(\n range(len(knn_clf_params['knn__n_neighbors'])), \n knn_clf_gs.cv_results_['mean_train_score'],\n label='train',\n color='blue'\n)\nax.plot(\n range(len(knn_clf_params['knn__n_neighbors'])),\n knn_clf_gs.cv_results_['mean_test_score'],\n label='test',\n color='orange'\n)\nax.set_xticks(\n ticks=range(len(knn_clf_params['knn__n_neighbors'])),\n labels=knn_clf_params['knn__n_neighbors']\n)\n\nii = knn_clf_params['knn__n_neighbors'].index(\n knn_clf_gs.best_params_['knn__n_neighbors']\n)\n\nplt.axvline(x=ii, linestyle='--', color='red', label='best', alpha=0.3)\nplt.legend()\nplt.xlabel(\"'n_neighbors' vals\")\nplt.ylabel('Accuracy')\nplt.title(\"PCA+KNN GridSearchCV Results\")\nplt.show()","metadata":{"_uuid":"2abf36ed-c1a2-4581-8a67-414fbfe556f5","_cell_guid":"38a69c14-7bf2-4328-abc9-45a9ef2c1b11","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:25.531729Z","iopub.execute_input":"2024-04-23T04:43:25.537248Z","iopub.status.idle":"2024-04-23T04:43:25.975876Z","shell.execute_reply.started":"2024-04-23T04:43:25.537166Z","shell.execute_reply":"2024-04-23T04:43:25.974661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Phần 5: Tạo một tệp tin chứa các dự đoán của mô hình học máy trên tập dữ liệu Test","metadata":{"_uuid":"2254c327-a8ec-41ca-9fa2-c41a850f4c75","_cell_guid":"adcdd26f-00ba-479f-8e33-e1bc6d65eafb","trusted":true}},{"cell_type":"code","source":"# Compute and display the Confusion Matrix\n# Note: the entire training dataset is used here\n\nbest_cv_clf = knn_clf_gs.best_estimator_\ny_pred = best_cv_clf.predict(X=X_train_rotate_pca)\nconf_matrix = confusion_matrix(y_true=y_train_rotate, y_pred=y_pred)\n\nconf_matrix_disp = ConfusionMatrixDisplay(conf_matrix)\nconf_matrix_disp.plot(cmap='Blues')","metadata":{"_uuid":"c3ac8535-57d6-4c40-9146-3b229c95284e","_cell_guid":"aa178b6c-9094-48f3-a36e-5cc5da6e3c50","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:25.977194Z","iopub.execute_input":"2024-04-23T04:43:25.977617Z","iopub.status.idle":"2024-04-23T04:43:26.965343Z","shell.execute_reply.started":"2024-04-23T04:43:25.977569Z","shell.execute_reply":"2024-04-23T04:43:26.964079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute the classification report using sklearn\n# Note: the entire training dataset is used here\n\nclf_report = classification_report(\n y_true=y_train_rotate,\n y_pred=y_pred,\n labels=range(10),\n)\n\n# Print the classification report\nprint(clf_report)","metadata":{"_uuid":"80ad0fa3-460a-4b42-aaf6-52edfa03afe9","_cell_guid":"04c9e15a-0874-4e08-b997-858cbcc9a37e","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-04-23T04:43:26.971243Z","iopub.execute_input":"2024-04-23T04:43:26.971676Z","iopub.status.idle":"2024-04-23T04:43:26.992293Z","shell.execute_reply.started":"2024-04-23T04:43:26.971637Z","shell.execute_reply":"2024-04-23T04:43:26.990648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📃 References 📃","metadata":{"_uuid":"0bf02057-e66b-4045-bf4b-b91fb25f67ba","_cell_guid":"b35209f1-8bed-48cd-986d-7c85a99a4519","trusted":true}},{"cell_type":"markdown","source":"
\n \n
","metadata":{"_uuid":"ba488a1a-be81-42a7-8169-c106278b9f85","_cell_guid":"7d3e9231-c38d-42ac-ae5b-8c18407ec2ff","trusted":true}}]}