{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30587,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Phần 1: Nhập các thư viện cần thiết\n------","metadata":{"_uuid":"b0093541-188e-43e9-acb2-d917ae3a5010","_cell_guid":"cadbfcb2-82c3-470f-bad6-eade54542125","trusted":true}},{"cell_type":"markdown","source":"\nThe
MNIST database (
Modified National Institute of Standards and Technology database) is a large database of
handwritten digits that is commonly used for training various image processing systems. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.
\nThe original
black and white images were normalized to fit into a
28x28 pixel bounding box and anti-aliased, which introduced
grayscale levels. The original MNIST database contains 60,000 training images and 10,000 testing images, but in this competition we are given
42,000 training images (70%) and
18,000 testing images (30%).
\nThe
state-of-the-art techniques to approach any Image Classification problem typically rely on
Convolutional Neural Networks. This kind of architecture are really good at extracting patterns from images. However, since the MNIST is a simple problem, in this notebook we will aplly
simple Machine Learning Algorithms (and we will get a decent accuracy!).
\nFirst, we will perform a
Principal Component Analysis to reduce the dimension of the data (from 784 pixel values to 10 principal components). Then, we will run our favourite
lazy learner algorithm:
K-Nearest Neighbors. This combinatino achieves a
Test Accuracy of 0.9271, which is pretty remarkable !!!
\n
Table of Contents:\n
\n - 🔎 Read & Explore the Data 🔍:
\n - Check if there are missing values.
\n - Check the range of pixel values.
\n - Visualize the data (Images).
\n - Check that the target is balanced.
\n - 🔶 PCA 🔶:
\n - Introduce the PCA algorithm.
\n - Preprocess (scale) the data.
\n - Implement the PCA algorithm using the Sklearn library.
\n - Embed the modules into a Sklearn Pipeline.
\n - Plot the explained variance of the components.
\n - Two-dimensional visualization of the data using seaborn.
\n - Three-dimensional dynamic visualization of the data using plotly.
\n - Visualization of the inverse transformation.
\n - Check that the components are uncorrelated.
\n - 🔶 KNN Classifier 🔶:
\n - Build a KNN Classifier using Sklearn.
\n - Use the PCA technique as a feature extractor for the KNN.
\n - Run a Grid-Search Cross-Validation to optimize the n_neighbors hyperparameters.
\n - Compare train and validation metrics to check for overfitting.
\n - Plot the Confusion Matrix.
\n - Compute the Classification Report.
\n
\n
\nIn this section, we will use the well-known Pandas Python library to read the data from .csv files.
\nWe will visualize the distribution of the pixel values (histogram) using the Seaborn and Matplotlib Python libraries. We will also check that there are no missing values, and that the target classes are balanced.
\nLastly, we will visualize the images by plotting one image from each class.\n\n
","metadata":{"_uuid":"d4c82fac-2a10-4c14-a95a-653bd16872cd","_cell_guid":"deaebcb3-115c-4e6b-b843-4522382a052d","trusted":true}},{"cell_type":"code","source":"# Read the data files using Pandas\ndf_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndf_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"_uuid":"4da57b46-cc50-4ae9-9495-d0dbeb6be5ee","_cell_guid":"8967621a-0c55-49ff-b6e2-05d9efdeedf2","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:37:54.499228Z","iopub.execute_input":"2024-06-11T02:37:54.499564Z","iopub.status.idle":"2024-06-11T02:37:59.191186Z","shell.execute_reply.started":"2024-06-11T02:37:54.499539Z","shell.execute_reply":"2024-06-11T02:37:59.190059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the size of the train and test DataFrames\nprint(\"df_train shape:\", df_train.shape) # 70%\nprint(\"df_test shape: \", df_test.shape) # 30%","metadata":{"_uuid":"5465a268-9ec4-4482-a0d7-e531d25f4917","_cell_guid":"da0e04b7-5ce3-48ec-be64-cee70c71af07","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:37:59.192500Z","iopub.execute_input":"2024-06-11T02:37:59.192773Z","iopub.status.idle":"2024-06-11T02:37:59.198015Z","shell.execute_reply.started":"2024-06-11T02:37:59.192750Z","shell.execute_reply":"2024-06-11T02:37:59.197122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fortunately there are no missing values in our data\nprint(\n \"Missing values in df_train: \", \\\n df_train.isna().sum().sum()\n)\nprint(\n \"Missing values in df_test: \", \\\n df_test.isna().sum().sum()\n)","metadata":{"_uuid":"f8324800-d007-4d21-9eb6-39bba50ce548","_cell_guid":"758f94a2-69c7-4c9b-b1a3-22cb0f0b1a43","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:37:59.201185Z","iopub.execute_input":"2024-06-11T02:37:59.201544Z","iopub.status.idle":"2024-06-11T02:37:59.248201Z","shell.execute_reply.started":"2024-06-11T02:37:59.201512Z","shell.execute_reply":"2024-06-11T02:37:59.247269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's have a look at the first five training rows\ndf_train.head()","metadata":{"_uuid":"b1a7f2dc-6cec-4610-ba22-6ad5a7fb06ac","_cell_guid":"f1a8c252-31ba-46a1-b25c-c6ffff3fa2a9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:37:59.249410Z","iopub.execute_input":"2024-06-11T02:37:59.249795Z","iopub.status.idle":"2024-06-11T02:37:59.267655Z","shell.execute_reply.started":"2024-06-11T02:37:59.249759Z","shell.execute_reply":"2024-06-11T02:37:59.266630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can see that the first colum is the label,\n# and the rest of the columns are the 784 pixels\n# that represent 28x28 gray-scale images.\n# Let's separate X and Y\n\nX_train = df_train.drop(columns=['label']).values\ny_train = df_train['label'].values\n","metadata":{"_uuid":"48085649-c225-4e39-824f-82474ea004ff","_cell_guid":"939c1008-c85e-4ee0-bdab-bcccc4a252e4","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:37:59.268903Z","iopub.execute_input":"2024-06-11T02:37:59.269241Z","iopub.status.idle":"2024-06-11T02:37:59.356051Z","shell.execute_reply.started":"2024-06-11T02:37:59.269210Z","shell.execute_reply":"2024-06-11T02:37:59.355210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's visualize the distribution of the pixel values\n\nax = sns.histplot(x=X_train.flatten(), color='#4895EE')\n_ = ax.set(title='Pixel Value Histogram', xlabel='Pixel Value')\n\n# As we can see, the 0 value represents the background\n# and the >0 values represent the digits in the images","metadata":{"_uuid":"f724afe4-d098-4f65-aaca-e97a096b8a32","_cell_guid":"36ec9986-72eb-4f04-abf8-84e2df3e3bfa","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:37:59.357155Z","iopub.execute_input":"2024-06-11T02:37:59.357510Z","iopub.status.idle":"2024-06-11T02:38:13.607081Z","shell.execute_reply.started":"2024-06-11T02:37:59.357482Z","shell.execute_reply":"2024-06-11T02:38:13.606165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's check that the target is balanced.\n# It means that the ten categories in the target\n# have roughly the same representation in the data.\n\nax = sns.countplot(x=y_train, color='#4895EE')\nax.set(\n title='Target distribution is balanced? Yes!',\n xlabel='Label'\n)\n# The expected number of samples for a balanced \n# class distribution is 42000/10 = 4200\n_ = ax.axhline(y=4200, color='#12086F', linestyle='--')\n\n# Since the target classes are balances, we don't have\n# worry about sampling methods and biases in our models.","metadata":{"_uuid":"cd488e69-2d9d-4209-ac52-f11e13ab443b","_cell_guid":"d681064d-2dbe-4c2b-8075-9ce2c254d755","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:13.608496Z","iopub.execute_input":"2024-06-11T02:38:13.608883Z","iopub.status.idle":"2024-06-11T02:38:13.953864Z","shell.execute_reply.started":"2024-06-11T02:38:13.608846Z","shell.execute_reply":"2024-06-11T02:38:13.952955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Next, let's visualize one image from each class.\n# We will use the matplotlib 'imshow' function.\n\nfig, axs = plt.subplots(nrows=2, ncols=5, figsize=(10,5))\n\nfor i in range(10):\n idx = np.where(y_train==i)[0][2]\n img = axs[int(i >= 5), i%5].imshow(\n X=X_train[idx].reshape(28, 28),\n cmap='Blues'\n )\n img.axes.get_yaxis().set_visible(False)\n img.axes.get_xaxis().set_visible(False)\n img.axes.set_title(f'Label: {i}')","metadata":{"_uuid":"9d6c909c-609f-4b5d-a907-2d75a3c17401","_cell_guid":"1fa8f38f-f989-4007-86c5-590612877157","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:13.955273Z","iopub.execute_input":"2024-06-11T02:38:13.955898Z","iopub.status.idle":"2024-06-11T02:38:14.434237Z","shell.execute_reply.started":"2024-06-11T02:38:13.955863Z","shell.execute_reply":"2024-06-11T02:38:14.433186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# transform data","metadata":{"_uuid":"dd9d9d94-7ea8-4f8d-aa91-866959d703e3","_cell_guid":"48cf8b51-1a3e-4d0e-94e5-da53b3aa7c90","trusted":true}},{"cell_type":"code","source":"# -------------------------------------------------- declare\nnumRotate_prior = 10\n\nangleOffset_prior = -20\nangleRange_prior = 40\n\nX_train_pick2rotate = {\n 'index': np.array([2500, 2501, 2502, 2503, 2504, 2505, 2506, 2507, 2508, 2509,\n 2510, 2511, 2512, 2513, 2514, 2515, 2516, 2517, 2518, 2519,\n 2520, 2521, 2522, 2523, 2524, 2525, 2526, 2527, 2528, 2529,\n 2530, 2531, 2532, 2533, 2534, 2535, 2536, 2537, 2538, 2539,\n 2540, 2541, 2542, 2543, 2544, 2545, 2546, 2547, 2548, 2549,\n 2550, 2551, 2552, 2553, 2554, 2555, 2556, 2557, 2558, 2559,\n 2560, 2561, 2562, 2563, 2564, 2565, 2566, 2567, 2568, 2569,\n 2570, 2571, 2572, 2573, 2574, 2575, 2576, 2577, 2578, 2579,\n 2580, 2581, 2582, 2583, 2584, 2585, 2586, 2587, 2588, 2589,\n 2590, 2591, 2592, 2593, 2594, 2595, 2596, 2597, 2598, 2599]),\n 'offset': np.array([\n #0\n [50,15,-10,20,20, \n 35,0,20,15,25],\n #1\n [45,18,20,-45,15, \n 10,-5,0,15,15],\n #2\n [80,10,-10,-5,20, \n 15,25,0,40,25],\n #3\n [5,10,0,-10,22, \n 0,10,-10,40,-45],\n #4\n [15,20,-15,10,10, \n 20,0,0,15,15],\n #5\n [0,5,-5,-10,5, \n 15,-20,10,15,0],\n #6\n [25,17,-7,-5,5, \n 10,40,0,-5,-19],\n #7\n [45,25,5,0,0, \n 18,-8,-5,0,0],\n #8\n [35,-8,-5,15,8, \n 0,-20,-20,-10,20],\n #9\n [20,25,-15,0,35, \n 5,5,0,25,15],\n #10\n [-5,30,22,5,5, \n 0,0,0,30,18],\n #11\n [5,25,0,5,30, \n 22,0,-5,5,0],\n #12\n [0,5,0,10,30, \n 30,-15,0,0,37],\n #13\n [40,22,5,5,30, \n 10,-5,0,0,0],\n #14\n [0,30,0,-5,0, \n 37,0,10,-5,33],\n #15\n [10,30,-10,5,-5, \n 27,0,-10,-5,0],\n #16\n [-5,-7,-18,0,0, \n 0,40,0,-8,30],\n #17\n [0,7,-48,-22,10, \n 45,32,0,16,17],\n #18\n [25,0,0,5,28, \n 40,7,0,0,0],\n #19\n [5,16,5,18,30, \n 29,35,0,45,5],\n #20\n [-40,12,-5,0,30, \n 30,-5,-30,20,-8],\n #21\n [49,25,-5,8,27, \n 5,0,0,0,0],\n #22\n [0,15,0,-5,0, \n 54,40,10,0,7],\n #23\n [45,5,-5,-30,27, \n 0,0,10,3,0],\n #24\n [30,0,-5,17,32, \n 8,0,0,10,26],\n #25\n [0,12,-10,-8,7, \n 28,28,0,15,2],\n #26\n [58,26,0,17,25, \n 0,0,-5,0,5],\n #27\n [26,-5,0,0,33, \n 20,0,12,0,15],\n #28\n [30,0,0,0,11, \n -15,0,0,30,0],\n #29\n [30,0,-10,-8,10, \n 0,0,0,0,4],\n #30\n [30,15,0,0,25, \n 0,-10,0,0,35],\n #31\n [52,44,0,-30,24, \n 0,0,64,-10,20],\n #32\n [32,-5,-5,0,20, \n -10,-10,0,0,33],\n #33\n [45,20,0,-15,-7, \n 42,0,0,20,0],\n #34\n [32,20,0,10,4, \n 28,0,-5,15,38],\n #35\n [35,22,-8,0,13, \n 30,10,17,17,25],\n #36\n [43,20,0,-5,39, \n 5,0,0,-5,15],\n #37\n [33,0,0,0,9, \n 0,0,0,0,5],\n #38\n [0,20,-8,-10,0, \n 0,-90,0,0,0],\n #39\n [30,0,20,0,20, \n -5,0,0,25,15],\n #40\n [0,21,-5,18,-5, \n 8,-50,-10,23,24],\n #41\n [40,10,0,-5,30, \n 30,0,10,5,0],\n #42\n [40,20,-5,10,4, \n 55,0,5,0,36],\n #43\n [48,0,0,28,-5, \n 28,18,0,32,30],\n #44\n [0,8,0,15,30, \n 10,-58,0,10,5],\n #45\n [33,20,0,-8,18, \n 5,0,0,5,23],\n #46\n [41,22,-10,0,10, \n 5,0,0,17,0],\n #47\n [0,0,0,18,20, \n 5,26,0,0,35],\n #48\n [-73,-5,0,-32,13, \n 0,0,-16,16,0],\n #49\n [86,0,0,-5,41, \n 0,28,0,29,15],\n #50\n [23,0,5,14,27, \n 27,-30,4,-20,22],\n #51\n [0,10,0,13,-10, \n 5,0,0,0,-12],\n #52\n [0,36,-10,5,-5, \n 0,-23,0,23,30],\n #53\n [32,35,5,5,13, \n 20,0,0,10,20],\n #54\n [68,30,0,5,32, \n 35,-56,25,20,0],\n #55\n [0,-5,0,5,18, \n 23,0,0,29,5],\n #56\n [35,15,-5,8,0, \n 41,0,0,31,0],\n #57\n [38,7,0,5,5, \n 0,-65,0,38,27],\n #58\n [0,0,-5,20,10, \n 31,26,0,-10,27],\n #59\n [0,8,-10,8,20, \n 33,0,0,5,0],\n #60\n [10,37,-5,0,0, \n 35,0,0,5,10],\n #61\n [32,23,0,0,26, \n 10,0,0,-10,-11],\n #62\n [18,15,0,-7,14, \n 40,30,7,-5,20],\n #63\n [43,30,-5,10,17, \n 40,32,0,-36,20],\n #64\n [-82,20,-5,-8,17, \n 5,-95,5,24,27],\n #65\n [0,0,0,-5,27, \n 33,0,0,20,27],\n #66\n [80,-10,0,5,-4, \n 5,0,5,25,42],\n #67\n [0,0,0,-25,16, \n -6,0,0,-5,26],\n #68\n [0,37,-5,18,8, \n 42,28,0,13,30],\n #69\n [90,3,0,0,15, \n 36,28,0,30,3],\n #70\n [35,20,0,5,16, \n 32,0,0,18,-5],\n #71\n [43,24,10,5,0, \n 0,25,0,28,6],\n #72\n [38,-3,0,-21,31, \n 5,35,0,0,27],\n #73\n [11,15,0,8,22, \n 10,-28,0,6,16],\n #74\n [45,15,-5,15,40, \n 22,5,0,27,11],\n #75\n [41,30,0,-23,27, \n -7,24,0,18,12],\n #76\n [48,-6,0,-13,16, \n 40,-30,0,5,-17],\n #77\n [23,14,-18,13,39, \n -6,-8,-10,11,20],\n #78\n [70,28,5,-14,31, \n 30,30,0,5,18],\n #79\n [30,16,-4,6,22, \n -30,0,0,4,4],\n #80\n [0,32,-4,16,-7, \n 31,0,0,14,30],\n #81\n [40,18,0,-35,6, \n 25,0,5,6,3],\n #82\n [4,-14,0,-30,4, \n 6,0,0,8,5],\n #83\n [37,4,-8,5,31, \n 38,-53,0,-13,7],\n #84\n [27,7,0,5,7, \n 15,-40,0,3,-2],\n #85\n [18,-10,0,12,17, \n 0,0,0,10,20],\n #86\n [0,17,5,0,18, \n 18,0,0,22,28],\n #87\n [0,21,-5,0,2, \n 5,48,0,8,0],\n #88\n [0,3,0,25,-3, \n 0,0,-3,0,33],\n #89\n [5,-2,5,-11,20, \n 0,12,0,0,-3],\n #90\n [60,4,-16,6,-13, \n 11,0,0,25,33],\n #91\n [0,23,-5,36,10, \n 16,30,0,2,22],\n #92\n [53,25,-15,5,17, \n -3,0,0,0,42],\n #93\n [55,47,0,-25,2, \n 25,15,0,17,15],\n #94\n [24,18,6,5,12, \n 0,8,0,16,20],\n #95\n [0,21,-5,4,3, \n 17,0,0,16,19],\n #96\n [36,-14,0,0,5, \n 35,30,-3,6,40],\n #97\n [0,30,5,20,20, \n 0,26,0,3,20],\n #98\n [-5,3,0,22,-11, \n 37,0,0,15,10],\n #99\n [27,5,0,0,30, \n 0,23,0,-11,14],\n ])\n}\n\nnumPickRotate = len(X_train_pick2rotate['index'])\n\nprint('numPickRotate = ', numPickRotate)\n\n# -------------------------------------------------- rotate\n\nX_sample_rotate = np.zeros((10,numRotate_prior,numPickRotate,X_train.shape[1]))\ny_sample_rotate = np.zeros((10,numRotate_prior,numPickRotate))\n\nfor k0 in range(10):\n for k1 in range(numRotate_prior):\n for k2 in range(numPickRotate):\n \n idxPick_X_train = X_train_pick2rotate['index'][k2]\n \n idx = np.where(y_train==k0)[0][idxPick_X_train]\n \n y_sample_rotate[k0,k1,k2] = k0 * numRotate_prior + k1\n \n angleRotate = (k1/numRotate_prior * angleRange_prior) + angleOffset_prior + X_train_pick2rotate['offset'][k2][k0]\n \n X_sample_rotate[k0,k1,k2,:] = func_array1d_rotate2d(X_train[idx][np.newaxis,:],angleRotate, 28)\n\n \nX_sample_rotate[X_sample_rotate < 10] = 0\n#X_sample_rotate[X_sample_rotate > 300] = 0\n\n# Let's normalize the pixel values to the interval [0,1]\n# Formula: MinMaxscale(X) = (X - X_min) / (X_max - X_min)\n\nprint(\"Before applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_sample_rotate.min())\nprint(\" - Maximum value:\", X_sample_rotate.max())\n\nminmax_scaler = MinMaxScaler()\n\nfor k0 in range(10):\n for k1 in range(numRotate_prior):\n X_sample_rotate[k0,k1,:,:] = minmax_scaler.fit_transform(X_sample_rotate[k0,k1,:,:]) * 255\n\nprint(\"\\nAfter applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_sample_rotate.min())\nprint(\" - Maximum value:\", X_sample_rotate.max())\n\n\n\n# Next, let's visualize one image from each class.\n# We will use the matplotlib 'imshow' function.\n\n# for k2 in range(numPickRotate):\nfor k2 in range(5,10):\n k1 = 0\n # k1 = numRotate_prior - 1\n\n fig, axs = plt.subplots(nrows=2, ncols=5, figsize=(10,5))\n\n for k0 in range(10):\n img = axs[int(k0 >= 5), k0%5].imshow(\n X=X_sample_rotate[k0,k1,k2,:].reshape(28, 28),\n cmap='Blues'\n )\n img.axes.get_yaxis().set_visible(False)\n img.axes.get_xaxis().set_visible(False)\n img.axes.set_title(f'Label: {k0}')\n \nX_train_rotate = np.reshape(X_sample_rotate, (-1,784))\ny_train_rotate = np.reshape(y_sample_rotate, (-1,1)).flatten()","metadata":{"_uuid":"2fd8cd6b-8db8-45dc-b5f7-6721c80f11fd","_cell_guid":"6b163e1f-7fee-43a6-b6f6-40c0d402a4fc","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:14.435461Z","iopub.execute_input":"2024-06-11T02:38:14.435748Z","iopub.status.idle":"2024-06-11T02:38:20.775118Z","shell.execute_reply.started":"2024-06-11T02:38:14.435723Z","shell.execute_reply":"2024-06-11T02:38:20.774108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------\n# Phần 3: Phân tích thành phần chính","metadata":{"_uuid":"f67ae6c7-11f8-4b88-a23e-bf62fb9fdb43","_cell_guid":"6e9093b4-f586-4545-b99e-b59d492b7958","trusted":true}},{"cell_type":"markdown","source":"\n Principal Comopnent Analysis (PCA) is a statistical technique for reducing the dimensionality of a dataset. This is accomplished by linearly transforming the data into a new coordinate system where (most of) the variation in the data can be described with fewer dimensions than the initial data.
\nThe first principal component can equivalently be defined as a direction that maximizes the variance of the projected data. The i-th principal component can be taken as a direction orthogonal to the first i-1 principal components that maximizes the variance of the projected data.
\nsklearn.decomposition.PCA: Linear dimensionality reduction using Singular Value Decomposition of the data to project it to a lower dimensional space.
\nHere we will use the top 10 principal components, which explain almost 50% of the variance in the data. This means that we can express 50% of the variance of the data with a space of features >98% smaller than the original one (10 features vs. 784 features)\n
","metadata":{"_uuid":"c9145741-6e7a-4419-bdb0-baca414ed4e7","_cell_guid":"54390ba7-75cb-4867-b577-97f23c85d6f2","trusted":true}},{"cell_type":"code","source":"# Let's normalize the pixel values to the interval [0,1]\n# Formula: MinMaxscale(X) = (X - X_min) / (X_max - X_min)\n\nprint(\"Before applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_train.min())\nprint(\" - Maximum value:\", X_train.max())\n\nminmax_scaler = MinMaxScaler()\nX_train_rotate_scaled = minmax_scaler.fit_transform(X_train_rotate)\n\nprint(\"\\nAfter applying MinMax Scaler:\")\nprint(\" - Minimum value:\", X_train_rotate_scaled.min())\nprint(\" - Maximum value:\", X_train_rotate_scaled.max())","metadata":{"_uuid":"aad53bf6-9b1f-4451-844d-ae51f509a68f","_cell_guid":"2f528db2-3b92-425b-a18c-2168dd8862d7","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:20.777346Z","iopub.execute_input":"2024-06-11T02:38:20.778283Z","iopub.status.idle":"2024-06-11T02:38:20.902777Z","shell.execute_reply.started":"2024-06-11T02:38:20.778240Z","shell.execute_reply":"2024-06-11T02:38:20.901774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Actually, we will embed the MinMaxScaler and the PCA\n# modules into a Sklearn Pipeline.\n\npca = Pipeline(steps=[\n ('scaler', MinMaxScaler()),\n ('pca', PCA(n_components=40))\n])\n\nX_train_rotate_pca = pca.fit_transform(X=X_train_rotate)\n\nprint('X_train_pca shape:', X_train_rotate_pca.shape)","metadata":{"_uuid":"6cae5170-3af3-4ed9-a9d8-372ef224e9ba","_cell_guid":"e423b865-ef1d-4c48-9152-917f8b258060","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:20.903871Z","iopub.execute_input":"2024-06-11T02:38:20.904160Z","iopub.status.idle":"2024-06-11T02:38:21.987683Z","shell.execute_reply.started":"2024-06-11T02:38:20.904138Z","shell.execute_reply":"2024-06-11T02:38:21.986333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's extract the explained variance ratio\n# And let's compute their cumulative sum\n\npca_object = pca.named_steps['pca']\nvar_ratio = pd.Series(\n data=pca_object.explained_variance_ratio_,\n index=range(1, 41) #41\n)\n\ncumsum_var_ratio = var_ratio.cumsum()","metadata":{"_uuid":"68b1ffab-f2de-43fe-a655-e04c6060ec60","_cell_guid":"c3cdf97b-6136-45e0-94c6-bdf23e750bf0","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:22.004079Z","iopub.execute_input":"2024-06-11T02:38:22.005925Z","iopub.status.idle":"2024-06-11T02:38:22.020035Z","shell.execute_reply.started":"2024-06-11T02:38:22.005878Z","shell.execute_reply":"2024-06-11T02:38:22.018629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the explained variance ratio for each component (left)\n# And plot their cumulative sum (right)\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))\n\nsns.barplot(\n x=var_ratio.index, \n y=var_ratio.values, \n color='#4895EE', \n ax=ax1\n)\nax1.set(\n title='Explained Variance Ratio PCA',\n xlabel='Component',\n ylabel='Var Ratio',\n)\nax1.set_ylim(0, 0.5)\n\nsns.barplot(\n x=cumsum_var_ratio.index, \n y=cumsum_var_ratio.values, \n color='#4895EE', \n ax=ax2\n)\nax2.set(\n title='Cumulative Explained Variance Ratio PCA',\n xlabel='Component',\n ylabel='Cumulative Var Ratio',\n)\n_ = ax2.set_ylim(0, 0.5)","metadata":{"_uuid":"9ea36679-af0c-40f1-bda2-641ef9c73a4e","_cell_guid":"6fe44c61-1ac9-4157-968d-022d42184ff6","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:22.022395Z","iopub.execute_input":"2024-06-11T02:38:22.024178Z","iopub.status.idle":"2024-06-11T02:38:24.536448Z","shell.execute_reply.started":"2024-06-11T02:38:22.024131Z","shell.execute_reply":"2024-06-11T02:38:24.535558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's put the components in Pandas DataFrame format\n# This way we can use the seaborn and plotly libraries\n# to visualize the data.\n# (these packages work better with Pandas DataFrames)\n\ndf_train_pca = pd.DataFrame(\n data=X_train_rotate_pca[:,:3], \n columns=['x', 'y', 'z']\n)\n# Let's also add the target to set the colors\ndf_train_pca['target'] = y_train_rotate\n\nprint(df_train_pca.shape)\ndf_train_pca.head()","metadata":{"_uuid":"9796e066-c6af-49c5-af8f-8ebde5e07430","_cell_guid":"4fb22220-cc87-410b-a637-55bc57791987","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:24.537874Z","iopub.execute_input":"2024-06-11T02:38:24.538586Z","iopub.status.idle":"2024-06-11T02:38:24.552973Z","shell.execute_reply.started":"2024-06-11T02:38:24.538549Z","shell.execute_reply":"2024-06-11T02:38:24.552057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Two-dimensional visualization of the data\n# Here the Top2 principal components are plotted\n\nsns.set_style(\"dark\")\nax = sns.scatterplot(\n data=df_train_pca,\n x='x', \n y='y', \n hue='target', \n palette='Blues',\n)\nax.set(\n title='PCA Two-Dimensional Projection',\n xlabel='1st Comp.',\n ylabel='2nd Comp.'\n)\nax.axhline(y=0, linestyle='--', color='#FFBF00', zorder=-1)\nax.axvline(x=0, linestyle='--', color='#FFBF00', zorder=-1)","metadata":{"_uuid":"30135dee-d7d0-4dc8-88f7-3a88a135ad5a","_cell_guid":"147fd5ee-e632-4611-b6de-35d64c6946e9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:24.554546Z","iopub.execute_input":"2024-06-11T02:38:24.554803Z","iopub.status.idle":"2024-06-11T02:38:25.941245Z","shell.execute_reply.started":"2024-06-11T02:38:24.554780Z","shell.execute_reply":"2024-06-11T02:38:25.940330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, let's apply the inverse_transform to compute\n# the approximation of images based on the 10 principal\n# compontents.\n# We can see that with just 10 components we convey a large\n# part of the information of each image.\n\npca_object = pca.named_steps['pca']\n\nfig, axs = plt.subplots(nrows=10, ncols=2, figsize=(6,20))\n\nfor i in range(10):\n idx = np.where(y_train==i)[0][0]\n original_img = axs[i, 0].imshow(\n X=X_train_rotate[idx].reshape(28, 28),\n cmap='Blues'\n )\n original_img.axes.get_yaxis().set_visible(False)\n original_img.axes.get_xaxis().set_visible(False)\n original_img.axes.set_title(f'Original [{i}]:')\n \n approx_img_ = pca_object.inverse_transform(X=X_train_rotate_pca[idx])\n approx_img = axs[i, 1].imshow(\n X=approx_img_.reshape(28, 28),\n cmap='Blues'\n )\n approx_img.axes.get_yaxis().set_visible(False)\n approx_img.axes.get_xaxis().set_visible(False)\n approx_img.axes.set_title(f'Approximation [{i}]:')","metadata":{"_uuid":"77146579-275e-4359-8ec9-7d175da30220","_cell_guid":"329c95b3-2406-402c-b1f8-38487c4fa2f3","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:25.942540Z","iopub.execute_input":"2024-06-11T02:38:25.942830Z","iopub.status.idle":"2024-06-11T02:38:26.917691Z","shell.execute_reply.started":"2024-06-11T02:38:25.942805Z","shell.execute_reply":"2024-06-11T02:38:26.916500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show that the PCA components are uncorrelated\n# (This statement is true by definition and construction\n# of the Principal Component Analysis algorithm)\n\ncorr = np.corrcoef(x=X_train_rotate_pca, rowvar=False)\ncorr = np.around(corr, decimals=2)\n\nax = sns.heatmap(\n data=corr, \n vmax=1, \n vmin=-1,\n square=True,\n linewidths=.5,\n annot=True,\n cmap='Blues',\n)\n\n_ = ax.set(\n title='Correlation PCA Components [Uncorrelated]',\n xlabel='Components',\n ylabel='Components'\n)","metadata":{"_uuid":"8c3fa8bb-61ab-43dc-bc7e-e5fecc13e6e2","_cell_guid":"ab321270-3628-423b-8fd6-a7ebaf6ccfe5","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:26.919090Z","iopub.execute_input":"2024-06-11T02:38:26.919548Z","iopub.status.idle":"2024-06-11T02:38:30.947278Z","shell.execute_reply.started":"2024-06-11T02:38:26.919501Z","shell.execute_reply":"2024-06-11T02:38:30.946288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"----\n# Phần 4: Sử dụng K-Nearest Neighbors Classifier phân loại 10 thành phần chính hàng đầu từ PCA.","metadata":{"_uuid":"3d69b1a4-faf3-4405-b740-9d5fc82b8424","_cell_guid":"16cefed3-22ad-4c84-ad87-230ed47df34d","trusted":true}},{"cell_type":"markdown","source":"\nThe K-Nearest Neighbors algorithm is a non-parametric, supervised learning classifier, which uses proximity to make classifications or predictions about the grouping of an individual data point. Prediction: A class label is assigned on the basis of a majority vote. the label that is most frequently represented around a given data point.
\nIn this project, the input of the KNN Classifier will be the top 10 principal components from the PCA.
\nsklearn.neighbors.KNeighborsClassifier: Classifier implementing the k-nearest neighbors vote. We will run a GridSearchCV to optimize the n_neighbors hyperparameter.\n
","metadata":{"_uuid":"1abd82c5-f4e1-4142-acb2-bea45e42488e","_cell_guid":"b9e0603d-4dc8-4761-baad-a34cf6709975","trusted":true}},{"cell_type":"markdown","source":"---\n# mô hình học máy trên tập dữ liệu Train","metadata":{"_uuid":"334a6d45-a6b2-499e-a95b-2477800b968f","_cell_guid":"e1b5a3d3-ced9-409c-b6f9-b07e7143bc66","trusted":true}},{"cell_type":"code","source":"knn_clf = Pipeline(steps=[\n ('scaler', MinMaxScaler()),\n ('pca', PCA(n_components=23)), #10\n ('knn', KNeighborsClassifier())\n])\n\nknn_clf_params = {\n 'knn__n_neighbors': [2, 4, 8, 16, 32]\n}\n\nknn_clf_gs = GridSearchCV(\n estimator=knn_clf, \n param_grid=knn_clf_params,\n scoring='accuracy',\n n_jobs=-1,\n cv=5,\n verbose=1,\n return_train_score=True\n)\n\nknn_clf_gs.fit(X=X_train_rotate_pca, y=y_train_rotate)\n\nprint('best PCA+KNN params:', knn_clf_gs.best_params_)\nprint('best PCA+KNN cv accuracy:', knn_clf_gs.best_score_)","metadata":{"_uuid":"7154a244-1406-4e55-9062-2a876ffcfb42","_cell_guid":"f8b23485-498b-446c-83a4-fdf1e50b8285","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:30.948562Z","iopub.execute_input":"2024-06-11T02:38:30.948893Z","iopub.status.idle":"2024-06-11T02:38:42.097052Z","shell.execute_reply.started":"2024-06-11T02:38:30.948866Z","shell.execute_reply":"2024-06-11T02:38:42.094821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the Training and Validation Accuracies\n# for each Hyperparamter value tested during the\n# Grid Search.\n# This plot allows us to check for overfitting.\n\nfig, ax = plt.subplots()\n\nax.plot(\n range(len(knn_clf_params['knn__n_neighbors'])), \n knn_clf_gs.cv_results_['mean_train_score'],\n label='train',\n color='blue'\n)\nax.plot(\n range(len(knn_clf_params['knn__n_neighbors'])),\n knn_clf_gs.cv_results_['mean_test_score'],\n label='test',\n color='orange'\n)\nax.set_xticks(\n ticks=range(len(knn_clf_params['knn__n_neighbors'])),\n labels=knn_clf_params['knn__n_neighbors']\n)\n\nii = knn_clf_params['knn__n_neighbors'].index(\n knn_clf_gs.best_params_['knn__n_neighbors']\n)\n\nplt.axvline(x=ii, linestyle='--', color='red', label='best', alpha=0.3)\nplt.legend()\nplt.xlabel(\"'n_neighbors' vals\")\nplt.ylabel('Accuracy')\nplt.title(\"PCA+KNN GridSearchCV Results\")\nplt.show()","metadata":{"_uuid":"43e1d701-a89a-463f-b860-3afbf14bbf36","_cell_guid":"b1316942-7e1c-4f06-8742-d3eb8f7918c3","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:42.100794Z","iopub.execute_input":"2024-06-11T02:38:42.103009Z","iopub.status.idle":"2024-06-11T02:38:42.528108Z","shell.execute_reply.started":"2024-06-11T02:38:42.102965Z","shell.execute_reply":"2024-06-11T02:38:42.527145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Phần 5: Tạo một tệp tin chứa các dự đoán của mô hình học máy trên tập dữ liệu Test","metadata":{"_uuid":"b609f283-8a5c-477c-9c29-941acfedc9aa","_cell_guid":"5d9eb764-a90a-48b5-b962-be1b347e4925","trusted":true}},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score\n\n# Compute and display the Confusion Matrix\n# Note: the entire training dataset is used here\n\nX_val = X_train_rotate_pca\nY_val = y_train_rotate\n\nbest_cv_clf = knn_clf_gs.best_estimator_\ny_pred = best_cv_clf.predict(X=X_val)\n\ny_val_number = Y_val // numRotate_prior\ny_pred_number = y_pred // numRotate_prior\n\ny_val_angle = (Y_val % numRotate_prior)\ny_pred_angle = (y_pred % numRotate_prior)\n\nprint('KNN number + angle, Acc({:.2f}%)', accuracy_score(Y_val, y_pred))\nprint('KNN number, Acc({:.2f}%)', accuracy_score(y_val_number, y_pred_number))\nprint('KNN angle, Acc({:.2f}%)', accuracy_score(y_val_angle, y_pred_angle))\n\n# print('y_val_number =', y_val_number)\n# print('y_pred_number =', y_pred_number)\n\ny_val_angle_degree = y_val_angle/numRotate_prior * angleRange_prior + angleOffset_prior\ny_pred_angle_degree = y_pred_angle/numRotate_prior * angleRange_prior + angleOffset_prior\n\n# print('y_val_angle_degree =', y_val_angle_degree)\n# print('y_pred_angle_degree =', y_pred_angle_degree)\n\nconf_matrix = confusion_matrix(y_val_number, y_pred_number)\nconf_matrix_disp = ConfusionMatrixDisplay(conf_matrix)\nconf_matrix_disp.plot(cmap='Blues')\n\nconf_matrix = confusion_matrix(y_val_angle, y_pred_angle)\nconf_matrix_disp = ConfusionMatrixDisplay(conf_matrix)\nconf_matrix_disp.plot(cmap='Blues')","metadata":{"_uuid":"62d7493a-d5c0-42ca-ad56-df419c3367ee","_cell_guid":"fdfdc4ce-7aa6-4c8a-b7c9-8a069cb98054","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:42.529364Z","iopub.execute_input":"2024-06-11T02:38:42.529666Z","iopub.status.idle":"2024-06-11T02:38:44.580269Z","shell.execute_reply.started":"2024-06-11T02:38:42.529640Z","shell.execute_reply":"2024-06-11T02:38:44.579368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_val = pca.transform(X_train)\nY_val = y_train\n\nbest_cv_clf = knn_clf_gs.best_estimator_\ny_pred = best_cv_clf.predict(X=X_val)\n\ny_pred_number = y_pred\ny_val_number = Y_val\n\nprint('KNN number, Acc({:.2f}%)', accuracy_score(y_val_number, y_pred_number))\n\nconf_matrix = confusion_matrix(y_val_number, y_pred_number)\nconf_matrix_disp = ConfusionMatrixDisplay(conf_matrix)\nconf_matrix_disp.plot(cmap='Blues')\n\ny_pred_angle = (y_pred % numRotate_prior)\n\ny_pred_angle_degree = y_pred_angle/numRotate_prior * angleRange_prior + angleOffset_prior\n\nprint('y_pred_angle_degree =', y_pred_angle_degree)","metadata":{"_uuid":"da419e38-d545-49ee-a8b6-32f3fa9760c3","_cell_guid":"fa5652c2-d596-4880-8396-b6e77177f4ff","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:38:44.581753Z","iopub.execute_input":"2024-06-11T02:38:44.582120Z","iopub.status.idle":"2024-06-11T02:39:09.784884Z","shell.execute_reply.started":"2024-06-11T02:38:44.582086Z","shell.execute_reply":"2024-06-11T02:39:09.783861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, let's apply the inverse_transform to compute\n# the approximation of images based on the 10 principal\n# compontents.\n# We can see that with just 10 components we convey a large\n# part of the information of each image.\n\npca_object = pca.named_steps['pca']\n\nfig, axs = plt.subplots(nrows=10, ncols=2, figsize=(6,20))\n\nfor i in range(10):\n idx = np.where(y_train == i )[0][0]\n original_img = axs[i, 0].imshow(\n X=X_train[idx].reshape(28, 28),\n cmap='Blues'\n )\n original_img.axes.get_yaxis().set_visible(False)\n original_img.axes.get_xaxis().set_visible(False)\n original_img.axes.set_title(f'Original [{i}]:')\n \n approx_img_ = pca_object.inverse_transform(X=X_val[idx])\n approx_img = axs[i, 1].imshow(\n X=approx_img_.reshape(28, 28),\n cmap='Blues'\n )\n approx_img.axes.get_yaxis().set_visible(False)\n approx_img.axes.get_xaxis().set_visible(False)\n approx_img.axes.set_title(f'Predicted angle [{y_pred_angle_degree[idx]}]:')","metadata":{"_uuid":"bc4e4c18-00b8-4441-900b-83786685d798","_cell_guid":"1393d26a-3a87-4a77-a642-c5b68c94fec3","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:39:09.786202Z","iopub.execute_input":"2024-06-11T02:39:09.786519Z","iopub.status.idle":"2024-06-11T02:39:10.769173Z","shell.execute_reply.started":"2024-06-11T02:39:09.786492Z","shell.execute_reply":"2024-06-11T02:39:10.768183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\nLight CNN Classifier\n
\n\n
![](\"https://taewanmerepo.github.io/2018/01/cnn/head.png\")
","metadata":{"_uuid":"5a238d9f-7fbe-4d2a-85c7-b38f1793fe47","_cell_guid":"d606a8da-01ba-4650-b234-842037898d97","trusted":true}},{"cell_type":"markdown","source":"### CNN is a very efficient ways for computer vision tasks. \na convolutional neural network (CNN, or ConvNet) is a class of Artificial Neural Network (ANN), most commonly applied to analyze visual imagery. They are also known as Shift Invariant or Space Invariant Artificial Neural Networks (SIANN), based on the shared-weight architecture of the convolution kernels or filters that slide along input features and provide translation equivariant responses known as feature maps. Counter-intuitively, most convolutional neural networks are only equivariant, as opposed to invariant, to translation. They have applications in image and video recognition, recommender systems, image classification, image segmentation, medical image analysis, natural language processing, brain-computer interfaces, and financial time series.","metadata":{"_uuid":"3f62053b-0364-4ff6-bb7b-5deba2a02280","_cell_guid":"1c349770-f9d8-4f7f-8efb-d87b402f2f10","trusted":true}},{"cell_type":"code","source":"import plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objects as go\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2024-06-11T02:39:10.770514Z","iopub.execute_input":"2024-06-11T02:39:10.770816Z","iopub.status.idle":"2024-06-11T02:39:10.775768Z","shell.execute_reply.started":"2024-06-11T02:39:10.770789Z","shell.execute_reply":"2024-06-11T02:39:10.774871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_confusion_matrix(cm, labels, title):\n # cm : confusion matrix list(list)\n # labels : name of the data list(str)\n # title : title for the heatmap\n data = go.Heatmap(z=cm, y=labels, x=labels)\n annotations = []\n for i, row in enumerate(cm):\n for j, value in enumerate(row):\n annotations.append(\n {\n \"x\": labels[i],\n \"y\": labels[j],\n \"font\": {\"color\": \"white\"},\n \"text\": str(value),\n \"xref\": \"x1\",\n \"yref\": \"y1\",\n \"showarrow\": False\n }\n )\n layout = {\n \"title\": title,\n \"xaxis\": {\"title\": \"Predicted value\"},\n \"yaxis\": {\"title\": \"Real value\"},\n \"annotations\": annotations\n }\n fig = go.Figure(data=data, layout=layout)\n return fig","metadata":{"execution":{"iopub.status.busy":"2024-06-11T02:39:10.776967Z","iopub.execute_input":"2024-06-11T02:39:10.777282Z","iopub.status.idle":"2024-06-11T02:39:10.790176Z","shell.execute_reply.started":"2024-06-11T02:39:10.777255Z","shell.execute_reply":"2024-06-11T02:39:10.789235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from tensorflow.keras.utils import np_utils\nfrom tensorflow.keras import utils\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\nfrom tensorflow.keras import datasets, layers, models","metadata":{"_uuid":"17fc4707-d986-4950-b0fe-ac152932ac00","_cell_guid":"0d0efe69-4e03-4204-86d8-3ee2e5125830","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:39:10.791282Z","iopub.execute_input":"2024-06-11T02:39:10.791622Z","iopub.status.idle":"2024-06-11T02:39:10.808137Z","shell.execute_reply.started":"2024-06-11T02:39:10.791598Z","shell.execute_reply":"2024-06-11T02:39:10.807183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_cnn = X_train_rotate.reshape(-1,28,28,1)\n# X_val = np.array(pca.transform(X_train)).reshape(-1,28,28,1)\ny_train_cnn = utils.to_categorical(np.array(y_train_rotate// numRotate_prior))\nX_val = X_train.reshape(-1,28,28,1)\nY_val = utils.to_categorical(np.array(y_train))","metadata":{"_uuid":"598e24f0-4665-4486-ba5d-fb550022bc96","_cell_guid":"b20ca6f3-b288-42fe-8210-9e804c27268e","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:39:10.809322Z","iopub.execute_input":"2024-06-11T02:39:10.809606Z","iopub.status.idle":"2024-06-11T02:39:10.827338Z","shell.execute_reply.started":"2024-06-11T02:39:10.809582Z","shell.execute_reply":"2024-06-11T02:39:10.826376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_cnn.shape","metadata":{"execution":{"iopub.status.busy":"2024-06-11T02:39:10.828529Z","iopub.execute_input":"2024-06-11T02:39:10.828847Z","iopub.status.idle":"2024-06-11T02:39:10.843472Z","shell.execute_reply.started":"2024-06-11T02:39:10.828822Z","shell.execute_reply":"2024-06-11T02:39:10.842494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# y_train_rotate// numRotate_prior","metadata":{"execution":{"iopub.status.busy":"2024-06-11T02:39:10.844785Z","iopub.execute_input":"2024-06-11T02:39:10.845412Z","iopub.status.idle":"2024-06-11T02:39:10.853442Z","shell.execute_reply.started":"2024-06-11T02:39:10.845378Z","shell.execute_reply":"2024-06-11T02:39:10.852695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_train_rotate = np.array(X_train_rotate).reshape(-1,28,28,1)\n# # X_val = np.array(pca.transform(X_train)).reshape(-1,28,28,1)\n# X_val = np.array(X_test_crop).reshape(-1,28,28,1)\n# y_train_rotate = utils.to_categorical(np.array(y_train_rotate))\n# Y_val = utils.to_categorical(np.array(y_test_crop))","metadata":{"_uuid":"f978abbd-867f-46a5-b6ec-35f64e740f64","_cell_guid":"9d27fd4b-f9a8-4c46-a856-7e8970246998","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:39:10.854472Z","iopub.execute_input":"2024-06-11T02:39:10.854733Z","iopub.status.idle":"2024-06-11T02:39:10.864026Z","shell.execute_reply.started":"2024-06-11T02:39:10.854710Z","shell.execute_reply":"2024-06-11T02:39:10.863274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model = models.Sequential()\n#model.add(layers.Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n# activation ='relu', input_shape = (28,28,1)))\n#model.add(layers.Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n# activation ='relu'))\n#model.add(layers.MaxPool2D(pool_size=(2,2)))\n#model.add(layers.BatchNormalization())\n#model.add(layers.Dropout(0.25))\n#model.add(layers.Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n# activation ='relu'))\n#model.add(layers.Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n# activation ='relu'))\n#model.add(layers.MaxPool2D(pool_size=(2,2), strides=(2,2)))\n#model.add(layers.BatchNormalization())\n#model.add(layers.Dropout(0.25))\n#model.add(layers.Flatten())\n#model.add(layers.Dense(256, activation = \"relu\"))\n#model.add(layers.Dropout(0.5))\n#model.add(layers.BatchNormalization())\n#model.add(layers.Dense(10, activation = \"softmax\"))\n\n#model.summary()\n\n\n\n\nmodel = models.Sequential()\n\nmodel.add(layers.Conv2D(filters = 16, kernel_size = (3, 3), activation='relu',\n input_shape = (28, 28, 1)))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.Conv2D(filters = 16, kernel_size = (3, 3), activation='relu'))\nmodel.add(layers.BatchNormalization())\n#model.add(Conv2D(filters = 16, kernel_size = (3, 3), activation='relu'))\n#model.add(BatchNormalization())\nmodel.add(layers.MaxPool2D(strides=(2,2)))\nmodel.add(layers.Dropout(0.25))\n\nmodel.add(layers.Conv2D(filters = 32, kernel_size = (3, 3), activation='relu'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.Conv2D(filters = 32, kernel_size = (3, 3), activation='relu'))\nmodel.add(layers.BatchNormalization())\n#model.add(Conv2D(filters = 32, kernel_size = (3, 3), activation='relu'))\n#model.add(BatchNormalization())\nmodel.add(layers.MaxPool2D(strides=(2,2)))\nmodel.add(layers.Dropout(0.25))\n\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.25))\nmodel.add(layers.Dense(1024, activation='relu'))\nmodel.add(layers.Dropout(0.5))\nmodel.add(layers.Dense(10, activation='softmax'))\nmodel.summary()","metadata":{"_uuid":"d30da279-3261-4e97-9b79-6e2a01d703d4","_cell_guid":"43ead369-05c5-4e2d-960c-d5773b8a7f7d","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:39:10.865195Z","iopub.execute_input":"2024-06-11T02:39:10.865492Z","iopub.status.idle":"2024-06-11T02:39:11.073787Z","shell.execute_reply.started":"2024-06-11T02:39:10.865458Z","shell.execute_reply":"2024-06-11T02:39:11.072843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.optimizers import Adam\nimport tensorflow as tf\nmodel_path = './best_model.h5'\n\ncheckpointer = ModelCheckpoint(\n filepath=model_path,\n monitor='val_accuracy',\n verbose=1,\n mode='max',\n save_best_only=True)\n\ndatagen = ImageDataGenerator(\n rotation_range=10, \n zoom_range = 0.10, \n width_shift_range=0.1, \n height_shift_range=0.1)\n\n# # model.optimizer.get_config()\n# datagen = ImageDataGenerator(zoom_range = 0.1,\n# height_shift_range = 0.1,\n# width_shift_range = 0.1,\n# rotation_range = 10)\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)\n\n\nmodel.compile(loss='categorical_crossentropy', optimizer = Adam(lr=1e-4), metrics=[\"accuracy\"])\nhistory = model.fit(\n datagen.flow(X_train_cnn, y_train_cnn, batch_size=32),\n steps_per_epoch=len(X_train_cnn) // 32,\n epochs=50, # Increase this when not on Kaggle kernel\n verbose=1, # 1 for ETA, 0 for silent\n validation_data=(X_val, Y_val), # For speed\n callbacks=[annealer, checkpointer]\n)","metadata":{"execution":{"iopub.status.busy":"2024-06-11T02:39:11.075144Z","iopub.execute_input":"2024-06-11T02:39:11.075812Z","iopub.status.idle":"2024-06-11T02:45:08.696831Z","shell.execute_reply.started":"2024-06-11T02:39:11.075777Z","shell.execute_reply":"2024-06-11T02:45:08.695989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model_path = './best_model.h5'\n#checkpointer = ModelCheckpoint(\n# filepath=model_path,\n# monitor='val_accuracy',\n# verbose=1,\n# mode='max',\n# save_best_only=True)\n\n#model.compile(loss='categorical_crossentropy',\n# optimizer='adam',\n# metrics=['accuracy'])","metadata":{"_uuid":"d131aa49-918d-43dd-be19-62d3110ca17a","_cell_guid":"db889d42-b8bb-4667-8f07-b831a3fbcf38","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:45:08.698221Z","iopub.execute_input":"2024-06-11T02:45:08.698537Z","iopub.status.idle":"2024-06-11T02:45:08.703129Z","shell.execute_reply.started":"2024-06-11T02:45:08.698510Z","shell.execute_reply":"2024-06-11T02:45:08.702176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#history = model.fit(X_train_cnn, y_train_cnn, validation_data = (X_val, Y_val), epochs=50, batch_size = 128, callbacks=[checkpointer])","metadata":{"_uuid":"620687bc-89d9-40dc-b133-006fef4e7c78","_cell_guid":"cd8420f8-3c35-407f-93de-b1691d1b534f","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:45:08.704206Z","iopub.execute_input":"2024-06-11T02:45:08.704553Z","iopub.status.idle":"2024-06-11T02:45:08.722057Z","shell.execute_reply.started":"2024-06-11T02:45:08.704525Z","shell.execute_reply":"2024-06-11T02:45:08.721339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport numpy as np\n\nfig = go.Figure()\n\nlabels = ['Loss', 'Accuracy']\ncolors = ['rgb(67,67,67)', 'rgb(49,130,189)']\n\nmode_size = [8, 8]\nline_size = [2, 2]\n\nx_data = np.vstack((np.arange(1, 51),)*2)\n\ny_data = np.array([\n np.array(history.history['loss']), \n np.array(history.history['accuracy'])\n])\n\nfor i in range(0, 2):\n fig.add_trace(go.Scatter(x=x_data[i], y=y_data[i], mode='lines',\n name=labels[i],\n line=dict(color=colors[i], width=line_size[i]),\n connectgaps=True,\n ))\n\n # endpoints\n fig.add_trace(go.Scatter(\n x=[x_data[i][0], x_data[i][-1]],\n y=[y_data[i][0], y_data[i][-1]],\n mode='markers',\n marker=dict(color=colors[i], size=mode_size[i])\n ))\n\nfig.update_layout(\n xaxis=dict(\n showline=True,\n showgrid=False,\n showticklabels=True,\n linecolor='rgb(204, 204, 204)',\n linewidth=2,\n ticks='outside',\n tickfont=dict(\n family='Arial',\n size=12,\n color='rgb(82, 82, 82)',\n ),\n ),\n yaxis=dict(\n showgrid=False,\n zeroline=False,\n showline=False,\n showticklabels=False,\n ),\n autosize=False,\n margin=dict(\n autoexpand=False,\n l=100,\n r=20,\n t=110,\n ),\n showlegend=False,\n plot_bgcolor='white'\n)\n\n#annotations = []\n\n# Adding labels\n#for y_trace, label, color in zip(y_data, labels, colors):\n # labeling the left_side of the plot\n# annotations.append(dict(xref='paper', x=0.05, y=y_trace[0],\n# xanchor='right', yanchor='middle',\n# text=label + ' {:.2f}'.format(y_trace[0]),\n # font=dict(family='Arial',\n# size=16),\n# showarrow=False))\n # labeling the right_side of the plot\n# annotations.append(dict(xref='paper', x=0.95, y=y_trace[49],\n # xanchor='left', yanchor='middle',\n # text='{:.4f}%'.format(y_trace[49]),\n # font=dict(family='Arial',\n # size=16),\n # showarrow=False))\n# Title\n#annotations.append(dict(xref='paper', yref='paper', x=0.0, y=1.05,\n# xanchor='left', yanchor='bottom',\n# text='Training Loss & Accuracy',\n# font=dict(family='Arial',\n# size=30,\n# color='rgb(37,37,37)'),\n# showarrow=False))\n# Source\n#annotations.append(dict(xref='paper', yref='paper', x=0.5, y=-0.1,\n# xanchor='center', yanchor='top',\n# text='Epochs',\n# font=dict(family='Arial',\n# size=12,\n# color='rgb(150,150,150)'),\n# showarrow=False))\n\n#fig.update_layout(annotations=annotations,height=600, width=1000)\n\n#fig.show()\n################################\n\nannotations = []\n\n# Adding labels\nfor y_trace, label, color in zip(y_data, labels, colors):\n # labeling the left_side of the plot\n annotations.append(dict(xref='paper', x=0.05, y=y_trace[0],\n xanchor='right', yanchor='middle',\n text=label + ' {:.2f}'.format(y_trace[0]),\n font=dict(family='Arial',\n size=16),\n showarrow=False))\n # labeling the right_side of the plot\n annotations.append(dict(xref='paper', x=0.95, y=y_trace[-1], # Use -1 to get the last element\n xanchor='left', yanchor='middle',\n text='{:.4f}%'.format(y_trace[-1]), # Use -1 here too\n font=dict(family='Arial',\n size=16),\n showarrow=False))\n# Title\nannotations.append(dict(xref='paper', yref='paper', x=0.0, y=1.05,\n xanchor='left', yanchor='bottom',\n text='Training Loss & Accuracy',\n font=dict(family='Arial',\n size=30,\n color='rgb(37,37,37)'),\n showarrow=False))\n# Source\nannotations.append(dict(xref='paper', yref='paper', x=0.5, y=-0.1,\n xanchor='center', yanchor='top',\n text='Epochs',\n font=dict(family='Arial',\n size=12,\n color='rgb(150,150,150)'),\n showarrow=False))\n\nfig.update_layout(annotations=annotations, height=600, width=1000)\n\nfig.show()\n","metadata":{"_uuid":"b2a98a55-6032-40d1-8bbf-bf6b35840730","_cell_guid":"7fcbf806-1db3-45b9-8bc0-3abf08dfabd9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:45:08.723478Z","iopub.execute_input":"2024-06-11T02:45:08.723843Z","iopub.status.idle":"2024-06-11T02:45:08.767767Z","shell.execute_reply.started":"2024-06-11T02:45:08.723811Z","shell.execute_reply":"2024-06-11T02:45:08.766869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"c51476fa-c9d6-4cbc-a5ad-f6ff9706d70e","_cell_guid":"8bdfa1eb-cd3b-4e48-a377-02b2b66506e1","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.models import load_model\nmodel = load_model('./best_model.h5')\ny_pred = model.predict(X_val)\nM = confusion_matrix(np.argmax(Y_val,1), np.argmax(y_pred,1))\nplot_confusion_matrix(M, sorted(np.unique(np.argmax(Y_val,1))), 'Simple CNN Confusion Matrix, Acc({:.2f}%)'.format(accuracy_score(np.argmax(Y_val,1), np.argmax(y_pred,1))*100))","metadata":{"_uuid":"57a8380a-11c2-45fb-8190-4e9da5bce1ed","_cell_guid":"bdeca8dc-2a69-4598-b508-873212a964dd","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-06-11T02:45:08.768938Z","iopub.execute_input":"2024-06-11T02:45:08.769276Z","iopub.status.idle":"2024-06-11T02:45:12.915538Z","shell.execute_reply.started":"2024-06-11T02:45:08.769243Z","shell.execute_reply":"2024-06-11T02:45:12.914572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\nChecking Failure Cases\n
\n\n
\n
\n
\nSubmit to Leaderboard\n
\n\n
\n
\n - MNIST database. (2023, September 8). In Wikipedia. [LINK]
\n - Principal component analysis. (2023, November 18). In Wikipedia. [LINK]
\n - PCA, scikit-learn documentation. [LINK]\n
- K. (2023, September 27). In Wikipedia. [LINK]
\n - KNNeighborsClassifier, scikit-learn documentation. [LINK]
\n
\n