{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
Simple Starter Model
\n\n
","metadata":{}},{"cell_type":"markdown","source":"#
Introduction
","metadata":{}},{"cell_type":"markdown","source":"The purpose of this notebook is to provide a starting point with a relatively simple, but high scoring network that users can experiment with. The following image illustrates that model which consists of 3 convolution layers using 3x3 filters. Max pooling is used after the 1st and 2nd convolution layers.","metadata":{}},{"cell_type":"code","source":"#\n# You can find the original code at: https://github.com/gwding/draw_convnet\n# My modifications are marked by commenting out the original code and applying\n# my changes.\n#\n\n# Get rid of warnings. My modification.\nimport warnings\nwarnings.filterwarnings('ignore')\n\n\"\"\"\nCopyright (c) 2017, Gavin Weiguang Ding\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\n modification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n3. Neither the name of the copyright holder nor the names of its contributors\n may be used to endorse or promote products derived from this software\n without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n POSSIBILITY OF SUCH DAMAGE.\n\"\"\"\n\n\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nplt.rcdefaults()\nfrom matplotlib.lines import Line2D\nfrom matplotlib.patches import Rectangle\nfrom matplotlib.patches import Circle\n\nNumDots = 4\nNumConvMax = 8\nNumFcMax = 20\nWhite = 1.\nLight = 0.7\nMedium = 0.5\nDark = 0.3\nDarker = 0.15\nBlack = 0.\n\n\ndef add_layer(patches, colors, size=(24, 24), num=5,\n top_left=[0, 0],\n loc_diff=[3, -3],\n ):\n # add a rectangle\n top_left = np.array(top_left)\n loc_diff = np.array(loc_diff)\n loc_start = top_left - np.array([0, size[0]])\n for ind in range(num):\n patches.append(Rectangle(loc_start + ind * loc_diff, size[1], size[0]))\n if ind % 2:\n colors.append(Medium)\n else:\n colors.append(Light)\n\n\ndef add_layer_with_omission(patches, colors, size=(24, 24),\n num=5, num_max=8,\n num_dots=4,\n top_left=[0, 0],\n loc_diff=[3, -3],\n ):\n # add a rectangle\n top_left = np.array(top_left)\n loc_diff = np.array(loc_diff)\n loc_start = top_left - np.array([0, size[0]])\n this_num = min(num, num_max)\n start_omit = (this_num - num_dots) // 2\n end_omit = this_num - start_omit\n start_omit -= 1\n for ind in range(this_num):\n if (num > num_max) and (start_omit < ind < end_omit):\n omit = True\n else:\n omit = False\n\n if omit:\n patches.append(\n Circle(loc_start + ind * loc_diff + np.array(size) / 2, 0.5))\n else:\n patches.append(Rectangle(loc_start + ind * loc_diff,\n size[1], size[0]))\n\n if omit:\n colors.append(Black)\n elif ind % 2:\n colors.append(Medium)\n else:\n colors.append(Light)\n\n\ndef add_mapping(patches, colors, start_ratio, end_ratio, patch_size, ind_bgn,\n top_left_list, loc_diff_list, num_show_list, size_list):\n\n start_loc = top_left_list[ind_bgn] \\\n + (num_show_list[ind_bgn] - 1) * np.array(loc_diff_list[ind_bgn]) \\\n + np.array([start_ratio[0] * (size_list[ind_bgn][1] - patch_size[1]),\n - start_ratio[1] * (size_list[ind_bgn][0] - patch_size[0])]\n )\n\n\n\n\n end_loc = top_left_list[ind_bgn + 1] \\\n + (num_show_list[ind_bgn + 1] - 1) * np.array(\n loc_diff_list[ind_bgn + 1]) \\\n + np.array([end_ratio[0] * size_list[ind_bgn + 1][1],\n - end_ratio[1] * size_list[ind_bgn + 1][0]])\n\n\n patches.append(Rectangle(start_loc, patch_size[1], -patch_size[0]))\n colors.append(Dark)\n patches.append(Line2D([start_loc[0], end_loc[0]],\n [start_loc[1], end_loc[1]]))\n colors.append(Darker)\n patches.append(Line2D([start_loc[0] + patch_size[1], end_loc[0]],\n [start_loc[1], end_loc[1]]))\n colors.append(Darker)\n patches.append(Line2D([start_loc[0], end_loc[0]],\n [start_loc[1] - patch_size[0], end_loc[1]]))\n colors.append(Darker)\n patches.append(Line2D([start_loc[0] + patch_size[1], end_loc[0]],\n [start_loc[1] - patch_size[0], end_loc[1]]))\n colors.append(Darker)\n\n\n\ndef label(xy, text, xy_off=[0, 4]):\n plt.text(xy[0] + xy_off[0], xy[1] + xy_off[1], text,\n family='sans-serif', size=8)\n\n\nif __name__ == '__main__':\n\n fc_unit_size = 2\n layer_width = 40\n# flag_omit = True\n flag_omit = False\n\n patches = []\n colors = []\n\n fig, ax = plt.subplots()\n\n\n ############################\n # conv layers\n# size_list = [(32, 32), (18, 18), (10, 10), (6, 6), (4, 4)]\n# num_list = [3, 32, 32, 48, 48]\n# x_diff_list = [0, layer_width, layer_width, layer_width, layer_width]\n size_list = [(28, 28), (28, 28), (14, 14), (14, 14), (7, 7), (7, 7)]\n num_list = [1, 16, 16, 32, 32, 64]\n x_diff_list = [0, layer_width, layer_width, layer_width, layer_width, layer_width]\n text_list = ['Inputs'] + ['Feature\\nmaps'] * (len(size_list) - 1)\n loc_diff_list = [[3, -3]] * len(size_list)\n\n num_show_list = list(map(min, num_list, [NumConvMax] * len(num_list)))\n top_left_list = np.c_[np.cumsum(x_diff_list), np.zeros(len(x_diff_list))]\n\n for ind in range(len(size_list)-1,-1,-1):\n if flag_omit:\n add_layer_with_omission(patches, colors, size=size_list[ind],\n num=num_list[ind],\n num_max=NumConvMax,\n num_dots=NumDots,\n top_left=top_left_list[ind],\n loc_diff=loc_diff_list[ind])\n else:\n add_layer(patches, colors, size=size_list[ind],\n num=num_show_list[ind],\n top_left=top_left_list[ind], loc_diff=loc_diff_list[ind])\n label(top_left_list[ind], text_list[ind] + '\\n{}@{}x{}'.format(\n num_list[ind], size_list[ind][0], size_list[ind][1]))\n\n ############################\n # in between layers\n# start_ratio_list = [[0.4, 0.5], [0.4, 0.8], [0.4, 0.5], [0.4, 0.8]]\n# end_ratio_list = [[0.4, 0.5], [0.4, 0.8], [0.4, 0.5], [0.4, 0.8]]\n# patch_size_list = [(5, 5), (2, 2), (5, 5), (2, 2)]\n start_ratio_list = [[0.4, 0.5], [0.4, 0.8], [0.4, 0.5], [0.4, 0.8], [0.4, 0.8]]\n end_ratio_list = [[0.4, 0.5], [0.4, 0.8], [0.4, 0.5], [0.4, 0.8], [0.4, 0.8]]\n patch_size_list = [(3, 3), (2, 2), (3, 3), (2, 2), (3, 3)]\n ind_bgn_list = range(len(patch_size_list))\n# text_list = ['Convolution', 'Max-pooling', 'Convolution', 'Max-pooling']\n text_list = ['Convolution', 'Max-pooling', 'Convolution', 'Max-pooling', 'Convolution']\n\n for ind in range(len(patch_size_list)):\n add_mapping(\n patches, colors, start_ratio_list[ind], end_ratio_list[ind],\n patch_size_list[ind], ind,\n top_left_list, loc_diff_list, num_show_list, size_list)\n label(top_left_list[ind], text_list[ind] + '\\n{}x{} kernel'.format(\n patch_size_list[ind][0], patch_size_list[ind][1]), xy_off=[26, -65]\n )\n\n\n ############################\n # fully connected layers\n# size_list = [(fc_unit_size, fc_unit_size)] * 3\n size_list = [(fc_unit_size, fc_unit_size)]\n# num_list = [768, 500, 2]\n num_list = [10]\n num_show_list = list(map(min, num_list, [NumFcMax] * len(num_list)))\n x_diff_list = [sum(x_diff_list) + layer_width, layer_width, layer_width]\n top_left_list = np.c_[np.cumsum(x_diff_list), np.zeros(len(x_diff_list))]\n loc_diff_list = [[fc_unit_size, -fc_unit_size]] * len(top_left_list)\n# text_list = ['Hidden\\nunits'] * (len(size_list) - 1) + ['Outputs']\n text_list = ['Outputs']\n\n for ind in range(len(size_list)):\n if flag_omit:\n add_layer_with_omission(patches, colors, size=size_list[ind],\n num=num_list[ind],\n num_max=NumFcMax,\n num_dots=NumDots,\n top_left=top_left_list[ind],\n loc_diff=loc_diff_list[ind])\n else:\n add_layer(patches, colors, size=size_list[ind],\n num=num_show_list[ind],\n top_left=top_left_list[ind],\n loc_diff=loc_diff_list[ind])\n label(top_left_list[ind], text_list[ind] + '\\n{}'.format(\n num_list[ind]))\n\n text_list = ['Flatten\\n', 'Fully\\nconnected', 'Fully\\nconnected']\n\n for ind in range(len(size_list)):\n label(top_left_list[ind], text_list[ind], xy_off=[-10, -65])\n\n ############################\n for patch, color in zip(patches, colors):\n patch.set_color(color * np.ones(3))\n if isinstance(patch, Line2D):\n ax.add_line(patch)\n else:\n patch.set_edgecolor(Black * np.ones(3))\n ax.add_patch(patch)\n\n# Get rid of warning.\n# plt.tight_layout()\n plt.axis('equal')\n plt.axis('off')\n# Swap the order.\n# plt.show()\n# fig.set_size_inches(8, 2.5)\n fig.set_size_inches(10, 2.5)\n# plt.show()\n\n# fig_dir = './'\n# fig_ext = '.png'\n# fig.savefig(os.path.join(fig_dir, 'convnet_fig' + fig_ext),\n# bbox_inches='tight', pad_inches=0)","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:36:56.630697Z","iopub.execute_input":"2024-11-05T20:36:56.631271Z","iopub.status.idle":"2024-11-05T20:36:57.000539Z","shell.execute_reply.started":"2024-11-05T20:36:56.631205Z","shell.execute_reply":"2024-11-05T20:36:56.999061Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Imports
","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\nimport pandas as pd\nimport seaborn as sns\nimport tensorflow as tf\nimport tensorflow.keras as keras\nimport time\n\nfrom scipy import ndimage\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras import layers, losses, models, optimizers, preprocessing, regularizers","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:36:57.002829Z","iopub.execute_input":"2024-11-05T20:36:57.004472Z","iopub.status.idle":"2024-11-05T20:37:16.366459Z","shell.execute_reply.started":"2024-11-05T20:36:57.004414Z","shell.execute_reply":"2024-11-05T20:37:16.365257Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_first_100(X):\n fig = plt.figure(figsize=(10, 1.6))\n for i in range(100):\n ax = plt.subplot(4, 25, i + 1)\n ax.axis('off')\n plt.imshow(np.array(X[i]).reshape((28,28)), cmap='gray_r')\n \ndef plot_transformations(image):\n generator = preprocessing.image.ImageDataGenerator()\n title_font_size = 10\n \n fig = plt.figure(figsize=(10,6))\n \n ax = plt.subplot(3, 5, 1)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(image, cmap='gray_r')\n plt.title('Original', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 2)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'theta':10}),\n cmap='gray_r')\n plt.title('Rotate 10', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 3)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'theta':-10}),\n cmap='gray_r')\n plt.title('Rotate -10', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 4)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'zx':0.8, 'zy':0.8}),\n cmap='gray_r')\n plt.title('Zoom 0.2', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 5)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'zx':1.2, 'zy':1.2}),\n cmap='gray_r')\n plt.title('Zoom -0.2', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 6)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'tx':-2.8}),\n cmap='gray_r')\n plt.title('Width Shift 0.2', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 7)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'tx':2.8}),\n cmap='gray_r')\n plt.title('Width Shift -0.2', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 8)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'ty':2.8}),\n cmap='gray_r')\n plt.title('Height Shift 0.2', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 9)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'ty':-2.8}),\n cmap='gray_r')\n plt.title('Height Shift -0.2', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 10)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'shear':10}),\n cmap='gray_r')\n plt.title('Shear 10', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 11)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'shear':-10}), cmap='gray_r')\n plt.title('Shear -10', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 12)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'theta':-45}), cmap='gray_r')\n plt.title('Rotate -20', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 13)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'shear':-45}), cmap='gray_r')\n plt.title('Shear -20', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 14)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'theta':10, 'zx':0.8, 'zy':0.8, 'tx':-2.8, 'ty':2.8, 'shear':10}), cmap='gray_r')\n plt.title('Combined +', fontsize=title_font_size)\n \n ax = plt.subplot(3, 5, 15)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(generator.apply_transform(image, transform_parameters={'theta':-10, 'zx':1.2, 'zy':1.2, 'tx':2.8, 'ty':-2.8, 'shear':-10}), cmap='gray_r')\n _ = plt.title('Combined -', fontsize=title_font_size)\n\ndef plot_augmentations(image):\n generator = preprocessing.image.ImageDataGenerator(\n rotation_range=6,\n zoom_range=0.05,\n width_shift_range=0.08,\n height_shift_range=0.07,\n shear_range=1\n )\n generator.fit(image.reshape(1,28,28,1))\n image_iterator = datagen.flow(image.reshape(1,28,28,1), np.array([0]), batch_size=1)\n \n fig = plt.figure(figsize=(10,1.6))\n for i in range(100):\n ax = plt.subplot(4, 25, i + 1)\n ax.axis('off')\n plt.imshow(next(image_iterator)[0][0], cmap='gray_r')\n\ndef plot_accuracy_and_loss(history):\n fig = plt.figure(figsize=(10,4))\n \n ax = plt.subplot(1, 2, 1)\n plt.plot(list(range(10,50)), history.history['accuracy'][10:], label=\"Training Accuracy\")\n plt.plot(list(range(10,50)), history.history['val_accuracy'][10:], label='Validation Accuracy')\n plt.title('Accuracy')\n plt.xlabel('Epoch')\n _ = plt.legend()\n \n ax = plt.subplot(1, 2, 2)\n plt.plot(list(range(10,50)), history.history['loss'][10:], label=\"Training Loss\")\n plt.plot(list(range(10,50)), history.history['val_loss'][10:], label='Validation Loss')\n plt.title('Loss')\n plt.xlabel('Epoch')\n _ = plt.legend()\n \n fig.tight_layout()\n\ndef plot_wrong(X, y, y_hat):\n X_wrong = X[y_hat != y]\n y_wrong_pred = y_hat[y_hat != y]\n y_wrong_actual = y[y_hat != y]\n fig = plt.figure(figsize=(10,2*len(X_wrong)//5))\n for i in range(len(X_wrong)):\n ax = plt.subplot(len(X_wrong)//5, 5, i + 1)\n plt.xticks([])\n plt.yticks([])\n plt.title(str(y_wrong_pred[i]) + '/' + str(y_wrong_actual[i]))\n plt.imshow(X_wrong[i], cmap='gray_r')\n\ndef plot_first_layer_filters(model):\n weights = model.get_layer('C1').get_weights()\n fig = plt.figure(figsize=(8,2))\n for i in range(16):\n ax = plt.subplot(2, 8, i + 1)\n ax.axis('off')\n plt.imshow(weights[0][:, :, 0, i], cmap='gray')\n\ndef plot_first_layer_output(model, input):\n first_layer_model = keras.Model(inputs=model.inputs, outputs=model.get_layer('A1').output)\n output = first_layer_model(input.reshape(1, 28, 28, 1))\n fig = plt.figure(figsize=(10,2.5))\n for i in range(16):\n ax = plt.subplot(2, 8, i + 1)\n #ax.axis('off')\n plt.xticks([])\n plt.yticks([])\n plt.imshow(output[0, :, :, i], cmap=\"gray\")\n\ndef plot_second_layer_output(model, input):\n second_layer_model = keras.Model(inputs=model.inputs, outputs=model.get_layer('A2').output)\n output = second_layer_model(input.reshape(1, 28, 28, 1))\n fig = plt.figure(figsize=(10,5))\n for i in range(32):\n ax = plt.subplot(4, 8, i + 1)\n #ax.axis('off')\n plt.xticks([])\n plt.yticks([])\n plt.imshow(output[0, :, :, i], cmap=\"gray\")\n\ndef plot_third_layer_output(model, input):\n third_layer_model = keras.Model(inputs=model.inputs, outputs=model.get_layer('A3').output)\n output = third_layer_model(input.reshape(1, 28, 28, 1))\n fig = plt.figure(figsize=(10,10))\n for i in range(64):\n ax = plt.subplot(8, 8, i + 1)\n #ax.axis('off')\n plt.xticks([])\n plt.yticks([])\n plt.imshow(output[0, :, :, i], cmap=\"gray\")\n\ndef plot_final_output(model, input):\n output = model(input.reshape(1, 28, 28, 1))\n fig = plt.figure(figsize=(10,1))\n plt.xticks([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n plt.yticks([])\n plt.imshow(output, cmap=\"gray\")\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:44:17.673169Z","iopub.execute_input":"2024-11-05T21:44:17.673604Z","iopub.status.idle":"2024-11-05T21:44:17.717587Z","shell.execute_reply.started":"2024-11-05T21:44:17.673552Z","shell.execute_reply":"2024-11-05T21:44:17.716376Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Load the Data
","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_df = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\nprint(train_df.shape)\nprint(test_df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:16.409418Z","iopub.execute_input":"2024-11-05T20:37:16.409851Z","iopub.status.idle":"2024-11-05T20:37:21.763382Z","shell.execute_reply.started":"2024-11-05T20:37:16.409801Z","shell.execute_reply":"2024-11-05T20:37:21.762187Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Deterministic vs Non-Deterministic Behavior
","metadata":{}},{"cell_type":"markdown","source":"For many machine learning algorithms, obtaining repeatable results is as simple as setting a seed value. For tensorflow, it's a little more involved, but still relatively straightforward.\n\n**Important Note:** If you use a GPU, you will need to set deterministic to False or you'll get an error.","metadata":{}},{"cell_type":"code","source":"# Set to False if using a GPU.\ndeterministic = True\nif deterministic:\n random_seed = 1\n tf.keras.utils.set_random_seed(random_seed)\n tf.config.experimental.enable_op_determinism()\nelse:\n random_seed = None","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:21.765080Z","iopub.execute_input":"2024-11-05T20:37:21.765572Z","iopub.status.idle":"2024-11-05T20:37:21.772560Z","shell.execute_reply.started":"2024-11-05T20:37:21.765522Z","shell.execute_reply":"2024-11-05T20:37:21.771037Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Training and Validation Sets
","metadata":{}},{"cell_type":"markdown","source":"For this notebook, the full training set is split 80/20 to create a smaller training set (80%) and validation set (20%). ","metadata":{}},{"cell_type":"code","source":"X_train_full = train_df.to_numpy()[:, 1:].reshape(42000, 28, 28, 1)\nX_train_full = X_train_full/255.0\nprint(X_train_full.shape)\n\ny_train_full = train_df.to_numpy()[:, 0]\nprint(y_train_full.shape)\n\nX_train, X_val, y_train, y_val = train_test_split(X_train_full, y_train_full, test_size=0.2,\n random_state=random_seed)\nprint(X_train.shape)\nprint(y_train.shape)\nprint(X_val.shape)\nprint(y_val.shape)\n\nX_test = test_df.to_numpy().reshape(28000, 28, 28, 1)\nX_test = X_test/255.0\nprint(X_test.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:21.774324Z","iopub.execute_input":"2024-11-05T20:37:21.775463Z","iopub.status.idle":"2024-11-05T20:37:22.468755Z","shell.execute_reply.started":"2024-11-05T20:37:21.775408Z","shell.execute_reply":"2024-11-05T20:37:22.467481Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
A Quick Look at Some Digits
","metadata":{}},{"cell_type":"markdown","source":"Here we plot the first 100 digits from the full training set.","metadata":{}},{"cell_type":"code","source":"plot_first_100(X_train_full)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:22.470501Z","iopub.execute_input":"2024-11-05T20:37:22.471006Z","iopub.status.idle":"2024-11-05T20:37:25.676676Z","shell.execute_reply.started":"2024-11-05T20:37:22.470954Z","shell.execute_reply":"2024-11-05T20:37:25.675437Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Data Augmentation
","metadata":{}},{"cell_type":"markdown","source":"Data augmentation is probably the most important thing to do to get higher than 99% accuracy on the MNIST dataset. It's very difficult to do so with just the 42,000 images in the training set. Keras ImageDataGenerator provides a convenient way to turn that 42,000 images into an essentially endless supply of unique images by applying what are known as affine transformations on each of those source images. The following code displays the effect of applying various affine transformations on one of the training images.","metadata":{}},{"cell_type":"code","source":"plot_transformations(X_train_full[81])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:25.678013Z","iopub.execute_input":"2024-11-05T20:37:25.678389Z","iopub.status.idle":"2024-11-05T20:37:26.952664Z","shell.execute_reply.started":"2024-11-05T20:37:25.678354Z","shell.execute_reply":"2024-11-05T20:37:26.951477Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The above tranformations are probably a little more extreme than what you'd want to use in practice but useful for explaining what they are. One reason more subtle changes may be preferable is illustrated in the image titled \"Combined +\" which applies the rotate 10, zoom 0.2, width shift 0.2, high shift 0.2, and shear 10 transformations simultaneously which causes part of the digit to run off the side of the image. It is important to note that ImageDataGenerator actually selects a random transformation from the provided ranges so the \"Combined +\" and \"Combined -\" images essentially represent the bounds of the possible images that might be generated with the parameters provided above.\n\nOne other thing to to note is that at low values shear and rotate result in similar output. The difference, however, becomes much more apparaent at high values as illustrated by the \"Rotate -20\" and \"Shear -20\" images.\n\nBelow, I use the following parameters which I've had some good luck with:\n* rotatation_range = 6\n* zoom_range = 0.05\n* width_shift_range = 0.08\n* height_shift_range = 0.07\n* shear_range = 1\n\nSome other common (and probably just as good) values I've seen include:\n* rotatation_range = 5 or 10\n* zoom_range = 0.1\n* width_shift_range = 0.1\n* height_shift_range = 0.1\n","metadata":{}},{"cell_type":"code","source":"datagen = preprocessing.image.ImageDataGenerator(\n rotation_range=6,\n zoom_range=0.05,\n width_shift_range=0.08,\n height_shift_range=0.07,\n shear_range=1\n)\ndatagen.fit(X_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:26.954067Z","iopub.execute_input":"2024-11-05T20:37:26.954438Z","iopub.status.idle":"2024-11-05T20:37:27.068658Z","shell.execute_reply.started":"2024-11-05T20:37:26.954402Z","shell.execute_reply":"2024-11-05T20:37:27.067410Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The follow code displays an example of what 100 augmented versions of the digit from above might look like with these parameters. Note how the tranformations are relatively subtle despite their effectiveness.","metadata":{}},{"cell_type":"code","source":"plot_augmentations(X_train_full[81])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:27.073056Z","iopub.execute_input":"2024-11-05T20:37:27.073455Z","iopub.status.idle":"2024-11-05T20:37:30.275270Z","shell.execute_reply.started":"2024-11-05T20:37:27.073419Z","shell.execute_reply":"2024-11-05T20:37:30.273773Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Learning Rate Schedule
","metadata":{}},{"cell_type":"markdown","source":"This isn't something you really need to do, but it helps reduce the variance at the end of training. The following schedule slowly decays from a learning rate of 0.001 to a value of 0.00001 over 50 epochs. The calculation for the parameters is relatively straightforward:\n\n$$decay\\_steps = floor\\left(\\frac{training\\_set\\_size}{batch\\_size}\\right)$$\n\n$$decay\\_steps = \\left(\\frac{final\\_rate}{initial\\_rate}\\right)^{\\frac{1}{epochs}}$$\n","metadata":{}},{"cell_type":"code","source":"lr_schedule = optimizers.schedules.ExponentialDecay(\n 0.001,\n decay_steps=132,\n decay_rate=0.912\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:30.276745Z","iopub.execute_input":"2024-11-05T20:37:30.277143Z","iopub.status.idle":"2024-11-05T20:37:30.282442Z","shell.execute_reply.started":"2024-11-05T20:37:30.277082Z","shell.execute_reply":"2024-11-05T20:37:30.281186Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
The Model
","metadata":{}},{"cell_type":"markdown","source":"The model is just a relatively straightforward 4-layer Convolutional Neural Network (CNN). Note how Batch Normalization is performed after the convolution step but before the activation step. Also note that the network contains both dropout and regularization, although both are set to zero. Choosing better values is left as an excercise. Finally, note how the output layer is linear instead of softmax. Supposely, this make the training a little more stable. It's relatively straightforward to apply softmax after the fact if necessary.","metadata":{}},{"cell_type":"code","source":"DROPOUT=0.0\nREG=0.0\nmodel = models.Sequential([\n layers.Input((28, 28, 1), name=\"In\"),\n\n # 28 X 28\n layers.Conv2D(16, (3, 3), padding='same', use_bias=False,\n kernel_regularizer=regularizers.L2(REG), name='C1'),\n layers.BatchNormalization(name='B1'),\n layers.Activation('relu', name='A1'),\n layers.Dropout(DROPOUT, name='D1'),\n\n layers.MaxPooling2D((2, 2), name='P1'),\n\n # 14 X 14\n layers.Conv2D(32, (3, 3), padding='same', use_bias=False,\n kernel_regularizer=regularizers.L2(REG), name='C2'),\n layers.BatchNormalization(name='B2'),\n layers.Activation('relu', name='A2'),\n layers.Dropout(DROPOUT, name='D2'),\n\n layers.MaxPooling2D((2, 2), name='P2'),\n\n # 7 X 7\n layers.Conv2D(64, (3, 3), padding='same', use_bias=False,\n kernel_regularizer=regularizers.L2(REG), name='C3'),\n layers.BatchNormalization(name='B3'),\n layers.Activation('relu', name='A3'),\n layers.Dropout(DROPOUT, name='D3'),\n \n # This is where you'd normally see some Dense layers. Feel free to add.\n \n layers.Flatten(),\n layers.Dense(10, activation='linear', name='Out'),\n\n])\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:30.283776Z","iopub.execute_input":"2024-11-05T20:37:30.284146Z","iopub.status.idle":"2024-11-05T20:37:30.451813Z","shell.execute_reply.started":"2024-11-05T20:37:30.284082Z","shell.execute_reply":"2024-11-05T20:37:30.450757Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Model Compilation
","metadata":{}},{"cell_type":"markdown","source":"The learning rate schedule is applied here at this point. If you want a constant learning rate, just add that instead.","metadata":{}},{"cell_type":"code","source":"model.compile(\n loss=losses.SparseCategoricalCrossentropy(from_logits=True),\n optimizer=optimizers.Adam(lr_schedule),\n metrics=['accuracy']\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:30.453179Z","iopub.execute_input":"2024-11-05T20:37:30.453538Z","iopub.status.idle":"2024-11-05T20:37:30.467822Z","shell.execute_reply.started":"2024-11-05T20:37:30.453503Z","shell.execute_reply":"2024-11-05T20:37:30.466582Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Model Fitting
","metadata":{}},{"cell_type":"markdown","source":"Note the batch_size of 256. For me this seemed to run a little more quickly than 32.","metadata":{}},{"cell_type":"code","source":"start = time.time()\nhistory = model.fit(\n datagen.flow(X_train, y_train, batch_size=256),\n epochs=50,\n batch_size=256,\n validation_data=(X_val, y_val)\n)\nprint('Training time:', time.time() - start)\n\ny_hat_train = np.argmax(model.predict(X_train), axis=1)\ntrain_accuracy = np.sum(y_hat_train == y_train)/y_train.shape[0]\n\ny_hat_val = np.argmax(model.predict(X_val), axis=1)\nval_accuracy = np.sum(y_hat_val == y_val)/y_val.shape[0]\n\nprint('Training Accuracy:', train_accuracy, 'Validation Accuracy', val_accuracy)\nprint('Learning Rate:', model.optimizer.learning_rate.numpy())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T20:37:30.469227Z","iopub.execute_input":"2024-11-05T20:37:30.469578Z","iopub.status.idle":"2024-11-05T21:04:42.834374Z","shell.execute_reply.started":"2024-11-05T20:37:30.469541Z","shell.execute_reply":"2024-11-05T21:04:42.833133Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Plotting the Results
","metadata":{}},{"cell_type":"markdown","source":"First we plot the values for accuracy and loss starting after the 10th epoch so the detail is more apparent. Note how the training accuracy continues to slope upwards indicating the possibility of better results with more epochs.","metadata":{}},{"cell_type":"code","source":"plot_accuracy_and_loss(history)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:04:42.835915Z","iopub.execute_input":"2024-11-05T21:04:42.836331Z","iopub.status.idle":"2024-11-05T21:04:43.543796Z","shell.execute_reply.started":"2024-11-05T21:04:42.836288Z","shell.execute_reply":"2024-11-05T21:04:43.542552Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Next we plot each of the 55 digits that the model predicted wrongly in the validation set. For each digit predicted/actual labels are printed in the title. Some of the images are obvious errors, but others, such as the digit in the first column of the last row, I would have guessed incorrecly as well. In fact, I would go as far as saying that several, including that one, appear to be mislabeled.","metadata":{}},{"cell_type":"code","source":"plot_wrong(X_val, y_val, y_hat_val)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:04:43.545307Z","iopub.execute_input":"2024-11-05T21:04:43.545704Z","iopub.status.idle":"2024-11-05T21:04:46.900794Z","shell.execute_reply.started":"2024-11-05T21:04:43.545662Z","shell.execute_reply":"2024-11-05T21:04:46.899586Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Intuition
","metadata":{}},{"cell_type":"markdown","source":"##
First Layer Filters
","metadata":{}},{"cell_type":"code","source":"plot_first_layer_filters(model)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:04:46.902322Z","iopub.execute_input":"2024-11-05T21:04:46.902695Z","iopub.status.idle":"2024-11-05T21:04:47.362924Z","shell.execute_reply.started":"2024-11-05T21:04:46.902657Z","shell.execute_reply":"2024-11-05T21:04:47.361825Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"##
First Layer Output
","metadata":{}},{"cell_type":"code","source":"plot_first_layer_output(model, X_train_full[81])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:11:40.383232Z","iopub.execute_input":"2024-11-05T21:11:40.383709Z","iopub.status.idle":"2024-11-05T21:11:41.158876Z","shell.execute_reply.started":"2024-11-05T21:11:40.383663Z","shell.execute_reply":"2024-11-05T21:11:41.157636Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"##
Second Layer Output
","metadata":{}},{"cell_type":"code","source":"plot_second_layer_output(model, X_train_full[81])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:29:16.147415Z","iopub.execute_input":"2024-11-05T21:29:16.147859Z","iopub.status.idle":"2024-11-05T21:29:17.563968Z","shell.execute_reply.started":"2024-11-05T21:29:16.147823Z","shell.execute_reply":"2024-11-05T21:29:17.562743Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"##
Third Layer Output
","metadata":{}},{"cell_type":"code","source":"plot_third_layer_output(model, X_train_full[81])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:31:55.092355Z","iopub.execute_input":"2024-11-05T21:31:55.092791Z","iopub.status.idle":"2024-11-05T21:31:58.253436Z","shell.execute_reply.started":"2024-11-05T21:31:55.092750Z","shell.execute_reply":"2024-11-05T21:31:58.252165Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"##
Final Output
","metadata":{}},{"cell_type":"code","source":"plot_final_output(model, X_train_full[81])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:44:41.282922Z","iopub.execute_input":"2024-11-05T21:44:41.283389Z","iopub.status.idle":"2024-11-05T21:44:41.446766Z","shell.execute_reply.started":"2024-11-05T21:44:41.283343Z","shell.execute_reply":"2024-11-05T21:44:41.445316Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Fine Tuning Against the Full Training Set
","metadata":{}},{"cell_type":"markdown","source":"So long as you don't need to use early stopping, there's not much point in training against only part of the training set so fine tune against the full set here. If you are using early stopping though, this gets much more difficult to do.\n\nNote also that the model is recompiled to change the learning rate.\n\n**Important Note:** In theory this should give better results, but I have so far had lackluster success doing this after recompiling. In the process of trying to understand why.","metadata":{}},{"cell_type":"code","source":"datagen.fit(X_train_full)\nmodel.compile(\n loss=losses.SparseCategoricalCrossentropy(from_logits=True),\n optimizer=optimizers.Adam(0.00001),\n metrics=['accuracy']\n)\nmodel.fit(\n datagen.flow(X_train_full, y_train_full, batch_size=256),\n epochs=20,\n batch_size=256\n)\ny_hat_train_full = np.argmax(model.predict(X_train_full), axis=1)\ntrain_accuracy_full = np.sum(y_hat_train_full == y_train_full)/y_train_full.shape[0]\n\nprint('Training Accuracy', train_accuracy_full)\nprint('Learning Rate:', model.optimizer.learning_rate.numpy())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:04:49.213258Z","iopub.status.idle":"2024-11-05T21:04:49.213655Z","shell.execute_reply.started":"2024-11-05T21:04:49.213458Z","shell.execute_reply":"2024-11-05T21:04:49.213477Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Generating the Submission
","metadata":{}},{"cell_type":"code","source":"y_hat_test = np.argmax(model.predict(X_test), axis=1)\nwith open('submission.csv', 'w') as f:\n f.write('ImageId,Label\\n')\n for i in range(len(y_hat_test)):\n f.write(str(i + 1) + ',' + str(y_hat_test[i]) + '\\n')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-05T21:04:49.215486Z","iopub.status.idle":"2024-11-05T21:04:49.216032Z","shell.execute_reply.started":"2024-11-05T21:04:49.215747Z","shell.execute_reply":"2024-11-05T21:04:49.215774Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
Things to Try
","metadata":{}},{"cell_type":"markdown","source":"* Increase the number of filters in each of the convolution layers.\n* Add one or more Dense layers after the convolution layers but before the output layer.\n* Try 5x5 and/or 7x7 kernel sizes.\n* Adjust Dropout and/or Regularization - probably do this after adding more layers or increasing the number of filters.\n* Use more epochs.\n* Try different augmentation parameters.","metadata":{}},{"cell_type":"markdown","source":"#
Papers to Read
","metadata":{}},{"cell_type":"markdown","source":"Pretty much any of the papers referenced in the Classifiers Section:\n\n[https://en.wikipedia.org/wiki/MNIST_database](https://en.wikipedia.org/wiki/MNIST_database)\n\nSo far I've looked at:\n\n[Deep Big Simple Neural Nets Excel on Hand-\nwritten Digit Recognition](https://arxiv.org/pdf/1003.0358)\n\n[LET’S KEEP IT SIMPLE,\nUSING SIMPLE ARCHITECTURES TO OUTPERFORM\nDEEPER AND MORE COMPLEX ARCHITECTURES](https://arxiv.org/pdf/1608.06037)\n\n[AN ENSEMBLE OF SIMPLE CONVOLUTIONAL NEURAL\nNETWORK MODELS FOR MNIST DIGIT RECOGNITION](https://arxiv.org/pdf/2008.10400)\n\nAnd of course:\n\n[Gradient\u0002Based Learning Applied to Document\nRecognition](http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf)","metadata":{}},{"cell_type":"markdown","source":"#
References
","metadata":{}},{"cell_type":"markdown","source":"[https://stackoverflow.com/questions/61552475/properly-set-up-exponential-decay-of-learning-rate-in-tensorflow](https://stackoverflow.com/questions/61552475/properly-set-up-exponential-decay-of-learning-rate-in-tensorflow)\n\n[https://towardsdatascience.com/going-beyond-99-mnist-handwritten-digits-recognition-cfff96337392](https://towardsdatascience.com/going-beyond-99-mnist-handwritten-digits-recognition-cfff96337392)\n\n[https://www.deeplearning.ai/courses/machine-learning-specialization/](https://www.deeplearning.ai/courses/machine-learning-specialization/)\n\n[https://www.deeplearning.ai/courses/deep-learning-specialization/](https://www.deeplearning.ai/courses/deep-learning-specialization/)","metadata":{}}]}