{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction\n\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “Hello World” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nIn this competition, we aim to correctly identify digits from a dataset of tens of thousands of handwritten images. Kaggle has curated a set of tutorial-style kernels which cover everything from regression to neural networks. They hope to encourage us to experiment with different algorithms to learn first-hand what works well and how techniques compare.\n\n## Approach\n\nFor this competition, we will be using Keras (with TensorFlow as our backend) as the main package to create a simple neural network to predict, as accurately as we can, digits from handwritten images. In particular, we will be calling the Functional Model API of Keras, and creating a 4-layered and 5-layered neural network.\n\nAlso, we will be experimenting with various optimizers: the plain vanilla Stochastic Gradient Descent optimizer and the Adam optimizer. However, there are many other parameters, such as training epochs which will we will not be experimenting with.\n\nIn addition, the choice of hidden layer units are completely arbitrary and may not be optimal. This is yet another parameter which we will not attempt to tinker with. Lastly, we introduce dropout, a form of regularisation, in our neural networks to prevent overfitting.\n\n## Result\n\nFollowing our simulations on the cross validation dataset, it appears that a 4-layered neural network, using 'Adam' as the optimizer along with a learning rate of 0.01, performs best. We proceed to introduce dropout in the model, and use the model to predict for the test set.\n\nThe test predictions (submitted to Kaggle) generated by our model predicts with an accuracy score of 97.600%, which places us at the top 55 percentile of the competition.","metadata":{"_cell_guid":"82418877-8407-45ef-9b76-86ca80e07abf","_uuid":"483f57b296f8d69647bbec1154acfa2da3c6fc2f","_execution_state":"idle"}},{"cell_type":"markdown","source":"Importing key libraries, and reading data","metadata":{"_cell_guid":"440129fa-bb2d-47b8-9415-714467d98743","_uuid":"44f63ca7fc70094c75598c27b433b316ea406237","_execution_state":"idle"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport cv2\n\nimport os\nnp.random.seed(1212)\nfrom itertools import chain\nfrom PIL import Image\nimport keras\nfrom keras.models import Model\nfrom keras.layers import *\nfrom keras import optimizers","metadata":{"_cell_guid":"55ec59e1-8a8c-42d2-8ccc-b6d95ea198b4","_uuid":"a6d625ede9e19ac1f146a5f63d6ca19c4b98e20c","_execution_state":"busy","execution":{"iopub.status.busy":"2023-06-11T14:04:23.230199Z","iopub.execute_input":"2023-06-11T14:04:23.230615Z","iopub.status.idle":"2023-06-11T14:04:33.926279Z","shell.execute_reply.started":"2023-06-11T14:04:23.230582Z","shell.execute_reply":"2023-06-11T14:04:33.924322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","metadata":{"_cell_guid":"daa0805e-e1dd-4806-b2db-ff0124febe7d","_uuid":"be0a7acbb3bfc83fd3103b4a401337aafe57c00d","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T14:04:36.038088Z","iopub.execute_input":"2023-06-11T14:04:36.038838Z","iopub.status.idle":"2023-06-11T14:04:41.687385Z","shell.execute_reply.started":"2023-06-11T14:04:36.038799Z","shell.execute_reply":"2023-06-11T14:04:41.686069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#df_train.head() # 784 features, 1 label\ndf_test.head()","metadata":{"_cell_guid":"dfae6a31-23ea-4d88-9fcd-06b8b705b614","_uuid":"0f7bfb5923e5b4ed5a5149c007a2cd39cb9b8282","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T14:17:06.846482Z","iopub.execute_input":"2023-06-11T14:17:06.846981Z","iopub.status.idle":"2023-06-11T14:17:06.867407Z","shell.execute_reply.started":"2023-06-11T14:17:06.846948Z","shell.execute_reply":"2023-06-11T14:17:06.866175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Splitting into training and validation dataset","metadata":{"_cell_guid":"45bd232a-4000-4a06-a3ed-0be9f993746c","_uuid":"f0970dcbc0b6791dc44936d8d63e6985dfb6960c","_execution_state":"idle"}},{"cell_type":"code","source":"df_features = df_train.iloc[:, 1:785]\ndf_label = df_train.iloc[:, 0]\n\nX_test = df_test.copy()\n\nprint(X_test.shape)","metadata":{"_cell_guid":"bb78680a-ce50-4ee9-a77f-265b9ee38aee","_uuid":"16e39b5512ed29c8f9d7db51331693e4689ce306","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T14:33:41.957154Z","iopub.execute_input":"2023-06-11T14:33:41.958407Z","iopub.status.idle":"2023-06-11T14:33:42.040573Z","shell.execute_reply.started":"2023-06-11T14:33:41.958367Z","shell.execute_reply":"2023-06-11T14:33:42.038757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_cv, y_train, y_cv = train_test_split(df_features, df_label, \n                                                test_size = 0.2,\n                                                random_state = 1212)\n\nX_train = X_train.to_numpy() #(33600, 784)\nX_cv = X_cv.to_numpy() #(8400, 784)\n\nX_test = X_test.to_numpy()\nX_test.shape","metadata":{"_cell_guid":"a3fb0b37-1edf-42bd-b951-86f254b78fc1","_uuid":"a98808f6188e9c44741ad2b5c25a894106406dfc","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T14:33:43.449673Z","iopub.execute_input":"2023-06-11T14:33:43.450048Z","iopub.status.idle":"2023-06-11T14:33:43.729956Z","shell.execute_reply.started":"2023-06-11T14:33:43.450021Z","shell.execute_reply":"2023-06-11T14:33:43.728438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data cleaning, normalization and selection","metadata":{"_cell_guid":"4486ba9e-b8b3-4f09-a597-3d5e6bbd6c1c","_uuid":"9a10a6e1aae9645a7c3e5a703d78d9c3b096f6d3","_execution_state":"idle"}},{"cell_type":"code","source":"print((min(X_train[1]), max(X_train[1])))","metadata":{"_cell_guid":"4d0e07b5-28dd-42f1-b425-6414aea90cde","_uuid":"cc2daf9f099fb62e3b470f913349428ef396b75b","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.014752Z","iopub.status.idle":"2023-06-11T02:39:25.015411Z","shell.execute_reply.started":"2023-06-11T02:39:25.015205Z","shell.execute_reply":"2023-06-11T02:39:25.015226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As the pixel intensities are currently between the range of 0 and 255, we proceed to normalize the features, using broadcasting. In addition, we proceed to convert our labels from a class vector to binary One Hot Encoded","metadata":{"_cell_guid":"60b333cc-a76e-4713-97f5-a9095e27e43b","_uuid":"11d4f22158c013b3f028f5c11340b4a4b7291d1e","_execution_state":"idle"}},{"cell_type":"code","source":"# Feature Normalization \nX_train = X_train.astype('float32'); X_cv= X_cv.astype('float32'); X_test = X_test.astype('float32')\nX_train /= 255; X_cv /= 255; X_test /= 255\n\n# Convert labels to One Hot Encoded\nnum_digits = 10\ny_train = keras.utils.to_categorical(y_train, num_digits)\ny_cv = keras.utils.to_categorical(y_cv, num_digits)","metadata":{"_cell_guid":"e9f8eaa0-c77e-4510-9ff1-bb56e1d5f76d","_uuid":"c24261ffd2384078a836a909ea4839a31d51dd66","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T14:05:09.452902Z","iopub.execute_input":"2023-06-11T14:05:09.453280Z","iopub.status.idle":"2023-06-11T14:05:09.608960Z","shell.execute_reply.started":"2023-06-11T14:05:09.453249Z","shell.execute_reply":"2023-06-11T14:05:09.607539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Printing 2 examples of labels after conversion\nprint(y_train[0]) # 2\nprint(y_train[3]) # 7","metadata":{"_cell_guid":"6667ca35-44ff-43f3-b88b-27b787be4ee0","_uuid":"3fdf37a538504f4e5288cbb8b70a719c2689caca","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.018354Z","iopub.status.idle":"2023-06-11T02:39:25.018917Z","shell.execute_reply.started":"2023-06-11T02:39:25.018711Z","shell.execute_reply":"2023-06-11T02:39:25.018730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Fitting","metadata":{"_cell_guid":"7d4e437c-12ea-4bad-a580-5504e8a77da0","_uuid":"0776b5f2aff1ebb6f12c2a9ee8e93013c690caf6","_execution_state":"idle"}},{"cell_type":"markdown","source":"We proceed by fitting several simple neural network models using Keras (with TensorFlow as our backend) and collect their accuracy. The model that performs the best on the validation set will be used as the model of choice for the competition.\n\nModel 1: Simple Neural Network with 4 layers (300, 100, 100, 200)\n\nIn our first model, we will use the Keras library to train a neural network with the activation function set as ReLu. To determine which class to output, we will rely on the SoftMax function","metadata":{"_uuid":"064d83d74bf2e0842b21dcb3cc93f8427b04b6ea","_execution_state":"idle","_cell_guid":"8f441392-a707-499c-8816-00c8aa1a7757"}},{"cell_type":"code","source":"# Input Parameters\nn_input = 784 # number of features\nn_hidden_1 = 300\nn_hidden_2 = 100\nn_hidden_3 = 100\nn_hidden_4 = 200\nnum_digits = 10","metadata":{"_cell_guid":"be9af3d4-ddac-4f85-945f-bd1cdf09fc15","_uuid":"b88da407b6dab83e54b5380e0e62459577949932","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.019938Z","iopub.status.idle":"2023-06-11T02:39:25.020301Z","shell.execute_reply.started":"2023-06-11T02:39:25.020121Z","shell.execute_reply":"2023-06-11T02:39:25.020137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)","metadata":{"_cell_guid":"74eac473-a6cb-4356-a982-ef3cb6c971de","_uuid":"f97a558010d88407363bbb08fa00e7c87a3d6ceb","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.021795Z","iopub.status.idle":"2023-06-11T02:39:25.022289Z","shell.execute_reply.started":"2023-06-11T02:39:25.022092Z","shell.execute_reply":"2023-06-11T02:39:25.022111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Our model would have '6' layers - input layer, 4 hidden layer and 1 output layer\nmodel = Model(Inp, output)\nmodel.summary() # We have 297,910 parameters to estimate","metadata":{"_cell_guid":"bf09781e-6376-4f2d-a872-6292d466f657","_uuid":"aaf8fe1cd6cb32675f0ed5009d7bc31d6f71e4ea","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.023361Z","iopub.status.idle":"2023-06-11T02:39:25.023748Z","shell.execute_reply.started":"2023-06-11T02:39:25.023562Z","shell.execute_reply":"2023-06-11T02:39:25.023579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Insert Hyperparameters\nlearning_rate = 0.1\ntraining_epochs = 20\nbatch_size = 100\nsgd = optimizers.SGD(learning_rate=learning_rate)","metadata":{"_cell_guid":"906807f1-ed62-4dee-a423-5f15842fb28f","_uuid":"c4ea7858fbdf41308dcdb5d373af11484d5b5140","_execution_state":"busy","execution":{"iopub.status.busy":"2023-06-11T03:05:28.206113Z","iopub.execute_input":"2023-06-11T03:05:28.206483Z","iopub.status.idle":"2023-06-11T03:05:28.212263Z","shell.execute_reply.started":"2023-06-11T03:05:28.206455Z","shell.execute_reply":"2023-06-11T03:05:28.210886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We rely on the plain vanilla Stochastic Gradient Descent as our optimizing methodology\nmodel.compile(loss='categorical_crossentropy',\n              optimizer='sgd',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"5678ce6d-58eb-4394-9a77-b67c94c2ca4f","_uuid":"775193de659c41c949495a2afa3b14024a2536a7","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.026860Z","iopub.status.idle":"2023-06-11T02:39:25.027249Z","shell.execute_reply.started":"2023-06-11T02:39:25.027048Z","shell.execute_reply":"2023-06-11T02:39:25.027066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history1 = model.fit(X_train, y_train,\n                     batch_size = batch_size,\n                     epochs = training_epochs,\n                     verbose = 2,\n                     validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"1a547d04-6aeb-4c76-8fbc-7fbca1b50b2a","_uuid":"4960dfa65394fb0314ce9cb7af63d48709a39d1a","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.028493Z","iopub.status.idle":"2023-06-11T02:39:25.028896Z","shell.execute_reply.started":"2023-06-11T02:39:25.028697Z","shell.execute_reply":"2023-06-11T02:39:25.028715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using a 4 layer neural network with:\n\n1. 20 training epochs\n2. A training batch size of 100\n3. Hidden layers set as (300, 100, 100, 200)\n4. Learning rate of 0.1\n\nAchieved a training score of around 96-98% and a test score of around 95 - 97%.\n\nCan we do better if we were to change the optimizer? To find out, we use the Adam optimizer for our second model, while maintaining the same parameter values for all other parameters.","metadata":{"_uuid":"6121bf7f8adbeaf85b3aef6167444b555b920440","_execution_state":"idle","_cell_guid":"2fee2dfc-23d2-480f-918e-54fb5386c4ea"}},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)\n\n# We rely on ADAM as our optimizing methodology\nadam = keras.optimizers.Adam(learning_rate=learning_rate)\nmodel2 = Model(Inp, output)\n\nmodel2.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"4e7a2ba9-ac08-4355-a37f-2336220cfda7","_uuid":"3d9c3ffb9722b640ea0682e5826971045091db66","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.030063Z","iopub.status.idle":"2023-06-11T02:39:25.030436Z","shell.execute_reply.started":"2023-06-11T02:39:25.030249Z","shell.execute_reply":"2023-06-11T02:39:25.030266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history2 = model2.fit(X_train, y_train,\n                      batch_size = batch_size,\n                      epochs = training_epochs,\n                      verbose = 2,\n                      validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"e4470ac7-6888-4b10-a4e5-8bd4e22301cf","_uuid":"7f94a2f9d7db89bf3fb28ecff64dbf7a496b0b9c","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.031483Z","iopub.status.idle":"2023-06-11T02:39:25.031904Z","shell.execute_reply.started":"2023-06-11T02:39:25.031708Z","shell.execute_reply":"2023-06-11T02:39:25.031726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As it turns out, it does appear to be the case that the optimizer plays a crucial part in the validation score. In particular, the model which relies on 'Adam' as its optimizer tend to perform 1.5 - 2.5% better on average. Going forward, we will use 'Adam' as our optimizer of choice.\n\nWhat if we changed the learning rate from 0.1 to 0.01, or 0.5? Will it have any impact on the accuracy?\nModel 2A","metadata":{"_uuid":"beae0734aefb9f26144b556c753032b66fe99911","_execution_state":"idle","_cell_guid":"e5b9ec27-6585-4f51-be9e-9e86120f723a"}},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)\n\nlearning_rate = 0.01\nadam = keras.optimizers.Adam(lr=learning_rate)\nmodel2a = Model(Inp, output)\n\nmodel2a.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"da528246-4223-42ff-af96-cd3419c34507","_uuid":"d2e88469ac4ed8ac19a195d71941178d8cdb7c32","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.032987Z","iopub.status.idle":"2023-06-11T02:39:25.033353Z","shell.execute_reply.started":"2023-06-11T02:39:25.033170Z","shell.execute_reply":"2023-06-11T02:39:25.033186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history2a = model2a.fit(X_train, y_train,\n                        batch_size = batch_size,\n                        epochs = training_epochs,\n                        verbose = 2,\n                        validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"847391bb-4640-4117-a35e-5593d823bbc4","_uuid":"8e397c8d027a4f27b1f72ac9333de51cbc335cd6","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.034514Z","iopub.status.idle":"2023-06-11T02:39:25.034909Z","shell.execute_reply.started":"2023-06-11T02:39:25.034717Z","shell.execute_reply":"2023-06-11T02:39:25.034735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Model 2B","metadata":{"_uuid":"42520a694b6caf71582c6d5aad5a8599d52e5096","_execution_state":"idle","_cell_guid":"590e8fcc-b3da-41e7-984d-d5d53aa5cb81"}},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)\n\nlearning_rate = 0.5\nadam = keras.optimizers.Adam(lr=learning_rate)\nmodel2b = Model(Inp, output)\n\nmodel2b.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"18ef2f7d-f9a6-4609-8fd4-7d98cb7bc82f","_uuid":"fbc0a1e06296ada3553c9328e48861645680bbf9","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.036022Z","iopub.status.idle":"2023-06-11T02:39:25.036396Z","shell.execute_reply.started":"2023-06-11T02:39:25.036210Z","shell.execute_reply":"2023-06-11T02:39:25.036227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history2b = model2b.fit(X_train, y_train,\n                        batch_size = batch_size,\n                        epochs = training_epochs,\n                            validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"ad3ac83b-a040-42ba-9d62-2203e62a82ec","_uuid":"c2e1a75c4a95acdfd3d5ce81af0c20bce995ca24","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.037397Z","iopub.status.idle":"2023-06-11T02:39:25.037780Z","shell.execute_reply.started":"2023-06-11T02:39:25.037594Z","shell.execute_reply":"2023-06-11T02:39:25.037613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The accuracy, as measured by the 3 different learning rates 0.01, 0.1 and 0.5 are around 98%, 97% and 98% respectively. As there are no considerable gains by changing the learning rates, we stick with the default learning rate of 0.01.\n\nWe proceed to fit a neural network with 5 hidden layers with the features in the hidden layer set as (300, 100, 100, 100, 200) respectively. To ensure that the two models are comparable, we will set the training epochs as 20, and the training batch size as 100.","metadata":{"_uuid":"a36eb4079d02e6d3997d107fc5e3dd3a452a0fc9","_execution_state":"idle","_cell_guid":"9189bb25-f243-4edf-a7a0-943ec633760d"}},{"cell_type":"code","source":"# Input Parameters\nn_input = 784 # number of features\nn_hidden_1 = 300\nn_hidden_2 = 100\nn_hidden_3 = 100\nn_hidden_4 = 100\nn_hidden_5 = 200\nnum_digits = 10","metadata":{"_cell_guid":"25d57f65-533c-432b-bd4a-a3127101d4bb","_uuid":"c19270be516858cca76b3479417e4017877896b1","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.038868Z","iopub.status.idle":"2023-06-11T02:39:25.039277Z","shell.execute_reply.started":"2023-06-11T02:39:25.039083Z","shell.execute_reply":"2023-06-11T02:39:25.039101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\nx = Dense(n_hidden_5, activation='relu', name = \"Hidden_Layer_5\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)","metadata":{"_cell_guid":"d179df94-04b5-4d1e-ab7f-882a72442837","_uuid":"c919e5d20e55f0c86152efbb729feeb0292abf0a","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.040372Z","iopub.status.idle":"2023-06-11T02:39:25.040756Z","shell.execute_reply.started":"2023-06-11T02:39:25.040569Z","shell.execute_reply":"2023-06-11T02:39:25.040586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Our model would have '7' layers - input layer, 5 hidden layer and 1 output layer\nmodel3 = Model(Inp, output)\nmodel3.summary() # We have 308,010 parameters to estimate","metadata":{"_cell_guid":"04f2f013-41ee-48a7-9204-cfe8b748e09c","_uuid":"db189665726670192df8ed9e9758e908821f629b","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.041754Z","iopub.status.idle":"2023-06-11T02:39:25.042122Z","shell.execute_reply.started":"2023-06-11T02:39:25.041939Z","shell.execute_reply":"2023-06-11T02:39:25.041956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We rely on 'Adam' as our optimizing methodology\nadam = keras.optimizers.Adam(lr=0.01)\n\nmodel3.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"3da4e128-5987-4fb3-a2c7-93420c844d1d","_uuid":"304ab7710bc6f2e56d698cc232fadbba6886b664","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.043089Z","iopub.status.idle":"2023-06-11T02:39:25.043443Z","shell.execute_reply.started":"2023-06-11T02:39:25.043260Z","shell.execute_reply":"2023-06-11T02:39:25.043277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history3 = model3.fit(X_train, y_train,\n                      batch_size = batch_size,\n                      epochs = training_epochs,\n                      validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"f7ac3e43-bf76-488d-8ec6-31ec98228ee6","_uuid":"986fc367bbd1fc9767c30a03324c4d18e3bfd0d5","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.044594Z","iopub.status.idle":"2023-06-11T02:39:25.044957Z","shell.execute_reply.started":"2023-06-11T02:39:25.044769Z","shell.execute_reply":"2023-06-11T02:39:25.044785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Compared to our first model, adding an additional layer did not significantly improve the accuracy from our previous model. However, there are computational costs (in terms of complexity) in implementing an additional layer in our neural network. Given that the benefits of an additional layer are low while the costs are high, we will stick with the 4 layer neural network.\n\nWe now proceed to include dropout (dropout rate of 0.3) in our second model to prevent overfitting.","metadata":{"_uuid":"595cd530b900cf25bb1774eabd22afff49b980df","_execution_state":"idle","_cell_guid":"d6d4f220-5431-4221-be1d-2a6ceae86ac9"}},{"cell_type":"code","source":"# Input Parameters\nn_input = 784 # number of features\nn_hidden_1 = 300\nn_hidden_2 = 100\nn_hidden_3 = 100\nn_hidden_4 = 200\nnum_digits = 10","metadata":{"_cell_guid":"9054eb9b-e655-415c-93e7-3f0ca45d84ac","_uuid":"608b71d5bd8dc23df6d916e7371887fd52dd24d8","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T03:05:40.958765Z","iopub.execute_input":"2023-06-11T03:05:40.959191Z","iopub.status.idle":"2023-06-11T03:05:40.965089Z","shell.execute_reply.started":"2023-06-11T03:05:40.959156Z","shell.execute_reply":"2023-06-11T03:05:40.963787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dropout(0.3)(x)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dropout(0.3)(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dropout(0.3)(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)","metadata":{"_cell_guid":"0541e860-b2f6-43cd-8d00-254e9daf775a","_uuid":"d6335bd113992a8898679d1a5d09ed2eda3fe5af","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T03:05:42.801853Z","iopub.execute_input":"2023-06-11T03:05:42.802236Z","iopub.status.idle":"2023-06-11T03:05:42.886797Z","shell.execute_reply.started":"2023-06-11T03:05:42.802206Z","shell.execute_reply":"2023-06-11T03:05:42.885340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Our model would have '6' layers - input layer, 4 hidden layer and 1 output layer\nmodel4 = Model(Inp, output)\nmodel4.summary() # We have 297,910 parameters to estimate","metadata":{"_cell_guid":"95f4299a-0e1c-4dca-b7d5-839e7a5e8fc8","_uuid":"08e56562d89bcd25756bc5bbbb4f33ceb87f106e","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T03:05:45.069193Z","iopub.execute_input":"2023-06-11T03:05:45.069595Z","iopub.status.idle":"2023-06-11T03:05:45.108772Z","shell.execute_reply.started":"2023-06-11T03:05:45.069566Z","shell.execute_reply":"2023-06-11T03:05:45.107370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model4.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"ed1c4a9d-730e-4e4d-8a25-4448d877d5b6","_uuid":"2328fa07b0f4c63448b0c30b7ae450df2b0fe3be","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T03:05:49.274228Z","iopub.execute_input":"2023-06-11T03:05:49.274685Z","iopub.status.idle":"2023-06-11T03:05:49.288917Z","shell.execute_reply.started":"2023-06-11T03:05:49.274645Z","shell.execute_reply":"2023-06-11T03:05:49.287558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model4.fit(X_train, y_train,\n                    batch_size = batch_size,\n                    epochs = training_epochs,\n                    validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"8543232e-da2b-46ce-9a5e-06e6283b854a","_uuid":"986af85bbaf1b212b00cf28dc9204339b717ee48","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T03:05:51.055015Z","iopub.execute_input":"2023-06-11T03:05:51.055626Z","iopub.status.idle":"2023-06-11T03:06:49.924479Z","shell.execute_reply.started":"2023-06-11T03:05:51.055580Z","shell.execute_reply":"2023-06-11T03:06:49.923225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With a validation score of close to 98%, we proceed to use this model to predict for the test set.","metadata":{"_uuid":"ad316467bedf355759a181d353850f1d365ee05b","_execution_state":"idle","_cell_guid":"07d922d5-ec8a-4b3b-8e02-a0aafc3ad52a"}},{"cell_type":"code","source":"test_pred = pd.DataFrame(model4.predict(X_test, batch_size=300))\ntest_pred = pd.DataFrame(test_pred.idxmax(axis = 1))\ntest_pred.index.name = 'ImageId'\ntest_pred = test_pred.rename(columns = {0: 'Label'}).reset_index()\ntest_pred['ImageId'] = test_pred['ImageId'] + 1\ntest_pred.head(10)\n\n","metadata":{"_cell_guid":"d07359be-0cb3-4e0b-9d3e-d8953f67211d","_uuid":"8ae888d3b351dca6417ed4e3830688f1b75701ec","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T03:20:34.202177Z","iopub.execute_input":"2023-06-11T03:20:34.203313Z","iopub.status.idle":"2023-06-11T03:20:35.090979Z","shell.execute_reply.started":"2023-06-11T03:20:34.203264Z","shell.execute_reply":"2023-06-11T03:20:35.089902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\ntest_img = X_test[1,:]\ntest_img2 = test_img.reshape(28,28)\nplt.imshow(test_img2)","metadata":{"execution":{"iopub.status.busy":"2023-06-11T14:05:48.231087Z","iopub.execute_input":"2023-06-11T14:05:48.231499Z","iopub.status.idle":"2023-06-11T14:05:48.492247Z","shell.execute_reply.started":"2023-06-11T14:05:48.231465Z","shell.execute_reply":"2023-06-11T14:05:48.490374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_pred.to_csv('mnist_submission.csv', index = False)","metadata":{"_cell_guid":"2a132e08-be9b-4efa-89bc-6fc9952b6fde","_uuid":"02492c073118b428bb2c4937840e12c35f1a0fd4","_execution_state":"busy","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-06-11T02:39:25.057996Z","iopub.status.idle":"2023-06-11T02:39:25.058375Z","shell.execute_reply.started":"2023-06-11T02:39:25.058189Z","shell.execute_reply":"2023-06-11T02:39:25.058207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using this model, we are able to achieve a score of 0.976, which places us at the top 55th percentile!","metadata":{"_uuid":"515b774eafeacae442e19e3fae989a49f6904837","_execution_state":"idle","_cell_guid":"1510902d-0b15-4b5b-9268-e4f8e1544222"}}]}