{"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3","language":"python"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction\n\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “Hello World” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nIn this competition, we aim to correctly identify digits from a dataset of tens of thousands of handwritten images. Kaggle has curated a set of tutorial-style kernels which cover everything from regression to neural networks. They hope to encourage us to experiment with different algorithms to learn first-hand what works well and how techniques compare.\n\n## Approach\n\nFor this competition, we will be using Keras (with TensorFlow as our backend) as the main package to create a simple neural network to predict, as accurately as we can, digits from handwritten images. In particular, we will be calling the Functional Model API of Keras, and creating a 4-layered and 5-layered neural network.\n\nAlso, we will be experimenting with various optimizers: the plain vanilla Stochastic Gradient Descent optimizer and the Adam optimizer. However, there are many other parameters, such as training epochs which will we will not be experimenting with.\n\nIn addition, the choice of hidden layer units are completely arbitrary and may not be optimal. This is yet another parameter which we will not attempt to tinker with. Lastly, we introduce dropout, a form of regularisation, in our neural networks to prevent overfitting.\n\n## Result\n\nFollowing our simulations on the cross validation dataset, it appears that a 4-layered neural network, using 'Adam' as the optimizer along with a learning rate of 0.01, performs best. We proceed to introduce dropout in the model, and use the model to predict for the test set.\n\nThe test predictions (submitted to Kaggle) generated by our model predicts with an accuracy score of 97.600%, which places us at the top 55 percentile of the competition.","metadata":{"_cell_guid":"82418877-8407-45ef-9b76-86ca80e07abf","_uuid":"483f57b296f8d69647bbec1154acfa2da3c6fc2f","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"Importing key libraries, and reading data","metadata":{"_cell_guid":"440129fa-bb2d-47b8-9415-714467d98743","_uuid":"44f63ca7fc70094c75598c27b433b316ea406237","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nnp.random.seed(1212) # Random number diye initialize variable.\n\nimport keras\nfrom keras.models import Model\nfrom keras.layers import *\nfrom keras import optimizers","metadata":{"_cell_guid":"55ec59e1-8a8c-42d2-8ccc-b6d95ea198b4","_uuid":"a6d625ede9e19ac1f146a5f63d6ca19c4b98e20c","_execution_state":"busy","trusted":true,"execution":{"iopub.status.busy":"2024-12-19T14:57:22.315929Z","iopub.execute_input":"2024-12-19T14:57:22.316237Z","iopub.status.idle":"2024-12-19T14:57:37.038541Z","shell.execute_reply.started":"2024-12-19T14:57:22.316207Z","shell.execute_reply":"2024-12-19T14:57:37.037401Z"}},"outputs":[],"execution_count":1},{"cell_type":"code","source":"!ls /kaggle/input/digit-recognizer/train.csv\n!ls /kaggle/input","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T14:57:37.041408Z","iopub.execute_input":"2024-12-19T14:57:37.042229Z","iopub.status.idle":"2024-12-19T14:57:39.398778Z","shell.execute_reply.started":"2024-12-19T14:57:37.042176Z","shell.execute_reply":"2024-12-19T14:57:39.397476Z"}},"outputs":[{"name":"stdout","text":"ls: cannot access '/kaggle/input/digit-recognizer/train.csv': No such file or directory\nsample_submission.csv  test.csv  train.csv\n","output_type":"stream"}],"execution_count":2},{"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/train.csv')\ndf_test = pd.read_csv('/kaggle/input/test.csv')","metadata":{"_cell_guid":"daa0805e-e1dd-4806-b2db-ff0124febe7d","_uuid":"be0a7acbb3bfc83fd3103b4a401337aafe57c00d","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:39.400831Z","iopub.execute_input":"2024-12-19T14:57:39.401358Z","iopub.status.idle":"2024-12-19T14:57:44.506764Z","shell.execute_reply.started":"2024-12-19T14:57:39.401308Z","shell.execute_reply":"2024-12-19T14:57:44.505614Z"}},"outputs":[],"execution_count":3},{"cell_type":"code","source":"df_train.head() # 784 features, 1 label","metadata":{"_cell_guid":"dfae6a31-23ea-4d88-9fcd-06b8b705b614","_uuid":"0f7bfb5923e5b4ed5a5149c007a2cd39cb9b8282","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:44.508213Z","iopub.execute_input":"2024-12-19T14:57:44.508547Z","iopub.status.idle":"2024-12-19T14:57:44.535653Z","shell.execute_reply.started":"2024-12-19T14:57:44.508515Z","shell.execute_reply":"2024-12-19T14:57:44.534589Z"}},"outputs":[{"execution_count":4,"output_type":"execute_result","data":{"text/plain":"   label  pixel0  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  \\\n0      1       0       0       0       0       0       0       0       0   \n1      0       0       0       0       0       0       0       0       0   \n2      1       0       0       0       0       0       0       0       0   \n3      4       0       0       0       0       0       0       0       0   \n4      0       0       0       0       0       0       0       0       0   \n\n   pixel8  ...  pixel774  pixel775  pixel776  pixel777  pixel778  pixel779  \\\n0       0  ...         0         0         0         0         0         0   \n1       0  ...         0         0         0         0         0         0   \n2       0  ...         0         0         0         0         0         0   \n3       0  ...         0         0         0         0         0         0   \n4       0  ...         0         0         0         0         0         0   \n\n   pixel780  pixel781  pixel782  pixel783  \n0         0         0         0         0  \n1         0         0         0         0  \n2         0         0         0         0  \n3         0         0         0         0  \n4         0         0         0         0  \n\n[5 rows x 785 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>label</th>\n      <th>pixel0</th>\n      <th>pixel1</th>\n      <th>pixel2</th>\n      <th>pixel3</th>\n      <th>pixel4</th>\n      <th>pixel5</th>\n      <th>pixel6</th>\n      <th>pixel7</th>\n      <th>pixel8</th>\n      <th>...</th>\n      <th>pixel774</th>\n      <th>pixel775</th>\n      <th>pixel776</th>\n      <th>pixel777</th>\n      <th>pixel778</th>\n      <th>pixel779</th>\n      <th>pixel780</th>\n      <th>pixel781</th>\n      <th>pixel782</th>\n      <th>pixel783</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 785 columns</p>\n</div>"},"metadata":{}}],"execution_count":4},{"cell_type":"markdown","source":"## Splitting into training and validation dataset","metadata":{"_cell_guid":"45bd232a-4000-4a06-a3ed-0be9f993746c","_uuid":"f0970dcbc0b6791dc44936d8d63e6985dfb6960c","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"df_features = df_train.iloc[:, 1:785] # total column = label + 784 = 785. Pixel = (1,785)\ndf_label = df_train.iloc[:, 0]  # 0th column = label\n\nX_test = df_test.iloc[:, 0:784] # Test dataset dont have labels. We predit the label. \n                                # So the pixels are in 0-784 columns\n\nprint(df_train.shape)","metadata":{"_cell_guid":"bb78680a-ce50-4ee9-a77f-265b9ee38aee","_uuid":"16e39b5512ed29c8f9d7db51331693e4689ce306","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:44.536863Z","iopub.execute_input":"2024-12-19T14:57:44.537225Z","iopub.status.idle":"2024-12-19T14:57:44.544016Z","shell.execute_reply.started":"2024-12-19T14:57:44.537195Z","shell.execute_reply":"2024-12-19T14:57:44.542914Z"}},"outputs":[{"name":"stdout","text":"(42000, 785)\n","output_type":"stream"}],"execution_count":5},{"cell_type":"markdown","source":"sklearn = sci kit learn = scientific kit learn. Open source community. so that it helps others.\n","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_cv, y_train, y_cv = train_test_split(df_features, df_label, \n                                                test_size=0.2,\n                                                random_state=1212)\nprint(X_train.shape)\n\nX_train = X_train.to_numpy().reshape(33600, 784)  \nX_cv = X_cv.to_numpy().reshape(8400, 784)         \n\nX_test = X_test.to_numpy().reshape(28000, 784)  ","metadata":{"_cell_guid":"a3fb0b37-1edf-42bd-b951-86f254b78fc1","_uuid":"a98808f6188e9c44741ad2b5c25a894106406dfc","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:44.545587Z","iopub.execute_input":"2024-12-19T14:57:44.546011Z","iopub.status.idle":"2024-12-19T14:57:45.290418Z","shell.execute_reply.started":"2024-12-19T14:57:44.545981Z","shell.execute_reply":"2024-12-19T14:57:45.289224Z"}},"outputs":[{"name":"stdout","text":"(33600, 784)\n","output_type":"stream"}],"execution_count":6},{"cell_type":"markdown","source":"## Data cleaning, normalization and selection","metadata":{"_cell_guid":"4486ba9e-b8b3-4f09-a597-3d5e6bbd6c1c","_uuid":"9a10a6e1aae9645a7c3e5a703d78d9c3b096f6d3","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"print((min(X_train[1]), max(X_train[1])))","metadata":{"_cell_guid":"4d0e07b5-28dd-42f1-b425-6414aea90cde","_uuid":"cc2daf9f099fb62e3b470f913349428ef396b75b","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.293718Z","iopub.execute_input":"2024-12-19T14:57:45.294347Z","iopub.status.idle":"2024-12-19T14:57:45.300334Z","shell.execute_reply.started":"2024-12-19T14:57:45.294312Z","shell.execute_reply":"2024-12-19T14:57:45.298848Z"}},"outputs":[{"name":"stdout","text":"(0, 255)\n","output_type":"stream"}],"execution_count":7},{"cell_type":"markdown","source":"As the pixel intensities are currently between the range of 0 and 255, we proceed to normalize the features, using broadcasting. In addition, we proceed to convert our labels from a class vector to binary One Hot Encoded","metadata":{"_cell_guid":"60b333cc-a76e-4713-97f5-a9095e27e43b","_uuid":"11d4f22158c013b3f028f5c11340b4a4b7291d1e","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"- 0/255 = 0\n- 110/255 = 0.4\n- 255/255 = 1","metadata":{}},{"cell_type":"markdown","source":"What is one hot encoding?\n\nUSA Russia Africa\n1    0       0\n0    0       1\n0    1       0\n\nCategorical data is expressed as one hot encoding (because model understands bits well).\n\noutput category: 0-9\n  0 1 2 3 4 5 6 7 8 9\n5 0 0 0 0 0 1 0 0 0 0","metadata":{}},{"cell_type":"code","source":"# Feature Normalization \nX_train = X_train.astype('float32') # 33600, the model only sees this.\nX_cv= X_cv.astype('float32') # 8400 , unknown to the model, it has not seen these before.\nX_test = X_test.astype('float32')\n\nX_train /= 255\nX_cv /= 255\nX_test /= 255\n\n# Convert labels to One Hot Encoded\nnum_digits = 10\n#y_train.shape\nfrom tensorflow.keras.utils import to_categorical\n\ny_train = keras.utils.to_categorical(y_train, num_digits)\ny_cv = keras.utils.to_categorical(y_cv, num_digits)","metadata":{"_cell_guid":"e9f8eaa0-c77e-4510-9ff1-bb56e1d5f76d","_uuid":"c24261ffd2384078a836a909ea4839a31d51dd66","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.302274Z","iopub.execute_input":"2024-12-19T14:57:45.302627Z","iopub.status.idle":"2024-12-19T14:57:45.773399Z","shell.execute_reply.started":"2024-12-19T14:57:45.302597Z","shell.execute_reply":"2024-12-19T14:57:45.772478Z"}},"outputs":[],"execution_count":8},{"cell_type":"code","source":"# Printing 2 examples of labels after conversion\nprint(y_train[0]) # 2\nprint(y_train[3]) # 7","metadata":{"_cell_guid":"6667ca35-44ff-43f3-b88b-27b787be4ee0","_uuid":"3fdf37a538504f4e5288cbb8b70a719c2689caca","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.774802Z","iopub.execute_input":"2024-12-19T14:57:45.775289Z","iopub.status.idle":"2024-12-19T14:57:45.782578Z","shell.execute_reply.started":"2024-12-19T14:57:45.775233Z","shell.execute_reply":"2024-12-19T14:57:45.781284Z"}},"outputs":[{"name":"stdout","text":"[0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n","output_type":"stream"}],"execution_count":9},{"cell_type":"markdown","source":"## Model Fitting","metadata":{"_cell_guid":"7d4e437c-12ea-4bad-a580-5504e8a77da0","_uuid":"0776b5f2aff1ebb6f12c2a9ee8e93013c690caf6","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"We proceed by fitting several simple neural network models using Keras (with TensorFlow as our backend) and collect their accuracy. The model that performs the best on the validation set will be used as the model of choice for the competition.\n\nModel 1: Simple Neural Network with 4 layers (300, 100, 100, 200)\n\nIn our first model, we will use the Keras library to train a neural network with the activation function set as ReLu. To determine which class to output, we will rely on the SoftMax function","metadata":{"collapsed":false,"_uuid":"064d83d74bf2e0842b21dcb3cc93f8427b04b6ea","_execution_state":"idle","_cell_guid":"8f441392-a707-499c-8816-00c8aa1a7757","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Input Parameters\nn_input = 784 # number of features\nn_hidden_1 = 300\nn_hidden_2 = 100\nn_hidden_3 = 100\nn_hidden_4 = 300\nn_hidden_5 = 100\nnum_digits = 10","metadata":{"_cell_guid":"be9af3d4-ddac-4f85-945f-bd1cdf09fc15","_uuid":"b88da407b6dab83e54b5380e0e62459577949932","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.784227Z","iopub.execute_input":"2024-12-19T14:57:45.785237Z","iopub.status.idle":"2024-12-19T14:57:45.797722Z","shell.execute_reply.started":"2024-12-19T14:57:45.785186Z","shell.execute_reply":"2024-12-19T14:57:45.796608Z"}},"outputs":[],"execution_count":10},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='sigmoid', name = \"Hidden_Layer_4\")(x)\nx = Dense(n_hidden_5, activation='relu', name = \"Hidden_Layer_5\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)","metadata":{"_cell_guid":"74eac473-a6cb-4356-a982-ef3cb6c971de","_uuid":"f97a558010d88407363bbb08fa00e7c87a3d6ceb","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.799056Z","iopub.execute_input":"2024-12-19T14:57:45.799432Z","iopub.status.idle":"2024-12-19T14:57:45.912289Z","shell.execute_reply.started":"2024-12-19T14:57:45.799401Z","shell.execute_reply":"2024-12-19T14:57:45.911127Z"}},"outputs":[],"execution_count":11},{"cell_type":"code","source":"# Our model would have '6' layers - input layer, 4 hidden layer and 1 output layer\nmodel = Model(Inp, output)\nmodel.summary() # We have 297,910 parameters to estimate","metadata":{"_cell_guid":"bf09781e-6376-4f2d-a872-6292d466f657","_uuid":"aaf8fe1cd6cb32675f0ed5009d7bc31d6f71e4ea","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.91367Z","iopub.execute_input":"2024-12-19T14:57:45.914025Z","iopub.status.idle":"2024-12-19T14:57:45.946512Z","shell.execute_reply.started":"2024-12-19T14:57:45.913991Z","shell.execute_reply":"2024-12-19T14:57:45.945154Z"}},"outputs":[{"output_type":"display_data","data":{"text/plain":"\u001b[1mModel: \"functional_1\"\u001b[0m\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional_1\"</span>\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ input_layer (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m784\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_1 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m300\u001b[0m)            │       \u001b[38;5;34m235,500\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_2 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m30,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_3 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m10,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_4 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m300\u001b[0m)            │        \u001b[38;5;34m30,300\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_5 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m30,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Output_Layer (\u001b[38;5;33mDense\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │         \u001b[38;5;34m1,010\u001b[0m │\n└─────────────────────────────────┴────────────────────────┴───────────────┘\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ input_layer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">784</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">300</span>)            │       <span style=\"color: #00af00; text-decoration-color: #00af00\">235,500</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">30,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_3 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">10,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_4 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">300</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">30,300</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_5 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">30,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Output_Layer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │         <span style=\"color: #00af00; text-decoration-color: #00af00\">1,010</span> │\n└─────────────────────────────────┴────────────────────────┴───────────────┘\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Total params: \u001b[0m\u001b[38;5;34m337,110\u001b[0m (1.29 MB)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">337,110</span> (1.29 MB)\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m337,110\u001b[0m (1.29 MB)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">337,110</span> (1.29 MB)\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n</pre>\n"},"metadata":{}}],"execution_count":12},{"cell_type":"code","source":"learning_rate = 0.1\ntraining_epochs = 20\nbatch_size = 100\n\n# Corrected SGD Optimizer initialization\nsgd = optimizers.SGD(learning_rate=learning_rate)","metadata":{"_cell_guid":"906807f1-ed62-4dee-a423-5f15842fb28f","_uuid":"c4ea7858fbdf41308dcdb5d373af11484d5b5140","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.948182Z","iopub.execute_input":"2024-12-19T14:57:45.948519Z","iopub.status.idle":"2024-12-19T14:57:45.963916Z","shell.execute_reply.started":"2024-12-19T14:57:45.948487Z","shell.execute_reply":"2024-12-19T14:57:45.962175Z"}},"outputs":[],"execution_count":13},{"cell_type":"code","source":"from tensorflow.keras.metrics import Accuracy\n# We rely on the plain vanilla Stochastic Gradient Descent as our optimizing methodology\nmodel.compile(loss='categorical_crossentropy',\n              optimizer='sgd',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"5678ce6d-58eb-4394-9a77-b67c94c2ca4f","_uuid":"775193de659c41c949495a2afa3b14024a2536a7","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.965959Z","iopub.execute_input":"2024-12-19T14:57:45.967239Z","iopub.status.idle":"2024-12-19T14:57:45.980591Z","shell.execute_reply.started":"2024-12-19T14:57:45.967182Z","shell.execute_reply":"2024-12-19T14:57:45.97873Z"}},"outputs":[],"execution_count":14},{"cell_type":"code","source":"history1 = model.fit(X_train, y_train,\n                     batch_size = batch_size,\n                     epochs = training_epochs,\n                     verbose = 2,\n                     validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"1a547d04-6aeb-4c76-8fbc-7fbca1b50b2a","_uuid":"4960dfa65394fb0314ce9cb7af63d48709a39d1a","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:57:45.98266Z","iopub.execute_input":"2024-12-19T14:57:45.983181Z","iopub.status.idle":"2024-12-19T14:58:26.348058Z","shell.execute_reply.started":"2024-12-19T14:57:45.983121Z","shell.execute_reply":"2024-12-19T14:58:26.347019Z"}},"outputs":[{"name":"stdout","text":"Epoch 1/20\n336/336 - 3s - 8ms/step - accuracy: 0.1713 - loss: 2.2795 - val_accuracy: 0.2468 - val_loss: 2.2425\nEpoch 2/20\n336/336 - 2s - 6ms/step - accuracy: 0.3801 - loss: 2.1291 - val_accuracy: 0.4404 - val_loss: 1.9186\nEpoch 3/20\n336/336 - 2s - 7ms/step - accuracy: 0.5462 - loss: 1.5533 - val_accuracy: 0.6286 - val_loss: 1.2072\nEpoch 4/20\n336/336 - 2s - 6ms/step - accuracy: 0.6924 - loss: 0.9953 - val_accuracy: 0.7412 - val_loss: 0.8327\nEpoch 5/20\n336/336 - 2s - 6ms/step - accuracy: 0.7868 - loss: 0.7289 - val_accuracy: 0.8231 - val_loss: 0.6316\nEpoch 6/20\n336/336 - 2s - 6ms/step - accuracy: 0.8371 - loss: 0.5685 - val_accuracy: 0.8564 - val_loss: 0.5030\nEpoch 7/20\n336/336 - 2s - 6ms/step - accuracy: 0.8639 - loss: 0.4689 - val_accuracy: 0.8735 - val_loss: 0.4282\nEpoch 8/20\n336/336 - 2s - 6ms/step - accuracy: 0.8826 - loss: 0.4048 - val_accuracy: 0.8861 - val_loss: 0.3766\nEpoch 9/20\n336/336 - 2s - 6ms/step - accuracy: 0.8955 - loss: 0.3610 - val_accuracy: 0.8974 - val_loss: 0.3423\nEpoch 10/20\n336/336 - 2s - 6ms/step - accuracy: 0.9038 - loss: 0.3297 - val_accuracy: 0.9064 - val_loss: 0.3181\nEpoch 11/20\n336/336 - 2s - 6ms/step - accuracy: 0.9110 - loss: 0.3052 - val_accuracy: 0.9115 - val_loss: 0.2963\nEpoch 12/20\n336/336 - 2s - 6ms/step - accuracy: 0.9160 - loss: 0.2849 - val_accuracy: 0.9176 - val_loss: 0.2796\nEpoch 13/20\n336/336 - 2s - 6ms/step - accuracy: 0.9213 - loss: 0.2673 - val_accuracy: 0.9188 - val_loss: 0.2657\nEpoch 14/20\n336/336 - 2s - 6ms/step - accuracy: 0.9267 - loss: 0.2518 - val_accuracy: 0.9235 - val_loss: 0.2558\nEpoch 15/20\n336/336 - 2s - 6ms/step - accuracy: 0.9314 - loss: 0.2379 - val_accuracy: 0.9263 - val_loss: 0.2422\nEpoch 16/20\n336/336 - 2s - 6ms/step - accuracy: 0.9343 - loss: 0.2254 - val_accuracy: 0.9296 - val_loss: 0.2310\nEpoch 17/20\n336/336 - 2s - 6ms/step - accuracy: 0.9369 - loss: 0.2135 - val_accuracy: 0.9312 - val_loss: 0.2237\nEpoch 18/20\n336/336 - 2s - 6ms/step - accuracy: 0.9412 - loss: 0.2020 - val_accuracy: 0.9362 - val_loss: 0.2118\nEpoch 19/20\n336/336 - 2s - 6ms/step - accuracy: 0.9441 - loss: 0.1920 - val_accuracy: 0.9396 - val_loss: 0.2034\nEpoch 20/20\n336/336 - 2s - 6ms/step - accuracy: 0.9471 - loss: 0.1829 - val_accuracy: 0.9387 - val_loss: 0.1991\n","output_type":"stream"}],"execution_count":15},{"cell_type":"markdown","source":"Using a 4 layer neural network with:\n\n1. 20 training epochs\n2. A training batch size of 100\n3. Hidden layers set as (300, 100, 100, 200)\n4. Learning rate of 0.1\n\nAchieved a training score of around 96-98% and a test score of around 95 - 97%.\n\nCan we do better if we were to change the optimizer? To find out, we use the Adam optimizer for our second model, while maintaining the same parameter values for all other parameters.","metadata":{"collapsed":false,"_uuid":"6121bf7f8adbeaf85b3aef6167444b555b920440","_execution_state":"idle","_cell_guid":"2fee2dfc-23d2-480f-918e-54fb5386c4ea","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)\n\n# We rely on ADAM as our optimizing methodology\nadam = keras.optimizers.Adam(learning_rate=learning_rate)\nmodel2 = Model(Inp, output)\n\nmodel2.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"4e7a2ba9-ac08-4355-a37f-2336220cfda7","_uuid":"3d9c3ffb9722b640ea0682e5826971045091db66","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:58:26.349516Z","iopub.execute_input":"2024-12-19T14:58:26.349819Z","iopub.status.idle":"2024-12-19T14:58:26.393925Z","shell.execute_reply.started":"2024-12-19T14:58:26.349791Z","shell.execute_reply":"2024-12-19T14:58:26.392906Z"}},"outputs":[],"execution_count":16},{"cell_type":"code","source":"history2 = model2.fit(X_train, y_train,\n                      batch_size = batch_size,\n                      epochs = training_epochs,\n                      verbose = 2,\n                      validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"e4470ac7-6888-4b10-a4e5-8bd4e22301cf","_uuid":"7f94a2f9d7db89bf3fb28ecff64dbf7a496b0b9c","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:58:26.395192Z","iopub.execute_input":"2024-12-19T14:58:26.395598Z","iopub.status.idle":"2024-12-19T14:59:11.548911Z","shell.execute_reply.started":"2024-12-19T14:58:26.395556Z","shell.execute_reply":"2024-12-19T14:59:11.547478Z"}},"outputs":[{"name":"stdout","text":"Epoch 1/20\n336/336 - 4s - 12ms/step - accuracy: 0.8967 - loss: 0.3375 - val_accuracy: 0.9562 - val_loss: 0.1477\nEpoch 2/20\n336/336 - 2s - 7ms/step - accuracy: 0.9639 - loss: 0.1191 - val_accuracy: 0.9635 - val_loss: 0.1228\nEpoch 3/20\n336/336 - 2s - 6ms/step - accuracy: 0.9744 - loss: 0.0828 - val_accuracy: 0.9660 - val_loss: 0.1135\nEpoch 4/20\n336/336 - 2s - 6ms/step - accuracy: 0.9813 - loss: 0.0569 - val_accuracy: 0.9661 - val_loss: 0.1089\nEpoch 5/20\n336/336 - 2s - 6ms/step - accuracy: 0.9851 - loss: 0.0455 - val_accuracy: 0.9744 - val_loss: 0.0905\nEpoch 6/20\n336/336 - 2s - 6ms/step - accuracy: 0.9880 - loss: 0.0367 - val_accuracy: 0.9733 - val_loss: 0.1048\nEpoch 7/20\n336/336 - 2s - 6ms/step - accuracy: 0.9904 - loss: 0.0283 - val_accuracy: 0.9771 - val_loss: 0.0929\nEpoch 8/20\n336/336 - 2s - 6ms/step - accuracy: 0.9905 - loss: 0.0294 - val_accuracy: 0.9717 - val_loss: 0.1089\nEpoch 9/20\n336/336 - 2s - 6ms/step - accuracy: 0.9918 - loss: 0.0243 - val_accuracy: 0.9712 - val_loss: 0.1195\nEpoch 10/20\n336/336 - 2s - 6ms/step - accuracy: 0.9939 - loss: 0.0188 - val_accuracy: 0.9727 - val_loss: 0.1199\nEpoch 11/20\n336/336 - 2s - 7ms/step - accuracy: 0.9932 - loss: 0.0212 - val_accuracy: 0.9736 - val_loss: 0.1011\nEpoch 12/20\n336/336 - 2s - 7ms/step - accuracy: 0.9937 - loss: 0.0203 - val_accuracy: 0.9730 - val_loss: 0.1061\nEpoch 13/20\n336/336 - 2s - 7ms/step - accuracy: 0.9948 - loss: 0.0175 - val_accuracy: 0.9767 - val_loss: 0.1068\nEpoch 14/20\n336/336 - 2s - 7ms/step - accuracy: 0.9962 - loss: 0.0128 - val_accuracy: 0.9757 - val_loss: 0.1135\nEpoch 15/20\n336/336 - 2s - 6ms/step - accuracy: 0.9954 - loss: 0.0143 - val_accuracy: 0.9707 - val_loss: 0.1321\nEpoch 16/20\n336/336 - 2s - 7ms/step - accuracy: 0.9950 - loss: 0.0164 - val_accuracy: 0.9754 - val_loss: 0.1238\nEpoch 17/20\n336/336 - 2s - 6ms/step - accuracy: 0.9968 - loss: 0.0098 - val_accuracy: 0.9755 - val_loss: 0.1318\nEpoch 18/20\n336/336 - 2s - 6ms/step - accuracy: 0.9954 - loss: 0.0155 - val_accuracy: 0.9768 - val_loss: 0.1058\nEpoch 19/20\n336/336 - 2s - 6ms/step - accuracy: 0.9974 - loss: 0.0076 - val_accuracy: 0.9750 - val_loss: 0.1380\nEpoch 20/20\n336/336 - 2s - 6ms/step - accuracy: 0.9935 - loss: 0.0199 - val_accuracy: 0.9763 - val_loss: 0.1162\n","output_type":"stream"}],"execution_count":17},{"cell_type":"markdown","source":"As it turns out, it does appear to be the case that the optimizer plays a crucial part in the validation score. In particular, the model which relies on 'Adam' as its optimizer tend to perform 1.5 - 2.5% better on average. Going forward, we will use 'Adam' as our optimizer of choice.\n\nWhat if we changed the learning rate from 0.1 to 0.01, or 0.5? Will it have any impact on the accuracy?\nModel 2A","metadata":{"collapsed":false,"_uuid":"beae0734aefb9f26144b556c753032b66fe99911","_execution_state":"idle","_cell_guid":"e5b9ec27-6585-4f51-be9e-9e86120f723a","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)\n\nlearning_rate = 0.01\nadam = keras.optimizers.Adam(learning_rate=learning_rate)\nmodel2a = Model(Inp, output)\n\nmodel2a.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"da528246-4223-42ff-af96-cd3419c34507","_uuid":"d2e88469ac4ed8ac19a195d71941178d8cdb7c32","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:59:11.550561Z","iopub.execute_input":"2024-12-19T14:59:11.550891Z","iopub.status.idle":"2024-12-19T14:59:11.590548Z","shell.execute_reply.started":"2024-12-19T14:59:11.550858Z","shell.execute_reply":"2024-12-19T14:59:11.589682Z"}},"outputs":[],"execution_count":18},{"cell_type":"code","source":"history2a = model2a.fit(X_train, y_train,\n                        batch_size = batch_size,\n                        epochs = training_epochs,\n                        verbose = 2,\n                        validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"847391bb-4640-4117-a35e-5593d823bbc4","_uuid":"8e397c8d027a4f27b1f72ac9333de51cbc335cd6","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:59:11.591703Z","iopub.execute_input":"2024-12-19T14:59:11.591989Z","iopub.status.idle":"2024-12-19T14:59:57.266599Z","shell.execute_reply.started":"2024-12-19T14:59:11.591962Z","shell.execute_reply":"2024-12-19T14:59:57.265377Z"}},"outputs":[{"name":"stdout","text":"Epoch 1/20\n336/336 - 4s - 11ms/step - accuracy: 0.8956 - loss: 0.3444 - val_accuracy: 0.9510 - val_loss: 0.1725\nEpoch 2/20\n336/336 - 2s - 7ms/step - accuracy: 0.9612 - loss: 0.1240 - val_accuracy: 0.9631 - val_loss: 0.1221\nEpoch 3/20\n336/336 - 2s - 6ms/step - accuracy: 0.9744 - loss: 0.0823 - val_accuracy: 0.9658 - val_loss: 0.1162\nEpoch 4/20\n336/336 - 2s - 6ms/step - accuracy: 0.9816 - loss: 0.0595 - val_accuracy: 0.9735 - val_loss: 0.0974\nEpoch 5/20\n336/336 - 2s - 6ms/step - accuracy: 0.9855 - loss: 0.0434 - val_accuracy: 0.9701 - val_loss: 0.1050\nEpoch 6/20\n336/336 - 2s - 6ms/step - accuracy: 0.9882 - loss: 0.0359 - val_accuracy: 0.9664 - val_loss: 0.1178\nEpoch 7/20\n336/336 - 2s - 6ms/step - accuracy: 0.9884 - loss: 0.0340 - val_accuracy: 0.9729 - val_loss: 0.1050\nEpoch 8/20\n336/336 - 2s - 7ms/step - accuracy: 0.9910 - loss: 0.0264 - val_accuracy: 0.9708 - val_loss: 0.1236\nEpoch 9/20\n336/336 - 2s - 6ms/step - accuracy: 0.9924 - loss: 0.0218 - val_accuracy: 0.9726 - val_loss: 0.1131\nEpoch 10/20\n336/336 - 2s - 6ms/step - accuracy: 0.9928 - loss: 0.0230 - val_accuracy: 0.9743 - val_loss: 0.1127\nEpoch 11/20\n336/336 - 2s - 6ms/step - accuracy: 0.9936 - loss: 0.0190 - val_accuracy: 0.9765 - val_loss: 0.0973\nEpoch 12/20\n336/336 - 2s - 6ms/step - accuracy: 0.9944 - loss: 0.0179 - val_accuracy: 0.9715 - val_loss: 0.1122\nEpoch 13/20\n336/336 - 2s - 6ms/step - accuracy: 0.9949 - loss: 0.0158 - val_accuracy: 0.9752 - val_loss: 0.1191\nEpoch 14/20\n336/336 - 2s - 7ms/step - accuracy: 0.9955 - loss: 0.0140 - val_accuracy: 0.9774 - val_loss: 0.1136\nEpoch 15/20\n336/336 - 2s - 7ms/step - accuracy: 0.9942 - loss: 0.0186 - val_accuracy: 0.9744 - val_loss: 0.1273\nEpoch 16/20\n336/336 - 2s - 7ms/step - accuracy: 0.9952 - loss: 0.0153 - val_accuracy: 0.9756 - val_loss: 0.1108\nEpoch 17/20\n336/336 - 2s - 7ms/step - accuracy: 0.9968 - loss: 0.0106 - val_accuracy: 0.9733 - val_loss: 0.1264\nEpoch 18/20\n336/336 - 2s - 6ms/step - accuracy: 0.9966 - loss: 0.0112 - val_accuracy: 0.9740 - val_loss: 0.1313\nEpoch 19/20\n336/336 - 2s - 6ms/step - accuracy: 0.9950 - loss: 0.0169 - val_accuracy: 0.9786 - val_loss: 0.1050\nEpoch 20/20\n336/336 - 2s - 7ms/step - accuracy: 0.9971 - loss: 0.0089 - val_accuracy: 0.9782 - val_loss: 0.1099\n","output_type":"stream"}],"execution_count":19},{"cell_type":"markdown","source":"Model 2B","metadata":{"collapsed":false,"_uuid":"42520a694b6caf71582c6d5aad5a8599d52e5096","_execution_state":"idle","_cell_guid":"590e8fcc-b3da-41e7-984d-d5d53aa5cb81","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)\n\nlearning_rate = 0.5\nadam = keras.optimizers.Adam(learning_rate=learning_rate)\nmodel2b = Model(Inp, output)\n\nmodel2b.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"18ef2f7d-f9a6-4609-8fd4-7d98cb7bc82f","_uuid":"fbc0a1e06296ada3553c9328e48861645680bbf9","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:59:57.268503Z","iopub.execute_input":"2024-12-19T14:59:57.269303Z","iopub.status.idle":"2024-12-19T14:59:57.313271Z","shell.execute_reply.started":"2024-12-19T14:59:57.269253Z","shell.execute_reply":"2024-12-19T14:59:57.312133Z"}},"outputs":[],"execution_count":20},{"cell_type":"code","source":"history2b = model2b.fit(X_train, y_train,\n                        batch_size = batch_size,\n                        epochs = training_epochs,\n                            validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"ad3ac83b-a040-42ba-9d62-2203e62a82ec","_uuid":"c2e1a75c4a95acdfd3d5ce81af0c20bce995ca24","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T14:59:57.314668Z","iopub.execute_input":"2024-12-19T14:59:57.315117Z","iopub.status.idle":"2024-12-19T15:00:44.295233Z","shell.execute_reply.started":"2024-12-19T14:59:57.315054Z","shell.execute_reply":"2024-12-19T15:00:44.293987Z"}},"outputs":[{"name":"stdout","text":"Epoch 1/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 9ms/step - accuracy: 0.8174 - loss: 0.6122 - val_accuracy: 0.9515 - val_loss: 0.1588\nEpoch 2/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 8ms/step - accuracy: 0.9599 - loss: 0.1301 - val_accuracy: 0.9663 - val_loss: 0.1065\nEpoch 3/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9756 - loss: 0.0768 - val_accuracy: 0.9717 - val_loss: 0.1024\nEpoch 4/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9846 - loss: 0.0497 - val_accuracy: 0.9725 - val_loss: 0.0954\nEpoch 5/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9874 - loss: 0.0392 - val_accuracy: 0.9740 - val_loss: 0.1026\nEpoch 6/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9857 - loss: 0.0427 - val_accuracy: 0.9725 - val_loss: 0.1007\nEpoch 7/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9913 - loss: 0.0286 - val_accuracy: 0.9687 - val_loss: 0.1141\nEpoch 8/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9926 - loss: 0.0220 - val_accuracy: 0.9751 - val_loss: 0.1088\nEpoch 9/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9940 - loss: 0.0194 - val_accuracy: 0.9711 - val_loss: 0.1179\nEpoch 10/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9925 - loss: 0.0223 - val_accuracy: 0.9762 - val_loss: 0.0994\nEpoch 11/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9941 - loss: 0.0184 - val_accuracy: 0.9749 - val_loss: 0.1228\nEpoch 12/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9944 - loss: 0.0179 - val_accuracy: 0.9694 - val_loss: 0.1415\nEpoch 13/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9938 - loss: 0.0229 - val_accuracy: 0.9726 - val_loss: 0.1218\nEpoch 14/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9953 - loss: 0.0148 - val_accuracy: 0.9792 - val_loss: 0.0930\nEpoch 15/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9972 - loss: 0.0091 - val_accuracy: 0.9754 - val_loss: 0.1067\nEpoch 16/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9959 - loss: 0.0114 - val_accuracy: 0.9779 - val_loss: 0.1025\nEpoch 17/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9959 - loss: 0.0127 - val_accuracy: 0.9779 - val_loss: 0.1003\nEpoch 18/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9960 - loss: 0.0124 - val_accuracy: 0.9729 - val_loss: 0.1310\nEpoch 19/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9965 - loss: 0.0113 - val_accuracy: 0.9768 - val_loss: 0.1099\nEpoch 20/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9964 - loss: 0.0109 - val_accuracy: 0.9758 - val_loss: 0.1200\n","output_type":"stream"}],"execution_count":21},{"cell_type":"markdown","source":"The accuracy, as measured by the 3 different learning rates 0.01, 0.1 and 0.5 are around 98%, 97% and 98% respectively. As there are no considerable gains by changing the learning rates, we stick with the default learning rate of 0.01.\n\nWe proceed to fit a neural network with 5 hidden layers with the features in the hidden layer set as (300, 100, 100, 100, 200) respectively. To ensure that the two models are comparable, we will set the training epochs as 20, and the training batch size as 100.","metadata":{"collapsed":false,"_uuid":"a36eb4079d02e6d3997d107fc5e3dd3a452a0fc9","_execution_state":"idle","_cell_guid":"9189bb25-f243-4edf-a7a0-943ec633760d","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Input Parameters\nn_input = 784 # number of features\nn_hidden_1 = 300\nn_hidden_2 = 100\nn_hidden_3 = 100\nn_hidden_4 = 100\nn_hidden_5 = 200\nnum_digits = 10","metadata":{"_cell_guid":"25d57f65-533c-432b-bd4a-a3127101d4bb","_uuid":"c19270be516858cca76b3479417e4017877896b1","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:00:44.300249Z","iopub.execute_input":"2024-12-19T15:00:44.300564Z","iopub.status.idle":"2024-12-19T15:00:44.30523Z","shell.execute_reply.started":"2024-12-19T15:00:44.300535Z","shell.execute_reply":"2024-12-19T15:00:44.304147Z"}},"outputs":[],"execution_count":22},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\nx = Dense(n_hidden_5, activation='relu', name = \"Hidden_Layer_5\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)","metadata":{"_cell_guid":"d179df94-04b5-4d1e-ab7f-882a72442837","_uuid":"c919e5d20e55f0c86152efbb729feeb0292abf0a","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:00:44.306448Z","iopub.execute_input":"2024-12-19T15:00:44.306825Z","iopub.status.idle":"2024-12-19T15:00:44.365097Z","shell.execute_reply.started":"2024-12-19T15:00:44.306794Z","shell.execute_reply":"2024-12-19T15:00:44.364119Z"}},"outputs":[],"execution_count":23},{"cell_type":"code","source":"# Our model would have '7' layers - input layer, 5 hidden layer and 1 output layer\nmodel3 = Model(Inp, output)\nmodel3.summary() # We have 308,010 parameters to estimate","metadata":{"_cell_guid":"04f2f013-41ee-48a7-9204-cfe8b748e09c","_uuid":"db189665726670192df8ed9e9758e908821f629b","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:00:44.366494Z","iopub.execute_input":"2024-12-19T15:00:44.367269Z","iopub.status.idle":"2024-12-19T15:00:44.3946Z","shell.execute_reply.started":"2024-12-19T15:00:44.367221Z","shell.execute_reply":"2024-12-19T15:00:44.39354Z"}},"outputs":[{"output_type":"display_data","data":{"text/plain":"\u001b[1mModel: \"functional_9\"\u001b[0m\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional_9\"</span>\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ input_layer_4 (\u001b[38;5;33mInputLayer\u001b[0m)      │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m784\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_1 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m300\u001b[0m)            │       \u001b[38;5;34m235,500\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_2 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m30,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_3 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m10,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_4 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m10,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_5 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m200\u001b[0m)            │        \u001b[38;5;34m20,200\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Output_Layer (\u001b[38;5;33mDense\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │         \u001b[38;5;34m2,010\u001b[0m │\n└─────────────────────────────────┴────────────────────────┴───────────────┘\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ input_layer_4 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)      │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">784</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">300</span>)            │       <span style=\"color: #00af00; text-decoration-color: #00af00\">235,500</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">30,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_3 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">10,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_4 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">10,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_5 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">200</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">20,200</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Output_Layer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │         <span style=\"color: #00af00; text-decoration-color: #00af00\">2,010</span> │\n└─────────────────────────────────┴────────────────────────┴───────────────┘\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Total params: \u001b[0m\u001b[38;5;34m308,010\u001b[0m (1.17 MB)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">308,010</span> (1.17 MB)\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m308,010\u001b[0m (1.17 MB)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">308,010</span> (1.17 MB)\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n</pre>\n"},"metadata":{}}],"execution_count":24},{"cell_type":"code","source":"# We rely on 'Adam' as our optimizing methodology\nadam = keras.optimizers.Adam(learning_rate=0.01)\n\nmodel3.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"3da4e128-5987-4fb3-a2c7-93420c844d1d","_uuid":"304ab7710bc6f2e56d698cc232fadbba6886b664","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:00:44.396157Z","iopub.execute_input":"2024-12-19T15:00:44.396596Z","iopub.status.idle":"2024-12-19T15:00:44.408799Z","shell.execute_reply.started":"2024-12-19T15:00:44.396549Z","shell.execute_reply":"2024-12-19T15:00:44.407771Z"}},"outputs":[],"execution_count":25},{"cell_type":"code","source":"history3 = model3.fit(X_train, y_train,\n                      batch_size = batch_size,\n                      epochs = training_epochs,\n                      validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"f7ac3e43-bf76-488d-8ec6-31ec98228ee6","_uuid":"986fc367bbd1fc9767c30a03324c4d18e3bfd0d5","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:00:44.41031Z","iopub.execute_input":"2024-12-19T15:00:44.410753Z","iopub.status.idle":"2024-12-19T15:01:31.043384Z","shell.execute_reply.started":"2024-12-19T15:00:44.410705Z","shell.execute_reply":"2024-12-19T15:01:31.042331Z"}},"outputs":[{"name":"stdout","text":"Epoch 1/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 7ms/step - accuracy: 0.7976 - loss: 0.6691 - val_accuracy: 0.9544 - val_loss: 0.1533\nEpoch 2/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9613 - loss: 0.1289 - val_accuracy: 0.9590 - val_loss: 0.1359\nEpoch 3/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9731 - loss: 0.0878 - val_accuracy: 0.9712 - val_loss: 0.0937\nEpoch 4/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9822 - loss: 0.0574 - val_accuracy: 0.9705 - val_loss: 0.1035\nEpoch 5/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9853 - loss: 0.0453 - val_accuracy: 0.9704 - val_loss: 0.1033\nEpoch 6/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9887 - loss: 0.0377 - val_accuracy: 0.9729 - val_loss: 0.0973\nEpoch 7/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9903 - loss: 0.0292 - val_accuracy: 0.9633 - val_loss: 0.1402\nEpoch 8/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9895 - loss: 0.0330 - val_accuracy: 0.9687 - val_loss: 0.1160\nEpoch 9/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9936 - loss: 0.0220 - val_accuracy: 0.9749 - val_loss: 0.1032\nEpoch 10/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9945 - loss: 0.0183 - val_accuracy: 0.9775 - val_loss: 0.0978\nEpoch 11/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9933 - loss: 0.0212 - val_accuracy: 0.9739 - val_loss: 0.1082\nEpoch 12/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9951 - loss: 0.0152 - val_accuracy: 0.9774 - val_loss: 0.1057\nEpoch 13/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9945 - loss: 0.0176 - val_accuracy: 0.9773 - val_loss: 0.1164\nEpoch 14/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9938 - loss: 0.0205 - val_accuracy: 0.9742 - val_loss: 0.1112\nEpoch 15/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9967 - loss: 0.0106 - val_accuracy: 0.9767 - val_loss: 0.1202\nEpoch 16/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9947 - loss: 0.0157 - val_accuracy: 0.9752 - val_loss: 0.1150\nEpoch 17/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 7ms/step - accuracy: 0.9963 - loss: 0.0114 - val_accuracy: 0.9769 - val_loss: 0.1102\nEpoch 18/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9962 - loss: 0.0123 - val_accuracy: 0.9771 - val_loss: 0.1180\nEpoch 19/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 6ms/step - accuracy: 0.9965 - loss: 0.0148 - val_accuracy: 0.9729 - val_loss: 0.1232\nEpoch 20/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9958 - loss: 0.0122 - val_accuracy: 0.9782 - val_loss: 0.1083\n","output_type":"stream"}],"execution_count":26},{"cell_type":"markdown","source":"Compared to our first model, adding an additional layer did not significantly improve the accuracy from our previous model. However, there are computational costs (in terms of complexity) in implementing an additional layer in our neural network. Given that the benefits of an additional layer are low while the costs are high, we will stick with the 4 layer neural network.\n\nWe now proceed to include dropout (dropout rate of 0.3) in our second model to prevent overfitting.","metadata":{"collapsed":false,"_uuid":"595cd530b900cf25bb1774eabd22afff49b980df","_execution_state":"idle","_cell_guid":"d6d4f220-5431-4221-be1d-2a6ceae86ac9","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Input Parameters\nn_input = 784 # number of features\nn_hidden_1 = 300\nn_hidden_2 = 100\nn_hidden_3 = 100\nn_hidden_4 = 200\nnum_digits = 10","metadata":{"_cell_guid":"9054eb9b-e655-415c-93e7-3f0ca45d84ac","_uuid":"608b71d5bd8dc23df6d916e7371887fd52dd24d8","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:01:31.045399Z","iopub.execute_input":"2024-12-19T15:01:31.045839Z","iopub.status.idle":"2024-12-19T15:01:31.051435Z","shell.execute_reply.started":"2024-12-19T15:01:31.045792Z","shell.execute_reply":"2024-12-19T15:01:31.050335Z"}},"outputs":[],"execution_count":27},{"cell_type":"code","source":"Inp = Input(shape=(784,))\nx = Dense(n_hidden_1, activation='relu', name = \"Hidden_Layer_1\")(Inp)\nx = Dropout(0.3)(x)\nx = Dense(n_hidden_2, activation='relu', name = \"Hidden_Layer_2\")(x)\nx = Dropout(0.3)(x)\nx = Dense(n_hidden_3, activation='relu', name = \"Hidden_Layer_3\")(x)\nx = Dropout(0.3)(x)\nx = Dense(n_hidden_4, activation='relu', name = \"Hidden_Layer_4\")(x)\noutput = Dense(num_digits, activation='softmax', name = \"Output_Layer\")(x)","metadata":{"_cell_guid":"0541e860-b2f6-43cd-8d00-254e9daf775a","_uuid":"d6335bd113992a8898679d1a5d09ed2eda3fe5af","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:01:31.053222Z","iopub.execute_input":"2024-12-19T15:01:31.053647Z","iopub.status.idle":"2024-12-19T15:01:31.106309Z","shell.execute_reply.started":"2024-12-19T15:01:31.053587Z","shell.execute_reply":"2024-12-19T15:01:31.105154Z"}},"outputs":[],"execution_count":28},{"cell_type":"code","source":"# Our model would have '6' layers - input layer, 4 hidden layer and 1 output layer\nmodel4 = Model(Inp, output)\nmodel4.summary() # We have 297,910 parameters to estimate","metadata":{"_cell_guid":"95f4299a-0e1c-4dca-b7d5-839e7a5e8fc8","_uuid":"08e56562d89bcd25756bc5bbbb4f33ceb87f106e","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:01:31.107707Z","iopub.execute_input":"2024-12-19T15:01:31.108106Z","iopub.status.idle":"2024-12-19T15:01:31.136871Z","shell.execute_reply.started":"2024-12-19T15:01:31.108047Z","shell.execute_reply":"2024-12-19T15:01:31.135893Z"}},"outputs":[{"output_type":"display_data","data":{"text/plain":"\u001b[1mModel: \"functional_11\"\u001b[0m\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional_11\"</span>\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ input_layer_5 (\u001b[38;5;33mInputLayer\u001b[0m)      │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m784\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_1 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m300\u001b[0m)            │       \u001b[38;5;34m235,500\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ dropout (\u001b[38;5;33mDropout\u001b[0m)               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m300\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_2 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m30,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ dropout_1 (\u001b[38;5;33mDropout\u001b[0m)             │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_3 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │        \u001b[38;5;34m10,100\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ dropout_2 (\u001b[38;5;33mDropout\u001b[0m)             │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_4 (\u001b[38;5;33mDense\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m200\u001b[0m)            │        \u001b[38;5;34m20,200\u001b[0m │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Output_Layer (\u001b[38;5;33mDense\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │         \u001b[38;5;34m2,010\u001b[0m │\n└─────────────────────────────────┴────────────────────────┴───────────────┘\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ input_layer_5 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)      │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">784</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">300</span>)            │       <span style=\"color: #00af00; text-decoration-color: #00af00\">235,500</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ dropout (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dropout</span>)               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">300</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">30,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ dropout_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dropout</span>)             │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_3 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">10,100</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ dropout_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dropout</span>)             │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Hidden_Layer_4 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">200</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">20,200</span> │\n├─────────────────────────────────┼────────────────────────┼───────────────┤\n│ Output_Layer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │         <span style=\"color: #00af00; text-decoration-color: #00af00\">2,010</span> │\n└─────────────────────────────────┴────────────────────────┴───────────────┘\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Total params: \u001b[0m\u001b[38;5;34m297,910\u001b[0m (1.14 MB)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">297,910</span> (1.14 MB)\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m297,910\u001b[0m (1.14 MB)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">297,910</span> (1.14 MB)\n</pre>\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n","text/html":"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n</pre>\n"},"metadata":{}}],"execution_count":29},{"cell_type":"code","source":"model4.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"_cell_guid":"ed1c4a9d-730e-4e4d-8a25-4448d877d5b6","_uuid":"2328fa07b0f4c63448b0c30b7ae450df2b0fe3be","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:01:31.138042Z","iopub.execute_input":"2024-12-19T15:01:31.138369Z","iopub.status.idle":"2024-12-19T15:01:31.14905Z","shell.execute_reply.started":"2024-12-19T15:01:31.13834Z","shell.execute_reply":"2024-12-19T15:01:31.147841Z"}},"outputs":[],"execution_count":30},{"cell_type":"code","source":"history = model4.fit(X_train, y_train,\n                    batch_size = batch_size,\n                    epochs = training_epochs,\n                    validation_data=(X_cv, y_cv))","metadata":{"_cell_guid":"8543232e-da2b-46ce-9a5e-06e6283b854a","_uuid":"986af85bbaf1b212b00cf28dc9204339b717ee48","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:01:31.150305Z","iopub.execute_input":"2024-12-19T15:01:31.150664Z","iopub.status.idle":"2024-12-19T15:02:21.009369Z","shell.execute_reply.started":"2024-12-19T15:01:31.150613Z","shell.execute_reply":"2024-12-19T15:02:21.008199Z"}},"outputs":[{"name":"stdout","text":"Epoch 1/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 7ms/step - accuracy: 0.6633 - loss: 1.0000 - val_accuracy: 0.9471 - val_loss: 0.1799\nEpoch 2/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9244 - loss: 0.2519 - val_accuracy: 0.9593 - val_loss: 0.1368\nEpoch 3/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 8ms/step - accuracy: 0.9477 - loss: 0.1767 - val_accuracy: 0.9660 - val_loss: 0.1173\nEpoch 4/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9595 - loss: 0.1374 - val_accuracy: 0.9700 - val_loss: 0.1039\nEpoch 5/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9645 - loss: 0.1226 - val_accuracy: 0.9701 - val_loss: 0.1038\nEpoch 6/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9676 - loss: 0.1061 - val_accuracy: 0.9696 - val_loss: 0.1064\nEpoch 7/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9726 - loss: 0.0940 - val_accuracy: 0.9739 - val_loss: 0.0909\nEpoch 8/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9743 - loss: 0.0861 - val_accuracy: 0.9743 - val_loss: 0.0930\nEpoch 9/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9750 - loss: 0.0825 - val_accuracy: 0.9745 - val_loss: 0.0917\nEpoch 10/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9804 - loss: 0.0672 - val_accuracy: 0.9763 - val_loss: 0.0881\nEpoch 11/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9821 - loss: 0.0634 - val_accuracy: 0.9774 - val_loss: 0.0855\nEpoch 12/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9816 - loss: 0.0602 - val_accuracy: 0.9774 - val_loss: 0.0854\nEpoch 13/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9832 - loss: 0.0564 - val_accuracy: 0.9755 - val_loss: 0.0957\nEpoch 14/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9837 - loss: 0.0559 - val_accuracy: 0.9786 - val_loss: 0.0821\nEpoch 15/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9844 - loss: 0.0525 - val_accuracy: 0.9761 - val_loss: 0.0986\nEpoch 16/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9835 - loss: 0.0531 - val_accuracy: 0.9779 - val_loss: 0.0913\nEpoch 17/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9846 - loss: 0.0517 - val_accuracy: 0.9779 - val_loss: 0.0863\nEpoch 18/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9870 - loss: 0.0453 - val_accuracy: 0.9789 - val_loss: 0.0893\nEpoch 19/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9854 - loss: 0.0475 - val_accuracy: 0.9796 - val_loss: 0.0877\nEpoch 20/20\n\u001b[1m336/336\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 8ms/step - accuracy: 0.9883 - loss: 0.0393 - val_accuracy: 0.9790 - val_loss: 0.0817\n","output_type":"stream"}],"execution_count":31},{"cell_type":"markdown","source":"With a validation score of close to 98%, we proceed to use this model to predict for the test set.","metadata":{"collapsed":false,"_uuid":"ad316467bedf355759a181d353850f1d365ee05b","_execution_state":"idle","_cell_guid":"07d922d5-ec8a-4b3b-8e02-a0aafc3ad52a","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"X_test.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T15:02:21.011224Z","iopub.execute_input":"2024-12-19T15:02:21.011561Z","iopub.status.idle":"2024-12-19T15:02:21.018792Z","shell.execute_reply.started":"2024-12-19T15:02:21.011529Z","shell.execute_reply":"2024-12-19T15:02:21.017737Z"}},"outputs":[{"execution_count":32,"output_type":"execute_result","data":{"text/plain":"(28000, 784)"},"metadata":{}}],"execution_count":32},{"cell_type":"markdown","source":"### How to predict using our trained model?","metadata":{}},{"cell_type":"code","source":"X_test.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T15:02:21.019954Z","iopub.execute_input":"2024-12-19T15:02:21.020289Z","iopub.status.idle":"2024-12-19T15:02:21.03983Z","shell.execute_reply.started":"2024-12-19T15:02:21.020258Z","shell.execute_reply":"2024-12-19T15:02:21.038317Z"}},"outputs":[{"execution_count":33,"output_type":"execute_result","data":{"text/plain":"(28000, 784)"},"metadata":{}}],"execution_count":33},{"cell_type":"code","source":"test_pred = pd.DataFrame(model4.predict(X_test, batch_size=100))\ntest_pred.head(5)","metadata":{"_cell_guid":"d07359be-0cb3-4e0b-9d3e-d8953f67211d","_uuid":"8ae888d3b351dca6417ed4e3830688f1b75701ec","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:02:21.041513Z","iopub.execute_input":"2024-12-19T15:02:21.041973Z","iopub.status.idle":"2024-12-19T15:02:22.171295Z","shell.execute_reply.started":"2024-12-19T15:02:21.041926Z","shell.execute_reply":"2024-12-19T15:02:22.170144Z"}},"outputs":[{"name":"stdout","text":"\u001b[1m280/280\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 3ms/step\n","output_type":"stream"},{"execution_count":34,"output_type":"execute_result","data":{"text/plain":"              0             1             2             3             4  \\\n0  3.402846e-13  2.338548e-13  9.999999e-01  9.678045e-10  2.332372e-15   \n1  9.999999e-01  1.064208e-18  1.248294e-13  1.146679e-13  4.235819e-15   \n2  1.265045e-06  6.706902e-07  5.052899e-06  8.536885e-06  3.444730e-03   \n3  4.064424e-02  1.196780e-03  3.090780e-02  2.364903e-02  3.387742e-02   \n4  1.903240e-09  2.186203e-07  2.468035e-05  9.998929e-01  3.309548e-11   \n\n              5             6             7             8             9  \n0  1.129738e-16  3.869711e-15  4.060446e-11  2.367969e-13  7.164737e-19  \n1  2.546703e-10  9.032844e-10  4.652869e-13  4.782681e-13  1.218489e-09  \n2  4.739626e-06  1.892163e-08  6.530016e-05  4.881574e-06  9.964648e-01  \n3  8.741324e-03  9.433114e-04  2.824686e-02  2.369678e-02  8.080965e-01  \n4  7.006975e-05  2.609030e-11  6.335559e-07  1.082946e-05  6.129208e-07  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n      <th>5</th>\n      <th>6</th>\n      <th>7</th>\n      <th>8</th>\n      <th>9</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3.402846e-13</td>\n      <td>2.338548e-13</td>\n      <td>9.999999e-01</td>\n      <td>9.678045e-10</td>\n      <td>2.332372e-15</td>\n      <td>1.129738e-16</td>\n      <td>3.869711e-15</td>\n      <td>4.060446e-11</td>\n      <td>2.367969e-13</td>\n      <td>7.164737e-19</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>9.999999e-01</td>\n      <td>1.064208e-18</td>\n      <td>1.248294e-13</td>\n      <td>1.146679e-13</td>\n      <td>4.235819e-15</td>\n      <td>2.546703e-10</td>\n      <td>9.032844e-10</td>\n      <td>4.652869e-13</td>\n      <td>4.782681e-13</td>\n      <td>1.218489e-09</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1.265045e-06</td>\n      <td>6.706902e-07</td>\n      <td>5.052899e-06</td>\n      <td>8.536885e-06</td>\n      <td>3.444730e-03</td>\n      <td>4.739626e-06</td>\n      <td>1.892163e-08</td>\n      <td>6.530016e-05</td>\n      <td>4.881574e-06</td>\n      <td>9.964648e-01</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>4.064424e-02</td>\n      <td>1.196780e-03</td>\n      <td>3.090780e-02</td>\n      <td>2.364903e-02</td>\n      <td>3.387742e-02</td>\n      <td>8.741324e-03</td>\n      <td>9.433114e-04</td>\n      <td>2.824686e-02</td>\n      <td>2.369678e-02</td>\n      <td>8.080965e-01</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1.903240e-09</td>\n      <td>2.186203e-07</td>\n      <td>2.468035e-05</td>\n      <td>9.998929e-01</td>\n      <td>3.309548e-11</td>\n      <td>7.006975e-05</td>\n      <td>2.609030e-11</td>\n      <td>6.335559e-07</td>\n      <td>1.082946e-05</td>\n      <td>6.129208e-07</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}],"execution_count":34},{"cell_type":"code","source":"test_pred = pd.DataFrame(test_pred.idxmax(axis = 1))\ntest_pred.head(15)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T15:02:22.172531Z","iopub.execute_input":"2024-12-19T15:02:22.17283Z","iopub.status.idle":"2024-12-19T15:02:22.190312Z","shell.execute_reply.started":"2024-12-19T15:02:22.172802Z","shell.execute_reply":"2024-12-19T15:02:22.189193Z"}},"outputs":[{"execution_count":35,"output_type":"execute_result","data":{"text/plain":"    0\n0   2\n1   0\n2   9\n3   9\n4   3\n5   7\n6   0\n7   3\n8   0\n9   3\n10  5\n11  7\n12  4\n13  0\n14  4","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>9</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>9</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>7</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>5</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>7</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}],"execution_count":35},{"cell_type":"code","source":"test_pred.index.name = 'ImageId'\ntest_pred = test_pred.rename(columns = {0: 'Label'}).reset_index()\ntest_pred['ImageId'] = test_pred['ImageId'] + 1\n\ntest_pred.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T15:02:22.191499Z","iopub.execute_input":"2024-12-19T15:02:22.191946Z","iopub.status.idle":"2024-12-19T15:02:22.204757Z","shell.execute_reply.started":"2024-12-19T15:02:22.191913Z","shell.execute_reply":"2024-12-19T15:02:22.203463Z"}},"outputs":[{"execution_count":36,"output_type":"execute_result","data":{"text/plain":"   ImageId  Label\n0        1      2\n1        2      0\n2        3      9\n3        4      9\n4        5      3","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>ImageId</th>\n      <th>Label</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>3</td>\n      <td>9</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>4</td>\n      <td>9</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>5</td>\n      <td>3</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}],"execution_count":36},{"cell_type":"code","source":"test_pred.to_csv('mnist_submission.csv', index = False)","metadata":{"_cell_guid":"2a132e08-be9b-4efa-89bc-6fc9952b6fde","_uuid":"02492c073118b428bb2c4937840e12c35f1a0fd4","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T15:02:22.206161Z","iopub.execute_input":"2024-12-19T15:02:22.206511Z","iopub.status.idle":"2024-12-19T15:02:22.241367Z","shell.execute_reply.started":"2024-12-19T15:02:22.206479Z","shell.execute_reply":"2024-12-19T15:02:22.240246Z"}},"outputs":[],"execution_count":37},{"cell_type":"code","source":"import joblib\nmypreciousmodel = joblib.load(\"model4.modelFile\") # I will sell model4.modelFile this file for 1M dollar\n#mypreciousmodel.predict(checkImage)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T15:05:54.588434Z","iopub.execute_input":"2024-12-19T15:05:54.588834Z","iopub.status.idle":"2024-12-19T15:05:54.727531Z","shell.execute_reply.started":"2024-12-19T15:05:54.588799Z","shell.execute_reply":"2024-12-19T15:05:54.726347Z"}},"outputs":[],"execution_count":40},{"cell_type":"markdown","source":"Using this model, we are able to achieve a score of 0.976, which places us at the top 55th percentile!","metadata":{"collapsed":false,"_uuid":"515b774eafeacae442e19e3fae989a49f6904837","_execution_state":"idle","_cell_guid":"1510902d-0b15-4b5b-9268-e4f8e1544222","jupyter":{"outputs_hidden":false}}}]}