{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"},{"sourceId":13686764,"sourceType":"datasetVersion","datasetId":8704735}],"dockerImageVersionId":29840,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0\"></a>\n# MNIST - Deep Neural Network with Keras","metadata":{"_uuid":"fc9d522c-fbe4-44d2-8bb6-17436445cad2","_cell_guid":"d5dcdbcf-5e41-4e84-9c10-8984f46c7a6e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"Hello friends,\n\n\nIn this notebook, I have built a deep neural network on MNIST handwritten digit images to classify them. \n\n\nMNIST is called Hello World of Deep Learning.\n\n\nSo, it is actually an image recognition task.\n\n\nIt helps me to understand and built a deep neural network with Keras. \n\n\nI hope that this notebook will help to understand the same.\n\nPlease take notes regarding the different concepts used here, and answer the questions from the comments in your notebook. \n- Aadhvika K","metadata":{"_uuid":"c7099ad7-4a42-4909-bbd9-43103f90beed","_cell_guid":"65d04a6c-e730-4932-8a5c-7ea6c3426bea","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"This kernel is based on the following Kaggle kernel:\n\n- [My previous kernel on Artificial Neural Network](https://www.kaggle.com/prashant111/comprehensive-guide-to-ann-with-keras)\n\n- [Deep Neural Network Keras way by Poonam Ligade](https://www.kaggle.com/poonaml/deep-neural-network-keras-way)\n\n- [MNIST with Keras for beginners by Aditya Soni](https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457)\n\n\nI thank the authors (Poonam Ligade and Aditya Soni) for their excellent work.","metadata":{"_uuid":"ab3c424b-6a7d-466b-b5b2-044e05c7fdb0","_cell_guid":"b277b571-30d7-4148-87d1-3bfcdbcca90a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"**I hope you find this kernel useful and some <font color=\"red\"><b>UPVOTES</b></font> would be very much appreciated**.","metadata":{"_uuid":"c7ee0280-1b50-4ab9-be27-74bb90921d29","_cell_guid":"eb5f474d-5644-4ea0-bf27-36cc4c43e95c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0.1\"></a>\n# Table of contents\n\n\n1.\t[Objective of the notebook](#1)\n1.\t[Import necessary libraries](#2)\n1.\t[MNIST dataset](#3)\n1.\t[Data visualization](#4)\n1.\t[Designing model architecture using Keras](#5)\n    - 5.1 [Import Keras layers](#5.1)\n    - 5.2 [Compute the number of labels](#5.2)\n    - 5.3 [One-Hot Encoding](#5.3)\n    - 5.4 [Data Preprocessing](#5.4)\n    - 5.5 [Setting network parameters](#5.5)\n    - 5.6 [Designing the model architecture](#5.6)\n    - 5.7 [View model summary](#5.7)\n1.\t[Implement MLP model using Keras](#6)\n    - 6.1 [Compile the model with compile() method](#6.1)\n    - 6.2 [Train the model with fit() method](#6.2)\n    - 6.3 [Evaluate model performance with evaluate() method](#6.3)\n1.\t[Overfitting and Regularization](#7)\n1.\t[Results and Conclusion](#8)\n1.\t[References](#9)","metadata":{"_uuid":"98ea5f33-f7d4-4f92-8461-9bb285e0401b","_cell_guid":"11f99feb-67ae-4957-bee7-37f8a5c7ffb4","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 1. Objective of the notebook <a class=\"anchor\" id=\"1\"></a>\n\n[Back to Table of Contents](#0.1)\n\n- In this notebook, we create a **Multilayer Perceptron (MLP)** model of the MNIST dataset. \n\n- **Multilayer Perceptrons (MLPs)** usually mean fully connected networks, that is, each neuron in one layer is connected to all neurons in the next layer. The \"fully-connectedness\" of these networks makes them prone to overfitting the data. \n\n- These MLP models are also referred to as either **deep feedforward networks** or **feedforward neural networks**. MLPs are common in simple logistic and linear regression problems.\n\n- So, the objective is to create a neural network for identifying numbers based on handwritten digits. For example, when the input to the network is an image of a handwritten number 8, the corresponding prediction must also be the digit 8. \n\n- To both train and validate a neural network, there must be a sufficiently large dataset of handwritten digits.\n\n- The Modified National Institute of Standards and Technology dataset or MNIST dataset for short, is often considered as the Hello World! of deep learning and is a suitable dataset for handwritten digit classification.\n\n- MNIST is used to explain and validate deep learning theories because the 70,000 samples it contains are small, yet sufficiently rich in information (MNIST dataset is described later).","metadata":{"_uuid":"36c6b585-7ef9-496d-8a10-53986712cb8d","_cell_guid":"d9708a01-1566-49ff-8bd9-dfda0c4ab76b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"So, we will start by importing the necessary libraries:","metadata":{"_uuid":"902e9b8b-5735-4031-ba86-6d85ab1184d4","_cell_guid":"fdc5098f-926b-453a-b6f5-932ad4a857f8","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 2. Import necessary libraries <a class=\"anchor\" id=\"2\"></a>\n\n[Back to Table of Contents](#0.1)","metadata":{"_uuid":"dcd5a803-0084-4ba3-afb5-4f4482246eb2","_cell_guid":"c128914f-9d91-4097-b7d5-355c563dcefa","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n# Keep an eye on the different modules/libraries you import as I will be questioning you about them later. \nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\nimport matplotlib.pyplot as plt # plotting library\n%matplotlib inline\n\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense , Activation, Dropout # Activation function lets the network learn complex function and non linear relationship\n                                                     # Dense function connects every neurons. It's also called a fully connected layer.\n                                                     # Dropout randomly turns off some neurons during the training\nfrom keras.optimizers import Adam ,RMSprop # RMSprop adjusts learning rate for each weight based on gradient size\n                                           # Adam has faster and more stable learning\n                                           # An optimizer decides how the model updates its weight during training so the predictions get better.\nfrom keras import  backend as K\n\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n\n# Any results you write to the current directory are saved as output.","metadata":{"_uuid":"0e424503-b4a8-4b96-8521-0775715d5c11","_cell_guid":"a736dd07-cf76-4075-ab33-8b0c7900748b","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:41.759211Z","iopub.execute_input":"2025-12-02T23:06:41.759497Z","iopub.status.idle":"2025-12-02T23:06:41.801976Z","shell.execute_reply.started":"2025-12-02T23:06:41.759450Z","shell.execute_reply":"2025-12-02T23:06:41.800592Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. MNIST dataset <a class=\"anchor\" id=\"3\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- MNIST is a collection of handwritten digits ranging from the number 0 to 9. \n\n- It has a training set of 60,000 images, and 10,000 test images that are classified into corresponding categories or labels. \n\n- To use the MNIST dataset in Keras, an API is provided to download and extract images and labels automatically. \n\n- The following Keras code shows how to access MNIST dataset, plot 25 random samples, and count the number of labels for train and test datasets.","metadata":{"_uuid":"a2d08fc7-c8df-42f9-9d43-21f3f4558f9f","_cell_guid":"10a124e0-6cb9-403f-a303-9178e355a5a8","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# path to your uploaded mnist.npz in Kaggle Dataset\npath = \"/kaggle/input/mnist-database/mnist.npz\"\n\n# load the MNIST file directly (by passing Keras download)\nwith np.load(path) as data:\n    x_train, y_train = data[\"x_train\"], data[\"y_train\"]\n    x_test, y_test = data[\"x_test\"], data[\"y_test\"]\n    \n# count the number of unique train labels\n# The unique function is part of NumPy, and it's used to find all the unique (distinct) elements in an array\nunique, counts = np.unique(y_train, return_counts=True)\nprint(\"Train labels: \", dict(zip(unique, counts)))\n# The zip() function combine two (or more) sequences element by element.\n# dict() creates a dictionary - data structure that stores key-value pairs.\n\n# count the number of unique test labels\nunique, counts = np.unique(y_test, return_counts=True)\nprint(\"\\nTest labels: \", dict(zip(unique, counts)))","metadata":{"_uuid":"d1239579-6da4-4238-b575-ca1be4344f04","_cell_guid":"3c4c720a-a510-4379-a196-094f16312924","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:41.804714Z","iopub.execute_input":"2025-12-02T23:06:41.805153Z","iopub.status.idle":"2025-12-02T23:06:42.149813Z","shell.execute_reply.started":"2025-12-02T23:06:41.805074Z","shell.execute_reply":"2025-12-02T23:06:42.148810Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Data visualization <a class=\"anchor\" id=\"4\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- The following code will help to sample the 25 random MNIST digits and visualize them.","metadata":{"_uuid":"27f22611-d7f2-4b76-a791-43719a9ff270","_cell_guid":"191b8953-17d5-4fcc-af5f-04024e1e55d6","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# 1. Randomly picks 25 MNIST images\n# 2. Creates a 5x5 grid\n# 3. Plots each digits in grayscale\n# 4. Displays and saves the figure\n\n\n# sample 25 mnist digits from train dataset\n# randomly selects 25 integers between 0 and the number of training samples\nindexes = np.random.randint(0, x_train.shape[0], size=25)\n# Each random interger is an index pointing to an image in the training set\nimages = x_train[indexes]\n# gets the 25 images from the training data\nlabels = y_train[indexes]\n\n\n# plot the 25 mnist digits\n# plt.figure is used to create a new figure object\n# This creates a new blank figure with size 5x5 inches.\nplt.figure(figsize=(5,5))\nfor i in range(len(indexes)):\n    # plt.subplot is used to divide a figure into smaller sections\n    # plt.subplot(5, 5, i + 1) Divides the figure into a 5x5 grid.\n    # (i + 1) tells it which slot to use for the current image.\n    plt.subplot(5, 5, i + 1)\n    image = images[i]\n    # plt.imshow() is the function used to display an image.\n    plt.imshow(image, cmap='gray') # displays the image in grayscale\n    plt.axis('off') # hides the x and y axis ticks to make the display cleaner\n    \nplt.show()\nplt.savefig(\"mnist-samples.png\")\nplt.close('all')","metadata":{"_uuid":"3b27c29d-4c20-4953-a37a-f1534b9105f4","_cell_guid":"3d080d23-fd98-44e9-bb8c-b834f9646bb0","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:42.151479Z","iopub.execute_input":"2025-12-02T23:06:42.151893Z","iopub.status.idle":"2025-12-02T23:06:43.316634Z","shell.execute_reply.started":"2025-12-02T23:06:42.151822Z","shell.execute_reply":"2025-12-02T23:06:43.315529Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Designing model architecture using Keras <a class=\"anchor\" id=\"5\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- The MLP model, discussed above can be used for MNIST digits classification. \n\n- When the units or perceptrons are exposed, the MLP model is a fully connected network.\n\n- The following code shows how to design the MLP model architecture using Keras.\n\n- The first step in designing the model architecture is to import the Keras layers. This can be done as follows:","metadata":{"_uuid":"ef562feb-3ed0-452a-bc2a-a921ad169d7e","_cell_guid":"c0f0ff2d-b26f-4f92-aba2-04bfb931230b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 5.1 Import Keras layers <a class=\"anchor\" id=\"5.1\"></a>","metadata":{"_uuid":"93139263-bd1e-4951-80b1-9ba68eb8a25f","_cell_guid":"ec3085b9-a92a-40d9-b5d2-ba65f177d78f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Sequential is a class in Keras that lets you build a neural network layer-by-layer, from iput -> hidden -> output, in a straight line sequence.\nfrom keras.models import Sequential\n\n# Research and record the function of a Dropout Layer\n# A Dense layer connects every neuron from the previous layer to every neuron in this layer. It's also called a fully connected layer.\n# Activation function lets the network learn complex, non-linear relationships.\n# Droppout randomly turns off some neurons during training.\n# Dense layer connects every input to every neuron and computes while dropout will drop neuron during training\nfrom keras.layers import Dense, Activation, Dropout\nfrom keras.utils import to_categorical, plot_model","metadata":{"_uuid":"a674c6ca-d461-4d69-a708-aa73d1bb6d4a","_cell_guid":"bf7fb8f5-efea-4826-94f9-93b03fa8ed83","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.318655Z","iopub.execute_input":"2025-12-02T23:06:43.319081Z","iopub.status.idle":"2025-12-02T23:06:43.325658Z","shell.execute_reply.started":"2025-12-02T23:06:43.319011Z","shell.execute_reply":"2025-12-02T23:06:43.324489Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.2 Compute the number of labels <a class=\"anchor\" id=\"5.2\"></a>\n\n- Now, the data must be in the correct shape and format. \n\n- After loading the MNIST dataset, the number of labels is computed as:","metadata":{"_uuid":"cc26eb39-ffb7-4a91-85e5-908937dc106b","_cell_guid":"9679c3cf-1c12-4e56-a513-c99a7f7d6d4c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# compute the number of labels\nnum_labels = len(np.unique(y_train))","metadata":{"_uuid":"3105f701-e42b-42da-b212-2ee7e4fda71d","_cell_guid":"2f18bf0c-90f7-44bb-92c3-ab558fc647bc","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.327420Z","iopub.execute_input":"2025-12-02T23:06:43.327745Z","iopub.status.idle":"2025-12-02T23:06:43.344324Z","shell.execute_reply.started":"2025-12-02T23:06:43.327670Z","shell.execute_reply":"2025-12-02T23:06:43.343004Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.3 One-Hot Encoding <a class=\"anchor\" id=\"5.3\"></a>\n\n- At this point, the labels are in digits format, 0 to 9. \n\n- This sparse scalar representation of labels is not suitable for the neural network prediction layer that outputs probabilities per class. \n\n- A more suitable format is called a one-hot vector, a 10-dim vector with all elements 0, except for the index of the digit class. \n\n- For example, if the label is 2, the equivalent one-hot vector is [0,0,1,0,0,0,0,0,0,0]. The first label has index 0.\n\n- The following lines convert each label into a one-hot vector:","metadata":{"_uuid":"c5c1fefb-a7ea-4fd2-83d7-7020ba165e9b","_cell_guid":"f2173a02-a216-40bb-9432-5bf030fa716b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# convert to one-hot vector\n# What does the function below do?\n# The function to_categorical() converts integer class labels into one-hot encoded vectors.\ny_train = to_categorical(y_train)\ny_test = to_categorical(y_test)","metadata":{"_uuid":"9f2c0970-ef5c-41ae-9fbb-0aaa4812e504","_cell_guid":"4649a3ad-21db-4682-8ada-e0bc97692591","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.346227Z","iopub.execute_input":"2025-12-02T23:06:43.346664Z","iopub.status.idle":"2025-12-02T23:06:43.356454Z","shell.execute_reply.started":"2025-12-02T23:06:43.346584Z","shell.execute_reply":"2025-12-02T23:06:43.355278Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.4 Data Preprocessing <a class=\"anchor\" id=\"5.4\"></a>\n\n\n- In deep learning, data is stored in tensors. The term tensor applies to a scalar (0D tensor), vector (1D tensor), matrix (2D tensor), and a multi-dimensional tensor.\n\n- The rest code computes the image dimensions, input_size of the first Dense layer and scales each pixel value from 0 to 255 to range from 0.0 to 1.0. Although raw pixel values can be used directly, it is better to normalize the input data as to avoid large gradient values that could make training difficult. \n\n- The output of the network is also normalized. After training, there is an option to put everything back to the integer pixel values by multiplying the output tensor by 255.  \n\n- The proposed model is based on MLP layers. Therefore, the input is expected to be a 1D tensor. So, x_train and x_test are reshaped to [60000, 28 * 28] and [10000, 28 * 28], respectively.","metadata":{"_uuid":"20a4348f-d772-4b23-ad59-40c21bae6397","_cell_guid":"9b752100-53f0-41bb-bd35-3ea72ba9173c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# image dimensions (assumed square)\nimage_size = x_train.shape[1]\ninput_size = image_size * image_size\ninput_size","metadata":{"_uuid":"4da6d6e0-cfbc-4506-b827-b00e0f1d96fa","_cell_guid":"7fd60eca-9932-4a48-b7d3-79cb5408aabf","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.358383Z","iopub.execute_input":"2025-12-02T23:06:43.358840Z","iopub.status.idle":"2025-12-02T23:06:43.374608Z","shell.execute_reply.started":"2025-12-02T23:06:43.358760Z","shell.execute_reply":"2025-12-02T23:06:43.373560Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# resize and normalize\n# reshape() changes the shape (dimensions) of a NumPy array without changing its data.\nx_train = np.reshape(x_train, [-1, input_size])\n# astype() changes the data type of the array elements.\nx_train = x_train.astype('float32') / 255\nx_test = np.reshape(x_test, [-1, input_size])\nx_test = x_test.astype('float32') / 255","metadata":{"_uuid":"0c603b21-7db0-4b27-a782-cf7d278f5233","_cell_guid":"81d19458-819a-4e80-bb5c-46870088271d","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.376129Z","iopub.execute_input":"2025-12-02T23:06:43.376416Z","iopub.status.idle":"2025-12-02T23:06:43.622014Z","shell.execute_reply.started":"2025-12-02T23:06:43.376357Z","shell.execute_reply":"2025-12-02T23:06:43.620887Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.5 Setting network parameters <a class=\"anchor\" id=\"5.5\"></a>\n\n- Now, we will set the network parameters as follows:","metadata":{"_uuid":"382bbd9e-254a-40d4-a669-f7b3bef3ee91","_cell_guid":"feba3d05-b150-4b05-b814-15a5dfc089f0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# network parameters\nbatch_size = 128\nhidden_units = 256\ndropout = 0.45","metadata":{"_uuid":"f14dd578-0e7a-411b-9452-f7f3e8e69cc9","_cell_guid":"58ddbf34-ecd7-4095-91a4-6c0e3e2acc3e","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.623529Z","iopub.execute_input":"2025-12-02T23:06:43.623920Z","iopub.status.idle":"2025-12-02T23:06:43.628863Z","shell.execute_reply.started":"2025-12-02T23:06:43.623842Z","shell.execute_reply":"2025-12-02T23:06:43.627861Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The **batch_size** argument indicates the number of data that we will use for each update of the model parameters.\n\n- **Hidden_units** shows the number of hidden units.\n\n- **Dropout** is the dropout rate (more on this in section 7 - **Overfitting and Regularization**).","metadata":{"_uuid":"cfca8b09-97e2-4d18-80ad-06362abd7d46","_cell_guid":"97bed5c4-4d97-4630-9d33-eed4fe20d695","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 5.6 Designing the model architecture <a class=\"anchor\" id=\"5.6\"></a>\n\n\n- The next step is to design the model architecture. The proposed model is made of three MLP layers. \n\n- In Keras, an MLP layer is referred to as Dense, which stands for the densely connected layer. \n\n- Both the first and second MLP layers are identical in nature with 256 units each, followed by relu activation and dropout. \n\n- 256 units are chosen since 128, 512 and 1,024 units have lower performance metrics. At 128 units, the network converges quickly, but has a lower test accuracy. The added number units for 512 or 1,024 does not increase the test accuracy significantly.","metadata":{"_uuid":"8824bb77-144f-4dfc-ba06-4dc9dbf98f47","_cell_guid":"6cf71473-4846-4aac-805a-4a5ab21c9af6","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"- The main data structure in Keras is the Sequential class, which allows the creation of a basic neural network.\n\n- The Sequential class of the Keras library is a wrapper for the sequential neural network model that Keras offers and can be created in the following way:\n\n`from keras.models import Sequential`\n\n`model = Sequential()`\n\n- The model in Keras is considered as a sequence of layers and each of them gradually “distills” the input data to obtain the desired output.\n\n- In Keras, we can add the required types of layers through the **add()** method.","metadata":{"_uuid":"df4fb220-dbb8-4ad1-bd09-fa0c207b8041","_cell_guid":"0b6013b0-b95c-4ed7-9ad0-dc85db31b5cc","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# model is a 3-layer MLP with ReLU and dropout after each layer\nmodel = Sequential()\nmodel.add(Dense(hidden_units, input_dim=input_size))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(dropout))\nmodel.add(Dense(hidden_units))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(dropout))\nmodel.add(Dense(num_labels))\nmodel.add(Activation('softmax'))\n# What is SoftMax? \n# SoftMax takes a list of numbers and turns them into probabilities","metadata":{"_uuid":"3dc6c475-8b66-438c-bc1d-5bcf275eb03c","_cell_guid":"864ab045-2cea-4ede-8299-6e00ba4a4e4f","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:09:17.051450Z","iopub.execute_input":"2025-12-02T23:09:17.051889Z","iopub.status.idle":"2025-12-02T23:09:17.133814Z","shell.execute_reply.started":"2025-12-02T23:09:17.051796Z","shell.execute_reply":"2025-12-02T23:09:17.132608Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Since a Dense layer is a linear operation, a sequence of Dense layers can only approximate a linear function. \n\n- The problem is that the MNIST digit classification is inherently a non-linear process. Inserting a relu activation between Dense layers will enable MLPs to model non-linear mappings. \n\n- relu or Rectified Linear Unit (ReLU) is a simple non-linear function. It allows positive inputs to pass through unchanged while clamping everything else to zero.","metadata":{"_uuid":"576f1dc6-286e-4c5d-ad72-3faf150d02f5","_cell_guid":"e70a6f56-46cc-43a2-8c89-5d49fbd35342","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 5.7 View model summary <a class=\"anchor\" id=\"5.7\"></a>\n\n- Keras library provides us **summary()** method to check the model description.","metadata":{"_uuid":"efb1cfb3-e05d-4442-9d58-3b8818794c37","_cell_guid":"b26d5a20-707d-4f7c-9a15-b9c7ac9aaa37","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"model.summary()","metadata":{"_uuid":"186d5038-28e6-4779-aff6-96b142596c3b","_cell_guid":"54e6fde4-ab0a-4999-93a3-c9a479f41d4d","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:09:21.593567Z","iopub.execute_input":"2025-12-02T23:09:21.593894Z","iopub.status.idle":"2025-12-02T23:09:21.601210Z","shell.execute_reply.started":"2025-12-02T23:09:21.593837Z","shell.execute_reply":"2025-12-02T23:09:21.600093Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The above listing shows the model summary of the proposed network. It requires a total of 269,322 parameters.\n\n- This is substantial considering that we have a simple task of classifying MNIST digits. So, MLPs are not parameter efficient. \n\n- The total number of parameters required can be computed as follows:\n\n  - From input to Dense layer: 784 × 256 + 256 = 200,960. \n  \n  - From first Dense to second Dense: 256 × 256 + 256 = 65,792. \n  \n  - From second Dense to the output layer: 10 × 256 + 10 = 2,570. \n  \n  - The total is 200,690 + 65,972 + 2,570 = 269,322.","metadata":{"_uuid":"19c3a879-6d2e-4c7c-87f7-0bef838cab0e","_cell_guid":"2af37506-303c-4e83-8f16-1dde583c70d0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"- Another way of verifying the network is by calling the **plot_model()** method as follows:","metadata":{"_uuid":"b204553a-2481-44a5-b10f-0096f27c6383","_cell_guid":"9aa4344c-2ab6-4011-9042-9d17fb0ca830","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"plot_model(model, to_file='mlp-mnist.png', show_shapes=True)","metadata":{"_uuid":"d7b5b23f-8ab2-4e06-8d1f-1d2bd497a3a6","_cell_guid":"d887a830-485c-432f-9776-4fa5cf221190","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.722779Z","iopub.execute_input":"2025-12-02T23:06:43.723168Z","iopub.status.idle":"2025-12-02T23:06:43.879017Z","shell.execute_reply.started":"2025-12-02T23:06:43.723090Z","shell.execute_reply":"2025-12-02T23:06:43.877789Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 6. Implement MLP model using Keras <a class=\"anchor\" id=\"6\"></a>\n\n\n[Back to Table of Contents](#0.1)\n\n\n- The implementation of MLP model in Keras comprises of three steps:-\n\n  - Compiling the model with the compile() method.\n  \n  - Training the model with fit() method.\n  \n  - Evaluating the model performance with evaluate() method.\n  \n  \n- For detailed discussion on implementation, please refer to my previous kernel [Comprehensive Guide to ANN with Keras](https://www.kaggle.com/prashant111/comprehensive-guide-to-ann-with-keras)","metadata":{"_uuid":"3cc9dac9-2502-4183-8440-c52fff9d4dac","_cell_guid":"66215287-2680-4781-b5e4-327d8b415d26","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 6.1 Compile the model with compile() method <a class=\"anchor\" id=\"6.1\"></a>\n\n\n- Compilation of model can be done as follows:","metadata":{"_uuid":"681a3b00-7a4e-470f-a03e-6380160f5bd2","_cell_guid":"67c20c3c-a717-4281-9b71-1ad4c24fd795","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Before compile(), the model doesn't know how to learn.\n# After compile(), the model knows: \"This is how I calculate error\", \"This is how I adjust my weights\", and \"This is what to report while training\"\nmodel.compile(loss='categorical_crossentropy',# How wrong the model is. It is the number the model tries to minimize while training\n              optimizer='adam',\n              metrics=['accuracy'])# Values you want to monitor during training \n\n# What does optimizer Adam do?\n# The Adam optimizer is one of the most popular optimization algorithms used in training neural networks.","metadata":{"_uuid":"940f4596-60bb-465b-b430-6f53c9d5f7b6","_cell_guid":"b631c32e-8032-4278-ac73-ad1e81418484","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.881169Z","iopub.execute_input":"2025-12-02T23:06:43.881484Z","iopub.status.idle":"2025-12-02T23:06:43.945513Z","shell.execute_reply.started":"2025-12-02T23:06:43.881423Z","shell.execute_reply":"2025-12-02T23:06:43.944514Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Loss function (categorical_crossentropy)\n\n- How far the predicted tensor is from the one-hot ground truth vector is called **loss**.\n\n- In this example, we use **categorical_crossentropy** as the loss function. It is the negative of the sum of the product of the target and the logarithm of the prediction. \n\n- There are other loss functions in Keras, such as mean_absolute_error and binary_crossentropy. The choice of the loss function is not arbitrary but should be a criterion that the model is learning. \n\n- For classification by category, categorical_crossentropy or mean_squared_error is a good choice after the softmax activation layer. The binary_crossentropy loss function is normally used after the sigmoid activation layer while mean_squared_error is an option for tanh output.","metadata":{"_uuid":"09e760e9-d5dc-4f2f-8dcc-fcb59e1e72ad","_cell_guid":"2716d156-d46b-4427-a3e2-96ca7a96da3a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### Optimization (optimizer adam)\n\n- With optimization, the objective is to minimize the loss function. The idea is that if the loss is reduced to an acceptable level, the model has indirectly learned the function mapping input to output.\n\n- In Keras, there are several choices for optimizers. The most commonly used optimizers are; **Stochastic Gradient Descent (SGD)**, **Adaptive Moments (Adam)** and **Root Mean Squared Propagation (RMSprop)**. \n\n- Each optimizer features tunable parameters like learning rate, momentum, and decay. \n\n- Adam and RMSprop are variations of SGD with adaptive learning rates. In the proposed classifier network, Adam is used since it has the highest test accuracy.","metadata":{"_uuid":"84f7090d-6b26-408f-874d-4f4f0e57bc84","_cell_guid":"5b50dd99-4a0b-4783-aa9b-651795c0b774","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### Metrics (accuracy)\n\n- Performance metrics are used to determine if a model has learned the underlying data distribution. The default metric in Keras is loss. \n\n- During training, validation, and testing, other metrics such as **accuracy** can also be included. \n\n- **Accuracy** is the percent, or fraction, of correct predictions based on ground truth.","metadata":{"_uuid":"26c9998c-ba56-4de4-9e28-ec52c5386122","_cell_guid":"f3ecbbc8-290e-4906-894b-de6094fbc8b9","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 6.2 Train the model with fit() method <a class=\"anchor\" id=\"6.2\"></a>","metadata":{"_uuid":"bbf47aa4-c670-450c-82e8-0c6fcc76955f","_cell_guid":"f32fafe5-042a-4fae-aedc-df01edc26a4b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"model.fit(x_train, y_train, epochs=21, batch_size=batch_size)","metadata":{"_uuid":"be3faf3b-f72e-4a23-b2e7-fc971601a340","_cell_guid":"d263b3c1-10a1-44f1-b7a0-21e1ced5ce6a","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:06:43.946739Z","iopub.execute_input":"2025-12-02T23:06:43.947038Z","iopub.status.idle":"2025-12-02T23:07:59.730000Z","shell.execute_reply.started":"2025-12-02T23:06:43.946981Z","shell.execute_reply":"2025-12-02T23:07:59.729086Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6.3 Evaluating model performance with evaluate() method <a class=\"anchor\" id=\"6.3\"></a>","metadata":{"_uuid":"5b6bc249-bebc-4590-ac83-e7dac6274256","_cell_guid":"3d466592-3f8a-48d2-ab82-94e09c3a0e51","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)\nprint(\"\\nTest accuracy: %.1f%%\" % (100.0 * acc))\n\n# Record results in Notebook","metadata":{"_uuid":"5fdefd2f-ae42-4a6e-b4b2-8fa43b21405b","_cell_guid":"bc9fb1d1-d8c4-4210-bb78-a766f9535b4b","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:07:59.731123Z","iopub.execute_input":"2025-12-02T23:07:59.731360Z","iopub.status.idle":"2025-12-02T23:08:00.018312Z","shell.execute_reply.started":"2025-12-02T23:07:59.731318Z","shell.execute_reply":"2025-12-02T23:08:00.017428Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 7. Overfitting and Regularization <a class=\"anchor\" id=\"7\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- A neural network has the property to memorize the characteristics of training data. This is called **overfitting**.\n\n- In such a case, the network fails to generalize when subject to the test data.\n\n- To avoid this tendency, the model uses a regularizing layer or function. A commonly used regularizing layer is referred to as a **Dropout layer**.\n\n- Given a dropout rate (dropout=0.45), the **Dropout layer** randomly removes the fraction of units from participating in the next layer. For example, if the first layer has 256 units, after dropout=0.45 is applied, only (1 - 0.45) * 256 units = 140 units from layer 1 participate in layer 2.\n\n- The Dropout layer makes neural networks robust to unforeseen input data because the network is trained to predict correctly, even if some units are missing. \n\n- The dropout is not used in the output layer and it is only active during training. Moreover, dropout is not present during prediction.\n\n- There are regularizers that can be used other than dropouts like l1 or l2. In Keras, the bias, weight and activation output can be regularized per layer. - l1 and l2 favor smaller parameter values by adding a penalty function. Both l1 and l2 enforce the penalty using a fraction of the sum of absolute (l1) or square (l2) of parameter values. \n\n- So, the penalty function forces the optimizer to find parameter values that are small. Neural networks with small parameter values are more insensitive to the presence of noise from within the input data.\n\n- So, the l2 weight regularizer with fraction=0.001 can be implemented as:","metadata":{"_uuid":"c91ed662-a0ae-41bd-8833-2854dfe379e5","_cell_guid":"6204202d-b135-4c70-bc64-b3db13372eaa","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from keras.regularizers import l2\nmodel.add(Dense(hidden_units,\n                # kernel_regularizer is a parameter you can add to a Keras layer to apply regularization to the layer's weights so the model does not overfit,\n                kernel_regularizer=l2(0.001),# 0.001 is the regularization coefficient\n                input_dim=input_size))","metadata":{"_uuid":"539f595a-5dbd-4759-bbb7-fd2415909db0","_cell_guid":"88c9c912-2db8-4f0c-9fe3-aecd6fbe9270","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:08:00.019674Z","iopub.execute_input":"2025-12-02T23:08:00.020025Z","iopub.status.idle":"2025-12-02T23:08:00.046109Z","shell.execute_reply.started":"2025-12-02T23:08:00.019964Z","shell.execute_reply":"2025-12-02T23:08:00.044919Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_predictions = model.predict(x_test)\npredicted_labels = np.argmax(test_predictions, axis = 1)\n\nsubmission = pd. DataFrame({\n    \"ImageId\": np.arange(1, len(predicted_labels) + 1),\n    \"Label\": predicted_labels\n})\n\nsubmission.to_csv(\"submission.csv\", index = False)\nprint(\"Submission file created successfully!\")","metadata":{"_uuid":"05862fb5-fa8e-4e7b-8d71-bfa8e8beb933","_cell_guid":"75ef2640-9ea6-4429-baf5-99529e9f0308","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2025-12-02T23:08:00.047480Z","iopub.execute_input":"2025-12-02T23:08:00.047895Z","iopub.status.idle":"2025-12-02T23:08:00.512549Z","shell.execute_reply.started":"2025-12-02T23:08:00.047820Z","shell.execute_reply":"2025-12-02T23:08:00.511680Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\n\n# pred_labels must be a list/array of your predictions (0–9)\nsubmission = pd.DataFrame({\n    \"ImageId\": range(1, len(predicted_labels) + 1),\n    \"Label\": predicted_labels\n})\n\nsubmission.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-02T23:08:00.514031Z","iopub.execute_input":"2025-12-02T23:08:00.514326Z","iopub.status.idle":"2025-12-02T23:08:00.558652Z","shell.execute_reply.started":"2025-12-02T23:08:00.514267Z","shell.execute_reply":"2025-12-02T23:08:00.557561Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- No additional layer is added if l1 or l2 regularization is used. The regularization is imposed in the Dense layer internally. For the proposed model, dropout still has a better performance than l2.","metadata":{"_uuid":"5102c632-16e7-4899-9e93-7f12bb376b23","_cell_guid":"a358a873-48d7-4d0d-81ad-050de7017833","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 8. Results and Conclusion <a class=\"anchor\" id=\"8\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- In this kernel, I build a dense neural network model to classify the MNIST digits and predict accuracy.\n\n- We get the test accuracy of 98.3%.\n\n- The accuracy tells us that our ANN model classifies the digits 98.3% of time correctly.","metadata":{"_uuid":"064fc043-4ae9-4d76-aca5-0acc83a63e2f","_cell_guid":"93120c66-d466-4f2e-b615-58f578909e1c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 9. References <a class=\"anchor\" id=\"9\"></a>\n\n\n[Back to Table of Contents](#0.1)\n\n\nThis kernel is based on the following Kaggle kernels and books:\n\n- [My previous kernel on Artificial Neural Network](https://www.kaggle.com/prashant111/comprehensive-guide-to-ann-with-keras)\n\n- [Deep Neural Network Keras way by Poonam Ligade](https://www.kaggle.com/poonaml/deep-neural-network-keras-way)\n\n- [MNIST with Keras for beginners by Aditya Soni](https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457)\n\n-\tDeep Learning with Python by Francois Chollet\n\n-\tAdvanced Deep Learning with Keras by Rowel Atienza","metadata":{"_uuid":"7203a022-dc4b-4bce-80eb-354d1b2f0538","_cell_guid":"f4cd84bd-bb9c-475d-b7e6-3b49d88d9841","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"Now we will come to the end of this kernel.\n\nI hope you find this kernel useful and enjoyable.\n\nThank you","metadata":{"_uuid":"3edf183a-b61a-4085-95fd-14540f87916d","_cell_guid":"d8fca82b-40b3-455b-a965-9cbb27231bc7","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"[Go to Top](#0)","metadata":{"_uuid":"f1296727-9791-4694-bac8-67ffc42ba5a6","_cell_guid":"20359636-d988-4974-a635-a1d8b0dbc49a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}}]}