{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Multi-class classification using Vanilla (simple) DL Models\n# Digit recognition (Pattern Recognition) using the MNIST Dataset- with Keras\n\n\n#","metadata":{"_uuid":"6386a860-32a4-46e9-aca3-9a22af29931e","_cell_guid":"36cac447-b351-4cca-9f1b-83c7cfb7b501","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# Objective of the notebook \n\n\n\n- In this notebook, we will design a **Multilayer Perceptron (MLP)** model of the MNIST dataset. \n\n- **Multilayer Perceptrons (MLPs)** usually mean fully connected networks, that is, each neuron in one layer is connected to all neurons in the next layer. The \"fully-connectedness\" of these networks makes them prone to overfitting the data. \n\n- These MLP models are also referred to as **feedforward neural networks**. MLPs are common in simple logistic and linear regression problems.\n\n- So, the objective is to create a neural network for identifying numbers based on handwritten digits. For example, when the input to the network is an image of a handwritten number 8, the corresponding prediction must also be the digit 8. \n\n- To both train and validate a neural network, there must be a sufficiently large dataset of handwritten digits.\n\n- The Modified National Institute of Standards and Technology dataset or MNIST dataset for short, is often considered as the Hello World! of deep learning and is a suitable dataset for handwritten digit classification.\n\n- MNIST is used to explain and validate deep learning theories because the 70,000 samples it contains are small, yet sufficiently rich in information (MNIST dataset is described later).","metadata":{"_uuid":"889c0d36-6b42-48d8-9456-6e286946af7b","_cell_guid":"819d7fb3-81a8-4b8f-bdf5-88d04336814f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"So, we will start by importing the necessary libraries:","metadata":{"_uuid":"e8945f2a-040c-4abf-9321-2fd9cdaf0d62","_cell_guid":"1c81ef64-fdef-428a-9f4f-a100dfe25568","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 1. Import necessary libraries ","metadata":{"_uuid":"a744b1dd-74c0-4c2b-92e7-f9156048650e","_cell_guid":"05727326-c515-4a8d-9d28-fca62cafe62c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\n# conda install TensorFlow\n# The Official Keras Deep Neural Network Example for CNN \n###############################\n# Setting Seed Random Number for Reproducibility\nimport numpy as np # linear algebra\nnp.random.seed(123) # for reproducibility\n\n# Setting Seed Random Number for the TensorFlow Backend\nimport tensorflow as tf\ntf.random.set_seed(123) # for reproducibility\n\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # plotting library\n#%matplotlib inline\n\nfrom tensorflow import keras\nfrom keras import backend as K","metadata":{"_uuid":"b7e0ae88-90df-4ff6-87b6-e573d3cd28f4","_cell_guid":"5fb5b942-8ba3-4348-a65f-85ca3521c1a4","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:36:22.09498Z","iopub.execute_input":"2024-12-11T09:36:22.095669Z","iopub.status.idle":"2024-12-11T09:36:35.245459Z","shell.execute_reply.started":"2024-12-11T09:36:22.095617Z","shell.execute_reply":"2024-12-11T09:36:35.244333Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Loading MNIST dataset \n\n\n\nThe **MNIST** dataset (**Modified National Institute of Standards and Technology database**) is a large collection of **handwritten digits** that is widely used for training and testing in the field of machine learning and image processing. Here are some key points about it:\n\n- MNIST is a collection of handwritten digits ranging from the number 0 to 9. \n\n- It consists of 60,000 training images and 10,000 test images of handwritten digits, each image being 28x28 pixels in grayscale.\n\n- The dataset is commonly used to **benchmark machine learning algorithms**, particularly in the areas of image recognition and classification.\n\n- To use the MNIST dataset in Keras, an API is provided to download and extract images and labels automatically.\n\n- Variants: There are extended versions like EMNIST, which includes handwritten letters, and Fashion MNIST, which contains images of fashion products.","metadata":{"_uuid":"d4153a75-1087-4e50-afec-cfe4dba7f911","_cell_guid":"27e01b21-73e5-4f0a-a51f-a4453927d63d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"The following Keras code shows how to load MNIST dataset, plot 25 random samples, and count the number of labels for train and test datasets.","metadata":{"_uuid":"1fa4c635-08dd-4b1d-b50e-0b576c06be08","_cell_guid":"4b2e43da-363e-4e74-b547-0b50112e82db","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# import dataset\nfrom keras.datasets import mnist\n\n# load dataset\n(x_train, y_train),(x_test, y_test) = mnist.load_data() # ouput numpy arrays","metadata":{"_uuid":"e9373630-dc3c-440a-ab8a-1cf5d61caf23","_cell_guid":"5698ae76-4596-4d60-9fd4-cd441d90b286","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:36:48.121791Z","iopub.execute_input":"2024-12-11T09:36:48.122182Z","iopub.status.idle":"2024-12-11T09:36:50.820117Z","shell.execute_reply.started":"2024-12-11T09:36:48.122148Z","shell.execute_reply":"2024-12-11T09:36:50.819253Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Data exploration and visualization \n\n\n\n\n- The following code will help to sample the 25 random MNIST digits and visualize them.","metadata":{"_uuid":"d2db5d60-8923-4576-922b-65c50d09360c","_cell_guid":"ba82fd4f-810c-4dca-9b5b-1e5d721fd622","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"print(type(x_train))\nprint(\"x_train shape:\", x_train.shape )\nprint(\"x_test shape:\",x_test.shape )\nprint(\"y_train shape:\",y_train.shape )\nprint(\"y_test shape:\",y_test.shape )","metadata":{"_uuid":"6023ae8a-927a-4a6d-ba05-5ca29cac042d","_cell_guid":"ecfb6786-b42e-403a-95ca-b8c842c4417a","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:00.907885Z","iopub.execute_input":"2024-12-11T09:37:00.908758Z","iopub.status.idle":"2024-12-11T09:37:00.91399Z","shell.execute_reply.started":"2024-12-11T09:37:00.908718Z","shell.execute_reply":"2024-12-11T09:37:00.912972Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"labels: \",np.unique(y_train)) # multi-classification\n\n# count the number of unique train labels\ncategory_name, counts = np.unique(y_train, return_counts=True)\nprint(\"Train labels: \", dict(zip(category_name, counts)))\n\n# count the number of unique test labels\ncategory_name, counts = np.unique(y_test, return_counts=True)\nprint(\"\\nTest labels: \", dict(zip(category_name, counts)))","metadata":{"_uuid":"cc4ec6ec-2e98-4777-9a68-1d5381dc10a9","_cell_guid":"ebccb08a-a103-45ea-bc78-7287b1494d4b","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:08.037575Z","iopub.execute_input":"2024-12-11T09:37:08.038416Z","iopub.status.idle":"2024-12-11T09:37:08.0492Z","shell.execute_reply.started":"2024-12-11T09:37:08.038373Z","shell.execute_reply":"2024-12-11T09:37:08.048129Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# sample 9 mnist digits from train dataset\nindexes = np.random.randint(0, x_train.shape[0], size=9)\nimages = x_train[indexes]\nlabels = y_train[indexes]\n\n# plot the 9 mnist digits\nplt.figure(figsize=(3,3))\nfor i in range(len(indexes)):\n plt.subplot(3, 3, i + 1)\n image = images[i]\n plt.imshow(image, cmap='gray')\n plt.axis('off')\n \nplt.show()\nplt.savefig(\"mnist-samples.png\")\nplt.close('all')","metadata":{"_uuid":"562aa2aa-50f8-4e09-a5b5-78127ae8eab8","_cell_guid":"dfe91f4a-1910-4fe5-bbbe-8ccab3905588","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:13.693216Z","iopub.execute_input":"2024-12-11T09:37:13.693614Z","iopub.status.idle":"2024-12-11T09:37:14.004032Z","shell.execute_reply.started":"2024-12-11T09:37:13.693554Z","shell.execute_reply":"2024-12-11T09:37:14.002729Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4 Data Preprocessing \n\n- **One-Hot Encode the Labels**: Convert the class labels (0-9) into one-hot encoded vectors. This is necessary for the output layer of the ANN.\n- **Reshape the Data**: Since ANNs expect 1D input, you need to flatten the 28x28 images into 1D vectors of size 784.\n- **Normalize the Data**: Scale the pixel values to the range [0, 1] by dividing by 255. This helps the network train faster and more effectively.","metadata":{"_uuid":"1eebd2cb-74ef-4139-8527-4fbc3d86c56a","_cell_guid":"7345a54f-ac15-4a19-aab6-53701d580b44","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### 4.1 One-Hot Encode the Labels \n\nThe number of labels is computed as:","metadata":{"_uuid":"2761af4a-bf70-47e1-b169-4ea176f85eb9","_cell_guid":"fc578e29-c561-4d36-b924-b650bf4057cf","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"np.unique(y_train)","metadata":{"_uuid":"8cb8f896-e7ed-41be-801a-e176a047b1c1","_cell_guid":"8b35b681-ed2b-4246-b389-81820fa5a64e","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:22.209936Z","iopub.execute_input":"2024-12-11T09:37:22.210323Z","iopub.status.idle":"2024-12-11T09:37:22.218264Z","shell.execute_reply.started":"2024-12-11T09:37:22.210286Z","shell.execute_reply":"2024-12-11T09:37:22.217312Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# compute the number of labels\nnum_labels = len(np.unique(y_train))\nprint(num_labels)","metadata":{"_uuid":"409b4d51-b5bd-428e-9b6d-066fda18906e","_cell_guid":"27392430-9acc-4f11-8dce-aae8a5497304","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:24.606486Z","iopub.execute_input":"2024-12-11T09:37:24.607423Z","iopub.status.idle":"2024-12-11T09:37:24.613997Z","shell.execute_reply.started":"2024-12-11T09:37:24.607383Z","shell.execute_reply":"2024-12-11T09:37:24.613067Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Keras requires your output variable to be one-hot encoded values in the case of multy-classification.\n\n\n\n**One-hot encoding** is necessary if you use **categorical_crossentropy loss** when training the model. However if you use **sparse_categorical_crossentropy** then you can skip this step.\n\n**The following lines convert each label into a one-hot vector**:","metadata":{"_uuid":"a46aaf73-59f8-40a5-8c9a-fb25d67df3b3","_cell_guid":"6b1fb47e-6493-484d-a2ed-6c02a3f8cfce","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from tensorflow.keras.utils import to_categorical, plot_model\n# convert to one-hot vector\nY_train = to_categorical(y_train)\nY_test = to_categorical(y_test)\n\nprint (\"A sample output before encoding: \",y_train[0])\nprint (\"A sample output after encoding: \",Y_train[0])","metadata":{"_uuid":"5e990a1a-80d5-4ca9-a4c7-f30cbb7257e4","_cell_guid":"fca0166f-87b0-4183-a859-33d8bf295071","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:35.170735Z","iopub.execute_input":"2024-12-11T09:37:35.171141Z","iopub.status.idle":"2024-12-11T09:37:35.251638Z","shell.execute_reply.started":"2024-12-11T09:37:35.171103Z","shell.execute_reply":"2024-12-11T09:37:35.250614Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.2 Reshape (Flatten) Images\nEach image is **28 pixels in height** and **28 pixels in width**, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\nThe training data set, (train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n\n","metadata":{"_uuid":"4fdd4df6-5524-4937-97ae-22e16dc11116","_cell_guid":"f5ac7042-a21c-4795-af95-acd40994b057","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# image dimensions\nimage_height = x_train.shape[1]\ninput_width = x_train.shape[2]\ninput_size = image_height * input_width\nprint(input_size)","metadata":{"_uuid":"1625f575-09aa-432e-855e-d15f833924bc","_cell_guid":"58f574c2-b6fe-4cb4-a108-30c2fd36d742","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:54.177039Z","iopub.execute_input":"2024-12-11T09:37:54.178041Z","iopub.status.idle":"2024-12-11T09:37:54.183134Z","shell.execute_reply.started":"2024-12-11T09:37:54.178Z","shell.execute_reply":"2024-12-11T09:37:54.182117Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Resize the images\nX_train = np.reshape(x_train, [-1, input_size])\nX_test = np.reshape(x_test, [-1, input_size])\n\nprint(\"x_train shape before the re-shaping:\", x_train.shape )\nprint(\"x_train shape after flattening:\", X_train.shape )","metadata":{"_uuid":"172e826a-67bd-480d-86c1-50d14af77623","_cell_guid":"c3629e94-16d8-4030-93bb-b67687d14522","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:37:57.64468Z","iopub.execute_input":"2024-12-11T09:37:57.645638Z","iopub.status.idle":"2024-12-11T09:37:57.650783Z","shell.execute_reply.started":"2024-12-11T09:37:57.645555Z","shell.execute_reply":"2024-12-11T09:37:57.649827Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.3 Normalize the Data","metadata":{"_uuid":"d6aea95d-c2b4-4465-b5e6-6c0ff5ccbe78","_cell_guid":"044f133d-2208-4634-b742-bab19d962725","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Preprocess input data\nprint(\"Range of X_train data before re-scaling: [ \", max(X_train[0].round(2)), ',', min(X_train[0]), ' ]' )\n# Scale image data to the [0, 1] range\nX_train_ = X_train.astype('float32') / 255\nX_test_ = X_test.astype('float32') / 255\n\nprint(\"Range of X_train data after re-scaling: [ \", max(X_train_[0].round(2)), ',', min(X_train[0]), ' ]' )","metadata":{"_uuid":"a6edaeaf-1f1e-44ef-97b1-92d2feaf3f17","_cell_guid":"e97eb9d8-5461-464d-94bf-aab4c35ebbe7","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:38:02.006616Z","iopub.execute_input":"2024-12-11T09:38:02.007032Z","iopub.status.idle":"2024-12-11T09:38:02.11439Z","shell.execute_reply.started":"2024-12-11T09:38:02.006998Z","shell.execute_reply":"2024-12-11T09:38:02.113368Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Designing ANN model using Keras Functional API \n\n\n- The MLP model, discussed above can be used for MNIST digits classification. \n\n- The MLP model is a fully connected network.","metadata":{"_uuid":"dacd5273-ab9b-4626-89bf-2f057d7116b3","_cell_guid":"589746da-78a1-459b-8126-d6690c170c88","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.layers import Dense, Input, Activation, Dropout\nfrom tensorflow.keras.utils import to_categorical, plot_model","metadata":{"_uuid":"6845e185-0f74-4239-aadb-3079c19bb90b","_cell_guid":"5990b52d-e41d-4b03-ab98-d4105c46d307","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:38:09.714533Z","iopub.execute_input":"2024-12-11T09:38:09.714967Z","iopub.status.idle":"2024-12-11T09:38:09.720087Z","shell.execute_reply.started":"2024-12-11T09:38:09.714933Z","shell.execute_reply":"2024-12-11T09:38:09.719054Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# network parameters\nbatch_size = 128","metadata":{"_uuid":"1272c5d7-c730-4093-8fbf-d33374b8e5c6","_cell_guid":"0f2b903e-7819-4003-a244-25a5e9c01873","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:38:12.54016Z","iopub.execute_input":"2024-12-11T09:38:12.540552Z","iopub.status.idle":"2024-12-11T09:38:12.545071Z","shell.execute_reply.started":"2024-12-11T09:38:12.540515Z","shell.execute_reply":"2024-12-11T09:38:12.543975Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The **batch_size** argument indicates the number of data that we will use for each update of the model parameters.","metadata":{"_uuid":"3ef2d7f3-ab75-4161-87ef-06f4fb5e760b","_cell_guid":"a66dc0d6-f568-4156-951d-6495dd43ac94","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 1. Designing ANN model using Keras DL Library ","metadata":{"_uuid":"4a351ec7-9100-4227-9e11-b79588763c93","_cell_guid":"c53d0b90-5c3d-4596-8ed5-ce215c4399c3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### 1.1 Designing the model architecture \n\n\n- The next step is to design the model architecture. The proposed model is made of three MLP layers. \n\n- In Keras, an MLP layer is referred to as Dense, which stands for the densely connected layer. \n\n- Both the first and second MLP layers are identical in nature with 256 neurons each and with relu activation. \n\n- Output layer has 10 neurons, which is equal to the number of the categories and with softmax activation.","metadata":{"_uuid":"5ccfc178-7a52-47e6-a5ed-98a74311a5c7","_cell_guid":"3ca6db59-5090-4867-9d29-77c1426e88ef","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"- The main data structure in Keras is the Sequential class, which allows the creation of a basic neural network.\n\n- The Sequential class of the Keras library is a wrapper for the sequential neural network model that Keras offers and can be created in the following way:\n\n`from keras.models import Sequential`\n\n`model = Sequential()`\n\n- The model in Keras is considered as a sequence of layers and each of them gradually “distills” the input data to obtain the desired output.\n\n- In Keras, we can add the required types of layers through the **add()** method.","metadata":{"_uuid":"472c8477-b05b-44d5-92c8-30cc5e226f32","_cell_guid":"f3f93fef-c6ce-4d9c-ac91-722bd551f6a5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.layers import Dense, Input, Activation, Dropout\n# Design the following sequential model using functional API\n# model is a 3-layer MLP (dense) layer\nmodel = Sequential()\nmodel.add(Input(shape=(input_size,))) # Input tensor\nmodel.add(Dense(256,activation='relu'))\nmodel.add(Dense(128,activation='relu'))\nmodel.add(Dense(64,activation='relu'))\nmodel.add(Dense(10, activation=\"softmax\"))\n\n# Define the input layer\ninputs = Input(shape=(input_size,))\n# First hidden layer\nx = Dense(256, activation='relu')(inputs)\n\n# Second hidden layer\nx = Dense(256, activation='relu')(x)\n\n# Third hidden layer\nx = Dense(256, activation='relu')(x)\n\n# Output layer\noutputs = Dense(10, activation='softmax')(x)\n...\n# Define Output layer\n# Implement the model\nmodel = Model(inputs=inputs, outputs=outputs)","metadata":{"_uuid":"fe268ea7-27ed-4141-b29d-2afe9854c78b","_cell_guid":"44a9e15e-7ba1-4572-91d6-aeb766f4eddf","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:42:37.389218Z","iopub.execute_input":"2024-12-11T09:42:37.389907Z","iopub.status.idle":"2024-12-11T09:42:37.464808Z","shell.execute_reply.started":"2024-12-11T09:42:37.389854Z","shell.execute_reply":"2024-12-11T09:42:37.463765Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Since a Dense layer is a linear operation, a sequence of Dense layers can only approximate a linear function. \n\n- The problem is that the MNIST digit classification is inherently a non-linear process. Inserting a relu activation between Dense layers will enable MLPs to model non-linear mappings. \n\n- relu or Rectified Linear Unit (ReLU) is a simple non-linear function. It allows positive inputs to pass through unchanged while clamping everything else to zero.","metadata":{"_uuid":"d03e4c28-ad8e-4c8e-a994-9f4a3d6ddd8d","_cell_guid":"4947a55d-83de-4e57-b6dc-8e6f283aeb77","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### 1.2 View model summary \n\n- Keras library provides us **summary()** method to check the model description.","metadata":{"_uuid":"37062bfc-2754-4f2b-aaab-16cb66a1c94a","_cell_guid":"ad136a38-0f9d-439f-83f6-b3ebcb52b481","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"model.summary()","metadata":{"_uuid":"78f8b6f7-f18b-483b-b267-ee3571f6847d","_cell_guid":"60aea5b6-1bf7-4af8-b7a5-71c861e7a779","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:42:41.396502Z","iopub.execute_input":"2024-12-11T09:42:41.396939Z","iopub.status.idle":"2024-12-11T09:42:41.415854Z","shell.execute_reply.started":"2024-12-11T09:42:41.396902Z","shell.execute_reply":"2024-12-11T09:42:41.41494Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The above listing shows the model summary of the proposed network. It requires a total of 269,322 parameters.\n\n- This is substantial considering that we have a simple task of classifying MNIST digits. So, MLPs are not parameter efficient.","metadata":{"_uuid":"fe511c55-fe74-407e-80a3-b1400711ceb3","_cell_guid":"ea272f54-e17a-42c3-82b3-7f35419f7850","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"- Another way of verifying the network is by calling the **plot_model()** method as follows:","metadata":{"_uuid":"c04e03d7-4ca5-435e-994e-7ada8c55b35c","_cell_guid":"b4331b1e-0fe3-4127-a367-55b5b55983e5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"plot_model(model, to_file='mlp-mnist_plot.png', \n show_shapes=True, \n show_layer_names=True, \n dpi=50, \n show_layer_activations=True,\n show_trainable=True)","metadata":{"_uuid":"a6bdc2ec-7282-4d85-8e4e-a6b3466b9e23","_cell_guid":"2c7cd9cb-6d17-4fcf-aa4a-c92664bc0451","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:42:47.646437Z","iopub.execute_input":"2024-12-11T09:42:47.647285Z","iopub.status.idle":"2024-12-11T09:42:47.907109Z","shell.execute_reply.started":"2024-12-11T09:42:47.647233Z","shell.execute_reply":"2024-12-11T09:42:47.906073Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.3 Compile the model with compile() method \n\n\n- Compilation of model can be done as follows:","metadata":{"_uuid":"7db4d1f3-798a-4bc1-9e74-94710402689a","_cell_guid":"7ab1ad7c-aa14-42f9-afdb-4d030f51c324","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"model.compile(loss='categorical_crossentropy', \n optimizer='adam',\n metrics=['accuracy'])","metadata":{"_uuid":"fa51a5c4-a506-4799-8a39-e8790effa555","_cell_guid":"479bd3d4-1b2e-41e9-a1da-d55e1d4b0672","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:43:19.52842Z","iopub.execute_input":"2024-12-11T09:43:19.529183Z","iopub.status.idle":"2024-12-11T09:43:19.537377Z","shell.execute_reply.started":"2024-12-11T09:43:19.529144Z","shell.execute_reply":"2024-12-11T09:43:19.536354Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Loss function (categorical_crossentropy)\n\n- How far the predicted tensor is from the one-hot ground truth vector is called **loss**.\n\n- In this example, we use **categorical_crossentropy** as the loss function. It is the negative of the sum of the product of the target and the logarithm of the prediction. \n\n- There are other loss functions in Keras, such as mean_absolute_error and binary_crossentropy. The choice of the loss function is not arbitrary but should be a criterion that the model is learning. \n\n- For classification by category, categorical_crossentropy or mean_squared_error is a good choice after the softmax activation layer. The binary_crossentropy loss function is normally used after the sigmoid activation layer while mean_squared_error is an option for tanh output.","metadata":{"_uuid":"ef670967-0f5a-4eec-8b04-294d20896d73","_cell_guid":"f2fb2e9e-783e-4192-9123-db7a099ac71d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### Optimization (optimizer adam)\n\n- With optimization, the objective is to minimize the loss function. The idea is that if the loss is reduced to an acceptable level, the model has indirectly learned the function mapping input to output.\n\n- In Keras, there are several choices for optimizers. The most commonly used optimizers are; **Stochastic Gradient Descent (SGD)**, **Adaptive Moments (Adam)** and **Root Mean Squared Propagation (RMSprop)**. \n\n- Each optimizer features tunable parameters like learning rate, momentum, and decay. \n\n- Adam and RMSprop are variations of SGD with adaptive learning rates. In the proposed classifier network, Adam is used since it has the highest test accuracy.","metadata":{"_uuid":"1cd636e4-dfd6-4f5c-af83-f75446277d05","_cell_guid":"1f8687c9-c7d9-4120-902c-a05b41dc1e5a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### Metrics (accuracy)\n\n- Performance metrics are used to determine if a model has learned the underlying data distribution. The default metric in Keras is loss. \n\n- During training, validation, and testing, other metrics such as **accuracy** can also be included. \n\n- **Accuracy** is the percent, or fraction, of correct predictions based on ground truth.","metadata":{"_uuid":"709e82bd-67fc-4d47-bef8-3dbc39677c9f","_cell_guid":"1fecec59-057b-43c3-a095-2b04e8c60a2d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### 1.4 Train the model with fit() method ","metadata":{"_uuid":"9aef1112-3e28-41a8-8c30-0713e2ec8379","_cell_guid":"9a757ed1-6f02-4a10-a6b1-5708dea02333","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Fit your model\nhistory = model.fit(X_train,Y_train , validation_split=0.1, epochs=20, # Number of epochs for training\n batch_size=32, # Number of samples per batch\n verbose=1\n )","metadata":{"_uuid":"82aaa58e-765a-4cb3-9ece-c9d1b4bf0e8f","_cell_guid":"39854e46-069d-44c1-891d-1c5966ce1b67","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:44:44.387737Z","iopub.execute_input":"2024-12-11T09:44:44.388476Z","iopub.status.idle":"2024-12-11T09:46:54.485181Z","shell.execute_reply.started":"2024-12-11T09:44:44.388438Z","shell.execute_reply":"2024-12-11T09:46:54.483991Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.5 Evaluating model performance with evaluate() method ","metadata":{"_uuid":"db2c47b3-d3f9-43a9-b0ca-baa2439dd48b","_cell_guid":"cc42a129-13c7-41a7-942e-ac79e0952c9e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"loss, acc = model.evaluate(X_train_, Y_train, batch_size=batch_size)\nprint(\"\\nTraining accuracy: %.1f%%\" % (100.0 * acc))\n\nloss, acc = model.evaluate(X_test_, Y_test, batch_size=batch_size)\nprint(\"\\nTest accuracy: %.1f%%\" % (100.0 * acc))","metadata":{"_uuid":"e6c0bcbe-ac06-400b-bce5-d2e4660be720","_cell_guid":"d035b7bd-a862-4168-b0a3-3a222b29f954","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:47:08.291269Z","iopub.execute_input":"2024-12-11T09:47:08.292232Z","iopub.status.idle":"2024-12-11T09:47:10.160514Z","shell.execute_reply.started":"2024-12-11T09:47:08.292188Z","shell.execute_reply":"2024-12-11T09:47:10.159508Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.6 Monitoring the training process ","metadata":{"_uuid":"699e87f9-0698-486a-b4d4-0ff1a5721ce6","_cell_guid":"8d125d65-d7bf-4709-af56-47cc816883e0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"pd.DataFrame(history.history).head()","metadata":{"_uuid":"a01340a9-f480-4090-bbf8-5c55c8105e12","_cell_guid":"5edb4d24-3d86-423d-9b0b-12cbf937af36","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:47:15.024247Z","iopub.execute_input":"2024-12-11T09:47:15.024668Z","iopub.status.idle":"2024-12-11T09:47:15.045217Z","shell.execute_reply.started":"2024-12-11T09:47:15.02463Z","shell.execute_reply":"2024-12-11T09:47:15.044268Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.title('Loss')\nplt.plot(history.history['loss'], label='train loss')\nplt.plot(history.history['val_loss'], label='validation loss')\n# Set x-axis ticks to be integers\nplt.xticks(np.arange(0, len(history.history['loss']), step=5)) # This sets ticks from 0 to 10\nplt.ylabel('Errors') \nplt.xlabel('Iterarions') \nplt.legend()\nplt.show()","metadata":{"_uuid":"1ceb9fa1-90ca-4899-83d0-db38f7052eba","_cell_guid":"7db394f7-aa5d-4e43-bb42-ddd5ea180480","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:47:19.1121Z","iopub.execute_input":"2024-12-11T09:47:19.112779Z","iopub.status.idle":"2024-12-11T09:47:19.309986Z","shell.execute_reply.started":"2024-12-11T09:47:19.112734Z","shell.execute_reply":"2024-12-11T09:47:19.308925Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.title('accuracy')\nplt.plot(history.history['accuracy'], label='train accuracy')\nplt.plot(history.history['val_accuracy'], label='val accuracy')\n# Set x-axis ticks to be integers\nplt.xticks(np.arange(0, len(history.history['loss']), step=5)) # This sets ticks from 0 to 10\nplt.ylabel('Accuracies') \nplt.xlabel('Iterarions') \nplt.legend()\nplt.show()","metadata":{"_uuid":"40efa8c1-343b-4b97-87e9-bc91ed5527bd","_cell_guid":"b8acfce7-57ac-4760-9cfa-fe120a1a995e","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:47:23.718993Z","iopub.execute_input":"2024-12-11T09:47:23.719369Z","iopub.status.idle":"2024-12-11T09:47:23.942292Z","shell.execute_reply.started":"2024-12-11T09:47:23.719335Z","shell.execute_reply":"2024-12-11T09:47:23.941377Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Results and Conclusion \n\n[Back to Table of Contents](#0.1)\n\n\n- In this notebook, We build a ANN neural network model using Keras DL library to classify the MNIST digits and predict accuracy.","metadata":{"_uuid":"1c6d721e-6657-4844-b409-8f2dc9bbc8ad","_cell_guid":"7e66974f-6f9f-454d-a0e1-a6d0648a9450","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Visualize Evaluation Metrics of ML Models\nfrom sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n# Make predictions\nY_pred = model.predict(X_test_)\nY_pred_categorical = np.argmax(Y_pred, axis=1)\nY_test_categorical = np.argmax(Y_test, axis=1)\n\n# Compute the confusion matrix\ncm = confusion_matrix(Y_test_categorical, Y_pred_categorical)\n\n# Plot the confusion matrix\ndisp = ConfusionMatrixDisplay(confusion_matrix=cm)\ndisp.plot(cmap=plt.cm.Blues)\nplt.show()","metadata":{"_uuid":"0a1155a0-bdbd-4615-b759-bdf9b71c0c82","_cell_guid":"03c8dc77-1ead-49f6-b330-33b7aadf2e7b","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:47:30.660155Z","iopub.execute_input":"2024-12-11T09:47:30.66057Z","iopub.status.idle":"2024-12-11T09:47:32.832809Z","shell.execute_reply.started":"2024-12-11T09:47:30.660528Z","shell.execute_reply":"2024-12-11T09:47:32.831654Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn import metrics\nprint(metrics.classification_report(Y_test_categorical, Y_pred_categorical))","metadata":{"_uuid":"4bca7f37-1347-438d-a3a9-d102eaf8ed09","_cell_guid":"64a3bed2-057c-4d44-9466-9e448bedc171","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:48:04.293544Z","iopub.execute_input":"2024-12-11T09:48:04.293971Z","iopub.status.idle":"2024-12-11T09:48:04.322892Z","shell.execute_reply.started":"2024-12-11T09:48:04.293937Z","shell.execute_reply":"2024-12-11T09:48:04.321902Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Developing a User Interface to interact with the trained model\n# 3. Using the trained model in the production environment","metadata":{"_uuid":"8c3ed275-c1a2-4928-9881-469163312a67","_cell_guid":"6bacdeca-36d1-40d4-b9f0-c774acea5609","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"*Draw a digit on the black background* using the Paint application and save it as **png** image. As seen the following example:\n\n","metadata":{"_uuid":"ccefe98e-04db-49b2-8da5-fc5dc58e4433","_cell_guid":"aed516dc-19df-484c-8fcf-da5ffd6fc455","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Draw a number on the black background using the Paint application and save it as png image. As seen in the example above:\n# Execute this cell to make available the upload button and upload your image from your computer to the kaggle notebook. \nimport ipywidgets as widgets\nfrom ipywidgets import interact\n# Define the widget\nfile_upload = widgets.FileUpload(\n accept='.png', # The type of files to accept\n multiple=False # Whether to allow to upload multiple files or not\n)\n\n# Display the widget\ndisplay(file_upload)","metadata":{"_uuid":"86e115c1-854b-47ed-b269-b345c09bf12c","_cell_guid":"16722b77-fa9e-4b5b-83f3-cf1ad8e2480f","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T10:08:33.313376Z","iopub.execute_input":"2024-12-11T10:08:33.31418Z","iopub.status.idle":"2024-12-11T10:08:33.326928Z","shell.execute_reply.started":"2024-12-11T10:08:33.314139Z","shell.execute_reply":"2024-12-11T10:08:33.325955Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Run this cell to see the prediction for your uploaded image.\n\nif len(file_upload.data)!=0:\n import io\n rawdata = file_upload.data[0]\n #When you need to handle binary data (like images or files) without saving them to disk, io.BytesIO can be very handy\n img = io.BytesIO(rawdata)\n # loading and displaying the sample image\n from PIL import Image\n import numpy as np\n import matplotlib.pyplot as plt \n img=Image.open(img).convert('L').resize((28, 28))\n \n # convert('L') loads the image as gray scale image.\n img=np.array(img) # convert the image into the Numpy array.\n \n # Plot the image\n plt.figure(figsize=(2, 2)) # Width: 2 inches, Height: 2 inches\n plt.imshow(img, cmap='gray')\n plt.axis('off') # Hide the axis\n plt.show()\n # Print the Shape of image as it is a Numpy Array\n print(\"Image is reshaped into:\", img.shape)\n\n # Scalling the image so that all pixel values can fall into \n # the range of 1 to 0 \n # Same as our training samples\n img=img/255\n # reshape our image data matrix \n # so that match input shape of the our trained perceptron network \n img=img.reshape(-1, 28*28)\n print('Flattened image shape:\\n',img.shape)\n prediction = model.predict(img)\n prediction = np.argmax(prediction, axis=1)\n print(\"Prediction for the uploaded image is: \", prediction)","metadata":{"_uuid":"cda6780c-63b8-47ea-856f-43d286d6e0e6","_cell_guid":"b4b7f33e-c828-4083-bf4d-c6d8257ccf12","trusted":true,"collapsed":false,"execution":{"iopub.status.busy":"2024-12-11T09:55:42.536492Z","iopub.execute_input":"2024-12-11T09:55:42.537212Z","iopub.status.idle":"2024-12-11T09:55:42.574775Z","shell.execute_reply.started":"2024-12-11T09:55:42.537172Z","shell.execute_reply":"2024-12-11T09:55:42.573424Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# References \n\n1. https://keras.io/api/datasets/mnist/","metadata":{"_uuid":"494d639d-b680-4b98-9751-5ed747aa393a","_cell_guid":"88e9f478-9ec6-488d-af3c-de0f8d2b28d1","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}}]}