{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30587,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Deep Learning Assignment**","metadata":{}},{"cell_type":"markdown","source":"## ***Submitted By: Arya B***","metadata":{}},{"cell_type":"markdown","source":"#  Deep Neural Network for Handwritten Digit Classification with MNIST Dataset\n\n### **Introduction:**\n\nIn this notebook, we will explore the fascinating world of deep learning by creating a deep neural network for the task of handwritten digit classification. In this project, we will leverage the power of a Multilayer Perceptron (MLP) to accurately identify and classify digits (0 through 9) from the renowned MNIST dataset.\n\n### **Objective:**\n\nThe primary objective of this notebook is to build a robust deep neural network that can generalize well on the MNIST dataset. Handwritten digit classification serves as an excellent entry point to understand the principles of deep learning, making it a fundamental problem in the realm of artificial intelligence.\n\n### **Dataset Overview:**\n\nThe MNIST dataset, considered the \"Hello World!\" of deep learning, consists of 70,000 grayscale images of handwritten digits, each measuring 28x28 pixels. These images serve as both our training and testing data. The challenge is to train our neural network to correctly predict the digit represented in each image.\n\n### **Approach:**\n\nOur approach involves designing and training a deep neural network architecture, specifically an MLP. The network will be tasked with learning hierarchical features from the input images to make accurate predictions. We'll delve into the architecture design, discuss hyperparameters, and implement training routines to optimize the network's performance.\n\n### **Key Steps:**\n\n**Data Loading:** Import and preprocess the MNIST dataset for training and testing.\n\n**Model Architecture:** Design a deep neural network with multiple hidden layers for effective feature learning.\n\n**Model Compilation:** Choose an appropriate loss function, optimizer, and metrics for training.\n\n**Training:** Train the model on the training data, monitoring its performance on validation sets.\n\n**Evaluation:** Evaluate the trained model on the test set and analyze its classification accuracy.\n\n### **Conclusion:**\n\nBy the end of this notebook, we will  gain hands-on experience in building, training, and evaluating a deep neural network for image classification using the MNIST dataset. This project serves as a stepping stone for those eager to dive into the exciting field of deep learning.","metadata":{}},{"cell_type":"markdown","source":"### **Import necessary libraries**","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\nimport matplotlib.pyplot as plt # plotting library\n%matplotlib inline\n\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense , Activation, Dropout\nfrom keras.optimizers import Adam ,RMSprop\nfrom keras import  backend as K\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-12-03T17:17:20.116722Z","iopub.execute_input":"2023-12-03T17:17:20.117226Z","iopub.status.idle":"2023-12-03T17:17:34.824538Z","shell.execute_reply.started":"2023-12-03T17:17:20.117172Z","shell.execute_reply":"2023-12-03T17:17:34.823125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import dataset\nfrom keras.datasets import mnist\n\n\n# load dataset\n(x_train, y_train),(x_test, y_test) = mnist.load_data()\n\n\n\n# count the number of unique train labels\nunique, counts = np.unique(y_train, return_counts=True)\nprint(\"Train labels: \", dict(zip(unique, counts)))\n\n\n\n# count the number of unique test labels\nunique, counts = np.unique(y_test, return_counts=True)\nprint(\"\\nTest labels: \", dict(zip(unique, counts)))","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:17:54.132275Z","iopub.execute_input":"2023-12-03T17:17:54.133028Z","iopub.status.idle":"2023-12-03T17:17:54.625064Z","shell.execute_reply.started":"2023-12-03T17:17:54.132991Z","shell.execute_reply":"2023-12-03T17:17:54.623841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ### **Data visualization**","metadata":{}},{"cell_type":"code","source":"# sample 25 mnist digits from train dataset\nindexes = np.random.randint(0, x_train.shape[0], size=25)\nimages = x_train[indexes]\nlabels = y_train[indexes]\n\n\n# plot the 25 mnist digits\nplt.figure(figsize=(5,5))\nfor i in range(len(indexes)):\n    plt.subplot(5, 5, i + 1)\n    image = images[i]\n    plt.imshow(image, cmap='gray')\n    plt.axis('off')\n    \nplt.show()\nplt.savefig(\"mnist-samples.png\")\nplt.close('all')","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:18:57.802106Z","iopub.execute_input":"2023-12-03T17:18:57.802578Z","iopub.status.idle":"2023-12-03T17:18:58.940844Z","shell.execute_reply.started":"2023-12-03T17:18:57.802541Z","shell.execute_reply":"2023-12-03T17:18:58.939640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ### **Import Keras layers**","metadata":{}},{"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, Activation, Dropout\nfrom keras.utils import to_categorical, plot_model","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:19:52.168666Z","iopub.execute_input":"2023-12-03T17:19:52.169149Z","iopub.status.idle":"2023-12-03T17:19:52.175108Z","shell.execute_reply.started":"2023-12-03T17:19:52.169113Z","shell.execute_reply":"2023-12-03T17:19:52.173781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compute the number of labels\nnum_labels = len(np.unique(y_train))","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:20:31.726183Z","iopub.execute_input":"2023-12-03T17:20:31.726619Z","iopub.status.idle":"2023-12-03T17:20:31.733381Z","shell.execute_reply.started":"2023-12-03T17:20:31.726584Z","shell.execute_reply":"2023-12-03T17:20:31.732066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **One-Hot Encoding**\nAt this point, the labels are in digits format, 0 to 9.\n\nThis sparse scalar representation of labels is not suitable for the neural network prediction layer that outputs probabilities per class.\n\nA more suitable format is called a one-hot vector, a 10-dim vector with all elements 0, except for the index of the digit class.\n\nFor example, if the label is 2, the equivalent one-hot vector is [0,0,1,0,0,0,0,0,0,0]. The first label has index 0.\n\nThe following lines convert each label into a one-hot vector:","metadata":{}},{"cell_type":"code","source":"# convert to one-hot vector\ny_train = to_categorical(y_train)\ny_test = to_categorical(y_test)","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:21:14.079832Z","iopub.execute_input":"2023-12-03T17:21:14.080294Z","iopub.status.idle":"2023-12-03T17:21:14.088919Z","shell.execute_reply.started":"2023-12-03T17:21:14.080261Z","shell.execute_reply":"2023-12-03T17:21:14.087551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Data Preprocessing**\nIn deep learning, data is stored in tensors. The term tensor applies to a scalar (0D tensor), vector (1D tensor), matrix (2D tensor), and a multi-dimensional tensor.\n\nThe rest code computes the image dimensions, input_size of the first Dense layer and scales each pixel value from 0 to 255 to range from 0.0 to 1.0. Although raw pixel values can be used directly, it is better to normalize the input data as to avoid large gradient values that could make training difficult.\n\nThe output of the network is also normalized. After training, there is an option to put everything back to the integer pixel values by multiplying the output tensor by 255.\n\nThe proposed model is based on MLP layers. Therefore, the input is expected to be a 1D tensor. So, x_train and x_test are reshaped to [60000, 28 28] and [10000, 28 28], respectively.","metadata":{}},{"cell_type":"code","source":"# image dimensions (assumed square)\nimage_size = x_train.shape[1]\ninput_size = image_size * image_size\ninput_size","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:21:59.747134Z","iopub.execute_input":"2023-12-03T17:21:59.747563Z","iopub.status.idle":"2023-12-03T17:21:59.755265Z","shell.execute_reply.started":"2023-12-03T17:21:59.747528Z","shell.execute_reply":"2023-12-03T17:21:59.754162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# resize and normalize\nx_train = np.reshape(x_train, [-1, input_size])\nx_train = x_train.astype('float32') / 255\nx_test = np.reshape(x_test, [-1, input_size])\nx_test = x_test.astype('float32') / 255","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:22:12.522017Z","iopub.execute_input":"2023-12-03T17:22:12.522419Z","iopub.status.idle":"2023-12-03T17:22:12.645706Z","shell.execute_reply.started":"2023-12-03T17:22:12.522389Z","shell.execute_reply":"2023-12-03T17:22:12.644495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Setting network parameters**\n\nNow, we will set the network parameters as follows:","metadata":{}},{"cell_type":"code","source":"# network parameters\nbatch_size = 128\nhidden_units = 256\ndropout = 0.45","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:23:31.297290Z","iopub.execute_input":"2023-12-03T17:23:31.297729Z","iopub.status.idle":"2023-12-03T17:23:31.303609Z","shell.execute_reply.started":"2023-12-03T17:23:31.297699Z","shell.execute_reply":"2023-12-03T17:23:31.302336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The batch_size argument indicates the number of data that we will use for each update of the model parameters.\n \n* Hidden_units shows the number of hidden units.\n\n*  Dropout is the dropout rate (more on this in section 7 - Overfitting and Regularization).","metadata":{}},{"cell_type":"markdown","source":"##  Designing the model architecture <a class=\"anchor\" id=\"5.6\"></a>\n\n\n- The next step is to design the model architecture. The proposed model is made of three MLP layers. \n\n- In Keras, an MLP layer is referred to as Dense, which stands for the densely connected layer. \n\n- Both the first and second MLP layers are identical in nature with 256 units each, followed by relu activation and dropout. \n\n- 256 units are chosen since 128, 512 and 1,024 units have lower performance metrics. At 128 units, the network converges quickly, but has a lower test accuracy. The added number units for 512 or 1,024 does not increase the test accuracy significantly.","metadata":{}},{"cell_type":"markdown","source":"- The main data structure in Keras is the Sequential class, which allows the creation of a basic neural network.\n​\n- The Sequential class of the Keras library is a wrapper for the sequential neural network model that Keras offers and can be created in the following way:\n​\n`from keras.models import Sequential`\n​\n`model = Sequential()`\n​\n- The model in Keras is considered as a sequence of layers and each of them gradually “distills” the input data to obtain the desired output.\n​\n- In Keras, we can add the required types of layers through the **add()** method.","metadata":{}},{"cell_type":"code","source":"# model is a 3-layer MLP with ReLU and dropout after each layer\nmodel = Sequential()\nmodel.add(Dense(hidden_units, input_dim=input_size))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(dropout))\nmodel.add(Dense(hidden_units))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(dropout))\nmodel.add(Dense(num_labels))\nmodel.add(Activation('softmax'))","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:26:17.449898Z","iopub.execute_input":"2023-12-03T17:26:17.450372Z","iopub.status.idle":"2023-12-03T17:26:17.682046Z","shell.execute_reply.started":"2023-12-03T17:26:17.450300Z","shell.execute_reply":"2023-12-03T17:26:17.680633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Since a Dense layer is a linear operation, a sequence of Dense layers can only approximate a linear function. \n\n- The problem is that the MNIST digit classification is inherently a non-linear process. Inserting a relu activation between Dense layers will enable MLPs to model non-linear mappings. \n\n- relu or Rectified Linear Unit (ReLU) is a simple non-linear function. It allows positive inputs to pass through unchanged while clamping everything else to zero.","metadata":{}},{"cell_type":"markdown","source":"##  View model summary \n\n- Keras library provides us **summary()** method to check the model description.","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:27:18.182786Z","iopub.execute_input":"2023-12-03T17:27:18.183248Z","iopub.status.idle":"2023-12-03T17:27:18.216983Z","shell.execute_reply.started":"2023-12-03T17:27:18.183218Z","shell.execute_reply":"2023-12-03T17:27:18.215709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The above listing shows the model summary of the proposed network. It requires a total of 269,322 parameters.\n\n- This is substantial considering that we have a simple task of classifying MNIST digits. So, MLPs are not parameter efficient. \n\n- The total number of parameters required can be computed as follows:\n\n  - From input to Dense layer: 784 × 256 + 256 = 200,960. \n  \n  - From first Dense to second Dense: 256 × 256 + 256 = 65,792. \n  \n  - From second Dense to the output layer: 10 × 256 + 10 = 2,570. \n  \n  - The total is 200,690 + 65,972 + 2,570 = 269,322.","metadata":{}},{"cell_type":"markdown","source":"- Another way of verifying the network is by calling the **plot_model()** method as follows:","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:27:52.340404Z","iopub.execute_input":"2023-12-03T17:27:52.340804Z","iopub.status.idle":"2023-12-03T17:27:52.348537Z","shell.execute_reply.started":"2023-12-03T17:27:52.340774Z","shell.execute_reply":"2023-12-03T17:27:52.346627Z"}}},{"cell_type":"code","source":"plot_model(model, to_file='mlp-mnist.png', show_shapes=True)","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:28:20.832476Z","iopub.execute_input":"2023-12-03T17:28:20.832918Z","iopub.status.idle":"2023-12-03T17:28:21.128302Z","shell.execute_reply.started":"2023-12-03T17:28:20.832884Z","shell.execute_reply":"2023-12-03T17:28:21.127121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Implement MLP model using Keras**\n\nThe implementation of MLP model in Keras comprises of three steps:-\n\nCompiling the model with the compile() method.\n\nTraining the model with fit() method.\n\nEvaluating the model performance with evaluate() method.","metadata":{}},{"cell_type":"markdown","source":"### **Compile the model with compile() method**","metadata":{}},{"cell_type":"code","source":"model.compile(loss='categorical_crossentropy', \n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:31:00.989890Z","iopub.execute_input":"2023-12-03T17:31:00.990331Z","iopub.status.idle":"2023-12-03T17:31:01.012633Z","shell.execute_reply.started":"2023-12-03T17:31:00.990300Z","shell.execute_reply":"2023-12-03T17:31:01.011580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loss function (categorical_crossentropy)\n\n- How far the predicted tensor is from the one-hot ground truth vector is called **loss**.\n\n- In this example, we use **categorical_crossentropy** as the loss function. It is the negative of the sum of the product of the target and the logarithm of the prediction. \n\n- There are other loss functions in Keras, such as mean_absolute_error and binary_crossentropy. The choice of the loss function is not arbitrary but should be a criterion that the model is learning. \n\n- For classification by category, categorical_crossentropy or mean_squared_error is a good choice after the softmax activation layer. The binary_crossentropy loss function is normally used after the sigmoid activation layer while mean_squared_error is an option for tanh output.","metadata":{}},{"cell_type":"markdown","source":"### Optimization (optimizer adam)\n\n- With optimization, the objective is to minimize the loss function. The idea is that if the loss is reduced to an acceptable level, the model has indirectly learned the function mapping input to output.\n\n- In Keras, there are several choices for optimizers. The most commonly used optimizers are; **Stochastic Gradient Descent (SGD)**, **Adaptive Moments (Adam)** and **Root Mean Squared Propagation (RMSprop)**. \n\n- Each optimizer features tunable parameters like learning rate, momentum, and decay. \n\n- Adam and RMSprop are variations of SGD with adaptive learning rates. In the proposed classifier network, Adam is used since it has the highest test accuracy.","metadata":{}},{"cell_type":"markdown","source":"### Metrics (accuracy)\n\n- Performance metrics are used to determine if a model has learned the underlying data distribution. The default metric in Keras is loss. \n\n- During training, validation, and testing, other metrics such as **accuracy** can also be included. \n\n- **Accuracy** is the percent, or fraction, of correct predictions based on ground truth.","metadata":{}},{"cell_type":"markdown","source":"### **Train the model with fit() method**","metadata":{}},{"cell_type":"code","source":"model.fit(x_train, y_train, epochs=20, batch_size=batch_size)","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:32:32.614386Z","iopub.execute_input":"2023-12-03T17:32:32.614811Z","iopub.status.idle":"2023-12-03T17:33:39.683079Z","shell.execute_reply.started":"2023-12-03T17:32:32.614770Z","shell.execute_reply":"2023-12-03T17:33:39.681835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Evaluating model performance with evaluate() method**","metadata":{}},{"cell_type":"code","source":"loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)\nprint(\"\\nTest accuracy: %.1f%%\" % (100.0 * acc))","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:33:39.685080Z","iopub.execute_input":"2023-12-03T17:33:39.685454Z","iopub.status.idle":"2023-12-03T17:33:40.244521Z","shell.execute_reply.started":"2023-12-03T17:33:39.685381Z","shell.execute_reply":"2023-12-03T17:33:40.243374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Overfitting and Regularization**\n\n\n\n\n- A neural network has the property to memorize the characteristics of training data. This is called **overfitting**.\n\n- In such a case, the network fails to generalize when subject to the test data.\n\n- To avoid this tendency, the model uses a regularizing layer or function. A commonly used regularizing layer is referred to as a **Dropout layer**.\n\n- Given a dropout rate (dropout=0.45), the **Dropout layer** randomly removes the fraction of units from participating in the next layer. For example, if the first layer has 256 units, after dropout=0.45 is applied, only (1 - 0.45) * 256 units = 140 units from layer 1 participate in layer 2.\n\n- The Dropout layer makes neural networks robust to unforeseen input data because the network is trained to predict correctly, even if some units are missing. \n\n- The dropout is not used in the output layer and it is only active during training. Moreover, dropout is not present during prediction.\n\n- There are regularizers that can be used other than dropouts like l1 or l2. In Keras, the bias, weight and activation output can be regularized per layer. - l1 and l2 favor smaller parameter values by adding a penalty function. Both l1 and l2 enforce the penalty using a fraction of the sum of absolute (l1) or square (l2) of parameter values. \n\n- So, the penalty function forces the optimizer to find parameter values that are small. Neural networks with small parameter values are more insensitive to the presence of noise from within the input data.\n\n- So, the l2 weight regularizer with fraction=0.001 can be implemented as:","metadata":{}},{"cell_type":"code","source":"from keras.regularizers import l2\nmodel.add(Dense(hidden_units,\n                kernel_regularizer=l2(0.001),\n                input_dim=input_size))","metadata":{"execution":{"iopub.status.busy":"2023-12-03T17:36:33.032623Z","iopub.execute_input":"2023-12-03T17:36:33.033104Z","iopub.status.idle":"2023-12-03T17:36:33.060071Z","shell.execute_reply.started":"2023-12-03T17:36:33.033069Z","shell.execute_reply":"2023-12-03T17:36:33.058693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- No additional layer is added if l1 or l2 regularization is used. The regularization is imposed in the Dense layer internally. For the proposed model, dropout still has a better performance than l2.","metadata":{}},{"cell_type":"markdown","source":"###  **Results and Conclusion**\n\n\n\n- In this kernel, a deep neural network model is built to classify the MNIST digits and predict accuracy.\n\n- We get the test accuracy of 98.3%.\n\n- The accuracy tells us that our ANN model classifies the digits 98.3% of time correctly.\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}