{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":29841,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0\"></a>\n# MNIST - Deep Neural Network with Keras","metadata":{}},{"cell_type":"markdown","source":"# 1. Objective of the notebook <a class=\"anchor\" id=\"1\"></a>\n\n[Back to Table of Contents](#0.1)\n\n- In this notebook, we create a **Multilayer Perceptron (MLP)** model of the MNIST dataset. \n\n- **Multilayer Perceptrons (MLPs)** usually mean fully connected networks, that is, each neuron in one layer is connected to all neurons in the next layer. The \"fully-connectedness\" of these networks makes them prone to overfitting the data. \n\n- These MLP models are also referred to as either **deep feedforward networks** or **feedforward neural networks**. MLPs are common in simple logistic and linear regression problems.\n\n- So, the objective is to create a neural network for identifying numbers based on handwritten digits. For example, when the input to the network is an image of a handwritten number 8, the corresponding prediction must also be the digit 8. \n\n- To both train and validate a neural network, there must be a sufficiently large dataset of handwritten digits.\n\n- The Modified National Institute of Standards and Technology dataset or MNIST dataset for short, is often considered as the Hello World! of deep learning and is a suitable dataset for handwritten digit classification.\n\n- MNIST is used to explain and validate deep learning theories because the 70,000 samples it contains are small, yet sufficiently rich in information (MNIST dataset is described later).","metadata":{}},{"cell_type":"markdown","source":"So, we will start by importing the necessary libraries:","metadata":{}},{"cell_type":"markdown","source":"# 2. Import necessary libraries <a class=\"anchor\" id=\"2\"></a>\n\n[Back to Table of Contents](#0.1)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\nimport matplotlib.pyplot as plt # plotting library\n%matplotlib inline\n\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense , Activation, Dropout\nfrom keras.optimizers import Adam ,RMSprop\nfrom keras import  backend as K\n\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n\n# Any results you write to the current directory are saved as output.\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:36.490932Z","iopub.execute_input":"2024-12-17T05:55:36.491267Z","iopub.status.idle":"2024-12-17T05:55:38.519431Z","shell.execute_reply.started":"2024-12-17T05:55:36.491217Z","shell.execute_reply":"2024-12-17T05:55:38.518645Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. MNIST dataset <a class=\"anchor\" id=\"3\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- MNIST is a collection of handwritten digits ranging from the number 0 to 9. \n\n- It has a training set of 60,000 images, and 10,000 test images that are classified into corresponding categories or labels. \n\n- To use the MNIST dataset in Keras, an API is provided to download and extract images and labels automatically. \n\n- The following Keras code shows how to access MNIST dataset, plot 25 random samples, and count the number of labels for train and test datasets.","metadata":{}},{"cell_type":"code","source":"# import dataset\nfrom keras.datasets import mnist\n\n\n# load dataset\n(x_train, y_train),(x_test, y_test) = mnist.load_data()\n\n\n\n# count the number of unique train labels\nunique, counts = np.unique(y_train, return_counts=True)\nprint(\"Train labels: \", dict(zip(unique, counts)))\n\n\n\n# count the number of unique test labels\nunique, counts = np.unique(y_test, return_counts=True)\nprint(\"\\nTest labels: \", dict(zip(unique, counts)))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:38.521384Z","iopub.execute_input":"2024-12-17T05:55:38.521638Z","iopub.status.idle":"2024-12-17T05:55:39.779658Z","shell.execute_reply.started":"2024-12-17T05:55:38.521590Z","shell.execute_reply":"2024-12-17T05:55:39.778949Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Data visualization <a class=\"anchor\" id=\"4\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- The following code will help to sample the 25 random MNIST digits and visualize them.","metadata":{}},{"cell_type":"code","source":"# sample 25 mnist digits from train dataset\nindexes = np.random.randint(0, x_train.shape[0], size=25)\nimages = x_train[indexes]\nlabels = y_train[indexes]\n\n\n# plot the 25 mnist digits\nplt.figure(figsize=(5,5))\nfor i in range(len(indexes)):\n    plt.subplot(5, 5, i + 1)\n    image = images[i]\n    plt.imshow(image, cmap='gray')\n    plt.axis('off')\n    \nplt.show()\nplt.savefig(\"mnist-samples.png\")\nplt.close('all')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:39.780807Z","iopub.execute_input":"2024-12-17T05:55:39.781052Z","iopub.status.idle":"2024-12-17T05:55:40.621124Z","shell.execute_reply.started":"2024-12-17T05:55:39.781005Z","shell.execute_reply":"2024-12-17T05:55:40.620591Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Designing model architecture using Keras <a class=\"anchor\" id=\"5\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- The MLP model, discussed above can be used for MNIST digits classification. \n\n- When the units or perceptrons are exposed, the MLP model is a fully connected network.\n\n- The following code shows how to design the MLP model architecture using Keras.\n\n- The first step in designing the model architecture is to import the Keras layers. This can be done as follows:\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## 5.1 Import Keras layers <a class=\"anchor\" id=\"5.1\"></a>","metadata":{}},{"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, Activation, Dropout\nfrom keras.utils import to_categorical, plot_model","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:40.622114Z","iopub.execute_input":"2024-12-17T05:55:40.622326Z","iopub.status.idle":"2024-12-17T05:55:40.625890Z","shell.execute_reply.started":"2024-12-17T05:55:40.622288Z","shell.execute_reply":"2024-12-17T05:55:40.625113Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.2 Compute the number of labels <a class=\"anchor\" id=\"5.2\"></a>\n\n- Now, the data must be in the correct shape and format. \n\n- After loading the MNIST dataset, the number of labels is computed as:","metadata":{}},{"cell_type":"code","source":"# compute the number of labels\nnum_labels = len(np.unique(y_train))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:40.627958Z","iopub.execute_input":"2024-12-17T05:55:40.628200Z","iopub.status.idle":"2024-12-17T05:55:40.637651Z","shell.execute_reply.started":"2024-12-17T05:55:40.628143Z","shell.execute_reply":"2024-12-17T05:55:40.636839Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.3 One-Hot Encoding <a class=\"anchor\" id=\"5.3\"></a>\n\n- At this point, the labels are in digits format, 0 to 9. \n\n- This sparse scalar representation of labels is not suitable for the neural network prediction layer that outputs probabilities per class. \n\n- A more suitable format is called a one-hot vector, a 10-dim vector with all elements 0, except for the index of the digit class. \n\n- For example, if the label is 2, the equivalent one-hot vector is [0,0,1,0,0,0,0,0,0,0]. The first label has index 0.\n\n- The following lines convert each label into a one-hot vector:","metadata":{}},{"cell_type":"code","source":"# convert to one-hot vector\ny_train = to_categorical(y_train)\ny_test = to_categorical(y_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:40.638854Z","iopub.execute_input":"2024-12-17T05:55:40.639072Z","iopub.status.idle":"2024-12-17T05:55:40.647505Z","shell.execute_reply.started":"2024-12-17T05:55:40.639026Z","shell.execute_reply":"2024-12-17T05:55:40.646886Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.4 Data Preprocessing <a class=\"anchor\" id=\"5.4\"></a>\n\n\n- In deep learning, data is stored in tensors. The term tensor applies to a scalar (0D tensor), vector (1D tensor), matrix (2D tensor), and a multi-dimensional tensor.\n\n- The rest code computes the image dimensions, input_size of the first Dense layer and scales each pixel value from 0 to 255 to range from 0.0 to 1.0. Although raw pixel values can be used directly, it is better to normalize the input data as to avoid large gradient values that could make training difficult. \n\n- The output of the network is also normalized. After training, there is an option to put everything back to the integer pixel values by multiplying the output tensor by 255.  \n\n- The proposed model is based on MLP layers. Therefore, the input is expected to be a 1D tensor. So, x_train and x_test are reshaped to [60000, 28 * 28] and [10000, 28 * 28], respectively.","metadata":{}},{"cell_type":"code","source":"# image dimensions (assumed square)\nimage_size = x_train.shape[1]\ninput_size = image_size * image_size\ninput_size","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:40.648768Z","iopub.execute_input":"2024-12-17T05:55:40.649057Z","iopub.status.idle":"2024-12-17T05:55:40.658326Z","shell.execute_reply.started":"2024-12-17T05:55:40.649004Z","shell.execute_reply":"2024-12-17T05:55:40.657747Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# resize and normalize\nx_train = np.reshape(x_train, [-1, input_size])\nx_train = x_train.astype('float32') / 255\nx_test = np.reshape(x_test, [-1, input_size])\nx_test = x_test.astype('float32') / 255","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:40.659373Z","iopub.execute_input":"2024-12-17T05:55:40.659613Z","iopub.status.idle":"2024-12-17T05:55:40.892620Z","shell.execute_reply.started":"2024-12-17T05:55:40.659568Z","shell.execute_reply":"2024-12-17T05:55:40.891972Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.5 Setting network parameters <a class=\"anchor\" id=\"5.5\"></a>\n\n- Now, we will set the network parameters as follows:","metadata":{}},{"cell_type":"code","source":"# network parameters\nbatch_size = 128\nhidden_units = 256\ndropout = 0.45","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:55:40.893687Z","iopub.execute_input":"2024-12-17T05:55:40.893926Z","iopub.status.idle":"2024-12-17T05:55:40.897472Z","shell.execute_reply.started":"2024-12-17T05:55:40.893880Z","shell.execute_reply":"2024-12-17T05:55:40.896855Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The **batch_size** argument indicates the number of data that we will use for each update of the model parameters.\n\n- **Hidden_units** shows the number of hidden units.\n\n- **Dropout** is the dropout rate (more on this in section 7 - **Overfitting and Regularization**).","metadata":{}},{"cell_type":"markdown","source":"## 5.6 Designing the model architecture <a class=\"anchor\" id=\"5.6\"></a>\n\n\n- The next step is to design the model architecture. The proposed model is made of three MLP layers. \n\n- In Keras, an MLP layer is referred to as Dense, which stands for the densely connected layer. \n\n- Both the first and second MLP layers are identical in nature with 256 units each, followed by relu activation and dropout. \n\n- 256 units are chosen since 128, 512 and 1,024 units have lower performance metrics. At 128 units, the network converges quickly, but has a lower test accuracy. The added number units for 512 or 1,024 does not increase the test accuracy significantly.","metadata":{}},{"cell_type":"markdown","source":"- The main data structure in Keras is the Sequential class, which allows the creation of a basic neural network.\n\n- The Sequential class of the Keras library is a wrapper for the sequential neural network model that Keras offers and can be created in the following way:\n\n`from keras.models import Sequential`\n\n`model = Sequential()`\n\n- The model in Keras is considered as a sequence of layers and each of them gradually “distills” the input data to obtain the desired output.\n\n- In Keras, we can add the required types of layers through the **add()** method.","metadata":{}},{"cell_type":"code","source":"# model is a 3-layer MLP with ReLU and dropout after each layer\nmodel = Sequential()\nmodel.add(Dense(hidden_units, input_dim=input_size))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(dropout))\nmodel.add(Dense(hidden_units))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(dropout))\nmodel.add(Dense(10))\nmodel.add(Activation('softmax'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:20:52.052203Z","iopub.execute_input":"2024-12-17T06:20:52.052525Z","iopub.status.idle":"2024-12-17T06:20:52.112637Z","shell.execute_reply.started":"2024-12-17T06:20:52.052467Z","shell.execute_reply":"2024-12-17T06:20:52.111900Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Since a Dense layer is a linear operation, a sequence of Dense layers can only approximate a linear function. \n\n- The problem is that the MNIST digit classification is inherently a non-linear process. Inserting a relu activation between Dense layers will enable MLPs to model non-linear mappings. \n\n- relu or Rectified Linear Unit (ReLU) is a simple non-linear function. It allows positive inputs to pass through unchanged while clamping everything else to zero.","metadata":{}},{"cell_type":"markdown","source":"## 5.7 View model summary <a class=\"anchor\" id=\"5.7\"></a>\n\n- Keras library provides us **summary()** method to check the model description.","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:20:54.015047Z","iopub.execute_input":"2024-12-17T06:20:54.015399Z","iopub.status.idle":"2024-12-17T06:20:54.021005Z","shell.execute_reply.started":"2024-12-17T06:20:54.015346Z","shell.execute_reply":"2024-12-17T06:20:54.020200Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The above listing shows the model summary of the proposed network. It requires a total of 269,322 parameters.\n\n- This is substantial considering that we have a simple task of classifying MNIST digits. So, MLPs are not parameter efficient. \n\n- The total number of parameters required can be computed as follows:\n\n  - From input to Dense layer: 784 × 256 + 256 = 200,960. \n  \n  - From first Dense to second Dense: 256 × 256 + 256 = 65,792. \n  \n  - From second Dense to the output layer: 10 × 256 + 10 = 2,570. \n  \n  - The total is 200,690 + 65,972 + 2,570 = 269,322.","metadata":{}},{"cell_type":"markdown","source":"- Another way of verifying the network is by calling the **plot_model()** method as follows:","metadata":{}},{"cell_type":"code","source":"plot_model(model, to_file='mlp-mnist.png', show_shapes=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:20:56.231908Z","iopub.execute_input":"2024-12-17T06:20:56.232190Z","iopub.status.idle":"2024-12-17T06:20:56.381198Z","shell.execute_reply.started":"2024-12-17T06:20:56.232150Z","shell.execute_reply":"2024-12-17T06:20:56.380342Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 6. Implement MLP model using Keras <a class=\"anchor\" id=\"6\"></a>\n\n\n[Back to Table of Contents](#0.1)\n\n\n- The implementation of MLP model in Keras comprises of three steps:-\n\n  - Compiling the model with the compile() method.\n  \n  - Training the model with fit() method.\n  \n  - Evaluating the model performance with evaluate() method.\n  \n  \n- For detailed discussion on implementation, please refer to my previous kernel [Comprehensive Guide to ANN with Keras](https://www.kaggle.com/prashant111/comprehensive-guide-to-ann-with-keras)","metadata":{}},{"cell_type":"markdown","source":"## 6.1 Compile the model with compile() method <a class=\"anchor\" id=\"6.1\"></a>\n\n\n- Compilation of model can be done as follows:","metadata":{}},{"cell_type":"code","source":"model.compile(loss='categorical_crossentropy', \n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:20:58.022223Z","iopub.execute_input":"2024-12-17T06:20:58.022511Z","iopub.status.idle":"2024-12-17T06:20:58.087937Z","shell.execute_reply.started":"2024-12-17T06:20:58.022470Z","shell.execute_reply":"2024-12-17T06:20:58.087357Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Loss function (categorical_crossentropy)\n\n- How far the predicted tensor is from the one-hot ground truth vector is called **loss**.\n\n- In this example, we use **categorical_crossentropy** as the loss function. It is the negative of the sum of the product of the target and the logarithm of the prediction. \n\n- There are other loss functions in Keras, such as mean_absolute_error and binary_crossentropy. The choice of the loss function is not arbitrary but should be a criterion that the model is learning. \n\n- For classification by category, categorical_crossentropy or mean_squared_error is a good choice after the softmax activation layer. The binary_crossentropy loss function is normally used after the sigmoid activation layer while mean_squared_error is an option for tanh output.","metadata":{}},{"cell_type":"markdown","source":"### Optimization (optimizer adam)\n\n- With optimization, the objective is to minimize the loss function. The idea is that if the loss is reduced to an acceptable level, the model has indirectly learned the function mapping input to output.\n\n- In Keras, there are several choices for optimizers. The most commonly used optimizers are; **Stochastic Gradient Descent (SGD)**, **Adaptive Moments (Adam)** and **Root Mean Squared Propagation (RMSprop)**. \n\n- Each optimizer features tunable parameters like learning rate, momentum, and decay. \n\n- Adam and RMSprop are variations of SGD with adaptive learning rates. In the proposed classifier network, Adam is used since it has the highest test accuracy.","metadata":{}},{"cell_type":"markdown","source":"### Metrics (accuracy)\n\n- Performance metrics are used to determine if a model has learned the underlying data distribution. The default metric in Keras is loss. \n\n- During training, validation, and testing, other metrics such as **accuracy** can also be included. \n\n- **Accuracy** is the percent, or fraction, of correct predictions based on ground truth.","metadata":{}},{"cell_type":"markdown","source":"## 6.2 Train the model with fit() method <a class=\"anchor\" id=\"6.2\"></a>","metadata":{}},{"cell_type":"code","source":"model.fit(x_train, y_train, epochs=20, batch_size=batch_size)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:21:00.196847Z","iopub.execute_input":"2024-12-17T06:21:00.197132Z","iopub.status.idle":"2024-12-17T06:21:28.465781Z","shell.execute_reply.started":"2024-12-17T06:21:00.197076Z","shell.execute_reply":"2024-12-17T06:21:28.465098Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6.3 Evaluating model performance with evaluate() method <a class=\"anchor\" id=\"6.3\"></a>","metadata":{}},{"cell_type":"code","source":"loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)\nprint(\"\\nTest accuracy: %.1f%%\" % (100.0 * acc))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:21:28.467650Z","iopub.execute_input":"2024-12-17T06:21:28.467847Z","iopub.status.idle":"2024-12-17T06:21:28.723535Z","shell.execute_reply.started":"2024-12-17T06:21:28.467813Z","shell.execute_reply":"2024-12-17T06:21:28.722712Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for i in range(len(x_test)):\n    if i < 30:  # Show the first 30 images\n        image = x_test[i]\n        label = y_test[i]\n        \n        # Reshape the image back to its original dimensions, assuming grayscale (e.g., 28x28)\n        image_reshaped = image.reshape(28, 28)  # Replace 28x28 with your original shape if different\n\n        # Show image using imshow\n        plt.imshow(image_reshaped, cmap='gray')  # Use cmap for grayscale images\n        prediction = model.predict(image.reshape(1, *image.shape))\n        print(prediction.shape)# Reshape for prediction if needed\n        plt.title(f\"True: {np.argmax(label)}\\nPred: {np.argmax(prediction)}\")\n        plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T06:21:53.673962Z","iopub.execute_input":"2024-12-17T06:21:53.674238Z","iopub.status.idle":"2024-12-17T06:21:57.110597Z","shell.execute_reply.started":"2024-12-17T06:21:53.674198Z","shell.execute_reply":"2024-12-17T06:21:57.109939Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 7. Overfitting and Regularization <a class=\"anchor\" id=\"7\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- A neural network has the property to memorize the characteristics of training data. This is called **overfitting**.\n\n- In such a case, the network fails to generalize when subject to the test data.\n\n- To avoid this tendency, the model uses a regularizing layer or function. A commonly used regularizing layer is referred to as a **Dropout layer**.\n\n- Given a dropout rate (dropout=0.45), the **Dropout layer** randomly removes the fraction of units from participating in the next layer. For example, if the first layer has 256 units, after dropout=0.45 is applied, only (1 - 0.45) * 256 units = 140 units from layer 1 participate in layer 2.\n\n- The Dropout layer makes neural networks robust to unforeseen input data because the network is trained to predict correctly, even if some units are missing. \n\n- The dropout is not used in the output layer and it is only active during training. Moreover, dropout is not present during prediction.\n\n- There are regularizers that can be used other than dropouts like l1 or l2. In Keras, the bias, weight and activation output can be regularized per layer. - l1 and l2 favor smaller parameter values by adding a penalty function. Both l1 and l2 enforce the penalty using a fraction of the sum of absolute (l1) or square (l2) of parameter values. \n\n- So, the penalty function forces the optimizer to find parameter values that are small. Neural networks with small parameter values are more insensitive to the presence of noise from within the input data.\n\n- So, the l2 weight regularizer with fraction=0.001 can be implemented as:","metadata":{}},{"cell_type":"code","source":"from keras.regularizers import l2\nmodel.add(Dense(hidden_units,\n                kernel_regularizer=l2(0.001),\n                input_dim=input_size))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-17T05:56:13.571367Z","iopub.execute_input":"2024-12-17T05:56:13.571672Z","iopub.status.idle":"2024-12-17T05:56:13.589237Z","shell.execute_reply.started":"2024-12-17T05:56:13.571614Z","shell.execute_reply":"2024-12-17T05:56:13.588596Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- No additional layer is added if l1 or l2 regularization is used. The regularization is imposed in the Dense layer internally. For the proposed model, dropout still has a better performance than l2.","metadata":{}},{"cell_type":"markdown","source":"# 8. Results and Conclusion <a class=\"anchor\" id=\"8\"></a>\n\n[Back to Table of Contents](#0.1)\n\n\n- In this kernel, I build a dense neural network model to classify the MNIST digits and predict accuracy.\n\n- We get the test accuracy of 98.3%.\n\n- The accuracy tells us that our ANN model classifies the digits 98.3% of time correctly.\n\n","metadata":{}},{"cell_type":"markdown","source":"# 9. References <a class=\"anchor\" id=\"9\"></a>\n\n\n[Back to Table of Contents](#0.1)\n\n\nThis kernel is based on the following Kaggle kernels and books:\n\n- [My previous kernel on Artificial Neural Network](https://www.kaggle.com/prashant111/comprehensive-guide-to-ann-with-keras)\n\n- [Deep Neural Network Keras way by Poonam Ligade](https://www.kaggle.com/poonaml/deep-neural-network-keras-way)\n\n- [MNIST with Keras for beginners by Aditya Soni](https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457)\n\n-\tDeep Learning with Python by Francois Chollet\n\n-\tAdvanced Deep Learning with Keras by Rowel Atienza\n","metadata":{}},{"cell_type":"markdown","source":"Now we will come to the end of this kernel.\n\nI hope you find this kernel useful and enjoyable.\n\nThank you\n","metadata":{}},{"cell_type":"markdown","source":"[Go to Top](#0)","metadata":{}}]}