{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Deep Learning - Prediction of Handwritten Digits

\n\n![](https://wallpaperbat.com/img/109996-machine-learning-wallpaper.jpg)\n\n#

Introduction 📖

\nThe main motivation behind creating this kernel was so I could explore more about the field of deep learning. In this kernel I'll try to explain some of the concepts I learnt and how they can be applied for our problem statement of identifying digits. We will explore neural networks in Python and then try to implement them using TensorFlow and Keras.\n\n### What is Tensorflow ?\n\nTensorFlow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks.\n\n\n\n### What is Keras ?\n\nKeras is a deep learning API written in Python, running on top of the machine learning platform `TensorFlow`. It was developed with a focus on enabling fast experimentation.\n\n\n\n#

Imports

","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport random\n\nfrom IPython.display import YouTubeVideo\n\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom keras.models import Model\nfrom keras.callbacks import ReduceLROnPlateau\n\nimport cv2\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:00.168961Z","iopub.execute_input":"2022-04-01T05:48:00.1692Z","iopub.status.idle":"2022-04-01T05:48:00.176112Z","shell.execute_reply.started":"2022-04-01T05:48:00.169167Z","shell.execute_reply":"2022-04-01T05:48:00.175078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

The Basics

\n
\nBefore we can start classifying our digits, lets try and understand some of the most basic things about deep learning.\n\n
\n\n#

What is a Tensor ?

\n**`Tensor`**: I've seen many complex definitions of tensors on Google and this was the simplest one I could find. A tensor is a container which can house data in N dimensions.\n\n\n\nSome of the common terms associated with `tensors` are:\n - `Rank`: It is defined as the number of dimensions of that tensor.\n - `Shape`: It represents the size of the each dimension.\n - `Data-type`: Type of data contained in the tensor, such as `int32`, `int64`, `float32`, `float64`, etc.\n\nLets take a look at creating tensors and then explore some of the operations we can perform with them.","metadata":{}},{"cell_type":"code","source":"# 1D tensor\na = tf.constant([0, 1, 2, 3])\nprint(a, '\\n')\n\n# 2D tensor\nb = tf.constant([[0, 1],\n [2, 3],\n [4, 5]])\nprint(b, '\\n')\n\n# 3D tensor\nc = tf.constant([[[0, 1, 2, 3, 4],\n [5, 6, 7, 8, 9]],\n [[10, 11, 12, 13, 14],\n [15, 16, 17, 18, 19]],\n [[20, 21, 22, 23, 24],\n [25, 26, 27, 28, 29]]])\nprint(c)","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-04-01T05:48:00.178428Z","iopub.execute_input":"2022-04-01T05:48:00.178726Z","iopub.status.idle":"2022-04-01T05:48:00.190783Z","shell.execute_reply.started":"2022-04-01T05:48:00.178691Z","shell.execute_reply":"2022-04-01T05:48:00.190012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we try to visualize the 3D tensor the break down would look like this:\n\n\n\nHigher dimensional tensors might be a bit tough to understand through code, so lets look at a visualization of what they look like:\n\n\n\n#

Tensor Operations

\n\nLet's take a look at a few operations that can be performed using `tensors`","metadata":{}},{"cell_type":"code","source":"# add two tensors\nprint(tf.add(5, 5), '\\n')\n\n# square the tensor\nprint(tf.square(10), '\\n')\n\n# adds all the tensors\nprint(tf.reduce_sum([5, 10, 100]), '\\n')\n\nt1 = tf.constant([[4, 5],\n [9, 10]])\nt2 = tf.constant([[3, 1],\n [15, 8]])\n\n# add 'n' tensors \nprint(tf.math.add_n([t1, t2, t2]), '\\n')\n\n# subtract the tensors\nprint(tf.subtract(t1, t2), '\\n')\n\nt3 = tf.constant([[1.5, 8],\n [16, 20]])\nt4 = tf.constant([[3.0, 4],\n [10, 25]])\n\n# multiply tensors\nprint(tf.multiply(t3, t4), '\\n')\n\n# divide the tensors\nprint(tf.divide(t3, t4), '\\n')","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:00.192297Z","iopub.execute_input":"2022-04-01T05:48:00.192578Z","iopub.status.idle":"2022-04-01T05:48:00.209728Z","shell.execute_reply.started":"2022-04-01T05:48:00.19252Z","shell.execute_reply":"2022-04-01T05:48:00.20906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

What is a Neural Network ?

\n\nNeural Networks (NNs) are one of the most important aspects of deep learning. [Neural Networks are a set of algorithms inspired by neurons in the human brain.](https://purnasaigudikandula.medium.com/a-beginner-intro-to-neural-networks-543267bda3c8)\n\nApart from this blog post, I found a few YouTube videos that explain what a neural network is. Definitely worth checking them out !!","metadata":{}},{"cell_type":"code","source":"YouTubeVideo('aircAruvnKk', width = 700, height = 500)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-01T05:48:00.21227Z","iopub.execute_input":"2022-04-01T05:48:00.212459Z","iopub.status.idle":"2022-04-01T05:48:00.345695Z","shell.execute_reply.started":"2022-04-01T05:48:00.212437Z","shell.execute_reply":"2022-04-01T05:48:00.34494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n A few more helpful videos on this topic:\n \n
\n\n#

The Perceptron

\n\nA perceptron is the basic unit of a neural network. It has only 1 layer. A perceptron consists of input values, weights, a bias, a weighted sum and activation function. This is what a **Perceptron** looks like:\n\n\nIt might seem a bit intimidating at first so lets break it down it smaller parts and understand what each part of a Perceptron really means.\n> - `Weights`: In simple terms weights decide how much influence the input will have on the output.\n> - `Bias`: Basically it is a constant value. Bias allows you to shift the activation function by adding a constant. \n> - `Weighted Sum`: It can be expressed according to this formula:
\n\n> - `Activation Function`: An Activation Function decides whether a neuron should be activated or not. Some of the most popular activation functions are:\n
    \n
  1. Sigmoid: It is especially used for models where we have to predict the probability as an output. Since probability of anything exists only between the range of 0 and 1, sigmoid is a good choice.\n \n
    2. \n
  2. ReLU (Rectified Linear Unit) : It will output the input directly if it is positive, otherwise, it will output zero. This is what it looks like:-\n

    \n
    3. \n
  3. Leaky ReLU: It is a type of activation function based on a ReLU, that has a small slope for negative values instead of a flat slope.\n

    \n
    4. \n
  4. Tanh: The function takes any real value as input and outputs values in the range -1 to 1.
    \n \n
    5. \n
\n \n#

Feed Forward Neural Networks (FFNN)

\nIn a feed forward neural network information flows only in **one direction**. It is one of the simplest forms of a neural network. It is a **Multi-Layered Perceptron (MLP)** as it is composed of multiple perceptrons. The opposite of a Feed Forward Neural Network is a **Recurrent Neural Network**, in which certain pathways are cycled.\n\n\n\nData enters at the inputs and passes through the network, layer by layer until it arrives at the output. During normal operation there is no feedback between the layers of the neural network.\n\nThe FFNN is the above example has the following architecture:\n> - 1 input layer with 4 neurons that accept inputs `x1`, `x2`, `x3` and `x4`\n> - 1st hidden layer with 4 neurons\n> - 2nd hidden layer with 5 neurons\n> - 3rd hidden layer with 6 neurons\n> - 4th hidden layer with 4 neurons\n> - 5th hidden layer with 3 neurons\n> - 1 output layer with 3 ouputs `y1`, `y2` and `y3`\n\nSince every neuron in a layer is connected to every other neuron in the next layer it is a Fully Connected Neural Network (FCNN).\n\n**Note**: Any neural network with more than 1 hidden layer is considered to be a **Deep Neural Network**.\n\n#

MNIST Dataset

\nThe MNIST (Modified National Institute of Standards and Technology) dataset for deep learning is like Titanic for machine learning. It is one of the simplest and easiest datasets to get started with deep learning and neural networks.","metadata":{}},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:00.346813Z","iopub.execute_input":"2022-04-01T05:48:00.347492Z","iopub.status.idle":"2022-04-01T05:48:00.353536Z","shell.execute_reply.started":"2022-04-01T05:48:00.347456Z","shell.execute_reply":"2022-04-01T05:48:00.35278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Taking a look at our data","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nprint(train_df)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:00.892122Z","iopub.execute_input":"2022-04-01T05:48:00.892718Z","iopub.status.idle":"2022-04-01T05:48:03.987707Z","shell.execute_reply.started":"2022-04-01T05:48:00.892683Z","shell.execute_reply":"2022-04-01T05:48:03.986962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets take a look at missing values in the MNIST dataset","metadata":{}},{"cell_type":"code","source":"print(train_df.isna().sum())","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:06.136495Z","iopub.execute_input":"2022-04-01T05:48:06.136956Z","iopub.status.idle":"2022-04-01T05:48:06.206349Z","shell.execute_reply.started":"2022-04-01T05:48:06.136912Z","shell.execute_reply":"2022-04-01T05:48:06.203701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fortunately there are no missing values present in the dataset.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (15, 10))\nax = sns.countplot(x = 'label', data = train_df)\nplt.title('Class Count', fontsize = 20)\nplt.xlabel('Labels', fontsize = 15)\nplt.ylabel('Count', fontsize = 15)\nfor p in ax.patches:\n ax.annotate('{:.0f}'.format(p.get_height()), (p.get_x() + 0.25, p.get_height() + 10))","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:09.245226Z","iopub.execute_input":"2022-04-01T05:48:09.245855Z","iopub.status.idle":"2022-04-01T05:48:09.551946Z","shell.execute_reply.started":"2022-04-01T05:48:09.245811Z","shell.execute_reply":"2022-04-01T05:48:09.55129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The number of images for each class is relatively similar. A huge class imbalance does not exist in the dataset, however there is a minor class imbalance that does exist.","metadata":{}},{"cell_type":"code","source":"X = train_df.drop(columns = ['label']).values.astype('float32')\ny = train_df['label'].values\nprint(X, '\\n')\nprint(y)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:11.40773Z","iopub.execute_input":"2022-04-01T05:48:11.408311Z","iopub.status.idle":"2022-04-01T05:48:11.534344Z","shell.execute_reply.started":"2022-04-01T05:48:11.408266Z","shell.execute_reply":"2022-04-01T05:48:11.533454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Splitting the Data

\nLets split our dataset using the `train_test_split` function. We set the `test_size` to 10 % of our dataset.","metadata":{}},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1, stratify = y, random_state = 0)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:12.417765Z","iopub.execute_input":"2022-04-01T05:48:12.418021Z","iopub.status.idle":"2022-04-01T05:48:13.038784Z","shell.execute_reply.started":"2022-04-01T05:48:12.41799Z","shell.execute_reply":"2022-04-01T05:48:13.038029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets take a look at the shape of `X_train`","metadata":{}},{"cell_type":"code","source":"print(X_train, '\\n')\nprint(X_train.shape)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-04-01T05:48:13.939903Z","iopub.execute_input":"2022-04-01T05:48:13.940449Z","iopub.status.idle":"2022-04-01T05:48:13.946287Z","shell.execute_reply.started":"2022-04-01T05:48:13.940411Z","shell.execute_reply":"2022-04-01T05:48:13.945384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The number of rows = 42000 - 0.1 * 42000 = 37800
\nThe number of columns = 784","metadata":{}},{"cell_type":"code","source":"X_train = X_train.reshape(X_train.shape[0], 28, 28)\nX_test = X_test.reshape(X_test.shape[0], 28, 28)\nprint(X_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:15.1103Z","iopub.execute_input":"2022-04-01T05:48:15.110559Z","iopub.status.idle":"2022-04-01T05:48:15.115838Z","shell.execute_reply.started":"2022-04-01T05:48:15.110528Z","shell.execute_reply":"2022-04-01T05:48:15.11498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`X_train` contains 37800 lists, in which each list has 28 sublists with 28 columns. This is a bit tricky to understand so let's take a look at the first list in `X_train`.","metadata":{}},{"cell_type":"code","source":"X_train[0]","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:16.271295Z","iopub.execute_input":"2022-04-01T05:48:16.271853Z","iopub.status.idle":"2022-04-01T05:48:16.286217Z","shell.execute_reply.started":"2022-04-01T05:48:16.271812Z","shell.execute_reply":"2022-04-01T05:48:16.285465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`X_train[0]` has 28 sublists that have 28 columns each. The shape has been modified since each image is 28 pixels in width and 28 pixels in height, having 784 pixels in total (28 x 28).","metadata":{}},{"cell_type":"code","source":"print(y_train)\nprint(y_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:17.464787Z","iopub.execute_input":"2022-04-01T05:48:17.465155Z","iopub.status.idle":"2022-04-01T05:48:17.471088Z","shell.execute_reply.started":"2022-04-01T05:48:17.46512Z","shell.execute_reply":"2022-04-01T05:48:17.470254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`y_train` is a single list with 37800 entries","metadata":{}},{"cell_type":"markdown","source":"#

Visualizing the Images

\n\nLets take a look at the what the images of our dataset actually look like","metadata":{}},{"cell_type":"code","source":"def show_random_images(color1, color2, color3, color4):\n \n '''Randomly prints images from the dataset with different color maps'''\n\n random_list = []\n for i in range(0, 4):\n random_number = random.randint(0, 37800)\n random_list.append(random_number)\n\n fig, axes = plt.subplots(2, 2, figsize = (10, 10))\n\n axes[0, 0].imshow(X_train[random_list[0]], cmap = color1)\n axes[0, 0].set_title('Label = ' + str(y_train[random_list[0]]))\n axes[0, 0].axis('off')\n\n axes[0, 1].imshow(X_train[random_list[1]], cmap = color2)\n axes[0, 1].set_title('Label = ' + str(y_train[random_list[1]]))\n axes[0, 1].axis('off')\n\n axes[1, 0].imshow(X_train[random_list[2]], cmap = color3)\n axes[1, 0].set_title('Label = ' + str(y_train[random_list[2]]))\n axes[1, 0].axis('off')\n\n axes[1, 1].imshow(X_train[random_list[3]], cmap = color4)\n axes[1, 1].set_title('Label = ' + str(y_train[random_list[3]]))\n axes[1, 1].axis('off')\n\n plt.show()\n\nshow_random_images('Greens', 'YlOrBr', 'gist_heat', 'winter')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-01T05:48:18.928061Z","iopub.execute_input":"2022-04-01T05:48:18.928638Z","iopub.status.idle":"2022-04-01T05:48:19.180902Z","shell.execute_reply.started":"2022-04-01T05:48:18.9286Z","shell.execute_reply":"2022-04-01T05:48:19.180136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_multiple_images(train_or_test, caption, title):\n \n '''Displays the first 25 images of the dataset'''\n\n r = 5\n c = 5 \n\n fig = plt.figure(figsize = (10, 10))\n\n for i in range(r * c): \n fig.add_subplot(r, c, i + 1)\n plt.imshow(train_or_test[i]) \n plt.title(caption + ' = ' + str(title[i]))\n plt.axis(\"off\")\n \ndisplay_multiple_images(X_train, 'Label', y_train)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-01T05:48:20.088846Z","iopub.execute_input":"2022-04-01T05:48:20.089282Z","iopub.status.idle":"2022-04-01T05:48:21.140809Z","shell.execute_reply.started":"2022-04-01T05:48:20.089227Z","shell.execute_reply":"2022-04-01T05:48:21.140103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def images_and_pixels(image, ax):\n \n '''Displays a random image along with its pixel values'''\n \n ax.imshow(image, cmap = 'PuBu')\n width, height = image.shape\n threshold = image.max() / 2.5\n \n for x in range(width):\n for y in range(height):\n ax.annotate(str(round(image[x][y],2)), \n xy = (y,x),\n horizontalalignment = 'center',\n verticalalignment = 'center',\n color = 'white' if image[x][y] < threshold else 'black')\n\nfig = plt.figure(figsize = (10,10)) \nax = fig.add_subplot(111)\nrandom_number = random.randint(0, 37800)\n\nimages_and_pixels(X_train[random_number], ax)\n\nplt.title('Label = ' + str(y_train[random_number]), fontsize = 20)\nplt.axis('off')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-01T05:48:22.282792Z","iopub.execute_input":"2022-04-01T05:48:22.283336Z","iopub.status.idle":"2022-04-01T05:48:26.080623Z","shell.execute_reply.started":"2022-04-01T05:48:22.283299Z","shell.execute_reply":"2022-04-01T05:48:26.07994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)\nX_test = X_test.reshape(X_test.shape[0], 28, 28, 1)\nprint(X_train.shape)\nprint(X_test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:26.081716Z","iopub.execute_input":"2022-04-01T05:48:26.081915Z","iopub.status.idle":"2022-04-01T05:48:26.087436Z","shell.execute_reply.started":"2022-04-01T05:48:26.08189Z","shell.execute_reply":"2022-04-01T05:48:26.086664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

CNN (Convolutional Neural Network)

\n\nA convolutional neural network is a class of deep neural networks, most commonly applied to analyze visual imagery. This is what a CNN looks like:\n\n\n\nSome of the common terms associated with it are:\n\n> - `Convolution`: It is the first layer to extract features from an input image. It preserves the relationship between pixels by learning image features using small squares of input data. It is a mathematical operation that takes two inputs:\n - An image matrix\n - Filter (or kernel)\n> - `Filter`: It is also called a `kernel`. Filters detect spatial patterns such as edges in an image by detecting the change in intensity values of the image. The values of the filters are learned automatically by the neural network through the training process.\n

\n> - `Feature Map`: This is also called the `Activation Map`. It is the result of the multiplication of the `image matrix` and the `filter`.\n

\n> - `Kernel Size`: It is the size of the `kernel` or the `filter`.\n> - `Stride`: It is the number of pixels shifted over the input image.\n

\n In the above example the image size is 5 x 5, the kernel size is 3 x 3 and the stride is 1.\n> - `Padding`: It extends the area of the image which is processed by the CNN. Adding padding to the image processed by the CNN allows for a more accurate analysis of the images.\n

\n> - `Max Pooling`: In `max pooling` the kernel extracts the maximum value of the area it convolves. It is advisable to perform `max pooling` when it is possible to extract some features from our image. It is not recommended to perform this operation in the initial stages of the CNN as the kernels would still be in the process of extracting edges and gradients.\n

\n> - `Batch Normalization`: This allows every layer of the network to do learning more independently. It is used to normalize the output of the previous layers.\n> - `Dropout`: This is a technique where randomly selected neurons are ignored during training. They are dropped out randomly. The effect is that the network becomes less sensitive to the specific weights of the neurons. This helps the network **generalize better** and makes it less likely to overfit the data.","metadata":{}},{"cell_type":"code","source":"model = keras.Sequential([\n keras.Input(shape = (28, 28, 1)),\n layers.Conv2D(32, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(32, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(32, 3, padding = 'same', activation = 'relu'),\n layers.MaxPooling2D(),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.MaxPooling2D(),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(128, 3, padding = 'same', activation = 'relu'),\n layers.Flatten(),\n layers.Dense(128, activation = 'relu'),\n layers.Dense(10, activation = 'softmax'),\n])\n\nmodel.compile(\n loss = keras.losses.SparseCategoricalCrossentropy(from_logits = True),\n optimizer = keras.optimizers.Adam(lr = 1e-4),\n metrics = ['acc'],\n)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:26.116884Z","iopub.execute_input":"2022-04-01T05:48:26.117732Z","iopub.status.idle":"2022-04-01T05:48:26.254888Z","shell.execute_reply.started":"2022-04-01T05:48:26.117689Z","shell.execute_reply":"2022-04-01T05:48:26.254195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have now defined our custom CNN model architecture, but what the **heck** does all this even mean?\n> - `Sequential` in Keras allows you to create models layer-by-layer for most problems. It is limited in that it does not allow you to create models that share layers or have multiple inputs or outputs.\n> - `Input` is used to accept the images in our dataset as the input. The shape has to be in the form of `width x height x channel`. Here the width = height = 28 and the number of channels = 1 since it is a blakc and white image. If we wanted to train the model for a colored images we would have set number of channels to 3. \n> - `Conv2D` is the Convolutional layer of our CNN. Once we understand the parameters of the first `Conv2D` layer it will become easy to understand the other convolutional layers as well. The first convolutional layer has:\n> > - `filters` = 32\n> > - `kernel_size` = 3\n> > - `padding = same`: It applies padding to the input image so that the input image gets fully covered by the filter and the specified stride. If the `stride` = 1 then the layer's output will have the same spatial dimensions as its input\n> > - `activation = 'relu'` is the activation function for the layer of the CNN\n> - `MaxPooling2D` as we saw earlier is used to help the kernel extract the maximum value of the area it convolves.\n> - `Flatten` helps you flatten the multi-dimension inputs into a single dimension\n> - `Dense` is the fully connected neural network layer. The first `Dense` layer has `units` = 128. This is the same as the number of neurons. It also has the `relu` activation function. It performs the following operation:\n> > - `output = activation(dot(input, kernel) + bias)`\n\nOnce we have defined the architecture to our CNN we have to `compile` the model. In our case our model is compiled having the following parameters:\n> - `loss`: The purpose of loss functions is to compute the quantity that a model should seek to minimize during training. We use `SparseCategoricalCrossentropy` when there are 2 or more classes in our dataset. You can take a look at the different types of loss functions provided by Keras over [here](https://keras.io/api/losses/).\n> - `optimizer`: We use the `Adam` optimizer which is an ADAptive Moment (ADAM) Estimation Algorithm. We set the `lr = 1e-4`. The **learning rate** is a hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are updated.\n> - `metrics` are functions that are used to judge the perfomance of the model. The `acc` metric computes the accuracy rates across all the predictions.\n\nKeras provies a way to looks at the summary of the model using `model.summary()`. The summary is textual and includes information about the layers and their order in the model, the output shape of each layer and the number of parameters (weights) in each layer.","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:28.394418Z","iopub.execute_input":"2022-04-01T05:48:28.394732Z","iopub.status.idle":"2022-04-01T05:48:28.406069Z","shell.execute_reply.started":"2022-04-01T05:48:28.394697Z","shell.execute_reply":"2022-04-01T05:48:28.405131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Using Learning Rate Reduction","metadata":{}},{"cell_type":"code","source":"reduce_lr = ReduceLROnPlateau(monitor = 'val_acc', patience = 3, verbose = 1, factor = 0.5, min_lr = 0.00001)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:48:40.175607Z","iopub.execute_input":"2022-04-01T05:48:40.175868Z","iopub.status.idle":"2022-04-01T05:48:40.181512Z","shell.execute_reply.started":"2022-04-01T05:48:40.175838Z","shell.execute_reply":"2022-04-01T05:48:40.179069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Training the CNN

","metadata":{}},{"cell_type":"code","source":"classifier = model.fit(X_train, y_train, batch_size = 64, \n epochs = 50, \n validation_data = (X_test, y_test), \n verbose = 1,\n steps_per_epoch = X_train.shape[0] // 64,\n callbacks = [reduce_lr])","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-04-01T05:48:41.943204Z","iopub.execute_input":"2022-04-01T05:48:41.943896Z","iopub.status.idle":"2022-04-01T05:52:04.728452Z","shell.execute_reply.started":"2022-04-01T05:48:41.943859Z","shell.execute_reply":"2022-04-01T05:52:04.727534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Visualizing CNN Layers

\nLets visualize the layers of our model and take a look at what our CNN is learning.","metadata":{}},{"cell_type":"code","source":"def CNN_layer_visualizer():\n \n layer_names = [layer.name for layer in model.layers]\n layer_outputs = [layer.output for layer in model.layers]\n \n activation_model = Model(inputs = model.input, outputs = layer_outputs)\n input = X_train[random_number]\n activations = activation_model.predict(input.reshape(-1, 28, 28, 1) / 255.0)\n \n for number in range(len(layer_names) - 4):\n fig = plt.figure(figsize = (22, 3))\n for img in range(30):\n ax = fig.add_subplot(2, 15, img + 1)\n ax = plt.imshow(activations[number][0, :, :, img], cmap = 'afmhot')\n main_title = fig.suptitle(str(layer_names[number]), fontsize = 20)\n plt.xticks([])\n plt.yticks([])\n fig.subplots_adjust(wspace = 0.05, hspace = 0.05)\n \nCNN_layer_visualizer()","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:52:08.414303Z","iopub.execute_input":"2022-04-01T05:52:08.414725Z","iopub.status.idle":"2022-04-01T05:52:19.395744Z","shell.execute_reply.started":"2022-04-01T05:52:08.414689Z","shell.execute_reply":"2022-04-01T05:52:19.39505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Model Performance

","metadata":{}},{"cell_type":"code","source":"def model_performance_graphs():\n \n fig, axes = plt.subplots(1, 2, figsize = (15, 8))\n\n axes[0].plot(classifier.epoch, classifier.history['acc'], label = 'acc')\n axes[0].plot(classifier.epoch, classifier.history['val_acc'], label = 'val_acc')\n axes[0].set_title('Accuracy vs Epochs', fontsize = 20)\n axes[0].set_xlabel('Epochs', fontsize = 15)\n axes[0].set_ylabel('Accuracy', fontsize = 15)\n axes[0].legend()\n\n axes[1].plot(classifier.epoch, classifier.history['loss'], label = 'loss')\n axes[1].plot(classifier.epoch, classifier.history['val_loss'], label=\"val_loss\")\n axes[1].set_title(\"Loss Curve\",fontsize=18)\n axes[1].set_xlabel(\"Epochs\",fontsize=15)\n axes[1].set_ylabel(\"Loss\",fontsize=15)\n axes[1].legend()\n\n plt.show()\n \nmodel_performance_graphs()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-01T05:52:22.751076Z","iopub.execute_input":"2022-04-01T05:52:22.751561Z","iopub.status.idle":"2022-04-01T05:52:23.254551Z","shell.execute_reply.started":"2022-04-01T05:52:22.751517Z","shell.execute_reply":"2022-04-01T05:52:23.25385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Evaluating the performance of our model on the training set","metadata":{}},{"cell_type":"code","source":"model.evaluate(X_train, y_train, batch_size = 64, verbose = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:52:29.442591Z","iopub.execute_input":"2022-04-01T05:52:29.443194Z","iopub.status.idle":"2022-04-01T05:52:32.198205Z","shell.execute_reply.started":"2022-04-01T05:52:29.443153Z","shell.execute_reply":"2022-04-01T05:52:32.197561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Evaluating the performance of our model on the testing set","metadata":{}},{"cell_type":"code","source":"model.evaluate(X_test, y_test, batch_size = 64, verbose = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:52:34.447544Z","iopub.execute_input":"2022-04-01T05:52:34.4481Z","iopub.status.idle":"2022-04-01T05:52:34.804857Z","shell.execute_reply.started":"2022-04-01T05:52:34.44806Z","shell.execute_reply":"2022-04-01T05:52:34.804128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Making the predictions using our trained model","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nfinal_labels = model.predict(((test_df.values).reshape(-1, 28, 28, 1) / 255.0))\nfinal_labels = np.argmax(final_labels, axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:52:39.540735Z","iopub.execute_input":"2022-04-01T05:52:39.541616Z","iopub.status.idle":"2022-04-01T05:52:44.552524Z","shell.execute_reply.started":"2022-04-01T05:52:39.541559Z","shell.execute_reply":"2022-04-01T05:52:44.551611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets see how well our predictions match up with the images from `test_df`","metadata":{}},{"cell_type":"code","source":"display_multiple_images(test_df.values.reshape(-1, 28, 28, 1), 'Prediction', final_labels)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:52:46.736225Z","iopub.execute_input":"2022-04-01T05:52:46.736925Z","iopub.status.idle":"2022-04-01T05:52:47.607823Z","shell.execute_reply.started":"2022-04-01T05:52:46.736888Z","shell.execute_reply":"2022-04-01T05:52:47.607123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Building 10 CNNs

","metadata":{}},{"cell_type":"code","source":"n = 10\nnew_models = [0] * n\n\nfor i in range(n):\n new_models[i] = keras.Sequential([\n keras.Input(shape = (28, 28, 1)),\n layers.Conv2D(32, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(32, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(32, 3, padding = 'same', activation = 'relu'),\n layers.MaxPooling2D(),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.MaxPooling2D(),\n layers.Conv2D(64, 3, padding = 'same', activation = 'relu'),\n layers.Conv2D(128, 3, padding = 'same', activation = 'relu'),\n layers.Flatten(),\n layers.Dense(128, activation = 'relu'),\n layers.Dense(10, activation = 'softmax'),\n ])\n\n new_models[i].compile(\n loss = keras.losses.SparseCategoricalCrossentropy(from_logits = True),\n optimizer = keras.optimizers.Adam(lr = 1e-4),\n metrics = ['acc'],\n )","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:52:52.637645Z","iopub.execute_input":"2022-04-01T05:52:52.637911Z","iopub.status.idle":"2022-04-01T05:52:53.441677Z","shell.execute_reply.started":"2022-04-01T05:52:52.63788Z","shell.execute_reply":"2022-04-01T05:52:53.440942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = [0] * n\nfor i in range(n):\n history[i] = new_models[i].fit(X_train, y_train, batch_size = 64, \n epochs = 50, \n validation_data = (X_test, y_test), \n verbose = 1,\n steps_per_epoch = X_train.shape[0] // 64,\n callbacks = [reduce_lr])\n print('Model {model_number}: Epochs = {epochs}: Training Accuracy = {training_accuracy}: Validation Accuracy = {validation_accuracy}'.format(model_number = i + 1,\n epochs = 50,\n training_accuracy = max(history[i].history['acc']),\n validation_accuracy = max(history[i].history['val_acc'])))","metadata":{"execution":{"iopub.status.busy":"2022-04-01T05:53:00.835348Z","iopub.execute_input":"2022-04-01T05:53:00.835611Z","iopub.status.idle":"2022-04-01T06:25:49.735411Z","shell.execute_reply.started":"2022-04-01T05:53:00.835581Z","shell.execute_reply":"2022-04-01T06:25:49.734658Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Performance of 10 CNNs

","metadata":{}},{"cell_type":"code","source":"model_nos = ['Model: ' + str(i + 1) for i in range(0, 10)]\nplt.figure(figsize = (15, 10))\nfor i in range(n):\n plt.plot(history[i].history['val_acc'])\nplt.title('CNN Accuracy', fontsize = 20)\nplt.legend(model_nos, loc = 'lower right')\nplt.xlabel('Epochs', fontsize = 15)\nplt.ylabel('Accuracy', fontsize = 15)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-01T06:25:54.965427Z","iopub.execute_input":"2022-04-01T06:25:54.966144Z","iopub.status.idle":"2022-04-01T06:25:55.241516Z","shell.execute_reply.started":"2022-04-01T06:25:54.966099Z","shell.execute_reply":"2022-04-01T06:25:55.240727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Ensemble 10 CNNs

","metadata":{}},{"cell_type":"code","source":"new_final_labels = np.zeros((test_df.shape[0], 10))\nfor i in range(n):\n new_final_labels = new_final_labels + new_models[i].predict(((test_df.values).reshape(-1, 28, 28, 1) / 255.0))\nnew_final_labels = np.argmax(new_final_labels, axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T06:26:07.014021Z","iopub.execute_input":"2022-04-01T06:26:07.014313Z","iopub.status.idle":"2022-04-01T06:26:30.343122Z","shell.execute_reply.started":"2022-04-01T06:26:07.01428Z","shell.execute_reply":"2022-04-01T06:26:30.342331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_multiple_images(test_df.values.reshape(-1, 28, 28, 1), 'Prediction', new_final_labels)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T06:26:39.605722Z","iopub.execute_input":"2022-04-01T06:26:39.606009Z","iopub.status.idle":"2022-04-01T06:26:40.76002Z","shell.execute_reply.started":"2022-04-01T06:26:39.605967Z","shell.execute_reply":"2022-04-01T06:26:40.759325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Creating the Submission File

","metadata":{}},{"cell_type":"code","source":"submission_df = pd.DataFrame()\nimage_id = [i for i in range(1, 28001)]\nsubmission_df['ImageId'] = image_id\nsubmission_df['Label'] = new_final_labels\nsubmission_df.to_csv('submission.csv', index = False)\nsubmission_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-04-01T06:26:45.137442Z","iopub.execute_input":"2022-04-01T06:26:45.137984Z","iopub.status.idle":"2022-04-01T06:26:45.216304Z","shell.execute_reply.started":"2022-04-01T06:26:45.137946Z","shell.execute_reply":"2022-04-01T06:26:45.215343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

References

\n\n> - https://www.kaggle.com/code/cdeotte/25-million-images-0-99757-mnist/notebook\n> - https://www.kaggle.com/code/anktplwl91/visualizing-what-your-convnet-learns/notebook\n> - https://www.kaggle.com/code/aishwarya2210/prediction-of-digits-from-handwritten-images\n> - https://www.kaggle.com/code/andradaolteanu/how-i-taught-myself-deep-learning-vanilla-nns\n> - https://www.kaggle.com/code/andradaolteanu/convolutional-neural-nets-cnns-explained\n> - https://www.kaggle.com/code/andradaolteanu/pytorch-rnns-and-lstms-explained-acc-0-99","metadata":{}},{"cell_type":"markdown","source":"
🚧 Work In Progress 🚧
","metadata":{}}]}