{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Comparing performance of ANNs and CNNs on Image Data

\n\nIn this notebook, we would train ANNs and CNNs on the MNIST Digit Classification Dataset. The data files contain gray-scale images of hand-drawn digits, from zero through nine.\n\n
\n\n
","metadata":{}},{"cell_type":"markdown","source":"Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive. The training data set, `(train.csv)`, has 785 columns. The first column, called `\"label\"`, is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n\nEach pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as $x = i * 28 + j$, where $i$ and $j$ are integers between 0 and 27, inclusive. Then pixelx is located on row $i$ and column $j$ of a 28 x 28 matrix, (indexing by zero)..\n\n## Artificial Neural Network (ANN)\nLet's look at what ANNs are.\n\nAn **Artificial Neural Network (ANN)** in the field of Artificial intelligence where it attempts to mimic the network of neurons makes up a human brain so that computers will have an option to understand things and make decisions in a human-like manner. The artificial neural network is designed by programming computers to behave simply like interconnected brain cells.\n\n**The architecture of an artificial neural network:**\n\nTo understand the concept of the architecture of an artificial neural network, we have to understand what a neural network consists of. In order to define a neural network that consists of a large number of artificial neurons, which are termed units arranged in a sequence of layers. Lets us look at various types of layers available in an artificial neural network.\n\nArtificial Neural Network primarily consists of three layers:\n\n
\n\n
\n\n1. **Input Layer**: As the name suggests, it accepts inputs in several different formats provided by the programmer.\n\n2. **Hidden Layer**: The hidden layer presents in-between input and output layers. It performs all the calculations to find hidden features and patterns.\n\n3. **Output Layer**: The input goes through a series of transformations using the hidden layer, which finally results in output that is conveyed using this layer.\n\nThe artificial neural network takes input and computes the weighted sum of the inputs and includes a bias. This computation is represented in the form of a transfer function.\n\n$$ \\huge z = \\Sigma_{i=1}^{n} w_{i} * x_{i} + b$$\n\nwhere, \n\n$x_{i}$ = Inputs of the dataset (aka `X_train` or `X_test`)\n\n$w_{i}$ = Weights of the various nodes in the neural network\n\n$b$ = Bias of the various nodes in the neural network\n\nArtificial Neural Network determines weighted total is passed as an input to an activation function to produce the output. Activation functions choose whether a node should fire or not. Only those who are fired make it to the output layer. \n\nThere are distinctive activation functions available that can be applied upon the sort of task we are performing. For regression, we can use `mean squared error` however for classification, we can use `sigmoid` (binary classification) or `softmax` (multiclass classification).","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport plotly.express as px\nimport matplotlib.pyplot as plt\n%matplotlib inline\nplt.rcParams['font.size'] = 12\nplt.rcParams['figure.figsize'] = (22,5)\nplt.rcParams['figure.dpi'] = 100\n\nimport tensorflow\nfrom tensorflow import keras\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, MaxPool2D, Conv2D, BatchNormalization\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom sklearn.metrics import confusion_matrix, accuracy_score, classification_report","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:18:54.450639Z","iopub.execute_input":"2023-10-31T07:18:54.451650Z","iopub.status.idle":"2023-10-31T07:18:57.956908Z","shell.execute_reply.started":"2023-10-31T07:18:54.451555Z","shell.execute_reply":"2023-10-31T07:18:57.955436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Downloading the dataset\n(X_train,y_train),(X_test,y_test) = keras.datasets.mnist.load_data()\n#defining the count of unique classes\nnum_classes = len(np.unique(y_train))\nprint(f'There are {num_classes} unique classes that we are supposed to be predicting')","metadata":{"id":"KRTH93VyMYF7","execution":{"iopub.status.busy":"2023-10-31T07:18:58.834840Z","iopub.execute_input":"2023-10-31T07:18:58.836019Z","iopub.status.idle":"2023-10-31T07:18:59.103111Z","shell.execute_reply.started":"2023-10-31T07:18:58.835982Z","shell.execute_reply":"2023-10-31T07:18:59.101973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('X train shape', X_train.shape)\nprint('y train shape', y_train.shape)\nprint('X test shape', X_test.shape)\nprint('y test shape', y_test.shape)","metadata":{"id":"m0hHUKoNMpsZ","outputId":"c48fd0c4-3edb-4cdc-8883-d6de097fbe98","execution":{"iopub.status.busy":"2023-10-31T07:19:03.584374Z","iopub.execute_input":"2023-10-31T07:19:03.585474Z","iopub.status.idle":"2023-10-31T07:19:03.591508Z","shell.execute_reply.started":"2023-10-31T07:19:03.585431Z","shell.execute_reply":"2023-10-31T07:19:03.590413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`y_train` and `y_test` basically contain the classes that are to be predicted with the help of the images provided","metadata":{}},{"cell_type":"code","source":"y_train","metadata":{"id":"skzwnglnMzl6","outputId":"ed1b62cf-97e9-44dd-fa63-2643180954a3","execution":{"iopub.status.busy":"2023-10-31T07:19:06.422098Z","iopub.execute_input":"2023-10-31T07:19:06.423122Z","iopub.status.idle":"2023-10-31T07:19:06.432178Z","shell.execute_reply.started":"2023-10-31T07:19:06.423081Z","shell.execute_reply":"2023-10-31T07:19:06.430962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:19:06.851221Z","iopub.execute_input":"2023-10-31T07:19:06.852006Z","iopub.status.idle":"2023-10-31T07:19:06.858208Z","shell.execute_reply.started":"2023-10-31T07:19:06.851974Z","shell.execute_reply":"2023-10-31T07:19:06.857202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's take a look at an image from the training dataset","metadata":{}},{"cell_type":"code","source":"_, axes = plt.subplots(nrows=1, ncols=4, figsize=(20,15))\nfor ax, image, label in zip(axes, X_train, y_train):\n ax.set_axis_off()\n image = image.reshape(28, 28)\n ax.imshow(image, cmap=plt.cm.gray_r, interpolation=\"nearest\")\n ax.set_title(f\"Training Dataset image Label: {label}\") ","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:19:07.828320Z","iopub.execute_input":"2023-10-31T07:19:07.829216Z","iopub.status.idle":"2023-10-31T07:19:08.366728Z","shell.execute_reply.started":"2023-10-31T07:19:07.829177Z","shell.execute_reply":"2023-10-31T07:19:08.365749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x=y_train)\nplt.gca().spines['top'].set_visible(False)\nplt.gca().spines['right'].set_visible(False)\nplt.title('Distribution of various target classes')\nplt.xlabel('Classes')\nplt.ylabel('Count')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:19:09.273297Z","iopub.execute_input":"2023-10-31T07:19:09.274035Z","iopub.status.idle":"2023-10-31T07:19:09.561721Z","shell.execute_reply.started":"2023-10-31T07:19:09.274000Z","shell.execute_reply":"2023-10-31T07:19:09.560775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing\n\nOne of the ways to account for the imbalance in our data is to use the `class_weight` parameter. Our model expects a dictionary like this where the keys are the classes and the values are the weight that the model should give to that particular class.\n\n```python\nclass_weights = {0: 1.13827, 1: 1.0, 2: 1.13158, 3: 1.09965, 4: 1.15405, 5: 1.24368, 6: 1.13923, 7: 1.07613, 8: 1.15228, 9: 1.13329}\n```\nSo this is exactly what we are going to do next.","metadata":{}},{"cell_type":"code","source":"label, count = np.unique(y_train, return_counts=True)\n# compute the class weights\ncounts = max(count) / count\nclass_weights = dict(zip(label, counts))\nprint('=== CLASS WEIGHTS ===')\nprint(class_weights)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:19:10.373382Z","iopub.execute_input":"2023-10-31T07:19:10.374196Z","iopub.status.idle":"2023-10-31T07:19:10.382757Z","shell.execute_reply.started":"2023-10-31T07:19:10.374151Z","shell.execute_reply":"2023-10-31T07:19:10.381605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" We are dividing the variables by 255 in order to scale them down within a certain range which will help the model converge faster and hence reduce the training time.","metadata":{}},{"cell_type":"code","source":"#normalizing the values (scaling the values between 0 and 1)\nX_train = X_train/255.0\nX_test = X_test/255.0","metadata":{"id":"dJmO2WLlRRT1","execution":{"iopub.status.busy":"2023-10-31T07:19:21.007792Z","iopub.execute_input":"2023-10-31T07:19:21.008211Z","iopub.status.idle":"2023-10-31T07:19:21.155124Z","shell.execute_reply.started":"2023-10-31T07:19:21.008173Z","shell.execute_reply":"2023-10-31T07:19:21.153923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The max value in the dataset after normalizing is {np.max(X_train)} and the min value is {np.min(X_train)}.')","metadata":{"id":"i1WA9FioRdtB","outputId":"5743362a-3ff6-4e8c-ae87-9849834e9bd3","execution":{"iopub.status.busy":"2023-10-31T07:19:21.416735Z","iopub.execute_input":"2023-10-31T07:19:21.417123Z","iopub.status.idle":"2023-10-31T07:19:21.485504Z","shell.execute_reply.started":"2023-10-31T07:19:21.417089Z","shell.execute_reply":"2023-10-31T07:19:21.484345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building an ANN\n\nWe are using the **ELU** activation function here.\n\nELU, also know as **Exponential Linear Unit** is an activation function which is somewhat similar to the ReLU with some differences. Similar to other non-saturating activation functions, ELU does not suffer from the problem of vanishing gradients and exploding gradients. And similar to Leaky-ReLU and PReLU, and unlike ReLU, ELU does not suffer from the problems of dying neurons. \n\nIt has proven to be better than ReLU and its variants like Leaky-ReLU(LReLU) and Parameterized-ReLU(PReLU). Using ELU leads to a lower training times but higher accuracies in Neural Networks as compared to ReLU, and its variants.ELU activation function is continuous and differentiable at all points. Mathematically, the ELU activation function can be written as-\n\n$$\\begin{equation*}\ny = \\text{ELU}(x) = \\begin{cases}\n\\exp(x) - 1 & \\text{if } x < 0 \\\\\nx & \\text{if } x \\ge 0\n\\end{cases}\n\\end{equation*}$$","metadata":{}},{"cell_type":"markdown","source":"Keras also helps setting the initial weights and biases which we can see below. These will later on be updated using `Backpropogation` and [Gradient Descent](https://www.kaggle.com/code/bhatnagardaksh/gradient-descent-from-scratch) and will keep on updating until convergence happens which means the loss is minimum but at the same time its also critical to understand if the data will overfit or not. Below is how the weights and biases are updated and this is what in Deep Learning is termed as `Backpropogation` \n\n![](https://miro.medium.com/max/1838/1*diQ7YbcLAZvqTMSOU8TYNw.png)","metadata":{}},{"cell_type":"markdown","source":"**Weight initialization** is an important design choice when developing deep learning neural network models. Historically, weight initialization involved using small random numbers, although over the last decade, more specific heuristics have been developed that use information, such as the type of activation function that is being used and the number of inputs to the node.\n\n**Why Not Set Weights to Zero?**\n\n- We can use the same set of weights each time we train the network; for example, you could use the values of 0.0 for all weights. In this case, the equations of the learning algorithm would fail to make any changes to the network weights, and the model will be stuck as it would face the problem of **vanishing gradient descent** . \n\n- It is important to note that the bias weight in each neuron is set to zero by default, not a small random value.\n\n- Specifically, nodes that are side-by-side in a hidden layer connected to the same inputs must have different weights for the learning algorithm to update the weights.\n\n**Initialization Methods**\n\n- The initialization of the weights of neural networks is a whole field of study as the careful initialization of the network can speed up the learning process.Modern deep learning libraries, such as Keras, offer a host of network initialization methods, all are variations of initializing the weights with small random numbers.\n\n- The methods are available in Keras at the moment for all network types. You can have a look at the them in the Keras documentation [here](https://keras.io/api/layers/initializers/)\n\n- Each layer has its own default value for initializing the weights. For most of the layers, such as Dense, convolution and RNN layers, the default `kernel initializer` is `glorot_uniform` and the default bias intializer is `zeros`.","metadata":{}},{"cell_type":"markdown","source":"**Loss Functions**\n\nWe will be using Sparse categorical crossentropy since we are dealing with a multiclass classification.\n\n\n| **Loss Function** | **Binary Classification** | **Multiclass Classification** |\n|----------------------------|--------------------------------------------------------------|--------------------------------------------------------------|\n| Binary Crossentropy | - Used for binary classification, where there are only two classes (e.g., 0 and 1). - Measures the cross-entropy between predicted probabilities and true binary labels. - Encourages predicted probabilities close to one for the true class and close to zero for the other class. | Not typically used in multiclass classification, as it's designed for binary scenarios. |\n| Categorical Crossentropy | Not commonly used in binary classification. | - Used for multiclass classification, where there are more than two classes. - Measures the cross-entropy between predicted probabilities and true one-hot encoded labels. - Encourages predicted probabilities close to one for the true class and close to zero for all other classes. |\n| Sparse Categorical Crossentropy | Not commonly used in binary classification. | - Used for multiclass classification with integer class labels. - Internally converts integer labels to one-hot encoded format. - More memory-efficient with a large number of classes. - Encourages predicted probabilities close to one for the true class. |\n| Hinge Loss (SVM) | - Used for binary classification, especially in Support Vector Machines (SVMs). - Encourages predicted scores to be greater than a margin for the true class and less than the margin for the other class. | Not commonly used in multiclass classification, as it's designed for binary scenarios. |\n\n- In binary classification, you typically use loss functions like Binary Crossentropy and Hinge Loss, while in multiclass classification, Categorical Crossentropy or Sparse Categorical Crossentropy are more commonly used, depending on the format of class labels.","metadata":{}},{"cell_type":"markdown","source":"### Setting Optimizations for training\n\nModels often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. The learning rate is reduced after every mentioned number of `decay_steps`. We use `ExponentialDecay` while compiling the model to apply the concept of **Learning Rate Scheduling**\n\nThe formula for Exponential Decay is \n\n$$f(x) = ab^{x}$$\n\nwhich when converted to our use case becomes\n\n$$\\begin{equation*}\n\\text{learning_rate} = lr_{0} \\times \\text{decay_rate} ^ {\\frac{\\text{step}}{\\text{decay_steps}}}\n\\end{equation*}$$\n\nThe idea is to get to the minima in a stable manner and not shoot out of the path haphazardly making the training unstable.","metadata":{}},{"cell_type":"code","source":"def get_model():\n model = Sequential()\n model.add(Flatten(input_shape=(28,28))) #Flattening the data\n\n model.add(Dense(128,activation='relu')) #input layer\n model.add(BatchNormalization())\n\n model.add(Dense(64,activation='elu')) #Hidden Layer\n model.add(BatchNormalization())\n\n model.add(Dense(32,activation='elu')) #Hidden Layer\n model.add(BatchNormalization())\n model.add(Dropout(0.25))\n\n model.add(Dense(16,activation='elu')) #Hidden Layer\n model.add(BatchNormalization())\n\n model.add(Dense(num_classes,activation='softmax')) #Output Layer\n \n return model\n\ninitial_learning_rate = 0.001\n\nlr_schedule = tensorflow.keras.optimizers.schedules.ExponentialDecay(\n initial_learning_rate,\n decay_steps=10000,\n decay_rate=0.96,\n staircase=True)\n\noptimizer = keras.optimizers.Adam(learning_rate=lr_schedule)\n\ncallback = EarlyStopping(\n monitor=\"loss\",\n min_delta=0.001,\n patience=5,\n verbose=1,\n mode=\"auto\",\n baseline=None,\n restore_best_weights=False\n)\n\nBATCH_SIZE = 200","metadata":{"id":"0yC51WgjSsLV","execution":{"iopub.status.busy":"2023-10-31T07:19:30.523832Z","iopub.execute_input":"2023-10-31T07:19:30.524255Z","iopub.status.idle":"2023-10-31T07:19:30.535874Z","shell.execute_reply.started":"2023-10-31T07:19:30.524218Z","shell.execute_reply":"2023-10-31T07:19:30.534831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = get_model()\n#compiling the model\nmodel.compile(loss='sparse_categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:19:32.281370Z","iopub.execute_input":"2023-10-31T07:19:32.282265Z","iopub.status.idle":"2023-10-31T07:19:33.542112Z","shell.execute_reply.started":"2023-10-31T07:19:32.282226Z","shell.execute_reply":"2023-10-31T07:19:33.541132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training ANN\n\nWe will be using the `BATCH_SIZE` to specify the size of the batch we want the training to be done on. We are passing the `validation_data` also (both `X_test` and `y_test`) and mentioning the validation batch size as well so that the validation is done on the same amount of images as the training. To induce the early stopping, we are passing the callback into `callbacks` argument as a list. The class weights are being passed in the `class_weight` argument and we are training the model for 100 epochs.","metadata":{}},{"cell_type":"code","source":"history = model.fit(x=X_train,y=y_train,epochs=100,batch_size=BATCH_SIZE,validation_data = (X_test,y_test), \n validation_batch_size=BATCH_SIZE, callbacks=[callback], class_weight=class_weights)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:19:48.543204Z","iopub.execute_input":"2023-10-31T07:19:48.543601Z","iopub.status.idle":"2023-10-31T07:20:25.152275Z","shell.execute_reply.started":"2023-10-31T07:19:48.543572Z","shell.execute_reply":"2023-10-31T07:20:25.151220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plotting the ANN Plots","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history['val_accuracy'], label='Validation Accuracy', marker='o')\nplt.plot(history.history['val_loss'], 'tab:orange', linestyle='dashed', label='Validation Loss')\nplt.gca().spines['top'].set_visible(False)\nplt.gca().spines['right'].set_visible(False)\nplt.title(\"Model Performance Plot, Validation\", fontsize=20)\nplt.xlabel('Epoch')\nplt.ylabel('Metrics')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:20:48.044630Z","iopub.execute_input":"2023-10-31T07:20:48.045394Z","iopub.status.idle":"2023-10-31T07:20:48.329383Z","shell.execute_reply.started":"2023-10-31T07:20:48.045358Z","shell.execute_reply":"2023-10-31T07:20:48.328385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Classification Report","metadata":{}},{"cell_type":"code","source":"y_prob = model.predict(X_test)\ny_pred = y_prob.argmax(axis=1)\nprint(classification_report(y_test, y_pred))","metadata":{"id":"CazuknkLW_LY","execution":{"iopub.status.busy":"2023-10-31T07:20:59.519319Z","iopub.execute_input":"2023-10-31T07:20:59.519735Z","iopub.status.idle":"2023-10-31T07:21:00.088452Z","shell.execute_reply.started":"2023-10-31T07:20:59.519698Z","shell.execute_reply":"2023-10-31T07:21:00.087402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ANN Confusion Matrix","metadata":{}},{"cell_type":"code","source":"confusionmatrix = np.around(confusion_matrix(y_test, y_pred, normalize='true'),4)\n\nfig = px.imshow(confusionmatrix, \n text_auto=True, \n color_continuous_scale=px.colors.sequential.Viridis, \n aspect=\"auto\", \n labels=dict(x=\"Predicted Label\", y=\"Correct Label\"))\n\nfig.update_xaxes(side=\"top\")\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:03.501424Z","iopub.execute_input":"2023-10-31T07:21:03.501825Z","iopub.status.idle":"2023-10-31T07:21:03.783069Z","shell.execute_reply.started":"2023-10-31T07:21:03.501788Z","shell.execute_reply":"2023-10-31T07:21:03.782063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Getting the Image labels that is misqualified the most\ndef get_value(item):\n return item[1]\ndata = y_test[y_test != y_pred]\ncount = {int(value): list(data).count(value) for value in set(data)}\nsorted_count = dict(sorted(count.items(), key=get_value, reverse=True))\nmaxvalue = max(count, key=count.get)\n#Setting the color of the most misqualified labels to 'indianred'\ncolors = {}\nfor i in range(len(np.unique(y_train))):\n colors[i] = 'lightgray'\n colors[maxvalue] = 'red'","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:07.957096Z","iopub.execute_input":"2023-10-31T07:21:07.957522Z","iopub.status.idle":"2023-10-31T07:21:07.966847Z","shell.execute_reply.started":"2023-10-31T07:21:07.957485Z","shell.execute_reply":"2023-10-31T07:21:07.965905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ANN Misclassified Plot","metadata":{}},{"cell_type":"code","source":"ax = sns.countplot(data,palette=colors, dodge=False, order=sorted_count)\nplt.xlabel('Labels')\nplt.title('ANN Misclassification Plot', fontsize=20)\n\nax.xaxis.set_ticks(np.unique(y_test))\nfor bar in ax.patches:\n ax.annotate(format(bar.get_height(), '.0f'),(bar.get_x() + bar.get_width() / 2,bar.get_height()), \n ha='center', va='center',size=15, xytext=(0, 8),textcoords='offset points')\nplt.gca().spines['top'].set_visible(False)\nplt.gca().spines['right'].set_visible(False)\nplt.legend(['Most Misclassified Class'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:11.739206Z","iopub.execute_input":"2023-10-31T07:21:11.739637Z","iopub.status.idle":"2023-10-31T07:21:12.087635Z","shell.execute_reply.started":"2023-10-31T07:21:11.739606Z","shell.execute_reply":"2023-10-31T07:21:12.086521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Cases where misclassification has been done\nerror_pct = np.around((len(data)/len(X_test)*100),4)\nprint(f'ANN Misclassified cases : {len(data)}/{len(X_test)}, Error Percentage : {error_pct}%')","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:15.060892Z","iopub.execute_input":"2023-10-31T07:21:15.061307Z","iopub.status.idle":"2023-10-31T07:21:15.067368Z","shell.execute_reply.started":"2023-10-31T07:21:15.061270Z","shell.execute_reply":"2023-10-31T07:21:15.066368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convolution Neural Network (CNN)\n\nNow that we are done training and improving the ANN. We will now use the CNN (Convolution Neural Network) for training and prediction purposes. \n\n### What are convolutions?\nA convolution is a filter that passes over an image, processes it, and extracts the important features.\n\nLet's say you have an image of a person wearing a sneaker. How would you detect that a sneaker is present in the image? In order for your program to \"see\" the image as a sneaker, you'll have to extract the important features, and blur the inessential features. This is called feature mapping.\n\nThe feature mapping process is theoretically simple. You'll scan every pixel in the image and then look at its neighboring pixels. You multiply the values of those pixels by the equivalent weights in a filter.\n\n*For example:*\n\n
\n\n
\n\nIn this case, a $3x3$ convolution matrix, or image kernel, is specified.\n\nThe current pixel value is 192. You can calculate the value of the new pixel by looking at the neighbor values, multiplying them by the values specified in the filter, and making the new pixel value the final amount.\n\n### Understanding Pooling\nSimilar to convolutions, pooling greatly helps with detecting features. Pooling layers reduce the overall amount of information in an image while maintaining the features that are detected as present.\n\n
\n\n
\n\nThere are a number of different types of pooling, but you'll use one called Maximum (Max) Pooling.\n\nIterate over the image and, at each point, consider the pixel and its immediate neighbors to the right, beneath, and right-beneath. Take the largest of those (hence max pooling) and load it into the new image. Thus, the new image will be one-fourth the size of the old.\n\n\n**Why CNNs are better than ANNs in case of Image Classification**\n1. ANNs are computationaly expensive because as you add more layers or neurons the weights and biases are going to increase which means there will be much more calculations of gradient descent happening.\n2. Loss Of Information or features in a way that if we try to reduce the layers or neurons the ANN model may not be able to pick up the features of the image as correctly as possible. If we keep on adding layers or neuron, we are gonna be facing the challenge of overfitting which is something we definitely don't want.\n\n\nLet's now train the CNN on the same data. We have reshaped the data and then flattened it out before giving it to the output layer which basically will classify the digit.","metadata":{}},{"cell_type":"code","source":"print('X train shape before reshaping :', X_train.shape)\nprint('X test shape before reshaping :', X_test.shape)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:28.445234Z","iopub.execute_input":"2023-10-31T07:21:28.446197Z","iopub.status.idle":"2023-10-31T07:21:28.451661Z","shell.execute_reply.started":"2023-10-31T07:21:28.446127Z","shell.execute_reply":"2023-10-31T07:21:28.450632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X_train.reshape(60000,28,28,1)\nX_test = X_test.reshape(10000,28,28,1)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:28.911689Z","iopub.execute_input":"2023-10-31T07:21:28.912086Z","iopub.status.idle":"2023-10-31T07:21:28.917170Z","shell.execute_reply.started":"2023-10-31T07:21:28.912051Z","shell.execute_reply":"2023-10-31T07:21:28.916191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our training and test data has been reshaped and is ready to be fed to the model. Let's first take a look if the reshaping has been done correctly or not.","metadata":{}},{"cell_type":"code","source":"print('X train shape after reshaping :', X_train.shape)\nprint('X test shape after reshaping :', X_test.shape)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:30.650134Z","iopub.execute_input":"2023-10-31T07:21:30.650547Z","iopub.status.idle":"2023-10-31T07:21:30.656531Z","shell.execute_reply.started":"2023-10-31T07:21:30.650510Z","shell.execute_reply":"2023-10-31T07:21:30.655508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CNN Model Architecture","metadata":{}},{"cell_type":"code","source":"model = Sequential()\n\n#1st Layer\nmodel.add(Conv2D(filters=64, kernel_size=(3, 3), activation='elu', padding = 'Same', input_shape=(28,28,1))),\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=2)),\nmodel.add(BatchNormalization())\n\n#2nd Layer\nmodel.add(Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same', \n activation ='elu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.25))\n\n#3rd Layer\nmodel.add(Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same', \n activation ='elu'))\nmodel.add(BatchNormalization())\n\n#Output Layer\nmodel.add(Flatten())\nmodel.add(Dense(num_classes, activation = \"softmax\"))\n\n#Model Summary\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:31.967880Z","iopub.execute_input":"2023-10-31T07:21:31.968406Z","iopub.status.idle":"2023-10-31T07:21:32.060205Z","shell.execute_reply.started":"2023-10-31T07:21:31.968367Z","shell.execute_reply":"2023-10-31T07:21:32.059191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are using the `class_weight` argument to help the imbalanced data. The argument was created before training the ANN and as per the results, its helping a lot make predictions better.","metadata":{}},{"cell_type":"code","source":"tensorflow.keras.utils.plot_model(\n model,\n show_shapes=True,\n show_dtype=False,\n show_layer_names=True,\n rankdir='TB',\n expand_nested=False,\n dpi=96,\n layer_range=None\n)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:37.987198Z","iopub.execute_input":"2023-10-31T07:21:37.987590Z","iopub.status.idle":"2023-10-31T07:21:38.235426Z","shell.execute_reply.started":"2023-10-31T07:21:37.987559Z","shell.execute_reply":"2023-10-31T07:21:38.234221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"initial_learning_rate = 0.001\n\nlr_schedule = tensorflow.keras.optimizers.schedules.ExponentialDecay(\n initial_learning_rate,\n decay_steps=10000,\n decay_rate=0.96,\n staircase=True)\n\noptimizer = keras.optimizers.Adam(learning_rate=lr_schedule)\n\ncallback = EarlyStopping(\n monitor=\"val_loss\",\n min_delta=0.001,\n patience=3,\n verbose=1,\n mode=\"auto\",\n baseline=None,\n restore_best_weights=False\n)\n\nmodel.compile(loss='sparse_categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:38.989294Z","iopub.execute_input":"2023-10-31T07:21:38.990233Z","iopub.status.idle":"2023-10-31T07:21:39.006922Z","shell.execute_reply.started":"2023-10-31T07:21:38.990190Z","shell.execute_reply":"2023-10-31T07:21:39.005800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training CNN\n\nWe will be using the `BATCH_SIZE` to specify the size of the batch we want the training to be done on. We are passing the `validation_data` also (both `X_test` and `y_test`) and mentioning the validation batch size as well so that the validation is done on the same amount of images as the training. To induce the early stopping, we are passing the callback into `callbacks` argument as a list. The class weights are being passed in the `class_weight` argument and we are training the model for 100 epochs.","metadata":{}},{"cell_type":"code","source":"history = model.fit(x=X_train,y=y_train,epochs=100,batch_size=BATCH_SIZE,validation_data = (X_test,y_test), \n validation_batch_size=BATCH_SIZE, callbacks=[callback], class_weight=class_weights)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:43.654933Z","iopub.execute_input":"2023-10-31T07:21:43.655641Z","iopub.status.idle":"2023-10-31T07:21:59.469632Z","shell.execute_reply.started":"2023-10-31T07:21:43.655603Z","shell.execute_reply":"2023-10-31T07:21:59.468588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.history['val_accuracy'], label='Validation Accuracy', marker='o')\nplt.plot(history.history['val_loss'], 'tab:orange', linestyle='dashed', label='Validation Loss')\nplt.gca().spines['top'].set_visible(False)\nplt.gca().spines['right'].set_visible(False)\nplt.title(\"CNN Performance Plot\", fontsize=20)\nplt.xlabel('Epoch')\nplt.ylabel('CNN Metrics, Validation')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:59.471883Z","iopub.execute_input":"2023-10-31T07:21:59.472377Z","iopub.status.idle":"2023-10-31T07:21:59.778244Z","shell.execute_reply.started":"2023-10-31T07:21:59.472335Z","shell.execute_reply":"2023-10-31T07:21:59.777247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### CNN Classification report","metadata":{}},{"cell_type":"code","source":"y_prob = model.predict(X_test)\ny_pred = y_prob.argmax(axis=1)\nprint(classification_report(y_test, y_pred))","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:21:59.779574Z","iopub.execute_input":"2023-10-31T07:21:59.779888Z","iopub.status.idle":"2023-10-31T07:22:00.358911Z","shell.execute_reply.started":"2023-10-31T07:21:59.779859Z","shell.execute_reply":"2023-10-31T07:22:00.357896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### CNN Misclassified Plot","metadata":{}},{"cell_type":"code","source":"data = y_test[y_test != y_pred]\ncount = {int(value): list(data).count(value) for value in set(data)}\nsorted_count = dict(sorted(count.items(), key=get_value, reverse=True))\nmaxvalue = max(count, key=count.get)\n#Setting the color of the most misqualified labels to 'indianred'\ncolors = {}\nfor i in range(len(np.unique(y_train))):\n colors[i] = 'lightgray'\n colors[maxvalue] = 'red'","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:22:06.706444Z","iopub.execute_input":"2023-10-31T07:22:06.706887Z","iopub.status.idle":"2023-10-31T07:22:06.716340Z","shell.execute_reply.started":"2023-10-31T07:22:06.706849Z","shell.execute_reply":"2023-10-31T07:22:06.715423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.countplot(y_test[y_test != y_pred],palette=colors, dodge=False, order=sorted_count)\nplt.xlabel('Labels')\nplt.title('CNN Misclassification Plot', fontsize=20)\n\nax.xaxis.set_ticks(np.unique(y_test))\nfor bar in ax.patches:\n ax.annotate(format(bar.get_height(), '.0f'),(bar.get_x() + bar.get_width() / 2,bar.get_height()), \n ha='center', va='center',size=15, xytext=(0, 8),textcoords='offset points')\nplt.gca().spines['top'].set_visible(False)\nplt.gca().spines['right'].set_visible(False)\nplt.legend(['Most Misclassified Class'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:22:07.928875Z","iopub.execute_input":"2023-10-31T07:22:07.929809Z","iopub.status.idle":"2023-10-31T07:22:08.203326Z","shell.execute_reply.started":"2023-10-31T07:22:07.929764Z","shell.execute_reply":"2023-10-31T07:22:08.202306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Confusion Matrix","metadata":{}},{"cell_type":"code","source":"confusionmatrix = np.around(confusion_matrix(y_test, y_pred, normalize='true'),4)\nfig = px.imshow(confusionmatrix, text_auto=True, aspect=\"auto\", \n color_continuous_scale=px.colors.sequential.Teal,\n labels=dict(x=\"Predicted Label\", y=\"Correct Label\", color='Value'))\n\nfig.update_xaxes(side=\"top\")\n\nfig.update_xaxes(ticktext=np.unique(y_test), tickvals = np.unique(y_test), ticklabelstep=1)\nfig.update_yaxes(ticktext=np.unique(y_test), tickvals = np.unique(y_test), ticklabelstep=1)\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:22:16.753079Z","iopub.execute_input":"2023-10-31T07:22:16.753498Z","iopub.status.idle":"2023-10-31T07:22:16.829026Z","shell.execute_reply.started":"2023-10-31T07:22:16.753462Z","shell.execute_reply":"2023-10-31T07:22:16.827950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a sample DataFrame\ndf = pd.DataFrame({'y_test': y_test, 'y_pred': y_pred})\n\n# Calculate the number of misclassified cases\nmismatched = df[df['y_test'] != df['y_pred']]\nmisclassified_cases = len(mismatched)\n\n# Calculate the total number of cases\ntotal_cases = len(df)\n\n# Calculate the error rate\nerror_rate = misclassified_cases / total_cases * 100\n\n# Print the results\nprint(f'Misclassification by CNN : {misclassified_cases} / {total_cases}, Error rate : {np.around(error_rate, 4)}%.')","metadata":{"execution":{"iopub.status.busy":"2023-10-31T07:22:17.865626Z","iopub.execute_input":"2023-10-31T07:22:17.866023Z","iopub.status.idle":"2023-10-31T07:22:17.875630Z","shell.execute_reply.started":"2023-10-31T07:22:17.865989Z","shell.execute_reply":"2023-10-31T07:22:17.874619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summary\n\n\n- In this Jupyter notebook, we have explored the topic of artificial neural networks (ANNs). We have compared how ANN and CNN can be used to train on the image data and found out that CNNs perform better and are efficient for the image processing.\n\n- We saw how to preprocess the data, set optimizations for the training of the model, setup the architecture of the model for both the ANNs and CNNs. We briefly discussed the various loss functions, weight initialization techniques, compared the performance of both the models using classification matrix, training loss and accuracy.\n\n- CNNs are a powerful tool that can be used for a variety of tasks. They have been used to achieve success in a wide range of fields, including computer vision, natural language processing, and machine translation.\n\n- We can clearly see that CNN is performing better on Image data as compared to the ANNs and the CNN is only working with 44,000 weights and the ANN is working with a whopping 110,000 weights. Imagine the number when it translates to images with better resolution and there is more training data. \n\n- Even when the ANN has more weights it still is making a higher error as compared to that of the CNN which outrightly states that CNN works better on Image Data. Two major benefits of CNN is that its computationally cheaper than ANN and the loss of spatial information is least to negligible.\n\nHence, we can conclude that ANNs are not a better choice when it comes to processing the image data.\n\nI HOPE THIS NOTEBOOK HELPED YOU IN SOME WAY OR THE OTHER. THANKS FOR GOING THROUGH THE NOTEBOOK!!!","metadata":{}},{"cell_type":"markdown","source":"## Further Reading\n\n1. https://colab.research.google.com/drive/1dJZZdhngq4eN83sQCupyh2QbyzrsBB-e?usp=sharing\n2. https://developers.google.com/codelabs/tensorflow-3-convolutions#5\n3. https://www.kaggle.com/code/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n4. https://machinelearningmastery.com/weight-initialization-for-deep-learning-neural-networks/\n5. https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler","metadata":{}}]}