{"metadata":{"colab":{"provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3649,"databundleVersionId":46718,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What is Cifar-10 Image Classifiction\n\nThe `CIFAR-10` dataset consists of `60000` `32x32` color images in `10` classes, with `6000` images per class. There are `50000` training images and `10000` test images.\n\n## Upon checking out the image data, we can define the problem as follows :\n","metadata":{"id":"YE5du-CaPpV7"}},{"cell_type":"markdown","source":"# Problem Definition:\n\nGiven an image, can we predict the correct class of this image?\n\nThe images are very small (`32x32`) and by visualizing them we notice how difficult it is to distinguish them even for a human.\n\nIn this notebook we are going to build a CNN model that can classify images of various objects. We have `10` class of images:\n1. Airplane\n2. Automobile\n3. Bird\n4. Cat\n5. Deer\n6. Dog\n7. Frog\n8. Horse\n9. Ship\n10. Truck\n\n# Evaluation:\n\nWe have `10` classes, so if we pick a image and we randomly gues it class, we have `1/10` probability to be true.\n\nSo our random guessing is 10% accuracy","metadata":{"id":"Ld-khf0XQheE"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport tensorflow as tf\nfrom tensorflow.keras.datasets import cifar10\nfrom tensorflow.keras.utils import to_categorical\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, Flatten, Dropout, BatchNormalization\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Activation\n\nfrom sklearn.metrics import ConfusionMatrixDisplay\nfrom sklearn.metrics import classification_report, confusion_matrix\n\nimport matplotlib.pyplot as plt\nimport numpy as np\n","metadata":{"id":"CFMuAnroOn-k"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading the data\n\nOkay, let's load the data and check what we are dealing with :","metadata":{"id":"bDWN0Cc4QuhQ"}},{"cell_type":"code","source":"(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n\nprint(f\"X_train shape: {X_train.shape}\")\nprint(f\"y_train shape: {y_train.shape}\")\nprint(f\"X_test shape: {X_test.shape}\")\nprint(f\"y_test shape: {y_test.shape}\")","metadata":{"id":"cvJlbBczO0c2","outputId":"fcc9d5ad-0e84-40b9-b156-5b9a952a13d1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing a sample\n\nLet's start by taking a sample and seeing how the images actually look. I found this nifty code sample which does that using grids as shown below.","metadata":{"id":"HvhmVflwRM0i"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Assuming x_train, y_train are loaded and contain CIFAR-10 data\nclass_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n               'dog', 'frog', 'horse', 'ship', 'truck']\n\nfig, axes = plt.subplots(3, 5, figsize=(12, 8))\naxes = axes.ravel()\n\nfor i in range(15):\n    index = np.random.randint(0, len(X_train))\n    axes[i].imshow(X_train[index])\n    axes[i].set_title(class_names[y_train[index][0]])\n    axes[i].axis('off')\n\nplt.subplots_adjust(hspace=0.5)\n","metadata":{"id":"k28ZZ3a3O1_M","outputId":"2c634d92-0a34-40fb-cd21-74fd6e47d05c"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Class distribution\n\nNext, let's take a look at if the classes are balanced or imbalanced in train and test","metadata":{"id":"FxSnT442R_2w"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\n# Assuming y_train and y_test are already defined\nclasses_name = ['Airplane', 'Automobile', 'Bird', 'Cat', 'Deer',\n                'Dog', 'Frog', 'Horse', 'Ship', 'Truck']\n\n# Calculate class distributions for both training and testing sets\ntrain_classes, train_counts = np.unique(y_train, return_counts=True)\ntest_classes, test_counts = np.unique(y_test, return_counts=True)\n\n# Create a figure and a set of subplots\nfig, axs = plt.subplots(1, 2, figsize=(12, 6), sharey=True)\n\n# Plot training set distribution\naxs[0].barh(classes_name, train_counts, color='skyblue')\naxs[0].set_title('Class Distribution in Training Set')\naxs[0].set_xlabel('Counts')\naxs[0].set_ylabel('Classes')\n\n# Plot testing set distribution\naxs[1].barh(classes_name, test_counts, color='salmon')\naxs[1].set_title('Class Distribution in Testing Set')\naxs[1].set_xlabel('Counts')\n\n# Add a legend to clarify the colors represent training and testing sets\nfig.legend(['Training Set', 'Testing Set'], loc='upper right')\n\nplt.tight_layout()\nplt.show()\n","metadata":{"id":"3T5PoMKuSuTi","outputId":"7762d9eb-d797-4261-dc43-a7ca2ec43364"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see the classes are equally distributed amongst in the train and test sets both. Thus, we don't need to perform class balancing.","metadata":{"id":"WsZtyBy3j2UD"}},{"cell_type":"markdown","source":"# Data Preprocessing\n\nOkay, let's begin with data preprocessing.\n\nThis preprocessing involves two main steps: normalization and one-hot encoding of the target variable.","metadata":{"id":"L8HC49kskYrS"}},{"cell_type":"markdown","source":"## Normalization\n\nSo I read about why we do Normalization. It involves scaling the pixel values of images to a range of 0 to 1. This process is essential because it helps in reducing the computational complexity by ensuring that the model trains on a standardized scale, leading to a more stable and faster convergence.\n\n","metadata":{"id":"gY_WH-7qloCO"}},{"cell_type":"code","source":"# Normalization\nX_train = X_train / 255.0\nX_test = X_test / 255.0\n","metadata":{"id":"E0skUtLdlnl5"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## One-Hot Encoding\nSince our task is a multi-class classification, we convert the categorical labels into a one-hot encoded format. This encoding transforms the categorical labels into a binary matrix representation, which is more suitable for training neural networks on classification tasks.\n\n","metadata":{"id":"XdF-PtYrl1cV"}},{"cell_type":"code","source":"from tensorflow.keras.utils import to_categorical\n\n# One-Hot Encoding\ny_cat_train = to_categorical(y_train, 10)\ny_cat_test = to_categorical(y_test, 10)\n","metadata":{"id":"QwnDU3lbSwdz"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Additionally, I read about data augmentation as an optional strategy to further enhance our model's ability to generalize from our training data.\n\nLet's try that out as well.","metadata":{"id":"9ZR8mD-mlgY7"}},{"cell_type":"markdown","source":"## Data Augmentation\n\n### Why do data augmentation ?\n\nTo improve the model's generalization ability, we implement data augmentation. This technique artificially expands the training dataset by applying random transformations (e.g., rotations, shifts, flips) to the training images. It helps the model learn from a more diversified set of data, reducing the risk of overfitting.","metadata":{"id":"jMItyQKLmG82"}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n\ntrain_datagen = ImageDataGenerator(\n    rotation_range=15,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=True,\n    zoom_range=0.1\n)\ntrain_datagen.fit(X_train)\n\n# Confirming the shape of the preprocessed data\nprint(f\"Training data shape: {X_train.shape}\")\nprint(f\"Training target shape: {y_cat_train.shape}\")\nprint(f\"Testing data shape: {X_test.shape}\")\nprint(f\"Testing target shape: {y_cat_test.shape}\")\n","metadata":{"id":"wo88onsmljso","outputId":"4d389417-2645-453b-e91b-287dda817234"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Building\n\n## Creating a Baseline Model\n\nInstead of directly writing a final model, let's start by testing a number of different methods.\n\nLet's start with a baseline model.\n\nHere are the key components of this model.\n\n**Convolutional** Layer: Just one convolutional layer with 16 filters, which is the most basic feature extraction layer for a CNN.\n\n**Flatten** **Layer**: Converts the 2D output of the convolutional layer into a 1D array.\n\n**Output** **Layer**: A dense layer with 10 neurons (one for each class) with a softmax activation for classification.\n","metadata":{"id":"rnczTK_ym04g"}},{"cell_type":"code","source":"from tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, Flatten, Dense\n\n\nmodel = Sequential([\n    # Single Convolutional Layer\n    Conv2D(16, (3, 3), activation='relu', input_shape=(32, 32, 3)),\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"oaypU5jUmBGu","outputId":"602cb3e6-d87a-46ce-a581-4b5e94678ac6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install visualkeras","metadata":{"id":"LvvQSmjgr6GT","outputId":"5108e37b-cb6a-4976-b1bb-c400f7e6b916"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"LzsulTKyLiMx","outputId":"b5ad82cb-9c64-45c8-9117-092765ce8dcf"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's go ahead and train/fit the model","metadata":{"id":"H801HKs_qgQH"}},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=5, validation_split=0.2)","metadata":{"id":"Od31DV0Eqf3I","outputId":"1d4a1b25-9338-4500-faf7-ea396ed33106"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, let's define a function to test the model's accuracy. And other one to see how is accuracy changing and converging over time.\n","metadata":{"id":"0uZW39IDpvba"}},{"cell_type":"code","source":"# Function to smooth curves (optional)\ndef smooth_curve(points, factor=0.8):\n    smoothed_points = []\n    for point in points:\n        if smoothed_points:\n            previous = smoothed_points[-1]\n            smoothed_points.append(previous * factor + point * (1 - factor))\n        else:\n            smoothed_points.append(point)\n    return smoothed_points\n\ndef plot_convergence():\n\n  plt.figure(figsize=(10, 5))\n  # Accuracy plot\n  plt.subplot(1, 2, 1)\n  plt.plot(smooth_curve(history.history['accuracy']), marker='o', label='Train Accuracy')\n  plt.plot(smooth_curve(history.history['val_accuracy']), marker='o', label='Validation Accuracy')\n  plt.title('Training and Validation Accuracy over Epochs')\n  plt.xlabel('Epoch')\n  plt.ylabel('Accuracy')\n  plt.grid(True)\n  plt.legend()\n\n  # Loss plot\n  plt.subplot(1, 2, 2)\n  plt.plot(smooth_curve(history.history['loss']), marker='o', label='Train Loss')\n  plt.plot(smooth_curve(history.history['val_loss']), marker='o', label='Validation Loss')\n  plt.title('Training and Validation Loss over Epochs')\n  plt.xlabel('Epoch')\n  plt.ylabel('Loss')\n  plt.grid(True)\n  plt.legend()\n\n  plt.tight_layout()\n  plt.show()\n\ndef test_model_accuracy(model):\n  test_loss, test_accuracy = model.evaluate(X_test, y_cat_test)\n  print(f\"Test Accuracy: {test_accuracy * 100:.2f}%\")\n\n  plot_convergence()\n","metadata":{"id":"7b6qs5kNphh0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"jXaH91IbnGxK","outputId":"eb6cbd1b-68f7-4368-cc61-34fe56c01c18"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Current Accuracy - 56% - Baseline Model","metadata":{"id":"ZeAY3kXeK8L4"}},{"cell_type":"markdown","source":"## Number of Epochs\n\nNow, An epoch represents one complete pass through the entire training dataset. When we set epochs to a certain number, it means the learning algorithm will work through the entire dataset that many times\n\nThus, the optimal number of epochs is not a fixed value but is determined empirically. It's based on when we start seeing diminishing returns in reduction of loss on a validation set.\n\nWe see that at 5 epochs Accuracy was still increasing, thus now we know to try 15 epochs","metadata":{"id":"7kiXAQv30koy"}},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=15, validation_split=0.2)","metadata":{"id":"hyyJiAnnp_D5","outputId":"54ffcc64-077f-4c32-df18-6fb99256a245"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"sLLrPHfz4aYC","outputId":"207bf8a9-90c6-4eea-80ac-a2d83d85c18e"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that validation accuracy is no longer exceeding 58% no matter how much we increase the number of epochs","metadata":{"id":"4g9uaNOM3mLG"}},{"cell_type":"markdown","source":"## Batch Size\n\nNext, Understanding Batch Size\n\nDefinition: batch_size specifies the number of samples that will be passed through the network at one time. In other words, it's the number of training examples used in one iteration.\n\nIn our case, a batch_size of 32 is a common choice for mini-batch gradient descent, providing a balance between computational efficiency and the benefits of stochastic gradient updates.","metadata":{"id":"QcyJYiPk71fw"}},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=15, validation_split=0.2, batch_size=32)","metadata":{"id":"DPzmraNAnNVx","outputId":"371ae9eb-f9a9-40fc-c1c4-194f0d9aa897"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"JxVB3z_jnOFf","outputId":"2e98e019-158e-42e7-f877-908fca1a1397"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pooling layer.\n\nSo A pooling layer is a neural network layer that performs a downsampling operation along the spatial dimensions (width and height) of the input volume.\n\nWe studied in class that there are 2 types of Pooling\n\nMax Pooling: This is the most commonly used approach. It partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value.\n\nAverage Pooling: This method, instead of taking the maximum value, computes the average value of the elements in the sub-region of the input.\n\n","metadata":{"id":"or_jtibHEech"}},{"cell_type":"code","source":"model = Sequential([\n    # Single Convolutional Layer\n    Conv2D(16, (3, 3), activation='relu', input_shape=(32, 32, 3)),\n    MaxPooling2D(2, 2),\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"8n-fksHZ8EKW","outputId":"d3313bbb-a97a-4caa-9cb1-a013f710b8a2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"24Y0Nph-Mv_H","outputId":"151cfadf-5f3c-48ef-83bf-d17ae3f5beab"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=8, validation_split=0.2, batch_size=32)","metadata":{"id":"3VHKGa2iHYXD","outputId":"e8318c16-c3df-44cd-9d7e-bce2284aec16"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"mu4URRIYIq2Q","outputId":"83565eaf-204d-4803-bcb8-9319dc7912eb"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I think we can increase the number of epochs further as the graphs show an upward trend","metadata":{"id":"j7cLOP1CJiI5"}},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=15, validation_split=0.2, batch_size=32)","metadata":{"id":"tP32ySudIrDL","outputId":"59216bc4-a311-46ec-e866-eea8e634be55"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"QP9DEOS4JpQR","outputId":"2a388743-985e-4809-cc52-27da1d9982fb"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Improved Accuracy with Pooling - 59%.","metadata":{"id":"UhzEyT7_LGe-"}},{"cell_type":"markdown","source":"## Batch Normalization\n\nAdd Batch Normalization layers, which can help in speeding up training and stabilizing the learning process by normalizing the input layer by re-centering and re-scaling.\n\nBatch Normalization normalizes the output of a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation. In other words, it transforms the outputs of the layer to have a mean of zero and a standard deviation of one (standard normal distribution).\n\nCouple of things to keep in mind :\n\n1. Position in the Network: There's some debate about whether to apply Batch Normalization before or after the activation function. The original paper suggests applying it before the non-linear activation, but some practitioners find benefits in applying it after.\n2. Running Mean and Variance: During training, Batch Normalization computes the mean and variance of each feature for each mini-batch. During inference (testing), it uses the running average of these mean and variance values computed during training.\n3. Batch Size Dependency: The effectiveness of Batch Normalization can depend on the size of your mini-batches. Very small batch sizes can reduce its effectiveness due to less reliable estimates of the mean and variance.\n","metadata":{"id":"rOxS-r9qMT5L"}},{"cell_type":"code","source":"model = Sequential([\n    # Single Convolutional Layer\n    Conv2D(16, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n\n    MaxPooling2D(2, 2),\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"TgRAUyTVJpaK","outputId":"6ca09343-4d46-46c4-ca27-6e548a0c7f57"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"HiRx4D3pMyaD","outputId":"eb384f6e-80c2-4c5a-a592-0138612d9762"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=15, validation_split=0.2, batch_size=32)","metadata":{"id":"6r51ILmRM7zp","outputId":"6d672f17-a737-47e7-fef8-5ffb2eeebff1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"lDPAg9c9NL-m","outputId":"7043a9f5-43fc-4fc2-ab18-cc65c99ddfe4"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# No major change in accuracy post Batch Normalization - 58%","metadata":{"id":"9QhBHPT3N6Z8"}},{"cell_type":"markdown","source":"## Additional Convolution and Pooling Layers\n\nWe know that convolutional layers are the core building blocks of CNNs. They apply a number of filters to the input to create feature maps. These filters help the network in detecting various features such as edges, textures, and more complex patterns in deeper layers.\n\nLet's try adding another set of Convolutional layer. Also let's try switching around the number of parameters for our first convolutional layer and see if this causes the accuracy to increase/decrease\n","metadata":{"id":"R13Ww4cKPT1w"}},{"cell_type":"code","source":"model = Sequential([\n    # Single Convolutional Layer\n    Conv2D(32, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n\n    # Second convolutional layer\n    Conv2D(64, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"2bopZHhbNMJb","outputId":"c997922c-3a2d-4e10-a65d-8a95dc197775"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"xHkoZouqM19Z","outputId":"1e2ba955-2794-4c8d-eec6-df552820256f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=10, validation_split=0.2, batch_size=32)","metadata":{"id":"flJgtBCNRYzI","outputId":"8653b897-d0bb-46d1-a3d2-44e6a034c4c6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"xxirZ2JoRZYk","outputId":"108e56b1-536c-4485-8119-b0baf507e5aa"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# No change after adding more conv layers - 57%","metadata":{"id":"5x2gkWjjOB7M"}},{"cell_type":"markdown","source":"## Dropout Layer\n\nWhile Batch Normalization does offer some regularization, you might consider adding Dropout layers, especially if you encounter overfitting. Dropout layers are typically placed after pooling layers or between dense layers","metadata":{"id":"BC0Mm9_UNwF6"}},{"cell_type":"code","source":"model = Sequential([\n    # Single Convolutional Layer\n    Conv2D(32, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n\n    # Second convolutional layer\n    Conv2D(64, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"16uy2lDjRlfE","outputId":"29c730f7-81bc-4363-a143-d164a97eb5ce"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"mWnmiNDqM4fZ","outputId":"e5cf8b96-b33e-4925-f05d-30d04be1b3ec"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=10, validation_split=0.2, batch_size=32)","metadata":{"id":"i05gb0oRLmT6","outputId":"69a32940-99d9-49cd-b105-a341b4a671c0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"rw5ChK-ZLqyf","outputId":"9efc2d1f-40a2-4f40-b8f4-a16b47e17ead"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Major Improvement after adding drop out layer - 69%","metadata":{"id":"f74aLxoKO1dX"}},{"cell_type":"markdown","source":"Since the graphs showed improvement, let's try increasing the number of epochs","metadata":{"id":"8W9_H51WO7lO"}},{"cell_type":"code","source":"history = model.fit(X_train, y_cat_train, epochs=20, validation_split=0.2, batch_size=32)","metadata":{"id":"KfZyqUoRO1NV","outputId":"6b9796f2-9680-4b2f-a168-50737d638a21"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"8NhmLqTSPDcV","outputId":"996ee57e-317c-4154-be2a-db17ee59db83"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Early Stopping\n\nSo early stopping callback is an effective way to reduce training time, especially when you've reached a point where further training does not meaningfully improve model performance. Early stopping monitors a specified metric (such as validation loss) and stops the training process once this metric stops improving for a specified number of epochs. This not only saves time but also helps prevent overfitting","metadata":{"id":"WwVfxNEKVdKI"}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.01,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=15,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"ZGkQUtv5Vcq-","outputId":"54609aff-2362-4b33-939b-e436dc9ea96e"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"fLKvidQXWL3M","outputId":"beebe523-c064-4aa0-c755-759bdceac053"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Improved after Early stopping - 70% accuracy on the 4th Epoch","metadata":{"id":"T7wT6HhzaleO"}},{"cell_type":"markdown","source":"## Addition of more dense layers\n\nNow dense layers, which is the same as fully connected layers, are a standard component in neural networks where all the neurons from the previous layer are connected to each neuron in the dense layer.\n\nLet's try adding a couple more dense layers. And dropout layers along side them to avoid overfitting.","metadata":{"id":"mS-GTfTlaw9J"}},{"cell_type":"code","source":"model = Sequential([\n    # Single Convolutional Layer\n    Conv2D(32, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n\n    # Second convolutional layer\n    Conv2D(64, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # First Dense Layer\n    Dense(128, activation='relu'),\n    Dropout(0.5),\n\n    # Additional Dense Layer\n    Dense(64, activation='relu'),\n    Dropout(0.5),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"tAy2mRGybBL6","outputId":"649a11f2-b7a6-495b-de7e-c394865bb8ea"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.01,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=15,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"9D2y6_4_bp_I","outputId":"08dcf672-8e23-4161-e945-f3fcf8954cd2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"7TCI3Lnw4LTS","outputId":"8669b156-d8d7-49e4-b76f-4cc42a375d6f"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pontential Underfitting\n\nOkay, so what do these graphs suggest ?\n\nThat our model is still underfitting. So, we need more epochs. Let's try 50 instead of 10.","metadata":{"id":"sfwhg88f51Bm"}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.001,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=50,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"_1riLozT5-Er","outputId":"7f408897-5adb-466b-be31-5cd0f3c5bcec"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"dS3QQkgN8QkP","outputId":"3a3dafef-55ae-4227-fc89-228c7e94e651"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Still no improvement, well let's try reducing one of the dense and reducing the dropout values as well to see if that does something.","metadata":{"id":"ZrXgZM1harvC"}},{"cell_type":"code","source":"from tensorflow.keras.layers import LeakyReLU\n\nmodel = Sequential([\n    # Single Convolutional Layer\n    Conv2D(32, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n\n    # Second convolutional layer\n    Conv2D(64, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # First Dense Layer\n    Dense(128, activation='relu'),\n    Dropout(0.15),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"12HRio1e8bWB","outputId":"e6b80ab9-9e87-4724-cb9f-8812209c40b9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"sHSZhOjzM9W1","outputId":"54a009bc-9a7c-4ebf-c62b-169a3a151d3c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.001,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=50,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"Ura6hizb8eFh","outputId":"a1e51139-3ecc-433f-fcbc-6c7b349ee6a2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"O3OL1RDk8f7y","outputId":"c7deab3f-6234-4bb4-9525-c7e2cf85afc9"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Activation Functions\n\nSo, the primary role of activation functions is to introduce non-linear properties to the network, enabling it to learn more complex functions.\n\nThere are a number of these activation functions.\n\n1. Sigmoid or Logistic Function:\n\n- Formula: σ(x) = 1 / (1 + exp(-x))\n- Output Range: (0, 1)\n- Use Case: Often used in the output layer of a binary classification model.\n- Pros: Smooth gradient, output values bound between 0 and - Cons: Prone to vanishing gradient problem, not zero-centered.\n\n2. Hyperbolic Tangent (tanh):\n\n- Formula: tanh(x) = 2σ(2x) - 1\n- Output Range: (-1, 1)\n- Use Case: Hidden layers of a neural network since data is centered around zero.\n- Pros: Zero-centered, making optimization easier in some cases.\n- Cons: Also prone to the vanishing gradient problem.\n\n3. Rectified Linear Unit (ReLU):\n\n- Formula: f(x) = max(0, x)\n- Output Range: [0, ∞)\n- Use Case: Most widely used in hidden layers of neural networks.\n- Pros: Solves vanishing gradient problem, computationally efficient.\n- Cons: Neurons can \"die\" during training (dead ReLU problem).\n\n4. Leaky ReLU:\n\n- Formula: f(x) = x if x > 0, else αx (α is a small constant)\n- Use Case: To address the dying neuron problem in ReLU.\n- Pros: Prevents neurons from dying.\n\n5. Softmax:\n\n- Formula: Converts a vector of values to a probability distribution.\n- Use Case: Output layer for multi-class classification problems.\n- Pros: Outputs sum to 1, providing a probabilistic interpretation.\n\n","metadata":{"id":"PcZ3A5NESWjI"}},{"cell_type":"markdown","source":"**Now, I tried a number of these functions but I couldn't get any better results. So I've included their names and properties here alongside an example.**","metadata":{"id":"itBnj1QOTNyB"}},{"cell_type":"code","source":"from tensorflow.keras.layers import LeakyReLU\n\nmodel = Sequential([\n    # Single Convolutional Layer\n    Conv2D(32, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n\n    # Second convolutional layer\n    Conv2D(64, (3, 3)),\n    BatchNormalization(),\n    LeakyReLU(alpha=0.01),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # First Dense Layer\n    Dense(128, activation='relu'),\n    Dropout(0.15),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nmodel.summary()\n","metadata":{"id":"iMzW8gbFLrGy","outputId":"15972c7a-0c6c-405e-bde3-aa430c2baa14"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.001,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=50,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"8tMbE8V5Un12","outputId":"a722b00f-c77d-4c4f-c676-3464687eca71"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"dnXO0AiXaQPZ","outputId":"7c2f42d8-6a5e-4530-f4c2-1eb6ebefdf7f"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### As we see here, LeakyRelu performs worse compared to relu.","metadata":{"id":"mEJ9U9qiAyo9"}},{"cell_type":"markdown","source":"## Learning Rate and Optimizer\n\nOkay, I tried modifying and adding layers of all sorts. Next, I moved to the compile code, specifically I learnt about :\n1. Optimizers are algorithms or methods used to change the attributes of the neural network, such as weights and learning rate, to reduce the losses.\n\n2. Learning Rate : it determines the size of the steps the model takes during training with the aim of reaching a minimum loss.\n\nLet's try and SGD Optimizer and different learning rates","metadata":{"id":"153Tjfk2BnnU"}},{"cell_type":"code","source":"from tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.optimizers import SGD\n\n\nmodel = Sequential([\n    # Single Convolutional Layer\n    Conv2D(32, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n\n    # Second convolutional layer\n    Conv2D(64, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # First Dense Layer\n    Dense(128, activation='relu'),\n    Dropout(0.15),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\n\noptimizer = SGD(learning_rate=0.01, momentum=0.9)\nmodel.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\n\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.001,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=50,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"fNmpoCdoF4hj","outputId":"b5672ee3-6d79-4e20-a75c-11f8cf2a84ec"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"F3XbexsjGRXI","outputId":"134969d3-b3f8-4629-fb91-96f73575f879"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Increasing Complexity of the model.\n\nNow, that we have tried out most of the things possible. Let's try a little bit more to get the accuracy to reach above 80%","metadata":{"id":"i6DEWy9KPvF9"}},{"cell_type":"code","source":"from tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.optimizers import SGD\n\n\nmodel = Sequential([\n    # Single Convolutional Layer\n    Conv2D(64, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    Conv2D(64, (3, 3), input_shape=(32, 32, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n\n    # Second convolutional layer\n    Conv2D(128, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    Conv2D(128, (3, 3)),\n    BatchNormalization(),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),  # Dropout layer after pooling\n\n    # Flattening the output to feed into a dense layer\n    Flatten(),\n\n    # First Dense Layer\n    Dense(128, activation='relu'),\n    Dropout(0.25),\n\n    # Output Layer with softmax activation for multi-class classification\n    Dense(10, activation='softmax')\n])\n\n\noptimizer = SGD(learning_rate=0.01, momentum=0.9)\nmodel.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\n\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss',  # Monitor validation loss\n    patience=3,         # Number of epochs with no improvement after which training will be stopped\n    min_delta=0.0005,    # Minimum change to qualify as an improvement\n    mode='min',         # The training will stop when the quantity monitored stops decreasing\n    verbose=1           # Print messages when early stopping is triggered\n)\n\nhistory = model.fit(\n    X_train, y_cat_train,\n    epochs=32,  # Set a high number of epochs\n    batch_size=32,\n    validation_split=0.2,\n    callbacks=[early_stopping]  # Include the early stopping callback\n)\n","metadata":{"id":"GXuSK966GRVH","outputId":"3944e468-162b-478a-9975-40159a37ff3d"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import visualkeras\n\nvisualkeras.layered_view(model, spacing=15, legend=True, scale_xy=5, to_file='output.png')","metadata":{"id":"VPdOIAZhNC0Y","outputId":"b1dfe550-0db6-4ae9-f0da-86088347d13c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_model_accuracy(model)","metadata":{"id":"wPSGRIawHc5C","outputId":"8e5cefee-2aa8-46dd-832d-a8840e682998"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Best Accuracy so far - 80%","metadata":{"id":"soRSEq8CuE4z"}},{"cell_type":"code","source":"from tensorflow.keras.models import load_model\n\nmodel.save('cnn_20_epochs.h5')","metadata":{"id":"eSHmtVbhuErM","outputId":"4e719bcb-7d5c-4090-c976-8cea1f70a424"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2 = load_model('cnn_20_epochs.h5')","metadata":{"id":"uktoCm3XwXhO"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2.predict(X_test)","metadata":{"id":"iOC9Lgfmwcsb","outputId":"078e8147-b3db-4f06-a41d-1d15431c230a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation of Model","metadata":{"id":"b-8T-B4Hub2H"}},{"cell_type":"code","source":"!pip install visualkeras","metadata":{"id":"DwFB5RBYLF-M","outputId":"3c68c42c-884c-4265-927f-10a935196aad"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n          'dog', 'frog', 'horse', 'ship', 'truck']\n\nevaluation = model.evaluate(X_test, y_cat_test)\nprint(f'Test Accuracy : {evaluation[1] * 100:.2f}%')\n\ny_pred = model.predict(X_test)\ny_pred = np.argmax(y_pred, axis=1)\ncm = confusion_matrix(y_test, y_pred)\n\n\ndisp = ConfusionMatrixDisplay(confusion_matrix=cm,\n                              display_labels=labels)\n\n\n# NOTE: Fill all variables here with default values of the plot_confusion_matrix\nfig, ax = plt.subplots(figsize=(10, 10))\ndisp = disp.plot(xticks_rotation='vertical', ax=ax,cmap='summer')\n\nplt.show()","metadata":{"id":"CVMsEMMkvpO8","outputId":"3f944fc2-fa22-4036-ab37-3a31841c367c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(classification_report(y_test, y_pred))","metadata":{"id":"6mLsgu66vrMf","outputId":"458474b1-7bb2-4f85-8ec6-278d99ced66f"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{"id":"GvveFD9lpdtZ"}},{"cell_type":"markdown","source":"## Testing against 1 image","metadata":{"id":"fXP1kaO7vUgI"}},{"cell_type":"code","source":"# Define the labels of the dataset\nlabels = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n          'dog', 'frog', 'horse', 'ship', 'truck']\n\nmy_image = X_test[100]\nplt.imshow(my_image)\n\n# that's a Deer\nprint(f\" Image 100 is {y_test[100]}\")\n\n# correctly predicted as a Deer\npred_100 = np.argmax(model.predict(my_image.reshape(1, 32, 32, 3)))\nprint(f\"The model predict that image 100 is {labels[pred_100]}\")","metadata":{"id":"C8bmH7LJue3Y","outputId":"59a5cc94-3b3c-46bf-9d68-4d45d10f1365"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's view more images in a grid format\n# Define the dimensions of the plot grid\nW_grid = 5\nL_grid = 5\n\n# fig, axes = plt.subplots(L_grid, W_grid)\n# subplot return the figure object and axes object\n# we can use the axes object to plot specific figures at various locations\n\nfig, axes = plt.subplots(L_grid, W_grid, figsize = (14,14))\n\naxes = axes.ravel() # flaten the 15 x 15 matrix into 225 array\n\nn_test = len(X_test) # get the length of the train dataset\n\n# Select a random number from 0 to n_train\nfor i in np.arange(0, W_grid * L_grid): # create evenly spaces variables\n\n    # Select a random number\n    index = np.random.randint(0, n_test)\n    # read and display an image with the selected index\n    axes[i].imshow(X_test[index,1:])\n    label_index = int(y_pred[index])\n    axes[i].set_title(labels[label_index], fontsize = 8)\n    axes[i].axis('off')\n\nplt.subplots_adjust(hspace=0.4)","metadata":{"id":"tAQr4aogvabL","outputId":"ecb8a639-adfe-432c-f820-507d2302d79c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = model.predict(X_test)\n","metadata":{"id":"D7fVOnJ7ASNe","outputId":"e066ab8e-77b9-4ee7-b1e5-2601cd5c3312"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualizing the output Probabilities","metadata":{"id":"AYyf1VweEFCq"}},{"cell_type":"code","source":"def plot_image(i, predictions_array, true_label, img):\n    predictions_array, true_label, img = predictions_array, true_label[i], img[i]\n    plt.grid(False)\n    plt.xticks([])\n    plt.yticks([])\n\n    plt.imshow(img, cmap=plt.cm.binary)\n\n    predicted_label = np.argmax(predictions_array)\n    if predicted_label == true_label:\n        color = 'blue'\n    else:\n        color = 'red'\n\n    plt.xlabel(f\"{labels[int(predicted_label)]} {100*np.max(predictions_array):2.0f}% ({labels[int(true_label)]})\",\n               color=color)\n\ndef plot_value_array(i, predictions_array, true_label):\n    predictions_array, true_label = predictions_array, int(true_label[i])\n    plt.grid(False)\n    plt.xticks(range(10))\n    plt.yticks([])\n    thisplot = plt.bar(range(10), predictions_array, color=\"#777777\")\n    plt.ylim([0, 1])\n    predicted_label = np.argmax(predictions_array)\n\n    thisplot[predicted_label].set_color('red')\n    thisplot[true_label].set_color('blue')\n\n# Plot the first X test images, their predicted labels, and the true labels.\n# Color correct predictions in blue and incorrect predictions in red.\nnum_rows = 4\nnum_cols = 3\nnum_images = num_rows * num_cols\nplt.figure(figsize=(2 * 2 * num_cols, 2 * num_rows))\nfor i in range(num_images):\n    plt.subplot(num_rows, 2 * num_cols, 2 * i + 1)\n    plot_image(i, predictions[i], y_test, X_test)\n    plt.subplot(num_rows, 2*num_cols, 2*i+2)\n    plot_value_array(i, predictions[i], y_test)\nplt.tight_layout()\nplt.show()\n\n","metadata":{"id":"oNXqT4dWvWBe","outputId":"8d72b07d-545d-4a9c-8c92-a3365321c1cf"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"jjdTVcca-0UM"},"execution_count":null,"outputs":[]}]}