{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Hi !!!! Obi-Wan \n\n![](https://media.giphy.com/media/6fScAIQR0P0xW/giphy.gif)\n\n## Please Do upVote if you like this Notebook","metadata":{"execution":{"iopub.status.busy":"2022-08-13T18:05:47.211453Z","iopub.execute_input":"2022-08-13T18:05:47.212435Z","iopub.status.idle":"2022-08-13T18:05:48.308627Z","shell.execute_reply.started":"2022-08-13T18:05:47.212279Z","shell.execute_reply":"2022-08-13T18:05:48.307183Z"}}},{"cell_type":"markdown","source":"Convolutional Neural Network (CNN) is a type of deep neural network which has proven to perform well in computer vision tasks such as image classification, object detection, object localization and neural style transfer. In this post, I will explain about the different layers that make up a convolutional neural network: convolution layer, pooling layer and fully connected layer.\n\n# Convolution Layer\nA convolution layer transforms the input image in order to extract features from it. In this transformation, the image is convolved with a **kernel (or filter).**\n\n\n\nA **kernel** is a small matrix, with its height and width smaller than the image to be convolved. It is also known as a convolution matrix or convolution mask. This kernel slides across the height and width of the image input and dot product of the kernel and the image are computed at every spatial position. The length by which the kernel slides is known as the stride length. **In the image below, the input image is of size 5X5, the kernel is of size 3X3 and the stride length is 1.** The output image is also referred to as the convolved feature.\n\n\n\nWhen convolving a coloured image (RGB image) with channels 3, the channel of the filters must be 3 as well. **In other words, in convolution, the number of channels in the kernel must be the same as the number of channels in the input image.**\n\n\n\nWhen we want to extract more than one feature from an image using convolution, we can use **multiple kernels** instead of using just one. In such a case, the size of all the kernels must be the same. The convolved features of the input image the output are stacked one after the other to create an output so that the number of channels is equal to the number of filters used. See the image below for reference.\n\n\n\n\nAn **activation function** is the last component of the convolutional layer to increase the non-linearity in the output. Generally, ReLu function or Tanh function is used as an activation function in a convolution layer. Here is an image of a simple convolution layer, where a 6X6X3 input image is convolved with two kernels of size 4X4X3 to get a convolved feature of size 3X3X2, to which activation function is applied to get the output, which is also referred to as feature map.\n\n\n\n## Pooling Layer\nPooling layer is used to reduce the size of the input image. In a convolutional neural network, a convolutional layer is usually followed by a pooling layer. Pooling layer is usually added to speed up computation and to make some of the detected features more robust.\n\nPooling operation uses kernel and stride as well. In the example image below, 2X2 filter is used for pooling the 4X4 input image of size, with a stride of 2.\n\nThere are different types of pooling. **Max pooling** and **average pooling** are the most commonly used pooling method a convolutional neural network.\n\n\n\n**Max Pooling:** In max pooling, from each patch of a feature map, the maximum value is selected to create a reduced map.\n\n**Average Pooling:** In average pooling, from each patch of a feature map, the average value is selected to create a reduced map.","metadata":{}},{"cell_type":"markdown","source":"# How can we understand what our convolutional neural network (black box) sees and understands when making a decision?\n\n![](https://miro.medium.com/max/1046/1*34rrZ9UH3vJ7Q7tUAWo7JA.png)\n\n\nWe can use Grad-CAM to identify bias in our model’s predictions which was passed by the training dataset. Considering the sample example as in the paper, suppose we’ve trained a nurse vs. doctor image classifier from images taken from the ImageNet dataset. The classifier would perform pretty well on the testing dataset but doesn’t generalize well. Well, how do we know this? Look at the Grad-CAM heatmaps in figure above, our the biased model, we can observe that the model focuses on unwanted features ( part of the face, hair ) which seem insignificant while predicting labels ‘nurse’ and ‘doctor’. The model seems to learn gender-based features which aren’t appropriate in our use-case.\n\nAs mentioned in the paper, the dataset was biased. 78% of the images, labelled as ‘doctor’, contained males whereas 93% of the images, labelled as ‘nurse’, contained women. Clearly, the gender bias present in the dataset was passed on to the model. Note, the model performed well on the testing dataset and had 82% accuracy on it.\n\nOnce we know that a specific bias is present in our dataset, we recollect the data and retrain the model. The unbiased model, as seen in figure 12, captures the exact features to identify the image as containing a ‘nurse’ and ‘doctor’. Observe, that the model focuses on the stethoscope for identifying the image of a ‘doctor’.","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport shutil\nfrom shutil import copyfile\nimport random\nimport zipfile\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array, load_img\n# Display\nfrom IPython.display import Image, display\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nfrom tensorflow import keras\nimport numpy as np\nimport random\nfrom tensorflow.keras.preprocessing.image import img_to_array, load_img","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-13T09:38:21.857524Z","iopub.execute_input":"2022-08-13T09:38:21.859057Z","iopub.status.idle":"2022-08-13T09:38:21.866751Z","shell.execute_reply.started":"2022-08-13T09:38:21.859008Z","shell.execute_reply":"2022-08-13T09:38:21.865765Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TensorFlow Based Approch to Visualization \n\n","metadata":{}},{"cell_type":"code","source":"test_path_cats_and_dogs ='../input/dogs-vs-cats/test1.zip'\n\n\nlocal_zip = test_path_cats_and_dogs\nzip_ref = zipfile.ZipFile(local_zip, 'r')\nzip_ref.extractall('./cats-vs-dogs/')\nzip_ref.close()\n\ntrain_path_cats_and_dogs ='../input/dogs-vs-cats/train.zip'\n\n\nlocal_zip = train_path_cats_and_dogs\nzip_ref = zipfile.ZipFile(local_zip, 'r')\nzip_ref.extractall('./cats-vs-dogs/')\nzip_ref.close()\n\nBASE_PATH = './cats-vs-dogs/train/'\nTRAIN_PATH='./cats-vs-dogs/train_data/'\nVAL_PATH='./cats-vs-dogs/validation_data/'\n\nos.mkdir(TRAIN_PATH)\nos.mkdir(VAL_PATH)\ntrain_dog=os.path.join(TRAIN_PATH , 'dog')\ntrain_cat=os.path.join(TRAIN_PATH , 'cat')\nval_dog=os.path.join(VAL_PATH , 'dog')\nval_cat=os.path.join(VAL_PATH , 'cat')\n# print(train_cat)\nos.mkdir(train_dog)\nos.mkdir(train_cat)\nos.mkdir(val_dog)\nos.mkdir(val_cat)\n\n#segregating cats and dog files for training and validation\ncat_file_list=[]\ndog_file_list=[]\nfor file in os.listdir(BASE_PATH):\n if file[:3]=='cat':\n cat_file_list.append(file)\n elif file[:3]=='dog':\n dog_file_list.append(file)\n#shuffling the list\n\ntrain_split=0.8\n\nrandom.sample(cat_file_list, len(cat_file_list))\nrandom.sample(dog_file_list, len(dog_file_list))\nnum_of_training_files_cat= int(len(cat_file_list)* train_split)\nnum_of_training_files_dog= int(len(dog_file_list)* train_split)\nprint(\"Total training cat files\", num_of_training_files_cat)\nprint(\"Training files dog\", num_of_training_files_dog)\nprint(\"Total filess\", len(cat_file_list) + len(dog_file_list) )\nprint(\"Total val Dog files\", len(dog_file_list) - num_of_training_files_dog)\n\nfor file in cat_file_list[:num_of_training_files_cat]:\n copyfile(os.path.join(BASE_PATH, file), os.path.join(train_cat, file))\n \nfor file in cat_file_list[num_of_training_files_cat:]:\n copyfile(os.path.join(BASE_PATH, file), os.path.join(val_cat, file))\n\nfor file in dog_file_list[:num_of_training_files_dog]:\n copyfile(os.path.join(BASE_PATH, file), os.path.join(train_dog, file))\n \nfor file in dog_file_list[num_of_training_files_dog:]:\n copyfile(os.path.join(BASE_PATH, file), os.path.join(val_dog, file))\n \nprint(\"Total val Dog files\", len(dog_file_list) - num_of_training_files_dog)\n\nbatch_size = 128 #128\nepochs = 20\nIMG_HEIGHT = 150\nIMG_WIDTH = 150\n\n\ntrain_image_generator = ImageDataGenerator(\n rescale=1./255,\n rotation_range=45,\n width_shift_range=.15,\n height_shift_range=.15,\n horizontal_flip=True,\n zoom_range=0.3 \n\n ) # Generator for our training data\n\n\nvalidation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data\n\ntrain_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,\n directory=TRAIN_PATH,\n shuffle=True,\n target_size=(IMG_HEIGHT, IMG_WIDTH),\n class_mode='binary')\nval_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,\n directory=VAL_PATH,\n target_size=(IMG_HEIGHT, IMG_WIDTH),\n class_mode='binary')\n\n\n# This function will plot images in the form of a grid with 1 row and 5 columns where images are placed in each column.\ndef plotImages(images_arr):\n fig, axes = plt.subplots(1, 5, figsize=(20,20))\n axes = axes.flatten()\n for img, ax in zip( images_arr, axes):\n ax.imshow(img)\n ax.axis('off')\n plt.tight_layout()\n plt.show()","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-13T09:35:25.730297Z","iopub.execute_input":"2022-08-13T09:35:25.730828Z","iopub.status.idle":"2022-08-13T09:35:47.555068Z","shell.execute_reply.started":"2022-08-13T09:35:25.730797Z","shell.execute_reply":"2022-08-13T09:35:47.554013Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"augmented_images = [train_data_gen[0][0][0] for i in range(5)]\nplotImages(augmented_images)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-13T09:35:47.556641Z","iopub.execute_input":"2022-08-13T09:35:47.556999Z","iopub.status.idle":"2022-08-13T09:35:53.517965Z","shell.execute_reply.started":"2022-08-13T09:35:47.556962Z","shell.execute_reply":"2022-08-13T09:35:53.516926Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_training_images, _ = next(train_data_gen)\nplotImages(sample_training_images[:5])","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-13T09:35:53.519704Z","iopub.execute_input":"2022-08-13T09:35:53.520805Z","iopub.status.idle":"2022-08-13T09:35:54.673319Z","shell.execute_reply.started":"2022-08-13T09:35:53.520764Z","shell.execute_reply":"2022-08-13T09:35:54.672296Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = Sequential([\n Conv2D(16, 3, padding='same', activation='relu', \n input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),\n MaxPooling2D(),\n Dropout(0.2),\n Conv2D(32, 3, padding='same', activation='relu'),\n MaxPooling2D(),\n Conv2D(64, 3, padding='same', activation='relu'),\n MaxPooling2D(),\n Dropout(0.2),\n Flatten(),\n Dense(512, activation='relu'),\n Dense(1)\n])\n\nmodel.compile(optimizer='adam',\n loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n metrics=['accuracy'])\n\nmodel.summary()\n\n\nbatch_size=100\nhistory = model.fit(\n train_data_gen,\n epochs=10,\n validation_data=val_data_gen,\n)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-13T09:38:29.249678Z","iopub.execute_input":"2022-08-13T09:38:29.250865Z","iopub.status.idle":"2022-08-13T10:06:52.009976Z","shell.execute_reply.started":"2022-08-13T09:38:29.250809Z","shell.execute_reply":"2022-08-13T10:06:52.008893Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plotFilters(conv_filter):\n fig, axes = plt.subplots(1, 3, figsize=(5,5))\n axes = axes.flatten()\n for img, ax in zip( conv_filter, axes):\n ax.imshow(img)\n ax.axis('off')\n plt.tight_layout()\n plt.show()\n\n\nnp.seterr(divide='ignore', invalid='ignore')\nimg_path='../input/cat-and-dogs/dataset/training_set/dogs/dog.1000.jpg' #dog\n# Let's define a new Model that will take an image as input, and will output\n# intermediate representations for all layers in the previous model after\n# the first.\nsuccessive_outputs = [layer.output for layer in model.layers[1:]]\n\n#visualization_model = Model(img_input, successive_outputs)\nvisualization_model = tf.keras.models.Model(inputs = model.input, outputs = successive_outputs)\n\n# Let's prepare a random input image of a cat or dog from the training set.\n#cat_img_files = [os.path.join(train_cats_dir, f) for f in train_cat_fnames]\n#dog_img_files = [os.path.join(train_dogs_dir, f) for f in train_dog_fnames]\n\n#img_path = random.choice(cat_img_files + dog_img_files)\n\nimg = load_img(img_path, target_size=(150, 150)) # this is a PIL image\n\nx = img_to_array(img) # Numpy array with shape (150, 150, 3)\nx = x.reshape((1,) + x.shape) # Numpy array with shape (1, 150, 150, 3)\n\n# Rescale by 1/255\nx /= 255.0\n\n# Let's run our image through our network, thus obtaining all\n# intermediate representations for this image.\nsuccessive_feature_maps = visualization_model.predict(x)\n\n# These are the names of the layers, so can have them as part of our plot\nlayer_names = [layer.name for layer in model.layers]\n\n# -----------------------------------------------------------------------\n# Now let's display our representations\n# -----------------------------------------------------------------------\nfor layer_name, feature_map in zip(layer_names, successive_feature_maps):\n# print(feature_map.shape)\n if len(feature_map.shape) == 4:\n \n #-------------------------------------------\n # Just do this for the conv / maxpool layers, not the fully-connected layers\n #-------------------------------------------\n n_features = feature_map.shape[-1] # number of features in the feature map\n size = feature_map.shape[ 1] # feature map shape (1, size, size, n_features)\n \n # We will tile our images in this matrix\n display_grid = np.zeros((size, size * n_features))\n \n #-------------------------------------------------\n # Postprocess the feature to be visually palatable\n #-------------------------------------------------\n for i in range(n_features):\n x = feature_map[0, :, :, i]\n x -= x.mean()\n x /= x.std ()\n x *= 64\n x += 128\n x = np.clip(x, 0, 255).astype('uint8')\n display_grid[:, i * size : (i + 1) * size] = x # Tile each filter into a horizontal grid\n\n #-----------------\n # Display the grid\n #-----------------\n\n scale = 20. / n_features\n plt.figure( figsize=(scale * n_features, scale) )\n plt.title ( layer_name )\n plt.grid ( False )\n plt.imshow( display_grid, aspect='auto', cmap='viridis' ) ","metadata":{"execution":{"iopub.status.busy":"2022-08-13T10:09:05.399568Z","iopub.execute_input":"2022-08-13T10:09:05.400005Z","iopub.status.idle":"2022-08-13T10:09:06.749759Z","shell.execute_reply.started":"2022-08-13T10:09:05.399969Z","shell.execute_reply":"2022-08-13T10:09:06.748691Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_path='../input/cat-and-dogs/dataset/training_set/cats/cat.1001.jpg' #dog\n# Define a new Model, Input= image \n# Output= intermediate representations for all layers in the \n# previous model after the first.\nsuccessive_outputs = [layer.output for layer in model.layers[1:]]\n#visualization_model = Model(img_input, successive_outputs)\nvisualization_model = tf.keras.models.Model(inputs = model.input, outputs = successive_outputs)\n#Load the input image\nimg = load_img(img_path, target_size=(150, 150)) \n# Convert ht image to Array of dimension (150,150,3)\nx = img_to_array(img) \nx = x.reshape((1,) + x.shape) \n# Rescale by 1/255\nx /= 255.0\n# Let's run input image through our vislauization network\n# to obtain all intermediate representations for the image.\nsuccessive_feature_maps = visualization_model.predict(x)\n# Retrieve are the names of the layers, so can have them as part of our plot\nlayer_names = [layer.name for layer in model.layers]\nfor layer_name, feature_map in zip(layer_names, successive_feature_maps):\n# print(feature_map.shape)\n if len(feature_map.shape) == 4:\n \n # Plot Feature maps for the conv / maxpool layers, not the fully-connected layers\n \n n_features = feature_map.shape[-1] # number of features in the feature map\n size = feature_map.shape[ 1] # feature map shape (1, size, size, n_features)\n \n # We will tile our images in this matrix\n display_grid = np.zeros((size, size * n_features))\n \n # Postprocess the feature to be visually palatable\n for i in range(n_features):\n x = feature_map[0, :, :, i]\n x -= x.mean()\n x /= x.std ()\n x *= 64\n x += 128\n x = np.clip(x, 0, 255).astype('uint8')\n # Tile each filter into a horizontal grid\n display_grid[:, i * size : (i + 1) * size] = x \n\n # Display the grid\n scale = 20. / n_features\n plt.figure( figsize=(scale * n_features, scale) )\n plt.title ( layer_name )\n plt.grid ( False )\n plt.imshow( display_grid, aspect='auto', cmap='viridis' )\n \nmodel.save('./model1.h5')","metadata":{"execution":{"iopub.status.busy":"2022-08-13T10:09:06.751862Z","iopub.execute_input":"2022-08-13T10:09:06.752210Z","iopub.status.idle":"2022-08-13T10:09:08.106592Z","shell.execute_reply.started":"2022-08-13T10:09:06.752175Z","shell.execute_reply":"2022-08-13T10:09:08.105675Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://media.giphy.com/media/5xaOcLOqNmWHaLeB14I/giphy.gif)\n\n# Grab Cam - Gradient-weighted Class Activation Mapping \n## Visual Explanations from Deep Networks\n\nResearch Paper : https://arxiv.org/abs/1610.02391 \n\nPapers with code : https://paperswithcode.com/paper/grad-cam-visual-explanations-from-deep\n\n\n\n\n\n\nThe technique is an improvement over previous approaches in versatility and accuracy. It is complex but, luckily, the output is intuitive. From a high-level, we take an image as input and create a model that is cut off at the layer for which we want to create a Grad-CAM heat-map. We attach the fully-connected layers for prediction. We then run the input through the model, grab the layer output, and loss. Next, we find the gradient of the output of our desired model layer w.r.t. the model loss. From there, we take sections of the gradient which contribute to the prediction, reduce, resize, and rescale so that the heat-map can be overlaid with the original image\n\nWe started out creating Grad-CAM heat-maps for the last convolutional layer in our model, Conv_1. In theory, the heat-map for this layer should display the most accurate visual explanation of the object being classified by the model.\n","metadata":{}},{"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('COjUB9Izk6E', width=800, height=450)","metadata":{"execution":{"iopub.status.busy":"2022-08-13T13:21:04.820404Z","iopub.execute_input":"2022-08-13T13:21:04.820833Z","iopub.status.idle":"2022-08-13T13:21:04.845372Z","shell.execute_reply.started":"2022-08-13T13:21:04.820796Z","shell.execute_reply":"2022-08-13T13:21:04.844442Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_builder = tf.keras.applications.xception.Xception\nimg_size = (299, 299)\npreprocess_input = tf.keras.applications.xception.preprocess_input\ndecode_predictions = tf.keras.applications.xception.decode_predictions\n\nlast_conv_layer_name = \"block14_sepconv2_act\"\n\n# The local path to our target image\nimg_path = tf.keras.utils.get_file(\n \"african_elephant.jpg\", \"https://i.imgur.com/Bvro0YD.png\"\n)\n\n\ndef get_img_array(img_path, size):\n # `img` is a PIL image of size 299x299\n img = keras.preprocessing.image.load_img(img_path, target_size=size)\n # `array` is a float32 Numpy array of shape (299, 299, 3)\n array = keras.preprocessing.image.img_to_array(img)\n # We add a dimension to transform our array into a \"batch\"\n # of size (1, 299, 299, 3)\n array = np.expand_dims(array, axis=0)\n return array\n\n\ndef make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=None):\n # First, we create a model that maps the input image to the activations\n # of the last conv layer as well as the output predictions\n grad_model = tf.keras.models.Model(\n [model.inputs], [model.get_layer(last_conv_layer_name).output, model.output]\n )\n\n # Then, we compute the gradient of the top predicted class for our input image\n # with respect to the activations of the last conv layer\n with tf.GradientTape() as tape:\n last_conv_layer_output, preds = grad_model(img_array)\n if pred_index is None:\n pred_index = tf.argmax(preds[0])\n class_channel = preds[:, pred_index]\n\n # This is the gradient of the output neuron (top predicted or chosen)\n # with regard to the output feature map of the last conv layer\n grads = tape.gradient(class_channel, last_conv_layer_output)\n\n # This is a vector where each entry is the mean intensity of the gradient\n # over a specific feature map channel\n pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n\n # We multiply each channel in the feature map array\n # by \"how important this channel is\" with regard to the top predicted class\n # then sum all the channels to obtain the heatmap class activation\n last_conv_layer_output = last_conv_layer_output[0]\n heatmap = last_conv_layer_output @ pooled_grads[..., tf.newaxis]\n heatmap = tf.squeeze(heatmap)\n\n # For visualization purpose, we will also normalize the heatmap between 0 & 1\n heatmap = tf.maximum(heatmap, 0) / tf.math.reduce_max(heatmap)\n return heatmap.numpy()\n\n# Prepare image\nimg_array = preprocess_input(get_img_array(img_path, size=img_size))\n\n# Make model\nmodel = model_builder(weights=\"imagenet\")\n\n# Remove last layer's softmax\nmodel.layers[-1].activation = None\n\n# Print what the top predicted class is\npreds = model.predict(img_array)\nprint(\"Predicted:\", decode_predictions(preds, top=1)[0])\n\n# Generate class activation heatmap\nheatmap = make_gradcam_heatmap(img_array, model, last_conv_layer_name)\n\n# Display heatmap\nplt.matshow(heatmap)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-13T10:09:27.291799Z","iopub.execute_input":"2022-08-13T10:09:27.292225Z","iopub.status.idle":"2022-08-13T10:09:31.243805Z","shell.execute_reply.started":"2022-08-13T10:09:27.292191Z","shell.execute_reply":"2022-08-13T10:09:31.242569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Gard-CAM Overview\n\nThe basic idea behind Grad-CAM is the same as the basic idea behind CAM: we want to exploit the spatial information that is preserved through convolutional layers, in order to understand which parts of an input image were important for a classification decision.\n\nSimilar to CAM, Grad-CAM uses the feature maps produced by the last convolutional layer of a CNN. The authors of Grad-CAM argue, “we can expect the last convolutional layers to have the best compromise between high-level semantics and detailed spatial information.”\n\nHere is a sketch showing the parts of a neural network model relevant to Grad-CAM:\n\n\n\n\nThe CNN is composed of some convolutional layers (shown as “conv” in the sketch). The feature maps produced by the final convolutional layer are shown as A1, A2, and A3, the same as in the CAM sketch.\n\nAt this point, for CAM we would need to do global average pooling followed by a fully connected layer. For Grad-CAM, we can do anything – for example, multiple fully connected layers – which is shown as “any neural network layers” in the sketch. The only requirement is that the layers we insert after A1, A2, and A3 have to be differentiable so that we can get a gradient. Finally, we have our classification outputs for airplane, dog, cat, person, etc.\n\nThe difference between CAM and Grad-CAM is in how the feature maps A1, A2, and A3 are weighted to make the final heatmap. In CAM, we weight these feature maps using weights taken out of the last fully-connected layer of the network. In Grad-CAM, we weight the feature maps using “alpha values” that are calculated based on gradients. Therefore, Grad-CAM does not require a particular architecture, because we can calculate gradients through any kind of neural network layer we want. The “Grad” in Grad-CAM stands for “gradient.”\n\nThe output of Grad-CAM is a “class-discriminative localization map”, i.e. a heatmap where the hot part corresponds to a particular class. If there are 10 possible output classes, then for a particular input image, you can make 10 different Grad-CAM heatmaps, one heatmap for each class.","metadata":{}},{"cell_type":"code","source":"def save_and_display_gradcam(img_path, heatmap, cam_path=\"cam.jpg\", alpha=0.4):\n # Load the original image\n img = tf.keras.preprocessing.image.load_img(img_path)\n img = tf.keras.preprocessing.image.img_to_array(img)\n\n # Rescale heatmap to a range 0-255\n heatmap = np.uint8(255 * heatmap)\n\n # Use jet colormap to colorize heatmap\n jet = cm.get_cmap(\"jet\")\n\n # Use RGB values of the colormap\n jet_colors = jet(np.arange(256))[:, :3]\n jet_heatmap = jet_colors[heatmap]\n\n # Create an image with RGB colorized heatmap\n jet_heatmap = keras.preprocessing.image.array_to_img(jet_heatmap)\n jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))\n jet_heatmap = keras.preprocessing.image.img_to_array(jet_heatmap)\n\n # Superimpose the heatmap on original image\n superimposed_img = jet_heatmap * alpha + img\n superimposed_img = keras.preprocessing.image.array_to_img(superimposed_img)\n\n # Save the superimposed image\n superimposed_img.save(cam_path)\n\n # Display Grad CAM\n display(Image(cam_path))\n\n\nsave_and_display_gradcam(img_path, heatmap)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-13T10:09:33.949975Z","iopub.execute_input":"2022-08-13T10:09:33.951010Z","iopub.status.idle":"2022-08-13T10:09:34.259706Z","shell.execute_reply.started":"2022-08-13T10:09:33.950957Z","shell.execute_reply":"2022-08-13T10:09:34.258571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_path = keras.utils.get_file(\n \"cat_and_dog.jpg\",\n \"https://storage.googleapis.com/petbacker/images/blog/2017/dog-and-cat-cover.jpg\",\n)\n\ndisplay(Image(img_path))\n\n# Prepare image\nimg_array = preprocess_input(get_img_array(img_path, size=img_size))\n\n# Print what the two top predicted classes are\npreds = model.predict(img_array)\nprint(\"Predicted:\", decode_predictions(preds, top=2)[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-13T10:09:41.095351Z","iopub.execute_input":"2022-08-13T10:09:41.095730Z","iopub.status.idle":"2022-08-13T10:09:41.677001Z","shell.execute_reply.started":"2022-08-13T10:09:41.095697Z","shell.execute_reply":"2022-08-13T10:09:41.676055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nAs Thanos says **\"Reality can be whatever I want\"** while training GradCAM can be used to keep a check on if the CNN sees a Dogs as a dog and not as a carpet 😆😆","metadata":{}},{"cell_type":"code","source":"#We generate class activation heatmap for \"chow,\" the class index is 260\nheatmap = make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=260)\n\nsave_and_display_gradcam(img_path, heatmap)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-13T10:09:42.230765Z","iopub.execute_input":"2022-08-13T10:09:42.231796Z","iopub.status.idle":"2022-08-13T10:09:42.381660Z","shell.execute_reply.started":"2022-08-13T10:09:42.231713Z","shell.execute_reply":"2022-08-13T10:09:42.380688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#We generate class activation heatmap for \"egyptian cat,\" the class index is 285\nheatmap = make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=285)\n\nsave_and_display_gradcam(img_path, heatmap)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-13T10:09:43.700200Z","iopub.execute_input":"2022-08-13T10:09:43.700895Z","iopub.status.idle":"2022-08-13T10:09:43.848707Z","shell.execute_reply.started":"2022-08-13T10:09:43.700856Z","shell.execute_reply":"2022-08-13T10:09:43.847691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Grad-CAM can be used for weakly-supervised localization, i.e. determining the location of particular objects using a model that was trained only on whole-image labels rather than explicit location annotations.\n\nGrad-CAM can also be used for weakly-supervised segmentation, in which the model predicts all of the pixels that belong to particular objects, without requiring pixel-level labels for training:\n\n\n\n\nwe’ve hopefully provided some useful insights into and tools for convolutional neural network visual explanation using the Grad-CAM. With the help of cats & dogs, we’ve explored how a model differentiates classes. We’ve improved the Grad-CAM results from the final convolutional layer by considering heat-maps from all layers of the model.Machine learning is an iterative process where our models are never good enough. We hope that the techniques presented here will help provide more transparency and create more trust in models throughout the improvement process. At Forsight, we use Grad-CAM heat-maps to identify erroneous areas of emphasis by our models.","metadata":{}},{"cell_type":"markdown","source":"# TF-Explain : A Tensorflow 2.0 library for deep learning model interpretability.\n\nThe library is adapted to the Tensorflow 2.0 workflow, using tf.keras API as possible. It provides:\n\n- Heatmaps Visualizations & Gradients Analysis\n- Both off-training and tf.keras.Callback Usages\n- Tensorboard Integration\n\n\n\n## Why Interpretability?\n\nThe main challenge when working with deep neural networks is **to understand the behavior of trained networks** . It is crucial both to the scientist to debug and improve current model, and to the users **to help them to trust the method**. As a human, it is difficult to get feedback from a neural network. Interpretability has emerged in research to help tackling this:\n\n- Analysis of decisions over a validation set helps identify issues with the network (for instance bias in dataset, mislabeled data)\n- Heatmap visualizations are often appreciated by non-initiated users\n\nHowever, those methods are not well-integrated in deep learning workflow as of today.\n\n### The solution: off the shelf analysis tools for your tf.keras models\ntf-explain implements interpretability methods as Tensorflow 2.0 callbacks to ease neural networks’ understanding.\n\nThe library was built to offer a comprehensive list of interpretability methods, directly usable in your Tensorflow workflow\n\n","metadata":{}},{"cell_type":"markdown","source":"## There are two Methods for using tf-explain with can be represeted such as : \n- After the model is trained to check the explanation of model \n- During model trainig in the form of Callbacks in TensorFlow\n","metadata":{}},{"cell_type":"markdown","source":"## After Training the model","metadata":{}},{"cell_type":"code","source":"!pip install tf-explain\nfrom tf_explain.core.grad_cam import GradCAM\n\n# Load pretrained model or your own\nmodel = tf.keras.applications.vgg16.VGG16(weights=\"imagenet\", include_top=True)\nIMAGE_PATH = \"../input/dogs-cats-images/dataset/training_set/dogs/dog.100.jpg\"\n# Load a sample image (or multiple ones)\nimg = tf.keras.preprocessing.image.load_img(IMAGE_PATH, target_size=(224, 224))\nimg = tf.keras.preprocessing.image.img_to_array(img)\ndata = ([img], None)\n\n# Start explainer\nexplainer = GradCAM()\ngrid = explainer.explain(data, model, class_index=281) # 281 is the tabby cat index in ImageNet\n\nexplainer.save(grid, \".\", \"grad_cam.png\")\ndisplay(Image(IMAGE_PATH))\ndisplay(Image(\"grad_cam.png\"))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-13T10:11:10.400414Z","iopub.execute_input":"2022-08-13T10:11:10.401021Z","iopub.status.idle":"2022-08-13T10:11:21.965782Z","shell.execute_reply.started":"2022-08-13T10:11:10.400973Z","shell.execute_reply":"2022-08-13T10:11:21.964626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## While Training Model :\n\n![](https://media.giphy.com/media/3o84sq21TxDH6PyYms/giphy.gif)\n\n## Available Methods\n\n1. [Activations Visualization](#activations-visualization)\n1. [Vanilla Gradients](#vanilla-gradients)\n1. [Gradients*Inputs](#gradients-inputs)\n1. [Occlusion Sensitivity](#occlusion-sensitivity)\n1. [Grad CAM (Class Activation Maps)](#grad-cam)\n1. [SmoothGrad](#smoothgrad)\n1. [Integrated Gradients](#integrated-gradients)\n\n### Activations Visualization\n\n**Visualize how a given input comes out of a specific activation layer**\n\n\n```python\nfrom tensorflow import keras\nmodel = keras.models.load_model('./model1.h5')\nfrom tf_explain.callbacks.activations_visualization import ActivationsVisualizationCallback\noutput_dir = \"./outputs/\"\n# model = [...]\n\nmodel.compile(optimizer='adam',\n loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n metrics=['accuracy'])\n\nimg_path='./cats-vs-dogs/train_data/cat/cat.9999.jpg'\n\nimg = load_img(img_path, target_size=(150, 150)) \n# Convert ht image to Array of dimension (150,150,3)\nx = img_to_array(img) \nx = x.reshape((1,) + x.shape) \n# Rescale by 1/255\nx /= 255.0\n\ncallbacks = [\n ActivationsVisualizationCallback(\n validation_data=(x,1),\n layers_name=[\"max_pooling2d_4\"],\n output_dir=output_dir,\n ),\n]\n\nmodel.fit(train_data_gen, epochs=1,callbacks=callbacks)\n```","metadata":{"execution":{"iopub.status.busy":"2022-08-13T10:44:56.316835Z","iopub.execute_input":"2022-08-13T10:44:56.317918Z","iopub.status.idle":"2022-08-13T10:47:24.895084Z","shell.execute_reply.started":"2022-08-13T10:44:56.317869Z","shell.execute_reply":"2022-08-13T10:47:24.893984Z"}}},{"cell_type":"markdown","source":"**Note**: TensorBoard requires a running kernel, so its output will only be available in an editor session. \n\nThis is a static image of an interactive graphic. For the interactive version, fork your own copy of this kernel and open it in the editor.\n","metadata":{}},{"cell_type":"markdown","source":"### Vanilla Gradients\n\nVisualize gradients importance on input image\n\n```python\nfrom tf_explain.callbacks.vanilla_gradients import VanillaGradientsCallback\n\nmodel = [...]\n\ncallbacks = [\n VanillaGradientsCallback(\n validation_data=(x_val, y_val),\n class_index=0,\n output_dir=output_dir,\n ),\n]\n\nmodel.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)\n\n```\n\n\n### Gradients*Inputs\n\nVariant of [Vanilla Gradients](#vanilla-gradients) ponderating gradients with input values\n\n```python\nfrom tf_explain.callbacks.gradients_inputs import GradientsInputsCallback\n\nmodel = [...]\n\ncallbacks = [\n GradientsInputsCallback(\n validation_data=(x_val, y_val),\n class_index=0,\n output_dir=output_dir,\n ),\n]\n\nmodel.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)\n```\n\n\n\n### Occlusion Sensitivity\n\nVisualize how parts of the image affects neural network's confidence by occluding parts iteratively\n\n```python\nfrom tf_explain.callbacks.occlusion_sensitivity import OcclusionSensitivityCallback\n\nmodel = [...]\n\ncallbacks = [\n OcclusionSensitivityCallback(\n validation_data=(x_val, y_val),\n class_index=0,\n patch_size=4,\n output_dir=output_dir,\n ),\n]\n\nmodel.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)\n```\n\n\n### Grad CAM\n\nVisualize how parts of the image affects neural network's output by looking into the activation maps\n\nFrom [Grad-CAM: Visual Explanations from Deep Networks\nvia Gradient-based Localization](https://arxiv.org/abs/1610.02391)\n\n```python\nfrom tf_explain.callbacks.grad_cam import GradCAMCallback\n\nmodel = [...]\n\ncallbacks = [\n GradCAMCallback(\n validation_data=(x_val, y_val),\n class_index=0,\n output_dir=output_dir,\n )\n]\n\nmodel.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)\n```\n\n### SmoothGrad\n\nVisualize stabilized gradients on the inputs towards the decision\n\nFrom [SmoothGrad: removing noise by adding noise](https://arxiv.org/abs/1706.03825)\n\n```python\nfrom tf_explain.callbacks.smoothgrad import SmoothGradCallback\n\nmodel = [...]\n\ncallbacks = [\n SmoothGradCallback(\n validation_data=(x_val, y_val),\n class_index=0,\n num_samples=20,\n noise=1.,\n output_dir=output_dir,\n )\n]\n\nmodel.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)\n```\n\n\n### Integrated Gradients\n\nVisualize an average of the gradients along the construction of the input towards the decision\n\nFrom [Axiomatic Attribution for Deep Networks](https://arxiv.org/pdf/1703.01365.pdf)\n\n```python\nfrom tf_explain.callbacks.integrated_gradients import IntegratedGradientsCallback\n\nmodel = [...]\n\ncallbacks = [\n IntegratedGradientsCallback(\n validation_data=(x_val, y_val),\n class_index=0,\n n_steps=20,\n output_dir=output_dir,\n )\n]\n\nmodel.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)\n```\n\n","metadata":{}},{"cell_type":"markdown","source":"# Summary\n\n- Grad-CAM is a popular technique for creating a class-specific heatmap based off of a particular input image, a trained CNN, and a chosen class of interest.\n- Grad-CAM is closely related to CAM.\n- Grad-CAM can be calculated on any CNN architecture as long the layers are differentiable.\n- Grad-CAM has been used for weakly-supervised localization and weakly-supervised segmentation.\n- Recent work identifies a fundamental problem with Grad-CAM: sometimes Grad-CAM highlights regions the model did not actually use. Thus, for model explanation, HiResCAM should be used instead of Grad-CAM.\n\n![](https://media.giphy.com/media/jd6TVgsph6w7e/giphy.gif)\n\n# !!!!!! Please Do comment your views on the same ","metadata":{}}]}