{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Dog-Cat Classifier (VGG) + GradCAM with TF 2.0\n\nIn this Notebook:\n* Observe CAM with ResNet50 trained model on ImageNet\n* Retrain output layer of ResNet50 model with Dog vs Cat data\n* Add Fully connected layers before output layer and train\n* Observe GradCAM with the re-trained models\n\nFollowing is the architecture of ResNet50. It does not have fully-connected layers (FC) between pooling layer và output layers. Therefore, to check if GradCAM is really effective with the presence of FC layers, I will train a model with additional FC layers to ResNet50 and see how GradCAM works.\n\nVisualization with Ipython Widget that can be run locally: https://github.com/nguyenhoa93/GradCAM_and_GuidedGradCAM_tf2\n\n**Image source:** [Qingge Ji et al.](https://www.researchgate.net/figure/Left-ResNet50-architecture-Blocks-with-dotted-line-represents-modules-that-might-be_fig3_331364877)\n![img](https://i.imgur.com/jRHPctT.png)\n\n\n# <a id = \"toc\"></a>\n# Table of contents\n[1. Data preparation](#dataprep)\n\n[2. Data exploration](#explore)\n\n[3. GradCAM & GuidedBackProp Class definition](#gradcam)\n* [GradCAM](#gradcam1)\n* [GuidedBackprop](#gb)\n\n[4. Observe GradCAM & Guided GradCAM with ResNet50 trained on ImageNet](#observe1)\n\n[5. Re-train output layer of ResNet50 model on dogs and cats data](#retrain1)\n* [Data generator](#generator)\n* [Model](#model1)\n* [Compile](#compile1)\n* [Train](#train1)\n* [Observe GradCAM & Guided GradCAM](#gradcam1)\n\n[6. Add FC layers and train](#retrain2)\n* [Training](#train2)\n* [Observe GradCAM & Guided GradCAM](#gradcam2)\n\n[7. References](#ref)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install --upgrade pip\n!pip install tensorflow==2.2.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"import numpy as np\nimport os\nimport zipfile\nimport pandas as pd\nimport random\nimport matplotlib.pyplot as plt\nimport cv2\n\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow.python.framework import ops\nfrom tensorflow.keras.applications import ResNet50V2\nfrom tensorflow.keras.models import Model, Sequential\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras.layers import Dense, Flatten, Activation\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array, ImageDataGenerator\nfrom tensorflow.keras.applications.resnet_v2 import preprocess_input\nfrom tensorflow.keras.applications.imagenet_utils import decode_predictions\nfrom tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint\nfrom tensorflow.keras import backend as K\nprint(\"Tensorflow version: \", tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Path and parameters\nIMAGE_DIR = \"../working/train\"\nH = 224\nW = 224\nepochs = 5\nbatch_size = 100\nSEED = 42","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"dataprep\"></a>\n## Data preparation\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Unzip data\nwith zipfile.ZipFile(\"../input/dogs-vs-cats/train.zip\",\"r\") as z:\n    z.extractall(\".\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Create dataframe\nfilenames = os.listdir(IMAGE_DIR)\nlabels = [x.split(\".\")[0] for x in filenames]\ndf = pd.DataFrame({\"filename\": filenames, \"label\": labels})\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train - Test list","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df, val_df = train_test_split(df, test_size=0.2, random_state = SEED, stratify = df.label)\ntrain_df.sample(frac=1, random_state=SEED)\ntrain_df = train_df.reset_index(drop=True)\nval_df = val_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train data distribution\ntrain_df.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Validation data distribution\nval_df.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"explore\"></a>\n# Data exploration\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dogs = list(df[df.label==\"dog\"].filename)\ncats = list(df[df.label==\"cat\"].filename)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Adapted with serveral modifications from https://www.kaggle.com/serkanpeldek/keras-cnn-transfer-learnings-on-cats-dogs-dataset\n\ndef get_side(img, side_type, n = 5):\n    h, w, c = img.shape\n    if side_type == \"horizontal\":\n        return np.ones((h,n,c))\n    return np.ones((n,w,c))\n\ndef show_gallery(im_ls,n=5, shuffle=True):\n    images = []\n    vertical_images = []\n    if shuffle:\n        random.shuffle(im_ls)\n    vertical_images = []\n    for i in range(n*n):\n        img = load_img(os.path.join(IMAGE_DIR,im_ls[i]), target_size=(W,H))\n        img = img_to_array(img)\n        hside = get_side(img,side_type=\"horizontal\")\n        images.append(img)\n        images.append(hside)\n        \n        if (i+1) % n == 0:\n            himage=np.hstack((images))\n            vside = get_side(himage, side_type=\"vertical\")\n            vertical_images.append(himage)\n            vertical_images.append(vside)\n            \n            images = []\n        \n    gallery = np.vstack((vertical_images))\n    plt.figure(figsize=(20,20))\n    plt.axis(\"off\")\n    plt.imshow(gallery.astype(np.uint8))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show dogs images\nshow_gallery(dogs, n=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show cat images\nshow_gallery(cats, n=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"gradcam\"></a>\n# GradCAM & GuidedGradCAM class defnition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"gradcam1\"></a>\n## GradCAM\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class GradCAM:\n    # Adapted with some modification from https://www.pyimagesearch.com/2020/03/09/grad-cam-visualize-class-activation-maps-with-keras-tensorflow-and-deep-learning/\n    def __init__(self, model, layerName=None):\n        \"\"\"\n        model: pre-softmax layer (logit layer)\n        \"\"\"\n        self.model = model\n        self.layerName = layerName\n            \n        if self.layerName == None:\n            self.layerName = self.find_target_layer()\n    \n    def find_target_layer(self):\n        for layer in reversed(self.model.layers):\n            if len(layer.output_shape) == 4:\n                return layer.name\n        raise ValueError(\"Could not find 4D layer. Cannot apply GradCAM\")\n            \n    def compute_heatmap(self, image, classIdx, upsample_size, eps=1e-5):\n        gradModel = Model(\n            inputs = [self.model.inputs],\n            outputs = [self.model.get_layer(self.layerName).output, self.model.output]\n        )\n        # record operations for automatic differentiation\n        \n        with tf.GradientTape() as tape:\n            inputs = tf.cast(image, tf.float32)\n            (convOuts, preds) = gradModel(inputs) # preds after softmax\n            loss = preds[:,classIdx]\n        \n        # compute gradients with automatic differentiation\n        grads = tape.gradient(loss, convOuts)\n        # discard batch\n        convOuts = convOuts[0]\n        grads = grads[0]\n        norm_grads = tf.divide(grads, tf.reduce_mean(tf.square(grads)) + tf.constant(eps))\n        \n        # compute weights\n        weights = tf.reduce_mean(norm_grads, axis=(0,1))\n        cam = tf.reduce_sum(tf.multiply(weights, convOuts), axis=-1)\n        \n        # Apply reLU\n        cam = np.maximum(cam, 0)\n        cam = cam/np.max(cam)\n        cam = cv2.resize(cam, upsample_size,interpolation=cv2.INTER_LINEAR)\n        \n        # convert to 3D\n        cam3 = np.expand_dims(cam, axis=2)\n        cam3 = np.tile(cam3, [1,1,3])\n        \n        return cam3\n    \ndef overlay_gradCAM(img, cam3):\n    cam3 = np.uint8(255*cam3)\n    cam3 = cv2.applyColorMap(cam3, cv2.COLORMAP_JET)\n    \n    new_img = 0.3*cam3 + 0.5*img\n    \n    return (new_img*255.0/new_img.max()).astype(\"uint8\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"gb\"></a>\n## GuidedBackprop\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"@tf.custom_gradient\ndef guidedRelu(x):\n    def grad(dy):\n        return tf.cast(dy>0,\"float32\") * tf.cast(x>0, \"float32\") * dy\n    return tf.nn.relu(x), grad\n\n# Reference: https://github.com/eclique/keras-gradcam with adaption to tensorflow 2.0  \nclass GuidedBackprop:\n    def __init__(self,model, layerName=None):\n        self.model = model\n        self.layerName = layerName\n        self.gbModel = self.build_guided_model()\n        \n        if self.layerName == None:\n            self.layerName = self.find_target_layer()\n\n    def find_target_layer(self):\n        for layer in reversed(self.model.layers):\n            if len(layer.output_shape) == 4:\n                return layer.name\n        raise ValueError(\"Could not find 4D layer. Cannot apply Guided Backpropagation\")\n\n    def build_guided_model(self):\n        gbModel = Model(\n            inputs = [self.model.inputs],\n            outputs = [self.model.get_layer(self.layerName).output]\n        )\n        layer_dict = [layer for layer in gbModel.layers[1:] if hasattr(layer,\"activation\")]\n        for layer in layer_dict:\n            if layer.activation == tf.keras.activations.relu:\n                layer.activation = guidedRelu\n        \n        return gbModel\n    \n    def guided_backprop(self, images, upsample_size):\n        \"\"\"Guided Backpropagation method for visualizing input saliency.\"\"\"\n        with tf.GradientTape() as tape:\n            inputs = tf.cast(images, tf.float32)\n            tape.watch(inputs)\n            outputs = self.gbModel(inputs)\n\n        grads = tape.gradient(outputs, inputs)[0]\n\n        saliency = cv2.resize(np.asarray(grads), upsample_size)\n\n        return saliency\n\ndef deprocess_image(x):\n    \"\"\"Same normalization as in:\n    https://github.com/fchollet/keras/blob/master/examples/conv_filter_visualization.py\n    \"\"\"\n    # normalize tensor: center on 0., ensure std is 0.25\n    x = x.copy()\n    x -= x.mean()\n    x /= (x.std() + K.epsilon())\n    x *= 0.25\n\n    # clip to [0, 1]\n    x += 0.5\n    x = np.clip(x, 0, 1)\n\n    # convert to RGB array\n    x *= 255\n    if K.image_data_format() == 'channels_first':\n        x = x.transpose((1, 2, 0))\n    x = np.clip(x, 0, 255).astype('uint8')\n    return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"vis\"></a>\n## Visualization function\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_gradCAMs(model, gradCAM, GuidedBP, im_ls, n=3, decode={}):\n    \"\"\"\n    model: softmax layer\n    \"\"\"\n    random.shuffle(im_ls)\n    plt.subplots(figsize=(30, 10*n))\n    k=1\n    for i in range(n):\n        img = cv2.imread(os.path.join(IMAGE_DIR,im_ls[i]))\n        upsample_size = (img.shape[1],img.shape[0])\n        if (i+1) == len(df):\n            break\n        # Show original image\n        plt.subplot(n,3,k)\n        plt.imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB))\n        plt.title(\"Filename: {}\".format(im_ls[i]), fontsize=20)\n        plt.axis(\"off\")\n        # Show overlayed grad\n        plt.subplot(n,3,k+1)\n        im = img_to_array(load_img(os.path.join(IMAGE_DIR,im_ls[i]), target_size=(W,H)))\n        x = np.expand_dims(im, axis=0)\n        x = preprocess_input(x)\n        preds = model.predict(x)\n        idx = preds.argmax()\n        if len(decode)==0:\n            res = decode_predictions(preds)[0][0][1:]\n        else:\n            res = [decode[idx],preds.max()]\n        cam3 = gradCAM.compute_heatmap(image=x, classIdx=idx, upsample_size=upsample_size)\n        new_img = overlay_gradCAM(img, cam3)\n        new_img = cv2.cvtColor(new_img, cv2.COLOR_BGR2RGB)\n        plt.imshow(new_img)\n        plt.title(\"GradCAM - Pred: {}. Prob: {}\".format(res[0],res[1]), fontsize=20)\n        plt.axis(\"off\")\n        \n        # Show guided GradCAM\n        plt.subplot(n,3,k+2)\n        gb = GuidedBP.guided_backprop(x, upsample_size)\n        guided_gradcam = deprocess_image(gb*cam3)\n        guided_gradcam = cv2.cvtColor(guided_gradcam, cv2.COLOR_BGR2RGB)\n        plt.imshow(guided_gradcam)\n        plt.title(\"Guided GradCAM\", fontsize=20)\n        plt.axis(\"off\")\n        \n        k += 3\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"observe1\"></a>\n# Observe GradCAM with ResNet50 trained model on ImageNet\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"resnet50_logit = ResNet50V2(include_top=True, weights='imagenet', classifier_activation=None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"resnet50 = ResNet50V2(include_top=True, weights='imagenet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gradCAM = GradCAM(model=resnet50_logit, layerName=\"conv5_block3_out\")\nguidedBP = GuidedBackprop(model=resnet50,layerName=\"conv5_block3_out\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### GradCAM for dog images\n\nNote:\n* These images may contain other objects rather than dogs so the network may focus on those objects rather than the animal.\n* The model may have wrong prediction.\n\nLook at the prediction result above the GradCAM to know which class it is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(resnet50, gradCAM,guidedBP,dogs, n=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### GradCAM for cat images\nNote:\n* These images may contain other objects rather than cats so the network may focus on those objects rather than the animal.\n* The model may have wrong prediction.\n\nLook at the prediction result above the GradCAM to know which class it is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(resnet50, gradCAM, guidedBP,cats, n=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"retrain1\"></a>\n# Re-train output layer of ResNet50 model on dogs and cats data\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"generator\"></a>\n## Data generator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train generator\ntrain_datagen = ImageDataGenerator(preprocessing_function = preprocess_input)\ntrain_generator = train_datagen.flow_from_dataframe(train_df, IMAGE_DIR,x_col=\"filename\", y_col=\"label\",\n                                                    target_size=(W,H), class_mode=\"categorical\",\n                                                   batch_size=batch_size, shuffle=True, seed=SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Validation generator\nval_datagen = ImageDataGenerator(preprocessing_function= preprocess_input)\nval_generator = val_datagen.flow_from_dataframe(val_df, IMAGE_DIR, x_col=\"filename\", y_col=\"label\",\n                                               target_size=(W,H), class_mode=\"categorical\",\n                                                batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at how data generator augment the data\nex_df = train_df.sample(n=15).reset_index(drop=True)\nex_gen = train_datagen.flow_from_dataframe(ex_df,IMAGE_DIR,x_col=\"filename\", y_col=\"label\",\n                                           target_size=(W,H), class_mode=\"categorical\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,15))\nfor i in range(0, 9):\n    plt.subplot(5,3,i+1)\n    for x, y in ex_gen:\n        im = x[0]\n        plt.imshow(cv2.cvtColor(im, cv2.COLOR_BGR2RGB))\n        plt.axis(\"off\")\n        break\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"model1\"></a>\n## Model","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"resnet = ResNet50V2(include_top=False, pooling=\"avg\", weights='imagenet')\nfor layer in resnet.layers:\n    layer.trainable=False\n\nlogits = Dense(2)(resnet.layers[-1].output)\noutput = Activation('softmax')(logits)\nmodel = Model(resnet.input, output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"compile1\"></a>\n## Compile","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)\nmodel.compile(optimizer=sgd, loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"train1\"></a>\n## Train","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"earlystoper = EarlyStopping(monitor=\"val_loss\", patience=3)\ncheckpointer = ModelCheckpoint(filepath=\"../working/resnet50best.hdf5\", monitor='val_loss', save_best_only=True, mode='auto')\ncallbacks = [earlystoper, checkpointer]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(\n    train_generator, \n    epochs=epochs,\n    validation_data=val_generator,\n    validation_steps=20,\n    steps_per_epoch=20,\n    callbacks=callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights(\"../working/resnet50best.hdf5\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training metrics","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(1, figsize = (15,8)) \n    \nplt.subplot(221)  \nplt.plot(history.history['accuracy'])  \nplt.plot(history.history['val_accuracy'])  \nplt.title('model accuracy')  \nplt.ylabel('accuracy')  \nplt.xlabel('epoch')  \nplt.legend(['train', 'valid']) \n    \nplt.subplot(222)  \nplt.plot(history.history['loss'])  \nplt.plot(history.history['val_loss'])  \nplt.title('model loss')  \nplt.ylabel('loss')  \nplt.xlabel('epoch')  \nplt.legend(['train', 'valid']) \n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"gradcam1\"></a>\n## Observe GradCAM with the re-trained model\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model_logit = Model(model.input,model.layers[-2].output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"retrained_gradCAM = GradCAM(model=model_logit, layerName=\"conv5_block3_out\")\nretrained_guidedBP = GuidedBackprop(model=model, layerName=\"conv5_block3_out\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_gen = ImageDataGenerator(preprocessing_function=preprocess_input)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_generator = data_gen.flow_from_dataframe(val_df, IMAGE_DIR, x_col=\"filename\",\n                                               target_size=(W,H), class_mode=None,\n                                                batch_size=1, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = model.predict_generator(test_generator, steps = len(test_generator), verbose = 1)\npred_indices = np.argmax(pred,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = val_df.copy()\nresults[\"pred\"] = pred_indices\ntrue_dogs = list(results[(results.label == \"dog\") & (results.pred ==1)].filename)\ntrue_cats = list(results[(results.label == \"cat\") & (results.pred ==0)].filename)\nwrong_class = [x for x in results.filename if x not in (true_cats+true_dogs)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dogs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(model, retrained_gradCAM,retrained_guidedBP,true_dogs, n=5, decode={0:\"cat\", 1:\"dog\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Cats","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(model, retrained_gradCAM,retrained_guidedBP,true_cats, n=5, decode={0:\"cat\", 1:\"dog\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Wrong classes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(wrong_class)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(model, retrained_gradCAM,retrained_guidedBP,wrong_class, n=5, decode={0:\"cat\", 1:\"dog\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"retrain2\"></a>\n# Add FC layers and train\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id = \"train2\"></a>\n## Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"resnet = ResNet50V2(include_top=False, pooling=\"avg\", weights='imagenet')\nfor layer in resnet.layers:\n    layer.trainable=False\n    \nfc1 = Dense(100)(resnet.layers[-1].output)\nfc2 = Dense(100)(fc1)\nlogits = Dense(2)(fc2)\noutput = Activation('softmax')(logits)\nmodel_with_fc = Model(resnet.input, output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)\nmodel_with_fc.compile(optimizer=sgd, loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"earlystoper = EarlyStopping(monitor=\"val_loss\", patience=3)\ncheckpointer = ModelCheckpoint(filepath=\"../working/resnet50fcbest.hdf5\", monitor='val_loss', save_best_only=True, mode='auto')\ncallbacks = [earlystoper, checkpointer]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model_with_fc.fit_generator(\n    train_generator, \n    epochs=5,\n    validation_data=val_generator,\n    validation_steps=20,\n    steps_per_epoch=20,\n    callbacks=callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_with_fc.load_weights(\"../working/resnet50fcbest.hdf5\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(1, figsize = (15,8)) \n    \nplt.subplot(221)  \nplt.plot(history.history['accuracy'])  \nplt.plot(history.history['val_accuracy'])  \nplt.title('model accuracy')  \nplt.ylabel('accuracy')  \nplt.xlabel('epoch')  \nplt.legend(['train', 'valid']) \n    \nplt.subplot(222)  \nplt.plot(history.history['loss'])  \nplt.plot(history.history['val_loss'])  \nplt.title('model loss')  \nplt.ylabel('loss')  \nplt.xlabel('epoch')  \nplt.legend(['train', 'valid']) \n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"gradcam2\"></a>\n## Observe GradCAM\n[Go back to Table of Contents](#toc)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fc_logit = Model(model_with_fc.input,model_with_fc.layers[-2].output)\nfctrained_gradCAM = GradCAM(model=model_fc_logit, layerName=\"conv5_block3_out\")\nfctrained_guidedBP = GuidedBackprop(model=model_with_fc, layerName=\"conv5_block3_out\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_generator = data_gen.flow_from_dataframe(val_df, IMAGE_DIR, x_col=\"filename\",\n                                               target_size=(W,H), class_mode=None,\n                                                batch_size=1, shuffle=False)\npred = model_with_fc.predict_generator(test_generator, steps = len(test_generator), verbose = 1)\npred_indices = np.argmax(pred,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = val_df.copy()\nresults[\"pred\"] = pred_indices\ntrue_dogs = list(results[(results.label == \"dog\") & (results.pred ==1)].filename)\ntrue_cats = list(results[(results.label == \"cat\") & (results.pred ==0)].filename)\nwrong_class = [x for x in results.filename if x not in (true_cats+true_dogs)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dogs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(model_with_fc, fctrained_gradCAM, fctrained_guidedBP,true_dogs, n=5, decode={0:\"cat\", 1:\"dog\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Cats","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(model_with_fc, fctrained_gradCAM, fctrained_guidedBP, true_cats, n=5, decode={0:\"cat\", 1:\"dog\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Wrong classes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(wrong_class)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gradCAMs(model_with_fc, fctrained_gradCAM, fctrained_guidedBP, wrong_class, n=5, decode={0:\"cat\", 1:\"dog\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"ref\"></a>\n# References\n[Go back to Table of Contents](#toc)\n\n1. https://github.com/eclique/keras-gradcam\n2. https://www.pyimagesearch.com/2020/03/09/grad-cam-visualize-class-activation-maps-with-keras-tensorflow-and-deep-learning/\n3. https://github.com/jacobgil/keras-grad-cam\n4. https://arxiv.org/abs/1610.02391","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}