{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.0.5"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":5441,"databundleVersionId":38425,"sourceType":"competition"}],"dockerImageVersionId":30618,"isInternetEnabled":true,"language":"r","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Table of content\n\n***\n\n#### [Section 1: Loading required libraries](#1)\n\n#### [Section 2: Loading DogsVScats dataset](#2)\n\n#### [Section 3: Image processing](#3)\n\n#### [Section 4: Loading Xception model](#4)\n\n#### [Section 5: Building a Grad-Cam function](#5)\n\n#### [Section 6: Visualizing results](#6)","metadata":{}},{"cell_type":"markdown","source":"# Disclaimer\n\n#### The following code is attempt to adapt the code provided by https://www.kaggle.com/code/tentotheminus9/where-is-xception-looking-grad-cam-via-r/notebook\n\n#### See also the original book: https://www.manning.com/books/deep-learning-with-r","metadata":{}},{"cell_type":"markdown","source":"# Loading libraries <a class = \"anchor\" id = 1><a/>","metadata":{}},{"cell_type":"code","source":"suppressPackageStartupMessages({\n\n    library(keras)\n    library(tensorflow)\n    library(magick)\n    library(viridis)\n})","metadata":{"execution":{"iopub.status.busy":"2024-06-21T15:38:06.467118Z","iopub.execute_input":"2024-06-21T15:38:06.525150Z","iopub.status.idle":"2024-06-21T15:38:06.538358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading data <a class = \"anchor\" id = 2><a/>","metadata":{}},{"cell_type":"code","source":"setwd(\"/kaggle/input/dogs-vs-cats-redux-kernels-edition/\")\n\nfor (index in 2:length(dir())) {\n    file <- paste0(getwd(), sep = \"/\", dir()[index])\n    unzip(zipfile = file, exdir = \"/kaggle/working/\")\n}","metadata":{"execution":{"iopub.status.busy":"2024-06-21T15:38:09.615984Z","iopub.execute_input":"2024-06-21T15:38:09.617624Z","iopub.status.idle":"2024-06-21T15:38:23.087825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Making a sample of 9 images from \"test\" directory\n\n#### Note: You must provide full image path","metadata":{}},{"cell_type":"code","source":"setwd(\"/kaggle/working/\")\n\ntest_images <- sample(paste(getwd(), \"test\", dir(\"test\"), sep = \"/\"), 9)\n\ntest_images","metadata":{"execution":{"iopub.status.busy":"2024-06-21T17:52:28.712817Z","iopub.execute_input":"2024-06-21T17:52:28.714624Z","iopub.status.idle":"2024-06-21T17:52:28.833632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Image loading and processing <a class = \"anchor\" id = 3><a/>\n\n#### Note: Here I change values to be in [-1, 1] range","metadata":{}},{"cell_type":"code","source":"load_and_process <- function(img) {\n        img <- img %>%\n                tf$io$read_file() %>%\n                tf$io$decode_jpeg() %>%\n                tf$image$resize(as.integer(c(299, 299))) %>%\n                tf$expand_dims(0L)\n    \n        img <- (img / 127.5) - 1\n    \n        return(img)\n}","metadata":{"execution":{"iopub.status.busy":"2024-06-21T19:34:44.412852Z","iopub.execute_input":"2024-06-21T19:34:44.414518Z","iopub.status.idle":"2024-06-21T19:34:44.427637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading Xception as a baseline model for illustration <a class = \"anchor\" id = 4><a/>\n\n#### Note #1: You can load any model","metadata":{}},{"cell_type":"code","source":"model_build <- application_xception(\n    weights = \"imagenet\",\n    include_top = TRUE,\n    input_shape = c(299, 299, 3),\n    classifier_activation = \"softmax\"\n)","metadata":{"execution":{"iopub.status.busy":"2024-06-21T19:34:50.197889Z","iopub.execute_input":"2024-06-21T19:34:50.199547Z","iopub.status.idle":"2024-06-21T19:34:51.978288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Establish a keras model with Xception model input/output and \"block14_sepconv2_act\" layer\n\n#### As far as I understand we need it in order to link variables with operations for GradientTape (reference: https://tensorflow.rstudio.com/guides/tensorflow/autodiff#gradients-with-respect-to-a-model)","metadata":{}},{"cell_type":"code","source":"grad_model <- keras_model(\n    model_build$inputs, c(model_build$get_layer(\"block14_sepconv2_act\")$output, model_build$output)\n)","metadata":{"execution":{"iopub.status.busy":"2024-06-21T19:34:53.529197Z","iopub.execute_input":"2024-06-21T19:34:53.530989Z","iopub.status.idle":"2024-06-21T19:34:53.562875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building a Grad-Cam body function <a class = \"anchor\" id = 5><a/>\n    \n#### Note #1: As for the reference code in Disclaimer, he/she uses `k_gradients` function for differentiation. but as for TF2 this is no longer the option, we have to use `GradientTape`\n    \n#### Note #2: You will not be able to produce results if you skipped the previous step with keras model build, your `grads` output will be NULL\n    \n#### Note #3: Here I create a for loop in order to multiply arrays, but I got the same results in VSCode with just `%*%` for matrix multiplication (I have no idea why it does not work in Kaggle notebook)","metadata":{}},{"cell_type":"code","source":"make_gradcam_heatmap <- function(img_array, y, test_images) {\n\n    with(\n        tf$GradientTape() %as% tape,\n        {\n            c(last_conv_layer_output, preds) %<-% grad_model(img_array)\n\n            pred_index <- tf$argmax(preds[1, ])\n\n            class_channel <- preds[, pred_index]\n        }\n    )\n\n    grads <- tape$gradient(class_channel, last_conv_layer_output)\n    \n    pooled_grads <- k_mean(grads, axis = c(1, 2, 3))\n    \n    last_conv_layer_output <- last_conv_layer_output[1, , , ]\n    \n    last_conv_layer_output <- as.array(last_conv_layer_output)\n    \n    pooled_grads <- as.array(pooled_grads)\n\n    for (i in 1:dim(last_conv_layer_output)[3]) {                                                         \n        last_conv_layer_output[,,i] <-\n                last_conv_layer_output[,,i] * pooled_grads[i]\n    }\n    \n    heatmap <- apply(last_conv_layer_output, c(1,2), mean) \n    \n    heatmap <- pmax(heatmap, 0)\n    \n    heatmap <- heatmap / max(heatmap) \n\n    write_heatmap <- function(heatmap, filename, width = 299, height = 299,\n                              bg = \"white\", col = terrain.colors(12)) {\n        png(filename, width = width, height = height, bg, units = \"px\", pointsize = 12)\n\n        op <- par(mar = c(0, 0, 0, 0))\n\n        on.exit(\n            {\n                par(op)\n                dev.off()\n            },\n            add = TRUE\n        )\n\n        rotate <- function(x) t(apply(x, 2, rev))\n\n        image(rotate(heatmap), axes = FALSE, asp = 1, col = col)\n    }\n\n    write_heatmap(heatmap, paste(\"img_heatmap_\", y, \".png\", sep = \"\"))\n\n    image <- image_read(test_images)\n\n    info <- image_info(image)\n\n    geometry <- sprintf(\"%dx%d!\", info$width, info$height)\n\n    pal <- col2rgb(viridis(20), alpha = TRUE)\n    alpha <- floor(seq(0, 255, length = ncol(pal)))\n    pal_col <- rgb(t(pal), alpha = alpha, maxColorValue = 255)\n\n    write_heatmap(heatmap, paste(\"img_overlay_\", y, \".png\", sep = \"\"),\n    width = 14, height = 14, bg = NA, col = pal_col)\n\n    return(geometry)\n}","metadata":{"execution":{"iopub.status.busy":"2024-06-21T20:03:54.755468Z","iopub.execute_input":"2024-06-21T20:03:54.757197Z","iopub.status.idle":"2024-06-21T20:03:54.771863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Let's visualize our results <a class = \"anchor\" id = 6><a/>","metadata":{}},{"cell_type":"code","source":"par(mfrow = c(3, 3), mar = c(rep(0.1, 4)))\n\ny <- 1\n\nwhile (y <= length(test_images)) {\n    image_temp <- load_and_process(test_images[y])\n\n    geom_temp <- make_gradcam_heatmap(image_temp, y, test_images[y])\n\n    image_temp <- image_read(paste(\"img_overlay_\", y, \".png\", sep = \"\"))\n\n    image_temp <- image_resize(image_temp, geom_temp, filter = \"quadratic\")\n\n    image_temp <- image_composite(\n        image = image_temp,\n        composite_image = image_convert(image_read(test_images[y])),\n        operator = \"blend\",\n        compose_args = \"20\"\n    )\n\n    plot(image_temp)\n    y <- y + 1\n}","metadata":{"execution":{"iopub.status.busy":"2024-06-21T20:03:57.503715Z","iopub.execute_input":"2024-06-21T20:03:57.505321Z","iopub.status.idle":"2024-06-21T20:04:02.917122Z"},"trusted":true},"execution_count":null,"outputs":[]}]}