{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cats vs Dogs - The Final Showdown\n\n[Dogs vs. Cats Redux: Kernels Edition](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition) is a `Kaggle`'s competition where using the dataset you have to create `classifier` that can classify images as `dog` or `cat`."},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nimport os\nfrom os import listdir\nfrom os.path import isfile, join\n\nimport json\nimport zipfile\nimport itertools\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport cv2\n\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import Callback\nfrom tensorflow.keras.optimizers import Adam, RMSprop\nfrom tensorflow.keras.applications import VGG16, InceptionV3\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout\nfrom tensorflow.keras.metrics import AUC, Precision, Recall\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 🏂 Getting the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extracting all of the zip files\n\n# Training dataset\nwith zipfile.ZipFile('../input/dogs-vs-cats-redux-kernels-edition/train.zip', 'r') as z:\n     z.extractall('../data')\n    \n# Testing dataset\nwith zipfile.ZipFile('../input/dogs-vs-cats-redux-kernels-edition/test.zip', 'r') as z:\n    z.extractall('../data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../data')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`create_df` will return a shuffled `pd.DataFrame` which will have two columns namely `label` and `img_path`(which will have individual img path).\n\nAlso `Cat = 0, Dog = 1` (labels info)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cat = 0, Dog = 1\ndef create_df():\n    path = '../data/train'\n    dataset = []\n    for f in listdir(path):\n        if isfile(join(path, f)):\n            if 'cat' in f:\n                dataset.append({'label': 0, 'img_path': join(path, f)})\n            elif 'dog' in f:\n                dataset.append({'label': 1, 'img_path': join(path, f)})\n\n    df = pd.DataFrame(dataset)\n\n    # shuffling the dataframe and resetting the index\n    df = df.sample(frac=1).reset_index(drop=True)\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting the dataset\ndf = create_df()\ndf.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting labels dtype to str\n# to use class_mode='binary' in ImageDataGenerators\ndf.label = df.label.astype('str')\ndf.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## 🤸‍♂️ Data preparation"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Dataset size: {len(df)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# See if dataset is balanced or not\n\nplt.figure(figsize=(8, 4))\nplt.title('Number of classes')\ng = sns.countplot(df.label, palette='icefire')\ng.set(xlabel='Classes', ylabel='Count')\nsns.despine(offset=5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://media.giphy.com/media/PlaOtXOHSpKtBwUw5X/giphy.gif)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Splitting the training and validation sets\nx_train, x_val, y_train, y_val = train_test_split(df.img_path, df.label, test_size=0.1, random_state=0)\n\n# Splitting the remaining training set into training and test sets\n# This set is different from the set ==> /kaggle/working/dogs-vs-cats/test/\nx_train, x_test, y_train, y_test = train_test_split(x_train, y_train, test_size=0.1, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting the above pandas series to dataframe\ntrain_df = pd.DataFrame({ 'label': y_train, 'img_path': x_train }).reset_index(drop=True)\nval_df   = pd.DataFrame({ 'label': y_val, 'img_path': x_val }).reset_index(drop=True)\ntest_df  = pd.DataFrame({ 'label': y_test, 'img_path': x_test }).reset_index(drop=True)\n\n# Printing the sets size\nprint(f'Training set size: {len(train_df)}')\nprint(f'Validation set size: {len(val_df)}')\nprint(f'Testing set size: {len(test_df)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Looking at first 25 training examples\n\nplt.figure(figsize=(15, 15))\nfor i, _img_path in enumerate(x_train[:25]):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.title('')\n    img = load_img(_img_path)\n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Global variables\n\nIMG_HEIGHT = 150\nIMG_WIDTH  = 150\nCHANNELS   = 3\n\nBATCH_SIZE = 16","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`process_images` function is used here for processing test data `(./dogs-vs-cats/test/test -- directory)`."},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_images(list_of_images):\n    x = []  # holds images\n    y = []  # hold labels\n    \n    for image in list_of_images:\n        x.append(\n            cv2.resize(\n                cv2.imread(image, cv2.IMREAD_COLOR),\n                (IMG_HEIGHT, IMG_WIDTH),\n                interpolation=cv2.INTER_CUBIC\n            )\n        )\n        \n        if 'dog' in image:\n            y.append(1)\n        if 'cat' in image:\n            y.append(0)\n    \n    return x, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### 🤼 Data augmentation\n\n![](https://media.giphy.com/media/10TB6QfNrahdhS/giphy.gif)"},{"metadata":{},"cell_type":"markdown","source":"**Instantiating ImageDataGenerators**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n    rescale=1./255, \n    rotation_range=10, \n    width_shift_range=0.2, \n    height_shift_range=0.2, \n    horizontal_flip=True,\n    zoom_range = 0.1,\n)\n\nval_datagen = ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`flow_from_dataframe` method of `ImageDataGenerator`s will take the path of the images and will `load`, `normalize` and `augment` them on fly while training."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = train_datagen.flow_from_dataframe(\n    train_df,\n    x_col='img_path',\n    y_col='label',\n    target_size=(IMG_HEIGHT, IMG_WIDTH),\n    batch_size=BATCH_SIZE,\n    shuffle=True,\n    class_mode='binary'\n)\n\nval_generator = val_datagen.flow_from_dataframe(\n    val_df,\n    x_col='img_path',\n    y_col='label',\n    target_size=(IMG_HEIGHT, IMG_WIDTH),\n    batch_size=BATCH_SIZE,\n    # shuffle=True,\n    class_mode='binary'\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 🏄‍♂️ Modelling"},{"metadata":{},"cell_type":"markdown","source":"Using the `VGG16` CNN architecture and training it for our classifier. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_VGG16_model(input_shape):\n    model = VGG16(weights='imagenet', include_top=False, input_shape=input_shape)\n    x = Flatten()(model.output)\n    Dropout(0.5)(x)\n    Dense(4096, activation='relu')(x)\n    Dense(4096, activation='relu')(x)\n    output = Dense(1, activation='sigmoid')(x)\n    model = Model(model.input, output)\n    \n    return model\n\n\ndef get_InceptionV3_model(input_shape):\n    model = InceptionV3(weights='imagenet', include_top=False, input_shape=input_shape)\n    x = Flatten()(model.output)\n    Dropout(0.5)(x)\n    Dense(4096, activation='relu')(x)\n    Dense(4096, activation='relu')(x)\n    output = Dense(1, activation='sigmoid')(x)\n    model = Model(model.input, output)\n    \n    return model\n\n\nmodel = get_InceptionV3_model(input_shape=(IMG_HEIGHT, IMG_WIDTH, CHANNELS))\nmodel.summary()\nprint()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Building the CustomCallback for getting more info on model's performance\nclass CustomCallback(Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        precision = logs['precision']\n        recall = logs['recall']\n        f1_score = (2 * (precision * recall)) / (precision + recall)\n        \n        loss = logs['loss']\n        accuracy = logs['accuracy']\n        auc_roc = logs['auc_roc']\n        \n        # Validation\n        val_loss = logs['val_loss']\n        val_accuracy = logs['val_accuracy']\n        val_auc_roc = logs['val_auc_roc']\n        val_precision = logs['val_precision']\n        val_recall = logs['val_recall']\n        val_f1_score = (2 * (val_precision * val_recall)) / (val_precision + val_recall)\n        \n        info = {\n            'loss': round(loss, 5),\n            'accuracy': round(accuracy, 4),\n            'auc_roc': round(auc_roc, 4),\n            'precision': round(precision, 4),\n            'recall': round(recall, 4),\n            'f1_score': round(f1_score, 4),\n            'val_loss': round(val_loss, 5),\n            'val_accuracy': round(val_accuracy, 4),\n            'val_auc_roc': round(val_auc_roc, 4),\n            'val_precision': round(val_precision, 4),\n            'val_recall': round(val_recall, 4),\n            'val_f1_score': round(val_f1_score, 4),\n        }\n        \n        print(f'\\n{json.dumps(info, indent=2)}')\n        print()\n\n        \n\ncallbacks = [\n    ReduceLROnPlateau(monitor='val_loss',factor=0.1, patience=2, min_lr=0.000001, verbose=1),\n    CustomCallback()\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_epoch = 1\noptimizer = Adam(learning_rate=0.0001)\nloss = 'binary_crossentropy'\n\nmetrics = [\n    'accuracy', \n    AUC(curve='ROC', name='auc_roc'), \n    Precision(name='precision'), \n    Recall(name='recall')\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    optimizer=optimizer,\n    loss=loss,\n    metrics=metrics\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Turn on the `GPU` accelerator to speed up `learning`."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    train_generator, \n    steps_per_epoch=x_train.shape[0] // BATCH_SIZE,\n    batch_size=BATCH_SIZE, \n    validation_data=val_generator, \n    epochs=num_epoch,\n    callbacks=callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Plotting model's performance**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Accuracy\n\nplt.plot(history.history['accuracy'][1:], label='train acc')\nplt.plot(history.history['val_accuracy'][1:], label='validation acc')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend(loc='lower right')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loss\n\nplt.plot(history.history['loss'][1:], label='train loss')\nplt.plot(history.history['val_loss'][1:], label='validation loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend(loc='lower right')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 🏇 Visualizing CNN\n\nTo know more read the following posts: [post_1](https://www.analyticsvidhya.com/blog/2018/03/essentials-of-deep-learning-visualizing-convolutional-neural-networks/), [post_2](https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/) and [post_3](https://www.kaggle.com/arpitjain007/guide-to-visualize-filters-and-feature-maps-in-cnn).\n\nTo `visualize how CNN and Max pooling works` go through the following [kernel](https://www.kaggle.com/akashsdas/how-does-convolutions-work)."},{"metadata":{},"cell_type":"markdown","source":"**Visualize filters**\n\nPlotting the `0th` filter of the `1st conv layer`."},{"metadata":{"trusted":true},"cell_type":"code","source":"top_layer = model.layers[1]\nplt.imshow(top_layer.get_weights()[0][:, :, :, 0].squeeze())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_filters_for_conv_layer(model, layer_index, num_columns=5, cmap='binary', how_many='all'):\n    layer = model.layers[layer_index]\n    filter_weights = layer.get_weights()[0]\n    \n    num_filters = layer.filters if how_many == 'all' else how_many\n    num_rows = (num_filters // num_columns) + (num_filters % num_columns)\n    # example:\n    # num_rows = (96 // 5) + (96 % 5) == 20 (to plot all the filters)\n    \n    f, axs = plt.subplots(num_rows, num_columns, figsize=(20, 5 * num_rows))\n    row_count = 0  # to plot num_columns figs in an individual row\n    \n    if not isinstance(axs, np.ndarray):\n        # When num_cloumns == how_many\n        axs = np.array(axs)  # to make axs iterable\n        # list can also be inplace np.array but since plt.subplots axs output is of type np.ndarray I kept \n        \n    for idx, row_ax in enumerate(axs):\n        # plotting filters in a row\n        for i, ax in enumerate(row_ax):\n            if row_count + i >= num_filters:\n                break\n                \n            if len(filter_weights.shape) == 4:\n                if filter_weights.shape[2] == 1:\n                    # For plotting filters whose weight shape is == (kernel_size_x, kernel_size_y, 1, #filters)\n                    # example: (11, 11, 1, 96)\n                    ax.imshow(filter_weights[:, :, :, row_count + i].squeeze(), cmap=cmap)\n                else:\n                    # For plotting filters whose weight shape is == (kernel_size_x, kernel_size_y, num > 1, #filters)\n                    # example: (5, 5, 96, 256)\n                    # because if ax.imshow(filter_weights[:, :, :, row_count + i].squeeze(), cmap=cmap)\n                    # is used then we'll have array of (5, 5, 96) which is invalid image data for plotting 2D image\n                    # (in above case where `filter_weights.shape[2] == 1` there we'll end up with (11, 11, 1) which\n                    # after applying the `squeeze` function will be (11, 11) which is valid image data) so in \n                    # that case we'll just plot (5, 5) plot in the first 3D array i.e. (5, 5, 0, row_count + i) \n                    # => this is what we'll plot. To plot (5, 5, row_count + i, 0) just change indexing from\n                    # [:, :, 0, row_count + i] to [:, :, row_count + i, 0]\n                    ax.imshow(filter_weights[:, :, 0, row_count + i].squeeze(), cmap=cmap)\n                    \n                # For generalization this can be used, but to understand why 0 need to be used,\n                # using the above way\n                # ax.imshow(filter_weights[:, :, 0, row_count + i].squeeze(), cmap=cmap)\n            else:\n                break\n                            \n        # increasing row_count by num_columns\n        row_count += num_columns ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"conv_layers_idxs = [idx for idx in range(len(model.layers)) if 'conv' in model.layers[idx].name]\nprint(len(conv_layers_idxs))\n\n' | '.join([str(idx) for idx in conv_layers_idxs])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using `cmap` as `sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)` instead of `binary`, just to make the plot look `beautiful as you`. Also there are `96` filters in the `1st conv` layer so only plotting the first `20` filters.\n\nVisualizing `only first 20` filters in the `1nd conv layer`. Here `sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)` is used as `cmap`."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_filters_for_conv_layer(\n    model, \n    conv_layers_idxs[0], \n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True), \n    how_many=20\n)  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualizing `only first 10` filters in the `2nd conv layer`. Here `binary` is used as `cmap`."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_filters_for_conv_layer(\n    model, \n    conv_layers_idxs[1], \n    num_columns=5,\n    how_many=10\n)  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizing feature maps**"},{"metadata":{},"cell_type":"markdown","source":"The `feature maps` of a `CNN` capture the result of `applying the filters` to an input image i.e at each layer, the feature map is the output of that layer. The reason for visualising a feature map for a specific input image is to try to gain some understanding of what features our CNN detects."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_feature_maps_for_single_conv_layer(model, layer_id, input_img, num_columns=10, cmap='binary'):\n    ref_model = Model(inputs=model.inputs, outputs=model.layers[layer_id].output)\n    feature_map = ref_model.predict(input_img)\n    \n    num_filters = feature_map[0].shape[2]\n    num_rows = (num_filters // num_columns) + (num_filters % num_columns)\n\n    fig = plt.figure(figsize=(16, 2 * num_rows))\n    ix = 1\n    for _ in range(num_rows):\n        for _ in range(num_columns):\n            if ix == num_filters:\n                break\n        \n            # specify subplot and turn of axis\n            ax = plt.subplot(num_rows, num_columns, ix)\n            ax.set_xticks([])\n            ax.set_yticks([])\n        \n            # plot filter channel in grayscale\n            plt.imshow(feature_map[0, :, :, ix-1], cmap=cmap)\n            ix += 1\n            \n    # show the figure\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize `feature maps` for `1st` image in `x_train`."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_feature_maps_for = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Doing as `x_train` pd.Series index are not uniform, so resetting the index\nimg_path = x_train.reset_index(drop=True)[visualize_feature_maps_for]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(load_img(img_path))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[0], \n    process_images([img_path])[0][0][np.newaxis, ...], \n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[2], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True),\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[len(conv_layers_idxs) // 2], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    cmap=sns.cubehelix_palette(as_cmap=True),\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[-1], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True),\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize `feature maps` for `2nd` image in `x_train`."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_feature_maps_for = 1\n\n# Doing as `x_train` pd.Series index are not uniform, so resetting the index\nimg_path = x_train.reset_index(drop=True)[visualize_feature_maps_for]\n\nplt.imshow(load_img(img_path))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[0], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True),\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[2], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    cmap=sns.cubehelix_palette(as_cmap=True),\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[len(conv_layers_idxs) // 2], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    conv_layers_idxs[-1], \n    process_images([img_path])[0][0][np.newaxis, ...],\n    cmap=sns.cubehelix_palette(as_cmap=True),\n    num_columns=8,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## 🎯 Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix'):\n    plt.figure(figsize=(4, 4))\n    \n    plt.imshow(cm, interpolation='nearest', cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True))\n    plt.title(title)\n    plt.colorbar(fraction=0.046, pad=0.04)\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Creating `ImageDataGenerator` for `test_df` which created by splitting the `training dataset`."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cat = 0, Dog = 1\ntest_datagen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_datagen.flow_from_dataframe(\n    test_df, \n    x_col='img_path',\n    y_col='label',    \n    batch_size=BATCH_SIZE, \n    target_size=(IMG_HEIGHT, IMG_WIDTH),\n    class_mode='binary'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluating labelled test data\ntest_results = model.evaluate(train_generator, steps=x_test.shape[0] // BATCH_SIZE)\n\nprint()\n\nprint(f'Test Loss: {test_results[0]}')\nprint(f'Test Accuracy: {test_results[1]}')\nprint(f'Test ACU: {test_results[2]}')\nprint(f'Test Precision: {test_results[3]}')\nprint(f'Test Recall: {test_results[4]}')\n\nf1_score_result = 2 * (test_results[3] * test_results[4]) / (test_results[3] + test_results[4])\nprint(f'Test F1 Score: {f1_score_result}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again creating `ImageDataGenerator` for `test_df` (which created by splitting the `training dataset`) but without `y_col` and `target_size`.\n\nDoing things in this way rather than \n\n```python\n# Cat = 0, Dog = 1\ntest_datagen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_datagen.flow_from_dataframe(\n    test_df, \n    x_col='img_path',\n    y_col='label',    \n    batch_size=BATCH_SIZE, \n    target_size=(IMG_HEIGHT, IMG_WIDTH),\n    class_mode='binary'\n)\n\npredictions = model.predict(test_generator, verbose=1)\npredictions = predictions.flatten()\n\nresults = []\nfor i in predictions:\n    if i >= 0.5:\n        results.append(1)\n    else:\n        results.append(0)\n\nplt.figure(figsize=(15, 15))\nfor i, _img_path in enumerate(test_imgs[:25]):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    if results[i] == 0:\n        plt.xlabel(f'Prediction: Cat')\n    else:\n        plt.xlabel(f'Prediction: Dog')\n    img = load_img(_img_path)\n    plt.imshow(img)\n```\n\nBecause results were very bad, model was giving wrong prediction for 50% to 75% of samples but the same model was giving 99% of `accuracy` and `f1-score` on `evaluation` of the **same dataset**. The cell below is what happened. If you see above cell, `evaluation` metrics then cell below makes no sense.\n\nNot sure why this was happening maybe because of changing size of the img or something else."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cat = 0, Dog = 1\ntest_datagen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_datagen.flow_from_dataframe(\n    test_df, \n    x_col='img_path',\n    y_col='label',    \n    batch_size=BATCH_SIZE, \n    target_size=(IMG_HEIGHT, IMG_WIDTH),\n    class_mode='binary'\n)\n\npredictions = model.predict(test_generator, verbose=1)\npredictions = predictions.flatten()\n\nresults = []\nfor i in predictions:\n    if i >= 0.5:\n        results.append(1)\n    else:\n        results.append(0)\n\nplt.figure(figsize=(15, 15))\nfor i, _img_path in enumerate(test_generator.filenames[:25]):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    if results[i] == 0:\n        plt.xlabel(f'Prediction: Cat')\n    else:\n        plt.xlabel(f'Prediction: Dog')\n    img = load_img(_img_path)\n    plt.imshow(img)\n    \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(test_generator.labels, results) \n\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes=range(2)) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So instead of using the above method for `prediction` on the test dataset (part of training dataset) I've used the other way where the image will flow directly from source using the `ImageDataGenerator`'s `flow` method and ImageDataGenerator won't have `y_col` (nor the `class_mode`) and `target_size`.\n\nBy doing we get results match our `evaluation` results which means that these are the right predictions that our model will make on `unseen` data.\n\nThe same process is used to make predictions of the `competitions` test dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting test data ready for prediction\ntest_imgs = [sample for sample in test_df.img_path]\nx_test, y_test = process_images(test_imgs)\nprint(f'x_test length: {len(x_test)}, y_test length: {len(y_test)}')\n\nx = np.asarray(x_test)\n\ndel x_test\ngc.collect()\n\ntest_datagen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_datagen.flow(x, batch_size=BATCH_SIZE, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test_generator, verbose=1)\npredictions = predictions.flatten()\n\nresults = []\nfor i in predictions:\n    if i >= 0.5:\n        results.append(1)\n    else:\n        results.append(0)\n\nplt.figure(figsize=(15, 15))\nfor i, _img_path in enumerate(test_imgs[:25]):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    if results[i] == 0:\n        plt.xlabel(f'Prediction: Cat')\n    else:\n        plt.xlabel(f'Prediction: Dog')\n    img = load_img(_img_path)\n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compute the confusion matrix\nlabels_list = test_df.label.astype('int')\nconfusion_mtx = confusion_matrix(labels_list, results) \n\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes=range(2)) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## 🔮 Predictions on test set and 📧 submitting the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_DIR = '../data/test/'\ntest_imgs = [TEST_DIR + i for i in os.listdir(TEST_DIR)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting test data ready for prediction\nx_test, y_test = process_images(test_imgs)\nprint(f'x_test length: {len(x_test)}, y_test length: {len(y_test)}')\n\nx = np.asarray(x_test)\n\ndel x_test\ngc.collect()\n\ntest_datagen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_datagen.flow(x, batch_size=BATCH_SIZE, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test_generator, verbose=1)\npredictions = predictions.flatten()\n\nresults = []\nfor i in predictions:\n    if i >= 0.5:\n        results.append(1)\n    else:\n        results.append(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(predictions[:10])\nprint()\nprint(results[:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cat = 0, Dog = 1\n\nplt.figure(figsize=(15, 15))\nfor i, _img_path in enumerate(test_imgs[:25]):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    if results[i] == 0:\n        plt.xlabel(f'Prediction: Cat')\n    else:\n        plt.xlabel(f'Prediction: Dog')\n    # plt.xlabel(f'Prediction: {predictions[i]}')\n    img = load_img(_img_path)\n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('../input/dogs-vs-cats-redux-kernels-edition/sample_submission.csv')\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in submission.index:\n    # submission['label'].iloc[i] = results[i]\n    submission['label'].iloc[i] = predictions[i]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('sample_submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **🎁 Saving the model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"---\n\nI'll wrap things up there. If you want to find some other answers then go ahead `edit` this kernel. If you have any `questions` then do let me know.\n\nIf this kernel helped you then don't forget to 🔼 `upvote` and share your 🎙 `feedback` on improvements of the kernel.\n\n![](https://media.giphy.com/media/RETg1tippXtNm/giphy.gif)\n\n---"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}