{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "764fe135-c09e-9769-5794-500867154d93"
      },
      "source": [
        "Practice notebook for learning CNN in the example of Dos vs Cats.\n",
        "\n",
        "Starter EDA and ConvNet implementation using Keras. \n",
        "\n",
        "Inspiration for this notebook comes from this [Keras blog post](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) and the [VGG ConvNet paper](https://arxiv.org/pdf/1409.1556.pdf). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3d458c15-e131-f3c4-f756-843d6454bb37"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns \n",
        "import matplotlib.pyplot as plt \n",
        "import os, cv2, random\n",
        "%matplotlib inline\n",
        "from matplotlib import ticker\n",
        "from keras.utils import np_utils\n",
        "from keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\n",
        "from keras.optimizers import RMSprop\n",
        "from keras.layers import Input, Dropout, Flatten, Convolution2D, MaxPooling2D, Dense, Activation\n",
        "from keras.models import Sequential\n",
        "\n",
        "\n",
        "#from keras.models import Sequential\n",
        "#from keras.layers import Input, Dropout, Flatten, Convolution2D, MaxPooling2D, Dense, Activation\n",
        "#from keras.optimizers import RMSprop\n",
        "#from keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\n",
        "#from keras.utils import np_utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "04681981-55ac-7820-a0ae-38f98c851c39"
      },
      "source": [
        "## Data Preparation\n",
        "\n",
        "This function resizes the images to 64x64 and samples 2000 images (8%) of the data to run efficiently as a Kaggle Kernel. Also separated cats and dogs for exploratory analysis. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "663d335e-1b84-a8cb-19ee-8f04839cf4e5"
      },
      "outputs": [],
      "source": [
        "TRAIN_DIR = '../input/train'\n",
        "TEST_DIR = '../input/test'\n",
        "\n",
        "ROWS = 64\n",
        "COLS = 64\n",
        "CHANNELS = 3\n",
        "\n",
        "train_images = [TRAIN_DIR+i for i in os.listdir(TRAIN_DIR)] # use this for full dataset\n",
        "train_dogs =   [TRAIN_DIR+i for i in os.listdir(TRAIN_DIR) if 'dog' in i]\n",
        "train_cats =   [TRAIN_DIR+i for i in os.listdir(TRAIN_DIR) if 'cat' in i]\n",
        "\n",
        "test_images =  [TEST_DIR+i for i in os.listdir(TEST_DIR)]\n",
        "\n",
        "\n",
        "# slice datasets for memory efficiency on Kaggle Kernels, delete if using full dataset\n",
        "train_images = train_dogs[:1000] + train_cats[:1000]\n",
        "random.shuffle(train_images)\n",
        "test_images =  test_images[:25]\n",
        "\n",
        "def read_image(file_path):\n",
        "    img = cv2.imread(file_path, cv2.IMREAD_COLOR) #cv2.IMREAD_GRAYSCALE\n",
        "    return cv2.resize(img, (ROWS, COLS), interpolation=cv2.INTER_CUBIC)\n",
        "\n",
        "\n",
        "def prep_data(images):\n",
        "    count = len(images)\n",
        "    data = np.ndarray((count, CHANNELS, ROWS, COLS), dtype=np.uint8)\n",
        "\n",
        "    for i, image_file in enumerate(images):\n",
        "        image = read_image(image_file)\n",
        "        data[i] = image.T\n",
        "        if i%250 == 0: print('Processed {} of {}'.format(i, count))\n",
        "    \n",
        "    return data\n",
        "\n",
        "train = prep_data(train_images)\n",
        "test = prep_data(test_images)\n",
        "\n",
        "print(\"Train shape: {}\".format(train.shape))\n",
        "print(\"Test shape: {}\".format(test.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ed0fc95a-53bc-3517-245d-cf1f5122bc2d"
      },
      "source": [
        "### Generating the Labels\n",
        "\n",
        "We're dealing with a binary classification problem here - (1) dog (0) cat. The lables can be created by looping over the file names in the train directory. It's nice to see the training data is perfectly balanced. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c0f2fbf6-8d78-7d42-6579-5486a36c1e60"
      },
      "outputs": [],
      "source": [
        "labels = []\n",
        "for i in train_images:\n",
        "    if 'dog' in i:\n",
        "        labels.append(1)\n",
        "    else:\n",
        "        labels.append(0)\n",
        "\n",
        "sns.countplot(labels)\n",
        "sns.plt.title('Cats and Dogs')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4bb0bd2a-0512-aa72-c628-5b6ac946d97c"
      },
      "source": [
        "### Checking out Cats and Dogs\n",
        "A quick side-by-side comparison of the animals."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0b7c79ae-6543-2ed8-e3f5-af4f5beb9cf7"
      },
      "outputs": [],
      "source": [
        "def show_cats_and_dogs(idx):\n",
        "    cat = read_image(train_cats[idx])\n",
        "    dog = read_image(train_dogs[idx])\n",
        "    pair = np.concatenate((cat, dog), axis=1)\n",
        "    plt.figure(figsize=(10,5))\n",
        "    plt.imshow(pair)\n",
        "    plt.show()\n",
        "    \n",
        "for idx in range(0,5):\n",
        "    show_cats_and_dogs(idx)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "599a11ba-aaa4-8266-03ea-d816518e2a81"
      },
      "source": [
        "### Your Average Cat and Dog Photo\n",
        "\n",
        "Just for fun, the mean pixel values for cats and dogs. I can almost see a resemblance, a ghost, if you will..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7d3c6eb7-c370-9752-67c8-c780bab14abe"
      },
      "outputs": [],
      "source": [
        "dog_avg = np.array([dog[0].T for i, dog in enumerate(train) if labels[i]==1]).mean(axis=0)\n",
        "plt.imshow(dog_avg)\n",
        "plt.title('Your Average Dog')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a5a93063-de92-18ce-b48f-68a193e90002"
      },
      "outputs": [],
      "source": [
        "cat_avg = np.array([cat[0].T for i, cat in enumerate(train) if labels[i]==0]).mean(axis=0)\n",
        "plt.imshow(cat_avg)\n",
        "plt.title('Your Average Cat')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "51e403b6-bcfc-b4fa-3770-9850cc86bae3"
      },
      "source": [
        "## CatdogNet-16\n",
        "\n",
        "A scaled down version of the VGG-16, with a few notable changes.\n",
        "\n",
        "- Number of convolution filters cut in half, fully connected (dense) layers scaled down. \n",
        "- Optimizer changed to `RMSprop`. \n",
        "- Output layer activation set to `sigmoid` for binary crossentropy. \n",
        "- Some layers commented out for efficiency.\n",
        "\n",
        "The full network takes about 80s per epoch on a GTX1070 (or 2hr+ on CPU) on the full dataset.  (This script only trains on 8% of the 25K images. )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4c477612-41d3-3112-5176-3c4ad9633080"
      },
      "outputs": [],
      "source": [
        "optimizer = RMSprop(lr=1e-4)\n",
        "objective = 'binary_crossentropy'\n",
        "\n",
        "\n",
        "def catdog():\n",
        "    \n",
        "    model = Sequential()\n",
        "\n",
        "    model.add(Convolution2D(32, 3, 3, border_mode='same', input_shape=(3, ROWS, COLS), activation='relu'))\n",
        "    model.add(Convolution2D(32, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "\n",
        "    model.add(Convolution2D(64, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(Convolution2D(64, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "    \n",
        "    model.add(Convolution2D(128, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(Convolution2D(128, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "    \n",
        "    model.add(Convolution2D(256, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(Convolution2D(256, 3, 3, border_mode='same', activation='relu'))\n",
        "#     model.add(Convolution2D(256, 3, 3, border_mode='same', activation='relu'))\n",
        "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "\n",
        "#     model.add(Convolution2D(256, 3, 3, border_mode='same', activation='relu'))\n",
        "#     model.add(Convolution2D(256, 3, 3, border_mode='same', activation='relu'))\n",
        "#     model.add(Convolution2D(256, 3, 3, border_mode='same', activation='relu'))\n",
        "#     model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "\n",
        "    model.add(Flatten())\n",
        "    model.add(Dense(256, activation='relu'))\n",
        "    model.add(Dropout(0.5))\n",
        "    \n",
        "    model.add(Dense(256, activation='relu'))\n",
        "    model.add(Dropout(0.5))\n",
        "\n",
        "    model.add(Dense(1))\n",
        "    model.add(Activation('sigmoid'))\n",
        "\n",
        "    model.compile(loss=objective, optimizer=optimizer, metrics=['accuracy'])\n",
        "    return model\n",
        "\n",
        "\n",
        "model = catdog()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b7fbf156-2268-62ce-5fed-52a09af5e6f7"
      },
      "source": [
        "### Train and Predict\n",
        "\n",
        "I'm using Keras's early stopping callback to end training when the validation loss stops improving, otherwise the model will overfit. I will also be tracking the loss history on each epoch to visualize the overfitting trend. \n",
        "\n",
        "Note: A slice of 1000 images was used to fit the model for CPU efficency. The model's perfrmance improves significantly when used on the entire dataset. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ca613169-4b41-e9d5-27c7-6629f8e035c8"
      },
      "outputs": [],
      "source": [
        "nb_epoch = 10\n",
        "batch_size = 16\n",
        "\n",
        "## Callback for loss logging per epoch\n",
        "class LossHistory(Callback):\n",
        "    def on_train_begin(self, logs={}):\n",
        "        self.losses = []\n",
        "        self.val_losses = []\n",
        "        \n",
        "    def on_epoch_end(self, batch, logs={}):\n",
        "        self.losses.append(logs.get('loss'))\n",
        "        self.val_losses.append(logs.get('val_loss'))\n",
        "\n",
        "early_stopping = EarlyStopping(monitor='val_loss', patience=3, verbose=1, mode='auto')        \n",
        "        \n",
        "def run_catdog():\n",
        "    \n",
        "    history = LossHistory()\n",
        "    model.fit(train, labels, batch_size=batch_size, nb_epoch=nb_epoch,\n",
        "              validation_split=0.25, verbose=0, shuffle=True, callbacks=[history, early_stopping])\n",
        "    \n",
        "\n",
        "    predictions = model.predict(test, verbose=0)\n",
        "    return predictions, history\n",
        "\n",
        "predictions, history = run_catdog()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6266d01c-9d4c-a1dc-8cb9-ce7b3107b536"
      },
      "outputs": [],
      "source": [
        "loss = history.losses\n",
        "val_loss = history.val_losses\n",
        "\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.title('VGG-16 Loss Trend')\n",
        "plt.plot(loss, 'blue', label='Training Loss')\n",
        "plt.plot(val_loss, 'green', label='Validation Loss')\n",
        "plt.xticks(range(0,nb_epoch)[0::2])\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d9382e7d-31db-0d75-6876-aaccb126e4f6"
      },
      "source": [
        "## How'd We Do?\n",
        "\n",
        "I'm pretty sure I can distinguish a cat from a dog 100% of the time, but how confident is the model?...\n",
        "\n",
        "Tip: Run on the full dataset with a GPU for a LB logloss of ~0.4 and accuracy at approx 90%. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c665d01b-a5a7-2f7e-e5d4-09e108920e40"
      },
      "outputs": [],
      "source": [
        "for i in range(0,10):\n",
        "    if predictions[i, 0] >= 0.5: \n",
        "        print('I am {:.2%} sure this is a Dog'.format(predictions[i][0]))\n",
        "    else: \n",
        "        print('I am {:.2%} sure this is a Cat'.format(1-predictions[i][0]))\n",
        "        \n",
        "    plt.imshow(test[i].T)\n",
        "    plt.show()"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}