{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('./'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# define location of zip dataset\nfolder_zip = '/kaggle/input/dogs-vs-cats/train.zip'\n# define location of dataset (where the unzipped files will be)\nfolder = './train/'\n\n#Extract the zip file\nimport zipfile\nwith zipfile.ZipFile(folder_zip,\"r\") as z:\n    z.extractall(\".\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot Dog and Cat Photos"},{"metadata":{},"cell_type":"markdown","source":"Looking at a few random photos in the directory, you can see that the photos are color and have different shapes and sizes. We can see that some photos are landscape format, some are portrait format, and some are square. We can also see a photo where the cat is barely visible (bottom left corner) and another that has two cats (lower right corner). This suggests that any classifier fit on this problem will have to be robust."},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot dog photos from the dogs vs cats dataset\nfrom matplotlib import pyplot\nfrom matplotlib.image import imread\n\ndef plot9FirstPictures(ofDog):\n    \"\"\" \n    Plot first 9 images of Cat or Dog\n  \n    Parameters: \n    ofDog (Boolean): if true we plot dog pictures else cat ones\n    \"\"\"\n    # plot first n images of Cat or Dog\n    animal = 'dog.' if ofDog else 'cat.'\n    for i in range(9):\n        # define subplot\n        pyplot.subplot(330 + 1 + i)\n        # define filename\n        filename = folder + animal + str(i) + '.jpg'\n        # load image pixels\n        image = imread(filename)\n        # plot raw pixel data\n        pyplot.imshow(image)\n    # show the figure\n    pyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot pictures of cat\nplot9FirstPictures(True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot9FirstPictures(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Select Standardized Photo Size"},{"metadata":{},"cell_type":"markdown","source":"We select a fixed size of 200×200 pixels. Smaller inputs mean a model that is faster to train."},{"metadata":{},"cell_type":"markdown","source":"Pre-Process Photos into Standard Directories using the Keras ImageDataGenerator class and flow_from_directory() API.\n\nThis API prefers data to be divided into separate train/ and test/ directories, and under each directory to have a subdirectory for each class, e.g. a train/dog/ and a train/cat/ subdirectories and the same for test. Images are then organized under the subdirectories.\n\nHere is a script to create a copy of the dataset with this preferred structure. We will randomly select 25% of the images (or 6,250) to be used in a test dataset.\n\nWe create the directory structure as follows:\n* dataset_dogs_vs_cats\n    * test\n        * dogs\n        * cas\n    * train\n        * dogs\n        * cats\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# create directories\ndataset_home = 'dataset_dogs_vs_cats/'\nsubdirs = ['train/', 'test/']\n# create label subdirectories\nlabeldirs = ['dogs/', 'cats/']\nfor subdir in subdirs:\n\tfor labldir in labeldirs:\n\t\tnewdir = dataset_home + subdir + labldir\n\t\tos.makedirs(newdir, exist_ok=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nfrom shutil import copyfile\n# seed random number generator\nrandom.seed(1)\n# define ratio of pictures to use for validation\nval_ratio = 0.25\n# copy training dataset images into subdirectories\nsrc_directory = 'train/'\nfor file in os.listdir(folder):\n\tsrc = folder + '/' + file\n\tdst_dir = 'train/'\n\tif random.random() < val_ratio:\n\t\tdst_dir = 'test/'\n\tif file.startswith('cat'):\n\t\tdst = dataset_home + dst_dir + 'cats/'  + file\n\t\tcopyfile(src, dst)\n\telif file.startswith('dog'):\n\t\tdst = dataset_home + dst_dir + 'dogs/'  + file\n\t\tcopyfile(src, dst)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Develop a Baseline CNN Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# baseline model for the dogs vs cats dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\nfrom keras.layers import Dropout\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nfrom keras.preprocessing.image import ImageDataGenerator","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Baseline\ndef define_model():\n    \n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(200, 200, 3)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(1, activation='sigmoid'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])\n    \n\treturn model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n    # plot loss\n    pyplot.subplot(211)\n    pyplot.title('Cross Entropy Loss')\n    pyplot.plot(history.history['loss'], color='blue', label='train')\n    pyplot.plot(history.history['val_loss'], color='orange', label='test')\n    # plot accuracy\n    pyplot.subplot(212)\n    pyplot.title('Classification Accuracy')\n    pyplot.plot(history.history['accuracy'], color='blue', label='train')\n    pyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n    # save plot to file\n    filename = sys.argv[0].split('/')[-1]\n    pyplot.savefig(filename + '_plot.png')\n    pyplot.close()\n    \n\n# run the test harness for evaluating a model\ndef run_test_harness(preloadWeights, model, number_epoch = 20, withDataAugmentation = False):\n    \"\"\" \n    Create a test harness of a model\n  \n    Parameters: \n    preloadWeights (Boolean): True if we preload the weights of the model \n    \n    model (int): 1 = one_block_vgg / 2 = two_blocks_vgg / 3 = three_block_vgg / 4 = three_block_vgg with dropout\n  \n    \"\"\"\n    # define model\n    if model == 1:\n        model = define_model_one_block_vgg()\n    elif model == 2:\n        model = define_model_two_blocks_vgg()\n    elif model == 3:\n        model = define_model_three_blocks_vgg()\n    elif model == 4:\n        model = define_model_three_blocks_vgg_with_dropout()\n    # create data generator\n    if withDataAugmentation:\n        train_datagen = ImageDataGenerator(rescale=1.0/255.0,\n        width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n    else:\n        train_datagen = ImageDataGenerator(rescale=1.0/255.0)\n    \n    test_datagen = ImageDataGenerator(rescale=1.0/255.0)\n    \n    # prepare iterators\n    #Explication of batch size : https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network\n    #https://stats.stackexchange.com/questions/230120/neural-networks-is-an-epoch-in-sgd-the-same-as-an-epoch-in-mini-batch\n    train_it = train_datagen.flow_from_directory('dataset_dogs_vs_cats/train/',\n        class_mode='binary', batch_size=64, target_size=(200, 200))\n    test_it = test_datagen.flow_from_directory('dataset_dogs_vs_cats/test/',\n        class_mode='binary', batch_size=64, target_size=(200, 200))\n    # fit model\n    \n    checkpoint_path = \"train_ckpt/cp.ckpt\"\n    # Create a callback that saves the model's weights\n    cp_callback = ModelCheckpoint(filepath=checkpoint_path,save_best_only=True, save_weights_only=True, verbose=1)\n    \n    #Early stopping\n    early_stop = EarlyStopping(monitor = 'val_loss', min_delta = 0.001, \n                           patience = 5, mode = 'min', verbose = 1,\n                           restore_best_weights = True)\n\n    if (preloadWeights):\n        model.load_weights(checkpoint_path)\n    \n    history = model.fit_generator(train_it, steps_per_epoch=len(train_it),\n        validation_data=test_it, validation_steps=len(test_it), epochs=number_epoch, callbacks=[cp_callback,early_stop])\n    # evaluate model\n    _, acc = model.evaluate_generator(test_it, steps=len(test_it))\n    print('> %.3f' % (acc * 100.0))\n    # learning curves\n    summarize_diagnostics(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run_test_harness(True,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# One Block VGG Model :\nThe one-block VGG model has a single convolutional layer with 32 filters followed by a max pooling layer.\nThe model achieved an accuracy of about 72% on the test dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"def define_model_one_block_vgg():\n    \n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(200, 200, 3)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(1, activation='sigmoid'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])\n    \n\treturn model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Two Block VGG Model :\nThe two-block VGG model extends the one block model and adds a second block with 64 filters.\nWe can see that the model achieved a small improvement in performance from about 72% with one block to about 76% accuracy with two blocks"},{"metadata":{"trusted":true},"cell_type":"code","source":"def define_model_two_blocks_vgg():\n    \n    model = Sequential()\n    \n    #First block\n    model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(200, 200, 3)))\n    model.add(MaxPooling2D((2, 2)))\n    #Second Block\n    model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n    model.add(MaxPooling2D((2, 2)))\n    \n    model.add(Flatten())\n    model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n    model.add(Dense(1, activation='sigmoid'))\n    # compile model\n    opt = SGD(lr=0.001, momentum=0.9)\n    model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run_test_harness(False,2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Three Blocks VGG Model :\nThe three-block VGG model extends the two block model and adds a third block with 128 filters.\nWe can see that the model achieved a small improvement in performance from about 72% with one block to about 76% accuracy with two blocks"},{"metadata":{"trusted":true},"cell_type":"code","source":"def define_model_three_blocks_vgg():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(200, 200, 3)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(1, activation='sigmoid'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])\n\treturn model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run_test_harness(False,3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Discussion: \nVGG 1: 72.331%\nVGG 2: 76.646%\nVGG 3: 80.184%\n\nWe observe a trend of improved performance with the increase in capacity, but also a similar case of overfitting occurring earlier and earlier in the run. -> The results suggest that the model will likely benefit from regularization techniques. Ex: dropout, weight decay, and data augmentation (can boost performance by encouraging the model to learn features that are further invariant to position by expanding the training dataset)?\n\n"},{"metadata":{},"cell_type":"markdown","source":"# Model Improvements\n\nBaseline : VGG 3\nLearning curves of the model showed strong signs of overfitting.\nWe can explore two approaches to attempt to address this overfitting: dropout regularization and data augmentation.\nWe will increase the number of training epochs from 20 to 50< to give the model more space for refinement."},{"metadata":{},"cell_type":"markdown","source":"**Dropout :**\nReviewing the learning curves, we can see that dropout has had an effect on the rate of improvement of the model on both the train and test sets.\n\nOverfitting has been reduced and delayed."},{"metadata":{"trusted":true},"cell_type":"code","source":"def define_model_three_blocks_vgg_with_dropout():\n    model = Sequential()\n    model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(200, 200, 3)))\n    model.add(MaxPooling2D((2, 2)))\n    model.add(Dropout(0.2))\n    model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n    model.add(MaxPooling2D((2, 2)))\n    model.add(Dropout(0.2))\n    model.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n    model.add(MaxPooling2D((2, 2)))\n    model.add(Dropout(0.2))\n    model.add(Flatten())\n    model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n    model.add(Dropout(0.5))\n    model.add(Dense(1, activation='sigmoid'))\n    # compile model\n    opt = SGD(lr=0.001, momentum=0.9)\n    model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run_test_harness(False,4,50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Augmentation:**\n\n* act as a regularization technique, adding noise to the training data, and encouraging the model to learn the same features, invariant to their position in the input.\n* an create variations of the images that can improve the ability of the fit models to generalize what they have learned to new images.\n\nHere we do small shifts and horizontal flips. Photos in the training dataset will be augmented with small (10%) random horizontal and vertical shifts and random horizontal flips that create a mirror image of a photo."},{"metadata":{"trusted":true},"cell_type":"code","source":"run_test_harness(False,4,50,True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}