{"cells":[{"metadata":{"colab_type":"text","id":"YHK6DyunSbs4"},"cell_type":"markdown","source":"# Distinguishing Cat and Dog Images with Convolutional Neural Networks\n\nHere I study the effects of successively adding components to a CNN. The goal is to label images as either Cat or Dog, based on training images taken from the [Dogs vs. Cats competition](https://www.kaggle.com/c/dogs-vs-cats). The evaluation metric used in the competition is simply the classification accuracy, though here I will also be examining the ROC curves of my models for a bit of extra insight.\n\nThe final model is inspired by the one provided in [this excellent notebook](https://www.kaggle.com/uysimty/keras-cnn-dog-or-cat-classification). (The only real changes I make are to use 150x150 images instead of 128x128, and a single sigmoid output instead of a two-output softmax.)"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"import os\nimport pickle\n\n%matplotlib inline\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\n\nfrom sklearn.metrics import roc_curve\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow import keras\nimport numpy as np\nimport pandas as pd\n\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"5oqBkNBJmtUv"},"cell_type":"markdown","source":"## Initial Model\n\nThe images are color and in a variety of shapes. In order to train a neural network to handle them, they will need to have uniform size. Note the input shape parameter: it is 150x150 for the size and 3 for the color depth. \n\nBecause we are facing a two-class classification problem, i.e. a *binary classification problem*, we will end our network with a [*sigmoid* activation](https://wikipedia.org/wiki/Sigmoid_function), so that the output of our network will be a single scalar between 0 and 1, encoding the probability that the current image is class 1 (as opposed to class 0)."},{"metadata":{"colab":{},"colab_type":"code","id":"PixZ2s5QbYQ3","scrolled":true,"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(150, 150, 3)),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2), \n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'), \n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Flatten(), \n    tf.keras.layers.Dense(512, activation='relu'), \n    tf.keras.layers.Dense(1, activation='sigmoid')  \n])","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"7ZKj8392nbgP","scrolled":true,"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"PEkKSpZlvJXA"},"cell_type":"markdown","source":"Next, we'll configure the specifications for model training. We will train our model with the `binary_crossentropy` loss, because it's a binary classification problem and our final activation is a sigmoid. We will use the `rmsprop` optimizer with a learning rate of `0.001`."},{"metadata":{"colab":{},"colab_type":"code","id":"8DHWhFP_uhq3","scrolled":true,"trusted":true},"cell_type":"code","source":"model.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',\n              metrics = ['acc'])","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"mu3Jdwkjwax4"},"cell_type":"markdown","source":"### Training\nI have packaged up the data into a couple of \"pickled\" DataFrames. The full training dataset comprises 25000 images, split evenly between cat and dog images. `train_df` comprises 90% of the full training data, and `validate_df` the remaining 10%. These have been split at random, in a stratified fashion so as to keep equal amounts of cats and dogs in the validation set (and hence the training set as well).\n\nLet's train on all training images available, for up to 15 epochs to start, and validate on all validation images. The raw image data has pixel activations between 0 and 255, which we will scale to lie between 0 and 1; this can help prevent our NN from multiplying (or dividing) by very large numbers."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load data\ntrain_df = pd.read_pickle('../input/cat-vs-dog-data/train_df.pkl')\ntrain_df.filename = train_df.filename.map(lambda s: s.split('\\\\')[1])\nvalidate_df = pd.read_pickle('../input/cat-vs-dog-data/validate_df.pkl')\nvalidate_df.filename = validate_df.filename.map(lambda s: s.split('\\\\')[1])\ntrain_dir = '../input/dogs-vs-cats/train/train'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255.)\nvalidation_datagen = ImageDataGenerator(rescale = 1./255.)\n\ntrain_generator = train_datagen.flow_from_dataframe(train_df,\n                                                    train_dir,\n                                                    x_col='filename',\n                                                    y_col='category',\n                                                    batch_size=50,\n                                                    class_mode='binary',\n                                                    target_size=(150, 150),\n                                                       seed=42)     \nvalidation_generator =  validation_datagen.flow_from_dataframe(validate_df,\n                                                          train_dir,\n                                                          x_col='filename',\n                                                          y_col='category',\n                                                         batch_size=50,\n                                                         class_mode  = 'binary',\n                                                         target_size = (150, 150),\n                                                            seed=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Early Stopping is basically always a good callback to use, especially if you have a good idea of how much patience is needed."},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=3, restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**NOTE:** Not actually training in this notebook, since that would take awhile to upload. Instead, I'll load up the results from training on my machine."},{"metadata":{"colab":{},"colab_type":"code","id":"Fb1_lgobv81m","scrolled":true,"trusted":true},"cell_type":"code","source":"# history = model.fit_generator(train_generator,\n#                               epochs=15,\n#                               validation_data=validation_generator,\n#                              callbacks=[earlystopping])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save model, so retraining is not necessary in future notebook runs"},{"metadata":{"trusted":true},"cell_type":"code","source":"def save_history(history, filename):\n    with open(filename, 'wb') as file_pi:\n        pickle.dump((history.epoch, history.history), file_pi)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_history(filename):\n    history = keras.callbacks.History()\n    with open(filename, 'rb') as file_pi:\n        (history.epoch, history.history) = pickle.load(file_pi)\n    return history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def save_earlystopping(earlystopping, filename):\n    with open(filename, 'wb') as file_pi:\n        pickle.dump((earlystopping.stopped_epoch, earlystopping.patience,\n                    earlystopping.monitor, earlystopping.min_delta,\n                    earlystopping.monitor_op, earlystopping.restore_best_weights,\n                    earlystopping.wait, earlystopping.baseline), file_pi)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_earlystopping(filename):\n    earlystopping = keras.callbacks.EarlyStopping()\n    with open(filename, 'rb') as file_pi:\n        (earlystopping.stopped_epoch, earlystopping.patience,\n         earlystopping.monitor, earlystopping.min_delta,\n         earlystopping.monitor_op, earlystopping.restore_best_weights,\n         earlystopping.wait, earlystopping.baseline) = pickle.load(file_pi)\n    return earlystopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Save model\n# model.save('initial_model.h5')\n# save_history(history, 'initial_history.pkl')\n# save_earlystopping(earlystopping, 'initial_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"# Load model\nmodel = keras.models.load_model('../input/cat-vs-dog-cnn-models/initial_model.h5')\nhistory = load_history('../input/cat-vs-dog-cnn-models/initial_history.pkl')\nearlystopping = load_earlystopping('../input/cat-vs-dog-cnn-models/initial_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define various helper methods, used for reporting training history and validation performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_early_stopping(history, earlystopping):\n    print('Early stopping')\n    monitor = history.history[earlystopping.monitor]\n    fun1, fun2 = np.min, np.argmin\n    best = 'Lowest'\n    if earlystopping.monitor_op == np.greater:\n        fun1, fun2 = np.max, np.argmax\n        best = 'Highest'\n    print(f'  Monitor: {earlystopping.monitor}')\n    print(f'    {best} value: {fun1(monitor)}')\n    print(f'    Epoch: {fun2(monitor)+1}')\n    stopped_epoch = earlystopping.stopped_epoch - earlystopping.patience + 1\n    if earlystopping.stopped_epoch == 0:\n        stopped_epoch = 'None (stopped_epoch==0)'\n    print(f'  Epoch detected by early stopping: {stopped_epoch}')\n    if not earlystopping.restore_best_weights or earlystopping.stopped_epoch == 0:\n        print('  Best weights NOT returned')\n    else:\n        print('  Best weights returned')","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"0oj0gTIy4k60","scrolled":true,"trusted":true},"cell_type":"code","source":"def plot_history(history):\n    acc      = history.history[     'acc' ]\n    val_acc  = history.history[ 'val_acc' ]\n    loss     = history.history[    'loss' ]\n    val_loss = history.history['val_loss' ]\n\n    epochs   = range(len(acc))\n\n    plt.plot  ( epochs,     acc , label='Training')\n    plt.plot  ( epochs, val_acc , label='Validation')\n    plt.xlabel ('Epoch')\n    plt.ylabel ('Accuracy')\n    plt.legend ()\n    plt.title ('Training and validation accuracy')\n    plt.figure()\n\n    plt.plot  ( epochs,     loss, label='Training')\n    plt.plot  ( epochs, val_loss, label='Validation')\n    plt.xlabel ('Epoch')\n    plt.ylabel ('Loss')\n    plt.legend ()\n    plt.title ('Training and validation loss'   )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def report(model, validation_generator, history=None, earlystopping=None):\n    if earlystopping is not None:\n        check_early_stopping(history, earlystopping)\n        print()\n    \n    if history is not None:\n        plot_history(history)\n    \n    # Evaluate trained model on validation set\n    validation_generator.reset()\n    [val_loss, val_acc] = model.evaluate_generator(validation_generator)\n    print('Model evaluation')\n    print(f'val_loss: {val_loss}, val_acc: {val_acc}')\n    print()\n    \n    # Compute ROC curve\n    validation_generator.reset()\n    validation_set = [validation_generator.next() for _ in range(len(validation_generator))]\n    val_images = np.concatenate([validation_set[i][0] for i in range(len(validation_set))])\n    val_y = np.concatenate([validation_set[i][1] for i in range(len(validation_set))])\n    fpr, tpr, thresholds = roc_curve(val_y, model.predict(val_images))\n    \n    return fpr, tpr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ROCCurveParams():\n    def __init__(self, fpr, tpr, color, linestyle):\n        self.fpr, self.tpr, self.color, self.linestyle = fpr, tpr, color, linestyle","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_roc_curves(roc_curves):\n    plt.figure()\n    for curvename in roc_curves:\n        c = roc_curves[curvename]\n        plt.plot(c.fpr, c.tpr, color=c.color, lw=2, label=curvename, linestyle=c.linestyle)\n    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n    plt.xlim([0.0, 1.0])\n    plt.ylim([0.0, 1.05])\n    plt.xlabel('False Positive Rate')\n    plt.ylabel('True Positive Rate')\n    plt.title('Validation set ROC')\n    plt.legend(loc=\"lower right\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"roc_curves = dict()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr = report(model, validation_generator, history, earlystopping)\nroc_curves['Initial model'] = ROCCurveParams(fpr, tpr, 'darkorange', '-')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_roc_curves(roc_curves)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\n\nInitial model with 83.5% validation accuracy. Trains quickly. Overfits considerably."},{"metadata":{},"cell_type":"markdown","source":"## Add Dropout, increase early stopping patience"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(150, 150, 3)),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'), \n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Flatten(), \n    tf.keras.layers.Dense(512, activation='relu'),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.Dense(1, activation='sigmoid')  \n])\nmodel.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',\n              metrics = ['acc'])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255.)\nvalidation_datagen = ImageDataGenerator(rescale = 1./255.)\n\ntrain_generator = train_datagen.flow_from_dataframe(train_df,\n                                                    train_dir,\n                                                    x_col='filename',\n                                                    y_col='category',\n                                                    batch_size=50,\n                                                    class_mode='binary',\n                                                    target_size=(150, 150),\n                                                       seed=42)     \nvalidation_generator =  validation_datagen.flow_from_dataframe(validate_df,\n                                                          train_dir,\n                                                          x_col='filename',\n                                                          y_col='category',\n                                                         batch_size=50,\n                                                         class_mode  = 'binary',\n                                                         target_size = (150, 150),\n                                                            seed=42)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=5, restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# history = model.fit_generator(train_generator,\n#                               epochs=30,\n#                               validation_data=validation_generator,\n#                              callbacks=[earlystopping])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Save model\n# model.save('dropout_model.h5')\n# save_history(history, 'dropout_history.pkl')\n# save_earlystopping(earlystopping, 'dropout_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"# Load model\nmodel = keras.models.load_model('../input/cat-vs-dog-cnn-models/dropout_model.h5')\nhistory = load_history('../input/cat-vs-dog-cnn-models/dropout_history.pkl')\nearlystopping = load_earlystopping('../input/cat-vs-dog-cnn-models/dropout_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr = report(model, validation_generator, history, earlystopping)\nroc_curves['Dropout'] = ROCCurveParams(fpr, tpr, 'green', '-')\nplot_roc_curves(roc_curves)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\n\nDropout has reduced overfitting and improved the validation set performance."},{"metadata":{},"cell_type":"markdown","source":"## Reduce LR on plateau, further increase early stopping patience"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(150, 150, 3)),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'), \n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Flatten(), \n    tf.keras.layers.Dense(512, activation='relu'),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.Dense(1, activation='sigmoid')  \n])\nmodel.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',\n              metrics = ['acc'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255.)\nvalidation_datagen = ImageDataGenerator(rescale = 1./255.)\n\ntrain_generator = train_datagen.flow_from_dataframe(train_df,\n                                                    train_dir,\n                                                    x_col='filename',\n                                                    y_col='category',\n                                                    batch_size=50,\n                                                    class_mode='binary',\n                                                    target_size=(150, 150),\n                                                       seed=42)     \nvalidation_generator =  validation_datagen.flow_from_dataframe(validate_df,\n                                                          train_dir,\n                                                          x_col='filename',\n                                                          y_col='category',\n                                                         batch_size=50,\n                                                         class_mode  = 'binary',\n                                                         target_size = (150, 150),\n                                                            seed=42)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=9, restore_best_weights=True)\nreducelr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', patience=2, factor=0.5, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# history = model.fit_generator(train_generator,\n#                               epochs=40,\n#                               validation_data=validation_generator,\n#                              callbacks=[earlystopping, reducelr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Save model\n# model.save('redlr_model.h5')\n# save_history(history, 'redlr_history.pkl')\n# save_earlystopping(earlystopping, 'redlr_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"# Load model\nmodel = keras.models.load_model('../input/cat-vs-dog-cnn-models/redlr_model.h5')\nhistory = load_history('../input/cat-vs-dog-cnn-models/redlr_history.pkl')\nearlystopping = load_earlystopping('../input/cat-vs-dog-cnn-models/redlr_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr = report(model, validation_generator, history, earlystopping)\nroc_curves['Reduce LR'] = ROCCurveParams(fpr, tpr, 'red', '--')\nplot_roc_curves(roc_curves)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\n\nLR Reduction on Plateaus stabilizes learning near the end. It takes several more epochs to reach optimum, and results in nearly identical validation set performance. The training set performance is markedly improved, and some amount of overfitting is now revealed.\n\n**NOTE:** From this point onward, the \"Dropout\" ROC curve (and all successive curves) includes LR Reduction on Plateaus."},{"metadata":{"trusted":true},"cell_type":"code","source":"roc_curves['Dropout'] = roc_curves.pop('Reduce LR')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Batch normalization"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(150, 150, 3)),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'), \n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Flatten(), \n    tf.keras.layers.Dense(512, activation='relu'),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.Dense(1, activation='sigmoid')  \n])\nmodel.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',\n              metrics = ['acc'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255.)\nvalidation_datagen = ImageDataGenerator(rescale = 1./255.)\n\ntrain_generator = train_datagen.flow_from_dataframe(train_df,\n                                                    train_dir,\n                                                    x_col='filename',\n                                                    y_col='category',\n                                                    batch_size=50,\n                                                    class_mode='binary',\n                                                    target_size=(150, 150),\n                                                       seed=42)     \nvalidation_generator =  validation_datagen.flow_from_dataframe(validate_df,\n                                                          train_dir,\n                                                          x_col='filename',\n                                                          y_col='category',\n                                                         batch_size=50,\n                                                         class_mode  = 'binary',\n                                                         target_size = (150, 150),\n                                                            seed=42)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=9, restore_best_weights=True)\nreducelr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', patience=2, factor=0.5, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# history = model.fit_generator(train_generator,\n#                               epochs=40,\n#                               validation_data=validation_generator,\n#                              callbacks=[earlystopping, reducelr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Save model\n# model.save('bnorm_model.h5')\n# save_history(history, 'bnorm_history.pkl')\n# save_earlystopping(earlystopping, 'bnorm_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"# Load model\nmodel = keras.models.load_model('../input/cat-vs-dog-cnn-models/bnorm_model.h5')\nhistory = load_history('../input/cat-vs-dog-cnn-models/bnorm_history.pkl')\nearlystopping = load_earlystopping('../input/cat-vs-dog-cnn-models/bnorm_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr = report(model, validation_generator, history, earlystopping)\nroc_curves['Batch Norm'] = ROCCurveParams(fpr, tpr, 'Blue', '-')\nplot_roc_curves(roc_curves)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\n\nBatch normalization sped up the training considerably, and has improved the training performance, but did not affect the validation performance very much. Overfitting is now extremely evident.\n\n**NOTE:** From now on, the \"Dropout\" and all successive ROC curves include Batch Normalization."},{"metadata":{"trusted":true},"cell_type":"code","source":"roc_curves['Dropout'] = roc_curves.pop('Batch Norm')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Double filters in all conv layers"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(150, 150, 3)),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(128, (3,3), activation='relu'), \n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Flatten(), \n    tf.keras.layers.Dense(512, activation='relu'),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.Dense(1, activation='sigmoid')  \n])\nmodel.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',\n              metrics = ['acc'])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255.)\nvalidation_datagen = ImageDataGenerator(rescale = 1./255.)\n\ntrain_generator = train_datagen.flow_from_dataframe(train_df,\n                                                    train_dir,\n                                                    x_col='filename',\n                                                    y_col='category',\n                                                    batch_size=50,\n                                                    class_mode='binary',\n                                                    target_size=(150, 150),\n                                                       seed=42)     \nvalidation_generator =  validation_datagen.flow_from_dataframe(validate_df,\n                                                          train_dir,\n                                                          x_col='filename',\n                                                          y_col='category',\n                                                         batch_size=50,\n                                                         class_mode  = 'binary',\n                                                         target_size = (150, 150),\n                                                            seed=42)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=9, restore_best_weights=True)\nreducelr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', patience=2, factor=0.5, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# history = model.fit_generator(train_generator,\n#                               epochs=50,\n#                               validation_data=validation_generator,\n#                              callbacks=[earlystopping, reducelr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Save model\n# model.save('big_model.h5')\n# save_history(history, 'big_history.pkl')\n# save_earlystopping(earlystopping, 'big_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"# Load model\nmodel = keras.models.load_model('../input/cat-vs-dog-cnn-models/big_model.h5')\nhistory = load_history('../input/cat-vs-dog-cnn-models/big_history.pkl')\nearlystopping = load_earlystopping('../input/cat-vs-dog-cnn-models/big_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr = report(model, validation_generator, history, earlystopping)\nroc_curves['Double filters'] = ROCCurveParams(fpr, tpr, 'Green', '-')\nplot_roc_curves(roc_curves)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\n\nDoubling the number of filters in each convolutional layer greatly improves training performance, and improves the speed of training to reach the same level of performance as the smaller model (even with longer epoch durations taken into account).\n\nValidation accuracy at the lowest validation loss is roughly the same, as illustrated in the ROC curve. But validation accuracy continues to increase for roughly constant loss, and in particular is almost two percentage points higher for nearly the same loss in epoch 15 compared to epoch 7. The lowest validation loss has slightly improved, but the learning curves still indicate major overfitting."},{"metadata":{},"cell_type":"markdown","source":"## Dataset augmentation"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(150, 150, 3)),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Conv2D(128, (3,3), activation='relu'), \n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.25),\n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(512, activation='relu'),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.Dense(1, activation='sigmoid')  \n])\nmodel.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',\n              metrics = ['acc'])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255.,\n                                   rotation_range=20,\n                                   width_shift_range=0.2,\n                                   height_shift_range=0.2,\n                                   shear_range=0.2,\n                                   zoom_range=0.2,\n                                   horizontal_flip=True)\nvalidation_datagen = ImageDataGenerator(rescale = 1./255.)\n\ntrain_generator = train_datagen.flow_from_dataframe(train_df,\n                                                    train_dir,\n                                                    x_col='filename',\n                                                    y_col='category',\n                                                    batch_size=50,\n                                                    class_mode='binary',\n                                                    target_size=(150, 150),\n                                                       seed=42)     \nvalidation_generator =  validation_datagen.flow_from_dataframe(validate_df,\n                                                          train_dir,\n                                                          x_col='filename',\n                                                          y_col='category',\n                                                         batch_size=50,\n                                                         class_mode  = 'binary',\n                                                         target_size = (150, 150),\n                                                              seed=42)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=9, restore_best_weights=True)\nreducelr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', patience=2, factor=0.5, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# history = model.fit_generator(train_generator,\n#                               epochs=50,\n#                               validation_data=validation_generator,\n#                              callbacks=[earlystopping, reducelr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Save model\n# model.save('daug_model.h5')\n# save_history(history, 'daug_history.pkl')\n# save_earlystopping(earlystopping, 'daug_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"# Load model\nmodel = keras.models.load_model('../input/cat-vs-dog-cnn-models/daug_model.h5')\nhistory = load_history('../input/cat-vs-dog-cnn-models/daug_history.pkl')\nearlystopping = load_earlystopping('../input/cat-vs-dog-cnn-models/daug_earlystopping.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr = report(model, validation_generator, history, earlystopping)\nroc_curves['Data augment'] = ROCCurveParams(fpr, tpr, 'Purple', '-')\nplot_roc_curves(roc_curves)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\n\nDue to the extra image preprocessing, the time per epoch is much longer. The training set accuracy has gone down, but of course the training data is now inconsistent, due to the random image manipulations.\n\nAfter a few initial epochs of variability, the validation performance improves quite a lot. Both validation loss and accuracy beat the corresponding training measures. In terms of validation performance, this is by far the best model in this notebook.\n\nSome mathematical speculation: In some sense, the training image manipulations introduce an extra dimension of variation to the probability manifold from which training images are drawn. The validation set is drawn from the center of all that variation (the mean; all variations exactly zero) and is enveloped by the expanding \"cloud\" of training set points. Thus it ends up sitting nicely in the middle of the training distribution and is therefore particularly well-described by a model trained on the training set."}],"metadata":{"colab":{"collapsed_sections":[],"name":"Course 2 - Part 4 - Lesson 2 - Notebook.ipynb","provenance":[],"toc_visible":true,"version":"0.3.2"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":1}