{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThis is a simple CNN kernel to do binary classification if an image contains a dog or a cat. 1 for dog, 0 for cat. The competition itself finished long time ago, but I found this a good dataset to try to learn some basics of CNN classification.\n\nI use the pre-trained ResNet50 model as a basis to try a little simple analysis of the given images, and try to different image augmentation methods from Keras. I was not familiar with any of this, so I tried most of them to see how they work and what is the modification to the original image they create."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nimport json\nimport math\nimport PIL\nfrom PIL import ImageOps\nfrom keras.models import Sequential, Model\nfrom keras.layers import Dense, Flatten, Activation, Dropout, GlobalAveragePooling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras import optimizers, applications\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, TensorBoard, EarlyStopping\nfrom keras import backend as K \nfrom sklearn.preprocessing import LabelEncoder\n\nfrom tqdm.auto import tqdm\ntqdm.pandas()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Including the cat and dog images, and the pretrained ResNet model weights."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls ../input/dogs-vs-cats/test1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!ls ../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = \"../input/dogs-vs-cats/train/train\"\nfile_list = os.listdir(train_dir)\nDOG = \"dog\"\nCAT = \"cat\"\nTRAIN_TOTAL = len(file_list)\nlabels = []\ndf_train = pd.DataFrame()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Collect the image labels (cat/dog), width/height, and aspect ratio to take a look at the shapes. How far are they from a square shape that the ResNet expects as input?"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nidx = 0\nimg_sizes = []\nwidths = np.zeros(TRAIN_TOTAL, dtype=int)\nheights = np.zeros(TRAIN_TOTAL, dtype=int)\naspect_ratios = np.zeros(TRAIN_TOTAL) #defaults to type float\nfor filename in file_list:\n    if \"cat\" in filename.lower():\n        labels.append(CAT)\n    else:\n        labels.append(DOG)\n    img = PIL.Image.open(f\"{train_dir}/{filename}\")\n    img_size = img.size\n    img_sizes.append(img_size)\n    widths[idx] = img_size[0]\n    heights[idx] = img_size[1]\n    aspect_ratios[idx] = img_size[0]/img_size[1]\n    img.close()\n    idx += 1\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[\"filename\"] = file_list\ndf_train[\"cat_or_dog\"] = labels\nlabel_encoder = LabelEncoder()\ndf_train[\"cd_label\"] = label_encoder.fit_transform(df_train[\"cat_or_dog\"])\ndf_train[\"size\"] = img_sizes\ndf_train[\"width\"] = widths\ndf_train[\"height\"] = heights\ndf_train[\"aspect_ratio\"] = aspect_ratios\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simple data exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[\"aspect_ratio\"].max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[\"aspect_ratio\"].min()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_idx = df_train[\"aspect_ratio\"].values.argmax()\nmax_idx","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.iloc[max_idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filename = df_train.iloc[max_idx][\"filename\"]\nimg = PIL.Image.open(f\"{train_dir}/{filename}\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### The Broadest Image in the Set","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above shows that the data can contain some other elements besides just cats and dogs. My guess is that this dataset was scraped somehow, and the Yahoo logo got left in there by mistake. Are there any other similar images in the set?"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sorted = df_train.sort_values(by=\"aspect_ratio\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_first_9(df_to_plot):\n    plt.figure(figsize=[30,30])\n    for x in range(9):\n        filename = df_to_plot.iloc[x].filename\n        img = PIL.Image.open(f\"{train_dir}/{filename}\")\n        print(filename)\n        plt.subplot(3, 3, x+1)\n        plt.imshow(img)\n        title_str = filename+\" \"+str(df_to_plot.iloc[x].aspect_ratio)\n        plt.title(title_str)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tallest Images\n\nThis should now show the \"tallest\" images in the dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_first_9(df_sorted)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first one is of multiple dogs in one, stacked vertically. The rest are just tight pics of cats and dogs."},{"metadata":{},"cell_type":"markdown","source":"## Widest Images\n\nA look at the broaders/widest similar to the \"tallest\" ones was above:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sorted = df_train.sort_values(by=\"aspect_ratio\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_first_9(df_sorted)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So there is another image that looks like it might have come from scraping some websites. The one with the pink flower/rose in it. Not sure what the third one is, looks a bit like a flea market sale outside. Similar to the stacked dog photo, there are also some with multiple cats next to each other as well.\n\nFunny thing is, someone seems to have labeled each of these as on of cat or dog. It might be worth a closer look at the overall dataset, but I just drop those 3 strange ones for now:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sorted.drop(df_sorted.index[:3], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And look again:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_first_9(df_sorted)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks quite fine now."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = df_sorted","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.dtypes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some basic attributes for training:\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#This batch size seemed to work without memory issues\nbatch_size = 32\n#299 is the input size for some of the pre-trained networks. I think ResNet50 is actually 224x224 but I left this as 299 anyway.\nimg_size = 299 #TODO: 224\n#I will try a few variations of training my model on top of ResNet, 5 seems to be enough to get results but leave some time to try the variants.\nepochs = 7","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Trying out the Keras Generators\n\nFirst a generic function to create generators with different augmentation configurations:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications.resnet50 import preprocess_input\n\ndef create_generators(validation_perc, shuffle=False, horizontal_flip=False, \n                      zoom_range=0, w_shift=0, h_shift=0, rotation_range=0, shear_range=0,\n                     fill_zeros=False, preprocess_func=None):\n    #the \"nearest\" mode copies image pixels on borders when shifting/rotation/etc to cover empty space\n    fill_mode = \"nearest\"\n    if fill_zeros:\n        #with constant mode, we fill created empty space with zeros\n        fill_mode = \"constant\"\n        \n    #rescale changes pixels from 1-255 integers to 0-1 floats suitable for neural nets\n    rescale = 1./255\n    if preprocess_func is not None:\n        #https://stackoverflow.com/questions/48677128/what-is-the-right-way-to-preprocess-images-in-keras-while-fine-tuning-pre-traine\n        #no need to rescale if using Keras in-built ResNet50 preprocess_func: https://github.com/keras-team/keras-applications/blob/master/keras_applications/imagenet_utils.py#L157\n        rescale = None\n\n    train_datagen=ImageDataGenerator(\n        rescale = rescale, \n        validation_split = validation_perc, #0.25, #subset for validation. seems to be subset='validation' in flow_from_dataframe\n        horizontal_flip = horizontal_flip,\n        zoom_range = zoom_range,\n        width_shift_range = w_shift,\n        height_shift_range=h_shift,\n        rotation_range=rotation_range,\n        shear_range=shear_range,\n        fill_mode=fill_mode,\n        cval=0,#this is the color value to fill with when \"constant\" mode used. 0=black\n        preprocessing_function=preprocess_func\n    )\n\n    #Keras has this two-part process of defining generators. \n    #First the generic properties above, then the actual generators with filenames and all.\n    train_generator=train_datagen.flow_from_dataframe(\n        dataframe=df_train,\n        directory=train_dir,\n        x_col=\"filename\", #the name of column containing image filename in dataframe\n        y_col=\"cat_or_dog\", #the y-col in dataframe\n        batch_size=batch_size, \n        shuffle=shuffle,\n        class_mode=\"binary\", #categorical if multiple. then y_col can be list or tuple also \n        #classes=lbls, #list of ouput classes. if not provided, inferred from data\n        target_size=(img_size,img_size),\n        subset='training') #the subset of data from the ImageDataGenerator definition above. The validation_split seems to produce these 2 values.\n\n    valid_generator=train_datagen.flow_from_dataframe(\n        dataframe=df_train,\n        directory=train_dir,\n        x_col=\"filename\",\n        y_col=\"cat_or_dog\",\n        batch_size=batch_size,\n        shuffle=shuffle,\n        class_mode=\"binary\",\n        #classes=lbls,\n        target_size=(img_size,img_size), #gave strange error about tuple cannot be interpreted as integer\n        subset='validation') #the subset of data from the ImageDataGenerator definition above. The validation_split seems to produce these 2 values.\n\n    return train_generator, valid_generator, train_datagen","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## For reference, the plain images, no transformation"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(0, False, False, 0, 0, 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator.class_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map = {v: k for k, v in train_generator.class_indices.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib\n\nfont = {'family' : 'normal',\n        'weight' : 'normal',\n        'size'   : 22}\n\nmatplotlib.rc('font', **font)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_batch_9():\n    train_generator.reset()\n    # configure batch size and retrieve one batch of images\n    plt.clf() #clears matplotlib data and axes\n    #for batch in train_generator:\n    plt.figure(figsize=[30,30])\n    batch = next(train_generator)\n    for x in range(0,9):\n    #    print(train_generator.filenames[x])\n        plt.subplot(3, 3, x+1)\n        plt.imshow(batch[0][x], interpolation='nearest')\n        item_label = batch[1][x]\n        item_label = class_map[int(item_label)]\n        plt.title(item_label)\n\n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the above images are the \"plain\" versions directly from the training data, no transformations. They may look a bit \"stretched\" because Keras resizes them all to the given target shape (299,299 here) on loading. But no other augmentation transformations were done.\n\nTo illustrate this stretching, lets see a few plain images without stretching:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_img(idx):\n    filename = df_train.iloc[idx][\"filename\"]\n    img = PIL.Image.open(f\"{train_dir}/{filename}\")\n    plt.imshow(img)\n    img.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_img(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_img(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Horizontal Flipping\n\nSo how does Keras apply the augmentation transformations? Lets try with only horizontal flipping enabled, and nothing else:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = True, \n                                                                    zoom_range = 0, \n                                                                    w_shift = 0, \n                                                                    h_shift = 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Comparing to the above plotted plain images, we can see that it has now randomly picked to flip the image left or right. Others are unchanged. So it applies the transformation sometimes, but not always."},{"metadata":{},"cell_type":"markdown","source":"## Horizontal Shift\n\nSo what does the horizontal shift look like? Here we set width-shift to max 20%, combined with the horizontal flip transformation."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = True, \n                                                                    zoom_range = 0, \n                                                                    w_shift = 0.2, \n                                                                    h_shift = 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems to pick one or both of the enabled transformations. Since it is random and I have not fixed a seed, it varies which one is applied above (between runs, so cannot say which one was applied now). Getting the original \"non-augmented\" seems rather rare, and likely much more rare if doing this with more transformations. \n\nThe horizontal shifting is filled now with the image pixes on that side, on a random set of 0-20% of the width. This pixel duplication is the \"nearest\" method.\n\n## Multiple in one\n\nLet's try with all the transformations I put in the function:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = True, \n                                                                    zoom_range = 0.2, \n                                                                    w_shift = 0.2, \n                                                                    h_shift = 0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So now we have combinations of zooming, horizontal flipping, horizontal and vertical shifting. And the choice of which are applied varies over epochs."},{"metadata":{},"cell_type":"markdown","source":"## Fill with Zeros\n\nTo try and see some of the effects better, change to use \"constant\" mode and fill empty space with black:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = True, \n                                                                    zoom_range = 0.2, \n                                                                    w_shift = 0.2, \n                                                                    h_shift = 0.2,\n                                                                   fill_zeros = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems clearer to see what happened. Not sure what is the impact on training."},{"metadata":{},"cell_type":"markdown","source":"## Rotation\n\nWell, later I also added rotation as an option, so lets see:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = False, \n                                                                    zoom_range = 0, \n                                                                    w_shift = 0, \n                                                                    h_shift = 0,\n                                                                    fill_zeros = True,\n                                                                   rotation_range=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Shearing\n\nAnd shearing. This one was a bit unclear to me what it does:\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = False, \n                                                                    zoom_range = 0, \n                                                                    w_shift = 0, \n                                                                    h_shift = 0,\n                                                                    fill_zeros = True,\n                                                                   shear_range=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So it vertically \"tilts\" the image. In this case I set 20 for 0-20% \"tilt\". What about a bigger value, what is the effect:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator, valid_generator, train_datagen = create_generators(validation_perc = 0, \n                                                                    shuffle = False, \n                                                                    horizontal_flip = False, \n                                                                    zoom_range = 0, \n                                                                    w_shift = 0, \n                                                                    h_shift = 0,\n                                                                    fill_zeros = True,\n                                                                   shear_range=90)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like smaller shear values might make sense sometimes but the bigger ones can produce a bit of a mess. I tried various values to see if I could make the shearing work also horizontally. No luck. I guess it only support vertical \"shearing\". There is probably some good reason, I just dont know it."},{"metadata":{},"cell_type":"markdown","source":"## ResNet50 Preprocessing function\n\nWhat dos the preprocessing function from Keras ResNet50 model itself do?"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications import resnet50\n\ntrain_generator, valid_generator, train_datagen = create_generators(validation_perc = 0.2, \n                                                                    shuffle = True, \n                                                                    horizontal_flip = True, \n                                                                    zoom_range = 0.2, \n                                                                    w_shift = 0.2, \n                                                                    h_shift = 0.2,\n                                                                    fill_zeros = True,\n                                                                    preprocess_func = resnet50.preprocess_input,\n                                                                   shear_range=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_batch_9()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It all looks rather psychedelic. Probably because the [docs](https://github.com/keras-team/keras-applications/blob/master/keras_applications/imagenet_utils.py#L157)\nsay the function changes values to scale of -1 to 1, and they get clipped on plotting.."},{"metadata":{},"cell_type":"markdown","source":"# Create the Model for Training\n\nRecapt the data and create and train some models:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#the total number of images we have:\ntrain_size = len(train_generator.filenames)\n#train_steps is how many steps per epoch Keras runs the genrator. One step is batch_size*images\ntrain_steps = train_size/batch_size\n#use 2* number of images to get more augmentations in. some do, some dont. up to you\ntrain_steps = int(2*train_steps)\n#same for the validation set\nvalid_size = len(valid_generator.filenames)\nvalid_steps = valid_size/batch_size\nvalid_steps = int(2*valid_steps) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now a method to create the mode(s) for training. Using a previous pre-trained model as a basis, this is called \"transfer learning\". Transferring the learned features and weights from before to now. \n\nDifferent number of the previously defined and trained layers are often set as \"trainable\" of \"fixed\" during training for a new dataset. This is the \"trainable_layer_count\" given here. I will try three variants of this to see how it performs. Hence the parameter.\n\nThis pre-trained model works as a form of readily provided \"feature engineering\" for us. On top of that, a custom classification layer, or a few, is provided. So we add those too here."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.regularizers import l2\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential, load_model\nfrom keras.layers import (Activation, Dropout, Flatten, Dense, GlobalMaxPooling2D,\n                          BatchNormalization, Input, Conv2D, GlobalAveragePooling2D)\nfrom keras.applications.resnet50 import ResNet50\n\ndef create_model(trainable_layer_count):\n    input_tensor = Input(shape=(img_size, img_size, 3))\n    base_model = ResNet50(include_top=False,\n                          #the weights value can apparently also be a file path..\n                   weights=None, #loading weights from dataset, avoiding need for internet conn\n                   input_tensor=input_tensor)\n    base_model.load_weights('../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')\n    if trainable_layer_count == \"all\":\n        #the full pre-trained model is fine-tuned in this case\n        for layer in base_model.layers:\n            layer.trainable = True\n    else:\n        #if not all should be trainable, first set them all as non-trainable (fixed)\n        for layer in base_model.layers:\n            layer.trainable = False\n        #and finally set the last N layers as trainable\n        #idea is to re-use higher level features and fine-tune the finer details\n        for layer in base_model.layers[-trainable_layer_count:]:\n            layer.trainable = True\n    print(\"base model has {} layers\".format(len(base_model.layers)))\n    #here on it is the fully custom classification on top of pre-trained layers above\n    x = GlobalAveragePooling2D()(base_model.output)\n    x = Dropout(0.5)(x)\n    x = Dense(1024, activation='relu', kernel_regularizer=l2(5e-4))(x)\n    x = Dropout(0.5)(x)\n    #doing binary prediction, so just 1 neuron is enough\n    final_output = Dense(1, activation='sigmoid', name='final_output')(x)\n    model = Model(input_tensor, final_output)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Callbacks\n\nSet up some general callbacks for all instances of training. These are:\n\n- checkpoints: save weights of model for best scores while training\n- reduce learning rate: if training gains hit a plateau, try lowering learning rate\n- early stopping: if no gain for several epochs in row, stop\n- logger: log some training details to a file\n\nNot all of these are really needed for this simple example, but I left them here since I find them useful to have around when generally training longer iterations."},{"metadata":{"trusted":true},"cell_type":"code","source":"# create callbacks list\nfrom keras.callbacks import (ModelCheckpoint, LearningRateScheduler,\n                             EarlyStopping, ReduceLROnPlateau,CSVLogger)\n                             \nfrom sklearn.model_selection import train_test_split\n\n\ncheckpoint = ModelCheckpoint('../working/Resnet50_best.h5', monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3,\n                                   verbose=1, mode='auto', epsilon=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n                      patience=7)\n\ncsv_logger = CSVLogger(filename='../working/training_log.csv',\n                       separator=',',\n                       append=True)\n\ncallbacks_list = [checkpoint, csv_logger, early]\n# callbacks_list = [checkpoint, csv_logger, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ResNet50 with all layers trainable"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = create_model(\"all\")\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit_history = model.fit_generator(\n        train_generator,\n        steps_per_epoch=train_steps,\n        epochs = epochs,\n        validation_data=valid_generator,\n        validation_steps=valid_steps,\n        callbacks=callbacks_list,\n    verbose = 1\n)\n#this would load the best scoring weights from above for prediction\nmodel.load_weights(\"../working/Resnet50_best.h5\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And this shows the training score evolution over the epochs:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fit_history.history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This scores around 90-91% validation accuracy at best when I ran it. It seems to have quite some diversity, even if final convergence is close. Still, 90% is not too bad really. Lets see the other two configurations though."},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(fit_history.history).head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_loss_and_accuracy(fit_history):\n    plt.clf()\n    plt.plot(fit_history.history['acc'])\n    plt.plot(fit_history.history['val_acc'])\n    plt.title('model accuracy')\n    plt.ylabel('accuracy')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'test'], loc='upper left')\n    plt.show()\n    plt.clf()\n    # summarize history for loss\n    plt.plot(fit_history.history['loss'])\n    plt.plot(fit_history.history['val_loss'])\n    plt.title('model loss')\n    plt.ylabel('loss')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'test'], loc='upper left')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss_and_accuracy(fit_history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ResNet50 all Layers Fixed\n\nThis version keeps the pre-trained weights as they are and just trains the custom classification layer on top."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = create_model(0)\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator.reset()\nvalid_generator.reset()\nfit_history = model.fit_generator(\n        train_generator,\n        steps_per_epoch=train_steps,\n        epochs = epochs,\n        validation_data=valid_generator,\n        validation_steps=valid_steps,\n        callbacks=callbacks_list,\n    verbose = 1\n)\nmodel.load_weights(\"../working/Resnet50_best.h5\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At best this scored around 92-93% validation accuracy during my test run. Slightly better than the fully trainable. Maybe both of these could use many more epochs, but not today."},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(fit_history.history).head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss_and_accuracy(fit_history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ResNet50 with last 5 layers trainable, others fixed\n\nSo keep all the rest of the pre-trained layers fixed, but keep finetuning the last 5 in the network along with custom classification layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = create_model(5)\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator.reset()\nvalid_generator.reset()\nfit_history = model.fit_generator(\n        train_generator,\n        steps_per_epoch=train_steps,\n        epochs = epochs,\n        validation_data=valid_generator,\n        validation_steps=valid_steps,\n        callbacks=callbacks_list,\n    verbose = 1\n)\nmodel.load_weights(\"../working/Resnet50_best.h5\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As visible, at best this scores close to 98% validation accuracy at best. Nice. Clearly better than the other two above."},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(fit_history.history).head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss_and_accuracy(fit_history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# A look at the models biggest mistakes\n\nAbove, I looked at the images in general. But can we learn anything more about the ones it gives biggest misclassifications for? Lets see what those are.\n\nFirst, re-run predictions on the whole validation set, collect probability of cat/dog:"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_generator.reset()\ndf_valid = pd.DataFrame()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.set_printoptions(suppress=True)\ndiffs = []\npredictions = []\ncat_or_dog = []\ncd_labels = []\nfor filename in tqdm(valid_generator.filenames):\n    img = PIL.Image.open(f'{train_dir}/{filename}')\n    resized = img.resize((img_size, img_size))\n    np_img = np.array(resized)\n    if \"cat\" in filename.lower():\n        reference = 0 #cat\n        cat_or_dog.append(CAT)\n    else:\n        reference = 1 #dog\n        cat_or_dog.append(DOG)\n    cd_labels.append(reference)\n    score_predict = model.predict(preprocess_input(np_img[np.newaxis]))\n#    print(reference)\n#    print(score_predict[0][0])\n    diffs.append(abs(reference-score_predict[0][0]))\n    predictions.append(score_predict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max(diffs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\"diffs\" now has the deviation of assigned label vs predicted, so sorting by that should give the most \"confident mistakes\" the model makes."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_valid[\"filename\"] = valid_generator.filenames\ndf_valid[\"cat_or_dog\"] = cat_or_dog\ndf_valid[\"cd_label\"] = cd_labels\ndf_valid[\"diff\"] = diffs\ndf_valid[\"prediction\"] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_valid.sort_values(by=\"diff\", ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This method plots the top N misclassifications:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_diff_imgs(n):\n    sorted_diffs = df_valid.sort_values(by=\"diff\", ascending=False)\n    x = 0\n    rows = int(math.ceil(n/3))\n    height = rows*10\n    plt.figure(figsize=[30,height])\n    for index, row in sorted_diffs.iterrows():\n        filename = row[\"filename\"]\n        cat_or_dog = row[\"cat_or_dog\"]\n        cd_label = row[\"cd_label\"]\n        diff = row[\"diff\"]\n        prediction = row[\"prediction\"]\n        #print(prediction)\n        pred_str = \"{:.2f}\".format(prediction[0][0])\n        img = PIL.Image.open(f\"{train_dir}/{filename}\")\n        print(filename+\" \"+cat_or_dog+\" \"+str(diff))\n        plt.subplot(3, rows, x+1)\n        plt.imshow(img)\n        title_str = f\"{cat_or_dog}: {cd_label} vs {pred_str}\"\n        plt.title(title_str)        \n        img.close()\n        x += 1\n        if x > n:\n            break\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_diff_imgs(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some of these seem like there is both a cat and a dog in the picture, and thus either label might be correct. Others show small and furry dogs that seem to get mistaken for cats. Overall, humans have no problem identifying these but the model did not. Still, the 5-layer version scores around 98% which is really quite good already."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Predictions for Submission\n\nThis competition ended long time ago, so no submitting anything. But it is always good to finish with a full set so I can come back and copy parts for the next real competition where I go take part and finish in the bottom 10% :).\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dir = \"../input/dogs-vs-cats/test1/test1\"\ntest_filenames = os.listdir(test_dir)\ntest_df = pd.DataFrame({\n    'filename': test_filenames\n})\nnb_samples = test_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.set_printoptions(suppress=True)\npredictions = []\nfor filename in tqdm(test_filenames):\n    img = PIL.Image.open(f'{test_dir}/{filename}')\n    resized = img.resize((img_size, img_size))\n    np_img = np.array(resized)\n    np_img = resnet50.preprocess_input(np_img)\n    score_predict = model.predict(np_img[np.newaxis])\n    predictions.append(score_predict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#1=dog,0=cat\nthreshold = 0.5\ntest_df['probability'] = predictions\ntest_df['category'] = np.where(test_df['probability'] > threshold, 1,0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just as a final sanity check, lets compare some image(s) from the test set and the predicted label. Did we get it right? So no silly mistake of giving does a 0 and cats 1 when it should have been the other way around.."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filename = test_df.iloc[1][\"filename\"]\nimg = PIL.Image.open(f'{test_dir}/{filename}')\nplt.imshow(img)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks like a cat, and the prediction set in test_df seems to match what a cat should have been given. Trying a few values in place of the .iloc(X) above shows similarly correct, so leaving it at that..\n\nAnd a submission file. Of course, I never tried to submit this, so take it with that:"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = test_df.copy()\nsubmission_df['id'] = submission_df['filename'].str.split('.').str[0]\nsubmission_df['label'] = submission_df['category']\nsubmission_df.drop(['filename', 'category'], axis=1, inplace=True)\nsubmission_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Summary\n\nMy goal here was to learn how to use CNNs to do image classification. I guess writing \"raw\" CNN models would be another good step, but this was still an interesting look into building an image classifier with Keras. Have to have a decent understanding to use transfer learning as well. Thats what I tell myself.\n\nThe augmentation experiments were helpful to see how they really work, and I guess I would use similar approaches to understand the transformations against the target domain when selecting the (hyper)parameters for augmentation.\n\nThe best score I got here was from the partly trainable ResNet model. I guess doing longer training sessions on the fully trainable model, and trying different depths of trainable vs fixed in the mixed model would be useful to try. My takeaway is to use transfer learning, or try it at least if I have large-ish datasets and problems, but limited computing power and resources.\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}