{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## 1. Introduction\n\nThis cats and dogs classifier, is based on a model covered in the Coursera 'TensorFlow in Practice Specialization' with some added visualisations and modifications. I am using InceptionV3 pretrained on the imagenet dataset as the main model architecture but I crop off the end of the model and add a new head for the current classification task. The model is trained for just 3 epochs which takes less than 15 minutes using a GPU and gives a good competition score of 0.21213 (though this could be improved by training for longer and optimizing the model more).\n\n\nIn this starter notebook I will go through the code I used to make my model. I will then explore the output results to see where the model made prediction mistakes and discuss some ideas for how to improve the model further. \n\n\n## 2. Contents\n\n1. Introduction\n\n2. Contents\n\n3. Data Pre-Processing\n\n4. EDA  \n\n5. Augmentations with ImageGenerators\n\n6. Creating the Model\n\n7. Training\n\n8. Inference\n\n9. Evaluating the Results\n\n10. Conclusions\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.applications.inception_v3 import InceptionV3\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import Model\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.preprocessing import image\nimport matplotlib.pyplot as plt\nimport zipfile\nimport pandas as pd\nimport os, os.path\nimport shutil\nimport numpy as np\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Data Pre-processing\n\nFirst we need to load and unzip the training and testing images which are compressed in zip folders.\nWe also need to seperate off a section of the training data as a validation set for model evaluation.\nWe do not have any labels provided, instead the names 'dog' and 'cat' form part of the name of each jpeg image.\nWe will be passing the images into the model using ImageDataGenerator. This can assign classification labels based on the subdirectory images are stored in, so by moving all dog images into a `Dog` subdirectory and cat images into a `Cat` subdirectory, we will automatically be labelling the data. This will be done for all images in the training and validation sets but not test (because we do not know the classes of these images yet!).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Opening data zip files\nroot = \"../input/dogs-vs-cats-redux-kernels-edition/\"\nsample_submission = pd.read_csv(root + \"sample_submission.csv\")\n\nwith zipfile.ZipFile(root + 'test.zip',\"r\") as test:\n    test.extractall(\".\")\n    \nwith zipfile.ZipFile(root + 'train.zip',\"r\") as train:\n    train.extractall(\".\")\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating paths to training and test data files\ndata_dir = '../working/train/'\ntest_dir_old = '../working/test/'\n\n# Lists of image file names in training and test files\ndata_list = os.listdir(data_dir)\ntest_list = os.listdir(test_dir_old)\n\n# Partitioning a section of training data for validation\ntrain_valid_split = 0.9\ntrain_number = int(train_valid_split * len(data_list))\n\n# Creating new subdirectories for classes (cats and dogs)\nos.mkdir('../working/train/cats/')\nos.mkdir('../working/train/dogs/')\nos.mkdir('../working/valid/')\nos.mkdir('../working/valid/cats/')\nos.mkdir('../working/valid/dogs/')\nos.mkdir('../working/test/test/')\n\n# Creating paths to new subdirectories\nvalid_dir = '../working/valid/'\ntest_dir_new = '../working/test/test/'\ntrain_cats = '../working/train/cats/'\ntrain_dogs = '../working/train/dogs/'\nvalid_cats = '../working/valid/cats/'\nvalid_dogs = '../working/valid/dogs/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Moving data into subdirectories\n\n# Moving image files into training files for cats and dogs\nfor filename in data_list[:train_number]:\n    category = filename.split('.')[0]\n    if category == 'dog':\n        shutil.move(data_dir+filename, train_dogs+filename)\n    if category == 'cat':\n        shutil.move(data_dir+filename, train_cats+filename)\n\n# Moving image files into validation files for cats and dogs      \nfor filename in data_list[train_number:]:\n    category = filename.split('.')[0]\n    if category == 'dog':\n        shutil.move(data_dir+filename, valid_dogs+filename)\n    if category == 'cat':\n        shutil.move(data_dir+filename, valid_cats+filename)\n\n# Moving train data into train subdirectory \nfor filename in test_list:\n    shutil.move(test_dir_old+filename, test_dir_new+filename)\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Verifying directories contents\nlen_train_cats, len_train_dogs, len_valid_cats, len_valid_dogs, len_test = len(os.listdir(train_cats)), len(os.listdir(train_dogs)), len(os.listdir(valid_cats)), len(os.listdir(valid_dogs)), len(os.listdir(test_dir_new))\nlen_train = len_train_cats + len_train_dogs\nlen_valid = len_valid_cats + len_valid_dogs\nprint(\"There are {} cats and {} dogs in the training set. There are {} cats and {} dogs in the validation set. There are {} images in the test set.\".format(len_train_cats, len_train_dogs, len_valid_cats, len_valid_dogs, len_test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. EDA\n\nLets take a look at our preprocessed data set:\n\nWe can first see the split of the different classes amongst the training, validation and testing sets. We can see that there are nearly equal numbers of cats and dogs images in both the training and validation set. We do not know how many cats or dogs are in the test set. This is good, the validation set should be representative of the training set. The test set is very large compared to the validation set, increasing the size of the validation set could be useful for more accurate evaluation, but for now I will leave it as it is. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the data to plot\ndata_sets = ['Training', 'Validation', 'Testing']\ncats = [len_train_cats, len_valid_cats, 0]\ndogs = [len_train_dogs, len_valid_dogs, 0]\nunlabelled = [0, 0, len_test]\nlabels = ['Cat Images', 'Dog Images', 'Unlabelled Images']\n\n# Set position and width of bars\npos = list(range(len(data_sets)))\nwidth = 0.25 \n    \n# Plotting the bars\nfig, ax = plt.subplots(figsize=(10,5))\n\n# Create a bar with cat data in position pos\nplt.bar(pos, cats, width, alpha=1, color='#EE3224', label=labels[0]) \n\n# Create a bar with dogs data in position pos + some width buffer,\nplt.bar([p + width for p in pos], dogs, width, alpha=1, color='#F78F1E', label=labels[1]) \n\n# Create a bar with unlabelled data,\n# in position pos + some width buffer,\nplt.bar([p + width*2 for p in pos], unlabelled, width, alpha=1, color='#FFC222', label=labels[2]) \n\n# Set the y axis label\nax.set_ylabel('Quantity in set', fontsize=14)\n\n# Set the chart's title\nax.set_title('Categories of data in each set', fontsize=16)\n\n# Set the position of the x ticks\nax.set_xticks([p + width for p in pos])\n\n# Set the labels for the x ticks\nax.set_xticklabels(data_sets, fontsize=14)\n\n# Remove grid lines\nax.grid(None)\n\n# Adding the legend and showing the plot\nplt.legend(labels, loc='upper center')\nplt.grid()\nplt.show()                       ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Sampling Images\nWe can also take a look at the images in the data sets. \nIf we look only at the dogs we can see that there are many different dog breeds of different colours and sizes in different poses. Some are even cropped and some have people in the images too, which makes them more compositionally complex. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting images of only dogs\n\ndimensions = 5\n\nfig, axs = plt.subplots(dimensions, dimensions, figsize=(dimensions * 4, dimensions * 4))\nfor i in range(0, dimensions * dimensions):\n    \n    doggy_id = os.listdir(train_dogs)[i]\n    \n    row = i // dimensions\n    col = i % dimensions\n    img_PIL = Image.open(train_dogs+doggy_id)\n    im_resized = img_PIL.resize((150,150))\n    ax = axs[row][col]\n    ax.imshow(im_resized)\n    ax.set_title(\"Dog\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Augmentations with ImageGenerators\n\nNow that we have sorted our data into subdirectories and confirmed that there is a good split of cats and dogs in each data set we will use ImageDataGenerator to load the data into the model. We can use this Generator to easily perform augmentations on the fly. I have normalised the images in all datasets by rescaling the pixel values (from 0 - 255 to 0 - 1). I have added several different rotations, image shifts, flips and zooms which should help to amplify the training data by simulating dogs of different sizes, angles and positions. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adding augmentations with datagenerator\n\ntrain_gen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=40,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest')\n\nvalid_gen = ImageDataGenerator(\n    rescale=1./255)\n\ntest_gen = ImageDataGenerator(\n    rescale=1./255)\n\n# Creating generators\ntrain_generator = train_gen.flow_from_directory(data_dir, batch_size=64, class_mode='binary', target_size=(150, 150))\nvalid_generator = valid_gen.flow_from_directory(valid_dir, batch_size=64, class_mode='binary', target_size=(150, 150))\ntest_generator = test_gen.flow_from_directory(test_dir_old, batch_size=64, class_mode='binary', shuffle=False, target_size=(150, 150))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Creating the Model\n\nNext we can load a pretrained model for transfer learning. I am using InceptionV3 pretrained on imagenet because it is available automatically with keras. I am ending the model early at the 'mixed7 layer' and instead adding my own layers at the end, flattening the output and using densely connected layers to get a binary prediction of 1 for 'dog' or 0 for 'cat'. The pretrained layers are set to non-trainable while training on the final newly added layers still take place. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Pretrained model\n\n# Loading InceptionV3 pretrained on imagenet\npre_trained_model = InceptionV3(input_shape=(150, 150, 3),\n                                include_top=False,\n                                weights='imagenet')\n    \n# Removing the last layers of the model\nlast_layer = 'mixed7'\nmodel = Model(inputs=pre_trained_model.input, outputs=pre_trained_model.get_layer(last_layer).output)\nmodel.summary()\n\n# Creating a new model by adding different layers to the end of the cropped pre-trained model\nnew_model = tf.keras.models.Sequential([\n    model,\n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(1024, activation='relu'),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(1, activation='sigmoid')\n])\n\n# Setting the pretrained model to non-trainable\nnew_model.layers[0].trainable = False\nnew_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Training\n\nA learning rate reducer is added to optimise the learning rate during training and the model is compiled.\n\nThe metrics of the data fitting process for both the validation and training sets are visualised below.\n\nInterestingly the validation metrics are better than the training set - perhaps due to the small sample size and augmentations only applied to the train set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add a learning rate scheduler\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compiling the model\nnew_model.compile(optimizer='rmsprop',\n              loss=tf.keras.losses.BinaryCrossentropy(),\n              metrics=['accuracy'])\n\n# Training\nNUM_EPOCHS = 3 \nhistory = new_model.fit(train_generator, steps_per_epoch=len_train/64, epochs=NUM_EPOCHS, validation_data=(valid_generator), validation_steps=len_valid/64, callbacks=[learning_rate_reduction], verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Metrics\n\nacc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\n# Plotting accuracy\nplt.figure(figsize=(18, 6))\nplt.subplot(1, 2, 1)\nplt.plot(acc, label='Training Accuracy')\nplt.plot(val_acc, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.ylabel('Accuracy', fontsize=12)\nplt.ylim([min(plt.ylim()),1])\nplt.title('Training and Validation Accuracy')\nplt.xlabel('epoch', fontsize=12)\n\n# Plotting loss\nplt.subplot(1, 2, 2)\nplt.plot(loss, label='Training Loss')\nplt.plot(val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.ylabel('Binary Cross Entropy', fontsize=12)\nplt.ylim([0,1.0])\nplt.title('Training and Validation Loss')\nplt.xlabel('epoch', fontsize=12)\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 8. Inference\n\nNow that the model is trained we can use it to make predictions of the contents of the images for the test set. \nWith 3 epochs training we saw that the validation and training graph's hadn't converged, and more training might have improved the model. However, we still get a pretty good results for log loss on the test set: 0.212. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predicting the test image classes\npredictions = new_model.predict(test_generator)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting the output predictions to the submission format for scoring\npredictions_list = predictions.tolist()\nfor index, item in enumerate(predictions_list):\n    predictions_list[index] = float(item[0])\n    \nresults_df = pd.DataFrame(\n    {\n        'id': pd.Series(test_generator.filenames), \n        'label': pd.Series(predictions_list)\n    })\nresults_df['id'] = results_df.id.str.extract('(\\d+)')\nresults_df['id'] = pd.to_numeric(results_df['id'], errors = 'coerce')\nresults_df.sort_values(by='id', inplace = True)\n\nresults_df.to_csv('submission.csv', index=False)\nresults_df.head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9. Evaluating the Results\n\nWe can visualise the results of the model by plotting some of the images and comapring them to their class probabilities. Values close to 1 mean that the model is very confident that the images is a dog. Values close to 0 mean that the model is confident the image contains a cat. Values in the middle i.e. 0.5 occur when the model is very unsure which animal class the image contains. \n\nLooking at a selection of these images below we can see that there is high confidence in the class of many images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting the images and labelling with class predicitions\ndimensions = 5\nfig, axs = plt.subplots(dimensions, dimensions, figsize=(dimensions * 4, dimensions * 4))\nfor i in range(0, dimensions * dimensions):\n    \n    doggy_id = i + 1 +100\n    \n    row = i // dimensions\n    col = i % dimensions\n    img_PIL = Image.open(test_dir_new+ f\"{doggy_id}.jpg\")\n    im_resized = img_PIL.resize((150,150))\n    ax = axs[row][col]\n    ax.imshow(im_resized)\n    ax.set_title(f\"Prediction: {results_df.label[results_df.id == doggy_id].values[0]:.4f} ID: {doggy_id}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we can look at how the prediction values are spread. The histogram plot below shows high confidence in the majority of the images classes (i.e. 1 or 0). \nHowever, there is a small spread of low confidence images in between these two histogram columns. \n\nWhy do these images confuse the model?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = results_df['label'].plot.hist(bins=10, alpha=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sorting the data to find the images with the lowest confidence is achieved by subtracting 0.5 from each class probability and sorting in ascending order. Images with values closes to 0.5 now appear at the top of the list, these are the images that the model had no idea which class it was. Next we can plot these images and take a look to see where the model was going wrong. \n\nThese images do seem much more difficult, there are some images which contain composite figures or text, very zoomed out cropped images, drawings of animals, images with many animals, pets wearing clothes and even some images which are not of animals at all. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sorted_list = results_df\nsorted_list['uncertainty'] = abs(sorted_list['label']-0.5)\nsorted_list = sorted_list.sort_values(by = ['uncertainty'])\nsorted_list = sorted_list.reset_index(drop=True)\nsorted_list.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dimensions = 6\nfrom PIL import Image\nfig, axs = plt.subplots(dimensions, dimensions, figsize=(dimensions * 4, dimensions * 4))\nfor i in range(0, dimensions * dimensions):\n    \n    doggy_id = sorted_list.id[i]\n    \n    row = i // dimensions\n    col = i % dimensions\n    img_PIL = Image.open(test_dir_new+ f\"{doggy_id}.jpg\")\n    im_resized = img_PIL.resize((150,150))\n    ax = axs[row][col]\n    ax.imshow(im_resized)\n    ax.set_title(f\"Prediction: {sorted_list.label[sorted_list.id == doggy_id].values[0]:.4f} ID: {doggy_id}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 10. Conclusions\n\n* Using a pretrained model with some small modifications, we can get a really good result with only 3 epochs training. \n\n* We took a look at which images the model was unsure of and these were of animals in unusual positions and image compositions. The model would probably really benefit from more training with more augmentatations:\n\n\n1. A bigger zoom range to replicate the cropped images/white space images.\n2. Blur to replicate the low resolution images.\n3. Making more composite images by combining different figures.\n4. Geater rotations/sheers to replicate pets in unusual positions. \n\n\n* Manually removing the 'bad' images from the training set i.e. those which are not of cats and dogs would probably be beneficial to training too. However, because these images are also found in the test set (which is likely considering that the test and train sets are both very large) then it may not help to improve score too much. \n\nI hope you enjoyed reading this starter notebook and eda. Hopefully you can replicate the work and try to improve the model by applying more augmentations or modifying the model in other ways. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}