{"cells":[{"metadata":{},"cell_type":"markdown","source":"The aim of this notebook is to illustrate how to create a classification model using transfer learning and keras."},{"metadata":{},"cell_type":"markdown","source":"# Import the data\n\nWe will use the dogs vs. cats dataset, which is composed of a total number of 25,000 images: 12,500 corresponds to dogs and 12,500 to cats. The test contains 12,500 images. All images are RGB.\n\nSince images are not inside subfolders with their respective class names, we have to create a pandas DataFrame specifying the correspondent filename and class.\n\nTo begin with, we have deleted some images (50) from the training set since some of them don't show real dogs or cats, or contain dogs and cats in the same image. (See [Relabeling mistakes in training data](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/discussion/29162#latest-302376)). \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nimport matplotlib.pyplot as plt\n\n# Train and test directory path\ntrain_dir = '../input/dogs-vs-cats-redux-kernels-edition/train/'\ntest_dir = '../input/dogs-vs-cats-redux-kernels-edition/test/'\n\n# Wrong images in the dataset\nwrong_images = ['dog.11731.jpg', 'dog.4334.jpg', 'cat.4688.jpg', \n                'cat.11222.jpg', 'cat.1450.jpg', 'cat.2159.jpg', \n                'cat.3822.jpg', 'cat.4104.jpg', 'cat.5355.jpg', \n                'cat.7194.jpg', 'cat.7920.jpg', 'cat.9250.jpg', \n                'cat.9444.jpg', 'cat.9882.jpg', 'dog.11538.jpg', \n                'dog.8507.jpg', 'cat.2939.jpg', 'cat.3216.jpg', \n                'cat.4833.jpg', 'cat.7968.jpg', 'cat.8470.jpg', \n                'dog.10161.jpg', 'dog.10190.jpg', 'dog.11186.jpg', \n                'dog.1308.jpg', 'dog.1895.jpg', 'dog.9188.jpg', \n                'cat.5351.jpg', 'cat.5418.jpg', 'cat.9171.jpg',\n                'dog.10747.jpg', 'dog.2614.jpg', 'dog.4367.jpg', \n                'dog.8736.jpg', 'cat.7377.jpg', 'dog.12376.jpg', \n                'dog.1773.jpg', 'cat.10712.jpg', 'cat.11184.jpg', \n                'cat.7564.jpg', 'cat.8456.jpg', 'dog.10237.jpg', \n                'dog.1043.jpg', 'dog.1194.jpg', 'dog.5604.jpg',\n                'dog.9517.jpg', 'cat.11565.jpg', 'dog.10797.jpg', \n                'dog.2877.jpg', 'dog.8898.jpg']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see some of these images. To do so, we write a function to visualize these images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_grid_images(images_directory, images_label, n, m):\n    \"\"\"\n    Shows a grid of images (5x5) with their corresponding label.\n    \n    Args:\n        images_directory (list of str): Contains the namefiles of each image\n        images_label (list of str): Contains the label of each image\n        n (int): Number of rows\n        m (int): Number of columns\n    \"\"\"\n    f, ax = plt.subplots(n, m, figsize = (10, 10))\n\n    for i in range(0,n*m):\n        imgBGR = cv2.imread(images_directory[i])\n        imgRGB = cv2.cvtColor(imgBGR, cv2.COLOR_BGR2RGB)\n        ax[i//5, i%5].imshow(imgRGB)\n        ax[i//5, i%5].axis('off')\n        ax[i//5, i%5].set_title(\"{}\".format(images_label[i]))\n\nimages_directory = [train_dir + image for image in wrong_images]\nplot_grid_images(images_directory, wrong_images, 5, 5)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we build our training and test dataframe:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\n\n# Extract the path to each image from directory\nX_train = ['../input/dogs-vs-cats-redux-kernels-edition/train/{}'.format(i) \n           for i in sorted(os.listdir(train_dir)) \n           if i not in wrong_images and i != 'train']\n\n# Set 0 for cat and 1 to dog\ny_train = ['cat' if 'cat.' in i else 'dog' for i in X_train]\n# Note: Keep '.cat' instead of 'cat' for proper functioning of os\nX_test = ['../input/dogs-vs-cats-redux-kernels-edition/test/{}'.format(i) \n          for i in sorted(os.listdir(test_dir)) if i != 'test']\n\n\n# Create dataframes\ntrain_df = pd.DataFrame({'filename':X_train, 'class':y_train})\ntest_df = pd.DataFrame({'filename':X_test})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training\nOnce we have our dataset, we have to address the training. For this notebook, we will use transfer learning: we take a trained neural network and keep all the weights except the ones from the last layers. Those last layers' weights will be trained for the current classification problem. \n\nFor assessing the performance during training, we will split the dataset into training (24,700) and validation (250). "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the sizes of each set\nNUM_VALIDATION = 500\nNUM_TRAIN = len(train_df) - NUM_VALIDATION\nNUM_TEST = len(test_df)\n\n# Split to train/validation/test\ntrain_df = train_df[:NUM_TRAIN+NUM_VALIDATION].sample(frac=1)\nvalidation_df = train_df[NUM_TRAIN:NUM_TRAIN+NUM_VALIDATION]\ntrain_df = train_df[:NUM_TRAIN]\ntest_df = test_df[:NUM_TEST]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Besides, we use data augmentation to increase the number of images for training. Also, the number of **epochs** will be set to 32. To set the batch size and the steps per epoch, we can follow this formula:\n\n$$\n\\text{no. images} = \\text{batch size} \\cdot \\text{batch steps}\n$$"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data augmentation specifications\nHORIZONTAL_FLIP = True\nWIDTH_SHIFT_RANGE = 0.2\nHEIGHT_SHIFT_RANGE = 0.2\nROTATION_RANGE = 40\n\n# Training specifications\nNUM_EPOCHS = 10\nBATCH_SIZE = 32 \nSTEPS_PER_EPOCH_TRAINING = NUM_TRAIN // BATCH_SIZE\nSTEPS_PER_EPOCH_VALIDATION = NUM_VALIDATION // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Two data generators are trained for training and validation. Here, we add the specification for data augmentation (just for the training set), as well as the correspondent processing function of the pre-trained network.\n\nThere are many different pre-trained neural networks for image classification. The various available networks in Keras are documented in [Keras' Applications](https://keras.io/applications/). Networks differ in size and performance. Bigger networks will need more time to compute a batch. At the same time, bigger networks don't guarantee better performance. Also, we have to tailor the input for the network we are using. For instance, if using an Xception network, the input has to be 299x299, but if we were to use NASNetLarge, it should be 331x331. For this notebook, we use the ResNet50 network (input size of 224x224). ResNet50 is a network with more than 25 million parameters with a top-1 accuracy of 0.749 in the ImageNet dataset.\n\nCats correspond to the label 0 and dogs to 1. This labeling happens due to the file ordering (cat files appear first)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.python import keras\nfrom tensorflow.python.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.python.keras.applications.resnet50 import preprocess_input\n\n# Image size for ResNET\nIMG_SIZE = 224\n\n# Create image generators to preprocess and group images into training \n# and validation\ndata_gen_train = ImageDataGenerator(preprocessing_function=preprocess_input, \n                                    horizontal_flip=HORIZONTAL_FLIP, \n                                    rotation_range=ROTATION_RANGE, \n                                    width_shift_range=WIDTH_SHIFT_RANGE, \n                                    height_shift_range=HEIGHT_SHIFT_RANGE)\ndata_gen_val = ImageDataGenerator(preprocessing_function=preprocess_input)\n\n# Use .flow_from_dataframe since images are not in subfolders. \n# Otherwise use .flow_from_subfolder\ntrain_gen = data_gen_train.flow_from_dataframe(dataframe=train_df, \n                                               x_col='filename', \n                                               y_col='class', \n                                               target_size=(IMG_SIZE, \n                                                            IMG_SIZE), \n                                               batch_size=BATCH_SIZE, \n                                               class_mode='categorical')\nval_gen = data_gen_val.flow_from_dataframe(dataframe=validation_df, \n                                           x_col='filename', \n                                           y_col='class', \n                                           target_size=(IMG_SIZE, \n                                                        IMG_SIZE), \n                                           batch_size=BATCH_SIZE, \n                                           class_mode='categorical')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we want the output to be probabilities (of the image being a dog or a cat), the last layer will be a 2-neuron **softmax**. We use the **adam optimizer** (RMSProp + momentum) together with **cross-entropy loss** (also called log loss) to assess the model evolution. Cross-entropy is defined as:\n\n$$\n\\mathcal{L} (w) = -\\frac{1}{N} \\sum^{N}_{i=1} \\big[y_i \\log{(\\hat y_i)} + (1-y_i)\\log{(1- \\hat y_i)} \\big],\n$$\n\nwhere the loss $\\mathcal{L}$ is a function of the weights $w$, $ N$ are the total number of images in the validation set, $y_i$ is the correct label for a given image, and $\\hat y_i$ is a prediction probability of the image being class $y_i$. If the closer is the probability to 1, the lesser will be the loss. The same happens for the other label, $1 - y_i$. The total loss is averaged out across images.\n\n\nIn the next cell, we build the network. We load ResNet50 and add two more layers: the first one with 128 neurons is a bottleneck layer; the second one with two neurons is in charge of the final classification. We freeze the top layers of the network since we are doing transfer learning."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.python.keras.applications.resnet50 import ResNet50\nfrom tensorflow.python.keras.models import Sequential\nfrom tensorflow.python.keras.layers import Dense\nfrom tensorflow.python.keras import optimizers\n\n# ResNet50 specifications\nNUM_CLASSES = 2\nPOOLING = 'avg'\nLAYER_ACTIVATION = 'softmax'\nLOSS = 'categorical_crossentropy'\nLOSS_METRICS = ['accuracy']\n\n# Create sequential model\nmodel = Sequential()\n\n# First layer: ResNet50. If not specified, weights are retrieved directly \n# from the repository (Internet must be ON)\nmodel.add(ResNet50(include_top=False, pooling=POOLING))\n\n# Second layer: Bottleneck layer\nmodel.add(Dense(128, activation=LAYER_ACTIVATION))\n\n# Third layer: Dense layer for the two different classes\nmodel.add(Dense(NUM_CLASSES, activation=LAYER_ACTIVATION))\n\n# Provided the ResNet50 has been already trained, we are interested in tuning \n# the weights of the last layer (classification). Therefore, we keep them \n# fixed\nmodel.layers[0].trainable = False\n\n# Specification of optimizer, compilation of the model and summary\nadam = optimizers.adam(lr=0.01)\nmodel.compile(optimizer=adam, loss=LOSS, metrics=LOSS_METRICS)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The total number of trainable parameters is 262,530, which quite large. However, nothing compared to the weights we had to train if we use were to train the full network: 23,587,712. \n\nWe want to stop training if there is not any substantial improvement in the validation set. This process is known as **early stopping** and helps to prevent overfitting."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint\n\n# Early stopping + checkpoints\nEARLY_STOPPING_PATIENCE = 3 # has to be smaller than NUM_EPOCHS\ncb_early_stopper = EarlyStopping(monitor='val_loss', \n                                 patience=EARLY_STOPPING_PATIENCE)\ncb_checkpointer = ModelCheckpoint(filepath='../working/best.hdf5', \n                                  monitor='val_loss', \n                                  save_best_only=True,\n                                  mode='auto')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fit the model and store the best weights for further retrieval."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Validate the model\nfit_history = model.fit_generator(\n        train_gen,\n        steps_per_epoch=STEPS_PER_EPOCH_TRAINING,\n        epochs=NUM_EPOCHS,\n        validation_data=val_gen,\n        validation_steps=STEPS_PER_EPOCH_VALIDATION,\n        callbacks=[cb_checkpointer, cb_early_stopper] \n)\nmodel.load_weights('../working/best.hdf5')\n\n# Print the different keys from the history fit\nprint(fit_history.history.keys())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot the accuracy and loss across epochs:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\n\n# Plot the responses for different events and regions\nsns.set(style=\"darkgrid\")\n\nplt.figure(figsize=(14,3.5))\nax1 = plt.subplot(121)\nsns.lineplot(x=range(len(fit_history.history['acc'])), \n             y=fit_history.history['acc'], label='acc')\nsns.lineplot(x=range(len(fit_history.history['val_acc'])), \n             y=fit_history.history['val_acc'], label='val_acc')\nax1.set(xlabel='epochs', ylabel='accuracy')\nax1.set_ylim(0.9,1)\nax1.set_title('Accuracy')\n\nax2 = plt.subplot(122)\nsns.lineplot(x=range(len(fit_history.history['loss'])), \n             y=fit_history.history['loss'], label='loss')\nsns.lineplot(x=range(len(fit_history.history['val_loss'])), \n             y=fit_history.history['val_loss'], label='val_loss')\nax2.set(xlabel='epochs', ylabel='loss')\nax2.set_title('Loss');\n\n# Note: x is set to be range(len(it_history.history['val_acc'])) and not range(NUM_EPOCHS) since early stopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nfrom sklearn.metrics import classification_report, confusion_matrix\n\n# Validation without shuffle\nval_gen = data_gen_val.flow_from_dataframe(dataframe=validation_df,\n                                           x_col='filename',\n                                           y_col='class',\n                                           target_size=(IMG_SIZE, \n                                                        IMG_SIZE),\n                                           batch_size=BATCH_SIZE,\n                                           class_mode='categorical',\n                                           shuffle=False,\n                                           seed=1234)\n# For reproducibility\nval_gen.reset()\n\n# Predict the validation set\ny_pred = np.argmax(model.predict(val_gen), axis=1)\n\n# Confusion Matrix\nsns.heatmap(confusion_matrix(val_gen.classes, y_pred), \n            cmap='RdBu', annot=True, annot_kws={\"size\": 15}, \n            fmt = '.0f', xticklabels=['cats', 'dogs'], \n            yticklabels=['cats', 'dogs']).set_title(\"Confusion Matrix\", fontsize=18);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Classification Report')\nprint(classification_report(val_gen.classes, \n                            y_pred, target_names=['Dogs', 'Cats']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Testing\n\nAfter training, we take the model and predict the class for the images in the test set. Theoretically, we have to train a model with all the train and validation set again. However, in this notebook, we will continue using the model on the training set only."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Full batch of testing\nBATCH_SIZE_TESTING = 1\n\n# Image generator for full dataset and the testing images (not labeled)\ndata_gen_test = ImageDataGenerator(preprocessing_function=preprocess_input)\ntest_gen = data_gen_test.flow_from_dataframe(dataframe=test_df,\n                                             x_col='filename',\n                                             class_mode = None,\n                                             target_size=(IMG_SIZE,\n                                                          IMG_SIZE),\n                                             batch_size=BATCH_SIZE_TESTING,\n                                             shuffle=False,\n                                             seed=1234)\n\n# Each time this cell is run, we start again to go through the images\ntest_gen.reset()\n\n# Predict the class of test images\nprediction = model.predict(test_gen, steps=len(test_gen), verbose=1)\npredicted_class_indices = np.argmax(prediction, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although we don't have the real labels, we can print some of the images with the labels and assess whether the model performed as expected or not."},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_labels = ['dog' if i == 1 else 'cat' \n                    for i in predicted_class_indices]\nplot_grid_images(test_gen.filenames, predicted_labels, 5, 5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save the data\n\nFinally, we store the data in *csv* for the submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Retrieve names and probabilities\nfilenames = test_gen.filenames\nprobabilities = prediction\n\n# Format according to competition submission requirements\nlabel = [int(filenames[i].split('/')[-1].split('.')[0]) \n         for i in range(len(probabilities))]\nprob = [probabilities[i][1] for i in range(len(probabilities))]\n\n# Save\noutput = pd.DataFrame({'id': label,\n                       'label': prob})\noutput.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References\n1. [Kaggle's Deep Learning Course](https://www.kaggle.com/learn/deep-learning)\n2. [Tutorial Keras Transfer Learning with ResNet50](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50)\n3. [Relabeling mistakes in training data](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/discussion/29162#latest-302376)\n4. [Tradeoff batch size vs number of iterations](https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network)\n5. [Keras Documentation](https://keras.io/)\n6. [Confusion Matrix with Keras](https://gist.github.com/RyanAkilos/3808c17f79e77c4117de35aa68447045)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}