{"cells":[{"metadata":{"_uuid":"2319ba962cda9b4a707ccb94e917e01304676c04"},"cell_type":"markdown","source":"

Transfer learning for dog breed classification - Part II

\n\nIn every single computer vision competition, top rank competitors almost never train their networks from scratch. They transfer knowledge from pretrained models to theirs. This is something I painfully learned in the first official competition I took place in (TGS Salt Identification), and I wish someone would have told me then. For that reason, I'm planning to create a series of kernels for beginners in which I'll cover the basic techniques in transfer learning. I will only touch the problem of image classification, but these techniques could be applied to any other computer vision problem (image segmentation, object detection, ...). At the same time, I will be covering some useful techniques which I also painfully learned about such as using ImageDataGenerator for reading images into memory from disk and data augmentation, and the Cyclic learning rates method.\n\nThe series will have three parts, in each of which I will cover three basic techniques and some of its variants. This is part II. In this part I've switched to the InceptionV3 model, as I wasn't getting very good results with the VGG16 model.\n\n\n
  • Part I - Extract features from the last convolutional block of VGG16 and use them to train a NN classifier, without data augmentation
    • \n
  • Part II - Freeze an InceptionV3 base model, put a classifier on top of it and train the model with data augmentation.
    • \n
  • Part III - Starting from the model trained in Part 2, unfreeze the last convolutional block and finetune the network with a small learning rate.
    • \n    \n
    \n

    \nIn this kernel I will be using cyclical learning rates. Thanks to Leslie N. Smith for his excellent contribution. You can find his article about cyclic learning rates here and the github repository with the code in Keras here. To use this library, you should upload the file to a dataset of your own and copy from there to your working directory (see cell below)."},{"metadata":{"trusted":true,"_uuid":"dfaaf1f79679b88e5089a597a01192884a82986b"},"cell_type":"code","source":"# Copy python scripts from my library my_python, which contains the clr_callback script\n# to the working directory.\n! cp ../input/my-python/* ../working/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"38f1edf12d03cb2b3523789b7761c175cc5f8924"},"cell_type":"code","source":"# Generic imports\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nimport cv2\nimport gc\nfrom tqdm import tqdm_notebook\n%matplotlib inline\nsns.set_style('whitegrid')\n\n# sklearn imports\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import OneHotEncoder\n\n# Keras imports\nfrom keras.applications import VGG16, ResNet50, ResNet50V2, InceptionResNetV2\nfrom keras.applications.inception_v3 import InceptionV3\nfrom keras.applications.xception import Xception\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, GlobalAveragePooling2D\nfrom keras.optimizers import RMSprop, Adam, SGD\nfrom keras.preprocessing.image import load_img, img_to_array\nfrom keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n\n# Other imports\nfrom clr_callback import * \nimport imgaug as ia\nfrom imgaug import augmenters as iaa\nfrom IPython.display import FileLink, FileLinks","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"267b3bd0bcb014ced495f0e5a66c58575ec59eeb"},"cell_type":"code","source":"# Constants\nDATA_DIR = '../input/dog-breed-identification/'\nTRAIN_DIR = DATA_DIR + 'train/'\nTEST_DIR = DATA_DIR + 'test/'\nBATCH_SIZE = 128\nINPUT_SIZE = 299\nNUM_CLASSES = 120\nSEED = 42","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d2e754a75c561d10af4baa14541dc838975d627"},"cell_type":"code","source":"# Let's check what's in the data directory\n! ls $DATA_DIR","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e82fcdad16d631412ba18fae9e867abe80f2791"},"cell_type":"code","source":"# Read the train data set, which has the ids of the images and their labels (breeds)\n# (adding the extension .jpg to the id becomes the file name of the image) \ntrain = pd.read_csv(DATA_DIR + 'labels.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64b100838feecd314621e132ce1a48038d17013f"},"cell_type":"code","source":"breeds_20 = train.breed.value_counts().head(NUM_CLASSES).index\ntrain = train[train.breed.isin(breeds_20)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c95772ecf8aee2b9f5b7f0a396069f0ec1876f01"},"cell_type":"code","source":"# The submission file contains one column for the image id, and then one column \n# each breed in alphabetical order, with the probability of the dog in the image beeing of that breed\nsubmission = pd.read_csv(DATA_DIR + 'sample_submission.csv')\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f92ecc0aea185d45b2df60d5bf276450bfc2f9ce"},"cell_type":"code","source":"# Get the breeds to pass them to the generators for creating the same labels for train set and validation set\n#breed_labels = list(submission.columns[1:].values)\nbreed_labels = list(breeds_20.values)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c51c5018554f37cd2f58f10765d5bf63386f64ae"},"cell_type":"markdown","source":"

    Transfer learning

    \n\nWe'll read the InceptionV3 base model and put on top of it a classifier suited for this problem (120 dog breeds). This way we'll leverage all the knowledge learned by this base model during training with the ImageNet data set (1.4 millio images). We'll freeze the base model it so its weights don't change during training."},{"metadata":{"trusted":true,"_uuid":"c58be4becf1b2a7f3a33fed6146f58e057975f4e"},"cell_type":"code","source":"# Now we create our model\n\ndef create_model(lr=0.0001, dropout=None):\n \n model = Sequential()\n \n #base = InceptionV3(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3))\n #base = Xception(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3))\n #base = VGG16(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3))\n #base = ResNet50(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3))\n base = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3))\n #base.layers.pop() # Remove the last layer (softmax classifier)\n\n for layer in base.layers:\n layer.trainable = False\n \n model.add(base)\n model.add(GlobalAveragePooling2D())\n model.add(Dense(NUM_CLASSES, activation='softmax'))\n\n adam = Adam(lr=lr)\n sgd = SGD(lr=0.1, momentum=0.95, nesterov=False)\n model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['acc'], )\n\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6922b23e021eb92bf59be82d7386343b73404153"},"cell_type":"code","source":"# We split images into training (80%) and validation (20%)\ntrain_set, val_set = train_test_split(train, test_size=0.20, stratify=train['breed'], random_state=SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"868d7a7ec6053f3feae10a0851502e0b0c34bee2"},"cell_type":"code","source":"# Create the generators and data augmenters\n\ndef keypoints(keypoints_on_images, random_state, parents, hooks):\n return keypoints_on_images\n\ndef rescale_imgs(images, random_state, parents, hooks):\n for img in images:\n img = img / 255.\n return images \n \naugs = iaa.Sequential([\n iaa.Fliplr(0.5),\n iaa.Sometimes(0.2, iaa.Affine(rotate=(-20,20), mode='edge')), \n iaa.SomeOf((0,4), [\n iaa.AdditiveGaussianNoise(scale=0.01*255), \n iaa.Sharpen(alpha=(0.0,0.3)),\n iaa.ContrastNormalization((0.8,1.2)),\n iaa.AverageBlur(k=(2,11)),\n iaa.Multiply((0.8,1.2)),\n iaa.Add((-20,20), per_channel=0.5),\n iaa.Grayscale(alpha=(0.0,1.0))\n ])\n #, iaa.Lambda(rescale_imgs, keypoints)\n]) \n\ntrain_datagen = ImageDataGenerator(preprocessing_function = augs.augment_image)\n\nval_datagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = train_datagen.flow_from_dataframe(dataframe=train_set,\n directory=TRAIN_DIR,\n x_col='id',\n y_col='breed',\n class_mode='categorical',\n classes=breed_labels,\n has_ext=False,\n batch_size=BATCH_SIZE,\n shuffle=True,\n seed=SEED,\n target_size=(INPUT_SIZE, INPUT_SIZE)\n )\n\nvalid_generator = val_datagen.flow_from_dataframe(dataframe=val_set,\n directory=TRAIN_DIR,\n x_col='id',\n y_col='breed',\n class_mode='categorical',\n classes=breed_labels,\n has_ext=False,\n batch_size=BATCH_SIZE,\n shuffle=False,\n seed=SEED,target_size=(INPUT_SIZE, INPUT_SIZE)\n )\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9dc64a7ac5e5b625a8cf20ebf6dc2b69eec51952"},"cell_type":"markdown","source":"

    Search for good min and max learning rates

    \n\nNow, I'll follow the guidelines to set good min and max learning rates for the cyclical learning rates approach."},{"metadata":{"trusted":true,"_uuid":"8c85c275abfe5da42ad88037636221b2b445d893","scrolled":false},"cell_type":"code","source":"gc.collect()\n\nmodel = create_model()\n\ntrain_generator.reset()\nvalid_generator.reset()\n\nSTEP_SIZE_TRAIN = train_set.shape[0] // BATCH_SIZE\nSTEP_SIZE_VALID = val_set.shape[0] // BATCH_SIZE\n\nEPOCHS = 1\nbase_lr=0.0001\nmax_lr=1\nstep_size = EPOCHS * STEP_SIZE_TRAIN \nclr = CyclicLR(base_lr=base_lr, max_lr=max_lr, step_size=step_size)\nmodel.fit_generator(train_generator, \n epochs=EPOCHS, \n steps_per_epoch=STEP_SIZE_TRAIN,\n validation_data=valid_generator,\n validation_steps=STEP_SIZE_VALID,\n callbacks=[clr]\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4df81ec0fa709bcbfc2b5330abdc90ab32832eb9","scrolled":false},"cell_type":"code","source":"# Let's plot accuracies against learning rates to select the base_lr and max_lr following the article\n\nresults = pd.DataFrame(clr.history)\nlr = results['lr']\nloss = results['loss']\nwindow=1\nrolling_loss = loss.rolling(window).mean().fillna(0)\n\nfig = plt.figure(figsize=(20, 10))\nticks = np.arange(0, 0.5, 0.005)\nlabels = ticks\nplt.xticks(ticks, ticks, rotation='vertical')\nplt.tick_params(axis='x', which='minor', colors='black')\n#plt.xscale('log')\nevery = 2\n#plt.ylim(4, 5.25)\nplt.xlim(0,0.5)\ntill=500\nplt.plot(lr[::every], loss[::every])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f3daae0231c7897b7684ab08a2fa1d3e361d242"},"cell_type":"markdown","source":"Following the guidelines in the cyclic learning rates article, from the previous plot, we can see that possible good values are base_lr=0.0001, max_lr=0.0005"},{"metadata":{"_uuid":"2a83741e3a5df46640f5c698bd0b76817baed54b"},"cell_type":"markdown","source":"

    Training

    "},{"metadata":{"trusted":true,"_uuid":"9d9e1bf6e5b7fda8c2e706736c03258f0a4719b5"},"cell_type":"code","source":"# Create the model in a different cell, just in case we want to train it several times\nmodel = create_model()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f86433ac4f050416c343d632e65ba7665d9956c7","scrolled":false},"cell_type":"code","source":"# Training\n\ngc.collect()\n\nEPOCHS=40\nSTEP_SIZE_TRAIN = train_set.shape[0] // BATCH_SIZE\nSTEP_SIZE_VALID = val_set.shape[0] // BATCH_SIZE\n\ntrain_generator.reset()\nvalid_generator.reset()\n\nclr = CyclicLR(base_lr=0.04, max_lr=0.11, step_size=2*STEP_SIZE_TRAIN, mode='triangular')\ncheckpoint = ModelCheckpoint('dog_breed_inceptionv3.hf5', monitor='val_acc', verbose=0, save_best_only=True, mode='max', save_weights_only=False)\nreduce = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5, verbose=1, min_lr = 0.00001)\nhistory = model.fit_generator(train_generator, \n epochs=EPOCHS, \n steps_per_epoch=STEP_SIZE_TRAIN,\n validation_data=valid_generator,\n validation_steps=STEP_SIZE_VALID,\n callbacks=[checkpoint, clr]\n \n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae7c36f8bbf06d84a35c7bfde5ac6471273a0117"},"cell_type":"code","source":"# Create a link to download the model weights\nFileLink('dog_breed_inceptionv3.hf5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"384483cee2a9110118933bf84b88b9bfd0028aa8"},"cell_type":"code","source":"# Plot loss and accuracy both for train and validateion sets\ndef plt_history(history, metric, title, ax, val=True):\n ax.plot(history[metric])\n if val:\n ax.plot(history['val_' + metric])\n ax.grid(True)\n ax.set_title(title)\n ax.xaxis.set_ticks(range(0,EPOCHS))\n ax.xaxis.set_ticklabels([str(i) for i in range(1,EPOCHS+1)])\n \n\n ax.set_xlabel('epoch')\n ax.set_ylabel(metric)\n \n \nhist = history.history\nfig, ax = plt.subplots(1,2, figsize=(18,6))\nplt_history(hist, 'loss', 'LOSS', ax[0])\nplt_history(hist, 'acc', 'ACCURACY', ax[1])\nplt.savefig('history.png')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec4e78c74662241b52879dff951a9af1c687b079"},"cell_type":"markdown","source":"

    Test prediction and submission

    "},{"metadata":{"trusted":true,"_uuid":"13e85051e60d4b295c6a80f58e2cc77fb893c1d0"},"cell_type":"code","source":"# Load the best model\n\nmodel.load_weights('dog_breed_inceptionv3.hf5')\n\n# Create a generator from the submission dataframe to leverage model.predict_generator to\n# make the predictions\n\ntest_datagen = ImageDataGenerator(rescale=1./255)\n\ntest_generator = test_datagen.flow_from_dataframe(dataframe=submission,\n directory=TEST_DIR,\n x_col='id',\n class_mode=None,\n has_ext=False,\n batch_size=BATCH_SIZE,\n shuffle=False,\n seed=SEED,\n target_size=(INPUT_SIZE, INPUT_SIZE)\n )\n\npredictions = model.predict_generator(test_generator, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16b5f69b85f5dae2394666b99b1d40e1e20a3683"},"cell_type":"code","source":"# Substitute the dummy predictions in submmission by the model predictions\nsubmission.loc[:,1:] = predictions\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b25c8ceaa75b627da74c31841940748a76b03a3c"},"cell_type":"code","source":"# Save the submission to a file and create a link to download it (without the need of commiting)\nsubmission.to_csv('submission.csv', index=False)\nFileLink('submission.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}