{"cells":[{"metadata":{"_uuid":"2319ba962cda9b4a707ccb94e917e01304676c04"},"cell_type":"markdown","source":"

Transfer learning for dog breed classification - Part I

\n\nIn every single computer vision competition, top rank competitors almost never train their networks from scratch. They transfer knowledge from pretrained models to theirs. This is something I painfully learned in the first official competition I took place in (TGS Salt Identification), and I wish someone would have told me then. For that reason, I'm planning to create a series of kernels for beginners in which I'll cover the basic techniques in transfer learning. I will only touch the problem of image classification, but these techniques could be applied to any other computer vision problem (image segmentation, object detection, ...). I will be using a pretrained model for the dog breed classification problem. At the same time, I will be covering some useful techniques which I also painfully learned about such as using ImageDataGenerator for reading images into memory from disk and data augmentation.\n\nThe series will have three parts, in each of which I will cover three basic techniques and some of its variants:\n\n\n
  • Part I - Extract features from the last convolutional block of VGG16 and use them to train a NN classifier, without data augmentation
    • \n
  • Part II - Freeze the VGG16 base model, put a classifier on top of it and train the model with data augmentation.
    • \n
  • Part III - Starting from the model trained in Part 2, unfreeze the last convolutional block and finetune the network with a small learning rate.
    • \n    "},{"metadata":{"trusted":true,"_uuid":"586b37413ed5d15a8f75a15e565c69e5123e5718"},"cell_type":"code","source":"! cp ../input/my-python/* ../working/\n! ls ../working ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"38f1edf12d03cb2b3523789b7761c175cc5f8924"},"cell_type":"code","source":"# Generic imports\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nimport cv2\nfrom tqdm import tqdm_notebook\n%matplotlib inline\nsns.set_style('whitegrid')\n\n# Sklearn imports\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Keras imports\nfrom keras.applications import VGG16, InceptionV3\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Input\nfrom keras.optimizers import RMSprop, Adam, SGD\nfrom keras.preprocessing.image import load_img,img_to_array\n\n# cyclical learning rates\nfrom clr import LRFinder\nfrom clr_callback import CyclicLR","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"267b3bd0bcb014ced495f0e5a66c58575ec59eeb"},"cell_type":"code","source":"# Constants\nDATA_DIR = '../input/dog-breed-identification/'\nTRAIN_DIR = DATA_DIR + 'train/'\nTEST_DIR = DATA_DIR + 'test/'\nBATCH_SIZE = 32\nINPUT_SIZE = 224\nNUM_CLASSES = 120\nSEED = 42","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d2e754a75c561d10af4baa14541dc838975d627"},"cell_type":"code","source":"# Let's check what's in the data directory\n! ls $DATA_DIR","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e82fcdad16d631412ba18fae9e867abe80f2791"},"cell_type":"code","source":"# Read the train data set, which has the ids of the images and their labels (breeds)\n# (adding the extension .jpg to the id becomes the file name of the image) \ntrain = pd.read_csv(DATA_DIR + 'labels.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9fb80951bf00b9b71800e41953b144104dff05d"},"cell_type":"code","source":"# The submission file contains one column for the image id, and then one column \n# each breed in alphabetical order, with the probability of the dog in the image beeing of that breed\nsubmission = pd.read_csv(DATA_DIR + 'sample_submission.csv')\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"daef164d8fc2cce606137648a8cfe963f8257834"},"cell_type":"code","source":"# Create a map of breeds to labels in the same order as the columns of the submission file\n# and create a new column 'label' in the train data frame with the breeds mapped to this labels.\n# This will make easier build the submission file from the predicted probabilities of the trained model\nbreed_labels = {breed:label for label,breed in enumerate(submission.columns[1:].values)}\ntrain['label'] = train['breed'].map(breed_labels)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fdaf52648c62df3777d91abd7ae2a0455441457"},"cell_type":"markdown","source":"

    A bit of EDA

    "},{"metadata":{"trusted":true,"_uuid":"68ff10690a062739cfee24381a9fa7e3fbc4b4c0","scrolled":false},"cell_type":"code","source":"# Frequency of each breed in the train set. We can see that the most frequent breed\n# has just above 120 images and the less frequent just above 60 images.\ncounts = train.breed.value_counts()\nplt.figure(figsize=(10,40))\nplt.xticks(np.arange(0, 130, 5))\nsns.barplot(x=counts.values, y=counts.index);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5273773781833976465f4b7abb3d1314c9617dc2"},"cell_type":"code","source":"# Let's plot some random images\nfig, axs = plt.subplots(5,5, figsize=(20,20), squeeze=True)\nplt.subplots_adjust(wspace=0.1, hspace=0.1)\naxs = axs.reshape(-1)\nindices = np.random.choice(train.shape[0], 25, replace=False)\nfor ax, i in zip(axs, indices):\n img = cv2.imread(os.path.join(DATA_DIR, 'train', train.iloc[i].id + '.jpg'))\n h, w, c = img.shape\n ax.set_xticks([])\n ax.set_yticks([])\n ax.set_xlabel(train.iloc[i].breed, fontsize=12)\n ax.imshow(img)\n ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c51c5018554f37cd2f58f10765d5bf63386f64ae"},"cell_type":"markdown","source":"*

    Variant 1 - Feature extraction from the last convolutional layer without pooling

    \n\nWe'll extract features directly from the last convolutional layer in VGG16, wich has shape (None, 7, 7, 512). This means that for every image\nwe'll extract 7*7*512 = 25088 features, and connect them to our classifier."},{"metadata":{"trusted":true,"_uuid":"c36694614c19f6c28e0ad87e73645ddbc75f8dea","scrolled":true},"cell_type":"code","source":"base = VGG16(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3))\nbase.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"722ed51c26c3f9a75441fe8aab6cec7f8e36556d"},"cell_type":"markdown","source":"

    flow_from_dataframe

    \nWe don't have train and validation folders, so we cannot use directly the method frow_from_directory from ImageDataGenerator. Instead we'll be using the method flow_from_dataframe. Here's an article on how to use it by the guy who wrote it, Vijayabhaskar J."},{"metadata":{"trusted":true,"_uuid":"868d7a7ec6053f3feae10a0851502e0b0c34bee2"},"cell_type":"code","source":"# Make the batchsize of the data generator a divisor of the number of images, as we have\n# to make just one pass for feature extraction\nbatch_size = 269 # 10222 = 2 * 19 * 269\n\n# No data augmentation, just rescaling the image\ndatagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = datagen.flow_from_dataframe(dataframe=train, \n directory= TRAIN_DIR,\n x_col='id',\n y_col='label',\n class_mode='categorical',\n has_ext=False,\n batch_size=batch_size, \n shuffle=False,\n seed=42,\n target_size=(224,224) \n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7eb2b48f80a134659239cf75ad903dccee4713ea"},"cell_type":"code","source":"%%time\n\n# Let's read all the images and labels into arrays in memory\n\ntrain_generator.reset()\ntrain_size = train.shape[0]\nfeatures = np.zeros(shape=(train_size, 7,7,512))\nlabels = np.zeros(shape=(train_size, NUM_CLASSES))\ni = 0\nfor inputs_batch, labels_batch in tqdm_notebook(train_generator):\n features_batch = base.predict(inputs_batch)\n features[i * batch_size:(i+1) * batch_size] = features_batch\n labels[i * batch_size:(i+1) * batch_size] = labels_batch\n i += 1\n if i * batch_size >= train_size:\n break;\n \n# Flatten the output of the VGG16 base model\nfeatures = features.reshape(train_size, -1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"584575bade84f0886ac400b26ae3d985f320ceef"},"cell_type":"markdown","source":"

    Fitting a CNN with the generated features

    "},{"metadata":{"trusted":true,"_uuid":"0488df6c976fa393e6cef78c4f9ccbe6c47af80b"},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(features, labels, test_size=0.20, random_state=SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d1e095698bb8a97634bb6480df65f21a856c8c1a"},"cell_type":"code","source":"# Build the classifier\n\ndef create_model(dropout=None):\n model = Sequential()\n model.add(Dense(1024, activation='relu'))\n if dropout:\n model.add(Dropout(dropout))\n model.add(Dense(512, activation='relu'))\n if dropout:\n model.add(Dropout(dropout))\n model.add(Dense(NUM_CLASSES, activation='softmax'))\n\n adam = Adam(lr=0.001)\n sgd = SGD(lr=0.01, momentum=0.9)\n model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['acc'], )\n return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70a95d66f0cbe60d02cdcacf4632c1103a34d838"},"cell_type":"code","source":"# Run a lr range test to find good learning rate margins\n\nmodel = create_model()\n\nBATCH_SIZE = 32\nSTEP_SIZE_TRAIN = X_train.shape[0] // BATCH_SIZE\nSTEP_SIZE_VALID = X_val.shape[0] // BATCH_SIZE\n\nEPOCHS = 1\nbase_lr=0.0001\nmax_lr=100\nstep_size = EPOCHS * STEP_SIZE_TRAIN \nlrf = LRFinder(X_train.shape[0], BATCH_SIZE,\n base_lr, max_lr,\n # validation_data=(X_val, Yb_val),\n lr_scale='exp', save_dir='./lr_find/', verbose=False)\n\nhistory = model.fit(X_train, y_train, epochs=EPOCHS, steps_per_epoch = STEP_SIZE_TRAIN,\n validation_data=[X_val, y_val], validation_steps = STEP_SIZE_VALID,\n callbacks=[lrf])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f86433ac4f050416c343d632e65ba7665d9956c7","collapsed":true},"cell_type":"code","source":"# Training\nhistory = model.fit(X_train, y_train, epochs=EPOCHS, batch_size=BATCH_SIZE, validation_data=[X_val, y_val])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3ca5f659c0c8ded70e8e42537f6e9adcf0f66ee"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,7))\nlrf.plot_schedule(clip_beginning=95, clip_endding=60)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c34f1bb6d6a2a54950e7d048fa5f217cc8b3d1d7"},"cell_type":"code","source":"10**(-1.7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"c31c3ad058e851e0383d303d14cd78122ac844c0"},"cell_type":"code","source":"model = create_model()\nEPOCHS=10\nBATCH_SIZE=32\nclr = CyclicLR(base_lr=0.005, max_lr=0.02, step_size=2*STEP_SIZE_TRAIN)\n\nhistory = model.fit(X_train, y_train, epochs=EPOCHS, steps_per_epoch=STEP_SIZE_TRAIN,\n validation_data=[X_val, y_val], validation_steps=STEP_SIZE_VALID,\n callbacks=[clr])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6dbd07b1e09486d85dce32df0b454ea8d24d04b"},"cell_type":"markdown","source":"As we can see in the history plot, there's a lot of overfitting. Varying the size of the FC layer in the network between 256 and 1024, and dropout between 0.1 and 0.5 doesn't change anything: train accuracy reaches around 0.99 and validation accuracy just below 0.3."},{"metadata":{"trusted":true,"_uuid":"384483cee2a9110118933bf84b88b9bfd0028aa8"},"cell_type":"code","source":"def plt_history(history, metric, title, ax, val=True):\n ax.plot(history[metric])\n if val:\n ax.plot(history['val_' + metric])\n ax.grid(True)\n ax.set_title(title)\n ax.set_xlabel('epoch')\n ax.set_ylabel(metric)\n \nhist = history.history\nfig, ax = plt.subplots(1,2, figsize=(15,6))\nplt_history(hist, 'loss', 'LOSS', ax[0])\nplt_history(hist, 'acc', 'ACCURACY', ax[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9ba26202b578a9813dd56b82d2a5a5406d9421d"},"cell_type":"markdown","source":"

    Create custom estimator

    \nBased on this excellent article by Daniel Hnyk, and the Scikit-learn documentation."},{"metadata":{"trusted":true,"_uuid":"832df67d2a09948757cfa895f1f10de834a125b0"},"cell_type":"markdown","source":"

    Variant 2 - Feature extraction from the last convolutional layer with average pooling

    \n\nWe'll extract features from the last convolutional layer in VGG16, wich has shape (None, 7, 7, 512), but average pooling this layer, so the output of the VGG16 model will be (None, 512)."},{"metadata":{"trusted":true,"_uuid":"4c3315960470d68640feadb69395aa0d6b64765f"},"cell_type":"code","source":"base = InceptionV3(weights='imagenet', include_top=False, input_shape=(INPUT_SIZE, INPUT_SIZE, 3), pooling='avg')\nbase.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5483bd54a881172c0d715240218d955f1328eb5"},"cell_type":"code","source":"# Make the batchsize of the data generator a divisor of the number of images, as we have\n# to make just one pass for feature extraction\nbatch_size = 269 # 10222 = 2 * 19 * 269\n\n# No data augmentation, just rescaling the image\ndatagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = datagen.flow_from_dataframe(dataframe=train, \n directory= TRAIN_DIR,\n x_col='id',\n y_col='label',\n class_mode='categorical',\n has_ext=False,\n batch_size=batch_size, \n shuffle=False,\n seed=42,\n target_size=(224,224) \n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0d46c1d86a7a1bdbe1a69b2ee9129d74714f9a5b"},"cell_type":"code","source":"%%time\n\n# Let's read all the images and labels into arrays in memory. We can use the same generator,\n# but this time the features array will have shape (train_size, 512) instead of (train_size, 7*7*512)\n\n\ntrain_generator.reset()\ntrain_size = train.shape[0]\nfeatures = np.zeros(shape=(train_size, 2048))\nlabels = np.zeros(shape=(train_size, NUM_CLASSES))\ni = 0\nfor inputs_batch, labels_batch in tqdm_notebook(train_generator):\n features_batch = base.predict(inputs_batch)\n features[i * batch_size:(i+1) * batch_size] = features_batch\n labels[i * batch_size:(i+1) * batch_size] = labels_batch\n i += 1\n if i * batch_size >= train_size:\n break;\n \n# This time the features array doesn't need flattening\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd2a344a27900479203f0381cb30dd3c9000d4f8"},"cell_type":"code","source":"# Build the classifier. This time, the classifier has a lot fewer parameters\ndef create_model_pool(lr=0.001):\n model = Sequential()\n model.add(Dense(512, activation='relu'))\n model.add(Dropout(0.7))\n model.add(Dense(NUM_CLASSES, activation='softmax'))\n\n sgd = SGD(lr=lr, momentum=0.9)\n model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['acc'], )\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9491d9d552a9ab65df4d50386511d1d14e086478"},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(features, labels, test_size=0.25, random_state=SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"74fc028cbd805aa829ae79728e47ff8c766faf91"},"cell_type":"code","source":"# Run a lr range test to find good learning rate margins\n\nmodel = create_model_pool()\n\nBATCH_SIZE = 32\nSTEP_SIZE_TRAIN = X_train.shape[0] // BATCH_SIZE\nSTEP_SIZE_VALID = X_val.shape[0] // BATCH_SIZE\n\nEPOCHS = 1\nbase_lr=0.001\nmax_lr=1\nstep_size = EPOCHS * STEP_SIZE_TRAIN \nlrf = LRFinder(X_train.shape[0], BATCH_SIZE,\n base_lr, max_lr,\n validation_data=(X_val, y_val),\n lr_scale='exp', save_dir='./lr_find/', verbose=False)\n\nhistory = model.fit(X_train, y_train, epochs=EPOCHS, steps_per_epoch = STEP_SIZE_TRAIN,\n validation_data=[X_val, y_val], validation_steps = STEP_SIZE_VALID,\n callbacks=[lrf])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"256ddb73c4d4266942060337b184eaa86208e881"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,7))\nlrf.plot_schedule(clip_beginning=50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ae15ad7cd45e91d8c3f500e511911f4949062f0"},"cell_type":"code","source":"10**(-1.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"09faf47d00afe3cb893fba237180af8aa9449eb8"},"cell_type":"code","source":"# Training\nmodel = create_model_pool()\nEPOCHS=20\nclr = CyclicLR(base_lr=0.01, max_lr=0.03, step_size=2*STEP_SIZE_TRAIN)\n\nhistory = model.fit(X_train, y_train, epochs=EPOCHS, steps_per_epoch=STEP_SIZE_TRAIN,\n validation_data=[X_val, y_val], validation_steps=STEP_SIZE_VALID,\n callbacks=[clr])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f70ec40341e7ca8848b1876ae54d53e7824a8991"},"cell_type":"code","source":"hist = history.history\nfig, ax = plt.subplots(1,2, figsize=(15,6))\nplt_history(hist, 'loss', 'LOSS', ax[0])\nplt_history(hist, 'acc', 'ACCURACY', ax[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90977299f90813b8b86e98db1de34f588d1e418a"},"cell_type":"markdown","source":"As we can see, with this approach we still have a lot of overfitting and the results are even worse for the train set. It seems that we will need a wiser strategy (data augmentation) for this problem in part II of this series."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}