{"cells": [{"metadata": {"_cell_guid": "7d5d8620-4728-4285-95e7-00dc8ef8eb00", "_uuid": "574728cda10a78a8a35840f02df2c811228e37c1"}, "source": ["# Fine-tuning VGG16, loss \u00b1 0.8\n", "Hi all, this is my first public notebook. If there are any issues, let me know!"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "f1e4afbf-9bc3-4483-8dd2-06d7f1113d3f", "_uuid": "b48a79dede79a78989878177c5c4934e4ee9d1cc", "collapsed": true}, "source": ["%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import matplotlib as mpl"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "d436d345-b96c-4981-85d5-a907971ad12e", "_uuid": "ef84eb8fc900e2a68ca6c7ccb0dc9611d3146c1b"}, "source": ["Import all the relevent Keras classes"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "54f71fc6-db27-446f-b241-17454c3c2c93", "_uuid": "b08a8c83ae1d249ec24d5b3534fdcdc053abc68b"}, "source": ["import keras\n", "from keras.layers import Input, Dense, Flatten, Dropout\n", "from keras.layers.normalization import BatchNormalization\n", "from keras.models import Model\n", "from keras.applications.vgg16 import preprocess_input\n", "from keras.preprocessing.image import ImageDataGenerator"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "3de5fb08-cea5-4169-af6f-4a5fac9dadfb", "_uuid": "36649a37da6c9e5fe17ff8a1c2e5540946978397"}, "source": ["Import some utility functions from other libraries"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "b5556a2d-aaf8-4a1a-bf0a-9a76a0965fcb", "_uuid": "8b1f193f2b543e78bed8d3a3f5649ecceb558862", "collapsed": true}, "source": ["import os, fnmatch\n", "from skimage import io, transform\n", "import numpy as np\n", "from tqdm import tqdm\n", "import pandas as pd\n", "import shutil"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "4c9b99ba-f691-42bb-9358-d773694a2ea3", "_uuid": "a81ecad2cb05026414e59adeaab8feef3b6be07f"}, "source": ["# Load VGG16 model"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "d0dfa0ae-e180-4d8e-a648-0cafe0cbdc9b", "_uuid": "a849abdd75953f3f19a6c893773090f7416e0cee"}, "source": ["I will fetch both the vgg16 model without the dense layers and the whole model, so I can play by adding the pre-trained dense layers later."], "cell_type": "markdown"}, {"metadata": {"scrolled": true, "_cell_guid": "448d400a-837b-4abc-981e-82396219740a", "_uuid": "923573ef733e7fa975e6a75c999c9a72a0a4e63d", "collapsed": true}, "source": ["vgg16 = keras.applications.vgg16.VGG16(include_top=False, weights=None,\n", "                                       # use weights='imagenet'\n", "                                       input_tensor=None, input_shape=(224,224,3))\n", "vgg16_full = keras.applications.vgg16.VGG16(include_top=True, weights=None, \n", "                                            # use weights='imagenet'\n", "                                            input_tensor=None, input_shape=(224,224,3))"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "122636c5-cc94-4d32-b63e-2c290080a559", "_uuid": "d5be579987b4a4ced0a41a13afbbfcc28b68efcf"}, "source": ["Extract the last dense layers."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "d3535543-19e7-48a2-ac9e-7e2bde0b3a1d", "_uuid": "cf39e8e6b9aba0b5c8b30f402a2ebe04f1c738e8"}, "source": ["fc1_layer = vgg16_full.get_layer(\"fc1\")\n", "fc1_layer"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "632f39d2-9767-44f6-ac1d-4940c41a6da8", "_uuid": "50aed1ab00cd56e327852463e70e1b292d1533fb"}, "source": ["fc2_layer = vgg16_full.get_layer(\"fc2\")\n", "fc2_layer"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "4fe97a20-11e3-4331-a77c-5194af894465", "_uuid": "7661ccb6b8b2672032cbeb875a0c400fc0f8559d"}, "source": ["# Preprocessing images\n", "We will use ImageDataGenerator so we do not need to keep all the images in memory. Also later this could be used for data augmentation."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "0bbbc7ee-4050-4c1a-8b6b-2fa6fe396471", "_uuid": "1211af6ea28185104d679de0c5a406889dbc1c4a"}, "source": ["!ls ../input"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "3d55bca5-7600-4618-a678-5ddc49d8b7ae", "_uuid": "325a23834aeeb0aea77a84a1b4170a02141b2e71", "scrolled": true}, "source": ["labels_csv = pd.read_csv(\"../input/labels.csv\")"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "1197cae9-4402-4bc8-b27b-027169ff7123", "_uuid": "7a30d559c418661385878411256ef953d0173a68", "collapsed": true}, "source": ["breeds = pd.Series(labels_csv['breed'])\n", "filenames = pd.Series(labels_csv['id'])"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "5f06c8f5-21bb-4a27-82c6-2fc0856e6bb9", "_uuid": "ae70f2529691a73914b87d6183a1dbcba0b30622"}, "source": ["Move the data in subfolders so we can use the Keras ImageDataGenerator. This way we can also later use Keras Data augmentation features."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "13a6f16d-5001-4600-988b-f99e5da19c62", "_uuid": "1d8b394186de8e1ed368278fbfa5cdd6913fbde1", "collapsed": true}, "source": ["unique_breeds = np.unique(breeds)\n", "labels = []\n", "for breed in breeds:\n", "    i = np.where(unique_breeds == breed)[0][0]\n", "    labels.append(i)\n", "\n", "n_breeds = np.max(labels) + 1\n", "labels = np.eye(n_breeds)[labels]"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "36dda0b4-c878-454b-8451-996498244a00", "_uuid": "a862ea7e0dabbbb7be71415eb65584bfae6f76d7"}, "source": ["Separate data in train and validation set"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "e7656770-63de-4a98-bcdb-2c232f152c45", "_uuid": "18c78319d61e15e84d863d65e1d696b73818b83b"}, "source": ["filenames_train = []\n", "filenames_validate = []\n", "\n", "# move to validate folder\n", "for i in tqdm(range(len(filenames))):\n", "    label = unique_breeds[np.where(labels[i]==1.)][0]\n", "    filename = '{}.jpg'.format(filenames[i])\n", "\n", "    if i < 8000:\n", "        new_dir = './sorted/train/{}/'.format(label)\n", "        filenames_train.append(new_dir + filename)\n", "    else:\n", "        new_dir = './sorted/validate/{}/'.format(label)\n", "        filenames_validate.append(new_dir + filename)\n", "        \n", "    if not os.path.exists(new_dir):\n", "        os.makedirs(new_dir)\n", "    \n", "    shutil.copy(\"../input/train/{}.jpg\".format(filenames[i]), new_dir + filename)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "bb877bff-2592-4e29-a5b1-5596f70b8d4f", "_uuid": "f8d48eeb49c0bcee7001f0b5f3a53e46161d6fe7"}, "source": ["We need to sort the filenames and labels array because ImageGenerator fetches the images alphabettic order."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "a0143641-1049-47bd-bdf9-2ab3865c8008", "_uuid": "37cd67ab641482cdae9197b4169173e3fa0718c3"}, "source": ["indices_train = np.argsort(filenames_train)\n", "indices_val = np.argsort(filenames_validate)\n", "\n", "sorted_filenames_train = np.array(filenames_train)[indices_train]\n", "sorted_filenames_validate = np.array(filenames_validate)[indices_val]\n", "sorted_labels_train = np.array(labels)[0:8000][indices_train]\n", "sorted_labels_validate = np.array(labels)[8000:][indices_val]"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "ea0c2db9-cd8d-4380-8353-a24f7182830d", "_uuid": "758a65d1af7313ef1c67302123a84f249df69913"}, "source": ["Check if the sorting is correct."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "8e09f7c5-53bb-49af-85d3-85360cd32786", "_uuid": "f501b5282984b3fe9ed21eeb015c30a470ab3cec"}, "source": ["print(unique_breeds[np.where(sorted_labels_train[50] == 1.)])\n", "# should be equal to:\n", "print(sorted_filenames_train[50])"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "6bdf54e3-9d92-4c6c-95c4-974716ff9680", "_uuid": "825c1c2b164037bf82fc15c4af55eaffc7cd5314", "collapsed": true}, "source": ["def preprocess(img):\n", "    input_img = preprocess_input(np.expand_dims(img, axis=0))\n", "    return input_img[0]\n", "\n", "train_datagen = ImageDataGenerator(preprocessing_function=preprocess)\n", "val_datagen = ImageDataGenerator(preprocessing_function=preprocess)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "8a3287ad-19d7-4f03-bcf0-1cf9cc9e43de", "_uuid": "ddf5294e3dae60de09b462fc6793dc34e6935d3d"}, "source": ["batch_size = 64\n", "\n", "train_gen = train_datagen.flow_from_directory(\"./sorted/train\", \n", "                                              batch_size=batch_size, \n", "                                              target_size=(224, 224), \n", "                                              shuffle=False)\n", "\n", "val_gen = val_datagen.flow_from_directory(\"./sorted/validate\", \n", "                                          batch_size=batch_size, \n", "                                          target_size=(224, 224), \n", "                                          shuffle=False)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "ba80bc7a-a290-4a26-bd61-cc0fa66d52b8", "_uuid": "92325b239046f1857325dc942ea8eaa67c254bcd"}, "source": ["# Generate Bottleneck features\n", "I only execute one step here because of the limited running time in Kaggle."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "c7d05b94-ac46-4121-818a-29e30d305cb9", "_uuid": "3003289f218fdd2937a8ce2ffc9273d637b6b2cf"}, "source": ["x_train = vgg16.predict_generator(train_gen, \n", "                                  # steps=8000 // batch_size, \n", "                                  steps=1, \n", "                                  verbose=1)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "b039342e-5357-48d2-b342-7bd4ad822930", "_uuid": "e1b926ee1ece6c832ce9c0183a3bf560beb1aee0"}, "source": ["x_val = vgg16.predict_generator(val_gen, \n", "                                # steps=2222 // batch_size, \n", "                                steps=1,\n", "                                verbose=1)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "d1f2b49a-56dc-42d9-996d-e1ade191e4ff", "_uuid": "996b5e1657070d329e9f32e3b3d9465c39395132", "collapsed": true}, "source": ["y_train = sorted_labels_train[0:len(x_train)]\n", "y_val = sorted_labels_validate[0:len(x_val)]"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "12eda2db-cbbe-4a69-84df-bfe5b78a30d5", "_uuid": "5f962de6133f5cbbc8d019d4be155c0a8410a576"}, "source": ["I found we need quite high dropout to make the model overfit less."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "99a75cfb-e807-4f63-9592-b6ace1bff05a", "_uuid": "2b02488b73e8b3be96e0e9557b7a701481ce0d6f"}, "source": ["inputs = Input(shape=(7,7,512))\n", "\n", "# Turn off training vgg16\n", "for layer in vgg16.layers:\n", "    layer.trainable = False\n", "fc1_layer.trainable = False\n", "\n", "x = Flatten()(inputs)\n", "x = fc1_layer(x)\n", "x = BatchNormalization()(x)\n", "x = Dropout(0.8)(x)\n", "x = Dense(512, activation='relu')(x)\n", "x = BatchNormalization()(x)\n", "x = Dropout(0.8)(x)\n", "x = Dense(120, activation='softmax')(x)\n", "\n", "model = Model(inputs=inputs, outputs=x)\n", "model.summary()"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"hidden": true, "_cell_guid": "753611a3-e84b-4c51-8825-ecf65d454018", "_uuid": "ab1d25d6b96accccfde1a5f3af364174a008f3aa", "collapsed": true}, "source": ["model.compile(optimizer=keras.optimizers.Adam(), \n", "              loss=keras.losses.categorical_crossentropy, \n", "              metrics=['accuracy'])"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "6443b7e7-a2a5-4d2f-9c2c-a8f9111b395c", "_kg_hide-output": false, "_uuid": "9f00f2e8657cc9c6840c2c748570e9b2ceb141b6"}, "source": ["history = model.fit(x_train, y_train, batch_size=128, epochs=30, verbose=1, \n", "                    validation_data=(x_val, y_val))"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "c080e125-f8d6-4d03-aa53-d635364b6ab3", "_uuid": "6cb8a0f0952120b8aeb60afa5633b69ae37ee5b7"}, "source": ["With all the examples I get:\n", "\n", "    Train on 8000 samples, validate on 2176 samples\n", "    Epoch 29/30\n", "    8000/8000 [==============================] - 2s 200us/step - loss: 0.8130 - acc: 0.7450 - val_loss: 0.8068 - val_acc: 0.7551\n", "    Epoch 30/30\n", "    8000/8000 [==============================] - 2s 201us/step - loss: 0.8329 - acc: 0.7354 - val_loss: 0.8173 - val_acc: 0.7541"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "740dfad9-a33c-4a20-a789-dbf038c5cdc9", "_uuid": "54f5210e3c372fa0f3956ebf4e74685da36eb4b6"}, "source": ["# summarize history for accuracy\n", "plt.plot(history.history['acc'])\n", "plt.plot(history.history['val_acc'])\n", "plt.title('model accuracy')\n", "plt.ylabel('accuracy')\n", "plt.xlabel('epoch')\n", "plt.legend(['train', 'test'], loc='upper left')\n", "plt.show()\n", "# summarize history for loss\n", "plt.plot(history.history['loss'])\n", "plt.plot(history.history['val_loss'])\n", "plt.title('model loss')\n", "plt.ylabel('loss')\n", "plt.xlabel('epoch')\n", "plt.legend(['train', 'test'], loc='upper left')\n", "plt.show()"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "a609c6df-f8aa-4f6f-8ad9-f74c809918c0", "_uuid": "6d99d74b16245680d13ea580677d62eea3e3d6d4", "collapsed": true}, "source": [], "execution_count": null, "cell_type": "code", "outputs": []}], "nbformat": 4, "nbformat_minor": 1, "metadata": {"language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "pygments_lexer": "ipython3", "name": "python", "version": "3.6.3"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}}