{"nbformat_minor": 1, "cells": [{"cell_type": "markdown", "metadata": {"_uuid": "cfa9823c603379256ab061977184c91bfad9fe3c", "_cell_guid": "7a5c03e4-8a34-42be-9b4d-1ab0d17db7bf", "collapsed": true}, "source": ["# Deep Ensemble learning: Stacking results from deep learning models\n", "### **Hyungsuk Kang, Sungkyunkwan University**\n", "#### 2017/10/21\n", "# Outline\n", "* **1. Introduction**\n", "\n", "* **2. Data preparation**\n", "\n", "    * 2.1 Load Data for Each Model\n", "    \n", "    * 2.2 Exploratory Data Analysis\n", "    \n", "    * 2.3 Feature Extraction\n", "\n", "* **3. Training**\n", "\n", "    * 3.1 Classifiers\n", "    \n", "    * 3.2 Evaluation\n", "    \n", "    * 3.3 Test predictions\n", "\n", "* **4. Prediction and submission**\n", "\n", "    * 4.1 Prediction\n", "    \n", "    * 4.2 Ensembling\n", "    \n", "    * 4.3 Results"]}, {"cell_type": "markdown", "metadata": {"_uuid": "845716fb9a1eda07027a6d49d050f57118bd4bcb", "_cell_guid": "68cfa51c-d7c2-4d1d-8af8-49ad982f848f"}, "source": ["# **1. Introduction**\n", "\n", "This is a full walkthrough for building the ensemble learning model for dog image dataset provided by Kaggle. Ensemble learning is a machine learning method which combines results from several model. It usually shows better results due to several reasons(reducing overfitting, etc) First, I will prepare the data (Dog images), get prediction for each model(InceptionV3, VGG, Xception), and combine each results from the model.\n", "\n", "For more information on Keras, click this link.\n", "\n", "# [Keras](https://keras.io/)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "98962defdf9bed2f52be321c293201ef7908eed8", "_cell_guid": "0bae0dff-4be2-4f44-b273-a07bfd551142", "collapsed": true}, "outputs": [], "source": ["%matplotlib inline\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from os import listdir, makedirs, getcwd, remove\n", "from os.path import isfile, join, abspath, exists, isdir, expanduser\n", "from tqdm import tqdm\n", "from keras.models import Model, Sequential\n", "from keras.layers import Input, GlobalAveragePooling2D, Dense, Conv2D, MaxPooling2D, Dropout, Lambda, Reshape, Flatten\n", "from keras import backend as K\n", "from keras.optimizers import Adam\n", "from keras.callbacks import ReduceLROnPlateau, ModelCheckpoint\n", "from sklearn.model_selection import train_test_split\n", "from keras.preprocessing import image\n", "from keras.applications.vgg19 import VGG19\n", "from keras.applications.resnet50 import ResNet50\n", "from keras.applications.inception_v3 import InceptionV3\n", "from keras.applications.xception import Xception\n", "import cv2\n", "from keras.applications.inception_v3 import preprocess_input\n", "import matplotlib.image as mpimg\n", "import seaborn as sns\n", "\n", "np.random.seed(2)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "b896a5d287e0387cc4a0574f275b0d1b07900da8", "_cell_guid": "bb8feb4c-a23b-4410-81b6-a772b8393629"}, "source": ["# 2. Data Preparation"]}, {"cell_type": "markdown", "metadata": {"_uuid": "1a72021f32396d45913386ef3932252eddff754a", "_cell_guid": "6cad74eb-a3a9-441b-92d9-f04cff0b4d1c"}, "source": ["## 2.1. Load Data for Each Model\n", "\n", "Each model requires certain image pixel size(299x299 for inceptionV3,  Xception, and VGG)\n", "\n", "### Set up metadata"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "a171db65ed6e662a0a496355cbab8abb306b34d0", "_cell_guid": "a1590450-01dd-451a-b9bc-f8658bc005ff", "collapsed": true}, "outputs": [], "source": ["df = pd.read_csv('../input/dog-breed-identification/labels.csv')\n", "df_test = pd.read_csv('../input/dog-breed-identification/sample_submission.csv')\n", "df.head()\n"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "744902374ceefff741f88ede0156e123f8f328d3", "_cell_guid": "7296f2ed-3abf-4a21-aa2f-a4ba3f999cca", "collapsed": true}, "outputs": [], "source": ["n = len(df)\n", "breed = set(df['breed'])\n", "n_class = len(breed)\n", "class_to_num = dict(zip(breed, range(n_class)))\n", "num_to_class = dict(zip(range(n_class), breed))"]}, {"cell_type": "markdown", "metadata": {"_uuid": "b641f256a0d90790ad22ef4b25a9df5109be041a", "_cell_guid": "0183677c-b867-4445-9626-3d8e28196f60"}, "source": ["### Load Images"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "ee981a7142ea3935774ad7b9f3499a0cf987f40e", "_cell_guid": "54fde1b8-aa57-4e53-ba08-2e20ad282211", "collapsed": true}, "outputs": [], "source": ["width = 299\n", "X = np.zeros((n, width, width, 3), dtype=np.uint8) # buffer for images\n", "y = np.zeros((n, n_class), dtype=np.uint8) # buffer for label\n", "for i in tqdm(range(n)): # fill them up; tqdm for progress bar in loop\n", "    X[i] = cv2.resize(cv2.imread('../input/dog-breed-identification/train/%s.jpg' % df['id'][i]), (width, width))\n", "    y[i][class_to_num[df['breed'][i]]] = 1"]}, {"cell_type": "markdown", "metadata": {"_uuid": "0a501135e5bab51b7f05f608d8be315be97cbdcb", "_cell_guid": "27a97350-862f-4de7-aaa4-72b995f89344"}, "source": ["### Load Images(test)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "aeaf5f034f0dca4fb953346791367056858ba007", "_cell_guid": "0617af63-d5e9-4f5c-9665-2ddee68b8f01", "collapsed": true}, "outputs": [], "source": ["width = 299\n", "n_test = len(df_test)\n", "X_test = np.zeros((n_test, width, width, 3), dtype=np.uint8)\n", "for i in tqdm(range(n_test)):\n", "    X_test[i] = cv2.resize(cv2.imread('../input/dog-breed-identification/test/%s.jpg' % df_test['id'][i]), (width, width))\n", "    \n"]}, {"cell_type": "markdown", "metadata": {"_uuid": "dfb69f4cfd680994ce19b771a9fa1d56621dbed7", "_cell_guid": "4a64a438-0439-438c-acf0-6dc5da441a90"}, "source": ["## 2.2 Exploratory Data Analysis\n", "\n", "### Distribution of Output Variable"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "05d0030ab0c05a0e08e640a120b3db53d390e2fd", "_cell_guid": "2ce6fdb7-8e4d-4caf-b3d3-18aa2d56480f", "collapsed": true}, "outputs": [], "source": ["y_eda = [list(i).index(1) for i in tqdm(y, total=n)]\n", "g = sns.countplot(y_eda)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "6b668b31abb591e5425f27f866e9b9b82fd6c819", "_cell_guid": "b2ba0c33-35d7-4d6e-af89-b8b92d825fc1", "collapsed": true}, "source": ["It can be seen that:\n", "    \n", "    - Labels are biased to some labels(15, 68, etc).\n", "    - Balancing weight for each class may improve LB score."]}, {"cell_type": "markdown", "metadata": {"_uuid": "74ff96c8c9ec925d2a5fe67b1547f934525b1553", "_cell_guid": "3b37cac5-0b2a-400d-837b-a61183f25e25", "collapsed": true}, "source": ["## 2.3 Feature Extraction\n", "\n", "### Extract Bottleneck Features\n", "\n", "Thanks to [Beluga](https://www.kaggle.com/gaborfodor), training with pretrained CNN models in Keras became possible in a Kaggle kernel.\n", "\n", "Add the [dataset](https://www.kaggle.com/gaborfodor/keras-pretrained-models) to it and execute shell code below\n", "\n", "Bottleneck features are saved just in case the computer is out of resources"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "b8daf0db0c983d3f2d1ee6661c78c89d07bf5556", "_cell_guid": "f6fa9bc2-3c29-4433-8913-b142b820b0a5", "collapsed": true}, "outputs": [], "source": ["cache_dir = expanduser(join('~', '.keras'))\n", "if not exists(cache_dir):\n", "    makedirs(cache_dir)\n", "models_dir = join(cache_dir, 'models')\n", "if not exists(models_dir):\n", "    makedirs(models_dir)\n", "\n", "!ls ../input/keras-pretrained-models/\n", "!cp ../input/keras-pretrained-models/* ~/.keras/models/\n"]}, {"cell_type": "markdown", "metadata": {"_uuid": "5e49b816edbf5472170d02faf8ce78e842a81cb7", "_cell_guid": "91712c5f-18d4-46cb-b8a4-90cb88d6da96", "collapsed": true}, "source": ["\n", "Function from [Yang Peiwan's kernel](https://www.kaggle.com/yangpeiwen/keras-inception-xception-0-47):"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "cc17fbcefe7452610421c0058eeddb4ff6ab4324", "_cell_guid": "7a1562a8-3481-46cd-a18b-471fbc2aba63", "collapsed": true}, "outputs": [], "source": ["\n", "def get_features(MODEL, data=X, batch_size=4):\n", "    cnn_model = MODEL(include_top=False, input_shape=(width, width, 3), weights='imagenet')\n", "    \n", "    inputs = Input((width, width, 3))\n", "    x = inputs\n", "    x = Lambda(preprocess_input, name='preprocessing')(x)\n", "    x = cnn_model(x)\n", "    x = GlobalAveragePooling2D()(x)\n", "    cnn_model = Model(inputs, x)\n", "\n", "    features = cnn_model.predict(data, batch_size=batch_size, verbose=1)\n", "    return features### InceptionV3"]}, {"cell_type": "markdown", "metadata": {"_uuid": "aa55a3b7a5cd64113649a17a63252fae72ba94cd", "_cell_guid": "df8e44e6-f250-47ad-8670-271786cde541"}, "source": ["### InceptionV3"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "a57043143e677bc6ff0adab572076cae14d17df0", "_cell_guid": "85d99c33-ce22-4077-97cd-ff83ac971d0a", "collapsed": true}, "outputs": [], "source": ["inception_features = get_features(InceptionV3, X)\n", "np.savez('bottleneck_features/inception_features.npz' , X=inception_features)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "53e4fc37c9fed325c784c93407d4d26441afc173", "_cell_guid": "84bd90d7-346e-4108-a17d-9630e88c8436"}, "source": ["### Xception"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "32880952a019d013e707c5ff5f43dc33454ed1ff", "_cell_guid": "85793da6-3e42-4f3d-8d8e-209999ce96c7", "collapsed": true}, "outputs": [], "source": ["xception_features = get_features(Xception, X)\n", "np.savez('bottleneck_features/xception_features.npz' , X=xception_features)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "dd501bd9927ac3a40d0c4a6cbad57a5959da49b2", "_cell_guid": "03559bc2-c9ec-4d0d-862d-2298eea6a86c"}, "source": ["### Resnet50"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "f89fd539547fc0d845d51814a619a5884ba015f1", "_cell_guid": "155a2932-8d22-4707-87a5-61c8c89929a6", "collapsed": true}, "outputs": [], "source": ["resnet_features = get_features(ResNet50, X)\n", "np.savez('bottleneck_features/resnet_features.npz' , X=resnet_features)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "81b9776c37eb0df121648805a590a82d9e407c8a", "_cell_guid": "51e974f7-7dec-4bf4-b029-02c62200b645"}, "source": ["### VGG19"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "66271b1051603b84482393fc7b0b5c08fbc2c4aa", "_cell_guid": "2e4cca67-76e3-401b-a2ff-efa11549d625", "collapsed": true}, "outputs": [], "source": ["vgg_features = get_features(VGG19, X)\n", "np.savez('bottleneck_features/vgg_features.npz' , X=vgg_features)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "e73ae264a0475f39461f1dd8baa0c3e042f02e97", "_cell_guid": "b89a1400-1b6e-4118-9030-2d95ea996302"}, "source": ["# 3. Training/Predicting Pipeline\n", "\n", "## 3.1. Split Train/Valid dataset"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "f14cea68e4644e0e53498fd5a36cb578fc68656a", "_cell_guid": "c93a3c29-1007-4439-89b3-6e2e0f5aeac1", "collapsed": true}, "outputs": [], "source": ["X_train_xception, X_valid_xception, y_train_xception, y_valid_xception =  train_test_split(xception_features, y, test_size=0.2, random_state=99)\n", "X_train_inception, X_valid_inception, y_train_inception, y_valid_inception = train_test_split(inception_features, y, test_size=0.2, random_state=99)\n", "X_train_vgg, X_valid_vgg, y_train_vgg, y_valid_vgg = train_test_split(vgg_features, y, test_size=0.2, random_state=99)\n", "X_train_resnet, X_valid_resnet, y_train_resnet, y_valid_resnet = train_test_split(resnet_features, y, test_size=0.2, random_state=99)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "2cbb4774364f86f1afb6de149f5a314f63f83c7e", "_cell_guid": "8782a2a2-e93c-4dcb-90ca-80a144b37c4c"}, "source": ["## 3.1. Classifiers\n", "\n", "\n", "### Neural Net"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "b39fa5054163133a78ae5d24e9d519e77b8f4fa4", "_cell_guid": "235d4597-392d-4451-969a-164f5744025e", "collapsed": true}, "outputs": [], "source": ["Inception_model = Sequential()\n", "Inception_model.add(Dropout(0.2, input_shape=inception_features.shape[1:]))\n", "Inception_model.add(Dense(n_class, activation='softmax'))\n", "\n", "Inception_model.compile(optimizer='adam',\n", "            loss='categorical_crossentropy',\n", "            metrics=['accuracy'],\n", "           )\n", "\n", "Inception_model.summary()\n", "\n", "\n", "Xception_model = Sequential()\n", "Xception_model.add(Dropout(0.2, input_shape=xception_features.shape[1:]))\n", "Xception_model.add(Dense(n_class, activation='softmax'))\n", "\n", "Xception_model.compile(optimizer='adam',\n", "            loss='categorical_crossentropy',\n", "            metrics=['accuracy'],\n", "           )\n", "\n", "Xception_model.summary()\n", "\n", "\n", "VGG_model = Sequential()\n", "VGG_model.add(Dropout(0.2, input_shape=vgg_features.shape[1:]))\n", "VGG_model.add(Dense(n_class, activation='softmax'))\n", "\n", "VGG_model.compile(optimizer='adam',\n", "            loss='categorical_crossentropy',\n", "            metrics=['accuracy'], \n", "           )\n", "\n", "VGG_model.summary()\n", "\n", "Resnet_model = Sequential()\n", "Resnet_model.add(Dropout(0.2, input_shape=resnet_features.shape[1:]))\n", "Resnet_model.add(Dense(n_class, activation='softmax'))\n", "\n", "Resnet_model.compile(optimizer='adam',\n", "            loss='categorical_crossentropy',\n", "            metrics=['accuracy'],\n", "           )\n", "\n", "Resnet_model.summary()"]}, {"cell_type": "markdown", "metadata": {"_uuid": "fe1e1cc1f8b71dea51bab2199e791d6b22a66b7f", "_cell_guid": "1e18d180-a6ce-4f03-9c31-11ec61604058"}, "source": ["### Callbacks\n", "\n", "- ModelCheckpoint is used to get the best model after epochs\n", "\n", "- ReduceLROnPlateau is used to manipulate learning rate for more delciate correction"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "2c4945fc15b827f9dc0a3520923d8859f2c0880b", "_cell_guid": "6963712c-f7b9-4b13-932d-a0d6a1d9e420", "collapsed": true}, "outputs": [], "source": ["inception_callbacks=[ReduceLROnPlateau(monitor='acc', \n", "                                            patience=3, \n", "                                            verbose=1, \n", "                                            factor=0.5, \n", "                                            min_lr=0.00001),\n", "                      ModelCheckpoint(filepath='saved_models/inception.best.from_features.hdf5', \n", "                               verbose=1, save_best_only=True)\n", "                     ]\n", "\n", "xception_callbacks=[ReduceLROnPlateau(monitor='acc', \n", "                                            patience=3, \n", "                                            verbose=1, \n", "                                            factor=0.5, \n", "                                            min_lr=0.00001),\n", "                      ModelCheckpoint(filepath='saved_models/xception.best.from_features.hdf5', \n", "                               verbose=1, save_best_only=True)\n", "                     ]\n", "\n", "resnet_callbacks=[ReduceLROnPlateau(monitor='acc', \n", "                                            patience=3, \n", "                                            verbose=1, \n", "                                            factor=0.5, \n", "                                            min_lr=0.00001),\n", "                      ModelCheckpoint(filepath='saved_models/resnet.best.from_features.hdf5', \n", "                               verbose=1, save_best_only=True)\n", "                     ]\n", "\n", "vgg_callbacks=[ReduceLROnPlateau(monitor='acc', \n", "                                            patience=3, \n", "                                            verbose=1, \n", "                                            factor=0.5, \n", "                                            min_lr=0.00001),\n", "                      ModelCheckpoint(filepath='saved_models/vgg.best.from_features.hdf5', \n", "                               verbose=1, save_best_only=True)\n", "                     ]"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "77dba3f3c6475d557f0345a8edb11db78f5053e1", "_cell_guid": "fdb72102-0967-4db5-9240-22c53dbe5b02", "collapsed": true}, "outputs": [], "source": ["epochs = 1 # Increase this if you want more accurate results(It is recommended to run on personal computer in this case)\n", "\n", "from sklearn.utils import class_weight\n", "\n", "class_weight = class_weight.compute_class_weight('balanced', np.unique(y_eda), y_eda)\n", "\n", "inception_history = Inception_model.fit(X_train_inception, y_train_inception, \n", "          validation_data=(X_valid_inception, y_valid_inception),\n", "          epochs=epochs, \n", "          callbacks=inception_callbacks,\n", "          class_weight=class_weight,\n", "          batch_size=8, verbose=1)\n", "\n", "xception_history = Xception_model.fit(X_train_xception, y_train_xception, \n", "          validation_data=(X_valid_xception, y_valid_xception),\n", "          epochs=epochs,                            \n", "          callbacks=xception_callbacks,\n", "          class_weight=class_weight,\n", "          batch_size=8, verbose=1)\n", "\n", "resnet_history = Resnet_model.fit(X_train_resnet, y_train_resnet, \n", "          validation_data=(X_valid_resnet, y_valid_resnet),\n", "          epochs=epochs, \n", "          callbacks=resnet_callbacks,\n", "          class_weight=class_weight,\n", "          batch_size=8, verbose=1)\n", "\n", "vgg_history = VGG_model.fit(X_train_vgg, y_train_vgg, \n", "          validation_data=(X_valid_vgg, y_valid_vgg),\n", "          epochs=epochs, \n", "          callbacks=vgg_callbacks,\n", "          class_weight=class_weight,\n", "          batch_size=8, verbose=1)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "5a9d345b4983f6a16fe4a1927ea6d69aa0ebba95", "_cell_guid": "98e3e4f6-e6a9-4ce9-bb02-1b9c356ce4a2"}, "source": [" ## 3.2 Evaluation"]}, {"cell_type": "markdown", "metadata": {"_uuid": "578c71bebfaa603d484693d870fe421cf4971956", "_cell_guid": "784f0634-ba2a-4270-83bb-d531e437c91b"}, "source": ["### Learning Curve\n", "\n", "Learning rate is the step by which the optimizer walks through the 'loss landscape'. The higher it is, the bigger are the steps and the quicker is the convergence. However, the sampling is very poor with an high LR and the optimizer could probably fall into a local minima. Low learning rate shows slower convergence and lower chance of falling into a local minima, but it leads to underfitting and requires more epochs.\n", "To detect this, learning curve plot is used. \n", "\n", "\n", "Picture below shows each case of the learning rate's status based on loss:\n", "\n", "![Learning Curve](http://img1.imagilive.com/0717/learningrates.jpg)\n", "\n", "\n", "For accuracy:\n", "![Learning Curve](http://img1.imagilive.com/0717/accuracies1de.jpg)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "730a917ad057c07a128fc6c12bb0bde94ad74ec2", "_cell_guid": "32987e1b-a904-4536-992e-0f3fa163012e", "collapsed": true}, "outputs": [], "source": ["# Plot the loss and accuracy curves for training and validation on InceptionV3\n", "fig, ax = plt.subplots(2,1)\n", "ax[0].plot(inception_history.history['loss'], color='b', label=\"Training loss\")\n", "ax[0].plot(inception_history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n", "legend = ax[0].legend(loc='best', shadow=True)\n", "\n", "ax[1].plot(inception_history.history['acc'], color='b', label=\"Training accuracy\")\n", "ax[1].plot(inception_history.history['val_acc'], color='r',label=\"Validation accuracy\")\n", "legend = ax[1].legend(loc='best', shadow=True)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "68deefb3708137bae798bea36fc66259ad40f00d", "_cell_guid": "e0c60808-15d8-4c09-9783-5967999cb837", "collapsed": true}, "outputs": [], "source": ["# Plot the loss and accuracy curves for training and validation on xception model\n", "fig, ax = plt.subplots(2,1)\n", "ax[0].plot(xception_history.history['loss'], color='b', label=\"Training loss\")\n", "ax[0].plot(xception_history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n", "legend = ax[0].legend(loc='best', shadow=True)\n", "\n", "ax[1].plot(xception_history.history['acc'], color='b', label=\"Training accuracy\")\n", "ax[1].plot(xception_history.history['val_acc'], color='r',label=\"Validation accuracy\")\n", "legend = ax[1].legend(loc='best', shadow=True)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "b3c5a50ff0756cb6d4270c75fe99ff672d7e0ca3", "_cell_guid": "0a0f6352-ec49-46f6-98c9-64612fcf4b09", "collapsed": true}, "outputs": [], "source": ["# Plot the loss and accuracy curves for training and validation on resnet model\n", "fig, ax = plt.subplots(2,1)\n", "ax[0].plot(resnet_history.history['loss'], color='b', label=\"Training loss\")\n", "ax[0].plot(resnet_history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n", "legend = ax[0].legend(loc='best', shadow=True)\n", "\n", "ax[1].plot(resnet_history.history['acc'], color='b', label=\"Training accuracy\")\n", "ax[1].plot(resnet_history.history['val_acc'], color='r',label=\"Validation accuracy\")\n", "legend = ax[1].legend(loc='best', shadow=True)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "348ed6910eb4f64d5fe36e235e18b9684c5e3bcd", "_cell_guid": "c91786da-1634-40aa-929c-817db604538c", "collapsed": true}, "outputs": [], "source": ["# Plot the loss and accuracy curves for training and validation on vgg model\n", "fig, ax = plt.subplots(2,1)\n", "ax[0].plot(vgg_history.history['loss'], color='b', label=\"Training loss\")\n", "ax[0].plot(vgg_history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n", "legend = ax[0].legend(loc='best', shadow=True)\n", "\n", "ax[1].plot(vgg_history.history['acc'], color='b', label=\"Training accuracy\")\n", "ax[1].plot(vgg_history.history['val_acc'], color='r',label=\"Validation accuracy\")\n", "legend = ax[1].legend(loc='best', shadow=True)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "ba0294fc8c16129ba771b330091d3f976f056ff1", "_cell_guid": "7b2e7a3a-9c40-4337-81da-25f9dd9a8c53"}, "source": ["### Confusion Matrix \n", "\n", "Confusion matrix can check false positives for each labels.\n", "\n", "This can visualize bias and variance of the model's prediction."]}, {"cell_type": "markdown", "metadata": {"_uuid": "d29a0de646735760652a38520ab5c5371a5713f2", "_cell_guid": "8f114794-8597-4f99-ab4b-6e693d41dbb1"}, "source": ["### InceptionV3"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "71a1e518ce15cdbbdeb25405903cb2f5a436d5f5", "_cell_guid": "340f9e3c-0a22-4f46-b6d5-6efb0695a3b4", "collapsed": true}, "outputs": [], "source": ["# Look at confusion matrix \n", "from sklearn.metrics import confusion_matrix\n", "\n", "# Predict the values from the validation dataset\n", "Y_pred = Inception_model.predict(X_valid_inception)\n", "# Convert predictions classes to one hot vectors \n", "Y_pred_classes = np.argmax(Y_pred,axis = 1) \n", "# Convert validation observations to one hot vectors\n", "Y_true = np.argmax(y_valid_inception,axis = 1) \n", "# compute the confusion matrix\n", "confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n", "# plot the confusion matrix\n", "ax = sns.heatmap(confusion_mtx)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "fc2e708e47790c98739b4bb2e15bf945fcdd1df0", "_cell_guid": "b276b7e9-3893-4bd6-aa60-c6eb567ccc5c"}, "source": ["### Xception"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "2c600946d502124331850a86c0c696a952716e31", "_cell_guid": "1deb9084-b7e0-43b9-a3cc-159440f074bb", "collapsed": true}, "outputs": [], "source": ["# Look at confusion matrix \n", "from sklearn.metrics import confusion_matrix\n", "\n", "# Predict the values from the validation dataset\n", "Y_pred = Xception_model.predict(X_valid_xception)\n", "# Convert predictions classes to one hot vectors \n", "Y_pred_classes = np.argmax(Y_pred,axis = 1) \n", "# Convert validation observations to one hot vectors\n", "Y_true = np.argmax(y_valid_xception,axis = 1) \n", "# compute the confusion matrix\n", "confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n", "# plot the confusion matrix\n", "ax = sns.heatmap(confusion_mtx)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "4ce00efc1117584df510cdc254256ec737d883be", "_cell_guid": "486591d0-d5ce-4af4-bec7-ebe24ffeccb2"}, "source": ["### Resnet50"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "b64ec0a343cdc2faab35b6d7b36015ac1cd597a7", "_cell_guid": "d0140326-2b10-44c0-9017-38a5077e4f2b", "collapsed": true}, "outputs": [], "source": ["# Look at confusion matrix \n", "from sklearn.metrics import confusion_matrix\n", "\n", "# Predict the values from the validation dataset\n", "Y_pred = Resnet_model.predict(X_valid_resnet)\n", "# Convert predictions classes to one hot vectors \n", "Y_pred_classes = np.argmax(Y_pred,axis = 1) \n", "# Convert validation observations to one hot vectors\n", "Y_true = np.argmax(y_valid_resnet,axis = 1) \n", "# compute the confusion matrix\n", "confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n", "# plot the confusion matrix\n", "ax = sns.heatmap(confusion_mtx)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "84d0144b292ce04d2384847fb29dcd9a9a19b049", "_cell_guid": "c47211d7-9a43-49cf-b4e1-99c64be0cfef"}, "source": ["### VGG19"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "8a5f72ca6424af429faa8b438d034f0bb80fc9df", "_cell_guid": "f54fc78b-c102-47ee-b921-550bb30a1921", "collapsed": true}, "outputs": [], "source": ["# Look at confusion matrix \n", "from sklearn.metrics import confusion_matrix\n", "\n", "# Predict the values from the validation dataset\n", "Y_pred = VGG_model.predict(X_valid_vgg)\n", "# Convert predictions classes to one hot vectors \n", "Y_pred_classes = np.argmax(Y_pred,axis = 1) \n", "# Convert validation observations to one hot vectors\n", "Y_true = np.argmax(y_valid_vgg,axis = 1) \n", "# compute the confusion matrix\n", "confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n", "# plot the confusion matrix\n", "ax = sns.heatmap(confusion_mtx)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "c4dec5522aaa5a2db12687e3bba44a82ea87371c", "_cell_guid": "1bdfed62-4209-4772-9d8b-6a5c15d0139e"}, "source": ["# 4. Prediction and Submission"]}, {"cell_type": "markdown", "metadata": {"_uuid": "cec407d302c09149d609fba7a5d08b94046a532c", "_cell_guid": "84e770a3-a806-46c9-bcb6-7a8c3d37b844"}, "source": ["## 4.1 Prediction"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "9241d8a951896af0e6b51b501a19a0aaf1a0ba3c", "_cell_guid": "c9a9b3a1-59d4-424b-b9fc-db0b5ba6074c", "collapsed": true}, "outputs": [], "source": ["inception_features_test = get_features(InceptionV3, X)\n", "np.savez('bottleneck_features/inception_features_test.npz' , X=inception_features_test)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "acb7c374a62c4c274648d054a1acc1322e909d09", "_cell_guid": "344dbcde-d1d1-4e82-b3fd-c9db31870f03", "collapsed": true}, "outputs": [], "source": ["xception_features_test = get_features(Xception, X)\n", "np.savez('bottleneck_features/xception_features_test.npz' , X=xception_features_test)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "3cef20026c526e0d44571f30da20dd83605db91f", "_cell_guid": "5fc419fe-658e-4608-b0ef-258b78879c2b", "collapsed": true}, "outputs": [], "source": ["resnet_features_test = get_features(ResNet50, X)\n", "np.savez('bottleneck_features/resnet_features_test.npz' , X=resnet_features_test)"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "b01d3495297adacaaf9238957ad06dd9062df163", "_cell_guid": "eb68fb2f-845a-47da-923d-4ee4fccd0ea1", "collapsed": true}, "outputs": [], "source": ["vgg_features_test = get_features(VGG19, X)\n", "np.savez('bottleneck_features/vgg_features_test.npz' , X=vgg_features_test)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "a765b471f634c45c98bd42906e6f88018171c4b4", "_cell_guid": "26aefc4c-12a9-4a7c-8e57-b9a5aa1f4b8a"}, "source": ["### Load weights with the highest accuracy results"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "75259bbe86fb0a3682e134c1aeb00a0199ada791", "_cell_guid": "6ad3fcf3-8a1f-4b5f-888b-22830b0f0388", "collapsed": true}, "outputs": [], "source": ["Inception_model.load_weights('saved_models/inception.best.from_features.hdf5')\n", "Xception_model.load_weights('saved_models/xception.best.from_features.hdf5')\n", "Resnet_model.load_weights('saved_models/resnet.best.from_features.hdf5')\n", "VGG_model.load_weights('saved_models/vgg.best.from_features.hdf5')\n"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "8a2ddcd2d9b5aff65e9365b4f64503ca3d98a090", "_cell_guid": "e0081cc3-7821-48cc-bb51-4638b94f7b1d", "collapsed": true}, "outputs": [], "source": ["y_pred = Inception_model.predict(inception_features_test, batch_size=128)\n", "for b in breed:\n", "    df_test[b] = y_pred[:,class_to_num[b]]\n", "inception_test = df_test.copy()\n", "df_test.to_csv('pred_inception.csv', index=None)\n", "\n", "y_pred = Xception_model.predict(xception_features_test, batch_size=128)\n", "for b in breed:\n", "    df_test[b] = y_pred[:,class_to_num[b]]\n", "xception_test = df_test.copy()\n", "df_test.to_csv('pred_xception.csv', index=None)\n", "\n", "y_pred = Resnet_model.predict(resnet_features_test, batch_size=128)\n", "for b in breed:\n", "    df_test[b] = y_pred[:,class_to_num[b]]\n", "resnet_test = df_test.copy()\n", "df_test.to_csv('pred_resnet.csv', index=None)\n", "\n", "y_pred = VGG_model.predict(vgg_features_test, batch_size=128)\n", "for b in breed:\n", "    df_test[b] = y_pred[:,class_to_num[b]]\n", "vgg_test = df_test.copy()\n", "df_test.to_csv('pred_vgg.csv', index=None)\n"]}, {"cell_type": "markdown", "metadata": {"_uuid": "54f6028b021b5ad59b6e1796bf9c20344f8329f2", "_cell_guid": "1af68236-a682-4344-a046-a86fbfd91cb3"}, "source": ["## 4.2 Ensembling\n", "\n", "I used stacking method for ensembling the results from the models."]}, {"execution_count": null, "cell_type": "code", "metadata": {"_uuid": "3494f19f274b94098509c9de50259778681c3b53", "_cell_guid": "dee4b531-10b4-443e-8195-8bf28e3d85bd", "collapsed": true}, "outputs": [], "source": ["n_model = 4\n", "id_test = inception_test['id']\n", "sum_test = inception_test.drop('id', axis=1) + xception_test.drop('id', axis=1) + resnet_test.drop('id', axis=1) + vgg_test.drop('id', axis=1)\n", "ensemble_test = (np.exp(sum_test / n_model) - 1)\n", "ensemble_test.insert(0, 'id', id_test)\n", "\n", "ensemble_test.to_csv('pred_stacked.csv', index=None)"]}, {"cell_type": "markdown", "metadata": {"_uuid": "eb530a14f921064d23b6ab278e09bca9fff3dffb", "_cell_guid": "dd4aa7f4-4b20-4268-a80e-c75fb7853040"}, "source": ["## 4.3 Results"]}, {"cell_type": "markdown", "metadata": {"_uuid": "ee3ee43281ff690b1e9ef346fe06a0ff0adc8c71", "_cell_guid": "bebbe65c-a1ec-4505-8f36-ee9bcd2acb77"}, "source": ["1. Inception LB 0.73604\n", "2. Xception LB 0.57665\n", "3. Resnet LB 4.62757 (This one had the fuzziest confusion matrix, needs a lot of epochs or bigger learning rate to prevent underfitting)\n", "4. VGG19 LB 1.54441 (This one too needs more adjustments than others)\n", "5. Stacked LB 0.85407\n", "\n", "Fine tune these models then you will get better result"]}], "nbformat": 4, "metadata": {"language_info": {"nbconvert_exporter": "python", "version": "3.6.3", "name": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3", "file_extension": ".py", "mimetype": "text/x-python"}, "kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}}}