{"metadata":{"kernelspec":{"name":"python3","language":"python","display_name":"Python 3"},"language_info":{"name":"python","pygments_lexer":"ipython3","mimetype":"text/x-python","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"version":"3.6.4","nbconvert_exporter":"python"}},"cells":[{"metadata":{"_kg_hide-output":true,"_uuid":"60f80d3d7a95ea743039eb897a404cd90e3904e5","_cell_guid":"e4791b71-6c29-4cb7-8c83-096c38cb5d0a"},"source":"# Intro\n\nWith some experimentation I found that VGG16 and VGG19 did not perform as well as Inception and XceptionV3 on the data. Therefore this kernel is about how to get the best out of the Xception and InceptionV3 pretrained weights using different ensembling methods.\n","cell_type":"markdown"},{"execution_count":null,"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"_uuid":"b5f0e7f31915743033c945839dd64770bca282ca","scrolled":true,"_cell_guid":"9a117fce-e7d6-41a2-8a8c-e73d27b8f5f3"},"source":"import numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications.xception import Xception\nfrom keras.applications.inception_v3 import InceptionV3\n\nfrom keras.applications.xception import preprocess_input as xception_preprocessor\nfrom keras.applications.inception_v3 import preprocess_input as inception_v3_preprocessor","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"c00bfad8f8e8e10b5778449cd40700f2f075d0c7","_cell_guid":"6a3d460a-0f16-4545-9fbf-a82deabec46d"},"source":"d# Data Exploration\n\nIf we look at the spread of frequencies of each class, the most common class has a frequency of almost double that of the least common class.\n\nNB. So that the kernel does not timeout we will limit to just the top 16 classes, but for submission we will predict on all classes.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"842d24fbe652fd120cff1b774c20eaf9445d2eec","_cell_guid":"02d7074e-e3db-428f-80e9-f4a92fe27e03"},"source":"LABELS = \"../input/dog-breed-identification/labels.csv\"\n\ntrain_df = pd.read_csv(LABELS)\n#return top 16 value counts and convert into list\nplt.figure(figsize=(13, 6))\ntrain_df['breed'].value_counts().plot(kind='bar')\nplt.show()\n\ntop_breeds = sorted(list(train_df['breed'].value_counts().head(16).index))\ntrain_df = train_df[train_df['breed'].isin(top_breeds)]\n\nprint(top_breeds)","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"2962e972c1fb7effecf2d332a8aca7b586d08d02","_cell_guid":"5a86b64a-99ca-478b-9dea-e2df60055be5"},"source":"# Train and Validation Split\n\nWe will load the images into an numpy array and split the data into train and  cross validation sets.\n\nI've chosen to take a 80:20 split of the data for cross validation. Strictly speaking we don't need to stratify split the dataset but it will ensure that the training and cross validation set are balanced.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"24c4289b8ec666db47335e607f997372a947167b","_cell_guid":"e87b2d23-cc29-4361-b960-143966571a6d"},"source":"from keras.preprocessing.image import img_to_array\nfrom keras.preprocessing.image import load_img\nfrom sklearn.model_selection import train_test_split\n\nSEED = 1234\n\nTRAIN_FOLDER = \"../input/dog-breed-identification/train/\"\nTEST_FOLDER = \"../input/dog-breed-identification/test/\"\n\nDIM = 299\n\ntrain_df['image_path'] = train_df.apply( lambda x: ( TRAIN_FOLDER + x[\"id\"] + \".jpg\" ), axis=1 )\n\ntrain_data = np.array([ img_to_array(load_img(img, target_size=(DIM, DIM))) for img in train_df['image_path'].values.tolist()]).astype('float32')\ntrain_labels = train_df['breed']\n\n\nx_train, x_validation, y_train, y_validation = train_test_split(train_data, train_labels, test_size=0.2, stratify=np.array(train_labels), random_state=SEED)\n\n#calculate the value counts for train and validation data\ndata = y_train.value_counts().sort_index().to_frame()\ndata.columns = ['train']\ndata['validation'] = y_validation.value_counts().sort_index().to_frame()\n\nnew_plot = data[['train','validation']].sort_values(['train']+['validation'], ascending=False)\nnew_plot.plot(kind='bar', stacked=True)\nplt.show()","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"dd688e94664a5061f7af657c26e9b1228a45fbfb","_cell_guid":"bb77e77b-e7d4-4a36-9c1c-1f6a402089b5"},"source":"## One-hot Encoding\n\nSince the output of our predictor for each input is a vector of probabilities for each class we must convert out label dataset to be the same format. That is for each input a row vector of length num_classes with a 1 at the index of the label and 0's everywhere else.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"42997e86530b345078d3498260b897bfa3734c36","_cell_guid":"06269a9f-ca34-4344-ae84-25adf97816b0"},"source":"# Let's convert our labels into one hot encoded format\n\ny_train = pd.get_dummies(y_train.reset_index(drop=True), columns=top_breeds).as_matrix()\ny_validation = pd.get_dummies(y_validation.reset_index(drop=True), columns=top_breeds).as_matrix()\n\nprint(y_train[0])","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"1775d285fc6849fa82e276ff964d727de9022292","_cell_guid":"3a6261bb-6351-40c5-a591-8bffc87a68e9"},"source":"Let's double check that our inputs and labels match.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"7874d898b930a42667728a61f5c584a02ff3f9ee","_cell_guid":"d3b181e7-471d-4257-9399-0850c0fce0cd"},"source":"plt.subplot(1, 2, 1)\nplt.title(top_breeds[np.where(y_train[5]==1)[0][0]])\nplt.axis('off')\nplt.imshow(x_train[5].astype(np.uint8))\n\nplt.subplot(1, 2, 2)\nplt.title(top_breeds[np.where(y_train[7]==1)[0][0]])\nplt.axis('off')\nplt.imshow(x_train[7].astype(np.uint8))\nplt.show()","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"eef1db5472d421abcdf149f9cd0d3ae7a1b5d778","_cell_guid":"fdfdbd10-afbc-4028-9634-a9dfb17a76ad"},"source":"# Generate bottleneck features\n\nSince kaggle kernels have no access to the internet we must use a pre-downloaded dataset and copy the files to the cache and models directory.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"5df2a9768533b4381be4c34a2b6e54076caec4be","_cell_guid":"da5212d7-0a8d-46df-9ea1-4c5c133760fe"},"source":"from os import makedirs\nfrom os.path import expanduser, exists, join\n\n!ls ../input/keras-pretrained-models/\n\ncache_dir = expanduser(join('~', '.keras'))\nif not exists(cache_dir):\n    makedirs(cache_dir)\nmodels_dir = join(cache_dir, 'models')\nif not exists(models_dir):\n    makedirs(models_dir)\n    \n!cp ../input/keras-pretrained-models/*notop* ~/.keras/models/\n!cp ../input/keras-pretrained-models/imagenet_class_index.json ~/.keras/models/\n!cp ../input/keras-pretrained-models/resnet50* ~/.keras/models/","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"9c3d3d73bd854a26f0887f98d6677680fe8be683","_cell_guid":"e37b91c0-3691-4e44-bd6d-b50b92038dae"},"source":"Let's define a function that will output bottleneck features from a given model. \n\nWe will use 'imagenet' weights and remove the final layers of the neural network so that we can use our own classifier.","cell_type":"markdown"},{"execution_count":null,"metadata":{"collapsed":true,"_uuid":"c3f767103e7b131dad138cf038caea029258dc49","_cell_guid":"e8cf9356-5ca4-4082-8840-84b3fa176622"},"source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import log_loss, accuracy_score\n\nbatch_size = 32\nepochs = 30\nnum_classes = len(top_breeds)\n\ndef generate_features(model_info, data, labels, datagen):\n    print(\"generating features...\")\n    datagen.preprocessing_function = model_info[\"preprocessor\"]\n    generator = datagen.flow(data, labels, shuffle=False, batch_size=batch_size, seed=model_info[\"seed\"])\n    bottleneck_model = model_info[\"model\"](weights='imagenet', include_top=False, input_shape=model_info[\"input_shape\"], pooling=model_info[\"pooling\"])\n    return bottleneck_model.predict_generator(generator)\n","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"a0845086c56f17489e854388273ba05927f310ab","_cell_guid":"0f0dc1f5-7e25-44b4-a67c-e084c54abce0"},"source":"## Define our models and run\n\nFirst we define the settings for our models such as the input shape and the preprocessor which we will feed into generate_features.\n\nThen let's generate our train features and validation features and save them to file so that we don't need to compute them again.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"49fe04a1adc1dc51bd109c51bb45eba154a49217","_cell_guid":"c3db3b42-6ee5-492c-bf0e-e3b81260c3b9"},"source":"import time\n\nmodels = {\n    \"InceptionV3\": {\n        \"model\": InceptionV3,\n        \"preprocessor\": inception_v3_preprocessor,\n        \"input_shape\": (299,299,3),\n        \"seed\": 1234,\n        \"pooling\": \"avg\"\n    },\n    \"Xception\": {\n        \"model\": Xception,\n        \"preprocessor\": xception_preprocessor,\n        \"input_shape\": (299,299,3),\n        \"seed\": 5512,\n        \"pooling\": \"avg\"\n    }\n}\n\nfor model_name, model in models.items():\n    print(\"Predicting : {}\".format(model_name))\n    filename = model_name + '_features.npy'\n    validfilename = model_name + '_validfeatures.npy'\n    if exists(filename):\n        features = np.load(filename)\n        validation_features = np.load(validfilename)\n    else:\n        train_datagen = ImageDataGenerator(\n                zoom_range = 0.3,\n                width_shift_range=0.1,\n                height_shift_range=0.1)\n        validation_datagen = ImageDataGenerator()\n        features = generate_features( model, x_train, y_train, train_datagen)\n        validation_features = generate_features(model, x_validation, y_validation, validation_datagen)\n        np.save(filename, features)\n        np.save(validfilename, validation_features)\n    \n    # Now that we have created or loaded the features  we need to do some predictions.\n    start_time = time.time()\n    \n    logreg = LogisticRegression(multi_class='multinomial', solver='lbfgs', random_state=SEED)\n    logreg.fit(features, (y_train * range(num_classes)).sum(axis=1))\n\n    model[\"predict_proba\"] = logreg.predict_proba(validation_features)\n    end_time = time.time()\n    print('Training time : {} {}'.format(np.round((end_time-start_time)/60, 2),' minutes'))","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"7f44936476cb6b624303784ade7ff5334bf52eca","_cell_guid":"926a2425-e8ac-4149-98a7-ea75e9ac2f08"},"source":"# Ensemble by average\n\nUsing a logistic regression classifier seems to yield good results so one method of esembling is to take the average probability from each prediction made from the logistric regression.\n\nWe have saved the predictions in \"predict_proba\" so it should be fairly easy to retrieve and ensemble.","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"e3fe4d5b0c6c6a23b4582ce868f9fc155b93264e","_cell_guid":"af86d51c-fc2c-402f-b04b-8a7df5878aad"},"source":"probas = [ model[\"predict_proba\"] for model_name, model in models.items() ]\n\navgprobas = np.average(probas, axis=0, weights=[1,1])\n\nprint('ensemble validation logLoss : {}'.format(log_loss(y_validation, avgprobas)))","outputs":[],"cell_type":"code"},{"execution_count":null,"metadata":{},"source":"import tensorflow as tf\n\nwith tf.Session() as sess:\n    result = sess.run(tf.one_hot(tf.argmax(avgprobas, dimension = 1), depth = 16))\n    print('ensemble validation accuracy : {}'.format(accuracy_score(y_validation, result)))","outputs":[],"cell_type":"code"},{"metadata":{"_uuid":"9e421d895515f6b58bb7ac8cbbcc04617af1503c","_cell_guid":"29373d33-3a25-4ef0-bf94-e8d23de8c7d0"},"source":"# Ensemble input features\n\nAnother way of ensembling is to merge the input features of the classifier together so we have more data to learn from. ","cell_type":"markdown"},{"execution_count":null,"metadata":{"_uuid":"d0cd2db54fb5c58cf160a8e697554ede182e37b0","_cell_guid":"1e11d450-6e7d-4fb2-b78d-a69bdd11c2ef"},"source":"features  = np.hstack( [ np.load(model_name + '_features.npy') for model_name, model in models.items() ])\nvalidation = np.hstack( [ np.load(model_name + '_validfeatures.npy') for model_name, model in models.items() ])\n\nlogreg = LogisticRegression(multi_class='multinomial', solver='lbfgs', random_state=SEED)\nlogreg.fit(features, (y_train * range(num_classes)).sum(axis=1))\n\npredict_probs = logreg.predict_proba(validation)\npredict_train = logreg.predict_proba(features)\n\nprint('ensemble of features va logLoss : {}'.format(log_loss(y_validation, predict_probs)))","outputs":[],"cell_type":"code"},{"metadata":{},"source":"# Accuracy\nAccuracy on train and validation sets","cell_type":"markdown"},{"execution_count":null,"metadata":{},"source":"with tf.Session() as sess:\n    result = sess.run(tf.one_hot(tf.argmax(predict_train, dimension = 1), depth = 16))\n    print('ensemble training accuracy : {}'.format(accuracy_score(y_train, result)))","outputs":[],"cell_type":"code"},{"execution_count":null,"metadata":{},"source":"with tf.Session() as sess:\n    result = sess.run(tf.one_hot(tf.argmax(predict_probs, dimension = 1), depth = 16))\n    print('ensemble validation accuracy : {}'.format(accuracy_score(y_validation, result)))","outputs":[],"cell_type":"code"}],"nbformat":4,"nbformat_minor":1}