{"cells": [{"metadata": {"_cell_guid": "c2324632-8d36-4bb3-bdc0-008fd1c26d3c", "_uuid": "3bf3ca0dbc0d1f05de52ca3c5a2f1a89b69f727d"}, "cell_type": "markdown", "source": ["# Introduction\n", "\n", "I was watching some of the videos of the [fast.ai](http://course.fast.ai/) deep learning course (highly recommended), and I had an idea for a small project. How many times you took a photo and, for some reason, it appeared upside down? I wanted to see whether I can train a Neural Net model to predict whether an image is rotated.  \n", "\n", "This notebook demonstrates how to create a neural net based classifier to predict the rotation of a given image using Python and Keras.\n", "\n", "I used the \"Dogs vs. Cats\" [dataset](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition). This dataset contains 25,000 dogs and cats images, however, since I wanted this notebook to run relatively fast, I only used 200 dogs images. \n", "\n", "The code below converts the images into tensors and rotates each image in one of 4 directions: 0\u00b0, 90\u00b0, 180\u00b0 and 270\u00b0. Then it assigns a label for each image according to the rotation that was applied to it: '0' for 0\u00b0 rotation, '1' for 90\u00b0 , '2' for 180\u00b0 rotation and '3' for 270\u00b0 rotation. Finally, it uses one-hot-encoding to represent the labels."]}, {"metadata": {"_cell_guid": "21d7bb42-fd80-455e-ade7-e69b3ecbeda9", "_uuid": "67914a160772b3cec1afa4613648d89e8d3eba9c"}, "source": ["%matplotlib inline\n", "from keras.preprocessing.image import array_to_img,img_to_array,load_img\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from glob import glob\n", "from keras.utils import to_categorical\n", "\n", "TRAIN_PATH = '../input/dogs-vs-cats-redux-kernels-edition/train/'\n", "NUM_OF_IMAGES = 200\n", "NUMBER_OF_ROTATIONS = 4\n", "\n", "images,images_arr,labels = [],[],[]\n", "label = 0\n", "for path in glob(TRAIN_PATH+'dog*')[:NUM_OF_IMAGES]:\n", "    #load the image\n", "    img = load_img(path,target_size=(224,224))\n", "    \n", "    #convert the image to a tensor\n", "    img_arr = img_to_array(img)\n", "    \n", "    #rotate the image according to the label\n", "    img_arr = np.rot90(img_arr,label)\n", "    \n", "    #compute the rotated image\n", "    img = array_to_img(img_arr)\n", "\n", "    #save the images,tensors and labels\n", "    images.append(img)\n", "    images_arr.append(img_arr)\n", "    labels.append(label)\n", "    \n", "    #next image will be rotated 90 degrees more\n", "    label = (label+1)%NUMBER_OF_ROTATIONS\n", "    \n", "images_arr = np.asarray(images_arr)\n", "labels = to_categorical(labels)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "53ce861a-e4a9-4f61-9e59-ad42d35cdc9c", "_uuid": "64b06e5dcb5641934cf1cadb9205294a23a2e286"}, "cell_type": "markdown", "source": ["Lets plot the first 8 images with their corresponding rotation "]}, {"metadata": {"_cell_guid": "abc3915f-01af-4564-ad93-9ced62ff4735", "_uuid": "d7521bfb40e721d8a41e2606bc5039f723845b04"}, "source": ["IMAGES_TO_PLOT = 8\n", "_,axis = plt.subplots(1, IMAGES_TO_PLOT,figsize=(15,15))\n", "for i,(img,label) in enumerate(zip(images,labels)):\n", "    if i==IMAGES_TO_PLOT:\n", "        break\n", "        \n", "    axis[i].imshow(img)\n", "    axis[i].xaxis.set_visible(False) \n", "    axis[i].yaxis.set_visible(False)\n", "    \n", "    label = np.argmax(label)\n", "    if label==0:\n", "        axis[i].set_title('Original Image')\n", "    else:\n", "        axis[i].set_title('{}\u00b0 Rotation'.format(label*90))"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "070cfbef-4de6-4414-92e3-f8f9c1259577", "_uuid": "8846f620c37e150ddc363bf941952d8a53f1eb5e"}, "cell_type": "markdown", "source": ["# Method"]}, {"metadata": {"_cell_guid": "f983ccf4-98c4-4732-bf85-baf1ed1c58dd", "_uuid": "832d0bcaad718e3bce73d3fcec99fc53ca5c08dc"}, "cell_type": "markdown", "source": ["The general idea is pretty simple. We will take a trained network that can classify images into some predifined classes. One option is the [VGG16]((https://arxiv.org/pdf/1409.1556.pdf)) network which looks like this (image taken from [here](http://book.paddlepaddle.org/03.image_classification/)):\n", "<img src=\"http://book.paddlepaddle.org/03.image_classification/image/vgg16.png\">\n", "\n", "As you can see, the network is pretty big (i.e., \"very deep\") and training such a model from scratch requires a lot more training examples, takes a few days, and requires more computer power than my laptop.\n", "\n", "Instead, we will use this network to generate features for our images. We will do that by keeping the weights of the network fixed and removing the last fully connected layers. Instead of these layers, we will add a new fully connected layer and a softmax layer that takes as an input the output of the VGG16 model and tries to predict the rotation of the image. \n", "\n", "The VGG16 network extracts useful features and we train a new classifier to separate those features into our four classes. This technique is called transfer learning as knowledge is transferred from one task another. You can read more about it [here](http://cs231n.github.io/transfer-learning/)."]}, {"metadata": {"_cell_guid": "92b4ce6a-94c4-4021-b02f-abe928a5a75a", "_uuid": "04d1d0338bd57fc8d4bbc836a024be68f05e0052"}, "cell_type": "markdown", "source": ["# Implementation"]}, {"metadata": {"_cell_guid": "eb1542c5-3852-4189-bd1c-a4db4548e12d", "_uuid": "713d74355f8b843058e316d87a0bcf4471fce70c"}, "cell_type": "markdown", "source": ["Lets first create a train and test sets:"]}, {"metadata": {"collapsed": true, "_cell_guid": "86e88586-0375-4bb6-ba03-2b7f94c26d10", "_uuid": "1153704a09eae22ed2ba07d4ace2ee51b60853f7"}, "source": ["images_train = images[:NUM_OF_IMAGES//2]\n", "images_test = images[NUM_OF_IMAGES//2:]\n", "\n", "images_arr_train = images_arr[:NUM_OF_IMAGES//2]\n", "images_arr_test = images_arr[NUM_OF_IMAGES//2:]\n", "\n", "labels_train = labels[:NUM_OF_IMAGES//2]\n", "labels_test = labels[NUM_OF_IMAGES//2:]"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "9846cb67-3a62-4a05-972e-0c42df3e994c", "_uuid": "d4eb6228f41e73d0092f2fa9ffdd390ead5c76a9"}, "cell_type": "markdown", "source": ["As we are not going to fine tune the weights of the VGG16 (we will keep them fixed during training), we can compute the extracted features a priori. This is going to make the training much faster as we will not have to recompute them in each forward pass. Unless you run this on a GPU, it is going to take a few minutes."]}, {"metadata": {"collapsed": true, "_cell_guid": "5060d1c0-a5b4-40a0-bf24-f07fed267d0a", "_uuid": "f4c4d33c4efac1f29845df7e4910d8be2f343f47"}, "source": ["from keras.applications.vgg16 import VGG16\n", "import h5py\n", "from keras.engine import topology\n", "\n", "def load_split_weights(model, model_path_pattern='model_%d.h5', memb_size=102400000):  \n", "    \"\"\"Loads weights from split hdf5 files.\n", "    \n", "    Parameters\n", "    ----------\n", "    model : keras.models.Model\n", "        Your model.\n", "    model_path_pattern : str\n", "        The path name should have a \"%d\" wild card in it.  For \"model_%d.h5\", the following\n", "        files will be expected:\n", "        model_0.h5\n", "        model_1.h5\n", "        model_2.h5\n", "        ...\n", "    memb_size : int\n", "        The number of bytes per hdf5 file.  \n", "    \"\"\"\n", "    model_f = h5py.File(model_path_pattern, \"r\", driver=\"family\", memb_size=memb_size)\n", "    topology.load_weights_from_hdf5_group_by_name(model_f, model.layers)\n", "    \n", "    return model\n", "\n", "'''\n", "This code is taken from https://www.kaggle.com/ekkus93/keras-models-as-datasets-test\n", "As we are running on Kaggle server, we can't download the VGG16 weights from github.\n", "If you are running it on your machine, you can simply replace this code with:\n", "base_model = VGG16(weights='imagenet', include_top=False)\n", "'''\n", "vgg16 = VGG16(include_top=False, weights=None)  \n", "keras_models_dir = '../input/keras-models'\n", "model_path_pattern = keras_models_dir + \"/vgg16_weights_tf_dim_ordering_tf_kernels_%d.h5\" \n", "base_model = load_split_weights(vgg16, model_path_pattern)\n", "\n", "def pretrained_features(arr,base_model):\n", "    features = base_model.predict(arr,batch_size=100, verbose=1)\n", "    return features.reshape((features.shape[0],-1))"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "76731cc5-a06c-4c92-a42f-ee6bb9342f53", "_uuid": "884f7fb6b1e3a473afcc68769c0bc1e8a728c221"}, "source": ["features_train = pretrained_features(images_arr_train,base_model)\n", "features_test = pretrained_features(images_arr_test,base_model)"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "548b709a-8fcd-4bf7-864f-eca802596414", "_uuid": "a181b60c1f2f498ece60145676e99794a6a374bf"}, "cell_type": "markdown", "source": ["Now we can create a new classifier that given the preprocessed features predicts the rotation of the corresponding image"]}, {"metadata": {"scrolled": false, "_cell_guid": "f7cf232e-f95d-4579-ab1d-b17956d4f6a7", "_uuid": "40177faeed4eb9e9dcd9d4864152fafa285c8c5d"}, "source": ["from keras.layers import Dense, GlobalAveragePooling2D,Dropout,Flatten\n", "from keras.applications.vgg16 import VGG16,preprocess_input\n", "from keras.models import Sequential\n", "from keras.optimizers import Adam\n", "from keras.regularizers import l2\n", "from keras.callbacks import EarlyStopping,ReduceLROnPlateau\n", "from keras.initializers import RandomNormal\n", "\n", "model = Sequential()\n", "model.add(Dense(128, input_dim=features_train.shape[1],activation='relu',\n", "                kernel_regularizer=l2(0.1),kernel_initializer=RandomNormal(stddev=0.001)))\n", "model.add(Dropout(0.5))\n", "model.add(Dense(4, activation='softmax',\n", "                kernel_regularizer=l2(0.1),kernel_initializer=RandomNormal(stddev=0.001)))\n", "model.compile(optimizer=Adam(lr=0.0001),\n", "              loss='categorical_crossentropy',\n", "              metrics=['accuracy'])\n", "\n", "model.fit(features_train, labels_train, batch_size=50, epochs=15,\n", "          validation_data=(features_test,labels_test),\n", "          callbacks=[EarlyStopping(patience=0)])"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "4dd19505-0fb7-4e14-8d98-f8cf207a11d4", "_uuid": "fba5c38fe3b51065bac696a6b230f3a78794b792"}, "cell_type": "markdown", "source": ["# Results"]}, {"metadata": {"_cell_guid": "4a2585b3-7c4e-451d-bcd6-eda65708beb4", "_uuid": "9d22327c8fc56d57ef8b8092ec3eadbec75b8166"}, "cell_type": "markdown", "source": ["The classifier achieves about 82% accuracy on the validation set (a random classifier would achieve 25%).\n", "\n", "Lets do some error analysis, starting from the confusion matrix."]}, {"metadata": {"_cell_guid": "51a8d1fc-d1e0-4fde-b93b-9855f054bf93", "_uuid": "65563ab713470d20f337567d452e950f661b943f"}, "source": ["from sklearn.metrics import confusion_matrix\n", "import seaborn as sns\n", "\n", "\n", "predictions = model.predict_classes(features_test)\n", "true_classes = np.argmax(labels_test,axis=1)\n", "cnf_matrix = confusion_matrix(true_classes, predictions)\n", "\n", "sns.heatmap(cnf_matrix,annot=True,annot_kws={\"size\": 14})\n", "plt.ylabel('True Class')\n", "plt.xlabel('Predicted Class')"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "e362285a-5f98-4711-8a5f-8ce717c4271c", "_uuid": "e8f5891099732eba784dd0e9982e5f716b219cbd"}, "cell_type": "markdown", "source": ["The numbers on the diagonal correspond to the class accuracy. The classifier is doing better on images with no rotation and images with 90 degrees rotation. Also, the classifier confuses 90 degrees rotations with 270 rotations. That make sense as these images are similar."]}, {"metadata": {"_cell_guid": "34f1108d-9bc4-4a0f-9466-fb652e827794", "_uuid": "5eb7e7ffac7605b9944d16ef347567d1c3d2f983"}, "cell_type": "markdown", "source": ["Lets examine the images with the lowest probability assigned by the model to the correct class (which will correspond to the highest cross entropy loss): "]}, {"metadata": {"_cell_guid": "7de6a639-6ae6-4622-834c-19602919b672", "_uuid": "6eaaffb3d3cbf52ccbca67ad23b6ee9095bf5ff7"}, "source": ["predictions = model.predict_proba(features_test)\n", "pred_true_class = predictions[range(len(predictions)),true_classes]\n", "sorted_images = [images_test[i] for i in np.argsort(pred_true_class)]\n", "\n", "_,axis = plt.subplots(1, 4,figsize=(15,15))\n", "pred_true_class.sort()\n", "for i in range(4):\n", "    axis[i].imshow(array_to_img(sorted_images[i]))\n", "    axis[i].xaxis.set_visible(False) \n", "    axis[i].yaxis.set_visible(False)\n", "    axis[i].set_title('predicted: {0:.3f}'.format(pred_true_class[i]))"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"collapsed": true, "_cell_guid": "ac9bc6c4-38da-45de-9d2d-0d9cb84b72d6", "_uuid": "f4638d63d3e1467385f2008e5bd3eeccb941dba9"}, "cell_type": "markdown", "source": ["These are difficult cases. Especially the second image from the left.\n", "\n", "Lets see the images for which the model has assigned the highest probability to the true class:"]}, {"metadata": {"_cell_guid": "2f992b9c-4445-4a6a-a61e-a2b3e5812488", "_uuid": "baaff473bcf2c9d423593ca772544078469af09f"}, "source": ["predictions = model.predict_proba(features_test)\n", "pred_true_class = predictions[range(len(predictions)),true_classes]\n", "sorted_images = [images_test[i] for i in np.argsort(pred_true_class)]\n", "\n", "_,axis = plt.subplots(1, 4,figsize=(15,15))\n", "pred_true_class.sort()\n", "for i in range(4):\n", "    axis[i].imshow(array_to_img(sorted_images[-1*(i+1)]))\n", "    axis[i].xaxis.set_visible(False) \n", "    axis[i].yaxis.set_visible(False)\n", "    axis[i].set_title('predicted: {0:.3f}'.format(pred_true_class[-1*(i+1)]))"], "execution_count": null, "cell_type": "code", "outputs": []}, {"metadata": {"_cell_guid": "758f0d9b-3bbc-4bec-bb64-c1236edbe70c", "_uuid": "d1a38d7471a5f4a3a9a66b08898411dcad9fe635"}, "cell_type": "markdown", "source": ["All images are clear and are not rotated."]}, {"metadata": {"_cell_guid": "96feec26-4514-419e-9265-fb920d6f067d", "_uuid": "8d49d4e652959877c8a372c179ae10d100e992be"}, "cell_type": "markdown", "source": ["# Next Steps\n", "\n", "- Train with more data - I've only used 100 images for training and more examples will probably improve the model. Additionally, the model used only dog images, and it is interesting to see whether it can perform as well on richer datasets (e.g., ImageNet).\n", "- Tune the hyper-parameters - I didn't invest much time in picking the best hyper-parameters. Grid or random search will probably yield a better model.\n", "- Predict more than 4 rotations - the model was trained to predict whether an image is rotated in 4 directions. An obvious extension would be to train it to predict finer grain rotations, i.e., 360 classes, or perhaps predict a continuous rotation (i.e., regression)."]}, {"metadata": {"collapsed": true, "_cell_guid": "75080b78-578d-412b-b3ce-2a5894779c88", "_uuid": "3fa79e4ce56ddd64eef3f9676a92ac154f4e5c97"}, "source": [], "execution_count": null, "cell_type": "code", "outputs": []}], "nbformat": 4, "metadata": {"language_info": {"pygments_lexer": "ipython3", "nbconvert_exporter": "python", "name": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "version": "3.6.1"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat_minor": 1}