{"nbformat_minor": 1, "metadata": {"language_info": {"codemirror_mode": {"version": 3, "name": "ipython"}, "name": "python", "pygments_lexer": "ipython3", "file_extension": ".py", "nbconvert_exporter": "python", "version": "3.6.3", "mimetype": "text/x-python"}, "kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}}, "cells": [{"metadata": {"_uuid": "9064a37581dee01c4405b9d3a7f20ceedf38c01f", "_cell_guid": "fbaac637-6aa1-41ac-91cf-4d3b97f1924b"}, "cell_type": "markdown", "source": ["# What is in this kernel?\n", "### 1. Data Exploration\n", "* Checking for missing values\n", "* Visualizing distribution of various dog breeds\n", "\n", "### 2. Useful Scripts\n", "* Script to segregate images into their respective breeds (to help with augmentation later on)"]}, {"metadata": {"_uuid": "e8d947b68efde8f8b70dbbcdedb43d2bb5c3cd49", "collapsed": false, "_cell_guid": "963f2d12-5d1b-411e-b98e-3b3cbaa33b77"}, "outputs": [], "cell_type": "code", "source": ["import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "import seaborn as sns\n", "import missingno as msno\n", "from tqdm import tqdm\n", "import cv2\n", "from keras.preprocessing import image\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "labels = pd.read_csv('../input/labels.csv')\n", "\n", "print (labels.shape)"], "execution_count": 2}, {"metadata": {"_uuid": "46117d68db9210c48c0f5c9982a8183490013d04", "_cell_guid": "ec32598a-dbed-4eed-8d84-35f507bc5370"}, "cell_type": "markdown", "source": ["# Data Exploration"]}, {"metadata": {"_uuid": "acd8f144455ffd5af160a568e4a690f1f2eac5d4", "_cell_guid": "74f482a6-0034-4d3c-81a7-455d4618ab58"}, "cell_type": "markdown", "source": ["## Checking for Missing Values"]}, {"metadata": {"_uuid": "11607c1a2b96965799772ef242d23e5206c3b0bd", "_cell_guid": "d74b4328-a656-4b87-b462-89cf4a7f6d2c"}, "cell_type": "markdown", "source": ["**Are there any missing values here?**"]}, {"metadata": {"_uuid": "5c88c22318cbacb3dc22b8664bff3c3946904016", "collapsed": true, "_cell_guid": "019e32af-c8ec-4704-939d-6108b838fd40"}, "outputs": [], "cell_type": "code", "source": ["labels.isnull().values.any()"], "execution_count": null}, {"metadata": {"_uuid": "cc717ae61bb1e1c50899455b9da1e18f221b260d", "collapsed": true, "_cell_guid": "ae1de701-270a-4338-b6a9-4ed5520f2a50"}, "outputs": [], "cell_type": "code", "source": ["msno.matrix(labels, width_ratios=(5,1),\\\n", "            figsize=(16,8), color=(0.2,0.2,0.2), fontsize=18, sparkline=True, labels=True)"], "execution_count": null}, {"metadata": {"_uuid": "c3b3b06ccd4194ed204d3005dab1c02b4f551cb0", "_cell_guid": "d1c229d2-b101-4071-9e1c-b7735146c97e"}, "cell_type": "markdown", "source": ["## Visualization"]}, {"metadata": {"_uuid": "09bcf94c38b2942ebce79df7359b5ae68758211b", "_cell_guid": "da6be84c-b730-4ba7-85ed-b96749329f40"}, "cell_type": "markdown", "source": ["**There are 120 dog breeds in our dataset. The following snippet confirms the fact**"]}, {"metadata": {"_uuid": "d7ae55ec67ca139d643ae01f4e4c5024c0768ce8", "collapsed": true, "_cell_guid": "b1fa4866-5f2b-4910-a037-87f2d8a84e05"}, "outputs": [], "cell_type": "code", "source": ["labels.breed.nunique()"], "execution_count": null}, {"metadata": {"_uuid": "1105d8e9632b777894c45b694062cf5745d0ce11", "_cell_guid": "26eb791e-7530-4890-972d-a7397676a266"}, "cell_type": "markdown", "source": ["**Distribution of various dog breeds**"]}, {"metadata": {"_uuid": "c72cde4c6dd39bb44c32e31468b689e1b1c02c25", "collapsed": true, "_cell_guid": "d063d364-1857-40a9-9650-43f247c542a0"}, "outputs": [], "cell_type": "code", "source": ["yy = pd.value_counts(labels['breed'])\n", "\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(15, 9)\n", "sns.set_style(\"whitegrid\")\n", "\n", "ax = sns.barplot(x = yy.index, y = yy, data = labels)\n", "ax.set_xticklabels(ax.get_xticklabels(), rotation = 90, fontsize = 8)\n", "ax.set(xlabel='Dog Breed', ylabel='Count')\n", "ax.set_title('Distribution of Dog breeds')"], "execution_count": null}, {"metadata": {"_uuid": "2c3fb581e0b0bf90d4ae64cd53462970e2792671", "_cell_guid": "386170c1-1984-4f71-bc09-4a87344fbc1f"}, "cell_type": "markdown", "source": ["**It is clear that the data set is highly skewed. In order to train a model to get good results, we have to augment images of the classes with lower images in order to avoid class imbalance.**"]}, {"metadata": {"_uuid": "213c3917c682c8c313014f0d79e31f8e7ddb0861", "_cell_guid": "7aae28fe-9da9-485d-b5d3-cdf08559c108"}, "cell_type": "markdown", "source": ["**Top 5 breeds with highest number of data samples**"]}, {"metadata": {"_uuid": "bd67affb4cc9f3171af1157f6e3fe133538ef4ca", "collapsed": true, "_cell_guid": "d9737358-ccc2-459d-a62c-9900bdae2b74"}, "outputs": [], "cell_type": "code", "source": ["yy.head()"], "execution_count": null}, {"metadata": {"_uuid": "884f104af56bdc5a2da39ead63efe8cb5556b48b", "_cell_guid": "908423e1-3927-4227-9d65-1caaf3abaaa2"}, "cell_type": "markdown", "source": ["**Bottom 5 breeds with lowest number of data samples**"]}, {"metadata": {"_uuid": "033fe351b9fbb442ef0bd9b3126661a6656fbcf3", "collapsed": true, "_cell_guid": "9fe9f319-f0a2-41f9-9905-db219c892d0c"}, "outputs": [], "cell_type": "code", "source": ["yy.tail()"], "execution_count": null}, {"metadata": {"_uuid": "d218d0d7c39a9b6aef9392f3558ddb6d51724577", "_cell_guid": "43d01991-f987-4d05-83d6-787379e22e85"}, "cell_type": "markdown", "source": ["# Useful Scripts\n", "**In this section I have a bunch of scripts that might be useful**"]}, {"metadata": {"_uuid": "e164f27adfe5f67d5efbffec8015ec20d6e48fce", "_cell_guid": "89f9957f-a554-46a9-8b3b-1774578c9bed"}, "cell_type": "markdown", "source": ["### Average height and width of all training images"]}, {"metadata": {"_uuid": "9e7f5b72023cb6a1530ce541906e97e5d39a8068", "collapsed": true, "_cell_guid": "e1a09a80-a45a-4f8f-8f57-cb8393615e94"}, "outputs": [], "cell_type": "code", "source": ["i = 0\n", "img_height = []\n", "img_width = []\n", "for f, breed in tqdm(labels.values):\n", "    img = cv2.imread('../input/train/{}.jpg'.format(f))\n", "    img_height.append(img.shape[0])\n", "    img_width.append(img.shape[1])\n", "    i = f\n", "i    "], "execution_count": null}, {"metadata": {"_uuid": "cf564608fcc0ed7a97a816c87f8fabb8e9ddb889", "collapsed": true, "_cell_guid": "4684eb34-93b5-401b-ab75-ed85233de7f8"}, "outputs": [], "cell_type": "code", "source": ["print(np.mean(img_height))\n", "print(np.mean(img_width))"], "execution_count": null}, {"metadata": {"_uuid": "5d3e2ac41787bf1425b7207022e9d92e4427ad07", "_cell_guid": "01a6bc05-c690-48da-a14f-a1a0a97c7ee9"}, "cell_type": "markdown", "source": ["### Segregating images into folders based on their breeds\n", "**The following code snippet helps segregate the images in the `train` folder to another folder `new_train` where each image of the dog is placed in its respective breed folder**\n", "\n", "### Why the following?\n", "\n", "**With the follwoing code you can now augment the breed you think has less images!**"]}, {"metadata": {"_uuid": "9dd3f3a53fb8d93e89892c2c83621a56f7a57b8b", "collapsed": true, "_cell_guid": "3cf948e5-cdbc-49c0-a48f-44cfee6a76c1"}, "outputs": [], "cell_type": "code", "source": ["'''\n", "import pandas as pd\n", "import numpy as np\n", "import os\n", "import shutil\n", "\n", "\n", "path = \"/home/ubuntu/Kaggle/Dog_Classification/Data/\"\n", "\n", "label = pd.read_csv(path + \"labels.csv\")\n", "train_path = '/home/ubuntu/Kaggle/Dog_Classification/Data/train/'\n", "new_train_path = '/home/ubuntu/Kaggle/Dog_Classification/Data/new_train/'\n", "#--- snippet to split train images into 120 folders ---\n", "\n", "c = 0\n", "for i in range(len(label)):\n", "    l = label.id[i]\n", "    for filename in os.listdir(train_path):\t    \n", "        f = filename[:-4]\n", "        if (l == f):\n", "            print c\n", "            c+=1\n", "            if not os.path.exists(new_train_path + label.breed[i]):\n", "                os.makedirs(new_train_path + label.breed[i])\n", "                shutil.copy2(train_path + filename, new_train_path + label.breed[i])\t\t\t\t\n", "            else:\n", "                shutil.copy2(train_path + filename, new_train_path + label.breed[i])\n", "'''                "], "execution_count": null}, {"metadata": {"_uuid": "2da2b80f816eb9e4670b110fe10a5b0aaa4c15d2", "_cell_guid": "3c384650-0b8f-409d-9146-5641ab3a0e00"}, "cell_type": "markdown", "source": ["### Image Augmentation"]}, {"metadata": {"_uuid": "509a101719c32959a20d252abb503cd6c1703e6c", "collapsed": true, "_cell_guid": "f14cd44a-0802-40ea-ab8b-85a60afd2eb9"}, "outputs": [], "cell_type": "code", "source": ["\n", "\n", "fig, ax = plt.subplots()\n", "img = image.load_img('../input/train/fff43b07992508bc822f33d8ffd902ae.jpg')\n", "img = image.img_to_array(img)\n", "ax.imshow(img / 255.) \n", "ax.axis('off')\n", "plt.show()\n", "\n", "fimg = np.fliplr(img)\n", "fimg = image.img_to_array(fimg)\n", "ax.imshow(fimg / 255.) \n", "ax.axis('off')\n", "plt.show()\n", "np.flipud\n", "\n", "fig, ax = plt.subplots()\n", "im = plt.imread('../input/train/fff43b07992508bc822f33d8ffd902ae.jpg')\n", "plt.subplot(2, 1, 1)\n", "ax.axis('off')\n", "plt.imshow(im)\n", "plt.subplot(2, 1, 2)\n", "plt.imshow(np.fliplr(im))\n", "ax.axis('off')\n", "plt.show()"], "execution_count": null}, {"metadata": {"_uuid": "cbecd398fc0d1af6d35c2d3edfb4594c5e198270", "collapsed": true, "_cell_guid": "4f1a996d-f646-430e-b19f-1c39c118f1c9"}, "outputs": [], "cell_type": "code", "source": ["\n"], "execution_count": null}, {"metadata": {"_uuid": "409fb5b5eccbfeb24d04c155c4babd3a98245327", "collapsed": true, "_cell_guid": "22c9f015-0756-43ab-9a46-0bc260998fc3"}, "outputs": [], "cell_type": "code", "source": ["fig, ax = plt.subplots()\n", "img = np.fliplr(img)\n", "img = image.img_to_array(img)\n", "ax.imshow(img / 255.) \n", "ax.axis('off')\n", "plt.imshow(img)\n", "\n", "img = image.img_to_array(fimg)\n", "ax.imshow(img / 255.) \n", "ax.axis('off')\n", "plt.show()\n", "#plt.show(rimg)"], "execution_count": null}], "nbformat": 4}