{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "c0a13489-68e7-fbfe-866b-77d78a378527" }, "source": [ "So far everyone (that I have seen) loads all the data into memory and trains away. But what if our data is too big or we are too lazy to have an large computer. Enter the ImageDataGenerator() from keras along with its companion flow_from_directory. Using these two tools I will \"stream\" data into the model. I cannot load all of the pics into memory at 128x128 (I can load all the pics at 64x64 barely) as I have 8Gb memory. That motivated me to get this solution working.\n", "\n", "A couple of assumptions. All of your training data starts in a /train sub-folder. This script will create the necessary subfolders, but the structure is pretty simple. You data needs to be divided into folders--the names are not important at the moment--here we use \"dog\" and \"Cat\", but it could be \"A\" and \"B\" or \"Bob\" and \"Felicia\"\n", "\n", "Lets get started\n", "First we make 2 subdirectories in \"train\" for our examples.\n", "\n", "Kaggle is read only, so there is a bit of pain in doing an example here, but I will do my best\n", "Please ignore the errors as its a product of auto-run. I tried to include my output where helpful" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "52d60232-5f0f-f499-a2ee-bb2f6c452a46" }, "outputs": [], "source": [ "import os\n", "%mkdir \"train/dog\"\n", "%mkdir \"train/cat\"" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "545abbbf-3a63-b281-449b-f6d29074cb5c" }, "source": [ "Now we will move pictures based on their name. We are fortunate in that the picture name contains the label" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "fb219416-6468-998d-7f45-e61d7a095b2b" }, "outputs": [], "source": [ "for dir, subdir, files in os.walk(\"train\"):\n", " if len(subdir) == 0:\n", " continue\n", " for file in files:\n", " category = file.split(\".\")[0]\n", " os.rename(\"{}/{}\".format(dir,file), \"{}/{}/{}\".format(dir, category, file))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "39425bec-d48a-c9da-618b-6979394647f0" }, "source": [ "Now we are going to make a validation set. First we make the folders" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4aa25e9f-e98a-bd8d-3fbd-4698b4180c48" }, "outputs": [], "source": [ "%mkdir \"valid\"\n", "%mkdir \"valid/dog\"\n", "%mkdir \"valid/cat\"" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9149c3b1-51e2-658b-71d6-0bebfc40823e" }, "source": [ "Then we get a list of all the dog and cat files in their respective directories" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "64b3d7fe-9179-b9f5-e3f6-ed223add7927" }, "outputs": [], "source": [ "dogs = [x for x in os.listdir(\"train/dog\")]\n", "cats = [x for x in os.listdir(\"train/cat\")]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "66e9f1bc-83f6-75b5-3599-fd6cfd2b1825" }, "source": [ "Now we pick out 1000 samples at random from each of the training categories and move them into the validation set" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "fd4fa385-ef5e-9645-3876-289a52114a9c" }, "outputs": [], "source": [ "import random\n", "if len(os.listdir(\"valid/dog\")) < 1:\n", " for n in random.sample(range(len(dogs)), 1000):\n", " os.rename(\"train/dog/{}\".format(dogs[n]), \"valid/dog/{}\".format(dogs[n]))\n", "\n", "if len(os.listdir(\"valid/cat\")) < 1:\n", " for n in random.sample(range(len(cats)), 1000):\n", " os.rename(\"train/cat/{}\".format(cats[n]), \"valid/cat/{}\".format(cats[n]))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6f66e45d-825d-5e88-d8e9-38a7b881ba2b" }, "source": [ "Now it is time to use ImageDataGenerator\n", "First I will define some variables so I can easily change my inputs, then I will build 3 generators\n", "(1) A training generator\n", "(2)A validation generator\n", "(3) A test sample generator\n", "\n", "The .flow_from_directory method is given a folder (for example (1) it is \"train\"). the method looks in this folder for subfolders and picks up those subfolders as labels (which is why we did the data separation first). In the output you will see it found 2 categories (aka 2 subfolders).\n", "\n", "The same thing happens for the \"valid\" folder. We tell the flow_from_directory method \"look in the folder marked 'valid' for examples\"\n", "\n", "Finally for 'test' we tell it there are no categories. We didn't setup any folders for test anyway! However, you need to put them in a subdirectory of test. It can be called anything. I like to make mine silly names like 'Steve' or 'Why-cant-I-get-a-job-as-a-Data-Scientist'\n", "\n", "The neat thing about the generators is you can do also sorts of transforms on the fly. Here I just did a very simple read in, but look at the documentation and you can flip samples, rotate them, do all sorts of nice tricks to help generalize your data set.\n", "\n", "https://keras.io/preprocessing/image/\n", "\n", "\n", "You should get output that looks like this:\n", "\n", " Found 23000 images belonging to 2 classes.\n", " Found 2000 images belonging to 2 classes.\n", " Found 12500 images belonging to 1 classes." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "89087a75-dd90-66e6-9c70-b9aef08b46a3" }, "outputs": [], "source": [ "from keras.preprocessing.image import ImageDataGenerator\n", "\n", "train_datagen = ImageDataGenerator(rescale=1./255)\n", "test_datagen = ImageDataGenerator(rescale=1./255)\n", "\n", "train_generator = train_datagen.flow_from_directory(\n", " 'train',\n", " target_size=(ROWS,COLS),\n", " batch_size=BATCH_SIZE,\n", " class_mode='categorical')\n", "\n", "validation_generator = test_datagen.flow_from_directory(\n", " 'valid',\n", " target_size=(ROWS, COLS),\n", " batch_size=BATCH_SIZE,\n", " class_mode='categorical')\n", "\n", "#don't forget, if you leave your images in /test the flow_from directory won't find them\n", "#put test images in /test/\n", "#also notice not quotes around the None in the class_mode!!\n", "test_generator=test_datagen.flow_from_directory(\n", " 'test',\n", " target_size=(ROWS,COLS),\n", " batch_size=BATCH_SIZE\n", " class_mode=None)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3cb9d6ed-3526-1210-e507-51a1b03db73c" }, "source": [ "\n", "Now I put together an Neural Net. This is one I was working with, it has no special signifigance other than it is roughly modeled on samples from Keras and the cs231n tutorial and the first two gents mention in my final comments. It is by no means optimized, and serves only as an example CNN" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1d6a7aa9-27d2-b766-3295-c544ad960ee9" }, "outputs": [], "source": [ "from keras.models import Sequential\n", "from keras.layers import Convolution2D\n", "from keras.layers import Dense, Activation, Flatten, Dropout, MaxPooling2D\n", "from keras.regularizers import l2\n", "\n", "model = Sequential()\n", "model.add(Convolution2D(16, 4, 4, border_mode='same', input_shape=(ROWS,COLS,CHANNELS),activation='relu'))\n", "model.add(Convolution2D(4, 3, 3, border_mode='same', activation='relu'))\n", "model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), border_mode='same'))\n", "model.add(Convolution2D(4, 3, 3, border_mode='same', activation='relu'))\n", "model.add(Convolution2D(4, 3, 3, border_mode='same', activation='relu'))\n", "model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), border_mode='same'))\n", "model.add(Flatten())\n", "model.add(Dense(output_dim=64, W_regularizer=l2(0.01)))\n", "model.add(Dropout(0.5))\n", "model.add(Activation('relu'))\n", "model.add(Dense(output_dim=2, W_regularizer=l2(0.01))) #binary classification\n", "model.add(Activation('softmax'))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f0661de1-27aa-b203-f8cb-93f8b0945b45" }, "source": [ "Now we compile the model and run using the .fit_generator() method. No laughing at my poor old graphics card. If you are doing CPU only, this may take a while, or you may want to scale back--FAIR WARNING!!\n", "\n", "Make sure your 'samples_per_epoch' match the number of samples found in your generator--keras will throw a warning otherwise.\n", "\n", "Notice also instead of \"model.fit\" we use \"model.fit_generator\" and our generator produces both X and Y inputs, as well as self-contained batch size." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a8209579-d087-7816-f2df-4c3c0f4ff11d" }, "outputs": [], "source": [ "from keras.utils.np_utils import to_categorical\n", "from keras.optimizers import SGD, RMSprop\n", "model.compile(loss='binary_crossentropy', optimizer=RMSprop(lr=1e-4), metrics=['accuracy'])\n", "model.fit_generator(generator=train_generator,samples_per_epoch=23000,nb_epoch=20)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d81aa508-e523-c715-3f1d-8eecf0839851" }, "source": [ " Epoch 1/20\n", " 23000/23000 [==============================] - 120s - loss: 0.9738 - acc: 0.5855 \n", " Epoch 2/20\n", " 23000/23000 [==============================] - 118s - loss: 0.7010 - acc: 0.6287 \n", " Epoch 3/20\n", " 23000/23000 [==============================] - 118s - loss: 0.6650 - acc: 0.6397 \n", " Epoch 4/20\n", " 23000/23000 [==============================] - 117s - loss: 0.6441 - acc: 0.6577 \n", " Epoch 5/20\n", " 23000/23000 [==============================] - 119s - loss: 0.6259 - acc: 0.6823 \n", " Epoch 6/20\n", " 23000/23000 [==============================] - 119s - loss: 0.6069 - acc: 0.7027 \n", " Epoch 7/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5984 - acc: 0.7124 \n", " Epoch 8/20\n", " 23000/23000 [==============================] - 119s - loss: 0.5909 - acc: 0.7145 \n", " Epoch 9/20\n", " 23000/23000 [==============================] - 119s - loss: 0.5839 - acc: 0.7234 \n", " Epoch 10/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5852 - acc: 0.7204 \n", " Epoch 11/20\n", " 23000/23000 [==============================] - 119s - loss: 0.5789 - acc: 0.7231 \n", " Epoch 12/20\n", " 23000/23000 [==============================] - 119s - loss: 0.5770 - acc: 0.7230 \n", " Epoch 13/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5740 - acc: 0.7273 \n", " Epoch 14/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5719 - acc: 0.7301 \n", " Epoch 15/20\n", " 23000/23000 [==============================] - 119s - loss: 0.5716 - acc: 0.7292 \n", " Epoch 16/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5676 - acc: 0.7338 \n", " Epoch 17/20\n", " 23000/23000 [==============================] - 117s - loss: 0.5653 - acc: 0.7351 \n", " Epoch 18/20\n", " 23000/23000 [==============================] - 120s - loss: 0.5646 - acc: 0.7353 \n", " Epoch 19/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5615 - acc: 0.7390 \n", " Epoch 20/20\n", " 23000/23000 [==============================] - 118s - loss: 0.5607 - acc: 0.7397 \n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0014e644-71ec-4a44-bf02-2af5af362bcb" }, "source": [ "Lets see how we did" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "379891a0-b7a5-9ca3-4340-77ac595e9ac4" }, "outputs": [], "source": [ "print(\"valid set :\", model.evaluate_generator(validation_generator,val_samples=2000)[1]*100, \"%\")\n", "print(\"--------------------\")\n", "print(\"train set :\", model.evaluate_generator(train_generator,val_samples=2000)[1]*100, \"%\")" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "754fae73-725c-8bcc-ebac-a593eb0e585a" }, "source": [ " valid set : 73.15 %\n", " --------------------\n", " train set : 75.1984126984 %" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "52927e2f-54ff-148d-a118-5b5fd45e4cc9" }, "source": [ "Source of where I pulled all this from\n", "\n", "http://cs231n.github.io/convolutional-networks/\n", "\n", "https://keras.io/\n", "\n", "user Jeff Delaney posted this good starter that got myself rolling and wondering why dogs always appeared blue (I found out, don't worry):\n", "\n", "https://www.kaggle.com/jeffd23/dogs-vs-cats-redux-kernels-edition/catdognet-keras-convnet-starter\n", "\n", "user Charles Gallay posted a black and white that really got me going on how to make subsets, and why his cat looked like it was on LSD (again, I figured out why)\n", "https://www.kaggle.com/cgallay/dogs-vs-cats-redux-kernels-edition/cat-dog-notebook\n", "\n", "Finally David de la Iglesia Castro posted a very nice example of a pre-trained network (which really is the way to go) that got me wondering how to \"stream\" the data as I couldn't fit his original model into my meager computer.\n", "https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/forums/t/27950/tutorial-how-to-get-0-06-loss-in-70-lines-of-code?forumMessageId=159119#post159119" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }