{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input/\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"246ca01e4609f59b0886ffaa654c8e9404dc24dc"},"cell_type":"code","source":"from os import listdir\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f38e76331204d648cbc267ff14c95cf139d2d4d"},"cell_type":"markdown","source":"The rule is \nSubmissions are scored on the log loss:\n\n\n# $$ loss(y, \\hat{y}) = -\\frac{1}{n}  \\sum_{i=1}^{n}{[ y_i log(\\hat{y_i}) + (1-y_i)  log(1-\\hat{y_i}})] $$\nwhere\n\n* n is the number of images in the test set\n* $y^i$ is the predicted probability of the image being a dog\n* $y_i$ is 1 if the image is a dog, 0 if cat\n* log() is the natural (base e) logarithm\nA smaller log loss is better.\n\n## Splitting data.\n\nFirst, I am going to split the data in train set and test set. Notice that the train set will be late split again for validation set. \n\nRight now, I will use 1000 images for testing, and the rest for training/validation"},{"metadata":{"trusted":true,"_uuid":"f25812d88c6a404effae3813d0b0481c662abbb6"},"cell_type":"code","source":"import random\ntrain_data = []  # This will later be split in validation too\ntest_data = []\nfor file in listdir(\"../input/train\"):\n    some_number = random.randint(1,100)\n    label = \"1\" if \"dog\" in file else \"0\" \n    if len(test_data) >= 1000 or some_number < 85:\n        train_data.append([file, label])\n    else:\n        test_data.append([file, label])\n        \ntrain = pd.DataFrame(train_data, columns=[\"filename\", \"class\"])\ntest = pd.DataFrame(test_data, columns = [\"filename\", \"class\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f65554fb228a9b3c362f2d9de3cab1b142c00a6a"},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec90f6ed68db7ee83820d52c293d72ca07ce845e"},"cell_type":"code","source":"test.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"542a90fe1670f71e9117c42b52b77b4d2ee1653b"},"cell_type":"code","source":"print(\"Train size\", len(train))\nprint(\"Test size\", len(test))\n\nfor label in [\"0\", \"1\"]:\n    print(\"------------\")\n    print(\"\\tTrain has\", len(train[train[\"class\"]==label]), label)\n    print(\"\\tTest has\", len(test[test[\"class\"]==label]), label)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e526b4dbe2241e0d54801c4fb453f395cb36c818"},"cell_type":"markdown","source":"## The data is quite balanced, ~50% are dogs (class 1), ~50% are cats (class 0). As this is a binary problem, we can output a sigmoid as the output function"},{"metadata":{"trusted":true,"_uuid":"4a2792c3438ca4a460c0fe3131336e83ab9bdfdc"},"cell_type":"code","source":"IMAGE_WIDTH = 224\nIMAGE_HEIGHT = 224\nBATCH_SIZE=32\ntrain_image_generator = ImageDataGenerator(rescale=1./255, \n                                           rotation_range=90, \n                                           horizontal_flip=True, \n                                           vertical_flip=True,\n                                           validation_split=0.15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41ec370eb467f20f43d396f10859534622610eba"},"cell_type":"code","source":"train_generator = train_image_generator.flow_from_dataframe(train, \"../input/train\", seed=42,\n                                                    target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),\n                                                    batch_size=BATCH_SIZE,\n                                                    class_mode=\"binary\",\n                                                    subset=\"training\",\n                                                    shuffle=True,      \n                                                    save_format=\"jpeg\")\n\nvalidation_generator = train_image_generator.flow_from_dataframe(train, \"../input/train\", seed=42,\n                                                    target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),\n                                                    batch_size=BATCH_SIZE,\n                                                    class_mode=\"binary\",\n                                                    subset=\"validation\",\n                                                    shuffle=False,                  \n                                                    save_format=\"jpeg\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"765f29c5b01ecdc61dd8178720e06fb3567d08a3"},"cell_type":"code","source":"from keras.applications import vgg16\nmodel = vgg16.VGG16(weights='imagenet', include_top=False, input_shape=(IMAGE_WIDTH, IMAGE_HEIGHT, 3), pooling=\"max\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6aae5d8c6c7e389331780d346730c7a7c5e55ae3"},"cell_type":"markdown","source":"### Now, we are going to only train the last 5 layers."},{"metadata":{"trusted":true,"_uuid":"00de3a89c67179ad12142291ccbc165335562a2c"},"cell_type":"code","source":"for layer in model.layers[:-5]:\n        layer.trainable = False","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d80ea21da4f5ccc8fe7e53d8c06adaaa7ad60ac"},"cell_type":"markdown","source":"## Finally, we are going to add a Dense layer of 512 units and then the output layer (a sigmoid function) at the end."},{"metadata":{"trusted":true,"_uuid":"72bec807f48188c3fa3eb7f325cded4cb6442fbb"},"cell_type":"code","source":"from keras.layers import Dense, GlobalAveragePooling2D, Dropout, BatchNormalization\nfrom keras.models import Model, Sequential\n\n# Although this part can be done also with the functional API, I found that for this simple models, this becomes more intuitive\ntransfer_model_vgg16 = Sequential()\nfor layer in model.layers:\n    transfer_model_vgg16.add(layer)\ntransfer_model_vgg16.add(Dense(512, activation=\"relu\"))  # Very important to use relu as activation function, search for \"vanishing gradiends\" :)\ntransfer_model_vgg16.add(Dense(1, activation=\"sigmoid\")) # Finally our activation layer! we use 2 outputs as we have either cats or dogs","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e793aeebac424218d44c2ccc1c3a2038f2186604"},"cell_type":"markdown","source":"## Lets display our model "},{"metadata":{"trusted":true,"_uuid":"ee72277ccf19ed97b6b964a8345c38e091f49f40"},"cell_type":"code","source":"from IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\n\nSVG(model_to_dot(transfer_model_vgg16).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9bb041d20dd1725a61441b38e978fb3722c582a3"},"cell_type":"code","source":"from keras import optimizers\nadam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.00001)\n\ntransfer_model_vgg16.compile(adam, \n                       loss=\"binary_crossentropy\",\n                      metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7434e621ad7546d04ae5dbb1a7cb4415611cd67a"},"cell_type":"code","source":"vgg16_model_history = transfer_model_vgg16.fit_generator(train_generator, \n                                             steps_per_epoch = train_generator.n // BATCH_SIZE,\n                                             validation_data = validation_generator,\n                                             validation_steps = validation_generator.n // BATCH_SIZE,\n                                            epochs=7)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d5745e1c0003714ee5c5027b2c51ce4ba4cc8e9"},"cell_type":"markdown","source":"## Lets define a small function to plot our predictions"},{"metadata":{"trusted":true,"_uuid":"ffec75d79e7f92d1a308291779f2102e811629f4"},"cell_type":"code","source":"from IPython.display import Image, display\ndef plot_prediction(image_path, label):\n    display(Image(filename=image_path, width=IMAGE_WIDTH, height=IMAGE_HEIGHT))\n    prediction = \"dog\"\n    confidence = label\n    if label < 0.5:\n        prediction = \"cat\"\n        confidence = (1-label)\n    legend = \"The image %s above is a %s with a confidence of %.2f%% %f\" % (image_path, prediction, confidence*100, label)\n    print(legend)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"202169372755992b3b1ac8766304f1b98b84d777"},"cell_type":"markdown","source":"## And another function to efficiently yield batches of images and (optionally) labels to predict"},{"metadata":{"trusted":true,"_uuid":"ad6167ea73442630ecdff4668cbbf00cf554afd9"},"cell_type":"code","source":"import cv2\nfrom skimage import io\n\ndef build_batches(df, has_labels=True, limit=500, batch_size=BATCH_SIZE, produce=\"images\"):\n    \"\"\"\n    produce: Can be either \"images\" in which case an array of normalized images is returned or \n             \"paths\" in which case, a string with the full dir is returned\n    \"\"\"\n    X = []\n    y = []\n    paths = []\n    i = 0\n    for _, row in df.iterrows():\n        if has_labels:\n            y.append(row[\"class\"])\n        raw_image_path = \"../input/train/\" if has_labels else \"../input/test/\"\n        raw_image_path += row[\"filename\"]\n        raw_image = io.imread(raw_image_path)\n        raw_image = cv2.resize(raw_image, (IMAGE_WIDTH, IMAGE_HEIGHT), interpolation=cv2.INTER_CUBIC)\n        X.append(raw_image)\n        paths.append(raw_image_path)\n        i += 1\n        if i == limit:\n            break\n        if i > 0 and i % batch_size == 0:\n            X = np.array(X)\n            y = np.array(y)\n            X = X / 255\n            \n            if produce == \"images\":\n                yield X, y\n            else:\n                yield paths, y\n            paths = []\n            X = []\n            y = []\n\n    X = np.array(X)\n    y = np.array(y)\n    \n    X = X / 255\n    \n    if produce == \"images\":\n        yield X, y         \n    else:\n        yield paths, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64801f354b7ac3fc287726c92c054641cc0f8e8f"},"cell_type":"code","source":"samples = 1000\ntransfer_model_vgg16.evaluate_generator(build_batches(test, limit=samples), steps=samples/BATCH_SIZE, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c3ba32de8c63dbcb0329b8fe52edde90a136822"},"cell_type":"markdown","source":"### Not a bad result, lets plot a couple of those images..."},{"metadata":{"trusted":true,"_uuid":"3ba0a06bc7fc5166dbc66798358a9730a775727b"},"cell_type":"code","source":"some_predictions = transfer_model_vgg16.predict_generator(build_batches(test, limit=12, batch_size=1), steps=12, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fcc6c5d001fe8a94d2765048354622b62f1bf4ee"},"cell_type":"code","source":"idx = 0\nfor mini_batch_files, mini_batch_labels in build_batches(test, limit=samples, batch_size=1, produce=\"paths\"):\n    mini_batch_file = mini_batch_files[0]\n    mini_batch_label = mini_batch_labels[0]\n    predicted_label = some_predictions[idx][0]\n    idx += 1\n    #print(mini_batch_file, mini_batch_label, predicted_label)\n    plot_prediction(mini_batch_file, predicted_label)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f1c702b8b8261ab8b3b8282e0561121d00d21bf"},"cell_type":"markdown","source":"### Now, it would be interesting to plot images that are NOT correctly predicted... lets do that too."},{"metadata":{"trusted":true,"_uuid":"51e05c72bde4e985d1d5746a43596d5a7881ab4f"},"cell_type":"code","source":"samples = 1000\nsome_predictions = transfer_model_vgg16.predict_generator(build_batches(test, limit=samples, batch_size=1), steps=samples, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"378ff3639ceb417f8da0788d7ccc1ccb91a8606a","scrolled":false},"cell_type":"code","source":"\nprint(\"Total predictions\", some_predictions.shape)\nidx = 0\nerrors = 0\nfor mini_batch_files, mini_batch_labels in build_batches(test, limit=samples, batch_size=1, produce=\"paths\"):\n    mini_batch_file = mini_batch_files[0]\n    mini_batch_label = mini_batch_labels[0]\n    predicted_label = some_predictions[idx][0]\n    if abs(float(mini_batch_label) - float(predicted_label)) > 0.5:\n        errors += 1\n        if errors < 10:\n            plot_prediction(mini_batch_file, predicted_label)\n    idx += 1\nprint(\"Total errors...\", errors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f87bd9d9d828dc53d8481a5696c778298d5606f0"},"cell_type":"code","source":"my_limit = 12500\ni = 0\noutput_df = []\nfor file in listdir(\"../input/test/\"):    \n    output_df.append([file, file.split(\".\")[0]])\n    i += 1\n    if i == my_limit:\n        break\noutput = pd.DataFrame(output_df, columns=[\"filename\", \"id\"])\nprint(len(output))\noutput.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d2a6b49ce2bb2ca1c9d7fd54780f87388f185d4"},"cell_type":"code","source":"results = transfer_model_vgg16.predict_generator(build_batches(output, limit=-1, has_labels=False, batch_size=64), steps=12500/64, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"baa5db0fb27d69634e7f8a8486ba135ca231708b"},"cell_type":"code","source":"results.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2d7183e441b8c3d7c0a6ce710107f277af1de1e"},"cell_type":"code","source":"output[\"label\"] = results\n\noutput.head(15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff5dd5dce9cdfb854a539d2f3c063c0f15b3afcf"},"cell_type":"markdown","source":"## Lets plot a couple of predictions..."},{"metadata":{"trusted":true,"_uuid":"30b477f22327462dfc8795e8e560b27e58b610a9"},"cell_type":"code","source":"stop = 10\nfor idx, row in output.iterrows():\n    path = \"../input/test/\" + row[\"id\"] + \".jpg\"\n    plot_prediction(path, row[\"label\"])\n    stop -= 1\n    if stop == 0:\n        break","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09e600d13c1341a3db00f20fb142b74ae5c5c6b6"},"cell_type":"markdown","source":"## And finally prepare the output file"},{"metadata":{"trusted":true,"_uuid":"24c6183f4a32f66c55e7e51f1ec0a29fed31a490"},"cell_type":"code","source":"del output[\"filename\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"80d64a408889a8443f377c91ab4994a4188de049"},"cell_type":"code","source":"output.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c268c3aa0f2201cc58c0f27628f4caaa78a390c"},"cell_type":"code","source":"len(output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b99fd902bc3bad486e509862439b2686a23fcf9"},"cell_type":"code","source":"output.to_csv(\"submission_file.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}