{"cells":[{"metadata":{"_uuid":"dc24e70b28aaee5362966df3f7db2267e2728dab"},"cell_type":"markdown","source":"# Cats vs. Dogs: Transfer Learning with little data\n\nThis kernel is inspired by the Keras blog post [Building powerful image classification models using very little data](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) (by Francois Chollet, Sun 05 June 2016). **This blog post is a must read for everyone interested in transfer learning with ConvNets.**\n\nFollowing, we will learn how to\n\n1. reproduce the results of the blog post - using limited training data (2000 images) and a pretrained VGG16 model.\n2. compare the performance of VGG16 with other pretrained Convolutional Nets (InceptionV3 and ResNet50).\n\n# PART I\n\n**Reproduce results from [Building powerful image classification models using very little data](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html)**\n\n1. Training a small convnet from scratch (~80% accuracy)\n2. Using the bottleneck features of a pre-trained VGG-16 network (~90% accuracy)\n3. Fine-tuning the top layers of a a pre-trained VGG-16 network (~93% accuracy)"},{"metadata":{"_uuid":"26b15f3753fca535013618ce5e2d3ac66763dec2"},"cell_type":"markdown","source":"## Pre-requisites\n\n### Dataset Preparation\n\nThe setup in the [Keras blog post](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) uses only 2000 training examples (1000 per class) and 400 validation examples (200 per class) which are located in separate folders. Therefore, we create the same folder structure and copy 1000+400 cat and dog images each from the full dataset of 25,000 images."},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir ../data\n!mkdir ../data/dogs-vs-cats-redux-kernels-edition\n!unzip -qq ../input/dogs-vs-cats-redux-kernels-edition/train.zip -d ../data/dogs-vs-cats-redux-kernels-edition/\n!unzip -qq ../input/dogs-vs-cats-redux-kernels-edition/test.zip -d ../data/dogs-vs-cats-redux-kernels-edition/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3cfde42e87baed8db20830e9cf30c1f754cbaea9"},"cell_type":"code","source":"!mkdir -p ../data/train/cats ../data/train/dogs ../data/validation/cats ../data/validation/dogs\n!cp `ls ../data/dogs-vs-cats-redux-kernels-edition/train/cat* | grep '\\.[0-9]\\{1,3\\}\\.'` ../data/train/cats/  # copy first 1000 cats\n!cp `ls ../data/dogs-vs-cats-redux-kernels-edition/train/dog* | grep '\\.[0-9]\\{1,3\\}\\.'` ../data/train/dogs/  # copy first 1000 dogs\n!cp ../data/dogs-vs-cats-redux-kernels-edition/train/cat.1[0-3]??.jpg ../data/validation/cats/  # copy next 400 cats\n!cp ../data/dogs-vs-cats-redux-kernels-edition/train/dog.1[0-3]??.jpg ../data/validation/dogs/  # copy next 400 dogs\n!du ../data/ -h","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Show Example Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport matplotlib.pyplot as plt\nfrom keras.preprocessing.image import ImageDataGenerator, img_to_array, load_img\n%matplotlib inline\n\nfiles = [[path+f for f in os.listdir(path)[:10]] for path in [f'../data/train/{x}/' for x in ['cats', 'dogs']]]\n\nfig, axs = plt.subplots(4, 5, figsize=(15,15), subplot_kw={'xticks': [], 'yticks': []})\n\nfor ax, img in zip(axs.flatten(), [item for sublist in files for item in sublist]):\n    ax.imshow(load_img(img))\n    ax.set_title(img.split('/')[-1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84a67a471ff65b0da7cce32f49c6f8a7f6867dc0"},"cell_type":"markdown","source":"### Data Augmentation\n\nData augmentation is a common approach to prevent overfitting by creating variations of the original training images. The  `DataImageGenerator` is used to apply different transformations on the images (e.g. rotating, zooming, flipping, shifting the images)."},{"metadata":{"trusted":true,"_uuid":"b97ce615be97450a3417f20925ea6ffad71a96d9"},"cell_type":"code","source":"# define image transformations\ndatagen = ImageDataGenerator(\n        rotation_range=40,\n        width_shift_range=0.2,\n        height_shift_range=0.2,\n        shear_range=0.2,\n        zoom_range=0.2,\n        horizontal_flip=True,\n        fill_mode='nearest')\n\nimg_samples = 16\nimage_dir = '../preview/'\nif not os.path.exists(image_dir):\n    os.makedirs(image_dir)\n\nimg = load_img('../data/train/cats/cat.8.jpg')\nx = img_to_array(img)\nx = x.reshape((1,) + x.shape)\n\nfrom itertools import islice\nlist(islice(datagen.flow(x, batch_size=1, save_to_dir=image_dir, save_prefix='cat', save_format='jpeg'), img_samples));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Show Augmented Images"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"103f5abe9f96ec0fc9cd0d3c275cf498d44383b8"},"cell_type":"code","source":"rows, cols = 2, img_samples // 2\nfig, axs = plt.subplots(rows, cols, figsize=(16,3), subplot_kw={'xticks': [], 'yticks': []})\n\n#for i, img in enumerate(os.listdir(image_dir)[:img_samples]):\n#    axs[i//cols][i%cols].imshow(load_img(image_dir+img))\n\nfor ax, img in zip(axs.flatten(), os.listdir(image_dir)[:img_samples]):\n    ax.imshow(load_img(image_dir+img))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b33419fd5551ab39f56983ebe4ec77ccd1161e13"},"cell_type":"markdown","source":"## Training a Small ConvNet from Scratch\n\nA classic setup of three convolution layers with a ReLU activation, followed by max-pooling layers. On top two fully-connected layers and a signle node with sigmoid activation for binary classification (cat or dog)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport keras, tensorflow\nimport time\n\nprint('Keras', keras.__version__)\nprint('TensorFlow', tensorflow.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"948273eedbae93d74c02d3f6a1f86c4699589bb5"},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Input, Conv2D, MaxPooling2D\nfrom keras.layers import Activation, Dropout, Flatten, Dense\n\ninput_tensor = Input(shape=(150,150,3))  # input_shape for Theano should be (3, 150,150)\n\nmodel = Sequential()\nmodel.add(Conv2D(32, (3, 3), input_shape=(150,150,3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(32, (3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(64, (3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1, activation='sigmoid'))\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46239d016178899ff64b7639fd2eefe4c458557f"},"cell_type":"code","source":"model.compile(loss='binary_crossentropy',\n              optimizer='rmsprop',\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"45b24e61088fc1f12b2ab610f53435e7d61c10d8"},"cell_type":"markdown","source":"### Provide Training and Validation Image Using ImageDataGenerator"},{"metadata":{"trusted":true,"_uuid":"c7cd585e0d124b13d1be76d3468ca8b7af21bd2f"},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n\ntrain_dir      = '../data/train'\nvalidation_dir = '../data/validation'\ntest_dir = \"../data/test\"\n\ntrain_samples      = 2000\nvalidation_samples = 800\n\ntarget_size    = (150,150)  # all images will be resized to 150x150\nbatch_size     = 16\n\n# rescale and augment training data\ntrain_datagen = ImageDataGenerator(\n        rescale=1./255,\n        shear_range=0.2,\n        zoom_range=0.2,\n        horizontal_flip=True)\n\n# rescale validation data\nvalidation_datagen = ImageDataGenerator(\n    rescale=1./255)\n\ntrain_generator = train_datagen.flow_from_directory(\n        train_dir,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='binary')\n\nvalidation_generator = validation_datagen.flow_from_directory(\n        validation_dir,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='binary')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1b7bd957aa38a63d0e65ab18fe203b78c1f6416"},"cell_type":"markdown","source":"Ready for training. ~16s per epoch on GPU, ~60s per epoch on CPU."},{"metadata":{"trusted":true,"_uuid":"4eb5aedbd5430016cce312962c3c471cc990a102"},"cell_type":"code","source":"epochs = 30\n\nstart_time = time.time()\nhistory_simple = model.fit_generator(\n    train_generator,\n    steps_per_epoch=train_samples // batch_size,\n    epochs=epochs,\n    verbose=2,\n    validation_data=validation_generator,\n    validation_steps=validation_samples // batch_size)\nprint(\"--- took %d:%.2d minutes ---\" % divmod(time.time() - start_time, 60))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b9f4162aa097211cb0d68dcf85aee00aa6fd342"},"cell_type":"markdown","source":"### Investigating Model Accuracy\n\nThe results can vary due to the random initialization. But the (validation) accuracy should be more or less close to 80% (like in the original blog post). Looking at the graphs we ses that the training accuracy continuously improves while the validation accuracy does not change much after ~15 epochs - a clear sign for overfitting."},{"metadata":{"trusted":true,"_uuid":"f5a578a0a8479ce56e54a1dd231d0f9bd0562af0"},"cell_type":"code","source":"def plot_history(history, acc_line=None, title=None, acc_lim=[0.5,1.0]):\n    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16,5))\n    if title:\n        fig.suptitle(title, fontsize=16)\n    \n    ax1.plot(history.history['acc'])\n    ax1.plot(history.history['val_acc'])\n    if acc_line:\n        ax1.axhline(y=acc_line, linewidth=2, linestyle='dashed', color='lightgrey')\n    ax1.set_title('Model Accuracy')\n    ax1.set_ylabel('Accuracy')\n    ax1.set_xlabel('Epoch')\n    #ax1.set_yticks(np.arange(0., 1.1, .1))\n    ax1.set_ylim(acc_lim)\n    ax1.legend(['Train', 'Test'])\n    ax1.grid(b=True, which='major', color='lightgrey', linestyle='dotted')\n    \n    ax2.plot(history.history['loss'])\n    ax2.plot(history.history['val_loss'])\n    ax2.set_title('Model Loss')\n    ax2.set_ylabel('Loss')\n    ax2.set_xlabel('Epoch')\n    ax2.set_ylim([0, 1.2])\n    ax2.legend(['Train', 'Test'])\n    ax2.grid(b=True, which='major', color='lightgrey', linestyle='dotted')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5a578a0a8479ce56e54a1dd231d0f9bd0562af0"},"cell_type":"code","source":"plot_history(history_simple, acc_line=0.8, title='Simple ConvNet')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a5ab5e89ee39ec79e8c3efd478f31581363eb25"},"cell_type":"markdown","source":"## Using bottleneck features of a pre-trained VGG16 network\n\nAs in the blog post we will be using VGG16 with pre-trained weights based on ImageNet divided into two steps:\n\n  * compute bottleneck features for all 2000 training images and 800 validation images of cats and dogs\n  * create new binary classifier and train it with bottleneck features.\n\n**Note: there is no data augmentation applied - so expect to see more overfitting.**"},{"metadata":{"_uuid":"d97ea91c86026f76eb52e10709d4b243d4224cbf"},"cell_type":"markdown","source":"### Create VGG16 model with pre-trained weights\n\n**Important**: Exclude the top layers (the dense classfication layers) because we are only interested in the output of the convolutional layers."},{"metadata":{"trusted":true,"_uuid":"5242dbcc524b5fb1dc538d9bdfca58def586e14f"},"cell_type":"code","source":"from keras.applications.vgg16 import VGG16\n\nweights_vgg16 = '../input/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5'\n\nbase_model_vgg16 = VGG16(weights=weights_vgg16, include_top=False, input_shape=(150,150,3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a5ab5e89ee39ec79e8c3efd478f31581363eb25"},"cell_type":"markdown","source":"### Extract Bottleneck Features\n\nSee also the [code referenced in Keras blog post](https://gist.github.com/fchollet/f35fbc80e066a49d65f1688a7e99f069)."},{"metadata":{"trusted":true,"_uuid":"b50366b0d9583c315049e43df0709578a980b55f"},"cell_type":"code","source":"def get_bottleneck_features(model, image_dir, target_size, samples, batch_size=16):\n    datagen   = ImageDataGenerator(rescale=1. / 255)\n    generator = datagen.flow_from_directory(image_dir,\n                                            target_size=target_size,\n                                            batch_size=batch_size,\n                                            class_mode=None,\n                                            shuffle=False)\n    return model.predict_generator(generator, samples // batch_size)\n\ntrain_data_vgg16 = get_bottleneck_features(base_model_vgg16, train_dir, target_size, train_samples, batch_size)\nprint('created bottleneck features for training:', train_data_vgg16.shape)\n\nvalidation_data_vgg16 = get_bottleneck_features(base_model_vgg16, validation_dir, target_size, validation_samples, batch_size)\nprint('created bottleneck features for validation:', validation_data_vgg16.shape)\n\ntrain_labels = np.array([0] * (train_samples // 2) + [1] * (train_samples // 2))\nvalidation_labels = np.array([0] * (validation_samples // 2) + [1] * (validation_samples // 2))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"48f22e8e38ee5e3ce8b65ecfbb2b6484eff80d96"},"cell_type":"markdown","source":"### Train a Classifier with Bottleneck Features\n\nClassifier Setup: A dense layer with 256 neurons plus Dropout of 0.5 with a final layer of one neuron."},{"metadata":{"trusted":true,"_uuid":"bf857a78b9e24ae569f840f139b2aa001ed88bdd"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Input, Flatten, Dense, Dropout\n\ndef get_top_model(input_shape):\n    input = Input(input_shape)\n    x = Flatten()(input)\n    x = Dense(256, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    prediction = Dense(1, activation='sigmoid')(x)\n    return Model(inputs=input, outputs=prediction)\n\ntop_model_vgg16 = get_top_model(train_data_vgg16.shape[1:])\n\ntop_model_vgg16.compile(loss='binary_crossentropy',\n                        optimizer='rmsprop',\n                        metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e2e87e5f9d1d48986baa208da1342d800125d50e"},"cell_type":"code","source":"epochs = 30\n\nstart_time = time.time()\nhistory_vgg16_top = top_model_vgg16.fit(train_data_vgg16, train_labels,\n                                        verbose=2,\n                                        epochs=epochs,\n                                        batch_size=batch_size,\n                                        validation_data=(validation_data_vgg16, validation_labels))\nprint(\"--- took %d:%.2d minutes ---\" % divmod(time.time() - start_time, 60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e2e87e5f9d1d48986baa208da1342d800125d50e"},"cell_type":"code","source":"model_weigths_file = 'bottleneck_fc_model_vgg16.h5'\ntop_model_vgg16.save_weights(model_weigths_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"996ab432343dbf7eb32797b11f87a20703f9e3cf"},"cell_type":"code","source":"plot_history(history_vgg16_top, acc_line=0.9, title='Pre-trained VGG16 with no augmentation', acc_lim=[0.75,1.0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5532bddd4ef26c3eb01208046b2a9fbcb86ca89"},"cell_type":"markdown","source":"## Fine-tuning the top layers of a pre-trained network\n\n**CAUTION:** The [original code](https://gist.github.com/fchollet/7eb39b44eb9e16e59632d25fb3119975) in the blog article using `base_model.add()` to join the pre-trained VGG16 model with the pre-trained classifier does not work (anymore). Instead, a new `Sequential` model has to be created. (see also https://github.com/keras-team/keras/issues/7338#issuecomment-315973050)."},{"metadata":{"trusted":true,"_uuid":"510162470a23c4d290ab5a948f901b5d9838bc00"},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Input, Flatten, Dense, Dropout","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load pre-trained VGG16 network (without classfication layers)\nbase_model_vgg16 = VGG16(weights=weights_vgg16, include_top=False, input_shape=(150,150,3))\nprint('Model loaded.')\n\n# set all but the last the last conv block to non-trainable (weights will not be updated)\nfor layer in base_model_vgg16.layers[:15]:\n    layer.trainable = False\n\n# create a classifier model to put on top of the convolutional model\ntop_model = Sequential()\ntop_model.add(Flatten(input_shape=base_model_vgg16.output_shape[1:]))\ntop_model.add(Dense(256, activation='relu'))\ntop_model.add(Dropout(0.5))\ntop_model.add(Dense(1, activation='sigmoid'))\ntop_model.load_weights(model_weigths_file)\n\n# add the model on top of the convolutional base\nmodel = Model(inputs = base_model_vgg16.input, outputs = top_model(base_model_vgg16.output))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import optimizers\n\n# compile model with a very slow learning rate\nmodel.compile(loss='binary_crossentropy',\n              #optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),\n              #optimizer=optimizers.RMSprop(lr=1e-5),\n              optimizer=optimizers.Adam(lr=1e-5),\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 30\n\nstart_time = time.time()\nhistory_tuned = model.fit_generator(\n    train_generator,\n    steps_per_epoch=train_samples // batch_size,\n    epochs=epochs,\n    verbose=2,\n    validation_data=validation_generator,\n    validation_steps=validation_samples // batch_size)\nprint(\"--- took %d:%.2d minutes ---\" % divmod(time.time() - start_time, 60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history_tuned, acc_line=0.93, title='Pre-trained VGG16 + pre-trained top model', acc_lim=[0.75,1.0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PART II\n\n**Compare different pretrained convolutional networks.**\n\n1. VGG-16 (~90% accuracy)\n2. Inception v3  (~95% accuracy)\n3. ResNet-50  (~98% accuracy)"},{"metadata":{"_uuid":"4289c9573499863153d3353897f42205e5339ca5"},"cell_type":"markdown","source":"### Create VGG16 model with pre-trained weights"},{"metadata":{"trusted":true,"_uuid":"0d852fbd759bf22717df745b7138aecec6bce260"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Flatten, Dense, Dropout\n\n# build the VGG16 network\nbase_model = VGG16(weights=weights_vgg16, include_top=False, input_shape=(150,150,3))\n\n# freeze convolutional layers\nfor layer in base_model.layers:\n    layer.trainable = False\n\nx = Flatten()(base_model.output)\nx = Dense(256, activation='relu')(x)\nx = Dropout(0.5)(x)\npredictions = Dense(1, activation='sigmoid')(x)\n\nmodel_vgg16base = Model(inputs=base_model.input, outputs=predictions)\n\nprint(model_vgg16base.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"513bb685f02fa434141767e043bfe377496baede"},"cell_type":"code","source":"model_vgg16base.compile(loss='binary_crossentropy',\n                        optimizer='rmsprop',\n                        metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4a6440ea3381ae2d7f7c343b7eb128ff409fbb91"},"cell_type":"code","source":"epochs = 30\n\nstart_time = time.time()\nhistory_vgg16base = model_vgg16base.fit_generator(\n    train_generator,\n    steps_per_epoch=train_samples // batch_size,\n    epochs=epochs,\n    validation_data=validation_generator,\n    validation_steps=validation_samples // batch_size)\nprint(\"--- took %d:%.2d minutes ---\" % divmod(time.time() - start_time, 60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"62325f061ad5ba8da49450cafe7333823889b4fd"},"cell_type":"code","source":"plot_history(history_vgg16base, acc_line=0.9, title='Pre-trained VGG16', acc_lim=[0.75, 1.0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a5ab5e89ee39ec79e8c3efd478f31581363eb25"},"cell_type":"markdown","source":"### Use Pre-trained InceptionV3 Model"},{"metadata":{"trusted":true,"_uuid":"dbc10f9f806a73bec2ed4beb3ffd676c5286b7d3"},"cell_type":"code","source":"from keras.applications.inception_v3 import InceptionV3\n\nweights_incv3 = '../input/inceptionv3/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5'\n\n# load pre-trained weights and add global average pooling layer\nbase_model_incv3 = InceptionV3(weights=weights_incv3, input_shape=(150,150,3), include_top=False, pooling='avg')\n\n# freeze convolutional layers\nfor layer in base_model_incv3.layers:\n    layer.trainable = False\n\n# define classification layers\n#x = Dense(1024, activation='relu')(base_model_incv3.output)\n#predictions = Dense(1, activation='sigmoid')(x)\nx = Dense(256, activation='relu')(base_model_incv3.output)\nx = Dropout(0.5)(x)\npredictions = Dense(1, activation='sigmoid')(x)\n\nmodel_incv3 = Model(inputs=base_model_incv3.input, outputs=predictions)\n#print(model_incv3.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70ef5db242608d4a7e68159bdc0535c5eed8e875"},"cell_type":"markdown","source":"### Prepare Model Training Setup"},{"metadata":{"trusted":true,"_uuid":"94820c6bd43b86671b4b8244ad3f20b840fba957"},"cell_type":"code","source":"model_incv3.compile(loss='binary_crossentropy',\n                    optimizer=optimizers.RMSprop(lr=0.0001),\n                    metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f277f7a940c1915a1085cd64d71aefdad0fa603d"},"cell_type":"markdown","source":"### Provide Training and Validation Images using ImageDataGenerator\n\n**Caution**: InceptionV3 requires a different image preprocessing than VGG."},{"metadata":{"trusted":true,"_uuid":"95ad61566c677bc4d9c615ba937e738e1a7789a4"},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications.inception_v3 import preprocess_input\n\ndef prep(image):\n    # copy image to prevent overwriting\n    return preprocess_input(image.copy())\n\ntrain_datagen = ImageDataGenerator(\n        shear_range=0.2,\n        zoom_range=0.2,\n        horizontal_flip=True,\n        preprocessing_function=prep)\n\ntest_datagen = ImageDataGenerator(preprocessing_function=prep)\n\ntrain_generator = train_datagen.flow_from_directory(\n        train_dir,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='binary')\n\nvalidation_generator = test_datagen.flow_from_directory(\n        validation_dir,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='binary')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea8cda5011697aba7c8391946fdd07833a427686","scrolled":false},"cell_type":"code","source":"epochs = 30\n\nstart_time = time.time()\nhistory_incv3 = model_incv3.fit_generator(\n    train_generator,\n    steps_per_epoch=train_samples // batch_size,\n    epochs=epochs,\n    validation_data=validation_generator,\n    validation_steps=validation_samples // batch_size)\nprint(\"--- took %d:%.2d minutes ---\" % divmod(time.time() - start_time, 60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history_incv3, acc_line=0.95, title=\"Pre-trained InceptionV3\", acc_lim=[0.75,1.0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ResNet50"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications.resnet50 import ResNet50, preprocess_input\n\nweights_resnet50 = '../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5'\n\nbase_model = ResNet50(weights = weights_resnet50, include_top = False, pooling = 'avg')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(base_model)\n\nmodel.add(Dense(256, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1, activation='sigmoid'))\n\nmodel.layers[0].trainable = False\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss='binary_crossentropy',\n                    optimizer=optimizers.RMSprop(lr=0.0001),\n                    metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prep(image):\n    # copy image to prevent overwriting\n    return preprocess_input(image.copy())\n\ntrain_datagen = ImageDataGenerator(\n        shear_range=0.2,\n        zoom_range=0.2,\n        horizontal_flip=True,\n        preprocessing_function=prep)\n\ntest_datagen = ImageDataGenerator(preprocessing_function=prep)\n\ntrain_generator = train_datagen.flow_from_directory(\n        train_dir,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='binary')\n\nvalidation_generator = test_datagen.flow_from_directory(\n        validation_dir,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='binary')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 30\n\nstart_time = time.time()\nhistory_resnet50 = model.fit_generator(\n    train_generator,\n    steps_per_epoch=train_samples // batch_size,\n    epochs=epochs,\n    validation_data=validation_generator,\n    validation_steps=validation_samples // batch_size)\nprint(\"--- took %d:%.2d minutes ---\" % divmod(time.time() - start_time, 60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history_resnet50, acc_line=0.97, title=\"Pre-trained ResNet-50\", acc_lim=[0.85,1.0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir ../output/\n!ls ../output/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save(\"model_resnet50.h5\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nsample_submission = pd.read_csv(\"../input/dogs-vs-cats-redux-kernels-edition/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nsample_submission = pd.read_csv(\"../input/dogs-vs-cats-redux-kernels-edition/sample_submission.csv\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}