{"cells":[{"metadata":{"_uuid":"cbd825e7cda03ad1826a8062ef7ede75df258a6c"},"cell_type":"markdown","source":"# Working with pretrained models\n\nThis notebook outlines how, and why, to work with pretrained models for image classification.\n\n## What are pretrained models\nPretrained models are models that where built and trained by others, usually for a different but related task. This process is called **transfer learning**. Transfer learning is especially popular in computer vision, but is gaining popularity in pretty much all other fields of machine learning, too. \n\nThe most popular benchmark for computer vision models is called [ImageNet](http://www.image-net.org/), a collection of millions of images showing 1000 different objects. There is an annual challenge to build a model that performs best on classifying ImageNet images and many useful models have come out of it. In this case, we will use the the [Xception model](https://arxiv.org/abs/1610.02357) as a basis from which we will build a cat and dog classifier.\n\n## Some preparation:\nWe will now copy the pretrained model in the right location to load it with Keras. This is only nessecary since Kaggle Kernels have no internet connection, otherwise Keras download the models automatically.","outputs":[],"execution_count":null},{"metadata":{"_cell_guid":"3efff49a-b436-4d21-98a7-9faf7befc1b3","_uuid":"46fa8c35fd5bb7ea0eb6413b7cd2382b25c2df4f","trusted":true},"cell_type":"code","source":"# Check that we have access to the models:\n!ls ../input/keras-pretrained-models","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d5832df348c5cacbe6cf88e09dd83bfd063e055","_cell_guid":"7acedaf0-1439-418c-8428-5a0ed29cd085","collapsed":true,"trusted":true},"cell_type":"code","source":"# Create paths for model\nimport os\ncache_dir = os.path.expanduser(os.path.join('~', '.keras'))\nif not os.path.exists(cache_dir):\n    os.makedirs(cache_dir)\nmodels_dir = os.path.join(cache_dir, 'models')\nif not os.path.exists(models_dir):\n    os.makedirs(models_dir)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f76985e0dea85f2c2a0a96b83c34217c81762b91","_cell_guid":"4ca2470f-4008-43c0-b85d-2115788c1a10","collapsed":true,"trusted":true},"cell_type":"code","source":"# Copy model over\n!cp ../input/keras-pretrained-models/xception* ~/.keras/models/","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5e041136-9522-4ca1-8131-23f8663f061a","_uuid":"88e7cd8bcbe61e5db3c241194bb807f49989cbdc","trusted":true},"cell_type":"code","source":"# Check that model is in place\n!ls ~/.keras/models","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8228f8a65518d1220562dffa9a8df0c4dd16dc14"},"cell_type":"markdown","source":"## Preparing the data\nIn this section we will copy and sort the data, just as we did when we built our image classifier from scratch. You can find the kernel explaining these steps here: https://www.kaggle.com/jannesklaas/14-building-an-image-classifier","outputs":[],"execution_count":null},{"metadata":{"_uuid":"20bdaca39a92f29a656a6ca19b022581e3280b40","_cell_guid":"78c7ae45-7066-45d7-8ef7-3eed2344a539","collapsed":true,"trusted":true},"cell_type":"code","source":"# Import matplotlib for plotting\nimport matplotlib.pyplot as plt\n# Import matplotlibs image tool\nimport matplotlib.image as mpimg\n# Flip the switch to get easier matplotlib rendering\n%matplotlib inline\n# for file listing\nimport os\n# for file moving\nfrom shutil import copyfile","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9ad7fc21-99b9-42ed-a81f-6ced236e35e7","_uuid":"dda7be034d00bcd49f1cce9d55793dbcd8309b3c","trusted":true},"cell_type":"code","source":"# Create destination directories\nif not os.path.exists('train'):\n    os.mkdir('train')\nif not os.path.exists('train/cat'):\n    os.mkdir('train/cat')\nif not os.path.exists('train/dog'):  \n    os.mkdir('train/dog')\nif not os.path.exists('validation'):    \n    os.mkdir('validation')\nif not os.path.exists('validation/cat'):\n    os.mkdir('validation/cat')\nif not os.path.exists('validation/dog'):\n    os.mkdir('validation/dog')\n# define paths\nsource_path = '../input/dogs-vs-cats-redux-kernels-edition/train/'\n\ncat_train_path = 'train/cat'\ndog_train_path = 'train/dog'\n\ncat_validation_path = 'validation/cat'\ndog_validation_path = 'validation/dog'\n# Loop over image numbering\nfor i in range(110):\n    cat = 'cat.' + str(i) + '.jpg'\n    dog = 'dog.' + str(i) + '.jpg'\n    # Get source paths\n    cat_source = os.path.join(source_path,cat)\n    dog_source = os.path.join(source_path,dog)\n    # Get destination paths\n    if i < 100:\n        cat_dest = os.path.join(cat_train_path,cat)\n        dog_dest = os.path.join(dog_train_path,dog)\n    else: \n        cat_dest = os.path.join(cat_validation_path,cat)\n        dog_dest = os.path.join(dog_validation_path,dog)\n    # Move file\n    copyfile(cat_source,cat_dest)\n    copyfile(dog_source,dog_dest)\n    print('Copied',(i+1)*2,'out of 220 files',end='\\r')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0f18f95e-fd9f-4bd9-a7d3-adab6bff46f2","_uuid":"23dcc98ade420208d6764fc7f0f69f8cd46a2531","trusted":true},"cell_type":"code","source":"# Check that images are in position\nimg=mpimg.imread('train/cat/cat.1.jpg')\nimgplot = plt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ec723b2eb1e61d748511ec4b277b69d8ad47e2a"},"cell_type":"markdown","source":"## Load the pretrained model\nIn this section we will load the pretrained model. The [Keras Applications](https://keras.io/applications/) model provides a range of useful pretrained model, but we will work with Xception in this case.\n\nModels trained on ImageNet usually have a structure consisting of a 'base' made out of convolutional layers which extract features and a 'top' made out of densely connected layers which do the classification. We want to use the feature extraction capabilities of our model but completely retrain the classification part of the model. Luckily, Keras offers us the option to load just the base of the model. You might want to experiement with retraining different layers though. For example it might be worth trying to retrain the last convolutional layer as well.","outputs":[],"execution_count":null},{"metadata":{"_cell_guid":"83a32863-bbb1-459e-b851-f1b97ac30e4c","_uuid":"94586550a8c96ae6d2da2ae207df06d7c7ed17c3","trusted":true},"cell_type":"code","source":"from keras.applications import Xception\nfrom keras.models import Sequential","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"aa4ada68-45da-4cb3-b49d-8aaa1ed49125","_uuid":"b5f66f12bb554a3e7713d09a1bbc760fad7c4bc1","trusted":true},"cell_type":"code","source":"model = Xception(weights='imagenet', include_top=False)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80c2f8656f958415aaad7f8c0e13cc275bc8f27e"},"cell_type":"markdown","source":"As you can see, the model is pretty large and contains some layer types you might not know. But don't worry, you don't have to understand how and why those bigger models work to use them.\n\n## Extracting bottleneck features\nWe will now extract some bottleneck features. Bottleneck features are the outputs of the last convolutional layer. We do this by running all of our training and validation images through the model, creating a new dataset which we will later use to train our new top.\n\nTo limit runtime, this kernel uses only 200 training and 20 validation images. The generators load images in order, so we know the labels and can create a label vector easily (the first 100 images are cats, the second 100 images are dogs).  To save runtime we also don't do much image augumentation besides rescaling.","outputs":[],"execution_count":null},{"metadata":{"_uuid":"8f38436019ef76fff0741830312dd9666f470491","_cell_guid":"1f225209-4f45-4a13-973c-b630da5cd614","collapsed":true,"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n\n# Only rescaling for training too this time\ndatagen = ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f1fa8b76f54e16ffb3e83b8367b41f2ec8b48358","_cell_guid":"340ed496-d9f1-4a2f-9e45-070e3d85fa43","collapsed":true,"trusted":true},"cell_type":"code","source":"# only rescaling\nvalidation_datagen = ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"09f20677-a0a7-4a14-a80c-a19d76773434","_uuid":"c422ba15b308cfbbba1bcdee95a582993ee4b3eb","trusted":true},"cell_type":"code","source":"# Set up batch size\nbatch_size = 1\ntrain_generator = datagen.flow_from_directory(\n        'train',  # this is the target directory\n        target_size=(150, 150),  # all images will be resized to 150x150\n        batch_size=batch_size, # How many images do we need at a time\n        class_mode=None, # Generator will yield data without labels\n        shuffle= False) # Generator will read files in order","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0db3d9b3-84c2-41cb-8db8-020e953838b5","_uuid":"bf7a6f137410f6992148cb1693477b6afa859e9f","trusted":true},"cell_type":"code","source":"validation_generator = validation_datagen.flow_from_directory(\n        'validation',  # this is the target directory\n        target_size=(150, 150),  # all images will be resized to 150x150\n        batch_size=batch_size, # How many images do we need at a time\n        class_mode=None,\n        shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b16117702a009a2925bc9348ff837c330d4bf7db"},"cell_type":"markdown","source":"Now we can create the bottleneck features of our training set","outputs":[],"execution_count":null},{"metadata":{"_uuid":"9559b0cd04f0399849962c3974edde0c57806f57","_cell_guid":"cc3861c9-bd75-4d94-bb6c-b9d546d34788","trusted":true},"cell_type":"code","source":"bottleneck_features_train = model.predict_generator(train_generator, 200, verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"678a7d1baf48c72d31ea0fd96ac94f6f391423a2"},"cell_type":"markdown","source":"as well as the label vector of our training set.","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"56a46a362ceb590435348e70e628cf284c00cb5e"},"cell_type":"code","source":"import numpy as np\ntrain_labels = np.array([0] * 100 + [1] * 100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac0d6e1f5064cdc086cc91c7de26f6ed34679302"},"cell_type":"markdown","source":"We do the same with the validation data:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"69ed6368a5ac3238f4823105764903ac4d67a791"},"cell_type":"code","source":"bottleneck_features_validation = model.predict_generator(validation_generator, 20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d58fd7d418ff79ffd5e91017712487f87dfa77ef"},"cell_type":"code","source":"validation_labels = np.array([0] * 10 + [1] * 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c55dc05f3365ca5f2c128fa31b7073892d67cc1"},"cell_type":"markdown","source":"## Building the new top\nNow we can build the new top of our model. We start with a `Flatten`layer, which reshapes the 3D convolutional outputs into 1D inputs that can be used by densely connected networks. Then we add a dense layer, relu, some dropout and a final classification layer.","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"bf58d496b9ff92f5197284523171c745023435cc"},"cell_type":"code","source":"from keras.layers import Flatten, Dense, Dropout, Activation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d7835d95251a831d82eabf66f1a1067efde32fd"},"cell_type":"code","source":"model = Sequential()\nmodel.add(Flatten(input_shape=bottleneck_features_train.shape[1:]))\nmodel.add(Dense(256))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1))\nmodel.add(Activation('sigmoid'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d92d4cbde5e7dbfe1341e2fcdb9f7c9133fa4f7e"},"cell_type":"code","source":"model.compile(optimizer='adam',\n              loss='binary_crossentropy',\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b14205dfa516d50ce1f7665f6413c8fd7baa719d"},"cell_type":"markdown","source":"Then we fit this top model to the bottleneck training data we created:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"7ec84f4cd504f3e058fcdcaac418ee31b0178b3f"},"cell_type":"code","source":"model.fit(bottleneck_features_train, train_labels,\n          epochs=10,\n          batch_size=32,\n          validation_data=(bottleneck_features_validation, validation_labels))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d864dc6252e42489c905f7f5b6aa2c07f969762"},"cell_type":"markdown","source":"Et voila! 95% accuracy after training from only 200 images!","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"f4e943ae993a961b446f12d70d9d360e4afe1239"},"cell_type":"code","source":"# Cleanup for kaggle\n!rm -r train\n!rm -r validation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"640b58cdddd901f9c26f0c9d9b35d50978a4d6e4"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}