{"cells":[{"metadata":{},"cell_type":"markdown","source":"Pre-trained NN models offer a great opportunity to build on other researchers' work. Inception which is a Deep Learning Convolutional Architecture presented by Szegedy et al 2014 is an important milestone in the development of CNN classifiers.\n\nIn this Notebook, I am applying a pre-trained model of Inception version 3 on the well-known Kaggle dataset of Cats vs Dogs. Inception V3 was trained using a dataset of 1,000 classes, including Cats and Dogs, from the original ImageNet dataset which was trained with over 1 million training images, the Tensorflow version has 1,001 classes which is due to an additional \"background' class not used in the original ImageNet.\n\nPre-trained model can downloaded from not less than the github of fChollet himself,\n\n!wget --no-check-certificate https://github.com/fchollet/deep-learning-models/releases/download/v0.5/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5 -O \n"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!wget --no-check-certificate https://github.com/fchollet/deep-learning-models/releases/download/v0.5/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5 -O inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport zipfile\nimport matplotlib.pyplot as plt\nfrom keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras import Model\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.applications.inception_v3 import InceptionV3\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initializing Inceptino3 Network"},{"metadata":{"trusted":true},"cell_type":"code","source":"local_weights_file = 'inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5'\npre_trained_model = InceptionV3(input_shape=(128, 128, 3),\n                                include_top=False,\n                                weights=None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now loading the model wieghts that we have dowanloaded to the Inception3 network we initiaized above"},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_trained_model.load_weights(local_weights_file)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are not going to train any of the layers of Inception3 network"},{"metadata":{"trusted":true},"cell_type":"code","source":"for layer in pre_trained_model.layers:\n    layer.trainable = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We would like to append couple of output layers configured specifically to our binary classification task of Cats vs Dogs. \nWe chose mixed9 block of Inception3 network to be where we will started appending output layers. "},{"metadata":{"trusted":true},"cell_type":"code","source":"last_layer = pre_trained_model.get_layer('mixed9')\nprint('last layer output shape: ', last_layer.output_shape)\nlast_output = last_layer.output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adding one output layer for our binary classification Cats vs Dogs task\n# Flatten the output layer to 1 dimension\nx = layers.Flatten()(last_output)\n# Add a fully connected layer with 1,024 hidden units and ReLU activation\nx = layers.Dense(1024, activation='relu')(x)\n# Add a dropout rate of 0.2\nx = layers.Dropout(0.2)(x)\n# Add a final sigmoid layer for classification\nx = layers.Dense(1, activation='sigmoid')(x)\n\n# Build the new model\nmodel = Model(pre_trained_model.input, x)\n\nmodel.compile(optimizer=RMSprop(lr=0.0001),\n              loss='binary_crossentropy',\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We prepare our data set folders, train and validation and fill them with images from tmp folder where we will extract train.zip"},{"metadata":{"trusted":true},"cell_type":"code","source":"from os import makedirs\nfrom shutil import rmtree\ndef make_directory(dir_path):\n    if os.path.exists(dir_path):\n        rmtree(dir_path)\n    makedirs(dir_path)\n    print(dir_path, ' folder is created')\n    \nbase_dir = '../output/cats-vs-dogs'\ntmp_dir = '../output/tmp'\n\nmake_directory(base_dir)\nmake_directory(tmp_dir)\n\n# extract train data\nwith zipfile.ZipFile('../input/dogs-vs-cats/train.zip', 'r') as zip_ref:\n    zip_ref.extractall(tmp_dir)\n    zip_ref.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy',\n                                            patience=2,\n                                            verbose=1,\n                                            factor=0.5,\n                                            min_lr=0.00001)\ncallbacks = [learning_rate_reduction]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = os.path.join(base_dir, 'train')\nvalidation_dir = os.path.join(base_dir, 'validation')\n\ntrain_cats_dir = os.path.join(train_dir, 'cats')  # Directory with our training cat pictures\ntrain_dogs_dir = os.path.join(train_dir, 'dogs')  # Directory with our training dog pictures\nvalidation_cats_dir = os.path.join(validation_dir, 'cats')  # Directory with our validation cat pictures\nvalidation_dogs_dir = os.path.join(validation_dir, 'dogs')  # Directory with our validation dog pictures\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Creating folders ....')\nmake_directory(train_dir)\nmake_directory(train_cats_dir)\nmake_directory(train_dogs_dir)\nmake_directory(validation_dir)\nmake_directory(validation_cats_dir)\nmake_directory(validation_dogs_dir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_of_fnames = os.listdir(os.path.join(tmp_dir,'train'))\nprint(len(list_of_fnames))\nlist_of_cats_fnames = [i for i in list_of_fnames if 'CAT' in i.upper()]\nprint('Found {0} CATS images in input folder tmp/train'.format(len(list_of_cats_fnames)))\nlist_of_dogs_fnames = [i for i in list_of_fnames if 'DOG' in i.upper()]\nprint('Found {0} DOGS images in input folder tmp/train'.format(len(list_of_dogs_fnames)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.shuffle(list_of_cats_fnames)\nnp.random.shuffle(list_of_dogs_fnames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TOTAL_CATS = len(list_of_cats_fnames)\nTOTAL_DOGS = len(list_of_dogs_fnames)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In my first runs I found that validation accuracy is higher that training accuracy, my interpretation for this is valiadtion set is small and the model is just go lucky figuring out the right labels, thereofore I increased the size of the validation set to 40%\nOne more comment, we train for only 5 Epochs and on 5000 training samples since we are relying on the pretrained Inception network"},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_VALIDATION_SPLIT_AT = 0.6\nBATCH_SIZE = 100\nTARGET_SIZE = (128, 128)\nNO_OF_EPOCHS = 5\nEXPERIMENT_SIZE = 5000  # Here we are using only 5000 samples, we relying to Inception pretrained network and therefore\n# we don't have to train on the full data set.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('\\nDistributing images to \\n {0} \\n {1} \\n {2} \\n {3}'\n      '\\nsuch that {4}% of total number of images goes to training and \\n'\n      '{5}% goes to validation'.format(\n    train_cats_dir, train_dogs_dir,\n    validation_cats_dir, validation_dogs_dir,\n    round(TRAIN_VALIDATION_SPLIT_AT * 100),\n    round((1 - TRAIN_VALIDATION_SPLIT_AT) * 100)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Copy images from tmp_dir to train/Cats, train/Dogs and to validation/Cats and validation/Dogs\n# according to the split percentage we decided.\n\nfrom shutil import copyfile\ntmp_train_dir = os.path.join(tmp_dir, 'train')\nc = 0\nfor i in list_of_cats_fnames:\n    if c < (round(TRAIN_VALIDATION_SPLIT_AT * EXPERIMENT_SIZE)):\n        copyfile(os.path.join(tmp_train_dir, i), os.path.join(train_cats_dir, i))\n    else:\n        copyfile(os.path.join(tmp_train_dir, i), os.path.join(validation_cats_dir, i))\n    c += 1\n    if c >= EXPERIMENT_SIZE:\n        break\n\nc = 0\nfor i in list_of_dogs_fnames:\n    if c < (round(TRAIN_VALIDATION_SPLIT_AT * EXPERIMENT_SIZE)):\n        copyfile(os.path.join(tmp_train_dir, i), os.path.join(train_dogs_dir, i))\n    else:\n        copyfile(os.path.join(tmp_train_dir, i), os.path.join(validation_dogs_dir, i))\n    c += 1\n    if c >= EXPERIMENT_SIZE:\n        break\n\nprint('Total training cat images :', len(os.listdir(train_cats_dir)))\nprint('Total training dog images :', len(os.listdir(train_dogs_dir)))\n\nprint('Total validation cat images :', len(os.listdir(validation_cats_dir)))\nprint('Total validation dog images :', len(os.listdir(validation_dogs_dir)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Loading images through generators ...')\ntrain_datagen = ImageDataGenerator(rescale=1. / 255.,\n                                   rotation_range=40,\n                                   width_shift_range=0.2,\n                                   height_shift_range=0.2,\n                                   shear_range=0.2,\n                                   zoom_range=0.2,\n                                   horizontal_flip=True)\n\n# Note that the validation data should not be augmented!\nvalidation_datagen = ImageDataGenerator(rescale=1.0 / 255.)\n\n# Flow training images in batches of 20 using train_datagen generator\ntrain_generator = train_datagen.flow_from_directory(train_dir,\n                                                    batch_size=BATCH_SIZE,\n                                                    target_size=TARGET_SIZE,\n                                                    class_mode='binary')\n\nTOTAL_TRAINING = len(train_generator.filenames)\n\nvalidation_generator = validation_datagen.flow_from_directory(validation_dir,\n                                                              batch_size=BATCH_SIZE,\n                                                              target_size=TARGET_SIZE,\n                                                              class_mode='binary')\n\nTOTAL_VALIDATION = len(validation_generator.filenames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    train_generator,\n    epochs=NO_OF_EPOCHS,\n    validation_data=validation_generator,\n    steps_per_epoch=TOTAL_TRAINING / BATCH_SIZE,\n    validation_steps=TOTAL_VALIDATION / BATCH_SIZE,\n    callbacks=[callbacks],\n    verbose=2)  # Found that this is the clearest, no annoying progress bars\n\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # -----------------------------------------------------------\n# To have a health training Loss should decrease while accuracy increases\n# if loss increase while accuracy increases then this is an overfitting case\nacc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n#\nepochs = range(len(acc))  # Get number of epochs\n# # # ------------------------------------------------\n# # # Plot training and validation accuracy per epoch\n# # # ------------------------------------------------\nplt.plot(epochs, acc, color='b', label=\"Training accuracy\")\nplt.plot(epochs, val_acc, color='r', label=\"Validation accuracy\")\nplt.title('Training and validation accuracy')\nplt.legend(loc='best', shadow=True)\nplt.show()\n# ------------------------------------------------\n# Plot training and validation loss per epoch\n# ------------------------------------------------\nplt.plot(epochs, loss, color='b', label=\"Training loss\")\nplt.plot(epochs, val_loss, color='r', label=\"Validation loss\")\nplt.title('Training and validation loss')\nplt.legend(loc='best', shadow=True)\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}