{"cells":[{"metadata":{},"cell_type":"markdown","source":"Here I am applying CNN to Cats vs. Dogs dataset with images read using ImageDataGenerator. \n\nIn this Kernel, I am trying to find the lest complex CNN architecture with the best performance by monitoring overfitting levels of the network. Hereunder is how I approached this problem:\n\n1. Start with simple CNN architecture ( one convolution layer of 16 filters and one dense 64 nodes) and an arbitrary small data set of 2000 images. This gave me a clear overfit in most cases with such simple network architecture. \n\n2. Increase the data set size and train again, and add more data until data balances the overfitting and the network performs well. \n\n3. Increase the complexity of the model, train again till it overfits, then increase data set size and include image augmentation, and repeat until we run out of data and have an overfit and the same time the least complicated CNN.\n\n4. Apply regularization by adding dropout to simplify the CNN architecture and git rid of overfitting.\n\nHere is the code which I tried to comment comprehensively. For brevity I am including only last step where I am training with full data set.","attachments":{}},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import shutil\nimport os\nfrom PIL import Image\ndef make_directory(dir_path):\n    if os.path.exists(dir_path):\n        shutil.rmtree(dir_path)\n    os.makedirs(dir_path)\n    print('folder ', dir_path, ' is created')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import zipfile\n\nDATASET_PATH = '../output/dogs-vs-cats'\nDATASET_PATH_TMP = '../output/dogs-vs-cats/tmp'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"make_directory(DATASET_PATH)\nmake_directory(DATASET_PATH_TMP)\n\n# extract train data\nwith zipfile.ZipFile('../input/dogs-vs-cats/train.zip', 'r') as zip_ref:\n    zip_ref.extractall(DATASET_PATH_TMP)\n\n# extract test data\nwith zipfile.ZipFile('../input/dogs-vs-cats/test1.zip', 'r') as zip_ref:\n    zip_ref.extractall(DATASET_PATH)\n\nprint('done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from shutil import copyfile\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nfrom keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc',\n                                            patience=2,\n                                            verbose=1,\n                                            factor=0.5,\n                                            min_lr=0.00001)\ncallbacks = [learning_rate_reduction]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# All files are here including train.zip and test1.zip\nbase_dir = DATASET_PATH\n# This is where I have extracted train.zip, will copy from here to train/cats and train/dogs and to validation/cats \n# and validation/dogs\ntmp_dir = os.path.join(base_dir, 'tmp/train')\n# This is training folder\ntrain_dir = os.path.join(base_dir, 'train')\n# This is validation folder. We will copy from train 20% of the dogs to validation/dogs and 20% of cats to validation/Cats\n# Validation is part of the development procss of the model, whether the data samples are assigned to validation set \n# dynamically through k-Fold Corss-Validation or it is a fixed set from the beginning of training till the end.\nvalidation_dir = os.path.join(base_dir, 'validation')\n# This is test folder, we extract test1.zip here. This is the 'Production' Dataset where you don't know the labels \n# of the data samples.\ntest_dir = os.path.join(base_dir, 'test1')\n# Directory with our training cat/dog pictures\ntrain_cats_dir = os.path.join(train_dir, 'cats')\ntrain_dogs_dir = os.path.join(train_dir, 'dogs')\nvalidation_cats_dir = os.path.join(validation_dir, 'cats')\nvalidation_dogs_dir = os.path.join(validation_dir, 'dogs')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Creating folders ....')\nmake_directory(train_dir)\nmake_directory(train_cats_dir)\nmake_directory(train_dogs_dir)\nmake_directory(validation_dir)\nmake_directory(validation_cats_dir)\nmake_directory(validation_dogs_dir)\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_of_fnames = os.listdir(tmp_dir)\nlist_of_cats_fnames = [i for i in list_of_fnames if 'CAT' in i.upper()]\nprint('Found {0} CATS images in input folder tmp/train'.format(len(list_of_cats_fnames)))\nlist_of_dogs_fnames = [i for i in list_of_fnames if 'DOG' in i.upper()]\nprint('Found {0} DOGS images in input folder tmp/train'.format(len(list_of_dogs_fnames)))\n\nnp.random.shuffle(list_of_cats_fnames)\nnp.random.shuffle(list_of_dogs_fnames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TOTAL_CATS = len(list_of_cats_fnames)\nTOTAL_DOGS = len(list_of_dogs_fnames)\n\nTRAIN_VALIDATION_SPLIT_AT = 0.8\n\nBATCH_SIZE = 100\nTARGET_SIZE = (128, 128)\nNO_OF_EPOCHS = 25\nEXPERIMENT_SIZE = 12500  # Size of the sample set per category, cats or doags.\n# This is to control how many samples we want to experiment the model on.\n# This helps to build the model incrementally by experimenting on smaller\n# set size, train untill over fit, then to seek better performance we increase complexity of the network\n# train again until we overfit, add more data, and so on untill we we make use of all data available.\n\nprint('\\nDistributing images to \\n {0} \\n {1} \\n {2} \\n {3}'\n      '\\nsuch that {4}% of total number of images goes to training and \\n'\n      '{5}% goes to validation'.format(\n    train_cats_dir, train_dogs_dir,\n    validation_cats_dir, validation_dogs_dir,\n    round(TRAIN_VALIDATION_SPLIT_AT * 100),\n    round((1 - TRAIN_VALIDATION_SPLIT_AT) * 100)))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"c = 0\nfor i in list_of_cats_fnames:\n    if c < (round(TRAIN_VALIDATION_SPLIT_AT * EXPERIMENT_SIZE)):\n        copyfile(os.path.join(tmp_dir, i), os.path.join(train_cats_dir, i))\n    else:\n        copyfile(os.path.join(tmp_dir, i), os.path.join(validation_cats_dir, i))\n    c += 1\n    if c >= EXPERIMENT_SIZE:\n        break\n\nc = 0\nfor i in list_of_dogs_fnames:\n    if c < (round(TRAIN_VALIDATION_SPLIT_AT * EXPERIMENT_SIZE)):\n        copyfile(os.path.join(tmp_dir, i), os.path.join(train_dogs_dir, i))\n    else:\n        copyfile(os.path.join(tmp_dir, i), os.path.join(validation_dogs_dir, i))\n    c += 1\n    if c >= EXPERIMENT_SIZE:\n        break\n\nprint('Total training cat images :', len(os.listdir(train_cats_dir)))\nprint('Total training dog images :', len(os.listdir(train_dogs_dir)))\n\nprint('Total validation cat images :', len(os.listdir(validation_cats_dir)))\nprint('Total validation dog images :', len(os.listdir(validation_dogs_dir)))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Loading images through generators ...')\n# # Here we create ImageDataGenerator and we normalize while loading\ntrain_datagen = ImageDataGenerator(\n    rescale=1.0 / 255,\n    rotation_range=15,\n    shear_range=0.1,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    width_shift_range=0.1,\n    height_shift_range=0.1\n)\nvalidation_datagen = ImageDataGenerator(\n    rescale=1.0 / 255,\n    rotation_range=15,\n    shear_range=0.1,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    width_shift_range=0.1,\n    height_shift_range=0.1)\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # # We then load data through the generator\ntrain_generator = train_datagen.flow_from_directory(\n    directory=train_dir,\n    target_size=TARGET_SIZE,  # Resize the image while loading\n    batch_size=BATCH_SIZE,  #\n    class_mode='binary')  # 1 Dimensional binary labels, generator assigns 0 to cats, and 1 to dogs\n# we can see that from train_generator.model.indicies\n\nTOTAL_TRAINING = len(train_generator.filenames)\n#\nvalidation_generator = validation_datagen.flow_from_directory(\n    directory=validation_dir,  # This is the source directory for training images\n    target_size=TARGET_SIZE,  # All images will be resized to 150x150\n    batch_size=BATCH_SIZE,\n    class_mode='binary')\n\nTOTAL_VALIDATION = len(validation_generator.filenames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# I have started with simpler networks than this and with less number of samples and smaller number of training epochs,\n# Now I am continuing from this network architecture Conve8MaxPool->Conv16MaxPool->Conv32MaxPool->Dense128\nprint('Constructing and compiling model ...')\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(8, (3, 3), activation='relu', input_shape=(128, 128, 3)),\n    tf.keras.layers.MaxPooling2D(2, 2),\n    tf.keras.layers.Conv2D(16, (3, 3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2, 2),\n    tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2, 2),\n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(128, activation='relu'),\n    tf.keras.layers.Dense(1, activation='sigmoid')\n])\n\nmodel.compile(optimizer=RMSprop(lr=0.001),\n              loss='binary_crossentropy',  # not sparse_crossentropy or categorical_corssentropy since\n              # we are doing two class which can ben handled as a binary classification\n              metrics=['accuracy'])\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# here we train the model\nprint('Training ....')\nhistory = model.fit(\n    train_generator,\n    epochs=NO_OF_EPOCHS,\n    validation_data=validation_generator,\n    steps_per_epoch=TOTAL_TRAINING / BATCH_SIZE,\n    validation_steps=TOTAL_VALIDATION / BATCH_SIZE,\n    callbacks = [callbacks],\n    verbose=2)  # Found that this is the clearest, no annoying progress bars\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # -----------------------------------------------------------\n# To have a healthy training, loss should decrease while accuracy increases\n# if loss increases while accuracy increases then this is an overfitting case\nacc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n#\nepochs = range(len(acc))  # Get number of epochs\n#\n# # ------------------------------------------------\n# # Plot training and validation accuracy per epoch\n# # ------------------------------------------------\nplt.plot(epochs, acc, color='b', label=\"Training accuracy\")\nplt.plot(epochs, val_acc, color='r', label=\"Validation accuracy\")\nplt.title('Training and validation accuracy')\nplt.legend(loc='best', shadow=True)\nplt.figure()\n# #\n# # ------------------------------------------------\n# # Plot training and validation loss per epoch\n# # ------------------------------------------------\nplt.plot(epochs, loss, color='b', label=\"Training loss\")\nplt.plot(epochs, val_loss, color='r', label=\"Validation loss\")\nplt.title('Training and validation loss')\nplt.legend(loc='best', shadow=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although the validation accuracy score fluctuates a little, no apparent overfitting and the model would probably give better performance with additional training. \n\n#### Conclusion\nThe least complex CNN architecture for the task at hand is more computationally efficient hence it is desirable. The objective of this notebook was to find a systematic approach to find that. Although I can not say for sure that my experiments took less time than what I would have taken if I tried random architectures but at least using this method gave some assurance that what I reached could be the optimum architecture from complexity point of view even if I have already spent more time to find it!\n\n#### Additional Experiments\nI am still working on this Notebook. Things I would like to try are:\n1. Try pushing the model to overfit by loading the model from the previous run and train again for another 25 epochs.\n2. Try cross-validation to reduce variability in validation accuracy score and learn from the entire data set.\n3. Once the model shows some symptoms of overfitting try to regularize the model. \n4. Compare this model performance with performances from pre-trained models say Inception Vx or VGGxx."},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}