{"cells":[{"metadata":{},"cell_type":"markdown","source":"The notebook will take you through the use of transfer learning to obtain a validation accuracy of 0.99 +. The code can be used as a basis for other implementions too with a few minor tweaks. This dataset is a good introduction to Deep learning with CNN and Transfer learning. Hopefully i have explained things well as we go on"},{"metadata":{"_uuid":"fe76d1d1ded592430e7548feacfa38dc42f085d9"},"cell_type":"markdown","source":"# Import Library and check data locations"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nfrom keras.preprocessing.image import ImageDataGenerator,load_img\nfrom keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\nimport random\nimport os\nprint(os.listdir(\"../input\"))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7335a579cc0268fba5d34d6f7558f33c187eedb3"},"cell_type":"markdown","source":"# Prepare Data"},{"metadata":{},"cell_type":"markdown","source":"This section we use the zipfile to unzip both the test and train files, these will get stored in the \\kaggle\\working\\ folders under there respective names"},{"metadata":{"trusted":true},"cell_type":"code","source":"import zipfile\n\n\nzip_files = ['test1', 'train']\n# Will unzip the files so that you can see them..\nfor zip_file in zip_files:\n    with zipfile.ZipFile(\"../input/{}.zip\".format(zip_file),\"r\") as z:\n        z.extractall(\".\")\n        print(\"{} unzipped\".format(zip_file))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"split the data in the training folder to give a category of cat (0) or dog (1) based on the image file name. this creates a list of 1's and 0's based on the image file name as shown in print categories"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"filenames = os.listdir(\"../working/train\")\ncategories = []\nfor filename in filenames:\n    category = filename.split('.')[0]\n    if category == 'dog':\n        categories.append(1)\n    else:\n        categories.append(0)\n\ndf = pd.DataFrame({\n    'filename': filenames,\n    'category': categories\n})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets look at the top and bottom of our dataframe, as you can see the category is 1 or 0 based on the image name"},{"metadata":{"trusted":true,"_uuid":"915bb9ba7063ab4d5c07c542419ae119003a5f98"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72bf69e817f67f5a2eaff8561217e22077248553"},"cell_type":"code","source":"df.tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a999484fc35b73373fafe2253ae9db7ff46fdb90"},"cell_type":"markdown","source":"### See total amount of each in the training data, the data is nicely balanced so there is no inherent bias in the model"},{"metadata":{"trusted":true,"_uuid":"fa26f0bc7a6d835a24989790b20f3c6f32946f45"},"cell_type":"code","source":"df['category'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a08da58107777a1dd05c4a4bf5c484484923cac"},"cell_type":"markdown","source":"From our data we have 12000 cats and 12000 dogs"},{"metadata":{"_uuid":"400a293df3c8499059d9175f3915187074efd971"},"cell_type":"markdown","source":"# See sample image"},{"metadata":{},"cell_type":"markdown","source":"lets look at a random example of an image, if you run this code a number of times you will see that the images vary in size a lot something that will be dealt with later on"},{"metadata":{"trusted":true,"_uuid":"602b40f7353871cb161c60b5237f0da0096b2f47"},"cell_type":"code","source":"sample = random.choice(filenames)\nimage = load_img(\"../working/train/\"+sample)\nplt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b244e6b7715a04fc6df92dd6dfa3d35c473ca600"},"cell_type":"markdown","source":"# Transfer Learning\n\n"},{"metadata":{},"cell_type":"markdown","source":"We will be using the Inception ResNet V2 pretrained model\n\nWe will initialise the model for an image shape of 256,256 with 3 channels and initialise the weights based of its weights for the imagenet challege.\n"},{"metadata":{"trusted":true,"_uuid":"8c9f833c1441b657c779844912d0b8028218d454"},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Flatten, Dense, Conv2D, MaxPool2D, ZeroPadding2D, Dropout, BatchNormalization\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.applications import inception_resnet_v2\n\nIMG_SHAPE = (256,256, 3)\n\n\nbase_model = tf.keras.applications.InceptionResNetV2(input_shape=IMG_SHAPE,\n                                               include_top=False,\n                                               weights='imagenet')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we allow the base model to be trainable from the 700th layer as there is 780 layers in the base model.\n\nWe allow some layers to be trained as the model will be pretrain to classify in image net, hence opening up a few layers will result in the model being able to adapt to the new task better, and the weights already being a fairly close match to the task.\n\nWe freeze all the layers below 700 and then flatten the out of the 780 layer called 'conv_7b_ac'. from this we add a dense layer of 500 and an l2 norm regularization as this layer has about 25 million parameters so we want to not over fit. We then add drop out layer before narrowing the model to 10 neurons before the final neuron where we pass a single neuron with a sigmoid function that helps classify the problem.\n\nThe number of neurons we added is a randomly chosen number and can be seen as a hyperparameter we can chose to explore later along with dropout rate. \n\nWe never want to intentially create bottlenecks in data only when we want to reduce the dimensionallity of the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.optimizers import Adam\nfrom tensorflow.python.keras import regularizers\nfrom tensorflow.keras.callbacks import EarlyStopping,ReduceLROnPlateau,ModelCheckpoint \n\nbase_model.trainable = True\nprint(\"Number of layers in the base model: \", len(base_model.layers))\n# Fine-tune from this layer onwards\nfine_tune_at = 700\n\n# Freeze all the layers before the `fine_tune_at` layer\nfor layer in base_model.layers[:fine_tune_at]:\n  layer.trainable =  False\n\nlast_layer = base_model.get_layer('conv_7b_ac')\nprint('last layer output shape: ', last_layer.output_shape)\nlast_output = last_layer.output\n\n# Flatten the output layer to 1 dimension\nx = layers.Flatten()(last_output)\n# Add a fully connected layer with 500 hidden units and ReLU activation\nx = layers.Dense(500, activation='relu',kernel_regularizer=regularizers.l2(0.1))(x)\n# Add a dropout rate of 0.3\nx = layers.Dropout(0.3)(x)   \nx = layers.Dense(10, activation='relu',kernel_regularizer=regularizers.l2(0.1))(x)          \n# Add a final sigmoid layer for classification\nx = layers.Dense  (1, activation='sigmoid')(x)\n\nmodel = Model( base_model.input, x) \nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd496f6c65888a969be3703135b0b03a8a1190c8"},"cell_type":"markdown","source":"# Callbacks"},{"metadata":{},"cell_type":"markdown","source":"Implement a number of call backs to aid in the training process, early stopping will stop the training if the validation accuracy doesnt improve after 5 epochs. Reduce on Plateau reduces the learning rate by 0.2 when the validation loss doesnt decrease after 3 epoch. we set a minimum learning rate of 1 * 10^-8"},{"metadata":{"trusted":true,"_uuid":"9aa032f0f6da539d23918890d2d131cc3aac8c7a"},"cell_type":"code","source":"from keras.callbacks import EarlyStopping, ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3421c5ec428da6c0d8cc1184179a9caff1e01d1c"},"cell_type":"code","source":"early_stopping = tf.keras.callbacks.EarlyStopping(\n    monitor='val_acc', \n    verbose=1,\n    patience=5,\n    restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8010a5661ad8924d2db24af0f3c00b1593b38901"},"cell_type":"code","source":"reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,\n                             patience=3, min_lr=0.00000001,verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_uuid":"a79cc604199469789f183096d863f7248e5f6aab"},"cell_type":"code","source":"callbacks = [early_stopping, reduce_lr]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare data"},{"metadata":{},"cell_type":"markdown","source":"Replace labels with string, and split the training data into 90% training and 10% validation and create a seperate dataframe for each and then count number in each.\nBatch size to be based through the image generators will be 32 at a time, this is an arbitory number and is a hyper parameter we can play with."},{"metadata":{"trusted":true},"cell_type":"code","source":"df[\"category\"] = df[\"category\"].replace({0: 'cat', 1: 'dog'}) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4eeb7af8dcf02c4ef5ca744c8305c51a2f5cedef"},"cell_type":"code","source":"train_df, validate_df = train_test_split(df, test_size=0.10, random_state=42)\ntrain_df = train_df.reset_index(drop=True)\nvalidate_df = validate_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b84836337441705eda9d2e655665ffa14d9feead"},"cell_type":"code","source":"train_df['category'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19cf03f9a3c39532d6e2d06bd30be49a5afd9d57"},"cell_type":"code","source":"validate_df['category'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae3dec0361f0443132d0309d3b883ee80070cf9f"},"cell_type":"code","source":"total_train = train_df.shape[0]\ntotal_validate = validate_df.shape[0]\nbatch_size=256","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff760be9104f7d9492467b8d9d3405011aa77d11"},"cell_type":"markdown","source":"* # Traning Generator\n\nwe will create and image generator that will cause random augmentations to occur to the data to allow the model to be invariant to change and positioning of the object (cat or dog). Image augmentation has been shown to aid in perfomance of training however we have to be careful to not augment the image too much from its original form. With the augmentation we are trying to create new data each time to the image is passed over to create the invariance. We do not apply augmentation to the validation set as this is the image we want to test the accuracy on."},{"metadata":{"trusted":true,"_uuid":"4d1c7818703a8a4bac5c036fdea45972aa9e5e9e"},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n    rotation_range=35,\n    rescale=1./255,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    width_shift_range=0.2,\n    height_shift_range=0.2\n)\n\ntrain_generator = train_datagen.flow_from_dataframe(\n    train_df, \n    \"../working/train\", \n    x_col='filename',\n    y_col='category',\n    target_size=(256,256),\n    class_mode='binary',\n    batch_size=batch_size\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"859c7b2857939c19fd2e3bb32839c9f7deb5aa3f"},"cell_type":"markdown","source":"### Validation Generator"},{"metadata":{"trusted":true,"_uuid":"7925e16bcacc89f4484fb6fe47e54d6420af732e"},"cell_type":"code","source":"validation_datagen = ImageDataGenerator(rescale=1./255)\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    validate_df, \n    \"../working/train/\", \n    x_col='filename',\n    y_col='category',\n    target_size=(256,256),\n    class_mode='binary',\n    batch_size=batch_size\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.optimizers import Adam\nopt = Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, amsgrad=True)\nmodel.compile(optimizer = opt, \n              loss = 'binary_crossentropy', \n              metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above we will use the Adam optimisation, you can change the optimisation to SGD ect but i just saw that Adam worked best for this particular task. AmsGrad is a useful tool to use. Details of different learning algorithms can be found on the keras documentation page."},{"metadata":{"_uuid":"6e17fc1f002fedd60febb78fee5e81770640b909"},"cell_type":"markdown","source":"# Example of generator\n\nIn this section we will look at an example of the image generator and how the image augmentation works. We input the image and thus get out 15 fairly different images. Hence this is wha the model will see if we train it on 15 epochs where 1 epoch it goes through the data once. We can therefor manually increase the dataset size by specifiying a batch size less than the number of examples and having the steps per epoch higher than is required to go through one epoch. Hence if we set the steps per epoch to be twice the size of length of training set // batch size, in one epoch the training set will be run through twice with a different image augmentation used for each image. The system never stores the image it flows the image from the location through the augmentation and then the model, so its a computationally efficent process. Making it far easier than hand creating new data."},{"metadata":{"trusted":true,"_uuid":"4252cce168ab65f88e44a8ebc2672607bc852af4"},"cell_type":"code","source":"example_df = train_df.sample(n=1).reset_index(drop=True)\nexample_generator = train_datagen.flow_from_dataframe(\n    example_df, \n    \"../working/train\", \n    x_col='filename',\n    y_col='category',\n    target_size=(256,256),\n    class_mode='categorical'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23d923dba747f8b47dc75569244cecc6f70df321"},"cell_type":"code","source":"plt.figure(figsize=(12, 12))\nfor i in range(0, 15):\n    plt.subplot(5, 3, i+1)\n    for X_batch, Y_batch in example_generator:\n        image = X_batch[0]\n        plt.imshow(image)\n        break\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"810ddf1373d9db470ed48da4f30ca5a6c1274435"},"cell_type":"markdown","source":"As we can see the image looks very diffrent in all 15 instances"},{"metadata":{"_uuid":"5cd8df64e794ed17de326b613a9819e7da977a0e"},"cell_type":"markdown","source":"# Fit Model\n\nnow we fit the model using 10 epochs or this can be changed to whatever you want.\n"},{"metadata":{"trusted":true,"_uuid":"0836a4cc8aa0abf603e0f96573c0c4ff383ad56b"},"cell_type":"code","source":"epochs=3\nhistory = model.fit_generator(\n    train_generator, \n    epochs=epochs,\n    validation_data=validation_generator,\n    validation_steps=22500//256,\n    steps_per_epoch=2500//256\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa1fbc4081ae0de2993188b2bf658a0be5bc0687"},"cell_type":"markdown","source":"# Save Model"},{"metadata":{"trusted":true,"_uuid":"67575a4decdaf79a915d23151626b784ffa82642"},"cell_type":"code","source":"model.save_weights(\"model.h5\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b76c0a9040bc0babf0a453e567e41e22f8a1e0e"},"cell_type":"markdown","source":"# Virtualize Training"},{"metadata":{"trusted":true,"_uuid":"79055f2dc3e2abb47bea758e0464c86ca42ab431"},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 12))\nax1.plot(history.history['loss'], color='b', label=\"Training loss\")\nax1.plot(history.history['val_loss'], color='r', label=\"validation loss\")\nax1.set_xticks(np.arange(1, epochs, 1))\nax1.set_yticks(np.arange(0, 1, 0.1))\n\nax2.plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax2.plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nax2.set_xticks(np.arange(1, epochs, 1))\n\nlegend = plt.legend(loc='best', shadow=True)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}