{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nConvolutional Neural Networks (CNN) are everywhere. It is arguably the most popular deep learning architecture. The main advantage of CNN compared to its predecessors is that it automatically detects the important features without any human supervision. For example, given many pictures of cats and dogs it learns distinctive features for each class by itself.\n\nCNN is also computationally efficient. It uses special convolution and pooling operations and performs parameter sharing. This enables CNN models to run on any device, making them universally attractive.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Import necessary libraries","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Import necessary libraries\nfrom matplotlib import pyplot\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport warnings\nimport os\nimport pickle\nimport random\nfrom shutil import copyfile\nfrom random import random, seed\nfrom matplotlib.image import imread\n\nwarnings.filterwarnings('ignore')\npd.options.display.float_format = '{:,.2f}'.format\npd.set_option('display.max_rows', 100)\npd.set_option('display.max_columns', 200)\n\n\n# baseline model with dropout for the dogs vs cats dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import Dropout\nfrom keras.optimizers import SGD\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau\n\n# make a prediction for a new image.\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array\nfrom keras.models import load_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Directory structure","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Unzipping data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Unzip the train and test data sets\n!unzip '../input/dogs-vs-cats/train.zip' -d '/tmp/cats-vs-dogs'\n!unzip '../input/dogs-vs-cats/test1.zip' -d '/tmp/cats-vs-dogs'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_dir = '/tmp/cats-vs-dogs/'\n\ntrain_dir = '/tmp/cats-vs-dogs/train/'\ntest_dir = '/tmp/cats-vs-dogs/test1/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating a dataframe for train set\nfilenames = os.listdir(train_dir)\ncategories = []\nfor filename in filenames:\n    category = filename.split('.')[0]\n    if category == 'dog':\n        categories.append(1)\n    else:\n        categories.append(0)\n\ndf = pd.DataFrame({\n    'filename': filenames,\n    'category': categories\n})","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.figure(figsize=(8,6))\nsns.countplot(x='category',data=df)\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['category'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot dog photos from the dogs vs cats dataset\nfig=pyplot.figure(figsize=(16, 16))\ncolumns = 4\nrows = 5\ni=1\nfor filename in os.listdir(train_dir)[0:20]:\n    image = imread(train_dir + filename)\n    fig.add_subplot(rows, columns, i)\n    pyplot.imshow(image)\n    i+=1\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot first few dog images\npyplot.figure(figsize=(12,12))\nfor i in range(9):\n    # define subplot\n    pyplot.subplot(330 + 1 + i)\n    # define filename\n    filename = train_dir + 'dog.' + str(i) + '.jpg'\n    # load image pixels\n    image = imread(filename)\n    # plot raw pixel data\n    pyplot.imshow(image)\n# show the figure\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Highlights**\n\nRunning the example creates a figure showing the first nine photos of dogs in the dataset.\n\nWe can see that some photos are landscape format, some are portrait format, and some are square.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot first few cat images\npyplot.figure(figsize=(12,12))\nfor i in range(9):\n    # define subplot\n    pyplot.subplot(330 + 1 + i)\n    # define filename\n    filename = train_dir + 'cat.' + str(i) + '.jpg'\n    # load image pixels\n    image = imread(filename)\n    # plot raw pixel data\n    pyplot.imshow(image)\n# show the figure\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Highlights**\n\nAgain, we can see that the photos are all different sizes.\n\nWe can also see a photo where the cat is barely visible (bottom left corner) and another that has two cats (lower right corner). This suggests that any classifier fit on this problem will have to be robust.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Select Standardized Photo Size \nThe photos will have to be reshaped prior to modeling so that all images have the same shape. This is often a small square image.\n\nThere are many ways to achieve this, although the most common is a simple resize operation that will stretch and deform the aspect ratio of each image and force it into the new shape.\n\nWe could load all photos and look at the distribution of the photo widths and heights, then design a new photo size that best reflects what we are most likely to see in practice.\n\nSmaller inputs mean a model that is faster to train, and typically this concern dominates the choice of image size. In this case, we will follow this approach and choose a fixed size of 200×200 pixels.\n\n### Pre-Process Photos into Standard Directories\n\nwe can load the images progressively using the **Keras ImageDataGenerator** class and **flow_from_directory()** API.\n\nThis API prefers data to be divided into separate training/ and validation/ directories, and under each directory to have a subdirectory for each class, e.g. a training/dogs/ and a training/cats/ subdirectories and the same for validation. Images are then organized under the subdirectories.\n\nWe can write a script to create a copy of the dataset with this preferred structure. We will randomly select 25% of the images (or 6,250) to be used in a test dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# create directories\nsubdirs = ['training/', 'validation/']\nfor subdir in subdirs:\n    # create label subdirectories\n    labeldirs = ['dogs/', 'cats/']\n    for labldir in labeldirs:\n        newdir = base_dir + subdir + labldir\n        os.makedirs(newdir, exist_ok=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(base_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we can enumerate all image files in the dataset and copy them into the dogs/ or cats/ subdirectory based on their filename.\n\nAdditionally, we can randomly decide to hold back 25% of the images into the validation dataset. This is done consistently by fixing the seed for the pseudorandom number generator so that we get the same split of data each time the code is run.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# seed random number generator\nseed(1)\n# define ratio of pictures to use for validation\nval_ratio = 0.25\n# copy training dataset images into subdirectories\nsrc_directory = 'train/'\nfor file in os.listdir(base_dir + src_directory):\n    src = base_dir + src_directory + '/' + file\n    dst_dir = base_dir + 'training/'\n    if random() < val_ratio:\n        dst_dir = base_dir + 'validation/'\n    if file.startswith('cat'):\n        dst =  dst_dir + 'cats/'  + file\n        copyfile(src, dst)\n    elif file.startswith('dog'):\n        dst =  dst_dir + 'dogs/'  + file\n        copyfile(src, dst)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now, Let's look at the filenames in cats and dogs training and validation directories.\nprint(os.listdir('/tmp/cats-vs-dogs/training/cats')[0:5])\nprint(os.listdir('/tmp/cats-vs-dogs/training/dogs')[0:5])\nprint(os.listdir('/tmp/cats-vs-dogs/validation/cats')[0:5])\nprint(os.listdir('/tmp/cats-vs-dogs/validation/dogs')[0:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = '/tmp/cats-vs-dogs/training/'\nvalid_dir = '/tmp/cats-vs-dogs/validation/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Develop CNN Model\n\nAll CNN models follow a similar architecture, as shown in the figure below.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://i.imgur.com/zxfn5Sy.png\" />\n\nThere is an input image that we’re working with. We perform a series convolution + pooling operations, followed by a number of fully connected layers. If we are performing multiclass classification the output is softmax. We will now dive into each component.\n\n* **Input Layer**: It represent input image data. It will reshape image into single diminsion array. For example image size is 64x64 = 4096, then it will convert to (4096,1) array.\n* **Conv Layer**: The main building block of CNN is the convolutional layer. Convolution is a mathematical operation to merge two sets of information. In our case the convolution is applied on the input data using a convolution filter to produce a feature map.\n* **Stride**: Stride specifies how much we move the convolution filter at each step. By default the value is 1.\n* **Padding**: We see that the size of the feature map is smaller than the input, because the convolution filter needs to be contained in the input. If we want to maintain the same dimensionality, we can use padding to surround the input with zeros.\n* **Pooling Layer**: After a convolution operation we usually perform pooling to reduce the dimensionality. This enables us to reduce the number of parameters, which both shortens the training time and combats overfitting. \n* **Fully Connected Layer**: It connect the network from a layer to another layer. After the convolution + pooling layers we add a couple of fully connected layers to wrap up the CNN architecture.\n* **Conv2D**: This method creates a convolutional layer. The first parameter is the filter count, and the second one is the filter size. For example in the first convolution layer we create 32 filters of size 3x3. We use relu non-linearity as activation. We also enable padding. In Keras there are two options for **padding: same or valid**. Same means we pad with the number on the edge and valid means no padding. **Stride** is 1 for convolution layers by default so we don’t change that. \n* **MaxPooling2D**: creates a maxpooling layer, the only argument is the window size. We use a 2x2 window as it’s the most common. By default **stride length is equal to the window size**, which is 2 in our case, so we don’t change that.\n* **Flatten**: After the convolution + pooling layers we flatten their output to feed into the fully connected layers as we discussed above.\n* **Dropout**: Dropout is by far the most popular **regularization technique** for deep neural networks. Dropout is used to prevent overfitting and the idea is very simple. During training time, at each iteration, a neuron is temporarily “dropped” or disabled with probability p. This means all the inputs and outputs to this neuron will be disabled at the current iteration.\n* **Output Layer**: It is the predicted values layer.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The architecture involves stacking convolutional layers with small 3×3 filters followed by a max pooling layer. Together, these layers form a block, and these blocks can be repeated where the number of filters in each block is increased with the depth of the network such as 32, 64, 128, 256 for the first four blocks of the model. Padding is used on the convolutional layers to ensure the height and width shapes of the output feature maps matches the inputs.\n\nWe can explore this architecture on the dogs vs cats problem and compare a model with this architecture with 1, 2, and 3 blocks.\n\nEach layer will use the ReLU activation function and the He weight initialization, which are generally best practices.\n\nThe model will be fit with stochastic gradient descent and we will start with a conservative learning rate of 0.001 and a momentum of 0.9.\n\nThe problem is a binary classification task, requiring the prediction of one value of either 0 or 1. An output layer with 1 node and a sigmoid activation will be used and the model will be optimized using the binary cross-entropy loss function.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# define cnn model\n\nmodel = Sequential()\n    \nmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', name='conv_1', padding='same', input_shape=(200, 200, 3)))\nmodel.add(MaxPooling2D((2, 2), name='maxpool_1'))\nmodel.add(Dropout(0.2))\n    \nmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', name='conv_2', padding='same'))\nmodel.add(MaxPooling2D((2, 2), name='maxpool_2'))\nmodel.add(Dropout(0.2))\n    \nmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', name='conv_3', padding='same'))\nmodel.add(MaxPooling2D((2, 2), name='maxpool_3'))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_uniform', name='conv_4', padding='same'))\nmodel.add(MaxPooling2D((2, 2), name='maxpool_4'))\nmodel.add(Dropout(0.2))\n    \nmodel.add(Flatten())\nmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform', name='dense_1'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1, activation='sigmoid', name='output'))\n    \n# compile model\nopt = SGD(lr=0.001, momentum=0.9)\nmodel.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Summary of the Model\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Augmentation\nOverfitting happens because of having too few examples to train on, resulting in a model that has poor generalization performance. If we had infinite training data, we wouldn’t overfit because we would see every possible instance. Data augmentation is a way to generate more training data from our current set.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Next, iterators need to be prepared for both the train and test datasets.\n\nWe can use the **flow_from_directory()** function on the data generator and create one iterator for each of the training/ and validation/ directories. We must specify that the problem is a binary classification problem via the **class_mode** argument, and to load the images with the size of 200×200 pixels via the **target_size** argument. We will fix the batch size at 64.\nThis involves first defining an instance of the **ImageDataGenerator** that will scale the pixel values to the range of 0-1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data Augmentation\n# datagen = ImageDataGenerator(rescale=1.0/255.0)\n    \n# # prepare iterator\n# train_it = datagen.flow_from_directory(train_dir,class_mode='binary', batch_size=64, target_size=(200, 200))\n# valid_it = datagen.flow_from_directory(valid_dir,class_mode='binary', batch_size=64, target_size=(200, 200))\n\n# create data generators\ntrain_datagen = ImageDataGenerator(rescale=1.0/255.0, width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\nvalid_datagen = ImageDataGenerator(rescale=1.0/255.0)\n\n# prepare iterators\ntrain_it = train_datagen.flow_from_directory(train_dir, class_mode='binary', batch_size=64, target_size=(200, 200))\nvalid_it = valid_datagen.flow_from_directory(valid_dir, class_mode='binary', batch_size=64, target_size=(200, 200))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case, photos in the training dataset will be augmented with small (10%)(width_shift_range=0.1) random horizontal and vertical shifts and random horizontal flips that create a mirror image of a photo. Photos in both the training and validation steps will have their pixel values scaled in the same way.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Callbacks","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Early Stop\n\nTo prevent over fitting we will stop the learning after 10 epochs and val_loss value not decreased","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"earlystop = EarlyStopping(patience=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning Rate Reduction\n\nWe will reduce the learning rate when then accuracy not increase for 2 steps","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, factor=0.5, min_lr=0.001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [earlystop, learning_rate_reduction]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can then fit the model using the train iterator (train_it) and use the validation iterator (valid_it) as a validation dataset during training.\n\nThe number of steps for the train and validation iterators must be specified. This is the number of batches that will comprise one epoch. This can be specified via the length of each iterator, and will be the total number of images in the train and test directories divided by the batch size (64).\n\nThe model will be fit for 20 epochs, a small number to check if the model can learn the problem.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# fit model\nhistory = model.fit_generator(train_it, steps_per_epoch=len(train_it),validation_data=valid_it, \n                              validation_steps=len(valid_it), epochs=70, callbacks=callbacks, verbose=1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save model\nmodel.save('final_model.h5')   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluate model\n\nOnce fit, the final model can be evaluated on the validation dataset directly and the classification accuracy reported.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# evaluate model\n_, acc = model.evaluate_generator(valid_it, steps=len(valid_it), verbose=1)\nprint('> %.3f' % (acc * 100.0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we can create a plot of the history collected during training stored in the **history** directory returned from the call to fit_generator().\n\nThe History contains the model accuracy and loss on the validation and training dataset at the end of each epoch. Line plots of these measures over training epochs provide learning curves that we can use to get an idea of whether the model is overfitting, underfitting, or has a good fit.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# A figure is also created showing a line plot for the loss and another for the accuracy of the model \n# on both the training (red) and validation (blue) datasets.\n\n# plot diagnostic learning curves\nacc = history.history[\"accuracy\"]\nval_acc = history.history[\"val_accuracy\"]\n\nloss = history.history[\"loss\"]\nval_loss = history.history[\"val_loss\"]\nepochs = range(len(acc))\n\nfig, (ax1, ax2) = pyplot.subplots(2, 1, figsize=(12, 12))\n\n# plot accuracy\nax1.plot(epochs, acc, \"r\", label=\"Training accuracy\")\nax1.plot(epochs, val_acc, \"b\", label=\"Validation accuracy\")\nax1.title.set_text(\"Training and Validation accuracy\")\n\n# plot loss\nax2.plot(epochs, loss, \"r\", label=\"Training loss\")\nax2.plot(epochs, val_loss, \"b\", label=\"Validation loss\")\nax2.title.set_text(\"Training and Validation loss\")\n\npyplot.legend(loc='best', shadow=True)\npyplot.tight_layout()\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction of Testing Data\n\n### Prepare Testing Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_filenames = os.listdir(test_dir)\ntest_df = pd.DataFrame({\n    'filename': test_filenames\n})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nb_samples = test_df.shape[0]\nnb_samples","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_gen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_gen.flow_from_dataframe(test_df, test_dir, x_col='filename',y_col=None,class_mode=None,\n                                              batch_size=64,target_size=(200, 200),shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict = model.predict_generator(test_generator, steps=np.ceil(nb_samples/64))\nthreshold = 0.6\ntest_df['category'] = np.where(predict > threshold, 1,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_test = test_df.sample(n=9).reset_index()\nsample_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.figure(figsize=(12, 12))\nfor index, row in sample_test.iterrows():\n    filename = row['filename']\n    category = row['category']\n    img = load_img(test_dir + filename, target_size=(256, 256))\n    pyplot.subplot(3, 3, index+1)\n    pyplot.imshow(img)\n    pyplot.xlabel(filename + '(' + \"{}\".format(category) + ')')\npyplot.tight_layout()\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = test_df.copy()\nsubmission_df['id'] = submission_df['filename'].str.split('.').str[0]\nsubmission_df['label'] = submission_df['category']\nsubmission_df.drop(['filename', 'category'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.figure(figsize=(10,5))\nsns.countplot(submission_df['label'])\npyplot.title(\"(Test data)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}