{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport cv2\nimport zipfile\n#import visualkeras\n\nfrom sklearn.model_selection import train_test_split","metadata":{"id":"XXSsX3lksrUl","execution":{"iopub.status.busy":"2021-11-19T00:33:10.335746Z","iopub.execute_input":"2021-11-19T00:33:10.336311Z","iopub.status.idle":"2021-11-19T00:33:11.336365Z","shell.execute_reply.started":"2021-11-19T00:33:10.336273Z","shell.execute_reply":"2021-11-19T00:33:11.335642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using the os library we can print the files contained inside a directory, this should contain 3 files as can be seen below:","metadata":{}},{"cell_type":"code","source":"print(os.listdir(\"../input/dogs-vs-cats-redux-kernels-edition\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:11.338276Z","iopub.execute_input":"2021-11-19T00:33:11.338669Z","iopub.status.idle":"2021-11-19T00:33:11.344456Z","shell.execute_reply.started":"2021-11-19T00:33:11.338631Z","shell.execute_reply":"2021-11-19T00:33:11.343528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"test.zip and train.zip contains the images to use in our project, because of this we have to extract them using the library zipfile, the images extracted will appear in the output directory of kaggle '/kaggle/working/', path which we have to take to explore each image.","metadata":{}},{"cell_type":"code","source":"with zipfile.ZipFile('../input/dogs-vs-cats-redux-kernels-edition/test.zip','r') as z:\n    z.extractall('.')\n    \nwith zipfile.ZipFile('../input/dogs-vs-cats-redux-kernels-edition/train.zip','r') as z:\n    z.extractall('.')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:11.345972Z","iopub.execute_input":"2021-11-19T00:33:11.346496Z","iopub.status.idle":"2021-11-19T00:33:29.800758Z","shell.execute_reply.started":"2021-11-19T00:33:11.346457Z","shell.execute_reply":"2021-11-19T00:33:29.800012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's print the list of images contained in the train folder, as we know this contain 25000 images which is huge and in order to just have an idea how they are named 10 samples of them will be shown:","metadata":{}},{"cell_type":"code","source":"os.listdir('/kaggle/working/train/')[:10]","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:29.803684Z","iopub.execute_input":"2021-11-19T00:33:29.804229Z","iopub.status.idle":"2021-11-19T00:33:29.826009Z","shell.execute_reply.started":"2021-11-19T00:33:29.804187Z","shell.execute_reply":"2021-11-19T00:33:29.825253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above the list of images is huge and we would be better if we print the length of such lists to see how many images each folder contain:","metadata":{}},{"cell_type":"code","source":"len(os.listdir('/kaggle/working/train/')), len(os.listdir('/kaggle/working/test/'))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:29.827251Z","iopub.execute_input":"2021-11-19T00:33:29.827572Z","iopub.status.idle":"2021-11-19T00:33:29.854213Z","shell.execute_reply.started":"2021-11-19T00:33:29.827533Z","shell.execute_reply":"2021-11-19T00:33:29.853454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Sample of file in train directory: ',os.listdir('/kaggle/working/train/')[0])\nprint('Sample of file in test directory: ',os.listdir('/kaggle/working/test/')[0])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:29.855437Z","iopub.execute_input":"2021-11-19T00:33:29.855747Z","iopub.status.idle":"2021-11-19T00:33:29.892950Z","shell.execute_reply.started":"2021-11-19T00:33:29.855705Z","shell.execute_reply":"2021-11-19T00:33:29.892124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our images in training dataset are labeled differently than we are used to see, in the current project the images contain their class in the name of the file. i.e. cat.11724.jpg means image of cat, index 11724, jpg format. For this reason we have to iterate through each image and assign its respective class according to the name as can be seen in the code below:","metadata":{}},{"cell_type":"code","source":"print('Label of image: ',os.listdir('/kaggle/working/train/')[0].split('.')[0])\nprint('Index of image: ',os.listdir('/kaggle/working/train/')[0].split('.')[1])\nprint('Format of image: ',os.listdir('/kaggle/working/train/')[0].split('.')[2])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:29.894284Z","iopub.execute_input":"2021-11-19T00:33:29.894590Z","iopub.status.idle":"2021-11-19T00:33:29.941832Z","shell.execute_reply.started":"2021-11-19T00:33:29.894554Z","shell.execute_reply":"2021-11-19T00:33:29.941045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the following line we will apply such idea were if the label corresponds to a cat the class will be '0', whereas if it's a dog the class will be '1', then the images will be read as RGB channels in resized to 150 pixels for width and height, to finally store the image and its class as one instance inside a list.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 120\nImages_train = []\nImages_label = []\nfor i in os.listdir('/kaggle/working/train/'):\n    label = i.split('.')[0]\n    if label == 'cat':\n        label = 0\n    elif label == 'dog':\n        label = 1\n    img = cv2.imread('/kaggle/working/train/'+i, cv2.IMREAD_COLOR)\n    img = cv2.resize(img,(IMG_SIZE,IMG_SIZE), interpolation=cv2.INTER_CUBIC)\n    Images_train.append([np.array(img), np.array(label)])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:33:29.943241Z","iopub.execute_input":"2021-11-19T00:33:29.943484Z","iopub.status.idle":"2021-11-19T00:34:24.664049Z","shell.execute_reply.started":"2021-11-19T00:33:29.943449Z","shell.execute_reply":"2021-11-19T00:34:24.663275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see the first instance of our training dataset preprocessed, this should contain 2 objects, the first one corresponds to the RGB image as numpy array and the second is the class 'label':","metadata":{}},{"cell_type":"code","source":"Images_train[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:24.665490Z","iopub.execute_input":"2021-11-19T00:34:24.665750Z","iopub.status.idle":"2021-11-19T00:34:24.674038Z","shell.execute_reply.started":"2021-11-19T00:34:24.665715Z","shell.execute_reply":"2021-11-19T00:34:24.673004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Images_train[0][0]","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:24.678002Z","iopub.execute_input":"2021-11-19T00:34:24.678321Z","iopub.status.idle":"2021-11-19T00:34:24.686348Z","shell.execute_reply.started":"2021-11-19T00:34:24.678235Z","shell.execute_reply":"2021-11-19T00:34:24.685567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Images_train[0][1]","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:24.687928Z","iopub.execute_input":"2021-11-19T00:34:24.688288Z","iopub.status.idle":"2021-11-19T00:34:24.695396Z","shell.execute_reply.started":"2021-11-19T00:34:24.688251Z","shell.execute_reply":"2021-11-19T00:34:24.694642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As the preprocessing was applied in the same order as the images were stored in the folder training we have to take into account that these were stored as the first 12500 images were cats and second 12500 were dogs, such sorting can make our model perform poorly and thus will be shuffled. ","metadata":{}},{"cell_type":"code","source":"import random\n\nrandom.shuffle(Images_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:24.696820Z","iopub.execute_input":"2021-11-19T00:34:24.697092Z","iopub.status.idle":"2021-11-19T00:34:24.727732Z","shell.execute_reply.started":"2021-11-19T00:34:24.697058Z","shell.execute_reply":"2021-11-19T00:34:24.727117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One we have finished the preprocessing step we have to reshape our images to a 4-dimentional array were (number of images, width, height, color channels), and our labels keep being the same:","metadata":{}},{"cell_type":"code","source":"Images = np.array([i[0] for i in Images_train]).reshape(-1,IMG_SIZE,IMG_SIZE,3)\nLabel = np.array([i[1] for i in Images_train])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:24.729167Z","iopub.execute_input":"2021-11-19T00:34:24.729352Z","iopub.status.idle":"2021-11-19T00:34:25.092507Z","shell.execute_reply.started":"2021-11-19T00:34:24.729330Z","shell.execute_reply":"2021-11-19T00:34:25.091793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Images.shape, Label.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:25.093869Z","iopub.execute_input":"2021-11-19T00:34:25.094155Z","iopub.status.idle":"2021-11-19T00:34:25.101978Z","shell.execute_reply.started":"2021-11-19T00:34:25.094118Z","shell.execute_reply":"2021-11-19T00:34:25.101245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(Label).value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:25.103194Z","iopub.execute_input":"2021-11-19T00:34:25.103927Z","iopub.status.idle":"2021-11-19T00:34:25.124054Z","shell.execute_reply.started":"2021-11-19T00:34:25.103875Z","shell.execute_reply":"2021-11-19T00:34:25.123361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's select 10 images randomly from our training dataset and show with their corresponding label:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15,15))\nfor k in range(10):\n    plt.subplot(2, 5, k+1)\n    img=random.randint(0,25000)\n    plt.imshow(Images[img])\n    plt.title('DOG' if Label[img]==1 else 'CAT')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:25.125278Z","iopub.execute_input":"2021-11-19T00:34:25.125726Z","iopub.status.idle":"2021-11-19T00:34:26.689387Z","shell.execute_reply.started":"2021-11-19T00:34:25.125691Z","shell.execute_reply":"2021-11-19T00:34:26.688683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see the images look perfect and the next step is splitting into training and validation sets to be used in modeling process:","metadata":{}},{"cell_type":"code","source":"X_train, X_val, Y_train, Y_val = train_test_split(Images, Label, test_size = 0.1)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:26.690285Z","iopub.execute_input":"2021-11-19T00:34:26.690494Z","iopub.status.idle":"2021-11-19T00:34:27.012714Z","shell.execute_reply.started":"2021-11-19T00:34:26.690467Z","shell.execute_reply":"2021-11-19T00:34:27.011964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.shape, Y_train.shape, X_val.shape, Y_val.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:27.014168Z","iopub.execute_input":"2021-11-19T00:34:27.014432Z","iopub.status.idle":"2021-11-19T00:34:27.020720Z","shell.execute_reply.started":"2021-11-19T00:34:27.014395Z","shell.execute_reply":"2021-11-19T00:34:27.019955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#X_train=X_train/255.0\n#X_val=X_val/255.0","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:27.022194Z","iopub.execute_input":"2021-11-19T00:34:27.022774Z","iopub.status.idle":"2021-11-19T00:34:27.029715Z","shell.execute_reply.started":"2021-11-19T00:34:27.022737Z","shell.execute_reply":"2021-11-19T00:34:27.029096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling:\n\nThe following models will be built and compared using their corresponding error measurements:\n\n- Convolutional Neural Network by scratch.\n- Pre-trained ResNet50.\n- Pre-trained VGG16.\n- Pre-trained EfficientNetB0.\n\nLet's import some libraries useful in the process of building the first network by scratch:","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom tensorflow.keras.layers import BatchNormalization\nfrom tensorflow.keras.optimizers import RMSprop,Adam,SGD,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:27.031188Z","iopub.execute_input":"2021-11-19T00:34:27.031505Z","iopub.status.idle":"2021-11-19T00:34:31.265158Z","shell.execute_reply.started":"2021-11-19T00:34:27.031470Z","shell.execute_reply":"2021-11-19T00:34:31.264415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to create some changes in our images we can use the ImageDataGenerator for data augmentation, as these correspond to images of animals and we know they can be in different positions we can play with lots of arguments of this function below:","metadata":{}},{"cell_type":"code","source":"datagen = ImageDataGenerator(\n    featurewise_center=False,\n    samplewise_center=False,\n    featurewise_std_normalization=False,\n    samplewise_std_normalization=False,\n    zca_whitening=False,\n    rotation_range=10,\n    zoom_range = 0.1,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=True,\n    vertical_flip=False) ","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:31.266306Z","iopub.execute_input":"2021-11-19T00:34:31.268413Z","iopub.status.idle":"2021-11-19T00:34:31.275754Z","shell.execute_reply.started":"2021-11-19T00:34:31.268382Z","shell.execute_reply":"2021-11-19T00:34:31.274629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's create two constraints or 'callbacks' which can help us improve the training (ReduceLROnPlateau) and stop the training once it has reached a high threshold (Callback):","metadata":{}},{"cell_type":"code","source":"class myCallback(tf.keras.callbacks.Callback):\n  def on_epoch_end(self, epoch, logs={}):\n    if(logs.get('val_accuracy')>0.98):\n      print(\"\\nReached 98% accuracy so cancelling training!\")\n      self.model.stop_training = True\n        \ncallbacks = myCallback()\n\nfrom keras.callbacks import ReduceLROnPlateau\nlr_reduction = ReduceLROnPlateau(monitor='val_accuracy',\n                                 patience=1, \n                                 verbose=1, \n                                 factor=0.5, \n                                 min_lr=0.000001)\n\nfrom keras.callbacks import EarlyStopping\nearly_stopping = EarlyStopping(monitor='val_accuracy', \n                               min_delta=0.005,\n                               patience=3, \n                               verbose=1, \n                               mode='auto')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:31.279587Z","iopub.execute_input":"2021-11-19T00:34:31.279834Z","iopub.status.idle":"2021-11-19T00:34:31.290171Z","shell.execute_reply.started":"2021-11-19T00:34:31.279807Z","shell.execute_reply":"2021-11-19T00:34:31.289463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following network was evaluated with four optimizers (Adam, SGD, RMSProp and Adadelta) and we got the highest train/validation accuracy using Adam, as can be seen below:","metadata":{}},{"cell_type":"code","source":"optimizer = Adam(learning_rate=0.001,beta_1=0.9,beta_2=0.999)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:31.291486Z","iopub.execute_input":"2021-11-19T00:34:31.292088Z","iopub.status.idle":"2021-11-19T00:34:31.308739Z","shell.execute_reply.started":"2021-11-19T00:34:31.292028Z","shell.execute_reply":"2021-11-19T00:34:31.308102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model=Sequential()\nmodel.add(Conv2D(64,(3,3),strides=1,padding='Same',activation='relu',input_shape=(X_train.shape[1],X_train.shape[2],3)))\nmodel.add(MaxPool2D(2,2))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128,(3,3), strides=1,padding= 'Same', activation='relu'))\nmodel.add(MaxPool2D(2,2))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128,(3,3), strides=1,padding= 'Same', activation='relu'))\nmodel.add(MaxPool2D(2,2))\nmodel.add(BatchNormalization())\nmodel.add(Flatten())\nmodel.add(Dense(1024, activation = \"relu\"))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(1, activation = \"sigmoid\"))\n\n\n#optimizer = SGD(learning_rate=0.01)\n#optimizer = RMSprop(learning_rate=0.001,rho=0.9,momentum=0.0,epsilon=1e-07)\n#optimizer = Adadelta(learning_rate=0.001,rho=0.95, epsilon=1e-07)\nmodel.compile(optimizer = optimizer , loss = \"binary_crossentropy\", metrics=[\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:16:45.967571Z","iopub.execute_input":"2021-11-18T23:16:45.968140Z","iopub.status.idle":"2021-11-18T23:16:48.830552Z","shell.execute_reply.started":"2021-11-18T23:16:45.968104Z","shell.execute_reply":"2021-11-18T23:16:48.829833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen.fit(X_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:31.309969Z","iopub.execute_input":"2021-11-19T00:34:31.310457Z","iopub.status.idle":"2021-11-19T00:34:36.428868Z","shell.execute_reply.started":"2021-11-19T00:34:31.310420Z","shell.execute_reply":"2021-11-19T00:34:36.428143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),\n                              validation_data=(X_val,Y_val), epochs=20, verbose=1,\n                              callbacks=[callbacks, lr_reduction])","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:16:54.831631Z","iopub.execute_input":"2021-11-18T23:16:54.831900Z","iopub.status.idle":"2021-11-18T23:41:37.173265Z","shell.execute_reply.started":"2021-11-18T23:16:54.831863Z","shell.execute_reply":"2021-11-18T23:41:37.172471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(history.history)","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:41:37.174709Z","iopub.execute_input":"2021-11-18T23:41:37.175054Z","iopub.status.idle":"2021-11-18T23:41:37.199928Z","shell.execute_reply.started":"2021-11-18T23:41:37.175015Z","shell.execute_reply":"2021-11-18T23:41:37.199033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def metrics_plot(history):\n  acc = history.history['accuracy']\n  val_acc = history.history['val_accuracy']\n  loss = history.history['loss']\n  val_loss = history.history['val_loss']\n\n  epochs = range(len(acc))\n\n  plt.plot(epochs, acc, 'r', label='Training accuracy')\n  plt.plot(epochs, val_acc, 'b', label='Validation accuracy')\n  plt.title('Training and validation accuracy')\n  plt.legend()\n  plt.figure()\n\n  plt.plot(epochs, loss, 'r', label='Training Loss')\n  plt.plot(epochs, val_loss, 'b', label='Validation Loss')\n  plt.title('Training and validation loss')\n  plt.legend()\n\n  plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:34:36.430276Z","iopub.execute_input":"2021-11-19T00:34:36.430527Z","iopub.status.idle":"2021-11-19T00:34:36.438746Z","shell.execute_reply.started":"2021-11-19T00:34:36.430493Z","shell.execute_reply":"2021-11-19T00:34:36.437869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics_plot(history)","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:41:37.210138Z","iopub.execute_input":"2021-11-18T23:41:37.210416Z","iopub.status.idle":"2021-11-18T23:41:37.635618Z","shell.execute_reply.started":"2021-11-18T23:41:37.210380Z","shell.execute_reply":"2021-11-18T23:41:37.634906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.models import load_model\n\nmodel.save('CNN_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:41:37.636790Z","iopub.execute_input":"2021-11-18T23:41:37.637235Z","iopub.status.idle":"2021-11-18T23:41:38.381110Z","shell.execute_reply.started":"2021-11-18T23:41:37.637195Z","shell.execute_reply":"2021-11-18T23:41:38.380243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ResNet50:\n\nEvery pre-trained network will make use of the weights belonging to imagenet.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras.applications.resnet50 import preprocess_input","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:26.905877Z","iopub.execute_input":"2021-11-18T23:49:26.906175Z","iopub.status.idle":"2021-11-18T23:49:26.913120Z","shell.execute_reply.started":"2021-11-18T23:49:26.906136Z","shell.execute_reply":"2021-11-18T23:49:26.912239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras import Model","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:26.914882Z","iopub.execute_input":"2021-11-18T23:49:26.915263Z","iopub.status.idle":"2021-11-18T23:49:26.922753Z","shell.execute_reply.started":"2021-11-18T23:49:26.915224Z","shell.execute_reply":"2021-11-18T23:49:26.922037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN=Sequential()\nmodel_RN.add(ResNet50(input_shape=(120,120,3),\n            include_top=False,\n            weights='imagenet',\n            pooling='max'))","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:26.924272Z","iopub.execute_input":"2021-11-18T23:49:26.924537Z","iopub.status.idle":"2021-11-18T23:49:31.813696Z","shell.execute_reply.started":"2021-11-18T23:49:26.924503Z","shell.execute_reply":"2021-11-18T23:49:31.812937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:35.125319Z","iopub.execute_input":"2021-11-18T23:49:35.125599Z","iopub.status.idle":"2021-11-18T23:49:35.144419Z","shell.execute_reply.started":"2021-11-18T23:49:35.125549Z","shell.execute_reply":"2021-11-18T23:49:35.143612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.layers[0].trainable=False\nmodel_RN.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:36.629346Z","iopub.execute_input":"2021-11-18T23:49:36.629632Z","iopub.status.idle":"2021-11-18T23:49:36.652977Z","shell.execute_reply.started":"2021-11-18T23:49:36.629602Z","shell.execute_reply":"2021-11-18T23:49:36.652171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.add(Dense(512,activation='relu'))\nmodel_RN.add(Dropout(0.2))\nmodel_RN.add(Dense(1,activation='sigmoid'))","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:39.556062Z","iopub.execute_input":"2021-11-18T23:49:39.556945Z","iopub.status.idle":"2021-11-18T23:49:39.582982Z","shell.execute_reply.started":"2021-11-18T23:49:39.556892Z","shell.execute_reply":"2021-11-18T23:49:39.582251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:41.355004Z","iopub.execute_input":"2021-11-18T23:49:41.355772Z","iopub.status.idle":"2021-11-18T23:49:41.375044Z","shell.execute_reply.started":"2021-11-18T23:49:41.355730Z","shell.execute_reply":"2021-11-18T23:49:41.374214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.layers","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:44.785958Z","iopub.execute_input":"2021-11-18T23:49:44.786213Z","iopub.status.idle":"2021-11-18T23:49:44.791256Z","shell.execute_reply.started":"2021-11-18T23:49:44.786183Z","shell.execute_reply":"2021-11-18T23:49:44.790505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:47.885136Z","iopub.execute_input":"2021-11-18T23:49:47.885689Z","iopub.status.idle":"2021-11-18T23:49:47.902579Z","shell.execute_reply.started":"2021-11-18T23:49:47.885654Z","shell.execute_reply":"2021-11-18T23:49:47.901889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history2 = model_RN.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),\n                                  validation_data=(X_val,Y_val), epochs=20, verbose=1,\n                                  callbacks=[callbacks, lr_reduction, early_stopping])","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:49:51.396371Z","iopub.execute_input":"2021-11-18T23:49:51.397108Z","iopub.status.idle":"2021-11-18T23:56:25.105855Z","shell.execute_reply.started":"2021-11-18T23:49:51.397069Z","shell.execute_reply":"2021-11-18T23:56:25.105110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(history2.history)","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:57:33.023092Z","iopub.execute_input":"2021-11-18T23:57:33.023374Z","iopub.status.idle":"2021-11-18T23:57:33.043115Z","shell.execute_reply.started":"2021-11-18T23:57:33.023339Z","shell.execute_reply":"2021-11-18T23:57:33.042432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics_plot(history2)","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:58:31.387096Z","iopub.execute_input":"2021-11-18T23:58:31.387301Z","iopub.status.idle":"2021-11-18T23:58:31.826100Z","shell.execute_reply.started":"2021-11-18T23:58:31.387276Z","shell.execute_reply":"2021-11-18T23:58:31.825417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_RN.save('ResNet_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-11-18T23:58:47.544559Z","iopub.execute_input":"2021-11-18T23:58:47.545246Z","iopub.status.idle":"2021-11-18T23:58:48.215263Z","shell.execute_reply.started":"2021-11-18T23:58:47.545207Z","shell.execute_reply":"2021-11-18T23:58:48.214427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## VGG16:","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import VGG16\nfrom tensorflow.keras.applications.vgg16 import preprocess_input","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:03:02.521159Z","iopub.execute_input":"2021-11-19T00:03:02.522661Z","iopub.status.idle":"2021-11-19T00:03:02.536605Z","shell.execute_reply.started":"2021-11-19T00:03:02.522623Z","shell.execute_reply":"2021-11-19T00:03:02.535918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG=Sequential()\nmodel_VGG.add(VGG16(input_shape=(120,120,3),\n                    include_top=False,\n                    pooling='max',\n                    weights='imagenet'))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:03:02.538061Z","iopub.execute_input":"2021-11-19T00:03:02.538587Z","iopub.status.idle":"2021-11-19T00:03:06.023934Z","shell.execute_reply.started":"2021-11-19T00:03:02.538549Z","shell.execute_reply":"2021-11-19T00:03:06.023001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:04:20.249436Z","iopub.execute_input":"2021-11-19T00:04:20.249989Z","iopub.status.idle":"2021-11-19T00:04:20.259901Z","shell.execute_reply.started":"2021-11-19T00:04:20.249953Z","shell.execute_reply":"2021-11-19T00:04:20.258999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.layers[0].trainable=False\nmodel_VGG.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:04:27.657260Z","iopub.execute_input":"2021-11-19T00:04:27.657798Z","iopub.status.idle":"2021-11-19T00:04:27.667097Z","shell.execute_reply.started":"2021-11-19T00:04:27.657759Z","shell.execute_reply":"2021-11-19T00:04:27.666425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.add(Dense(512,activation='relu'))\nmodel_VGG.add(Dropout(0.2))\nmodel_VGG.add(Dense(1,activation='sigmoid'))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:04:35.057019Z","iopub.execute_input":"2021-11-19T00:04:35.057553Z","iopub.status.idle":"2021-11-19T00:04:35.085965Z","shell.execute_reply.started":"2021-11-19T00:04:35.057517Z","shell.execute_reply":"2021-11-19T00:04:35.085215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:04:47.835875Z","iopub.execute_input":"2021-11-19T00:04:47.836139Z","iopub.status.idle":"2021-11-19T00:04:47.844853Z","shell.execute_reply.started":"2021-11-19T00:04:47.836109Z","shell.execute_reply":"2021-11-19T00:04:47.844074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.layers","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:04:56.004020Z","iopub.execute_input":"2021-11-19T00:04:56.004865Z","iopub.status.idle":"2021-11-19T00:04:56.012151Z","shell.execute_reply.started":"2021-11-19T00:04:56.004826Z","shell.execute_reply":"2021-11-19T00:04:56.011391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:04:59.486218Z","iopub.execute_input":"2021-11-19T00:04:59.486526Z","iopub.status.idle":"2021-11-19T00:04:59.501862Z","shell.execute_reply.started":"2021-11-19T00:04:59.486494Z","shell.execute_reply":"2021-11-19T00:04:59.500980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history3 = model_VGG.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),\n                                  validation_data=(X_val,Y_val), epochs=20, verbose=1,\n                                  callbacks=[callbacks, lr_reduction, early_stopping])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:05:05.132744Z","iopub.execute_input":"2021-11-19T00:05:05.133287Z","iopub.status.idle":"2021-11-19T00:10:17.307709Z","shell.execute_reply.started":"2021-11-19T00:05:05.133247Z","shell.execute_reply":"2021-11-19T00:10:17.306944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(history3.history)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:12:00.249202Z","iopub.execute_input":"2021-11-19T00:12:00.249814Z","iopub.status.idle":"2021-11-19T00:12:00.274627Z","shell.execute_reply.started":"2021-11-19T00:12:00.249772Z","shell.execute_reply":"2021-11-19T00:12:00.273916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics_plot(history3)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:12:05.417128Z","iopub.execute_input":"2021-11-19T00:12:05.417382Z","iopub.status.idle":"2021-11-19T00:12:24.883959Z","shell.execute_reply.started":"2021-11-19T00:12:05.417353Z","shell.execute_reply":"2021-11-19T00:12:24.883109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_VGG.save('VGG_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:12:24.886920Z","iopub.execute_input":"2021-11-19T00:12:24.887113Z","iopub.status.idle":"2021-11-19T00:12:25.033435Z","shell.execute_reply.started":"2021-11-19T00:12:24.887089Z","shell.execute_reply":"2021-11-19T00:12:25.032663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EfficientNetB0:\n\nImportant to mention that InceptionV3, Xception, ResNet152V2, DenseNet201 didn't work well, I mean reached no more than 0.75 in validation accuracy, I tried with EfficientNetB0 and finally I reached a satisfactory performance as you will see below:","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import EfficientNetB0\nfrom tensorflow.keras.applications.efficientnet import preprocess_input","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:16:13.299066Z","iopub.execute_input":"2021-11-19T00:16:13.299483Z","iopub.status.idle":"2021-11-19T00:16:13.313606Z","shell.execute_reply.started":"2021-11-19T00:16:13.299446Z","shell.execute_reply":"2021-11-19T00:16:13.312895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF=Sequential()\nmodel_EF.add(EfficientNetB0(input_shape=(120,120,3),\n                            include_top=False,\n                            pooling='max',\n                            weights='imagenet'))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:16:13.317701Z","iopub.execute_input":"2021-11-19T00:16:13.317899Z","iopub.status.idle":"2021-11-19T00:16:18.510320Z","shell.execute_reply.started":"2021-11-19T00:16:13.317869Z","shell.execute_reply":"2021-11-19T00:16:18.509539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:18:49.206527Z","iopub.execute_input":"2021-11-19T00:18:49.207267Z","iopub.status.idle":"2021-11-19T00:18:49.228171Z","shell.execute_reply.started":"2021-11-19T00:18:49.207212Z","shell.execute_reply":"2021-11-19T00:18:49.227358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.layers[0].trainable=False\nmodel_EF.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:18:51.606279Z","iopub.execute_input":"2021-11-19T00:18:51.606536Z","iopub.status.idle":"2021-11-19T00:18:51.634716Z","shell.execute_reply.started":"2021-11-19T00:18:51.606500Z","shell.execute_reply":"2021-11-19T00:18:51.633934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.add(Dense(512,activation='relu'))\nmodel_EF.add(Dropout(0.2))\nmodel_EF.add(Dense(1,activation='sigmoid'))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:18:55.030352Z","iopub.execute_input":"2021-11-19T00:18:55.031101Z","iopub.status.idle":"2021-11-19T00:18:55.058846Z","shell.execute_reply.started":"2021-11-19T00:18:55.031048Z","shell.execute_reply":"2021-11-19T00:18:55.058186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:18:56.724317Z","iopub.execute_input":"2021-11-19T00:18:56.725171Z","iopub.status.idle":"2021-11-19T00:18:56.747217Z","shell.execute_reply.started":"2021-11-19T00:18:56.725122Z","shell.execute_reply":"2021-11-19T00:18:56.746332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.layers","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:19:01.231718Z","iopub.execute_input":"2021-11-19T00:19:01.232458Z","iopub.status.idle":"2021-11-19T00:19:01.238173Z","shell.execute_reply.started":"2021-11-19T00:19:01.232409Z","shell.execute_reply":"2021-11-19T00:19:01.237252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:19:02.817235Z","iopub.execute_input":"2021-11-19T00:19:02.817482Z","iopub.status.idle":"2021-11-19T00:19:02.835342Z","shell.execute_reply.started":"2021-11-19T00:19:02.817454Z","shell.execute_reply":"2021-11-19T00:19:02.834095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history4 = model_EF.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),\n                                  validation_data=(X_val,Y_val), epochs=10, verbose=1,\n                                  callbacks=[callbacks, lr_reduction, early_stopping])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:19:05.955505Z","iopub.execute_input":"2021-11-19T00:19:05.956554Z","iopub.status.idle":"2021-11-19T00:25:30.826141Z","shell.execute_reply.started":"2021-11-19T00:19:05.956502Z","shell.execute_reply":"2021-11-19T00:25:30.824991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(history4.history)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:25:52.015948Z","iopub.execute_input":"2021-11-19T00:25:52.016224Z","iopub.status.idle":"2021-11-19T00:25:52.036319Z","shell.execute_reply.started":"2021-11-19T00:25:52.016193Z","shell.execute_reply":"2021-11-19T00:25:52.035555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics_plot(history4)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:25:57.766439Z","iopub.execute_input":"2021-11-19T00:25:57.767083Z","iopub.status.idle":"2021-11-19T00:26:39.665287Z","shell.execute_reply.started":"2021-11-19T00:25:57.767049Z","shell.execute_reply":"2021-11-19T00:26:39.664610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_EF.save('EF_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:26:39.668532Z","iopub.execute_input":"2021-11-19T00:26:39.668746Z","iopub.status.idle":"2021-11-19T00:26:40.020117Z","shell.execute_reply.started":"2021-11-19T00:26:39.668721Z","shell.execute_reply":"2021-11-19T00:26:40.019300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I'm going to load the 4 models saved and compute their corresponding metrics which should match those from the last epoch for each one.","metadata":{}},{"cell_type":"code","source":"from keras.models import load_model","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:40:25.730094Z","iopub.execute_input":"2021-11-19T00:40:25.730872Z","iopub.status.idle":"2021-11-19T00:40:25.735069Z","shell.execute_reply.started":"2021-11-19T00:40:25.730817Z","shell.execute_reply":"2021-11-19T00:40:25.734235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model1 = load_model('../input/models-saved/CNN_model.h5')\nmodel2 = load_model('../input/models-saved/ResNet_model.h5')\nmodel3 = load_model('../input/models-saved/VGG_model.h5')\nmodel4 = load_model('../input/models-saved/EF_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:51:57.947686Z","iopub.execute_input":"2021-11-19T00:51:57.947949Z","iopub.status.idle":"2021-11-19T00:52:09.450614Z","shell.execute_reply.started":"2021-11-19T00:51:57.947918Z","shell.execute_reply":"2021-11-19T00:52:09.449821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loss_cnn, train_acc_cnn = model1.evaluate(X_train,  Y_train, verbose=2)\ntest_loss_cnn, test_acc_cnn = model1.evaluate(X_val,  Y_val, verbose=2)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:52:09.452477Z","iopub.execute_input":"2021-11-19T00:52:09.452735Z","iopub.status.idle":"2021-11-19T00:52:16.203104Z","shell.execute_reply.started":"2021-11-19T00:52:09.452700Z","shell.execute_reply":"2021-11-19T00:52:16.202347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loss_rn, train_acc_rn = model2.evaluate(X_train,  Y_train, verbose=2)\ntest_loss_rn, test_acc_rn = model2.evaluate(X_val,  Y_val, verbose=2)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:52:16.205739Z","iopub.execute_input":"2021-11-19T00:52:16.206343Z","iopub.status.idle":"2021-11-19T00:52:37.090249Z","shell.execute_reply.started":"2021-11-19T00:52:16.206312Z","shell.execute_reply":"2021-11-19T00:52:37.089482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loss_vgg, train_acc_vgg = model3.evaluate(X_train,  Y_train, verbose=2)\ntest_loss_vgg, test_acc_vgg = model3.evaluate(X_val,  Y_val, verbose=2)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:52:37.092226Z","iopub.execute_input":"2021-11-19T00:52:37.092561Z","iopub.status.idle":"2021-11-19T00:52:55.316807Z","shell.execute_reply.started":"2021-11-19T00:52:37.092521Z","shell.execute_reply":"2021-11-19T00:52:55.316048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loss_ef, train_acc_ef = model4.evaluate(X_train,  Y_train, verbose=2)\ntest_loss_ef, test_acc_ef = model4.evaluate(X_val,  Y_val, verbose=2)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T00:52:55.318112Z","iopub.execute_input":"2021-11-19T00:52:55.318444Z","iopub.status.idle":"2021-11-19T00:53:20.328153Z","shell.execute_reply.started":"2021-11-19T00:52:55.318405Z","shell.execute_reply":"2021-11-19T00:53:20.327372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Given the computed metrics for every model loaded I will summarize and show a table which can facilitate comparing them:","metadata":{}},{"cell_type":"code","source":"data = {'Scratch model':[train_acc_cnn,train_loss_cnn,test_acc_cnn,test_loss_cnn],\n        'ResNet50':[train_acc_rn,train_loss_rn,test_acc_rn,test_loss_rn],\n        'VGG16': [train_acc_vgg,train_loss_vgg,test_acc_vgg,test_loss_vgg],\n        'EfficientNet': [train_acc_ef,train_loss_ef,test_acc_ef,test_loss_ef]}\n \npd.DataFrame(data, index=['Train accuracy','Train loss','Val accuracy','Val loss'])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:02:04.921271Z","iopub.execute_input":"2021-11-19T01:02:04.921972Z","iopub.status.idle":"2021-11-19T01:02:04.945048Z","shell.execute_reply.started":"2021-11-19T01:02:04.921932Z","shell.execute_reply":"2021-11-19T01:02:04.944324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the table above we can see the model build by scrath is the one with lowest performance, despite it is not too far from the others the difference is significant. About the pre-trained models again there is not a big difference, but something important to look and take into account is that when we trained these models the metrics were different than in the table above, the reason of such change could be the stochastic nature of convolutional networks have when evaluating on different batches. Having said that I will continue with VGG16 for having the best accuracy on training and validation sets, obviously you can choose whatever you want, but such model outstandingly!\n\nLet's see the summary of the model chosen:","metadata":{}},{"cell_type":"code","source":"model3.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:18:42.369924Z","iopub.execute_input":"2021-11-19T01:18:42.370619Z","iopub.status.idle":"2021-11-19T01:18:42.377834Z","shell.execute_reply.started":"2021-11-19T01:18:42.370578Z","shell.execute_reply":"2021-11-19T01:18:42.377115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_val_prob = model3.predict(X_val, batch_size=32)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:18:52.971748Z","iopub.execute_input":"2021-11-19T01:18:52.972036Z","iopub.status.idle":"2021-11-19T01:18:54.622174Z","shell.execute_reply.started":"2021-11-19T01:18:52.971996Z","shell.execute_reply":"2021-11-19T01:18:54.621388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Remember that as this project is binary classification the last layer had sigmoid activation function, therefore our predicted output will have values of probabilities between 0-1, where 1 means 'Dog' and 0 means 'Cat':","metadata":{}},{"cell_type":"code","source":"predicted_val_prob","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:18:55.472955Z","iopub.execute_input":"2021-11-19T01:18:55.473626Z","iopub.status.idle":"2021-11-19T01:18:55.478946Z","shell.execute_reply.started":"2021-11-19T01:18:55.473588Z","shell.execute_reply":"2021-11-19T01:18:55.478286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Once we have this probabilities we have to convert it to discrete values, either 1 or 0, such task can be achieved by using the np.round function:","metadata":{}},{"cell_type":"code","source":"Y_val_pred= np.round(predicted_val_prob)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:18:56.719720Z","iopub.execute_input":"2021-11-19T01:18:56.720432Z","iopub.status.idle":"2021-11-19T01:18:56.723926Z","shell.execute_reply.started":"2021-11-19T01:18:56.720391Z","shell.execute_reply":"2021-11-19T01:18:56.723241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_val_pred","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:18:58.942494Z","iopub.execute_input":"2021-11-19T01:18:58.942942Z","iopub.status.idle":"2021-11-19T01:18:58.968479Z","shell.execute_reply.started":"2021-11-19T01:18:58.942894Z","shell.execute_reply":"2021-11-19T01:18:58.962520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see the classification report for this model, we can see the considerably high metrics, later we will in more detail why every model didn't achieve a perfect performance.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report\n\nreport = classification_report(Y_val, Y_val_pred)\n\nprint(report)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:19:01.533744Z","iopub.execute_input":"2021-11-19T01:19:01.534039Z","iopub.status.idle":"2021-11-19T01:19:01.550621Z","shell.execute_reply.started":"2021-11-19T01:19:01.533998Z","shell.execute_reply":"2021-11-19T01:19:01.549709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\nf,ax = plt.subplots(figsize=(15, 15))\nconfusion_mtx = confusion_matrix(Y_val, Y_val_pred)\nsns.set(font_scale=1.4)\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.01,cmap=\"Greens\",linecolor=\"gray\",ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix Validation set\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:19:05.633849Z","iopub.execute_input":"2021-11-19T01:19:05.634128Z","iopub.status.idle":"2021-11-19T01:19:05.929544Z","shell.execute_reply.started":"2021-11-19T01:19:05.634095Z","shell.execute_reply":"2021-11-19T01:19:05.928917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model which had the best performance was the VGG16 with 119 misclassification out of 2500 instances, as I said before such number can change due to stochastic nature of these models, since now I will use VGG16 to predict the classes of instances stored in testing folder, but firstly let's plot a sample of the images misclassified and see if there is a pattern or general reason of this problem.","metadata":{}},{"cell_type":"code","source":"l = []\nfor i in range(len(Y_val_pred)):\n    if Y_val[i] != Y_val_pred[i]:\n        l.append(i)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:30:03.294066Z","iopub.execute_input":"2021-11-19T01:30:03.294819Z","iopub.status.idle":"2021-11-19T01:30:03.306290Z","shell.execute_reply.started":"2021-11-19T01:30:03.294775Z","shell.execute_reply":"2021-11-19T01:30:03.305518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Number of misclassifications in validation dataset: ', len(l))","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:30:17.839386Z","iopub.execute_input":"2021-11-19T01:30:17.840130Z","iopub.status.idle":"2021-11-19T01:30:17.845074Z","shell.execute_reply.started":"2021-11-19T01:30:17.840086Z","shell.execute_reply":"2021-11-19T01:30:17.844043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The list 'l' is storing the indexes of those instances which were misclassified, having said that let's see the first 20 misclassifications and then show these images with their corresponding sigmoid output (probability):","metadata":{}},{"cell_type":"code","source":"l[:20]","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:31:26.877790Z","iopub.execute_input":"2021-11-19T01:31:26.878251Z","iopub.status.idle":"2021-11-19T01:31:26.889134Z","shell.execute_reply.started":"2021-11-19T01:31:26.878190Z","shell.execute_reply":"2021-11-19T01:31:26.888038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(35,35))\nc = 1\nfor i in l[:20]:\n    plt.subplot(4,5, c)\n    plt.imshow(X_val[i])\n    plt.title('DOG:{}\\nTrue label:{}'.format(predicted_val_prob[i], Y_val[i])\n              if predicted_val_prob[i]>= 0.5 else 'CAT:{}\\nTrue label:{}'.format(predicted_val_prob[i],Y_val[i]))\n    plt.axis('off')\n    c = c+1","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:31:36.543576Z","iopub.execute_input":"2021-11-19T01:31:36.544091Z","iopub.status.idle":"2021-11-19T01:31:40.706474Z","shell.execute_reply.started":"2021-11-19T01:31:36.544049Z","shell.execute_reply":"2021-11-19T01:31:40.704062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above we see only 20 misclassifications, what is more worrying is that some of these images had a relatively high probability of prediction (let's say ~0.8 and ~0.2), which is bad if we want to have a high accuracy when predicting the classes of testing images. One big reason could be that some of them belong to uncommon or less frequent breeds of dogs and cats, thus there are present only a few pictures of them. This problem could be solved by adding more images in order to balance the breeds in the training set. Another big problem is that some of the images do not contain only one animal or 'object', some of them contain humans close to animals, some contain cats and dogs in same image, some contain multiple images, etc. This without a doubt sidetracks the prediction of every model and for such images the method to use should be 'Object detection', one of the well-known models are YOLO's which as output of any image highlights each object contained drawing a bounding box and the class predicted with corresponding probability.\n\nAs I said in order to correct such problems YOLO approaches should be used and this is a challenge for a next project, which I will be working on.\n\nAs a final step let's compute and show the error metrics (recall, precision, f1-score, accuracy and AUC) for the best model.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report\nfrom sklearn.metrics import precision_recall_fscore_support as score\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.preprocessing import label_binarize","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:35:37.586249Z","iopub.execute_input":"2021-11-19T01:35:37.586732Z","iopub.status.idle":"2021-11-19T01:35:37.590970Z","shell.execute_reply.started":"2021-11-19T01:35:37.586692Z","shell.execute_reply":"2021-11-19T01:35:37.590284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics = []\nmodels = ['VGG16 model']\npredictions=[Y_val_pred]\n\nfor lab,i in zip(models, predictions):\n    precision, recall, fscore, _ = score(Y_val, i, average='weighted')\n    accuracy = accuracy_score(Y_val, i)\n    auc = roc_auc_score(label_binarize(Y_val, classes=[0,1]),\n                        label_binarize(i, classes=[0,1]),\n                        average='weighted')\n    metrics.append(pd.Series({'precision':precision, 'recall':recall,\n                              'fscore':fscore, 'accuracy':accuracy,\n                              'auc':auc}, name=lab))\n    \nmetrics = pd.concat(metrics, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:36:02.986044Z","iopub.execute_input":"2021-11-19T01:36:02.986718Z","iopub.status.idle":"2021-11-19T01:36:03.006191Z","shell.execute_reply.started":"2021-11-19T01:36:02.986678Z","shell.execute_reply":"2021-11-19T01:36:03.005472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:36:04.765229Z","iopub.execute_input":"2021-11-19T01:36:04.765759Z","iopub.status.idle":"2021-11-19T01:36:04.773554Z","shell.execute_reply.started":"2021-11-19T01:36:04.765721Z","shell.execute_reply":"2021-11-19T01:36:04.772813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction of testing images:","metadata":{}},{"cell_type":"markdown","source":"Using the os.listdir function let's print the name of some images contained in the test folder: ","metadata":{}},{"cell_type":"code","source":"os.listdir('/kaggle/working/test/')[:20] ","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:36:40.431077Z","iopub.execute_input":"2021-11-19T01:36:40.431624Z","iopub.status.idle":"2021-11-19T01:36:40.448201Z","shell.execute_reply.started":"2021-11-19T01:36:40.431584Z","shell.execute_reply":"2021-11-19T01:36:40.447552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As the testing dataset does not contain the labels included in the name of the image we will see how they are presented printing one sample of them:","metadata":{}},{"cell_type":"code","source":"print('Sample of image: ',os.listdir('/kaggle/working/test/')[0])\nprint('Index of image: ',os.listdir('/kaggle/working/test/')[0].split('.')[0])\nprint('Format of image: ',os.listdir('/kaggle/working/test/')[0].split('.')[1])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:36:57.092849Z","iopub.execute_input":"2021-11-19T01:36:57.093203Z","iopub.status.idle":"2021-11-19T01:36:57.137686Z","shell.execute_reply.started":"2021-11-19T01:36:57.093163Z","shell.execute_reply":"2021-11-19T01:36:57.136971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following function will extract the index of the image and the image as numpy array and save both as an instance in the list Images_test:","metadata":{}},{"cell_type":"code","source":"Images_test = []\nfor j in os.listdir('/kaggle/working/test/'):\n    index = j.split('.')[0]\n    img = cv2.imread('/kaggle/working/test/'+j, cv2.IMREAD_COLOR)\n    img = cv2.resize(img,(IMG_SIZE,IMG_SIZE), interpolation = cv2.INTER_CUBIC)\n    Images_test.append([np.array(img), np.array(index)])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:37:11.938115Z","iopub.execute_input":"2021-11-19T01:37:11.938858Z","iopub.status.idle":"2021-11-19T01:37:39.610689Z","shell.execute_reply.started":"2021-11-19T01:37:11.938808Z","shell.execute_reply":"2021-11-19T01:37:39.609898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The numpy arrays representing the images will be stored in a new variable 'X_test' from which will be predicted the class. ","metadata":{}},{"cell_type":"code","source":"X_test = np.array([j[0] for j in Images_test]).reshape(-1,IMG_SIZE, IMG_SIZE, 3)\nIndex = np.array([j[1] for j in Images_test])","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:37:53.738498Z","iopub.execute_input":"2021-11-19T01:37:53.738762Z","iopub.status.idle":"2021-11-19T01:37:53.921365Z","shell.execute_reply.started":"2021-11-19T01:37:53.738730Z","shell.execute_reply":"2021-11-19T01:37:53.920458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_prediction = model3.predict(X_test, batch_size = 32)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T01:37:59.666384Z","iopub.execute_input":"2021-11-19T01:37:59.666650Z","iopub.status.idle":"2021-11-19T01:38:08.269726Z","shell.execute_reply.started":"2021-11-19T01:37:59.666616Z","shell.execute_reply":"2021-11-19T01:38:08.268968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission=pd.DataFrame(test_prediction, columns=['label'], index=pd.Series(Index, name='id'))\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-19T02:06:07.330813Z","iopub.execute_input":"2021-11-19T02:06:07.331093Z","iopub.status.idle":"2021-11-19T02:06:07.346148Z","shell.execute_reply.started":"2021-11-19T02:06:07.331055Z","shell.execute_reply":"2021-11-19T02:06:07.345319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T02:06:09.506374Z","iopub.execute_input":"2021-11-19T02:06:09.506906Z","iopub.status.idle":"2021-11-19T02:06:09.536314Z","shell.execute_reply.started":"2021-11-19T02:06:09.506868Z","shell.execute_reply":"2021-11-19T02:06:09.535632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I would like to know any feedback in order to increase the performance of the models or tell me if you found a different one even better!\n\nIf you liked this notebook I would appreciate so much your upvote if you want to see more projects/tutorials like this one. I encourage you to see my projects portfolio, am sure you will love it.\n\nThank you!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}