{"cells":[{"metadata":{},"cell_type":"markdown","source":"Contents\n==\n\n[Introduction to The Kernel](#intro)<br/>\n[1. Preprocessing Data](#pre) <br/>\n&emsp;&emsp; [1.1 Organizing & Splitting Data](#reshape) <br/>\n&emsp;&emsp; [1.2 Augmenting Data](#augment)<br/>\n[2. Building The Model](#model) <br/>\n&emsp;&emsp; [2.1 Model's Callbacks](#call)<br/>\n&emsp;&emsp; [2.2 Model Architecture](#arch)<br/>\n&emsp;&emsp; [2.3 Hyperparameters Tuning](#params)<br/>\n&emsp;&emsp; [2.4 Model Training](#train)<br/>\n[3. Model Evaluation](#eval) <br/>\n[4. Submission](#submit)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"intro\"></a>\n# Introduction to The Kernel\nIn this kernel, I'll use a pretrained encoder from another architecture trained on ImageNet and will just train the the decoder part which consist of multiple fully connected layers with their activations and dropouts.\nCNNs are the best choice when it comes to image classification or any computer vision task, as well as providing relatively small number of weight parameters compared to MLP"},{"metadata":{},"cell_type":"markdown","source":"## Steps of The Kernel\n1. Preprocessing the data, which includes organizing, reshaping, normalization, and images augmentation\n2. Building the model, trying different architectures and hyperparameters tuning\n3. Model Evaluation and calculating the metrics of the model's performance"},{"metadata":{},"cell_type":"markdown","source":"## Importing Python Modules\n* **Data Handling:** numpy, pandas\n* **DL Model Building:** keras \n* **Preprocessing:** keras, sklearn, cv2\n* **Visualization:** matplotlib, seaborn, IPython.display"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","id":"UE-pbpUHm46s","trusted":true,"outputId":"35c3eb2a-f09f-4c52-8843-507d06ed8325","executionInfo":{"status":"ok","timestamp":1585739582682,"user_tz":-120,"elapsed":40292,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# For data handling and manipulation\nimport numpy as np\nimport pandas as pd\nimport cv2\n\n# For Visualiztion\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom IPython.display import clear_output\n\n\n# for model building and trining\nfrom keras import backend as K\nfrom keras.layers import Dense, Activation, Dropout, BatchNormalization, Input, Flatten, Conv2D, MaxPooling2D, Lambda, UpSampling2D, Concatenate\nfrom keras.models import Model\nfrom keras.optimizers import Adam\nfrom keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2\nfrom keras.initializers import he_normal\nfrom keras.preprocessing.image import ImageDataGenerator\n\n# For organizing data\nimport os\nimport shutil","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that any command preceeded by !, is a bash command"},{"metadata":{"id":"5JP7R1CudAkn","outputId":"822f1de5-0b03-45cd-b2f4-9c731ac58ce6","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739586054,"user_tz":-120,"elapsed":43411,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# view sample images filenames - bash command\n!ls ../input/dog-breed-identification/train/ -U | head -5 ","execution_count":null,"outputs":[]},{"metadata":{"id":"tbAm_aZDEMCN","outputId":"73efc5c9-c9c2-4e5d-cd8a-3cb45499e566","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739586056,"user_tz":-120,"elapsed":43179,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# Firstly, let's use pandas to load the labels and see how they are mapped to data\nlabels = pd.read_csv('../input/dog-breed-identification/labels.csv')\nlabels.head(5)","execution_count":null,"outputs":[]},{"metadata":{"id":"SXn8uytjZ7Mz","outputId":"e0a0aaf7-5ac5-4c1c-ae79-fd203ec64e64","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739586058,"user_tz":-120,"elapsed":42981,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# get number of available classes\nclasses = np.unique(labels.breed)\nclasses_num = classes.size\nclasses_num","execution_count":null,"outputs":[]},{"metadata":{"id":"OkeqeB8-mDft","outputId":"0945cd9a-a511-431d-a8da-2d4669acb804","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739586060,"user_tz":-120,"elapsed":42728,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"train_dir = '../input/dog-breed-identification/train'  # the images directory\nimages_names = os.listdir(train_dir)  # names of the files in the directory\nimages_num = len(images_names)\nprint(f'Number of images: {images_num}')  # Number of training images","execution_count":null,"outputs":[]},{"metadata":{"id":"YNRnmyKFaDrI"},"cell_type":"markdown","source":"Since there's a large number of images given the kernel's resources, we'll just make keras feed the model from the directory directly instead of loading them into an np.array\n\n**But Firstly we need to organize the data.**"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"pre\"></a>\n# 1. Preprocessing Data\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"reshape\"></a>\n## 1.1 Organizing & Splitting Data\n\nThe first step to do is to split the data into training, validation, and test sets, and in each of them the images are organized in subdirectories corresponging to their labels.\n"},{"metadata":{"id":"D5HR8NNgRebh","trusted":true},"cell_type":"code","source":"new_train_dir = '/root/new_train/'  # parent directoiry of the training set\nnew_test_dir = '/root/new_test/'  # parent directory of the validation set\nnew_valid_dir = '/root/new_valid/'  # parent directory of the test set\n!mkdir $new_train_dir\n!mkdir $new_test_dir\n!mkdir $new_valid_dir","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"GVPiYjTCqF8w","outputId":"c67af022-c883-4357-a18d-15b4dff706aa","executionInfo":{"status":"ok","timestamp":1585739597485,"user_tz":-120,"elapsed":53083,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# for each of the parent directories of the sets, we'll create subdirectories for the breeds\nfor sub_dir in classes:\n    os.mkdir(new_train_dir+sub_dir)\n    os.mkdir(new_test_dir+sub_dir)\n    os.mkdir(new_valid_dir+sub_dir)\n!ls $new_train_dir","execution_count":null,"outputs":[]},{"metadata":{"id":"tAvjQzeEVbsH","outputId":"6c2dba82-0300-46c7-b234-684f88fb1876","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739597487,"user_tz":-120,"elapsed":52890,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"labels_jpg = labels.copy(deep=True)\nlabels_jpg['id'] += '.jpg'  # add .jpg to each image id to get its filename\n\n# group the images filenames of each breed\ngrouped_ids = labels_jpg.groupby('breed')['id'].apply(list).to_dict()\nprint(classes[0], grouped_ids[classes[0]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"fxyzK4WnqF84"},"cell_type":"code","source":"# specify the required split ratios\ntest_split = 0.1\nvalid_split = 0.2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Splitting the Data**"},{"metadata":{"id":"knir-76OeD8S","outputId":"9ea063b3-8619-4258-e127-900e62394fc5","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739598862,"user_tz":-120,"elapsed":13,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# iterators to track the final sizes of the sets\ntrain_size = 0\nvalid_size = 0\ntest_size = 0\n\n# loop on the images of each breed and using the defined probabilities assign each image to one of the 3 sets\nfor breed_idx, (breed, breed_images) in enumerate(grouped_ids.items()):\n    for img in breed_images:\n        rnd_prob = np.random.rand()  # give the current image a random number in the range [0, 1]\n        if rnd_prob <= test_split: \n            # copy to the corresponding breed subdirectory in the test directory\n            shutil.copy(train_dir+'/'+img, new_test_dir+'/'+breed) \n            test_size += 1\n            \n        elif rnd_prob <= (test_split + valid_split):\n            # copy to the corresponding breed subdirectory in the validation directory\n            shutil.copy(train_dir+'/'+img, new_valid_dir+'/'+breed)\n            valid_size += 1\n            \n        else:\n            # copy to the corresponding breed subdirectory in the training directory\n            shutil.copy(train_dir+'/'+img, new_train_dir+'/'+breed)\n            train_size += 1\n            \n    clear_output(wait=True)\n    print(f'Organized {breed_idx+1} out of {classes_num} breeds: {breed}')","execution_count":null,"outputs":[]},{"metadata":{"id":"rhiAymaECWAi","outputId":"85ddb587-58dc-4ede-e8c7-b0d19a4a6e66","executionInfo":{"status":"ok","timestamp":1585739598863,"user_tz":-120,"elapsed":53567,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}},"trusted":true},"cell_type":"code","source":"# let's check the final sizes of the sets\nprint(train_size, valid_size, test_size)","execution_count":null,"outputs":[]},{"metadata":{"id":"YtD3lE_QIONZ","trusted":true,"outputId":"eb13340d-6935-4e2b-a634-fb9aa8d568a9","executionInfo":{"status":"ok","timestamp":1585739602884,"user_tz":-120,"elapsed":57403,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# let's check if the organizing process ended as we intended\ntest_breed = classes[0]\n!ls $new_train_dir/$test_breed | head -5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"augment\"></a>\n## 1.4 Augmenting Data"},{"metadata":{"id":"UykCyUoOhdtt","trusted":true},"cell_type":"code","source":"# dimensions of images to use for plt.imshow\nwidth, height, channels = 512, 512, 3","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","id":"QxcXkBAom46x","trusted":true},"cell_type":"code","source":"# initialize images and labels samples\nimages_samples = np.zeros((4, height, width, 3), dtype=float)\nsamples_labels = []\n\n# get random 4 images\nrnd_indexes = np.random.randint(0, images_num, 4)\nfor i, rnd_idx in enumerate(rnd_indexes):\n    img_filename = images_names[rnd_idx]\n    img_id = img_filename[:-4]\n    img_bgr = cv2.imread(train_dir + '/' + img_filename)  # loads the images channels in (blue, green, red) order\n    images_samples[i] = cv2.resize(src=img_bgr[:, :, [2, 1, 0]], dsize=(width, height)) / 255  # store the random image\n    img_label = labels.breed[labels.id == img_id].values[0]\n    samples_labels.append(img_label)  # store the random images' label","execution_count":null,"outputs":[]},{"metadata":{"id":"h_1PUQ5gm46-","trusted":true,"outputId":"87ad17e6-ab81-4f8c-9952-bfa5dbc76d46","executionInfo":{"status":"ok","timestamp":1585739608594,"user_tz":-120,"elapsed":62245,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# view the 4 samples\nfig, axs = plt.subplots(1, 4, figsize=(20, 5))\nfor ax, img, label in zip(axs.ravel(), images_samples, samples_labels):\n    ax.imshow(img)\n    ax.axis('off')\n    ax.set_title(f'Class: {label}', size=15);","execution_count":null,"outputs":[]},{"metadata":{"id":"n3ga0IE2aqxl"},"cell_type":"markdown","source":"Note that there are mislabelled data in the dataset so don't worry if you see an image with the wrong label. You just got unlucky while picking random samples."},{"metadata":{"id":"EUmdySDHm47h","trusted":true},"cell_type":"code","source":"norm_factor = 1 / 255\n\n# Augmentation Ranges\ntransform_params = {\n    'featurewise_center': False,\n    'featurewise_std_normalization': False,\n    'samplewise_center': False,\n    'samplewise_std_normalization': False,\n    'rotation_range': 30, \n    'width_shift_range': 0.15,\n    'height_shift_range': 0.15,\n    'horizontal_flip': True,\n    'rescale': norm_factor\n}\n\n# the generator used for training - gives augmented images\nimg_gen = ImageDataGenerator(**transform_params) ","execution_count":null,"outputs":[]},{"metadata":{"id":"fSzXtTmHTtc1","trusted":true},"cell_type":"code","source":"# the generator used for validaiton - gives the images unchanged so that the validation error becomes a good\n# indication of the test error\nimg_feed = ImageDataGenerator(rescale=1/255)","execution_count":null,"outputs":[]},{"metadata":{"id":"_lITVFYVm47o","trusted":true,"outputId":"3900ca9e-bb1c-4dd6-9a21-bea4ca612605","executionInfo":{"status":"ok","timestamp":1585739624098,"user_tz":-120,"elapsed":77043,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"fig, axs = plt.subplots(2, 4, figsize=(20,10))  # let's see 4 augmentation examples\nfig.suptitle('Augmentation Results', size=32)\n\nfor axs_col, img in enumerate(images_samples):\n    viz_transoform_params = {  # defined each iteration to get new augmentation values each time\n        'theta': np.random.randint(-transform_params['rotation_range'], transform_params['rotation_range']),\n        'tx': np.random.uniform(0, transform_params['width_shift_range']),\n        'ty': np.random.uniform(0, transform_params['height_shift_range']),\n        'flip_horizontal': np.random.choice([True, False], p=[0.5, 0.5])\n    }\n\n    aug_img = img_gen.apply_transform(img, viz_transoform_params)  # the same image after augmentation\n    \n    axs[0, axs_col].imshow(img);\n    axs[0, axs_col].axis('off')\n    axs[0, axs_col].set_title('Original Image', size=15)\n    \n    axs[1, axs_col].imshow(aug_img);\n    axs[1, axs_col].axis('off')\n    axs[1, axs_col].set_title('Augmented Image', size=15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"model\"></a>\n# 2. Building The Model"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"call\"></a>\n## 2.1 Model's Callback"},{"metadata":{"id":"35sWNtvlm48P","trusted":true},"cell_type":"code","source":"# used to plot training curves (accuracy, loss) while model is training\nclass Plotter(Callback):\n    def plot(self):  # Updates the graph\n        clear_output(wait=True)\n        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 8))\n        \n        # plot the losses\n        ax1.plot(self.epochs, self.losses, label='train_loss')\n        ax1.plot(self.epochs, self.val_losses, label='val_loss')\n        \n        # plot the accuracies\n        ax2.plot(self.epochs, self.acc, label='train_acc')\n        ax2.plot(self.epochs, self.val_acc, label='val_acc')\n    \n        ax1.set_title(f'Loss vs Epochs')\n        ax1.set_xlabel(\"Epochs\")\n        ax1.set_ylabel(\"Loss\")\n        \n        ax2.set_title(f'Accuracy vs Epochs')\n        ax2.set_xlabel(\"Epoches\")\n        ax2.set_ylabel(\"Accuracy\")\n        \n        ax1.legend()\n        ax2.legend()\n        plt.show()\n        \n        # print out the accuracies at each epoch\n        print(f'Epoch #{self.epochs[-1]+1} >> train_acc={self.acc[-1]*100:.3f}%, train_loss={self.losses[-1]:.5f}')\n        print(f'Epoch #{self.epochs[-1]+1} >> val_acc={self.val_acc[-1]*100:.3f}%, val_loss={self.val_losses[-1]:.5f}')\n        \n    def on_train_begin(self, logs={}):\n        # initialize lists to store values from training\n        self.losses = []\n        self.val_losses = []\n        self.epochs = []\n        self.batch_no = []\n        self.acc = []\n        self.val_acc = []\n    \n    def on_epoch_end(self, epoch, logs={}):\n        # append values from the last epoch\n        self.losses.append(logs.get('loss'))\n        self.val_losses.append(logs.get('val_loss'))\n        self.acc.append(logs.get('acc'))\n        self.val_acc.append(logs.get('val_acc'))\n        self.epochs.append(epoch)\n        self.plot()  # update the graph\n        \n    def on_train_end(self, logs={}):\n        self.plot()\n        \n    def load_plot_data(self, data):\n        self.losses, self.val_losses, self.epochs, self.batch_no, self.acc, self.val_acc = data\n    \n    def get_plot_data(self):\n        return [self.losses, self.val_losses, self.epochs, self.batch_no, self.acc, self.val_acc]\n               \nplotter = Plotter()","execution_count":null,"outputs":[]},{"metadata":{"id":"aoPAkm6ym48L","trusted":true},"cell_type":"code","source":"# used to decrease the learning rate if val_acc doesn't enhance\nplateau_reduce = ReduceLROnPlateau(monitor='val_loss', factor=0.01,\n                              patience=1, min_lr=1e-20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"GrS4aYSDqF9s"},"cell_type":"code","source":"# not used for early stopping, but to rollback to the best weights obtained during training\ne_stop = EarlyStopping(monitor='val_loss', patience=15, mode='min', restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"YidIMiyjm48U","trusted":true},"cell_type":"code","source":"callbacks = [plotter, plateau_reduce, e_stop]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"arch\"></a>\n## 2.2 Model Architecture"},{"metadata":{},"cell_type":"markdown","source":"Because training large models can be very computationally expensive, we'll use transfer learning and load the encoder layers from a pretrained model on a similar dataset"},{"metadata":{"id":"EYegF7sim47z","trusted":true},"cell_type":"code","source":"# a Fully connected layer with activation, batchnorm and dropout\ndef dense_block(x, neurons, layer_no):\n    x = Dense(neurons, kernel_initializer=he_normal(layer_no), name=f'topDense{layer_no}')(x)\n    x = Activation('relu', name=f'Relu{layer_no}')(x)\n    x = BatchNormalization(name=f'BatchNorm{layer_no}')(x)\n    x = Dropout(0.5, name=f'Dropout{layer_no}')(x)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"id":"l1QYGqp0m473","trusted":true},"cell_type":"code","source":"def create_model(shape):\n    input_layer = Input(shape, name='input_layer')  # input layer with given shape\n    \n    # load InceptionResNetV2 with initialized weights and remove final dense layers - frozen layers\n    incep_res = InceptionResNetV2(include_top=False, weights='imagenet', input_tensor=input_layer)\n    for layer in incep_res.layers:\n        layer.trainable = False\n\n    # pooling to reduce dimensionality of each feature map\n    pool = MaxPooling2D(pool_size=[3, 3], strides=[3, 3], padding='same')(incep_res.output)\n    flat1 = Flatten(name='Flatten1')(pool)\n    flat1_bn = BatchNormalization(name='BatchNormFlat')(flat1)\n \n    # dense layers after the InceptionResNetV2 initialized layers\n    dens1 = dense_block(flat1_bn, neurons=512, layer_no=1)\n    dens2 = dense_block(dens1, neurons=512, layer_no=2)\n    dens3 = dense_block(dens2, neurons=1024, layer_no=3)\n    \n    dens_final = Dense(classes_num, name='Dense4')(dens3)\n    output_layer = Activation('softmax', name='Softmax')(dens_final)\n    \n    model = Model(inputs=[input_layer], outputs=[output_layer])\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"params\"></a>\n## 2.3 Hyperparameters Tuning"},{"metadata":{"id":"ue17z2dYR-6q","trusted":true},"cell_type":"code","source":"# hyperparameters\nheight, width, channels_num = 512, 512, 3\nlearning_rate = 0.004\nepochs = 15\nbatch_size = 32  # if increased you may run out of gpu memory - reduce image size if to increase the batch size","execution_count":null,"outputs":[]},{"metadata":{"id":"ne2rrapUm48C","outputId":"bfedcc3e-23e3-45a2-fb6d-aaf9cdec7d7e","trusted":true,"executionInfo":{"status":"ok","timestamp":1585739633567,"user_tz":-120,"elapsed":81430,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"model = create_model((height, width, channels_num))\noptimizer = Adam(learning_rate)\n\nmodel.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['acc'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"train\"></a>\n## 2.4 Model Training\nInstead of training the model here in kaggle, I've trained it in Colab and will use the weights from there due to limited GPU Quota and availability of higher resources on colab\n\nNote that I've used the same architecture, hyperparameters, and callbacks as the ones in this notebook"},{"metadata":{"id":"cQjS2O0lm48f","trusted":true,"outputId":"4adcc0c1-f450-4863-ace8-668ac708e281","executionInfo":{"status":"ok","timestamp":1585745317045,"user_tz":-120,"elapsed":43,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# used to feed the model augmented training data after being loaded from the directory\ntrain_gen = img_gen.flow_from_directory(directory=new_train_dir, target_size=(height, width), color_mode='rgb', classes=list(classes), \n                                        class_mode='categorical', batch_size=batch_size, shuffle=True, interpolation='nearest')\n\n# used to feed the model augmented validation data after being loaded from the directory\nvalid_gen = img_feed.flow_from_directory(directory=new_valid_dir, target_size=(height, width), color_mode='rgb', classes=list(classes), \n                                        class_mode='categorical', batch_size=batch_size, shuffle=True, interpolation='nearest')\n\n\n# # I'll load the decoder layers weights from the same model I trained on colab\n# # fit the model using the defined generators\n# model.fit_generator(train_gen, validation_data=valid_gen, epochs=epochs, \n#                         steps_per_epoch=train_size//batch_size + 1, \n#                         validation_steps=valid_size//batch_size + 1, callbacks=callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are the training curves I obtained from trining this model\n\nThe plot data are saved in a numpy file"},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget \"https://drive.google.com/uc?id=1-3Y-DB5uhOaY69pvVl5rmfirw9aKCYYB&export=download\" -O '/root/training_curves.npy'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_data = np.load('/root/training_curves.npy', allow_pickle=True)\nplotter.load_plot_data(plot_data)  # load the data into the plotter\nplotter.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"this is the code used to save the weights of specific layers\n\nmake sure the layers names are the same as the ones loaded or you can just load them in order"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# decoder_weights = {}\n# for layer in model.layers[-15:]:\n#     decoder_weights[layer.name] = layer.get_weights()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's load download and load the trained layers of the decoder"},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget \"https://drive.google.com/uc?id=1Omp4wFOWc2WslToduWAgBkyRv9poQVKb&export=download\" -O '/root/decoder_weights.npy'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"decoder_weights = np.load('/root/decoder_weights.npy', allow_pickle=True).item()\nfor layer_name, layer_weights in decoder_weights.items():\n    model.get_layer(layer_name).set_weights(layer_weights)  # set each layer of the decoder with its weights","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the dictionary\ndecoder_weights.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"eval\"></a>\n# 3. Model Evaluation"},{"metadata":{"trusted":true,"id":"xhG9T-uTqF96","outputId":"cb3c51de-583d-473d-f288-bd3a51dcf7d6","executionInfo":{"status":"ok","timestamp":1585745372189,"user_tz":-120,"elapsed":1696,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# define the test image generator that feeds the labelled test images to the model to evaluate it\ntest_gen = ImageDataGenerator(rescale=1/255)\ntest_flow = test_gen.flow_from_directory(new_test_dir,\n        target_size=(512, 512),\n        batch_size=1,\n        shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"JLKPO52KqF9-","outputId":"45ab78ef-1a92-4df2-b79b-15f8c2c0761b","executionInfo":{"status":"ok","timestamp":1585745438033,"user_tz":-120,"elapsed":66028,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# evaluate the model on the labelled test data\nmetrics = model.evaluate_generator(test_flow, steps=test_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"8jkiJrXCqF-B","outputId":"b51bc26c-182c-48aa-a79d-62514cfe46a8","executionInfo":{"status":"ok","timestamp":1585745438035,"user_tz":-120,"elapsed":65407,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"m_names = model.metrics_names\nprint(f'{m_names[0]} = {metrics[0]}\\n{m_names[1]} = {metrics[1]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Great! We've got 90% top-1 accuracy on a classification problem that has 0.83% accuracy from random chance"},{"metadata":{"id":"iesfBnzH9qvw","trusted":true,"outputId":"e34ffd44-129f-4ff4-dc7e-c6ff96ae3166","executionInfo":{"status":"ok","timestamp":1585745979772,"user_tz":-120,"elapsed":3108,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# check the corresponding classes of the encoding and ensure it matches the sample submission columns order\none_hot_map = train_gen.class_indices\none_hot_map","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"submit\"></a>\n# 4. Submission"},{"metadata":{"trusted":true,"id":"G1yzVqREqF-I"},"cell_type":"code","source":"input_dir = '../input/dog-breed-identification'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"rEwrsbekqF-L","outputId":"94f0ba10-6dce-4fde-b038-dcce394cb456","executionInfo":{"status":"ok","timestamp":1585745981279,"user_tz":-120,"elapsed":1768,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"test_names = os.listdir(input_dir+'/test')  # names of the files in the directory\ntest_names.sort()\ntest_size = len(test_names)\ntest_size  # number of test images to predict their labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"aAIKCvriqF-O","outputId":"24d59842-082f-4bc8-d907-bf6069c01733","executionInfo":{"status":"ok","timestamp":1585745982795,"user_tz":-120,"elapsed":2288,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# this flow just returns the test images, one by one, in order\ntest_flow = test_gen.flow_from_directory(input_dir,\n        target_size=(512, 512),\n        batch_size=1,\n        shuffle=False,\n        classes=['test'])  # added test folder as class because keras' flow needs subdirectories hierarchy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Due to floating point precision difference between the GPUs of Colab and Kaggle, the model will give a 0.34959 loss if you run it in kaggle, while it will give 0.28106 if you use colab. This is very important when dealing with such large models where this difference stacks up.\n\n### So I'll just load the submission file from colab to get a better model performance and save GPU hours.\n\nI'm open to advice on how to specify the floating point precision so that it's consistent on both GPUs. is setting the precision for numpy enough?"},{"metadata":{},"cell_type":"markdown","source":"### Uncomment the following cells to run the model using kaggle"},{"metadata":{"trusted":true,"id":"daylQ5BGqF-Y"},"cell_type":"code","source":"# # obtain the model's predictions\n# y_pred = model.predict_generator(test_flow, steps=test_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"79YkFakIqF-b","outputId":"f4e5c791-160a-4a32-a0f4-8dc69b7bac7b","executionInfo":{"status":"ok","timestamp":1585746619194,"user_tz":-120,"elapsed":634069,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# # check the shape\n# y_pred.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"PQTywDnQqF-f"},"cell_type":"code","source":"# submission = pd.DataFrame(data=y_pred, columns=classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0ojsHnZrqF-i"},"cell_type":"code","source":"# submission.insert(0, \"id\", test_names, True) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"fa-hRJECqF-l"},"cell_type":"code","source":"# submission.id = submission.id.apply(lambda x: x[:-4])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"nPCGSJw0qF-p","outputId":"073e8518-5cc8-4131-afb4-943c79d51524","executionInfo":{"status":"ok","timestamp":1585746619197,"user_tz":-120,"elapsed":633283,"user":{"displayName":"Mohammed Asfour","photoUrl":"","userId":"14357519656998048291"}}},"cell_type":"code","source":"# # check the submission is in the required format\n# submission.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"g9WDlthPqF-s"},"cell_type":"code","source":"# # save the submission\n# submission.to_csv('submission_file.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# download the submission file obtained from colab\n!wget 'https://drive.google.com/uc?id=18IQ9WVf1KcVTgwTgGZXKzx61W0ISayI8&export=download' -O 'submission_colab.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"accelerator":"GPU","colab":{"name":"kernel75a1f2720c0_rop.ipynb","provenance":[],"collapsed_sections":[],"machine_shape":"hm"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.2"}},"nbformat":4,"nbformat_minor":4}