{"cells":[{"metadata":{"_uuid":"e48fca5b06135c4b0aee1ccb7295773ceb909bdf"},"cell_type":"markdown","source":"Based on [_1](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50),  [_2](https://www.kaggle.com/abhiksark/introduction-to-transfer-learning-cats-dogs), [_3](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50/notebook) and [_4](https://www.kaggle.com/johnfarrell/dvc-pretrained-model-finetune/notebook)\n\n## Dataset\nThe train folder contains 25,000 images of dogs and cats. Each image in this folder has the label as part of the filename. The test folder contains 12,500 images, named according to a numeric id.\nFor each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).\n## Transfer learning\nIt is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task."},{"metadata":{"trusted":true,"_uuid":"6bc39767d719ce394356529f5fce8b72e2810bb1"},"cell_type":"code","source":"# Load packages\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\nfrom random import shuffle\nimport os, gc, time, cv2, random, math\n\n%matplotlib inline\nimport warnings\nwarnings.filterwarnings('ignore')\n\n####################\n# Global Constants #\n####################\nINCEPTION_V3_WEIGHTS_PATH = '../input/inceptionv3/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5'\nPATH = '../input/dogs-vs-cats-redux-kernels-edition/'\nTRAIN_DIR = PATH+'train'\nTEST_DIR =  PATH+'test'\nNUM_CLASSES = 2\nIMG_SIZE = 145  ###\nCHANNELS = 3\nEPOCHS = 30\nBATCH_SIZE = 32\n\ntrain_images = os.listdir(TRAIN_DIR)\ntest_images = os.listdir(TEST_DIR)\n\n# # For testing purposes\n# train_images = train_images[:10000]\n# test_images = test_images[:100]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"758cf4494d143da7e7683040a2b17dc5a3612567","scrolled":true},"cell_type":"code","source":"def label_img(img):\n    word_label = img.split('.')[-3]\n    if word_label == 'cat': return 0  ###\n    elif word_label == 'dog' : return 1  ###\n\n# Return a numpy array of train and test data\ndef process_data(data_image_list, DATA_FOLDER, isTrain=True):\n    data_df = []\n    for img in tqdm(data_image_list):\n        path = os.path.join(DATA_FOLDER,img)\n        if(isTrain):\n            label = label_img(img)\n        else:\n            label = img.split('.')[0]\n        img = cv2.imread(path,cv2.IMREAD_COLOR)\n        img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))\n        data_df.append([np.array(img), label])\n    shuffle(data_df)\n    return data_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3155b033f3e94c9f29c8ed26b962fdc29781560f"},"cell_type":"code","source":"# Prepare the train data\ntrain_data = process_data(train_images, TRAIN_DIR, isTrain=True)\nX = np.array([i[0] for i in train_data]).reshape(-1, IMG_SIZE, IMG_SIZE, 3)\ny = np.array([i[1] for i in train_data])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"902d5f3ffdd6b4ffce59b0fbb19967326f2721bb"},"cell_type":"markdown","source":"## Build the Model"},{"metadata":{"trusted":true,"_uuid":"ff69114b31aebd24406b02edbe93a11e19178019"},"cell_type":"code","source":"import keras.backend as K\nfrom keras.applications.inception_v3 import InceptionV3\nfrom keras.models import Model\nfrom keras import optimizers\nfrom keras.layers import Conv2D, Dense, Input, Flatten, Concatenate, Dropout, Activation\nfrom keras.layers import BatchNormalization, MaxPooling2D, GlobalAveragePooling2D\nfrom keras import applications\n\ndef get_pretrained_model(Weights_path='imagenet', trainable=False, Input_Shape=None):\n    input_shape = Input_Shape\n    base_model = InceptionV3(weights=None, include_top=False, input_shape= input_shape)\n    base_model.load_weights(Weights_path)\n    for l in base_model.layers:\n        l.trainable = trainable\n    return base_model\n    \ndef build_model(PreModel, LearningRate=1e-3, Decay=1e-8):\n    \n    input_x = PreModel.inputs\n    \n    x_model = PreModel.output\n    #x_model = GlobalAveragePooling2D()(x_model)\n    \n    x_model = Flatten()(x_model)\n    \n    x_model = Dense(64, activation='relu',name='fc1_Dense')(x_model)\n    x_model = Dropout(0.5, name='dropout_1')(x_model)\n    x_model = BatchNormalization()(x_model)\n    \n    x_model = Dense(32, activation='relu',name='fc2_Dense')(x_model)\n    x_model = Dropout(0.5, name='dropout_2')(x_model)\n    x_model = BatchNormalization()(x_model)\n    \n    predictions = Dense(1, activation='sigmoid',name='output_layer')(x_model)\n    model = Model(inputs=input_x, outputs=predictions)\n    optimizer = optimizers.SGD(lr=LearningRate, decay=Decay)\n    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n    return model\n\nPreModel = get_pretrained_model(Weights_path=INCEPTION_V3_WEIGHTS_PATH,\n                                trainable=False,\n                                Input_Shape=(IMG_SIZE, IMG_SIZE, CHANNELS))\nmodel = build_model(PreModel, LearningRate=1e-3, Decay=1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9671caab043c20d8b0162bac3983be819082c920","scrolled":true},"cell_type":"code","source":"# Model Summary\nfrom IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\nfrom keras.utils import plot_model\n\n# model.summary()\n# plot_model(model, to_file='model.png')\n# SVG(model_to_dot(model).create(prog='dot', format='svg'))\n# Trainable layers\nfor l in model.layers:\n    if l.trainable: print(l.name)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b0f0d21cdb0461f2c70e1069c05a9f5a2679953"},"cell_type":"markdown","source":"## Training"},{"metadata":{"trusted":true,"_uuid":"f98e407621ef794e416d68a1e180fcc719b235d3"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom keras.preprocessing.image import ImageDataGenerator\n\nX_train, X_val, y_train, y_val = train_test_split(X,y, test_size=0.2, random_state=1)\n\n# Augmentation configuration to use for training and validation\ntrain_datagen = ImageDataGenerator(\n        rescale=1./255,#!!!!!\n        shear_range=0.2,\n        zoom_range=0.2,\n        rotation_range=20, \n        horizontal_flip=True,\n#         preprocessing_function=preprocess_input\n)\ntest_datagen = ImageDataGenerator(\n    rescale=1./255,#!!!!!\n#     preprocessing_function=preprocess_input\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0993c6ead13b57c5eeefa0e7100804a9acc6285f"},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nBestModelWeightsPath = 'BestModel.hdf5'\ncheck_point = ModelCheckpoint(\n    BestModelWeightsPath, monitor='val_loss', verbose=1,\n    save_best_only=True, \n    save_weights_only=True,\n    mode='min'\n)\nlr_reduce = ReduceLROnPlateau(monitor='val_acc', factor=0.1, min_delta=0.0001, patience=3, verbose=1)\nearlyStop = EarlyStopping(monitor='val_loss', mode='min', patience=30)\ncallbacks_list = [check_point, lr_reduce, earlyStop]\n\nK.set_value(model.optimizer.lr, 0.1) ####\ngc.collect()\nhistory = model.fit_generator(\n    train_datagen.flow(np.array(X_train), y_train, batch_size=BATCH_SIZE, shuffle=True),\n    steps_per_epoch= len(X) // BATCH_SIZE,\n    validation_data = test_datagen.flow(np.array(X_val), y_val, batch_size=BATCH_SIZE*3, shuffle=False),\n    validation_steps = len(X_val) // (BATCH_SIZE*3),\n    epochs=EPOCHS,\n    shuffle=False,\n    verbose=1,\n    callbacks=callbacks_list\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"455ea546f78bc1294e0936029826c0c5c5fae1d1"},"cell_type":"code","source":"# Plotting loss and accuracy for the model\ndef plot_accuracy_and_loss(history):\n    eval_res = pd.DataFrame(history.history)\n    f, ax = plt.subplots(1,2, figsize=(18,5))\n    for i, c in enumerate(['acc', 'loss']):\n        ax[i].plot(eval_res[[c]], label=f'Training {c}')\n        ax[i].plot(eval_res[[f'val_{c}']], label=f'Validation {c}')\n        ax[i].set_xlabel('Epoch'); ax[i].set_ylabel(c);\n        ax[i].legend(); ax[i].set_title(f'Training and validation {c}'); plt.grid();\n    plt.show()\nplot_accuracy_and_loss(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"508e813cc0fcb80b8150ccff35025e15a27831f7"},"cell_type":"markdown","source":"## Fine-tuning"},{"metadata":{"trusted":true,"_uuid":"b34fdcc2caba3a6015560e9970c2e85e0dc0461d"},"cell_type":"code","source":"last_5_layer_names = [_.name for _ in PreModel.layers[::-1][:5]]\nprint(f'Pretrained have {len(PreModel.layers)} layers')\nprint(f'My model have {len(model.layers)} layers')\nprint(f'Pretrained last 5 layers: ', last_5_layer_names, '\\n')\n\n# for l in model.layers[:]: # enable training just for all layers\nfor l in model.layers[::-1][6:12]: # enable training just for last five layers of the Restnet50\n    print('Fine-tune', l.name);\n    l.trainable = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f19cab6393ddf8c863124a508eef52d9359b9773"},"cell_type":"code","source":"BestModelWeightsPath = 'BestModel.hdf5'\ncheck_point = ModelCheckpoint(\n    BestModelWeightsPath, monitor='val_loss', verbose=1,\n    save_best_only=True, \n    save_weights_only=True,\n    mode='min'\n)\nlr_reduce = ReduceLROnPlateau(monitor='val_acc', factor=0.1, min_delta=0.0001, patience=3, verbose=1)\nearlyStop = EarlyStopping(monitor='val_loss', mode='min', patience=30)\ncallbacks_list = [check_point, lr_reduce, earlyStop]\n\nK.set_value(model.optimizer.lr, 1e-5) ###\nK.set_value(model.optimizer.decay, 1e-8)\ngc.collect()\nhistory = model.fit_generator(\n    train_datagen.flow(np.array(X_train), y_train, batch_size=BATCH_SIZE, shuffle=True),\n    steps_per_epoch= len(X) // BATCH_SIZE,\n    validation_data = test_datagen.flow(np.array(X_val), y_val, batch_size=BATCH_SIZE*3, shuffle=False),\n    validation_steps = len(X_val) // (BATCH_SIZE*3),\n    epochs=math.ceil(EPOCHS*1.6), ###\n    verbose=1,\n    callbacks=callbacks_list\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6d28b4af0fb9e298fa076d4ab86c93fc0b81638"},"cell_type":"code","source":"plot_accuracy_and_loss(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f3a8a0fc848356e60ccbab01fbe6a22a4cb7b6a"},"cell_type":"markdown","source":"## Testing"},{"metadata":{"trusted":true,"_uuid":"0c191b7fabc05fac3434446facfb24b498504d0d","scrolled":false},"cell_type":"code","source":"# Free some memory\ndel X, y, train_data; gc.collect()\n\n# Load Best model weights\nmodel.load_weights(BestModelWeightsPath)\n\n# Testing Model on Test Data\ntest_data = process_data(test_images, TEST_DIR, isTrain=False)\nf, ax = plt.subplots(5,5, figsize=(18,18))\nfor i,data in enumerate(test_data[:25]):\n    img_num = data[1]\n    img_data = data[0]\n    orig = img_data\n    data = img_data.reshape(-1,IMG_SIZE,IMG_SIZE,3)\n    data = data* 1./255\n    model_out = model.predict([data])[0]\n    if model_out[0] >= 0.5: \n        str_predicted='Dog'\n    else: \n        str_predicted='Cat'\n    ax[i//5, i%5].imshow(orig)\n    ax[i//5, i%5].axis('off')\n    ax[i//5, i%5].set_title(\"Confident :{:.2%} as {} \".format(abs(0.5-model_out[0])*2, str_predicted))    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3aba644ae36bdafe4f720005c646adf6233846fa"},"cell_type":"markdown","source":"## Generate .csv for submission"},{"metadata":{"trusted":true,"_uuid":"1b564f68d331f27d9d85403002bd878af7d52fc1"},"cell_type":"code","source":"prob = []\nimg_list = []\nfor data in tqdm(test_data):\n        img_num = data[1]\n        img_data = data[0]\n        orig = img_data\n        data = img_data.reshape(-1,IMG_SIZE,IMG_SIZE,3)\n        data = data* 1./255\n        model_out = model.predict([data])[0]\n        img_list.append(img_num)\n        prob.append(model_out[0])\n    \nsubmission = pd.DataFrame({'id':img_list , 'label':prob})\nprint(submission.head())\nsubmission.to_csv(\"submit.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"226b45f9282d17819a5871c05e403e12e5b9d777"},"cell_type":"markdown","source":"## Comments\n- I was getting small training loss and large validation loss, so i removed the keras.applications.resnet50.preprocess_input preprocessing_function and replaced it with just , **rescale=1./255**.   \n(It may be because, training_steps is 3 times smaller then validation_steps and also because we do data augmentation on training and not on testing data)\n\n"},{"metadata":{"trusted":true,"_uuid":"0f805898db3afc6d601a26eb74592a6e139a010d"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}