{"cells":[{"metadata":{"_uuid":"e48fca5b06135c4b0aee1ccb7295773ceb909bdf"},"cell_type":"markdown","source":"Based on [_1](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50),  [_2](https://www.kaggle.com/abhiksark/introduction-to-transfer-learning-cats-dogs), [_3](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50/notebook) and [_4](https://www.kaggle.com/johnfarrell/dvc-pretrained-model-finetune/notebook)\n\n## Dataset\nThe train folder contains 25,000 images of dogs and cats. Each image in this folder has the label as part of the filename. The test folder contains 12,500 images, named according to a numeric id.\nFor each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).\n## Transfer learning\nIt is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task."},{"metadata":{"trusted":true,"_uuid":"6bc39767d719ce394356529f5fce8b72e2810bb1"},"cell_type":"code","source":"# Load packages\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\nfrom random import shuffle\nimport os, gc, time, cv2, random, math\n\n%matplotlib inline\nimport warnings\nwarnings.filterwarnings('ignore')\n\n####################\n# Global Constants #\n####################\nRESNET_WEIGHTS_PATH = '../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5'\nPATH = '../input/dogs-vs-cats-redux-kernels-edition/'\nTRAIN_DIR = PATH+'train'\nTEST_DIR =  PATH+'test'\nNUM_CLASSES = 2\nIMG_SIZE = 145  ###\nCHANNELS = 3\nEPOCHS = 15\nBATCH_SIZE = 32\n\ntrain_images = os.listdir(TRAIN_DIR)\ntest_images = os.listdir(TEST_DIR)\n\n# # For testing purposes\n# train_images = train_images[:10000]\n# test_images = test_images[:100]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"758cf4494d143da7e7683040a2b17dc5a3612567","scrolled":true},"cell_type":"code","source":"def label_img(img):\n    word_label = img.split('.')[-3]\n    if word_label == 'cat': return 0  ###\n    elif word_label == 'dog' : return 1  ###\n\n# Return a numpy array of train and test data\ndef process_data(data_image_list, DATA_FOLDER, isTrain=True):\n    data_df = []\n    for img in tqdm(data_image_list):\n        path = os.path.join(DATA_FOLDER,img)\n        if(isTrain):\n            label = label_img(img)\n        else:\n            label = img.split('.')[0]\n        img = cv2.imread(path,cv2.IMREAD_COLOR)\n        img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))\n        data_df.append([np.array(img), label])\n    shuffle(data_df)\n    return data_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3155b033f3e94c9f29c8ed26b962fdc29781560f"},"cell_type":"code","source":"# Prepare the train data\ntrain_data = process_data(train_images, TRAIN_DIR, isTrain=True)\nX = np.array([i[0] for i in train_data]).reshape(-1, IMG_SIZE, IMG_SIZE, 3)\ny = np.array([i[1] for i in train_data])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"902d5f3ffdd6b4ffce59b0fbb19967326f2721bb"},"cell_type":"markdown","source":"## Build the Model"},{"metadata":{"trusted":true,"_uuid":"ff69114b31aebd24406b02edbe93a11e19178019"},"cell_type":"code","source":"import keras.backend as K\nfrom keras.applications import ResNet50\nfrom keras.models import Model\nfrom keras import optimizers\nfrom keras.layers import Conv2D, Dense, Input, Flatten, Concatenate, Dropout, Activation\nfrom keras.layers import BatchNormalization, MaxPooling2D, GlobalAveragePooling2D\nfrom keras import applications\n\ndef get_pretrained_model(Weights_path='imagenet', trainable=False, Input_Shape=None):\n    input_shape = Input_Shape\n    base_model = ResNet50(weights=None, include_top=False, input_shape= input_shape)\n    base_model.load_weights(Weights_path)\n    for l in base_model.layers:\n        l.trainable = trainable\n    return base_model\n    \ndef build_model(PreModel, LearningRate=1e-3, Decay=1e-8):\n    \n    input_x = PreModel.inputs\n    \n    x_model = PreModel.output #(None, 1, 1, 2048)\n    x_model = Flatten()(x_model)\n    \n    x_model = Dense(64, activation='relu',name='fc1_Dense')(x_model)\n    #x_model = Dropout(0.5, name='dropout_1')(x_model)\n    x_model = BatchNormalization()(x_model)\n    \n    x_model = Dense(32, activation='relu',name='fc2_Dense')(x_model)\n    #x_model = Dropout(0.5, name='dropout_2')(x_model)\n    x_model = BatchNormalization()(x_model)\n    \n    predictions = Dense(1, activation='sigmoid',name='output_layer')(x_model)\n    model = Model(inputs=input_x, outputs=predictions)\n    optimizer = optimizers.Adam(lr=LearningRate, decay=Decay)\n    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n    return model\n\nPreModel = get_pretrained_model(Weights_path=RESNET_WEIGHTS_PATH,\n                                trainable=False,\n                                Input_Shape=(IMG_SIZE, IMG_SIZE, CHANNELS))\nmodel = build_model(PreModel, LearningRate=1e-3, Decay=1e-8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9671caab043c20d8b0162bac3983be819082c920","scrolled":true},"cell_type":"code","source":"# Model Summary\nfrom IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\nfrom keras.utils import plot_model\n\n# model.summary()\n# plot_model(model, to_file='model.png')\n# SVG(model_to_dot(model).create(prog='dot', format='svg'))\n# Trainable layers\nfor l in model.layers:\n    if l.trainable: print(l.name)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b0f0d21cdb0461f2c70e1069c05a9f5a2679953"},"cell_type":"markdown","source":"## Training"},{"metadata":{"trusted":true,"_uuid":"f98e407621ef794e416d68a1e180fcc719b235d3"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom keras.applications.resnet50 import preprocess_input\nfrom keras.preprocessing.image import ImageDataGenerator\n\nX_train, X_val, y_train, y_val = train_test_split(X,y, test_size=0.2, random_state=1)\n\n# Augmentation configuration to use for training and validation\ntrain_datagen = ImageDataGenerator(\n        #rescale=1./255,#!!!!!\n        shear_range=0.2,\n        zoom_range=0.2,\n        rotation_range=20, \n        horizontal_flip=True,\n        preprocessing_function=preprocess_input\n)\ntest_datagen = ImageDataGenerator(\n    #rescale=1./255,#!!!!!\n    preprocessing_function=preprocess_input\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0993c6ead13b57c5eeefa0e7100804a9acc6285f"},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nBestModelWeightsPath = 'BestModel.hdf5'\ncheck_point = ModelCheckpoint(\n    BestModelWeightsPath, monitor='val_loss', verbose=1,\n    save_best_only=True, \n    save_weights_only=True,\n    mode='min'\n)\nlr_reduce = ReduceLROnPlateau(monitor='val_acc', factor=0.1, min_delta=0.0001, patience=3, verbose=1)\nearlyStop = EarlyStopping(monitor='val_loss', mode='min', patience=30)\ncallbacks_list = [check_point, lr_reduce, earlyStop]\n\nK.set_value(model.optimizer.lr, 0.0001)\ngc.collect()\nhistory = model.fit_generator(\n    train_datagen.flow(np.array(X_train), y_train, batch_size=BATCH_SIZE, shuffle=True),\n    steps_per_epoch= len(X) // BATCH_SIZE,\n    validation_data = test_datagen.flow(np.array(X_val), y_val, batch_size=BATCH_SIZE*3, shuffle=False),\n    validation_steps = len(X_val) // (BATCH_SIZE*3),\n    epochs=EPOCHS,\n    shuffle=False,\n    verbose=1,\n    callbacks=callbacks_list\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"455ea546f78bc1294e0936029826c0c5c5fae1d1"},"cell_type":"code","source":"# Plotting loss and accuracy for the model\ndef plot_accuracy_and_loss(history):\n    eval_res = pd.DataFrame(history.history)\n    f, ax = plt.subplots(1,2, figsize=(18,5))\n    for i, c in enumerate(['acc', 'loss']):\n        ax[i].plot(eval_res[[c]], label=f'Training {c}')\n        ax[i].plot(eval_res[[f'val_{c}']], label=f'Validation {c}')\n        ax[i].set_xlabel('Epoch'); ax[i].set_ylabel(c);\n        ax[i].legend(); ax[i].set_title(f'Training and validation {c}'); plt.grid();\n    plt.show()\nplot_accuracy_and_loss(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"508e813cc0fcb80b8150ccff35025e15a27831f7"},"cell_type":"markdown","source":"## Fine-tuning"},{"metadata":{"trusted":true,"_uuid":"b34fdcc2caba3a6015560e9970c2e85e0dc0461d"},"cell_type":"code","source":"last_5_layer_names = [_.name for _ in PreModel.layers[::-1][:5]]\nprint(f'Pretrained have {len(PreModel.layers)} layers')\nprint(f'My model have {len(model.layers)} layers')\nprint(f'Pretrained last 5 layers: ', last_5_layer_names, '\\n')\n\n# for l in model.layers[:]: # enable training just for all layers\nfor l in model.layers[::-1][6:12]: # enable training just for last five layers of the Restnet50\n    print('Fine-tune', l.name);\n    l.trainable = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f19cab6393ddf8c863124a508eef52d9359b9773"},"cell_type":"code","source":"BestModelWeightsPath = 'BestModel.hdf5'\ncheck_point = ModelCheckpoint(\n    BestModelWeightsPath, monitor='val_loss', verbose=1,\n    save_best_only=True, \n    save_weights_only=True,\n    mode='min'\n)\nlr_reduce = ReduceLROnPlateau(monitor='val_acc', factor=0.1, min_delta=0.0001, patience=3, verbose=1)\nearlyStop = EarlyStopping(monitor='val_loss', mode='min', patience=30)\ncallbacks_list = [check_point, lr_reduce, earlyStop]\n\nK.set_value(model.optimizer.lr, 1e-6) ###\nK.set_value(model.optimizer.decay, 1e-9)\ngc.collect()\nhistory = model.fit_generator(\n    train_datagen.flow(np.array(X_train), y_train, batch_size=BATCH_SIZE, shuffle=True),\n    steps_per_epoch= len(X) // BATCH_SIZE,\n    validation_data = test_datagen.flow(np.array(X_val), y_val, batch_size=BATCH_SIZE*3, shuffle=False),\n    validation_steps = len(X_val) // (BATCH_SIZE*3),\n    epochs=math.ceil(EPOCHS*1.6), ###\n    verbose=1,\n    callbacks=callbacks_list\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6d28b4af0fb9e298fa076d4ab86c93fc0b81638"},"cell_type":"code","source":"plot_accuracy_and_loss(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f3a8a0fc848356e60ccbab01fbe6a22a4cb7b6a"},"cell_type":"markdown","source":"## Testing"},{"metadata":{"trusted":true,"_uuid":"0c191b7fabc05fac3434446facfb24b498504d0d","scrolled":false},"cell_type":"code","source":"# Free some memory\ndel X, y, train_data; gc.collect()\n\n# Load Best model weights\nmodel.load_weights(BestModelWeightsPath)\n\n# Testing Model on Test Data\ntest_data = process_data(test_images, TEST_DIR, isTrain=False)\nf, ax = plt.subplots(5,5, figsize=(18,18))\nfor i,data in enumerate(test_data[:25]):\n    img_num = data[1]\n    img_data = data[0]\n    orig = img_data\n    data = img_data.reshape(-1,IMG_SIZE,IMG_SIZE,3)\n    data = preprocess_input(data)\n    model_out = model.predict([data])[0]\n    if model_out[0] >= 0.5: \n        str_predicted='Dog'\n    else: \n        str_predicted='Cat'\n    ax[i//5, i%5].imshow(orig)\n    ax[i//5, i%5].axis('off')\n    ax[i//5, i%5].set_title(\"Confident :{:.2%} as {} \".format(abs(0.5-model_out[0])*2, str_predicted))    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3aba644ae36bdafe4f720005c646adf6233846fa"},"cell_type":"markdown","source":"## Generate .csv for submission"},{"metadata":{"trusted":true,"_uuid":"1b564f68d331f27d9d85403002bd878af7d52fc1"},"cell_type":"code","source":"prob = []\nimg_list = []\nfor data in tqdm(test_data):\n        img_num = data[1]\n        img_data = data[0]\n        orig = img_data\n        data = img_data.reshape(-1,IMG_SIZE,IMG_SIZE,3)\n        data = preprocess_input(data)\n        model_out = model.predict([data])[0]\n        img_list.append(img_num)\n        prob.append(model_out[0])\n    \nsubmission = pd.DataFrame({'id':img_list , 'label':prob})\nprint(submission.head())\nsubmission.to_csv(\"submit.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"226b45f9282d17819a5871c05e403e12e5b9d777"},"cell_type":"markdown","source":"## Comments\n- Due to memory limitations, I was obliged to reduce the Batch-size and Image-size.\n- Same data processing for validation and test images, i used this function **preprocess_input(data)**.\n- I was having vaidation accuracy higher then training accuracy, so i corrected some suffle values, so my model will not use some data in both training and validation.\n- **Fine-Tuning**: unfreezing some or all layers from Resnet 50 base layers and retraining using the previous best weights and a very small learning rate\n- I passed all day predicing Dog as 0 and cat as 1 and getting scores greeter then 1. I was testing different parameters, but predicting the reverse was a stupid thing\n- \n"},{"metadata":{"trusted":true,"_uuid":"0f805898db3afc6d601a26eb74592a6e139a010d"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}