{"cells":[{"metadata":{"_uuid":"2af9b615f4bf20eac4c1dce058551d9000eac181"},"cell_type":"markdown","source":"## This Notebook Introduces How to apply 'Transfer Learning' in Kaggle\n\nThank you for opening this Notebook!\n\nPress \"Fork\" at the top-right of this screen to run this notebook yourself and build each of the examples.\n\nI have made all efforts to document each and every step involved so that this notebook acts as a good starting point for new Kagglers who hope to apply **Transfer Learning** to their problem."},{"metadata":{"_uuid":"2a25d06d3c778d4272b808099c45690058447c74"},"cell_type":"markdown","source":"## Transfer learning\nIt is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task. \n\n<img src=\"https://elearningindustry.com/wp-content/uploads/2016/09/5-tips-improve-knowledge-transfer-elearning-e1475138920743.jpeg\" width=\"400px\"/>\n\nThe above image exactly represets what the model does for you.\nIt remembers the learning from a fairly related problem and applies it to the problem having new data.\n\n\nPre-trained Model Approach :-\n\n   1.  Select Source Model. A pre-trained source model is chosen from available models. Many research institutions release models on large and challenging datasets that may be included in the pool of candidate models from which to choose from.\n   <br/><br/>\n   2.  Reuse Model. The model pre-trained model can then be used as the starting point for a model on the second task of interest. This may involve using all or parts of the model, depending on the modeling technique used.  \n   <br/>\n   3.  Tune Model. Optionally, the model may need to be adapted or refined on the input-output pair data available for the task of interest.\n   <br/>\n![](http://https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2017/09/Three-ways-in-which-transfer-might-improve-learning.png)    \n\nPre-trained Models <br/>\n<a href=\"https://github.com/KaimingHe/deep-residual-networks\">Microsoft ResNet Model</a> <br/>\n<a href=\"https://github.com/tensorflow/models/tree/master/inception\">Google Inception Model</a> <br/>\n<a href=\"https://github.com/BVLC/caffe/wiki/Model-Zoo\">Caffe Model Zoo</a>\n\n"},{"metadata":{"_uuid":"bf0ba281ef46d9c17d618bc7ec1f0102179132f1"},"cell_type":"markdown","source":"## Libraries and settings ##\nImporting the Required libraries and assignment of few constants(such as Directories)"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"import os, cv2, random\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm    #Helps in visualization\nfrom random import shuffle #to shuffle the images \n\n%matplotlib inline \n\nTRAIN_DIR = '../input/dogs-vs-cats-redux-kernels-edition/train/'\nTEST_DIR = '../input/dogs-vs-cats-redux-kernels-edition/test/'\nIMG_SIZE = 224\n\nSHORT_LIST_TRAIN = os.listdir(TRAIN_DIR)[0:10000] #using a subset of data as resouces as limited. \nSHORT_LIST_TEST = os.listdir(TEST_DIR)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78bec5a9ce267aa7d910f44ce22839fbc9e163b9","trusted":false},"cell_type":"code","source":"def label_img(img): \n    word_label = img.split('.')[-3]\n    # conversion to one-hot array [cat,dog]\n    #                            [much cat, no dog]\n    if word_label == 'cat': return [1,0]\n    #                             [no cat, very doggo]\n    elif word_label == 'dog': return [0,1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8ae1548ea6e2d6167785fda2ee2168446560110","trusted":false},"cell_type":"code","source":"#returns an numpy array of train and test data\ndef create_train_data():\n    training_data = []\n    for img in tqdm(SHORT_LIST_TRAIN):\n        label = label_img(img)\n        path = os.path.join(TRAIN_DIR,img)\n        img = cv2.imread(path,cv2.IMREAD_COLOR)\n        img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))\n        training_data.append([np.array(img),np.array(label)])\n    shuffle(training_data)\n    return training_data\n\ndef process_test_data():\n    testing_data = []\n    for img in tqdm(os.listdir(TEST_DIR)):\n        path = os.path.join(TEST_DIR,img)\n        img_num = img.split('.')[0]\n        img = cv2.imread(path,cv2.IMREAD_COLOR)\n        img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))\n        testing_data.append([np.array(img), img_num])\n    shuffle(testing_data)\n    return testing_data","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0ca9ae75c6ee2da2791212b1d3790e5c30ab78c"},"cell_type":"markdown","source":"## Visualizing Classes ##\nVisulization is important to compare differnet classes and their number of occurances."},{"metadata":{"_uuid":"7112c62ae337855b5c17180970f37bff526dbf37","trusted":false},"cell_type":"code","source":"\nlabels = []\nfor i in SHORT_LIST_TRAIN:\n    if 'dog' in i:\n        labels.append(1)\n    else:\n        labels.append(0)\n\nsns.countplot(labels)\nplt.title('Cats and Dogs')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"012a523c1b1dee44f6bed9885d563cff5f4295a3"},"cell_type":"markdown","source":"## Creating a Training Set Data##"},{"metadata":{"_uuid":"f1aa5d1aef17d7a50e3163fd3076456fb9b47d70","trusted":false},"cell_type":"code","source":"train = create_train_data()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3548879f48ccfb0b73f1515d02a2fb4e28c14aa7"},"cell_type":"markdown","source":"## From Train Dividing X and Y##"},{"metadata":{"_uuid":"0171a2515458e1f6194e18ec48ab44aa041eb6bf","trusted":false},"cell_type":"code","source":"X = np.array([i[0] for i in train]).reshape(-1,IMG_SIZE,IMG_SIZE,3)\nY = np.array([i[1] for i in train])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92570dba082df8054b188acb31a38e2b39ea6dce"},"cell_type":"markdown","source":"## Specify Model##"},{"metadata":{"_uuid":"d6c9d86a101db152f9e431fd2ebe9ca8503a12de","collapsed":true,"trusted":false},"cell_type":"code","source":"from tensorflow.python.keras.applications import ResNet50\nfrom tensorflow.python.keras.models import Sequential\nfrom tensorflow.python.keras.layers import Dense, Flatten, GlobalAveragePooling2D\n\nNUM_CLASSES = 2\nRESNET_WEIGHTS_PATH = '../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5' #importing a pretrained model\nmy_new_model = Sequential()\nmy_new_model.add(ResNet50(include_top=False, pooling='max', weights=RESNET_WEIGHTS_PATH))\nmy_new_model.add(Dense(NUM_CLASSES, activation='softmax'))\n\n# Say not to train first layer (ResNet) model. It is already trained\nmy_new_model.layers[0].trainable = True","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0425612ac8e9bd947cc8f27772cca79890916289"},"cell_type":"markdown","source":"## Compile Model##"},{"metadata":{"_uuid":"50af9a9bed2bc9e5ac493db6b84bd9b76db5bc64","collapsed":true,"trusted":false},"cell_type":"code","source":"my_new_model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"edc9dbc6937b84f2a2596d21f1470c42099f2b99"},"cell_type":"markdown","source":"## Model Sumary##"},{"metadata":{"_kg_hide-input":true,"_uuid":"780be7320cc68646c105e4eb4196f40d76bde419","collapsed":true,"trusted":false},"cell_type":"code","source":"my_new_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"baa535766e60046b4d25cecba8fc5d3e1347c925"},"cell_type":"markdown","source":"## Fit Model##"},{"metadata":{"_kg_hide-input":true,"_uuid":"dc4a2d4933a0b7fbab3ffe15216479c6bb2a6875","collapsed":true,"trusted":false},"cell_type":"code","source":"history = my_new_model.fit(X, Y, validation_split=0.20, epochs=4, batch_size=64)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74de353ffaf5e86c8828c85baad4dc5cb3d62dda"},"cell_type":"markdown","source":"## Plotting loss and accuracy for the model##"},{"metadata":{"_kg_hide-input":true,"_uuid":"2d96ecaee49e71e14c86d293aff3c032b18519e8","collapsed":true,"trusted":false},"cell_type":"code","source":"plt.plot(history.history['acc'])\nplt.plot(history.history['val_acc'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"707fb10f7a890e3880ae37aadeb6e28e37bf50c5","collapsed":true,"trusted":false},"cell_type":"code","source":"\nSHORT_LIST_TRAIN = os.listdir(TRAIN_DIR)[-5000:]\ntrain = create_train_data()\nX = np.array([i[0] for i in train]).reshape(-1,IMG_SIZE,IMG_SIZE,3)\nY = np.array([i[1] for i in train])\nhistory = my_new_model.fit(X, Y, validation_split=0.5, epochs=20, batch_size=64)\n\nplt.plot(history.history['acc'])\nplt.plot(history.history['val_acc'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"958e0c3fb4e3b8bbc74c3eeb0556446c2ac9acf1"},"cell_type":"markdown","source":"## Testing Model on the Test Data##"},{"metadata":{"_kg_hide-input":true,"_uuid":"a52303a40be898802543f761a49feea890cf0fe3","collapsed":true,"trusted":false},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# if you need to create the data:\ntest_data = process_test_data()\n# if you already have some saved:\n#test_data = np.load('test_data.npy')\n\nfig=plt.figure()\n\nfor num,data in enumerate(test_data[:12]):\n    # cat: [1,0]\n    # dog: [0,1]\n    \n    img_num = data[1]\n    img_data = data[0]\n    \n    y = fig.add_subplot(3,4,num+1)\n    orig = img_data\n    data = img_data.reshape(-1,IMG_SIZE,IMG_SIZE,3)\n    #model_out = model.predict([data])[0]\n    model_out = my_new_model.predict([data])[0]\n    \n    if np.argmax(model_out) == 1: str_label='Dog'\n    else: str_label='Cat'\n        \n    y.imshow(orig)\n    plt.title(str_label)\n    y.axes.get_xaxis().set_visible(False)\n    y.axes.get_yaxis().set_visible(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ffa8b3ae50d60ee6a347fa2d7c1372f08703411","collapsed":true,"trusted":false},"cell_type":"code","source":"prob = []\nimg_list = []\nfor data in tqdm(test_data):\n        img_num = data[1]\n        img_data = data[0]\n        orig = img_data\n        data = img_data.reshape(-1,IMG_SIZE,IMG_SIZE,3)\n        model_out = my_new_model.predict([data])[0]\n        img_list.append(img_num)\n        prob.append(model_out[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1eb4058867b49c1f688edff0956e25ff2a828a06","collapsed":true,"trusted":false},"cell_type":"code","source":"submission = pd.DataFrame({'id':img_list , 'label':prob})\nprint(submission.head())\nsubmission.to_csv(\"../working/submit.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f44e0e1ff6ff7fe011f1d6dc5413fa6f9d35c5b1"},"cell_type":"markdown","source":"## Sources\n1. https://machinelearningmastery.com/transfer-learning-for-deep-learning/ <br/>\n2. http://cs231n.github.io/transfer-learning/ <br/>\n3. https://arxiv.org/abs/1411.1792 <br/>\nIf you are Interested in Research.\n\n\n"},{"metadata":{"_uuid":"be56ce986f213a15f65bba387e37da1d61d8cf6a","collapsed":true},"cell_type":"markdown","source":"Please upvote this kernel so that it reaches the top of the chart and is easily locatable by new users. Your comments on how we can improve this kernel is welcome. Thanks You For your Support."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}