{"cells":[{"metadata":{"_uuid":"3d6171ac9cc0b2f51656418c1e0511538cab06ba"},"cell_type":"markdown","source":"## Fast.ai Learning through Cats & Dogs Image Classification\n\n#### Sunil Kumar"},{"metadata":{"_uuid":"3492bd30529193bcf8cab076fa97f3057f31471f"},"cell_type":"markdown","source":"This kernel can be used to submit to Cats & Dogs Kaggle competition. One can try 1st option for training just the last layer, i,e., Dense FC layer while re-using pretrained ResNet* layers. Other option is to try SGDR. Then another approach given below is to try SGDR with Differential Learning Rate. Observe the validation loss, validation accuracy and Kaggle score. NOTE that try this with GPU ON and Internet CONNECTED in your Kaggle kernel. One can try fast.ai Images Augmentation while preparing the ConvNet model before model learning and it is one of the ways of better generalizing our deep learning - refer to https://becominghuman.ai/data-augmentation-using-fastai-aefa88ca03f1 .\n\nEven before moving to lectures beyond Lesson #1, I got curious to explore learning performance with various options given in fast.ai Lesson #1 notebook and explored if all fast.ai tricks can be implemented in Keras & TensorFlow. Refer to tutorial on [Tutorial Keras: Transfer Learning with ResNet50 for image classification on Cats & Dogs dataset](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50).\n\nNotes about special observations noticed while practicing with fast.ai on Cats & Dogs dataset from an old Kaggle competition with reference to fast.ai Lesson#1: - \n* Refer to http://ruder.io/deep-learning-optimization-2017/ for a comprehensive practical background in Deep Learning Optimizations with respect to Optimizers, Learning Rate schedule, etc.\n* Refer to SGDR in https://abdel.me/2018/01/05/fastai-part1/ for a better understanding as what fast.ai is doing under the hood.\n* SGDR\n    * Refer to the below plot of learning schedule for cycle_len=1 (notice iteration count as ~313, i.e., 10k/32)\n    * It is based on the concept of Cyclical Learning, a.k.a., Learning Rate Annealing with Warm Restart. SGD with Learning Rate Annealing schedule is a common practice to compete with adaptive Adam optimizer.\n    * Learning Rate decays as per half cosine with each Epoch Step (ref model.py >> step fn) and it restarts with specified Learning Rate value\n*  Differential Learning Rate for Fine Tuning PreTrained ResNet*    \n    * It is not enough to just train and re-train the Dense FC last layer while keeping ResNet layers frozen.\n    * We idenity best Learning Rate for Dense FC last layer and apply sclaed down Learning Rate for ResNet* grouped layers (Fast.ai seem to have treated ResNet* in 2 grouped layers & I'm yet to figure out as how this grouping has been done).\n    * SGDR + DLR + cyclt_mult affects in stretching Learning Rate Annealing to span across epochs. Refer to below plot for DLR.    \n    "},{"metadata":{"trusted":true,"_uuid":"83b494829684010098f0e4e4c2549c443a33ce63"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\n%matplotlib inline \n\nimport cv2\n\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f3c207ad48bceca8512c1c442635e23cc1df4583"},"cell_type":"code","source":"os.listdir(\"../input\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"from fastai.imports import *\nfrom fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"807cb0ff8db5410a4afd08ed1dfd1f5a8c3864ba"},"cell_type":"code","source":"PATH = \"../input/\"\nTMP_PATH = \"/tmp/tmp\"\nMODEL_PATH = \"/tmp/model/\"\nsz=224","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc43520d579c50f3158faf5181cb24d3ea91a1cc"},"cell_type":"code","source":"fnames = np.array([f'train/{f}' for f in sorted(os.listdir(f'{PATH}train'))])\nlabels = np.array([(0 if 'cat' in fname else 1) for fname in fnames])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"360888b7acb357f9c530da7242a53a5c828cbf2f"},"cell_type":"code","source":"arch=resnet50\ndata = ImageClassifierData.from_names_and_array(\n    path=PATH, \n    fnames=fnames, \n    y=labels, \n    classes=['dogs', 'cats'], \n    test_name='test', \n    tfms=tfms_from_model(arch, sz)\n)\nlearn = ConvLearner.pretrained(arch, data, precompute=True, tmp_name=TMP_PATH, models_name=MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c789ac75db552f9b7b95b0b69e79f1db23c2e6d"},"cell_type":"code","source":"###\n### Search for suitable, i.e., best Learning Rate for our-newly-added-Last Layer (as we have used 'precompute=True', i.e., ResNet50-minus-its-last-layer weights are being re-used as is)\n###\n#lrf=learn.lr_find()\n#learn.sched.plot_lr()\n#learn.sched.plot()\n\n###\n### Use the identified best Learning Rate for our-newly-added-Last Layer\n### Note that even without running above 3 lines of Learning Rate Finder, it is well known that best learning rate is 0.01 for Cats & Dogs images with 224x224 size\n### Kaggle Score obtained is 0.38683 (v7)\n###\n#learn.fit(0.01, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d2adfa8e89886003a1bbc9290a58068453f5af4"},"cell_type":"code","source":"###\n### SGDR (SGD with warm Resrart): fast.ai uses half Cosine shape decay (start with 0.01 & decay till 0) of LR during each epoch and then it restarts with 1e-02\n### Kaggle score obtained is 0.37578 (v8)\n###\nlearn.fit(1e-2, 10, cycle_len=1)\nlearn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1b6d4fdf924bef43327cd6c67352c5ec68d6f17"},"cell_type":"code","source":"###\n### Continue from Last Layer learned model with PreCompute=TRUE\n### Unfreeze all layers (all weights learned so far are retained) => it sets PreCompute=FALSE making all layers learnable\n### Effectively, the network weights are intialized as (ResNet-minus-last-layer with its original pre-trained weight & Last Layer as per above model learning while keeping ResNet as frozen)\n### Now, all layers are FURTHER learnable\n### Kaggle score obtained is 0.34815 (v9)\n###\nlearn.unfreeze()\n\n# Differential LR (above identified best LR for last layer, x0.1 to middle layer, x0.01 to inner layer)\nlr=np.array([1e-4,1e-3,1e-2])\n\nlearn.fit(lr, 3, cycle_len=1, cycle_mult=2)\n\nlearn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67b5c712b0bf0bbfc32d380a8b4126056d08ca56"},"cell_type":"code","source":"temp = learn.predict(is_test=True)\npred = np.argmax(temp, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26d521f382455e51ec30ee2ce6e0087dab1d6050"},"cell_type":"code","source":"import cv2\n\n# learn.predict works on unsorted os.listdir, hence listing filenames without sorting\nfnames_test = np.array([f'test/{f}' for f in os.listdir(f'{PATH}test')])\n\nf, ax = plt.subplots(5, 5, figsize = (15, 15))\n\nfor i in range(0,25):\n    imgBGR = cv2.imread(f'{PATH}{fnames_test[i]}')\n    imgRGB = cv2.cvtColor(imgBGR, cv2.COLOR_BGR2RGB)\n    \n    # a if condition else b\n    predicted_class = \"Dog\" if pred[i] else \"Cat\"\n\n    ax[i//5, i%5].imshow(imgRGB)\n    ax[i//5, i%5].axis('off')\n    ax[i//5, i%5].set_title(\"Predicted:{}\".format(predicted_class))    \n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d0390a1c932ee9f7303007b94523d49c75b36cd"},"cell_type":"code","source":"results_df = pd.DataFrame(\n    {\n        'id': pd.Series(fnames_test), \n        'label': pd.Series(pred)\n    })\nresults_df['id'] = results_df.id.str.extract('(\\d+)')\nresults_df['id'] = pd.to_numeric(results_df['id'], errors = 'coerce')\nresults_df.sort_values(by='id', inplace = True)\n\nresults_df.to_csv('submission.csv', index=False)\nresults_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f30fef9e396d330f151d856e4010e366b94cc049"},"cell_type":"markdown","source":"### References\n\n1. [Kaggle kernel about implementing Cats & Dogs images classification in Keras](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}