{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Introduction\n\nI wanted to use Kaggle GPU resources to learn fastai but unfortunately, the current lessons are using fastai 0.7, while kaggle has upgraded to fastai 1.0.  Downgrading is not possible without switching off GPU, and I did not want to subscribe any payment option (not affordable for me), like vectordash to be able to use GPU. So I set out to use latest fastai 1.0 for the currently available lessons and started with first one. This was very challenging for me, as I am just a rookie in fastai with no ML experience, with just few months of python coding. That said, this notebook would most probably contain potential bugs in the transformation. Also note, fastai's new tutorial (DL Part 1 v3) would be available by Jan 2019 which would use latest fastai 1.0, and might be totally different than this. So, please use this with caution and feel free to correct it. This is also the reason, I have not added documentation part yet. This notebook is mainly for those like me who cannot wait till new tutorials are up, and want to start immediately, and also want free GPU resources. \n\nThis notebook does not use loaded dataset from kaggle, but downloads on the fly from fastai url., so please ensure in kaggle settings, to enable internet\n\n# Initial Setup"},{"metadata":{"trusted":true,"_uuid":"89b194649f22cc23819fc34e8a46deda5306e73b","scrolled":true},"cell_type":"code","source":"!pip freeze | grep fastai","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2deddd227d02a90c5952d2c955a26bf56552097c"},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import * \nfrom fastai.datasets import *  # this is needed for untar_data\nfrom fastai.metrics import * # for accuracy\nfrom fastai.vision.data import * ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f8dd5e4fef3c235e473c6d9d7299ee461d96aad"},"cell_type":"markdown","source":"Check if GPU is enabled properly"},{"metadata":{"trusted":true,"_uuid":"9524e54c902f219316e97fc10afbedfbbca430b8"},"cell_type":"code","source":"import torch\ntorch.cuda.is_available()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b3e62a531a510ef579b3ededf0358bc3006ccd3"},"cell_type":"code","source":"torch.backends.cudnn.enabled","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d4a9758f7d6b2089471b2000756d5c2c1ebd453"},"cell_type":"markdown","source":"# First look at cat pictures"},{"metadata":{"trusted":true,"_uuid":"6c705ef260755ea35acecb075819e37ae8ebefb5"},"cell_type":"code","source":"path = untar_data(URLs.DOGS)\nsz=112  #size\npath","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69c619f1fa8c2817a64961bc5a23e63c1207f41e","_kg_hide-input":false},"cell_type":"code","source":"import os\nos.listdir(path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1303acd94b53510c1d5f0dc379267e42f54d5613"},"cell_type":"code","source":"from subprocess import check_output\nprint(check_output([\"ls\", \"/tmp/.fastai/data/dogscats/train\"]).decode(\"utf8\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0684b79b45245a537ec1f25b2d50e291380557ec"},"cell_type":"code","source":"import numpy as np\nfnames = np.array([f'train/cats/{f}' for f in sorted(os.listdir(f'{path}/train/cats'))] + [f'train/{f}' for f in sorted(os.listdir(f'{path}/train/dogs'))])\nlabels = np.array([(0 if 'cat' in fname else 1) for fname in fnames])\nprint(fnames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c1dec3ef6e74255e2edb706b3e3ab5c4873a7d34"},"cell_type":"code","source":"import matplotlib.pyplot as plt \nimg = plt.imread(f'{path}/{fnames[0]}')\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba3d0ff3601ab66050b50a18dcbcaafacc3c89d7"},"cell_type":"code","source":"img.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"8ff28a4d661cb4a5213455bed254e5be13841b9f"},"cell_type":"code","source":"img[:4,:4]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f733c03ac4a7d4a7429ccfdfd66e81f8763ea1c"},"cell_type":"markdown","source":"# Our first model: quick start"},{"metadata":{"trusted":true,"_uuid":"2df0eae59d5071d0620d24d209119434e5efd5e7"},"cell_type":"code","source":"data = ImageDataBunch.from_folder(path, ds_tfms=get_transforms(), size=sz, num_workers=1)\nlearn = create_cnn(data, models.resnet34, metrics=accuracy)\nlearn.fit(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b17dad180e1001b78fe491914929354b3ffa05ff"},"cell_type":"markdown","source":"## Analyzing results: looking at pictures"},{"metadata":{"trusted":true,"_uuid":"737b8cf96e6ba567a5139e1109e3fe6ef66e00c7"},"cell_type":"code","source":"# this gives prediction for test set  - afaik!\nraw_preds,val_y,losses = learn.get_preds(with_loss=True) # predicting on test set\nraw_preds.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"308f7032f34f66a7c4b6752272f4bca1de814f43"},"cell_type":"code","source":"# This is the label for a val data - afaik\nval_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27400373896b24844fb09ad020012925216cf3eb"},"cell_type":"code","source":"# from here we know that 'dogs' is label 0 and 'cats' is label 1.\n# Note this is opposite of original lesson 1 assignment\ndata.classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"188b8ded2ca25b0b2110a2ba6ce38df3914a0fb5"},"cell_type":"code","source":"# few predication values - probability in linear scale\nraw_preds[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ed05479bd15902d5d9f7b16a7dc50f3a59c98b86"},"cell_type":"code","source":"preds = np.argmax(raw_preds, axis=1)  # from probabilities to 0 or 1\npreds[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48b1dd6f6d09a5bcb5906f6e6ae26dd551dbf9b0"},"cell_type":"code","source":"probs = raw_preds[:,1]        # pr(dog)  # no exp operator caz tensor was in linear scale\nnp.around(probs[:10],3)       # printing in decimal as I could not tolerate exp format","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec5b48dd94788d48534c51e28bfc2c127710ab44"},"cell_type":"code","source":"def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], 4, replace=False)\ndef rand_by_correct(is_correct): return rand_by_mask((preds == val_y)==is_correct)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18110cea1100e6d0605f99315438394c8b534118"},"cell_type":"code","source":"def plots(ims, figsize=(12,6), rows=1, titles=None):\n    f = plt.figure(figsize=figsize)\n    for i in range(len(ims)):\n        sp = f.add_subplot(rows, len(ims)//rows, i+1)\n        sp.axis('Off')\n        if titles is not None: sp.set_title(titles[i], fontsize=16)\n        plt.imshow(ims[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee526fc5dab59940ec957a1d92a4998257ff2d92"},"cell_type":"code","source":"def load_img_id(ds, idx): return np.array(PIL.Image.open(ds.x.items[idx]))\n\ndef plot_val_with_title(idxs, title):\n    imgs = [load_img_id(data.valid_ds,x) for x in idxs]\n    title_probs = [probs[x] for x in idxs]\n    print(title)\n    return plots(imgs, rows=1, titles=title_probs, figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c52b3696972b30e417c1f4c8d4932ebc5bfc9f4","scrolled":true},"cell_type":"code","source":"# 1. A few correct labels at random\nplot_val_with_title(rand_by_correct(True), \"Correctly classified\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f81bc84883eec03693436c3c3a6d74d69fe45d6b"},"cell_type":"code","source":"# 2. A few incorrect labels at random\nplot_val_with_title(rand_by_correct(False), \"Incorrectly classified\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85dac53bd3d952d1d12dc8f84bbae2df67d6d510"},"cell_type":"code","source":"def most_by_mask(mask, mult):\n    idxs = np.where(mask)[0]\n    return idxs[np.argsort(mult * probs[idxs])[:4]]\n\ndef most_by_correct(y, is_correct): \n    mult = -1 if (y==1)==is_correct else 1\n    return most_by_mask(((preds == val_y)==is_correct) & (val_y == y), mult)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f736ce214c663724b9c868b7e0fff69c2a05c4e"},"cell_type":"code","source":"plot_val_with_title(most_by_correct(0, True), \"Most correct cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7dca327eccc2343c5126bf27de684097ec391498"},"cell_type":"code","source":"plot_val_with_title(most_by_correct(1, True), \"Most correct dogs\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3de262d7f9df06a48076c1a50f05e7c9dd30737"},"cell_type":"code","source":"plot_val_with_title(most_by_correct(0, False), \"Most incorrect cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86d510895b5a867709d3cde6b53bd91f20f2d8c2"},"cell_type":"code","source":"plot_val_with_title(most_by_correct(1, False), \"Most incorrect dogs\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5c8485356c71dbd589982627ebcccdec3c0c033"},"cell_type":"code","source":"most_uncertain = np.argsort(np.abs(probs -0.5))[:4]\nplot_val_with_title(most_uncertain, \"Most uncertain predictions\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66a4ab5240eda08d2d27a578356fc9a215cc4b48"},"cell_type":"markdown","source":"## Choosing a learning rate"},{"metadata":{"trusted":true,"_uuid":"4e975d8fce3a9fe8e53b3ec4b97edc517ca65b87"},"cell_type":"code","source":"def simple_learner(): return Learner(data, simple_cnn((3,16,16,2)), metrics=[accuracy])\nlearn_s = simple_learner()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"93c7b96914f10d930aabedb288e81da11460241f"},"cell_type":"code","source":"learn_s.lr_find(stop_div=False, num_it=250)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f01f8cb6a61ed25bc3e2d05a410652636f75730"},"cell_type":"code","source":"learn_s.recorder.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"80d20bc198ac6c38e5533ee448f814fbb16977da"},"cell_type":"code","source":"learn_s.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b33ccecee4c25bed6cccb17d15cf482b677aa74"},"cell_type":"markdown","source":"<font color='blue'>Unlike part 1 lesson 1 v2 using fastai 0.7 which has smoother graph, above graph turns out to be kinda rough and ugly</font> I tried using create_cnn, but that resulted in totally different graph so sticking with simple learner for now. "},{"metadata":{"_uuid":"ecfa28b608ea7432a9df67708f17dad212568eb2"},"cell_type":"markdown","source":"## Improving our model\n\n### Data augmentation"},{"metadata":{"_uuid":"0399a98082f3bbe3e84fbefbe742d50102fd76a1"},"cell_type":"markdown","source":"<font color='blue'>\n    <li>could not find lighting and flip function applicable without image input</li>\n    <li>could not apply rotate as well as that is member of image class not separate function</li>\n</font>"},{"metadata":{"trusted":true,"_uuid":"03fbdf33959bccde528fd2848f23f4a6254ef79f"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom random import randint\n\n# fastai v0.7 definition for transforms_side_on not available in fastai v1.0\n#transforms_basic    = [RandomRotate(10), RandomLighting(0.05, 0.05)]\n#transforms_side_on  = transforms_basic + [RandomFlip()]\n\ndef get_augs():\n    rand_deg = randint(0,10)\n    rand_light = randint(0,5)/100\n    tfms = get_transforms(xtra_tfms = [rotate(degrees=rand_deg),flip_lr()], max_zoom=1.1, max_lighting=rand_light)\n    data_a = ImageDataBunch.from_folder(\n        path, \n        ds_tfms = tfms,\n        size=sz, \n        num_workers=1)     \n    \n    x,_ = next(iter(data_a.train_ds))\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17042be1df9a1f8af917194cf848e3a9574a77bb"},"cell_type":"code","source":"ims = np.stack([get_augs() for i in range(6)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c68c540f349383af013b2f6677bcfb599f56af1d"},"cell_type":"code","source":"%matplotlib inline\n# plt.imshow(image2np(ims[0].data))\n# plt.show()\nfig, axr = plt.subplots(2,3, figsize=(8,8))\nk = 0\nfor i in range(2): # rows\n    for j in range(3): # cols\n        ims[k].show(ax=axr[i,j])\n        k += 1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fda82aa61b813e545bbdf4b31cf96ce8296f6ee5"},"cell_type":"markdown","source":"creating a new ```data``` object that includes the augmentation (note due to missing random functions as explained above, we are proceeding with a fixed transformation probably)"},{"metadata":{"trusted":true,"_uuid":"60c27066a81d19303417c30e3fe5582b968faf03"},"cell_type":"code","source":"tfms = get_transforms(xtra_tfms = [rotate(degrees=10),flip_lr()], max_zoom=1.1, max_lighting=0.05)\ndata_a = ImageDataBunch.from_folder(\n    path, \n    ds_tfms = tfms,\n    size=sz, \n    num_workers=1)     \nlearn_a = create_cnn(data_a, models.resnet34, metrics=accuracy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec444c287483c259fb34653b904d612b40c10f4c"},"cell_type":"code","source":"learn_a.fit(epochs=1, lr=1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca996f9415013cc5674368eba2d2001c7bf91598"},"cell_type":"code","source":"learn_a.fit_one_cycle(3, 1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f11fb5cbd35e14aa344c716b57f97f468bcaad4f"},"cell_type":"code","source":"learn_a.recorder.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92e66ccf82791194b019140ccbd7748af134f813"},"cell_type":"markdown","source":"<font color='blue'>I could not find how to bring 3 tops on the curve, only managed to get single top unlike the lesson 1 of part 1 v2</font>"},{"metadata":{"_uuid":"10d9d9fa3f30608e65b7243ebcb361580cdf7283"},"cell_type":"markdown","source":"#### saving and loading the model"},{"metadata":{"trusted":true,"_uuid":"6fd2c1908de51333b183b94d1179738fee046e78"},"cell_type":"code","source":"learn_a.save('224_lastlayer')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9791d0c5ad94f2f6b5fe6260c3e24fb33a718f1"},"cell_type":"code","source":"learn_a.load('224_lastlayer')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d0c6fb30d653dfc6234268814c4a717123b50b9"},"cell_type":"markdown","source":"## Fine tuning and differential learning rate annealing"},{"metadata":{"trusted":true,"_uuid":"94081c6c63a0144272891aa205db19e9d51d387b"},"cell_type":"code","source":"learn_a.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f19ca56637e1214705ef772da13f883a9331084"},"cell_type":"code","source":"from fastai.callbacks import * \n\ndef fit_sgd_warm(learn, n_cycles, lr, mom, cycle_len, cycle_mult):\n    n = len(learn.data.train_dl)\n    phases = [TrainingPhase(n * (cycle_len * cycle_mult**i), lr, mom, lr_anneal=annealing_cos) for i in range(n_cycles)]\n    sched = GeneralScheduler(learn, phases)\n    learn.callbacks.append(sched)\n    if cycle_mult != 1:\n        total_epochs = int(cycle_len * (1 - (cycle_mult)**n_cycles)/(1-cycle_mult)) \n    else: total_epochs = n_cycles * cycle_len\n    learn.fit(total_epochs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f16fb2ca30266230277d1cbb764c808755014b9f"},"cell_type":"code","source":"fit_sgd_warm(learn_a, 3, 1e-3, 0.9, 1, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"95c250b76d2c4953e46bf8e81bdfed40ffa6e9d5"},"cell_type":"code","source":"learn_a.recorder.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"460fdd25b97b636259e5c800dc40a10b5ca3829e"},"cell_type":"code","source":"learn_a.save('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"3f5f6e6ebec30bf343e0e501357453367fec0780"},"cell_type":"code","source":"learn_a.load('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40d3900062c16ee1a3d616b023140175b523bf43"},"cell_type":"code","source":"preds_a, y_a = learn_a.TTA()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac78e7b61bdf0ab9a8f028d5cc5c26f86c0c1f5a"},"cell_type":"code","source":"def accuracy_np(preds, targs):\n    preds = np.argmax(preds, 1)\n    return (preds==targs).mean()\n\naccuracy_np(preds_a.numpy(), y_a.numpy())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e525a8b253b1e408407c626036a535732cd87333"},"cell_type":"markdown","source":"## Analyzing results\n\n### Confusion matrix"},{"metadata":{"trusted":true,"_uuid":"6ea45f52f43a8b8cb900db87ffbfc9343f8bb24b"},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn_a)\ninterp.plot_confusion_matrix()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12c6b61ba136428976a2dccabeea8d516b99ad13"},"cell_type":"markdown","source":"### Looking at pictures again"},{"metadata":{"trusted":true,"_uuid":"787e6185a903c7169a94c5bb0c3586d6fd357ca3"},"cell_type":"code","source":"plot_val_with_title(most_by_correct(0, False), \"Most incorrect cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b36093312c1f797df7f75e5e99934e21eee24fc"},"cell_type":"code","source":"plot_val_with_title(most_by_correct(1, False), \"Most incorrect dogs\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c9b65bd260f583e9ca0aedcc2947acc4626a00f"},"cell_type":"markdown","source":"### Understanding the code for our first model"},{"metadata":{"trusted":true,"_uuid":"88abe9585fea48c6597d0e76a7a890b4e44be79f"},"cell_type":"code","source":"tfms = get_transforms(xtra_tfms = [rotate(degrees=10),flip_lr()], max_zoom=1.1, max_lighting=0.05)\ndata_z = ImageDataBunch.from_folder(\n    path, \n    ds_tfms = tfms,\n    size=sz, \n    num_workers=1)     \nlearn_z = create_cnn(data_z, models.resnet34, metrics=accuracy)\nlearn_z.fit(epochs=1, lr=1e-2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c24a2bf30ff5daffbfbe40c1ca2b3484de34a69c"},"cell_type":"markdown","source":"## Analyzing results: loss and accuracy"},{"metadata":{"trusted":true,"_uuid":"94795083d21b80242276cbfd59d0615034f27e01"},"cell_type":"code","source":"def binary_loss(y, p):\n    return np.mean(-(y * np.log(p) + (1-y)*np.log(1-p)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"97238b7ca1b4e855c71a82036d2d24ba1db8757f"},"cell_type":"code","source":"acts_z = np.array([1, 0, 0, 1])\npreds_z = np.array([0.9, 0.1, 0.2, 0.8])\nbinary_loss(acts_z, preds_z)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4319ade50a4f585091f9dc775f71a8b7376daf4"},"cell_type":"markdown","source":"### Location to retrieve the trained models\n\nYou should see .pth files there. Is this location persistent? or should we transfer it to \"output\" of kaggle? "},{"metadata":{"trusted":true,"_uuid":"b66ba37dcf824b0550ec698ff154fce08eb96bcd"},"cell_type":"code","source":"!ls '/tmp/.fastai/data/dogscats/models'","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}