{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"feb563af4d76e5dd3d74a7c554137930e6abe26a"},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"collapsed":true},"cell_type":"code","source":"from fastai.imports import *\nfrom fastai.torch_imports import *\nfrom fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"60858e8d0dffebd62e156f724b185f88ca06423b"},"cell_type":"code","source":"torch.cuda.set_device(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"046310be46c028d3dc6e3d2e87b58b49601fbab3"},"cell_type":"code","source":"PATH = \"data/dogscats/\"\nsz=224","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8841aea9f8aaeaac2d9951358a8a7f5f54d88ae"},"cell_type":"markdown","source":"**First Look at Cat Pictures**\n\nOur library will assume that you have train and valid directories. It also assumes that each dir will have subdirs for each class you wish to recognize (in this case, 'cats' and 'dogs').\n"},{"metadata":{"trusted":true,"_uuid":"3f575c1ecd63af5cd1fcde126e4cf0aea76c4f76","collapsed":true},"cell_type":"code","source":"os.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ebbb434c5915fadf46756c0b37837b3de02d091","collapsed":true},"cell_type":"code","source":"os.listdir(f'{PATH}valid')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c6e9deda4052203f32cec9377c03ed7ed2b7e21b","collapsed":true},"cell_type":"code","source":"files = os.listdir(f'{PATH}valid/cats')[:5]\nfiles","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17e93ba2ea660ede614f714f1bab91578bc5d8a3","collapsed":true},"cell_type":"code","source":"img = plt.imread(f'{PATH}valid/cats/{files[0]}')\nplt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fcc24fe000e52bc3059bb86955d32b5c6e5e4cfb","collapsed":true},"cell_type":"code","source":"img.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f3d469edeabfaf6dc9b3efd8b73a249a9b4c6ad1","collapsed":true},"cell_type":"code","source":"img[:4,:4]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c55340edccf1a6d237a6bf1affbb76205358a563"},"cell_type":"markdown","source":"**Our first model: quick start**\nWe're going to use a **pre-trained model**, that is, a model created by some one else to solve a different problem. Instead of building a model from scratch to solve a similar problem, we'll use a model trained on ImageNet (1.2 million images and 1000 classes) as a starting point. The model is a Convolutional Neural Network (CNN), a type of Neural Network that builds state-of-the-art models for computer vision. We'll be learning all about CNNs during this course.\nWe will be using the **resnet34** model. resnet34 is a version of the model that won the 2015 ImageNet competition. Here is more info on resnet models. We'll be studying them in depth later, but for now we'll focus on using them effectively.\n\nHere's how to train and evalulate a dogs vs cats model in 3 lines of code, and under 20 seconds:"},{"metadata":{"trusted":true,"_uuid":"6f58b5488b47033a6a106655b517b349528f5b21","collapsed":true},"cell_type":"code","source":"arch=resnet34\ndata = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))\nlearn = ConvLearner.pretrained(arch, data, precompute=True)\nlearn.fit(0.01, 2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"99a7016b9047720cc46913cdb8312578e3215da7"},"cell_type":"markdown","source":"**Analyzing results: looking at pictures**"},{"metadata":{"trusted":true,"_uuid":"7a7b30ef6773a6854b413e2cf3d47ffa541cfe80","collapsed":true},"cell_type":"code","source":"# This is the label for a val data# This i \ndata.val_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fdd80ad5fc93b6026f72056c0991f877c3b71c36","collapsed":true},"cell_type":"code","source":"# from here we know that 'cats' is label 0 and 'dogs' is label 1.\ndata.classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2c1a9e8b88e3136b9352a1dad88a026f9a65c51","collapsed":true},"cell_type":"code","source":"# this gives prediction for validation set. Predictions are in log scale\nlog_preds = learn.predict()\nlog_preds.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af941d15308aafbd7fbb8c038ddb66a13c8d6f5e","collapsed":true},"cell_type":"code","source":"log_preds[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"683f230afd9fbfb39909e0c21e3b366472d7a3b6"},"cell_type":"code","source":"preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1\nprobs = np.exp(log_preds[:,1])        # pr(dog)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"5bc0220a57a01c5cccfdba86d78ac467a58682d8"},"cell_type":"code","source":"def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], 4, replace=False)\ndef rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"5f61080bbb285c56e6ded946b4a935a3fa627f59"},"cell_type":"code","source":"def plots(ims, figsize=(12,6), rows=1, titles=None):\n    f = plt.figure(figsize=figsize)\n    for i in range(len(ims)):\n        sp = f.add_subplot(rows, len(ims)//rows, i+1)\n        sp.axis('Off')\n        if titles is not None: sp.set_title(titles[i], fontsize=16)\n        plt.imshow(ims[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"688b38c19cf992526445891c357dc75bb8212346"},"cell_type":"code","source":"def load_img_id(ds, idx): return np.array(PIL.Image.open(PATH+ds.fnames[idx]))\n\ndef plot_val_with_title(idxs, title):\n    imgs = [load_img_id(data.val_ds,x) for x in idxs]\n    title_probs = [probs[x] for x in idxs]\n    print(title)\n    return plots(imgs, rows=1, titles=title_probs, figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9c06a589f536efcf308bcfc0d39b984a4831516","collapsed":true},"cell_type":"code","source":"# 1. A few correct labels at random\nplot_val_with_title(rand_by_correct(True), \"Correctly classified\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2ef4d83ec4175e7182162eb77cc0bf1e28d73443","collapsed":true},"cell_type":"code","source":"# 2. A few incorrect labels at random\nplot_val_with_title(rand_by_correct(False), \"Incorrectly classified\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d1550fe513a57c88cd4725955741862280284ee6"},"cell_type":"code","source":"def most_by_mask(mask, mult):\n    idxs = np.where(mask)[0]\n    return idxs[np.argsort(mult * probs[idxs])[:4]]\n\ndef most_by_correct(y, is_correct): \n    mult = -1 if (y==1)==is_correct else 1\n    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"7c4a6f28c801cd4df063e382b98ad938b64c73d4","collapsed":true},"cell_type":"code","source":"plot_val_with_title(most_by_correct(0, True), \"Most correct cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da0c63361bc2acfac9977744ad1704ca89453e20","collapsed":true},"cell_type":"code","source":"plot_val_with_title(most_by_correct(1, True), \"Most correct dogs\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b154f68d16a9ee8a0f258cdc1e1e64f448a58a6a","collapsed":true},"cell_type":"code","source":"\nplot_val_with_title (most_by_correct(0, False), \"Most incorrect cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8255dcd784070e24abdb13d7b0d654df45ce304d","collapsed":true},"cell_type":"code","source":"plot_val_with_title(most_by_correct(1, False), \"Most incorrect dogs\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bfce3223d488a61c844e603efa40200cd0e4d5ae","collapsed":true},"cell_type":"code","source":"most_uncertain = np.argsort(np.abs(probs -0.5))[:4]\nplot_val_with_title(most_uncertain, \"Most uncertain predictions\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"813ead76c6aa89d4ad1a38037a799a88e714e94c"},"cell_type":"markdown","source":"**Choosing a learning rate**\n\nThe learning rate determines how quickly or how slowly you want to update the weights (or parameters). Learning rate is one of the most difficult parameters to set, because it significantly affects model performance.\n\nThe method learn.lr_find() helps you find an optimal learning rate. It uses the technique developed in the 2015 paper Cyclical Learning Rates for Training Neural Networks, where we simply keep increasing the learning rate from a very small value, until the loss stops decreasing. We can plot the learning rate across batches to see what this looks like.\n\nWe first create a new learner, since we want to know how to set the learning rate for a new (untrained) model."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"bbd8c00df344697c3e8c66cb4d1d5af8079711da"},"cell_type":"code","source":"learn = ConvLearner.pretrained(arch, data, precompute=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a4238ef28762ad43e7b84c7730ad36b0097d0f4","collapsed":true},"cell_type":"code","source":"lrf=learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f51f836e0804d6879852e55394568cd94366983"},"cell_type":"markdown","source":"\n Our learn *object* contains an attribute sched that contains our learning rate scheduler, and has some convenient plotting functionality including this one:"},{"metadata":{"trusted":true,"_uuid":"ecd2ae619337596b6a7d1c850be315e3b5072b10","collapsed":true},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"16df43cf403751451359fcdc3a8d69a1060a392b"},"cell_type":"markdown","source":"\nNote that in the previous plot iteration is one *iteration* (or *minibatch*) of SGD. In one epoch there are (num_train_samples/num_iterations) of SGD.\n\nWe can see the plot of loss versus learning rate to see where our loss stops decreasing:"},{"metadata":{"trusted":true,"_uuid":"d54650560a1f000aec89366234cc0bbebd642cbc","collapsed":true},"cell_type":"code","source":"learn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8565aae998739e004d552b464464f0cb16952f19"},"cell_type":"markdown","source":"The loss is still clearly improving at lr=1e-2 (0.01), so that's what we use. Note that the optimal learning rate can change as we train the model, so you may want to re-run this function from time to time."},{"metadata":{"_uuid":"050b500863dae7509c2d160ffa7c998acf926595"},"cell_type":"markdown","source":"**Improving our model**\n\n*Data Augmentation*\nIf you try training for more epochs, you'll notice that we start to *overfit*, which means that our model is learning to recognize the specific images in the training set, rather than generalizing such that we also get good results on the validation set. One way to fix this is to effectively create more data, through *data augmentation*. This refers to randomly changing the images in ways that shouldn't impact their interpretation, such as horizontal flipping, zooming, and rotating.\n\nWe can do this by passing aug_tfms (*augmentation transforms*) to tfms_from_model, with a list of functions to apply that randomly change the image however we wish. For photos that are largely taken from the side (e.g. most photos of dogs and cats, as opposed to photos taken from the top down, such as satellite imagery) we can use the pre-defined list of functions transforms_side_on. We can also specify random zooming of images up to specified scale by adding the max_zoom parameter."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"4a3027c65e68118ea09083b974ad66ea71ee0d3a"},"cell_type":"code","source":"tfms = tfms_from_model(resnet34, sz, aug_tfms=transforms_side_on, max_zoom=1.1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"f1416d39641331f6a944b8f870007436ff79421b"},"cell_type":"code","source":"def get_augs():\n    data = ImageClassifierData.from_paths(PATH, bs=2, tfms=tfms, num_workers=1)\n    x,_ = next(iter(data.aug_dl))\n    return data.trn_ds.denorm(x)[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"9bff12e8a0f933df7126e556a08789793bcdd651"},"cell_type":"code","source":"ims = np.stack([get_augs() for i in range(6)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9674797ee2c80a7be3e0123e981b3d434ae86698","collapsed":true},"cell_type":"code","source":"plots(ims, rows=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d42dd36f23d560ea83442c49d26d2edc717c40d3"},"cell_type":"markdown","source":"Let's create a new data object that includes this augmentation in the transforms."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"f613b2d299ecac8a4b8389c09ca52f14f71b1757"},"cell_type":"code","source":"data = ImageClassifierData.from_paths(PATH, tfms=tfms)\nlearn = ConvLearner.pretrained(arch, data, precompute=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e1768246f713c50a5aff13373a903b3c10515517","collapsed":true},"cell_type":"code","source":"learn.fit(1e-2, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"e1c4ea59806212fdbdee9141987720c008571b00"},"cell_type":"code","source":"learn.precompute=False","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"281e23a7dd6efa47f4c07b8973f4add63bc539b7"},"cell_type":"markdown","source":"By default when we create a learner, it sets all but the last layer to *frozen*. That means that it's still only updating the weights in the last layer when we call *fit*."},{"metadata":{"trusted":true,"_uuid":"7d8993508483c88105e7c602b0b5564ffa5fb6dd","collapsed":true},"cell_type":"code","source":"learn.fit(1e-2, 3, cycle_len=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"896161e926037baab0544e7b66e322441e9e58f6"},"cell_type":"markdown","source":"What is that *cycle_len* parameter? What we've done here is used a technique called stochastic gradient descent with restarts (SGDR), a variant of *learning rate annealing*, which gradually decreases the learning rate as training progresses. This is helpful because as we get closer to the optimal weights, we want to take smaller steps.\n\nHowever, we may find ourselves in a part of the weight space that isn't very resilient - that is, small changes to the weights may result in big changes to the loss. We want to encourage our model to find parts of the weight space that are both accurate and stable. Therefore, from time to time we increase the learning rate (this is the 'restarts' in 'SGDR'), which will force the model to jump to a different part of the weight space if the current area is \"spikey\". Here's a picture of how that might look if we reset the learning rates 3 times (in this paper they call it a \"cyclic LR schedule\"):\n![](https://github.com/fastai/fastai/raw/4649f74e8f365f09fe7ed4a94535d688b5b2d7d5/courses/dl1/images/sgdr.png)\n(From the paper [Snapshot Ensembles](https://arxiv.org/abs/1704.00109)).\n\nThe number of epochs between resetting the learning rate is set by *cycle_len*, and the number of times this happens is refered to as the number of cycles, and is what we're actually passing as the 2nd parameter to *fit()*. So here's what our actual learning rates looked like:\n\n"},{"metadata":{"trusted":true,"_uuid":"ac8370626809905842e4969d6deb297ac9e90846","collapsed":true},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70e44bc8a4cfd2334e3065ecb61d1806ac3f292d"},"cell_type":"markdown","source":"Our validation loss isn't improving much, so there's probably no point further training the last layer on its own.\n\nSince we've got a pretty good model at this point, we might want to save it so we can load it again later without training it from scratch."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"7a3feb457239b48374c1ed5828fcc3b83e2d19f6"},"cell_type":"code","source":"learn.save('224_lastlayer')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"35b1371149a29ba259f692a70e07b2f0f74117da"},"cell_type":"code","source":"learn.load('224_lastlayer')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f30c7997249b2a4601347fc6ad416690389e8a9"},"cell_type":"markdown","source":"**Fine-tuning and differential learning rate annealing**\n\nNow that we have a good final layer trained, we can try fine-tuning the other layers. To tell the learner that we want to unfreeze the remaining layers, just call (surprise surprise!) unfreeze()."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c127b9f6ec7ba5f37400b67178ab4d7a9891009f"},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9000a9cde5bdd5096ddec4009d213f8358019d07"},"cell_type":"markdown","source":"Note that the other layers have *already* been trained to recognize imagenet photos (whereas our final layers where randomly initialized), so we want to be careful of not destroying the carefully tuned weights that are already there.\n\nGenerally speaking, the earlier layers (as we've seen) have more general-purpose features. Therefore we would expect them to need less fine-tuning for new datasets. For this reason we will use different learning rates for different layers: the first few layers will be at 1e-4, the middle layers at 1e-3, and our FC layers we'll leave at 1e-2 as before. We refer to this as differential learning rates, although there's no standard name for this techique in the literature that we're aware of."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"a674ca9ac630c3997599c46e1dcfea4d7a283d33"},"cell_type":"code","source":"lr=np.array([1e-4,1e-3,1e-2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f25ecd2c57e0736d483ef434a024d5938079ac9a","_kg_hide-output":false,"collapsed":true},"cell_type":"code","source":"learn.fit(lr, 3, cycle_len=1, cycle_mult=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f616a8625819dc30b66addfd348eef2ee26c105a","collapsed":true},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"6a43bce051c1761aa32791adb37e5692f02aeeea"},"cell_type":"code","source":"learn.save('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"93510875fde0fcc98175bf769b0ac5fcc6acef3a"},"cell_type":"code","source":"learn.load('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"37da4cd67af9919e51599654d75fabac119a60f8"},"cell_type":"markdown","source":"There is something else we can do with data augmentation: use it at *inference* time (also known as test time). Not surprisingly, this is known as test time augmentation, or just TTA.\n\nTTA simply makes predictions not just on the images in your validation set, but also makes predictions on a number of randomly augmented versions of them too (by default, it uses the original image along with 4 randomly augmented versions). It then takes the average prediction from these images, and uses that. To use TTA on the validation set, we can use the learner's TTA() method."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"7a47120cdfb4e366dd2a302800dc4961518e8cdb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}