{"cells":[{"metadata":{"_uuid":"5d52707f344f0dcfe5baa73532d64fc3456e06c6"},"cell_type":"markdown","source":"## Image classification with Convolutional Neural Networks"},{"metadata":{"_uuid":"c1a51714a4b27163ad6421b9661ba6c872eed83b"},"cell_type":"markdown","source":"Welcome to the first week of the second deep learning certificate! We're going to use convolutional neural networks (CNNs) to allow our computer to see - something that is only possible thanks to deep learning."},{"metadata":{"_uuid":"fe57324f4745fb9d4383198da02a202031625fbb"},"cell_type":"markdown","source":"## Introduction to our first task: 'Dogs vs Cats'"},{"metadata":{"_uuid":"dbac44c46ab037faf84621b6c1a8d582b7ef31a4"},"cell_type":"markdown","source":"We're going to try to create a model to enter the Dogs vs Cats competition at Kaggle. There are 25,000 labelled dog and cat photos available for training, and 12,500 in the test set that we have to try to label for this competition. According to the Kaggle web-site, when this competition was launched (end of 2013): \"State of the art: The current literature suggests machine classifiers can score above 80% accuracy on this task\". So if we can beat 80%, then we will be at the cutting edge as of 2013!"},{"metadata":{"_uuid":"acec03a5349f652365134989c150cbdf28ba2076","trusted":true},"cell_type":"code","source":"# Put these at the top of every notebook, to get automatic reloading and inline plotting\n%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"600da1f542a804eca39699c43f0b3fb4246d8a6e"},"cell_type":"markdown","source":"Here we import the libraries we need. We'll learn about what each does during the course."},{"metadata":{"trusted":true,"_uuid":"4b6d214ff3acee7ea972c49613f0d020c5bf9a9e"},"cell_type":"code","source":"import pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25cb4b4f0b501fa618f0e3b82d6c8f0420173d6a"},"cell_type":"code","source":"!pip freeze | grep fastai","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23458afac17c6c5ab691b5bad3e1292082539bfc","trusted":true},"cell_type":"code","source":"# This file contains all the main external libs we'll use\nfrom fastai.imports import *\nfrom fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"188e90203f18be6de43fd149dbe94af0da40eb03"},"cell_type":"markdown","source":"`PATH` is the path to your data - if you use the recommended setup approaches from the lesson, you won't need to change this. `sz` is the size that the images will be resized to in order to ensure that the training runs quickly. We'll be talking about this parameter a lot during the course. Leave it at `224` for now."},{"metadata":{"trusted":true,"_uuid":"446f36ba16d4f8560b898f744d2bd12fa6308f50"},"cell_type":"code","source":"ConvLearner","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"276daac1c68af55ffc03e89d20c853bdf23e8522","trusted":true},"cell_type":"code","source":"PATH = \"../input/\"\nTMP_PATH = \"/tmp/tmp\"\nMODEL_PATH = \"/tmp/model/\"\nsz=224","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a6564a6294ee094ba60038c5a94074318f18337"},"cell_type":"markdown","source":"It's important that you have a working NVidia GPU set up. The programming framework used to behind the scenes to work with NVidia GPUs is called CUDA. Therefore, you need to ensure the following line returns `True` before you proceed. If you have problems with this, please check the FAQ and ask for help on [the forums](http://forums.fast.ai)."},{"metadata":{"_uuid":"9ba310c51d959cab508081d01d87a3aaa2ab5085","trusted":true},"cell_type":"code","source":"torch.cuda.is_available()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13b963795d3ecd9c86cffa9044eaf5145df56e56"},"cell_type":"markdown","source":"In addition, NVidia provides special accelerated functions for deep learning in a package called CuDNN. Although not strictly necessary, it will improve training performance significantly, and is included by default in all supported fastai configurations. Therefore, if the following does not return `True`, you may want to look into why."},{"metadata":{"_uuid":"aeffd45f60ecfcb90aa0cd37f09bbb89cbee8c01","trusted":true},"cell_type":"code","source":"torch.backends.cudnn.enabled","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true,"_uuid":"e7371d1918f4903352b82bb8f4b2ca55fa4386ca"},"cell_type":"markdown","source":"## First look at cat pictures"},{"metadata":{"hidden":true,"scrolled":true,"_uuid":"ee3a6d31ba52de6d1f294998d48b56073ee31604","trusted":true},"cell_type":"code","source":"os.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"a9fb6a8048fa18c65800b1865876bf8cccfa0e86","trusted":true},"cell_type":"code","source":"fnames = np.array([f'train/{f}' for f in sorted(os.listdir(f'{PATH}train'))])\n# labels = np.array([(0 if 'cat' in fname else 1) for fname in fnames])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"701892de64ac6b86b58337c429544781b77e1c43"},"cell_type":"code","source":"fnames[:10]","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"09f87570c2cd82eb3eed603565240021a5a888fe","trusted":true},"cell_type":"code","source":"img = plt.imread(f'{PATH}{fnames[10]}')\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"1d87111c18c0cb0b3326f8e5ff61e18651c13b40"},"cell_type":"markdown","source":"Here is how the raw data looks like"},{"metadata":{"hidden":true,"_uuid":"5ba79cbbb1fbab9dbcbc35834f62a8b024942d2e","trusted":true},"cell_type":"code","source":"img.shape","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"cfef74b040d59c22cb769c6cdfa5cb9cf40066cf","trusted":true},"cell_type":"code","source":"img[:4,:4]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"068c9d47b0032d276dba5f128f6b7684c1a5dd07"},"cell_type":"markdown","source":"## Label df"},{"metadata":{"trusted":true,"_uuid":"16369beade3d42662b7dbebc2818fc634d84c06e"},"cell_type":"code","source":"!head ../input/labels.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82d6f4ff7cdc9e61fab720aa2500979bbe029ba8"},"cell_type":"code","source":"!ls ../input/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12c8a410f6a1c804895159c58b85aef812e9a42a"},"cell_type":"code","source":"label_df = pd.read_csv(f'{PATH}/labels.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d503f6467e0e3f4d46b43c4b8629a3a91b2c5b98"},"cell_type":"code","source":"label_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"a921fd897bbcaa20c8aa7d5aba67a0ee249c91f9"},"cell_type":"code","source":"label_df['breed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true,"_uuid":"500f86792c2bd9c9373d883eb301b74219a9b2a7"},"cell_type":"markdown","source":"## Our first model: quick start"},{"metadata":{"hidden":true,"_uuid":"98cddf2e4f8c750fb55756749af75acb62500c73"},"cell_type":"markdown","source":"We're going to use a <b>pre-trained</b> model, that is, a model created by some one else to solve a different problem. Instead of building a model from scratch to solve a similar problem, we'll use a model trained on ImageNet (1.2 million images and 1000 classes) as a starting point. The model is a Convolutional Neural Network (CNN), a type of Neural Network that builds state-of-the-art models for computer vision. We'll be learning all about CNNs during this course.\n\nWe will be using the <b>resnet34</b> model. resnet34 is a version of the model that won the 2015 ImageNet competition. Here is more info on [resnet models](https://github.com/KaimingHe/deep-residual-networks). We'll be studying them in depth later, but for now we'll focus on using them effectively.\n\nHere's how to train and evalulate a *dogs vs cats* model in 3 lines of code, and under 20 seconds:"},{"metadata":{"trusted":true,"_uuid":"7047f8fc4ca947728151d5f0e28496102aacdfd3"},"cell_type":"code","source":"# arch=resnet101\n# arch = resnet152\narch = resnet50\nbs = 56\nsz = 224","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a6657aa00e8c0997c2b8af7c771fd6f94397e5e"},"cell_type":"code","source":"n = len(label_df)\nval_idxs = get_cv_idxs(n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7aa60a8f5fecaf1e2054fbb53029fa881108ea24","trusted":true},"cell_type":"code","source":"# Uncomment the below if you need to reset your precomputed activations\n# shutil.rmtree(f'{PATH}tmp', ignore_errors=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b288b58d88a50b2eb5054b7eb679ea34165f6d58"},"cell_type":"code","source":"tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)\ndata = ImageClassifierData.from_csv(f'{PATH}', 'train', f'{PATH}labels.csv', bs=bs, test_name='test', \n                                    val_idxs=val_idxs, tfms=tfms, suffix='.jpg')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"79cae42cbaa193cda4e2971d5b25d6c52f9533ba"},"cell_type":"markdown","source":"### Look at data"},{"metadata":{"trusted":true,"_uuid":"cf7a5f37f9f5bc048d593a7fc2f44c1dcf9358ec"},"cell_type":"code","source":"fn = PATH + data.trn_ds.fnames[0]; fn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea164266741f325312d40ddb71d490d2adeab5fa"},"cell_type":"code","source":"img = PIL.Image.open(fn); img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"483f0a34b06b869d3613c7ef85df764dbfcf7d3a"},"cell_type":"code","source":"img.size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6939d4d0fdb5ad1393e06342feb8272df43c629c"},"cell_type":"code","source":"size_d = {k: PIL.Image.open(PATH + k).size for k in data.trn_ds.fnames}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee9ab31c62f4b861973a25272408617c28c7eb35"},"cell_type":"code","source":"row_sz, col_sz = zip(*size_d.values())\nrow_sz = np.array(row_sz); col_sz = np.array(col_sz)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e31c798d01fb4130e5bdc7bba35fca7efa58f5ec"},"cell_type":"code","source":"from collections import Counter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"689e375e7b872fc9558041ed005d2090458d3cec"},"cell_type":"code","source":"len(Counter(row_sz))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c8e70f4909231e2c22ce09bd074ff1286aa53732"},"cell_type":"code","source":"len(Counter(col_sz))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c405300e9f7eab8a6f48bf848c7ad739122e0a6e"},"cell_type":"code","source":"plt.hist(row_sz)\nplt.title('Row size histogram')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1db4fcfa8685d18d97ea14062fa11b8c2a60469e"},"cell_type":"code","source":"plt.hist(row_sz[row_sz < 1000])\nplt.title('Row size histogram')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3bbcc2a7f0429088e5900665badedb54ea36bfb"},"cell_type":"code","source":"plt.hist(col_sz[col_sz < 1000])\nplt.title('Col size histogram')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"203773766c3ad96bb843d1bcf4eab6ec6d97060b"},"cell_type":"markdown","source":"### Traning\n"},{"metadata":{"trusted":true,"_uuid":"c5047c58ec7b43a0c31a2837f75c5919e21e727b"},"cell_type":"code","source":"len(label_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6f81c592513edeb611975ac2849c61bee669206"},"cell_type":"code","source":"def get_data(sz, bs):\n    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)\n    data = ImageClassifierData.from_csv(f'{PATH}', 'train', f'{PATH}labels.csv', bs=bs, test_name='test', \n                                    val_idxs=val_idxs, tfms=tfms, suffix='.jpg')\n    return data if sz > 300 else data.resize(340, TMP_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"00faecd319cbe66d4d841dbcf04b4e6ad5c6eba8"},"cell_type":"markdown","source":"## 3.1. Precompute "},{"metadata":{"trusted":true,"_uuid":"f73a19326a01d95a2af440173cbdad9a79e758ea"},"cell_type":"code","source":"data = get_data(sz, bs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"463227cd4ea6dd1f510fb4bbe4b8bc0cc87b4480"},"cell_type":"code","source":"learn = ConvLearner.pretrained(arch, data, precompute=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9b396d4076cf4c2f4b101581a907fd61ddc36d8c"},"cell_type":"code","source":"learn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e831691d65d607c7023887405247f419592a1fd"},"cell_type":"code","source":"learn.fit(0.01, 5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cdac880ae15d0679fab90a2c19022083f2d1023a"},"cell_type":"markdown","source":"## 3.2. Augment"},{"metadata":{"trusted":true,"_uuid":"a441a27d3bf938b6884411090f4bba7124d27c58"},"cell_type":"code","source":"data = get_data(sz, bs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"704599dea424767aa9e328cb3000dafb6e9d530f","scrolled":true},"cell_type":"code","source":"learn = ConvLearner.pretrained(arch, data, precompute=True, ps=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aaef3500d7d308da7b59ef5ca901f8094abc4b24"},"cell_type":"code","source":"learn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"837af047b919e726a1eb39d9bc3c9fe158756136"},"cell_type":"code","source":"lrf=learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eaa3250f6a0caa20dafb48218c7145e16af124c4"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b80f0d0546c40dd80e3f3f1cf8984e83c4c8ae50"},"cell_type":"code","source":"learn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e13fad5914d63b04ccbab329665073727996bad1"},"cell_type":"code","source":"learn.fit(1e-2, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5eb013b19848a6d1cce6764381fb2557e728eb7"},"cell_type":"code","source":"learn.precompute = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e13fad5914d63b04ccbab329665073727996bad1"},"cell_type":"code","source":"learn.fit(1e-2, 5, cycle_len=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9acdac8182d647a79f3fc77de51c8c3458bb8e4"},"cell_type":"code","source":"learn.save('224_pre')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5c90fb368f82a21c4fcd08293c75a369cf47635"},"cell_type":"markdown","source":"## 3.3 Increase image size"},{"metadata":{"trusted":true,"_uuid":"3d8246d1772ebebea8102b875464d29c4e01640a"},"cell_type":"code","source":"learn.set_data(get_data(299, bs))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e57ad25990a430540aacd7df1849c1ce09c5e0d"},"cell_type":"code","source":"learn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c898261c409c5967d21d01eacd9a68a2df4576c"},"cell_type":"code","source":"tfms_from_model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41eb49054f91d78034061fb128a141f7745b81b0"},"cell_type":"code","source":"learn.fit(1e-2, n_cycle=3, cycle_len=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41eb49054f91d78034061fb128a141f7745b81b0"},"cell_type":"code","source":"learn.fit(1e-2, n_cycle=3, cycle_len=1, cycle_mult=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bf5338e13d8bb4cb35c93842ce2ed26b10218f7"},"cell_type":"code","source":"learn.save('299_pre')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19dfea211019ec4b4465e2604004150cf608007f"},"cell_type":"code","source":"learn.load('299_pre')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee13a4cbb7fa3b66f8a0abe572d98cf79e560585"},"cell_type":"code","source":"learn.fit(1e-2, n_cycle=1, cycle_len=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70a4398996a484a4bc937eb6c1c812254c7c5210"},"cell_type":"code","source":"log_preds, y = learn.TTA()\nprobs = np.mean(np.exp(log_preds),0)\npreds = np.argmax(probs, axis=1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a261b66ac2fb0ee47f43a95bab93c76f987eca65"},"cell_type":"code","source":"log_preds.shape, y.shape, probs.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02f3e89d9cfac168cc166ac53bb465c40fb71b6b"},"cell_type":"code","source":"# probs = np.exp(log_preds)\naccuracy_np(probs, y), metrics.log_loss(y, probs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"79ea922e9313d9e9801bef36105cd61ac9b555c6"},"cell_type":"code","source":"accuracy(preds, y)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"3fd2cdb6e89c1163a19008ce233445b192db8d9a"},"cell_type":"markdown","source":"How good is this model? Well, as we mentioned, prior to this competition, the state of the art was 80% accuracy. But the competition resulted in a huge jump to 98.9% accuracy, with the author of a popular deep learning library winning the competition. Extraordinarily, less than 4 years later, we can now beat that result in seconds! Even last year in this same course, our initial model had 98.3% accuracy, which is nearly double the error we're getting just a year later, and that took around 10 minutes to compute."},{"metadata":{"heading_collapsed":true,"_uuid":"17f0e5c8170bbcd4a0371c95d5a136e9674f7c31"},"cell_type":"markdown","source":"## Analyzing results: looking at pictures"},{"metadata":{"hidden":true,"_uuid":"b986461621fef09d1aadfe6c7c04ff21f18eca6e"},"cell_type":"markdown","source":"As well as looking at the overall metrics, it's also a good idea to look at examples of each of:\n1. A few correct labels at random\n2. A few incorrect labels at random\n3. The most correct labels of each class (i.e. those with highest probability that are correct)\n4. The most incorrect labels of each class (i.e. those with highest probability that are incorrect)\n5. The most uncertain labels (i.e. those with probability closest to 0.5)."},{"metadata":{"hidden":true,"_uuid":"942e14027848dcad6d6c80422b5bbf56f63bc1c7","trusted":true},"cell_type":"code","source":"# This is the label for a val data\ndata.val_y","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"6378a93c7e03118e0b12d423d31b2be8f95c1005","trusted":true},"cell_type":"code","source":"# from here we know that 'cats' is label 0 and 'dogs' is label 1.\ndata.classes","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"1326d53d70acad37272afa71c20d6ff39e165c89","trusted":true},"cell_type":"code","source":"# this gives prediction for validation set. Predictions are in log scale\nlog_preds = learn.predict()\nlog_preds.shape","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"b7ff16accb5fa5104de318bd261abcdee72530aa","trusted":true},"cell_type":"code","source":"log_preds[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc7b16d158b44355be9663fcf4b0ea835ab493c4"},"cell_type":"code","source":"probs.shape","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"b8709f3bac05d50ee2f7b0b774eb86af8f0f709c","trusted":true},"cell_type":"code","source":"preds = np.argmax(probs, axis=1)  # from log probabilities to 0 or 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08e16f1689f0c09929c24ded9025ab9f9900d936"},"cell_type":"code","source":"preds.shape","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"37901ee29c7a90714b6385d51eb4126b890f4f8e","trusted":true},"cell_type":"code","source":"def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], 4, replace=False)\ndef rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"f9de40f3a17b608fb5c4baea893e138c88464fd7","trusted":true},"cell_type":"code","source":"def plots(ims, figsize=(12,6), rows=1, titles=None):\n    f = plt.figure(figsize=figsize)\n    for i in range(len(ims)):\n        sp = f.add_subplot(rows, len(ims)//rows, i+1)\n        sp.axis('Off')\n        if titles is not None: sp.set_title(titles[i], fontsize=16)\n        plt.imshow(ims[i])","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"f64c7bef125f50701884273ed8d161de25b9703b","trusted":true},"cell_type":"code","source":"def load_img_id(ds, idx): return np.array(PIL.Image.open(PATH+ds.fnames[idx]))\n\ndef plot_val_with_title(idxs, title):\n    imgs = [load_img_id(data.val_ds,x) for x in idxs]\n#     title_probs = [max(probs[x]) for x in idxs]\n    title_probs = [f'Label: {data.classes[data.val_y[x]]} \\n Pred: {data.classes[preds[x]]} \\n {max(probs[x])}' for x in idxs ]\n    print(title)\n    return plots(imgs, rows=1, titles=title_probs, figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"9b2fac071b6f72c82cb4a499dffe9dc77b5b57e3","trusted":true},"cell_type":"code","source":"# 1. A few correct labels at random\nplot_val_with_title(rand_by_correct(True), \"Correctly classified\")","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"2380c1cc7a75c19c96a72fdafed058c7fbd92815","trusted":true},"cell_type":"code","source":"# 2. A few incorrect labels at random\nplot_val_with_title(rand_by_correct(False), \"Incorrectly classified\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c46297623e4ebd4fd6023028475b1575cf67ba1"},"cell_type":"markdown","source":"## 4. Predict"},{"metadata":{"trusted":true,"_uuid":"d881cf74478598b80a9e09193160c4c3f9c6f54f"},"cell_type":"code","source":"log_preds, y = learn.TTA(is_test=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ccdd8133320aebc45eb9b8375794dd61d8dfffee"},"cell_type":"code","source":"len(data.test_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e8714e2875381c3fb5e406a1bb1939aee460b69b"},"cell_type":"code","source":"probs = np.mean(np.exp(log_preds),0)\npreds = np.argmax(probs, axis=1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"616266a54e481d38c029e8c22ffcb9797d939ae3"},"cell_type":"code","source":"df = pd.DataFrame(probs, columns=data.classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d6007c07449111716b96ac70c3281caf246527f"},"cell_type":"code","source":"iddf = pd.DataFrame([x.split('/')[1].split('.')[0] for x in data.test_ds.fnames], columns=['id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f76113a402073c0e921f6f365a75a0dd229b2a0f"},"cell_type":"code","source":"iddf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e98b9478d3e85e2a267780c2b427cdca7340199"},"cell_type":"code","source":"testdf = pd.concat([iddf, df], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a93081b65692b9fe5e5efcc44a73006a1a27f0fd"},"cell_type":"code","source":"testdf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6333aabf89a0168204906bc4d6dc6525488ba0b8"},"cell_type":"code","source":"testdf.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"cb6102e09fd3c172d9f3ff644b97a6fa488b24c9"},"cell_type":"markdown","source":"There is something else we can do with data augmentation: use it at *inference* time (also known as *test* time). Not surprisingly, this is known as *test time augmentation*, or just *TTA*.\n\nTTA simply makes predictions not just on the images in your validation set, but also makes predictions on a number of randomly augmented versions of them too (by default, it uses the original image along with 4 randomly augmented versions). It then takes the average prediction from these images, and uses that. To use TTA on the validation set, we can use the learner's `TTA()` method."},{"metadata":{"hidden":true,"_uuid":"fcb92d519aaa2d5a8982a6f7c280ec6ee6a0916d"},"cell_type":"markdown","source":"I generally see about a 10-20% reduction in error on this dataset when using TTA at this point, which is an amazing result for such a quick and easy technique!"},{"metadata":{"_uuid":"3068e3a8473a2c1ac558780c107d0158bd338e51"},"cell_type":"markdown","source":"## Analyzing results"},{"metadata":{"heading_collapsed":true,"_uuid":"667093c0f80a6477a5d25c4fbe3f0e071a9520ab"},"cell_type":"markdown","source":"### Confusion matrix "},{"metadata":{"heading_collapsed":true,"_uuid":"3e3db3bd9fe817641063ff0e37bfb4bcf6b33967"},"cell_type":"markdown","source":"## Review: easy steps to train a world-class image classifier"},{"metadata":{"hidden":true,"_uuid":"54eef5b76bc4e37be5aee47b7eab68fe04bea567"},"cell_type":"markdown","source":"1. precompute=True\n1. Use `lr_find()` to find highest learning rate where loss is still clearly improving\n1. Train last layer from precomputed activations for 1-2 epochs\n1. Train last layer with data augmentation (i.e. precompute=False) for 2-3 epochs with cycle_len=1\n1. Unfreeze all layers\n1. Set earlier layers to 3x-10x lower learning rate than next higher layer\n1. Use `lr_find()` again\n1. Train full network with cycle_mult=2 until over-fitting"},{"metadata":{"heading_collapsed":true,"_uuid":"1c8befa45315236b9d6dd4a914dc0f92e796539e"},"cell_type":"markdown","source":"## Analyzing results: loss and accuracy"},{"metadata":{"hidden":true,"_uuid":"3b04656ea2ab9401544240311d337f5978c18b51"},"cell_type":"markdown","source":"When we run `learn.fit` we print 3 performance values (see above.) Here 0.03 is the value of the **loss** in the training set, 0.0226 is the value of the loss in the validation set and 0.9927 is the validation accuracy. What is the loss? What is accuracy? Why not to just show accuracy?\n\n**Accuracy** is the ratio of correct prediction to the total number of predictions.\n\nIn machine learning the **loss** function or cost function is representing the price paid for inaccuracy of predictions.\n\nThe loss associated with one example in binary classification is given by:\n`-(y * log(p) + (1-y) * log (1-p))`\nwhere `y` is the true label of `x` and `p` is the probability predicted by our model that the label is 1."},{"metadata":{"trusted":true,"_uuid":"d19fa5f25f5351c7aec80a4fc53f82b71104e92a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"colors":{"hover_highlight":"#DAA520","navigate_num":"#000000","navigate_text":"#333333","running_highlight":"#FF0000","selected_highlight":"#FFD700","sidebar_border":"#EEEEEE","wrapper_background":"#FFFFFF"},"moveMenuLeft":true,"nav_menu":{"height":"266px","width":"252px"},"navigate_menu":true,"number_sections":true,"sideBar":true,"threshold":4,"toc_cell":false,"toc_section_display":"block","toc_window_display":false,"widenNotebook":false}},"nbformat":4,"nbformat_minor":1}