{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n1. [Data Extraction & Label Generation](#section_1)\n2. [{Training Set + Validation Set} Creation](#section_2)\n3. [Mixed Precision Training](#section_3)\n4. [Validation Set Results](#section_4)\n5. [Improving the Classifier](#section_5)\n    1. [Data Cleaning](#section_5_1)\n    2. [Progressive Image Re-sizing](#section_5_2)\n    3. [Unfreezing & Discriminative Layer Training](#section_5_3)\n6. [Test Set Predictions & Submission](#section_6)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---\n\nThis notebook attempts to replicate a few tricks mentioned by <a href=\"https://www.kaggle.com/jhoward\" target=\"_blank\">Jeremy Howard</a> in <a href=\"https://course.fast.ai/\" target=\"_blank\">Practical Deep Learning for Coders</a> to improve the performance of an image classifier.\n\nModule imports and other preliminaries:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nimport zipfile\nfrom fastai.vision import *\nfrom fastai.metrics import error_rate\nfrom fastai.widgets import *\nimport pandas as pd\nimport base64\nfrom IPython.display import HTML\nimport re\n\nwarnings.filterwarnings('ignore') # Suppress warning messages.\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n<a id=\"section_1\"></a>\n# 1. Data Extraction & Label Generation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the competition data files.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.chdir('/kaggle/input/dogs-vs-cats-redux-kernels-edition')\n\nos.listdir()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's extract the files in `'train.zip'` and `'test.zip'` to the `'/kaggle/working/'` directory.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with zipfile.ZipFile('/kaggle/input/dogs-vs-cats-redux-kernels-edition/train.zip', 'r') as zip_ref:\n    zip_ref.extractall('/kaggle/working/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with zipfile.ZipFile('/kaggle/input/dogs-vs-cats-redux-kernels-edition/test.zip', 'r') as zip_ref:\n    zip_ref.extractall('/kaggle/working/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's change our working directory to `'/kaggle/working'` and take a look at its contents.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.chdir('/kaggle/working/')\n\nos.listdir()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The folders `'train'` and `'test'` contain the images.\n\nLet's get the filenames in `'train'`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_fnames = get_image_files('/kaggle/working/train')\n\nlen(train_fnames)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are `25000` files. Let's examine the first `5` filenames.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_fnames[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's use a list comprehension to generate the labels.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = [('cat' if 'cat.' in str(fname) else 'dog') for fname in train_fnames]\n\nlabels[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n<a id=\"section_2\"></a>\n# 2. {Training Set + Validation Set} Creation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's create an `ImageDataBunch` object (containing both the training set and the validation set).","execution_count":null},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"np.random.seed(123) # Ensure reproducibility.\ndata = ImageDataBunch.from_lists(\n    path='/kaggle/working/train', \n    fnames=train_fnames, \n    labels=labels, \n    valid_pct=0.2, # Put 20% of the images in the validation set.\n    ds_tfms=get_transforms(flip_vert=False), # Perform data augmentation.\n    size=224, # Resize all images to the same size (224px by 224px).\n    bs=32 # Set the batch size for training.\n).normalize(imagenet_stats) # Normalize all images with ImageNet statistics.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(data.train_ds), len(data.valid_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training set contains `20000` images and the validation set contains `5000` images.\n\nClasses:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data.classes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A random sample of observations:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3, figsize=(12, 12))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n<a id=\"section_3\"></a>\n# 3. Mixed Precision Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"A couple of checks:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.is_available()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.backends.cudnn.enabled","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create our learner by specifying:\n\n- `data` {Training Set + Validation Set}\n- `models.resnet50` (model with ResNet50 architecture pre-trained on ImageNet images)\n- `error_rate` (metric to show during training)\n- `to_fp16()` (mixed precision training)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make sure Internet is on.\nlearner = cnn_learner(data, models.resnet50, metrics=error_rate).to_fp16()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's run `fastai`'s learning rate finder.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.lr_find(start_lr=1e-6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After eyeballing the graph, let's choose a *maximum learning rate*.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"max_lr_choice = 5e-4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's train our model for `4` epochs.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.fit_one_cycle(4, max_lr=max_lr_choice)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's a pretty good fit.\n\nPlot of training & validation losses:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's save our model's weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.save('imgsize224-stage1')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n<a id=\"section_4\"></a>\n# 4. Validation Set Results","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's create a `ClassificationInterpretation` object.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learner)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot the confusion matrix.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_confusion_matrix()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Accuracy:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy = (interp.confusion_matrix()[0, 0] + interp.confusion_matrix()[1, 1]) / len(data.valid_ds)\n\naccuracy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's use the `plot_top_losses()` method to examine images which have the biggest losses along with:\n\n- predicted class\n- actual class\n- loss\n- probability assigned by model to actual class","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_top_losses(20, figsize=(16, 16))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, some of the misclassified images are noisy / irrelevant.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---\n\n<a id=\"section_5\"></a>\n# 5. Improving the Classifier","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"section_5_1\"></a>\n## 5.1. Data Cleaning","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now, we'll use `fastai`'s `ImageCleaner` Jupyter widget to re-label / delete images which are mislabeled / noisy / irrelevant.\n\nFirst, let's create a new `ImageDataBunch` without a training, validation split.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_no_split = ImageDataBunch.from_lists(\n    path='/kaggle/working/train', \n    fnames=train_fnames, \n    labels=labels, \n    valid_pct=0, # Don't put any images in the validation set.\n    ds_tfms=get_transforms(flip_vert=False),\n    size=224,\n    bs=32\n).normalize(imagenet_stats)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's create a new learner with `data_no_split` and load the `'imgsize224-stage1'` weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\nlearner_no_split = cnn_learner(data_no_split, models.resnet50).to_fp16()\nlearner_no_split.load('imgsize224-stage1')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's pass in the new learner to `DatasetFormatter.from_toplosses()`. It will return a *formatted* dataset and file indices in descending order of top losses.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset, file_indices = DatasetFormatter.from_toplosses(learner_no_split)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we can use ImageCleaner to:\n\n1. **Re-label:** Mis-labeled images (`'cat'` as `'dog'` or `'dog'` as `'cat'`).\n2. **Delete:**\n    - Images containing both a cat and a dog (since this is not a multi-label classification problem).\n    - Images where it isn't clear whether the animal is a cat or a dog (e.g., due to the animal's posture / image blur).\n    - Clip art / cartoons.\n    - Irrelevant images (e.g., house pictures, landscapes, company logos, etc.)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use in interactive mode only. When committing notebook, cleaning isn't possible.\nImageCleaner(dataset, file_indices, Path('/kaggle/working/train'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A new CSV file `'cleaned.csv'` has been created in the `'/kaggle/working/train'` folder. Let's read it in and take a look.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use in interactive mode only.\ncleaned = pd.read_csv('/kaggle/working/train/cleaned.csv')\n\ncleaned.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note:** Since it isn't possible to use the `ImageCleaner` Jupyter widget when committing the notebook, we need to save the `cleaned` data frame to a persistent storage location (e.g., Google Cloud Storage / AWS S3 / Dropbox). We'll then continue our workflow with this persistent file (instead of the interactive one above).\n\nLet's create a function to download the data frame.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel\ndef create_download_link(df, title=\"Download CSV file\", filename=\"dogs_vs_cats_cleaned.csv\"):\n    csv = df.to_csv(index=False)\n    b64 = base64.b64encode(csv.encode())\n    payload = b64.decode()\n    html = '<a download=\"{filename}\" href=\"data:text/csv;base64,{payload}\" target=\"_blank\">{title}</a>'\n    html = html.format(payload=payload, title=title, filename=filename)\n    return HTML(html)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now download the data frame.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use in interactive mode only.\ncreate_download_link(cleaned)\n\n# See 'Download CSV file' link below.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's download the file, and upload it to a bucket in Google Cloud Storage (or AWS S3 / Dropbox / other similar service).\n\nAfter the file has been uploaded, we can read it in.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make sure Internet is on.\ncleaned = pd.read_csv('https://storage.googleapis.com/cleaned-data/dogs_vs_cats_cleaned.csv')\n\ncleaned.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's recreate our `ImageDataBunch` using the `cleaned` data frame.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(123)\ndata_cleaned = ImageDataBunch.from_df(\n    path='/kaggle/working/train', \n    df=cleaned, \n    valid_pct=0.2, # Put 20% of the images in the validation set.\n    ds_tfms=get_transforms(flip_vert=False),\n    size=224,\n    bs=32\n).normalize(imagenet_stats)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create a new learner with `data_cleaned`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner = cnn_learner(data_cleaned, models.resnet50, metrics=error_rate).to_fp16()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's run `fastai`'s learning rate finder.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.lr_find(start_lr=1e-6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After eyeballing the graph, let's choose a maximum learning rate.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"max_lr_choice = 5e-4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's train our model for `4` epochs.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.fit_one_cycle(4, max_lr=max_lr_choice)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's save our model's weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.save('imgsize224-stage2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"section_5_2\"></a>\n## 5.2. Progressive Image Re-sizing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now we have a model (`'imgsize224-stage2'`) that is pretty good at classifying dogs vs. cats.\n\n**Trick to create an even better model:** \n\n1. Re-size all images to 300px by 300px and create a new `ImageDataBunch`.\n2. Perform transfer learning on this new `ImageDataBunch` using `'imgsize224-stage2'` as our pre-trained model.\n\nBy using a larger image size, we'll lose most of the overfitting of the previous model, but transfer its 'learning' to the new model.\n\nLet's create the new `ImageDataBunch`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(123)\ndata = ImageDataBunch.from_df(\n    path='/kaggle/working/train', \n    df=cleaned, \n    valid_pct=0.2,\n    ds_tfms=get_transforms(flip_vert=False),\n    size=300, # Re-size all images to 300px by 300px.\n    bs=32\n).normalize(imagenet_stats)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create a new learner with the new `ImageDataBunch` and load the previous model's weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\nlearner = cnn_learner(data, models.resnet50, metrics=error_rate).to_fp16()\nlearner.load('imgsize224-stage2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's run `fastai`'s learning rate finder.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.lr_find(start_lr=1e-6, end_lr=1e-2, stop_div=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After eyeballing the graph, let's choose a maximum learning rate.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"max_lr_choice = 5e-5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's train our model for `4` epochs.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.fit_one_cycle(4, max_lr=max_lr_choice)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's save our model's weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.save('imgsize300-stage1')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"section_5_3\"></a>\n## 5.3. Unfreezing & Discriminative Layer Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's unfreeze our model and run `fastai`'s learning rate finder.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.unfreeze()\nlearner.lr_find(start_lr=1e-6, end_lr=1e-2, stop_div=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We shall use `slice(1e-5, 1e-4)` as our sequence of learning rates.\n\nLet's perform *discriminative layer training* for `4` epochs. (This will apply a different learning rate to each *layer group*.)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.fit_one_cycle(4, max_lr=slice(1e-5, 1e-4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's save our model's weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.save('imgsize300-stage2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's export our model as a pickle.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner = learner.to_fp32() # Convert back to default precision for safe export.\nlearner.export()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n<a id=\"section_6\"></a>\n# 6. Test Set Predictions & Submission","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's get the filenames in the `'test'` folder.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_fnames = get_image_files('/kaggle/working/test')\n\nlen(test_fnames)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are `12500` files. Let's examine the first `5` filenames.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_fnames[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's use a list comprehension to extract the ids.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = [int(re.findall(r'\\d+', str(fname))[0]) for fname in test_fnames]\n\nids[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's load our learner from the exported pickle (specifying the test set this time).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.path # Location of pickle.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner = load_learner(path=learner.path, test=test_fnames)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's obtain our test set predictions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, labels = learner.get_preds(ds_type=DatasetType.Test)\n\npreds[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first column is the probability of `'cat'` and the second column is the probability of `'dog'`.\n\nLet's create our submission data frame and sort its rows by `id`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"d = {'id': ids, 'label': preds[:, 1]}\nsubmission = pd.DataFrame(data=d)\nsubmission = submission.sort_values(by='id')\n\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's write it to disk.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}