{"cells":[{"metadata":{"_uuid":"bff92b0cf1761266783a68c2c0bdf650ffcc7305"},"cell_type":"markdown","source":"# Fast.AI.MNIST\n\nAfter playing with plain `numpy` neutral network implementation in NNNN - a [Naive Neutral Network iNtroduction](https://www.kaggle.com/alukashenkov/nnnn-a-naive-neutral-network-introduction) kernel I intended to try to implement a similar model using a popular deep learning framework. However, I really got inspired by Fast.AI approach demonstrated in Lesson 1 of the [course](https://course.fast.ai). Also, it helps a lot in the learning process to take a concept and to extend it to a similar, but still different case.\n\nSo, here is my take on MNIST dataset with the Fast.AI toolbox.\n\nAs always, if you see a bug, please let me know at alukashenkov@gmail.com. If you find this piece of use and interest, please upvote."},{"metadata":{"_uuid":"af4303cd916337ea5b9e1984d7c99866b7d15558"},"cell_type":"markdown","source":"## Acknowledgments\n\nI took initial inspiration from [MNIST Classification using Fast AI V2](https://www.kaggle.com/vijaykris/mnist-classification-using-fast-ai-v2) kernel. \n\n[Practical Deep Learning for Coders](https://course.fast.ai) course materials were of great help. I guess I'm using inputs from first three lessons in this kernel. There are [incredible notes](https://medium.com/@hiromi_suenaga/deep-learning-2-part-1-lesson-1-602f73869197) for this course created by [Hiromi Suenaga](https://medium.com/@hiromi_suenaga) that make the content of video lectures \"searchable.\"\n\n[Data Augmentation Experimentation](https://towardsdatascience.com/data-augmentation-experimentation-3e274504f04b) article by [Amrit Virdee](https://medium.com/@pillview) helped a lot to understand how to work with data augmentation in Fast.AI."},{"metadata":{"_uuid":"bc9d989e94dc787951fb9dc400592088b6ac807b"},"cell_type":"markdown","source":"## Environment"},{"metadata":{"_uuid":"cdd08788dfc3bae05340abe4bfd2b9c00489d1be"},"cell_type":"markdown","source":"Apparently, it is hard to manage the environment, I ran into this issue both here on Kaggle and on Google Colab. Although it is possible to change packages versions with `!pip`, it would work only for a particular session. However, having these installation steps in the script is not that practical, as it takes precious kernel runtime."},{"metadata":{"_kg_hide-output":false,"trusted":true,"_uuid":"7dcf6afcee248cdde7506fed76c3ed4aaab86d7a","scrolled":true},"cell_type":"code","source":"!pip show fastai","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1da37351e74307cc6ba60c567eef9d62762f62cd"},"cell_type":"markdown","source":"## Getting Data and Preparing Train/CV Sets\nImporting libraries."},{"metadata":{"trusted":true,"_uuid":"0a7d6f1b8e0e6ef64d5410e2e0e574951b62314a"},"cell_type":"code","source":"# Put these at the top of every notebook, to get automatic reloading and inline plotting\n%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b15cf223ebd968342407a01135215c4e5b6a1629"},"cell_type":"code","source":"# This file contains all the main external libs we'll use\nfrom fastai.imports import *\nfrom fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *\n\nimport torch.optim as optim\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nimport numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nplt.rcParams['image.interpolation'] = 'nearest'\nplt.rcParams['image.cmap'] = 'gray'\n\nimport seaborn as sns\nsns.set(style='white', context='notebook', palette='deep')\n\nimport gc\nimport datetime as dt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"adf8c4d26cf8201800771c049f21fed24ffb0798"},"cell_type":"code","source":"# Start timer\nstart = dt.datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68037264bfe209db62bc1c8c72a0e5d1f405cfd2"},"cell_type":"markdown","source":"Definig some useful constants."},{"metadata":{"trusted":true,"_uuid":"7f0df203208915a055eef3d7a0e04738ec39c314"},"cell_type":"code","source":"#set the path\nPATH = \"../input\"\nos.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"1d50bfde13b9e4a68520fc31c7fa22eb08ecdf76"},"cell_type":"code","source":"img_size = 28               # Known from dataset description\nbatch_size = 64             # Opting for a default value for now","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13704c1d62a35bed7086554adcfc0085e9694435"},"cell_type":"markdown","source":"Load train and test data."},{"metadata":{"trusted":true,"_uuid":"59495f589d5dd6afdddcb0a766e49865d34bafb1"},"cell_type":"code","source":"train_data = pd.read_csv(f'{PATH}/train.csv', header='infer')\ntest_data  = pd.read_csv(f'{PATH}/test.csv', header='infer')\n\n# Getting number of training examples from train data shape\nm = train_data.shape[0]\nprint(\"Number of samples in training data is:\", m)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"19e35d8bb20b74f9ccc8afc0c321cc7976120ac6"},"cell_type":"markdown","source":"The above training file contains both images and labels. These have to be split. The first column is a label."},{"metadata":{"trusted":true,"_uuid":"72afec5b0271ddebb0b76575a73898288d67f16d"},"cell_type":"code","source":"#Pixels\nX_train_data = train_data.iloc[:,1:]\nX_train_data = X_train_data.values\nX_test_data = test_data.values       # this one has no labels to be removed\n\n# Labels\nY_train_data = train_data.iloc[:,0:1]\nY_train_data = Y_train_data.values\n\n# Little clean up\ndel train_data\ndel test_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a87be0e2f1d2d2691aff0db51296b17046fce7a"},"cell_type":"markdown","source":"Since the pre-trained model resnet has 3 channels, we will have to multiply the channels by 3 the test and train data."},{"metadata":{"trusted":true,"_uuid":"b830f6d72784533c938c6a9f3a4c7af1556da23d"},"cell_type":"code","source":"# Converting to the square image\nprint(\"Original train image data shape:\", X_train_data.shape)\nX_train_data = X_train_data.reshape(-1, img_size, img_size)\n\n#Add missing color channels to previously reshaped image\nX_train_data = np.stack((X_train_data,) * 3, axis = -1).astype('uint8') \nX_train_data = X_train_data/255                                           # Normalizing data as we go, it seems to make a lot of difference\nprint(\"Resulting train image data shape:\", X_train_data.shape)\n\n# Same conversion for test data as well\nX_test = X_test_data.reshape(-1, img_size, img_size)\nX_test = np.stack((X_test,) * 3, axis = -1).astype('uint8')\nX_test = X_test/255                                                      # Same processing for test as for training","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"620f62482213704eb3e7c693ccc19800c82b6df1"},"cell_type":"code","source":"def print_10_images(X, y):\n    plt.rcParams['figure.figsize'] = (50.0, 50.0) # set default size of plots\n    plt.axis('off')\n    \n    num_images_to_show = 10\n    m = X.shape[0]\n    \n    for i in range(num_images_to_show):\n        sample = np.random.randint(0, m - 1)\n        \n        plt.subplot(2, num_images_to_show, i + 1)\n        plt.imshow(X[sample], interpolation = 'nearest')\n        plt.rcParams['figure.figsize'] = (50.0, 50.0) # set default size of plots\n        \n        plt.title(\"Index: \" + str(sample) + \" \\n Label: \" + str(y[sample]), fontsize=25)\n            \n    plt.axis('on')\n    plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots\n        \ndef plot_labels_distribution(Y_train, Y_cv, sharey = False):\n    _, axes = plt.subplots(1, 2, figsize = (10, 3), sharey = sharey)\n    sns.countplot(Y_train.flatten(), ax = axes[0])\n    sns.countplot(Y_cv.flatten(), ax = axes[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66c1aebcb5b87c6c01ab6902fa906ee21a56c3c2"},"cell_type":"markdown","source":"Let's see what we've got to work with."},{"metadata":{"trusted":true,"_uuid":"750655171f6b337fb2ba1a8a0c4e1f512e063317"},"cell_type":"code","source":"print_10_images(X_train_data, Y_train_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b09bb4dcad25ea6b6f2c3d41b5de7df558961af"},"cell_type":"markdown","source":"We need to split `train_data` into training and cross-validation sets."},{"metadata":{"trusted":true,"_uuid":"ea8a6dd19da3bcfb7c6538a18021fe02debdf42e"},"cell_type":"code","source":"test_size = 0.05      # Proportion of CV set to be extracted from training data\n\nY_train, Y_cv = train_test_split(Y_train_data, test_size = test_size, shuffle = True, stratify = None, random_state = 1974)\nplot_labels_distribution(Y_train, Y_cv)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d490a2330cfba1d80f4df3ad4a6056ed1b71af5e"},"cell_type":"markdown","source":"The distribution of classes looks fine, but it is not perfect. Stratifying should help, so also including image data into split along with the labels."},{"metadata":{"trusted":true,"_uuid":"68aa7c165bbc6c25966a68fdba9e755cf96fd642"},"cell_type":"code","source":"X_train, X_cv, Y_train, Y_cv = train_test_split(X_train_data, Y_train_data, test_size = test_size, shuffle = True, stratify = Y_train_data, random_state = 1974)\nplot_labels_distribution(Y_train, Y_cv)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3eb4d6abe5338dab231d757a0400a8a2922a5768"},"cell_type":"code","source":"# Final touches\nY_train = Y_train.flatten()\nY_cv = Y_cv.flatten()\n\ndel X_train_data\ndel Y_train_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89f20b1a52fedd27a599daa3bab8019c88ac4d6f"},"cell_type":"markdown","source":"## The Model\n\nNow we are coming to the famous \"three lines of code\" model definition. Let's follow the steps outlined in the course to get it working."},{"metadata":{"trusted":true,"_uuid":"2edb55c2f4d3020851bf38f26e74f5014d79ad95","scrolled":false},"cell_type":"code","source":"# Loading data into the Fast.AI structure\narch = resnet34\ndata = ImageClassifierData.from_arrays('tmp/', (X_train, Y_train), (X_cv, Y_cv), classes = np.unique(Y_cv), bs = batch_size, tfms = tfms_from_model(arch, img_size))\nlearn = ConvLearner.pretrained(arch, data, precompute = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40b8878546e14e4b94f2464829c775a13688e7a6","scrolled":false},"cell_type":"code","source":"# Find learning rate\nlearn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"960a246ac6c55e383b4f6cb9542aafd6d3ea6103"},"cell_type":"code","source":"learn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa239c390b359e11834a3705cbe595c1933563f2"},"cell_type":"markdown","source":"Learning rate of 0.01 seems to be a good default choice. It is important to notice that by default only the output layer of the model is affected by fitting."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"e6845c9b5f845a6728027b6e0e15c1008be1cc9f"},"cell_type":"code","source":"learn.fit(lrs = 0.01, n_cycle = 3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2d6ba8c947b6cf78967a06c5c974e70ea265b44"},"cell_type":"markdown","source":"All in all, accuracy is not that impressive. After all, the simplest possible model with one output layer (inputs pixels are directly mapped to softmax activations) would give accuracy at around 90%. However, with only the output layer being fitted by default we are very close to that."},{"metadata":{"_uuid":"ff52e0dcddb26ba6d1a04b72ffd78613967a42bd","trusted":true},"cell_type":"code","source":"# This gives prediction for validation set. Predictions are in log scale\nlog_preds = learn.predict()\n\n# From log probabilities to 0 or 1\nprobs = np.exp(log_preds)  \n\n# From probabilities to classes\npreds = np.argmax(probs, axis=1)  ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2364ed6f7e138f895ad5aaf980e7d7860e49e775"},"cell_type":"markdown","source":"Sanity check - what images the model gets wrong? In my case, most of the pictures I see still easily recognisable."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ae41d0073997800c86774f902a522de2f4fbd209"},"cell_type":"code","source":"def print_10_mislabeled_images(X, y, p):\n    \n    plt.rcParams['figure.figsize'] = (50.0, 10.0) # set default size of plots\n    plt.axis('off')\n    \n    num_images_to_show = 10\n    \n    a = p - y\n    mislabeled_indices = np.asarray(np.where(a != 0))\n\n    for i in range(num_images_to_show):\n        sample = np.random.randint(0, mislabeled_indices.shape[1])\n        index = mislabeled_indices[0][sample]\n        \n        plt.subplot(2, num_images_to_show, i + 1)\n        plt.imshow(X[index], interpolation = 'nearest')\n        plt.title(\"Prediction: \" + str(p[index]) + \" \\n Label: \" + str(y[index]), fontsize=25)\n        \n    plt.axis('on')\n    plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots\n    \ndef print_all_mislabeled_images(X, y, p):\n    \n    plt.rcParams['figure.figsize'] = (50.0, 15.0) # set default size of plots\n    plt.axis('off')\n    \n    num_images_in_row = 10\n    \n    a = p - y\n    mislabeled_indices = np.asarray(np.where(a != 0))\n    num_rows = mislabeled_indices.shape[1] // num_images_in_row + 1\n    \n    for i in range(mislabeled_indices.shape[1]):\n        index = mislabeled_indices[0][i]\n        \n        plt.subplot(num_rows, num_images_in_row, i + 1)\n        plt.imshow(X[index], interpolation = 'nearest')\n        plt.title(\"Prediction: \" + str(p[index]) + \" \\n Label: \" + str(y[index]), fontsize=25)\n        \n    plt.axis('on')\n    plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"14fc198a0186951962fb7a82905d8a9e9db3502f"},"cell_type":"code","source":"print_10_mislabeled_images(X_cv, Y_cv, preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa17b4250e8dab851ab75df4593a37b69b4e0816"},"cell_type":"markdown","source":"## Improving the Model"},{"metadata":{"_uuid":"ae5408cff6af26218810e01c712ca3ee32b42dc7"},"cell_type":"markdown","source":"By default when we create a learner, it sets all but the last layer to *frozen*. That means that it's still only updating the weights in the last layer when we call `fit`."},{"metadata":{"trusted":true,"_uuid":"103eb27b02765132bac16e11ae6cffa9065b22a0"},"cell_type":"code","source":"learn.precompute = False\nlearn.fit(lrs = 1e-2, n_cycle = 3, cycle_len = 1)\n\nlearn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5572f6e3bff2c0d8ffa5ee3db5dd610e60adf17"},"cell_type":"markdown","source":"Now that we have a good final layer trained, we can try fine-tuning the other layers. To tell the learner that we want to unfreeze the remaining layers, just call `unfreeze()`."},{"metadata":{"trusted":true,"_uuid":"df2312a2b7e24b15f79384a05b7b6abec42d2133"},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3de6e5a71b52f08e3677b9b97eaba80b719ae6a8"},"cell_type":"markdown","source":"Generally speaking, the earlier layers (as we've seen) have more general-purpose features. Therefore we would expect them to need less fine-tuning for new datasets. For this reason, we will use different learning rates for different layers. We refer to this as differential learning rates, although there's no standard name for this technique in the literature that we're aware of."},{"metadata":{"trusted":true,"_uuid":"3ab434beccd4d0bb6677bf305a584fcfc545cb06"},"cell_type":"code","source":"lr = 0.01\nlrs = np.array([lr/9,lr/3,lr])                # Using bigger LR for early layers as our images are quite different from the ImageNet\n\nlearn.fit(lrs = lr, n_cycle = 3, cycle_len = 1, cycle_mult = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"29aa967b3209b31476e9b78ff6e18989a0d91c48"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ab385ae38b72553789db4b5e01625e8fd623a6a"},"cell_type":"markdown","source":"Note that's what being plotted above is the learning rate of the final layers. The learning rates of the earlier layers are fixed at the same multiples of the last layer rates as we initially requested.\n\nLoss graph for reference."},{"metadata":{"trusted":true,"_uuid":"8cb8a94715e78cbfffe37c869784d84326d553fe"},"cell_type":"code","source":"learn.sched.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa1d71654ff3c9e9b2f9dd8da5be38080fcbe86a"},"cell_type":"markdown","source":"Let's see what we still get wrong? Actually, there are only a bunch of weird images in the validation set that the model couldn't classify correctly."},{"metadata":{"trusted":true,"_uuid":"0ccf67895fde961c9b7e2e35950e6be77b029082","scrolled":false},"cell_type":"code","source":"# This gives prediction for validation set. Predictions are in log scale\nlog_preds = learn.predict()\n\n# From log probabilities to 0 or 1\nprobs = np.exp(log_preds)  \n\n# From probabilities to classes\npreds = np.argmax(probs, axis=1)  \n\nprint_all_mislabeled_images(X_cv, Y_cv, preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ddc37bff91b9ce6add1e36f486e7b41efbd73e1"},"cell_type":"markdown","source":"Also, looking at `trn_loss` and `val_loss` it is clear that the model starts to overfit. So, I would assume that further training wouldn't yield much better results - we need to implement data augmentation."},{"metadata":{"_uuid":"bba028474feab3df9d8382b74fdb2c9ca0aa4202"},"cell_type":"markdown","source":"## Data Augmentation"},{"metadata":{"_uuid":"7d1a2c5b692bcfbeabde526402e78f53505bd9fa"},"cell_type":"markdown","source":"Let's first look at the confusion matrix."},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"b95e121d455186cf3db1b2047595f38dc4288b73"},"cell_type":"code","source":"cm = confusion_matrix(Y_cv, preds)\n\nplt.rcParams['figure.figsize'] = (8.0, 8.0) # set default size of plots\nplot_confusion_matrix(cm, data.classes)\nplt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ffb693af9d0974c0ce83c2d775e20806084f4d78"},"cell_type":"markdown","source":"To implement data augmentation, we need to create new learner wish updated transformations structure `tfms`. As we are working with digits, we should be careful selecting augmentations as compared to simple images classification (e.g., we can't flip images). As we work with relatively small images, all augmentation parameters should be fairly subtle."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e856301d66dd4a89f6ab02ff545dfef33bbc741e"},"cell_type":"code","source":"def show_img(ims, idx, figsize = (50, 50), ax = None):\n    if ax is None: fig, ax = plt.subplots(figsize = figsize)\n    ims = np.rollaxis(to_np(ims), 1, 4)\n    ax.imshow(np.clip(ims, 0, 1)[idx], interpolation = 'nearest')\n    ax.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca79c21aef3a29afddc244a41686614a78f8560c"},"cell_type":"code","source":"# Loading data into the Fast.AI structure\narch = resnet34\n\naug_tfms = [RandomRotate(30, p = 0.75, mode = cv2.BORDER_REFLECT, tfm_y = TfmType.NO),\n            AddPadding(pad = 2, mode = cv2.BORDER_REPLICATE)]\n\ntfms = tfms_from_model(arch, img_size, aug_tfms = aug_tfms, max_zoom = 1.1)\ndata = ImageClassifierData.from_arrays('tmp/', (X_train, Y_train), (X_cv, Y_cv), classes = np.unique(Y_cv), bs = batch_size, tfms = tfms)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5bc94823d75669902c704bd2901a501b25f572e"},"cell_type":"markdown","source":"Let's look at samples of augmented images."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"cdc20aa786a2a02f7ea96d8e56f71863dc6699af"},"cell_type":"code","source":"batches = [next(iter(data.aug_dl)) for i in range(10)]\n\nfor pos in range(len(batches)):\n    fig, axes = plt.subplots(1, 10, figsize=(25,8))\n    for i,(x,y) in enumerate(batches):\n        show_img(x, pos, ax = axes.flat[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e49de68660c69e01ea34edd3501c4127666b8420"},"cell_type":"code","source":"learn = ConvLearner.pretrained(arch, data, precompute = True)\n\n# Find learning rate\nlearn.lr_find()\n\nlearn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9bcf9aa0fc09301c1a19d37d385d6eb8044c39c0"},"cell_type":"code","source":"lr = 0.02                                \nlearn.fit(lrs = lr, n_cycle = 3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b319020a718074c39cf4de7b0a4581ae22093c8a"},"cell_type":"code","source":"learn.precompute = False\nlearn.fit(lrs = lr, n_cycle = 3, cycle_len = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d11c24cb21d9e3fd4fe672c4d399bdb7f75589a2"},"cell_type":"code","source":"learn.unfreeze()\n\nlearn.lr_find()\nlearn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61cbcb754406255c749dbfa1e94cdc7dbbfc486f"},"cell_type":"code","source":"lr = 0.003\nlrs = np.array([lr/9, lr/3, lr])\nlearn.fit(lrs = lrs, n_cycle = 3, cycle_len = 1, cycle_mult = 2)\nlearn.sched.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"62f715f7ee76dabb32ac046073c26380cac9567f"},"cell_type":"markdown","source":"It is clear that the model still can benefit from the time invested in further training. But first, let's check out TTA."},{"metadata":{"trusted":true,"_uuid":"48ba8618f06b66af873eecc3c2dcd74fe994be71"},"cell_type":"code","source":"# This gives prediction for validation set. Predictions are in log scale\nlog_preds, y = learn.TTA()\n\n# From log probabilities to 0 or 1\nprobs = np.exp(log_preds) \n\n# From probabilities to classes\npreds = np.argmax(np.mean(probs, axis = 0), axis = 1)\n\nprint_all_mislabeled_images(X_cv, Y_cv, preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4e0252a17b5b03b3fa7055e70f36b0d049c0a96"},"cell_type":"code","source":"cm = confusion_matrix(y, preds)\n\nplt.rcParams['figure.figsize'] = (8.0, 8.0) # set default size of plots\nplot_confusion_matrix(cm, data.classes)\nplt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"881d13cffff20678ca274b46bf9c4b1dd8db3354"},"cell_type":"markdown","source":"## Putting everything together.\n\nI now want to start from scratch and put together all I have learned. The goal is to get to accuracy > 0.999 or better."},{"metadata":{"trusted":true,"_uuid":"d45ef21bfa5de1899d37bb5bd731c43bd21ed799"},"cell_type":"code","source":"arch = resnet152\nbatch_size = 64\n\ndef get_data(sz, bs, trn = (X_train, Y_train), val = (X_cv, Y_cv), test = X_test):\n    aug_tfms = [RandomRotate(30, p = 0.75, mode = cv2.BORDER_REFLECT, tfm_y = TfmType.NO),\n                RandomZoom(zoom_max = 0.05),\n                #RandomStretch(max_stretch = 0.05),\n                AddPadding(pad = 2, mode = cv2.BORDER_REPLICATE)]\n    tfms = tfms_from_model(arch, sz, aug_tfms = aug_tfms, max_zoom = 1.1)\n    data = ImageClassifierData.from_arrays('tmp/', trn, val, classes = np.unique(Y_cv), bs = bs, tfms = tfms, test = test)\n    return data\n\ndata = get_data(img_size, batch_size, (X_train, Y_train), (X_cv, Y_cv), X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1bd71ecaa4781dc2a892a67af3f2fc8ba5aad161"},"cell_type":"code","source":"# Using Adam optimizer\nlearn = ConvLearner.pretrained(arch, data, opt_fn = optim.Adam, precompute = True)\n\nlearn.lr_find()\nlearn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a82581d66e524035b2b489dc673d4da47e81622c"},"cell_type":"code","source":"lr = 0.001\n\nlearn.fit(lr, n_cycle = 1, cycle_len = 1)\n\nlearn.precompute = False\nlearn.fit(lr, n_cycle = 4, cycle_len = 1, cycle_mult = 2)\nlearn.save('pre_fit')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24feae2ebd2e024e9d9a019d25c86fae2a51ab2d"},"cell_type":"markdown","source":"I observed no material improvements in model accuracy after 4 cycles (15 epochs). So, let's now unfreeze and fit model with all layers, but first, let's see if any changes to the learning rate are needed."},{"metadata":{"trusted":true,"_uuid":"32f95155a0a081c0b5572c52c20e60bb2b093824"},"cell_type":"code","source":"learn.load('pre_fit')\nlearn.unfreeze()\n\nlearn.lr_find()\nlearn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"212a0c15ecb614d23be71ea1b55e409920c91765"},"cell_type":"markdown","source":"I'm also using weight decay parameter (as described in https://www.fast.ai/2018/07/02/adam-weight-decay/)"},{"metadata":{"trusted":true,"_uuid":"72942c6db6d0304f0250920c4f7c6e460db9fabc"},"cell_type":"code","source":"lr = 0.0003\nwd = 0.0001\n\nlearn.fit(lrs = [lr/9, lr/3, lr], n_cycle = 5, wds = [wd/100, wd/10, wd], use_wd_sched = True, cycle_len = 1, cycle_mult = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fd673c8938d6b49fcd5928613f8d78735fc4898"},"cell_type":"code","source":"learn.sched.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f7493355a72f2d5c6438824837feb62cb414fd5"},"cell_type":"code","source":"# This gives prediction for validation set. Predictions are in log scale\nlog_preds, y = learn.TTA()\n\n# From log probabilities to 0 or 1\nprobs = np.exp(log_preds) \n\n# From probabilities to classes\npreds = np.argmax(np.mean(probs, axis = 0), axis = 1)\n\nprint_all_mislabeled_images(X_cv, Y_cv, preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f25100c58b35409f2c3c2c98204d107ba0f20c4"},"cell_type":"code","source":"cm = confusion_matrix(Y_cv, preds)\n\nplt.rcParams['figure.figsize'] = (8.0, 8.0) # set default size of plots\nplot_confusion_matrix(cm, data.classes)\nplt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c49a65b2e1cfff25b8f0871808e33b1852d9c582"},"cell_type":"markdown","source":"## Testing and Submitting"},{"metadata":{"trusted":true,"_uuid":"cc26d819754f62f36691baef361e3c211c6e93a3"},"cell_type":"code","source":"# This gives prediction for validation set. Predictions are in log scale\nlog_preds, y = learn.TTA(is_test = True)\n\n# From log probabilities to 0 or 1\nprobs = np.exp(log_preds) \n\n# From probabilities to classes\npreds = np.argmax(np.mean(probs, axis = 0), axis = 1)\n\n# Saving predictions\nresults = pd.Series(preds, name = \"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001), name = \"ImageId\"), results], axis = 1)\nsubmission.to_csv(\"submission.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d35cccbbac6b2a086103e83549448fb264605603"},"cell_type":"code","source":"# Remove temp files as it seems to help to avoid error on the kernel submission\n!rm -rf '../working/tmp'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2e5b003a6d97064e3770e2252dbf293dac263545"},"cell_type":"code","source":"# Collect and report execution timings\nend = dt.datetime.now()\nprint('Latest run {}.\\nTotal time {}s'.format(end, (end - start).seconds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6dc87539eb55a8439cad8bcd3c7d1881321f55bf"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}