{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Mixup training data augmentation, Test time Augmentation"},{"metadata":{},"cell_type":"markdown","source":"## Brief Background"},{"metadata":{},"cell_type":"markdown","source":"The goal of this challenge is to to decet Kannada digits. Kannada is a language spoken predominantly by people of Karnataka in southwestern India. We are given a .csv file containing pixel data and labels. After transforming the data, we train a network to label the numbers from omdu (1) to hattu (10)."},{"metadata":{},"cell_type":"markdown","source":"## Let's Mix it up!"},{"metadata":{},"cell_type":"markdown","source":"This notebook implements two types of data augmentation: Mix Up Augmentation for training and Test Time Augmentation (TTA) for testing. \n\nWith [mixup](https://arxiv.org/pdf/1710.09412.pdf) augmentation, two images are combined during testing. For example, imagine a toy example with the digits \"1\" and \"2\", Instead of training the model with an image containing the digit \"1,\" we give the model an image containing a linear combination of digits \"1\" and \"2\". As you might expect the outcome variable needs to be changed, too. In contrast to a target of [1,0] for the digit \"1\", we would instead have a target of [.3,.7] for an image that is 30% digit \"1\" and 70% digit \"2\". The model is given a harder tasks, and is forced to generalize. Sounds tricky, but fastai lets us implement this augmentation easily.\n\nIn [Test Time Augmentation](https://machinelearningmastery.com/how-to-use-test-time-augmentation-to-improve-model-performance-for-image-classification/), we use the same data augmenatation commonly used for training the model (like rotating, zooming, etc.) when obtaining the test set predictions. This gives us multiple potential digits per image, and we take the most commonly predicted digit as the outcome.\n\nThis kernel proceeds as follows:\n\n1. Loading the data into a DataBunch\n1. Train the model using Mixup Augmentation\n1. Get model predictions using Test time Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nfrom fastai.callbacks import *\n\nDATAPATH = Path('/kaggle/input/Kannada-MNIST/')\n\nimport os\nfor dirname, _, filenames in os.walk(DATAPATH):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(DATAPATH/'train.csv')\ndf_train['fn'] = df_train.index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Making a DataBunch"},{"metadata":{},"cell_type":"markdown","source":"In this notebook, I'm going to be using the Fastai APIs. Fastai uses the `DataBunch` as its container for the training and validation data. For this project we need to create a custom `ImageList` to handle our data because it is stored in a csv file. This custom ImageList for pixels was inspired by this very [helpful kernel](https://www.kaggle.com/heye0507/fastai-1-0-with-customized-itemlist). To open an image, the function is passed a 'filename' (fn), which is a string containing the index number with '../' in front. This code finds the correct row, and selects the needed colums of that row. The resulting array is reshaped into a 28x28 matrix, and returned as an image."},{"metadata":{"trusted":true},"cell_type":"code","source":"class PixelImageItemList(ImageList):\n    def open(self,fn):\n        img_pixel = self.inner_df.loc[self.inner_df['fn'] == int(fn[2:])].values[0,1:785]\n        img_pixel = img_pixel.reshape(28,28)\n        return vision.Image(pil2tensor(img_pixel,np.float32).div_(255))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My data source is created from the `PixelImageList`. In this code, I randomly obtain a validation set, and label my training data from the `DataFrame`'"},{"metadata":{"trusted":true},"cell_type":"code","source":"src = (PixelImageItemList.from_df(df_train,'.',cols='fn')\n      .split_by_rand_pct()\n      .label_from_df(cols='label'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since I want to augment only my testing data, I first get the regular fastai transforms. Fastai transformations are a tuple with two values: the trainig transforms and the testing transforms. Copying the 'training' transformations to the 'test' transformations to allow for test time augmentation. "},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms=get_transforms(do_flip=False)\nwith_tta = ([],tfms[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `DataBunch` holds all of my training and validation data, along with the transformations I want to use. "},{"metadata":{"trusted":true},"cell_type":"code","source":"bs = 1024\ndata = (src.transform(tfms=with_tta)\n       .databunch(num_workers=2,bs=bs)\n       .normalize())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a peek at our data and make sure the digits look as we expect."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3,figsize=(4,4),cmap='bone')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training a model "},{"metadata":{},"cell_type":"markdown","source":"Before we can train a model, we need to define which ones we are using. I used a custom model, instead of a Resnet or Densenet, because of the size of the images. Typically, pretrained models are designed for images of size 224x224. Here, our images are 28x28. Larger models have more of a chance of overfitting. Below is an implementation of the 'best' original MNIST architecture found [here](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist). The tutorial was created using Keras, and here I've re-implemented it using a combination of Pytorch and Fastai. The Fastai `conv_layer` function returns a sequence of convolutional, ReLU and batchnorm layers. "},{"metadata":{"trusted":true},"cell_type":"code","source":"best_architecture = nn.Sequential(\n    conv_layer(1,32,stride=1,ks=3),\n    conv_layer(32,32,stride=1,ks=3),\n    conv_layer(32,32,stride=2,ks=5),\n    nn.Dropout(0.4),\n    \n    conv_layer(32,64,stride=1,ks=3),\n    conv_layer(64,64,stride=1,ks=3),\n    conv_layer(64,64,stride=2,ks=5),\n    nn.Dropout(0.4),\n    \n    Flatten(),\n    nn.Linear(3136, 128),\n    relu(inplace=True),\n    nn.BatchNorm1d(128),\n    nn.Dropout(0.4),\n    nn.Linear(128,10),\n    nn.Softmax(-1)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This 'learner' in Fastai holds the data, model, loss function, and metric of interest. It's at this point that we specify the mixup augmentation by adding `.mixup` when defining the learner."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(data, best_architecture, loss_func = nn.CrossEntropyLoss(), metrics=[accuracy]).mixup()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Implement the callbacks that save the best model, and implement early stopping. Honestly, I haven't played around with these very much, so I can't say if the training process is that much more efficient by using these. Any links to good articles with a way to systematically evaluate this would be appreciated."},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\nSaveModelCallback(learn, monitor='valid_loss', mode='min',name='bestweights'),\nShowGraph(learn),\nEarlyStoppingCallback(learn, min_delta=1e-5, patience=3),\n]\nlearn.callbacks = callbacks","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize the loss at different learning rates.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model is trained using a [one cycle policy](https://docs.fast.ai/callbacks.one_cycle.html), which is easily implemented in Fastai. I set this model to train for an arbitrary 50 epochs, but may be stopped earlier."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(50,1e-2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Time Augmentation"},{"metadata":{},"cell_type":"markdown","source":"Now, we can get the predictions for the test set. But, since we are doing test time augmentation, we'll run the digits through an arbitrary 15 times, and take the 'winner' of all the runs.  \n\nLoad the data like the training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv(DATAPATH/'test.csv')\ndf_test.rename(columns={'id':'label'}, inplace=True)\ndf_test['fn'] = df_test.index\ntest_set = PixelImageItemList.from_df(df_test,path='.',cols='fn')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add the test set to the DataBunch."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.data.add_test(test_set)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get the predictions for the first run of the test set. After we get the predictions, 'unsqueeze' the tensor to give it an extra dimension. The extra dimension is needed because we will stack the predictions on top of one another."},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, _ = learn.get_preds(DatasetType.Test)\npreds.unsqueeze_(1);preds.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get 14 more predictions, for a total of 15 (potentially) different predictions on the test set. Unsqueeze, and stack on top of each other."},{"metadata":{"trusted":true},"cell_type":"code","source":"num_preds = 14\nfor x in range(num_preds):\n    new_preds, _ = learn.get_preds(DatasetType.Test)\n    new_preds.unsqueeze_(1);preds.shape\n    preds = torch.cat((preds,new_preds),1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, the resulting predictions are DIGITS X NUMBER_PREDS X ACTUAL_PREDS. In other words, we have 5,000 digits in the test set, 15 different predictions (each using test time augmentation) from the model, and 10 outputs from the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"preds.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get the most likely prediction from each of the 15 prediction runs. Now we have 500 digits, along with the 15 specific predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"indv_preds = torch.argmax(preds, dim=2);indv_preds.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we get the mode (most frequent value) of the predictions. This gives us our overall digit predictions. "},{"metadata":{"trusted":true},"cell_type":"code","source":"winner = torch.mode(indv_preds, dim=1).values;winner.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, all that's left is to write the results to a .csv file to submit."},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({ 'id': np.arange(0,len(winner)),'label': winner })","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(path_or_buf =\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"markdown","source":"fin."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}