{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Model Ensembling and Transfer Learning with the Kannada MNIST datasets (and fastai)"},{"metadata":{},"cell_type":"markdown","source":"## Brief background: Kannada MNIST\nThis Playground Code Competition is to detect Kannada digits. Kannada is a language spoken predominantly by people of Karnataka in southwestern India. The digits range from omdu (1) to hattu (10)."},{"metadata":{},"cell_type":"markdown","source":"## Playground Competiton: Let's play!\nThe goal of this notebook is to implement two techniques: model ensembling and transfer learning. \n\nIn [model ensembling](https://towardsdatascience.com/simple-guide-for-ensemble-learning-methods-d87cc68705a2), the decisions of many models is combined to make a final prediction. All of the kernels I have submitted so far for this competition have been a single model that predicts one of 10 categories for the digits. In this kernel, instead of creating one model for 10 digits, I created ten models, each predicting a single digit. For example, the first model would predict whether the digit was of category \"0\" or \"not 0,\" the second model would predict category \"1\" or \"not one,\" etc. \n\nHowever, I don't want to fully train 10 models. This is what brings me to the second part of my goal: [transfer learning](https://machinelearningmastery.com/how-to-improve-performance-with-transfer-learning-for-deep-learning-neural-networks/). In transfer learning, a model that has learned in one setting is used to improve generalization in another setting. For example, a model might be trained on the images in [ImageNet](http://www.image-net.org/), and then used to perform another task, like identifying the subset of [dog breeds](https://www.kaggle.com/c/dog-breed-identification). Here, I want to create a 10 Kannada digit model, and transfer the trained convolutional portion to my single digit models.\n\nThis kernel follows the following steps:\n1. Define commmonly used methods for reading and creating data structures\n1. Train the model I will use for transfer learning (aka, the 'backbone' model)\n1. Create a new model for single digit detection that incorporates the pretrained 'backbone' model\n1. Train the single digit models and retain their predictions\n1. Ensemble the results of all models to obtain the submission\n\nI've used fastai in this kernel, but my hope is that you can take the ideas presented here, and apply it to whatever framework you like best."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nimport gc; \nDATAPATH = Path('/kaggle/input/Kannada-MNIST/')\n#PATH = Path.cwd()\n#DATAPATH = PATH/'data'\n\nimport os\nfor dirname, _, filenames in os.walk(DATAPATH):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"csv_train = DATAPATH/'train.csv'\ncsv_test = DATAPATH/'test.csv'\ncsv_extra = DATAPATH/'Dig-MNIST.csv'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Commonly used methods"},{"metadata":{},"cell_type":"markdown","source":"The data is provided in a comma separated value file, and I need it to be a Pandas `DataFrame`. When creating the DataFrame, I add extra columns:\n- fn: is a duplicate of the index, which is the 'image number'\n- is_valid: defines a 10% validity set\n- binary: this column will be used in the single digit models"},{"metadata":{"trusted":false},"cell_type":"code","source":"def panda_from_csv(csv):\n    panda = pd.read_csv(csv)\n    panda['fn'] = panda.index\n    panda['is_valid'] = np.random.random(len(panda))\n    panda['is_valid'] = panda['is_valid'] > 0.9\n    panda['binary'] = 1\n    return panda","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this notebook, I'm going to be using the Fastai APIs. This custom ImageList for pixels [was inspired by this very helpful kernel](https://www.kaggle.com/heye0507/fastai-1-0-with-customized-itemlist). To open an image, the function is passed a 'filename' (fn), which is a string containing the index number with '../' in front. This code finds the correct row, and selects the needed colums of that row. The resulting array is reshaped into a 28x28 matrix, and returned as an image. \n\nPreviously, I used [this super interesting kernel](https://www.kaggle.com/joatom/kannada-mnist-speed-up-fastai-image-processing), but I was getting random performance after ensembling. After debugging I'm still stumped as to why, so I reverted back to the slower version."},{"metadata":{"trusted":false},"cell_type":"code","source":"class PixelImageItemList(ImageList):\n    \n    def open(self,fn):\n        img_pixel = self.inner_df.loc[self.inner_df['fn'] == int(fn[2:])].values[0,1:785]\n        img_pixel = img_pixel.reshape(28,28)\n        return vision.Image(pil2tensor(img_pixel,np.float32).div_(255))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`bs` is the batch size, and `tfms` gets the standard fastai transforms, minus the image flipping. I did not flip the images because that would create noise instead of helping the model generalize (think about a '3')."},{"metadata":{"trusted":false},"cell_type":"code","source":"bs = 512\ntfms=get_transforms(do_flip=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In fastai, the data and dataloaders are stored as a `DataBunch`. This codde will get the images, create the training/validation sets and add the labels. Next, I specify whether I want test time augmentation or not. Finally, I create the databunch, and normalize."},{"metadata":{"trusted":false},"cell_type":"code","source":"def get_databunch(df, col_label, tta=True):\n    \n    src = (PixelImageItemList.from_df(df,'.',cols='fn')\n          .split_from_df('is_valid')\n          .label_from_df(cols=col_label))\n    with_tta = (tfms[0],tfms[0])\n    if not tta:\n        with_tta = (tfms[0],[])\n    data = (src.transform(tfms=with_tta)\n           .databunch(num_workers=2,bs=bs)\n           .normalize())\n    return data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Train the 'backbone' model"},{"metadata":{},"cell_type":"markdown","source":"The model I am using is an implementation of the 'best' original MNIST architecture found [here](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist). This great tutorial was created using Keras, and here I've re-implemented it using a combination of Pytorch and Fastai. The Fastai `conv_layer` function returns a sequence of convolutional, ReLU and batchnorm layers. \n\nNote that this model is divided into two sections. The first is the backbone, that contains the convolutional layers, and the second is the 'head', that contains the linear layers. Since I will use the backbone of this model as the pretrained portion of my single digit models, I am calling this the 'backbone model'."},{"metadata":{"trusted":false},"cell_type":"code","source":"best_architecture = nn.Sequential(\n    # backbone\n    nn.Sequential(\n        conv_layer(1,32,stride=1,ks=3),\n        conv_layer(32,32,stride=1,ks=3),\n        conv_layer(32,32,stride=2,ks=5),\n        nn.Dropout(0.4),\n    \n        conv_layer(32,64,stride=1,ks=3),\n        conv_layer(64,64,stride=1,ks=3),\n        conv_layer(64,64,stride=2,ks=5),\n        nn.Dropout(0.4),),\n    \n    # head\n    nn.Sequential(\n        Flatten(),\n        nn.Linear(3136, 128),\n        relu(inplace=True),\n        nn.BatchNorm1d(128),\n        nn.Dropout(0.4),\n        nn.Linear(128,10),\n        nn.Softmax(dim=-1))\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This methods returns a `Learner`. In fastai the `Learner` is a container that holds the data, the model, and other associated details. I have chosen to use weighted cross entropy as my loss function across all models. The second line of this function specifies where the 'backbone' and 'head' of my model are."},{"metadata":{"trusted":false},"cell_type":"code","source":"def get_learner(dbch, arch, weights):\n    learn = Learner(dbch, arch, loss_func = nn.CrossEntropyLoss(weight=class_weights), \n                    metrics=[accuracy])\n    learn.split([learn.model[0], learn.model[1]]);\n    return learn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since I am using weighted cross entropy, I want to pass in the weights of my classes. In the 10 digit datasets, the classes are even, so I have a tensor of 10 1's. This is probably not needed at this point, but the weights become especially important in the single digit models."},{"metadata":{"trusted":false},"cell_type":"code","source":"weights = [1.,1.,1.,1.,1.,1.,1.,1.,1.,1.]\nclass_weights=torch.FloatTensor(weights).cuda()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train the out of domain Kannada digits"},{"metadata":{},"cell_type":"markdown","source":"The first training set is the out of domain Kannada digits. From the [website](https://www.kaggle.com/c/Kannada-MNIST):\n> ...the author also disseminated an additional real world handwritten dataset (with 10k images), termed as the 'Dig-MNIST dataset' that can serve as an out-of-domain test dataset. It was created with the help of volunteers that were non-native users of the language, authored on a smaller sheet and scanned with different scanner settings compared to the main dataset. This 'dig-MNIST' dataset serves as a more difficult test-set.\n\nHere, I'm using it as a noisy, small, initial training set."},{"metadata":{"trusted":false},"cell_type":"code","source":"dbch_extra = get_databunch(panda_from_csv(csv_extra), 'label', False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Show what numbers are in a single batch to make sure everything seems ok. A characteristic of this dataset is noisy vertical bars that appears to be artifacts."},{"metadata":{"trusted":false},"cell_type":"code","source":"dbch_extra.show_batch(rows=3,figsize=(4,4),cmap='bone')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"learn = get_learner(dbch_extra, best_architecture, class_weights)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The backbone model is completely untrained so far. Therefore, I want to train all of the layers from the beginning. Since I told the learner which part of the model was my 'backbone' and which is my 'head,' the default behavior is to freeze (don't train) the backbone. To preven this, I will 'unfreeze' to train the entire model."},{"metadata":{},"cell_type":"markdown","source":"Before training, I want to find an acceptable learning rate. The following graph helps to find one in the right ballpark.\n- commented out to reduce time for competition, 1e-2 was a good learning rate"},{"metadata":{"trusted":false},"cell_type":"code","source":"#learn.lr_find()\n#learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def train_kannada_mnist(u_epochs,lr):\n    learn.unfreeze()\n    learn.fit_one_cycle(u_epochs,lr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a small dataset, so I've made the arbitrary choice to train for 10 epochs. When I ran this on my computer, a learning rate of 1e-2 was acceptable."},{"metadata":{"trusted":false},"cell_type":"code","source":"train_kannada_mnist(10,1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"learn.save('after_extra_kannada')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train on the Kannada MNIST training set"},{"metadata":{},"cell_type":"markdown","source":"Now the model will be pretrained on the Kannada MNIST training set. It is very important that I keep the training dataframe because it specifies my training/validation split. If I created a new training/testing split for my single digit models, some of the digits from the training set could then be in the validation set. I'm not entirely sure this would matter because I am using a different model head, but I'm not going to risk it."},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = panda_from_csv(csv_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"dbch_train = get_databunch(train_df, 'label', False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"dbch_train.show_batch(rows=3,figsize=(4,4),cmap='bone')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"learn = get_learner(dbch_train, best_architecture, class_weights)\nlearn.load('after_extra_kannada');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Find an appropriate learning rate for this pretrained model.\n- commented out due to time constraints in the competition, 1e-2 was fine for the lr"},{"metadata":{"trusted":false},"cell_type":"code","source":"#learn.lr_find()\n#learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This model has quite a different loss curve. However, 1e-2 still seems to be a good choice.\n\nNow the backbone model is already partially trained. Therefore, we will first train the 'head' on the training digits by using `freeze()` to not train the 'backbone' layers. Then, the entire model is 'unfrozen' and fine tuned for a few epochs. The number of frozen and unfrozen epochs is arbitrary chosen. Since I'm using this as my pretrained model, I'm not concerned about training for lots of epochs. 99% accuracy tells me that the model is good at recognizing the digits. "},{"metadata":{"trusted":false},"cell_type":"code","source":"def train_kannada_mnist(f_epochs,u_epochs,lr):\n    learn.freeze()\n    learn.fit_one_cycle(f_epochs,lr)\n    learn.unfreeze()\n    learn.fit_one_cycle(u_epochs,lr/2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_kannada_mnist(1,3,1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"learn.save('after_train_kannada')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Create the single digit model"},{"metadata":{},"cell_type":"markdown","source":"To create a single digit classifier, I create a new 'head' with two output categories. I reduced the amount of dropout from 0.4 to 0.1 because of the reduction in categories. In addition, I removed the number of nodes in the linear layer from 128 to 64."},{"metadata":{"trusted":false},"cell_type":"code","source":"binary_head = nn.Sequential(\n    Flatten(),\n    nn.Linear(3136, 64),\n    relu(inplace=True),\n    nn.BatchNorm1d(64),\n    nn.Dropout(0.1),\n    nn.Linear(64,2),\n    nn.Softmax(dim=-1)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, my new architecture would be the 'backbone' of the first model, combined with the new two category 'head'. "},{"metadata":{"trusted":false},"cell_type":"code","source":"empty_binary = nn.Sequential(best_architecture[0],binary_head)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, the model I just created is empty. My pretrained model contains the previously trained backbone, with the new 'head'. "},{"metadata":{"trusted":false},"cell_type":"code","source":"pretrained_model = nn.Sequential(learn.model[0],binary_head)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before training the single digit classifiers, I want to ensure that I have an appropriate learning rate. Here, I am creating a the labels for the 0th category, and creating a learner with the new pretrained single digit model."},{"metadata":{"trusted":false},"cell_type":"code","source":"is_0 = train_df['label'] == 0\ntrain_df['binary'] = np.multiply(is_0,1)\ndbch_bin = get_databunch(train_df, 'binary', True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we can see that now the labels are either 1 or 0."},{"metadata":{"trusted":false},"cell_type":"code","source":"dbch_bin.show_batch(rows=3,figsize=(4,4),cmap='bone')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here it's important to note that the categories of the digits are mostly (if not all) 0. This makes sense because now instead of having 10 equally sized categories, we have one positive category as 1 and the nine other categories combined to make up the 0 category. This creates an imbalanced data set. To account for the data imbalance, I will adjust the weights of the categories for the cross entropy loss."},{"metadata":{"trusted":false},"cell_type":"code","source":"weights = [1.,9.]\nclass_weights=torch.FloatTensor(weights).cuda()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"binary_model = get_learner(dbch_bin, pretrained_model, class_weights)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Find an acceptable lr for this new model.\n- This is commented out due to time constraints in the competition."},{"metadata":{"trusted":false},"cell_type":"code","source":"#binary_model.lr_find()\n#binary_model.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like 1e-2 will work for these models, too. Save the model so it can be used for single digit recognition later."},{"metadata":{"trusted":false},"cell_type":"code","source":"binary_model.save('pretrained_binary')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'm a bit worried about running out of RAM on the Kaggle kernel, although I'm not sure how founded that fear is. Here's two variables I know I won't need below."},{"metadata":{"trusted":false},"cell_type":"code","source":"del(binary_model)\ndel(dbch_bin)\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Train the single digit models"},{"metadata":{},"cell_type":"markdown","source":"First the test set is created as a PixelImageList. We will add the test set to the learner after it is trained."},{"metadata":{"trusted":false},"cell_type":"code","source":"df_test = panda_from_csv(csv_test)\ndf_test.rename(columns={'id':'label'}, inplace=True)\ntest_set = PixelImageItemList.from_df(df_test,path='.',cols='fn')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The predictions from the models will be ensembled after all 10 models have been trained. I make a numpy array that will hold all of the model predictions. Here, there are 5,000 test images, 10 single digit models, n_tta_aug test time augmentations, and 2 outcomes of each model. For the test time augmentation, augmentation are applied to the test set, allowing the same model to make multiple predictions about a digit."},{"metadata":{"trusted":false},"cell_type":"code","source":"n_tta_aug = 13\npreds = np.zeros([len(df_test),10,n_tta_aug,2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This functions trains all of the single digit models and records the output.The following process is repeated for each digit:\n1. New labels are created from the trainin_df. These labels are stored in the 'binary' column.\n1. A databunch is created with the 'binary' column as the label, and test time augmentation.\n1. The model 'head' is trained\n    - Note that I chose not to unfreeze the model and train the whole thing. I have pretrained my 'backbone' to decent accuracy, and I note that unfreezing causes a decrease in accuracy.\n1. The test set is added, and the test time augmentation predictions are recorded"},{"metadata":{"trusted":false},"cell_type":"code","source":"def train_binary_models(f_epochs, lr):\n\n    for model_num in range(0,10):\n        print('training model for',model_num)\n        # create the new binary labels\n        is_num = train_df['label'] == model_num\n        train_df['binary'] = np.multiply(is_num,1)\n        \n        # get a new data bunch\n        dbch_bin = get_databunch(train_df, 'binary', True)\n        \n        # create a learner and load the empty binary model\n        learn = get_learner(dbch_bin, empty_binary, class_weights)\n        learn.load('pretrained_binary')\n        \n        # train the model\n        learn.freeze()\n        learn.fit_one_cycle(f_epochs,lr)\n        \n        # add the test set\n        learn.data.add_test(test_set)\n        for npred in range(n_tta_aug):\n            new_preds, _ = learn.get_preds(DatasetType.Test)\n            preds[:,model_num,npred,:] = new_preds\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train and test each model for an arbitrary number of epochs."},{"metadata":{"trusted":false},"cell_type":"code","source":"train_binary_models(3, 1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"np.save('preds.npy',preds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Ensemble the results of the model"},{"metadata":{"trusted":false},"cell_type":"code","source":"preds = np.load('preds.npy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"preds = torch.tensor(preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"preds.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With one line, we can get the ensembled predictions. Broken down from the inside out:\n- preds[:,:,:,1] = the probability of True digit\n- torch.mean(preds[:,:,:,1],dim=2 = the mean probability of True across aurmentations\n- torch.argmax(torch.mean(preds[:,:,:,1],dim=2),dim=1) = the digit classifier with highest (argmax) mean True across augmentations "},{"metadata":{"trusted":false},"cell_type":"code","source":"winner = torch.argmax(torch.mean(preds[:,:,:,1],dim=2),dim=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"winner.size(),winner[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another way to find the 'winner' of the models"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Find digit classifier (=argmax) with highest fraction of augmentation runs where the true class shows higher response\n#torch.argmax(torch.mean(torch.argmax(preds,dim=-1).double(),dim=-1),dim=-1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create a dataframe of the submission, and turn it into a csv file."},{"metadata":{"trusted":false},"cell_type":"code","source":"submission = pd.DataFrame({ 'id': np.arange(0,len(winner)),'label': winner })","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"fin."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.5"}},"nbformat":4,"nbformat_minor":1}