{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this kernel"},{"metadata":{},"cell_type":"markdown","source":"Hello everyone, in this kernel, I will share some information I discovered in the training process of kannadamnist dataset. And some tips to get the stable result with 0.99+ accuracy. <br />\n\n<br />\nThis kernel will cover: <br /> <br />\n**1. The training results before I got 0.99 accuracy and some thoughts I would like to share regarding to previous result** <br />\n**2. Some useful techiques, which I heavily use in other competitions or projects** <br />\n**3. How I improve the kernel to get better result** <br />\n**4. What else might be helpful to get better result than my current result** <br />\n\nAt the begining, I will show you the results from my previous training and what I discovered.<br />\nI already trained 3 models with different split of data in colab and save their valid accuracy for 30 epochs. <br />\nAnd the valid accuracy had been save every 6 epochs. <br />\nI found some interesting things in the result. And I change some places in the kernel upon these information to get my current result. <br />\nSo after showing the results of previous training, I will show the new results after I adjusted the kernel. <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#This csv file saved all the validation result of my previous training.\n\nold_result_df = pd.read_csv('../input/kmnist-data/Old_training_result.csv')\nold_result_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here are two helper function to show the validation results"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def display_single_fold_val_accuracy(final_df, fold):\n    epochs = [6,18,30]\n    color = ['y','g','b']\n    plt.figure(figsize=(12,6))\n    for c, e in zip(color,epochs):\n        for fold in [fold]:\n            df = final_df[(final_df['fold'] == fold) & (final_df['epoch'] == e)]\n            label = 'fold:'+str(fold)+', epoch:' +str(e) \n            plt.plot(df['class'].values, df['valid_accuracy'].values, label = label, c =c)\n    plt.plot(np.arange(0,10,1), np.ones(10), '^', label = 'Top accuracy', color='c')\n    plt.plot(np.arange(0,10,1), np.ones(10)*0.995, '--', label = 'baseline', color='r')\n    plt.xticks(np.arange(0,10,1))\n    plt.title('Valid accuracy on fold'+str(fold)+' model')\n    plt.legend()\n    plt.show()\n\ndef display_top_accuracy_on_final_epoch(final_df):\n    color = ['y','g','b']\n    plt.figure(figsize=(12,6))\n    for c,f in zip(color, [1,2,3]):\n        df = final_df[(final_df['fold'] == f) & (final_df['epoch'] == 30)]\n        label = 'fold:'+str(f)+', epoch:' +str(30) \n        plt.plot(df['class'].values, df['valid_accuracy'].values, label = label, c =c)\n    plt.plot(np.arange(0,10,1), np.ones(10), '^', label = 'Top accuracy', color='c')\n    plt.plot(np.arange(0,10,1), np.ones(10)*0.995, '--', label = 'baseline', color='r')\n    plt.xticks(np.arange(0,10,1))\n    plt.title('Valid accuracy on all models at epoch 30')\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results of my first model\nThe following picture is the validation accuracy of my first model on epoch 6, 18 and 30. <br />\nAlso 0.995 valid accuracy is the baseline of the results. <br />\nFrom the picture, we can discover some information: <br /> <br />\n**1. Model 1 perform worse on class 0 and class 6, both of them have valid accuracy below baseline at epoch 30.** <br />\n**2. Model 1 can reach almost 1.0 valid accuracy on class 5.** <br />\n**3. Some classes got better valid accuracy when epoch is low: class 1, class6, class9** <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_single_fold_val_accuracy(old_result_df, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results of my second model\nThe following picture is the validation accuracy of my second model on epoch 6, 18 and 30. <br />\nAlso 0.995 valid accuracy is the baseline of the results. <br />\nFrom the picture, we can discover some information: <br /> <br />\n**1. Model 2 perform worse on class 0 and class 6 either! Also below the baseline at epoch 30** <br />\n**2. Model 2 can reach almost 1.0 valid accuracy on class 5 at epoch 6.** <br />\n**3. Model 2 can reach almost 1.0 valid accuracy on class 9 at epoch 18** <br />\n**4. Also some classes got better valid accuracy when epoch is low: class 4, class 5, class6 and class 9** <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_single_fold_val_accuracy(old_result_df, 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results of my third model\nThe following picture is the validation accuracy of my third model on epoch 6, 18 and 30. <br />\nAlso 0.995 valid accuracy is the baseline of the results. <br />\nFrom the picture, we can discover some information: <br /> <br />\n**1. Model 3 perform worse on class 0, class 6, class7 and class 9! All of them are lower than the baseline at epoch 30** <br />\n**2. Model 3 can reach 1.0 valid accuracy on class 5 at epoch 18.** <br />\n**3. Model 3 has other 2 classes are very close to 1.0 valid accuracy : class 1 and class 8.** <br />\n**4. Also some classes got better valid accuracy when epoch is low: class 0, class 5** <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_single_fold_val_accuracy(old_result_df, 3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results of all my models in last epoch\n\nThe following picture is the valid accuracy at the epoch 30 of all my trained models. <br /> \nWe can simply get some infromation here: <br />\n\n**1. There are certrain classes have worse valid accuracy in every model.** <br />\n**2. Also there are some classes have quite different results on different model.**<br />\n**3. It is hard to tell which model perform the best on average valid accuracy.**<br />"},{"metadata":{},"cell_type":"markdown","source":"## Conclusion and what might happen\n\nFrom previous information, we can have some simple conclusions\n\n**1. Some classes are not that easy to train compare to others** <br />\n**2. Model might perform different performance, since the training data are slightly different(with different split)** <br />\n**3. Some classes got better valid accuracy when epoch is bigger, some didn't.** <br />\n\n\nLet's talk about first and third concolusions above. Why valid accuracy decrease when epoch increase? Does this sound familiar to you? <br />\nThis is overfitting, right? From the training result, we can see some classes are much harder to train, some are not. <br />\nEasy classes can easily get high valid accuracy when epoch is low. But at the same time, hard classes are still underfitting. <br />\nSo when the model get too confidence on easy classes, it might cause some error when it classifying the hard classes. <br />\nSince some of the kannada numbers are a little similar, so model migth classify the hard examples to easy classes. <br />\nThus, some of the easy classes start to overfitting, and of course, the accuracy start to decrease. <br />\n\nHow can we solve this problem? <br />\n\n**1. Use some speical loss fucntion, which will help model to not getting too confidence on easy class and improve underfitting on hard class.** <br />\n**2. Use more data. This is the most obvious one.** <br />\n**3. Ensemble more models might help.** <br />"},{"metadata":{},"cell_type":"markdown","source":"## How did I modify the kernel to get better result\n\n\nSince I can't get more data, so I try with 1 and 3 options to improve the kernel. <br />\nAnd I did get much stable results after I apply 1 and 3 options to my kernel. <br />\n\n\nFor option 1 above, I try with Symmetric Cross Entropy loss function, which is used to prevent noisy data and model get too confidence on easy class. The code implementation of Symmetric Cross entropy can be easlily find on github, or you can just refer to the following loss function section of this kernel. <br />\n\n\nFor option 3, I increased the folding number of KFold to train more model with different data spliting. I will explain the Multi-Fold training in training section of this kernel<br />\n\n\n**Following part is the training validation accuracy after I adjusted the kernel**<br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"#This csv file saved all the validation result of my current training.\nnew_result_df = pd.read_csv('../input/kmnist-data/New_training_result.csv')\nnew_result_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Result of my new models\n\nWe can see there are many valid accuracy still remain 1.0 when the epoch is 30. <br />\nThis means our new loss function might did prevent the model to get too confidence on easy classes. <br />\nAlthough there are still some models perform bad on cetrain classes, but since we have trained more models with more folds.\nSo the models will balance each other with their own prediction, which might help the score <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nfor fold in range(1,9,1):\n    df = new_result_df[new_result_df['fold'] == fold]\n    plt.plot(df.Class.values-1, df.Valid_accuracy.values, label = 'fold_'+str(fold)+'_model_epoch_'+str(30))\nplt.plot(np.arange(0,10,1), np.ones(10)*0.995, '--', label = 'baseline', c='r')\nplt.plot(np.arange(0,10,1), np.ones(10), '^', label = 'Top accuracy', color='c')\nplt.legend()\nplt.xticks(np.arange(0,10,1))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Let's begin!\n\nThis kernel is using small convolution neural network with tf-keras framework. <br />\nSince this will be a starter kernel, so I will explan the detail as more as possible. <br />\nAlthough this is a starter kernel, I will use slightly advanced architecture and some practical tips to enhance the performance. <br />\n\nHere I will simply introduce the kernel first<br />\n\n## This kernel include\n**1. Very small portion of EDA** <br />\n**2. Albumentation augmentation library** <br />\n**3. Squeeze and Excitiation neural network** <br />\n**4. Swish acitvation function** <br />\n**5. Symmetric cross entropy loss function** <br />\n**6. Multi-Fold Cross-validation and model ensembles** <br />\n**7. Test time augmentation** <br />"},{"metadata":{},"cell_type":"markdown","source":"## Import the modules"},{"metadata":{"colab":{},"colab_type":"code","id":"A-wsJlLv2IS8","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom sklearn.metrics import confusion_matrix, accuracy_score, classification_report\nimport seaborn as sn\nimport albumentations as albu\nfrom sklearn.model_selection import train_test_split, KFold\nfrom tqdm import tqdm_notebook\nimport gc\nimport os\nimport warnings \nwarnings.filterwarnings('ignore')\nmain_dir = '../input/Kannada-MNIST/'\ntf.keras.__version__","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data prepare/EDA/Visualization"},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nHere I read the csv files with pandas api. <br />\nThere are 3 datasets <br />\n1. train.csv -> training dataset\n2. Dig-MNIST.csv -> validation dataset\n3. test.csv -> testing dataset\n\nI will merge training dataset with validation dataset first then use train_test_split or KFold function  to produce training & validation dataset. <br />\nSince I'm not sure whether there will be difference between original training dataset and validation dataset. <br />\nSo I merge them together and trying to make the data more random. <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"pretrain_weights_path = [\n    '../input/kmnist-data/0_model.hdf5',\n    '../input/kmnist-data/1_model.hdf5',\n    '../input/kmnist-data/2_model.hdf5',\n    '../input/kmnist-data/3_model.hdf5',\n    '../input/kmnist-data/4_model.hdf5',\n    '../input/kmnist-data/5_model.hdf5',\n    '../input/kmnist-data/6_model.hdf5',\n    '../input/kmnist-data/7_model.hdf5',\n]\n\n\nuptrain = False\nsubmit = True\nnum_classes = 10\nnum_features = (28,28,1)\nbatch_size = 1024\nlr = 3e-4\nepochs = 33\n\n\nk_fold_split = 8","execution_count":null,"outputs":[]},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":244},"colab_type":"code","id":"yaDeoDY11ySM","outputId":"6e560988-8ab8-4ac8-bec8-afd6072e5239","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(main_dir + 'train.csv')\nvalid_df = pd.read_csv(main_dir + 'Dig-MNIST.csv')\ntest_df = pd.read_csv(main_dir + 'test.csv')\n\n#Check out some training data\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nFrom the print out data above. We will know\n1. The first column is the label of dataset.\n2. Other columns are the digital value of pixels of kannada mnist. And there are 784 columns of digital value in 1 row.\n<br />\n\nThe dataset is the same as regular mnist dataset."},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":67},"colab_type":"code","id":"knEzC2Nc1_i4","outputId":"646c8f59-5cff-4420-d33d-692049f7c07e","trusted":true},"cell_type":"code","source":"#Extract the label from training dataframe and discard the label column\ntrain_label = train_df['label']\ntest_indices = test_df['id']\n\ntrain_df = train_df.drop(['label'], axis = 1)\ntest_df = test_df.drop(['id'], axis = 1)\n\n#Convert dataframe into numpy array \ntrain_x = train_df.values\ntrain_y = train_label.values\n\ntest_x = test_df.values\n\nprint(\"shape of train_x :\", train_x.shape)\nprint(\"shape of train_y :\", train_y.shape)\nprint(\"shape of test_x :\", test_x.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nFirst we turn the original data columns into square dimension, which is 28x28x1(height x width x channel) <br />\nSince we will use the convolution neural network to train the data, so we need to transform the data into image dimension. <br />\nAnd I do one-hot encoding on the original labels <br />"},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":101},"colab_type":"code","id":"ULFjjY0p2RMz","outputId":"149fa8d9-2f32-455e-a1f1-5e9f97d57353","trusted":true},"cell_type":"code","source":"train_x = train_x.reshape(-1,28,28,1)\n# One-hot encode the original label\ntrain_y = tf.keras.utils.to_categorical(train_y, num_classes)\ntest_x = test_x.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":390},"colab_type":"code","id":"EZ-e99hU2ZS0","outputId":"09368970-a625-4d12-c094-bbc7ec0b03ec","trusted":true},"cell_type":"code","source":"#check some image data\ntemp_imgs = train_x[8:12]\ntemp_labels = train_y[8:12]\n\nnrows = 2\nncols = 2\nplt.figure(figsize=(6,6))\n\nfor idx, (img, label) in enumerate(zip(temp_imgs, temp_labels)):\n    plt.subplot(nrows, ncols, idx+1)\n    plt.imshow(np.squeeze(img,axis=2), cmap = 'gray')\n    plt.title(\"label : \" + str(np.argmax(label)))\n    plt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":295},"colab_type":"code","id":"fXSQ-P6V2a0q","outputId":"92fb3317-dfd9-437f-a225-321fd27c6cbb","trusted":true},"cell_type":"code","source":"#check the quantity of each labels\nlabel_counts = train_label.value_counts().reset_index()\n#rename the columns of label_counts\nlabel_counts.columns = ['label', 'quantity']\n#sort the value in label_counts on label column\nlabel_counts = label_counts.sort_values('label')\n\nplt.figure(figsize = (8,4))\nplt.bar(label_counts['label'], label_counts['quantity'])\nplt.xlabel('label')\nplt.ylabel('quantity')\nplt.title('label quantity in training dataset')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nSplit the training and validation dataset with train_test_split function. <br />\nIt is very important to separate your training and validation dataset. <br />\nSince your model might learn the pattern/feature of your training data, but it still have chance to perform bad on the dataset it didnt see before.<br />\nSo we split the original training dataset into two parts for training/validating, thus we can know the model handle the data it didnt see before well or not. <br />\nThis just one of the way to check your model is overfitting or underfitting, there are also KFold, cross-validation.... <br />\n<br />\n<br />\nAlso random state is quite important here, fix the random_state parameter will keep every time train/valid dataset split in same way. <br />\nWhich is easy for your to tune your model. \n\nBut I didn't use train_test_split in this kernel, instead I use K-Fold to do the cross-validation. (at the trainging section)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#x_train, x_test, y_train, y_test = train_test_split(train_x, train_y, test_size = 0.2, random_state = 2019)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data generator and Data augmentation"},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nHere I using albumentation library to do the data augmentation. <br />\nAlbumentation is an excellent library for data augmentation, I prefer to use it than ImageDataGenerator in Keras. <br />\nSince there are more augmentation options in albumentation library, like GridDistortion, RandomGamma, RandomContrast.... <br />\nSo we can make the data more diversity, which can increase the generalization ability of our model. <br />\nGeneralization abilty of model is like how well of our model can handle the data it didnt see before. This is pretty important to the machine learning model, since we hope our model not only can predict the training data right, but also get good prediction ability on real world data. <br />\n"},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nI created two data generator for training and validation. And there are no augmentation operation on validation dataset. Since we want to observe the performance on data which is more original. But there is one techique called TTA(Test Time Augmentation). TTA is a techique that we do augmentation operation on testing data. For instance, we have a testing data we would like to predict, we can perform the augumentation on it and make several versions of this original testing data. Then we use our model to predict all the testing data we just created to average the result. This can make the prediction more stable since we average the error. I will perform a simple version of TTA at the end for predicting submission csv file."},{"metadata":{},"cell_type":"markdown","source":"## Check some augmentation effect"},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_aug_effect(img, aug, repeat=3, aug_item = 'rotate'):\n    '''\n    img : input image for display\n    aug : augmentation object to perform image multiplication\n    repeat : how much time you want to perfrom multiplication\n    aug_item : certain multiplication you want to apply to input image\n    '''\n    plt.figure(figsize=(int(4*(repeat+1)),4))\n    plt.subplot(1,repeat+1, 1)\n    plt.imshow(img, cmap='gray')\n    plt.title('original image')\n    \n    for i in range(repeat):\n        plt.subplot(1, repeat+1, i+2)\n        temp_aug_img = aug(image = img.astype('uint8'))['image']\n        plt.imshow(temp_aug_img, cmap='gray')\n        plt.title(aug_item + ' : ' + str(i+1))\n    \n    plt.axis('off')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Shift Scale Rotate**"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_aug = albu.ShiftScaleRotate(scale_limit=0.2, rotate_limit=20, shift_limit=0.15, p=1, border_mode=0)\ndisplay_aug_effect(np.squeeze(temp_imgs[0], axis=2), temp_aug, aug_item = 'ShiftScaleRotate')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Grid Distortion**"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_aug = albu.GridDistortion(p=1)\ndisplay_aug_effect(np.squeeze(temp_imgs[1], axis=2), temp_aug, aug_item = 'GridDistortion')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Random Brightness/Random Gamma/Random Contrast**"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_aug = albu.OneOf([ albu.RandomBrightness(limit=10), albu.RandomGamma(gamma_limit=(80, 120)), albu.RandomContrast(limit=1.5) ], p=1)\ndisplay_aug_effect(np.squeeze(temp_imgs[2], axis=2), temp_aug, aug_item = 'Gamma/Brightness/Contrast')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Random Crop**"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_aug = albu.RandomCrop(height=24,width=24,p=1)\ndisplay_aug_effect(np.squeeze(temp_imgs[3], axis=2), temp_aug, aug_item = 'RandomCrop')","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"kDyrEnSt2fih","trusted":true},"cell_type":"code","source":"class InputGenerator(tf.keras.utils.Sequence):\n    \n    def __init__(self,\n                 x,\n                 y=None,\n                 aug=None,\n                 batch_size=128,\n                 training=True):\n        \n        self.x = x\n        self.y = y\n        self.aug = aug\n        self.batch_size = batch_size\n        self.training = training\n        self.indices = range(len(x))\n    \n    def __len__(self):\n        return len(self.x) // self.batch_size\n    \n    def __getitem__(self,index):\n        \n        batch_indices = self.indices[index * self.batch_size : (index+1)*self.batch_size]\n        batch_data = self.__get_batch_x(batch_indices)\n        \n        if self.training:\n            batch_label = self.__get_batch_y(batch_indices)\n            return batch_data, batch_label\n        else:\n            return batch_data\n    \n    def on_epoch_start(self):\n        \n        if self.training:\n            np.random.shuffle(self.indices)\n            \n    def __get_batch_x(self, batch_indices):\n        \n        batch_data = []\n        for idx in batch_indices:\n            cur_data = self.x[idx].astype('uint8')\n            cur_data = self.aug(image = cur_data)['image']\n            batch_data.append(cur_data)\n            \n        return np.stack(batch_data)/255.0\n    \n    def __get_batch_y(self, batch_indices):\n        \n        batch_label = []\n        for idx in batch_indices:\n            batch_label.append(self.y[idx])\n            \n        return np.stack(batch_label)\n\n\n        \ntrain_aug = albu.Compose([\n                    albu.ShiftScaleRotate(scale_limit=0.2, rotate_limit=15.0, shift_limit=0.15, p=0.5, border_mode=0, value = 0)]\n                    )\n\nvalid_aug = albu.Compose([])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the model"},{"metadata":{},"cell_type":"markdown","source":"**Little summary of model** <br />\n<br />\n**Model architure** : Use 8 convolution layers with 1 squeeze-and-excitiation blocks and 3 fully-connected layers. <br />\n**Activation function** : The activation function in the hidden layers of  neural network is Swish, and the activation function in output layer is softmax. <br />\n**Optimizer** : RMSProp. <br />\n**Loss function** : Symmetric categorical cross entropy <br />\n<br />\n<br />\n**Note**\n\nHere I would like to introduce two parts of below code block. <br />\n1. Swish activation\n2. Squeeze-and-Excitiation nerual network\n\n<br />\n**Swish activation** <br />\n<br />\nSwish activation is proposed by google brain, which just simply multiply the input tensor with sigmoid tensor( f(x) = x * sigmoid(x) ). The ramp function of swish is quite similar to relu, but  it is not so monotonic as relu. We all know relu will make the input tensor to zero once its value smaller than zero. This behavior will make the parameter of neuron won't be updated and just assign to its original value. But swish can create much more difference during the training since it will not simply make the value to zero. Although swish activation might wont help too much on small architecture(like I created in this kernel), but it might be helpful on deeper neural network architecture.\n\n<br />\n**Squeeze and Excitation neural network** <br />\n<br />\nSE net is an architecture proposed by J Hu. It can be divide in two parts, squeeze and excitation parts. \nIn very simple explanation, it aggregates feature maps across their spatial dimension and makes them as channel descriptor. And it can learn the importance of dependencies of each channels during training. \nThen multiply the weights to its corresponding channels of input tensor to enhance/decrease the importance. <br />\n<br />\nOriginal article : https://arxiv.org/abs/1709.01507"},{"metadata":{"trusted":true},"cell_type":"code","source":"def _swish(x):\n    '''\n    x : input tensor\n    \n    return : swish activated tensor\n    '''\n    return tf.keras.backend.sigmoid(x) * x\n\n#helper function of Squeeze and Excitation block.\ndef _seblock(input_channels=32, se_ratio=2):\n    '''\n    input_channels : the channels of input tensor\n    se_ratio : the ratio for reducing the first fully-conntected layer\n    \n    return : helper function for entire se block\n    '''\n    def f(input_x):\n        \n        reduced_channels = input_channels // se_ratio\n        \n        x = tf.keras.layers.GlobalAveragePooling2D()(input_x)\n        x = tf.keras.layers.Dense(units=reduced_channels, kernel_initializer='he_normal')(x)\n        x = tf.keras.layers.Activation(_swish)(x)\n        x = tf.keras.layers.Dense(units=input_channels, kernel_initializer='he_normal', activation='sigmoid')(x)\n        \n        x = tf.keras.layers.multiply([x,input_x])\n        \n        return x\n    return f\n\ndef _cn_bn_act(filters=64, kernel_size=(3,3), strides=(1,1)):\n    '''\n    filters : filter number of convolution layer\n    kernel_size : filter/kernel size of convolution layer\n    strides : stride size of convolution layer\n    \n    return : helper function for convolution -> batch normalization -> activation\n    '''\n    def f(input_x):\n        \n        x = tf.keras.layers.Conv2D(filters=filters, kernel_size=kernel_size, strides=strides, padding='same', kernel_initializer='he_normal')(input_x)\n        x = tf.keras.layers.BatchNormalization()(x)\n        x = tf.keras.layers.Activation(_swish)(x)\n        \n        return x\n    \n    return f\n\n\ndef _dn_bn_act(units=128):\n    '''\n    units : units for fully-connected layer\n    \n    return : helper function for fully-connected -> batch normalization -> activation\n    '''\n    def f(input_x):\n        \n        x = tf.keras.layers.Dense(units=units, kernel_initializer='he_normal')(input_x)\n        x = tf.keras.layers.BatchNormalization()(x)\n        x = tf.keras.layers.Activation(_swish)(x)\n        return x\n    \n    return f\n\ndef build_model(input_shape = (28,28,1), classes = 10):\n    '''\n    input_shape : input dimension of single data\n    classes : class number of label\n    \n    return : cnn model\n    '''\n    input_layer = tf.keras.layers.Input(shape = input_shape)\n    \n    x = _cn_bn_act(filters=64)(input_layer)\n    #x = _seblock(input_channels=64)(x)\n    x = _cn_bn_act(filters=64)(x)\n    x = _cn_bn_act(filters=64)(x)\n        \n    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2))(x)\n    x = tf.keras.layers.Dropout(0.2)(x)\n    \n    x = _cn_bn_act(filters=128)(x)\n    #x = _seblock(input_channels=128)(x)\n    x = _cn_bn_act(filters=128)(x)\n    x = _cn_bn_act(filters=128)(x)\n    \n    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2))(x)\n    x = tf.keras.layers.Dropout(0.2)(x)\n    \n    x = _cn_bn_act(filters=256)(x)    \n    x = _seblock(input_channels=256)(x)\n    x = _cn_bn_act(filters=256)(x)    \n    \n    x = tf.keras.layers.Dropout(0.2)(x)\n    x = tf.keras.layers.Flatten()(x)\n    \n    x = _dn_bn_act(units=256)(x)\n    x = _dn_bn_act(units=128)(x)\n    output_layer = tf.keras.layers.Dense(units=classes, kernel_initializer='he_normal', activation = 'softmax')(x)\n    \n    model = tf.keras.models.Model(inputs=[input_layer], outputs=[output_layer])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nI prefer to add Precision and Recall into metrics. Since I like to monitor the trending of false positive and negative positive during training. <br />\nYou can always customize your own metrics/loss fucntion for model training. <br />\nAs long as the function has y_true and y_pred parameters(Remember that the y_true and y_pred send in function are tensors)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Precision(y_true, y_pred, epsilon=1e-7):\n    \n    y_true_f = tf.keras.backend.flatten(y_true)\n    y_pred_f = tf.keras.backend.flatten(y_pred)\n    \n    y_pred_f = tf.keras.backend.round(y_pred_f)\n    \n    TP = tf.keras.backend.sum(y_true_f * y_pred_f)\n    FP = tf.keras.backend.sum((1-y_true_f) * y_pred_f)\n    \n    return TP/(TP+FP+epsilon)\n\ndef Recall(y_true, y_pred, epsilon=1e-7):\n    \n    y_true_f = tf.keras.backend.flatten(y_true)\n    y_pred_f = tf.keras.backend.flatten(y_pred)\n    \n    y_pred_f = tf.keras.backend.round(y_pred_f)\n    \n    TP = tf.keras.backend.sum(y_true_f * y_pred_f)\n    TN = tf.keras.backend.sum(y_true_f * (1-y_pred_f))\n    \n    return TP/(TP+TN+epsilon)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def symmetric_cross_entropy(alpha=1.0, beta=1.0, epsilon=1e-7):\n    def loss(y_true, y_pred):\n        \n        y_pred_ce = tf.clip_by_value(y_pred, epsilon, 1.0)\n        y_true_rce = tf.clip_by_value(y_true, epsilon, 1.0)\n\n        ce = alpha*tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred_ce), axis = -1))\n        rce = beta*tf.reduce_mean(-tf.reduce_sum(y_pred * tf.math.log(y_true_rce), axis = -1))\n        \n        return  ce + rce\n    return loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the model"},{"metadata":{},"cell_type":"markdown","source":"## callbacks"},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nCallback functions are very important during training. Here I use the callback functions that already finished by keras. But you can write your own callback function if you want. <br />\ncustom callbacks in keras : https://keras.io/callbacks/ <br />\n\n<br />\n**ReduceLROnPlateau** : This callback function can monitor certain value you want to check and reduce the learning rate when it didnt improve in cetrain epochs. <br />\n**ModelCheckpoint** : This callback function also can monitor ctratin value you want and save the weights of your model once this certrain value hit the best performance than before. Or maybe you don't have enough time to train the current model, you can always save the weight and uptrain the model when you are available. <br />\n**EarlyStopping** : This callback function will stop the training if your monitor value didnt imporve in certain epochs. <br />"},{"metadata":{},"cell_type":"markdown","source":"# KFold Cross-Validation(CV)/Ensemble models"},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nKFold is a very common cross-validation method. It will split your original training data into several pieces(depends on the n_splits parameters). <br />\nAnd use 1/n_splits portion of data to be the validation data and rest of data to be training data. <br />\nBut this validation data will keep change until it go through all your original training data, for instance <br />\n<br />\noriginal data = [1,2,3] <br />\nn_splits = 3 <br />\n<br />\n**1/3 training:**<br />\nvalidatng data : [1] <br />\ntraining data : [2,3] <br />\n<br />\n**2/3 training:** <br />\nvalidating data : [2] <br />\ntraining data : [1,3]<br />\n<br />\n**3/3 training:** <br />\nvalidating data : [3] <br />\ntraining data : [1,2] <br />\n<br />\nBut we also know, it is not a good idea to blend the training data with validating data. <br />\nSo We can use several models to train on each different spliting situation. <br />\nAfter training, we will have n_splits models, which all of them are training on (1-1/n_splits) of original training data and validating on 1/n_splits of original training data. <br />\nSo all of these models were training on different(partially different) datasets. <br /> \nThen we can use these models to predict the data we want to predict, thus we get n_splits different prediction results. <br />\nIn the end, we can blend these results by voting or averaging them and get a much stable result than use only one model. <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Before training, I would like to shuffle the training dataset to make the data more random\n# But we also need to shuffle the training label and keep it has same corresponding relation with training dataset.\n# So I use permutation in numpy to create the new indices for both new training data & label\npermutation = np.random.RandomState(2019).permutation(len(train_x))\ntrain_x = train_x[permutation]\ntrain_y = train_y[permutation]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kf = KFold(n_splits=k_fold_split, random_state=2019)\n\n#list to save all the models we are going to train\nmodel_members = []\ncheck_points_path = []\n\nfor idx in range(k_fold_split):\n    check_points_path.append(str(idx) + '_model.hdf5')\n    \nfor model_index, (train_indices, valid_indices) in enumerate(kf.split(train_x)):\n    \n    #data generator for training\n    train_aug_gen = InputGenerator(train_x[train_indices], train_y[train_indices], train_aug, batch_size)\n    #data generator for validating\n    valid_aug_gen = InputGenerator(train_x[valid_indices], train_y[valid_indices], valid_aug, batch_size)\n    \n    model = build_model()\n    \n    optimizer = tf.keras.optimizers.RMSprop(lr=lr)\n    model.compile(optimizer=optimizer, metrics=['accuracy',Precision,Recall], loss=symmetric_cross_entropy())\n\n    steps_per_epoch = len(train_x[train_indices]) // batch_size\n    validation_steps = len(train_x[valid_indices]) // batch_size\n    \n    reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', patience=3, factor=0.5, min_lr=1e-8, mode='min', verbose=2)\n    check_point = tf.keras.callbacks.ModelCheckpoint(monitor='val_loss', filepath = check_points_path[model_index], mode='min', save_best_only=True, verbose=2)\n    early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)\n    \n    training_callbacks = [ check_point, reduce_lr ]\n    \n    if pretrain_weights_path == None or uptrain == True:\n        \n        if pretrain_weights_path != None:\n            print('*'*10,'Load ',model_index,'-fold pretrain weights','*'*10)\n            model.load_weights(pretrain_weights_path[model_index])\n            \n        \n        \n        print('*'*30,'Training model ', model_index+1,'*'*30)\n        print('Train on ', len(train_indices),' data')\n        print('Valid on ', len(valid_indices),' data')\n\n        history = model.fit_generator(generator=train_aug_gen,\n                                      steps_per_epoch=steps_per_epoch,\n                                      epochs=epochs,\n                                      validation_data=valid_aug_gen,\n                                      validation_steps=validation_steps,\n                                      workers=-1,\n                                      verbose=2,\n                                      callbacks=training_callbacks)\n        \n        print('*'*30,'Validating model ', model_index+1,'*'*30)\n        val_gen = InputGenerator(train_x[valid_indices], None, albu.Compose([]), 1, training=False)\n        \n        print('\\n\\n')\n        preds = model.predict_generator(val_gen, workers=-1, verbose=1)\n        truths = train_y[valid_indices]\n        preds = np.round( np.array(preds) )\n        truths = np.array(truths)\n        valid_results = []\n        for c in range(num_classes):\n            valid_results.append((c+1, np.sum(preds[:,c] * truths[:,c])/ np.sum(truths[:,c])))\n        \n        valid_results_df = pd.DataFrame(valid_results, columns=['Class', 'Valid_accuracy'])\n        valid_results_df.to_csv('new_ValidationResults'+str(model_index+1)+'.csv', index=False)\n        print(valid_results_df)\n\n        del history, val_gen, train_aug_gen, valid_aug_gen, preds, truths, valid_results_df\n        gc.collect()\n        \n        \n    else:\n        print('Load the pretrain weights '+str(model_index+1))\n        model.load_weights(pretrain_weights_path[model_index])\n        \n    model_members.append(model)\n    del model\n    print('\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Time Augmentation"},{"metadata":{},"cell_type":"markdown","source":"**Note**\n\nHere I still use albumentations library to do the TTA. And I created a simple tta_wrapper class. It will wrap the original model you just trained then use it to perform tta. <br />\nThe concept is quite simple, it took the model then predict the results of original test image and images after some multiplications. <br />\nSince the output of the model is the probability of each class in one-hot-encoded form, which already normalized by softmax activation. <br />\nSo I simply add up the probabilities then average them. Thus I got a new result by averaging the prediction results of several different view of original test image. <br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"tta_aug = albu.Compose([\n                    albu.ShiftScaleRotate(scale_limit=0.1, rotate_limit=10, shift_limit=0.10, p=1.0, border_mode=0, value=0)\n                    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class tta_wrapper():\n\n    def __init__(self,\n                 model,\n                 normal_generator,\n                 aug_generator,\n                 repeats = 1\n                 ):\n        '''\n        model : model you trained on your original data\n        normal_generator : generator for data without augmentation\n        aug_generator : generator for data with augmentation\n        repeats : how many times you want to use model to predict on augmentation data\n        '''\n        self.model = model\n        self.normal_generator = normal_generator\n        self.aug_generator = aug_generator\n        self.repeats = repeats\n    \n        \n    def predict(self):\n        \n        '''\n        return : Averaging results of several different version of original test images\n        '''\n\n        batch_label = self.model.predict_generator( normal_generator,\n                                                    workers=-1,\n                                                    verbose=1)\n        \n        for idx in range(self.repeats):\n\n            batch_label += self.model.predict_generator( aug_generator,\n                                                         workers=-1,\n                                                         verbose=1)\n        batch_label /= (self.repeats+1)\n        return batch_label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":false},"cell_type":"code","source":"if submit == True:\n    predictions = np.zeros((len(test_x), num_classes))\n\n    for model_index, model in enumerate(model_members):\n        print(str(model_index+1) + '_model predicting')\n    \n        normal_generator = InputGenerator(test_x,\n                                          aug = albu.Compose([]),\n                                          training=False,\n                                          batch_size=250)\n        \n        aug_generator = InputGenerator(x=test_x,\n                                       aug = tta_aug,\n                                       training=False,\n                                       batch_size=250)\n    \n        tta_model = tta_wrapper(model=model,\n                                normal_generator=normal_generator,\n                                aug_generator = aug_generator,\n                                repeats=1)\n    \n        predictions += tta_model.predict()\n\n    predictions = predictions / k_fold_split\n    predictions = np.argmax(predictions, axis=1)\n    submission = pd.DataFrame({'id' : test_indices, 'label':predictions})\n    submission.to_csv('submission.csv', index = False)\n    submission.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation on real world data"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_labels = valid_df['label']\nvalid_datas = valid_df.drop(['label'],axis=1).values\nvalid_datas = valid_datas.reshape(-1,28,28,1)\nvalid_predictions = np.zeros((len(valid_datas), num_classes))\n\nfor model_index, model in enumerate(model_members):\n    print(str(model_index+1) + '_model predicting')\n    \n    normal_generator = InputGenerator(x=valid_datas,\n                                      aug = albu.Compose([]),\n                                      training=False,\n                                      batch_size=160)\n        \n    aug_generator = InputGenerator(x=valid_datas,\n                                   aug = tta_aug,\n                                   training=False,\n                                   batch_size=160)\n    \n    tta_model = tta_wrapper(model=model,\n                            normal_generator=normal_generator,\n                            aug_generator = aug_generator,\n                            repeats=1)\n    \n    valid_predictions += tta_model.predict()\n    \nvalid_predictions = valid_predictions/k_fold_split\nvalid_predictions = np.argmax(valid_predictions,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint('Real world validation accuracy : {}'.format(accuracy_score(valid_labels, valid_predictions)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(classification_report(valid_labels, valid_predictions))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,12))\nconfusion_mat = confusion_matrix(valid_labels, valid_predictions)\nsn.heatmap(confusion_mat, annot=True, cmap='YlGnBu')\nplt.title('Confusion matrix of Real World validation result')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above validation result of real world data, it is very obvious my models didn't perform well on real world data. <br />\nWe might need to use more data augmentation when training the model to improve the model generalization. <br />\nAlso from the result we can know one thing, my models were still overfitting to ideal data. <br />\nWhich means the public testing dataset might have very few portion of real data, or even none. <br />\nBut maybe there are some real world data in the private testing dataset.<br />\nSo I think we might need to train the model on public validation dataset as well.<br />\nWhich may help you to stay in good position after private score is release."},{"metadata":{},"cell_type":"markdown","source":"# What might help the kernel to further improve?\n\n1. Transfer learning, you can train the model on other dataset(like original mnist) then use the pretain model to train kannada mnist dataset.\n2. More fold\n3. Try with different loss function \n4. Dig into the dataset\n\n**Thanks for reading!**"}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"NewMnist","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"10fe50f7a8d3438e9196a101c528fc17":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"1b1546350fe44aa7bd8969cca5f59b0f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"20d49567ed554e57874fe0d860715838":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"","description":" 64%","description_tooltip":null,"layout":"IPY_MODEL_a8d66e6f69b54636964edc6122a9dd44","max":5000,"min":0,"orientation":"horizontal","style":"IPY_MODEL_efbc64e02b554ad9974c288ff68679e3","value":3179}},"2351f31467784301aaf33563045195eb":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"3ae7fc424f2840da9716b4d9bfb33f0f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_10fe50f7a8d3438e9196a101c528fc17","placeholder":"​","style":"IPY_MODEL_ba1e327234ab400080cf38a6e555c814","value":" 2870/5000 [05:24&lt;03:50,  9.23it/s]"}},"3af7dd535f8d404a9e93906b52d5642c":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_b882cab5c0604ea7bd36ddf622e77862","placeholder":"​","style":"IPY_MODEL_2351f31467784301aaf33563045195eb","value":" 3179/5000 [05:48&lt;03:40,  8.27it/s]"}},"9ff20c91cae449d9b5a797f230e7ad26":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"","description":" 57%","description_tooltip":null,"layout":"IPY_MODEL_aa8adc88dcc64735bed54967c83c31d7","max":5000,"min":0,"orientation":"horizontal","style":"IPY_MODEL_af6acf89d52f48a7aada85a51ffad6f4","value":2870}},"a8d66e6f69b54636964edc6122a9dd44":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"aa8adc88dcc64735bed54967c83c31d7":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"af6acf89d52f48a7aada85a51ffad6f4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"b2933e692a144f859ae7549543c2b2c4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_9ff20c91cae449d9b5a797f230e7ad26","IPY_MODEL_3ae7fc424f2840da9716b4d9bfb33f0f"],"layout":"IPY_MODEL_1b1546350fe44aa7bd8969cca5f59b0f"}},"b63465f2a8974f41afdf4201d60f4c06":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_20d49567ed554e57874fe0d860715838","IPY_MODEL_3af7dd535f8d404a9e93906b52d5642c"],"layout":"IPY_MODEL_d1cf3df13828465786b251f26a0bab44"}},"b882cab5c0604ea7bd36ddf622e77862":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"ba1e327234ab400080cf38a6e555c814":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"d1cf3df13828465786b251f26a0bab44":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"efbc64e02b554ad9974c288ff68679e3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":1}