{"cells":[{"metadata":{},"cell_type":"markdown","source":"Introduction\n
\n\nIn this kernel we will be exploring different model architectures, their performance (against one another) and create an ensembled submission of all of our models and submit it to the Leaderboard.\n\nI've picked 4 of my best performing models so far for this kernel, everything coded up using the **fast.ai** library and PyTorch.\n\n> **And... Please don't forget to smash that UPVOTE button.**\n\nSo lets jump right into it...\n\n\n\n\n## What are we doing in this kernel?\n\n* **1. Getting Started**\n\n* **2. Loading Data + EDA**\n\n* **3. Data Processing**\n\n* **4. Defining our Models**\n\n* **5. Training and Inference**\n\n* **6. Comparing Model Performances**\n\n* **7. Creating Average Ensembled Submission**\n"},{"metadata":{},"cell_type":"markdown","source":"Show your appreciation by giving this kernel an UPVOTE"},{"metadata":{},"cell_type":"markdown","source":"## Getting Started\nHere we import all the requires libraries and utillity functions."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nplt.style.use(\"ggplot\")\nimport seaborn as sns\n\nfrom fastai.vision import *\nfrom fastai.callbacks import SaveModelCallback\n\nfrom radam_optimizer_pytorch import RAdam\nfrom torch.nn import Conv2d\nfrom torch.optim import Adam\n\nimport os\nPATH = Path('../input/Kannada-MNIST/')\nos.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Set the same random seeds for all libraries to ensure reproducibility."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting Global Random Seed\ndef random_seed(seed_value, use_cuda): \n np.random.seed(seed_value) # cpu vars\n torch.manual_seed(seed_value) # cpu vars\n random.seed(seed_value) # Python\n if use_cuda: torch.cuda.manual_seed_all(seed_value) # gpu \n\nrandom_seed(42, True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data + EDA"},{"metadata":{},"cell_type":"markdown","source":"The images are given in standard format with a CSV for refenrence. Let's read that in first. The CSV file contains pixel values for all individual pixels in the image. The Image dimension is 28x28 so we have 784 pixel values."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv(PATH/'train.csv')\ntrain_csv.head().T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The labels aren't given in the CSV file so we'll go fetch that."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_data_labels(csv,label):\n fileraw = pd.read_csv(csv)\n labels = fileraw[label].to_numpy()\n data = fileraw.drop([label],axis=1).to_numpy(dtype=np.float32).reshape((fileraw.shape[0],28,28))\n data = np.expand_dims(data, axis=1)\n return data, labels\n\ntrain_data, train_labels = get_data_labels(PATH/'train.csv','label')\ntest_data, test_labels = get_data_labels(PATH/'test.csv','id')\nother_data, other_labels = get_data_labels(PATH/'Dig-MNIST.csv','label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f' Train:\\tdata shape {train_data.shape}\\tlabel shape {train_labels.shape}\\n \\\nTest:\\tdata shape {test_data.shape}\\tlabel shape {test_labels.shape}\\n \\\nOther:\\tdata shape {other_data.shape}\\tlabel shape {other_labels.shape}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.title(f'Training Label: {train_labels[43]}')\nplt.imshow(train_data[43,0],cmap='gray');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wait don't get confused that's not a 2. Remember the numbers are in a language called Kannada which is one of many languages spoken in the southern part of India."},{"metadata":{},"cell_type":"markdown","source":"## Processing Data for Training"},{"metadata":{},"cell_type":"markdown","source":"Splitting the Full Train set into\n\n**80% - Training**\n\n**20% - Validation**"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)\nran_20_pct_idx = (np.random.random_sample(train_labels.shape)) < .2\n\ntrain_80_labels = train_labels[np.invert(ran_20_pct_idx)]\ntrain_80_data = train_data[np.invert(ran_20_pct_idx)]\n\nvalid_20_labels = train_labels[ran_20_pct_idx]\nvalid_20_data = train_data[ran_20_pct_idx]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we create a fastai databunch. I strongly recommend reading the documentation here [docs.fast.ai](https://www.docs.fast.ai) if you have any problems understanding what's happening in the next few lines."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ArrayDataset(Dataset):\n \"Dataset for numpy arrays based on fastai example: \"\n def __init__(self, x, y):\n self.x, self.y = x, y\n self.c = len(np.unique(y))\n \n def __len__(self):\n return len(self.x)\n \n def __getitem__(self, i):\n return self.x[i], self.y[i]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = ArrayDataset(train_80_data,train_80_labels)\nvalid_ds = ArrayDataset(valid_20_data,valid_20_labels)\nother_ds = ArrayDataset(other_data, other_labels)\ntest_ds = ArrayDataset(test_data, test_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bs = 64 # Batch Size\ndata = DataBunch.create(train_ds, valid_ds, test_ds=test_ds, bs=bs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Models"},{"metadata":{},"cell_type":"markdown","source":"We create a models directory for fast.ai to save and load our models."},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MODEL_DIR = Path('../working/models/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model 1** in our experiment is just a small custom Convolutional Neural Net that has previously worked very well for me in the original MNIST dataset. The architecture of this model is pretty straight forward.\n\nIf you have troubles understanding how CNNs are coded in PyTorch you can check out some of my other tutorial kernels listed below\n* [MNIST: Introduction to Computer Vision with PyTorch](https://www.kaggle.com/abhinand05/in-depth-guide-to-convolutional-neural-networks)\n\n* [In-Depth Guide to Convolutional Neural Networks](https://www.kaggle.com/abhinand05/in-depth-guide-to-convolutional-neural-networks)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ConvNet1(nn.Module):\n def __init__(self):\n super(ConvNet1, self).__init__()\n self.layer1 = nn.Sequential(\n nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),\n nn.ReLU(),\n nn.BatchNorm2d(32)\n )\n self.layer2 = nn.Sequential(\n nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, padding=1),\n nn.ReLU(),\n nn.BatchNorm2d(32)\n )\n self.layer3 = nn.Sequential(\n nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5, padding=1, stride=2),\n nn.ReLU(),\n nn.BatchNorm2d(64) \n )\n self.layer4 = nn.Sequential(\n nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1),\n nn.ReLU(),\n nn.BatchNorm2d(64)\n )\n self.layer5 = nn.Sequential(\n nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1),\n nn.ReLU(),\n nn.BatchNorm2d(64)\n )\n self.layer6 = nn.Sequential(\n nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5, padding=1, stride=2),\n nn.ReLU(),\n nn.BatchNorm2d(128),\n )\n \n self.drop_out = nn.Dropout(0.2)\n self.relu = nn.ReLU()\n\n self.fc1 = nn.Linear(4608, 128)\n self.fc2 = nn.Linear(128, 10)\n self.bn1d = nn.BatchNorm1d(128)\n self.output = nn.Softmax(dim=1)\n \n def forward(self, x):\n # conv layers\n out = self.layer1(x)\n out = self.layer2(out)\n out = self.drop_out(self.layer3(out))\n out = self.layer4(out)\n out = self.layer5(out)\n out = self.drop_out(self.layer6(out))\n out = out.view(out.shape[0], -1)\n# print(out.shape) # Life Saving Debuggung Step\n # FC Layer 1\n out = self.fc1(out)\n out = self.relu(out)\n out = self.bn1d(out)\n out = self.drop_out(out)\n # Output layer\n out = self.fc2(out)\n out = self.output(out)\n return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# Move Model to GPU\nconv_net_1 = ConvNet1()\nconv_net_1 = conv_net_1.to(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model 2** is also one of those that has worked well on the original MNIST Dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ConvNet2(nn.Module):\n def __init__(self):\n super(ConvNet2, self).__init__()\n self.layer1 = nn.Sequential(\n nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2, stride=2))\n self.layer2 = nn.Sequential(\n nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2, stride=2))\n self.drop_out = nn.Dropout(0.5)\n self.fc1 = nn.Linear(7 * 7 * 64, 1000)\n self.fc2 = nn.Linear(1000, 10)\n self.relu = nn.ReLU()\n \n def forward(self, x):\n out = self.layer1(x)\n out = self.layer2(out)\n out = out.reshape(out.size(0), -1)\n out = self.drop_out(out)\n out = self.fc1(out)\n out = self.relu(out)\n out = self.drop_out(out)\n out = self.fc2(out)\n return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Move to GPU\nconv_net_2 = ConvNet2()\nconv_net_2 = conv_net_2.to(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model 3**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Helper Function for Model 3\ndef conv2(ni,nf,stride=2,ks=3): return conv_layer(ni,nf,stride=stride,ks=ks)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"conv_net_3 = nn.Sequential(\n conv2(1,32,stride=1,ks=3),\n conv2(32,32,stride=1,ks=3),\n conv2(32,32,stride=2,ks=5),\n nn.Dropout(0.4),\n \n conv2(32,64,stride=1,ks=3),\n conv2(64,64,stride=1,ks=3),\n conv2(64,64,stride=2,ks=5),\n nn.Dropout(0.4),\n \n Flatten(),\n nn.Linear(3136, 128),\n relu(inplace=True),\n nn.BatchNorm1d(128),\n nn.Dropout(0.4),\n nn.Linear(128,10)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model 4: (ResNet18)**\n\nThrowing proven architectures like ResNet18 into the mix might help to improve our ensemble. "},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!ls ../input/pytorch-pretrained-models","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We load the ResNet Model and change the in_channels to 1 because this dataset only contains Greyscale images."},{"metadata":{"trusted":true},"cell_type":"code","source":"rn18 = models.resnet18(pretrained=False)\nrn18.load_state_dict(torch.load('../input/pytorch-pretrained-models/resnet18-5c106cde.pth'))\nrn18.conv1 = Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Show your appreciation by giving this kernel an UPVOTE if you liked."},{"metadata":{},"cell_type":"markdown","source":"## Training and Inference"},{"metadata":{},"cell_type":"markdown","source":"Now we train the models using fastai library one by one."},{"metadata":{},"cell_type":"markdown","source":"### Model - 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"learner1 = Learner(data, \n conv_net_1, \n metrics=accuracy, \n model_dir=MODEL_DIR,\n opt_func=Adam,\n loss_func=nn.CrossEntropyLoss()\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner1.lr_find()\nlearner1.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearner1.fit_one_cycle(50, \n slice(1e-03),\n callbacks=[SaveModelCallback(learner1, \n every='improvement', \n monitor='accuracy', \n name='best_model_1')]\n ) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner1.recorder.plot_losses(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner1.recorder.plot_metrics(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner1.load('best_model_1')\npreds, ids = learner1.get_preds(DatasetType.Test)\ny = torch.argmax(preds, dim=1)\n\nsubmission_1 = pd.DataFrame({ 'id': ids,'label': y })\nsubmission_1.to_csv(\"submission.csv\", index=False)\nsubmission_1.to_csv(\"submission_1.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model - 2"},{"metadata":{"trusted":true},"cell_type":"code","source":"learner2 = Learner(data, \n conv_net_2, \n metrics=accuracy, \n model_dir=MODEL_DIR,\n opt_func=RAdam,\n loss_func=nn.CrossEntropyLoss()\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner2.lr_find()\nlearner2.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearner2.fit_one_cycle(50, \n slice(1e-03),\n callbacks=[SaveModelCallback(learner2, \n every='improvement', \n monitor='accuracy', \n name='best_model_2')]\n ) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner2.recorder.plot_losses(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner2.recorder.plot_metrics(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner2.load('best_model_2')\npreds, ids = learner2.get_preds(DatasetType.Test)\ny = torch.argmax(preds, dim=1)\n\nsubmission_2 = pd.DataFrame({ 'id': ids,'label': y })\nsubmission_2.to_csv(\"submission_2.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model - 3"},{"metadata":{"trusted":true},"cell_type":"code","source":"learner3 = Learner(data, \n conv_net_3, \n metrics=accuracy, \n model_dir=MODEL_DIR,\n opt_func=Adam,\n loss_func=nn.CrossEntropyLoss()\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner3.lr_find()\nlearner3.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearner3.fit_one_cycle(50, \n slice(8e-03),\n callbacks=[SaveModelCallback(learner3, \n every='improvement', \n monitor='accuracy', \n name='best_model_3')]\n ) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner3.recorder.plot_losses(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner3.recorder.plot_metrics(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner3.load('best_model_3')\npreds, ids = learner3.get_preds(DatasetType.Test)\ny = torch.argmax(torch.exp(preds), dim=1)\n\nsubmission_3 = pd.DataFrame({ 'id': ids,'label': y })\nsubmission_3.to_csv(\"submission_3.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model - 4"},{"metadata":{"trusted":true},"cell_type":"code","source":"learner4 = Learner(data, \n rn18, \n metrics=accuracy, \n model_dir=MODEL_DIR,\n opt_func=Adam,\n loss_func=nn.CrossEntropyLoss()\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner4.lr_find()\nlearner4.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearner4.fit_one_cycle(50, \n slice(1e-03),\n callbacks=[SaveModelCallback(learner4, \n every='improvement', \n monitor='accuracy', \n name='best_model_4')]\n ) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner4.recorder.plot_losses(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner4.recorder.plot_metrics(skip_start=800)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner4.load('best_model_4')\npreds, ids = learner4.get_preds(DatasetType.Test)\ny = torch.argmax(torch.exp(preds), dim=1)\n\nsubmission_4 = pd.DataFrame({ 'id': ids,'label': y })\nsubmission_4.to_csv(\"submission_4.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Comparing Models"},{"metadata":{},"cell_type":"markdown","source":"We extract the data stored in `learner.recorder` after training to plot a few graphs to evaluate the performance of our models."},{"metadata":{"trusted":true},"cell_type":"code","source":"flatten = lambda l: [np.float32(item) for sublist in l for item in sublist]\nmetrics_list_1 = flatten(learner1.recorder.metrics)\nmetrics_list_2 = flatten(learner2.recorder.metrics)\nmetrics_list_3 = flatten(learner3.recorder.metrics)\nmetrics_list_4 = flatten(learner4.recorder.metrics)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"losses_1 = pd.DataFrame({'loss':learner1.recorder.val_losses, 'accuracy': metrics_list_1})\nlosses_2 = pd.DataFrame({'loss':learner2.recorder.val_losses, 'accuracy': metrics_list_2})\nlosses_3 = pd.DataFrame({'loss':learner3.recorder.val_losses, 'accuracy': metrics_list_3})\nlosses_4 = pd.DataFrame({'loss':learner4.recorder.val_losses, 'accuracy': metrics_list_4})\n\nfig, ax = plt.subplots(1,1,figsize=(14, 6))\nax.set(xlabel='Epochs Processed', ylabel='Loss', title='Comparing Validation Losses')\n# losses_1['loss'].sort_index().plot(ax=ax)\nlosses_2['loss'].sort_index().plot(ax=ax)\nlosses_3['loss'].sort_index().plot(ax=ax)\nlosses_4['loss'].sort_index().plot(ax=ax)\n\nax.legend(['Model 2', 'Model 3', 'Model 4'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since Model 1 has validation losses in a completely different range we plot that below seperately.\n\n> But don't think it's not a good model just yet, it gave me scores close to 0.985 in Public LB and as you can see in the forthcoming plot, it has a pretty good validation accuracy as well which looks pretty stable."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,1,figsize=(14, 6))\nax.set(xlabel='Epochs Processed', ylabel='Loss', title='Validation Losses for Model 1')\n\nlosses_1['loss'].sort_index().plot(ax=ax)\nax.legend(['Model 1'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,1,figsize=(14, 6))\nax.set(xlabel='Epochs Processed', ylabel='Loss', title='Comparing Validation Accuracy')\nlosses_1['accuracy'].sort_index().plot(ax=ax)\nlosses_2['accuracy'].sort_index().plot(ax=ax)\nlosses_3['accuracy'].sort_index().plot(ax=ax)\nlosses_4['accuracy'].sort_index().plot(ax=ax)\n\nax.legend(['Model 1', 'Model 2', 'Model 3', 'Model 4'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"preds_1 = (submission_1.label.value_counts()).rename('Model_1')\npreds_2 = (submission_2.label.value_counts()).rename('Model_2')\npreds_3 = (submission_3.label.value_counts()).rename('Model_3')\npreds_4 = (submission_4.label.value_counts()).rename('Model_4')\n\npreds_data = pd.concat([preds_1, preds_2, preds_3, preds_4], axis=1)\npreds_data['category'] = preds_data.index\npreds_data = pd.melt(preds_data, id_vars='category', var_name='model', value_name='preds')\n\nfig = sns.catplot(x='category', y='preds', hue='model',data=preds_data, kind='bar', height=4, aspect=3)\nfig.set(title='Distribution of predictions for each model per category')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Average Ensembled Submission"},{"metadata":{},"cell_type":"markdown","source":"I'm using a simple averaged ensemble and creating a submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"blended_preds = np.round((submission_1['label'] + submission_2['label'] + \n submission_3['label'] + submission_4['label'])/4)\n\nblended_submission = pd.DataFrame({'id': ids, 'label': blended_preds})\nblended_submission['label'] = blended_submission['label'].astype(np.uint8)\nblended_submission.to_csv(\"blended_submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Show your appreciation by giving this kernel an UPVOTE if you liked."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}