{"cells":[{"metadata":{"id":"Jme-NJAJYvnW","outputId":"1ed720e9-d7cd-4a68-a5c7-9f52281e6442"},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/catalyst-team/catalyst-pics/master/pics/catalyst_logo.png)"},{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction\n\nIn this Kernel I will develop my take on the classic competition of Digit Recognizer using the nice Python module called [Catalyst](https://github.com/catalyst-team/catalyst). It's good because you can write modular and reproducible code to future experiments.<br>\nNow import all required modules."},{"metadata":{"_kg_hide-input":true,"id":"FYQC9cqwY9Xy","outputId":"7809cf7f-9fcf-465c-8b90-e4836faa629e","trusted":true},"cell_type":"code","source":"# Standard Modules\nimport numpy as np\nimport pandas as pd\n# Graphic Plots\nimport matplotlib.pyplot as plt\n%matplotlib inline\nplt.style.use('ggplot')\nimport seaborn as sns\nsns.set_style('darkgrid')\n# Extra Tools\nimport os\nimport warnings\nimport math\nimport random\nfrom sklearn.model_selection import train_test_split\nwarnings.filterwarnings('ignore')\n# PyTorch and Catalyst\nimport collections\nimport torch\nimport torchvision\nimport torchvision.transforms as transforms\nfrom torch.utils.data import DataLoader, Dataset, TensorDataset\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom catalyst.dl import utils\nfrom catalyst.dl.runner import SupervisedRunner\nfrom catalyst.dl.callbacks import AccuracyCallback\nfrom catalyst.utils import set_global_seed, prepare_cudnn\n# For better plot visualization\n%config InlineBackend.figure_format = 'retina'\n# Enviroment PATH\nPATH = '../input/digit-recognizer/'\nprint(os.listdir(PATH))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And set the enviroment settings for reprodutibility."},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 42\nos.environ['CUDA_VISIBLE_DEVICES'] = \"0\"\nset_global_seed(SEED)                       \nprepare_cudnn(deterministic=True)           ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's use [NVIDIA Apex](https://github.com/NVIDIA/apex) to work with [Mixed Precision Training](https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html), also called Automatic Mixed Precision (AMP).\nSet it to use FP16 and install Apex."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"FP16_PARAMS = None","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%writefile setup.sh\ngit clone https://github.com/NVIDIA/apex\ncd apex\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\nrm -rf ./apex","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\n!sh setup.sh\nFP16_PARAMS = dict(opt_level=\"O1\") ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Load Datasets and Explore the Images\n\nLoad the datasets in Pandas Dataframes and take a look on the Train Set to check the dataset format."},{"metadata":{"_kg_hide-input":true,"id":"QBegVYWtwTRn","trusted":true},"cell_type":"code","source":"train_raw = pd.read_csv(PATH+'train.csv')\ntest_raw = pd.read_csv(PATH+'test.csv')\nsubmission = pd.read_csv(PATH+'sample_submission.csv')\ntrain_raw.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although this is an image competition, is provided tabular data, where each feature column represents an image pixel. <br>\nLet's see how many samples (images) it have."},{"metadata":{"_kg_hide-input":true,"id":"iRtK3DzM2eBz","outputId":"caed51e5-676c-4143-d4a1-67990bb3017f","trusted":true},"cell_type":"code","source":"print(f'Train Samples: {train_raw.shape[0]}')\nprint(f'Test Samples: {test_raw.shape[0]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check the label distribution to find any disbalance."},{"metadata":{"_kg_hide-input":true,"id":"FlN2yv66273H","outputId":"758822f1-3325-4dce-af3e-0e23b6ff2d39","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.countplot(x='label', data=train_raw)\nplt.title('Digit Label Distribution');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Time to see how the samples look like images. For this, they will reshaped for images with 28x28 pixels, because the product of this results in 784, the number of columns/pixels in the dataset (except label column). "},{"metadata":{"_kg_hide-input":true,"id":"r3FT-bHc1GWB","outputId":"1cd9b37f-8e82-4c45-daad-15c643125f70","trusted":true},"cell_type":"code","source":"label_index = []\nfor i in range(10):\n  label_index.append(train_raw.index[train_raw['label'] == i][0])\n\nfig = plt.figure(figsize=(10,6))\n\nidx = 1\n\nfor i in label_index:\n  plt.subplot(2, 5, idx)\n  plt.imshow(train_raw.iloc[i,1:].values.reshape(28, 28))\n  plt.axis('off')\n  plt.suptitle('MNIST Digits')\n  plt.title(str(train_raw['label'].loc[i]))\n  idx = idx+1\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Data Preparation for Catalyst\n\nWe start splitting the train dataframe in train and validations sets."},{"metadata":{"_kg_hide-input":true,"id":"zToQ4BwoyEMX","trusted":true},"cell_type":"code","source":"x_train, x_val, y_train, y_val = train_test_split(train_raw.iloc[:, 1:], \n                                                  train_raw['label'], \n                                                  test_size=0.3, \n                                                  random_state=42)\n\nx_train.reset_index(drop=True, inplace=True)\ny_train.reset_index(drop=True, inplace=True)\nx_val.reset_index(drop=True, inplace=True)\ny_val.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we have a Pandas Dataframe with pixel values, it's needed to transform in tensors, they will be input of the Network. <br>\nThe pixels are normalized with the transform.\nLet's just write how the transform will work."},{"metadata":{"_kg_hide-input":true,"id":"N8szhSG8K876","trusted":true},"cell_type":"code","source":"data_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,), (0.5,))])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the Catalyst, we have to write a class called dataset to prepare the data for the network. <br>\nOn the Dataset class is defined the input, label and any process, like tranformations, in the data."},{"metadata":{"_kg_hide-input":true,"id":"Vr1Yz5XAC132","trusted":true},"cell_type":"code","source":"class MnistDataset(Dataset):\n# Get input, labels and transforms\n  def __init__(self, images, labels=None, transform=None):\n    self.x = images\n    self.y = labels\n    self.transform = transform\n# Get data length\n  def __len__(self):\n    return len(self.x)\n# Get data index\n  def __getitem__(self, i):\n    data = np.array(self.x.iloc[i, :], dtype='uint8').reshape([28, 28, 1])\n\n    if self.transform is not None:\n      data = self.transform(data)\n    if self.y is not None:\n      target = self.y.iloc[i]\n      return data, target\n    else:\n      return data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we define the parameters of the experiment, like Batch Size, Iterations, etc."},{"metadata":{"_kg_hide-input":false,"id":"yRi8KADSnXjN","trusted":true},"cell_type":"code","source":"num_class = 10\nbatch_size = 32\nnum_workers = 4\nn_iters = 5000\nn_epochs = int(n_iters/(len(x_train)/batch_size))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's a personal choice, I create fake labels for the test set for avoid any error on Catalyst."},{"metadata":{"_kg_hide-input":false,"id":"75Hbwo-eVTC5","trusted":true},"cell_type":"code","source":"fake_labels = np.zeros(test_raw.shape[0])\nfake_labels = pd.Series(fake_labels)\nfake_labels = fake_labels.astype(int)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Time to convert the data on Catalyst Dataset."},{"metadata":{"_kg_hide-input":false,"id":"YHOkWYXNMEn4","trusted":true},"cell_type":"code","source":"train = MnistDataset(x_train, y_train, transform=data_transform)\nval = MnistDataset(x_val, y_val, transform=data_transform)\ntest = MnistDataset(test_raw, fake_labels, transform=data_transform)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And get these datasets and put they on Loaders. <br>\nLoaders in Catalyst provide parameters like batch size together the data."},{"metadata":{"_kg_hide-input":true,"id":"qERBENjSy_aO","trusted":true},"cell_type":"code","source":"train_loader = DataLoader(train, batch_size=batch_size, shuffle=False, num_workers=num_workers)\nval_loader = DataLoader(val, batch_size=batch_size, shuffle=False, num_workers=num_workers)\ntest_loader = DataLoader(test, batch_size=batch_size, shuffle=False, num_workers=num_workers)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Put the train and validation loaders in dictionares to feed the network."},{"metadata":{"_kg_hide-input":true,"id":"MBCs3SqJNVwe","trusted":true},"cell_type":"code","source":"loaders = collections.OrderedDict()\nloaders[\"train\"] = train_loader\nloaders[\"valid\"] = val_loader","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have the advantage of use GPU, let's check for any available and save it for use on Catalyst."},{"metadata":{"_kg_hide-input":true,"id":"SAlSzLgI8CR2","outputId":"04b70d0b-e60a-4f82-b80e-2f3665a1c507","trusted":true},"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Model\n\nTime to build the model! This is a nice part on Catalyst, it's modular, is easy to change any part like the network. <br>\nIf you see on this competition, you will find a lot of interesting model networks, I think that the model that is provided [here](https://www.kaggle.com/juiyangchang/cnn-with-pytorch-0-995-accuracy) is good enough."},{"metadata":{"id":"JE3YsTaV9htk","trusted":true},"cell_type":"code","source":"class Net(nn.Module):    \n    def __init__(self):\n        super(Net, self).__init__()\n          \n        self.features = nn.Sequential(\n            nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),\n            nn.BatchNorm2d(32),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(32, 32, kernel_size=3, stride=1, padding=1),\n            nn.BatchNorm2d(32),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(kernel_size=2, stride=2),\n            nn.Conv2d(32, 64, kernel_size=3, padding=1),\n            nn.BatchNorm2d(64),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(64, 64, kernel_size=3, padding=1),\n            nn.BatchNorm2d(64),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(kernel_size=2, stride=2)\n        )\n          \n        self.classifier = nn.Sequential(\n            nn.Dropout(p = 0.5),\n            nn.Linear(64 * 7 * 7, 512),\n            nn.BatchNorm1d(512),\n            nn.ReLU(inplace=True),\n            nn.Dropout(p = 0.5),\n            nn.Linear(512, 512),\n            nn.BatchNorm1d(512),\n            nn.ReLU(inplace=True),\n            nn.Dropout(p = 0.5),\n            nn.Linear(512, 10),\n        )\n          \n        for m in self.features.children():\n            if isinstance(m, nn.Conv2d):\n                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels\n                m.weight.data.normal_(0, math.sqrt(2. / n))\n            elif isinstance(m, nn.BatchNorm2d):\n                m.weight.data.fill_(1)\n                m.bias.data.zero_()\n        \n        for m in self.classifier.children():\n            if isinstance(m, nn.Linear):\n                nn.init.xavier_uniform(m.weight)\n            elif isinstance(m, nn.BatchNorm1d):\n                m.weight.data.fill_(1)\n                m.bias.data.zero_()\n                \n\n    def forward(self, x):\n        x = self.features(x)\n        x = x.view(x.size(0), -1)\n        x = self.classifier(x)\n        \n        return x    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we assemble each component created before and run the expirement on Catalyst. <br>\nLet's start training the network."},{"metadata":{"_kg_hide-input":false,"id":"ANJTmLtc_Yvg","outputId":"9d67fe03-37be-4d1c-d2b5-148a5b6bcb43","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n# Experiment\nlogdir = './logs/mnist'\n\nmodel = Net().to(device)\ncriterion = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model.parameters())\n\nrunner = SupervisedRunner()\n\nrunner.train(\n    model=model,\n    criterion=criterion,\n    optimizer=optimizer,\n    loaders=loaders,\n    callbacks=[\n        AccuracyCallback(num_classes=num_class),\n    ],\n    logdir=logdir,\n    num_epochs=n_epochs,\n    fp16=FP16_PARAMS,\n    verbose=False\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's possible to check the saved models on the `logdir` directory defined early."},{"metadata":{"_kg_hide-input":true,"id":"MsmCzDB9rs3J","outputId":"22958127-4803-476b-94ad-4a49983da58d","trusted":true},"cell_type":"code","source":"!ls ./logs/mnist/checkpoints","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Predictions\n\nOk, let's run our prediction. Remember the fake labels? I used here to don't get errors on predict."},{"metadata":{"_kg_hide-input":false,"id":"u-rjPyEJrEBr","outputId":"639ac2e9-74d4-448e-8586-b90cb5f2a52b","trusted":true},"cell_type":"code","source":"runner_out = runner.predict_loader(model=model, loader=test_loader, resume=f\"{logdir}/checkpoints/best.pth\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And save the results on submission file for the competition."},{"metadata":{"_kg_hide-input":true,"id":"eMtbpoLDoX5y","trusted":true},"cell_type":"code","source":"_,results = torch.topk(torch.from_numpy(runner_out),1)\nsubmission.columns \nsubmission_final = pd.DataFrame({'ImageId':submission.ImageId,'Label':np.squeeze(results.numpy())},columns=['ImageId','Label'])\nsubmission_final.to_csv('submission_final.csv',index=False)\nprint(\"Predictions made and Submission file saved!\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Conclusion\n\nCatalyst is awesome module to work with Deep Learning, it is based on the great PyTorch, but without use any complicated code lines and is friendly user. <br>\nAnother advantage, due the PyTorch, is that easy to change and classification and segmentation tasks. Catalyst is really worth to give a try!\nAbout the Mixed Precision Training, it is so good, because the task gain more power to deal with big datasets, like images."}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"MNIST_Catalyst","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":4}