{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{},"cell_type":"markdown","source":"Kaggle recently released a new dataset called the Kannada MNIST. This dataset features images of the 10 digits used by the Kannada language (whichprimarily iresides in southwestern India). More information here: https://www.kaggle.com/c/Kannada-MNIST/overview"},{"metadata":{},"cell_type":"markdown","source":"This dataset looks like it will pose some problems that the original MNIST dataset didn't as the numbers in general look a little more complicated. However, this leads to the numbers looking a bit more distinct. One thing to note is that there is not a digit for 0 in this dataset; it is instead 1-10. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import Time\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\nimport torchvision\nfrom torchvision import transforms\nfrom torch.utils.data import DataLoader\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\n%matplotlib inline\n\n# Gets the GPU if there is one, otherwise the cpu\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nprint(device) # Might as well find out! ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Quick function that gets how many out of 'preds' match 'labels'\ndef get_num_correct(preds, labels):\n    return preds.argmax(dim=1).eq(labels).sum().item()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, there is a separate csv for the validation set. For now, I will mix it together with the training set and resplit it (slightly smaller 10,200 to ~7000). "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read in the train set and the test set\ntrain_set = pd.read_csv(\"../input/Kannada-MNIST/train.csv\")\ntest_images = pd.read_csv(\"../input/Kannada-MNIST/test.csv\")\nval_set = pd.read_csv(\"../input/Kannada-MNIST/Dig-MNIST.csv\")\ntrain_set = pd.concat([train_set, val_set], axis=0)\n\n# Don't have to anymore with the separate val set given\ntrain_images, val_images, train_labels, val_labels = train_test_split(train_set.iloc[:, 1:], \n                                                                     train_set.iloc[:, 0], \n                                                                     test_size=0.1)\n\n# Reset indices so the Dataset can find them with __getitem__ easily\ntrain_images.reset_index(drop=True, inplace=True)\nval_images.reset_index(drop=True, inplace=True)\ntrain_labels.reset_index(drop=True, inplace=True)\nval_labels.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_images.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The test set has an ID column, however they're already in order. So we'll dispose of it quickly. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get rid of ID column for the test set, they're already in order\ntest_images = test_images.iloc[:, 1:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's just display a number as a dataframe for fun\npd.set_option('display.max_columns', 28)\npd.DataFrame(train_images.iloc[3, :].to_numpy().reshape(28, 28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_labels[3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some quick data visualization\n# (not a terribly exciting dataset)\n# First 25 images in training set\n# Note: label is above the image\n\nfig, ax = plt.subplots(nrows=5, ncols=5, figsize=(8,8))\nfig.subplots_adjust(hspace=.3)\nfor i in range(5):\n    for j in range(5):\n        ax[i][j].axis('off')\n        ax[i][j].imshow(train_images.iloc[[i+(j*5)], :].to_numpy().astype(np.uint8).reshape(28, 28), cmap='gray')\n        ax[i][j].set_title(train_labels[i+(j*5)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some quick data visualization\n# First 10 images of each class in the training set\n\nfig, ax = plt.subplots(nrows=10, ncols=10, figsize=(10,10))\n\n# I know these for loops look weird, but this way num_i is only computed once for each class\nfor i in range(10): # Column by column\n    num_i = train_images[train_labels == i]\n    ax[0][i].set_title(i)\n    for j in range(10): # Row by row\n        ax[j][i].axis('off')\n        ax[j][i].imshow(num_i.iloc[j, :].to_numpy().astype(np.uint8).reshape(28, 28), cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images definitely look more interesting than the normal MNIST dataset. We'll if that translates to the problem being significantly harder. "},{"metadata":{},"cell_type":"markdown","source":"# Transformations, PyTorch DataSet, and the CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transformations\n\nIMG_SIZE = 28 # size of images in MNIST\n# Also the images only have one color channel\n# So 3D size = (1, 28, 28)\n\n# Transformations for the train\ntrain_trans = transforms.Compose(([\n    transforms.ToPILImage(),\n    transforms.RandomCrop(IMG_SIZE),\n    transforms.RandomAffine(degrees=5, translate=(0.1, 0.1)),\n    transforms.ToTensor(), # divides by 255\n  #  transforms.Normalize((0.5,), (0.5,))\n]))\n\n# Transformations for the validation & test sets\nval_trans = transforms.Compose(([\n    transforms.ToPILImage(),\n    transforms.ToTensor(), # divides by 255\n   # transforms.Normalize((0.1307,), (0.3081,))\n]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Custom class for the new Kannada MNIST dataset on Kaggle\n# Images come in a csv, not as actual images\n# Training set is split before given to this class\n\nclass KannadaDataSet(torch.utils.data.Dataset):\n    # images df, labels df, transforms\n    # uses labels to determine if it needs to return X & y or just X in __getitem__\n    def __init__(self, images, labels, transforms=None):\n        self.X = images\n        self.y = labels\n        self.transforms = transforms\n                    \n    def __len__(self):\n        return len(self.X)\n    \n    def __getitem__(self, i):\n        data = self.X.iloc[i, :] # gets the row\n        # reshape the row into the image size \n        # (numpy arrays have the color channels dim last)\n        data = np.array(data).astype(np.uint8).reshape(28, 28, 1) \n        \n        # perform transforms if there are any\n        if self.transforms:\n            data = self.transforms(data)\n        \n        # if !test_set return the label as well, otherwise don't\n        if self.y is not None: # train/val\n            return (data, self.y[i])\n        else: # test\n            return data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get datasets using the custom Kannada MNIST Dataset for the train, val, and test images\ntrain_set = KannadaDataSet(train_images, train_labels, train_trans)\nval_set = KannadaDataSet(val_images, val_labels, val_trans)\ntest_set = KannadaDataSet(test_images, None, val_trans)\n\nnum_classes = 10 # 1-10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Network(nn.Module):\n    def __init__(self):\n        super(Network, self).__init__()\n        # image starts as (1, 28, 28)\n        # Formula to compute size of image after conv/pool\n        # (size-filter+2*padding / stride) + 1\n        #                      inputs         # of filters    filter size    \n        self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=5, stride=1, padding=2) # conv1\n        self.conv1_bn = nn.BatchNorm2d(num_features=32)\n        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5, stride=1, padding=2) # conv2\n        self.conv2_bn = nn.BatchNorm2d(num_features=64)\n        self.conv3 = nn.Conv2d(in_channels=64, out_channels= 128, kernel_size=3, stride=1, padding=1) # conv3\n        self.conv3_bn = nn.BatchNorm2d(num_features=128)\n        \n        self.fc1 = nn.Linear(in_features=128*6*6, out_features=1024) # linear 1\n        self.fc1_bn = nn.BatchNorm1d(num_features=1024)\n        self.fc2 = nn.Linear(in_features=1024, out_features=512) # linear 2\n        self.fc2_bn = nn.BatchNorm1d(num_features=512)\n        self.fc3 = nn.Linear(in_features=512, out_features=256) # linear 3\n        self.fc3_bn = nn.BatchNorm1d(num_features=256)\n        self.fc4 = nn.Linear(in_features=256, out_features=64) # linear 4\n        self.fc4_bn = nn.BatchNorm1d(num_features=64)\n        self.out = nn.Linear(in_features=64, out_features=10) # output\n    \n    def forward(self, t):\n        t = F.relu(self.conv1_bn(self.conv1(t)))\n        t = F.max_pool2d(t, kernel_size=2, stride=2) # (1, 14, 14)\n        \n        t = F.relu(self.conv2_bn(self.conv2(t)))\n        t = F.max_pool2d(t, kernel_size=2, stride=2) # (1, 7, 7)\n        \n        t = F.relu(self.conv3_bn(self.conv3(t)))\n        t = F.max_pool2d(t, kernel_size=2, stride=1) # (1, 6, 6)\n        \n        t = F.relu(self.fc1_bn(self.fc1(t.reshape(-1, 128*6*6))))\n        t = F.relu(self.fc2_bn(self.fc2(t)))\n        t = F.relu(self.fc3_bn(self.fc3(t)))\n        t = F.relu(self.fc4_bn(self.fc4(t)))\n        t = self.out(t)\n        \n        return t","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Time"},{"metadata":{"trusted":true},"cell_type":"code","source":"lr = 0.001 # initial learning rate\nbatch_size = 100 # batch size\nepochs = 10 # number of epochs to run\n\nnetwork = Network().to(device) # put the model on device (hopefully a GPU!)\ntrain_dl = DataLoader(train_set, batch_size=batch_size, shuffle=True)\nval_dl = DataLoader(val_set, batch_size=batch_size, shuffle=False)\noptimizer = optim.Adam(network.parameters(), lr=lr)\n\nfor epoch in range(epochs):\n    print(\"Epoch: \", epoch+1)\n    epoch_loss = 0\n    epoch_correct = 0\n    network.train() # training mode\n    \n    # lessen the learning rate after 4 epochs (0,...,3)\n    if epoch == 4:\n        print(\" decreasing lr\")\n        optimizer = optim.Adam(network.parameters(), lr=0.00001)\n    \n    if epoch==10:\n        print(\" decreasing lr again\")\n        optimizer = optim.Adam(network.parameters(), lr=0.0000000000001)\n    \n    for images, labels in train_dl:\n        X, y = images.to(device), labels.to(device) # put X & y on device\n        y_ = network(X) # get predictions\n        \n        optimizer.zero_grad() # zeros out the gradients\n        loss = F.cross_entropy(y_, y) # computes the loss\n        loss.backward() # computes the gradients\n        optimizer.step() # updates weights\n        \n        epoch_loss += loss.item() * batch_size\n        epoch_correct += get_num_correct(y_, y)\n        \n    # Evaluation with the validation set\n    network.eval() # eval mode\n    val_loss = 0\n    val_correct = 0\n    with torch.no_grad():\n        for images, labels in val_dl:\n            X, y = images.to(device), labels.to(device) # to device\n            \n            preds = network(X) # get predictions\n            loss = F.cross_entropy(preds, y) # calculate the loss\n            \n            val_correct += get_num_correct(preds, y)\n            val_loss = loss.item() * batch_size\n    # Print the loss and accuracy for the validation set\n    print(\" Val Loss: \", val_loss)\n    print(\" Val Acc: \", (val_correct/len(val_images))*100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation & Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use the validation set to make a confusion matrix\n\nnetwork.eval() # good habit I suppose\npredictions = torch.LongTensor().to(device) # Tensor for all predictions\n\n# Goes through the val set\nfor images, _ in val_dl:\n    preds = network(images.to(device))\n    predictions = torch.cat((predictions, preds.argmax(dim=1)), dim=0)\n\n# Make the confusion matrix\ncmt = torch.zeros(num_classes, num_classes, dtype=torch.int32)\nfor i in range(len(val_labels)):\n    cmt[val_labels[i], predictions[i]] += 1\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Doesn't look fancy but it works\n# Actual class down the column (0-9), predicted class across the rows (0-9)\ncmt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Time to get the network's predictions on the test set\n# Put the test set in a DataLoader\ntest_dl = DataLoader(test_set, batch_size=batch_size, shuffle=False)\n\nnetwork.eval() # Safety first\npredictions = torch.LongTensor().to(device) # Tensor for all predictions\n\n# Go through the test set, saving the predictions in... 'predictions'\nfor images in test_dl:\n    preds = network(images.to(device))\n    predictions = torch.cat((predictions, preds.argmax(dim=1)), dim=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read in the sample submission\nsubmission = pd.read_csv(\"../input/Kannada-MNIST/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Change the label column to our predictions \n# Have to make sure the predictions Tensor is on the cpu\nsubmission['label'] = predictions.cpu().numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Write the dataframe to a new csv, not including the index\nsubmission.to_csv(\"predictions.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Save the model (if you really want)\n# torch.save(network, \"Model/kannada_network\") ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"}},"nbformat":4,"nbformat_minor":1}