{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Library Imports","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\n\nimport torch\nfrom torch import nn, optim\nfrom torch.utils.data import DataLoader, Dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Data Loading and Preliminary Processing","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Load train data\ndata = pd.read_csv('/kaggle/input/Kannada-MNIST/train.csv')\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Get simple description of the data set\ndata.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the rows of our dataset are composed of 784 features, equivalent to 28x28 pixels images. Additionally, our target, *label*, assumes 10 different values. Let us now separate features and target in order to proceed the data processing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x_data = data.drop(columns = 'label')\ny_data = data['label']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First of all, we should take a look at our data as more than just a bunch of pixels, to get a sense of what we are working with. We can do that simply by printing some of the images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(2, 2)\nfig.tight_layout(pad = 3.0)\nfor i in range(2):\n    for j in range(2):\n        img = np.random.randint(0, len(x_data))\n        axs[i,j].imshow(x_data.loc[img].values.reshape(28, 28))\n        axs[i,j].set_title(y_data[img])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Feature Engineering","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We now take a look at our dataset and try to find forms of improving how it is going o help us classificate digits. From the previous section, two aspects of this data have grabbed our attention:\n\n* The values for each pixel basically range from 0 to 255; \n* The image data is presented in an one-dimensional format.\n\nThe first topic is fairly common. It is widely known that we need to scale our features in order to achieve better and faster models. Additionally, the second point is a very interesting one. It is possible to treat each image as 1D array of features and to obtain fairly accurarte results with this approach, as seen on previous versions of this notebook. However, if we want to take advantage of the CNN capabilities, it is better to represente the images in their original two-dimensional format. Hence, our first step is to apply this transformation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define function to scale the dataset\ndef scale_data(data, scaler, opt = 0): #opt: 0 -> apply fit_trandform, 1 -> apply only transform\n    if opt == 0:\n        return scaler.fit_transform(data)\n    else:\n        return scaler.transform(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Scale train dataset\nmy_scaler = MinMaxScaler()\nscaled_x_data = scale_data(x_data, my_scaler, 0)\n\nprint('Original dataset: \\n', x_data.values[0:3], '\\n\\n')\nprint('Scaled dataset: \\n', scaled_x_data[0:3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Convert data entries from 1D to 2D\nscaled_x_data = np.asarray([x.reshape(28, 28) for x in scaled_x_data])\nprint('Sample converted image: \\n')\nprint(scaled_x_data[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This concludes the feature engineering section of this notebook. We now proceed to constructing and training our predictive model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 4. Data Modeling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this section we are using the PyTorch library in order to construct a MLP classification model. First we are going to construct the neural network topology and then try to optimize it while training it with our data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.1. Data Preparation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Selecting device\nif torch.cuda.is_available():\n    device = torch.device('cuda')\nelse:\n    device = torch.device('cpu')\n\nprint('Device: ', device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When using the *torch* package, we are working with its on data type: tensors. Then, the first thing we are going to do is to adjust our data to this format. Additionally, it is important to take advantage of the tools provided by this library. One of them is the *DataLoader* module, which allows for better management of the data packages on the training procedure. Here, we are also creating a dataloader for our dataset.\n\nAdditionally, the *test.csv* file contains the data using in the final scoring of the model. So, if we want to really understand how our model is going to behave on production, we have to separate our data set into three parts as follows:\n\n* Train data: 75%;\n* Test data: 25%.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Split data into train, test and validation\nx_train, x_test, y_train, y_test = train_test_split(scaled_x_data, y_data,\n                                                    test_size = 0.25,\n                                                    stratify = y_data)\n\nprint('Train size: %d \\n Test size: %d' %(len(x_train), len(x_test)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define model hyperparameters\nparam = {\n    'num_jobs': 2,\n    'batch_size': 128,\n    'num_epochs': 100\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define class to load the dataset as a tensor\nclass MnistData(Dataset):\n    \n    def __init__(self, x_data, y_data):\n        self.x_data = x_data\n        self.y_data = y_data\n        \n    def __getitem__(self, index):\n        sample = torch.from_numpy(self.x_data[index].astype(np.float32).reshape(1, 28, 28))\n        target = torch.from_numpy(self.y_data[index].astype(np.float32))\n        return (sample, target)\n    \n    def __len__(self):\n        return len(self.x_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Load data as tensors\ntrain_data = MnistData(x_train, y_train.values.reshape(len(y_train), 1))\ntest_data = MnistData(x_test, y_test.values.reshape(len(y_test), 1))\n#validation_data = MnistData(x_validation, y_validation.values.reshape(len(y_validation), 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create DataLoader\ntrain_loader = DataLoader(train_data,\n                          batch_size = param['batch_size'],\n                          shuffle = True,\n                          num_workers = param['num_jobs'])\ntest_loader = DataLoader(test_data,\n                         batch_size = param['batch_size'],\n                         shuffle = True,\n                         num_workers = param['num_jobs'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2. Neural Network Topology Construction ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MnistClassification(nn.Module):\n    \n    def __init__(self, input_dim, output_dim):\n        #Inicializar classe pai\n        super(MnistClassification, self).__init__()\n        self.input_dim = input_dim\n        \n        #Preprocessing layers\n        self.conv_01 = nn.Conv2d(in_channels = 1, out_channels = 16, kernel_size = 6, stride = 1, padding = 1)\n        self.conv_bn_01 = nn.BatchNorm2d(num_features = 16)\n        self.pool_01 = nn.MaxPool2d(kernel_size = 3, stride = 1)\n        \n        self.conv_02 = nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 5, stride = 1, padding = 1)\n        self.conv_bn_02 = nn.BatchNorm2d(num_features = 32)\n        self.pool_02 = nn.MaxPool2d(kernel_size = 3, stride = 1)\n        \n        self.conv_03 = nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 5, stride = 1, padding = 1)\n        self.conv_bn_03 = nn.BatchNorm2d(num_features = 64)\n        self.pool_03 = nn.MaxPool2d(kernel_size = 3, stride = 2)\n        \n        #Linear an normalization layers\n        self.norm_01 = nn.BatchNorm1d(num_features = 2048)\n        self.layer_01 = nn.Linear(in_features = 4096, out_features = 2048)\n        self.norm_02 = nn.BatchNorm1d(num_features = 1024)\n        self.layer_02 = nn.Linear(in_features = 2048, out_features = 1024)\n        self.norm_03 = nn.BatchNorm1d(num_features = 512)\n        self.layer_03 = nn.Linear(in_features = 1024, out_features = 512)\n        self.norm_04 = nn.BatchNorm1d(num_features = 256)\n        self.layer_04 = nn.Linear(in_features = 512, out_features = 256)\n        self.norm_05 = nn.BatchNorm1d(num_features = 128)\n        self.layer_05 = nn.Linear(in_features = 256, out_features = 128)\n        self.norm_06 = nn.BatchNorm1d(num_features = 64)\n        self.layer_06 = nn.Linear(in_features = 128, out_features = 64)\n        self.output_layer = nn.Linear(in_features = 64, out_features = output_dim)\n        \n        #Activation and dropout layers\n        self.dropout = nn.Dropout(0.2)\n        self.relu = nn.ReLU()\n        self.softmax = nn.Softmax(dim = 1)\n            \n    def forward(self, x_data):\n        \n        x_data = x_data.to(device)\n        response = self.relu(self.conv_bn_01(self.conv_01(x_data)))\n        response = self.pool_01(response)\n        response = self.relu(self.conv_bn_02(self.conv_02(response)))\n        response = self.pool_02(response)\n        response = self.relu(self.conv_bn_03(self.conv_03(response)))\n        response = self.pool_03(response)\n        \n        response = self.dropout(self.relu(self.norm_01(self.layer_01(response.reshape(len(x_data),4096)))))\n        response = self.dropout(self.relu(self.norm_02(self.layer_02(response))))\n        response = self.dropout(self.relu(self.norm_03(self.layer_03(response))))\n        response = self.dropout(self.relu(self.norm_04(self.layer_04(response))))\n        response = self.dropout(self.relu(self.norm_05(self.layer_05(response))))\n        response = self.dropout(self.relu(self.norm_06(self.layer_06(response))))\n        \n        response = self.softmax(self.output_layer(response))\n        \n        return response","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create MLP instance\nparam['input_dim'] = x_data.shape[1]\nparam['output_dim'] = len(y_data.unique())\nparam['num_layers'] = 3\n\nmodel = MnistClassification(input_dim = param['input_dim'],\n                            output_dim = param['output_dim']).to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define loss function and optimizer\nparam['learning_rate'] = 1e-4\nparam['weight_decay'] = 5e-3\n\nloss_function = nn.CrossEntropyLoss().to(device)\noptimizer = optim.Adam(params = model.parameters(),\n                       lr = param['learning_rate'],\n                       weight_decay = param['weight_decay'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3. Training Setup Construction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define training model function\ndef train_model(estimator, train_data, epoch):\n    \n    #Toogle training mode\n    model.train()\n    \n    epoch_error = []\n    epoch_accuracy = []\n    \n    for batch in train_data:\n        \n        #Update learning rate \n        optimizer = optim.Adam(params = model.parameters(),\n                               lr = param['learning_rate'] * (1/(10**((epoch/10) + 1))),\n                               weight_decay = param['weight_decay'])\n\n        #Forward process\n        x_batch, y_batch = batch\n        x_batch = x_batch.to(device)\n        y_batch = y_batch.to(device)\n        batch_response = model(x_batch)\n                \n        #Compute error\n        batch_error = loss_function(batch_response, torch.squeeze(y_batch, 1).type(torch.LongTensor).to(device))\n        epoch_error.append(batch_error.cpu().data)\n        \n        batch_accuracy = accuracy_score(torch.squeeze(y_batch, 1).type(torch.LongTensor).cpu().numpy(), \n                                        np.argmax(batch_response.detach().cpu().numpy(), 1))\n        epoch_accuracy.append(batch_accuracy)\n        \n        #Backward process\n        optimizer.zero_grad()\n        batch_error.backward()\n        optimizer.step()\n        \n    epoch_error = np.asarray(epoch_error)\n    epoch_accuracy = np.asarray(epoch_accuracy)\n    \n    print('Epoch %d TRAIN error: %.4f +/- %.4f / accuracy: %.4f' %(epoch+1, epoch_error.mean(), \n                                                                   epoch_error.std(), epoch_accuracy.mean()))\n    \n    return [epoch_error.mean(), epoch_accuracy.mean()] ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"#Define testing model function\ndef test_model(estimator, test_data, epoch):\n    \n    #Toogle training mode\n    model.eval()\n    with torch.no_grad():\n    \n        epoch_error = []\n        epoch_accuracy = []\n        for batch in test_data:\n            #Forward process\n            x_batch, y_batch = batch\n            x_batch = x_batch.to(device)\n            y_batch = y_batch.to(device)\n            batch_response = model(x_batch)\n\n            #Compute error\n            batch_error = loss_function(batch_response, torch.squeeze(y_batch, 1).type(torch.LongTensor).to(device))\n            epoch_error.append(batch_error.cpu().data)\n            \n            batch_accuracy = accuracy_score(torch.squeeze(y_batch, 1).type(torch.LongTensor).cpu().numpy(), \n                                            np.argmax(batch_response.detach().cpu().numpy(), 1))\n            epoch_accuracy.append(batch_accuracy)\n\n        epoch_error = np.asarray(epoch_error)\n        epoch_accuracy = np.asarray(epoch_accuracy)\n        \n        print('Epoch %d TEST error: %.4f +/- %.4f / accuracy: %.4f' %(epoch+1, epoch_error.mean(), \n                                                                      epoch_error.std(), epoch_accuracy.mean()))\n\n        return [epoch_error.mean(), epoch_accuracy.mean()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4. Train Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train model\nparam['num_epochs'] = 50\n\ntrain_error = []\ntest_error = []\n\nfor epoch in range(param['num_epochs']):\n    train_error.append(train_model(model, train_loader, epoch))\n    test_error.append(test_model(model, test_loader, epoch))\n    print('-----------------------------------------------------------')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot loss function profile\nplt.plot(list(range(param['num_epochs'])), [x[0] for x in train_error])\nplt.plot(list(range(param['num_epochs'])), [x[0] for x in test_error])\nplt.legend(['Train', 'Test'])\nplt.xlabel('Epochs')\nplt.ylabel('Loss')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot accuracy profile\nplt.plot(list(range(param['num_epochs'])), [x[1] for x in train_error])\nplt.plot(list(range(param['num_epochs'])), [x[1] for x in test_error])\nplt.legend(['Train', 'Test'])\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.5. Predict Test Data Target","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Load test data\ntest_data = pd.read_csv('/kaggle/input/Kannada-MNIST/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Adjust test data\ntest_data_id = test_data['id']\n\nscaled_test_data = scale_data(test_data.drop(columns = 'id'), my_scaler, 1)\nscaled_test_data = np.asarray([x.reshape(1, 28, 28) for x in scaled_test_data])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = np.argmax(model.forward(torch.from_numpy(scaled_test_data.astype(np.float32))).detach().cpu().numpy(), 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_data = pd.DataFrame(data = test_data_id.values, columns = ['id'])\nsubmission_data['label'] = pred\nprint(submission_data.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_data.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}