{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30673,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Digit recognition in MNIST Dataset is considered as a classic example of deep learning and computer vision. It is a dataset of handwritten digits. I have already make a model for this dataset with [Keras](https://keras.io/guides/) which can make predictions with 99.9% accuracy. Here is the link of the Notebook [here](https://www.kaggle.com/code/bibekbhusal0/digit-recognizer-with-keras-99-978-accurate), do check it out and give a upvote if you learn something new from it. Main purpose of these projects is to learn abour deep learning and computer vision. In this notebook I will make a model for digit recognition with [PyTorch](https://pytorch.org/docs). This is part of my learning journey on deep learning. I want to make this notebook public so that anyone can learn from it.","metadata":{}},{"cell_type":"markdown","source":"# 1.Importing Libraries ","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns \nimport os \nfrom sklearn.metrics import confusion_matrix\nimport warnings \nwarnings.filterwarnings(\"ignore\")\n\nimport torch \nfrom torch import nn \nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision import datasets, transforms\nfrom torchvision.utils import make_grid, save_image\nimport torchvision as tv \nfrom torchvision.datasets import MNIST","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:20.705302Z","iopub.execute_input":"2024-04-27T03:54:20.705894Z","iopub.status.idle":"2024-04-27T03:54:29.661303Z","shell.execute_reply.started":"2024-04-27T03:54:20.705860Z","shell.execute_reply":"2024-04-27T03:54:29.660340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2.Importing Dataset ","metadata":{}},{"cell_type":"code","source":"validation  = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest= pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nsample_submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:29.663067Z","iopub.execute_input":"2024-04-27T03:54:29.663496Z","iopub.status.idle":"2024-04-27T03:54:35.311639Z","shell.execute_reply.started":"2024-04-27T03:54:29.663471Z","shell.execute_reply":"2024-04-27T03:54:35.310825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## let's take a look on some images on dataset","metadata":{}},{"cell_type":"code","source":"rows = 5\nfig, axes = plt.subplots(rows, 10, figsize=(10 ,rows))\n\nfor i in range (10 ):\n    ds  = validation[validation.label == i]\n    for j in range(rows):\n        ax = axes[j,i]\n        ax.imshow(ds.iloc[j,1:].values.reshape(28,28), cmap='gray')\n        ax.axis('off')\n        \nplt.show() ","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:35.312779Z","iopub.execute_input":"2024-04-27T03:54:35.313067Z","iopub.status.idle":"2024-04-27T03:54:37.225088Z","shell.execute_reply.started":"2024-04-27T03:54:35.313042Z","shell.execute_reply":"2024-04-27T03:54:37.224173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Preprocessing of Data\n- Loading extra data from PyTorch\n- Construction of data loader \n- Normalizing the data\n- Converting data into tensor ","metadata":{}},{"cell_type":"code","source":"\ntransform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Lambda(lambda x: x / 255)\n])\n\n\nx1 = MNIST(\"/data\", download = True, train=True, transform=transform)\nx2 = MNIST(\"/data\", train = False, transform=transform)\ntrain = torch.utils.data.ConcatDataset([x1, x2])","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:37.227141Z","iopub.execute_input":"2024-04-27T03:54:37.227436Z","iopub.status.idle":"2024-04-27T03:54:41.766308Z","shell.execute_reply.started":"2024-04-27T03:54:37.227411Z","shell.execute_reply":"2024-04-27T03:54:41.765316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are converting data into `tensor` first and than `normalize` it by `dividing by 255`.\n\n`torch.cuda.is_available` function checks if a GPU is available. We are using GPU when available. Later in code we are sinding the model and each data to the GPU. ","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:41.767443Z","iopub.execute_input":"2024-04-27T03:54:41.767767Z","iopub.status.idle":"2024-04-27T03:54:41.793310Z","shell.execute_reply.started":"2024-04-27T03:54:41.767741Z","shell.execute_reply":"2024-04-27T03:54:41.792423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our data is in pandas dataframe we need to convert it to tensor and also normalize it to fit the model. I think best way to do it by using [PyTorch Dataset](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html). I am also making batch data loader.","metadata":{}},{"cell_type":"code","source":"class TensorFromDataset(Dataset):\n    def __init__ (self, dataFrame, transform = transforms.ToTensor()):\n        self.dataFrame = dataFrame\n        \n    def __len__ (self):\n        return len(self.dataFrame)\n    \n    def __getitem__ (self, index):\n        label = self.dataFrame.iloc[index, 0]\n        image = self.dataFrame.iloc[index, 1:].values.astype(np.uint8).reshape(28, 28)\n        label = torch.tensor(label)\n        if transform is not None:\n            image = transform(image)\n        return (image, label)\n\nvalidation_data = TensorFromDataset(validation, transform = transform)\ntest_data = transform(test.values.astype(np.uint8))","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:41.794777Z","iopub.execute_input":"2024-04-27T03:54:41.795460Z","iopub.status.idle":"2024-04-27T03:54:41.948068Z","shell.execute_reply.started":"2024-04-27T03:54:41.795426Z","shell.execute_reply":"2024-04-27T03:54:41.946991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 32 ** 2\n\ntrain_loader = DataLoader(train, batch_size = batch_size, shuffle = True)\nvalidation_loader = DataLoader(validation_data, batch_size = batch_size, shuffle = True)\ntest_loader = DataLoader(test_data, batch_size = batch_size, shuffle = True)","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:41.949285Z","iopub.execute_input":"2024-04-27T03:54:41.949569Z","iopub.status.idle":"2024-04-27T03:54:41.955636Z","shell.execute_reply.started":"2024-04-27T03:54:41.949544Z","shell.execute_reply":"2024-04-27T03:54:41.954649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Making Model","metadata":{}},{"cell_type":"code","source":"class Model(nn.Module):\n    def __init__(self):\n        super(Model, self).__init__()\n        self.main =  nn.Sequential(\n            nn.Conv2d(1, 32, kernel_size  = 3, stride = 1, padding = 1),\n            nn.ReLU(),\n            nn.BatchNorm2d(32),\n            nn.Conv2d(32, 64, kernel_size  = 3, stride = 1, padding = 1),\n            nn.MaxPool2d(2),\n            nn.Conv2d(64, 128, kernel_size  = 3, stride = 1, padding = 1),\n            nn.MaxPool2d(2),\n            nn.Flatten(),\n            nn.Linear(128 * 7 * 7, 128 * 3),\n            nn.ReLU(),\n            nn.Linear(128 * 3 , 10),\n            nn.Softmax()\n        )\n        self.to(device)\n        self.loss = nn.CrossEntropyLoss()\n        self.optimizer = torch.optim.Adam(self.parameters(), lr = 0.0001)\n        self.validation_step = self.training_step\n        self.history = [] \n        self.apply(self.init_weights)\n    \n    def init_weights(self, m):\n        if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):\n            torch.nn.init.xavier_uniform_(m.weight)\n            \n    def forward(self, x):\n        return self.main(x)\n    \n    def accuracy(self, outputs, labels):\n        _, preds = torch.max(outputs, dim = 1)\n        return torch.tensor(torch.sum(preds == labels).item() / len(preds))\n    \n    def training_step(self, batch):\n        images, labels = batch\n        images = images.to(device)\n        labels = labels.to(device)\n        out = self(images)\n        loss = self.loss(out, labels)\n        accuracy = self.accuracy(out, labels) \n        \n        self.optimizer.zero_grad()\n        loss.backward()\n        self.optimizer.step()\n        \n        return loss.item(), accuracy.item() \n    \n    def validation_step(self, batch):\n        images, labels = batch\n        images = images.to(device)\n        labels = labels.to(device)\n        with torch.no_grad():\n            out = self(images)\n            loss = self.loss(out, labels)\n            accuracy = self.accuracy(out, labels)\n        \n            return loss.item() , accuracy.item()\n    \n    def training_epoch_end(self, t_accuracy, t_loss, v_accuracy, v_loss):\n        epoch = len(self.history)\n        self.history.append({\n            'epoch' : epoch + 1,\n            'training_accuracy' : t_accuracy,\n            'training_loss' : t_loss,\n            'validation_accuracy' : v_accuracy,\n            'validation_loss' : v_loss,\n        })\n        print (self.history[-1])\n        \n        \n    def plot_history(self):\n        df = pd.DataFrame(self.history)\n        fig, axes  = plt.subplots(1, 2, figsize = (15, 5))\n        losses = df[['training_loss', 'validation_loss']]\n        accuracies = df[['training_accuracy', 'validation_accuracy']]\n        losses.plot(ax = axes[0])\n        accuracies.plot(ax = axes[1])\n        axes[0].set_title('Loss over epochs')\n        axes[1].set_title('Accuracy over epochs')\n        axes[0].set_xlabel('Epoch')\n        axes[1].set_xlabel('Epoch')\n        axes[0].set_ylabel('Loss')\n        axes[1].set_ylabel('Accuracy')\n        plt.show()\n        \n        \n    def training_loop(self, epochs, ):\n        for epoch in range(epochs):\n            for training_batch in train_loader:\n                t_loss, t_accuracy = self.training_step(training_batch)\n                \n            for validation_batch in validation_loader:\n                v_loss, v_accuracy = self.validation_step(validation_batch)\n                    \n            self.training_epoch_end(t_accuracy, t_loss, v_accuracy, v_loss)\n                    \ndigitRecognizer = Model()","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:41.957171Z","iopub.execute_input":"2024-04-27T03:54:41.957427Z","iopub.status.idle":"2024-04-27T03:54:42.185485Z","shell.execute_reply.started":"2024-04-27T03:54:41.957405Z","shell.execute_reply":"2024-04-27T03:54:42.184637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Training model","metadata":{}},{"cell_type":"code","source":"digitRecognizer.training_loop(500)","metadata":{"execution":{"iopub.status.busy":"2024-04-26T17:41:16.444754Z","iopub.execute_input":"2024-04-26T17:41:16.445042Z","iopub.status.idle":"2024-04-26T17:41:40.467047Z","shell.execute_reply.started":"2024-04-26T17:41:16.445017Z","shell.execute_reply":"2024-04-26T17:41:40.466094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# i am also saveing the model after training it for 500 epochs\ntorch.save({'model' : digitRecognizer.state_dict(),\n            'optimizer' : digitRecognizer.optimizer.state_dict() }, 'digitRecognizer.pth')","metadata":{"execution":{"iopub.status.busy":"2024-04-27T03:54:42.186594Z","iopub.execute_input":"2024-04-27T03:54:42.186933Z","iopub.status.idle":"2024-04-27T03:54:42.210414Z","shell.execute_reply.started":"2024-04-27T03:54:42.186908Z","shell.execute_reply":"2024-04-27T03:54:42.209702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Looking at model performance\nI am using similar approch to test the model to previous [keras project](https://www.kaggle.com/code/bibekbhusal0/digit-recognizer-with-keras-99-978-accurate#Looking-at-the-model-performance-and-predictions).","metadata":{}},{"cell_type":"code","source":"digitRecognizer.plot_history()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def random_test(rows= 1, cols =1 ):\n    total = rows * cols\n    correct = total\n    random_indices = np.random.randint(len(validation), size=total)\n    fig, axes = plt.subplots(rows, cols, figsize=(cols * 1.3  ,1.8* rows) )\n    \n    random_images = validation.iloc[random_indices, 1:].values.astype(np.uint8)\n    labels = validation.iloc[random_indices, 0].values\n    images_tensor = transform(random_images)\n    images_tensor = images_tensor.to(device)\n    images_tensor = images_tensor.reshape(-1, 1, 28, 28)\n    \n    predictions_p = digitRecognizer(images_tensor)\n    _, predictions = torch.max(predictions_p, dim = 1)\n    predictions = predictions.cpu().numpy()\n    \n    for i in range(rows):\n        for j in range(cols):\n            index = i * cols + j\n            image = random_images[index]\n            label = labels[index]\n            prediction = predictions[index]\n            if rows == 1 and cols ==1:\n                ax = axes\n            elif rows ==1 :\n                ax = axes[j]\n            elif cols == 1:\n                ax = axes[i]\n            else:\n                ax = axes[i][j]\n            cmap = 'gray'\n            if label != prediction:\n                correct -= 1\n                cmap = 'OrRd'\n            ax.imshow(image.reshape(28, 28), cmap=cmap)\n            ax.set_title(f'L: {label}, P: {prediction}')\n            ax.axis(False)\n        fig.suptitle(f'correct:{correct}/{total}', fontsize=16)\n        fig.tight_layout(rect=[0, 0.03, 1, 0.95])\n    del images_tensor\n    del predictions_p\n    del predictions\n    torch.cuda.empty_cache()\n    plt.show()\n\nrandom_test(3,3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_test(10 ,10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. submission","metadata":{}},{"cell_type":"code","source":"test_data = test_data.to(device)\ntest_data_batch = test_data.reshape(1000, 28, 1, 28, 28)\n\npredictions_p = np.array([])\nfor t in test_data_batch:\n    predictions_p = np.append(predictions_p, digitRecognizer(t).detach().cpu().numpy())\n\npredictions_p = torch.from_numpy(predictions_p.reshape(-1, 10))\n\n_, predictions = torch.max(predictions_p, dim = 1)\npredictions = predictions.cpu().numpy()\n\nsample_submission['Label'] = predictions\nsample_submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thanks for reading till the end of this notebook. I hope you have learned something new, if yes then please do upvote and share. You can also checkout my [previous project](https://www.kaggle.com/bibekbhusal0/digit-recognizer-with-keras-99-978-accurate) which is [digit recognizer with keras](https://www.kaggle.com/bibekbhusal0/digit-recognizer-with-keras-99-978-accurate).\n\nMy next project will be Generative Adversarial Networks on this MNIST dataset with PyTorch, I will make it public so that others can also learn from it.","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}