{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Importing the necessary Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision.models.resnet import ResNet, BasicBlock\nfrom tqdm.autonotebook import tqdm\nfrom sklearn.model_selection import train_test_split\nfrom torch import nn, optim\nimport torch\nfrom torch.utils.data import DataLoader\nfrom matplotlib import pyplot as plt\nfrom torch.autograd import Variable\nimport seaborn as sns\nimport numpy as np \nimport pandas as pd\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\npd_train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\", dtype = np.float32)\npd_train.tail(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(11.7,8.27)})\nsns.countplot(pd_train[\"label\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = pd_train.label.values\nx_train = pd_train.loc[:,pd_train.columns != \"label\"].values/255\n\n#splitting the Data 80/20\nx_train, x_test, y_train, y_test = train_test_split(x_train,\n                                                    y_train,\n                                                    test_size = 0.2,\n                                                    random_state = 42) \n# converting to tensor\nx_train = torch.from_numpy(x_train)\ny_train = torch.from_numpy(y_train).type(torch.LongTensor)\nx_test = torch.from_numpy(x_test)\ny_test = torch.from_numpy(y_test).type(torch.LongTensor)\n\n#Binding the tests together\ntrain = torch.utils.data.TensorDataset(x_train,y_train)\ntest = torch.utils.data.TensorDataset(x_test,y_test)\n\nbatch_size = 100\ntrain_loader = DataLoader(train, batch_size=batch_size , shuffle=True)\ntest_loader = DataLoader(test, batch_size=batch_size , shuffle=True)\n\nplt.imshow(x_train[80].reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.reshape(-1,1,28,28).shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Architecture\n---\nNow for those of you who have worked with custom and readymade architectures, here's also a fun way to understand any architecture given in pytorch and how to mould it to your benefit.\n\nI have taken resnet18 for inspection purposes. It's available in the torchvision library so head over to the github link below so you can understand how the resnet18 model class is like in torchvison.\n<br>\n<br>\n[resnet18 in pytorch](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py)\n\n### The function resnet18 will look something like this (In the Resnet class)<br>\n\n    def _resnet(arch, block, layers, pretrained, progress, **kwargs):\n    model = ResNet(block, layers, **kwargs)\n    if pretrained:\n        state_dict = load_state_dict_from_url(model_urls[arch], progress=progress)\n        model.load_state_dict(state_dict)<br>\n    return model\n    \n    def resnet18(pretrained=False, progress=True, **kwargs):\n    Args:\n        pretrained (bool): If True, returns a model pre-trained on ImageNet\n        progress (bool): If True, displays a progress bar of the download to stderr\n    \"\"\"\n    return _resnet('resnet18', BasicBlock, [2, 2, 2, 2], pretrained, progress,\n                   **kwargs)\n\nNow in it we can see that for resnet18 function takes parameters for the Resnet class as:\n> model = ResNet(block, layers, **kwargs) \n  <br>where Block = BasicBlock and layers = \\[ 2, 2, 2, 2 \\] \n  \n### So now we are ready to use Resnet class as a parent gving the parameters from it as mentioned above\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### There's just one more thing\n\nNow, what happens when we train the network? In PyTorch, the forward function of network class is called - it represent forward pass of data through the network. ResNets forward looks like this:\n\n    def _forward_impl(self, x):\n        # See note [TorchScript super()]\n        x = self.conv1(x)\n        x = self.bn1(x)\n        x = self.relu(x)\n        x = self.maxpool(x)\n\n        x = self.layer1(x)\n        x = self.layer2(x)\n        x = self.layer3(x)\n        x = self.layer4(x)\n\n        x = self.avgpool(x)\n        x = torch.flatten(x, 1)\n        x = self.fc(x)\n\n        return x\n\n    def forward(self, x):\n        return self._forward_impl(x)\n        \nWe know that the resnet model takes 3 intout chanels so we will write the self.conv1 layer to according to our input:\n> self.conv1 = torch.nn.Conv2d(1, 64,  kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MnistResNet(ResNet):\n    def __init__(self):\n        super(MnistResNet, self).__init__(BasicBlock, [2, 2, 2, 2], num_classes=10)\n        self.conv1 = torch.nn.Conv2d(1, 64, \n                            kernel_size=(7, 7), \n                            stride=(2, 2), \n                            padding=(3, 3), bias=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training the model on GPU","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# For Training on GPU\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = MnistResNet()\nmodel = model.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = optim.Adagrad(model.parameters(),lr=0.01)\n\nloss_function = nn.CrossEntropyLoss()\nloss_function = loss_function.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_list = []\niterate_list = []\naccuracy_list = []\nepochs = 40\ncount = 0 \n\nfor a in tqdm(range(epochs)):\n    for i , (images, labels) in enumerate(train_loader):\n        \n        X, y = Variable(images.view(100,1,28,28)).to(device) , Variable(labels).to(device)\n        \n        optimizer.zero_grad()\n        outputs = model(X)\n        loss = loss_function(outputs,y)\n        loss.backward()\n        optimizer.step()\n        \n        count +=1\n        if count %50 == 0:\n            correct = 0\n            total = 0\n            with torch.no_grad():\n                for images ,labels in test_loader:\n                    test, labels = Variable(images.view(100,1,28,28)).to(device) , labels.to(device) \n\n                    outputs = model(test)\n                    predicted = torch.max(outputs.data, 1)[1]\n                    total += len(labels)\n                    correct += (predicted == labels).sum()\n                accuracy = 100 * correct / float(total)\n            \n            loss_list.append(loss.data)\n            iterate_list.append(count)\n            accuracy_list.append(accuracy)\n        if count % 200 == 0:\n            print(\"Epoch: {} \".format(a))\n            print('Iteration: {} ||  Loss: {} || Accuracy: {}%'.format(count, loss.data, accuracy))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualization loss \nplt.figure(figsize=[20,11])\nplt.plot(iterate_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"CNN: Loss vs Number of iteration\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualization accuracy \nplt.figure(figsize=[20,11])\nplt.plot(iterate_list,accuracy_list,color = \"red\")\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"CNN: Accuracy vs Number of iteration\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submissions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\",dtype = np.float32)\npd_sample_submission = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\n\ntesting = pd_test.loc[:,:].values/255\ntest = torch.from_numpy(testing)\ntest = test.to(device)\n\nsubmit = model(test.view(-1,1,28,28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_sample_submission[\"Label\"] = torch.max(submit.data.cpu(),1).indices\npd_sample_submission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Honorable Mentions\n- [Pytorch Tutorial for Deep Learning Lovers](https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers)\n- [MNIST Competition: PyTorch NN](https://www.kaggle.com/tarunpaparaju/mnist-competition-pytorch-nn)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}