{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction to Pytorch - CIFAR-10 Image Classification\nThis notebook serves as an introduction to Pytorch using the CIFAR-10 dataset.","metadata":{}},{"cell_type":"markdown","source":"### Import Libraries\n* torch: Build and train the dataset and model\n* torchvision: Computer vision & data augmentation\n* tqdm: Fancy loading bar\n* matplotlib: Displaying images","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader\nfrom torch.optim import Adam\nfrom torchvision.datasets import ImageFolder\nfrom torchvision import transforms\nfrom tqdm.auto import tqdm\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:37:39.408662Z","iopub.execute_input":"2023-10-24T16:37:39.409260Z","iopub.status.idle":"2023-10-24T16:37:42.892391Z","shell.execute_reply.started":"2023-10-24T16:37:39.409226Z","shell.execute_reply":"2023-10-24T16:37:42.891422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ### Helper function: show_image\n Create a helper function to display images.","metadata":{}},{"cell_type":"code","source":"inv_normalize = transforms.Normalize(\n    mean=[-0.4914/0.247, -0.4822/0.243, -0.4465/0.261],\n    std=[1/0.247, 1/0.243, 1/0.261]\n)\ndef show_image(image, normalized=True):\n    if normalized:\n        image = inv_normalize(image)\n    image = torch.squeeze(image)\n    image = torch.permute(image, (1, 2, 0))\n    image = image.cpu()\n    plt.imshow(image)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:37:42.893927Z","iopub.execute_input":"2023-10-24T16:37:42.894326Z","iopub.status.idle":"2023-10-24T16:37:42.900633Z","shell.execute_reply.started":"2023-10-24T16:37:42.894300Z","shell.execute_reply":"2023-10-24T16:37:42.899517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Parameters\nParameters of the model.\n\n* `lr`: Learning rate of our model\n* `batch_size`: Batch size\n* `criterion`: Loss function (cross-entropy)\n* `n_epochs`: Number of epochs\n* `device`: Whether we are using a GPU or CPU\n* `display_step`: How often to show training results\n\n---\nYou may notice that we are not using `nn.CrossEntropyLoss()` as we did before. We use the `nn.NLLLoss()` and `F.log_softmax()` functions for computational efficiency. This is equivalent to `nn.CrossEntropyLoss()` with `F.softmax()`.","metadata":{}},{"cell_type":"code","source":"lr = 1e-3\nbatch_size = 128\ncriterion = nn.NLLLoss()\nn_epochs = 15\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\ndisplay_step = 100","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:37:42.901663Z","iopub.execute_input":"2023-10-24T16:37:42.901929Z","iopub.status.idle":"2023-10-24T16:37:42.935553Z","shell.execute_reply.started":"2023-10-24T16:37:42.901905Z","shell.execute_reply":"2023-10-24T16:37:42.934673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_path = \"/kaggle/input/cifar10-pngs-in-folders/cifar10/train\"\ntest_path = \"/kaggle/input/cifar10-pngs-in-folders/cifar10/test\"","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:37:42.937410Z","iopub.execute_input":"2023-10-24T16:37:42.937678Z","iopub.status.idle":"2023-10-24T16:37:42.956882Z","shell.execute_reply.started":"2023-10-24T16:37:42.937655Z","shell.execute_reply":"2023-10-24T16:37:42.956035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [TASK 1] Data Augmentation\nUse the Pytorch documentation page (https://pytorch.org/vision/0.9/transforms.html) to randomly flip the images horizontally with probability 0.5 (the default), and randomly resize and crop images to 32x32 (you will have to put the `antialias=True` argument).\n\n---\n\nThe code below standardizes the data to follow a standard normal distribution (zero mean and unit variance).\nIf you do not know the mean or standard deviation, use 0.5 for all values to normalize values to be in the interval [-1, 1].","metadata":{}},{"cell_type":"code","source":"# Preprocess images\ntrain_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.RandomHorizontalFlip(),\n    transforms.RandomResizedCrop(32, antialias=True),\n    transforms.Normalize(mean=[0.4914, 0.4822, 0.4465], std=[0.247, 0.243, 0.261])\n])\ntest_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.4914, 0.4822, 0.4465], std=[0.247, 0.243, 0.261])\n])","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:37:42.957721Z","iopub.execute_input":"2023-10-24T16:37:42.957968Z","iopub.status.idle":"2023-10-24T16:37:42.969905Z","shell.execute_reply.started":"2023-10-24T16:37:42.957947Z","shell.execute_reply":"2023-10-24T16:37:42.968904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [TASK 2] Create dataset\nUsing the Pytorch documentation page (https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html), create the datasets for the train and test sets.\n\n--- \n\nHere we are using a different function to create the dataset because each class is separated into folders of images.","metadata":{}},{"cell_type":"code","source":"train_dataset = ImageFolder(train_path, train_transform)\ntest_dataset = ImageFolder(test_path, test_transform)","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:37:42.971038Z","iopub.execute_input":"2023-10-24T16:37:42.971454Z","iopub.status.idle":"2023-10-24T16:38:16.114308Z","shell.execute_reply.started":"2023-10-24T16:37:42.971422Z","shell.execute_reply":"2023-10-24T16:38:16.113516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [TASK 3] Create convolution blocks\nUsing the Pytorch documentation (https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html), create 2 layers of 2D convolutions. The first layer `conv1` takes `input_dim` input channels and returns a tensor with `output_dim` channels. The second layer `conv2` takes `output_dim` channels to `output_dim` channels. Both layers have a kernel size of 3 and a padding of 1.\n\n---\n\nWe will use these blocks to create our full model.\n\nFun fact: The ReLU and max pooling layer commute with each other and can be applied in any order to yield the same result. We use ReLU before the max pooling layer for slight computational efficiency.","metadata":{}},{"cell_type":"code","source":"class ConvBlock(nn.Module):\n    def __init__(self, input_dim, output_dim):\n        super(ConvBlock, self).__init__()\n        self.conv1 = nn.Conv2d(input_dim, output_dim, kernel_size=3, padding=1)\n        self.conv2 = nn.Conv2d(output_dim, output_dim, kernel_size=3, padding=1)\n        \n    def forward(self, x):\n        x = self.conv1(x)\n        x = F.relu(x)\n        x = self.conv2(x)\n        x = F.relu(x)\n        x = F.max_pool2d(x, 2)\n        \n        return x","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:38:16.115340Z","iopub.execute_input":"2023-10-24T16:38:16.115616Z","iopub.status.idle":"2023-10-24T16:38:16.122202Z","shell.execute_reply.started":"2023-10-24T16:38:16.115593Z","shell.execute_reply":"2023-10-24T16:38:16.121136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [TASK 4] Implement the `forward` function\nWrite the code to pass the tensor `x` through the final perceptron layers. Apply `self.fc1`, `self.fc2`, `self.fc3` sequentially with `F.relu(x)` in between.\n\n---\n\nThe below model is a modified LeNet (https://en.wikipedia.org/wiki/LeNet).","metadata":{}},{"cell_type":"code","source":"class Model(nn.Module):\n    def __init__(self):\n        super(Model, self).__init__()\n        \n        self.block1 = ConvBlock(3, 64)\n        self.block2 = ConvBlock(64, 128)\n        self.block3 = ConvBlock(128, 256)\n        \n        self.fc1 = nn.Linear(4096, 1024)\n        self.fc2 = nn.Linear(1024, 512)\n        self.fc3 = nn.Linear(512, 10)\n        \n        \n    def forward(self, x):\n        x = self.block1(x)\n        x = self.block2(x)\n        x = self.block3(x)\n        \n        x = torch.flatten(x, start_dim=1)  # Turn the tensor into a vector\n        \n        x = self.fc1(x)\n        x = F.relu(x)\n        \n        x = self.fc2(x)\n        x = F.relu(x)\n        \n        x = self.fc3(x)\n        x = F.log_softmax(x, dim=1)\n        \n        return x","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:38:16.123456Z","iopub.execute_input":"2023-10-24T16:38:16.123794Z","iopub.status.idle":"2023-10-24T16:38:16.137601Z","shell.execute_reply.started":"2023-10-24T16:38:16.123763Z","shell.execute_reply":"2023-10-24T16:38:16.136777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = Model().to(device)\n\noptimizer = Adam(model.parameters(), lr=lr)","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:38:16.138613Z","iopub.execute_input":"2023-10-24T16:38:16.138870Z","iopub.status.idle":"2023-10-24T16:38:19.071804Z","shell.execute_reply.started":"2023-10-24T16:38:16.138848Z","shell.execute_reply":"2023-10-24T16:38:19.070980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [TASK 5] Write backpropagation code\nUsing the MNIST notebook, perform the following steps:\n1. Zero out the gradient of the optimizer.\n2. Backpropagate the loss\n3. Step the optimizer to update the model weights","metadata":{}},{"cell_type":"code","source":"def train():\n    # Create the dataloader\n    # The pin_memory and num_workers arguments make loading the images faster\n    dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=4, persistent_workers=True)\n    cur_step = 0\n    total_loss = 0\n    \n    for _ in range(n_epochs):\n        for image, label in tqdm(dataloader):\n            # Run prediction\n            image = image.to(device)\n            label = label.to(device)\n            label_pred = model(image)\n            loss = criterion(label_pred, label)\n            total_loss += loss.item()\n            \n            optimizer.zero_grad()\n            loss.backward()\n            optimizer.step()\n            \n            # Display the loss after every display_step steps\n            if cur_step == display_step:\n                print(f\"Loss:{total_loss / cur_step}\")\n                total_loss = 0\n                cur_step = 0\n            \n            cur_step += 1","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:38:19.074311Z","iopub.execute_input":"2023-10-24T16:38:19.074721Z","iopub.status.idle":"2023-10-24T16:38:19.081797Z","shell.execute_reply.started":"2023-10-24T16:38:19.074682Z","shell.execute_reply":"2023-10-24T16:38:19.080859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:38:19.082767Z","iopub.execute_input":"2023-10-24T16:38:19.083067Z","iopub.status.idle":"2023-10-24T16:44:39.159953Z","shell.execute_reply.started":"2023-10-24T16:38:19.083042Z","shell.execute_reply":"2023-10-24T16:44:39.158882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Validate the model\nUse the validation dataset to compute the accuracy of the model on the unseen validation data. If the model is trained correctly, the accuracy should be relatively high (and should have a validation loss that matches the training loss closely).","metadata":{}},{"cell_type":"code","source":"def validate():\n    dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=4, persistent_workers=True)\n    accuracy = 0\n    count = 0\n    \n    with torch.no_grad():\n        for image, label in tqdm(dataloader):\n            count += 1\n            \n            # Run prediction\n            image = image.to(device)\n            label = label.to(device)\n            label_pred = model(image)\n            \n            # Get top prediction\n            label_pred = torch.argmax(label_pred, dim=1)\n            \n            # Calculate accuracy\n            equals = label_pred == label\n            accuracy += torch.mean(equals.type(torch.FloatTensor))\n    accuracy /= count\n    print(f\"Accuracy: {accuracy.item() * 100}%\")","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:44:39.161574Z","iopub.execute_input":"2023-10-24T16:44:39.161866Z","iopub.status.idle":"2023-10-24T16:44:39.169894Z","shell.execute_reply.started":"2023-10-24T16:44:39.161838Z","shell.execute_reply":"2023-10-24T16:44:39.169069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"validate()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:44:39.171039Z","iopub.execute_input":"2023-10-24T16:44:39.171292Z","iopub.status.idle":"2023-10-24T16:44:49.507843Z","shell.execute_reply.started":"2023-10-24T16:44:39.171269Z","shell.execute_reply":"2023-10-24T16:44:49.506797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize predictions\nRun the model on some images","metadata":{}},{"cell_type":"code","source":"classes = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\nfor i in range(0, 10000, 1000):\n    image = torch.unsqueeze(test_dataset[i][0], dim=0).to(device)\n    \n    label = torch.Tensor([test_dataset[i][1]]).to(device)\n    label = classes[int(label.item())]\n    \n    label_pred = model(image)\n    label_pred = classes[torch.argmax(label_pred)]\n    \n    \n    show_image(image)\n    print(f\"Label: {label} \\t Predicted Label: {label_pred}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-24T16:47:52.692625Z","iopub.execute_input":"2023-10-24T16:47:52.693607Z","iopub.status.idle":"2023-10-24T16:47:54.922708Z","shell.execute_reply.started":"2023-10-24T16:47:52.693574Z","shell.execute_reply":"2023-10-24T16:47:54.921781Z"},"trusted":true},"execution_count":null,"outputs":[]}]}