{"metadata":{"accelerator":"GPU","colab":{"name":"HW03.ipynb","provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":86375,"databundleVersionId":9802769,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Experiment 2 - Convolutional Neural Network**\n\n\nIn this experiment, you are required to build a convolutional neural network for image classification, possibly with some advanced training tips.","metadata":{"id":"D_a2USyd4giE"}},{"cell_type":"markdown","source":"## **About the Dataset**\n\nThe dataset used here is food-11, a collection of food images in 11 classes.","metadata":{"id":"VHpJocsDr6iA"}},{"cell_type":"markdown","source":"## **Import Packages**\n\nFirst, we need to import packages that will be used later.\n\nIn this homework, we highly rely on **torchvision**, a library of PyTorch.","metadata":{"id":"BBVSCWWhp6uq"}},{"cell_type":"code","source":"# Import necessary packages.\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport torchvision.transforms as transforms\nfrom PIL import Image\n# \"ConcatDataset\" and \"Subset\" are possibly useful when doing semi-supervised learning.\nfrom torch.utils.data import ConcatDataset, DataLoader, Subset\nfrom torchvision.datasets import DatasetFolder\n\n# This is for the progress bar.\nfrom tqdm.auto import tqdm","metadata":{"id":"9sVrKci4PUFW","execution":{"iopub.status.busy":"2024-10-12T08:06:57.146832Z","iopub.execute_input":"2024-10-12T08:06:57.147182Z","iopub.status.idle":"2024-10-12T08:07:01.964641Z","shell.execute_reply.started":"2024-10-12T08:06:57.147145Z","shell.execute_reply":"2024-10-12T08:07:01.963424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Dataset, Data Loader, and Transforms**\n\nTorchvision provides lots of useful utilities for image preprocessing, data wrapping as well as data augmentation.\n\nHere, since our data are stored in folders by class labels, we can directly apply **torchvision.datasets.DatasetFolder** for wrapping data without much effort.\n\nPlease refer to [PyTorch official website](https://pytorch.org/vision/stable/transforms.html) for details about different transforms.","metadata":{"id":"F0i9ZCPrOVN_"}},{"cell_type":"code","source":"# It is important to do data augmentation in training.\n# However, not every augmentation is useful.\n# Please think about what kind of augmentation is helpful for food recognition.\ntrain_tfm = transforms.Compose([\n    # Resize the image into a fixed shape (height = width = 128)\n    transforms.Resize((128, 128)),\n    # You may add some transforms here.\n    # ToTensor() should be the last one of the transforms.\n    transforms.ToTensor(),\n])\n\n# We don't need augmentations in testing and validation.\n# All we need here is to resize the PIL image and transform it into Tensor.\ntest_tfm = transforms.Compose([\n    transforms.Resize((128, 128)),\n    transforms.ToTensor(),\n])\n","metadata":{"id":"gKd2abixQghI","execution":{"iopub.status.busy":"2024-10-12T08:07:01.966713Z","iopub.execute_input":"2024-10-12T08:07:01.967333Z","iopub.status.idle":"2024-10-12T08:07:01.974348Z","shell.execute_reply.started":"2024-10-12T08:07:01.967295Z","shell.execute_reply":"2024-10-12T08:07:01.973128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Batch size for training, validation, and testing.\n# A greater batch size usually gives a more stable gradient.\n# But the GPU memory is limited, so please adjust it carefully.\nbatch_size = 128\n\n# Construct datasets.\n# The argument \"loader\" tells how torchvision reads the data.\ntrain_set = DatasetFolder(\"/kaggle/input/dl-2024-autum-exp-2/food-11_DL_2024/training/labeled\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=train_tfm)\nvalid_set = DatasetFolder(\"/kaggle/input/dl-2024-autum-exp-2/food-11_DL_2024/validation\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=test_tfm)\nunlabeled_set = DatasetFolder(\"/kaggle/input/dl-2024-autum-exp-2/food-11_DL_2024/training/unlabeled\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=train_tfm)\ntest_set = DatasetFolder(\"/kaggle/input/dl-2024-autum-exp-2/food-11_DL_2024/testing\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=test_tfm)\n\n# Construct data loaders.\ntrain_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True, pin_memory=True)\nvalid_loader = DataLoader(valid_set, batch_size=batch_size, shuffle=True, pin_memory=True)\ntest_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False)","metadata":{"id":"qz6jeMnkQl0_","execution":{"iopub.status.busy":"2024-10-12T08:07:01.976543Z","iopub.execute_input":"2024-10-12T08:07:01.976964Z","iopub.status.idle":"2024-10-12T08:07:08.956084Z","shell.execute_reply.started":"2024-10-12T08:07:01.976926Z","shell.execute_reply":"2024-10-12T08:07:08.955191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Model**\n\nThe basic model here is simply a stack of convolutional layers followed by some fully-connected layers.\n\nSince there are three channels for a color image (RGB), the input channels of the network must be three.\nIn each convolutional layer, typically the channels of inputs grow, while the height and width shrink (or remain unchanged, according to some hyperparameters like stride and padding).\n\nBefore fed into fully-connected layers, the feature map must be flattened into a single one-dimensional vector (for each image).\nThese features are then transformed by the fully-connected layers, and finally, we obtain the \"logits\" for each class.\n\n### **WARNING -- You Must Know**\nYou are free to modify the model architecture here for further improvement.\nHowever, if you want to use some well-known architectures such as ResNet50, please make sure **NOT** to load the pre-trained weights.\nUsing such pre-trained models is considered cheating and therefore you will be punished.\nSimilarly, it is your responsibility to make sure no pre-trained weights are used if you use **torch.hub** to load any modules.\n\nFor example, if you use ResNet-18 as your model:\n\nmodel = torchvision.models.resnet18(pretrained=**False**) → This is fine.\n\nmodel = torchvision.models.resnet18(pretrained=**True**)  → This is **NOT** allowed.","metadata":{"id":"j9YhZo7POPYG"}},{"cell_type":"code","source":"class Classifier(nn.Module):\n    def __init__(self):\n        super(Classifier, self).__init__()\n        # The arguments for commonly used modules:\n        # torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)\n        # torch.nn.MaxPool2d(kernel_size, stride, padding)\n\n        # input image size: [3, 128, 128]\n        self.cnn_layers = nn.Sequential(\n            nn.Conv2d(3, 64, 3, 1, 1),\n            nn.BatchNorm2d(64),\n            nn.ReLU(),\n            nn.MaxPool2d(2, 2, 0),\n\n            nn.Conv2d(64, 128, 3, 1, 1),\n            nn.BatchNorm2d(128),\n            nn.ReLU(),\n            nn.MaxPool2d(2, 2, 0),\n\n            nn.Conv2d(128, 256, 3, 1, 1),\n            nn.BatchNorm2d(256),\n            nn.ReLU(),\n            nn.MaxPool2d(4, 4, 0),\n        )\n        self.fc_layers = nn.Sequential(\n            nn.Linear(256 * 8 * 8, 256),\n            nn.ReLU(),\n            nn.Linear(256, 256),\n            nn.ReLU(),\n            nn.Linear(256, 11)\n        )\n\n    def forward(self, x):\n        # input (x): [batch_size, 3, 128, 128]\n        # output: [batch_size, 11]\n\n        # Extract features by convolutional layers.\n        x = self.cnn_layers(x)\n\n        # The extracted feature map must be flatten before going to fully-connected layers.\n        x = x.flatten(1)\n\n        # The features are transformed by fully-connected layers to obtain the final logits.\n        x = self.fc_layers(x)\n        return x","metadata":{"id":"Y1c-GwrMQqMl","execution":{"iopub.status.busy":"2024-10-12T08:07:08.958830Z","iopub.execute_input":"2024-10-12T08:07:08.959197Z","iopub.status.idle":"2024-10-12T08:07:08.970124Z","shell.execute_reply.started":"2024-10-12T08:07:08.959161Z","shell.execute_reply":"2024-10-12T08:07:08.969067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Training**\n\nYou can finish supervised learning by simply running the provided code without any modification.\n\nThe function \"get_pseudo_labels\" is used for semi-supervised learning.\nIt is expected to get better performance if you use unlabeled data for semi-supervised learning.\nHowever, you have to implement the function on your own and need to adjust several hyperparameters manually.\n\nFor more details about semi-supervised learning, please refer to [Prof. Lee's slides](https://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/semi%20(v3).pdf).\n\nAgain, please notice that utilizing external data (or pre-trained model) for training is **prohibited**.","metadata":{"id":"aEnGbriXORN3"}},{"cell_type":"code","source":"def get_pseudo_labels(dataset, model, threshold=0.65):\n    # This functions generates pseudo-labels of a dataset using given model.\n    # It returns an instance of DatasetFolder containing images whose prediction confidences exceed a given threshold.\n    # You are NOT allowed to use any models trained on external data for pseudo-labeling.\n    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n    # Construct a data loader.\n    data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)\n\n    # Make sure the model is in eval mode.\n    model.eval()\n    # Define softmax function.\n    softmax = nn.Softmax(dim=-1)\n\n    # Iterate over the dataset by batches.\n    for batch in tqdm(data_loader):\n        img, _ = batch\n\n        # Forward the data\n        # Using torch.no_grad() accelerates the forward process.\n        with torch.no_grad():\n            logits = model(img.to(device))\n\n        # Obtain the probability distributions by applying softmax on logits.\n        probs = softmax(logits)\n\n        # ---------- TODO ----------\n        # Filter the data and construct a new dataset.\n\n    # # Turn off the eval mode.\n    model.train()\n    return dataset","metadata":{"id":"swlf5EwA-hxA","execution":{"iopub.status.busy":"2024-10-12T08:07:08.971676Z","iopub.execute_input":"2024-10-12T08:07:08.972017Z","iopub.status.idle":"2024-10-12T08:07:08.988055Z","shell.execute_reply.started":"2024-10-12T08:07:08.971974Z","shell.execute_reply":"2024-10-12T08:07:08.986854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Testing**\n\nFor inference, we need to make sure the model is in eval mode, and the order of the dataset should not be shuffled (\"shuffle=False\" in test_loader).\n\nLast but not least, don't forget to save the predictions into a single CSV file.\nThe format of CSV file should follow the rules mentioned in the slides.\n\n### **WARNING -- Keep in Mind**\n\nCheating includes but not limited to:\n1.   using testing labels,\n2.   submitting results to previous Kaggle competitions,\n3.   sharing predictions with others,\n4.   copying codes from any creatures on Earth,\n5.   asking other people to do it for you.\n\nAny violations bring you punishments from getting a discount on the final grade to failing the course.\n\nIt is your responsibility to check whether your code violates the rules.\nWhen citing codes from the Internet, you should know what these codes exactly do.\nYou will **NOT** be tolerated if you break the rule and claim you don't know what these codes do.\n","metadata":{"id":"2o1oCMXy61_3"}},{"cell_type":"code","source":"# \"cuda\" only when GPUs are available.\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n# Initialize a model, and put it on the device specified.\nmodel = Classifier().to(device)\nmodel.device = device\n\n# For the classification task, we use cross-entropy as the measurement of performance.\ncriterion = nn.CrossEntropyLoss()\n\n# Initialize optimizer, you may fine-tune some hyperparameters such as learning rate on your own.\noptimizer = torch.optim.Adam(model.parameters(), lr=0.0003, weight_decay=1e-5)\n\n# The number of training epochs.\nn_epochs = 80\n\n# Whether to do semi-supervised learning.\ndo_semi = False\n\nfor epoch in range(n_epochs):\n    # ---------- TODO ----------\n    # In each epoch, relabel the unlabeled dataset for semi-supervised learning.\n    # Then you can combine the labeled dataset and pseudo-labeled dataset for the training.\n    if do_semi:\n        # Obtain pseudo-labels for unlabeled data using trained model.\n        pseudo_set = get_pseudo_labels(unlabeled_set, model)\n\n        # Construct a new dataset and a data loader for training.\n        # This is used in semi-supervised learning only.\n        concat_dataset = ConcatDataset([train_set, pseudo_set])\n        train_loader = DataLoader(concat_dataset, batch_size=batch_size, shuffle=True, num_workers=8, pin_memory=True)\n\n    # ---------- Training ----------\n    # Make sure the model is in train mode before training.\n    model.train()\n\n    # These are used to record information in training.\n    train_loss = []\n    train_accs = []\n\n    # Iterate the training set by batches.\n    for batch in tqdm(train_loader):\n\n        # A batch consists of image data and corresponding labels.\n        imgs, labels = batch\n\n        # Forward the data. (Make sure data and model are on the same device.)\n        logits = model(imgs.to(device))\n\n        # Calculate the cross-entropy loss.\n        # We don't need to apply softmax before computing cross-entropy as it is done automatically.\n        loss = criterion(logits, labels.to(device))\n\n        # Gradients stored in the parameters in the previous step should be cleared out first.\n        optimizer.zero_grad()\n\n        # Compute the gradients for parameters.\n        loss.backward()\n\n        # Clip the gradient norms for stable training.\n        grad_norm = nn.utils.clip_grad_norm_(model.parameters(), max_norm=10)\n\n        # Update the parameters with computed gradients.\n        optimizer.step()\n\n        # Compute the accuracy for current batch.\n        acc = (logits.argmax(dim=-1) == labels.to(device)).float().mean()\n\n        # Record the loss and accuracy.\n        train_loss.append(loss.item())\n        train_accs.append(acc)\n\n    # The average loss and accuracy of the training set is the average of the recorded values.\n    train_loss = sum(train_loss) / len(train_loss)\n    train_acc = sum(train_accs) / len(train_accs)\n\n    # Print the information.\n    print(f\"[ Train | {epoch + 1:03d}/{n_epochs:03d} ] loss = {train_loss:.5f}, acc = {train_acc:.5f}\")\n\n    # ---------- Validation ----------\n    # Make sure the model is in eval mode so that some modules like dropout are disabled and work normally.\n    model.eval()\n\n    # These are used to record information in validation.\n    valid_loss = []\n    valid_accs = []\n\n    # Iterate the validation set by batches.\n    for batch in tqdm(valid_loader):\n\n        # A batch consists of image data and corresponding labels.\n        imgs, labels = batch\n\n        # We don't need gradient in validation.\n        # Using torch.no_grad() accelerates the forward process.\n        with torch.no_grad():\n          logits = model(imgs.to(device))\n\n        # We can still compute the loss (but not the gradient).\n        loss = criterion(logits, labels.to(device))\n\n        # Compute the accuracy for current batch.\n        acc = (logits.argmax(dim=-1) == labels.to(device)).float().mean()\n\n        # Record the loss and accuracy.\n        valid_loss.append(loss.item())\n        valid_accs.append(acc)\n\n    # The average loss and accuracy for entire validation set is the average of the recorded values.\n    valid_loss = sum(valid_loss) / len(valid_loss)\n    valid_acc = sum(valid_accs) / len(valid_accs)\n\n    # Print the information.\n    print(f\"[ Valid | {epoch + 1:03d}/{n_epochs:03d} ] loss = {valid_loss:.5f}, acc = {valid_acc:.5f}\")","metadata":{"id":"PHaFE-8oQtkC","execution":{"iopub.status.busy":"2024-10-12T08:07:08.991732Z","iopub.execute_input":"2024-10-12T08:07:08.992061Z","iopub.status.idle":"2024-10-12T13:22:30.215532Z","shell.execute_reply.started":"2024-10-12T08:07:08.992007Z","shell.execute_reply":"2024-10-12T13:22:30.214198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make sure the model is in eval mode.\n# Some modules like Dropout or BatchNorm affect if the model is in training mode.\nmodel.eval()\n\n# Initialize a list to store the predictions.\npredictions = []\n\n# Iterate the testing set by batches.\nfor batch in tqdm(test_loader):\n    # A batch consists of image data and corresponding labels.\n    # But here the variable \"labels\" is useless since we do not have the ground-truth.\n    # If printing out the labels, you will find that it is always 0.\n    # This is because the wrapper (DatasetFolder) returns images and labels for each batch,\n    # so we have to create fake labels to make it work normally.\n    imgs, labels = batch\n\n    # We don't need gradient in testing, and we don't even have labels to compute loss.\n    # Using torch.no_grad() accelerates the forward process.\n    with torch.no_grad():\n        logits = model(imgs.to(device))\n\n    # Take the class with greatest logit as prediction and record it.\n    predictions.extend(logits.argmax(dim=-1).cpu().numpy().tolist())","metadata":{"id":"4HznI9_-ocrq","execution":{"iopub.status.busy":"2024-10-12T13:22:30.217592Z","iopub.execute_input":"2024-10-12T13:22:30.218339Z","iopub.status.idle":"2024-10-12T13:23:02.725869Z","shell.execute_reply.started":"2024-10-12T13:22:30.218278Z","shell.execute_reply":"2024-10-12T13:23:02.724447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save predictions into the file.\nwith open(\"predict.csv\", \"w\") as f:\n\n    # The first row must be \"Id, Category\"\n    f.write(\"Id,Category\\n\")\n\n    # For the rest of the rows, each image id corresponds to a predicted class.\n    for i, pred in  enumerate(predictions):\n         f.write(f\"{i},{pred}\\n\")","metadata":{"id":"3t2q2Th85ZUE","execution":{"iopub.status.busy":"2024-10-12T13:23:02.727384Z","iopub.execute_input":"2024-10-12T13:23:02.727767Z","iopub.status.idle":"2024-10-12T13:23:02.734797Z","shell.execute_reply.started":"2024-10-12T13:23:02.727726Z","shell.execute_reply":"2024-10-12T13:23:02.733540Z"},"trusted":true},"execution_count":null,"outputs":[]}]}