{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HW2P2: Face Classification and Verification\n","metadata":{"id":"XBITN0M_LKds"}},{"cell_type":"markdown","source":"Congrats on coming to the second homework in 11785: Introduction to Deep Learning. This homework significantly longer and tougher than the previous homework. You have 2 sub-parts as outlined below. Please start early!\n\n\n*   Face Recognition: You will be writing your own CNN model to tackle the problem of classification, consisting of 7001 identities\n*   Face Verification: You use the model trained for classification to evaluate the quality of its feature embeddings, by comparing the similarity of known and unknown identities","metadata":{"id":"-NH4P-HzLRQs"}},{"cell_type":"markdown","source":"Common errors which you may face in this homeworks (because of the size of the model)\n\n\n*   CUDA Out of Memory (OOM): You can tackle this problem by (1) Reducing the batch size (2) Calling `torch.cuda.empty_cache()` and `gc.collect()` (3) Finally restarting the runtime\n\n","metadata":{"id":"i1B_m84_cU6c"}},{"cell_type":"markdown","source":"# Preliminaries","metadata":{"id":"BdoDIKWOMF59"}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"!nvidia-smi # to see what GPU you have","metadata":{"id":"Jza7lwiScUhb","execution":{"iopub.status.busy":"2023-10-23T18:17:20.8837Z","iopub.execute_input":"2023-10-23T18:17:20.88455Z","iopub.status.idle":"2023-10-23T18:17:21.960048Z","shell.execute_reply.started":"2023-10-23T18:17:20.884512Z","shell.execute_reply":"2023-10-23T18:17:21.958893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install torchsummary wandb --quiet","metadata":{"id":"bTxfd_nqFnL9","outputId":"1bc4fd52-c665-472d-ed41-43565d71a970","execution":{"iopub.status.busy":"2023-10-23T18:18:21.410566Z","iopub.execute_input":"2023-10-23T18:18:21.410975Z","iopub.status.idle":"2023-10-23T18:18:33.333745Z","shell.execute_reply.started":"2023-10-23T18:18:21.410935Z","shell.execute_reply":"2023-10-23T18:18:33.33225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nfrom torchsummary import summary\nimport torchvision #This library is used for image-based operations (Augmentations)\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nimport os\nimport gc\nfrom tqdm import tqdm\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom sklearn.metrics import accuracy_score\nimport glob\nimport wandb\nimport matplotlib.pyplot as plt\nDEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint(\"Device: \", DEVICE)","metadata":{"id":"jwLEd0gdPbSc","outputId":"9d52e17f-25a6-46c6-8559-8cfc5fe29f1b","execution":{"iopub.status.busy":"2023-10-23T18:18:04.553128Z","iopub.execute_input":"2023-10-23T18:18:04.553492Z","iopub.status.idle":"2023-10-23T18:18:09.484095Z","shell.execute_reply.started":"2023-10-23T18:18:04.553462Z","shell.execute_reply":"2023-10-23T18:18:09.48307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#from google.colab import drive # Link your drive if you are a colab user\n#drive.mount('/content/drive') # Models in this HW take a long time to get trained and make sure to save it her","metadata":{"id":"SRz9et3SZnbO","outputId":"33ef3f5c-c01a-4775-de49-fde8184eb18e"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TODOs\nAs you go, please read the code and keep an eye out for TODOs!","metadata":{"id":"1oxQNl-YVWHc"}},{"cell_type":"markdown","source":"# Download Data from Kaggle","metadata":{"id":"scOnMklwWBY6"}},{"cell_type":"code","source":"# TODO: Use the same Kaggle code from HW1P2","metadata":{"id":"6BksgPdkQwwb"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!mkdir '/content/data'\n\n# !kaggle competitions download -c 11-785-f23-hw2p2-classification\n# !unzip -qo '11-785-f23-hw2p2-classification.zip' -d '/content/data'\n\n# !kaggle competitions download -c 11-785-f23-hw2p2-verification\n# !unzip -qo '11-785-f23-hw2p2-verification.zip' -d '/content/data'","metadata":{"id":"3oFjaJTaRjT7"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configs","metadata":{"id":"O68hT27SXClj"}},{"cell_type":"code","source":"config = {\n    'batch_size': 64, # Increase this if your GPU can handle it\n    'lr': 0.1,\n    'epochs': 100, # 20 epochs is recommended ONLY for the early submission - you will have to train for much longer typically.\n    'model' : 'convnextv1'\n    # Include other parameters as needed.\n}","metadata":{"id":"S7qpMxG0XCJz","execution":{"iopub.status.busy":"2023-10-23T18:18:34.867016Z","iopub.execute_input":"2023-10-23T18:18:34.867412Z","iopub.status.idle":"2023-10-23T18:18:34.873155Z","shell.execute_reply.started":"2023-10-23T18:18:34.867372Z","shell.execute_reply":"2023-10-23T18:18:34.872184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Classification Dataset","metadata":{"id":"sSeiKHYrM-6b"}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_DIR    = '/kaggle/input/11-785-f23-hw2p2-classification/11-785-f23-hw2p2-classification'# TODO: Path where you have downloaded the data\nTRAIN_DIR   = os.path.join(DATA_DIR, \"train\")\nVAL_DIR     = os.path.join(DATA_DIR, \"dev\")\nTEST_DIR    = os.path.join(DATA_DIR, \"test\")\n\n# Transforms using torchvision - Refer https://pytorch.org/vision/stable/transforms.html\ntest_transforms = torchvision.transforms.Compose([\n    torchvision.transforms.ToTensor(),\n    torchvision.transforms.Normalize(mean=[0.5102, 0.4013, 0.3507], std=[0.3074, 0.2701, 0.2590])\n])\n\ntrain_transforms = torchvision.transforms.Compose([\n    torchvision.transforms.RandomHorizontalFlip(p=0.5),\n    torchvision.transforms.RandomRotation(20), # Slight rotation\n    torchvision.transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),\n    torchvision.transforms.ToTensor(),\n    torchvision.transforms.Normalize(mean=[0.5102, 0.4013, 0.3507], std=[0.3074, 0.2701, 0.2590])\n])# Implementing the right train transforms/augmentation methods is key to improving performance.\n\nvalid_transforms = torchvision.transforms.Compose([\n    torchvision.transforms.ToTensor(),\n    torchvision.transforms.Normalize(mean=[0.5102, 0.4013, 0.3507], std=[0.3074, 0.2701, 0.2590]) # Values for ImageNet\n])\n\ntrain_dataset   = torchvision.datasets.ImageFolder(TRAIN_DIR, transform= train_transforms)\nvalid_dataset   = torchvision.datasets.ImageFolder(VAL_DIR, transform= valid_transforms)\n# You should NOT have data augmentation on the validation set. Why?\n\n\n# Create data loaders\ntrain_loader = torch.utils.data.DataLoader(\n    dataset     = train_dataset,\n    batch_size  = config['batch_size'],\n    shuffle     = True,\n    num_workers = 4,\n    pin_memory  = True\n)\n\nvalid_loader = torch.utils.data.DataLoader(\n    dataset     = valid_dataset,\n    batch_size  = config['batch_size'],\n    shuffle     = False,\n    num_workers = 2\n)","metadata":{"id":"tmRX5omaNDEZ","execution":{"iopub.status.busy":"2023-10-23T18:18:36.646915Z","iopub.execute_input":"2023-10-23T18:18:36.647794Z","iopub.status.idle":"2023-10-23T18:19:42.266457Z","shell.execute_reply.started":"2023-10-23T18:18:36.64776Z","shell.execute_reply":"2023-10-23T18:19:42.2654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# You can do this with ImageFolder as well, but it requires some tweaking\nclass ClassificationTestDataset(torch.utils.data.Dataset):\n\n    def __init__(self, data_dir, transforms):\n        self.data_dir   = data_dir\n        self.transforms = transforms\n\n        # This one-liner basically generates a sorted list of full paths to each image in the test directory\n        self.img_paths  = list(map(lambda fname: os.path.join(self.data_dir, fname), sorted(os.listdir(self.data_dir))))\n\n    def __len__(self):\n        return len(self.img_paths)\n\n    def __getitem__(self, idx):\n        return self.transforms(Image.open(self.img_paths[idx]))","metadata":{"id":"SqSR063BGE2e","execution":{"iopub.status.busy":"2023-10-23T18:21:48.386763Z","iopub.execute_input":"2023-10-23T18:21:48.387651Z","iopub.status.idle":"2023-10-23T18:21:48.394357Z","shell.execute_reply.started":"2023-10-23T18:21:48.387615Z","shell.execute_reply":"2023-10-23T18:21:48.393394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset = ClassificationTestDataset(TEST_DIR, transforms = valid_transforms) #Why are we using val_transforms for Test Data?\ntest_loader = torch.utils.data.DataLoader(test_dataset, batch_size = config['batch_size'], shuffle = False,\n                         drop_last = False, num_workers = 2)","metadata":{"id":"fVLB41KtGC2o","execution":{"iopub.status.busy":"2023-10-23T18:21:50.641592Z","iopub.execute_input":"2023-10-23T18:21:50.642446Z","iopub.status.idle":"2023-10-23T18:21:51.074631Z","shell.execute_reply.started":"2023-10-23T18:21:50.642414Z","shell.execute_reply":"2023-10-23T18:21:51.073741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of classes    : \", len(train_dataset.classes))\nprint(\"No. of train images  : \", train_dataset.__len__())\nprint(\"Shape of image       : \", train_dataset[0][0].shape)\nprint(\"Batch size           : \", config['batch_size'])\nprint(\"Train batches        : \", train_loader.__len__())\nprint(\"Val batches          : \", valid_loader.__len__())","metadata":{"id":"x4t8eU9gY0Jy","execution":{"iopub.status.busy":"2023-10-23T18:21:52.285839Z","iopub.execute_input":"2023-10-23T18:21:52.286234Z","iopub.status.idle":"2023-10-23T18:21:52.389699Z","shell.execute_reply.started":"2023-10-23T18:21:52.286205Z","shell.execute_reply":"2023-10-23T18:21:52.388633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data visualization","metadata":{"id":"zs2Xw_tl0IQ8"}},{"cell_type":"code","source":"# Visualize a few images in the dataset\n# You can write your own code, and you don't need to understand the code\n# It is highly recommended that you visualize your data augmentation as sanity check\n\nr, c    = [5, 5]\nfig, ax = plt.subplots(r, c, figsize= (15, 15))\n\nk       = 0\ndtl     = torch.utils.data.DataLoader(\n    dataset     = torchvision.datasets.ImageFolder(TRAIN_DIR, transform= train_transforms), # dont wanna see the images with transforms\n    batch_size  = config['batch_size'],\n    shuffle     = True,\n)\n\nfor data in dtl:\n    x, y = data\n\n    for i in range(r):\n        for j in range(c):\n            img = x[k].numpy().transpose(1, 2, 0)\n            ax[i, j].imshow(img)\n            ax[i, j].axis('off')\n            k+=1\n    break\n\ndel dtl","metadata":{"id":"xIoRUzCbz85y","execution":{"iopub.status.busy":"2023-10-23T18:21:55.268221Z","iopub.execute_input":"2023-10-23T18:21:55.268591Z","iopub.status.idle":"2023-10-23T18:22:02.049128Z","shell.execute_reply.started":"2023-10-23T18:21:55.268562Z","shell.execute_reply":"2023-10-23T18:22:02.047602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Very Simple Network (for Mandatory Early Submission)","metadata":{"id":"mIqmojPaWD0H"}},{"cell_type":"code","source":"class ConvNextBlock(torch.nn.Module):\n    def __init__(self, in_channels, out_channels, stochastic_depth = 0.0, expansion = 4, stride = 1):\n        super().__init__()\n        \n        self.cnnblock = torch.nn.Sequential(\n            torch.nn.Conv2d(in_channels, in_channels, kernel_size = 7, stride = stride, padding = 3, groups = in_channels),\n            torch.nn.BatchNorm2d(in_channels),\n            torch.nn.Conv2d(in_channels, in_channels * expansion, kernel_size = 1),\n            torch.nn.GELU(),\n            torch.nn.Conv2d(in_channels * expansion, out_channels, kernel_size = 1)\n        )\n    def forward(self, x):\n        out = self.cnnblock(x)\n        return out + x\n\nclass ConvNextNetwork(torch.nn.Module):\n    def __init__(self, num_classes = 7001):\n        super().__init__()\n        self.backbone = torch.nn.Sequential(\n            torch.nn.Conv2d(3,96, kernel_size = 4, stride = 4),\n            torch.nn.BatchNorm2d(96),\n            ConvNextBlock(96,96),\n            ConvNextBlock(96,96),\n            torch.nn.BatchNorm2d(96),\n            torch.nn.Conv2d(96, 192, kernel_size=2, stride=2),\n            ConvNextBlock(192,192),\n            ConvNextBlock(192,192),\n            torch.nn.BatchNorm2d(192),\n            torch.nn.Conv2d(192, 384, kernel_size=2, stride=2),\n            ConvNextBlock(384,384),\n            ConvNextBlock(384,384),\n            ConvNextBlock(384,384),\n            torch.nn.BatchNorm2d(384),\n            torch.nn.Conv2d(384, 768, kernel_size=2, stride=2),\n            ConvNextBlock(768,768),\n            ConvNextBlock(768,768),\n            torch.nn.AdaptiveAvgPool2d(1),\n            torch.nn.Flatten(),\n            torch.nn.BatchNorm1d(768),\n            torch.nn.Dropout(0.2)\n        )\n        self.close = torch.nn.Linear(768, num_classes)\n        \n    def forward(self, x, returnf = False):\n        f = self.backbone(x)\n        out = self.close(f)\n        if returnf:\n            return f\n        else:\n            return out\nmodel = ConvNextNetwork().to(DEVICE)\nsummary(model, (3,224, 224))","metadata":{"id":"Ny-mh_ocWIJR","execution":{"iopub.status.busy":"2023-10-23T18:25:08.30132Z","iopub.execute_input":"2023-10-23T18:25:08.301701Z","iopub.status.idle":"2023-10-23T18:25:08.515619Z","shell.execute_reply.started":"2023-10-23T18:25:08.301673Z","shell.execute_reply":"2023-10-23T18:25:08.514646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setup everything for training","metadata":{"id":"KZCn0qHuZRKj"}},{"cell_type":"code","source":"criterion = torch.nn.CrossEntropyLoss(label_smoothing=0.2)# TODO: What loss do you need for a multi class classification problem?\noptimizer = torch.optim.SGD(model.parameters(), lr=config['lr'], momentum=0.9, weight_decay=1e-4)\n# TODO: Implement a scheduler (Optional but Highly Recommended)\n# You can try ReduceLRonPlateau, StepLR, MultistepLR, CosineAnnealing, etc.\n# Define the optimizer\n\n# Define the learning rate scheduler\nscheduler = ReduceLROnPlateau(optimizer, 'max', patience=3, factor=0.7)\nscaler = torch.cuda.amp.GradScaler() # Good news. We have FP16 (Mixed precision training) implemented for you\n# It is useful only in the case of compatible GPUs such as T4/V100","metadata":{"id":"UowI9OcUYPjP","execution":{"iopub.status.busy":"2023-10-21T19:40:58.846848Z","iopub.execute_input":"2023-10-21T19:40:58.847245Z","iopub.status.idle":"2023-10-21T19:40:58.85452Z","shell.execute_reply.started":"2023-10-21T19:40:58.847207Z","shell.execute_reply":"2023-10-21T19:40:58.853594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model.load_state_dict(torch.load('/kaggle/input/lowmodel/h2p2low')['model_state_dict'])","metadata":{"execution":{"iopub.status.busy":"2023-10-05T16:38:31.176986Z","iopub.execute_input":"2023-10-05T16:38:31.177332Z","iopub.status.idle":"2023-10-05T16:38:31.946519Z","shell.execute_reply.started":"2023-10-05T16:38:31.177305Z","shell.execute_reply":"2023-10-05T16:38:31.945589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Let's train!","metadata":{"id":"dzM11HtcboYv"}},{"cell_type":"code","source":"def train(model, dataloader, optimizer, criterion):\n\n    model.train()\n\n    # Progress Bar\n    batch_bar   = tqdm(total=len(dataloader), dynamic_ncols=True, leave=False, position=0, desc='Train', ncols=5)\n\n    num_correct = 0\n    total_loss  = 0\n\n    for i, (images, labels) in enumerate(dataloader):\n\n        optimizer.zero_grad() # Zero gradients\n\n        images, labels = images.to(DEVICE), labels.to(DEVICE)\n\n        with torch.cuda.amp.autocast(): # This implements mixed precision. Thats it!\n            outputs = model(images)\n            loss    = criterion(outputs, labels)\n\n        # Update no. of correct predictions & loss as we iterate\n        num_correct     += int((torch.argmax(outputs, axis=1) == labels).sum())\n        total_loss      += float(loss.item())\n\n        # tqdm lets you add some details so you can monitor training as you train.\n        batch_bar.set_postfix(\n            acc         = \"{:.04f}%\".format(100 * num_correct / (config['batch_size']*(i + 1))),\n            loss        = \"{:.04f}\".format(float(total_loss / (i + 1))),\n            num_correct = num_correct,\n            lr          = \"{:.04f}\".format(float(optimizer.param_groups[0]['lr']))\n        )\n\n        scaler.scale(loss).backward() # This is a replacement for loss.backward()\n        scaler.step(optimizer) # This is a replacement for optimizer.step()\n        scaler.update()\n\n        # TODO? Depending on your choice of scheduler,\n        # You may want to call some schdulers inside the train function. What are these?\n\n        batch_bar.update() # Update tqdm bar\n\n    batch_bar.close() # You need this to close the tqdm bar\n\n    acc         = 100 * num_correct / (config['batch_size']* len(dataloader))\n    total_loss  = float(total_loss / len(dataloader))\n\n    return acc, total_loss","metadata":{"id":"bgSw6iJJavBZ","execution":{"iopub.status.busy":"2023-10-21T19:41:09.616166Z","iopub.execute_input":"2023-10-21T19:41:09.616942Z","iopub.status.idle":"2023-10-21T19:41:09.627269Z","shell.execute_reply.started":"2023-10-21T19:41:09.616909Z","shell.execute_reply":"2023-10-21T19:41:09.626283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def validate(model, dataloader, criterion):\n\n    model.eval()\n    batch_bar = tqdm(total=len(dataloader), dynamic_ncols=True, position=0, leave=False, desc='Val', ncols=5)\n\n    num_correct = 0.0\n    total_loss = 0.0\n\n    for i, (images, labels) in enumerate(dataloader):\n\n        # Move images to device\n        images, labels = images.to(DEVICE), labels.to(DEVICE)\n\n        # Get model outputs\n        with torch.inference_mode():\n            outputs = model(images)\n            loss = criterion(outputs, labels)\n\n        num_correct += int((torch.argmax(outputs, axis=1) == labels).sum())\n        total_loss += float(loss.item())\n\n        batch_bar.set_postfix(\n            acc=\"{:.04f}%\".format(100 * num_correct / (config['batch_size']*(i + 1))),\n            loss=\"{:.04f}\".format(float(total_loss / (i + 1))),\n            num_correct=num_correct)\n\n        batch_bar.update()\n\n    batch_bar.close()\n    acc = 100 * num_correct / (config['batch_size']* len(dataloader))\n    total_loss = float(total_loss / len(dataloader))\n    return acc, total_loss","metadata":{"id":"m5V2UdnpdEoK","execution":{"iopub.status.busy":"2023-10-21T19:41:11.956154Z","iopub.execute_input":"2023-10-21T19:41:11.957018Z","iopub.status.idle":"2023-10-21T19:41:11.965909Z","shell.execute_reply.started":"2023-10-21T19:41:11.956982Z","shell.execute_reply":"2023-10-21T19:41:11.964852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect() # These commands help you when you face CUDA OOM error\ntorch.cuda.empty_cache()","metadata":{"id":"cmotca6pcLLY","execution":{"iopub.status.busy":"2023-10-05T04:45:22.130563Z","iopub.execute_input":"2023-10-05T04:45:22.131269Z","iopub.status.idle":"2023-10-05T04:45:22.316365Z","shell.execute_reply.started":"2023-10-05T04:45:22.131236Z","shell.execute_reply":"2023-10-05T04:45:22.315369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Wandb","metadata":{"id":"2mBgKGkXLrdJ"}},{"cell_type":"code","source":"wandb.login(key=\"b78f5a0a316b8f09525ad5c671b527f4b95c1695\") #API Key is in your wandb account, under settings (wandb.ai/settings)","metadata":{"id":"Ix62_BkaLr_D","execution":{"iopub.status.busy":"2023-10-21T19:41:14.691627Z","iopub.execute_input":"2023-10-21T19:41:14.692507Z","iopub.status.idle":"2023-10-21T19:41:18.279819Z","shell.execute_reply.started":"2023-10-21T19:41:14.692471Z","shell.execute_reply":"2023-10-21T19:41:18.278769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create your wandb run\nrun = wandb.init(\n    name = \"high-archi\", ## Wandb creates random run names if you skip this field\n    reinit = True, ### Allows reinitalizing runs when you re-run this cell\n    # run_id = ### Insert specific run id here if you want to resume a previous run\n    # resume = \"must\" ### You need this to resume previous runs, but comment out reinit = True when using this\n    project = \"hw2p2-ablations\", ### Project should be created in your wandb account\n    config = config ### Wandb Config for your run\n)","metadata":{"id":"VG0vmsmbRYEi","execution":{"iopub.status.busy":"2023-10-21T19:42:16.995227Z","iopub.execute_input":"2023-10-21T19:42:16.996105Z","iopub.status.idle":"2023-10-21T19:42:48.923654Z","shell.execute_reply.started":"2023-10-21T19:42:16.996045Z","shell.execute_reply":"2023-10-21T19:42:48.922724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Experiments","metadata":{"id":"SQkRw1FvLqYe"}},{"cell_type":"code","source":"best_valacc = 0.0\n\nfor epoch in range(config['epochs']):\n    curr_lr = float(optimizer.param_groups[0]['lr'])\n\n    train_acc, train_loss = train(model, train_loader, optimizer, criterion)\n\n    print(\"\\nEpoch {}/{}: \\nTrain Acc {:.04f}%\\t Train Loss {:.04f}\\t Learning Rate {:.04f}\".format(\n        epoch + 1,\n        config['epochs'],\n        train_acc,\n        train_loss,\n        curr_lr))\n\n    val_acc, val_loss = validate(model, valid_loader, criterion)\n    scheduler.step(val_acc)\n    print(\"Val Acc {:.04f}%\\t Val Loss {:.04f}\".format(val_acc, val_loss))\n\n    wandb.log({\"train_loss\": train_loss, 'train_Acc': train_acc, 'validation_Acc': val_acc,\n               'validation_loss': val_loss, \"learning_Rate\": curr_lr})\n\n    # If you are using a scheduler in your train function within your iteration loop, you may want to log\n    # your learning rate differently\n\n    # Save model in the drive location if val_acc is better than the best recorded val_acc\n    if val_acc >= best_valacc:\n        # path = os.path.join(root, model_directory, 'checkpoint' + '.pth')\n        print(\"Saving model\")\n        torch.save({'model_state_dict': model.state_dict(),\n                    'optimizer_state_dict': optimizer.state_dict(),\n                    'scheduler_state_dict': scheduler.state_dict(),\n                    'val_acc': val_acc,\n                    'epoch': epoch}, './checkpoint.pth')\n        best_valacc = val_acc\n        torch.save({'model_state_dict': model.state_dict(),\n                    'optimizer_state_dict': optimizer.state_dict(),\n                    'scheduler_state_dict': scheduler.state_dict(),\n                    }, \"Model\")\n\n        # Creating Artifact\n        model_artifact = wandb.Artifact(config['model'], type='model')\n\n        # Adding model file to Artifact\n        model_artifact.add_file(\"Model\")\n\n        # Saving Artifact to WandB\n        run.log_artifact(model_artifact)\n        wandb.save('checkpoint.pth')\n\n# You may find it interesting to explore Wandb Artifacts to version your models\nrun.finish()\n","metadata":{"id":"EqWO8Edb0BK2","execution":{"iopub.status.busy":"2023-10-21T19:48:12.968403Z","iopub.execute_input":"2023-10-21T19:48:12.968798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Classification Task: Testing","metadata":{"id":"UpgCHImRkYQW"}},{"cell_type":"code","source":"def test(model,dataloader):\n\n    model.eval()\n    batch_bar = tqdm(total=len(dataloader), dynamic_ncols=True, position=0, leave=False, desc='Test')\n    test_results = []\n\n    for i, (images) in enumerate(dataloader):\n      # TODO: Finish predicting on the test set.\n        images = images.to(DEVICE)\n\n        with torch.inference_mode():\n            outputs = model(images)\n\n        outputs = torch.argmax(outputs, axis=1).detach().cpu().numpy().tolist()\n        test_results.extend(outputs)\n\n        batch_bar.update()\n\n    batch_bar.close()\n    return test_results","metadata":{"id":"U2WQEUjXkWvo","execution":{"iopub.status.busy":"2023-10-21T19:45:34.366213Z","iopub.execute_input":"2023-10-21T19:45:34.366585Z","iopub.status.idle":"2023-10-21T19:45:34.373417Z","shell.execute_reply.started":"2023-10-21T19:45:34.366556Z","shell.execute_reply":"2023-10-21T19:45:34.372489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_results = test(model, test_loader)","metadata":{"id":"K7R1lcCAzULc","outputId":"ed1d1147-46d3-4a6c-832f-6dbb2b0bde8b","execution":{"iopub.status.busy":"2023-10-21T19:45:37.75254Z","iopub.execute_input":"2023-10-21T19:45:37.753313Z","iopub.status.idle":"2023-10-21T19:47:32.349674Z","shell.execute_reply.started":"2023-10-21T19:45:37.753278Z","shell.execute_reply":"2023-10-21T19:47:32.348593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generate csv to submit to Kaggle","metadata":{"id":"zqfUzwS2L1gx"}},{"cell_type":"code","source":"with open(\"classification_early_submission.csv\", \"w+\") as f:\n    f.write(\"id,label\\n\")\n    for i in range(len(test_dataset)):\n        f.write(\"{},{}\\n\".format(str(i).zfill(6) + \".jpg\", test_results[i]))","metadata":{"id":"Vob9a2-HkW_V"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !kaggle competitions submit -c 11-785-f23-hw2p2-classification -f classification_early_submission.csv -m \"early submission\"","metadata":{"id":"GnRUN53CZMTf"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Verification Task: Validation","metadata":{"id":"7WYgUjJzUiGU"}},{"cell_type":"markdown","source":"The verification task consists of the following generalized scenario:\n- You are given X unknown identitites\n- You are given Y known identitites\n- Your goal is to match X unknown identities to Y known identities.\n\nWe have given you a verification dataset, that consists of 960 known identities, and 1080 unknown identities. The 1080 unknown identities are split into dev (360) and test (720). Your goal is to compare the unknown identities to the 1080 known identities and assign an identity to each image from the set of unknown identities. Some unknown identities do not have correspondence in known identities, you also need to identify these and label them with a special label n000000.\n\nYour will use/finetune your model trained for classification to compare images between known and unknown identities using a similarity metric and assign labels to the unknown identities.\n\nThis will judge your model's performance in terms of the quality of embeddings/features it generates on images/faces it has never seen during training for classification.","metadata":{"id":"FoBFFF8-Lpvj"}},{"cell_type":"code","source":"# This obtains the list of known identities from the known folder\nknown_regex = \"/kaggle/input/11-785-f23-hw2p2-verification/11-785-f23-hw2p2-verification/known/*/*\"\nknown_paths = [i.split('/')[-2] for i in sorted(glob.glob(known_regex))]\n\n# Obtain a list of images from unknown folders\nunknown_dev_regex = \"/kaggle/input/11-785-f23-hw2p2-verification/11-785-f23-hw2p2-verification/unknown_dev/*\"\nunknown_test_regex = \"/kaggle/input/11-785-f23-hw2p2-verification/11-785-f23-hw2p2-verification/unknown_test/*\"\n\n# We load the images from known and unknown folders\nunknown_dev_images = [Image.open(p) for p in tqdm(sorted(glob.glob(unknown_dev_regex)))]\nunknown_test_images = [Image.open(p) for p in tqdm(sorted(glob.glob(unknown_test_regex)))]\nknown_images = [Image.open(p) for p in tqdm(sorted(glob.glob(known_regex)))]\n\n# Why do you need only ToTensor() here?\ntransforms = torchvision.transforms.Compose([\n    torchvision.transforms.ToTensor()])\n\nunknown_dev_images = torch.stack([transforms(x) for x in unknown_dev_images])\nunknown_test_images = torch.stack([transforms(x) for x in unknown_test_images])\nknown_images  = torch.stack([transforms(y) for y in known_images ])\n#Print your shapes here to understand what we have done\n\n# You can use other similarity metrics like Euclidean Distance if you wish\nsimilarity_metric = torch.nn.CosineSimilarity(dim= 1, eps= 1e-6)","metadata":{"id":"f9aY5o-suWdn","execution":{"iopub.status.busy":"2023-10-05T16:42:47.186693Z","iopub.execute_input":"2023-10-05T16:42:47.187057Z","iopub.status.idle":"2023-10-05T16:42:57.865622Z","shell.execute_reply.started":"2023-10-05T16:42:47.187029Z","shell.execute_reply":"2023-10-05T16:42:57.864693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def eval_verification(unknown_images, known_images, model, similarity, batch_size= config['batch_size'], mode='val'):\n\n    unknown_feats, known_feats = [], []\n\n    batch_bar = tqdm(total=len(unknown_images)//batch_size, dynamic_ncols=True, position=0, leave=False, desc=mode)\n    model.eval()\n\n    # We load the images as batches for memory optimization and avoiding CUDA OOM errors\n    for i in range(0, unknown_images.shape[0], batch_size):\n        unknown_batch = unknown_images[i:i+batch_size] # Slice a given portion upto batch_size\n\n        with torch.no_grad():\n            unknown_feat = model(unknown_batch.float().to(DEVICE), return_feats=True) #Get features from model\n        unknown_feats.append(unknown_feat)\n        batch_bar.update()\n\n    batch_bar.close()\n\n    batch_bar = tqdm(total=len(known_images)//batch_size, dynamic_ncols=True, position=0, leave=False, desc=mode)\n\n    for i in range(0, known_images.shape[0], batch_size):\n        known_batch = known_images[i:i+batch_size]\n        with torch.no_grad():\n              known_feat = model(known_batch.float().to(DEVICE), return_feats=True)\n\n        known_feats.append(known_feat)\n        batch_bar.update()\n\n    batch_bar.close()\n\n    # Concatenate all the batches\n    unknown_feats = torch.cat(unknown_feats, dim=0)\n    known_feats = torch.cat(known_feats, dim=0)\n\n    similarity_values = torch.stack([similarity(unknown_feats, known_feature) for known_feature in known_feats])\n    # Print the inner list comprehension in a separate cell - what is really happening?\n\n    max_similarity_values, predictions = similarity_values.max(0) #Why are we doing an max here, where are the return values?\n    max_similarity_values, predictions = max_similarity_values.cpu().numpy(), predictions.cpu().numpy()\n\n\n    # Note that in unknown identities, there are identities without correspondence in known identities.\n    # Therefore, these identities should be not similar to all the known identities, i.e. max similarity will be below a certain\n    # threshold compared with those identities with correspondence.\n\n    # In early submission, you can ignore identities without correspondence, simply taking identity with max similarity value\n    pred_id_strings = [known_paths[i] for i in predictions] # Map argmax indices to identity strings\n\n    # After early submission, remove the previous line and uncomment the following code\n\n    # threshold = # Choose a proper threshold\n    # NO_CORRESPONDENCE_LABEL = 'n000000'\n    # pred_id_strings = []\n    # for idx, prediction in enumerate(predictions):\n    #     if max_similarity_values[idx] < threshold: # why < ? Thank about what is your similarity metric\n    #         pred_id_strings.append(NO_CORRESPONDENCE_LABEL)\n    #     else:\n    #         pred_id_strings.append(known_paths[prediction])\n\n    if mode == 'val':\n        true_ids = pd.read_csv('/kaggle/input/11-785-f23-hw2p2-verification/11-785-f23-hw2p2-verification/verification_dev.csv')['label'].tolist()\n        accuracy = accuracy_score(pred_id_strings, true_ids)\n        print(\"Verification Accuracy = {}\".format(accuracy))\n\n    return pred_id_strings","metadata":{"id":"rk1LS0BRxFHM","outputId":"ad9694ee-1075-4709-b50d-1ce6a049dc14","execution":{"iopub.status.busy":"2023-10-05T16:43:00.351797Z","iopub.execute_input":"2023-10-05T16:43:00.352163Z","iopub.status.idle":"2023-10-05T16:43:00.361799Z","shell.execute_reply.started":"2023-10-05T16:43:00.352135Z","shell.execute_reply":"2023-10-05T16:43:00.360798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# verification eval\npred_id_strings = eval_verification(unknown_dev_images, known_images, model, similarity_metric, config['batch_size'], mode='val')\n# verification test\npred_id_strings = eval_verification(unknown_test_images, known_images, model, similarity_metric, config['batch_size'], mode='test')","metadata":{"id":"zMC7FacaUnJ7","execution":{"iopub.status.busy":"2023-10-05T16:43:15.642357Z","iopub.execute_input":"2023-10-05T16:43:15.642961Z","iopub.status.idle":"2023-10-05T16:43:16.608398Z","shell.execute_reply.started":"2023-10-05T16:43:15.642932Z","shell.execute_reply":"2023-10-05T16:43:16.607471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# add your finetune/retrain code here","metadata":{"id":"1bCmzKfH7XgG"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generate csv to submit to Kaggle","metadata":{"id":"iTLW0RPD7XGC"}},{"cell_type":"code","source":"with open(\"verification_early_submission.csv\", \"w+\") as f:\n    f.write(\"id,label\\n\")\n    for i in range(len(pred_id_strings)):\n        f.write(\"{},{}\\n\".format(i, pred_id_strings[i]))","metadata":{"id":"fD-r-HmsAeWV","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !kaggle competitions submit -c 11-785-f23-hw2p2-verification -f verification_early_submission.csv -m \"early submission\"","metadata":{"id":"jPIgq0tMZ8qk"},"execution_count":null,"outputs":[]}]}