{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3649,"databundleVersionId":46718,"sourceType":"competition"}],"dockerImageVersionId":30636,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Kaggle Link: https://www.kaggle.com/lorenzozanolin/object-recognition-zanolin\n        \nW&B: https://wandb.ai/lorenzozanolin-52/object_recognition/table?workspace=user-lorenzozanolin-52","metadata":{}},{"cell_type":"code","source":"#This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\nfrom tqdm.notebook import trange, tqdm\nimport wandb\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"a\")\nwandb.login(key=secret_value_0)\nwandb.init(project='object_recognition', save_code=True)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2024-01-22T21:32:35.607358Z","iopub.execute_input":"2024-01-22T21:32:35.608002Z","iopub.status.idle":"2024-01-22T21:33:12.863842Z","shell.execute_reply.started":"2024-01-22T21:32:35.607959Z","shell.execute_reply":"2024-01-22T21:33:12.862589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import everything needed","metadata":{}},{"cell_type":"code","source":"import glob\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport collections\nimport math\nimport os\nimport shutil\nfrom sklearn.model_selection import train_test_split\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader, Dataset\nimport torchvision\nfrom torchvision import datasets, transforms\nnp.random.seed(0)\ntorch.manual_seed(0)\ntorch.cuda.manual_seed(0)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:33:12.866251Z","iopub.execute_input":"2024-01-22T21:33:12.866658Z","iopub.status.idle":"2024-01-22T21:33:12.875415Z","shell.execute_reply.started":"2024-01-22T21:33:12.866623Z","shell.execute_reply":"2024-01-22T21:33:12.874436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Unzip datasets","metadata":{}},{"cell_type":"code","source":"!pip install py7zr","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:33:12.877035Z","iopub.execute_input":"2024-01-22T21:33:12.877748Z","iopub.status.idle":"2024-01-22T21:33:25.361509Z","shell.execute_reply.started":"2024-01-22T21:33:12.877718Z","shell.execute_reply":"2024-01-22T21:33:25.360315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# WARNING: It can take a lot of time to uncompress!","metadata":{}},{"cell_type":"code","source":"!python -m py7zr x /kaggle/input/cifar-10/train.7z","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:33:25.365488Z","iopub.execute_input":"2024-01-22T21:33:25.366437Z","iopub.status.idle":"2024-01-22T21:34:25.121589Z","shell.execute_reply.started":"2024-01-22T21:33:25.366375Z","shell.execute_reply":"2024-01-22T21:34:25.120340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python -m py7zr x /kaggle/input/cifar-10/test.7z","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:34:25.123053Z","iopub.execute_input":"2024-01-22T21:34:25.123406Z","iopub.status.idle":"2024-01-22T21:54:03.129592Z","shell.execute_reply.started":"2024-01-22T21:34:25.123356Z","shell.execute_reply":"2024-01-22T21:54:03.128307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_dir = '/kaggle/working/'","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:54:03.131265Z","iopub.execute_input":"2024-01-22T21:54:03.131658Z","iopub.status.idle":"2024-01-22T21:54:03.136936Z","shell.execute_reply.started":"2024-01-22T21:54:03.131624Z","shell.execute_reply":"2024-01-22T21:54:03.136059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_csv_labels(fname):\n    \"\"\"Read `fname` to return a filename to label dictionary.\"\"\"\n    with open(fname, 'r') as f:\n        # Skip the file header line (column name)\n        lines = f.readlines()[1:]\n    tokens = [l.rstrip().split(',') for l in lines]\n    return dict(((name, label) for name, label in tokens))\n\nlabels = read_csv_labels(os.path.join(data_dir, '/kaggle/input/cifar-10/trainLabels.csv'))\nprint(f'Number training examples: {len(labels)}')\nprint(f'Number classes: {len(set(labels.values()))}')","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:54:03.138006Z","iopub.execute_input":"2024-01-22T21:54:03.138265Z","iopub.status.idle":"2024-01-22T21:54:03.403977Z","shell.execute_reply.started":"2024-01-22T21:54:03.138243Z","shell.execute_reply":"2024-01-22T21:54:03.403058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def copyfile(filename, target_dir):\n    \"\"\"Copy a file into a target directory.\"\"\"\n    os.makedirs(target_dir, exist_ok=True)\n    shutil.copy(filename, target_dir)\n\ndef reorg_train_valid(data_dir, labels, valid_ratio):\n    \"\"\"Split the validation set out of the original training set.\"\"\"\n    # The number of examples of the class that has the fewest examples in the\n    # training dataset\n    n = collections.Counter(labels.values()).most_common()[-1][1]\n    # The number of examples per class for the validation set\n    n_valid_per_label = max(1, math.floor(n * valid_ratio))\n    label_count = {}\n    for train_file in os.listdir(os.path.join(data_dir, 'train')):\n        label = labels[train_file.split('.')[0]]\n        fname = os.path.join(data_dir, 'train', train_file)\n        copyfile(fname, os.path.join(data_dir, 'train_valid_test',\n                                     'train_valid', label))\n        if label not in label_count or label_count[label] < n_valid_per_label:\n            copyfile(fname, os.path.join(data_dir, 'train_valid_test',\n                                         'valid', label))\n            label_count[label] = label_count.get(label, 0) + 1\n        else:\n            copyfile(fname, os.path.join(data_dir, 'train_valid_test',\n                                         'train', label))\n    return n_valid_per_label\n\ndef reorg_test(data_dir):\n    \"\"\"Organize the testing set for data loading during prediction.\"\"\"\n    for test_file in os.listdir(os.path.join(data_dir, 'test')):\n        copyfile(os.path.join(data_dir, 'test', test_file),\n                 os.path.join(data_dir, 'train_valid_test', 'test',\n                              'unknown'))\n        \ndef reorg_cifar10_data(data_dir, valid_ratio):\n    labels = read_csv_labels('/kaggle/input/cifar-10/trainLabels.csv')\n    reorg_train_valid(data_dir, labels, valid_ratio)\n    reorg_test(data_dir)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:54:03.405293Z","iopub.execute_input":"2024-01-22T21:54:03.405679Z","iopub.status.idle":"2024-01-22T21:54:03.417227Z","shell.execute_reply.started":"2024-01-22T21:54:03.405645Z","shell.execute_reply":"2024-01-22T21:54:03.416370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 64\nvalid_ratio = 0.1\nreorg_cifar10_data(data_dir, valid_ratio)\nwandb.log({'batch_size': batch_size})","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:54:03.418464Z","iopub.execute_input":"2024-01-22T21:54:03.418799Z","iopub.status.idle":"2024-01-22T21:55:17.885492Z","shell.execute_reply.started":"2024-01-22T21:54:03.418767Z","shell.execute_reply":"2024-01-22T21:55:17.884435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Transformations\nWe will do some Image Augmentation:","metadata":{}},{"cell_type":"code","source":"transform_train = torchvision.transforms.Compose([\n    transforms.RandomCrop(32, padding=4),  #Random crop can help to make the NN invariant to the position of the element\n    transforms.RandomHorizontalFlip(),  #we will use horizontal flip\n    torchvision.transforms.ToTensor(), #transform the image into a tensor\n    torchvision.transforms.Normalize([0.4914, 0.4822, 0.4465],  #each channel (3) will be normalized using the mean (first tuple) and the std (second tuple)\n                                     [0.2023, 0.1994, 0.2010])\n])\n\ntransform_test = torchvision.transforms.Compose([  #since it is the dataset, we will not do data augmentation, other than normalization\n    torchvision.transforms.ToTensor(),\n    torchvision.transforms.Normalize([0.4914, 0.4822, 0.4465], #each channel (3) will be normalized using the mean (first tuple) and the std (second tuple)\n                                     [0.2023, 0.1994, 0.2010])]) ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-01-22T21:55:17.889261Z","iopub.execute_input":"2024-01-22T21:55:17.889939Z","iopub.status.idle":"2024-01-22T21:55:17.896406Z","shell.execute_reply.started":"2024-01-22T21:55:17.889904Z","shell.execute_reply":"2024-01-22T21:55:17.895554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds, train_valid_ds = [torchvision.datasets.ImageFolder(\n    os.path.join(data_dir, 'train_valid_test', folder),\n    transform=transform_train) for folder in ['train', 'train_valid']]\n\nvalid_ds, test_ds = [torchvision.datasets.ImageFolder(\n    os.path.join(data_dir, 'train_valid_test', folder),\n    transform=transform_test) for folder in ['valid', 'test']]\n\ntrain_iter, train_valid_iter = [torch.utils.data.DataLoader(\n    dataset, batch_size, shuffle=True, drop_last=True)\n    for dataset in (train_ds, train_valid_ds)]\n\nvalid_iter = torch.utils.data.DataLoader(valid_ds, batch_size, shuffle=False,\n                                         drop_last=True)\n\ntest_iter = torch.utils.data.DataLoader(test_ds, batch_size, shuffle=False,\n                                        drop_last=False)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T21:55:17.897731Z","iopub.execute_input":"2024-01-22T21:55:17.897997Z","iopub.status.idle":"2024-01-22T21:55:19.827178Z","shell.execute_reply.started":"2024-01-22T21:55:17.897975Z","shell.execute_reply":"2024-01-22T21:55:19.826399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ResNet18 Fine Tuning\nWe will now use transfer learning on a pretrained net (ResNet18) modifying the last layer to predict the classes of the images.","metadata":{}},{"cell_type":"code","source":"net = torchvision.models.resnet18(pretrained=True)  #load the net with pretrained weigths\nnet.fc = nn.Linear(net.fc.in_features, 10)  #modify the fc layer to have an output of 10 classes\nnn.init.xavier_normal_(net.fc.weight)  #random initialization of the weights \nnn.init.constant_(net.fc.bias, 0);  # bias\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'  #we want to move the net on the GPU\nnet = net.to(device)\nif device == 'cuda':\n    net = torch.nn.DataParallel(net) # if multiple GPUs use them","metadata":{"execution":{"iopub.status.busy":"2024-01-22T22:41:34.701433Z","iopub.execute_input":"2024-01-22T22:41:34.701827Z","iopub.status.idle":"2024-01-22T22:41:34.920328Z","shell.execute_reply.started":"2024-01-22T22:41:34.701788Z","shell.execute_reply":"2024-01-22T22:41:34.919436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will use:\n- Cross Entropy as the $Loss function$\n- Adam with Reduced LR as $Optimizer$","metadata":{}},{"cell_type":"code","source":"lr = 1e-4\nwandb.log({'optimizer': 'Adam'})\nwandb.log({'lr': lr})\ncriterion = nn.CrossEntropyLoss(label_smoothing=0.1)\noptimizer = torch.optim.AdamW(net.parameters(), lr=lr, weight_decay=0.0001)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', patience=3, verbose=True, min_lr=1e-5, factor=0.5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us train the net!","metadata":{}},{"cell_type":"code","source":"# we will train for 10 epochs\nepochs = 20\nwandb.log({'epochs': epochs})\n\nfor epoch in trange(epochs):\n    mean_train_losses = []\n    mean_valid_losses = []\n    valid_acc_list = []\n    train_losses = []\n    valid_losses = []\n    total = 0\n    val_acc = 0\n    # TRAINING\n    for X, y in tqdm(train_iter):\n        tr_loss = 0\n        X = X.to(device)  #move data on the GPU\n        y = y.to(device)\n        y_pred = net(X)  #calculate the value using the net, y_pred in this case will be an array of 10 values (probability for each class)\n        loss = criterion(y_pred, y)  # calculate the loss wrt the ground truth\n        \n        # zero the gradients before running\n        # the backward pass.\n        optimizer.zero_grad()\n\n        # Backward pass to compute the gradient\n        # of loss w.r.t our learnable params. \n        loss.backward()\n\n        # Update params\n        optimizer.step()\n        \n        train_losses.append(loss.item())\n        \n    # VALIDATION\n    with torch.no_grad():\n        net.eval()  # put network in train mode for Dropout and Batch Normalization\n        \n        for X, y in tqdm(valid_iter):\n            X = X.to(device)\n            y = y.to(device)\n            y_pred = net(X)\n            loss = criterion(y_pred, y)\n            _, predicted = torch.max(y_pred, 1)\n            val_acc += (y == predicted).sum().float()\n            valid_losses.append(loss.item())\n            total += len(y)\n\n    mean_train_losses.append(np.mean(train_losses))\n    mean_valid_losses.append(np.mean(valid_losses))\n    \n    accuracy = 100*val_acc/total\n    valid_acc_list.append(accuracy)\n    \n    print('epoch : {}, train loss : {:.4f}, valid loss : {:.4f}, valid acc : {:.2f}%'\\\n         .format(epoch+1, np.mean(train_losses), np.mean(valid_losses), accuracy))\n    \n    wandb.log({'train_loss':mean_train_losses[-1]})\n    wandb.log({'val_loss':mean_valid_losses[-1]})\n    wandb.log({'val_accuracy':valid_acc_list[-1]})","metadata":{"execution":{"iopub.status.busy":"2024-01-22T22:41:39.762563Z","iopub.execute_input":"2024-01-22T22:41:39.763590Z","iopub.status.idle":"2024-01-22T23:02:04.286895Z","shell.execute_reply.started":"2024-01-22T22:41:39.763556Z","shell.execute_reply":"2024-01-22T23:02:04.285743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submission generation\nWe will generate the submission.csv file based on the test set","metadata":{}},{"cell_type":"code","source":"preds = []\n\nnet.eval()\nwith torch.no_grad():\n    for X, _ in test_iter:\n        X = X.to(device)\n        preds.extend(net(X).argmax(dim=1).type(torch.int32).cpu().numpy())\nids = list(range(1, len(test_ds)+1))\nids.sort(key=lambda x: str(x))\ndf = pd.DataFrame({'id': ids, 'label': preds})\ndf['label'] = df['label'].apply(lambda x: train_ds.classes[x])\ndf.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T22:10:14.010163Z","iopub.execute_input":"2024-01-22T22:10:14.010454Z","iopub.status.idle":"2024-01-22T22:13:53.861256Z","shell.execute_reply.started":"2024-01-22T22:10:14.010429Z","shell.execute_reply":"2024-01-22T22:13:53.860236Z"},"trusted":true},"execution_count":null,"outputs":[]}]}