{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Dog Breed Identification (ImageNet Dogs) on Kaggle\n\nIn this section, we will practice\nthe dog breed identification problem on\nKaggle. (**The web address of this competition is https://www.kaggle.com/c/dog-breed-identification**)\n\nIn this competition,\n120 different breeds of dogs will be recognized.\nIn fact,\nthe dataset for this competition is\na subset of the ImageNet dataset.\nUnlike the images in the CIFAR-10 dataset in :numref:`sec_kaggle_cifar10`,\nthe images in the ImageNet dataset are both higher and wider in varying dimensions.\n:numref:`fig_kaggle_dog` shows the information on the competition's webpage. You need a Kaggle account\nto submit your results.\n\n\n![The dog breed identification competition website. The competition dataset can be obtained by clicking the \"Data\" tab.](../img/kaggle-dog.jpg)\n:width:`400px`\n:label:`fig_kaggle_dog`\n","metadata":{"origin_pos":0}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2023-03-02T01:38:24.126963Z","iopub.execute_input":"2023-03-02T01:38:24.127332Z","iopub.status.idle":"2023-03-02T01:38:39.743128Z","shell.execute_reply.started":"2023-03-02T01:38:24.127231Z","shell.execute_reply":"2023-03-02T01:38:39.742291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gym\n%pip install d2l==1.0.0b0","metadata":{"execution":{"iopub.status.busy":"2023-03-02T01:39:21.753051Z","iopub.execute_input":"2023-03-02T01:39:21.753418Z","iopub.status.idle":"2023-03-02T01:39:36.464100Z","shell.execute_reply.started":"2023-03-02T01:39:21.753386Z","shell.execute_reply":"2023-03-02T01:39:36.462959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport torch\nimport torchvision\nfrom torch import nn\nfrom d2l import torch as d2l","metadata":{"origin_pos":2,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:39:56.090626Z","iopub.execute_input":"2023-03-02T01:39:56.091005Z","iopub.status.idle":"2023-03-02T01:39:58.397781Z","shell.execute_reply.started":"2023-03-02T01:39:56.090969Z","shell.execute_reply":"2023-03-02T01:39:58.396646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Obtaining and Organizing the Dataset\n\nThe competition dataset is divided into a training set and a test set, which contain 10222 and 10357 JPEG images\nof three RGB (color) channels, respectively.\nAmong the training dataset,\nthere are 120 breeds of dogs\nsuch as Labradors, Poodles, Dachshunds, Samoyeds, Huskies, Chihuahuas, and Yorkshire Terriers.\n\n\n### Downloading the Dataset\n\nAfter logging into Kaggle,\nyou can click on the \"Data\" tab on the\ncompetition webpage shown in :numref:`fig_kaggle_dog` and download the dataset by clicking the \"Download All\" button.\nAfter unzipping the downloaded file in `../data`, you will find the entire dataset in the following paths:\n\n* ../data/dog-breed-identification/labels.csv\n* ../data/dog-breed-identification/sample_submission.csv\n* ../data/dog-breed-identification/train\n* ../data/dog-breed-identification/test\n\nYou may have noticed that the above structure is\nsimilar to that of the CIFAR-10 competition in :numref:`sec_kaggle_cifar10`, where folders `train/` and `test/` contain training and testing dog images, respectively, and `labels.csv` contains\nthe labels for the training images.\n","metadata":{"origin_pos":3}},{"cell_type":"markdown","source":"The data is in the input folder and is read-only. To manipulate the data, we first copy it to a working folder.","metadata":{}},{"cell_type":"code","source":"!cp -r '/kaggle/input/dog-breed-identification' '/kaggle/working/dog-breed-identification'","metadata":{"execution":{"iopub.status.busy":"2023-03-02T01:40:01.249290Z","iopub.execute_input":"2023-03-02T01:40:01.249867Z","iopub.status.idle":"2023-03-02T01:41:48.703082Z","shell.execute_reply.started":"2023-03-02T01:40:01.249834Z","shell.execute_reply":"2023-03-02T01:41:48.701723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From now on we set the directory as the working directory.","metadata":{}},{"cell_type":"code","source":"data_dir='/kaggle/working/dog-breed-identification'","metadata":{"execution":{"iopub.status.busy":"2023-03-02T01:42:50.473823Z","iopub.execute_input":"2023-03-02T01:42:50.474231Z","iopub.status.idle":"2023-03-02T01:42:50.479903Z","shell.execute_reply.started":"2023-03-02T01:42:50.474194Z","shell.execute_reply":"2023-03-02T01:42:50.478922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [**Organizing the Dataset**]\n\nWe can organize the dataset similarly to what we did in :numref:`sec_kaggle_cifar10`, namely splitting out\na validation set from the original training set, and moving images into subfolders grouped by labels.\n\nThe `reorg_dog_data` function below reads\nthe training data labels, splits out the validation set, and organizes the training set.\n","metadata":{"origin_pos":5}},{"cell_type":"code","source":"def reorg_dog_data(data_dir, valid_ratio):\n    labels = d2l.read_csv_labels(os.path.join(data_dir, 'labels.csv'))\n    d2l.reorg_train_valid(data_dir, labels, valid_ratio)\n    d2l.reorg_test(data_dir)\n\n\nbatch_size = 128\nvalid_ratio = 0.1\nreorg_dog_data(data_dir, valid_ratio)","metadata":{"origin_pos":6,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:42:57.618351Z","iopub.execute_input":"2023-03-02T01:42:57.618743Z","iopub.status.idle":"2023-03-02T01:43:02.084223Z","shell.execute_reply.started":"2023-03-02T01:42:57.618712Z","shell.execute_reply":"2023-03-02T01:43:02.083205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [**Image Augmentation**]\n\nRecall that this dog breed dataset\nis a subset of the ImageNet dataset,\nwhose images\nare larger than those of the CIFAR-10 dataset\nin :numref:`sec_kaggle_cifar10`.\nThe following\nlists a few image augmentation operations\nthat might be useful for relatively larger images.\n","metadata":{"origin_pos":7}},{"cell_type":"code","source":"transform_train = torchvision.transforms.Compose([\n    torchvision.transforms.RandomResizedCrop(224, scale=(0.08, 1.0),\n                                             ratio=(3.0/4.0, 4.0/3.0)),\n    torchvision.transforms.RandomHorizontalFlip(),\n    torchvision.transforms.ColorJitter(brightness=0.4,\n                                       contrast=0.4,\n                                       saturation=0.4),\n    torchvision.transforms.ToTensor(),\n    torchvision.transforms.Normalize([0.485, 0.456, 0.406],\n                                     [0.229, 0.224, 0.225])])","metadata":{"origin_pos":9,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:43:07.304512Z","iopub.execute_input":"2023-03-02T01:43:07.304876Z","iopub.status.idle":"2023-03-02T01:43:07.311334Z","shell.execute_reply.started":"2023-03-02T01:43:07.304843Z","shell.execute_reply":"2023-03-02T01:43:07.310307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"During prediction,\nwe only use image preprocessing operations\nwithout randomness.\n","metadata":{"origin_pos":10}},{"cell_type":"code","source":"transform_test = torchvision.transforms.Compose([\n    torchvision.transforms.Resize(256),\n    torchvision.transforms.CenterCrop(224),\n    torchvision.transforms.ToTensor(),\n    torchvision.transforms.Normalize([0.485, 0.456, 0.406],\n                                     [0.229, 0.224, 0.225])])","metadata":{"origin_pos":12,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:43:11.137175Z","iopub.execute_input":"2023-03-02T01:43:11.137874Z","iopub.status.idle":"2023-03-02T01:43:11.143497Z","shell.execute_reply.started":"2023-03-02T01:43:11.137838Z","shell.execute_reply":"2023-03-02T01:43:11.142541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [**Reading the Dataset**]\n\nAs in :numref:`sec_kaggle_cifar10`,\nwe can read the organized dataset\nconsisting of raw image files.\n","metadata":{"origin_pos":13}},{"cell_type":"code","source":"train_ds, train_valid_ds = [torchvision.datasets.ImageFolder(\n    os.path.join(data_dir, 'train_valid_test', folder),\n    transform=transform_train) for folder in ['train', 'train_valid']]\n\nvalid_ds, test_ds = [torchvision.datasets.ImageFolder(\n    os.path.join(data_dir, 'train_valid_test', folder),\n    transform=transform_test) for folder in ['valid', 'test']]","metadata":{"origin_pos":15,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:44:42.236090Z","iopub.execute_input":"2023-03-02T01:44:42.236792Z","iopub.status.idle":"2023-03-02T01:44:42.374702Z","shell.execute_reply.started":"2023-03-02T01:44:42.236757Z","shell.execute_reply":"2023-03-02T01:44:42.373664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below we create data iterator instances\nthe same way\nas in :numref:`sec_kaggle_cifar10`.\n","metadata":{"origin_pos":16}},{"cell_type":"code","source":"train_iter, train_valid_iter = [torch.utils.data.DataLoader(\n    dataset, batch_size, shuffle=True, drop_last=True)\n    for dataset in (train_ds, train_valid_ds)]\n\nvalid_iter = torch.utils.data.DataLoader(valid_ds, batch_size, shuffle=False,\n                                         drop_last=True)\n\ntest_iter = torch.utils.data.DataLoader(test_ds, batch_size, shuffle=False,\n                                        drop_last=False)","metadata":{"origin_pos":18,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:44:47.915505Z","iopub.execute_input":"2023-03-02T01:44:47.915868Z","iopub.status.idle":"2023-03-02T01:44:47.922073Z","shell.execute_reply.started":"2023-03-02T01:44:47.915838Z","shell.execute_reply":"2023-03-02T01:44:47.921097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [**Fine-Tuning a Pretrained Model**]\n\nAgain,\nthe dataset for this competition is a subset of the ImageNet dataset.\nTherefore, we can use the approach discussed in\n:numref:`sec_fine_tuning`\nto select a model pretrained on the\nfull ImageNet dataset and use it to extract image features to be fed into a\ncustom small-scale output network.\nHigh-level APIs of deep learning frameworks\nprovide a wide range of models\npretrained on the ImageNet dataset.\nHere, we choose\na pretrained ResNet-34 model,\nwhere we simply reuse\nthe input of this model's output layer\n(i.e., the extracted\nfeatures).\nThen we can replace the original output layer with a small custom\noutput network that can be trained,\nsuch as stacking two\nfully connected layers.\nDifferent from the experiment in\n:numref:`sec_fine_tuning`,\nthe following does\nnot retrain the pretrained model used for feature\nextraction. This reduces training time and\nmemory for storing gradients.\n\nRecall that we\nstandardized images using\nthe means and standard deviations of the three RGB channels for the full ImageNet dataset.\nIn fact,\nthis is also consistent with the standardization operation\nby the pretrained model on ImageNet.\n","metadata":{"origin_pos":19}},{"cell_type":"code","source":"def get_net(devices):\n    finetune_net = nn.Sequential()\n    finetune_net.features = torchvision.models.resnet34(pretrained=True)\n    finetune_net.output_new = nn.Sequential(nn.Linear(1000, 256),\n                                            nn.ReLU(),\n                                            nn.Linear(256, 120))\n    finetune_net = finetune_net.to(devices[0])\n    for param in finetune_net.features.parameters():\n        param.requires_grad = False\n    return finetune_net","metadata":{"origin_pos":21,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:45:21.109022Z","iopub.execute_input":"2023-03-02T01:45:21.109417Z","iopub.status.idle":"2023-03-02T01:45:21.115808Z","shell.execute_reply.started":"2023-03-02T01:45:21.109385Z","shell.execute_reply":"2023-03-02T01:45:21.114817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before [**calculating the loss**],\nwe first obtain the input of the pretrained model's output layer, i.e., the extracted feature.\nThen we use this feature as input for our small custom output network to calculate the loss.\n","metadata":{"origin_pos":22}},{"cell_type":"code","source":"loss = nn.CrossEntropyLoss(reduction='none')\n\ndef evaluate_loss(data_iter, net, devices):\n    l_sum, n = 0.0, 0\n    for features, labels in data_iter:\n        features, labels = features.to(devices[0]), labels.to(devices[0])\n        outputs = net(features)\n        l = loss(outputs, labels)\n        l_sum += l.sum()\n        n += labels.numel()\n    return l_sum / n","metadata":{"origin_pos":24,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:45:36.408056Z","iopub.execute_input":"2023-03-02T01:45:36.408454Z","iopub.status.idle":"2023-03-02T01:45:36.417745Z","shell.execute_reply.started":"2023-03-02T01:45:36.408420Z","shell.execute_reply":"2023-03-02T01:45:36.416763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Defining [**the Training Function**]\n\nWe will select the model and tune hyperparameters according to the model's performance on the validation set. The model training function `train` only\niterates parameters of the small custom output network.\n","metadata":{"origin_pos":25}},{"cell_type":"code","source":"def train(net, train_iter, valid_iter, num_epochs, lr, wd, devices, lr_period,\n          lr_decay):\n    net = nn.DataParallel(net, device_ids=devices).to(devices[0])\n    trainer = torch.optim.SGD((param for param in net.parameters()\n                               if param.requires_grad), lr=lr,\n                              momentum=0.9, weight_decay=wd)\n    scheduler = torch.optim.lr_scheduler.StepLR(trainer, lr_period, lr_decay)\n    num_batches, timer = len(train_iter), d2l.Timer()\n    legend = ['train loss']\n    if valid_iter is not None:\n        legend.append('valid loss')\n    animator = d2l.Animator(xlabel='epoch', xlim=[1, num_epochs],\n                            legend=legend)\n    for epoch in range(num_epochs):\n        metric = d2l.Accumulator(2)\n        for i, (features, labels) in enumerate(train_iter):\n            timer.start()\n            features, labels = features.to(devices[0]), labels.to(devices[0])\n            trainer.zero_grad()\n            output = net(features)\n            l = loss(output, labels).sum()\n            l.backward()\n            trainer.step()\n            metric.add(l, labels.shape[0])\n            timer.stop()\n            if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:\n                animator.add(epoch + (i + 1) / num_batches,\n                             (metric[0] / metric[1], None))\n        measures = f'train loss {metric[0] / metric[1]:.3f}'\n        if valid_iter is not None:\n            valid_loss = evaluate_loss(valid_iter, net, devices)\n            animator.add(epoch + 1, (None, valid_loss.detach().cpu()))\n        scheduler.step()\n    if valid_iter is not None:\n        measures += f', valid loss {valid_loss:.3f}'\n    print(measures + f'\\n{metric[1] * num_epochs / timer.sum():.1f}'\n          f' examples/sec on {str(devices)}')","metadata":{"origin_pos":27,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:45:56.640204Z","iopub.execute_input":"2023-03-02T01:45:56.640613Z","iopub.status.idle":"2023-03-02T01:45:56.652851Z","shell.execute_reply.started":"2023-03-02T01:45:56.640580Z","shell.execute_reply":"2023-03-02T01:45:56.651766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [**Training and Validating the Model**]\n\nNow we can train and validate the model.\nThe following hyperparameters are all tunable.\nFor example, the number of epochs can be increased. Because `lr_period` and `lr_decay` are set to 2 and 0.9, respectively, the learning rate of the optimization algorithm will be multiplied by 0.9 after every 2 epochs.\n","metadata":{"origin_pos":28}},{"cell_type":"code","source":"devices, num_epochs, lr, wd = d2l.try_all_gpus(), 10, 1e-4, 1e-4\nlr_period, lr_decay, net = 2, 0.9, get_net(devices)\ntrain(net, train_iter, valid_iter, num_epochs, lr, wd, devices, lr_period,\n      lr_decay)","metadata":{"origin_pos":30,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T01:46:07.726055Z","iopub.execute_input":"2023-03-02T01:46:07.726431Z","iopub.status.idle":"2023-03-02T02:00:12.383842Z","shell.execute_reply.started":"2023-03-02T01:46:07.726396Z","shell.execute_reply":"2023-03-02T02:00:12.382821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [**Classifying the Testing Set**] and Submitting Results on Kaggle\n\n\nSimilar to the final step in :numref:`sec_kaggle_cifar10`,\nin the end all the labeled data (including the validation set) are used for training the model and classifying the testing set.\nWe will use the trained custom output network\nfor classification.\n","metadata":{"origin_pos":31}},{"cell_type":"code","source":"net = get_net(devices)\ntrain(net, train_valid_iter, None, num_epochs, lr, wd, devices, lr_period,\n      lr_decay)\n\npreds = []\nfor data, label in test_iter:\n    output = torch.nn.functional.softmax(net(data.to(devices[0])), dim=1)\n    preds.extend(output.cpu().detach().numpy())\nids = sorted(os.listdir(\n    os.path.join(data_dir, 'train_valid_test', 'test', 'unknown')))\nwith open('submission.csv', 'w') as f:\n    f.write('id,' + ','.join(train_valid_ds.classes) + '\\n')\n    for i, output in zip(ids, preds):\n        f.write(i.split('.')[0] + ',' + ','.join(\n            [str(num) for num in output]) + '\\n')","metadata":{"origin_pos":33,"tab":["pytorch"],"execution":{"iopub.status.busy":"2023-03-02T02:11:50.399758Z","iopub.execute_input":"2023-03-02T02:11:50.400126Z","iopub.status.idle":"2023-03-02T02:27:03.907743Z","shell.execute_reply.started":"2023-03-02T02:11:50.400094Z","shell.execute_reply":"2023-03-02T02:27:03.906642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above code\nwill generate a `submission.csv` file\nto be submitted\nto Kaggle in the same way described in :numref:`sec_kaggle_house`.\n\n\n## Summary\n\n\n* Images in the ImageNet dataset are larger (with varying dimensions) than CIFAR-10 images. We may modify image augmentation operations for tasks on a different dataset.\n* To classify a subset of the ImageNet dataset, we can leverage pre-trained models on the full ImageNet dataset to extract features and only train a custom small-scale output network. This will lead to less computational time and memory cost.\n\n","metadata":{"origin_pos":34}},{"cell_type":"markdown","source":"[Discussions](https://discuss.d2l.ai/t/1481)\n","metadata":{"origin_pos":36,"tab":["pytorch"]}}]}