{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note:** It is easier to redownload the dataset, then figure out how to extract .7z files in Kaggle**","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Dowload the dataset\nfrom torchvision.datasets.utils import download_url\ndataset_url = \"http://files.fast.ai/data/cifar10.tgz\"\ndownload_url(dataset_url, '.')\nimport tarfile\n# Extract from archive\nwith tarfile.open('./cifar10.tgz', 'r:gz') as tar:\n    tar.extractall(path='./data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls data/cifar10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = pd.read_csv('/kaggle/input/cifar-10/trainLabels.csv')\nlabels.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Are Datasets these days this Unbiased?\n\nThe CIFAR-10 Dataset has *equal number* of labels for each class (ie, a balanced dataset), which is often good for training :). The number of images for each label (a.k.a. class) are printed below. We see there are 5000 images for each of the 10 classes.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"labels['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading Datasets in PyTorch","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import tqdm\nimport torch\nimport torchvision\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torchvision.datasets import ImageFolder\nfrom torchvision.transforms import ToTensor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = ImageFolder('./data/cifar10/train', transform=ToTensor())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split data into Train + Validation\n\n- An **80 : 20** split seems fair to start with. This means 80% of the data will be used for training, while 20% will be kept aside and used to determine how well the model performs on unseen data. \n    \n    It is important to keep training our model as long as both the training and validation loss decrease (this prevents underfitting). When we see that our validation loss starts increasing after a certain point, while training loss decreases - we must STOP training immediately. This prevents overfitting and enables our model to generalize well on unseen data.  \n\n\n- You may also choose splits of 60:40 or 90:10 or <*insert favorite_number : 100 - fav_num*>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> ### Calculating Split Sizes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val_split = 0.20\nval_size = int(val_split * len(dataset))\ntrain_size = int(len(dataset) - val_size)\nprint(train_size, val_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Performing the Stratified Split\n\nIn the previous cell, we calculated the sizes of our train and validation splits. Now we actually split the dataset.\n\nIt would make sense to ensure our train and validation splits have the same class proportions as the original balanced dataset. Stratified Splitting does exactly this!\n\nImagine what would happen during a random split where **nearly all images of a particular class** got into your validation set. Since, no image from that class would be in the train set, your model would almost surely have a huge validation error - and you'd be left wondering if theres a bug in your code xP","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Collect Indices and their targets(\"labels\"). \n# We later feed this to sklearn.train_test_split ready-made function to get the indices to be put in the train_set and those in the validation_set\nindices = np.arange(len(dataset))\ntargets = np.array(dataset.targets)\nindices.shape, targets.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ntrain_idx, val_idx, train_labels, val_labels = train_test_split(indices, targets, test_size=val_size, stratify=targets)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see our 80:20 stratified split ensures there are 4000 train images per class and 1000 validation images per class.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Train Split: \\n\", pd.Series(train_labels).value_counts())\nprint(\"Validation Split: \\n\", pd.Series(val_labels).value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Sampling the Train and Validation set from Torch.dataset using Indices\n\nIn the previous section, we generated `train_idx` and `val_idx` as arrays containing the train indices and validation indices from the whole dataset. We now use these indices to retrieve the actual train_data and val_data from the torch.dataset (CIFAR 10 read by Torch API) with the help of the `SubsetRandomSampler`.\n\nWe set the batch_size to 64. This parameter is restricted by the memory limitations of your GPU. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 64\n\ntrain_sampler = torch.utils.data.SubsetRandomSampler(train_idx)\nval_sampler = torch.utils.data.SubsetRandomSampler(val_idx)\n\ntrain_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)\nval_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=val_sampler)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Defining our CNN Model\n\nExperimenting the performance of a tiny inception-based network with dilations. We first code the inception block below, where we replace the traditional conv5x5 with a dilated conv3x3 in the third branch.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class InceptionBlock(nn.Module):\n    def __init__(self, in_channels, o1, o2, o3, o4, **kwargs):\n        \"\"\"\n        @Params:\n            in_channels: Number of input channels from previous layer/ inception block\n            o1(int): Number of output channels from branch_1\n            o2(tuple): Number of output channels from branch 2. (b2_0 -> b2_1, b2_1 -> output)\n            o3(tuple): Number of output channels from branch_3. (b3_0 -> b3_1, b3_1 -> output)\n            o4(int): Number of output channels from branch_4.\n            \n        Note: branch_2 and branch_3 have two layers: conv1x1 followed by conv3x3, and thus, takes a tuple argument o2/ o3.\n        \n        Outputs a tensor with (o1 + o2[1] + o3[1] + o4) channels.\n        \"\"\"\n        super().__init__(**kwargs)\n        \n        # Branch 1 is a conv1x1 layer, outputs o1 channels\n        self.b1 = nn.Conv2d(in_channels, o1, kernel_size=1)\n        \n        # Branch 2 is a conv1x1 layer followed by a conv3x3\n        self.b2_0 = nn.Conv2d(in_channels, o2[0], kernel_size=1)\n        self.b2_1 = nn.Conv2d(o2[0], o2[1], kernel_size=3, padding=1)\n        \n        # Branch 3 is a conv1x1 followed by a dilated conv3x3\n        self.b3_0 = nn.Conv2d(in_channels, o3[0], kernel_size=1)\n        self.b3_1 = nn.Conv2d(o3[0], o3[1], kernel_size=3, padding=2, dilation=2)\n        \n        # Branch 4 is a 3x3 Max Pooling layer followed by a conv1x1\n        self.b4_0 = nn.MaxPool2d(kernel_size=3, stride=1, padding=1)\n        self.b4_1 = nn.Conv2d(in_channels, o4, kernel_size=1)\n\n    def forward(self, x):\n        branch_1 = F.relu(self.b1(x))\n        branch_2 = F.relu(self.b2_1(F.relu(self.b2_0(x))))\n        branch_3 = F.relu(self.b3_1(F.relu(self.b3_0(x))))\n        branch_4 = F.relu(self.b4_1(self.b4_0(x)))\n        # Concatenate the outputs on the channel dimension\n        return torch.cat((branch_1, branch_2, branch_3, branch_4), dim=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now build our TinyNet model using the `InceptionBlock` from above.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class TinyNet(nn.Module):\n    \n    def __init__(self, input_channels=3):\n        super().__init__()\n        \n        self.input_block = nn.Sequential(\n            nn.Conv2d(input_channels, 32, kernel_size=5, stride=1, padding=2),\n            nn.ReLU(),\n            nn.MaxPool2d(kernel_size=3, stride=1, padding=1)\n        )\n        \n        self.block_1 = nn.Sequential(\n            nn.Conv2d(32, 64, kernel_size=1),\n            nn.ReLU(),\n            nn.Conv2d(64, 192, kernel_size=3, padding=1),\n            nn.MaxPool2d(kernel_size=3, stride=2, padding=1)\n        )\n        \n        self.inception_1 = nn.Sequential(\n            InceptionBlock(192, 64, (96, 128), (16, 32), 32),\n            InceptionBlock(256, 128, (128, 192), (32, 96), 64),\n            nn.MaxPool2d(kernel_size=3, stride=2, padding=1)\n        )\n        \n        self.inception_2 = nn.Sequential(\n            InceptionBlock(480, 192, (96, 208), (16, 48), 64),\n            InceptionBlock(512, 160, (112, 224), (24, 64), 64),\n            nn.AdaptiveMaxPool2d((1,1)),\n            nn.Flatten()\n        )\n        \n        self.classifier = nn.Linear(512, 10)\n        \n    def forward(self, x):        \n        x = self.input_block(x)\n        x = self.block_1(x)\n        x = self.inception_1(x)\n        x = self.inception_2(x)\n        x = self.classifier(x)\n        return x\n        \n            ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Instantiate the model we defined above. Also, we tell pytorch to use a GPU (if available) as it speeds up training time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\nmodel = TinyNet().to(device)\n# print(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us see the outputs dimensions from each layer. Helps to keep track of the dimensions while creating the model and also gives us an idea of the number of parameters to learn.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install torchsummary","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchsummary import summary\n\nsample_input_size = (3, 32, 32)\nsummary(model, sample_input_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**More than 1 million parameters to learn!** Why would any one call this a \"TinyNet\"? :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Training the Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Choosing an Optimizer\n\nA good optimizer changes the learning_rate and weights of the network to minimize the loss function(\"criterion\").\n\nI chose to try AdaBound [ICLR 2019] , an optimizer that claims to behave like Adam at the beginning of training, and gradually transforms to SGD at the end. See more here :  [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://openreview.net/forum?id=Bkg3g2R9FX)\n\nYou may also want to see Rectified Adam. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install adabound","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import adabound\noptimizer = adabound.AdaBound(model.parameters(), lr=1e-3, final_lr=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define a Loss Function\n\nWe use the standard cross entropy loss for this image classification problem. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = nn.CrossEntropyLoss()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training Loop\n\n`num_epochs`: An epoch is a run over the entire training dataset. Remember, that we defined `batch_size = 64` and, therefore, only look at 64 images per iteration over the training set.\n\n`history`: Dictionary that stores train_acc, train_loss, val_acc, val_loss for each epoch.\n\nFor every epoch, we first train our model on the train_set only. We then run predictions on the validation_set, and print the scores to see how well our model is training. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"num_epochs = 50\nhistory = {\n    'train': { 'acc': [], 'loss': [] },\n    'val'  : { 'acc': [], 'loss': [] }\n}\n\nprint(\"TinyNet training...\")\n\nfor epoch in range(num_epochs):\n    \n    #Epoch Statistics\n    train_loss, train_acc, val_loss, val_acc = 0.0, 0.0, 0.0, 0.0\n    \n    #Train\n    correct = 0\n    model.train()\n    for images, labels in train_loader:\n        \n        images, labels = images.to(device), labels.to(device) #Shift to GPU\n        \n        optimizer.zero_grad() #Zero Parameter Gradients!\n        \n        #Forward Pass -> Back Propagation -> Optimize\n        outputs = model(images)\n        batch_loss = criterion(outputs, labels)\n        batch_loss.backward()\n        optimizer.step()\n        \n        #Accuracy\n        _, predicted = torch.max(outputs.data, 1)\n        correct += (predicted == labels).sum().item()\n        \n        train_loss += batch_loss.item()\n    \n    train_acc = correct/train_size\n    train_loss = train_loss/train_size\n    \n    #Validation\n    with torch.no_grad():\n        correct = 0\n        model.eval()\n        for images, labels in val_loader:\n            images, labels = images.to(device), labels.to(device) #Shift to GPU\n            \n            outputs = model(images)\n            batch_loss = criterion(outputs, labels)\n            \n            _, predicted = torch.max(outputs.data, 1)\n            correct += (predicted == labels).sum().item()\n            \n            val_loss += batch_loss.item()\n            \n    val_acc = correct/val_size\n    val_loss = val_loss/val_size\n    \n    #Save Epoch Statistics\n    history['train']['acc'] = train_acc\n    history['train']['loss'] = train_loss\n    history['val']['acc'] = val_acc\n    history['val']['loss'] = val_loss\n    print(f\"Epoch {epoch}: Train_acc: {train_acc} , \\t Train_loss: {train_loss} , \\t Val_acc: {val_acc} , \\t Val_loss: {val_loss}\")\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}