{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CIFAR-10  is an established computer-vision dataset used for object recognition. It is a subset of the 80 million tiny images dataset and consists of 60,000 32x32 color images containing one of 10 object classes, with 6000 images per class. It was collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.\n\nThe label classes in the dataset are:\n\n* airplane \n* automobile \n* bird \n* cat \n* deer \n* dog \n* frog \n* horse \n* ship \n* truck","metadata":{}},{"cell_type":"markdown","source":"# Import Required Libraries and Load Data from Pytorch Dataset","metadata":{}},{"cell_type":"code","source":"import os\nimport torch\nimport torchvision\n# To read zipped .tar, .tz file\nimport tarfile","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torchvision.datasets.utils import download_url","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can download data from below given URL","metadata":{}},{"cell_type":"code","source":"url = 'https://s3.amazonaws.com/fast-ai-imageclas/cifar10.tgz'\ndownload_url(url, '.')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with tarfile.open('./cifar10.tgz', 'r:gz') as tar:\n    tar.extractall(path = './data')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data is there in \\data folder","metadata":{}},{"cell_type":"code","source":"data_dir = './data/cifar10'\nprint(os.listdir(data_dir))\n\nclasses = os.listdir(data_dir + '/train')\nprint(classes)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we have 10 Classes, 2 folders available Train and Test","metadata":{}},{"cell_type":"markdown","source":"Let's read some of Images and plot","metadata":{}},{"cell_type":"code","source":"airplane_file = os.listdir(data_dir + '/train/airplane')\nprint('Number of Training Examples: ', len(airplane_file))\nprint(airplane_file[:5])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ship_file = os.listdir(data_dir + '/test/ship')\nprint('Number of Testing Examples: ', len(ship_file))\nprint(ship_file[:5])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torchvision.datasets import ImageFolder\nfrom torchvision.transforms import ToTensor\nimport torchvision.transforms as transforms","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To reduce chance of overfitting, lets carryout Normalization and Regularization as below","metadata":{}},{"cell_type":"code","source":"trans = transforms.Compose([\n    # this operation was done, to make Image size compatible with AlexNet Model\n    transforms.Resize((70,70)),\n    # To focus on only primary component of Image\n    transforms.RandomCrop((64,64)),\n    transforms.ToTensor(),\n    # To nullify dominance of one of Color channel\n    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets load train and test data","metadata":{}},{"cell_type":"code","source":"train_ds = ImageFolder(data_dir+'/train', transform = trans)\nval_ds = ImageFolder(data_dir+'/test', transform = trans)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Below library will help us to pass batch of data, as feeding all data at once will lead to OverLoading and System will get Hang\nfrom torch.utils.data.dataloader import DataLoader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this is one of Hyper parameter, let's start with\nbatch_size = 512","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PyTorch data loaders\ntrain_dl = DataLoader(train_ds, batch_size, shuffle=True, num_workers=3, pin_memory=True)\nvalid_dl = DataLoader(val_ds, batch_size*2, num_workers=3, pin_memory=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot and observe sets","metadata":{}},{"cell_type":"code","source":"# These libraries will help us to plot images\nimport matplotlib.pyplot as plt\nfrom torchvision.utils import make_grid","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_batch(dl):\n    for images, labels in dl:\n        fig, ax = plt.subplots(figsize = (12,12))\n        ax.set_xticks([]); ax.set_yticks([])\n        ax.imshow(make_grid(images[: 100], 10).permute(1,2,0))\n        break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_batch(train_dl)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_batch(valid_dl)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see some changes in images from original image from dataset","metadata":{}},{"cell_type":"code","source":"img, label = train_ds[0]\nimg.shape, label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Building","metadata":{}},{"cell_type":"markdown","source":"These libraries will help us to build Neural Network","metadata":{}},{"cell_type":"code","source":"import torch.nn as nn\nimport torch.nn.functional as F","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AlexNet(nn.Module):\n    \n    def __init__(self, num_classes):\n        \n        super().__init__()\n        \n        self.features = nn.Sequential(\n            nn.Conv2d(3, 64, kernel_size = 11, stride = 4, padding = 2),\n            nn.ReLU(inplace = True),\n            nn.MaxPool2d(kernel_size = 3, stride = 2),\n            \n            nn.Conv2d(64, 192, kernel_size = 5, padding = 2),\n            nn.ReLU(inplace = True),\n            nn.MaxPool2d(kernel_size = 3, stride = 2),\n            \n            nn.Conv2d(192, 384, kernel_size = 3, padding = 1),\n            nn.ReLU(inplace = True),\n            \n            nn.Conv2d(384, 256, kernel_size = 3, padding = 1),\n            nn.ReLU(inplace = True),\n            \n            nn.Conv2d(256, 256, kernel_size = 3, padding = 1),\n            nn.ReLU(inplace = True),\n            nn.MaxPool2d(kernel_size = 3, stride = 2)\n        )\n        \n        self.avgpool = nn.AdaptiveAvgPool2d((6,6))\n        \n        self.classifier = nn.Sequential(\n            nn.Dropout(0.5),\n            \n            nn.Linear(256*6*6, 4096),\n            nn.ReLU(inplace = True),\n            \n            nn.Linear(4096, 4096),\n            nn.ReLU(inplace = True),\n            \n            nn.Linear(4096, num_classes)\n        )\n        \n    def forward(self, x):\n        \n        x = self.features(x)\n        x = self.avgpool(x)\n        x = x.view(x.size(0), 256*6*6)\n        logit = self.classifier(x)\n        \n        return logit","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets Initialize Model\nmodel = AlexNet(num_classes = 10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# to get all details of Model\nmodel","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's Verify Models","metadata":{}},{"cell_type":"code","source":"sample = next(iter(train_ds))\nimg = sample[0]\nimg.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img.unsqueeze(0).shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"out = model(img.unsqueeze(0))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"F.softmax(out)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" # Demo plot\n\nfor images, labels in train_dl:\n    print('Image Shape', images.shape)\n    out = model(images)\n    print('output shape', out.shape)\n    print('out[0]', out[0])\n    break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, model has initial guess, it seems all options have been given same weightage i.e. 10%","metadata":{}},{"cell_type":"code","source":"probs = F.softmax(out[0], dim = 0)\nprobs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"m = torch.argmax(probs)\nm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(img.permute(1,2,0))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, it was wrong prediction by model, but it was just initial guess !!","metadata":{}},{"cell_type":"markdown","source":"# Device Selection","metadata":{}},{"cell_type":"code","source":"def get_default_device():\n    \"\"\"Pick GPU if available, else CPU\"\"\"\n    if torch.cuda.is_available():\n        return torch.device('cuda')\n    else:\n        return torch.device('cpu')\n    \ndef to_device(data, device):\n    \"\"\"Move tensor(s) to chosen device\"\"\"\n    if isinstance(data, (list,tuple)):\n        return [to_device(x, device) for x in data]\n    return data.to(device, non_blocking=True)\n\nclass DeviceDataLoader():\n    \"\"\"Wrap a dataloader to move data to a device\"\"\"\n    def __init__(self, dl, device):\n        self.dl = dl\n        self.device = device\n        \n    def __iter__(self):\n        \"\"\"Yield a batch of data after moving it to device\"\"\"\n        for b in self.dl: \n            yield to_device(b, self.device)\n\n    def __len__(self):\n        \"\"\"Number of batches\"\"\"\n        return len(self.dl)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = get_default_device()\ndevice","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dl = DeviceDataLoader(train_dl, device)\nval_dl = DeviceDataLoader(valid_dl, device)\nto_device(model, device)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define Helper Functions","metadata":{}},{"cell_type":"code","source":"def loss_batch(model, loss_func, x, y, opt = None, metric = None):\n    \n    pred = model(x)\n    \n    loss = loss_func(pred, y)\n    \n    if opt is not None:\n        \n        loss.backward()\n        opt.step()\n        opt.zero_grad()\n        \n    metric_result = None\n    \n    if metric is not None:\n        \n        metric_result = metric(pred, y)\n        \n    return loss.item(), len(x), metric_result","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evaluate(model, loss_fn, val_dl, metric = None):\n    \n    with torch.no_grad():\n        \n        results = [loss_batch(model, loss_fn, x, y, metric = metric) for x, y in val_dl]\n        \n        losses, nums, metrics = zip(*results)\n        \n        total = np.sum(nums)\n        \n        avg_loss = np.sum(np.multiply(losses, nums)) / total\n        \n        avg_metric = None\n        \n        if metric is not None:\n            avg_metric = np.sum(np.multiply(metrics, nums)) / total\n            \n    return avg_loss, total, avg_metric","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have defined Scheduler to handle Learning rate, as with SGD, we would be looking for Learning rate that changes based on Accuracy\nIf Model finds no change in Accuracy, it will reduce learning rate by factor of 0.1. Controling parameter would be validation accuracy.","metadata":{}},{"cell_type":"code","source":"def fit(epochs, model, loss_fn, train_dl, val_dl, opt_fn = None, metric = None, scheduler = None, scheduler_on = 'val_metric'):\n    \n    train_losses, val_losses, val_metrics = [], [], []\n    \n    \n    for epoch in range(epochs):\n        \n        model.train()\n        for x, y in train_dl:\n            train_loss, _, _ = loss_batch(model, loss_fn, x, y, opt_fn)\n            \n        model.eval()\n        result = evaluate(model, loss_fn, val_dl, metric)\n        val_loss, total, val_metric = result\n        \n        train_losses.append(train_loss)\n        val_losses.append(val_losses)\n        val_metrics.append(val_metric)\n        \n        if metric is None:\n            print('Epoch{}/{}, train_loss: {:.4f}, val_loss: {:.4f}' \n                 .format(epoch+1, epochs, train_loss, val_loss))\n            \n        else:\n            print('Epoch {}/{}, train_loss: {:.4f}, val_loss: {:.4f}, val_{}: {:.4f}'\n                 .format(epoch+1, epochs, train_loss, val_loss, metric.__name__, val_metric))\n            \n        if scheduler is not None:\n            if scheduler_on == 'val_metric':\n                scheduler.step(val_metrics[-1])\n        \n            \n    return train_losses, val_losses, val_metrics","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Function to find accuracy, as there is no built in function in Pytorh to find accuracy, its simple code to define accuracy as below","metadata":{}},{"cell_type":"code","source":"def accuracy(output, labels):\n    _, preds = torch.max(output, dim = 1)\n    \n    return torch.sum(preds == labels).item() / len(preds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets verify initial guess on validation","metadata":{}},{"cell_type":"code","source":"val_loss, _, val_acc = evaluate(model, F.cross_entropy, val_dl, metric = accuracy)\n\nprint(val_loss, val_acc)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Model","metadata":{}},{"cell_type":"code","source":"num_epochs = 25\n\noptimizer = torch.optim.SGD(model.parameters(), lr = 0.1, momentum = 0.9)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor = 0.1, mode = 'max', verbose = True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = fit(num_epochs, model, F.cross_entropy, train_dl, val_dl, optimizer, accuracy, scheduler, 'val_metric')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"76% Accuracy for 10 class classification model, is not bad, adding few more dropout and batchnormalization, would have increased accuracy.\n\n\nBut, moto of this ntebook is not to carry out hyperparameter Tuning, it was to run this Revolutionary Model AlexNet.","metadata":{}},{"cell_type":"markdown","source":"# Plotting result","metadata":{}},{"cell_type":"markdown","source":"Plotting Train Losses","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (8,8))\nplt.plot(history[0], '-x')\nplt.xlabel('Epochs')\nplt.ylabel('Training Loss')\nplt.title('Plot between Training Loss vs Epochs')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (8,8))\nplt.plot(history[2], '-x')\nplt.xlabel('Epochs')\nplt.ylabel('Validation Loss')\nplt.title('Plot between Validation Loss vs Epochs')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems like running few more epochs could have increased Accuracy to 80%.\n\nBut I will stop here,\n\nThanks for reading and I hop you have found this useful.\n\nKindly consider Upvoting and happy Learning!!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}