{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nIn this project, you will build a neural network of your own design to evaluate the CIFAR-10 dataset.\nOur target accuracy is 70%, but any accuracy over 50% is a great start.\nSome of the benchmark results on CIFAR-10 include:\n\n78.9% Accuracy | [Deep Belief Networks; Krizhevsky, 2010](https://www.cs.toronto.edu/~kriz/conv-cifar10-aug2010.pdf)\n\n90.6% Accuracy | [Maxout Networks; Goodfellow et al., 2013](https://arxiv.org/pdf/1302.4389.pdf)\n\n96.0% Accuracy | [Wide Residual Networks; Zagoruyko et al., 2016](https://arxiv.org/pdf/1605.07146.pdf)\n\n99.0% Accuracy | [GPipe; Huang et al., 2018](https://arxiv.org/pdf/1811.06965.pdf)\n\n98.5% Accuracy | [Rethinking Recurrent Neural Networks and other Improvements for ImageClassification; Nguyen et al., 2020](https://arxiv.org/pdf/2007.15161.pdf)\n\nResearch with this dataset is ongoing. Notably, many of these networks are quite large and quite expensive to train. \n\nThis notebook achieves 71.8% w/ only basic knowledge of CNNs and pytorch.\n\n## Imports","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2023-08-25T13:12:38.615461Z","iopub.execute_input":"2023-08-25T13:12:38.615893Z","iopub.status.idle":"2023-08-25T13:12:38.621980Z","shell.execute_reply.started":"2023-08-25T13:12:38.615858Z","shell.execute_reply":"2023-08-25T13:12:38.620626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## This cell contains the essential imports you will need – DO NOT CHANGE THE CONTENTS! ##\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader\nimport torchvision\nimport torchvision.transforms as transforms\nfrom torchvision import datasets\nimport matplotlib.pyplot as plt\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2023-08-25T13:12:38.629541Z","iopub.execute_input":"2023-08-25T13:12:38.630207Z","iopub.status.idle":"2023-08-25T13:12:42.441803Z","shell.execute_reply.started":"2023-08-25T13:12:38.630163Z","shell.execute_reply":"2023-08-25T13:12:42.440659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load the Dataset\n\nSpecify your transforms as a list first.\nThe transforms module is already loaded as `transforms`.\n\nCIFAR-10 is fortunately included in the torchvision module.\nThen, you can create your dataset using the `CIFAR10` object from `torchvision.datasets` ([the documentation is available here](https://pytorch.org/docs/stable/torchvision/datasets.html#cifar)).\nMake sure to specify `download=True`! \n\nOnce your dataset is created, you'll also need to define a `DataLoader` from the `torch.utils.data` module for both the train and the test set.","metadata":{}},{"cell_type":"code","source":"# Define transforms\n## YOUR CODE HERE ##\n\n# Create training set and define training dataloader\n## YOUR CODE HERE ##\n\n# Create test set and define test dataloader\n## YOUR CODE HERE ##\n\n# Define transforms\ntrain_transform = transforms.Compose([\n    transforms.RandomHorizontalFlip(),\n    transforms.RandomCrop(size=32, padding=4),  #arguable for such tiny ings...\n    transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n])\n\nval_test_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n])\n\n# Create training set and define training dataloader\ntrain_dataset = datasets.CIFAR10('./kaggle/input/cifar-10/train.7z', train=True, transform=train_transform, download=True)\ntrain_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)\n\n# Create test set and define test dataloader\ntest_dataset = datasets.CIFAR10('./kaggle/input/cifar-10/test.7z', train=False, transform=val_test_transform, download=True)\ntest_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)\n\n# Define validation set and validation dataloader\nval_dataset = datasets.CIFAR10('./kaggle/input/cifar-10/test.7z', train=False, transform=val_test_transform, download=True)\nval_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)\n\n# The 10 classes in the dataset\nclasses = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T13:12:42.444173Z","iopub.execute_input":"2023-08-25T13:12:42.444873Z","iopub.status.idle":"2023-08-25T13:12:52.587433Z","shell.execute_reply.started":"2023-08-25T13:12:42.444835Z","shell.execute_reply":"2023-08-25T13:12:52.586104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore the Dataset\nUsing matplotlib, numpy, and torch, explore the dimensions of your data.\n\nYou can view images using the `show5` function defined below – it takes a data loader as an argument.\nRemember that normalized images will look really weird to you! You may want to try changing your transforms to view images.\nTypically using no transforms other than `toTensor()` works well for viewing – but not as well for training your network.\nIf `show5` doesn't work, go back and check your code for creating your data loaders and your training/test sets.","metadata":{}},{"cell_type":"code","source":"def show5(img_loader):\n    dataiter = iter(img_loader)\n    \n    batch = next(dataiter)\n    labels = batch[1][0:5]\n    images = batch[0][0:5]\n    for i in range(5):\n        print(classes[labels[i]])\n    \n        image = images[i].numpy()\n        plt.imshow(image.T)\n        plt.show()\n        \n# Show 5 sample images from the test dataset (less intrusive transf)\nshow5(test_loader)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:53:03.377083Z","iopub.execute_input":"2023-08-25T14:53:03.377503Z","iopub.status.idle":"2023-08-25T14:53:04.668003Z","shell.execute_reply.started":"2023-08-25T14:53:03.377469Z","shell.execute_reply":"2023-08-25T14:53:04.667017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Explore data\n## YOUR CODE HERE ##\n\n# Get class labels\nlabels = np.array(train_dataset.targets)\n\n# Get class names\nclasses = train_dataset.classes\n\n# Calculate class frequencies\nclass_counts = np.bincount(labels)\nclass_frequencies = class_counts / class_counts.sum()\n\n# Plot class distribution\nplt.figure(figsize=(10, 5))\nplt.bar(classes, class_frequencies)\nplt.xlabel('Class')\nplt.ylabel('Frequency')\nplt.title('Class Distribution')\nplt.xticks(rotation=45)\nplt.show()\n\n# Plot label distribution\nplt.figure(figsize=(10, 5))\nplt.hist(labels, bins=len(classes), edgecolor='black')\nplt.xlabel('Label')\nplt.ylabel('Frequency')\nplt.title('Label Distribution')\nplt.xticks(np.arange(len(classes)), classes, rotation=45)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:53:09.442703Z","iopub.execute_input":"2023-08-25T14:53:09.445520Z","iopub.status.idle":"2023-08-25T14:53:10.108720Z","shell.execute_reply.started":"2023-08-25T14:53:09.445482Z","shell.execute_reply":"2023-08-25T14:53:10.107681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Perfect balance on both...!","metadata":{}},{"cell_type":"markdown","source":"## Build your Neural Network\nUsing the layers in `torch.nn` (which has been imported as `nn`) and the `torch.nn.functional` module (imported as `F`), construct a neural network based on the parameters of the dataset. \nFeel free to construct a model of any architecture – feedforward, convolutional, or even something more advanced!","metadata":{}},{"cell_type":"code","source":"## YOUR CODE HERE ##\n\nclass CNNClassifier(nn.Module):\n    def __init__(self):\n        super(CNNClassifier, self).__init__()\n        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)\n        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)\n        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)\n        self.fc1 = nn.Linear(64 * 8 * 8, 128)  # Adjust this based on your input image size\n        self.fc2 = nn.Linear(128, 10)  # 10 classes in CIFAR-10\n        \n    def forward(self, x):\n        x = self.pool(F.relu(self.conv1(x)))\n        x = self.pool(F.relu(self.conv2(x)))\n        x = x.view(-1, 64 * 8 * 8)  # Flatten the tensor\n        x = F.relu(self.fc1(x))\n        x = self.fc2(x)\n        return x\n\n# Create an instance of the CNNClassifier\nmodel = CNNClassifier()\n\n# Print the model's architecture\nprint(model)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-25T13:12:53.688495Z","iopub.execute_input":"2023-08-25T13:12:53.689667Z","iopub.status.idle":"2023-08-25T13:12:53.731939Z","shell.execute_reply.started":"2023-08-25T13:12:53.689626Z","shell.execute_reply":"2023-08-25T13:12:53.730725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Input (3, H, W)\n#  |\n#  | Conv2d (32 filters, 3x3, padding=1)\n#  | ReLU\n#  | Conv2d (64 filters, 3x3, padding=1)\n#  | ReLU\n#  | MaxPool2d (2x2)\n#  |\n#  | Flatten\n#  |\n#  | Fully Connected (128 units)\n#  | ReLU\n#  |\n#  | Fully Connected (10 units - output classes)\n#  |\n# Output (10 classes)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Specify a loss function and an optimizer, and instantiate the model.\n\nIf you use a less common loss function, please note why you chose that loss function in a comment.","metadata":{}},{"cell_type":"code","source":"## YOUR CODE HERE ##\nimport torch.optim as optim\n\n# Specify the loss function (Cross-Entropy Loss)\ncriterion = nn.CrossEntropyLoss()\n\n# Specify the optimizer (Stochastic Gradient Descent)\noptimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)\n\n# Print the loss function and optimizer\nprint(\"Loss Function:\", criterion)\nprint(\"Optimizer:\", optimizer)\n\n# Create an instance of the CNNClassifier => was already instantiated with the CNN arch. above\n# model = CNNClassifier()","metadata":{"execution":{"iopub.status.busy":"2023-08-25T13:14:15.520085Z","iopub.execute_input":"2023-08-25T13:14:15.520555Z","iopub.status.idle":"2023-08-25T13:14:15.533035Z","shell.execute_reply.started":"2023-08-25T13:14:15.520515Z","shell.execute_reply":"2023-08-25T13:14:15.531937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Running your Neural Network\nUse whatever method you like to train your neural network, and ensure you record the average loss at each epoch. \nDon't forget to use `torch.device()` and the `.to()` method for both your model and your data if you are using GPU!\n\nIf you want to print your loss during each epoch, you can use the `enumerate` function and print the loss after a set number of batches. 250 batches works well for most people!","metadata":{}},{"cell_type":"markdown","source":"Plot the training loss (and validation loss/accuracy, if recorded).","metadata":{}},{"cell_type":"code","source":"## YOUR CODE HERE ##\n\n# Redo: a new clean re-train d'a cappo for GPU\n# Define the CNNClassifier class and other necessary code\n\n# Define the device (GPU if available)\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using {device}\")\n\n# Instantiate the model and move it to the device\nmodel = CNNClassifier().to(device)\n\n# Define the loss function and optimizer\ncriterion = nn.CrossEntropyLoss() \noptimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.95)  #  lr=0.01, momentum=0.9 => 71%\n\n# # Define the hyperparameters for the Adam optimizer\n# learning_rate = 0.001\n# weight_decay = 1e-4  # or 5e-4\n# # Instantiate the Adam optimizer\n# optimizer = optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)\n\n# Lists to store training loss and validation loss/accuracy\ntrain_loss_list = []\nval_loss_list = []\nval_acc_list = []\n\n# Define the number of epochs\nepochs = 5\n\n# Set the model to training mode\nmodel.train()\n\n# Loop over epochs\nfor epoch in range(epochs):\n    running_loss = 0.0\n    epoch_loss = 0.0 # Initialize epoch_loss to 0.0\n    \n    # Loop over batches\n    for batch_idx, (inputs, labels) in enumerate(train_loader, 1):\n    \n        # Move data to the device\n        inputs, labels = inputs.to(device), labels.to(device)        \n        # Zero the gradients\n        optimizer.zero_grad()        \n        # Forward pass\n        outputs = model(inputs)        \n        # Compute the loss\n        loss = criterion(outputs, labels)        \n        # Backpropagation\n        loss.backward()        \n        # Update weights\n        optimizer.step() \n        #update the run and epoch losses\n        running_loss += loss.item()\n        epoch_loss += loss.item() # Update epoch_loss\n        if batch_idx % 250 == 0:\n            print(f\"Epoch [{epoch+1}/{epochs}], Batch [{batch_idx}/{len(train_loader)}], Loss: {running_loss/250:.4f}\")\n            running_loss = 0.0\n    \n    # Record training loss for this epoch\n    train_loss_list.append(epoch_loss / len(train_loader)) # Record average training loss for this epoch\n    \n    # Evaluate the model on validation data\n    model.eval()\n    val_loss = 0.0\n    correct = 0\n    total = 0\n    with torch.no_grad():\n        for inputs, labels in val_loader:\n            inputs, labels = inputs.to(device), labels.to(device)\n            outputs = model(inputs)\n            val_loss += criterion(outputs, labels).item()\n            _, predicted = outputs.max(1)\n            total += labels.size(0)\n            correct += predicted.eq(labels).sum().item()\n    \n    # Record validation loss and accuracy for this epoch\n    val_loss_list.append(val_loss / len(val_loader))\n    val_acc_list.append(correct / total)\n    \n    # Print validation metrics\n    print(f\"Epoch [{epoch+1}/{epochs}], Validation Loss: {val_loss/len(val_loader):.4f}, Validation Accuracy: {correct/total:.4f}\")\n    \n    # Set the model back to training mode\n    model.train()\n\n# Plot the training and validation loss/accuracy\nplt.figure(figsize=(12, 5))\nplt.subplot(1, 2, 1)\nplt.plot(range(1, epochs+1), train_loss_list, label='Training Loss')\nplt.plot(range(1, epochs+1), val_loss_list, label='Validation Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title('Training and Validation Loss')\nplt.legend()\n\nplt.subplot(1, 2, 2)\nplt.plot(range(1, epochs+1), val_acc_list, label='Validation Accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.title('Validation Accuracy')\nplt.legend()\n\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:30:01.581004Z","iopub.execute_input":"2023-08-25T14:30:01.581607Z","iopub.status.idle":"2023-08-25T14:34:20.826997Z","shell.execute_reply.started":"2023-08-25T14:30:01.581568Z","shell.execute_reply":"2023-08-25T14:34:20.825913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Testing your model\nUsing the previously created `DataLoader` for the test set, compute the percentage of correct predictions using the highest probability prediction. \n\nIf your accuracy is over 70%, great work! \nThis is a hard task to exceed 70% on.\n\nIf your accuracy is under 45%, you'll need to make improvements.\nGo back and check your model architecture, loss function, and optimizer to make sure they're appropriate for an image classification task.","metadata":{}},{"cell_type":"code","source":"## YOUR CODE HERE ##\n# Set the model to evaluation mode\nmodel.eval()\n\n# Initialize variables to keep track of correct and total predictions\ncorrect_predictions = 0\ntotal_predictions = 0\n\n# Loop over batches in the test/validation dataset\nwith torch.no_grad():\n    for inputs, labels in test_loader:  # use test_loader here\n        inputs, labels = inputs.to(device), labels.to(device)\n        outputs = model(inputs)\n        _, predicted = outputs.max(1)\n        total_predictions += labels.size(0)\n        correct_predictions += predicted.eq(labels).sum().item()\n\n# Compute the percentage of correct predictions\naccuracy = correct_predictions / total_predictions * 100\n\n# Print the accuracy\nprint(f\"Accuracy on test dataset: {accuracy:.2f}%\")\n","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:36:08.499990Z","iopub.execute_input":"2023-08-25T14:36:08.500389Z","iopub.status.idle":"2023-08-25T14:36:11.279033Z","shell.execute_reply.started":"2023-08-25T14:36:08.500357Z","shell.execute_reply":"2023-08-25T14:36:11.277987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Saving your model\nUsing `torch.save`, save your model for future loading.","metadata":{}},{"cell_type":"code","source":"## YOUR CODE HERE ##\n# Specify the file path for saving the checkpoint\ncheckpoint_path = 'cifar10_checkpoint.pth'\n\n# Create a dictionary to store the checkpoint data\ncheckpoint = {\n    'epoch': epochs,\n    'model_state_dict': model.state_dict(),\n    'optimizer_state_dict': optimizer.state_dict(),\n    'train_loss_list': train_loss_list,\n    'val_loss_list': val_loss_list,\n    'val_acc_list': val_acc_list\n}\n\n# Save the checkpoint dictionary to the specified file\ntorch.save(checkpoint, checkpoint_path)\n\nprint(f\"Checkpoint saved to {checkpoint_path}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:41:38.519875Z","iopub.execute_input":"2023-08-25T14:41:38.520284Z","iopub.status.idle":"2023-08-25T14:41:38.543643Z","shell.execute_reply.started":"2023-08-25T14:41:38.520252Z","shell.execute_reply":"2023-08-25T14:41:38.542521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make a Recommendation\n\nBased on your evaluation, what is your recommendation on whether to build or buy? Explain your reasoning below.","metadata":{}},{"cell_type":"markdown","source":"### Build your own\nBased on our frugal evaluation of the achieved ~72% accuracy with a basic CNN model, relatively low PyTorch skills, and a relatively short training time of approximately 200 seconds on a GPU (free kaggle cloud), it's evident that there's substantial room for improvement as low-hanging fruit. The fact that a basic model has already reached this accuracy level suggests that with more expertise and optimization, it's quite achievable even for startups to exceed 80% accuracy on the CIFAR-10 dataset.\n\nHence it would be recommended to \"build\" the model rather than \"buy.\" Building the model in-house allows for greater flexibility and control over the architecture, hyperparameters, and training process. The fact that we were able to achieve a satisfactory 71% accuracy with minimal effort and resources indicates that with more advanced techniques, fine-tuning of hyperparameters, and perhaps exploring more complex architectures (such as deeper CNNs or using techniques like advanced data augmentation), significant performance gains are likely at moderate in-house costs.\n\nFurthermore, since the performance could potentially be improved at a relatively low cost (in terms of time and expertise) compared to purchasing a pre-built commercial solution, building the model aligns well with the goals of achieving a higher accuracy while maintaining in-house control and customization.\n\nIn summary, based on the achieved accuracy, available resources, and the potential for improvement, building locally a more advanced model with expertise and optimized techniques is a recommended course of action.","metadata":{}},{"cell_type":"markdown","source":"\n\n**Double click this cell to modify it**\n\n","metadata":{}}]}