{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":31012,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Fundamentals of Convolutional Networks - LeNet5 with Wandb Experiment Tracking","metadata":{}},{"cell_type":"markdown","source":"#### Author: Antonio Rueda-Toicen\n\n**antonio.rueda.toicen 'at' hpi 'dot' de**\n\n\n[![Creative Commons License](https://i.creativecommons.org/l/by/4.0/88x31.png)](http://creativecommons.org/licenses/by/4.0/)\n\nThis work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).","metadata":{}},{"cell_type":"markdown","source":"[LeNet-5](https://en.wikipedia.org/wiki/LeNet), from [\"Gradient Based Learning Applied to Document\nRecognition\"](http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf) is a convolutional neural network that can recognize complex patterns. It was originally designed to recognize the handwritten digits of the [MNIST dataset](https://try.fiftyone.ai/datasets/mnist/samples). \n\n","metadata":{}},{"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/andandandand/practical-computer-vision/refs/heads/main/images/lenet5-architecture.png)","metadata":{}},{"cell_type":"markdown","source":"LeNet-5 has a simple structure: input layer, three convolutional layers (C1, C3, C5) interleaved with two pooling layers (S2, S4), followed by fully connected layers (F6 as a layer connecting the activations coming from C5 to the output). \n\nLeNet-5 recognizes handwritten digits by using local receptive fields with shared weights and spatial subsampling (pooling) before feeding the resulting downsampled images into fully connected layers. \n\nThis architecture became a blueprint for modern CNNs.\n\nIn this notebook, we build it and use it to submit an entry into [Kaggle's Digit Recognition Competition](https://www.kaggle.com/competitions/digit-recognizer) to provide classifications to the MNIST dataset. We will also integrate [Weights & Biases (wandb)](https://wandb.ai/home) to track the experiment, log metrics, and save the best performing model.","metadata":{}},{"cell_type":"markdown","source":"## Set up imports","metadata":{"_uuid":"3bf074cd-2ee4-4ddb-b99a-d8bc47c7dc6d","_cell_guid":"e9640e21-4918-4222-86a7-1baa9ff82edd","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.optim import Adam\n\n# Visualization tools\nimport torchvision\nimport torchvision.transforms.v2 as transforms\nimport torchvision.transforms.functional as F\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nfrom PIL import Image\n\n# Utilities for I/O and file manipulation\nimport numpy as np\nimport pandas as pd\nimport os \nfrom contextlib import redirect_stdout\n\n# Weights and Biases import to track experiments\nimport wandb \nfrom kaggle_secrets import UserSecretsClient\n# Metrics to track\nfrom sklearn.metrics import confusion_matrix, f1_score\n","metadata":{"_uuid":"a8404a12-e32d-478d-b359-d0f91e77c06e","_cell_guid":"166a85b1-0964-4e30-b3c8-40c992ed7054","trusted":true,"collapsed":false,"execution":{"execution_failed":"2025-05-06T13:34:09.489Z"},"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Reproducibility seed and compute device (GPU or CPU)","metadata":{"_uuid":"a4b0436c-d9dc-4fd5-8ae6-ae2e00cc7e7b","_cell_guid":"f7c4260c-93a3-4538-87ef-161d8fdf89bd","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Set random seed for reproducibility of all random operations on PyTorch\ntorch.manual_seed(42)\n\n# Set computing device, \"cuda\" stands for an Nvidia GPU \n# Data and models will be sent to this device when training\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ndevice","metadata":{"_uuid":"adee5c16-949f-4be4-85a0-0043e7c41076","_cell_guid":"528e22c0-a848-4495-8b8b-7b2becac5008","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Prepare dataset\n\n* Explore the MNIST dataset [through FiftyOne](https://try.fiftyone.ai/datasets/mnist/samples)","metadata":{"_uuid":"73b1d80a-85ce-49b3-9f96-2d0ce1c980ca","_cell_guid":"3f770ff4-e1e4-45d1-b5eb-8e390898ba4b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Create an initial transform from PIL to PyTorch tensor to compute mean and standard deviation of the training set\ninitial_transform = transforms.Compose([transforms.ToImage(),\n                                        transforms.ToDtype(torch.float32, scale=True)])","metadata":{"_uuid":"42621d09-9152-4e3f-8ebe-d270eefd2b9b","_cell_guid":"0bd81053-dbb2-40f6-9556-e2078538e6e8","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Define the MNIST Dataset class\nclass MNISTDataset(Dataset):\n    def __init__(self, dataframe, labels=True, transform=None):\n        self.data = dataframe\n        self.labels = labels\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        if self.labels:\n            # Reshape image data and get label\n            image = self.data.iloc[idx, 1:].values.astype(np.uint8).reshape(28, 28)\n            label = self.data.iloc[idx, 0]\n            image = Image.fromarray(image) # Convert numpy array to PIL Image\n        else:\n            # Reshape image data, assign placeholder label\n            image = self.data.iloc[idx, :].values.astype(np.uint8).reshape(28, 28)\n            label = -1 # Placeholder for test data (or any value not in 0-9)\n            image = Image.fromarray(image) # Convert numpy array to PIL Image\n\n        # Apply transformations if any\n        if self.transform:\n            image = self.transform(image)\n            \n        # Ensure label is a tensor\n        label_tensor = torch.tensor(label).long()\n\n        # Return image tensor and label tensor\n        return image, label_tensor\n\n\n# Load the data\ntrain_data_df = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_data_df  = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"_uuid":"6da95d28-247e-41e6-ae0d-2628b96f0bfc","_cell_guid":"375b9c61-f669-4fbf-8e98-302b4dde6852","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_data_df.head(1)","metadata":{"_uuid":"99abce3d-2885-45f1-b67b-7318b5467b3e","_cell_guid":"125cb50f-c4cd-4580-9721-85a4936b1cf5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_data_df.head(1)","metadata":{"_uuid":"7f654045-0448-407f-82f4-428a0f5d1f35","_cell_guid":"dbf655a9-d0c0-4c7f-9258-1fed48200da2","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create datasets\n# Pass the initial transform to calculate stats correctly from original data range\ntrain_set_for_stats = MNISTDataset(train_data_df, labels=True, transform=initial_transform)\n\n# We'll apply the final transform later after calculating stats\ntrain_set = MNISTDataset(train_data_df, labels=True, transform=None)\ntest_set = MNISTDataset(test_data_df, labels=False, transform=None)","metadata":{"_uuid":"9b9c4494-8980-4917-a36a-0fff199a85a4","_cell_guid":"e0cd1bdc-ef70-4b48-b07e-1d34e6c9c178","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# The images are initially pairs of PIL (Python Image Library) images and their labels\n# Get an example using the non-transformed dataset first\nimg_pil, label_pil = MNISTDataset(train_data_df, labels=True, transform=None)[7]\nplt.imshow(img_pil, cmap='gray');\nprint(\"Label:\", label_pil.item())","metadata":{"_uuid":"d33578e5-0c60-4ea6-916b-0f35c5e7e497","_cell_guid":"d202ff94-b38b-4d89-8ad6-b58236f626ac","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Each image is 28x28 pixels and represented in grayscale\n# Show the transformed version for stats calculation\nim_tensor, _ = train_set_for_stats[7] \nplt.imshow(im_tensor.squeeze(), cmap='gray');","metadata":{"_uuid":"524bb208-c014-48ab-bd40-b6eaf7cef52c","_cell_guid":"8209bd3d-4f48-4188-9ec5-16503e998538","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.490Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# We use 0 to represent black pixels and 255 to represent white in uint8 (unsigned integers of 8 bits)\n# Display the non-transformed PIL image data as a heatmap\ndf = pd.DataFrame(np.array(img_pil))\nplt.figure(figsize=(12, 10))\nsns.heatmap(df, cmap=\"Greys_r\", annot=True, fmt=\".0f\", cbar=False)\nplt.show()","metadata":{"_uuid":"54b31887-fc44-4fc4-aa6c-53a334837190","_cell_guid":"a254da39-a18b-4f23-920f-874f9383ff18","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.491Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Input normalization with mean and standard deviation of the training set\n\nThe classic paper [Efficient Backprop](https://www.researchgate.net/publication/2811922_Efficient_BackProp) (LeCun et al., 2000) shows empirically that input normalization using the mean and standard deviation of the training set can lead to a 10x improvement in training speed. This practice is widely spread in modern deep learning. Standard scaling transforms the data such that it has a mean of 0 and a standard deviation of 1. This is useful in machine learning algorithms sensitive to the magnitude of features.\n\nWe use the standard scaling normalization (also called [z-score normalization](https://en.wikipedia.org/wiki/Standard_score)):\n\n$x^{normalized} = \\frac{x - \\mu}{\\sigma}$\n\nWhere:\n- $x$ is the input variable\n- $\\mu$ is the mean of inputs across the training set\n- $\\sigma$ is the standard deviation across the training set\n\nIt represents the distance between the raw input x and the population mean in units of the standard deviation. This distance is negative when the raw score is below the mean and positive when above. \n\nImplementation in PyTorch or NumPy:\n```python\nmean = training_data.mean()\nstd = training_data.std()\nnormalized_data = (data - mean) / std\n```\n\nInput normalization is useful for neural networks for these key reasons:\n\n1. Training Stability: Normalized inputs improve gradient flow through the network. Without normalization, large input values can cascade into exploding gradients (values that go outside of the range that we can represent with the number of bits that we have to represent our activations) and unstable training.\n\n2. Faster Learning: Normalization creates a more spherical loss surface, enabling more direct gradient descent paths and allowing higher learning rates.\n\n3. Better Initialization: Most normalized values fall in ranges where activation functions have stronger gradients, preventing early neuron saturation.\n\n**Important**: Always use training set statistics to normalize both training and test data to maintain consistent transformations.","metadata":{"_uuid":"f068646d-d373-483f-b41b-aae89965f628","_cell_guid":"ef7d4bc7-1d9c-49c1-8b2a-9ff880103757","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Calculate mean and std of training set using the initially transformed data\n# This ensures stats are based on the 0.0-1.0 float range\ntrain_tensor_list = [img for img, _ in train_set_for_stats] # Extract tensors\n\n# Check if the list is not empty\nif train_tensor_list:\n    train_data_tensors = torch.stack(train_tensor_list)\n    mean = train_data_tensors.mean().item()\n    std = train_data_tensors.std().item()\n    print(f\"Dataset statistics (0-1 scale) - Mean: {mean:.4f}, Std: {std:.4f}\")\nelse:\n    print(\"Error: No tensors found in train_set_for_stats to calculate statistics.\")\n","metadata":{"_uuid":"7cacdef6-2eb0-48dd-ad62-a9ac530bf5e3","_cell_guid":"eb352c08-45d2-4ca4-bbb0-e30aad5c5631","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.491Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Now we create a final transformation that will convert the PIL Images to Tensors, scale them (0-1), and normalize them\nnorm_transform = transforms.Compose([\n                                     transforms.ToImage(), # Converts PIL to tensor (uint8)\n                                     transforms.ToDtype(torch.float32, scale=True), # Converts to float32 and scales to [0.0, 1.0]\n                                     transforms.Normalize((mean,), (std,)) # Normalizes using calculated stats                      \n                                    ])","metadata":{"_uuid":"bd751b7f-2a15-4733-a645-ab6b0aed481d","_cell_guid":"6fd24159-5ade-4b93-9950-cb3977d06480","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.491Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Apply transformation to datasets","metadata":{"_uuid":"a058b408-eecc-477a-a5f6-ba1d9808d7fe","_cell_guid":"9711c496-9120-4508-bcad-47a5617c8448","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# This converts every datapoint in the original dataset to scaled and normalized torch tensor\ntrain_set.transform = norm_transform\ntest_set.transform = norm_transform","metadata":{"_uuid":"99f3b45d-c2dd-463b-9939-fa4ae6341104","_cell_guid":"665e438f-b22d-4e29-9105-1fdb53b3e385","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# If we inspect an image from the training_set, we see that it has been normalized (mean approx 0, std approx 1)\nimg_transformed, _ = train_set[0]\nimg_transformed.min(), img_transformed.max()","metadata":{"_uuid":"092e2d94-85e6-4694-9d69-b696f2ee7216","_cell_guid":"1cdc6884-9ef3-4bf0-9fe3-97953453759c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Display the normalized image (it might look weird due to normalization)\nplt.imshow(img_transformed.squeeze(), cmap='gray');","metadata":{"_uuid":"a6ac0ce0-a623-4eee-8701-23b3dba3a578","_cell_guid":"a3e1c810-fd84-43c8-b2d1-7bd4b4464e27","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create a validation set","metadata":{"_uuid":"88c1c42b-e690-410b-9f4a-9498c2df4bb1","_cell_guid":"65b978dd-f91a-4fc5-9dcc-ac3830e6ab95","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Create validation/train split from the original training set\nvalidation_size = int(0.20 * len(train_set)) # Use 20% for validation\ntrain_size = len(train_set) - validation_size\n\n# Make sure to split the dataset that has the final transform applied\ntrain_subset, valid_subset = torch.utils.data.random_split(\n    train_set, [train_size, validation_size], generator=torch.Generator().manual_seed(42)\n)\nlen(train_subset), len(valid_subset)","metadata":{"_uuid":"ed9a110c-8c08-4527-bd94-7548afe24b5c","_cell_guid":"d26ca423-3aad-4efc-856d-677ab08a77ee","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create dataloaders","metadata":{"_uuid":"00ca351a-ce0f-4f1c-81b2-d3db4cec67cf","_cell_guid":"9807eb4c-1f46-4556-9b35-af8c0c89a3cf","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Create data loaders\nbatch_size = 64 # Increased batch size often helps with GPU utilization\n# Notice that we shuffle the training loader, but not the validation or test loaders.\n# This practice of shuffling the training set is one of the techniques that have been shown to improve training.\n# It is described on LeCun's Efficient Backprop publication, but its origin is likely much earlier.\ntrain_loader = DataLoader(train_subset, batch_size=batch_size, shuffle=True, num_workers=2, pin_memory=True)\nvalid_loader = DataLoader(valid_subset, batch_size=batch_size * 2, shuffle=False, num_workers=2, pin_memory=True) # Larger batch for validation\ntest_loader  = DataLoader(test_set, batch_size=batch_size * 2, shuffle=False, num_workers=2, pin_memory=True)","metadata":{"_uuid":"ed05f82b-828d-4c75-9175-fcb37a80acbe","_cell_guid":"2655e8ec-597d-4ec0-9bc5-50077b4cb99f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# The batch size is the first dimension of the images tensor. Next comes the number of channels and x, y dimensions.\n# We have labels equal to batch size, one for each datapoint in the batch. \nimages, labels = next(iter(train_loader))\nimages.shape, labels.shape","metadata":{"_uuid":"68de1780-8225-4b77-9754-4146e922ac7a","_cell_guid":"910f62ee-e6c8-4b60-b1ca-1d80212b20fa","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Defining the model","metadata":{"_uuid":"66a17385-226a-4a46-a57f-1c8c5e613ce3","_cell_guid":"90bf53fc-0bf0-4923-a4f7-053b19011202","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/andandandand/practical-computer-vision/refs/heads/main/images/lenet5-architecture.png)\n","metadata":{}},{"cell_type":"markdown","source":"There are some important details that we need to consider and change from the architecture described in the diagram.\n\n1. The input is being shown as being the letter \"A\", but we have 10 output units (each matching one of the digits from 0 to 9). If we wanted to recognize the letters of the alphabet (A-Z), we would need 26 output units. (This might be an error/omission from the original paper.)\n2. The data that we have has input size 28x28, not 32x32. In order to create feature maps as those produced on C1, we apply a convolution kernel with size 5x5, stride = 1, and padding = 2.\n3. In the original paper the [hyperbolic tangent activation function](https://paperswithcode.com/method/tanh-activation) was used between hidden layers. We will try first the [ReLU activation function](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) as it creates less numerical errors, however feel free to change the architecture below and replace [ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) with [Tanh](https://pytorch.org/docs/stable/generated/torch.nn.Tanh.html).","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass LeNet5_MNIST(nn.Module):\n    \"\"\"\n    LeNet-5 architecture adapted for MNIST with 28x28 input image size.\n    Follows the layer dimensions specified in the architecture diagram.\n    \"\"\"\n    def __init__(self):\n        super(LeNet5_MNIST, self).__init__()\n        # Convolutional Layer 1 (C1)\n        # Input: (N, 1, 28, 28)\n        # Output: (N, 6, 28, 28)\n        # Kernel: 5x5, Stride: 1, Padding: 2 (to maintain size 28x28)\n        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5, stride=1, padding=2)\n\n        # Subsampling Layer 2 (S2) - Average Pooling\n        # Input: (N, 6, 28, 28)\n        # Output: (N, 6, 14, 14)\n        # Kernel: 2x2, Stride: 2\n        self.pool1 = nn.AvgPool2d(kernel_size=2, stride=2)\n\n        # Convolutional Layer 3 (C3)\n        # Input: (N, 6, 14, 14)\n        # Output: (N, 16, 10, 10)\n        # Kernel: 5x5, Stride: 1, Padding: 0\n        self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5, stride=1, padding=0)\n\n        # Subsampling Layer 4 (S4) - Average Pooling\n        # Input: (N, 16, 10, 10)\n        # Output: (N, 16, 5, 5)\n        # Kernel: 2x2, Stride: 2\n        self.pool2 = nn.AvgPool2d(kernel_size=2, stride=2)\n\n        # Convolutional Layer 5 (C5) - Often treated as a FC layer precursor\n        # Input: (N, 16, 5, 5)\n        # Output: (N, 120, 1, 1)\n        # Kernel: 5x5, Stride: 1, Padding: 0\n        self.conv3 = nn.Conv2d(in_channels=16, out_channels=120, kernel_size=5, stride=1, padding=0)\n\n        # Fully Connected Layer 6 (F6)\n        # Input: 120 features (flattened from C5 output)\n        # Output: 84 features\n        self.fc1 = nn.Linear(in_features=120, out_features=84)\n\n        # Output Layer (Fully Connected)\n        # Input: 84 features\n        # Output: 10 features (for MNIST classes 0-9)\n        self.fc2 = nn.Linear(in_features=84, out_features=10)\n\n    def forward(self, x):\n        # Apply C1 -> ReLU -> S2\n        x = self.pool1(F.relu(self.conv1(x)))\n        # Apply C3 -> ReLU -> S4\n        x = self.pool2(F.relu(self.conv2(x)))\n        # Apply C5 -> ReLU\n        x = F.relu(self.conv3(x))\n\n        # Flatten the output for the fully connected layers\n        # Input shape: (N, 120, 1, 1) -> Output shape: (N, 120)\n        x = torch.flatten(x, 1) # Flatten all dimensions except batch\n\n        # Apply F6 -> ReLU\n        x = F.relu(self.fc1(x))\n        # Apply Output layer (logits)\n        x = self.fc2(x)\n        return x","metadata":{"_uuid":"42257382-45e9-4de3-ae6a-398f92e1d868","_cell_guid":"70b34051-862c-45f5-a626-f9a6bc20faf3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Checking the Model \n","metadata":{}},{"cell_type":"code","source":"model = LeNet5_MNIST()\nprint(model)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Checking that expected input and output shapes match\ndummy_input = torch.randn(1, 1, 28, 28)\noutput = model(dummy_input)\n# Should be [1, 10]\nprint(\"Output shape:\", output.shape) ","metadata":{"trusted":true,"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Loading the Model to the GPU","metadata":{}},{"cell_type":"code","source":"## Send the model to the GPU's VRAM\nmodel.to(device)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Check where the model's weights are \nnext(model.parameters()).device","metadata":{"_uuid":"b0220efa-83a9-4883-bbd6-5179f03c3ccf","_cell_guid":"1fd82e45-ba49-48f3-ad15-2d5ec9299f16","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Define Cross Entropy as the Loss Function\nPyTorch's [`nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) computes the disagreement between the probabilities of the ground truth labels and the predicted probabilities of the model. It works on logits and applies the log function to be more 'numerically stable' (i.e., do less computations with rounding errors). It takes advantage of the fact that $\\log(a / b) = \\log(a) - \\log(b)$","metadata":{"_uuid":"344a8a68-b3a9-4ab6-a9bb-fbd7e191abaa","_cell_guid":"a1705c6f-e103-4087-9cf6-a1843f4ab232","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"\\begin{align}\n\\text{CrossEntropyLoss}(x, y) &= \\text{NLLLoss}(\\log(\\text{Softmax}(x)), y) \\\\\n&= -\\log\\left(\\frac{\\exp(x_{y})}{\\sum_j \\exp(x_j)}\\right) \\\\\n&= -x_{y} + \\log\\left(\\sum_j \\exp(x_j)\\right)\n\\end{align}","metadata":{}},{"cell_type":"code","source":"loss_function = nn.CrossEntropyLoss()","metadata":{"_uuid":"0c70e367-8112-42b3-a57f-d9863919b963","_cell_guid":"1a9e7398-0b96-4247-a22b-e6091eabb731","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Inspecting the class balance in samples","metadata":{}},{"cell_type":"code","source":"# Plot class balance of the MNIST training set\nplt.figure(figsize=(10, 6))\n# The first column of train_data_df is 'label'\nclass_counts = train_data_df['label'].value_counts().sort_index()\n# Use the 'color' parameter to set a uniform color for all bars.\n# You can use a named color (e.g., 'skyblue', 'steelblue') or a hex code.\nsns.barplot(x=class_counts.index, y=class_counts.values, color=\"steelblue\") #. [2]\nplt.title('Class Distribution in MNIST Training Set')\nplt.xlabel('Digit Class')\nplt.ylabel('Number of Samples')\nplt.xticks(rotation=0) # Keep digit labels horizontal\nfor i, count in enumerate(class_counts.values):\n    plt.text(i, count + 50, str(count), ha='center', va='bottom') # Add count labels on top of bars\nplt.show()\n","metadata":{"trusted":true,"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Class counts:\\n\", class_counts)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n\n### Why Accuracy is 'Acceptable' as a Performance Metric for MNIST\n\nAccuracy is considered 'good enough' for MNIST classification because:\n\n1. The dataset has balanced classes (approximately equal numbers of each digit 0-9)\n2. Classification boundaries between digits are distinct with minimal ambiguity\n3. Misclassification errors have roughly equal importance\n4. It's easy to interpret and compare against published benchmarks\n\nFor deeper analysis, confusion matrices can reveal which specific digits get misclassified and for unbalanced datasets metrics like the [F1 score](https://en.wikipedia.org/wiki/F-score) are preferable. \n\nNote that the digit 1 is sligthly over-represented and that the digit 5 is slightly under-represented in the MNIST training set. To address issues of class imbalance, using precision and recall and the F1 metric is considered a better approach than the 'traditional' accuracy. ","metadata":{"_uuid":"fbeab32d-ea85-42f7-8d45-00e815f3ef84","_cell_guid":"11258f1c-0d81-45a9-a6ec-a854dd6f9d04","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"$$\n\\text{Accuracy} = \\frac{\\text{Number of Correct Predictions}}{\\text{Total Number of Predictions}}\n$$","metadata":{"_uuid":"e8412bc1-a840-47f2-8eb8-4a0d4b552a74","_cell_guid":"831ce24b-b28d-4548-a18e-625a67c33c4c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def get_batch_accuracy(output, y):\n    \"\"\"Implementing accuracy in a batch of samples\"\"\"\n    pred = output.argmax(dim=1, keepdim=True)\n    correct = pred.eq(y.view_as(pred)).sum().item()\n    return correct / output.shape[0]","metadata":{"_uuid":"6ec18a63-b0a9-4213-921f-06755f18cc18","_cell_guid":"2883fcfe-fff8-4845-82b6-16092d187bdd","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# F1 Macro Score in MNIST Context\n\nThe F1 macro score is a performance metric that averages the F1 scores across all classes, giving equal weight to each class regardless of its frequency in the dataset. In the MNIST context, where we have 10 digit classes (0-9), the F1 macro score helps evaluate how well a classifier performs across all digits. It's defined as \n\n$$F1_{macro} = \\frac{1}{n}\\sum_{i=1}^{n}F1_i$$ \n\nwhere $n=10$ for MNIST and each $F1_i$ is calculated as \n\n\n$$\\frac{2 \\times precision_i \\times recall_i}{precision_i + recall_i}$$ \n\nThis metric is valuable for MNIST because it prevents the model's performance on common digits from masking potential weaknesses in classifying less frequent digits.","metadata":{}},{"cell_type":"markdown","source":"## Lists to Track Metrics (Optional with Wandb)","metadata":{"_uuid":"01d49e9a-5fcc-4e74-97b7-fef9b6a8a70f","_cell_guid":"9ad3e663-abb7-4930-96b4-29dedb12cc18","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# We will use these for plotting locally \ntrain_losses = []\ntrain_accuracies = [] \nvalid_losses = []\nvalid_accuracies = []","metadata":{"_uuid":"1d7e3db7-809f-450e-9cdd-9e6ecc00424d","_cell_guid":"d65dca82-853d-4d08-8f93-762a5d59c825","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.492Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Training Config and Wandb Login/Initialization","metadata":{"_uuid":"4d650952-8f13-4351-97fb-4e9f4e5e217e","_cell_guid":"6c392bde-5a88-40f8-bccd-986e6c0372e9","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def get_training_config():\n    config = {\n        'learning_rate': 0.001,  # Learning rate for Adam\n        'epochs': 10, #  Number of training epochs\n        'min_lr': 1e-6,  # Minimum learning rate for scheduler\n        'batch_size': batch_size, # Log batch size\n        'optimizer': 'Adam', # Log optimizer type\n        'scheduler': 'CosineAnnealingLR', # Log scheduler type\n        'architecture': 'LeNet5',\n        'dataset': 'MNIST-Kaggle',\n        'mean': mean, # Log dataset mean\n        'std': std # Log dataset std\n    }\n    return config\n\nconfig = get_training_config()","metadata":{"_uuid":"ff5f14ed-ffab-4aba-9ddb-7ce8167100b3","_cell_guid":"0c7ef00a-c3df-4476-9ed0-1f6b0aa7d511","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Get wandb API key","metadata":{}},{"cell_type":"markdown","source":"Go to **Add ons -> Secrets -> Add Secret** to include the `wandb_api` secret.  Make sure to have the checkox on when starting the notebook. \n\nIf this step is missing, you will get errors such as 'No user secrets exist for kernel id 82334504 and label wandb_api.'","metadata":{}},{"cell_type":"code","source":"user_secrets = UserSecretsClient()\nwandb_api_key = user_secrets.get_secret(\"wandb_api\") \nwandb.login(key=wandb_api_key)\nanonymous = None\nprint(\"Wandb login successful.\")","metadata":{"_uuid":"ff5f14ed-ffab-4aba-9ddb-7ce8167100b3","_cell_guid":"0c7ef00a-c3df-4476-9ed0-1f6b0aa7d511","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- WANDB INIT --- \nwandb.init(\n    project=\"kaggle-lenet5-digit-recognition\", \n    name=f\"lenet5_adam_cos_{config['epochs']}epochs\", \n    config=config,\n    settings=wandb.Settings(start_method=\"fork\")\n)","metadata":{"_uuid":"ff5f14ed-ffab-4aba-9ddb-7ce8167100b3","_cell_guid":"0c7ef00a-c3df-4476-9ed0-1f6b0aa7d511","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Define Training (with Wandb Logging)","metadata":{"_uuid":"3de4f04c-1be8-4d06-95f3-32316bf41a75","_cell_guid":"ffb6cd5a-aae5-4039-a977-21333c053e3f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def train(epoch): # Pass epoch for logging step\n    total_loss = 0\n    total_accuracy = 0\n    \n    model.train()\n    for x, y in train_loader:\n        x, y = x.to(device), y.to(device)\n        output = model(x)\n        optimizer.zero_grad()\n        batch_loss = loss_function(output, y)\n        batch_loss.backward()\n        optimizer.step()\n        \n        total_loss += batch_loss.item()\n        total_accuracy += get_batch_accuracy(output, y)\n    \n    # Calculate average loss and accuracy\n    avg_loss = total_loss / len(train_loader)\n    avg_accuracy = total_accuracy / len(train_loader)\n    \n    # Store metrics (optional if only using wandb)\n    train_losses.append(avg_loss)\n    train_accuracies.append(avg_accuracy)\n    \n    # ---> WANDB LOGGING <--- \n    # Log training metrics along with the current learning rate\n    current_lr = optimizer.param_groups[0]['lr']\n    wandb.log({\"epoch\": epoch, \n               \"train_loss\": avg_loss, \n               \"train_accuracy\": avg_accuracy,\n               \"learning_rate\": current_lr})\n    \n    print(f'Train - Loss: {avg_loss:.4f} Accuracy: {avg_accuracy:.4f}')\n    return avg_loss, avg_accuracy","metadata":{"_uuid":"362f071f-db5a-4560-9551-a18d22957405","_cell_guid":"d85203cf-c227-40a6-afd3-d92cc96ee66d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Define Validation (with Wandb Logging)","metadata":{"_uuid":"484a5cd8-21d7-48ff-afb7-8687e41cfa4d","_cell_guid":"1c9dce2e-6418-4fdc-a833-1e6c7d161f0e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def validate(epoch): # Pass epoch for logging step\n    total_loss = 0\n    total_accuracy = 0\n    all_preds = [] # Store all predictions for F1/CM\n    all_targets = [] # Store all true labels for F1/CM\n\n    model.eval()\n    with torch.no_grad():\n        for x, y in valid_loader:\n            x, y = x.to(device), y.to(device)\n            output = model(x)\n\n            batch_loss = loss_function(output, y)\n            total_loss += batch_loss.item()\n            total_accuracy += get_batch_accuracy(output, y)\n\n            # --- ADDED FOR F1/CM ---\n            _, batch_preds = torch.max(output, 1)\n            all_preds.extend(batch_preds.cpu().numpy())\n            all_targets.extend(y.cpu().numpy())\n            # --- END ADDED ---\n\n    # Calculate average loss and accuracy\n    avg_loss = total_loss / len(valid_loader)\n    avg_accuracy = total_accuracy / len(valid_loader)\n\n    # Store metrics (optional if only using wandb)\n    valid_losses.append(avg_loss)\n    valid_accuracies.append(avg_accuracy)\n\n    # --- ADDED F1/CM CALCULATION ---\n    # Ensure there are predictions before calculating metrics\n    if len(all_targets) > 0 and len(all_preds) > 0:\n        f1 = f1_score(all_targets, all_preds, average='macro') # Use 'macro' or 'weighted' as needed\n        # cm = confusion_matrix(all_targets, all_preds) # Sklearn confusion matrix (optional if just logging wandb)\n        class_names = [str(i) for i in range(10)] # MNIST class names\n        # Log confusion matrix directly using wandb's utility\n        wandb_cm = wandb.plot.confusion_matrix(\n            preds=all_preds,\n            y_true=all_targets,\n            class_names=class_names,\n            title=f\"Validation Confusion Matrix, Epoch {epoch+1}\"\n        )\n    else:\n        f1 = 0.0 # Default value if no validation data processed\n        wandb_cm = None # No confusion matrix to log\n    # --- END ADDED ---\n\n\n    # ---> WANDB LOGGING <---\n    log_dict = {\n        \"epoch\": epoch,\n        \"val_loss\": avg_loss,\n        \"val_accuracy\": avg_accuracy,\n        \"val_f1_macro\": f1 # Log F1 score\n    }\n    if wandb_cm is not None:\n        log_dict[\"val_confusion_matrix\"] = wandb_cm # Log confusion matrix plot\n\n    wandb.log(log_dict)\n\n    print(f'Valid - Loss: {avg_loss:.4f} Accuracy: {avg_accuracy:.4f} F1 (macro): {f1:.4f}')\n\n    return avg_loss, avg_accuracy ","metadata":{"_uuid":"e15c11d0-d100-464f-a664-9ac7a0a6e65f","_cell_guid":"50fe912d-6178-45a4-8c74-deb9fc4a8b0f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Training loop with model checkpointing and Wandb Artifacts","metadata":{"_uuid":"79646f46-2c62-4ba6-bf86-c66e76dd3c9f","_cell_guid":"e3e9163b-75b7-4a94-919b-44d5a5aec4e6","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Training loop with model checkpointing and Wandb Artifacts\nepochs = config['epochs']\nbest_valid_accuracy = 0\n\n# Re-initialize optimizer and scheduler with config values\noptimizer = torch.optim.Adam(\n    model.parameters(),\n    lr=config['learning_rate'],\n)\nscheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n    optimizer, \n    T_max=config['epochs'],\n    eta_min=config['min_lr']\n)\n\n# ---> WATCH MODEL WITH WANDB <--- \n# Optional: gradients=False, log_freq=100 saves computational resources.\nwandb.watch(model, loss_function, log=\"all\", log_freq=len(train_loader)//2) # Log approx twice per epoch\n\n\nfor epoch in tqdm(range(epochs), desc='Epochs'):\n    print(f'\\nEpoch: {epoch+1}/{epochs}')\n    \n    train_loss, train_acc = train(epoch) # Pass epoch\n    valid_loss, valid_acc = validate(epoch) # Pass epoch\n\n    # Step the scheduler to change the learning rate\n    scheduler.step()\n    \n    # Save model if validation accuracy improves\n    if valid_acc > best_valid_accuracy:\n        best_valid_accuracy = valid_acc\n        model_save_path = 'best_model.pth' # Define path\n        torch.save(model.state_dict(), model_save_path)\n        print(f'Model saved locally! New best validation accuracy: {best_valid_accuracy:.4f}')\n        \n        # ---> SAVE BEST MODEL AS WANDB ARTIFACT <--- \n        artifact_name = f'lenet5-best-epoch{epoch+1}' # Use epoch+1 for 1-based indexing\n        artifact = wandb.Artifact(\n            artifact_name, \n            type='model',\n            description=f'Best LeNet5 model based on validation accuracy up to epoch {epoch+1}',\n            metadata={'epoch': epoch+1, 'val_accuracy': best_valid_accuracy, 'val_loss': valid_loss} \n        )\n        artifact.add_file(model_save_path) # Add the saved model file\n        wandb.log_artifact(artifact, aliases=[\"best\", f\"epoch-{epoch+1}\"]) # Log the artifact, use aliases (overwrite best)\n        print(f'Best model saved as wandb artifact: {artifact_name}')\n\n","metadata":{"_uuid":"d2fd8221-b548-4fac-ba39-47c1f060b980","_cell_guid":"ee4ecdd5-24e8-4144-a610-4b944770209f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Plot training curves","metadata":{"_uuid":"1b10d1c9-162d-4066-b164-b9cb49b564eb","_cell_guid":"0ab64a58-d170-40bf-a088-dd7b998638a4","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# We want to see both validation and training loss going down\nplt.figure(figsize=(10, 5))\n\n# Plot losses\nplt.subplot(1, 2, 1)\nplt.plot(train_losses, label='Training Loss')\nplt.plot(valid_losses, label='Validation Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title('Training vs Validation Loss')\nplt.legend()\n\n# Plot accuracies \nplt.subplot(1, 2, 2)\nplt.plot(train_accuracies, label='Training Accuracy')\nplt.plot(valid_accuracies, label='Validation Accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.title('Training vs Validation Accuracy')\nplt.legend()\n\nplt.tight_layout()\nplt.show()","metadata":{"_uuid":"3598fb45-12e1-48b4-a063-e428db499e09","_cell_guid":"e4449b8a-a456-46c1-8e7d-5762962b0384","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load best model (from local file for submission, artifact is in wandb)\n# Ensure the file exists before loading\nif os.path.exists('best_model.pth'):\n    model.load_state_dict(torch.load('best_model.pth', weights_only=True))\n    print(f'Loaded best model with validation accuracy: {best_valid_accuracy:.4f}')\nelse:\n    print(\"Best model file 'best_model.pth' not found. Using the model from the last epoch.\")","metadata":{"_uuid":"3598fb45-12e1-48b4-a063-e428db499e09","_cell_guid":"e4449b8a-a456-46c1-8e7d-5762962b0384","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Prepare submission.csv file","metadata":{}},{"cell_type":"code","source":"\nif wandb.run is not None: # Check if a run is active (optional, good practice)\n    # If you called wandb.finish() before this, wandb.run will be None.\n    # The hooks are still on the model regardless.\n    # So, unwatch is generally needed if watch was ever called on this model instance.\n    pass # No need to do anything with wandb.run itself if it's None\n\n# Crucially, unwatch the model to remove the hooks\n# This should be done even if wandb.run is None, if watch was called previously.\ntry:\n    wandb.unwatch(model)\nexcept AttributeError:\n    # This can happen if wandb.watch was never called, or if wandb itself\n    # is not in a state where unwatch is meaningful (e.g., wandb.run is None\n    # and it tries to do something internally that fails).\n    # It's usually safe to ignore this specific error during unwatch if\n    # the main goal is to prevent the logging hooks from firing.\n    print(\"wandb.unwatch(model) failed, possibly because watch was not called or run already finished.\")\n    pass\n\n\n","metadata":{"_uuid":"8d264f99-114c-4ed5-b323-28399604c1ec","_cell_guid":"a0ca1e39-d51b-4d08-8ff8-f73f9b605a91","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Evaluate on the test data and prepare a submission\nmodel.eval()\npredictions = []\nwith torch.no_grad():\n    for images, _ in tqdm(test_loader, desc=\"Generating Predictions\"):\n        # Ensure images are on the correct device\n        images = images.to(device) # Make sure this was here or add it\n        outputs = model(images)    # Now this should not trigger wandb hooks\n        _, preds = torch.max(outputs, 1)\n        predictions.extend(preds.cpu().numpy())\n\n# Prepare submission\nsubmission = pd.DataFrame({'ImageId': np.arange(1, len(predictions) + 1), 'Label': predictions})\nsubmission.to_csv('submission.csv', index=False)\n\nprint(\"\\nSubmission file created: 'submission.csv'\")","metadata":{"_uuid":"8d264f99-114c-4ed5-b323-28399604c1ec","_cell_guid":"a0ca1e39-d51b-4d08-8ff8-f73f9b605a91","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check out the format of the submission file\npd.read_csv('submission.csv').head(5)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-05-06T13:34:09.493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Exercises\n\n* Modify the notebook and log your experiments with Weights and Biases. For this you will need to create an account on https://wandb.ai/. Refer to [this notebook](https://www.kaggle.com/code/andandand/wandb-resnet-applebananaorange) to give you guidance on how to log experiments on wandb through Kaggle.\n* Try checkpointing the best model based on the [F1 (macro) metric](https://www.kdnuggets.com/2023/01/micro-macro-weighted-averages-f1-score-clearly-explained.html), instead of accuracy. \n* Think about the problem of [data leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning)), that is having testing data in the training set. How could a Kaggler use it to cheat and achieve a \"perfect\" score of 1.0 accuracy? You can refer to this [Github repo](https://github.com/tesla-is/MNIST-Kaggle-Competition-The-Winning-Solution) and [its discussion on Kaggle](https://www.kaggle.com/c/digit-recognizer/discussion/61480) for ideas.","metadata":{"_uuid":"bcc3116e-1649-4ae7-a046-f875637c0a36","_cell_guid":"9f72386a-7e91-4ac2-9cac-11381dd8a8c9","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}