{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Convolutional Neural Network\n\n*This is my PyTorch CNN Models submission for the Digit Recognizer competition.*\n\nI will define three CNN models and train them on augmented and normalized data.\n\nBut before we start what are CNNs ?\n\n## What is a CNN\n\nA **Convolutional Neural Network (CNN)**, also called a **ConvNet**, is a specialized type of deep learning architecture primarily designed for tasks that involve object recognition, such as image classification, detection, and segmentation.\n\n## Architectural Foundation\n\nCNNs are constructed using a combination of specialized layers:\n- **Convolutional layer:**\n - designed to automatically and adaptively learn spatial hierarchies of features through layers of convolutions and *detect and extract specific features such as edges, textures, or patterns*. \n - It operates by applying a set of learnable filters (kernels) across the input data, performing element-wise multiplications and summing the results to produce feature maps.\n - These filters *convert the pixels within their receptive fields into single values*, effectively ***reducing the spatial dimensions*** of the data ***while preserving essential information***.\n - Compresses the input and highlights the most salient features. ***Avoiding overfitting without sacrificing model performance. It makes the model focus on general aspects, features and patterns rather than details***.\n- **Non Linearity (ReLU):** Rectified Linear Unit (ReLU) *introduces ***non-linearity****, *enabling the network to learn complex patterns and representations*.\n- **Pooling layer:** **reduces spatial dimensions** while **preserving essential information**, decreasing the number of parameters and computations in the network. This downsampling *helps control overfitting*, enhances computational efficiency, and *promotes spatial invariance*.\n- **Fully-connected (FC) layer:** plays a pivotal role in *interpreting the features* extracted by preceding layers to make final predictions. After convolutional and pooling layers have distilled the input data into *high-level feature representations*, these features are flattened into a one-dimensional vector and passed to the FC layer. It allows the network to learn complex, non-linear combinations of features.\n\n## Feature Hierarchy and Representation Learning\n\nAs data propagates through the network, CNNs perform hierarchical feature extraction:\n\n- Initial layers capture ***low-level features*** (edges, textures, color gradients).\n- Middle layers detect ***intermediate structures*** (corners, contours, simple shapes).\n- Deeper layers identify ***high-level abstractions*** (object parts, complex patterns).\n- Final layers integrate these representations for comprehensive scene understanding.\n\n## Project Implementation\n\nThis project explores **CNN** performance on the ***MNIST*** handwritten digit dataset using ***PyTorch***. We'll implement a systematic comparison across multiple architectural variations and preprocessing techniques.\n\n### Experimental Design\n- ***Data Preparation:***\n 1. *Normalization:* Scaling pixel values from the integer range *$[0-255]$* to floating-point range *$[0.0-1.0]$*.\n 2. *Data Augmentation:* Improving the dataset with applying variations on the current data without changing the label.\n- ***Model Architectures:***\n - **SmallCNN:** Lightweight architecture with minimal layers and parameters.\n - **MediumCNN:** Balanced network with moderate complexity.\n - **BigCNN:** Deep architecture with enhanced representational capacity.\n\nWe will rigorously evaluate each model's performance, analyzing metrics including accuracy, training efficiency, convergence behavior, and generalization capabilities.\n\nWe will submit the three prediction files to the competition.\n\n

Contact Me

\n\n

\n If you notice anything lacking, spot an issue with this notebook, or have suggestions for improvements, feel free to reach out through any of the platforms below:\n

\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n \n \"LinkedIn\n \n
\n \n \"Twitter\n \n
\n \n \"Gmail\n \n
\n \n \"GitHub\n \n
","metadata":{}},{"cell_type":"markdown","source":"# Lab Implementation\n\n## Lab Setup\n\n1. Importing needed packages.\n2. Importing the datasets.\n3. Setting up the Dataset class to enable data augmentation.\n4. Setting a function that returns the loaders for training and test.\n5. Setting up functions for evaluation and data exploration.","metadata":{}},{"cell_type":"code","source":"import os\nimport time\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import TensorDataset, DataLoader, Dataset\n\nfrom torchvision import transforms\nfrom PIL import Image\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:05.607219Z","iopub.execute_input":"2025-05-20T14:29:05.607661Z","iopub.status.idle":"2025-05-20T14:29:18.616912Z","shell.execute_reply.started":"2025-05-20T14:29:05.607630Z","shell.execute_reply":"2025-05-20T14:29:18.615975Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nprint(f\"Dataset shape: {df.shape}\")\nprint(f\"Data types:\\n{df.dtypes.head()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:18.617826Z","iopub.execute_input":"2025-05-20T14:29:18.618252Z","iopub.status.idle":"2025-05-20T14:29:21.499699Z","shell.execute_reply.started":"2025-05-20T14:29:18.618221Z","shell.execute_reply":"2025-05-20T14:29:21.498901Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"y = df['label']\nX = df.drop(['label'], axis=1).values.reshape(-1, 28, 28).astype(np.float32) / 255.0\n\ndel df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:21.501554Z","iopub.execute_input":"2025-05-20T14:29:21.502011Z","iopub.status.idle":"2025-05-20T14:29:21.734083Z","shell.execute_reply.started":"2025-05-20T14:29:21.501989Z","shell.execute_reply":"2025-05-20T14:29:21.733415Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Understanding Labels distribution:\")\nprint(y.value_counts())\ny = y.values.astype(np.int64)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:21.734864Z","iopub.execute_input":"2025-05-20T14:29:21.735127Z","iopub.status.idle":"2025-05-20T14:29:21.746435Z","shell.execute_reply.started":"2025-05-20T14:29:21.735099Z","shell.execute_reply":"2025-05-20T14:29:21.745473Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(x=pd.Series(y),palette=\"viridis\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:21.747193Z","iopub.execute_input":"2025-05-20T14:29:21.747467Z","iopub.status.idle":"2025-05-20T14:29:22.011358Z","shell.execute_reply.started":"2025-05-20T14:29:21.747441Z","shell.execute_reply":"2025-05-20T14:29:22.010590Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Checking Null values in the dataset\")\nprint(np.isnan(X).any())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.012180Z","iopub.execute_input":"2025-05-20T14:29:22.012452Z","iopub.status.idle":"2025-05-20T14:29:22.063718Z","shell.execute_reply.started":"2025-05-20T14:29:22.012430Z","shell.execute_reply":"2025-05-20T14:29:22.062959Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"There are no nulls and the labels are distributed evenly.\n\nThis verification is needed: If the labels are not evenly distributed this will lead to a **bias** toward a certain class making our model less trustworthy.\n\nIf this was the case, we can mitigate for this by augmenting our data a bit: mainly using translation vectors. This method do not affect the label of the picture. So using it is preferable over other means. This also helps us not worrying about deleting the excess data.\n\nLuckily, this is not the case.","metadata":{}},{"cell_type":"markdown","source":"### Data Augmentation and Data Loaders\n\nHere we define the `MNISTDataset` class that inherits from the `Dataset` class to enable data augmentation.","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 64","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.064533Z","iopub.execute_input":"2025-05-20T14:29:22.064839Z","iopub.status.idle":"2025-05-20T14:29:22.069021Z","shell.execute_reply.started":"2025-05-20T14:29:22.064810Z","shell.execute_reply":"2025-05-20T14:29:22.068307Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class MNISTDataset(Dataset):\n def __init__(self, X, y, transform=None, augment=False, augment_factor=4):\n \"\"\"\n Custom dataset for MNIST with optional augmentation.\n \n Parameters:\n - X: Input features (numpy array, shape: [n_samples, 28, 28], normalized to [0, 1])\n - y: Labels (numpy array, shape: [n_samples])\n - transform: PyTorch transform for tensor conversion\n - augment: Whether to apply augmentation at runtime\n - augment_factor: Number of augmented versions per sample (if augment=True)\n \"\"\"\n assert X.shape[1:] == (28, 28), f\"Expected X shape [n, 28, 28], got {X.shape}\"\n assert X.dtype == np.float32, f\"Expected X dtype float32, got {X.dtype}\"\n assert X.min() >= 0.0 and X.max() <= 1.0, f\"Expected X in [0, 1], got [{X.min()}, {X.max()}]\"\n \n self.X = X\n self.y = y\n self.transform = transform\n self.augment = augment\n self.augment_factor = augment_factor if augment else 1\n\n if self.augment:\n self.aug_transform = transforms.Compose([\n transforms.RandomAffine(\n degrees=5,\n translate=(0.04, 0.04),\n scale=(0.98, 1.02),\n )\n ])\n\n def __len__(self):\n return len(self.X) * self.augment_factor\n\n def __getitem__(self, idx):\n orig_idx = idx // self.augment_factor\n image = self.X[orig_idx]\n label = self.y[orig_idx]\n\n # Convert to PIL for augmentation or transform\n pil_image = Image.fromarray((image * 255).astype(np.uint8))\n\n if self.augment and idx % self.augment_factor != 0:\n # Apply augmentation for non-original samples\n image = self.aug_transform(pil_image)\n else:\n image = pil_image\n\n # Apply transform (includes ToTensor for normalization to [0, 1])\n if self.transform:\n image = self.transform(image)\n else:\n # Fallback to tensor conversion if transform is None\n image = transforms.ToTensor()(image)\n \n return image, torch.tensor(label, dtype=torch.long)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.069920Z","iopub.execute_input":"2025-05-20T14:29:22.070200Z","iopub.status.idle":"2025-05-20T14:29:22.084429Z","shell.execute_reply.started":"2025-05-20T14:29:22.070179Z","shell.execute_reply":"2025-05-20T14:29:22.083579Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_loaders(X, y, test_size=0.2, augment_factor=4):\n \"\"\"\n Prepare data loaders for training and testing with optional augmentation.\n \n Parameters:\n - X: Features (shape: [n_samples, 784] or [n_samples, 28, 28])\n - y: Labels\n - test_size: Proportion of data for testing\n - augment_factor: Number of augmented versions per training sample\n \n Returns:\n - train_loader, test_loader: PyTorch data loaders\n \"\"\"\n assert isinstance(X, np.ndarray) and isinstance(y, np.ndarray), \"X and y must be NumPy arrays\"\n assert X.shape[1:] == (28, 28), f\"Expected X shape [n, 28, 28], got {X.shape}\"\n assert X.dtype == np.float32, f\"Expected X dtype float32, got {X.dtype}\"\n assert X.min() >= 0.0 and X.max() <= 1.0, f\"Expected X in [0, 1], got [{X.min()}, {X.max()}]\"\n\n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, shuffle=True)\n print(f\"X shape: {X.shape}, dtype: {X.dtype}\")\n print(f\"y shape: {y.shape}, dtype: {y.dtype}\")\n print(f\"Training set shape: {X_train.shape}\")\n print(f\"Test set shape: {X_test.shape}\")\n\n train_transform = transforms.Compose([\n transforms.ToTensor(),\n ])\n test_transform = transforms.Compose([transforms.ToTensor()])\n \n train_dataset = MNISTDataset(X_train, y_train, transform=train_transform, augment=True, augment_factor=augment_factor)\n test_dataset = MNISTDataset(X_test, y_test, transform=test_transform, augment=False)\n \n print(f\"Train dataset size: {len(train_dataset)}\")\n print(f\"Test dataset size: {len(test_dataset)}\")\n \n # Create data loaders\n train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4, pin_memory=True)\n test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=4, pin_memory=True)\n \n return train_loader, test_loader","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.087930Z","iopub.execute_input":"2025-05-20T14:29:22.088218Z","iopub.status.idle":"2025-05-20T14:29:22.101847Z","shell.execute_reply.started":"2025-05-20T14:29:22.088190Z","shell.execute_reply":"2025-05-20T14:29:22.100984Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visualization functions\n\n1. ``visualize_samples``: Display sample images from the dataset.\n2. ``plot_training_history``: Plot training history for multiple models.\n3. ``plot_model_comparison``: Compare test accuracies of different models.\n4. ``plot_confusion_matrix``: Generate and plot confusion matrix for a model.","metadata":{}},{"cell_type":"code","source":"def visualize_samples(loader, nrow=2, ncol=5):\n \"\"\"Quickly visualize a grid of sample images from a DataLoader.\"\"\"\n images, labels = next(iter(loader))\n images, labels = images[:nrow*ncol], labels[:nrow*ncol]\n\n fig, axes = plt.subplots(nrow, ncol, figsize=(8, 6))\n axes = axes.flatten()\n\n for img, label, ax in zip(images, labels, axes):\n ax.imshow(img.squeeze(0), cmap='gray')\n ax.set_title(f\"{label.item()}\")\n ax.axis('off')\n\n plt.tight_layout()\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.102738Z","iopub.execute_input":"2025-05-20T14:29:22.103010Z","iopub.status.idle":"2025-05-20T14:29:22.117542Z","shell.execute_reply.started":"2025-05-20T14:29:22.102983Z","shell.execute_reply":"2025-05-20T14:29:22.116689Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_training_history(histories, model_names):\n \"\"\"Plot training loss and accuracy for multiple models.\"\"\"\n fig, axes = plt.subplots(1, 2, figsize=(8, 5))\n titles = ['Training Loss', 'Training Accuracy']\n y_labels = ['Loss', 'Accuracy (%)']\n metrics = ['train_loss', 'train_acc']\n\n for ax, title, y_label, metric in zip(axes, titles, y_labels, metrics):\n for history, name in zip(histories, model_names):\n epochs = range(1, len(history[metric]) + 1)\n ax.plot(epochs, history[metric], marker='o', label=name)\n ax.set_title(title)\n ax.set_xlabel('Epochs')\n ax.set_ylabel(y_label)\n ax.legend()\n ax.grid(True)\n\n plt.tight_layout()\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.118388Z","iopub.execute_input":"2025-05-20T14:29:22.118770Z","iopub.status.idle":"2025-05-20T14:29:22.135469Z","shell.execute_reply.started":"2025-05-20T14:29:22.118748Z","shell.execute_reply":"2025-05-20T14:29:22.134587Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_model_comparison(model_results):\n \"\"\"Compare test accuracies of different models.\"\"\"\n models = list(model_results.keys())\n accuracies = [model_results[model]['accuracy'] for model in models]\n\n fig, ax = plt.subplots(figsize=(8, 5))\n bars = ax.bar(models, accuracies, color='skyblue')\n max_height = 0\n \n # Add value labels on top of bars\n for bar in bars:\n height = bar.get_height()\n if height > max_height:\n max_height = height\n ax.text(bar.get_x() + bar.get_width() / 2, height + 0.5,\n f'{height:.2f}%', ha='center', va='bottom')\n\n ax.set_title('Model Accuracy Comparison')\n ax.set_ylabel('Test Accuracy (%)')\n ax.grid(axis='y', linestyle='--', alpha=0.7)\n ax.set_ylim(0, max_height + 4)\n plt.tight_layout()\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.136315Z","iopub.execute_input":"2025-05-20T14:29:22.136755Z","iopub.status.idle":"2025-05-20T14:29:22.150352Z","shell.execute_reply.started":"2025-05-20T14:29:22.136728Z","shell.execute_reply":"2025-05-20T14:29:22.149530Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_confusion_matrix(model, test_loader, device, model_name):\n \"\"\"Generate and plot confusion matrix for a model.\"\"\"\n model.eval()\n y_true, y_pred = [], []\n\n with torch.no_grad():\n for X_batch, y_batch in test_loader:\n X_batch = X_batch.to(device)\n y_batch = y_batch.to(device)\n outputs = model(X_batch)\n preds = outputs.argmax(dim=1)\n y_true.extend(y_batch.cpu().numpy())\n y_pred.extend(preds.cpu().numpy())\n\n cm = confusion_matrix(y_true, y_pred)\n cm_normalized = cm.astype('float') / cm.sum(axis=1, keepdims=True)\n\n plt.figure(figsize=(8, 5))\n sns.heatmap(cm_normalized, annot=True, fmt='.2f', cmap='Blues')\n plt.title(f'Confusion Matrix - {model_name}')\n plt.xlabel('Predicted')\n plt.ylabel('True')\n plt.tight_layout()\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.151201Z","iopub.execute_input":"2025-05-20T14:29:22.151501Z","iopub.status.idle":"2025-05-20T14:29:22.168858Z","shell.execute_reply.started":"2025-05-20T14:29:22.151475Z","shell.execute_reply":"2025-05-20T14:29:22.168035Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Models Definition\n\nHere we define our 3 models followed by their analysis and comparison based on their architectures.\n\nWe will later compare them based on their performance.","metadata":{}},{"cell_type":"code","source":"class SmallCNN(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 64, kernel_size=3)\n self.conv2 = nn.Conv2d(64, 32, kernel_size=3)\n self.flatten = nn.Flatten()\n self.fc = nn.Linear(32 * 24 * 24, 10)\n self.name = \"SmallCNN\"\n \n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.relu(self.conv2(x))\n x = self.flatten(x)\n return F.log_softmax(self.fc(x), dim=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.169726Z","iopub.execute_input":"2025-05-20T14:29:22.170024Z","iopub.status.idle":"2025-05-20T14:29:22.185036Z","shell.execute_reply.started":"2025-05-20T14:29:22.169998Z","shell.execute_reply":"2025-05-20T14:29:22.183942Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### SmallCNN Architecture\n\n| Layer | Type | Parameters | Input Shape | Output Shape | Activation | Notes |\n|-------|------|------------|-------------|--------------|------------|-------|\n| conv1 | Conv2d | 1→64, 3×3 kernel | [B, 1, 28, 28] | [B, 64, 26, 26] | ReLU | No padding, stride=1 |\n| conv2 | Conv2d | 64→32, 3×3 kernel | [B, 64, 26, 26] | [B, 32, 24, 24] | ReLU | No padding, stride=1 |\n| flatten | Flatten | - | [B, 32, 24, 24] | [B, 18432] | - | Flattens 3D to 1D |\n| fc | Linear | 18432→10 | [B, 18432] | [B, 10] | Log Softmax | Output layer |\n\n***Total Parameters:***\n1. **``conv1``**: 1 input channel → 64 output channels with 3×3 kernel:\n - Weights: 1 × 64 × 3 × 3 = 576\n - Biases: 64\n - Total: 576 + 64 = 640 parameters.\n2. **``conv2``**: 64 input channels → 32 output channels with 3×3 kernel:\n - Weights: 64 × 32 × 3 × 3 = 18,432\n - Biases: 32\n - Total: 18,432 + 32 = 18,464 parameters.\n3. **``fc``**: 32 × 24 × 24 input neurons → 10 output neurons:\n - Weights: (32 × 24 × 24) × 10 = 18,432 × 10 = 184,320\n - Biases: 10\n - Total: 184,320 + 10 = 184,330 parameters.\n\n**Total SmallCNN Parameters**: *640 + 18,464 + 184,330 = **203,434 parameters***.\n\n**Architecture Pattern:** Simple 2-layer CNN without pooling.\n\n**Notable Features:** \n- No pooling layers (maintains spatial dimensions).\n- Single fully connected layer.\n- No dropout regularization.\n- Higher initial filter count (64) compared to other models.","metadata":{}},{"cell_type":"code","source":"class MediumCNN(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, kernel_size=5)\n self.pool = nn.MaxPool2d(2, 2)\n self.drop = nn.Dropout(0.2)\n self.flatten = nn.Flatten()\n self.fc1 = nn.Linear(32 * 12 * 12, 128)\n self.fc2 = nn.Linear(128, 10)\n self.name = \"MediumCNN\"\n \n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = self.pool(x)\n x = self.drop(x)\n x = self.flatten(x)\n x = F.relu(self.fc1(x))\n return F.log_softmax(self.fc2(x), dim=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.186255Z","iopub.execute_input":"2025-05-20T14:29:22.186586Z","iopub.status.idle":"2025-05-20T14:29:22.222628Z","shell.execute_reply.started":"2025-05-20T14:29:22.186558Z","shell.execute_reply":"2025-05-20T14:29:22.221862Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### MediumCNN Architecture\n\n| Layer | Type | Parameters | Input Shape | Output Shape | Activation | Notes |\n|-------|------|------------|-------------|--------------|------------|-------|\n| conv1 | Conv2d | 1→32, 5×5 kernel | [B, 1, 28, 28] | [B, 32, 24, 24] | ReLU | No padding, stride=1 |\n| pool | MaxPool2d | 2×2, stride=2 | [B, 32, 24, 24] | [B, 32, 12, 12] | - | Reduces spatial dimensions |\n| drop | Dropout | p=0.2 | [B, 32, 12, 12] | [B, 32, 12, 12] | - | Regularization |\n| flatten | Flatten | - | [B, 32, 12, 12] | [B, 4608] | - | Flattens 3D to 1D |\n| fc1 | Linear | 4608→128 | [B, 4608] | [B, 128] | ReLU | Hidden layer |\n| fc2 | Linear | 128→10 | [B, 128] | [B, 10] | Log Softmax | Output layer |\n\n***Total Parameters:***\n1. **``conv1``**: 1 input channel → 32 output channels with 5×5 kernel:\n - Weights: 1 × 32 × 5 × 5 = 800\n - Biases: 32\n - Total: 800 + 32 = 832 parameters.\n2. **``fc1``**: 32 × 12 × 12 input neurons → 128 output neurons:\n - Weights: (32 × 12 × 12) × 128 = 4,608 × 128 = 589,824\n - Biases: 128\n - Total: 589,824 + 128 = 589,952 parameters.\n3. **``fc2``**: 128 input neurons → 10 output neurons:\n - Weights: 128 × 10 = 1,280\n - Biases: 10\n - Total: 1,280 + 10 = 1,290 parameters.\n\n**Total MediumCNN Parameters**: *832 + 589,952 + 1,290 = **592,074 parameters***.\n\n**Architecture Pattern:** Single conv layer with pooling + 2 FC layers.\n\n**Notable Features:**\n- Includes MaxPooling for dimensionality reduction.\n- Dropout regularization (20%).\n- Two fully connected layers creating a bottleneck.\n- Larger 5×5 kernel in conv layer.","metadata":{}},{"cell_type":"code","source":"class BigCNN(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 30, kernel_size=5)\n self.pool1 = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(30, 15, kernel_size=3)\n self.drop = nn.Dropout(0.2)\n self.flatten = nn.Flatten()\n self.fc1 = nn.Linear(15 * 10 * 10, 128)\n self.fc2 = nn.Linear(128, 50)\n self.fc3 = nn.Linear(50, 10)\n self.name = \"BigCNN\"\n \n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = self.pool1(x)\n x = F.relu(self.conv2(x))\n x = self.drop(x)\n x = self.flatten(x)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n return F.log_softmax(self.fc3(x), dim=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.223502Z","iopub.execute_input":"2025-05-20T14:29:22.223783Z","iopub.status.idle":"2025-05-20T14:29:22.237751Z","shell.execute_reply.started":"2025-05-20T14:29:22.223758Z","shell.execute_reply":"2025-05-20T14:29:22.236886Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### BigCNN Architecture\n\n| Layer | Type | Parameters | Input Shape | Output Shape | Activation | Notes |\n|-------|------|------------|-------------|--------------|------------|-------|\n| conv1 | Conv2d | 1→30, 5×5 kernel | [B, 1, 28, 28] | [B, 30, 24, 24] | ReLU | No padding, stride=1 |\n| pool1 | MaxPool2d | 2×2, stride=2 | [B, 30, 24, 24] | [B, 30, 12, 12] | - | Reduces spatial dimensions |\n| conv2 | Conv2d | 30→15, 3×3 kernel | [B, 30, 12, 12] | [B, 15, 10, 10] | ReLU | No padding, stride=1 |\n| drop | Dropout | p=0.2 | [B, 15, 10, 10] | [B, 15, 10, 10] | - | Regularization |\n| flatten | Flatten | - | [B, 15, 10, 10] | [B, 1500] | - | Flattens 3D to 1D |\n| fc1 | Linear | 1500→128 | [B, 1500] | [B, 128] | ReLU | First hidden layer |\n| fc2 | Linear | 128→50 | [B, 128] | [B, 50] | ReLU | Second hidden layer |\n| fc3 | Linear | 50→10 | [B, 50] | [B, 10] | Log Softmax | Output layer |\n\n***Total Parameters:***\n1. **``conv1``**: 1 input channel → 30 output channels with 5×5 kernel:\n - Weights: 1 × 30 × 5 × 5 = 750\n - Biases: 30\n - Total: 750 + 30 = 780 parameters.\n2. **``conv2``**: 30 input channels → 15 output channels with 3×3 kernel:\n - Weights: 30 × 15 × 3 × 3 = 4,050\n - Biases: 15\n - Total: 4,050 + 15 = 4,065 parameters.\n3. **``fc1``**: 15 × 10 × 10 input neurons → 128 output neurons:\n - Weights: (15 × 10 × 10) × 128 = 1,500 × 128 = 192,000\n - Biases: 128\n - Total: 192,000 + 128 = 192,128 parameters.\n4. **``fc2``**: 128 input neurons → 50 output neurons:\n - Weights: 128 × 50 = 6,400\n - Biases: 50\n - Total: 6,400 + 50 = 6,450 parameters.\n5. **``fc3``**: 50 input neurons → 10 output neurons:\n - Weights: 50 × 10 = 500\n - Biases: 10\n - Total: 500 + 10 = 510 parameters.\n\n**Total BigCNN Parameters**: *780 + 4,065 + 192,128 + 6,450 + 510 = **203,933 parameters***.\n\n**Architecture Pattern:** 2 conv layers with pooling + 3 FC layers.\n\n**Notable Features:**\n- Deepest architecture with multiple conv and FC layers.\n- Progressive filter reduction (1→30→15).\n- Multiple fully connected layers (3 total).\n- Progressive dimensionality reduction.\n- Dropout regularization (20%).","metadata":{}},{"cell_type":"markdown","source":"### Pytorch Neural Network Library ``(torch.nn)`` Functions Explained\n\n| Function | Description |\n|----------|-------------|\n| nn.Conv2d | 2D convolutional layer that applies learnable filters to input data. Parameters include input/output channels and kernel size. |\n| nn.MaxPool2d | Downsamples input by taking maximum value in each window, reducing spatial dimensions while preserving important features. |\n| nn.Dropout | Randomly zeroes some elements during training to prevent overfitting by forcing the network to learn redundant representations. |\n| nn.Flatten | Reshapes input tensor by flattening all dimensions except batch, converting multi-dimensional data to 1D for fully connected layers. |\n| nn.Linear | Fully connected layer applying linear transformation (y = xA^T + b) to input data. |\n| F.relu | Rectified Linear Unit activation function that outputs max(0,x), introducing non-linearity to the model. |\n| F.log_softmax | Applies log of softmax function to normalize outputs into probability distribution while improving numerical stability. |\n\n## Model Architecture Comparison\n\n| Feature | SmallCNN | MediumCNN | BigCNN |\n|---------|----------|-----------|--------|\n| Convolutional Layers | 2 | 1 | 2 |\n| Max Pooling Layers | 0 | 1 | 1 |\n| Fully Connected Layers | 1 | 2 | 3 |\n| Dropout | No | Yes (0.2) | Yes (0.2) |\n| Parameter Count | 203,434 | 592,074 | 203,933 |\n| Input Filter Size | 3×3 | 5×5 | 5×5, 3×3 |\n| Filter Progression | 1→64→32 | 1→32 | 1→30→15 |\n| Output Activation | Log Softmax | Log Softmax | Log Softmax |","metadata":{}},{"cell_type":"markdown","source":"## Model Training\n\nHere we will define the functions for model training and evaluation.","metadata":{}},{"cell_type":"code","source":"def fit(model, train_loader, epochs=20, lr=0.001, use_gpu=True):\n \"\"\"Train the model and return training history and the trained model.\"\"\"\n \n device = torch.device(\"cuda\" if use_gpu and torch.cuda.is_available() else \"cpu\")\n if use_gpu and torch.cuda.is_available():\n print(f\"Using GPU: {torch.cuda.get_device_name(0)}\")\n else:\n print(\"Using CPU (GPU is not requested).\")\n \n model.to(device)\n optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n criterion = nn.NLLLoss()\n history = {'train_loss': [], 'train_acc': []}\n \n start = time.time()\n \n for epoch in range(1, epochs + 1):\n model.train()\n total_loss, correct = 0, 0\n \n for X, y in train_loader:\n X, y = X.to(device), y.to(device)\n optimizer.zero_grad()\n output = model(X)\n loss = criterion(output, y)\n loss.backward()\n optimizer.step()\n \n total_loss += loss.item()\n correct += (output.argmax(dim=1) == y).sum().item()\n \n avg_loss = total_loss / len(train_loader)\n accuracy = 100 * correct / len(train_loader.dataset)\n \n history['train_loss'].append(avg_loss)\n history['train_acc'].append(accuracy)\n \n print(f\"Epoch {epoch}: Loss={avg_loss:.4f}, Accuracy={accuracy:.2f}%\")\n \n print(f\"Training completed in {time.time() - start:.2f} seconds\")\n return history, model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.238731Z","iopub.execute_input":"2025-05-20T14:29:22.239002Z","iopub.status.idle":"2025-05-20T14:29:22.252443Z","shell.execute_reply.started":"2025-05-20T14:29:22.238981Z","shell.execute_reply":"2025-05-20T14:29:22.251681Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training Loop Components\n\nThe training loop in the *`fit`* function involves several key components:\n\n1. **Setup Phase**:\n - Device configuration with `torch.device`.\n - Optimizer initialization with `torch.optim.Adam`.\n - Loss function definition with `nn.NLLLoss`:***Negative Log Likelihood Loss function***.\n - History tracking initialization.\n2. **Per Epoch**:\n - Set model to training mode with `model.train()`.\n - Track loss and accuracy metrics.\n3. **Per Batch**:\n - Move data to device with `.to(device)`.\n - Reset gradients with `optimizer.zero_grad()`.\n - Forward pass with `model(X_batch)`.\n - Loss calculation with `criterion(output, y_batch)`.\n - Backward pass with `loss.backward()`.\n - Parameter update with `optimizer.step()`.\n - Calculate metrics with `output.argmax(dim=1)` and comparison operations.","metadata":{}},{"cell_type":"code","source":"def evaluate(model, test_loader, use_gpu=True):\n \"\"\"\n Evaluate a model and return accuracy and predictions\n \"\"\"\n device = torch.device(\"cuda\" if use_gpu and torch.cuda.is_available() else \"cpu\")\n if use_gpu and torch.cuda.is_available():\n print(f\"Evaluating on GPU: {torch.cuda.get_device_name(0)}\")\n else:\n print(\"Evaluating on CPU (GPU is not requested).\")\n \n model.to(device).eval()\n correct = 0\n predictions = []\n\n with torch.no_grad():\n for X_batch, y_batch in test_loader:\n X_batch, y_batch = X_batch.to(device), y_batch.to(device)\n output = model(X_batch)\n preds = output.argmax(dim=1)\n correct += (preds == y_batch).sum().item()\n predictions.extend(list(zip(y_batch.cpu().numpy(), preds.cpu().numpy())))\n \n acc = 100 * correct / len(test_loader.dataset)\n error_rate = 100 - acc\n print(f\"Test accuracy: {acc:.2f}%, Error rate: {error_rate:.2f}%\")\n\n return {\n 'accuracy': acc,\n 'error_rate': error_rate,\n 'predictions': predictions\n }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.253168Z","iopub.execute_input":"2025-05-20T14:29:22.253386Z","iopub.status.idle":"2025-05-20T14:29:22.266962Z","shell.execute_reply.started":"2025-05-20T14:29:22.253368Z","shell.execute_reply":"2025-05-20T14:29:22.266288Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Evaluation Components\n\nThe evaluation function uses:\n1. **Setup Phase**:\n - Device configuration similar to training.\n - Set model to evaluation mode with `.eval()`.\n2. **Inference**:\n - Disable gradient calculation with `torch.no_grad()`.\n - Forward pass through the model.\n - Calculate predictions with `output.argmax(dim=1)`.\n - Calculate accuracy metrics.","metadata":{}},{"cell_type":"code","source":"def workflow(model, train_loader, test_loader, use_gpu=True, epochs=20):\n \"\"\"\n Complete workflow for training and evaluating a model\n \"\"\"\n print(f\"\\n{'='*60}\")\n print(f\"Model: {model.name}\")\n print(f\"Architecture:\\n{model}\")\n print(f\"{'='*60}\")\n \n print(f\"\\nTraining and Fitting the Model:\")\n history, trained_model = fit(model, train_loader=train_loader, use_gpu=use_gpu, epochs=epochs)\n \n print(f\"\\nEvaluating the Model:\")\n results = evaluate(trained_model, test_loader=test_loader, use_gpu=use_gpu)\n \n return history, results, trained_model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.268269Z","iopub.execute_input":"2025-05-20T14:29:22.268572Z","iopub.status.idle":"2025-05-20T14:29:22.286390Z","shell.execute_reply.started":"2025-05-20T14:29:22.268550Z","shell.execute_reply":"2025-05-20T14:29:22.285634Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training and Evaluation Functions\n\n| Function | Description |\n|----------|-------------|\n| torch.device | Creates a device object representing where tensors and models will be allocated (CPU or GPU). |\n| torch.no_grad | Context manager that disables gradient calculation to reduce memory usage and speed up computations during evaluation. |\n| model.to(device) | Moves model parameters and buffers to specified device (CPU or GPU). |\n| model.to(device).eval() | Puts the model in evaluation mode, which disables layers like dropout and batch normalization from training behavior. |\n| torch.cuda.get_device_name | Returns the name of the CUDA device at the specified index, used to display GPU information. |\n| torch.optim.Adam(model.parameters(), lr=lr) | Creates Adam optimizer that implements adaptive learning rate optimization algorithm for stochastic gradient descent. |\n| nn.NLLLoss() | Negative Log Likelihood Loss function, commonly used with log_softmax outputs for classification tasks. |\n| model.train() | Sets the model to training mode, enabling layers like dropout and batch normalization to behave differently than in evaluation. |\n| optimizer.zero_grad() | Resets gradients of all model parameters to zero, preventing accumulated gradients from previous backward passes. |\n| output = model(X) | Forward pass of input through the model, producing predictions from the current model parameters. |\n| loss = criterion(output, y) | Calculates loss between model predictions and target labels using the specified criterion (loss function). |\n| loss.backward() | Computes gradients of the loss with respect to model parameters through backpropagation. |\n| optimizer.step() | Updates model parameters based on current gradients, using the optimization algorithm's update rule. |\n| output.argmax(dim=1) | Returns indices of maximum values along dimension 1, effectively giving the predicted class for each sample. |\n| (preds == y_batch).sum().item() | Counts correct predictions by comparing predicted classes with true labels. |\n\n### Related PyTorch Concepts\n- **Tensors:** Multi-dimensional arrays similar to NumPy arrays but can run on GPUs.\n- **Computational Graph:** Dynamic graph of operations tracked for automatic differentiation.\n- **Autograd:** PyTorch's automatic differentiation engine that calculates gradients.","metadata":{}},{"cell_type":"markdown","source":"## Setup for the experiments","metadata":{}},{"cell_type":"code","source":"def run_experiment(epochs=20):\n \"\"\"\n Run complete experiment with all models\n \"\"\"\n\n small_cnn = SmallCNN()\n medium_cnn = MediumCNN()\n big_cnn = BigCNN()\n \n models = [small_cnn, medium_cnn, big_cnn]\n\n model_names = [model.name for model in models]\n \n train_loader, test_loader = get_loaders(X, y)\n \n print(\"\\nVisualizing sample images from the dataset:\")\n visualize_samples(train_loader)\n \n # Run workflow for each model\n histories = []\n results = {}\n trained_models = {}\n \n for model in models:\n history, result, trained_model = workflow(model, train_loader, test_loader, epochs=epochs)\n histories.append(history)\n results[model.name] = result\n trained_models[model.name] = trained_model\n \n plot_training_history(histories, model_names)\n plot_model_comparison(results)\n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n for model_name, model in trained_models.items():\n plot_confusion_matrix(model, test_loader, device, model_name)\n plt.pause(0.5)\n return results, histories, trained_models","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.287191Z","iopub.execute_input":"2025-05-20T14:29:22.287434Z","iopub.status.idle":"2025-05-20T14:29:22.301839Z","shell.execute_reply.started":"2025-05-20T14:29:22.287409Z","shell.execute_reply":"2025-05-20T14:29:22.301004Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Running the experiments","metadata":{}},{"cell_type":"code","source":"results, histories, trained_models = run_experiment(epochs=20)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T14:29:22.302754Z","iopub.execute_input":"2025-05-20T14:29:22.303023Z","iopub.status.idle":"2025-05-20T15:57:18.809491Z","shell.execute_reply.started":"2025-05-20T14:29:22.302998Z","shell.execute_reply":"2025-05-20T15:57:18.808399Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Generate Predictions\n\nNow we will generate predictions for the test dataset.","metadata":{}},{"cell_type":"code","source":"def predict_with_models(models_dict, test_csv_path, output_dir='./submissions'):\n \"\"\"\n Make predictions on test data using trained models and export to CSV\n \n Parameters:\n - models_dict: Dictionary of trained models\n - test_csv_path: Path to the test CSV file\n - output_dir: Directory to save prediction CSV files\n \"\"\" \n os.makedirs(output_dir, exist_ok=True)\n \n # Load test data\n print(f\"Loading test data from {test_csv_path}...\")\n test_df = pd.read_csv(test_csv_path)\n print(f\"Test data shape: {test_df.shape}\")\n \n X_test = test_df.values.astype(np.float32)\n print(f\"X_test shape: {X_test.shape}\")\n \n # Always normalize test data\n print(\"Normalizing test data...\")\n X_test = X_test / 255.0\n \n print(f\"X_test values range: {X_test.min()} to {X_test.max()}\")\n \n # Define a simple dataset for the test data\n class TestDataset(Dataset):\n def __init__(self, X):\n self.X = X\n \n def __len__(self):\n return len(self.X)\n \n def __getitem__(self, idx):\n image = self.X[idx].reshape(28, 28, 1).astype(np.float32)\n image_tensor = torch.from_numpy(image).permute(2, 0, 1)\n return image_tensor\n \n test_dataset = TestDataset(X_test)\n test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=4, pin_memory=True)\n \n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n print(f\"Using device: {device}\")\n \n for model_name, model in models_dict.items():\n print(f\"\\nMaking predictions with {model_name}...\")\n \n model.to(device)\n model.eval()\n \n predictions = []\n start_time = time.time()\n \n with torch.no_grad():\n for batch in test_loader:\n batch = batch.to(device)\n outputs = model(batch)\n pred = outputs.argmax(dim=1)\n predictions.extend(pred.cpu().numpy())\n \n prediction_time = time.time() - start_time\n print(f\"Prediction completed in {prediction_time:.2f} seconds\")\n \n # Create submission DataFrame\n submission = pd.DataFrame({\n 'ImageId': range(1, len(predictions) + 1), # 1-indexed\n 'Label': predictions\n })\n \n # Save to CSV\n output_path = os.path.join(output_dir, f\"{model_name}_submission.csv\")\n submission.to_csv(output_path, index=False)\n print(f\"Saved predictions to {output_path}\")\n \n # Print summary statistics\n print(f\"Prediction statistics:\")\n print(submission['Label'].value_counts().sort_index())\n \n print(\"\\nAll predictions completed!\")\n return True","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T15:57:18.810981Z","iopub.execute_input":"2025-05-20T15:57:18.811394Z","iopub.status.idle":"2025-05-20T15:57:18.825090Z","shell.execute_reply.started":"2025-05-20T15:57:18.811351Z","shell.execute_reply":"2025-05-20T15:57:18.824213Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_test_data_and_predict(trained_models, test_csv_path='/kaggle/input/digit-recognizer/test.csv'):\n \"\"\"\n Load test data and make predictions with all models\n \n Parameters:\n - trained_models: Dictionary of trained models\n - test_csv_path: Path to the test CSV file\n \"\"\"\n \n output_dir = './submissions'\n \n print(\"\\n\" + \"=\"*80)\n print(\"GENERATING PREDICTIONS FOR SUBMISSION\")\n print(\"=\"*80)\n \n # Make predictions with normalized models on normalized data\n predict_with_models(trained_models, test_csv_path=test_csv_path, output_dir=output_dir)\n \n print(\"\\n\" + \"=\"*80)\n print(f\"All predictions completed and saved to {output_dir}\")\n print(\"=\"*80)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T15:57:18.825996Z","iopub.execute_input":"2025-05-20T15:57:18.826408Z","iopub.status.idle":"2025-05-20T15:57:18.845882Z","shell.execute_reply.started":"2025-05-20T15:57:18.826372Z","shell.execute_reply":"2025-05-20T15:57:18.844937Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"load_test_data_and_predict(trained_models)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-20T15:57:18.846930Z","iopub.execute_input":"2025-05-20T15:57:18.847259Z","iopub.status.idle":"2025-05-20T15:57:37.434890Z","shell.execute_reply.started":"2025-05-20T15:57:18.847231Z","shell.execute_reply":"2025-05-20T15:57:37.433807Z"}},"outputs":[],"execution_count":null}]}