{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"f5d4cf28","cell_type":"markdown","source":"# CIFAR-10 (Kaggle) — Challenge Assignment: Custom CNN, AdamW, Cosine Warmup, CutMix\n\nThis notebook satisfies the **Challenge**-tier requirements (as distinct from the earlier\nbaseline assignment):\n\n* Weights randomly initialized (no pretrained weights) — same requirement as before.\n* **A self-designed architecture** (`CIFARSEConvNet`), not a copy of a named published network\n  such as ResNet / VGG / MobileNet.\n* **AdamW** optimizer (not plain SGD / SGD+Momentum).\n* **Cosine annealing learning-rate schedule with a linear warmup phase** (not a constant LR).\n* **CutMix** advanced data augmentation (not just basic crop/flip, and not only Cutout).","metadata":{}},{"id":"58d8328f","cell_type":"markdown","source":"## 1. Setup","metadata":{}},{"id":"ede6779d","cell_type":"code","source":"import os, glob, time, math, random\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader, Dataset\nimport torchvision.transforms as T\nfrom PIL import Image\n\nSEED = 42\nrandom.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED); torch.cuda.manual_seed_all(SEED)\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Using device:', device)\nprint('Torch version:', torch.__version__)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"55958853","cell_type":"markdown","source":"## 2. Load the official CIFAR-10 training data\n\nSame as the baseline assignment: train on the official 50,000-image CIFAR-10 training set, hold\nout the official 10,000-image test set purely to estimate the Kaggle score offline. Loaded via\n`tf.keras.datasets.cifar10` (reliable on Kaggle's network) rather than the Kaggle PNG files.\n\nPer-image transforms stay basic (random crop + flip + normalize) — the *advanced* augmentation\n(CutMix) is applied at the **batch level inside the training loop** in Section 6, since CutMix\nmixes pairs of already-loaded images and their labels together.\n","metadata":{}},{"id":"d776b71f","cell_type":"code","source":"CIFAR_MEAN = (0.4914, 0.4822, 0.4465)\nCIFAR_STD  = (0.2470, 0.2435, 0.2616)\n\ntrain_tfms = T.Compose([\n    T.RandomCrop(32, padding=4, padding_mode='reflect'),\n    T.RandomHorizontalFlip(),\n    T.ToTensor(),\n    T.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\n\ntest_tfms = T.Compose([\n    T.ToTensor(),\n    T.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\n\nfrom tensorflow.keras.datasets import cifar10 as keras_cifar10\n(x_train, y_train), (x_test, y_test) = keras_cifar10.load_data()\ny_train = y_train.flatten()\ny_test = y_test.flatten()\n\nclass ArrayCIFAR10(Dataset):\n    def __init__(self, images, labels, transform):\n        self.images = images\n        self.labels = labels\n        self.transform = transform\n    def __len__(self):\n        return len(self.images)\n    def __getitem__(self, idx):\n        img = Image.fromarray(self.images[idx])\n        return self.transform(img), int(self.labels[idx])\n\nclasses = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n\ntrain_ds = ArrayCIFAR10(x_train, y_train, train_tfms)\ntest_ds  = ArrayCIFAR10(x_test,  y_test,  test_tfms)\nprint(classes)\n\nBATCH_SIZE = 512\ntrain_dl = DataLoader(train_ds, batch_size=BATCH_SIZE, shuffle=True,  num_workers=2, pin_memory=True, drop_last=True)\ntest_dl  = DataLoader(test_ds,  batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)\n\nprint(f'Train: {len(train_ds)}  Test: {len(test_ds)}')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"b3143068","cell_type":"markdown","source":"## 3. Model — `CIFARSEConvNet` (self-designed, randomly initialized)\n\nThis is an **original architecture** for this assignment, not a copy of a named published network\n(it is not ResNet, VGG, MobileNet, etc.). It combines standard, generic building blocks — 3×3\nconvolutions, batch norm, residual connections, and Squeeze-and-Excitation (SE) channel attention\n— into a configuration authored specifically for 32×32 CIFAR-10 images:\n\n* **Downsampling by strided convolution** (stride-2 3×3 conv) rather than max-pooling at every\n  stage, so the network learns how to downsample instead of using a fixed pooling rule.\n* **One residual+SE block per resolution stage** (4 stages: 64 → 128 → 256 → 512 channels), each\n  block containing two 3×3 conv-BN-ReLU layers plus a channel-attention (SE) gate before the\n  residual add.\n* **Global average pooling + dropout + a single linear head**, avoiding a large flattened\n  fully-connected layer.\n\nAll weights are randomly initialized (PyTorch's default init) — no checkpoint is loaded anywhere.\n","metadata":{}},{"id":"6d6bca19","cell_type":"code","source":"def conv_bn(ch_in, ch_out):\n    return nn.Sequential(\n        nn.Conv2d(ch_in, ch_out, kernel_size=3, padding=1, bias=False),\n        nn.BatchNorm2d(ch_out),\n        nn.ReLU(inplace=True),\n    )\n\nclass SEBlock(nn.Module):\n    \"\"\"Squeeze-and-Excitation channel attention gate.\"\"\"\n    def __init__(self, channels, reduction=16):\n        super().__init__()\n        reduced = max(channels // reduction, 4)\n        self.pool = nn.AdaptiveAvgPool2d(1)\n        self.fc1 = nn.Linear(channels, reduced)\n        self.fc2 = nn.Linear(reduced, channels)\n        self.relu = nn.ReLU(inplace=True)\n        self.sigmoid = nn.Sigmoid()\n\n    def forward(self, x):\n        b, c, _, _ = x.shape\n        s = self.pool(x).view(b, c)\n        s = self.relu(self.fc1(s))\n        s = self.sigmoid(self.fc2(s))\n        return x * s.view(b, c, 1, 1)\n\nclass ResSEBlock(nn.Module):\n    \"\"\"Two conv-BN-ReLU layers, an SE gate, and a residual (identity) skip connection.\"\"\"\n    def __init__(self, ch, reduction=16):\n        super().__init__()\n        self.conv1 = conv_bn(ch, ch)\n        self.conv2 = conv_bn(ch, ch)\n        self.se = SEBlock(ch, reduction)\n\n    def forward(self, x):\n        out = self.conv1(x)\n        out = self.conv2(out)\n        out = self.se(out)\n        return x + out\n\nclass DownBlock(nn.Module):\n    \"\"\"Learned downsampling: stride-2 3x3 conv instead of max-pooling.\"\"\"\n    def __init__(self, ch_in, ch_out):\n        super().__init__()\n        self.block = nn.Sequential(\n            nn.Conv2d(ch_in, ch_out, kernel_size=3, stride=2, padding=1, bias=False),\n            nn.BatchNorm2d(ch_out),\n            nn.ReLU(inplace=True),\n        )\n\n    def forward(self, x):\n        return self.block(x)\n\nclass CIFARSEConvNet(nn.Module):\n    def __init__(self, num_classes=10):\n        super().__init__()\n        self.stem   = conv_bn(3, 64)\n        self.block1 = ResSEBlock(64)\n        self.down1  = DownBlock(64, 128)\n        self.block2 = ResSEBlock(128)\n        self.down2  = DownBlock(128, 256)\n        self.block3 = ResSEBlock(256)\n        self.down3  = DownBlock(256, 512)\n        self.block4 = ResSEBlock(512)\n        self.pool    = nn.AdaptiveAvgPool2d(1)\n        self.dropout = nn.Dropout(0.2)\n        self.fc      = nn.Linear(512, num_classes)\n\n    def forward(self, x):\n        x = self.stem(x)\n        x = self.block1(x)\n        x = self.down1(x)\n        x = self.block2(x)\n        x = self.down2(x)\n        x = self.block3(x)\n        x = self.down3(x)\n        x = self.block4(x)\n        x = self.pool(x).flatten(1)\n        x = self.dropout(x)\n        return self.fc(x)\n\nmodel = CIFARSEConvNet().to(device)\nn_params = sum(p.numel() for p in model.parameters())\nprint(f'CIFARSEConvNet parameters: {n_params:,} (all randomly initialized)')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"aec7bf79","cell_type":"markdown","source":"## 4. CutMix (advanced data augmentation)\n\nStandard CutMix (Yun et al., 2019): for a fraction of training batches, a random rectangular\nregion is cut from one image in the batch and pasted onto another; the label becomes a\nlambda-weighted mix of the two images' labels, where lambda is the fraction of the image\n*not* replaced. The loss is a matching lambda-weighted mix of cross-entropy against both labels.\n","metadata":{}},{"id":"7e63f59a","cell_type":"code","source":"CUTMIX_PROB = 0.5     # fraction of batches that get CutMix applied\nCUTMIX_ALPHA = 1.0    # Beta(alpha, alpha) distribution for the mixing ratio\n\ndef rand_bbox(size, lam):\n    W, H = size[2], size[3]\n    cut_rat = np.sqrt(1. - lam)\n    cut_w, cut_h = int(W * cut_rat), int(H * cut_rat)\n    cx, cy = np.random.randint(W), np.random.randint(H)\n    x1 = np.clip(cx - cut_w // 2, 0, W)\n    y1 = np.clip(cy - cut_h // 2, 0, H)\n    x2 = np.clip(cx + cut_w // 2, 0, W)\n    y2 = np.clip(cy + cut_h // 2, 0, H)\n    return x1, y1, x2, y2\n\ndef cutmix_batch(xb, yb, alpha=CUTMIX_ALPHA):\n    lam = float(np.random.beta(alpha, alpha))\n    rand_index = torch.randperm(xb.size(0), device=xb.device)\n    y_a, y_b = yb, yb[rand_index]\n    x1, y1, x2, y2 = rand_bbox(xb.size(), lam)\n    xb[:, :, y1:y2, x1:x2] = xb[rand_index, :, y1:y2, x1:x2]\n    lam = 1 - ((x2 - x1) * (y2 - y1) / (xb.size(-1) * xb.size(-2)))\n    return xb, y_a, y_b, lam\n","metadata":{},"outputs":[],"execution_count":null},{"id":"d8cf85d7","cell_type":"markdown","source":"## 5. Training setup\n\n* **Optimizer:** AdamW (Adam with decoupled weight decay) — satisfies the \"advanced optimizer\"\n  requirement (plain SGD / SGD+Momentum would score 0 on this assignment).\n* **LR schedule:** linear warmup for the first few epochs, then cosine annealing down to a small\n  final LR for the rest of training — built from two standard, non-deprecated PyTorch schedulers\n  (`LinearLR` + `CosineAnnealingLR`) chained with `SequentialLR`. A constant LR would score 0 here.\n* **Loss:** cross-entropy with label smoothing (0.1), evaluated against both CutMix labels when\n  CutMix is applied to a batch.\n* **Mixed precision (AMP):** current `torch.amp` API (not the deprecated `torch.cuda.amp`).\n","metadata":{}},{"id":"eeb68a70","cell_type":"code","source":"EPOCHS = 60\nPEAK_LR = 2e-3\nWEIGHT_DECAY = 0.05\nWARMUP_EPOCHS = 5\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=PEAK_LR, weight_decay=WEIGHT_DECAY)\n\nsteps_per_epoch = len(train_dl)\nwarmup_iters = WARMUP_EPOCHS * steps_per_epoch\ntotal_iters = EPOCHS * steps_per_epoch\n\nwarmup_sched = torch.optim.lr_scheduler.LinearLR(\n    optimizer, start_factor=0.01, end_factor=1.0, total_iters=warmup_iters)\ncosine_sched = torch.optim.lr_scheduler.CosineAnnealingLR(\n    optimizer, T_max=total_iters - warmup_iters, eta_min=PEAK_LR * 0.01)\nscheduler = torch.optim.lr_scheduler.SequentialLR(\n    optimizer, schedulers=[warmup_sched, cosine_sched], milestones=[warmup_iters])\n\ncriterion = nn.CrossEntropyLoss(label_smoothing=0.1)\nscaler = torch.amp.GradScaler('cuda')\n\ndef evaluate(dl):\n    model.eval()\n    correct, total = 0, 0\n    with torch.no_grad():\n        for xb, yb in dl:\n            xb, yb = xb.to(device, non_blocking=True), yb.to(device, non_blocking=True)\n            with torch.amp.autocast('cuda'):\n                out = model(xb)\n            correct += (out.argmax(1) == yb).sum().item()\n            total += yb.size(0)\n    return correct / total\n","metadata":{},"outputs":[],"execution_count":null},{"id":"ed452499","cell_type":"markdown","source":"## 6. Training loop","metadata":{}},{"id":"62e65f8f","cell_type":"code","source":"history = []\nt0 = time.time()\n\nfor epoch in range(EPOCHS):\n    model.train()\n    running_loss, seen = 0.0, 0\n    for xb, yb in train_dl:\n        xb, yb = xb.to(device, non_blocking=True), yb.to(device, non_blocking=True)\n        optimizer.zero_grad(set_to_none=True)\n\n        use_cutmix = np.random.rand() < CUTMIX_PROB\n        if use_cutmix:\n            xb, y_a, y_b, lam = cutmix_batch(xb, yb)\n\n        with torch.amp.autocast('cuda'):\n            out = model(xb)\n            if use_cutmix:\n                loss = lam * criterion(out, y_a) + (1 - lam) * criterion(out, y_b)\n            else:\n                loss = criterion(out, yb)\n\n        scaler.scale(loss).backward()\n        scaler.unscale_(optimizer)\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)\n        scaler.step(optimizer)\n        scaler.update()\n        scheduler.step()\n\n        running_loss += loss.item() * yb.size(0)\n        seen += yb.size(0)\n\n    train_loss = running_loss / seen\n    test_acc = evaluate(test_dl)\n    history.append((epoch + 1, train_loss, test_acc))\n    elapsed = time.time() - t0\n    cur_lr = scheduler.get_last_lr()[0]\n    print(f'epoch {epoch+1:2d}/{EPOCHS} | lr {cur_lr:.5f} | train_loss {train_loss:.4f} | '\n          f'official CIFAR-10 test acc {test_acc*100:.2f}% | {elapsed/60:.1f} min elapsed')\n\nprint('Training done.')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"a18a6665","cell_type":"markdown","source":"## 7. Offline evaluation on the held-out official CIFAR-10 test set\n\nThis is our estimate of the Kaggle score, since the Kaggle-scored subset of the 300k test images\nis exactly this official 10,000-image test set.\n","metadata":{}},{"id":"775ef8f7","cell_type":"code","source":"final_test_acc = evaluate(test_dl)\nprint(f'Final accuracy on the OFFICIAL CIFAR-10 test set (10,000 images): {final_test_acc*100:.2f}%')\nprint('This is our estimate of the Kaggle score.')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"3bcedca8","cell_type":"code","source":"import matplotlib.pyplot as plt\n\nepochs_r  = [h[0] for h in history]\nlosses_r  = [h[1] for h in history]\naccs_r    = [h[2] for h in history]\n\nfig, ax1 = plt.subplots(figsize=(7,4))\nax1.plot(epochs_r, losses_r, color='tab:red', label='train loss')\nax1.set_xlabel('epoch'); ax1.set_ylabel('train loss (CutMix-mixed)', color='tab:red')\nax2 = ax1.twinx()\nax2.plot(epochs_r, [a*100 for a in accs_r], color='tab:blue', label='test acc (%)')\nax2.set_ylabel('official CIFAR-10 test accuracy (%)', color='tab:blue')\nplt.title('Training loss and held-out accuracy per epoch')\nplt.tight_layout(); plt.show()\n","metadata":{},"outputs":[],"execution_count":null},{"id":"95e9fae8","cell_type":"markdown","source":"## 8. Locate and extract the Kaggle `test.7z` (300,000 images)\n\nCurrent Kaggle notebooks mount an attached competition under\n`/kaggle/input/competitions/<slug>/`. Run the diagnostic cell first to confirm the exact path on\nyour own notebook, since this layout has changed before.\n","metadata":{}},{"id":"99717d02","cell_type":"code","source":"# Diagnostic: find exactly where the competition files live on THIS notebook.\nfor root, dirs, files in os.walk('/kaggle/input'):\n    for f in files:\n        print(os.path.join(root, f))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"f4816375","cell_type":"code","source":"!pip install -q py7zr\n\nimport shutil\nimport py7zr\n\nKAGGLE_INPUT = '/kaggle/input/competitions/cifar-10'  # adjust if the diagnostic cell above shows a different path\nWORK_DIR = '/kaggle/working'\n\nsrc_7z = os.path.join(KAGGLE_INPUT, 'test.7z')\ndst_7z = os.path.join(WORK_DIR, 'test.7z')\nif os.path.exists(src_7z) and not os.path.exists(dst_7z):\n    shutil.copy(src_7z, dst_7z)\n\nos.chdir(WORK_DIR)\n\nif not os.path.isdir('test') or len(glob.glob('test/*')) < 300000:\n    with py7zr.SevenZipFile(dst_7z, mode='r') as archive:\n        archive.extractall(path=WORK_DIR)\n\nprint('Extracted test images:', len(glob.glob('test/*')))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"221b38fc","cell_type":"markdown","source":"## 9. Predict on the Kaggle test set and build `submission.csv`\n","metadata":{}},{"id":"4d319918","cell_type":"code","source":"class KaggleTestDataset(Dataset):\n    def __init__(self, folder, transform):\n        self.paths = sorted(\n            glob.glob(os.path.join(folder, '*.png')),\n            key=lambda p: int(os.path.splitext(os.path.basename(p))[0])\n        )\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.paths)\n\n    def __getitem__(self, idx):\n        path = self.paths[idx]\n        img_id = int(os.path.splitext(os.path.basename(path))[0])\n        img = Image.open(path).convert('RGB')\n        return self.transform(img), img_id\n\nkaggle_test_ds = KaggleTestDataset(os.path.join(WORK_DIR, 'test'), test_tfms)\nkaggle_test_dl = DataLoader(kaggle_test_ds, batch_size=1024, shuffle=False, num_workers=2, pin_memory=True)\nprint('Kaggle test images:', len(kaggle_test_ds))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"ddeb8e01","cell_type":"code","source":"model.eval()\nall_ids, all_preds = [], []\n\nwith torch.no_grad():\n    for xb, ids in kaggle_test_dl:\n        xb = xb.to(device, non_blocking=True)\n        with torch.amp.autocast('cuda'):\n            out = model(xb)\n        preds = out.argmax(1).cpu().numpy()\n        all_ids.extend(ids.numpy().tolist())\n        all_preds.extend(preds.tolist())\n\nimport pandas as pd\nsub = pd.DataFrame({'id': all_ids, 'label': [classes[p] for p in all_preds]})\nsub = sub.sort_values('id').reset_index(drop=True)\nsub.to_csv('/kaggle/working/submission.csv', index=False)\nsub.head()\n","metadata":{},"outputs":[],"execution_count":null}]}