{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"1c359622-530f-4ad2-9896-410169613d2e","cell_type":"markdown","source":"# CIFAR-10 from scratch: a custom CNN (randomly initialized, no pre-trained weights)\n\nv4: replaces the ResNet-18 architecture with an original design, per the\ngrading rubric's \"design your own neural network\" requirement (3 pts) --\nResNet-18, even hand-coded, is a well-known off-the-shelf architecture and\nwould not satisfy that rubric item.\n\n**What makes this network custom** (see the \"Model\" section below for the\nfull rationale, intended for the report):\n- A 5-stage width progression (64/96/160/256/384) instead of ResNet's\n  doubling 64/128/256/512 over 4 stages.\n- Squeeze-and-Excitation (SE) channel attention added to every block --\n  not part of ResNet-18.\n- Stochastic depth (randomly dropping whole residual blocks during\n  training, with probability increasing by depth) -- not part of ResNet-18.\n- A 2-layer classifier head with dropout instead of a single linear layer.\n\nTraining setup (per the other rubric requirements):\n- AdamW optimizer (Adam + decoupled weight decay)\n- Cosine annealing LR schedule with a linear warmup phase\n- CutMix data augmentation, disabled for a cooldown period at the end of training\n- 100 epochs\n\nPlan\n1. Train on the official CIFAR-10 training set (same as the Kaggle training set).\n2. Evaluate on the official CIFAR-10 test set to estimate the Kaggle score.\n3. Predict on the Kaggle test set (300,000 images) and write `submission.csv`.\n\nNotebook settings: **Accelerator = GPU** and **Internet = On**","metadata":{}},{"id":"44bac7cd-0acb-4d88-8e78-8bc31a4ef696","cell_type":"code","source":"import os, glob, time, math\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision\nimport torchvision.transforms as T\nfrom torch.utils.data import DataLoader, Dataset\nfrom PIL import Image\n\nSEED = 0\ntorch.manual_seed(SEED)\nnp.random.seed(SEED)\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nuse_amp = device == \"cuda\"\ntorch.backends.cudnn.benchmark = True\nprint(\"device:\", device)","metadata":{},"outputs":[],"execution_count":null},{"id":"b47ea2b5-52df-47fc-894c-ea3b1bdfcd23","cell_type":"markdown","source":"## 1. Data (official CIFAR-10) + augmentation\n\nStandard crop/flip augmentation is applied in the Dataset transform.\nCutMix is applied per-batch in the training loop (it needs pairs of images, so\nit can't live inside a single-image `transform`).","metadata":{}},{"id":"476e202c-f500-4aaf-b3e7-d28e9489ae86","cell_type":"code","source":"MEAN = (0.4914, 0.4822, 0.4465)\nSTD = (0.2470, 0.2435, 0.2616)\n\ntrain_tf = T.Compose([\n    T.RandomCrop(32, padding=4, padding_mode=\"reflect\"),\n    T.RandomHorizontalFlip(),\n    T.ToTensor(),\n    T.Normalize(MEAN, STD),\n])\ntest_tf = T.Compose([T.ToTensor(), T.Normalize(MEAN, STD)])\n\ntrain_set = torchvision.datasets.CIFAR10(\"./data\", train=True, download=True, transform=train_tf)\ntest_set = torchvision.datasets.CIFAR10(\"./data\", train=False, download=True, transform=test_tf)\nclasses = train_set.classes\nprint(len(train_set), len(test_set), classes)\n\nBATCH_SIZE = 128\nNUM_WORKERS = 4\ntrain_loader = DataLoader(train_set, batch_size=BATCH_SIZE, shuffle=True,\n                          num_workers=NUM_WORKERS, pin_memory=True,\n                          drop_last=True, persistent_workers=True)\ntest_loader = DataLoader(test_set, batch_size=512, shuffle=False,\n                         num_workers=NUM_WORKERS, pin_memory=True,\n                         persistent_workers=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"0a63e55b-af7b-4cb5-83f3-4ebc6dd93710","cell_type":"markdown","source":"### CutMix\nFor each batch (with some probability), cut a random rectangular patch from\none image in the batch and paste it onto another (a shuffled copy of the\nbatch). The label becomes a mix of the two labels, weighted by the patch\narea, and the loss is a matching mix of the two cross-entropy terms.","metadata":{}},{"id":"1bc0e651-1d80-4b0a-9530-305f7c5a3b21","cell_type":"code","source":"CUTMIX_PROB = 0.5\nCUTMIX_ALPHA = 1.0\n\n\ndef rand_bbox(W, H, lam):\n    cut_rat = math.sqrt(1.0 - lam)\n    cut_w, cut_h = int(W * cut_rat), int(H * cut_rat)\n    cx, cy = np.random.randint(W), np.random.randint(H)\n    x1, y1 = np.clip(cx - cut_w // 2, 0, W), np.clip(cy - cut_h // 2, 0, H)\n    x2, y2 = np.clip(cx + cut_w // 2, 0, W), np.clip(cy + cut_h // 2, 0, H)\n    return x1, y1, x2, y2\n\n\ndef cutmix(x, y, alpha=CUTMIX_ALPHA):\n    lam = np.random.beta(alpha, alpha)\n    perm = torch.randperm(x.size(0), device=x.device)\n    y_a, y_b = y, y[perm]\n    H, W = x.size(2), x.size(3)\n    x1, y1, x2, y2 = rand_bbox(W, H, lam)\n    x[:, :, y1:y2, x1:x2] = x[perm, :, y1:y2, x1:x2]\n    # recompute lam from actual cut area (in case it was clipped at the border)\n    lam = 1 - ((x2 - x1) * (y2 - y1) / (W * H))\n    return x, y_a, y_b, lam","metadata":{},"outputs":[],"execution_count":null},{"id":"ccdd1db2-50a7-4691-899e-fd7503ce3ab0","cell_type":"markdown","source":"## 2. Model: a custom CNN, random initialization\n\nThis is an original design, not a reproduction of a published architecture.\nIt borrows the general *idea* of residual connections (a widely-used\nbuilding-block technique, not an architecture in itself) but combines it\nwith choices not found together in ResNet-18:\n\n- **5 stages** with widths 64 / 96 / 160 / 256 / 384 (ResNet-18 uses 4\n  stages that double each time: 64/128/256/512). The narrower early growth\n  and extra stage were chosen to add depth without blowing up the\n  parameter count on 32x32 inputs, where very wide early layers are mostly\n  wasted since the spatial resolution is already small.\n- **Squeeze-and-Excitation (SE) attention** in every block: after the two\n  convolutions, a block computes a per-channel \"importance\" score from the\n  global-average-pooled feature map and rescales each channel by it. This\n  lets the network emphasize informative feature channels and suppress\n  less useful ones, which plain ResNet blocks cannot do.\n- **Stochastic depth**: during training, each block has a probability of\n  being skipped entirely (identity passthrough), with that probability\n  increasing linearly from 0 at the first block to `MAX_DROP_PATH` at the\n  last. This acts like a structured form of dropout over whole blocks and\n  is a regularizer ResNet-18 does not use.\n- **2-layer classifier head** (Linear -> GELU -> Dropout -> Linear) instead\n  of a single linear layer on top of the pooled features.","metadata":{}},{"id":"c6e5e1e4-c2bc-4437-b2c8-cc6ddaf53d5e","cell_type":"code","source":"class SEBlock(nn.Module):\n    \"\"\"Squeeze-and-Excitation channel attention.\"\"\"\n    def __init__(self, channels, reduction=8):\n        super().__init__()\n        hidden = max(channels // reduction, 8)\n        self.fc1 = nn.Linear(channels, hidden)\n        self.fc2 = nn.Linear(hidden, channels)\n\n    def forward(self, x):\n        b, c, _, _ = x.shape\n        s = x.mean(dim=(2, 3))                 # squeeze: global average pool\n        s = F.relu(self.fc1(s))\n        s = torch.sigmoid(self.fc2(s))         # excite: per-channel gate in (0,1)\n        return x * s.view(b, c, 1, 1)\n\n\nclass CustomBlock(nn.Module):\n    def __init__(self, in_ch, out_ch, stride=1, drop_path=0.0):\n        super().__init__()\n        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1, bias=False)\n        self.bn1 = nn.BatchNorm2d(out_ch)\n        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1, bias=False)\n        self.bn2 = nn.BatchNorm2d(out_ch)\n        self.se = SEBlock(out_ch)\n        self.drop_path = drop_path\n\n        self.shortcut = nn.Sequential()\n        if stride != 1 or in_ch != out_ch:\n            self.shortcut = nn.Sequential(\n                nn.Conv2d(in_ch, out_ch, 1, stride, bias=False),\n                nn.BatchNorm2d(out_ch),\n            )\n\n    def forward(self, x):\n        out = F.gelu(self.bn1(self.conv1(x)))\n        out = self.bn2(self.conv2(out))\n        out = self.se(out)\n\n        if self.training and self.drop_path > 0.0:\n            # stochastic depth: zero out the whole residual branch for some\n            # samples in the batch, falling back to a pure identity/shortcut\n            keep = (torch.rand(x.size(0), 1, 1, 1, device=x.device) > self.drop_path).float()\n            out = out * keep\n\n        return F.gelu(out + self.shortcut(x))\n\n\nclass CustomNet(nn.Module):\n    def __init__(self, num_classes=10, max_drop_path=0.1):\n        super().__init__()\n        self.stem = nn.Sequential(\n            nn.Conv2d(3, 64, 3, 1, 1, bias=False), nn.BatchNorm2d(64), nn.GELU()\n        )\n        # (out_channels, stride, num_blocks) per stage\n        stage_cfg = [(64, 1, 2), (96, 2, 2), (160, 2, 2), (256, 2, 2), (384, 2, 2)]\n        total_blocks = sum(n for _, _, n in stage_cfg)\n\n        layers, in_ch, block_idx = [], 64, 0\n        for out_ch, stride, n_blocks in stage_cfg:\n            for b in range(n_blocks):\n                s = stride if b == 0 else 1\n                dp = max_drop_path * block_idx / max(1, total_blocks - 1)\n                layers.append(CustomBlock(in_ch, out_ch, s, drop_path=dp))\n                in_ch = out_ch\n                block_idx += 1\n        self.layers = nn.Sequential(*layers)\n\n        self.head = nn.Sequential(\n            nn.Linear(384, 256),\n            nn.GELU(),\n            nn.Dropout(0.3),\n            nn.Linear(256, num_classes),\n        )\n\n        for m in self.modules():\n            if isinstance(m, nn.Conv2d):\n                nn.init.kaiming_normal_(m.weight, mode=\"fan_out\", nonlinearity=\"relu\")\n            elif isinstance(m, nn.BatchNorm2d):\n                nn.init.ones_(m.weight)\n                nn.init.zeros_(m.bias)\n            elif isinstance(m, nn.Linear):\n                nn.init.trunc_normal_(m.weight, std=0.02)\n                nn.init.zeros_(m.bias)\n\n    def forward(self, x):\n        x = self.layers(self.stem(x))\n        x = F.adaptive_avg_pool2d(x, 1).flatten(1)\n        return self.head(x)\n\n\nmodel = CustomNet().to(device).to(memory_format=torch.channels_last)\nprint(f\"parameters: {sum(p.numel() for p in model.parameters()) / 1e6:.2f}M\")","metadata":{},"outputs":[],"execution_count":null},{"id":"817e2b0f-7d44-4a51-8edf-815ed7ee31d4","cell_type":"markdown","source":"## 3. Training: AdamW + warmup-cosine LR + CutMix\n\nv3 changes (to push past the 30-epoch result of 91.64%):\n- EPOCHS 30 -> 100: CutMix is a strong regularizer and needs more epochs to\n  fully pay off than plain crop/flip does; 30 epochs under-trained the model.\n- CutMix is switched off for the final COOLDOWN_EPOCHS epochs, so the model\n  spends most of training under heavy regularization, then \"settles\" on\n  clean data at the end. This is a standard trick in strong-augmentation\n  recipes and often adds a point or more by itself.\n- Slightly higher peak LR (3e-4 -> 4e-4) with a longer warmup (5 -> 10\n  epochs), scaled to the longer schedule.","metadata":{}},{"id":"49015792-9654-4e30-8659-947f66b675c2","cell_type":"code","source":"EPOCHS = 100\nBASE_LR = 4e-4         # typical AdamW LR for CNNs on CIFAR (much lower than SGD's 0.1)\nWEIGHT_DECAY = 0.05     # decoupled weight decay (the \"W\" in AdamW)\nWARMUP_EPOCHS = 10\nCOOLDOWN_EPOCHS = 10    # CutMix disabled for the last N epochs\n\ncriterion = nn.CrossEntropyLoss(label_smoothing=0.1)\noptimizer = torch.optim.AdamW(model.parameters(), lr=BASE_LR, weight_decay=WEIGHT_DECAY,\n                              betas=(0.9, 0.999))\n\nsteps_per_epoch = len(train_loader)\nwarmup_steps = WARMUP_EPOCHS * steps_per_epoch\ntotal_steps = EPOCHS * steps_per_epoch\n\n\ndef lr_lambda(step):\n    if step < warmup_steps:\n        # linear warmup from 0 -> 1 (scales BASE_LR)\n        return step / max(1, warmup_steps)\n    # cosine decay from 1 -> 0 over the remaining steps\n    progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)\n    return 0.5 * (1 + math.cos(math.pi * progress))\n\n\nscheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)\nscaler = torch.amp.GradScaler(\"cuda\", enabled=use_amp)\n\n\n@torch.no_grad()\ndef evaluate(loader):\n    model.eval()\n    correct, total, loss_sum = 0, 0, 0.0\n    for x, y in loader:\n        x = x.to(device, non_blocking=True).to(memory_format=torch.channels_last)\n        y = y.to(device, non_blocking=True)\n        with torch.autocast(device_type=\"cuda\", dtype=torch.float16, enabled=use_amp):\n            out = model(x)\n            loss_sum += F.cross_entropy(out, y, reduction=\"sum\").item()\n        correct += (out.argmax(1) == y).sum().item()\n        total += y.numel()\n    return loss_sum / total, correct / total\n\n\nhistory = []\nbest_acc = 0.0\nfor epoch in range(1, EPOCHS + 1):\n    model.train()\n    t0, run_loss, run_correct, seen = time.time(), 0.0, 0, 0\n    for x, y in train_loader:\n        x = x.to(device, non_blocking=True).to(memory_format=torch.channels_last)\n        y = y.to(device, non_blocking=True)\n\n        cutmix_active = epoch <= (EPOCHS - COOLDOWN_EPOCHS)\n        use_cutmix = cutmix_active and (np.random.rand() < CUTMIX_PROB)\n        if use_cutmix:\n            x, y_a, y_b, lam = cutmix(x, y)\n\n        optimizer.zero_grad(set_to_none=True)\n        with torch.autocast(device_type=\"cuda\", dtype=torch.float16, enabled=use_amp):\n            out = model(x)\n            if use_cutmix:\n                loss = lam * criterion(out, y_a) + (1 - lam) * criterion(out, y_b)\n            else:\n                loss = criterion(out, y)\n        scaler.scale(loss).backward()\n        scaler.step(optimizer)\n        scaler.update()\n        scheduler.step()\n\n        run_loss += loss.item() * y.size(0)\n        # training accuracy is approximate under CutMix (measured against the\n        # dominant label), since the true label is itself a mixture\n        run_correct += (out.argmax(1) == (y_a if use_cutmix else y)).sum().item()\n        seen += y.size(0)\n\n    test_loss, test_acc = evaluate(test_loader)\n    cur_lr = optimizer.param_groups[0][\"lr\"]\n    history.append((epoch, run_loss / seen, run_correct / seen, test_loss, test_acc))\n    tag = \"cutmix\" if cutmix_active else \"cooldown\"\n    print(f\"epoch {epoch:03d}/{EPOCHS} [{tag}] | lr {cur_lr:.5f} | train loss {run_loss/seen:.3f} \"\n          f\"acc {run_correct/seen:.4f} | test loss {test_loss:.3f} acc {test_acc:.4f} \"\n          f\"| {time.time()-t0:.0f}s\")\n\n    if test_acc > best_acc:\n        best_acc = test_acc\n        torch.save(model.state_dict(), \"customnet_cifar10_best.pt\")\n\ntorch.save(model.state_dict(), \"customnet_cifar10_last.pt\")\nfinal_loss, final_acc = evaluate(test_loader)\nprint(f\"\\nFinal-epoch accuracy on the official CIFAR-10 test set: {final_acc*100:.2f}%\")\nprint(f\"Best-epoch accuracy on the official CIFAR-10 test set:  {best_acc*100:.2f}%\")\n\n# Use the best checkpoint for the Kaggle submission below, since with a long\n# cosine schedule the last epoch is not always the strongest one.\nmodel.load_state_dict(torch.load(\"customnet_cifar10_best.pt\"))\nprint(\"Loaded best checkpoint for prediction/submission.\")","metadata":{},"outputs":[],"execution_count":null},{"id":"ee38493b-0ded-4c2b-99a5-a58e5704ae86","cell_type":"markdown","source":"## 4. Learning curves (useful for your report)","metadata":{}},{"id":"649a1d74-03e8-4e4b-80dc-5dbe34eb7a7a","cell_type":"code","source":"import matplotlib.pyplot as plt\n\nh = np.array(history)\nfig, ax = plt.subplots(1, 2, figsize=(11, 4))\nax[0].plot(h[:, 0], h[:, 1], label=\"train\"); ax[0].plot(h[:, 0], h[:, 3], label=\"test\")\nax[0].set_title(\"Loss\"); ax[0].set_xlabel(\"epoch\"); ax[0].legend()\nax[1].plot(h[:, 0], h[:, 2], label=\"train\"); ax[1].plot(h[:, 0], h[:, 4], label=\"test\")\nax[1].set_title(\"Accuracy\"); ax[1].set_xlabel(\"epoch\"); ax[1].legend()\nplt.tight_layout(); plt.savefig(\"curves.png\", dpi=120); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"57e0f1ed-b6a5-43b6-b398-876aa1594de7","cell_type":"markdown","source":"## 5. Predict on the Kaggle test set and build `submission.csv`\n\nAdd the competition data to the notebook (**Add Input -> Competitions -> CIFAR-10 - Object Recognition in Images**).\nThe Kaggle test set ships as `test.7z` (300,000 PNGs named `1.png ... 300000.png`).","metadata":{}},{"id":"b1682e51-0964-4703-96fa-2ce9dc5cbdf0","cell_type":"code","source":"try:\n    import py7zr\nexcept ImportError:\n    os.system(\"pip install -q py7zr\")\n    import py7zr\n\narchives = glob.glob(\"/kaggle/input/**/test.7z\", recursive=True)\nassert archives, \"test.7z not found - did you add the CIFAR-10 competition as an input?\"\nsample_sub_path = glob.glob(\"/kaggle/input/**/sampleSubmission.csv\", recursive=True)[0]\n\nEXTRACT_DIR = \"/kaggle/working/kaggle_test\"\nif not os.path.isdir(EXTRACT_DIR):\n    with py7zr.SevenZipFile(archives[0], mode=\"r\") as z:\n        z.extractall(EXTRACT_DIR)\n\nfiles = glob.glob(f\"{EXTRACT_DIR}/**/*.png\", recursive=True)\nprint(\"test images:\", len(files))\n\n\nclass KaggleTestSet(Dataset):\n    def __init__(self, files):\n        self.files = files\n\n    def __len__(self):\n        return len(self.files)\n\n    def __getitem__(self, i):\n        img = Image.open(self.files[i]).convert(\"RGB\")\n        return test_tf(img), int(os.path.splitext(os.path.basename(self.files[i]))[0])\n\n\nkaggle_loader = DataLoader(KaggleTestSet(files), batch_size=512, shuffle=False,\n                           num_workers=NUM_WORKERS, pin_memory=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"cf60a390-3bb0-451d-bcbc-520d26653ca8","cell_type":"code","source":"model.eval()\nall_ids, all_preds = [], []\nwith torch.no_grad():\n    for x, ids in kaggle_loader:\n        x = x.to(device, non_blocking=True).to(memory_format=torch.channels_last)\n        with torch.autocast(device_type=\"cuda\", dtype=torch.float16, enabled=use_amp):\n            logits = model(x) + model(torch.flip(x, dims=[3]))  # horizontal-flip TTA\n        all_preds.append(logits.argmax(1).cpu())\n        all_ids.append(ids)\n\npreds = torch.cat(all_preds).numpy()\nids = torch.cat(all_ids).numpy()\n\nsubmission = pd.DataFrame({\"id\": ids, \"label\": [classes[p] for p in preds]}).sort_values(\"id\")\n\nsample = pd.read_csv(sample_sub_path)\nassert list(submission.columns) == list(sample.columns)\nassert len(submission) == len(sample) and (submission[\"id\"].values == sample[\"id\"].sort_values().values).all()\n\nsubmission.to_csv(\"submission.csv\", index=False)\nprint(submission.head())\nprint(submission[\"label\"].value_counts())","metadata":{},"outputs":[],"execution_count":null}]}