{"nbformat":4,"nbformat_minor":5,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","pygments_lexer":"ipython3"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":3649}]}},"cells":[{"cell_type":"markdown","id":"02aa59dd7b92","metadata":{},"source":"# CIFAR-10 — Deep Learning Assignment 5 (challenge)\n\nKaggle competition: [CIFAR-10 – Object Recognition in Images](https://www.kaggle.com/c/cifar-10).\n\n**Constraints for this assignment**\n- Network weights are **randomly initialized**. No ImageNet (or other) pretrained checkpoint is loaded.\n- Optimizer is **AdamW** (not SGD).\n- Learning-rate schedule is **cosine annealing with a linear warmup**.\n- Augmentation includes **CutMix** and **Cutout** (not only crop/flip).\n- The network is a **custom DualGateNet**, not a torchvision / paper off-the-shelf ResNet, WRN, EfficientNet, or ConvNeXt.\n\n**Data.** Training uses the official CIFAR-10 train split (50,000 images) via `torchvision.datasets.CIFAR10`. The official 10,000-image test split is the Kaggle-score estimate. The leaderboard file is built by predicting the 300,000 images in `test.7z`.\n"},{"cell_type":"markdown","id":"47bacc31d9cb","metadata":{},"source":"## 1. Imports, device, and reproducibility\n"},{"cell_type":"code","id":"19ec8178196a","metadata":{},"outputs":[],"execution_count":null,"source":"import glob\nimport json\nimport math\nimport os\nimport shutil\nimport subprocess\nimport threading\nimport time\nfrom concurrent.futures import ThreadPoolExecutor\nfrom pathlib import Path\n\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader\nfrom torchvision import datasets, transforms\nfrom PIL import Image\n\nprint(\"PyTorch\", torch.__version__)\nprint(\"CUDA available:\", torch.cuda.is_available())\nassert torch.cuda.is_available(), \"This notebook is meant to run with a Kaggle GPU.\"\nprint(\"GPU count:\", torch.cuda.device_count())\nfor i in range(torch.cuda.device_count()):\n    print(f\"  GPU {i}:\", torch.cuda.get_device_name(i))\nos.system(\"nvidia-smi\")\n\nSEED = 42\ntorch.manual_seed(SEED)\nnp.random.seed(SEED)\ntorch.cuda.manual_seed_all(SEED)\ntorch.backends.cudnn.benchmark = True\ntorch.backends.cudnn.deterministic = False\n\nDEVICE = torch.device(\"cuda:0\")\nN_GPUS = 1\nprint(\"Using single GPU only (DataParallel is avoided; it killed the previous run).\")\nCLASSES = [\n    \"airplane\",\n    \"automobile\",\n    \"bird\",\n    \"cat\",\n    \"deer\",\n    \"dog\",\n    \"frog\",\n    \"horse\",\n    \"ship\",\n    \"truck\",\n]\nprint(\"Classes:\", CLASSES)\nprint(\"PRETRAINED_WEIGHTS_USED = False\")\n"},{"cell_type":"markdown","id":"cd04958819a9","metadata":{},"source":"## 2. Start extracting Kaggle `test.7z` in the background\n\nLocate `test.7z` now, but **do not extract yet**. Unpacking 300,000 PNGs during training has crashed this kernel (process killed, no Python traceback). Extraction runs after the weights are saved.\n"},{"cell_type":"code","id":"7d670d4d7f09","metadata":{},"outputs":[],"execution_count":null,"source":"TEST_7Z_CANDIDATES = [\n    Path(\"/kaggle/input/cifar-10/test.7z\"),\n    Path(\"/kaggle/input/cifar-10/test/test.7z\"),\n    Path(\"/kaggle/input/competitions/cifar-10/test.7z\"),\n]\ntest_7z = None\nif Path(\"/kaggle/input\").exists():\n    found = list(Path(\"/kaggle/input\").rglob(\"test.7z\"))\n    if found:\n        test_7z = found[0]\nfor p in TEST_7Z_CANDIDATES:\n    if p.is_file():\n        test_7z = p\n        break\n\nEXTRACT_ROOT = Path(\"/tmp/cifar10_kaggle\")\nEXTRACT_ROOT.mkdir(parents=True, exist_ok=True)\nprint(\"test.7z:\", test_7z)\n\n\ndef extract_with_7z(archive: Path, dest: Path) -> None:\n    cmd = [\"7z\", \"x\", str(archive), f\"-o{dest}\", \"-y\"]\n    print(\"Running\", \" \".join(cmd), flush=True)\n    subprocess.run(cmd, check=True)\n\n\ndef extract_with_libarchive(archive: Path, dest: Path) -> None:\n    subprocess.run([\"apt-get\", \"update\", \"-qq\"], check=False)\n    subprocess.run([\"apt-get\", \"install\", \"-y\", \"-qq\", \"libarchive-dev\"], check=False)\n    subprocess.run([\"pip\", \"install\", \"-q\", \"libarchive\"], check=False)\n    import libarchive.public\n\n    dest.mkdir(parents=True, exist_ok=True)\n    prev = os.getcwd()\n    os.chdir(dest)\n    try:\n        cnt = 0\n        for _entry in libarchive.public.file_pour(str(archive)):\n            cnt += 1\n            if cnt % 1000 == 0:\n                print(cnt, flush=True)\n        print(\"libarchive extracted\", cnt, \"entries\", flush=True)\n    finally:\n        os.chdir(prev)\n\n\ndef extract_test_archive() -> None:\n    if test_7z is None or not Path(test_7z).is_file():\n        print(\"WARNING: test.7z not found yet; attach the cifar-10 competition data.\", flush=True)\n        return\n    try:\n        if shutil.which(\"7z\") is None:\n            subprocess.run([\"apt-get\", \"update\", \"-qq\"], check=False)\n            subprocess.run([\"apt-get\", \"install\", \"-y\", \"-qq\", \"p7zip-full\"], check=False)\n        extract_with_7z(Path(test_7z), EXTRACT_ROOT)\n    except Exception as exc:\n        print(\"7z failed:\", exc, \"— falling back to libarchive\", flush=True)\n        extract_with_libarchive(Path(test_7z), EXTRACT_ROOT)\n    print(\"Extraction finished.\", flush=True)\n\n\nprint(\"Extraction deferred until after training.\")\n"},{"cell_type":"markdown","id":"59aeeb77a260","metadata":{},"source":"## 3. Official CIFAR-10 data and augmentation\n\nBase transforms: 4-pixel pad + random crop, horizontal flip, CIFAR-10 normalize.\nAdvanced methods used in the training loop: **Cutout** (DeVries & Taylor, 2017) and **CutMix** (Yun et al., 2019).\n"},{"cell_type":"code","id":"6a7c81d4ce78","metadata":{},"outputs":[],"execution_count":null,"source":"class Cutout:\n    \"\"\"Zero a random square (DeVries & Taylor, 2017). Applied after Normalize.\"\"\"\n\n    def __init__(self, length: int = 16):\n        self.length = length\n\n    def __call__(self, img: torch.Tensor) -> torch.Tensor:\n        _, h, w = img.shape\n        y = torch.randint(0, h, (1,)).item()\n        x = torch.randint(0, w, (1,)).item()\n        y1 = max(0, y - self.length // 2)\n        y2 = min(h, y + self.length // 2)\n        x1 = max(0, x - self.length // 2)\n        x2 = min(w, x + self.length // 2)\n        img[:, y1:y2, x1:x2] = 0.0\n        return img\n\n\ndef rand_bbox(h: int, w: int, lam: float):\n    cut_rat = math.sqrt(1.0 - lam)\n    cut_w = int(w * cut_rat)\n    cut_h = int(h * cut_rat)\n    cx = int(np.random.randint(w))\n    cy = int(np.random.randint(h))\n    x1 = int(np.clip(cx - cut_w // 2, 0, w))\n    y1 = int(np.clip(cy - cut_h // 2, 0, h))\n    x2 = int(np.clip(cx + cut_w // 2, 0, w))\n    y2 = int(np.clip(cy + cut_h // 2, 0, h))\n    return x1, y1, x2, y2\n\n\ndef cutmix_batch(x: torch.Tensor, y: torch.Tensor, alpha: float = 1.0):\n    \"\"\"Replace a random patch with pixels from another sample (Yun et al., ICCV 2019).\"\"\"\n    lam = float(np.random.beta(alpha, alpha))\n    index = torch.randperm(x.size(0), device=x.device)\n    _, _, h, w = x.shape\n    x1, y1, x2, y2 = rand_bbox(h, w, lam)\n    mixed = x.clone()\n    mixed[:, :, y1:y2, x1:x2] = x[index, :, y1:y2, x1:x2]\n    lam_adj = 1.0 - ((x2 - x1) * (y2 - y1) / float(w * h))\n    return mixed, y, y[index], lam_adj\n\n\nCIFAR_MEAN = (0.4914, 0.4822, 0.4465)\nCIFAR_STD = (0.2023, 0.1994, 0.2010)\n\ntrain_tf = transforms.Compose(\n    [\n        transforms.RandomCrop(32, padding=4),\n        transforms.RandomHorizontalFlip(),\n        transforms.ToTensor(),\n        transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n        Cutout(16),\n    ]\n)\neval_tf = transforms.Compose(\n    [\n        transforms.ToTensor(),\n        transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n    ]\n)\n\nDATA_ROOT = \"/tmp/official_cifar10\"\ntrain_set = datasets.CIFAR10(DATA_ROOT, train=True, download=True, transform=train_tf)\ntest_set = datasets.CIFAR10(DATA_ROOT, train=False, download=True, transform=eval_tf)\n\nPER_GPU_BATCH = 128\nBATCH = PER_GPU_BATCH * N_GPUS\ntrain_loader = DataLoader(\n    train_set, batch_size=BATCH, shuffle=True, num_workers=0, pin_memory=True, drop_last=False\n)\ntest_loader = DataLoader(\n    test_set, batch_size=256, shuffle=False, num_workers=0, pin_memory=True\n)\nprint(\"Train\", len(train_set), \"Test\", len(test_set), \"batch\", BATCH, \"batches/epoch\", len(train_loader))\n"},{"cell_type":"markdown","id":"1eeece555a07","metadata":{},"source":"## 4. DualGateNet — a custom network (random initialization)\n\nThis is **not** ResNet-18/50, Wide-ResNet, ResNeXt, DenseNet, EfficientNet, or ConvNeXt.\n\nEach **DualGateBlock** has two parallel residual paths:\n- a **local** path of two $3\\times 3$ convolutions;\n- a **context** path of $1\\times 1$ → depthwise $5\\times 5$ → $1\\times 1$;\n- a **channel gate** (squeeze-excitation with two sigmoid heads) that mixes the paths;\n- an identity skip (a $1\\times 1$ projection when the shape changes);\n- stochastic depth.\n\nThe stage layout is a non-standard width ladder **64–128–256–384** with block counts **2–3–3–2**. The stem is a stride-1 $3\\times 3$ convolution (no ImageNet $7\\times 7$ / max-pool). Every Conv2d / Linear layer uses PyTorch’s default Kaiming initialization. Nothing is loaded from a checkpoint.\n"},{"cell_type":"code","id":"24af55a00f50","metadata":{},"outputs":[],"execution_count":null,"source":"def drop_path(x: torch.Tensor, drop_prob: float, training: bool) -> torch.Tensor:\n    if drop_prob == 0.0 or not training:\n        return x\n    keep = 1.0 - drop_prob\n    shape = (x.shape[0],) + (1,) * (x.ndim - 1)\n    mask = x.new_empty(shape).bernoulli_(keep)\n    if keep > 0.0:\n        mask.div_(keep)\n    return x * mask\n\n\nclass ChannelGate(nn.Module):\n    \"\"\"Squeeze-excitation that emits two channel-wise mixing gates.\"\"\"\n\n    def __init__(self, channels: int, reduction: int = 8):\n        super().__init__()\n        hidden = max(8, channels // reduction)\n        self.pool = nn.AdaptiveAvgPool2d(1)\n        self.fc = nn.Sequential(\n            nn.Conv2d(channels, hidden, 1),\n            nn.SiLU(),\n            nn.Conv2d(hidden, 2 * channels, 1),\n        )\n\n    def forward(self, x: torch.Tensor):\n        w = self.fc(self.pool(x))\n        a, b = w.chunk(2, dim=1)\n        return torch.sigmoid(a), torch.sigmoid(b)\n\n\nclass DualGateBlock(nn.Module):\n    \"\"\"Custom residual unit: local 3x3 pair + 5x5 depthwise context, mixed by ChannelGate.\"\"\"\n\n    def __init__(self, in_ch: int, out_ch: int, stride: int = 1, drop_path_rate: float = 0.0):\n        super().__init__()\n        self.drop_path_rate = drop_path_rate\n        self.local = nn.Sequential(\n            nn.Conv2d(in_ch, out_ch, 3, stride=stride, padding=1, bias=False),\n            nn.BatchNorm2d(out_ch),\n            nn.SiLU(),\n            nn.Conv2d(out_ch, out_ch, 3, padding=1, bias=False),\n            nn.BatchNorm2d(out_ch),\n        )\n        self.context = nn.Sequential(\n            nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False),\n            nn.BatchNorm2d(out_ch),\n            nn.SiLU(),\n            nn.Conv2d(out_ch, out_ch, 5, padding=2, groups=out_ch, bias=False),\n            nn.BatchNorm2d(out_ch),\n            nn.SiLU(),\n            nn.Conv2d(out_ch, out_ch, 1, bias=False),\n            nn.BatchNorm2d(out_ch),\n        )\n        self.gate = ChannelGate(out_ch)\n        if stride != 1 or in_ch != out_ch:\n            self.skip = nn.Sequential(\n                nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False),\n                nn.BatchNorm2d(out_ch),\n            )\n        else:\n            self.skip = nn.Identity()\n        self.act = nn.SiLU()\n        nn.init.zeros_(self.local[-1].weight)\n        nn.init.zeros_(self.context[-1].weight)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        local = self.local(x)\n        context = self.context(x)\n        ga, gb = self.gate(local + context)\n        mixed = ga * local + gb * context\n        mixed = drop_path(mixed, self.drop_path_rate, self.training)\n        return self.act(self.skip(x) + mixed)\n\n\nclass DualGateNet(nn.Module):\n    \"\"\"CIFAR DualGateNet — designed for this assignment, not an off-the-shelf model.\n\n    All layers use PyTorch default random init. No pretrained weights.\n    \"\"\"\n\n    def __init__(\n        self,\n        num_classes: int = 10,\n        drop_path_rate: float = 0.12,\n        dropout: float = 0.15,\n    ):\n        super().__init__()\n        widths = [64, 128, 256, 384]\n        n_blocks = [2, 3, 3, 2]\n        strides = [1, 2, 2, 2]\n        self.stem = nn.Sequential(\n            nn.Conv2d(3, widths[0], 3, stride=1, padding=1, bias=False),\n            nn.BatchNorm2d(widths[0]),\n            nn.SiLU(),\n        )\n        total = sum(n_blocks)\n        dpr = [drop_path_rate * i / max(1, total - 1) for i in range(total)]\n        stages = []\n        in_ch = widths[0]\n        idx = 0\n        for w, n, s in zip(widths, n_blocks, strides):\n            blocks = []\n            for i in range(n):\n                stride = s if i == 0 else 1\n                blocks.append(DualGateBlock(in_ch, w, stride=stride, drop_path_rate=dpr[idx]))\n                in_ch = w\n                idx += 1\n            stages.append(nn.Sequential(*blocks))\n        self.stage1, self.stage2, self.stage3, self.stage4 = stages\n        self.head = nn.Sequential(\n            nn.AdaptiveAvgPool2d(1),\n            nn.Flatten(),\n            nn.Dropout(dropout),\n            nn.Linear(widths[-1], num_classes),\n        )\n        self.widths = widths\n        self.n_blocks = n_blocks\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        x = self.stem(x)\n        x = self.stage1(x)\n        x = self.stage2(x)\n        x = self.stage3(x)\n        x = self.stage4(x)\n        return self.head(x)\n\n\ndef unwrap(model: nn.Module) -> nn.Module:\n    return model.module if isinstance(model, nn.DataParallel) else model\n\n\nnet = DualGateNet()\nn_params = sum(p.numel() for p in net.parameters() if p.requires_grad)\nstem_w = net.stem[0].weight.detach()\nINIT_CONV_MEAN = float(stem_w.mean().item())\nINIT_CONV_STD = float(stem_w.std().item())\nprint(f\"Parameters: {n_params:,}\")\nprint(\"Architecture: DualGateNet widths={} blocks={}\".format(net.widths, net.n_blocks))\nprint(\"First conv weight: mean={:.5f} std={:.5f} (random, not ImageNet)\".format(\n    INIT_CONV_MEAN, INIT_CONV_STD\n))\nprint(\"PRETRAINED_WEIGHTS_USED = False\")\nmodel = net.to(DEVICE)\nprint(\"Using single GPU:\", torch.cuda.get_device_name(0))\n"},{"cell_type":"markdown","id":"028c8cf0cf0a","metadata":{},"source":"## 5. AdamW, cosine annealing with warmup, CutMix training\n\n- **Optimizer:** AdamW, decoupled weight decay $5\\times 10^{-2}$ on weights (0 on biases / BatchNorm).\n- **LR:** linear warmup for 10 epochs from $1\\%$ of the base rate up to $10^{-3}$ (scaled with batch size), then cosine annealing to $10^{-6}$.\n- **Loss:** cross-entropy; CutMix mixes two labels in the same mini-batch (`$\\lambda \\ell(y_a) + (1-\\lambda)\\ell(y_b)$`).\n- **CutMix** is applied with probability 0.8 (`$\\alpha=1$`); otherwise the sample already has Cutout from the transform.\n- Mixed-precision (`torch.cuda.amp`) on the T4. An exponential moving average of the weights is kept for evaluation.\n"},{"cell_type":"code","id":"858f0e0ff821","metadata":{},"outputs":[],"execution_count":null,"source":"EPOCHS = 160\nWARMUP_EPOCHS = 10\nBASE_LR = 1.0e-3 * (BATCH / 128.0)\nETA_MIN = 1.0e-6\nWEIGHT_DECAY = 5e-2\nCUTMIX_PROB = 0.8\nCUTMIX_ALPHA = 1.0\nEMA_DECAY = 0.9998\n\ncriterion = nn.CrossEntropyLoss()\n\n\ndef adamw_param_groups(module: nn.Module, weight_decay: float):\n    decay, no_decay = [], []\n    for name, p in module.named_parameters():\n        if not p.requires_grad:\n            continue\n        if p.ndim == 1 or name.endswith(\"bias\"):\n            no_decay.append(p)\n        else:\n            decay.append(p)\n    return [\n        {\"params\": decay, \"weight_decay\": weight_decay},\n        {\"params\": no_decay, \"weight_decay\": 0.0},\n    ]\n\n\noptimizer = torch.optim.AdamW(adamw_param_groups(unwrap(model), WEIGHT_DECAY), lr=BASE_LR, betas=(0.9, 0.999))\n\n\ndef lr_lambda(epoch: int) -> float:\n    if epoch < WARMUP_EPOCHS:\n        return float(epoch + 1) / float(WARMUP_EPOCHS)\n    progress = (epoch - WARMUP_EPOCHS) / max(1, EPOCHS - WARMUP_EPOCHS)\n    cosine = 0.5 * (1.0 + math.cos(math.pi * progress))\n    min_factor = ETA_MIN / BASE_LR\n    return min_factor + (1.0 - min_factor) * cosine\n\n\nscheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)\nscaler = torch.amp.GradScaler(\"cuda\")\n\n\nclass ModelEMA:\n    def __init__(self, module: nn.Module, decay: float):\n        self.decay = decay\n        self.shadow = {k: v.detach().clone() for k, v in unwrap(module).state_dict().items()}\n\n    @torch.no_grad()\n    def update(self, module: nn.Module) -> None:\n        for k, v in unwrap(module).state_dict().items():\n            if v.dtype.is_floating_point:\n                self.shadow[k].mul_(self.decay).add_(v.detach(), alpha=1.0 - self.decay)\n            else:\n                self.shadow[k].copy_(v)\n\n    def copy_to(self, module: nn.Module) -> None:\n        unwrap(module).load_state_dict(self.shadow, strict=True)\n\n\nema = ModelEMA(model, EMA_DECAY)\n\n\n@torch.no_grad()\ndef accuracy_on(loader) -> float:\n    model.eval()\n    correct = 0\n    total = 0\n    for x, y in loader:\n        x = x.to(DEVICE, non_blocking=True)\n        y = y.to(DEVICE, non_blocking=True)\n        with torch.amp.autocast(\"cuda\"):\n            pred = model(x).argmax(1)\n        correct += (pred == y).sum().item()\n        total += y.size(0)\n    return correct / total\n\n\nhistory = {\"loss\": [], \"train_acc\": [], \"test_acc\": [], \"lr\": [], \"cutmix_frac\": []}\nt0 = time.perf_counter()\n\nfor epoch in range(1, EPOCHS + 1):\n    model.train()\n    running_loss = 0.0\n    running_correct = 0\n    running_n = 0\n    n_cutmix = 0\n    n_batches = 0\n    for x, y in train_loader:\n        x = x.to(DEVICE, non_blocking=True)\n        y = y.to(DEVICE, non_blocking=True)\n        use_cutmix = bool(np.random.rand() < CUTMIX_PROB)\n        if use_cutmix:\n            x, ya, yb, lam = cutmix_batch(x, y, CUTMIX_ALPHA)\n            n_cutmix += 1\n        else:\n            ya, yb, lam = y, y, 1.0\n        n_batches += 1\n        optimizer.zero_grad(set_to_none=True)\n        with torch.amp.autocast(\"cuda\"):\n            logits = model(x)\n            loss = lam * criterion(logits, ya) + (1.0 - lam) * criterion(logits, yb)\n        scaler.scale(loss).backward()\n        scaler.step(optimizer)\n        scaler.update()\n        ema.update(model)\n        running_loss += loss.item() * y.size(0)\n        running_correct += (logits.argmax(1) == y).sum().item()\n        running_n += y.size(0)\n    scheduler.step()\n    train_loss = running_loss / running_n\n    train_acc = running_correct / running_n\n    lr_now = optimizer.param_groups[0][\"lr\"]\n    history[\"loss\"].append(train_loss)\n    history[\"train_acc\"].append(train_acc)\n    history[\"lr\"].append(lr_now)\n    history[\"cutmix_frac\"].append(n_cutmix / max(1, n_batches))\n    log = (\n        f\"epoch {epoch:03d}/{EPOCHS}  loss {train_loss:.4f}  \"\n        f\"train {train_acc:.4f}  lr {lr_now:.6f}  cutmix {history['cutmix_frac'][-1]:.2f}\"\n    )\n    if epoch % 5 == 0 or epoch == 1 or epoch == EPOCHS:\n        test_acc = accuracy_on(test_loader)\n        history[\"test_acc\"].append((epoch, test_acc))\n        log += f\"  official_test {test_acc:.4f}\"\n    print(log, flush=True)\n    if epoch % 20 == 0:\n        ckpt = f\"/kaggle/working/dualgatenet_ep{epoch:03d}.pt\"\n        torch.save(unwrap(model).state_dict(), ckpt)\n        print(\"saved\", ckpt, flush=True)\n\ntrain_s = time.perf_counter() - t0\nraw_test_acc = accuracy_on(test_loader)\ntorch.save(unwrap(model).state_dict(), \"/kaggle/working/dualgatenet_raw.pt\")\nema.copy_to(model)\ntorch.save(unwrap(model).state_dict(), \"/kaggle/working/dualgatenet_ema.pt\")\nofficial_test_acc = accuracy_on(test_loader)\nprint(f\"Training wall time: {train_s:.1f}s ({train_s/60:.1f} min)\")\nprint(f\"Official test (raw weights): {raw_test_acc:.4f}\")\nprint(f\"Official test (EMA weights, used for submission): {official_test_acc:.4f}\")\nprint(\"PRETRAINED_WEIGHTS_USED = False\")\n"},{"cell_type":"markdown","id":"50f2dccea2f3","metadata":{},"source":"## 6. Curves, learning-rate schedule, and confusion matrix\n"},{"cell_type":"code","id":"d7799c3f93d2","metadata":{},"outputs":[],"execution_count":null,"source":"from sklearn.metrics import classification_report, confusion_matrix\n\nepochs_x = list(range(1, EPOCHS + 1))\nfig, ax = plt.subplots(1, 3, figsize=(12.5, 3.3))\nax[0].plot(epochs_x, history[\"loss\"], color=\"black\")\nax[0].set_title(\"Train loss\")\nax[0].set_xlabel(\"Epoch\")\nax[1].plot(epochs_x, history[\"train_acc\"], label=\"Train (hard labels)\", color=\"black\")\nif history[\"test_acc\"]:\n    te, ta = zip(*history[\"test_acc\"])\n    ax[1].plot(te, ta, label=\"Official test\", color=\"0.45\", marker=\"o\", markersize=3)\nax[1].set_title(\"Accuracy\")\nax[1].set_xlabel(\"Epoch\")\nax[1].legend()\nax[2].plot(epochs_x, history[\"lr\"], color=\"black\")\nax[2].axvline(WARMUP_EPOCHS, color=\"0.6\", linestyle=\"--\", linewidth=1, label=\"warmup end\")\nax[2].set_title(\"AdamW learning rate\")\nax[2].set_xlabel(\"Epoch\")\nax[2].legend(fontsize=8)\nfig.tight_layout()\nfig.savefig(\"/kaggle/working/training_curves.png\", dpi=140, bbox_inches=\"tight\")\nplt.show()\n\nmodel.eval()\ny_true, y_pred = [], []\nwith torch.no_grad():\n    for x, y in test_loader:\n        with torch.amp.autocast(\"cuda\"):\n            pred = model(x.to(DEVICE)).argmax(1).cpu().numpy()\n        y_pred.append(pred)\n        y_true.append(y.numpy())\ny_true = np.concatenate(y_true)\ny_pred = np.concatenate(y_pred)\ncm = confusion_matrix(y_true, y_pred)\nprint(\"Official test errors:\", int((y_true != y_pred).sum()), \"/\", len(y_true))\nprint(classification_report(y_true, y_pred, target_names=CLASSES, digits=4))\n\nplt.figure(figsize=(5.6, 5.0))\nplt.imshow(cm, cmap=\"Greys\")\nplt.title(f\"Official CIFAR-10 test, EMA (acc={official_test_acc:.2%})\")\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"True\")\nplt.xticks(range(10), CLASSES, rotation=45, ha=\"right\", fontsize=8)\nplt.yticks(range(10), CLASSES, fontsize=8)\nfor i in range(10):\n    for j in range(10):\n        plt.text(j, i, cm[i, j], ha=\"center\", va=\"center\", fontsize=6)\nplt.colorbar()\nplt.tight_layout()\nplt.savefig(\"/kaggle/working/confusion_matrix.png\", dpi=140, bbox_inches=\"tight\")\nplt.show()\n"},{"cell_type":"markdown","id":"1227147b6f4b","metadata":{},"source":"## 7. Extract `test.7z`, then predict 300,000 images (flip TTA)\n"},{"cell_type":"code","id":"c261f6df9f44","metadata":{},"outputs":[],"execution_count":null,"source":"print(\"Extracting competition test images after training...\")\nextract_test_archive()\n\n\ndef find_test_dir():\n    for png in EXTRACT_ROOT.rglob(\"1.png\"):\n        return png.parent\n    hits = list(Path(\"/tmp\").rglob(\"1.png\"))\n    if hits:\n        return hits[0].parent\n    cwd_hits = list(Path(\".\").rglob(\"1.png\"))\n    if cwd_hits:\n        return cwd_hits[0].parent\n    return None\n\n\nTEST_DIR = find_test_dir()\nif TEST_DIR is None:\n    print(\"Competition test images not found — skipping 300k Kaggle inference.\")\n    print(\"Official CIFAR-10 test accuracy (Kaggle estimate):\", official_test_acc)\nelse:\n    n_files = len(glob.glob(str(TEST_DIR / \"*\")))\n    print(\"Test dir:\", TEST_DIR, \"n_files:\", n_files)\n    assert n_files >= 300000, n_files\n\n    MEAN = torch.tensor(CIFAR_MEAN, device=DEVICE).view(1, 3, 1, 1)\n    STD = torch.tensor(CIFAR_STD, device=DEVICE).view(1, 3, 1, 1)\n\n    def load_one(image_id: int) -> np.ndarray:\n        path = str(TEST_DIR / f\"{image_id}.png\")\n        img = cv2.imread(path, cv2.IMREAD_COLOR)\n        if img is None:\n            return np.asarray(Image.open(path).convert(\"RGB\"))\n        return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\n    def load_ids(ids):\n        with ThreadPoolExecutor(max_workers=8) as pool:\n            parts = list(pool.map(load_one, ids))\n        return np.stack(parts, axis=0)\n\n    def to_tensor(nhwc: np.ndarray) -> torch.Tensor:\n        x = torch.from_numpy(nhwc).to(DEVICE)\n        x = x.permute(0, 3, 1, 2).float() / 255.0\n        x = (x - MEAN) / STD\n        return x\n\n    @torch.no_grad()\n    def predict_tta(x: torch.Tensor) -> np.ndarray:\n        model.eval()\n        with torch.amp.autocast(\"cuda\"):\n            logits = model(x)\n            logits_flip = model(torch.flip(x, dims=[3]))\n        return ((logits.float() + logits_flip.float()) / 2).argmax(1).cpu().numpy()\n\n    N_KAGGLE = 300000\n    PRED_BATCH = 512\n    preds = np.empty(N_KAGGLE, dtype=np.int64)\n    t_pred = time.perf_counter()\n    for start in range(1, N_KAGGLE + 1, PRED_BATCH):\n        end = min(start + PRED_BATCH, N_KAGGLE + 1)\n        ids = list(range(start, end))\n        arr = load_ids(ids)\n        preds[start - 1 : end - 1] = predict_tta(to_tensor(arr))\n        if start == 1 or end - 1 == N_KAGGLE or (end - 1) % 20000 == 0:\n            print(f\"predicted {end - 1}/{N_KAGGLE}\", flush=True)\n\n    print(f\"Kaggle test inference: {time.perf_counter() - t_pred:.1f}s\")\n    labels = [CLASSES[int(i)] for i in preds]\n    sub = pd.DataFrame({\"id\": np.arange(1, N_KAGGLE + 1), \"label\": labels})\n    sub_path = Path(\"/kaggle/working/submission.csv\")\n    sub.to_csv(sub_path, index=False)\n    print(sub.head())\n    print(\"Wrote\", sub_path, \"rows\", len(sub))\n    print(sub[\"label\"].value_counts())\n"},{"cell_type":"markdown","id":"8b3740f4ba69","metadata":{},"source":"## 8. Save metrics (no pretrained weights)\n"},{"cell_type":"code","id":"a44664b2fc39","metadata":{},"outputs":[],"execution_count":null,"source":"metrics = {\n    \"pretrained_weights_used\": False,\n    \"architecture\": \"DualGateNet (custom dual-path gated residual, 64-128-256-384)\",\n    \"parameters\": int(n_params),\n    \"epochs\": EPOCHS,\n    \"warmup_epochs\": WARMUP_EPOCHS,\n    \"batch_size\": BATCH,\n    \"n_gpus\": int(N_GPUS),\n    \"optimizer\": \"AdamW betas=(0.9, 0.999) weight_decay=5e-2 (0 on bias/BN)\",\n    \"base_lr\": float(BASE_LR),\n    \"lr_schedule\": \"linear warmup 10 epochs then cosine to 1e-6\",\n    \"augmentation\": \"pad4 crop, hflip, Cutout 16, CutMix p=0.8 alpha=1.0\",\n    \"ema_decay\": EMA_DECAY,\n    \"tta\": \"horizontal flip average\",\n    \"official_test_accuracy\": float(official_test_acc),\n    \"official_test_accuracy_raw\": float(raw_test_acc),\n    \"official_test_errors\": int((y_true != y_pred).sum()),\n    \"train_seconds\": float(train_s),\n    \"gpu\": torch.cuda.get_device_name(0),\n    \"seed\": SEED,\n    \"first_conv_weight_mean\": INIT_CONV_MEAN,\n    \"first_conv_weight_std\": INIT_CONV_STD,\n}\nPath(\"/kaggle/working/metrics.json\").write_text(json.dumps(metrics, indent=2), encoding=\"utf-8\")\nprint(json.dumps(metrics, indent=2))\nprint(\"Done. Estimated Kaggle score (EMA):\", round(official_test_acc, 4))\nprint(\"PRETRAINED_WEIGHTS_USED = False\")\n"}]}