{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"964de90e","cell_type":"markdown","source":"**Course:** Applied Deep Learning, HBKU, Fall 2026  \n**Assignment:** CIFAR10 challenge on Kaggle  ","metadata":{}},{"id":"75e5f877","cell_type":"markdown","source":"# CIFAR10 challenge on Kaggle with a custom network, AdamW, cosine warmup and CutMix\n\nCompetition: https://www.kaggle.com/c/cifar-10\n\nThis notebook is the challenge follow up to the first CIFAR10 assignment. It trains a\nconvolutional network that was designed for this assignment, not taken off the shelf, and\npredicts the 300,000 images of the Kaggle test set. Every weight starts from a random value.\nNo pretrained weights are used.\n\nThe requirements of the challenge and how this notebook meets them:\n\n1. **Own network design.** CifarSENet, a pre activation residual network with squeeze and\n   excitation attention and stochastic depth. Section 3 describes it.\n2. **Advanced optimizer.** AdamW with decoupled weight decay. Weight decay is not applied to\n   batch normalization weights and biases.\n3. **Cosine annealing with warmup.** The learning rate rises linearly for 5 epochs and then\n   follows a cosine curve down to almost zero. It is updated after every batch.\n4. **Advanced data augmentation.** CutMix on half of the batches and Cutout on the other half,\n   on top of random crop and horizontal flip.\n\nTwo more tricks raise the score a little: an exponential moving average of the weights is kept\nduring training and used for the final predictions, and the class scores of each image and its\nmirror image are added at test time.\n\n**Kaggle settings.** Add the competition data as an input. Turn on the GPU accelerator.\nTurn on internet so the notebook can download the official CIFAR10 files and the 7z tool.","metadata":{}},{"id":"f5777847","cell_type":"code","source":"import os, sys, glob, time, math, shutil, subprocess, tarfile, pickle, urllib.request, copy\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib\n\nIN_NOTEBOOK = \"ipykernel\" in sys.modules\nif not IN_NOTEBOOK:\n    matplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\nfrom PIL import Image\n\nSEED = 0\ntorch.manual_seed(SEED)\nnp.random.seed(SEED)\n\nif torch.cuda.is_available():\n    device = torch.device(\"cuda\")\nelif torch.backends.mps.is_available():\n    device = torch.device(\"mps\")\nelse:\n    device = torch.device(\"cpu\")\n\nON_KAGGLE = os.path.exists(\"/kaggle/input\")\nOUT_DIR = \"/kaggle/working\" if ON_KAGGLE else \"outputs\"\nSCRATCH_DIR = \"/tmp/cifar10_scratch\" if ON_KAGGLE else \"data\"\nos.makedirs(OUT_DIR, exist_ok=True)\nos.makedirs(SCRATCH_DIR, exist_ok=True)\n\n# QUICK mode runs a tiny version of everything. It only checks that the code works.\nQUICK = os.environ.get(\"CIFAR_QUICK\") == \"1\"\nif QUICK:\n    EPOCHS = 2\nelif device.type == \"cuda\":\n    EPOCHS = 100\nelif device.type == \"mps\":\n    EPOCHS = 10\nelse:\n    EPOCHS = 2\nEPOCHS = int(os.environ.get(\"CIFAR_EPOCHS\", EPOCHS))\nBATCH_SIZE = 128\nWARMUP_EPOCHS = 1 if QUICK else 5\n\nif device.type == \"cuda\":\n    torch.backends.cudnn.benchmark = True\n    print(\"GPU\", torch.cuda.get_device_name(0))\nprint(\"Python\", sys.version.split()[0])\nprint(\"PyTorch\", torch.__version__)\nprint(\"Device\", device)\nprint(\"Epochs\", EPOCHS, \" Warmup epochs\", WARMUP_EPOCHS, \" Batch size\", BATCH_SIZE, \" Quick mode\", QUICK)\nif device.type == \"cpu\":\n    print(\"No GPU found. Training on a CPU is very slow. Turn on the GPU accelerator.\")","metadata":{},"outputs":[],"execution_count":null},{"id":"4c34591c","cell_type":"markdown","source":"## 1. Data\n\nThe official CIFAR10 archive has 50,000 training images and 10,000 test images. The Kaggle\ntraining set holds the same 50,000 images. The Kaggle test set holds 300,000 images. Only\nthe 10,000 official test images inside it count for the Kaggle score. So the accuracy on the\nofficial test set is a good estimate of the Kaggle score.\n\nThe notebook downloads the official archive with several connections at once, because the\nserver is slow per connection. If a Kaggle dataset with the file cifar-10-python.tar.gz is\nattached as an input, that copy is used instead. If neither works, it reads the Kaggle\ntraining archive and keeps 5,000 images aside as a validation set.","metadata":{}},{"id":"44d9d839","cell_type":"code","source":"CLASSES = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\nCIFAR_URL = \"https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\"\n\n\ndef find_file(name):\n    \"\"\"Find a competition file on Kaggle or in the local data folder.\"\"\"\n    candidates = [f\"/kaggle/input/cifar-10/{name}\"]\n    candidates += sorted(glob.glob(f\"/kaggle/input/**/{name}\", recursive=True))\n    candidates += [os.path.join(\"data\", name), os.path.join(SCRATCH_DIR, name)]\n    for path in candidates:\n        if os.path.exists(path):\n            return path\n    return None\n\n\nif ON_KAGGLE:\n    # Show what Kaggle attached, so that a missing input is easy to spot.\n    print(\"Files under /kaggle/input:\")\n    shown = 0\n    for root, dirs, files in os.walk(\"/kaggle/input\"):\n        for f in sorted(files):\n            if shown < 20:\n                print(\"  \", os.path.join(root, f))\n            shown += 1\n    if shown == 0:\n        print(\"   nothing. Click Add Input and attach the CIFAR-10 competition.\")\n    elif shown > 20:\n        print(f\"   and {shown - 20} more files\")\n\n\ndef extract_7z(archive, dest):\n    \"\"\"Extract a 7z archive into dest. The 7z program is fast. The pure Python tools are slow.\"\"\"\n    os.makedirs(dest, exist_ok=True)\n    if shutil.which(\"7z\") is None and ON_KAGGLE:\n        print(\"  installing the 7z program\")\n        subprocess.run(\"apt-get update -qq > /dev/null 2>&1; apt-get install -y -qq p7zip-full > /dev/null 2>&1\",\n                       shell=True)\n    if shutil.which(\"7z\"):\n        print(\"  extracting with 7z\")\n        subprocess.run([\"7z\", \"x\", \"-y\", f\"-o{dest}\", archive], check=True, stdout=subprocess.DEVNULL)\n        return dest\n    if shutil.which(\"bsdtar\"):\n        print(\"  extracting with bsdtar\")\n        subprocess.run([\"bsdtar\", \"-xf\", archive, \"-C\", dest], check=True)\n        return dest\n    # The libarchive method from the assignment text. The C library does the work, so it is fast.\n    try:\n        try:\n            import libarchive.public\n        except ImportError:\n            subprocess.run(\"apt-get install -y -qq libarchive-dev > /dev/null 2>&1\", shell=True)\n            subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"libarchive\"], check=True)\n            import libarchive.public\n        print(\"  extracting with libarchive\")\n        cwd = os.getcwd()\n        os.chdir(dest)\n        try:\n            for _ in libarchive.public.file_pour(archive):\n                pass\n        finally:\n            os.chdir(cwd)\n        return dest\n    except Exception as e:\n        print(\"  libarchive failed:\", e)\n    # Last option. py7zr is pure Python and needs about half an hour for 300,000 files.\n    try:\n        import py7zr\n    except ImportError:\n        subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"py7zr\"], check=True)\n        import py7zr\n    print(\"  extracting with py7zr, this is slow\")\n    with py7zr.SevenZipFile(archive, \"r\") as z:\n        z.extractall(dest)\n    return dest\n\n\ndef read_png_folder(paths):\n    \"\"\"Read PNG files into one uint8 array of shape N by 3 by 32 by 32.\"\"\"\n    out = np.empty((len(paths), 3, 32, 32), dtype=np.uint8)\n    for i, p in enumerate(paths):\n        out[i] = np.asarray(Image.open(p).convert(\"RGB\"), dtype=np.uint8).transpose(2, 0, 1)\n        if (i + 1) % 50000 == 0:\n            print(f\"  read {i + 1} images\")\n    return out\n\n\ndef download(url, path, workers=8, chunk=2 * 1024 * 1024):\n    \"\"\"Download a file with several connections at once. A partial file is resumed.\n\n    The CIFAR10 server gives each connection only about 75 KB per second. Eight parallel\n    range requests bring the 170 MB archive in a few minutes instead of half an hour.\n    \"\"\"\n    import threading\n    import queue\n    part = path + \".part\"\n    with urllib.request.urlopen(url, timeout=60) as r:\n        size = int(r.headers.get(\"Content-Length\") or 0)\n        final_url = r.geturl()\n    have = os.path.getsize(part) if os.path.exists(part) else 0\n    if have > size:\n        have = 0\n    ranges = [(a, min(a + chunk, size) - 1) for a in range(have, size, chunk)]\n    print(f\"Downloading {url}\")\n    print(f\"  {size / 1e6:.0f} MB in total, {have / 1e6:.0f} MB already present, {len(ranges)} chunks to fetch\")\n    todo = queue.Queue()\n    for item in enumerate(ranges):\n        todo.put(item)\n    results = {}\n    failed = []\n    lock = threading.Lock()\n    t0 = time.time()\n\n    def worker():\n        while True:\n            try:\n                i, (a, b) = todo.get_nowait()\n            except queue.Empty:\n                return\n            for attempt in range(5):\n                try:\n                    req = urllib.request.Request(final_url, headers={\"Range\": f\"bytes={a}-{b}\"})\n                    with urllib.request.urlopen(req, timeout=120) as r:\n                        data = r.read()\n                        status = r.status\n                    if status == 206 and len(data) == b - a + 1:\n                        with lock:\n                            results[i] = data\n                            if len(results) % 10 == 0:\n                                print(f\"  {len(results)} of {len(ranges)} chunks after {time.time() - t0:.0f} s\")\n                        break\n                except Exception:\n                    time.sleep(2)\n            else:\n                failed.append(i)\n\n    threads = [threading.Thread(target=worker) for _ in range(workers)]\n    for t in threads:\n        t.start()\n    for t in threads:\n        t.join()\n    if failed:\n        raise RuntimeError(f\"{len(failed)} chunks could not be downloaded\")\n    with open(part, \"ab\") as f:\n        for i in range(len(ranges)):\n            f.write(results[i])\n    os.replace(part, path)\n    print(f\"  done in {time.time() - t0:.0f} s\")\n\n\ndef load_official_cifar10():\n    \"\"\"Download the official CIFAR10 archive and return numpy arrays.\"\"\"\n    archive = find_file(\"cifar-10-python.tar.gz\") or os.path.join(SCRATCH_DIR, \"cifar-10-python.tar.gz\")\n    if not os.path.exists(archive):\n        download(CIFAR_URL, archive)\n\n    def read_batch(tf, name):\n        d = pickle.load(tf.extractfile(name), encoding=\"bytes\")\n        x = d[b\"data\"].reshape(-1, 3, 32, 32)\n        y = np.array(d[b\"labels\"], dtype=np.int64)\n        return x, y\n\n    with tarfile.open(archive) as tf:\n        parts = [read_batch(tf, f\"cifar-10-batches-py/data_batch_{i}\") for i in range(1, 6)]\n        x_train = np.concatenate([p[0] for p in parts])\n        y_train = np.concatenate([p[1] for p in parts])\n        x_test, y_test = read_batch(tf, \"cifar-10-batches-py/test_batch\")\n    return x_train, y_train, x_test, y_test\n\n\ndef load_kaggle_train():\n    \"\"\"Read the Kaggle training archive. Used only when the download is not possible.\"\"\"\n    archive = find_file(\"train.7z\")\n    labels_csv = find_file(\"trainLabels.csv\")\n    if archive is None or labels_csv is None:\n        raise FileNotFoundError(\"train.7z or trainLabels.csv was not found\")\n    folder = extract_7z(archive, os.path.join(SCRATCH_DIR, \"kaggle_train\"))\n    files = {int(os.path.splitext(os.path.basename(f))[0]): f\n             for f in glob.glob(os.path.join(folder, \"**\", \"*.png\"), recursive=True)}\n    labels = pd.read_csv(labels_csv)\n    paths = [files[i] for i in labels[\"id\"]]\n    x = read_png_folder(paths)\n    y = labels[\"label\"].map({c: i for i, c in enumerate(CLASSES)}).to_numpy(dtype=np.int64)\n    return x, y\n\n\ntry:\n    x_train, y_train, x_test, y_test = load_official_cifar10()\n    HAVE_OFFICIAL_TEST = True\n    print(\"Loaded the official CIFAR10 training and test sets.\")\nexcept Exception as e:\n    print(\"Could not load the official archive:\", e)\n    print(\"Reading the Kaggle training archive instead.\")\n    x_all, y_all = load_kaggle_train()\n    perm = np.random.RandomState(SEED).permutation(len(x_all))\n    x_test, y_test = x_all[perm[:5000]], y_all[perm[:5000]]\n    x_train, y_train = x_all[perm[5000:]], y_all[perm[5000:]]\n    HAVE_OFFICIAL_TEST = False\n    print(\"The test set below is a 5,000 image validation split of the training data.\")\n\nif QUICK:\n    x_train, y_train = x_train[:2000], y_train[:2000]\n    x_test, y_test = x_test[:1000], y_test[:1000]\n\nprint(\"Train images\", x_train.shape, \" Test images\", x_test.shape)\nprint(\"Images per class in the training set:\", np.bincount(y_train).tolist())","metadata":{},"outputs":[],"execution_count":null},{"id":"cb6f939f","cell_type":"code","source":"fig, axes = plt.subplots(10, 8, figsize=(8, 10.5))\nfor c in range(10):\n    idx = np.where(y_train == c)[0][:8]\n    for k, i in enumerate(idx):\n        ax = axes[c, k]\n        ax.imshow(x_train[i].transpose(1, 2, 0))\n        ax.set_xticks([])\n        ax.set_yticks([])\n        if k == 0:\n            ax.set_ylabel(CLASSES[c], rotation=0, ha=\"right\", va=\"center\", fontsize=10)\nfig.suptitle(\"Training images, one row per class\")\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"samples.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"3b1c6f25","cell_type":"markdown","source":"## 2. Batches and data augmentation\n\nThe whole dataset fits in GPU memory as 8 bit integers. Each batch is normalized and augmented\non the GPU, so no data loader is needed. Every training image gets a random crop after 4 pixel\npadding and a random horizontal flip. Then one of two stronger methods is applied to the batch:\n\n* **CutMix** on half of the batches. A random rectangle of every image is replaced by the same\n  rectangle from another image of the batch, and the two labels are mixed in proportion to the\n  area. The rectangle size follows a Beta(1, 1) distribution.\n* **Cutout** on the other half. One 8 by 8 square of every image is set to zero.","metadata":{}},{"id":"708626d6","cell_type":"code","source":"MEAN = torch.tensor([0.4914, 0.4822, 0.4465], device=device).view(1, 3, 1, 1)\nSTD = torch.tensor([0.2470, 0.2435, 0.2616], device=device).view(1, 3, 1, 1)\n\ntrain_x = torch.from_numpy(x_train).to(device)\ntrain_y = torch.from_numpy(y_train).to(device)\ntest_x = torch.from_numpy(x_test).to(device)\ntest_y = torch.from_numpy(y_test).to(device)\n\n\ndef normalize(x_uint8):\n    \"\"\"Scale 8 bit images to floats with zero mean and unit variance per channel.\"\"\"\n    return (x_uint8.float() / 255.0 - MEAN) / STD\n\n\ndef crop_and_flip(x):\n    \"\"\"Random crop with 4 pixel padding and a random horizontal flip, on the GPU.\"\"\"\n    n = x.size(0)\n    ar = torch.arange(32, device=x.device)\n    padded = F.pad(x, (4, 4, 4, 4))\n    top = torch.randint(0, 9, (n, 1), device=x.device)\n    left = torch.randint(0, 9, (n, 1), device=x.device)\n    rows = top + ar\n    cols = left + ar\n    x = padded[torch.arange(n, device=x.device)[:, None, None, None],\n               torch.arange(3, device=x.device)[None, :, None, None],\n               rows[:, None, :, None],\n               cols[:, None, None, :]]\n    flip = torch.rand(n, device=x.device) < 0.5\n    return torch.where(flip[:, None, None, None], x.flip(3), x)\n\n\ndef cutout(x, size=8):\n    \"\"\"Set one random square of each image to zero.\"\"\"\n    n = x.size(0)\n    ar = torch.arange(32, device=x.device)\n    cy = torch.randint(0, 32, (n, 1, 1), device=x.device)\n    cx = torch.randint(0, 32, (n, 1, 1), device=x.device)\n    dy = ar[None, :, None] - cy\n    dx = ar[None, None, :] - cx\n    half = size // 2\n    box = (dy >= -half) & (dy < half) & (dx >= -half) & (dx < half)\n    return x.masked_fill(box[:, None], 0.0)\n\n\ndef cutmix(x, y, alpha=1.0):\n    \"\"\"Paste one random box from a shuffled copy of the batch and mix the labels by area.\n\n    Returns the mixed images, the two label sets, and the weight of the first label set.\n    \"\"\"\n    n = x.size(0)\n    lam = float(np.random.beta(alpha, alpha))\n    perm = torch.randperm(n, device=x.device)\n    cut = math.sqrt(1.0 - lam)\n    rh, rw = int(32 * cut), int(32 * cut)\n    cy, cx = np.random.randint(32), np.random.randint(32)\n    y1, y2 = max(cy - rh // 2, 0), min(cy + rh // 2, 32)\n    x1, x2 = max(cx - rw // 2, 0), min(cx + rw // 2, 32)\n    x = x.clone()\n    x[:, :, y1:y2, x1:x2] = x[perm][:, :, y1:y2, x1:x2]\n    lam = 1.0 - (y2 - y1) * (x2 - x1) / (32 * 32)\n    return x, y, y[perm], lam\n\n\ndef augment(x, y):\n    \"\"\"Crop and flip every image, then CutMix or Cutout the batch. Returns mixed labels too.\"\"\"\n    x = crop_and_flip(x)\n    if np.random.rand() < 0.5:\n        return cutmix(x, y)\n    return cutout(x), y, y, 1.0\n\n\ndef to_image(x):\n    \"\"\"Undo the normalization so that a tensor can be drawn.\"\"\"\n    x = (x * STD + MEAN).clamp(0, 1)\n    return x.permute(0, 2, 3, 1).cpu().numpy()\n\n\nsample = normalize(train_x[:8])\nsample_y = train_y[:8]\nfig, axes = plt.subplots(3, 8, figsize=(10, 4.2))\nrows = [(\"original\", sample),\n        (\"cutout\", cutout(crop_and_flip(sample))),\n        (\"cutmix\", cutmix(crop_and_flip(sample), sample_y)[0])]\nfor r, (name, imgs) in enumerate(rows):\n    for k, img in enumerate(to_image(imgs)):\n        axes[r, k].imshow(img)\n        axes[r, k].axis(\"off\")\n    axes[r, 0].set_title(name, loc=\"left\")\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"augmentation.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"779fe7a9","cell_type":"markdown","source":"## 3. Model: CifarSENet\n\nThe network was designed for this assignment. It is a residual network, but its parts are\nchosen for 32 by 32 images and differ from the standard ResNet family in five ways.\n\n1. **Two convolution stem.** Two 3 by 3 convolutions (3 to 64 to 128 channels) read the image\n   at full resolution. There is no max pooling, so no detail is lost before the first block.\n2. **Pre activation blocks.** Each block applies batch normalization and the SiLU activation\n   before every convolution. The shortcut path is a pure identity, which keeps gradients clean.\n3. **Squeeze and excitation attention.** Every block ends with a small gate. A global average\n   pool summarizes each channel, two tiny layers turn the summary into a weight between 0 and 1\n   per channel, and the block output is scaled by these weights. The network learns which\n   channels matter for the current image.\n4. **Average pool shortcuts.** When a stage halves the resolution, the shortcut uses a 2 by 2\n   average pool followed by a 1 by 1 convolution, instead of a strided 1 by 1 convolution that\n   would skip three quarters of the pixels.\n5. **Stochastic depth.** During training each residual branch is dropped for a random subset of\n   the images, with a drop rate that grows linearly from 0 at the first block to 0.1 at the\n   last. This regularizes the network like an ensemble of shallower networks.\n\nThe three stages have 128, 256 and 512 channels at resolutions 32, 16 and 8, with two blocks\neach. A final batch normalization, SiLU, global average pool, dropout and one linear layer\ngive the ten class scores. The network has about 11 million parameters, all randomly\ninitialized by PyTorch.","metadata":{}},{"id":"1da16dda","cell_type":"code","source":"class SqueezeExcite(nn.Module):\n    \"\"\"Channel attention. A per channel weight in (0, 1) is computed from the pooled features.\"\"\"\n    def __init__(self, channels, reduction=16):\n        super().__init__()\n        self.fc1 = nn.Conv2d(channels, channels // reduction, 1)\n        self.fc2 = nn.Conv2d(channels // reduction, channels, 1)\n\n    def forward(self, x):\n        s = F.adaptive_avg_pool2d(x, 1)\n        s = torch.sigmoid(self.fc2(F.silu(self.fc1(s))))\n        return x * s\n\n\nclass PreActSEBlock(nn.Module):\n    \"\"\"Pre activation residual block with squeeze and excitation and stochastic depth.\"\"\"\n    def __init__(self, cin, cout, stride, drop_path):\n        super().__init__()\n        self.bn1 = nn.BatchNorm2d(cin)\n        self.conv1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False)\n        self.bn2 = nn.BatchNorm2d(cout)\n        self.conv2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False)\n        self.se = SqueezeExcite(cout)\n        self.drop_path = drop_path\n        self.shortcut = None\n        if stride != 1 or cin != cout:\n            layers = [nn.AvgPool2d(stride)] if stride != 1 else []\n            layers.append(nn.Conv2d(cin, cout, 1, bias=False))\n            self.shortcut = nn.Sequential(*layers)\n\n    def forward(self, x):\n        out = F.silu(self.bn1(x))\n        shortcut = x if self.shortcut is None else self.shortcut(out)\n        out = self.conv1(out)\n        out = self.conv2(F.silu(self.bn2(out)))\n        out = self.se(out)\n        if self.training and self.drop_path > 0:\n            keep = (torch.rand(out.size(0), 1, 1, 1, device=out.device) >= self.drop_path).to(out.dtype)\n            out = out * keep / (1.0 - self.drop_path)\n        return out + shortcut\n\n\nclass CifarSENet(nn.Module):\n    \"\"\"The network designed for this assignment. See the text above for the design choices.\"\"\"\n    def __init__(self, widths=(128, 256, 512), blocks=(2, 2, 2), num_classes=10,\n                 drop_path=0.1, dropout=0.1):\n        super().__init__()\n        self.stem = nn.Sequential(\n            nn.Conv2d(3, 64, 3, 1, 1, bias=False), nn.BatchNorm2d(64), nn.SiLU(inplace=True),\n            nn.Conv2d(64, widths[0], 3, 1, 1, bias=False))\n        stages = []\n        total = sum(blocks)\n        index = 0\n        cin = widths[0]\n        for stage, (width, count) in enumerate(zip(widths, blocks)):\n            for b in range(count):\n                stride = 2 if (stage > 0 and b == 0) else 1\n                rate = drop_path * index / max(total - 1, 1)\n                stages.append(PreActSEBlock(cin, width, stride, rate))\n                cin = width\n                index += 1\n        self.stages = nn.Sequential(*stages)\n        self.head_bn = nn.BatchNorm2d(cin)\n        self.dropout = nn.Dropout(dropout)\n        self.fc = nn.Linear(cin, num_classes)\n\n    def forward(self, x):\n        x = self.stages(self.stem(x))\n        x = F.silu(self.head_bn(x))\n        x = F.adaptive_avg_pool2d(x, 1).flatten(1)\n        return self.fc(self.dropout(x))\n\n\nmodel = CifarSENet().to(device)\nCHANNELS_LAST = device.type == \"cuda\"\nif CHANNELS_LAST:\n    model = model.to(memory_format=torch.channels_last)\nn_params = sum(p.numel() for p in model.parameters())\nprint(f\"CifarSENet with {n_params / 1e6:.2f} million parameters, all randomly initialized\")\nprint(model)","metadata":{},"outputs":[],"execution_count":null},{"id":"5f0ac0ad","cell_type":"markdown","source":"## 4. Training\n\n* **Optimizer.** AdamW with learning rate 0.002, betas (0.9, 0.999) and weight decay 0.05.\n  The decay is decoupled from the gradient, which is what makes AdamW different from Adam with\n  L2 regularization. Batch normalization weights and all biases are excluded from the decay.\n* **Schedule.** Linear warmup from zero to the peak learning rate over the first 5 epochs, then\n  cosine annealing down to one percent of the peak at the last step. The learning rate changes\n  after every batch, not once per epoch.\n* **Loss.** Cross entropy with label smoothing 0.1. For a CutMix batch the loss is the weighted\n  sum of the losses against the two label sets.\n* **Weight averaging.** An exponential moving average of the weights with decay 0.999 per step\n  is updated after every optimizer step. The averaged model is evaluated at the end and is used\n  for the Kaggle predictions. The raw model is also evaluated for comparison.\n* **Precision.** On a CUDA GPU the forward and backward passes run in float16 with loss scaling.\n\nThe training accuracy printed below is measured on the augmented batches, so it stays well\nbelow the test accuracy. No epoch is chosen with the test set. The last epoch is used.","metadata":{}},{"id":"458f8d3f","cell_type":"code","source":"LR = 2e-3\nWEIGHT_DECAY = 0.05\nLABEL_SMOOTHING = 0.1\nEMA_DECAY = 0.999\n\nsteps_per_epoch = math.ceil(len(train_x) / BATCH_SIZE)\ntotal_steps = EPOCHS * steps_per_epoch\nwarmup_steps = WARMUP_EPOCHS * steps_per_epoch\n\ndecay_params, no_decay_params = [], []\nfor name, p in model.named_parameters():\n    (no_decay_params if p.ndim <= 1 else decay_params).append(p)\noptimizer = torch.optim.AdamW([{\"params\": decay_params, \"weight_decay\": WEIGHT_DECAY},\n                               {\"params\": no_decay_params, \"weight_decay\": 0.0}],\n                              lr=LR, betas=(0.9, 0.999))\n\n\ndef lr_at(step):\n    \"\"\"Linear warmup, then cosine annealing to one percent of the peak learning rate.\"\"\"\n    if step < warmup_steps:\n        return LR * (step + 1) / warmup_steps\n    progress = (step - warmup_steps) / max(total_steps - warmup_steps, 1)\n    return LR * (0.01 + 0.99 * 0.5 * (1.0 + math.cos(math.pi * progress)))\n\n\ncriterion = nn.CrossEntropyLoss(label_smoothing=LABEL_SMOOTHING)\nUSE_AMP = device.type == \"cuda\"\nscaler = torch.amp.GradScaler(\"cuda\", enabled=USE_AMP)\nema_model = copy.deepcopy(model).eval()\nfor p in ema_model.parameters():\n    p.requires_grad_(False)\n\n\n@torch.no_grad()\ndef ema_update(decay):\n    \"\"\"Move the averaged weights a small step towards the current weights. Buffers are copied.\"\"\"\n    for ema_p, p in zip(ema_model.parameters(), model.parameters()):\n        ema_p.mul_(decay).add_(p.detach(), alpha=1.0 - decay)\n    for ema_b, b in zip(ema_model.buffers(), model.buffers()):\n        ema_b.copy_(b)\n\n\ndef prepare(x):\n    \"\"\"Move a normalized batch to the memory layout that the GPU prefers.\"\"\"\n    if CHANNELS_LAST:\n        return x.contiguous(memory_format=torch.channels_last)\n    return x\n\n\ndef predict_logits(xb, tta, net=None):\n    \"\"\"Return class scores of a network. With tta the scores of the flipped image are added.\"\"\"\n    net = model if net is None else net\n    with torch.autocast(\"cuda\", dtype=torch.float16, enabled=USE_AMP):\n        logits = net(xb).float()\n        if tta:\n            logits = logits + net(xb.flip(3)).float()\n    return logits\n\n\ndef evaluate(x_uint8, y, tta=False, net=None, batch=1000):\n    \"\"\"Accuracy and predictions on a set of images stored as 8 bit tensors.\"\"\"\n    net = model if net is None else net\n    was_training = net.training\n    net.eval()\n    preds = []\n    with torch.no_grad():\n        for i in range(0, len(x_uint8), batch):\n            xb = prepare(normalize(x_uint8[i:i + batch]))\n            preds.append(predict_logits(xb, tta, net).argmax(1))\n    if was_training:\n        net.train()\n    preds = torch.cat(preds)\n    return (preds == y).float().mean().item(), preds\n\n\nhistory = []\nlr_trace = []\nstep = 0\ntrain_start = time.time()\nfor epoch in range(1, EPOCHS + 1):\n    model.train()\n    perm = torch.randperm(len(train_x), device=device)\n    loss_sum = torch.zeros((), device=device)\n    correct = torch.zeros((), device=device, dtype=torch.long)\n    for i in range(0, len(perm), BATCH_SIZE):\n        idx = perm[i:i + BATCH_SIZE]\n        xb, ya, yb, lam = augment(normalize(train_x[idx]), train_y[idx])\n        xb = prepare(xb)\n        lr = lr_at(step)\n        for group in optimizer.param_groups:\n            group[\"lr\"] = lr\n        with torch.autocast(\"cuda\", dtype=torch.float16, enabled=USE_AMP):\n            logits = model(xb)\n            loss = lam * criterion(logits, ya) + (1.0 - lam) * criterion(logits, yb)\n        optimizer.zero_grad(set_to_none=True)\n        scaler.scale(loss).backward()\n        scaler.step(optimizer)\n        scaler.update()\n        ema_update(EMA_DECAY)\n        loss_sum += loss.detach() * len(idx)\n        correct += (logits.argmax(1) == ya).sum()\n        lr_trace.append(lr)\n        step += 1\n    train_loss = loss_sum.item() / len(perm)\n    train_acc = correct.item() / len(perm)\n    test_acc, _ = evaluate(test_x, test_y)\n    ema_acc, _ = evaluate(test_x, test_y, net=ema_model)\n    elapsed = time.time() - train_start\n    history.append(dict(epoch=epoch, lr=lr, train_loss=train_loss, train_acc=train_acc,\n                        test_acc=test_acc, ema_acc=ema_acc, time=elapsed))\n    print(f\"epoch {epoch:3d}  lr {lr:.5f}  train loss {train_loss:.3f}  train acc {train_acc:.4f}  \"\n          f\"test acc {test_acc:.4f}  ema acc {ema_acc:.4f}  time {elapsed:6.0f}s\")\n\nTRAIN_MINUTES = (time.time() - train_start) / 60\nprint(f\"Training took {TRAIN_MINUTES:.1f} minutes\")\ntorch.save(ema_model.state_dict(), os.path.join(OUT_DIR, \"cifar_senet_ema.pt\"))","metadata":{},"outputs":[],"execution_count":null},{"id":"9b6a866b","cell_type":"code","source":"hist = pd.DataFrame(history)\nfig, axes = plt.subplots(1, 3, figsize=(13, 3.6))\naxes[0].plot(np.arange(1, len(lr_trace) + 1) / steps_per_epoch, lr_trace)\naxes[0].set_xlabel(\"epoch\")\naxes[0].set_ylabel(\"learning rate\")\naxes[0].set_title(\"warmup and cosine annealing\")\naxes[0].grid(alpha=0.3)\naxes[1].plot(hist.epoch, hist.train_loss, marker=\"o\", markersize=3)\naxes[1].set_xlabel(\"epoch\")\naxes[1].set_ylabel(\"training loss\")\naxes[1].grid(alpha=0.3)\naxes[2].plot(hist.epoch, hist.train_acc, marker=\"o\", markersize=3, label=\"train accuracy, augmented batches\")\naxes[2].plot(hist.epoch, hist.test_acc, marker=\"o\", markersize=3, label=\"test accuracy, raw weights\")\naxes[2].plot(hist.epoch, hist.ema_acc, marker=\"o\", markersize=3, label=\"test accuracy, averaged weights\")\naxes[2].set_xlabel(\"epoch\")\naxes[2].set_ylabel(\"accuracy\")\naxes[2].grid(alpha=0.3)\naxes[2].legend(fontsize=8)\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"training_curves.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"69986f6f","cell_type":"markdown","source":"## 5. Accuracy on the official CIFAR10 test set\n\nThis is the estimate of the Kaggle score. The averaged weights are used with flip averaging.\nThe raw weights are shown for comparison.","metadata":{}},{"id":"b0b0418f","cell_type":"code","source":"acc_raw, _ = evaluate(test_x, test_y, tta=False)\nacc_raw_tta, _ = evaluate(test_x, test_y, tta=True)\nacc_ema, _ = evaluate(test_x, test_y, tta=False, net=ema_model)\nacc_ema_tta, preds = evaluate(test_x, test_y, tta=True, net=ema_model)\nset_name = \"official CIFAR10 test set\" if HAVE_OFFICIAL_TEST else \"validation split\"\nprint(f\"Accuracy on the {set_name}, raw weights, no flip averaging:      {acc_raw:.4f}\")\nprint(f\"Accuracy on the {set_name}, raw weights, flip averaging:         {acc_raw_tta:.4f}\")\nprint(f\"Accuracy on the {set_name}, averaged weights, no flip averaging: {acc_ema:.4f}\")\nprint(f\"Accuracy on the {set_name}, averaged weights, flip averaging:    {acc_ema_tta:.4f}\")\nprint(f\"Estimated Kaggle score: {acc_ema_tta:.4f}\")\nfinal_model = ema_model\n\ncm = torch.zeros(10, 10, dtype=torch.long)\nfor t, p in zip(test_y.cpu(), preds.cpu()):\n    cm[t, p] += 1\ncm = cm.numpy()\nper_class = pd.DataFrame({\"class\": CLASSES, \"accuracy\": cm.diagonal() / cm.sum(1)})\nprint(per_class.to_string(index=False, float_format=lambda v: f\"{v:.4f}\"))\n\nfig, ax = plt.subplots(figsize=(6.5, 5.5))\nim = ax.imshow(cm, cmap=\"Blues\")\nax.set_xticks(range(10))\nax.set_yticks(range(10))\nax.set_xticklabels(CLASSES, rotation=45, ha=\"right\")\nax.set_yticklabels(CLASSES)\nax.set_xlabel(\"predicted\")\nax.set_ylabel(\"true\")\nfor r in range(10):\n    for c in range(10):\n        ax.text(c, r, cm[r, c], ha=\"center\", va=\"center\", fontsize=7,\n                color=\"white\" if cm[r, c] > cm.max() / 2 else \"black\")\nax.set_title(f\"Confusion matrix, accuracy {acc_ema_tta:.4f}\")\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"confusion_matrix.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"2964e2c5","cell_type":"markdown","source":"## 6. Predict the Kaggle test set and write the submission file\n\nThe archive test.7z holds 300,000 PNG files named 1.png to 300000.png. They are extracted\nto a scratch folder, read in batches, and predicted with the averaged weights and flip\naveraging. The submission file has two columns, id and label, with the class name as the label.","metadata":{}},{"id":"2cf88a05","cell_type":"code","source":"test_archive = os.environ.get(\"CIFAR_TEST_7Z\") or find_file(\"test.7z\")\nif test_archive is None:\n    print(\"test.7z was not found. On Kaggle add the competition data as an input. \"\n          \"Locally put test.7z in the data folder.\")\n    submission = None\nelse:\n    extract_dir = os.path.join(SCRATCH_DIR, \"kaggle_test\")\n    files = glob.glob(os.path.join(extract_dir, \"**\", \"*.png\"), recursive=True)\n    if len(files) < 300000:\n        t0 = time.time()\n        print(\"Extracting\", test_archive)\n        extract_7z(test_archive, extract_dir)\n        files = glob.glob(os.path.join(extract_dir, \"**\", \"*.png\"), recursive=True)\n        print(f\"Extracted {len(files)} images in {time.time() - t0:.0f} seconds\")\n    by_id = {int(os.path.splitext(os.path.basename(f))[0]): f for f in files}\n    ids = sorted(by_id)\n    print(\"Number of test images:\", len(ids))\n    if len(ids) != 300000:\n        print(\"Warning: the Kaggle test set should have 300000 images.\")\n\n    class PngFolder(torch.utils.data.Dataset):\n        def __init__(self, paths):\n            self.paths = paths\n\n        def __len__(self):\n            return len(self.paths)\n\n        def __getitem__(self, i):\n            img = np.asarray(Image.open(self.paths[i]).convert(\"RGB\"), dtype=np.uint8)\n            return torch.from_numpy(img.copy()).permute(2, 0, 1)\n\n    workers = min(4, os.cpu_count() or 1) if device.type == \"cuda\" else 0\n    loader = torch.utils.data.DataLoader(PngFolder([by_id[i] for i in ids]), batch_size=1000,\n                                         shuffle=False, num_workers=workers)\n    final_model.eval()\n    preds = []\n    t0 = time.time()\n    with torch.no_grad():\n        for b, xb in enumerate(loader):\n            xb = prepare(normalize(xb.to(device, non_blocking=True)))\n            preds.append(predict_logits(xb, tta=True, net=final_model).argmax(1).cpu())\n            if (b + 1) % 50 == 0:\n                print(f\"  predicted {(b + 1) * 1000} images in {time.time() - t0:.0f} seconds\")\n    preds = torch.cat(preds).numpy()\n    submission = pd.DataFrame({\"id\": ids, \"label\": [CLASSES[p] for p in preds]})\n    sub_path = os.path.join(OUT_DIR, \"submission.csv\")\n    submission.to_csv(sub_path, index=False)\n    print(f\"Wrote {sub_path} with {len(submission)} rows in {time.time() - t0:.0f} seconds\")\n    print(submission.head())\n    print(\"Predicted labels per class:\")\n    print(submission[\"label\"].value_counts().to_string())","metadata":{},"outputs":[],"execution_count":null},{"id":"123523e4","cell_type":"markdown","source":"## 7. Summary\n\nThe table below repeats the main numbers. Submit the file submission.csv on the competition\npage as a late submission. The Kaggle score should be close to the estimate above.","metadata":{}},{"id":"57757f60","cell_type":"code","source":"summary = pd.DataFrame({\n    \"item\": [\"model\", \"parameters\", \"optimizer\", \"schedule\", \"augmentation\", \"epochs\", \"batch size\",\n             \"training minutes\", f\"accuracy on the {set_name}, raw weights\",\n             f\"accuracy on the {set_name}, averaged weights\", \"estimated Kaggle score\", \"submission rows\"],\n    \"value\": [\"CifarSENet, own design, random initialization\", f\"{n_params / 1e6:.2f} million\",\n              f\"AdamW, lr {LR}, weight decay {WEIGHT_DECAY}\",\n              f\"linear warmup {WARMUP_EPOCHS} epochs, then cosine annealing\",\n              \"random crop, flip, CutMix or Cutout\", EPOCHS, BATCH_SIZE, f\"{TRAIN_MINUTES:.1f}\",\n              f\"{acc_raw_tta:.4f}\", f\"{acc_ema_tta:.4f}\", f\"{acc_ema_tta:.4f}\",\n              0 if submission is None else len(submission)],\n})\nprint(summary.to_string(index=False))","metadata":{},"outputs":[],"execution_count":null}]}