{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"bd2e1c0f","cell_type":"markdown","source":"**Course:** Applied Deep Learning, HBKU, Fall 2026  \n**Assignment:** CIFAR10 competition on Kaggle  ","metadata":{}},{"id":"f8fdeddf","cell_type":"markdown","source":"# CIFAR10 on Kaggle with a ResNet18 trained from scratch\n\nCompetition: https://www.kaggle.com/c/cifar-10\n\nThis notebook trains a convolutional network on the CIFAR10 training set. It then predicts\nthe labels of the 300,000 images in the Kaggle test set. The network is a ResNet18 adapted to\n32 by 32 images. Every weight starts from a random value. No pretrained weights are used.\n\nThe steps are:\n\n1. Load the official CIFAR10 data. The Kaggle training set is the same data.\n2. Train the ResNet18 with data augmentation.\n3. Measure the accuracy on the official CIFAR10 test set. This estimates the Kaggle score.\n4. Extract the Kaggle test archive, predict all 300,000 images, and write submission.csv.\n\n**Kaggle settings.** Add the competition data as an input. Turn on the GPU accelerator.\nTurn on internet so the notebook can download the official CIFAR10 files.","metadata":{}},{"id":"df77a0b0","cell_type":"code","source":"import os, sys, glob, time, math, shutil, subprocess, tarfile, pickle, urllib.request\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib\n\nIN_NOTEBOOK = \"ipykernel\" in sys.modules\nif not IN_NOTEBOOK:\n    matplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\nfrom PIL import Image\n\nSEED = 0\ntorch.manual_seed(SEED)\nnp.random.seed(SEED)\n\nif torch.cuda.is_available():\n    device = torch.device(\"cuda\")\nelif torch.backends.mps.is_available():\n    device = torch.device(\"mps\")\nelse:\n    device = torch.device(\"cpu\")\n\nON_KAGGLE = os.path.exists(\"/kaggle/input\")\nOUT_DIR = \"/kaggle/working\" if ON_KAGGLE else \"outputs\"\nSCRATCH_DIR = \"/tmp/cifar10_scratch\" if ON_KAGGLE else \"data\"\nos.makedirs(OUT_DIR, exist_ok=True)\nos.makedirs(SCRATCH_DIR, exist_ok=True)\n\n# QUICK mode runs a tiny version of everything. It only checks that the code works.\nQUICK = os.environ.get(\"CIFAR_QUICK\") == \"1\"\nif QUICK:\n    EPOCHS = 2\nelif device.type == \"cuda\":\n    EPOCHS = 40\nelif device.type == \"mps\":\n    EPOCHS = 10\nelse:\n    EPOCHS = 2\nEPOCHS = int(os.environ.get(\"CIFAR_EPOCHS\", EPOCHS))\nBATCH_SIZE = 128\n\nif device.type == \"cuda\":\n    torch.backends.cudnn.benchmark = True\n    print(\"GPU\", torch.cuda.get_device_name(0))\nprint(\"Python\", sys.version.split()[0])\nprint(\"PyTorch\", torch.__version__)\nprint(\"Device\", device)\nprint(\"Epochs\", EPOCHS, \" Batch size\", BATCH_SIZE, \" Quick mode\", QUICK)\nif device.type == \"cpu\":\n    print(\"No GPU found. Training on a CPU is very slow. Turn on the GPU accelerator.\")","metadata":{},"outputs":[],"execution_count":null},{"id":"cba10f7c","cell_type":"markdown","source":"## 1. Data\n\nThe official CIFAR10 archive has 50,000 training images and 10,000 test images. The Kaggle\ntraining set holds the same 50,000 images. The Kaggle test set holds 300,000 images. Only\nthe 10,000 official test images inside it count for the Kaggle score. So the accuracy on the\nofficial test set is a good estimate of the Kaggle score.\n\nThe notebook downloads the official archive. If a Kaggle dataset with the file\ncifar-10-python.tar.gz is attached as an input, that copy is used and nothing is downloaded.\nIf neither works, it reads the Kaggle training archive instead and keeps 5,000 images aside\nas a validation set.","metadata":{}},{"id":"61889432","cell_type":"code","source":"CLASSES = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\nCIFAR_URL = \"https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\"\n\n\ndef find_file(name):\n    \"\"\"Find a competition file on Kaggle or in the local data folder.\"\"\"\n    candidates = [f\"/kaggle/input/cifar-10/{name}\"]\n    candidates += sorted(glob.glob(f\"/kaggle/input/**/{name}\", recursive=True))\n    candidates += [os.path.join(\"data\", name), os.path.join(SCRATCH_DIR, name)]\n    for path in candidates:\n        if os.path.exists(path):\n            return path\n    return None\n\n\nif ON_KAGGLE:\n    # Show what Kaggle attached, so that a missing input is easy to spot.\n    print(\"Files under /kaggle/input:\")\n    shown = 0\n    for root, dirs, files in os.walk(\"/kaggle/input\"):\n        for f in sorted(files):\n            if shown < 20:\n                print(\"  \", os.path.join(root, f))\n            shown += 1\n    if shown == 0:\n        print(\"   nothing. Click Add Input and attach the CIFAR-10 competition.\")\n    elif shown > 20:\n        print(f\"   and {shown - 20} more files\")\n\n\ndef extract_7z(archive, dest):\n    \"\"\"Extract a 7z archive into dest. Try each tool that may exist on this machine.\"\"\"\n    os.makedirs(dest, exist_ok=True)\n    if shutil.which(\"7z\"):\n        subprocess.run([\"7z\", \"x\", \"-y\", f\"-o{dest}\", archive], check=True, stdout=subprocess.DEVNULL)\n        return dest\n    if shutil.which(\"bsdtar\"):\n        subprocess.run([\"bsdtar\", \"-xf\", archive, \"-C\", dest], check=True)\n        return dest\n    try:\n        try:\n            import py7zr\n        except ImportError:\n            subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"py7zr\"], check=True)\n            import py7zr\n        with py7zr.SevenZipFile(archive, \"r\") as z:\n            z.extractall(dest)\n        return dest\n    except Exception as e:\n        print(\"py7zr failed:\", e)\n    # Last option. This is the method from the assignment text.\n    try:\n        import libarchive.public\n    except ImportError:\n        subprocess.run(\"apt-get install -y -q libarchive-dev > /dev/null\", shell=True)\n        subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"libarchive\"], check=True)\n        import libarchive.public\n    cwd = os.getcwd()\n    os.chdir(dest)\n    try:\n        for _ in libarchive.public.file_pour(archive):\n            pass\n    finally:\n        os.chdir(cwd)\n    return dest\n\n\ndef read_png_folder(paths):\n    \"\"\"Read PNG files into one uint8 array of shape N by 3 by 32 by 32.\"\"\"\n    out = np.empty((len(paths), 3, 32, 32), dtype=np.uint8)\n    for i, p in enumerate(paths):\n        out[i] = np.asarray(Image.open(p).convert(\"RGB\"), dtype=np.uint8).transpose(2, 0, 1)\n        if (i + 1) % 50000 == 0:\n            print(f\"  read {i + 1} images\")\n    return out\n\n\ndef download(url, path, workers=8, chunk=2 * 1024 * 1024):\n    \"\"\"Download a file with several connections at once. A partial file is resumed.\n\n    The CIFAR10 server gives each connection only about 75 KB per second. Eight parallel\n    range requests bring the 170 MB archive in a few minutes instead of half an hour.\n    \"\"\"\n    import threading\n    import queue\n    part = path + \".part\"\n    with urllib.request.urlopen(url, timeout=60) as r:\n        size = int(r.headers.get(\"Content-Length\") or 0)\n        final_url = r.geturl()\n    have = os.path.getsize(part) if os.path.exists(part) else 0\n    if have > size:\n        have = 0\n    ranges = [(a, min(a + chunk, size) - 1) for a in range(have, size, chunk)]\n    print(f\"Downloading {url}\")\n    print(f\"  {size / 1e6:.0f} MB in total, {have / 1e6:.0f} MB already present, {len(ranges)} chunks to fetch\")\n    todo = queue.Queue()\n    for item in enumerate(ranges):\n        todo.put(item)\n    results = {}\n    failed = []\n    lock = threading.Lock()\n    t0 = time.time()\n\n    def worker():\n        while True:\n            try:\n                i, (a, b) = todo.get_nowait()\n            except queue.Empty:\n                return\n            for attempt in range(5):\n                try:\n                    req = urllib.request.Request(final_url, headers={\"Range\": f\"bytes={a}-{b}\"})\n                    with urllib.request.urlopen(req, timeout=120) as r:\n                        data = r.read()\n                        status = r.status\n                    if status == 206 and len(data) == b - a + 1:\n                        with lock:\n                            results[i] = data\n                            if len(results) % 10 == 0:\n                                print(f\"  {len(results)} of {len(ranges)} chunks after {time.time() - t0:.0f} s\")\n                        break\n                except Exception:\n                    time.sleep(2)\n            else:\n                failed.append(i)\n\n    threads = [threading.Thread(target=worker) for _ in range(workers)]\n    for t in threads:\n        t.start()\n    for t in threads:\n        t.join()\n    if failed:\n        raise RuntimeError(f\"{len(failed)} chunks could not be downloaded\")\n    with open(part, \"ab\") as f:\n        for i in range(len(ranges)):\n            f.write(results[i])\n    os.replace(part, path)\n    print(f\"  done in {time.time() - t0:.0f} s\")\n\n\ndef load_official_cifar10():\n    \"\"\"Download the official CIFAR10 archive and return numpy arrays.\"\"\"\n    archive = find_file(\"cifar-10-python.tar.gz\") or os.path.join(SCRATCH_DIR, \"cifar-10-python.tar.gz\")\n    if not os.path.exists(archive):\n        download(CIFAR_URL, archive)\n\n    def read_batch(tf, name):\n        d = pickle.load(tf.extractfile(name), encoding=\"bytes\")\n        x = d[b\"data\"].reshape(-1, 3, 32, 32)\n        y = np.array(d[b\"labels\"], dtype=np.int64)\n        return x, y\n\n    with tarfile.open(archive) as tf:\n        parts = [read_batch(tf, f\"cifar-10-batches-py/data_batch_{i}\") for i in range(1, 6)]\n        x_train = np.concatenate([p[0] for p in parts])\n        y_train = np.concatenate([p[1] for p in parts])\n        x_test, y_test = read_batch(tf, \"cifar-10-batches-py/test_batch\")\n    return x_train, y_train, x_test, y_test\n\n\ndef load_kaggle_train():\n    \"\"\"Read the Kaggle training archive. Used only when the download is not possible.\"\"\"\n    archive = find_file(\"train.7z\")\n    labels_csv = find_file(\"trainLabels.csv\")\n    if archive is None or labels_csv is None:\n        raise FileNotFoundError(\"train.7z or trainLabels.csv was not found\")\n    folder = extract_7z(archive, os.path.join(SCRATCH_DIR, \"kaggle_train\"))\n    files = {int(os.path.splitext(os.path.basename(f))[0]): f\n             for f in glob.glob(os.path.join(folder, \"**\", \"*.png\"), recursive=True)}\n    labels = pd.read_csv(labels_csv)\n    paths = [files[i] for i in labels[\"id\"]]\n    x = read_png_folder(paths)\n    y = labels[\"label\"].map({c: i for i, c in enumerate(CLASSES)}).to_numpy(dtype=np.int64)\n    return x, y\n\n\ntry:\n    x_train, y_train, x_test, y_test = load_official_cifar10()\n    HAVE_OFFICIAL_TEST = True\n    print(\"Loaded the official CIFAR10 training and test sets.\")\nexcept Exception as e:\n    print(\"Could not load the official archive:\", e)\n    print(\"Reading the Kaggle training archive instead.\")\n    x_all, y_all = load_kaggle_train()\n    perm = np.random.RandomState(SEED).permutation(len(x_all))\n    x_test, y_test = x_all[perm[:5000]], y_all[perm[:5000]]\n    x_train, y_train = x_all[perm[5000:]], y_all[perm[5000:]]\n    HAVE_OFFICIAL_TEST = False\n    print(\"The test set below is a 5,000 image validation split of the training data.\")\n\nif QUICK:\n    x_train, y_train = x_train[:2000], y_train[:2000]\n    x_test, y_test = x_test[:1000], y_test[:1000]\n\nprint(\"Train images\", x_train.shape, \" Test images\", x_test.shape)\nprint(\"Images per class in the training set:\", np.bincount(y_train).tolist())","metadata":{},"outputs":[],"execution_count":null},{"id":"244349c8","cell_type":"code","source":"fig, axes = plt.subplots(10, 8, figsize=(8, 10.5))\nfor c in range(10):\n    idx = np.where(y_train == c)[0][:8]\n    for k, i in enumerate(idx):\n        ax = axes[c, k]\n        ax.imshow(x_train[i].transpose(1, 2, 0))\n        ax.set_xticks([])\n        ax.set_yticks([])\n        if k == 0:\n            ax.set_ylabel(CLASSES[c], rotation=0, ha=\"right\", va=\"center\", fontsize=10)\nfig.suptitle(\"Training images, one row per class\")\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"samples.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"e7a4146d","cell_type":"markdown","source":"## 2. Batches and data augmentation\n\nThe whole dataset fits in GPU memory as 8 bit integers. Each batch is normalized and\naugmented on the GPU. This avoids a slow data loader. Three augmentations are used:\n\n* Random crop. The image is padded by 4 pixels on each side and a 32 by 32 window is cut out.\n* Random horizontal flip with probability one half.\n* Cutout. One 8 by 8 square is set to zero.","metadata":{}},{"id":"7defb4f7","cell_type":"code","source":"MEAN = torch.tensor([0.4914, 0.4822, 0.4465], device=device).view(1, 3, 1, 1)\nSTD = torch.tensor([0.2470, 0.2435, 0.2616], device=device).view(1, 3, 1, 1)\n\ntrain_x = torch.from_numpy(x_train).to(device)\ntrain_y = torch.from_numpy(y_train).to(device)\ntest_x = torch.from_numpy(x_test).to(device)\ntest_y = torch.from_numpy(y_test).to(device)\n\n\ndef normalize(x_uint8):\n    \"\"\"Scale 8 bit images to floats with zero mean and unit variance per channel.\"\"\"\n    return (x_uint8.float() / 255.0 - MEAN) / STD\n\n\ndef augment(x):\n    \"\"\"Random crop with 4 pixel padding, random horizontal flip, and one 8 by 8 cutout box.\"\"\"\n    n = x.size(0)\n    ar = torch.arange(32, device=x.device)\n    padded = F.pad(x, (4, 4, 4, 4))\n    top = torch.randint(0, 9, (n, 1), device=x.device)\n    left = torch.randint(0, 9, (n, 1), device=x.device)\n    rows = top + ar\n    cols = left + ar\n    x = padded[torch.arange(n, device=x.device)[:, None, None, None],\n               torch.arange(3, device=x.device)[None, :, None, None],\n               rows[:, None, :, None],\n               cols[:, None, None, :]]\n    flip = torch.rand(n, device=x.device) < 0.5\n    x = torch.where(flip[:, None, None, None], x.flip(3), x)\n    cy = torch.randint(0, 32, (n, 1, 1), device=x.device)\n    cx = torch.randint(0, 32, (n, 1, 1), device=x.device)\n    dy = ar[None, :, None] - cy\n    dx = ar[None, None, :] - cx\n    box = (dy >= -4) & (dy < 4) & (dx >= -4) & (dx < 4)\n    return x.masked_fill(box[:, None], 0.0)\n\n\ndef to_image(x):\n    \"\"\"Undo the normalization so that a tensor can be drawn.\"\"\"\n    x = (x * STD + MEAN).clamp(0, 1)\n    return x.permute(0, 2, 3, 1).cpu().numpy()\n\n\nsample = train_x[:8]\nfig, axes = plt.subplots(2, 8, figsize=(10, 2.8))\nfor k, (o, a) in enumerate(zip(to_image(normalize(sample)), to_image(augment(normalize(sample))))):\n    axes[0, k].imshow(o)\n    axes[1, k].imshow(a)\n    axes[0, k].axis(\"off\")\n    axes[1, k].axis(\"off\")\naxes[0, 0].set_title(\"original\", loc=\"left\")\naxes[1, 0].set_title(\"augmented\", loc=\"left\")\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"augmentation.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"57ef265b","cell_type":"markdown","source":"## 3. Model\n\nThe model is a ResNet18 for small images. The first layer is a 3 by 3 convolution with stride\none, and there is no max pooling. The four stages have 64, 128, 256 and 512 channels. Each stage\nhas two residual blocks. Each block has two 3 by 3 convolutions with batch normalization and\nReLU. A global average pool and one linear layer give the ten class scores. The network has\nabout 11 million parameters. PyTorch initializes all of them randomly.","metadata":{}},{"id":"f8c7b0a1","cell_type":"code","source":"class BasicBlock(nn.Module):\n    def __init__(self, cin, cout, stride):\n        super().__init__()\n        self.conv1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False)\n        self.bn1 = nn.BatchNorm2d(cout)\n        self.conv2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False)\n        self.bn2 = nn.BatchNorm2d(cout)\n        self.shortcut = nn.Sequential()\n        if stride != 1 or cin != cout:\n            self.shortcut = nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False),\n                                          nn.BatchNorm2d(cout))\n\n    def forward(self, x):\n        out = F.relu(self.bn1(self.conv1(x)))\n        out = self.bn2(self.conv2(out))\n        return F.relu(out + self.shortcut(x))\n\n\nclass ResNet18(nn.Module):\n    def __init__(self, num_classes=10):\n        super().__init__()\n        self.stem = nn.Sequential(nn.Conv2d(3, 64, 3, 1, 1, bias=False),\n                                  nn.BatchNorm2d(64), nn.ReLU(inplace=True))\n        self.layer1 = self.make_stage(64, 64, 1)\n        self.layer2 = self.make_stage(64, 128, 2)\n        self.layer3 = self.make_stage(128, 256, 2)\n        self.layer4 = self.make_stage(256, 512, 2)\n        self.fc = nn.Linear(512, num_classes)\n\n    @staticmethod\n    def make_stage(cin, cout, stride):\n        return nn.Sequential(BasicBlock(cin, cout, stride), BasicBlock(cout, cout, 1))\n\n    def forward(self, x):\n        x = self.stem(x)\n        x = self.layer4(self.layer3(self.layer2(self.layer1(x))))\n        x = F.adaptive_avg_pool2d(x, 1).flatten(1)\n        return self.fc(x)\n\n\nmodel = ResNet18().to(device)\nCHANNELS_LAST = device.type == \"cuda\"\nif CHANNELS_LAST:\n    model = model.to(memory_format=torch.channels_last)\nn_params = sum(p.numel() for p in model.parameters())\nprint(f\"ResNet18 with {n_params / 1e6:.2f} million parameters, all randomly initialized\")","metadata":{},"outputs":[],"execution_count":null},{"id":"b1bd46b8","cell_type":"markdown","source":"## 4. Training\n\nThe optimizer is SGD with Nesterov momentum 0.9 and weight decay 0.0005. The learning rate\nfollows a one cycle schedule. It rises from 0.01 to 0.1 in the first 20 percent of the steps\nand then falls to almost zero with a cosine curve. The loss is cross entropy with label\nsmoothing 0.1. On a CUDA GPU the forward and backward passes run in mixed precision.\nThe model from the last epoch is used. No epoch is chosen with the test set.","metadata":{}},{"id":"eb9d8998","cell_type":"code","source":"LR = 0.1\nWEIGHT_DECAY = 5e-4\nLABEL_SMOOTHING = 0.1\nsteps_per_epoch = math.ceil(len(train_x) / BATCH_SIZE)\n\noptimizer = torch.optim.SGD(model.parameters(), lr=LR, momentum=0.9, nesterov=True,\n                            weight_decay=WEIGHT_DECAY)\nscheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=LR, epochs=EPOCHS,\n                                                steps_per_epoch=steps_per_epoch, pct_start=0.2,\n                                                div_factor=10, final_div_factor=100)\ncriterion = nn.CrossEntropyLoss(label_smoothing=LABEL_SMOOTHING)\nUSE_AMP = device.type == \"cuda\"\nscaler = torch.amp.GradScaler(\"cuda\", enabled=USE_AMP)\n\n\ndef prepare(x):\n    \"\"\"Move a normalized batch to the memory layout that the GPU prefers.\"\"\"\n    if CHANNELS_LAST:\n        return x.contiguous(memory_format=torch.channels_last)\n    return x\n\n\ndef predict_logits(xb, tta):\n    \"\"\"Return class scores. With tta the scores of the flipped image are added.\"\"\"\n    with torch.autocast(\"cuda\", dtype=torch.float16, enabled=USE_AMP):\n        logits = model(xb).float()\n        if tta:\n            logits = logits + model(xb.flip(3)).float()\n    return logits\n\n\ndef evaluate(x_uint8, y, tta=False, batch=1000):\n    \"\"\"Accuracy and predictions on a set of images stored as 8 bit tensors.\"\"\"\n    model.eval()\n    preds = []\n    with torch.no_grad():\n        for i in range(0, len(x_uint8), batch):\n            xb = prepare(normalize(x_uint8[i:i + batch]))\n            preds.append(predict_logits(xb, tta).argmax(1))\n    preds = torch.cat(preds)\n    return (preds == y).float().mean().item(), preds\n\n\nhistory = []\ntrain_start = time.time()\nfor epoch in range(1, EPOCHS + 1):\n    model.train()\n    perm = torch.randperm(len(train_x), device=device)\n    loss_sum = torch.zeros((), device=device)\n    correct = torch.zeros((), device=device, dtype=torch.long)\n    for i in range(0, len(perm), BATCH_SIZE):\n        idx = perm[i:i + BATCH_SIZE]\n        xb = prepare(augment(normalize(train_x[idx])))\n        yb = train_y[idx]\n        with torch.autocast(\"cuda\", dtype=torch.float16, enabled=USE_AMP):\n            logits = model(xb)\n            loss = criterion(logits, yb)\n        optimizer.zero_grad(set_to_none=True)\n        scaler.scale(loss).backward()\n        scaler.step(optimizer)\n        scaler.update()\n        scheduler.step()\n        loss_sum += loss.detach() * len(idx)\n        correct += (logits.argmax(1) == yb).sum()\n    train_loss = loss_sum.item() / len(perm)\n    train_acc = correct.item() / len(perm)\n    test_acc, _ = evaluate(test_x, test_y)\n    elapsed = time.time() - train_start\n    history.append(dict(epoch=epoch, lr=optimizer.param_groups[0][\"lr\"], train_loss=train_loss,\n                        train_acc=train_acc, test_acc=test_acc, time=elapsed))\n    print(f\"epoch {epoch:3d}  train loss {train_loss:.3f}  train acc {train_acc:.4f}  \"\n          f\"test acc {test_acc:.4f}  time {elapsed:6.0f}s\")\n\nTRAIN_MINUTES = (time.time() - train_start) / 60\nprint(f\"Training took {TRAIN_MINUTES:.1f} minutes\")\ntorch.save(model.state_dict(), os.path.join(OUT_DIR, \"cifar10_resnet18.pt\"))","metadata":{},"outputs":[],"execution_count":null},{"id":"f5e9e06d","cell_type":"code","source":"hist = pd.DataFrame(history)\nfig, axes = plt.subplots(1, 2, figsize=(10, 3.6))\naxes[0].plot(hist.epoch, hist.train_loss, marker=\"o\")\naxes[0].set_xlabel(\"epoch\")\naxes[0].set_ylabel(\"training loss\")\naxes[0].grid(alpha=0.3)\naxes[1].plot(hist.epoch, hist.train_acc, marker=\"o\", label=\"train accuracy\")\naxes[1].plot(hist.epoch, hist.test_acc, marker=\"o\", label=\"test accuracy\")\naxes[1].set_xlabel(\"epoch\")\naxes[1].set_ylabel(\"accuracy\")\naxes[1].grid(alpha=0.3)\naxes[1].legend()\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"training_curves.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"ca565177","cell_type":"markdown","source":"## 5. Accuracy on the official CIFAR10 test set\n\nThis is the estimate of the Kaggle score. The prediction also uses test time augmentation.\nThe class scores of the image and of its mirror image are added before the argmax.","metadata":{}},{"id":"14459227","cell_type":"code","source":"acc_plain, _ = evaluate(test_x, test_y, tta=False)\nacc_tta, preds = evaluate(test_x, test_y, tta=True)\nset_name = \"official CIFAR10 test set\" if HAVE_OFFICIAL_TEST else \"validation split\"\nprint(f\"Accuracy on the {set_name} without flip averaging: {acc_plain:.4f}\")\nprint(f\"Accuracy on the {set_name} with flip averaging:    {acc_tta:.4f}\")\nprint(f\"Estimated Kaggle score: {acc_tta:.4f}\")\n\ncm = torch.zeros(10, 10, dtype=torch.long)\nfor t, p in zip(test_y.cpu(), preds.cpu()):\n    cm[t, p] += 1\ncm = cm.numpy()\nper_class = pd.DataFrame({\"class\": CLASSES, \"accuracy\": cm.diagonal() / cm.sum(1)})\nprint(per_class.to_string(index=False, float_format=lambda v: f\"{v:.4f}\"))\n\nfig, ax = plt.subplots(figsize=(6.5, 5.5))\nim = ax.imshow(cm, cmap=\"Blues\")\nax.set_xticks(range(10))\nax.set_yticks(range(10))\nax.set_xticklabels(CLASSES, rotation=45, ha=\"right\")\nax.set_yticklabels(CLASSES)\nax.set_xlabel(\"predicted\")\nax.set_ylabel(\"true\")\nfor r in range(10):\n    for c in range(10):\n        ax.text(c, r, cm[r, c], ha=\"center\", va=\"center\", fontsize=7,\n                color=\"white\" if cm[r, c] > cm.max() / 2 else \"black\")\nax.set_title(f\"Confusion matrix, accuracy {acc_tta:.4f}\")\nfig.tight_layout()\nfig.savefig(os.path.join(OUT_DIR, \"confusion_matrix.png\"), dpi=120)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"7f3f14e5","cell_type":"markdown","source":"## 6. Predict the Kaggle test set and write the submission file\n\nThe archive test.7z holds 300,000 PNG files named 1.png to 300000.png. They are extracted\nto a scratch folder, read in batches, and predicted with the same flip averaging as above.\nThe submission file has two columns, id and label, with the class name as the label.","metadata":{}},{"id":"21e7cfa1","cell_type":"code","source":"test_archive = os.environ.get(\"CIFAR_TEST_7Z\") or find_file(\"test.7z\")\nif test_archive is None:\n    print(\"test.7z was not found. On Kaggle add the competition data as an input. \"\n          \"Locally put test.7z in the data folder.\")\n    submission = None\nelse:\n    extract_dir = os.path.join(SCRATCH_DIR, \"kaggle_test\")\n    files = glob.glob(os.path.join(extract_dir, \"**\", \"*.png\"), recursive=True)\n    if len(files) < 300000:\n        t0 = time.time()\n        print(\"Extracting\", test_archive)\n        extract_7z(test_archive, extract_dir)\n        files = glob.glob(os.path.join(extract_dir, \"**\", \"*.png\"), recursive=True)\n        print(f\"Extracted {len(files)} images in {time.time() - t0:.0f} seconds\")\n    by_id = {int(os.path.splitext(os.path.basename(f))[0]): f for f in files}\n    ids = sorted(by_id)\n    print(\"Number of test images:\", len(ids))\n    if len(ids) != 300000:\n        print(\"Warning: the Kaggle test set should have 300000 images.\")\n\n    class PngFolder(torch.utils.data.Dataset):\n        def __init__(self, paths):\n            self.paths = paths\n\n        def __len__(self):\n            return len(self.paths)\n\n        def __getitem__(self, i):\n            img = np.asarray(Image.open(self.paths[i]).convert(\"RGB\"), dtype=np.uint8)\n            return torch.from_numpy(img.copy()).permute(2, 0, 1)\n\n    workers = min(4, os.cpu_count() or 1) if device.type == \"cuda\" else 0\n    loader = torch.utils.data.DataLoader(PngFolder([by_id[i] for i in ids]), batch_size=1000,\n                                         shuffle=False, num_workers=workers)\n    model.eval()\n    preds = []\n    t0 = time.time()\n    with torch.no_grad():\n        for b, xb in enumerate(loader):\n            xb = prepare(normalize(xb.to(device, non_blocking=True)))\n            preds.append(predict_logits(xb, tta=True).argmax(1).cpu())\n            if (b + 1) % 50 == 0:\n                print(f\"  predicted {(b + 1) * 1000} images in {time.time() - t0:.0f} seconds\")\n    preds = torch.cat(preds).numpy()\n    submission = pd.DataFrame({\"id\": ids, \"label\": [CLASSES[p] for p in preds]})\n    sub_path = os.path.join(OUT_DIR, \"submission.csv\")\n    submission.to_csv(sub_path, index=False)\n    print(f\"Wrote {sub_path} with {len(submission)} rows in {time.time() - t0:.0f} seconds\")\n    print(submission.head())\n    print(\"Predicted labels per class:\")\n    print(submission[\"label\"].value_counts().to_string())","metadata":{},"outputs":[],"execution_count":null},{"id":"6be90563","cell_type":"markdown","source":"## 7. Summary\n\nThe table below repeats the main numbers. Submit the file submission.csv on the competition\npage as a late submission. The Kaggle score should be close to the estimate above.","metadata":{}},{"id":"c7872a30","cell_type":"code","source":"summary = pd.DataFrame({\n    \"item\": [\"model\", \"parameters\", \"epochs\", \"batch size\", \"training minutes\",\n             f\"accuracy on the {set_name}\", \"estimated Kaggle score\", \"submission rows\"],\n    \"value\": [\"ResNet18, random initialization\", f\"{n_params / 1e6:.2f} million\", EPOCHS,\n              BATCH_SIZE, f\"{TRAIN_MINUTES:.1f}\", f\"{acc_plain:.4f}\", f\"{acc_tta:.4f}\",\n              0 if submission is None else len(submission)],\n})\nprint(summary.to_string(index=False))","metadata":{},"outputs":[],"execution_count":null}]}