{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"2967348e","cell_type":"markdown","source":"# CIFAR-10 (Kaggle) — ResNet9, trained from scratch\n\nThis notebook trains a convolutional neural network **from random initialization** (no pretrained\nweights) on the official CIFAR-10 training set, evaluates it on a held-out official CIFAR-10 test\nset (to estimate the Kaggle score before submitting), and then generates predictions for the\n**Kaggle `cifar-10` competition** test set (300,000 images, of which the official 10,000 test\nimages are the ones actually scored).\n\n**How to run this on Kaggle:**\n1. Create a new Kaggle Notebook.\n2. In the right-hand panel, click *Add Input* → search for the competition **`cifar-10`** → add it.\n   On current Kaggle, this mounts the competition files under\n   `/kaggle/input/competitions/cifar-10/` (verify with the diagnostic cell in section 6 if the\n   path has moved again).\n3. Turn on a GPU accelerator: *Settings → Accelerator → GPU T4 x2* (or P100). Do this **before**\n   running any cells — switching accelerators restarts the kernel and clears `/kaggle/working`,\n   so anything downloaded before the switch is lost and must be re-fetched.\n4. Run all cells top to bottom (Run → Run All). Training + inference on 300k test images takes\n   roughly 15–25 minutes on a T4.\n5. The last cell writes `submission.csv`. Go to the competition page and click **Submit\n   Predictions** (recorded as a *late submission* since the competition is closed).\n","metadata":{}},{"id":"439ee04b","cell_type":"markdown","source":"## 1. Setup","metadata":{}},{"id":"a52e3006","cell_type":"code","source":"import os, glob, time, math, random\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader, Dataset\nimport torchvision.transforms as T\nfrom PIL import Image\n\nSEED = 42\nrandom.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED); torch.cuda.manual_seed_all(SEED)\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Using device:', device)\nprint('Torch version:', torch.__version__)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:23:09.656729Z","iopub.execute_input":"2026-09-21T16:23:09.657281Z","iopub.status.idle":"2026-09-21T16:23:09.666464Z","shell.execute_reply.started":"2026-09-21T16:23:09.657245Z","shell.execute_reply":"2026-09-21T16:23:09.665575Z"}},"outputs":[],"execution_count":null},{"id":"fe5f9b30","cell_type":"markdown","source":"## 2. Load the official CIFAR-10 training data\n\nWe train on the **official CIFAR-10 training set** (50,000 images) and hold out the official\nCIFAR-10 test set (10,000 images) purely to **estimate** the Kaggle score offline before\nsubmitting — we never train on the held-out set.\n\nData is loaded via `tf.keras.datasets.cifar10`, which downloads the same canonical CIFAR-10\ntarball used everywhere else (`cs.toronto.edu`), then wrapped in a small custom `Dataset` so it\nplugs into standard PyTorch `DataLoader`s with our own augmentation pipeline.\n","metadata":{}},{"id":"4762e924","cell_type":"code","source":"CIFAR_MEAN = (0.4914, 0.4822, 0.4465)\nCIFAR_STD  = (0.2470, 0.2435, 0.2616)\n\ntrain_tfms = T.Compose([\n    T.RandomCrop(32, padding=4, padding_mode='reflect'),\n    T.RandomHorizontalFlip(),\n    T.ToTensor(),\n    T.Normalize(CIFAR_MEAN, CIFAR_STD),\n    T.RandomErasing(p=0.25, scale=(0.02, 0.2)),  # cutout-style augmentation\n])\n\ntest_tfms = T.Compose([\n    T.ToTensor(),\n    T.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\n\nfrom tensorflow.keras.datasets import cifar10 as keras_cifar10\n(x_train, y_train), (x_test, y_test) = keras_cifar10.load_data()\ny_train = y_train.flatten()\ny_test = y_test.flatten()\n\nclass ArrayCIFAR10(Dataset):\n    def __init__(self, images, labels, transform):\n        self.images = images   # uint8, shape (N, 32, 32, 3)\n        self.labels = labels\n        self.transform = transform\n    def __len__(self):\n        return len(self.images)\n    def __getitem__(self, idx):\n        img = Image.fromarray(self.images[idx])\n        return self.transform(img), int(self.labels[idx])\n\nclasses = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n\ntrain_ds = ArrayCIFAR10(x_train, y_train, train_tfms)\ntest_ds  = ArrayCIFAR10(x_test,  y_test,  test_tfms)\nprint(classes)\n\nBATCH_SIZE = 512\ntrain_dl = DataLoader(train_ds, batch_size=BATCH_SIZE, shuffle=True,  num_workers=2, pin_memory=True, drop_last=True)\ntest_dl  = DataLoader(test_ds,  batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)\n\nprint(f'Train: {len(train_ds)}  Test: {len(test_ds)}')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:23:09.668019Z","iopub.execute_input":"2026-09-21T16:23:09.66835Z","iopub.status.idle":"2026-09-21T16:47:04.874041Z","shell.execute_reply.started":"2026-09-21T16:23:09.668316Z","shell.execute_reply":"2026-09-21T16:47:04.873065Z"}},"outputs":[],"execution_count":null},{"id":"d6a722fe","cell_type":"markdown","source":"## 3. Model — ResNet9 (randomly initialized, no pretrained weights)\n\nWe use a compact residual CNN (the \"ResNet9\" architecture popularized by the fast.ai / DAWNBench\nCIFAR-10 speed-training entries). It is a well-known existing architecture, but **all weights are\nrandomly initialized here** — nothing is loaded from a checkpoint, so this satisfies the\nassignment's \"no pretrained weights\" requirement.\n","metadata":{}},{"id":"07c92ea7","cell_type":"code","source":"def conv_bn(ch_in, ch_out, pool=False):\n    layers = [\n        nn.Conv2d(ch_in, ch_out, kernel_size=3, padding=1, bias=False),\n        nn.BatchNorm2d(ch_out),\n        nn.ReLU(inplace=True),\n    ]\n    if pool:\n        layers.append(nn.MaxPool2d(2))\n    return nn.Sequential(*layers)\n\nclass Residual(nn.Module):\n    def __init__(self, ch):\n        super().__init__()\n        self.block = nn.Sequential(conv_bn(ch, ch), conv_bn(ch, ch))\n    def forward(self, x):\n        return x + self.block(x)\n\nclass ResNet9(nn.Module):\n    def __init__(self, in_ch=3, num_classes=10):\n        super().__init__()\n        self.stem   = conv_bn(in_ch, 64)\n        self.layer1 = conv_bn(64, 128, pool=True)\n        self.res1   = Residual(128)\n        self.layer2 = conv_bn(128, 256, pool=True)\n        self.layer3 = conv_bn(256, 512, pool=True)\n        self.res2   = Residual(512)\n        self.pool   = nn.Sequential(nn.MaxPool2d(4), nn.Flatten())\n        self.fc     = nn.Linear(512, num_classes)\n\n    def forward(self, x):\n        x = self.stem(x)\n        x = self.layer1(x)\n        x = self.res1(x)\n        x = self.layer2(x)\n        x = self.layer3(x)\n        x = self.res2(x)\n        x = self.pool(x)\n        return self.fc(x)\n\nmodel = ResNet9().to(device)\nn_params = sum(p.numel() for p in model.parameters())\nprint(f'ResNet9 parameters: {n_params:,} (all randomly initialized)')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:47:04.875299Z","iopub.execute_input":"2026-09-21T16:47:04.876045Z","iopub.status.idle":"2026-09-21T16:47:05.350509Z","shell.execute_reply.started":"2026-09-21T16:47:04.876002Z","shell.execute_reply":"2026-09-21T16:47:05.349316Z"}},"outputs":[],"execution_count":null},{"id":"d6f7e9ab","cell_type":"markdown","source":"## 4. Training\n\n* **Optimizer:** SGD with Nesterov momentum, weight decay.\n* **LR schedule:** One-Cycle policy (linear warmup, cosine decay).\n* **Loss:** cross-entropy with label smoothing (0.1).\n* **Mixed precision (AMP):** uses the current, non-deprecated `torch.amp` API (PyTorch ≥ 2.x),\n  not the legacy `torch.cuda.amp` module.\n* **Epochs:** 30.\n","metadata":{}},{"id":"ccc6c1a4","cell_type":"code","source":"EPOCHS = 30\nMAX_LR = 0.4\nWEIGHT_DECAY = 5e-4\n\noptimizer = torch.optim.SGD(model.parameters(), lr=MAX_LR, momentum=0.9,\n                             nesterov=True, weight_decay=WEIGHT_DECAY)\nsteps_per_epoch = len(train_dl)\nscheduler = torch.optim.lr_scheduler.OneCycleLR(\n    optimizer, max_lr=MAX_LR, epochs=EPOCHS, steps_per_epoch=steps_per_epoch,\n    pct_start=0.3, div_factor=10, final_div_factor=100)\n\ncriterion = nn.CrossEntropyLoss(label_smoothing=0.1)\nscaler = torch.amp.GradScaler('cuda')\n\ndef evaluate(dl):\n    model.eval()\n    correct, total = 0, 0\n    with torch.no_grad():\n        for xb, yb in dl:\n            xb, yb = xb.to(device, non_blocking=True), yb.to(device, non_blocking=True)\n            with torch.amp.autocast('cuda'):\n                out = model(xb)\n            correct += (out.argmax(1) == yb).sum().item()\n            total += yb.size(0)\n    return correct / total\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:47:05.352747Z","iopub.execute_input":"2026-09-21T16:47:05.353055Z","iopub.status.idle":"2026-09-21T16:47:05.361398Z","shell.execute_reply.started":"2026-09-21T16:47:05.353013Z","shell.execute_reply":"2026-09-21T16:47:05.36067Z"}},"outputs":[],"execution_count":null},{"id":"cd5a1958","cell_type":"code","source":"history = []\nt0 = time.time()\n\nfor epoch in range(EPOCHS):\n    model.train()\n    running_loss, seen = 0.0, 0\n    for xb, yb in train_dl:\n        xb, yb = xb.to(device, non_blocking=True), yb.to(device, non_blocking=True)\n        optimizer.zero_grad(set_to_none=True)\n        with torch.amp.autocast('cuda'):\n            out = model(xb)\n            loss = criterion(out, yb)\n        scaler.scale(loss).backward()\n        scaler.unscale_(optimizer)\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)\n        scaler.step(optimizer)\n        scaler.update()\n        scheduler.step()\n        running_loss += loss.item() * yb.size(0)\n        seen += yb.size(0)\n\n    train_loss = running_loss / seen\n    test_acc = evaluate(test_dl)\n    history.append((epoch + 1, train_loss, test_acc))\n    elapsed = time.time() - t0\n    print(f'epoch {epoch+1:2d}/{EPOCHS} | train_loss {train_loss:.4f} | '\n          f'official CIFAR-10 test acc {test_acc*100:.2f}% | {elapsed/60:.1f} min elapsed')\n\nprint('Training done.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:47:05.369202Z","iopub.execute_input":"2026-09-21T16:47:05.369559Z","iopub.status.idle":"2026-09-21T16:58:00.582845Z"}},"outputs":[],"execution_count":null},{"id":"60f55448","cell_type":"markdown","source":"## 5. Offline evaluation on the held-out official CIFAR-10 test set\n\nBecause the Kaggle test set is a superset of the official CIFAR-10 test set, and only the images\noverlapping with the official 10,000-image test set are scored, this offline accuracy is a good\n**estimate of the Kaggle leaderboard score** before we even submit.\n","metadata":{}},{"id":"699bb06d","cell_type":"code","source":"final_test_acc = evaluate(test_dl)\nprint(f'Final accuracy on the OFFICIAL CIFAR-10 test set (10,000 images): {final_test_acc*100:.2f}%')\nprint('This is our estimate of the Kaggle score.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:58:00.588212Z","iopub.execute_input":"2026-09-21T16:58:00.588464Z","iopub.status.idle":"2026-09-21T16:58:02.193406Z"}},"outputs":[],"execution_count":null},{"id":"096c1323","cell_type":"code","source":"import matplotlib.pyplot as plt\n\nepochs_r  = [h[0] for h in history]\nlosses_r  = [h[1] for h in history]\naccs_r    = [h[2] for h in history]\n\nfig, ax1 = plt.subplots(figsize=(7,4))\nax1.plot(epochs_r, losses_r, color='tab:red', label='train loss')\nax1.set_xlabel('epoch'); ax1.set_ylabel('train loss', color='tab:red')\nax2 = ax1.twinx()\nax2.plot(epochs_r, [a*100 for a in accs_r], color='tab:blue', label='test acc (%)')\nax2.set_ylabel('official CIFAR-10 test accuracy (%)', color='tab:blue')\nplt.title('Training loss and held-out accuracy per epoch')\nplt.tight_layout(); plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:58:02.202254Z","iopub.execute_input":"2026-09-21T16:58:02.202579Z","iopub.status.idle":"2026-09-21T16:58:02.560152Z"}},"outputs":[],"execution_count":null},{"id":"3f1082a4","cell_type":"markdown","source":"## 6. Locate and extract the Kaggle `test.7z` (300,000 images)\n\nOn current Kaggle notebooks, an attached competition's files are mounted under\n`/kaggle/input/competitions/<competition-slug>/` rather than directly under `/kaggle/input/`.\nRun the diagnostic cell first if you're unsure of the exact path on your own notebook — Kaggle's\nmount layout has changed over time and may change again.\n\nWe use `py7zr` (pure Python) to extract the archive rather than the system `libarchive` package,\nsince the `libarchive-dev` Ubuntu package is not reliably installable on Kaggle's current base\nimage (the apt mirror can 404 on the pinned version).\n","metadata":{}},{"id":"cb3d6590","cell_type":"code","source":"# Diagnostic: find exactly where the competition files live on THIS notebook.\nfor root, dirs, files in os.walk('/kaggle/input'):\n    for f in files:\n        print(os.path.join(root, f))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T16:58:02.56458Z","iopub.execute_input":"2026-09-21T16:58:02.564919Z","iopub.status.idle":"2026-09-21T16:58:02.579155Z"}},"outputs":[],"execution_count":null},{"id":"72624fd3","cell_type":"code","source":"!pip install -q py7zr\n\nimport shutil\nimport py7zr\n\nKAGGLE_INPUT = '/kaggle/input/competitions/cifar-10'  # adjust if the diagnostic cell above shows a different path\nWORK_DIR = '/kaggle/working'\n\nsrc_7z = os.path.join(KAGGLE_INPUT, 'test.7z')\ndst_7z = os.path.join(WORK_DIR, 'test.7z')\nif os.path.exists(src_7z) and not os.path.exists(dst_7z):\n    shutil.copy(src_7z, dst_7z)\n\nos.chdir(WORK_DIR)\n\nif not os.path.isdir('test') or len(glob.glob('test/*')) < 300000:\n    with py7zr.SevenZipFile(dst_7z, mode='r') as archive:\n        archive.extractall(path=WORK_DIR)\n\nprint('Extracted test images:', len(glob.glob('test/*')))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T17:18:45.844789Z","iopub.execute_input":"2026-09-21T17:18:45.845536Z","iopub.status.idle":"2026-09-21T17:18:50.851065Z","shell.execute_reply.started":"2026-09-21T17:18:45.845496Z","shell.execute_reply":"2026-09-21T17:18:50.849802Z"}},"outputs":[],"execution_count":null},{"id":"a68f33b7","cell_type":"markdown","source":"## 7. Predict on the Kaggle test set and build `submission.csv`\n\nThe Kaggle submission format is a CSV with columns `id` (matching the test image filename,\n`<id>.png`) and `label` (the predicted class **name**, e.g. `cat`). We use CIFAR-10's alphabetical\nclass ordering, which matches the label names Kaggle expects.\n","metadata":{}},{"id":"61e13760","cell_type":"code","source":"class KaggleTestDataset(Dataset):\n    def __init__(self, folder, transform):\n        self.paths = sorted(\n            glob.glob(os.path.join(folder, '*.png')),\n            key=lambda p: int(os.path.splitext(os.path.basename(p))[0])\n        )\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.paths)\n\n    def __getitem__(self, idx):\n        path = self.paths[idx]\n        img_id = int(os.path.splitext(os.path.basename(path))[0])\n        img = Image.open(path).convert('RGB')\n        return self.transform(img), img_id\n\nkaggle_test_ds = KaggleTestDataset(os.path.join(WORK_DIR, 'test'), test_tfms)\nkaggle_test_dl = DataLoader(kaggle_test_ds, batch_size=1024, shuffle=False, num_workers=2, pin_memory=True)\nprint('Kaggle test images:', len(kaggle_test_ds))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-21T17:18:50.85324Z","iopub.execute_input":"2026-09-21T17:18:50.853633Z","iopub.status.idle":"2026-09-21T17:18:51.896111Z","shell.execute_reply.started":"2026-09-21T17:18:50.853595Z","shell.execute_reply":"2026-09-21T17:18:51.895433Z"}},"outputs":[],"execution_count":null},{"id":"31262f39","cell_type":"code","source":"model.eval()\nall_ids, all_preds = [], []\n\nwith torch.no_grad():\n    for xb, ids in kaggle_test_dl:\n        xb = xb.to(device, non_blocking=True)\n        with torch.amp.autocast('cuda'):\n            out = model(xb)\n        preds = out.argmax(1).cpu().numpy()\n        all_ids.extend(ids.numpy().tolist())\n        all_preds.extend(preds.tolist())\n\nimport pandas as pd\nsub = pd.DataFrame({'id': all_ids, 'label': [classes[p] for p in all_preds]})\nsub = sub.sort_values('id').reset_index(drop=True)\nsub.to_csv('/kaggle/working/submission.csv', index=False)\nsub.head()\n","metadata":{"trusted":true,"execution":{"iopub.status.idle":"2026-09-21T17:03:26.603859Z"}},"outputs":[],"execution_count":null}]}