{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","pygments_lexer":"ipython3"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"37436f96-f67d-423a-9644-4cf000c76ab5","cell_type":"markdown","source":"# Dog Breed Identification — 0.1177 on the leaderboard from frozen features + a LoRA branch\n\n**Public leaderboard: 0.11766** multi-class log loss, scored on the `submission.csv` this\nnotebook writes in section 8 · **0.1062 log loss and 97.36% accuracy** on a held-out 10% split.\n\nAlmost nothing here is trained by gradient descent on the labels: three\ndropout-plus-`Linear` heads, and one rank-4 LoRA adapter that is discarded the moment it has\ndone its job. Everything else is about *what goes into those heads* and *how their outputs get\ncombined*.\n\nThe competition gives 10,222 labelled photos across 120 breeds — about 85 images per class.\nThat is far too little to fine-tune a 300M-parameter backbone without it memorising the training\nset in three epochs. So this pipeline mostly does not fine-tune anything: it freezes several\nImageNet backbones, dumps their penultimate activations to disk once, and then spends the\ncompute budget on the cheap part — searching, seed-averaging and calibrating a linear head over\nthose frozen vectors.\n\nThree feature branches feed three separate heads, and the heads are blended:\n\n| branch | features | eval transform | val log loss |\n|:--|:--|:--|--:|\n| **A** — ViT-L/32 + EfficientNet-B3 + Inception-ResNet-v2, concatenated | 4096-d | resize 384² | 0.1134 |\n| **B** — ViT-L/32 alone | 1024-d | resize 416, center-crop 384 | 0.1212 |\n| **C** — ViT-L/32 after LoRA fine-tuning, then re-frozen | 1024-d | resize 416, center-crop 384 | 0.1242 |\n| **blend** — weights 0.59 / 0.13 / 0.28, temperature 0.95 | | | **0.1062** |\n\n**Two numbers, two datasets.** Everything in the table is measured on the 1,023-image\nvalidation split held out of the 10,222 labelled photos. The 0.11766 above is a different\nmeasurement entirely: the leaderboard's own scoring of the 10,357 test images, whose labels\nare not public. Validation is the friendlier of the two by construction — the temperature and\nthe blend weights are *fitted* on exactly those 1,023 images and then applied to test\nunchanged — and that is most of the 0.011 gap between them. Read the leaderboard number as\nthe result; the validation number is for comparing variants inside the notebook.\n\nEvery stage is seeded — the split, the LoRA fine-tune, and all fifteen heads — and the table\nabove is what this notebook printed on the run you are reading. Seeds do not buy bit-identical\nresults across *environments*, though. The same code on Colab lands at 0.1046–0.1049; the\n0.1062 here is a Kaggle T4 with a newer `timm` and different cuDNN kernel choices. Inside that\nspread, MC-Dropout resampling accounts for roughly 0.0003 — section 10 explains why that part\nis expected, and why it is the least interesting knob in the notebook.\n\n---\n\n### Running this notebook\n\nTurn on **GPU** (the published run used a single T4) and **Internet** in the settings\npanel — internet is needed to pull `timm` weights and to `pip install peft`. Then run top\nto bottom.\n\nFeature extraction is where the time goes: about 350k backbone forward passes at 384², of which\n270k belong to branch A, plus the LoRA fine-tune — roughly 150k forward+backward passes, which\nearly-stops at epoch 14. Expect a few hours on a T4 rather than a few minutes. Every\nextraction step writes a `.npy` into `CFG.cache_dir`, so if a session does drop you can set\n`CFG.keep_cache = True`, commit, and the next run resumes instead of re-extracting.\n\nTo make it much cheaper: set `CFG.st_tta = 0`, `CFG.st_train_aug = 1`, and drop `irv2` from\n`BACKBONES`. That costs a few thousandths of log loss and cuts the runtime to a fraction of it.\n\n### Contents\n\n1. [Setup](#s1)\n2. [The data](#s2)\n3. [Feature extraction](#s3)\n4. [The classifier head](#s4)\n5. [Hyperparameter search](#s5)\n6. [Final training](#s6)\n7. [Calibration and blending](#s7)\n8. [Test inference and submission](#s8)\n9. [Error analysis](#s9)\n10. [Notes from building this](#s10)","metadata":{}},{"id":"6de5530b-80b8-424b-a935-aa54978f82f2","cell_type":"markdown","source":"<a id=\"s1\"></a>\n## 1. Setup","metadata":{}},{"id":"6a566d94-88bc-4e8b-964b-93445ba4b5d0","cell_type":"code","source":"# timm ships with the Kaggle image but is usually a few versions behind; peft and optuna do not.\n!pip install -q -U timm\n!pip install -q peft optuna","metadata":{},"outputs":[],"execution_count":null},{"id":"439bd945-3d86-4633-80c8-59de5ce39ad2","cell_type":"code","source":"import os, gc, copy, json, random, shutil, warnings\nfrom pathlib import Path\nfrom contextlib import contextmanager\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nfrom tqdm.auto import tqdm\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader, TensorDataset\nfrom torchvision import transforms\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, precision_recall_fscore_support\n\nimport timm\nfrom peft import LoraConfig, get_peft_model\n\n# peft runs every LoRA target through its quantisation dispatchers, and the torchao one\n# hard-raises when the installed torchao is older than the minimum it wants (the Kaggle\n# image ships 0.10.0 against a 0.16.0 floor).  Nothing here is quantised, so report torchao\n# as absent and let the dispatcher fall through to the plain nn.Linear path.\nimport sys, peft.import_utils\ntry:\n    import peft.tuners.lora.torchao          # the module that holds the version check\nexcept Exception:\n    pass\n\ndef _torchao_unavailable():\n    return False\n\nfor _m in list(sys.modules.values()):\n    if getattr(_m, '__name__', '').startswith('peft') and hasattr(_m, 'is_torchao_available'):\n        _m.is_torchao_available = _torchao_unavailable\n\nwarnings.filterwarnings('ignore')\nplt.rcParams['figure.dpi'] = 110\n\nDEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nAMP    = DEVICE.type == 'cuda'\n\nprint(f'torch {torch.__version__} | timm {timm.__version__}')\nprint('device:', torch.cuda.get_device_name(0) if AMP else 'cpu')","metadata":{},"outputs":[],"execution_count":null},{"id":"cfe804bd-59c2-449b-bdbc-a656b5384200","cell_type":"markdown","source":"Everything adjustable lives in one place. The three `BEST` dictionaries are the output of the\nOptuna study in [section 5](#s5), pasted back in so that reproducing the submission does not\nrequire re-running a multi-hour search.","metadata":{}},{"id":"115e762d-bdbc-410b-8b36-3964318f95d2","cell_type":"code","source":"class CFG:\n    # --- paths ---------------------------------------------------------------\n    input_dir = Path('/kaggle/input/dog-breed-identification')\n    work_dir  = Path('/kaggle/working')\n    cache_dir = Path('/kaggle/working/features')\n\n    # --- data ----------------------------------------------------------------\n    seed        = 1234\n    image_size  = 384\n    num_classes = 120\n    val_frac    = 0.10\n\n    # --- augmented copies of the training set to extract ----------------------\n    st_train_aug   = 4     # branch A\n    st_tta         = 4     # branch A, augmented views of the *test* set\n    vit_train_aug  = 2     # branch B\n    lora_train_aug = 4     # branch C\n\n    # --- MC-Dropout -----------------------------------------------------------\n    mc_passes = 80         # stochastic passes to average\n    mc_chunk  = 40         # how many of those get batched into one matmul\n    mc_rows   = 2048       # rows of the feature matrix held on the GPU at once\n\n    # --- LoRA fine-tuning -----------------------------------------------------\n    lora_r, lora_alpha, lora_dropout = 4, 8, 0.5\n    lora_lr, lora_wd                 = 2e-4, 0.05\n    lora_epochs, lora_patience       = 50, 10\n    lora_warmup, lora_batch          = 2, 32\n    lora_clip                        = 1.0\n\n    # --- classifier heads ------------------------------------------------------\n    seeds       = [1234, 2345, 3456, 4567, 5678]\n    head_epochs = 50\n\n    # --- switches ---------------------------------------------------------------\n    run_optuna = False     # True re-runs the search in section 5 (hours)\n    n_trials   = 40        # the published params came from 300 trials per branch\n    optuna_mc  = 40\n    keep_cache = False     # True keeps ~1.8 GB of features in the notebook output\n                           # so that a re-run resumes instead of re-extracting\n\n    num_workers = 0        # forked DataLoader workers crash on Kaggle (shared-memory limit).\n                           # In-process loading is stable, but decode no longer overlaps with\n                           # the GPU, so extraction gets slower -- cut st_tta/st_train_aug or\n                           # drop irv2 (see the intro) if the session runs long.\n\nCFG.cache_dir.mkdir(parents=True, exist_ok=True)\n\n# Head hyperparameters found by Optuna (300 trials per branch, 5 seeds per trial).\nBEST = {\n    'stack': dict(dropout=0.8270, lr=1.3585e-3, sched_factor=0.5704, sched_patience=4, es_patience=4),\n    'vit'  : dict(dropout=0.6882, lr=2.1157e-3, sched_factor=0.5264, sched_patience=6, es_patience=12),\n    'lora' : dict(dropout=0.8502, lr=1.1412e-3, sched_factor=0.5813, sched_patience=6, es_patience=3),\n}\n\n# Whether each branch averages stochastic (MC-Dropout) passes or takes one clean\n# deterministic pass. Section 10 explains why this matters less than it looks.\nUSE_MC = {'stack': True, 'vit': False, 'lora': False}\n\n\ndef seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n\nseed_everything(CFG.seed)","metadata":{},"outputs":[],"execution_count":null},{"id":"721155c9","cell_type":"code","source":"# Attached data mounts either at /kaggle/input/<slug>/ or at\n# /kaggle/input/competitions/<slug>/ depending on how it was added, so find the folder\n# that actually holds the competition instead of trusting a hard-coded path.\ndef is_competition_dir(p):\n    return (p / 'labels.csv').exists() and (p / 'train').is_dir()\n\nINPUT_ROOT = Path('/kaggle/input')\ncandidates = ([INPUT_ROOT, *INPUT_ROOT.glob('*'), *INPUT_ROOT.glob('*/*')]\n              if INPUT_ROOT.exists() else [])\ncandidates = [p for p in candidates if p.is_dir()]\n\nif not is_competition_dir(CFG.input_dir):\n    match = next((p for p in candidates if is_competition_dir(p)), None)\n    if match is not None:\n        print(f'competition data found at {match}')\n        CFG.input_dir = match\n\nif not is_competition_dir(CFG.input_dir):\n    raise FileNotFoundError(\n        'The Dog Breed Identification data is not attached to this notebook.\\n'\n        '  In the editor: right-hand panel -> Input -> \"+ Add Input\" -> Competitions tab\\n'\n        '  -> search \"Dog Breed Identification\" -> Add, then restart the session so the\\n'\n        '  mount appears.  If it does not show up, accept the rules once at\\n'\n        '  kaggle.com/c/dog-breed-identification/rules.\\n'\n        f'  Searched: {[str(p) for p in candidates] or \"/kaggle/input does not exist\"}')\n\nTRAIN_DIR, TEST_DIR = CFG.input_dir / 'train', CFG.input_dir / 'test'\nfor p in [TRAIN_DIR, TEST_DIR, CFG.input_dir / 'labels.csv', CFG.input_dir / 'sample_submission.csv']:\n    print(f'{\"found  \" if p.exists() else \"MISSING\"}  {p}')","metadata":{},"outputs":[],"execution_count":null},{"id":"826591ac-4d50-4ff6-9a66-da450337412e","cell_type":"markdown","source":"<a id=\"s2\"></a>\n## 2. The data\n\n`labels.csv` maps an image id to a breed; the images sit flat in `train/` and `test/`. Many\nnotebooks copy all 10,222 files into 120 per-breed subdirectories so `ImageFolder` can be used.\nThat costs ~350 MB of disk and a couple of minutes for no benefit — a six-line `Dataset` reads\nthe ids directly.\n\nOne detail that matters for reproducibility: the split is taken over rows sorted by\n`(breed, id)`, which is the order `ImageFolder` would have produced. The same `random_state`\ntherefore gives the same 9,199 / 1,023 split no matter how `labels.csv` happens to be ordered.","metadata":{}},{"id":"7c6d9ae8-490e-4159-b49e-e2ed6ad7d343","cell_type":"code","source":"labels_df    = pd.read_csv(CFG.input_dir / 'labels.csv')\nCLASSES      = sorted(labels_df.breed.unique())\nCLASS_TO_IDX = {b: i for i, b in enumerate(CLASSES)}\n\n# The submission is the test ids plus one column per breed in alphabetical order, which\n# is exactly `CLASSES` -- so it can be rebuilt if the sample file is not in the mount.\nsub_path = CFG.input_dir / 'sample_submission.csv'\nif sub_path.exists():\n    sample_sub = pd.read_csv(sub_path)\n    assert CLASSES == list(sample_sub.columns[1:]), 'class order must match submission columns'\nelse:\n    print('sample_submission.csv not in the mount - rebuilding it from the test folder')\n    sample_sub = pd.DataFrame({'id': sorted(p.stem for p in TEST_DIR.glob('*.jpg'))})\n    sample_sub[CLASSES] = 0.0\n\nlabels_df = labels_df.sort_values(['breed', 'id']).reset_index(drop=True)\nlabels_df['target'] = labels_df.breed.map(CLASS_TO_IDX)\n\ntr_i, va_i = train_test_split(np.arange(len(labels_df)), test_size=CFG.val_frac,\n                              random_state=CFG.seed, stratify=labels_df.target.values)\n\ntrain_df = labels_df.iloc[np.sort(tr_i)].reset_index(drop=True)\nvalid_df = labels_df.iloc[np.sort(va_i)].reset_index(drop=True)\nvalid_labels = valid_df.target.values\n\n# Test rows follow sample_submission's order, so nothing has to be re-sorted at the end.\ntest_df = pd.DataFrame({'id': sample_sub['id']})\n\nprint(f'train {len(train_df)}   valid {len(valid_df)}   test {len(test_df)}   classes {len(CLASSES)}')","metadata":{},"outputs":[],"execution_count":null},{"id":"c349349c-4366-4acd-a546-5e3cb2f3b95d","cell_type":"code","source":"sizes  = np.array([Image.open(TRAIN_DIR / f'{i}.jpg').size for i in labels_df.id])\nw, h   = sizes[:, 0], sizes[:, 1]\ncounts = labels_df.breed.value_counts()\n\nprint(f'width   min {w.min():5d}   max {w.max():5d}   mean {w.mean():.0f}')\nprint(f'height  min {h.min():5d}   max {h.max():5d}   mean {h.mean():.0f}')\nprint(f'images per breed   min {counts.min()}   max {counts.max()}')\n\nfig, ax = plt.subplots(1, 3, figsize=(15, 3.4))\nax[0].hist(w, bins=40, color='steelblue', edgecolor='white'); ax[0].set_title('image width')\nax[1].hist(h, bins=40, color='salmon',    edgecolor='white'); ax[1].set_title('image height')\nax[2].hist(counts.values, bins=25, color='seagreen', edgecolor='white'); ax[2].set_title('images per breed')\nfor a in ax:\n    a.grid(alpha=.25)\nplt.tight_layout(); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"14fde2ed-53b9-4cd4-80ab-00dab3dc9694","cell_type":"markdown","source":"The average image is roughly 440 × 390, so 384² is close to native resolution — the backbones are\nnot being fed upsampled mush. Class balance is mild (66 to 126 images per breed), mild enough\nthat a stratified split and plain cross-entropy are enough; no resampling or class weighting is\nused anywhere below.","metadata":{}},{"id":"58f8bfef-1a09-4d06-aa3f-a4fd5ddb8fd7","cell_type":"code","source":"fig, axes = plt.subplots(2, 6, figsize=(15, 5.2))\nfor ax, breed in zip(axes.flat, CLASSES[:12]):\n    row = labels_df[labels_df.breed == breed].iloc[0]\n    ax.imshow(Image.open(TRAIN_DIR / f'{row.id}.jpg'))\n    ax.set_title(breed.replace('_', ' '), fontsize=8)\n    ax.axis('off')\nplt.tight_layout(); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"c56b7d24-d9a7-44d7-aae6-06d3a2660d9e","cell_type":"markdown","source":"<a id=\"s3\"></a>\n## 3. Feature extraction\n\n### Why frozen features\n\nWith 85 images per class, full fine-tuning has to be fought rather than used. A frozen backbone\nplus a linear head has almost no capacity to overfit, and — more usefully — it makes the search\nbudget affordable. Once the features are on disk, training one head takes seconds, so 300 Optuna\ntrials × 5 seeds is a coffee break rather than a week. That trade is the entire idea here:\nspend the GPU once on features, then buy accuracy with search and averaging instead of with\ngradient steps.\n\nThe three branches are deliberately not independent — branch B is the *same* backbone as one\nthird of branch A. It earns its place in the blend anyway because it sees a different eval\ntransform (resize-then-center-crop instead of a plain squashing resize) and therefore makes\ndifferent mistakes. The weight search settles on 0.13 for it, which is about right for a branch\nthat is mostly redundant but not entirely.\n\nAll three backbones run at 384², including EfficientNet-B3 (native 300²) and\nInception-ResNet-v2 (native 299²). Running them above their training resolution is fine for pure\nfeature extraction, and a single image size means one decode-and-resize pipeline serves all of\nthem.","metadata":{}},{"id":"589af888-8845-4e66-9df3-86bb79677ad6","cell_type":"code","source":"class DogDataset(Dataset):\n    \"\"\"Reads jpgs straight out of the competition folder, keyed by id.\"\"\"\n\n    def __init__(self, df, folder, transform):\n        self.ids       = df['id'].tolist()\n        self.targets   = df['target'].tolist() if 'target' in df.columns else None\n        self.folder    = Path(folder)\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.ids)\n\n    def __getitem__(self, i):\n        img = Image.open(self.folder / f'{self.ids[i]}.jpg').convert('RGB')\n        y   = self.targets[i] if self.targets is not None else -1\n        return self.transform(img), y\n\n\ndef build_transforms(size, mean, std, center_crop=False):\n    \"\"\"`train_tf` is the augmented view (also reused for test-time augmentation);\n    `eval_tf` is the clean one.\"\"\"\n    train_tf = transforms.Compose([\n        transforms.Resize((size, size)),\n        transforms.RandomHorizontalFlip(),\n        transforms.RandomAffine(degrees=18, scale=(0.9, 1.1)),\n        transforms.ToTensor(),\n        transforms.Normalize(mean, std),\n    ])\n    eval_steps = ([transforms.Resize(size + 32), transforms.CenterCrop(size)] if center_crop\n                  else [transforms.Resize((size, size))])\n    eval_tf = transforms.Compose(eval_steps + [\n        transforms.ToTensor(),\n        transforms.Normalize(mean, std),\n    ])\n    return train_tf, eval_tf\n\n\n@torch.no_grad()\ndef extract(model, df, folder, transform, batch_size, desc=''):\n    loader = DataLoader(DogDataset(df, folder, transform), batch_size=batch_size,\n                        shuffle=False, num_workers=CFG.num_workers, pin_memory=True)\n    out = []\n    for x, _ in tqdm(loader, desc=desc, leave=False):\n        with torch.amp.autocast('cuda', enabled=AMP):\n            out.append(model(x.to(DEVICE, non_blocking=True)).float().cpu().numpy())\n    return np.concatenate(out)\n\n\ndef cached(name, fn):\n    \"\"\"Run `fn` once and memoise the array on disk, so a restarted session skips\n    whatever already finished.\"\"\"\n    path = CFG.cache_dir / f'{name}.npy'\n    if path.exists():\n        return np.load(path)\n    arr = fn()\n    np.save(path, arr)\n    return arr\n\n\ndef build_backbone(name):\n    model = timm.create_model(name, pretrained=True, num_classes=0).to(DEVICE).eval()\n    dc    = timm.data.resolve_data_config(pretrained_cfg=model.pretrained_cfg)\n    return model, dc['mean'], dc['std']\n\n\ndef free():\n    torch.cuda.empty_cache()\n    gc.collect()","metadata":{},"outputs":[],"execution_count":null},{"id":"696c6fc1-2d54-467c-89a0-71b026ad4152","cell_type":"markdown","source":"### 3.1 Branch A — three backbones, concatenated\n\nA ViT, a ConvNet and an Inception hybrid disagree in useful ways, and concatenating their\npenultimate vectors lets one linear layer decide how to weigh them per class. That is cheaper\nand better behaved than training three classifiers and averaging their probabilities.\n\nThe training set is extracted **four times** through the augmentation pipeline, giving the head\n36,796 rows instead of 9,199. Since the backbone is frozen, this is not really augmentation in\nthe usual sense — it is a fixed, pre-computed enlargement of the training distribution the linear\nlayer sees. The test set gets the same treatment: one clean view plus four augmented ones, all\naveraged at inference.\n\nBatch sizes below are sized for a 16 GB card. Halve them if you hit OOM.","metadata":{}},{"id":"fa222ab7-4636-467e-888c-621b04d40e9a","cell_type":"code","source":"BACKBONES = {\n    'vitL' : dict(name='vit_large_patch32_384',       dim=1024, bs=128),\n    'effB3': dict(name='efficientnet_b3.ra2_in1k',    dim=1536, bs=192),\n    'irv2' : dict(name='inception_resnet_v2.tf_in1k', dim=1536, bs=128),\n}\nST_DIM = sum(b['dim'] for b in BACKBONES.values())\n\nparts = {k: [] for k in ('train', 'valid', 'test')}\ntta_parts = [[] for _ in range(CFG.st_tta)]\n\nfor key, bb in BACKBONES.items():\n    names = ([f'{key}_train', f'{key}_valid', f'{key}_test']\n             + [f'{key}_tta{t}' for t in range(CFG.st_tta)])\n\n    if all((CFG.cache_dir / f'{n}.npy').exists() for n in names):\n        print(f'[{key}] cached')\n        model = train_tf = eval_tf = None\n    else:\n        print(f'[{key}] {bb[\"name\"]}')\n        model, mean, std  = build_backbone(bb['name'])\n        train_tf, eval_tf = build_transforms(CFG.image_size, mean, std)\n\n    parts['train'].append(cached(f'{key}_train', lambda: np.concatenate(\n        [extract(model, train_df, TRAIN_DIR, train_tf, bb['bs'], f'{key} train {a + 1}')\n         for a in range(CFG.st_train_aug)])))\n    parts['valid'].append(cached(f'{key}_valid',\n        lambda: extract(model, valid_df, TRAIN_DIR, eval_tf, bb['bs'], f'{key} valid')))\n    parts['test'].append(cached(f'{key}_test',\n        lambda: extract(model, test_df, TEST_DIR, eval_tf, bb['bs'], f'{key} test')))\n    for t in range(CFG.st_tta):\n        tta_parts[t].append(cached(f'{key}_tta{t}',\n            lambda t=t: extract(model, test_df, TEST_DIR, train_tf, bb['bs'], f'{key} tta {t + 1}')))\n\n    del model\n    free()\n\nX_st_train = np.concatenate(parts['train'], axis=1)\nX_st_valid = np.concatenate(parts['valid'], axis=1)\nX_st_test  = np.concatenate(parts['test'],  axis=1)\nX_st_tta   = [np.concatenate(p, axis=1) for p in tta_parts]\ny_st_train = np.tile(train_df.target.values, CFG.st_train_aug)\n\ndel parts, tta_parts\ngc.collect()\n\nprint(f'branch A   train {X_st_train.shape}   valid {X_st_valid.shape}   '\n      f'test {X_st_test.shape} (+{CFG.st_tta} TTA views)')","metadata":{},"outputs":[],"execution_count":null},{"id":"93581e8f-a936-4507-828f-848e567fe224","cell_type":"markdown","source":"### 3.2 Branch B — ViT-L on its own, center-cropped\n\nSame weights as one third of branch A, but evaluated with resize-to-416 then center-crop-384\nrather than a squashing resize to 384×384. Aspect ratio is preserved and the border is discarded,\nwhich changes the errors enough to be worth blending in.","metadata":{}},{"id":"6a79c9dd-f739-4543-8e4b-4c55fee4a789","cell_type":"code","source":"VIT_NAME, VIT_DIM, VIT_BS = 'vit_large_patch32_384', 1024, 128\n\nif all((CFG.cache_dir / f'{n}.npy').exists() for n in ('vit_train', 'vit_valid', 'vit_test')):\n    print('branch B cached')\n    model = train_tf = eval_tf = None\nelse:\n    model, mean, std  = build_backbone(VIT_NAME)\n    train_tf, eval_tf = build_transforms(CFG.image_size, mean, std, center_crop=True)\n\nX_vit_train = cached('vit_train', lambda: np.concatenate(\n    [extract(model, train_df, TRAIN_DIR, train_tf, VIT_BS, f'vit train {a + 1}')\n     for a in range(CFG.vit_train_aug)]))\nX_vit_valid = cached('vit_valid', lambda: extract(model, valid_df, TRAIN_DIR, eval_tf, VIT_BS, 'vit valid'))\nX_vit_test  = cached('vit_test',  lambda: extract(model, test_df,  TEST_DIR,  eval_tf, VIT_BS, 'vit test'))\ny_vit_train = np.tile(train_df.target.values, CFG.vit_train_aug)\n\ndel model\nfree()\n\nprint(f'branch B   train {X_vit_train.shape}   valid {X_vit_valid.shape}   test {X_vit_test.shape}')","metadata":{},"outputs":[],"execution_count":null},{"id":"090fc461-76f6-4644-bd30-d1404f2302e6","cell_type":"markdown","source":"### 3.3 Branch C — LoRA, then freeze again\n\nThis is the one branch where the backbone actually sees the labels. LoRA adapters of rank 4 are\nattached to every submodule named `qkv` or `proj` — the attention QKV and output projections in\nall 24 blocks, plus the patch-embedding convolution — and the classifier head is unfrozen. That\nis **729,208 trainable parameters out of 306M, or 0.24%**, which is small enough that 9,199\nimages can move it without destroying the pretrained representation.\n\nThen comes the part that is easy to miss: the fine-tuned classifier is not the product. The\nadapters are merged into the base weights, the head is stripped off, and the result is used as\n*another frozen feature extractor*. The adapter's job is to pull the representation toward\ndog-specific structure; the classifier attached to it during that process is scaffolding.\n\nOnly the LoRA and head tensors are checkpointed (about 3 MB), not the full merged model, so a\nrestarted session can skip the expensive part.","metadata":{}},{"id":"09f2284c-7eaf-4b08-abd4-1625c3de835e","cell_type":"code","source":"seed_everything(CFG.seed)\n\nlora_base = timm.create_model(VIT_NAME, pretrained=True, num_classes=CFG.num_classes).to(DEVICE)\ndc_lora   = timm.data.resolve_data_config(pretrained_cfg=lora_base.pretrained_cfg)\nlora_train_tf, lora_eval_tf = build_transforms(\n    CFG.image_size, dc_lora['mean'], dc_lora['std'], center_crop=True)\n\nlora_model = get_peft_model(lora_base, LoraConfig(\n    r=CFG.lora_r, lora_alpha=CFG.lora_alpha,\n    target_modules=['qkv', 'proj'],\n    lora_dropout=CFG.lora_dropout, bias='none',\n))\nfor n, p in lora_model.named_parameters():\n    if 'head' in n or 'classifier' in n:\n        p.requires_grad = True\n\nlora_model.print_trainable_parameters()\n\n\ndef adapter_state(model):\n    \"\"\"Just the tensors that actually changed — the frozen base comes from timm.\"\"\"\n    return {k: v.detach().cpu().clone()\n            for k, v in model.state_dict().items() if 'lora_' in k or 'head' in k}","metadata":{},"outputs":[],"execution_count":null},{"id":"e36636c7-ad15-445c-955d-88d9be0cf385","cell_type":"code","source":"LORA_CKPT = CFG.cache_dir / 'lora_adapter.pt'\nLORA_HIST = CFG.cache_dir / 'lora_history.json'\n\nif LORA_CKPT.exists():\n    lora_model.load_state_dict(torch.load(LORA_CKPT, map_location=DEVICE), strict=False)\n    lora_hist = json.loads(LORA_HIST.read_text()) if LORA_HIST.exists() else None\n    print('LoRA checkpoint restored, skipping fine-tuning')\nelse:\n    tr_loader = DataLoader(DogDataset(train_df, TRAIN_DIR, lora_train_tf),\n                           batch_size=CFG.lora_batch, shuffle=True,\n                           num_workers=CFG.num_workers, pin_memory=True)\n    va_loader = DataLoader(DogDataset(valid_df, TRAIN_DIR, lora_eval_tf),\n                           batch_size=CFG.lora_batch * 2, shuffle=False,\n                           num_workers=CFG.num_workers, pin_memory=True)\n\n    opt    = torch.optim.AdamW([p for p in lora_model.parameters() if p.requires_grad],\n                               lr=CFG.lora_lr, weight_decay=CFG.lora_wd)\n    warm   = CFG.lora_warmup * len(tr_loader)\n    total  = CFG.lora_epochs * len(tr_loader)\n    sched  = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: (\n        s / max(warm, 1) if s < warm\n        else 0.5 * (1 + np.cos(np.pi * (s - warm) / max(total - warm, 1)))))\n    crit   = nn.CrossEntropyLoss()\n    scaler = torch.amp.GradScaler('cuda', enabled=AMP)\n\n    best, best_state, bad = float('inf'), None, 0\n    lora_hist = {'train': [], 'val': [], 'acc': []}\n\n    for epoch in range(CFG.lora_epochs):\n        lora_model.train()\n        run = n = 0\n        for x, y in tqdm(tr_loader, desc=f'lora epoch {epoch + 1}', leave=False):\n            x, y = x.to(DEVICE, non_blocking=True), y.to(DEVICE, non_blocking=True)\n            opt.zero_grad()\n            with torch.amp.autocast('cuda', enabled=AMP):\n                loss = crit(lora_model(x), y)\n            scaler.scale(loss).backward()\n            scaler.unscale_(opt)\n            torch.nn.utils.clip_grad_norm_(lora_model.parameters(), CFG.lora_clip)\n            scaler.step(opt); scaler.update(); sched.step()\n            run += loss.item() * len(y); n += len(y)\n        train_loss = run / n\n\n        lora_model.eval()\n        run = correct = n = 0\n        with torch.no_grad():\n            for x, y in va_loader:\n                x, y = x.to(DEVICE, non_blocking=True), y.to(DEVICE, non_blocking=True)\n                with torch.amp.autocast('cuda', enabled=AMP):\n                    out = lora_model(x)\n                    run += crit(out, y).item() * len(y)\n                correct += (out.argmax(1) == y).sum().item(); n += len(y)\n        val_loss, val_acc = run / n, correct / n\n\n        lora_hist['train'].append(train_loss)\n        lora_hist['val'].append(val_loss)\n        lora_hist['acc'].append(val_acc)\n\n        flag = ''\n        if val_loss < best:\n            best, best_state, bad, flag = val_loss, adapter_state(lora_model), 0, '  *'\n        else:\n            bad += 1\n        print(f'  epoch {epoch + 1:2d}   train {train_loss:.4f}   val {val_loss:.4f}   '\n              f'acc {val_acc:.4f}   lr {opt.param_groups[0][\"lr\"]:.2e}{flag}')\n        if bad >= CFG.lora_patience:\n            print(f'  early stop (patience {CFG.lora_patience})')\n            break\n\n    lora_model.load_state_dict(best_state, strict=False)\n    torch.save(best_state, LORA_CKPT)\n    LORA_HIST.write_text(json.dumps(lora_hist))\n    print(f'\\nbest val loss {best:.4f} at epoch {int(np.argmin(lora_hist[\"val\"])) + 1}')","metadata":{},"outputs":[],"execution_count":null},{"id":"7a18ba08-e1db-40c0-8916-8c3ad110d3d4","cell_type":"code","source":"if lora_hist:\n    best_ep = int(np.argmin(lora_hist['val']))\n    fig, ax = plt.subplots(1, 2, figsize=(12, 3.6))\n    ax[0].plot(lora_hist['train'], label='train')\n    ax[0].plot(lora_hist['val'],   label='validation')\n    ax[0].set_title('LoRA fine-tuning loss'); ax[0].legend()\n    ax[1].plot(lora_hist['acc'], color='seagreen')\n    ax[1].set_title('LoRA validation accuracy')\n    for a in ax:\n        a.axvline(best_ep, color='grey', ls='--', alpha=.6)\n        a.set_xlabel('epoch'); a.grid(alpha=.25)\n    plt.tight_layout(); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"547f0747-4c3d-44c6-a71d-2419d3ce123d","cell_type":"markdown","source":"The adapter converges quickly and stops: **val loss 0.1414 / accuracy 95.9%** at epoch 4, with\ntraining loss under 0.01 ten epochs later. That gap is the expected shape at 85 images per\nclass, and it is why the branch is used the way it is — four epochs is enough to specialise the\nrepresentation, and going further would only specialise it to these 9,199 photographs.\n\nThe payoff shows up later: a head trained on these features takes **28% of the final blend\nweight**, the second-largest share of the three branches.","metadata":{}},{"id":"cafdba9c-2a93-47a9-83ab-f218996275ff","cell_type":"code","source":"lora_backbone = lora_model.merge_and_unload()\nlora_backbone.reset_classifier(0)\nlora_backbone.eval()\n\nX_lora_train = cached('lora_train', lambda: np.concatenate(\n    [extract(lora_backbone, train_df, TRAIN_DIR, lora_train_tf, VIT_BS, f'lora train {a + 1}')\n     for a in range(CFG.lora_train_aug)]))\nX_lora_valid = cached('lora_valid',\n    lambda: extract(lora_backbone, valid_df, TRAIN_DIR, lora_eval_tf, VIT_BS, 'lora valid'))\nX_lora_test  = cached('lora_test',\n    lambda: extract(lora_backbone, test_df,  TEST_DIR,  lora_eval_tf, VIT_BS, 'lora test'))\ny_lora_train = np.tile(train_df.target.values, CFG.lora_train_aug)\n\ndel lora_model, lora_base, lora_backbone\nfree()\n\nprint(f'branch C   train {X_lora_train.shape}   valid {X_lora_valid.shape}   test {X_lora_test.shape}')","metadata":{},"outputs":[],"execution_count":null},{"id":"be02178b-367d-4dea-a08d-df8d8e15914e","cell_type":"markdown","source":"<a id=\"s4\"></a>\n## 4. The classifier head\n\nDropout, then one linear layer. That is the whole model.\n\nThe dropout is subclassed so it stays active in `eval()`, which allows sampling the head at\ninference time (MC-Dropout). The dropout rates the search settles on are high — 0.83, 0.69, 0.85\n— which is what you would expect when a 4096-dimensional input meets 9,199 distinct source\nimages: the regulariser is doing most of the work of keeping the layer honest.","metadata":{}},{"id":"7392e4d3-5c81-4bd6-bf6b-47bd65cf8685","cell_type":"code","source":"class MCDropout(nn.Dropout):\n    \"\"\"Dropout that stays on in eval(), so the head can be sampled.\"\"\"\n\n    def forward(self, x):\n        return F.dropout(x, self.p, training=True, inplace=self.inplace)\n\n\nclass Head(nn.Module):\n    def __init__(self, in_dim, num_classes, p=0.7):\n        super().__init__()\n        self.mc_dropout = MCDropout(p)\n        self.classifier = nn.Linear(in_dim, num_classes)\n\n    def forward(self, x):\n        return self.classifier(self.mc_dropout(x))\n\n\n@contextmanager\ndef dropout_off(model):\n    \"\"\"Temporarily make the head deterministic.\"\"\"\n    p, model.mc_dropout.p = model.mc_dropout.p, 0.0\n    try:\n        yield\n    finally:\n        model.mc_dropout.p = p","metadata":{},"outputs":[],"execution_count":null},{"id":"113dcc79-7412-471f-8f62-a68819c63306","cell_type":"code","source":"def train_head(model, Xtr, ytr, Xva, yva, lr, es_patience, sched_factor, sched_patience,\n               seed, epochs=CFG.head_epochs, verbose=True):\n    \"\"\"Adam + ReduceLROnPlateau + early stopping on validation loss.\n    Dropout is on for training and off for the validation pass.\"\"\"\n    seed_everything(seed)\n\n    tr = DataLoader(TensorDataset(torch.from_numpy(Xtr).float(), torch.from_numpy(ytr).long()),\n                    batch_size=512, shuffle=True)\n    va = DataLoader(TensorDataset(torch.from_numpy(Xva).float(), torch.from_numpy(yva).long()),\n                    batch_size=1024, shuffle=False)\n\n    crit  = nn.CrossEntropyLoss()\n    opt   = torch.optim.Adam(model.parameters(), lr=lr)\n    sched = torch.optim.lr_scheduler.ReduceLROnPlateau(\n        opt, 'min', factor=sched_factor, patience=sched_patience)\n\n    best, best_w, bad = float('inf'), None, 0\n    hist = {'train': [], 'val': [], 'acc': []}\n\n    for epoch in range(epochs):\n        model.train()\n        run = n = 0\n        for xb, yb in tr:\n            xb, yb = xb.to(DEVICE), yb.to(DEVICE)\n            opt.zero_grad()\n            loss = crit(model(xb), yb)\n            loss.backward()\n            opt.step()\n            run += loss.item() * len(yb); n += len(yb)\n        train_loss = run / n\n\n        model.eval()\n        run = correct = n = 0\n        with dropout_off(model), torch.no_grad():\n            for xb, yb in va:\n                xb, yb = xb.to(DEVICE), yb.to(DEVICE)\n                out = model(xb)\n                run += crit(out, yb).item() * len(yb)\n                correct += (out.argmax(1) == yb).sum().item(); n += len(yb)\n        val_loss, val_acc = run / n, correct / n\n        sched.step(val_loss)\n\n        hist['train'].append(train_loss)\n        hist['val'].append(val_loss)\n        hist['acc'].append(val_acc)\n\n        if val_loss < best:\n            best, best_w, bad = val_loss, copy.deepcopy(model.state_dict()), 0\n            if verbose:\n                print(f'  epoch {epoch + 1:2d}   train {train_loss:.4f}   '\n                      f'val {val_loss:.4f}   acc {val_acc:.4f}   *')\n        else:\n            bad += 1\n            if bad >= es_patience:\n                if verbose:\n                    print(f'  early stop at epoch {epoch + 1}, best {best:.4f}')\n                break\n\n    model.load_state_dict(best_w)\n    return hist","metadata":{},"outputs":[],"execution_count":null},{"id":"57f609e3-192a-4661-b688-f6ad57ec92e5","cell_type":"markdown","source":"### Inference, calibration and a note on speed\n\nThe original version of this pipeline searched temperature with `sklearn.log_loss` inside a\nPython loop, and the three-way blend search called it roughly 1.4 million times — around forty\nminutes of pure overhead. `sweep_temperatures` below evaluates the entire temperature grid in a\nsingle batched `log_softmax` on the GPU, which takes it down to about a minute.","metadata":{}},{"id":"0957aea7-ee48-4311-82d4-162579deb9c0","cell_type":"code","source":"T_GRID = torch.arange(0.30, 3.0001, 0.01)\n\n\n@torch.no_grad()\ndef mc_logits(model, X, passes=None, chunk=None, rows=None):\n    \"\"\"Average `passes` stochastic forward passes. The head is one linear layer, so the\n    batch can simply be tiled `chunk` times to get `chunk` samples out of one matmul.\"\"\"\n    passes = passes or CFG.mc_passes\n    chunk  = chunk  or CFG.mc_chunk\n    rows   = rows   or CFG.mc_rows\n    model.eval()\n    out = np.empty((len(X), CFG.num_classes), dtype=np.float32)\n    for s in range(0, len(X), rows):\n        e = min(s + rows, len(X))\n        x = torch.from_numpy(X[s:e]).float().to(DEVICE)\n        acc, done = torch.zeros(e - s, CFG.num_classes, device=DEVICE), 0\n        while done < passes:\n            k = min(chunk, passes - done)\n            acc += model(x.repeat(k, 1)).view(k, e - s, -1).sum(0)\n            done += k\n        out[s:e] = (acc / passes).cpu().numpy()\n    return out\n\n\n@torch.no_grad()\ndef det_logits(model, X, rows=8192):\n    \"\"\"One clean pass, dropout disabled.\"\"\"\n    model.eval()\n    out = []\n    with dropout_off(model):\n        for s in range(0, len(X), rows):\n            out.append(model(torch.from_numpy(X[s:s + rows]).float().to(DEVICE)).cpu().numpy())\n    return np.concatenate(out)\n\n\ndef branch_logits(models, X, use_mc):\n    \"\"\"Seed ensemble: average the logits of every head trained on this branch.\"\"\"\n    fn = mc_logits if use_mc else det_logits\n    return np.mean([fn(m, X) for m in models], axis=0)\n\n\n@torch.no_grad()\ndef _sweep(z, y, temps):\n    \"\"\"Log loss of logits `z` at every temperature in `temps`, in one GPU op.\"\"\"\n    lp = torch.log_softmax(z.unsqueeze(0) / temps.view(-1, 1, 1), dim=-1)\n    return -lp[:, torch.arange(z.shape[0], device=z.device), y].mean(1)\n\n\ndef sweep_temperatures(logits, labels):\n    z    = torch.as_tensor(logits, dtype=torch.float32, device=DEVICE)\n    y    = torch.as_tensor(labels, dtype=torch.long,    device=DEVICE)\n    lls  = _sweep(z, y, T_GRID.to(DEVICE)).cpu().numpy()\n    best = int(lls.argmin())\n    return float(T_GRID[best]), float(lls[best]), lls\n\n\ndef to_probs(logits, T=1.0, eps=1e-7):\n    \"\"\"Temperature scaling, softmax, and a clip so a single confident mistake cannot\n    dominate the log loss.\"\"\"\n    z = logits / T\n    z = z - z.max(1, keepdims=True)\n    p = np.exp(z)\n    p = p / p.sum(1, keepdims=True)\n    p = np.clip(p, eps, 1 - eps)\n    return p / p.sum(1, keepdims=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"702870e7-c81f-4775-9969-68e44fa0abe0","cell_type":"markdown","source":"<a id=\"s5\"></a>\n## 5. Hyperparameter search\n\nEach trial trains **five heads** (one per seed), seed-averages their validation logits, fits a\ntemperature, and reports the resulting log loss. Scoring the ensemble rather than a single head\nmatters: a configuration that produces five diverse heads beats one that produces five good but\nidentical heads, and a single-model objective cannot see that difference.\n\nThis is off by default — `BEST` in section 1 already holds the result of 300 trials per branch.\nSet `CFG.run_optuna = True` to re-run it.","metadata":{}},{"id":"d55012f8-397b-4f15-9b4f-d9e345327a8f","cell_type":"code","source":"if CFG.run_optuna:\n    import optuna\n    optuna.logging.set_verbosity(optuna.logging.WARNING)\n\n    def make_objective(Xtr, ytr, Xva, dim, dropout_range):\n        def objective(trial):\n            hp = dict(\n                dropout        = trial.suggest_float('dropout', *dropout_range),\n                lr             = trial.suggest_float('lr', 5e-5, 1e-2, log=True),\n                sched_factor   = trial.suggest_float('sched_factor', 0.05, 0.6),\n                sched_patience = trial.suggest_int('sched_patience', 1, 7),\n                es_patience    = trial.suggest_int('es_patience', 3, 20),\n            )\n            det, mc = [], []\n            for seed in CFG.seeds:\n                head = Head(dim, CFG.num_classes, hp['dropout']).to(DEVICE)\n                train_head(head, Xtr, ytr, Xva, valid_labels, hp['lr'], hp['es_patience'],\n                           hp['sched_factor'], hp['sched_patience'], seed, verbose=False)\n                det.append(det_logits(head, Xva))\n                mc.append(mc_logits(head, Xva, passes=CFG.optuna_mc))\n                del head\n            free()\n\n            _, ll_det, _ = sweep_temperatures(np.mean(det, axis=0), valid_labels)\n            _, ll_mc,  _ = sweep_temperatures(np.mean(mc,  axis=0), valid_labels)\n            trial.set_user_attr('use_mc', bool(ll_mc < ll_det))\n            return min(ll_det, ll_mc)\n        return objective\n\n    SPACES = {\n        'stack': (X_st_train,   y_st_train,   X_st_valid,   ST_DIM,  (0.30, 0.95)),\n        'vit'  : (X_vit_train,  y_vit_train,  X_vit_valid,  VIT_DIM, (0.30, 0.95)),\n        'lora' : (X_lora_train, y_lora_train, X_lora_valid, VIT_DIM, (0.85, 0.97)),\n    }\n\n    studies = {}\n    for name, (Xtr, ytr, Xva, dim, drange) in SPACES.items():\n        study = optuna.create_study(direction='minimize', study_name=name)\n        study.enqueue_trial(BEST[name])                      # start from the known-good point\n        study.optimize(make_objective(Xtr, ytr, Xva, dim, drange),\n                       n_trials=CFG.n_trials, show_progress_bar=True)\n        studies[name] = study\n        BEST[name]    = study.best_params\n        USE_MC[name]  = study.best_trial.user_attrs.get('use_mc', False)\n        print(f'{name}: {study.best_value:.4f}   {study.best_params}   MC={USE_MC[name]}')\n\n    fig, axes = plt.subplots(1, 3, figsize=(16, 3.6))\n    for ax, (name, study) in zip(axes, studies.items()):\n        vals = [t.value for t in study.trials if t.value is not None]\n        ax.plot(vals, '.-', alpha=.6)\n        ax.axhline(study.best_value, color='crimson', ls='--',\n                   label=f'best {study.best_value:.4f}')\n        ax.set_title(name); ax.set_xlabel('trial'); ax.set_ylabel('val log loss')\n        ax.legend(); ax.grid(alpha=.25)\n    plt.tight_layout(); plt.show()\nelse:\n    print('Optuna skipped. Using the stored parameters:')\n    print(pd.DataFrame(BEST).T.to_string())","metadata":{},"outputs":[],"execution_count":null},{"id":"de254424-92ff-4bdb-bc90-10e7bf4450ce","cell_type":"markdown","source":"Parameter importances from the 300-trial studies, if you would rather not spend the GPU hours\nfinding this out yourself:\n\n| | learning rate | dropout | es_patience | sched_factor | sched_patience |\n|:--|--:|--:|--:|--:|--:|\n| branch A | 0.73 | 0.19 | 0.05 | 0.03 | 0.00 |\n| branch B | 0.74 | 0.21 | 0.01 | 0.01 | 0.03 |\n| branch C | 0.51 | 0.24 | 0.12 | 0.10 | 0.04 |\n\nLearning rate and dropout account for over 90% of the variance in every branch. The scheduler\nsettings are noise. If your search budget is limited, tune two things.","metadata":{}},{"id":"0935047c-5d22-4d1e-8173-73fd128207e5","cell_type":"markdown","source":"<a id=\"s6\"></a>\n## 6. Final training — 3 branches × 5 seeds\n\nFifteen heads. Each takes a few seconds, because all the expensive work already happened in\nsection 3.","metadata":{}},{"id":"6791dcb3-f8db-4a42-b3e7-1f1162c6e9a5","cell_type":"code","source":"BRANCHES = {\n    'stack': dict(Xtr=X_st_train,   ytr=y_st_train,   Xva=X_st_valid,   dim=ST_DIM),\n    'vit'  : dict(Xtr=X_vit_train,  ytr=y_vit_train,  Xva=X_vit_valid,  dim=VIT_DIM),\n    'lora' : dict(Xtr=X_lora_train, ytr=y_lora_train, Xva=X_lora_valid, dim=VIT_DIM),\n}\n\nheads, histories, rows = {}, {}, []\n\nfor name, b in BRANCHES.items():\n    hp = BEST[name]\n    heads[name], histories[name] = [], []\n    print(f'\\n{name}   dropout {hp[\"dropout\"]:.3f}   lr {hp[\"lr\"]:.2e}')\n    for seed in CFG.seeds:\n        head = Head(b['dim'], CFG.num_classes, hp['dropout']).to(DEVICE)\n        hist = train_head(head, b['Xtr'], b['ytr'], b['Xva'], valid_labels,\n                          lr=hp['lr'], es_patience=hp['es_patience'],\n                          sched_factor=hp['sched_factor'], sched_patience=hp['sched_patience'],\n                          seed=seed, verbose=False)\n        torch.save(head.state_dict(), CFG.work_dir / f'{name}_head_seed{seed}.pth')\n        heads[name].append(head)\n        histories[name].append(hist)\n\n        k = int(np.argmin(hist['val']))\n        rows.append((name, seed, k + 1, hist['val'][k], hist['acc'][k]))\n        print(f'  seed {seed}   best epoch {k + 1:2d}   val {hist[\"val\"][k]:.4f}   acc {hist[\"acc\"][k]:.4f}')\n\nseed_table = pd.DataFrame(rows, columns=['branch', 'seed', 'best_epoch', 'val_loss', 'val_acc'])","metadata":{},"outputs":[],"execution_count":null},{"id":"74d683b7-6fe6-4aba-b2ec-72a1c952f807","cell_type":"markdown","source":"Note the best epoch numbers: usually 1 to 3. A linear layer over pre-computed features converges\nalmost immediately, and everything after that is overfitting. This is also why the `es_patience`\nvalues the search picked are small.","metadata":{}},{"id":"0e9d3d60-286b-449e-b62d-7b7fac33ace9","cell_type":"code","source":"fig, axes = plt.subplots(1, 3, figsize=(16, 3.6), sharey=True)\nfor ax, (name, hs) in zip(axes, histories.items()):\n    for h in hs:\n        ax.plot(h['val'], alpha=.7)\n    ax.set_title(f'{name} — validation loss per seed')\n    ax.set_xlabel('epoch'); ax.grid(alpha=.25)\naxes[0].set_ylabel('log loss')\nplt.tight_layout(); plt.show()\n\nprint(seed_table.groupby('branch')[['val_loss', 'val_acc']].agg(['mean', 'std']).round(4).to_string())","metadata":{},"outputs":[],"execution_count":null},{"id":"256e83be-5df3-49c4-b7d3-1a3500c858cc","cell_type":"markdown","source":"<a id=\"s7\"></a>\n## 7. Calibration and blending\n\nTwo steps, in this order.\n\n**Temperature.** Log loss punishes confident mistakes much harder than it rewards confident\nhits, so a single scalar divisor on the logits is fitted on the validation set. It is one\nparameter and it cannot change a single prediction — only how strongly they are asserted.\n\n**Blend weights.** Logits are combined *before* the softmax, not probabilities after it.\nAveraging in logit space keeps the operation linear with respect to what the heads actually\nproduce; averaging probabilities pulls the result toward whichever model is least confident.\nThe weights live on the 2-simplex and are searched on a 0.01 grid, with the temperature\nre-fitted for every weight triple — 5,151 weight combinations × 271 temperatures.","metadata":{}},{"id":"3a04f937-101c-442e-a522-e70ef9103467","cell_type":"code","source":"val_logits, summary = {}, []\n\nfor name, b in BRANCHES.items():\n    z       = branch_logits(heads[name], b['Xva'], USE_MC[name])\n    T, ll, _ = sweep_temperatures(z, valid_labels)\n    acc     = (to_probs(z, T).argmax(1) == valid_labels).mean()\n    val_logits[name] = z\n    summary.append((name, 'MC-Dropout' if USE_MC[name] else 'deterministic', ll, acc, T))\n\nprint(f'{\"branch\":<8}{\"head inference\":<16}{\"log loss\":>10}{\"accuracy\":>11}{\"T\":>7}')\nprint('-' * 52)\nfor n, m, ll, a, T in summary:\n    print(f'{n:<8}{m:<16}{ll:>10.4f}{a:>10.2%}{T:>7.2f}')","metadata":{},"outputs":[],"execution_count":null},{"id":"ef09d375-3c6d-4a9c-9143-26c388f8a8f3","cell_type":"code","source":"def blend_search(logits, labels, step=0.01):\n    \"\"\"Grid over the 2-simplex of blend weights, re-fitting temperature at each point.\"\"\"\n    Z = [torch.as_tensor(z, dtype=torch.float32, device=DEVICE) for z in logits]\n    y = torch.as_tensor(labels, dtype=torch.long, device=DEVICE)\n    T = T_GRID.to(DEVICE)\n    n = int(round(1 / step))\n\n    best, curve = dict(ll=float('inf')), {}\n    for i in tqdm(range(n + 1), desc='blend grid'):\n        for j in range(n + 1 - i):\n            w  = (i / n, j / n, (n - i - j) / n)\n            ll = _sweep(w[0] * Z[0] + w[1] * Z[1] + w[2] * Z[2], y, T)\n            k  = int(ll.argmin())\n            v  = float(ll[k])\n            curve[w[2]] = min(curve.get(w[2], np.inf), v)\n            if v < best['ll']:\n                best = dict(ll=v, w=w, T=float(T[k]))\n    return best, curve\n\n\nbest, curve = blend_search([val_logits['stack'], val_logits['vit'], val_logits['lora']], valid_labels)\n\nW, T_BLEND = best['w'], best['T']\nval_blend  = W[0] * val_logits['stack'] + W[1] * val_logits['vit'] + W[2] * val_logits['lora']\nval_probs  = to_probs(val_blend, T_BLEND)\nval_pred   = val_probs.argmax(1)\nval_acc    = (val_pred == valid_labels).mean()\n\nprint(f'\\nweights   stack {W[0]:.2f}   vit {W[1]:.2f}   lora {W[2]:.2f}   temperature {T_BLEND:.2f}')\nprint(f'validation log loss {best[\"ll\"]:.4f}   accuracy {val_acc:.2%}')\nprint(f'best single branch  {min(s[2] for s in summary):.4f}')","metadata":{},"outputs":[],"execution_count":null},{"id":"2c5a6857-997b-4545-ba45-4a2bec3fb18d","cell_type":"code","source":"_, _, st_curve = sweep_temperatures(val_logits['stack'], valid_labels)\n\nfig, ax = plt.subplots(1, 2, figsize=(12, 3.6))\nxs = sorted(curve)\nax[0].plot(xs, [curve[x] for x in xs], color='rebeccapurple')\nax[0].scatter([W[2]], [best['ll']], color='crimson', zorder=5)\nax[0].set_title('best achievable log loss vs branch C weight')\nax[0].set_xlabel('weight on the LoRA branch')\n\nax[1].plot(T_GRID.numpy(), st_curve, color='steelblue')\nax[1].axvline(summary[0][4], color='crimson', ls='--', label=f'T = {summary[0][4]:.2f}')\nax[1].set_title('temperature sweep, branch A')\nax[1].set_xlabel('temperature'); ax[1].legend()\n\nfor a in ax:\n    a.set_ylabel('log loss'); a.grid(alpha=.25)\nplt.tight_layout(); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"56609695-aa2e-4b40-b66d-9438601214b8","cell_type":"markdown","source":"The blend lands at **0.59 / 0.13 / 0.28** — branch A carries it, but the LoRA branch that\n*lost* to it head-to-head still takes 28% of the weight. That is the useful lesson: a model does\nnot have to be good to be worth blending, it has to be wrong in a different direction. Blending\ntakes the score from 0.1134 to 0.1062, which is a larger gain than anything the hyperparameter\nsearch produced.\n\nThe fitted temperature is 0.95 — that is, seed-averaged logits over frozen features were already\nclose to calibrated, and this step is cheap insurance rather than a real source of gain.","metadata":{}},{"id":"1f5611f9-8e71-49c7-8b4f-4a388679c9c6","cell_type":"markdown","source":"<a id=\"s8\"></a>\n## 8. Test inference and submission\n\nBranch A averages over five views of the test set (one clean, four augmented). Branches B and C\nuse the clean view only. Then the same weights and temperature fitted on validation are applied\nunchanged.","metadata":{}},{"id":"99cb634f-ce91-4f52-9647-a37920674ad5","cell_type":"code","source":"st_test_logits = np.mean(\n    [branch_logits(heads['stack'], X, USE_MC['stack'])\n     for X in tqdm([X_st_test] + X_st_tta, desc='branch A test views')], axis=0)\n\nvit_test_logits  = branch_logits(heads['vit'],  X_vit_test,  USE_MC['vit'])\nlora_test_logits = branch_logits(heads['lora'], X_lora_test, USE_MC['lora'])\n\ntest_blend = (W[0] * st_test_logits + W[1] * vit_test_logits + W[2] * lora_test_logits)\ntest_probs = to_probs(test_blend, T_BLEND)\n\nprint('test logits', test_blend.shape)","metadata":{},"outputs":[],"execution_count":null},{"id":"3d62d074-d74e-4406-879f-5422804a7d4c","cell_type":"code","source":"sub = sample_sub.copy()\nsub.iloc[:, 1:] = test_probs[:, [CLASS_TO_IDX[b] for b in sub.columns[1:]]]\nsub.to_csv(CFG.work_dir / 'submission.csv', index=False)\n\nrow_sums = sub.iloc[:, 1:].sum(axis=1)\nprint(f'{sub.shape[0]} rows, row sums in [{row_sums.min():.6f}, {row_sums.max():.6f}]')\nsub.iloc[:5, :6]","metadata":{},"outputs":[],"execution_count":null},{"id":"6670bfc2-6fbd-49bb-80f0-e5b6d069db81","cell_type":"markdown","source":"<a id=\"s9\"></a>\n## 9. Error analysis\n\nTwenty-seven mistakes out of 1,023. At that point the confusion matrix stops being a\ndiagnostic and starts being a list of genuinely ambiguous photographs, but it is still worth\nlooking at which pairs survive.","metadata":{}},{"id":"eb246c0e-c407-4b82-92cc-c0535f7b20f9","cell_type":"code","source":"_, recall, _, support = precision_recall_fscore_support(\n    valid_labels, val_pred, average=None, zero_division=0, labels=np.arange(CFG.num_classes))\n\nper_breed = (pd.DataFrame({'breed': CLASSES, 'accuracy': recall, 'n_val': support})\n             .sort_values('accuracy'))\n\nn_wrong = int((val_pred != valid_labels).sum())\nprint(f'{n_wrong} errors out of {len(valid_labels)}  ({n_wrong / len(valid_labels):.2%})')\nprint(f'{(per_breed.accuracy == 1.0).sum()} of {CFG.num_classes} breeds classified perfectly\\n')\nprint('hardest breeds')\nprint(per_breed.head(10).to_string(index=False))","metadata":{},"outputs":[],"execution_count":null},{"id":"79c83094-052f-43c2-9004-b6d4e8e0dea1","cell_type":"code","source":"cm = confusion_matrix(valid_labels, val_pred, labels=np.arange(CFG.num_classes))\n\nfig, ax = plt.subplots(1, 2, figsize=(15, 5.6),\n                       gridspec_kw={'width_ratios': [1.15, 1]})\n\nim = ax[0].imshow(cm, cmap='Blues', vmax=cm.max())\nax[0].set_title('confusion matrix, 120 breeds')\nax[0].set_xlabel('predicted'); ax[0].set_ylabel('true')\nfig.colorbar(im, ax=ax[0], shrink=.8)\n\nworst = per_breed.head(15)\nax[1].barh(worst.breed.str.replace('_', ' '), worst.accuracy, color='salmon')\nax[1].set_xlim(0, 1); ax[1].invert_yaxis()\nax[1].set_title('15 lowest-accuracy breeds'); ax[1].grid(axis='x', alpha=.25)\n\nplt.tight_layout(); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"fb84601a-b441-4ee8-a4f2-aca5a2bca1e0","cell_type":"code","source":"off = cm.copy()\nnp.fill_diagonal(off, 0)\npairs = [(CLASSES[i], CLASSES[j], off[i, j])\n         for i, j in zip(*np.unravel_index(np.argsort(off, axis=None)[::-1][:10], off.shape))\n         if off[i, j] > 0]\n\nprint('most frequent confusions')\nfor true_b, pred_b, count in pairs:\n    print(f'  {true_b:>22s}  ->  {pred_b:<22s} {count}')","metadata":{},"outputs":[],"execution_count":null},{"id":"6cbced2a-fe2f-421c-b31b-140a92c9a412","cell_type":"markdown","source":"The recurring pairs are the ones a person would also hesitate over: collie against border\ncollie, Siberian husky against Eskimo dog, Afghan hound against borzoi. Several of\nthese are not cleanly separable from a single photograph at all — the breeds differ mainly in\nsize and build, which a tight crop removes. That is roughly the point at which more modelling\nstops paying.","metadata":{}},{"id":"49705c09-0c6d-4c8b-bf24-43909cfbc109","cell_type":"code","source":"wrong = np.where(val_pred != valid_labels)[0]\norder = wrong[np.argsort(-val_probs[wrong].max(1))][:8]\n\nfig, axes = plt.subplots(2, 4, figsize=(14, 7.4))\nfor ax, i in zip(axes.flat, order):\n    ax.imshow(Image.open(TRAIN_DIR / f'{valid_df.id.iloc[i]}.jpg'))\n    ax.set_title(f'true: {CLASSES[valid_labels[i]].replace(\"_\", \" \")}\\n'\n                 f'predicted: {CLASSES[val_pred[i]].replace(\"_\", \" \")} '\n                 f'({val_probs[i].max():.0%})', fontsize=8)\n    ax.axis('off')\nfor ax in axes.flat[len(order):]:\n    ax.axis('off')\nfig.suptitle('the eight mistakes the model was most confident about', y=1.0)\nplt.tight_layout(); plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"7598e878-7e27-45c4-b9a2-bd5ef311857a","cell_type":"markdown","source":"<a id=\"s10\"></a>\n## 10. Notes from building this\n\n**Frozen features are the right shape for this dataset.** Three backbones and one linear layer\nreach 0.1134 before any blending, while the LoRA-adapted ViT with its own classifier reaches\n0.1414. At 85 images per class that ordering is what you should expect, and it is the reason the\nadapter is deployed as a feature extractor rather than as a model. Capacity spent on the backbone\ndoes not come back here; capacity spent on diversity does.\n\n**Deterministic and MC-Dropout inference were compared head to head, per branch.** The head is\n`Linear(Dropout(x))` and inverted dropout is unbiased, so the expected logit\n`E[W(m ⊙ x)/(1-p)]` is exactly `Wx` — averaging stochastic passes converges back to the\ndeterministic answer rather than adding information. The measurements agree with the algebra:\n0.1267 against 0.1269 on branch C, 0.1184 against 0.1174 for branch B during the search. Both\npaths are kept in the code so the comparison is reproducible, and there is a free speedup in it —\nset every entry of `USE_MC` to `False` and most of the inference time disappears for a change in\nthe fourth decimal. (Re-drawing the masks alone moved the final blend between 0.1046 and 0.1049\non the Colab run — the same statement measured a different way, and an order of magnitude\nsmaller than the gap to the 0.1062 printed above, which is environment rather than masks.)\n\n**Diversity paid more than tuning.** The best single branch finishes at 0.1134. Adding two\nbranches that are individually *worse* — 0.1212 and 0.1242 — and searching three blend weights\ntakes it to 0.1062. That 0.0072 came from combining models, not from making any one of them\nbetter, and it is the single largest improvement anywhere in this notebook.\n\n**Learning rate is 70% of the hyperparameter story.** Optuna's importance scores put `lr` at\n0.73 / 0.74 / 0.51 and dropout second. The scheduler parameters had importances near zero and\ncould have been fixed constants.\n\n**A 1,023-image holdout resolves about half a percent.** One image is worth 0.1% of accuracy and\nthe finished model makes 27 mistakes in total, so two variants closer than roughly half a point\nare indistinguishable on this split. That is worth knowing before trusting a small delta, and it\nis why the blend weights are fitted against a seed-averaged, temperature-calibrated objective\nrather than against a single validation number.\n\n**The validation number did not rank the two runs.** This same code on Colab reached 0.1046\non validation and 0.11870 on the leaderboard. The Kaggle run you are reading reached 0.1062\non validation — worse — and 0.11766 on the leaderboard — better. Validation preferred the\nfirst, the leaderboard preferred the second. With 1,023 images, and with the temperature and\nthe blend weights fitted on those same 1,023, a 0.0016 validation difference carries no\ninformation about which run generalises better. That is the caveat above, measured.\n\n### Why the frozen-feature approach fits this competition so well\n\nThe 120 breeds here are 120 of the 1,000 ImageNet-1k classes, and the images come from Stanford\nDogs, which was built from ImageNet. The pretrained backbones have therefore already learned this\nexact distribution. That is precisely what makes freezing them and training a linear probe the\nefficient move, and it is why the whole leaderboard sits at low log loss: the representations\nalready encode these classes, so the work worth doing is combining and calibrating them rather\nthan relearning them.\n\nIt also means the methods carry further than the number does. Seed ensembling, logit-space\nblending and temperature scaling are worth just as much on a dataset your backbone has never\nseen — the score itself is a property of this particular dataset.\n\n### Where I would go next\n\nHigher-resolution backbones (ConvNeXt or EVA-02 at 448²) would probably help branch A more than\nanything else in this notebook. Proper k-fold instead of a single 10% holdout would make the\nblend weights trustworthy — with 1,023 validation images they are fitted on very little, and\nsome of that 0.59 / 0.13 / 0.28 is certainly noise.\n\n---\n\n### References\n\n- Hu et al., **LoRA: Low-Rank Adaptation of Large Language Models** — [arXiv:2106.09685](https://arxiv.org/abs/2106.09685)\n- Guo et al., **On Calibration of Modern Neural Networks** (temperature scaling) — [arXiv:1706.04599](https://arxiv.org/abs/1706.04599)\n- Gal & Ghahramani, **Dropout as a Bayesian Approximation** — [arXiv:1506.02142](https://arxiv.org/abs/1506.02142)\n- Khosla et al., **Novel Dataset for Fine-Grained Image Categorization: Stanford Dogs** — [the source of this competition's images](http://vision.stanford.edu/aditya86/ImageNetDogs/)\n- [`timm`](https://github.com/huggingface/pytorch-image-models) · [`peft`](https://github.com/huggingface/peft) · [`optuna`](https://github.com/optuna/optuna)\n\nIf this was useful, an upvote is appreciated — and if you spot something wrong in it, please say\nso in the comments.","metadata":{}},{"id":"366b7188-3636-43e7-9b70-0334074112e3","cell_type":"code","source":"if not CFG.keep_cache:\n    shutil.rmtree(CFG.cache_dir, ignore_errors=True)\n    print('feature cache removed; set CFG.keep_cache = True to keep it for a warm restart')\n\nprint('\\noutput files:')\nfor p in sorted(CFG.work_dir.iterdir()):\n    print(f'  {p.name:<28} {p.stat().st_size / 1e6:8.2f} MB')","metadata":{},"outputs":[],"execution_count":null}]}