{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"b36a35ca-8092-43d7-9035-9d7b0621525a","cell_type":"markdown","source":"# Digit Recognizer — NumPy MLP (from scratch)\n\nA two-layer fully-connected network (784 → 512 ReLU → 10 softmax), written\ndirectly in NumPy — no ML framework. Training uses:\n\n- **mini-batch SGD + momentum** with **step learning-rate decay**\n- **dropout** on the hidden layer\n- **data augmentation** — every training image is randomly rotated / scaled /\n  shifted a little each epoch\n\n**GPU:** turn on the *GPU T4* accelerator and the array backend switches from\nNumPy to **CuPy** automatically (same code, runs on the GPU) — roughly 3–5×\nfaster. With the accelerator off it runs on CPU exactly as before.\n\nRuns unchanged on Kaggle (add the *Digit Recognizer* data as input) or locally\nwith `train.csv` / `test.csv` in `./data/`.\n\n**Output:** `submission.csv` in the working directory.","metadata":{}},{"id":"5c57031a-0ef3-4967-b8f6-0351453fa797","cell_type":"code","source":"from pathlib import Path\n\nimport numpy as np\nimport pandas as pd\n\n# Array backend: CuPy on the GPU if it is available and working, else NumPy.\n# `import cupy` also fails (ImportError) on a CPU-only kernel, so this covers\n# \"accelerator off\" too.\ntry:\n    import cupy as xp\n    xp.zeros(1) + 1          # force a real GPU op; raises if no device\n    BACKEND = \"cupy (GPU)\"\nexcept Exception:\n    import numpy as xp\n    BACKEND = \"numpy (CPU)\"\nprint(\"array backend:\", BACKEND)\n\ndef to_cpu(a):\n    \"\"\"Bring an array back to the CPU as plain NumPy (no-op under NumPy).\"\"\"\n    return xp.asnumpy(a) if BACKEND.startswith(\"cupy\") else np.asarray(a)\n\ndef make_rng(seed):\n    \"\"\"Seeded RNG that works on every NumPy and CuPy version.\n\n    (CuPy's newer ``default_rng`` is missing on Kaggle's image.) Exposes the\n    handful of methods this notebook uses: standard_normal, uniform,\n    permutation, random_sample.\n    \"\"\"\n    return xp.random.RandomState(seed)\n\nDTYPE = np.float32\nSEED = 0\n\nCANDIDATES = [Path(\"/kaggle/input/digit-recognizer\"), Path(\"data\"), Path(\".\")]\nDATA_DIR = next((p for p in CANDIDATES if (p / \"train.csv\").exists()), None)\nassert DATA_DIR is not None, (\n    \"train.csv not found. On Kaggle: 'Add Input' -> Digit Recognizer. \"\n    \"Locally: put train.csv/test.csv in ./data/\"\n)\nprint(\"Using data from:\", DATA_DIR)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-29T16:36:14.806809Z","iopub.execute_input":"2026-08-29T16:36:14.807020Z","iopub.status.idle":"2026-08-29T16:36:16.405633Z","shell.execute_reply.started":"2026-08-29T16:36:14.806998Z","shell.execute_reply":"2026-08-29T16:36:16.404873Z"}},"outputs":[],"execution_count":null},{"id":"eee0a42e-0ee0-43c3-8c1f-2ba7904e1e28","cell_type":"code","source":"train_df = pd.read_csv(DATA_DIR / \"train.csv\")\ntest_df = pd.read_csv(DATA_DIR / \"test.csv\")\nprint(\"train:\", train_df.shape, \" test:\", test_df.shape)\n\n# Pixels -> [0, 1]; labels -> one-hot. Built on the CPU with pandas/NumPy first.\nX_np = train_df.drop(columns=\"label\").to_numpy(DTYPE) / DTYPE(255.0)\nlabels = train_df[\"label\"].to_numpy()\nY_np = np.eye(10, dtype=DTYPE)[labels]\nX_test_np = test_df.to_numpy(DTYPE) / DTYPE(255.0)\n\n# One shuffle, then hold out a validation slice for early stopping.\nperm = np.random.default_rng(SEED).permutation(len(X_np))\nX_np, Y_np, labels = X_np[perm], Y_np[perm], labels[perm]\n\nN_VAL = 4000\n# Move the arrays onto the backend (GPU when available) once, up front.\nX_tr, Y_tr = xp.asarray(X_np[N_VAL:]), xp.asarray(Y_np[N_VAL:])\nX, Y = xp.asarray(X_np), xp.asarray(Y_np)\nX_val = xp.asarray(X_np[:N_VAL])\ny_val = labels[:N_VAL]                 # kept on the CPU for accuracy comparisons\nX_test = xp.asarray(X_test_np)\nprint(\"train split:\", X_tr.shape, \" val split:\", X_val.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-29T16:36:16.408009Z","iopub.execute_input":"2026-08-29T16:36:16.408235Z","iopub.status.idle":"2026-08-29T16:36:21.646371Z","shell.execute_reply.started":"2026-08-29T16:36:16.408213Z","shell.execute_reply":"2026-08-29T16:36:21.645550Z"}},"outputs":[],"execution_count":null},{"id":"6e32d72e-a088-4386-b057-8191c64fc8d7","cell_type":"code","source":"def augment_batch(batch, rng, max_shift=2.0, max_rotate_deg=12.0, max_scale=0.12):\n    \"\"\"Random per-image rotation / scale / shift of a (B, side*side) batch.\n\n    Bilinear sampling; pixels that map outside the source become 0. Runs on\n    whatever backend ``batch`` lives on (NumPy or CuPy), all in float32. Labels\n    are unchanged.\n    \"\"\"\n    b, n = batch.shape\n    side = int(round(n ** 0.5))\n    imgs = batch.reshape(b, side, side)\n    c = (side - 1) / 2.0\n\n    grid = xp.arange(side, dtype=DTYPE)\n    yy, xx = xp.meshgrid(grid, grid, indexing=\"ij\")\n    xc, yc = (xx - c)[None], (yy - c)[None]\n\n    ang = xp.deg2rad(rng.uniform(-max_rotate_deg, max_rotate_deg, b).astype(DTYPE))[:, None, None]\n    inv_s = (1.0 / (1.0 + rng.uniform(-max_scale, max_scale, b).astype(DTYPE)))[:, None, None]\n    tx = rng.uniform(-max_shift, max_shift, b).astype(DTYPE)[:, None, None]\n    ty = rng.uniform(-max_shift, max_shift, b).astype(DTYPE)[:, None, None]\n    cos, sin = xp.cos(ang) * inv_s, xp.sin(ang) * inv_s\n\n    src_x = cos * xc + sin * yc + c - tx\n    src_y = -sin * xc + cos * yc + c - ty\n    x0, y0 = xp.floor(src_x).astype(xp.int32), xp.floor(src_y).astype(xp.int32)\n    wx, wy = src_x - x0, src_y - y0\n    bi = xp.arange(b)[:, None, None]\n\n    def s(xi, yi):\n        inb = ((xi >= 0) & (xi < side) & (yi >= 0) & (yi < side)).astype(DTYPE)\n        v = imgs[bi, xp.clip(yi, 0, side - 1), xp.clip(xi, 0, side - 1)]\n        return v * inb          # zero out-of-bounds samples; stays float32\n\n    top = s(x0, y0) * (1 - wx) + s(x0 + 1, y0) * wx\n    bot = s(x0, y0 + 1) * (1 - wx) + s(x0 + 1, y0 + 1) * wx\n    return xp.clip(top * (1 - wy) + bot * wy, 0.0, 1.0).reshape(b, n).astype(DTYPE)\n\n\nclass MLP:\n    \"\"\"784 -> n_hidden (ReLU, dropout) -> 10 (softmax), SGD + momentum.\"\"\"\n\n    def __init__(self, n_in=784, n_hidden=512, n_out=10, lr=0.08,\n                 momentum=0.9, dropout=0.25, seed=SEED):\n        r = make_rng(seed)\n        self.lr, self.mu, self.p_drop = lr, momentum, dropout\n        self.drop_rng = make_rng(seed + 1)\n        # randn / rand are on every NumPy and CuPy RandomState.\n        self.w1 = (r.randn(n_in, n_hidden) * np.sqrt(2 / n_in)).astype(DTYPE)\n        self.b1 = xp.zeros(n_hidden, DTYPE)\n        self.w2 = (r.randn(n_hidden, n_out) * np.sqrt(2 / n_hidden)).astype(DTYPE)\n        self.b2 = xp.zeros(n_out, DTYPE)\n        self.vw1 = xp.zeros_like(self.w1); self.vb1 = xp.zeros_like(self.b1)\n        self.vw2 = xp.zeros_like(self.w2); self.vb2 = xp.zeros_like(self.b2)\n\n    def forward(self, x, training=False):\n        self.x = x\n        self.z1 = x @ self.w1 + self.b1\n        a1 = xp.maximum(0, self.z1)\n        if training and self.p_drop > 0:\n            keep = 1.0 - self.p_drop\n            # inverted dropout, kept in float32 (bool / float would upcast to f64)\n            self.mask = (self.drop_rng.rand(*a1.shape) < keep).astype(DTYPE)\n            self.mask *= DTYPE(1.0 / keep)\n            a1 = a1 * self.mask\n        else:\n            self.mask = None\n        self.a1 = a1\n        z2 = self.a1 @ self.w2 + self.b2\n        z2 -= z2.max(axis=1, keepdims=True)\n        e = xp.exp(z2)\n        self.p = e / e.sum(axis=1, keepdims=True)\n        return self.p\n\n    def backward(self, y):\n        n = len(self.x)\n        dz2 = (self.p - y) / n\n        dw2 = self.a1.T @ dz2\n        db2 = dz2.sum(axis=0)\n        da1 = dz2 @ self.w2.T\n        if self.mask is not None:\n            da1 = da1 * self.mask\n        dz1 = da1 * (self.z1 > 0)\n        dw1 = self.x.T @ dz1\n        db1 = dz1.sum(axis=0)\n        self.vw1 = self.mu * self.vw1 - self.lr * dw1\n        self.vb1 = self.mu * self.vb1 - self.lr * db1\n        self.vw2 = self.mu * self.vw2 - self.lr * dw2\n        self.vb2 = self.mu * self.vb2 - self.lr * db2\n        self.w1 += self.vw1; self.b1 += self.vb1\n        self.w2 += self.vw2; self.b2 += self.vb2\n\n    def weights(self):\n        return [w.copy() for w in (self.w1, self.b1, self.w2, self.b2)]\n\n    def load(self, ws):\n        self.w1, self.b1, self.w2, self.b2 = (w.copy() for w in ws)\n\n    def predict(self, x):\n        return xp.argmax(self.forward(x), axis=1)   # training=False -> no dropout\n\n\ndef val_accuracy(model, X_val, y_val_cpu):\n    return float((to_cpu(model.predict(X_val)) == y_val_cpu).mean())\n\n\ndef train(model, X, Y, X_val=None, y_val=None, epochs=260, batch=128,\n          eval_every=5, patience=16, lr_decay=0.5, lr_decay_every=70,\n          augment=True, seed=SEED):\n    \"\"\"Mini-batch SGD with LR decay and (optionally) per-batch augmentation.\n\n    With validation data: check accuracy every ``eval_every`` epochs, keep the\n    best weights, stop after ``patience`` checks without improvement, and always\n    restore the best weights at the end. Returns ``(best_epoch, best_val_acc)``.\n    \"\"\"\n    r = make_rng(seed)\n    aug_rng = make_rng(seed + 2)\n    base_lr = model.lr\n    best_acc, best_epoch, best_w, stale = -1.0, epochs, None, 0\n\n    for epoch in range(1, epochs + 1):\n        if lr_decay_every:\n            model.lr = base_lr * (lr_decay ** ((epoch - 1) // lr_decay_every))\n        order = r.permutation(len(X))\n        for s in range(0, len(X), batch):\n            idx = order[s : s + batch]\n            bx = augment_batch(X[idx], aug_rng) if augment else X[idx]\n            model.forward(bx, training=True)\n            model.backward(Y[idx])\n\n        if X_val is None or epoch % eval_every != 0:\n            continue\n\n        acc = val_accuracy(model, X_val, y_val)\n        improved = acc > best_acc\n        print(f\"epoch {epoch:3d}/{epochs}  lr {model.lr:.4f}  val_acc {acc:.4f}{'  *' if improved else ''}\")\n        if improved:\n            best_acc, best_epoch, best_w, stale = acc, epoch, model.weights(), 0\n        else:\n            stale += 1\n            if patience and stale >= patience:\n                print(f\"early stop at epoch {epoch} - best {best_acc:.4f} @ epoch {best_epoch}\")\n                break\n\n    if best_w is not None:\n        model.load(best_w)\n    return best_epoch, best_acc","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-29T16:36:21.647942Z","iopub.execute_input":"2026-08-29T16:36:21.648166Z","iopub.status.idle":"2026-08-29T16:36:21.682446Z","shell.execute_reply.started":"2026-08-29T16:36:21.648144Z","shell.execute_reply":"2026-08-29T16:36:21.681319Z"}},"outputs":[],"execution_count":null},{"id":"a9d1ac02-3b0f-452f-b5da-8d28b654afb6","cell_type":"code","source":"# --- hyper-parameters (tuned locally against full MNIST) ---\nHIDDEN, LR, MOMENTUM, DROPOUT = 512, 0.08, 0.9, 0.25\nEPOCHS, BATCH = 260, 128\nEVAL_EVERY, PATIENCE = 5, 16\nLR_DECAY, LR_DECAY_EVERY = 0.5, 70\n\n# 1) Fit on the train split; early stopping picks the epoch count.\nmodel = MLP(n_hidden=HIDDEN, lr=LR, momentum=MOMENTUM, dropout=DROPOUT)\nbest_epoch, best_acc = train(\n    model, X_tr, Y_tr, X_val, y_val,\n    epochs=EPOCHS, batch=BATCH, eval_every=EVAL_EVERY, patience=PATIENCE,\n    lr_decay=LR_DECAY, lr_decay_every=LR_DECAY_EVERY, augment=True,\n)\nprint(f\"\\nbest validation accuracy: {best_acc:.4f} at epoch {best_epoch}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-29T16:36:21.683741Z","iopub.execute_input":"2026-08-29T16:36:21.683997Z","iopub.status.idle":"2026-08-29T16:40:27.727160Z","shell.execute_reply.started":"2026-08-29T16:36:21.683973Z","shell.execute_reply":"2026-08-29T16:40:27.726377Z"}},"outputs":[],"execution_count":null},{"id":"dc5193e5-8579-4d48-a3b5-d2c6863b8525","cell_type":"code","source":"# 2) Retrain on ALL labelled data (train + val) for the chosen epoch count.\nfinal = MLP(n_hidden=HIDDEN, lr=LR, momentum=MOMENTUM, dropout=DROPOUT)\ntrain(final, X, Y, epochs=best_epoch, batch=BATCH,\n      lr_decay=LR_DECAY, lr_decay_every=LR_DECAY_EVERY, augment=True)\nprint(\"full-data train accuracy:\", float((to_cpu(final.predict(X)) == labels).mean()))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-29T16:40:27.728522Z","iopub.execute_input":"2026-08-29T16:40:27.728828Z","iopub.status.idle":"2026-08-29T16:43:21.239749Z","shell.execute_reply.started":"2026-08-29T16:40:27.728801Z","shell.execute_reply":"2026-08-29T16:43:21.238944Z"}},"outputs":[],"execution_count":null},{"id":"3039f19e-8a90-4cbe-9dc7-8845191c2269","cell_type":"code","source":"# 3) Predict the test set and write the submission file.\npreds = to_cpu(final.predict(X_test))\nsubmission = pd.DataFrame({\"ImageId\": np.arange(1, len(preds) + 1), \"Label\": preds})\nsubmission.to_csv(\"submission.csv\", index=False)\nprint(submission.shape)\nsubmission.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-29T16:43:21.240966Z","iopub.execute_input":"2026-08-29T16:43:21.241184Z","iopub.status.idle":"2026-08-29T16:43:21.702892Z","shell.execute_reply.started":"2026-08-29T16:43:21.241162Z","shell.execute_reply":"2026-08-29T16:43:21.701994Z"}},"outputs":[],"execution_count":null},{"id":"52f17ed4-a75a-4647-9503-7d7131c08ab8","cell_type":"markdown","source":"## GPU\n\nEnable the **GPU T4** accelerator (Settings → Accelerator) before *Run All*.\nCell 1 prints `array backend: cupy (GPU)` when it took — the whole model then\nruns on the GPU with no other change, roughly **3–5× faster** (~3–5 min instead\nof 10–20). With the accelerator off it prints `numpy (CPU)` and runs as before.\n\nOnly one GPU is used (CuPy uses device 0); the second T4 sits idle, which is\nfine for a model this small.\n\nCuPy's random number generator differs from NumPy's, so a GPU run's exact\naccuracy will differ from a CPU run by a hair — same ballpark, not bit-identical.\n\n## How to submit\n\n**On Kaggle:** *Run All*, then click **Submit** (top-right), or *Save Version* →\nopen it → **Submit** from the Output tab. Kaggle picks up `submission.csv`\nautomatically.\n\n**Locally:** the last cell wrote `submission.csv` — upload it on the *Submit\nPredictions* page, or `kaggle competitions submit -c digit-recognizer -f submission.csv -m \"numpy MLP + augmentation\"`.\n\nLocal full-MNIST test accuracy for this config is ~0.993; on Kaggle expect\nroughly **0.986–0.989** (it trains on 42k rows vs MNIST's 60k, and the public\nleaderboard scores a subset).","metadata":{}}]}