{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":31090,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# From Scratch GAN on MNIST (Pytorch)\n\n**abstract:** Here a Generative adversarial network is trained on MNIST dataset using DCGAN architecture adapted to 28x28 grayscale images. The discriminator is a binary classifier trained to distinguish real images from fake images, while the generator learns to synthesize the realistic digits from random noise. ","metadata":{}},{"cell_type":"markdown","source":"$$\n\\begin{align*}\n& \\textbf{1. GAN Objective (Theory)} \\\\\n& \\text{A vanilla GAN consists of a \\textbf{generator} } G_\\theta \\text{ that maps latent noise } z \\sim p_z(z) \\text{ to images } G(z), \\\\\n& \\text{and a \\textbf{discriminator} } D_\\phi \\text{ that outputs the probability that an input image is real. The original minimax objective is:} \\\\\n& \\min_{\\theta}\\max_{\\phi}\\; V(D_\\phi,G_\\theta) = \\mathbb{E}_{x \\sim p_{\\text{data}}}[\\log D_\\phi(x)] + \\mathbb{E}_{z \\sim p_z}[\\log(1 - D_\\phi(G_\\theta(z)))]. \\\\\n\\\\\n& \\text{In practice, the \\textbf{non-saturating} generator loss is used to provide stronger gradients:} \\\\\n& \\mathcal{L}_D = -\\mathbb{E}_{x \\sim p_{\\text{data}}}[\\log D_\\phi(x)] - \\mathbb{E}_{z \\sim p_z}[\\log(1 - D_\\phi(G_\\theta(z)))], \\\\\n& \\mathcal{L}_G = -\\mathbb{E}_{z \\sim p_z}[\\log D_\\phi(G_\\theta(z))]. \\\\\n\\\\\n& \\text{I implement these with \\textbf{binary cross-entropy with logits} (BCEWithLogitsLoss), letting the discriminator} \\\\\n& \\text{output raw logits } s \\text{ where } D(x) = \\sigma(s) \\text{ and } \\sigma(\\cdot) \\text{ is the sigmoid function.} \\\\\n\\\\\n& \\textbf{2. DCGAN Heuristics (adapted to 28×28 MNIST)} \\\\\n& \\text{- \\textbf{Generator:} latent } z \\in \\mathbb{R}^{100} \\to \\text{linear} \\to \\text{reshape to } 7 \\times 7 \\text{ feature map} \\to \\text{transposed convolutions to } 14 \\times 14 \\\\\n& \\quad \\text{and } 28 \\times 28 \\to \\text{final } 1 \\text{ channel with \\textbf{tanh}.} \\\\\n& \\text{- \\textbf{Discriminator:} strided convolutions } 28 \\to 14 \\to 7, \\text{ LeakyReLU activations, batch norm in deeper layers, and a} \\\\\n& \\quad \\text{single-logit output (real vs fake).} \\\\\n& \\text{- \\textbf{Optimizer:} Adam with } \\beta_1=0.5, \\beta_2=0.999. \\\\\n& \\text{- \\textbf{Input normalization:} images scaled to } [-1,1] \\text{ to match generator \\textbf{tanh} output.} \\\\\n\\\\\n& \\textbf{3. Labels and Stabilization} \\\\\n& \\text{I use target labels } 1 \\text{ for real and } 0 \\text{ for fake. Optional tricks like one-sided label smoothing or random label} \\\\\n& \\text{flips can be added later to improve stability.}\n\\end{align*}\n$$","metadata":{}},{"cell_type":"markdown","source":"# Import the necessary libraries","metadata":{}},{"cell_type":"code","source":"import os, math, random\nimport numpy as np\nimport torch \nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader\nfrom torchvision import datasets,transforms, utils\nimport matplotlib.pyplot as plt","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:43.157244Z","iopub.execute_input":"2025-09-07T01:10:43.157530Z","iopub.status.idle":"2025-09-07T01:10:50.033936Z","shell.execute_reply.started":"2025-09-07T01:10:43.157513Z","shell.execute_reply":"2025-09-07T01:10:50.033396Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"SEED = 42\nrandom.seed(SEED); np.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\ntorch.backends.cudnn.benchmark = False","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:50.034558Z","iopub.execute_input":"2025-09-07T01:10:50.034820Z","iopub.status.idle":"2025-09-07T01:10:50.043480Z","shell.execute_reply.started":"2025-09-07T01:10:50.034804Z","shell.execute_reply":"2025-09-07T01:10:50.042953Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(\"Device:\", DEVICE)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:50.045114Z","iopub.execute_input":"2025-09-07T01:10:50.045381Z","iopub.status.idle":"2025-09-07T01:10:50.110228Z","shell.execute_reply.started":"2025-09-07T01:10:50.045340Z","shell.execute_reply":"2025-09-07T01:10:50.109360Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"OUT_DIR = \"/kaggle/working/gan_mnist\"\nSAMPLES_DIR = os.path.join(OUT_DIR, \"samples\")\nos.makedirs(SAMPLES_DIR, exist_ok=True)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:50.110959Z","iopub.execute_input":"2025-09-07T01:10:50.111287Z","iopub.status.idle":"2025-09-07T01:10:50.124378Z","shell.execute_reply.started":"2025-09-07T01:10:50.111262Z","shell.execute_reply":"2025-09-07T01:10:50.123838Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# MNIST dataloader","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 128\n\ntransform = transforms.Compose([\n    transforms.ToTensor(),                # -> [0,1]\n    transforms.Normalize((0.5,), (0.5,))  # -> [-1,1] for GAN (tanh output)\n])\n\ntrainset = datasets.MNIST(\n    root=\"./data\",   # will download here\n    train=True,\n    download=True,\n    transform=transform\n)\n\ntrainloader = DataLoader(\n    trainset,\n    batch_size=BATCH_SIZE,\n    shuffle=True,\n    num_workers=2,\n    pin_memory=True\n)\n\nprint(\"Total samples:\", len(trainset))\nprint(\"Batches per epoch:\", len(trainloader))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:50.125035Z","iopub.execute_input":"2025-09-07T01:10:50.125296Z","iopub.status.idle":"2025-09-07T01:10:53.902352Z","shell.execute_reply.started":"2025-09-07T01:10:50.125273Z","shell.execute_reply":"2025-09-07T01:10:53.901609Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# models (Generator and Discriminators)","metadata":{}},{"cell_type":"code","source":"LATENT_DIM=100\n\nclass Generator(nn.Module):\n    def __init__(self,z_dim=LATENT_DIM,base=64):\n        super().__init__()\n        self.fc=nn.Sequential(\n            nn.Linear(z_dim,256*7*7),\n            nn.BatchNorm1d(256*7*7),\n            nn.ReLU(True)\n        )\n        self.deconv=nn.Sequential(\n            nn.ConvTranspose2d(256, 128, kernel_size=4, stride=2, padding=1),  # 7->14\n            nn.BatchNorm2d(128),\n            nn.ReLU(True),\n\n            nn.ConvTranspose2d(128, 64, kernel_size=4, stride=2, padding=1),   # 14->28\n            nn.BatchNorm2d(64),\n            nn.ReLU(True),\n\n            nn.Conv2d(64, 1, kernel_size=3, stride=1, padding=1),\n            nn.Tanh()\n        )\n    def forward(self,z):\n        x=self.fc(z)\n        x=x.view(z.size(0),256,7,7)\n        x=self.deconv(x)\n        return x","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:53.903162Z","iopub.execute_input":"2025-09-07T01:10:53.903420Z","iopub.status.idle":"2025-09-07T01:10:53.909115Z","shell.execute_reply.started":"2025-09-07T01:10:53.903396Z","shell.execute_reply":"2025-09-07T01:10:53.908466Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class Discriminator(nn.Module):\n    # (1,28,28) -> logit\n    def __init__(self):\n        super().__init__()\n        self.conv = nn.Sequential(\n            nn.Conv2d(1, 64, kernel_size=4, stride=2, padding=1),   # 28->14\n            nn.LeakyReLU(0.2, inplace=True),\n\n            nn.Conv2d(64, 128, kernel_size=4, stride=2, padding=1), # 14->7\n            nn.BatchNorm2d(128),\n            nn.LeakyReLU(0.2, inplace=True),\n        )\n        self.fc = nn.Linear(128*7*7, 1)  \n\n    def forward(self, x):\n        h = self.conv(x)                 # (N, 128, 7, 7)\n        h = h.view(x.size(0), -1)        # (N, 128*7*7)\n        logit = self.fc(h)               # (N, 1)\n        return logit\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:53.909743Z","iopub.execute_input":"2025-09-07T01:10:53.909977Z","iopub.status.idle":"2025-09-07T01:10:53.927173Z","shell.execute_reply.started":"2025-09-07T01:10:53.909955Z","shell.execute_reply":"2025-09-07T01:10:53.926404Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"G = Generator().to(DEVICE)\nD = Discriminator().to(DEVICE)\n\nprint(\"Generator params:\", sum(p.numel() for p in G.parameters())/1e6, \"M\")\nprint(\"Discriminator params:\", sum(p.numel() for p in D.parameters())/1e6, \"M\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:53.927804Z","iopub.execute_input":"2025-09-07T01:10:53.928036Z","iopub.status.idle":"2025-09-07T01:10:54.120108Z","shell.execute_reply.started":"2025-09-07T01:10:53.928021Z","shell.execute_reply":"2025-09-07T01:10:54.119479Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Loss and optimizers","metadata":{}},{"cell_type":"code","source":"lr = 2e-4\nG_OPT = torch.optim.Adam(G.parameters(), lr=lr, betas=(0.5, 0.999))\nD_OPT = torch.optim.Adam(D.parameters(), lr=lr, betas=(0.5, 0.999))\ncriterion = nn.BCEWithLogitsLoss()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:54.121944Z","iopub.execute_input":"2025-09-07T01:10:54.122174Z","iopub.status.idle":"2025-09-07T01:10:54.126581Z","shell.execute_reply.started":"2025-09-07T01:10:54.122156Z","shell.execute_reply":"2025-09-07T01:10:54.125854Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"FIXED_Z = torch.randn(64, LATENT_DIM, device=DEVICE)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:54.127433Z","iopub.execute_input":"2025-09-07T01:10:54.127668Z","iopub.status.idle":"2025-09-07T01:10:54.152017Z","shell.execute_reply.started":"2025-09-07T01:10:54.127647Z","shell.execute_reply":"2025-09-07T01:10:54.151358Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def save_grid(tensor, fname, nrow=8):\n    grid = utils.make_grid(tensor, nrow=nrow, normalize=True, value_range=(-1, 1))\n    utils.save_image(grid, fname)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:54.152744Z","iopub.execute_input":"2025-09-07T01:10:54.152923Z","iopub.status.idle":"2025-09-07T01:10:54.156490Z","shell.execute_reply.started":"2025-09-07T01:10:54.152909Z","shell.execute_reply":"2025-09-07T01:10:54.155903Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"$$\n\\begin{align*}\n& \\textbf{4. Training Losses (with BCE logits)} \\\\\n\\\\\n& \\text{Let } s_r = D_\\phi(x) \\text{ and } s_f = D_\\phi(G_\\theta(z)) \\text{ denote discriminator logits.} \\\\\n& \\text{Using \\textbf{BCEWithLogitsLoss}, we minimize:} \\\\\n\\\\\n& \\mathcal{L}_D = \\mathrm{BCEWithLogits}(s_r, 1) + \\mathrm{BCEWithLogits}(s_f, 0), \\\\\n& \\mathcal{L}_G = \\mathrm{BCEWithLogits}(s_f, 1). \\\\\n\\\\\n& \\text{These correspond to the non-saturating GAN formulation and yield stronger} \\\\\n& \\text{generator gradients early in training.}\n\\end{align*}\n$$","metadata":{}},{"cell_type":"markdown","source":"# Training Loop","metadata":{}},{"cell_type":"code","source":"EPOCHS = 200\nlog_every = 200\n\nG_losses, D_losses = [], []\n\nfor epoch in range(1, EPOCHS+1):\n    G.train(); D.train()\n    for i, (x_real, _) in enumerate(trainloader, start=1):\n        x_real = x_real.to(DEVICE)  \n        bs = x_real.size(0)\n\n        z = torch.randn(bs, LATENT_DIM, device=DEVICE)\n        x_fake = G(z).detach()\n\n        D_OPT.zero_grad(set_to_none=True)\n\n\n        real_logits = D(x_real)\n        real_labels = torch.ones(bs, 1, device=DEVICE)\n        d_real = criterion(real_logits, real_labels)\n\n        fake_logits = D(x_fake)\n        fake_labels = torch.zeros(bs, 1, device=DEVICE)\n        d_fake = criterion(fake_logits, fake_labels)\n\n        d_loss = d_real + d_fake\n        d_loss.backward()\n        D_OPT.step()\n\n        z = torch.randn(bs, LATENT_DIM, device=DEVICE)\n        x_fake = G(z)\n        G_OPT.zero_grad(set_to_none=True)\n\n        fake_logits = D(x_fake)\n        g_loss = criterion(fake_logits, real_labels)\n        g_loss.backward()\n        G_OPT.step()\n\n        # Logging\n        if i % log_every == 0:\n            print(f\"[Epoch {epoch:02d}/{EPOCHS}] Step {i:04d} | D: {d_loss.item():.3f} | G: {g_loss.item():.3f}\")\n\n    with torch.no_grad():\n        samples = G(FIXED_Z).cpu()\n    save_grid(samples, os.path.join(SAMPLES_DIR, f\"epoch_{epoch:03d}.png\"), nrow=8)\n\n    G_losses.append(g_loss.item())\n    D_losses.append(d_loss.item())\n\nprint(\"Training complete. Samples saved to:\", SAMPLES_DIR)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:10:54.157134Z","iopub.execute_input":"2025-09-07T01:10:54.157326Z","iopub.status.idle":"2025-09-07T01:19:07.939474Z","shell.execute_reply.started":"2025-09-07T01:10:54.157306Z","shell.execute_reply":"2025-09-07T01:19:07.938653Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Visualizations","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(6,4))\nplt.plot(G_losses, label=\"G loss\")\nplt.plot(D_losses, label=\"D loss\")\nplt.xlabel(\"Epoch\"); plt.ylabel(\"Loss\"); plt.title(\"GAN Training Losses\")\nplt.legend(); plt.tight_layout()\nplt.savefig(os.path.join(OUT_DIR, \"loss_curves.png\"), dpi=150)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:19:07.940417Z","iopub.execute_input":"2025-09-07T01:19:07.940643Z","iopub.status.idle":"2025-09-07T01:19:08.310569Z","shell.execute_reply.started":"2025-09-07T01:19:07.940617Z","shell.execute_reply":"2025-09-07T01:19:08.309828Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from IPython.display import Image, display\nlatest = os.path.join(SAMPLES_DIR, sorted(os.listdir(SAMPLES_DIR))[-1])\ndisplay(Image(latest))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:19:08.311408Z","iopub.execute_input":"2025-09-07T01:19:08.311611Z","iopub.status.idle":"2025-09-07T01:19:08.317749Z","shell.execute_reply.started":"2025-09-07T01:19:08.311589Z","shell.execute_reply":"2025-09-07T01:19:08.317101Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"$$\n\\begin{align*}\n& \\textbf{5. Results \\& Notes} \\\\\n\\\\\n& \\text{- \\textbf{Qualitative Samples.} The grids saved at \\texttt{samples/epoch\\_XXX.png} show clear progression} \\\\\n& \\quad \\text{from noisy blobs to digit-like structures.} \\\\\n\\\\\n& \\text{- \\textbf{Loss Behavior.} The discriminator loss } \\mathcal{L}_D \\text{ and generator loss } \\mathcal{L}_G \\text{ may oscillate.} \\\\\n& \\quad \\text{This is expected due to the adversarial nature of training.} \\\\\n\\\\\n& \\text{- \\textbf{Stability Tips.}} \\\\\n& \\quad \\text{- Keep } \\beta_1=0.5 \\text{ for Adam (DCGAN recommendation).} \\\\\n& \\quad \\text{- Normalize inputs to } [-1,1] \\text{ and use \\textbf{tanh} at the generator output.} \\\\\n& \\quad \\text{- If collapse occurs, try smaller learning rate, add one-sided label smoothing} \\\\\n& \\qquad \\text{(e.g., real label } =0.9\\text{), or add instance noise.}\n\\end{align*}\n$$","metadata":{}},{"cell_type":"code","source":"G.eval()\nwith torch.no_grad():\n    z = torch.randn(64, LATENT_DIM, device=DEVICE)\n    imgs = G(z).cpu()\nsave_grid(imgs, os.path.join(SAMPLES_DIR, \"inference_grid.png\"), nrow=8)\nfrom IPython.display import Image, display\ndisplay(Image(os.path.join(SAMPLES_DIR, \"inference_grid.png\")))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-07T01:19:08.318551Z","iopub.execute_input":"2025-09-07T01:19:08.318792Z","iopub.status.idle":"2025-09-07T01:19:08.355737Z","shell.execute_reply.started":"2025-09-07T01:19:08.318771Z","shell.execute_reply":"2025-09-07T01:19:08.355027Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}