![\"image\"](\"http://playagricola.com/Kaggle/DIS.jpg\")
There are currently no Kaggle kernels that use RaLSGAN. A few kernels have RaLSGAN in their name, but they are using basic DCGAN. What is RaLSGAN? RaLSGAN is just basic GAN with a different loss function.
\n\n
A Discriminator can either output sigmoid activation or linear activation. If sigmoid, we have the probability of an image being real, Pr(Real). If linear, we have the log odds of being real, Logit. Probabilities are between 0 and 1 inclusive. And Logits can be any number. A positive logit implies image is real, a negative logit implies image is fake.
Define x_r to be a real image, and x_f to be a fake image. Define D(x) = Pr(Real) and C(x) = Logit to be the two different outputs from a Discriminator when you input an image. The simplest possible loss is below.
# Take AVG over x_r and x_f in batch\ndisc_loss = (1 - D(x_r)) + (D(x_f) - 0)\ngen_loss = (1 - D(x_f))\nMinimizing these losses says the following. The Discriminator wants D(x_r)=1 and D(x_f)=0, i.e. the Discriminator wants real images to have probably 1 of being real. And wants fake images to have probability 0 of being real. The Generator wants D(x_f)=1, i.e. the Generator wants fakes images to have probability 1 of being real.
Basic GAN or DCGAN use the following loss functions. Note that they use D(x), i.e. probs.
# Take AVG over x_r and x_f in batch\ndisc_loss = -log (D(x_r)) - log (1-D(x_f))\ngen_loss = -log (D(x_f))\nRaLSGAN uses the following loss functions. Note that they use C(x), i.e. logits.
# Take AVG over x_r and x_f in batch\ndisc_loss = (C(x_r) - AVG(C(x_f)) - 1)^2 + (C(x_f) - AVG(C(x_r)) + 1)^2\ngen_loss = (C(x_r) - AVG(C(x_f)) + 1)^2 + (C(x_f) - AVG(C(x_r)) - 1)^2\nHere's how to update the Kaggle kernel starter code. First, if you fork Kaggle's Starter Code, turn GPU on! Second, change the last line of the Discriminator to output linear activation (Logit) by removing sigmoid
\n\n #x = torch.sigmoid(self.conv5(x))\n x = self.conv5(x)\nNext, change the following
\n\n ############################\n # (1) Update D network\n ###########################\n netD.zero_grad()\n real_images = real_images.to(device)\n batch_size = real_images.size(0)\n labels = torch.full((batch_size, 1), real_label, device=device)\n outputR = netD(real_images)\n noise = torch.randn(batch_size, nz, 1, 1, device=device)\n fake = netG(noise)\n outputF = netD(fake.detach())\n errD = (torch.mean((outputR - torch.mean(outputF) - labels) ** 2) + \n torch.mean((outputF - torch.mean(outputR) + labels) ** 2))/2\n errD.backward(retain_graph=True)\n optimizerD.step()\n ############################\n # (2) Update G network\n ###########################\n netG.zero_grad()\n outputF = netD(fake) \n errG = (torch.mean((outputR - torch.mean(outputF) + labels) ** 2) +\n torch.mean((outputF - torch.mean(outputR) - labels) ** 2))/2\n errG.backward()\n optimizerG.step()\nI notice that many kernels have dark images. This is because the image pixels have been normalized to [ -1, 1 ]. If you want nice looking images, then un-normalize them with plt.imshow((img+1.)/2.).
Have fun and enjoy the power of RaLSGAN!
", "messages": [ { "id": "573007", "postDate": "07/11/2019 16:43:25", "content": "There are currently no Kaggle kernels that use RaLSGAN. A few kernels have RaLSGAN in their name, but they are using basic DCGAN. What is RaLSGAN? RaLSGAN is just basic GAN with a different loss function.
\n\n
A Discriminator can either output sigmoid activation or linear activation. If sigmoid, we have the probability of an image being real, Pr(Real). If linear, we have the log odds of being real, Logit. Probabilities are between 0 and 1 inclusive. And Logits can be any number. A positive logit implies image is real, a negative logit implies image is fake.
Define x_r to be a real image, and x_f to be a fake image. Define D(x) = Pr(Real) and C(x) = Logit to be the two different outputs from a Discriminator when you input an image. The simplest possible loss is below.
# Take AVG over x_r and x_f in batch\ndisc_loss = (1 - D(x_r)) + (D(x_f) - 0)\ngen_loss = (1 - D(x_f))\nMinimizing these losses says the following. The Discriminator wants D(x_r)=1 and D(x_f)=0, i.e. the Discriminator wants real images to have probably 1 of being real. And wants fake images to have probability 0 of being real. The Generator wants D(x_f)=1, i.e. the Generator wants fakes images to have probability 1 of being real.
Basic GAN or DCGAN use the following loss functions. Note that they use D(x), i.e. probs.
# Take AVG over x_r and x_f in batch\ndisc_loss = -log (D(x_r)) - log (1-D(x_f))\ngen_loss = -log (D(x_f))\nRaLSGAN uses the following loss functions. Note that they use C(x), i.e. logits.
# Take AVG over x_r and x_f in batch\ndisc_loss = (C(x_r) - AVG(C(x_f)) - 1)^2 + (C(x_f) - AVG(C(x_r)) + 1)^2\ngen_loss = (C(x_r) - AVG(C(x_f)) + 1)^2 + (C(x_f) - AVG(C(x_r)) - 1)^2\nHere's how to update the Kaggle kernel starter code. First, if you fork Kaggle's Starter Code, turn GPU on! Second, change the last line of the Discriminator to output linear activation (Logit) by removing sigmoid
\n\n #x = torch.sigmoid(self.conv5(x))\n x = self.conv5(x)\nNext, change the following
\n\n ############################\n # (1) Update D network\n ###########################\n netD.zero_grad()\n real_images = real_images.to(device)\n batch_size = real_images.size(0)\n labels = torch.full((batch_size, 1), real_label, device=device)\n outputR = netD(real_images)\n noise = torch.randn(batch_size, nz, 1, 1, device=device)\n fake = netG(noise)\n outputF = netD(fake.detach())\n errD = (torch.mean((outputR - torch.mean(outputF) - labels) ** 2) + \n torch.mean((outputF - torch.mean(outputR) + labels) ** 2))/2\n errD.backward(retain_graph=True)\n optimizerD.step()\n ############################\n # (2) Update G network\n ###########################\n netG.zero_grad()\n outputF = netD(fake) \n errG = (torch.mean((outputR - torch.mean(outputF) + labels) ** 2) +\n torch.mean((outputF - torch.mean(outputR) - labels) ** 2))/2\n errG.backward()\n optimizerG.step()\nI notice that many kernels have dark images. This is because the image pixels have been normalized to [ -1, 1 ]. If you want nice looking images, then un-normalize them with plt.imshow((img+1.)/2.).
Have fun and enjoy the power of RaLSGAN!
", "rawMarkdown": "# Loss Functions\nThere are currently no Kaggle kernels that use RaLSGAN. A few kernels have RaLSGAN in their name, but they are using basic DCGAN. **What is RaLSGAN?** RaLSGAN is just basic GAN with a different loss function.\n\n\n\nA Discriminator can either output `sigmoid` activation or `linear` activation. If sigmoid, we have the probability of an image being real, `Pr(Real)`. If linear, we have the [log odds][1] of being real, `Logit`. Probabilities are between 0 and 1 inclusive. And Logits can be any number. A positive logit implies image is real, a negative logit implies image is fake.\n# Simple Loss\nDefine `x_r` to be a real image, and `x_f` to be a fake image. Define `D(x) = Pr(Real)` and `C(x) = Logit` to be the two different outputs from a Discriminator when you input an image. The simplest possible loss is below. \n \n # Take AVG over x_r and x_f in batch\n disc_loss = (1 - D(x_r)) + (D(x_f) - 0)\n gen_loss = (1 - D(x_f))\n\nMinimizing these losses says the following. The Discriminator wants `D(x_r)=1` and `D(x_f)=0`, i.e. the Discriminator wants real images to have probably 1 of being real. And wants fake images to have probability 0 of being real. The Generator wants `D(x_f)=1`, i.e. the Generator wants fakes images to have probability 1 of being real.\n \n# DCGAN Loss\nBasic GAN or DCGAN use the following loss functions. Note that they use `D(x)`, i.e. probs.\n\n # Take AVG over x_r and x_f in batch\n disc_loss = -log (D(x_r)) - log (1-D(x_f))\n gen_loss = -log (D(x_f))\n\n# RaLSGAN Loss\nRaLSGAN uses the following loss functions. Note that they use `C(x)`, i.e. logits.\n\n # Take AVG over x_r and x_f in batch\n disc_loss = (C(x_r) - AVG(C(x_f)) - 1)^2 + (C(x_f) - AVG(C(x_r)) + 1)^2\n gen_loss = (C(x_r) - AVG(C(x_f)) + 1)^2 + (C(x_f) - AVG(C(x_r)) - 1)^2\n\n# Code\nHere's how to update the Kaggle kernel starter code. First, if you fork Kaggle's Starter Code, **turn GPU on**! Second, change the last line of the Discriminator to output linear activation (Logit) by removing sigmoid\n\n #x = torch.sigmoid(self.conv5(x))\n x = self.conv5(x)\n\nNext, change the following\n\n ############################\n # (1) Update D network\n ###########################\n netD.zero_grad()\n real_images = real_images.to(device)\n batch_size = real_images.size(0)\n labels = torch.full((batch_size, 1), real_label, device=device)\n outputR = netD(real_images)\n noise = torch.randn(batch_size, nz, 1, 1, device=device)\n fake = netG(noise)\n outputF = netD(fake.detach())\n errD = (torch.mean((outputR - torch.mean(outputF) - labels) ** 2) + \n torch.mean((outputF - torch.mean(outputR) + labels) ** 2))/2\n errD.backward(retain_graph=True)\n optimizerD.step()\n ############################\n # (2) Update G network\n ###########################\n netG.zero_grad()\n outputF = netD(fake) \n errG = (torch.mean((outputR - torch.mean(outputF) + labels) ** 2) +\n torch.mean((outputF - torch.mean(outputR) - labels) ** 2))/2\n errG.backward()\n optimizerG.step()\n\n# Image Display\nI notice that many kernels have dark images. This is because the image pixels have been normalized to `[ -1, 1 ]`. If you want nice looking images, then un-normalize them with `plt.imshow((img+1.)/2.)`.\n\n# Have Fun\nHave fun and enjoy the power of RaLSGAN!\n\n[1]: https://en.wikipedia.org/wiki/Logit", "votes": null }, { "id": "573010", "postDate": "07/11/2019 16:48:03", "content": "I knew something was wrong with this kernel! \nI only started learning GANs a few days ago. This kernel confused me. \nThank you for the clarifications! :)
", "rawMarkdown": "I knew something was wrong with [this](https://www.kaggle.com/speedwagon/ralsgan-dogs) kernel! \nI only started learning GANs a few days ago. This kernel confused me. \nThank you for the clarifications! :)", "votes": null }, { "id": "573018", "postDate": "07/11/2019 17:03:14", "content": "", "rawMarkdown": "", "votes": null }, { "id": "573019", "postDate": "07/11/2019 17:04:44", "content": "Excellent . About time someone clarified this :) .Thanks a lot .
\n\nI read it from here .
\n\nhttps://github.com/AlexiaJM/RelativisticGAN
\n\n```
\n\ncriterion = torch.nn.BCELoss()
\n\nerrD_real = criterion(y_pred, y)\nerrD_real.backward()
\n\nerrD_fake = criterion(y_pred_fake, y2)\nerrD_fake.backward()
\n\nerrG = criterion(y_pred_fake, y)\nerrG.backward()
\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()
\n\nerrD = BCE_stable(y_pred - y_pred_fake, y)\nerrD.backward()
\n\nerrG = BCE_stable(y_pred_fake - y_pred, y)\nerrG.backward()
\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()
\n\nerrD = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y) + BCE_stable(y_pred_fake - torch.mean(y_pred), y2))/2\nerrD.backward()
\n\nerrG = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y2) + BCE_stable(y_pred_fake - torch.mean(y_pred), y))/2\nerrG.backward()
\n\nerrD = (torch.mean((y_pred - torch.mean(y_pred_fake) - y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) + y) ** 2))/2\nerrD.backward()
\n\nerrG = (torch.mean((y_pred - torch.mean(y_pred_fake) + y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) - y) ** 2))/2\nerrG.backward()
\n\nerrD = (torch.mean(torch.nn.ReLU()(1.0 - (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 + (y_pred_fake - torch.mean(y_pred)))))/2\nerrD.backward()
\n\nerrG = (torch.mean(torch.nn.ReLU()(1.0 + (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 - (y_pred_fake - torch.mean(y_pred)))))/2\nerrG.backward()\n```
", "rawMarkdown": "Excellent . About time someone clarified this :) .Thanks a lot .\n\nI read it from here .\n\nhttps://github.com/AlexiaJM/RelativisticGAN \n\n```\n### Standard GAN (non-saturating)\n# Use torch.nn.Sigmoid() as last layer in discriminator\n\ncriterion = torch.nn.BCELoss()\n\n# Real data Discriminator loss\nerrD_real = criterion(y_pred, y)\nerrD_real.backward()\n\n# Fake data Discriminator loss\nerrD_fake = criterion(y_pred_fake, y2)\nerrD_fake.backward()\n\n# Generator loss\nerrG = criterion(y_pred_fake, y)\nerrG.backward()\n\n\n### Relativistic Standard GAN\n\n# No sigmoid activation in last layer of discriminator because BCEWithLogitsLoss() already adds it\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()\n\n# Discriminator loss\nerrD = BCE_stable(y_pred - y_pred_fake, y)\nerrD.backward()\n\n# Generator loss (You may want to resample again from real and fake data)\nerrG = BCE_stable(y_pred_fake - y_pred, y)\nerrG.backward()\n\n\n### Relativistic average Standard GAN\n\n# No sigmoid activation in last layer of discriminator because BCEWithLogitsLoss() already adds it\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()\n\n# Discriminator loss\nerrD = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y) + BCE_stable(y_pred_fake - torch.mean(y_pred), y2))/2\nerrD.backward()\n\n# Generator loss (You may want to resample again from real and fake data)\nerrG = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y2) + BCE_stable(y_pred_fake - torch.mean(y_pred), y))/2\nerrG.backward()\n\n\n### Relativistic average LSGAN\n\n# No activation in discriminator\n\n# Discriminator loss\nerrD = (torch.mean((y_pred - torch.mean(y_pred_fake) - y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) + y) ** 2))/2\nerrD.backward()\n\n# Generator loss (You may want to resample again from real and fake data)\nerrG = (torch.mean((y_pred - torch.mean(y_pred_fake) + y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) - y) ** 2))/2\nerrG.backward()\n\n\n### Relativistic average HingeGAN\n\n# No activation in discriminator\n\n# Discriminator loss\nerrD = (torch.mean(torch.nn.ReLU()(1.0 - (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 + (y_pred_fake - torch.mean(y_pred)))))/2\nerrD.backward()\n \n# Generator loss (You may want to resample again from real and fake data)\nerrG = (torch.mean(torch.nn.ReLU()(1.0 + (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 - (y_pred_fake - torch.mean(y_pred)))))/2\nerrG.backward()\n```", "votes": null }, { "id": "573020", "postDate": "07/11/2019 17:06:34", "content": "That's where I originally saw the formula for PyTorch too. For TensorFlow, I wrote it myself.
\n\ndef TF_errD(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) - tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) + tf.ones_like(real))**2,0 ) )/2.\n\ndef TF_errG(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) + tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) - tf.ones_like(real))**2,0 ) )/2.\nThanks, @cdeotte 👍
", "rawMarkdown": "Thanks, @cdeotte 👍", "votes": null }, { "id": "573087", "postDate": "07/11/2019 19:00:52", "content": "Thanks for the explanation
", "rawMarkdown": "Thanks for the explanation", "votes": null }, { "id": "573111", "postDate": "07/11/2019 19:48:49", "content": "Thanks for the clarification @cdeotte .\nI finally caught the dog.... 😅
", "rawMarkdown": "Thanks for the clarification @cdeotte .\nI finally caught the dog.... 😅", "votes": null }, { "id": "573129", "postDate": "07/11/2019 20:21:30", "content": "Awesome. It's super exciting when you see your first dogs being drawn. Congrats.
", "rawMarkdown": "Awesome. It's super exciting when you see your first dogs being drawn. Congrats.", "votes": null }, { "id": "573460", "postDate": "07/12/2019 09:53:54", "content": "Thank you very much @cdeotte for the clear explanation and the TF implementation. Have you found that the RaLSGAN loss definitely outperforms the DCGAN one?
", "rawMarkdown": "Thank you very much @cdeotte for the clear explanation and the TF implementation. Have you found that the RaLSGAN loss definitely outperforms the DCGAN one?", "votes": null }, { "id": "573486", "postDate": "07/12/2019 10:34:46", "content": "I've tried relativistic loss on DCGAN. Not sure if there were any significant improvements but it didn't hurt metric either.
", "rawMarkdown": "I've tried relativistic loss on DCGAN. Not sure if there were any significant improvements but it didn't hurt metric either.", "votes": null }, { "id": "573497", "postDate": "07/12/2019 10:49:02", "content": "The code above is both relativistic average loss (Ra) and least squares loss (LS) combined. For me, RaLS has helped my nets converge faster. I get better results in my 150 epochs. Some participants are managing to perform 1000s of epochs so maybe they don't need RaLS.
\n\n(Note that there is also relativistic loss (R) that doesn't use averages).
", "rawMarkdown": "The code above is both relativistic average loss (Ra) and least squares loss (LS) combined. For me, RaLS has helped my nets converge faster. I get better results in my 150 epochs. Some participants are managing to perform 1000s of epochs so maybe they don't need RaLS.\n\n(Note that there is also relativistic loss (R) that doesn't use averages).", "votes": null }, { "id": "573753", "postDate": "07/12/2019 17:46:35", "content": "Inspired by this post , I modified the starter code a bit and able to implement the RaLSGAN and RaHingeGAN . I will clean up the code and share the Kernel shortly . Below are the images generated and loss graph. I didnt run longer like DCGAN , got score 82 . I am optimistic , that it will be better . Visually , what do you think is better , is there any difference at all ?\n*RaLSGAN generated images *\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2Fd647d5a83c333d281c3f4cf2dc960427%2FRaLSGAN_Sample.png?generation=1562953386528345&alt=media\")
*RaHingeGAN generated images *\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F0efc70d52e17a797ce9dcafad2e54186%2FRaHingeGAN_Sample.png?generation=1562953518238732&alt=media\")
RaLSGAN loss :\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F25e0fda4b4ba9c377814725111a45ff3%2FRaLSGAN_loss.png?generation=1562953553951388&alt=media\")
\nRaHingeGAN loss :\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F3725ccb421815c0f88d827496612ab91%2FRaHingeGAN_loss.png?generation=1562953591095385&alt=media\")
@cdeotte You are so super helpful. Thanks.
", "rawMarkdown": "cdeotte You are so super helpful. Thanks.", "votes": null }, { "id": "579998", "postDate": "07/19/2019 14:21:01", "content": "Hello Chris,\nWanted to know how should be the ideal loss curves look for RaLSGAN G and D.\nThey dont behave as normal GAN's for sure, due to how the loss eqn's are formulated
", "rawMarkdown": "Hello Chris,\nWanted to know how should be the ideal loss curves look for RaLSGAN G and D.\nThey dont behave as normal GAN's for sure, due to how the loss eqn's are formulated", "votes": null }, { "id": "580430", "postDate": "07/20/2019 06:18:06", "content": "Thanks @cdeotte
", "rawMarkdown": "Thanks @cdeotte", "votes": null }, { "id": "582165", "postDate": "07/22/2019 20:36:34", "content": "Can anyone please explain how to implement this loss function in Keras?
", "rawMarkdown": "Can anyone please explain how to implement this loss function in Keras?", "votes": null }, { "id": "582167", "postDate": "07/22/2019 20:44:32", "content": "Code is posted in comments below. Here is a direct link.
\n\nThe function header is TF_errD(fake,real). The variables fake and real are the outputs from your Discriminator network where the last neuron activation is linear, i.e. this loss function takes Logits. Variable fake is array of length BATCH-SIZE when generated images are inputted into Discriminator and real is length BATCH-SIZE of real images inputted.
Thank you @cdeotte
", "rawMarkdown": "Thank you @cdeotte", "votes": null }, { "id": "589268", "postDate": "07/31/2019 17:22:51", "content": "Excellent work. Thanks for sharing !!!
", "rawMarkdown": "Excellent work. Thanks for sharing !!!", "votes": null }, { "id": "597003", "postDate": "08/11/2019 16:34:42", "content": "@cdeotte \nThanks for your sharing, may I know how to implant the loss into keras?\nsorry I am new in keras, here is my naive idea:\n```
\n\nDiscriminator = Sequential()\nDiscriminator.add(Conv2D(32,4,padding='same',strides= 2, input_shape=(64,64,3)))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(64,4,strides= 2, padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(128,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(256,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Dropout(0.3))\nDiscriminator.add(Conv2D(1,4,strides= 1,padding='valid'))\nDiscriminator.compile(loss=TF_errD, optimizer=optimizer_dis, metrics=['accuracy'])\n...
\n\ntrue_label = Discriminator.predict(images) # images is from a batch of real image\nnoise = np.random.normal(0,1,(batch,input_dims))\nsynthesis_img = Generator.predict(noise)\nd_loss = Discriminator.train_on_batch(synthesis_img,true_label)``\n```
\n\nhowever, this does not work...\nthe loss is very large and generate meaningless image.
", "rawMarkdown": "cdeotte \nThanks for your sharing, may I know how to implant the loss into keras?\nsorry I am new in keras, here is my naive idea:\n```\n#discriminator compile as \nDiscriminator = Sequential()\nDiscriminator.add(Conv2D(32,4,padding='same',strides= 2, input_shape=(64,64,3)))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(64,4,strides= 2, padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(128,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(256,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Dropout(0.3))\nDiscriminator.add(Conv2D(1,4,strides= 1,padding='valid'))\nDiscriminator.compile(loss=TF_errD, optimizer=optimizer_dis, metrics=['accuracy'])\n...\n#and then train like this\ntrue_label = Discriminator.predict(images) # images is from a batch of real image\nnoise = np.random.normal(0,1,(batch,input_dims))\nsynthesis_img = Generator.predict(noise)\nd_loss = Discriminator.train_on_batch(synthesis_img,true_label)``\n```\n\nhowever, this does not work...\nthe loss is very large and generate meaningless image.", "votes": null } ], "comments": [ { "id": 573010, "author_name": "timetraveller98", "author_url": "", "post_date": "07/11/2019 16:48:03", "content": "I knew something was wrong with this kernel! \nI only started learning GANs a few days ago. This kernel confused me. \nThank you for the clarifications! :)
", "votes": null, "replies": [ { "id": 573018, "author_name": "phoenix9032", "author_url": "", "post_date": "07/11/2019 17:03:14", "content": "", "votes": null, "replies": [] } ] }, { "id": 573019, "author_name": "phoenix9032", "author_url": "", "post_date": "07/11/2019 17:04:44", "content": "Excellent . About time someone clarified this :) .Thanks a lot .
\n\nI read it from here .
\n\nhttps://github.com/AlexiaJM/RelativisticGAN
\n\n```
\n\ncriterion = torch.nn.BCELoss()
\n\nerrD_real = criterion(y_pred, y)\nerrD_real.backward()
\n\nerrD_fake = criterion(y_pred_fake, y2)\nerrD_fake.backward()
\n\nerrG = criterion(y_pred_fake, y)\nerrG.backward()
\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()
\n\nerrD = BCE_stable(y_pred - y_pred_fake, y)\nerrD.backward()
\n\nerrG = BCE_stable(y_pred_fake - y_pred, y)\nerrG.backward()
\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()
\n\nerrD = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y) + BCE_stable(y_pred_fake - torch.mean(y_pred), y2))/2\nerrD.backward()
\n\nerrG = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y2) + BCE_stable(y_pred_fake - torch.mean(y_pred), y))/2\nerrG.backward()
\n\nerrD = (torch.mean((y_pred - torch.mean(y_pred_fake) - y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) + y) ** 2))/2\nerrD.backward()
\n\nerrG = (torch.mean((y_pred - torch.mean(y_pred_fake) + y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) - y) ** 2))/2\nerrG.backward()
\n\nerrD = (torch.mean(torch.nn.ReLU()(1.0 - (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 + (y_pred_fake - torch.mean(y_pred)))))/2\nerrD.backward()
\n\nerrG = (torch.mean(torch.nn.ReLU()(1.0 + (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 - (y_pred_fake - torch.mean(y_pred)))))/2\nerrG.backward()\n```
", "votes": null, "replies": [ { "id": 573020, "author_name": "cdeotte", "author_url": "", "post_date": "07/11/2019 17:06:34", "content": "That's where I originally saw the formula for PyTorch too. For TensorFlow, I wrote it myself.
\n\ndef TF_errD(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) - tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) + tf.ones_like(real))**2,0 ) )/2.\n\ndef TF_errG(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) + tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) - tf.ones_like(real))**2,0 ) )/2.\n@cdeotte You are so super helpful. Thanks.
", "votes": null, "replies": [] }, { "id": 597003, "author_name": "leolewis", "author_url": "", "post_date": "08/11/2019 16:34:42", "content": "@cdeotte \nThanks for your sharing, may I know how to implant the loss into keras?\nsorry I am new in keras, here is my naive idea:\n```
\n\nDiscriminator = Sequential()\nDiscriminator.add(Conv2D(32,4,padding='same',strides= 2, input_shape=(64,64,3)))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(64,4,strides= 2, padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(128,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(256,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Dropout(0.3))\nDiscriminator.add(Conv2D(1,4,strides= 1,padding='valid'))\nDiscriminator.compile(loss=TF_errD, optimizer=optimizer_dis, metrics=['accuracy'])\n...
\n\ntrue_label = Discriminator.predict(images) # images is from a batch of real image\nnoise = np.random.normal(0,1,(batch,input_dims))\nsynthesis_img = Generator.predict(noise)\nd_loss = Discriminator.train_on_batch(synthesis_img,true_label)``\n```
\n\nhowever, this does not work...\nthe loss is very large and generate meaningless image.
", "votes": null, "replies": [] } ] }, { "id": 573039, "author_name": "coolcoder22", "author_url": "", "post_date": "07/11/2019 17:52:30", "content": "Thanks, @cdeotte 👍
", "votes": null, "replies": [] }, { "id": 573087, "author_name": "khursani8", "author_url": "", "post_date": "07/11/2019 19:00:52", "content": "Thanks for the explanation
", "votes": null, "replies": [] }, { "id": 573111, "author_name": "jolasa", "author_url": "", "post_date": "07/11/2019 19:48:49", "content": "Thanks for the clarification @cdeotte .\nI finally caught the dog.... 😅
", "votes": null, "replies": [ { "id": 573129, "author_name": "cdeotte", "author_url": "", "post_date": "07/11/2019 20:21:30", "content": "Awesome. It's super exciting when you see your first dogs being drawn. Congrats.
", "votes": null, "replies": [] } ] }, { "id": 573460, "author_name": "lucamassaron", "author_url": "", "post_date": "07/12/2019 09:53:54", "content": "Thank you very much @cdeotte for the clear explanation and the TF implementation. Have you found that the RaLSGAN loss definitely outperforms the DCGAN one?
", "votes": null, "replies": [ { "id": 573486, "author_name": "dvorobiev", "author_url": "", "post_date": "07/12/2019 10:34:46", "content": "I've tried relativistic loss on DCGAN. Not sure if there were any significant improvements but it didn't hurt metric either.
", "votes": null, "replies": [] }, { "id": 573497, "author_name": "cdeotte", "author_url": "", "post_date": "07/12/2019 10:49:02", "content": "The code above is both relativistic average loss (Ra) and least squares loss (LS) combined. For me, RaLS has helped my nets converge faster. I get better results in my 150 epochs. Some participants are managing to perform 1000s of epochs so maybe they don't need RaLS.
\n\n(Note that there is also relativistic loss (R) that doesn't use averages).
", "votes": null, "replies": [] } ] }, { "id": 573753, "author_name": "phoenix9032", "author_url": "", "post_date": "07/12/2019 17:46:35", "content": "Inspired by this post , I modified the starter code a bit and able to implement the RaLSGAN and RaHingeGAN . I will clean up the code and share the Kernel shortly . Below are the images generated and loss graph. I didnt run longer like DCGAN , got score 82 . I am optimistic , that it will be better . Visually , what do you think is better , is there any difference at all ?\n*RaLSGAN generated images *\n
*RaHingeGAN generated images *\n
RaLSGAN loss :\n\nRaHingeGAN loss :\n
Hello Chris,\nWanted to know how should be the ideal loss curves look for RaLSGAN G and D.\nThey dont behave as normal GAN's for sure, due to how the loss eqn's are formulated
", "votes": null, "replies": [] }, { "id": 580430, "author_name": "akhileshrai", "author_url": "", "post_date": "07/20/2019 06:18:06", "content": "Thanks @cdeotte
", "votes": null, "replies": [] }, { "id": 582165, "author_name": "techytushar", "author_url": "", "post_date": "07/22/2019 20:36:34", "content": "Can anyone please explain how to implement this loss function in Keras?
", "votes": null, "replies": [ { "id": 582167, "author_name": "cdeotte", "author_url": "", "post_date": "07/22/2019 20:44:32", "content": "Code is posted in comments below. Here is a direct link.
\n\nThe function header is TF_errD(fake,real). The variables fake and real are the outputs from your Discriminator network where the last neuron activation is linear, i.e. this loss function takes Logits. Variable fake is array of length BATCH-SIZE when generated images are inputted into Discriminator and real is length BATCH-SIZE of real images inputted.
Thank you @cdeotte
", "votes": null, "replies": [] } ] }, { "id": 589268, "author_name": "suneelpatel", "author_url": "", "post_date": "07/31/2019 17:22:51", "content": "Excellent work. Thanks for sharing !!!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "573007": "# Loss Functions\nThere are currently no Kaggle kernels that use RaLSGAN. A few kernels have RaLSGAN in their name, but they are using basic DCGAN. **What is RaLSGAN?** RaLSGAN is just basic GAN with a different loss function.\n\n\n\nA Discriminator can either output `sigmoid` activation or `linear` activation. If sigmoid, we have the probability of an image being real, `Pr(Real)`. If linear, we have the [log odds][1] of being real, `Logit`. Probabilities are between 0 and 1 inclusive. And Logits can be any number. A positive logit implies image is real, a negative logit implies image is fake.\n# Simple Loss\nDefine `x_r` to be a real image, and `x_f` to be a fake image. Define `D(x) = Pr(Real)` and `C(x) = Logit` to be the two different outputs from a Discriminator when you input an image. The simplest possible loss is below. \n \n # Take AVG over x_r and x_f in batch\n disc_loss = (1 - D(x_r)) + (D(x_f) - 0)\n gen_loss = (1 - D(x_f))\n\nMinimizing these losses says the following. The Discriminator wants `D(x_r)=1` and `D(x_f)=0`, i.e. the Discriminator wants real images to have probably 1 of being real. And wants fake images to have probability 0 of being real. The Generator wants `D(x_f)=1`, i.e. the Generator wants fakes images to have probability 1 of being real.\n \n# DCGAN Loss\nBasic GAN or DCGAN use the following loss functions. Note that they use `D(x)`, i.e. probs.\n\n # Take AVG over x_r and x_f in batch\n disc_loss = -log (D(x_r)) - log (1-D(x_f))\n gen_loss = -log (D(x_f))\n\n# RaLSGAN Loss\nRaLSGAN uses the following loss functions. Note that they use `C(x)`, i.e. logits.\n\n # Take AVG over x_r and x_f in batch\n disc_loss = (C(x_r) - AVG(C(x_f)) - 1)^2 + (C(x_f) - AVG(C(x_r)) + 1)^2\n gen_loss = (C(x_r) - AVG(C(x_f)) + 1)^2 + (C(x_f) - AVG(C(x_r)) - 1)^2\n\n# Code\nHere's how to update the Kaggle kernel starter code. First, if you fork Kaggle's Starter Code, **turn GPU on**! Second, change the last line of the Discriminator to output linear activation (Logit) by removing sigmoid\n\n #x = torch.sigmoid(self.conv5(x))\n x = self.conv5(x)\n\nNext, change the following\n\n ############################\n # (1) Update D network\n ###########################\n netD.zero_grad()\n real_images = real_images.to(device)\n batch_size = real_images.size(0)\n labels = torch.full((batch_size, 1), real_label, device=device)\n outputR = netD(real_images)\n noise = torch.randn(batch_size, nz, 1, 1, device=device)\n fake = netG(noise)\n outputF = netD(fake.detach())\n errD = (torch.mean((outputR - torch.mean(outputF) - labels) ** 2) + \n torch.mean((outputF - torch.mean(outputR) + labels) ** 2))/2\n errD.backward(retain_graph=True)\n optimizerD.step()\n ############################\n # (2) Update G network\n ###########################\n netG.zero_grad()\n outputF = netD(fake) \n errG = (torch.mean((outputR - torch.mean(outputF) + labels) ** 2) +\n torch.mean((outputF - torch.mean(outputR) - labels) ** 2))/2\n errG.backward()\n optimizerG.step()\n\n# Image Display\nI notice that many kernels have dark images. This is because the image pixels have been normalized to `[ -1, 1 ]`. If you want nice looking images, then un-normalize them with `plt.imshow((img+1.)/2.)`.\n\n# Have Fun\nHave fun and enjoy the power of RaLSGAN!\n\n[1]: https://en.wikipedia.org/wiki/Logit", "573010": "I knew something was wrong with [this](https://www.kaggle.com/speedwagon/ralsgan-dogs) kernel! \nI only started learning GANs a few days ago. This kernel confused me. \nThank you for the clarifications! :)", "573018": "", "573019": "Excellent . About time someone clarified this :) .Thanks a lot .\n\nI read it from here .\n\nhttps://github.com/AlexiaJM/RelativisticGAN \n\n```\n### Standard GAN (non-saturating)\n# Use torch.nn.Sigmoid() as last layer in discriminator\n\ncriterion = torch.nn.BCELoss()\n\n# Real data Discriminator loss\nerrD_real = criterion(y_pred, y)\nerrD_real.backward()\n\n# Fake data Discriminator loss\nerrD_fake = criterion(y_pred_fake, y2)\nerrD_fake.backward()\n\n# Generator loss\nerrG = criterion(y_pred_fake, y)\nerrG.backward()\n\n\n### Relativistic Standard GAN\n\n# No sigmoid activation in last layer of discriminator because BCEWithLogitsLoss() already adds it\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()\n\n# Discriminator loss\nerrD = BCE_stable(y_pred - y_pred_fake, y)\nerrD.backward()\n\n# Generator loss (You may want to resample again from real and fake data)\nerrG = BCE_stable(y_pred_fake - y_pred, y)\nerrG.backward()\n\n\n### Relativistic average Standard GAN\n\n# No sigmoid activation in last layer of discriminator because BCEWithLogitsLoss() already adds it\n\nBCE_stable = torch.nn.BCEWithLogitsLoss()\n\n# Discriminator loss\nerrD = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y) + BCE_stable(y_pred_fake - torch.mean(y_pred), y2))/2\nerrD.backward()\n\n# Generator loss (You may want to resample again from real and fake data)\nerrG = ((BCE_stable(y_pred - torch.mean(y_pred_fake), y2) + BCE_stable(y_pred_fake - torch.mean(y_pred), y))/2\nerrG.backward()\n\n\n### Relativistic average LSGAN\n\n# No activation in discriminator\n\n# Discriminator loss\nerrD = (torch.mean((y_pred - torch.mean(y_pred_fake) - y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) + y) ** 2))/2\nerrD.backward()\n\n# Generator loss (You may want to resample again from real and fake data)\nerrG = (torch.mean((y_pred - torch.mean(y_pred_fake) + y) ** 2) + torch.mean((y_pred_fake - torch.mean(y_pred) - y) ** 2))/2\nerrG.backward()\n\n\n### Relativistic average HingeGAN\n\n# No activation in discriminator\n\n# Discriminator loss\nerrD = (torch.mean(torch.nn.ReLU()(1.0 - (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 + (y_pred_fake - torch.mean(y_pred)))))/2\nerrD.backward()\n \n# Generator loss (You may want to resample again from real and fake data)\nerrG = (torch.mean(torch.nn.ReLU()(1.0 + (y_pred - torch.mean(y_pred_fake)))) + torch.mean(torch.nn.ReLU()(1.0 - (y_pred_fake - torch.mean(y_pred)))))/2\nerrG.backward()\n```", "573020": "That's where I originally saw the formula for PyTorch too. For TensorFlow, I wrote it myself.\n\n def TF_errD(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) - tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) + tf.ones_like(real))**2,0 ) )/2.\n\n def TF_errG(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) + tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) - tf.ones_like(real))**2,0 ) )/2.", "573039": "Thanks, @cdeotte 👍", "573087": "Thanks for the explanation", "573111": "Thanks for the clarification @cdeotte .\nI finally caught the dog.... 😅", "573129": "Awesome. It's super exciting when you see your first dogs being drawn. Congrats.", "573460": "Thank you very much @cdeotte for the clear explanation and the TF implementation. Have you found that the RaLSGAN loss definitely outperforms the DCGAN one?", "573486": "I've tried relativistic loss on DCGAN. Not sure if there were any significant improvements but it didn't hurt metric either.", "573497": "The code above is both relativistic average loss (Ra) and least squares loss (LS) combined. For me, RaLS has helped my nets converge faster. I get better results in my 150 epochs. Some participants are managing to perform 1000s of epochs so maybe they don't need RaLS.\n\n(Note that there is also relativistic loss (R) that doesn't use averages).", "573753": "Inspired by this post , I modified the starter code a bit and able to implement the RaLSGAN and RaHingeGAN . I will clean up the code and share the Kernel shortly . Below are the images generated and loss graph. I didnt run longer like DCGAN , got score 82 . I am optimistic , that it will be better . Visually , what do you think is better , is there any difference at all ?\n**RaLSGAN generated images **\n\n\n**RaHingeGAN generated images **\n\n\n**RaLSGAN loss :**\n\n**RaHingeGAN loss :**\n", "574501": "cdeotte You are so super helpful. Thanks.", "579998": "Hello Chris,\nWanted to know how should be the ideal loss curves look for RaLSGAN G and D.\nThey dont behave as normal GAN's for sure, due to how the loss eqn's are formulated", "580430": "Thanks @cdeotte", "582165": "Can anyone please explain how to implement this loss function in Keras?", "582167": "Code is posted in comments below. [Here][1] is a direct link.\n\nThe function header is `TF_errD(fake,real)`. The variables `fake` and `real` are the outputs from your Discriminator network where the last neuron activation is `linear`, i.e. this loss function takes `Logits`. Variable `fake` is array of length BATCH-SIZE when generated images are inputted into Discriminator and `real` is length BATCH-SIZE of real images inputted.\n\n[1]: https://www.kaggle.com/c/generative-dog-images/discussion/99485#573020", "582535": "Thank you @cdeotte", "589268": "Excellent work. Thanks for sharing !!!", "597003": "cdeotte \nThanks for your sharing, may I know how to implant the loss into keras?\nsorry I am new in keras, here is my naive idea:\n```\n#discriminator compile as \nDiscriminator = Sequential()\nDiscriminator.add(Conv2D(32,4,padding='same',strides= 2, input_shape=(64,64,3)))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(64,4,strides= 2, padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(128,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Conv2D(256,4,strides= 2,padding='same'))\nDiscriminator.add(LeakyReLU(alpha=0.2))\nDiscriminator.add(Dropout(0.3))\nDiscriminator.add(Conv2D(1,4,strides= 1,padding='valid'))\nDiscriminator.compile(loss=TF_errD, optimizer=optimizer_dis, metrics=['accuracy'])\n...\n#and then train like this\ntrue_label = Discriminator.predict(images) # images is from a batch of real image\nnoise = np.random.normal(0,1,(batch,input_dims))\nsynthesis_img = Generator.predict(noise)\nd_loss = Discriminator.train_on_batch(synthesis_img,true_label)``\n```\n\nhowever, this does not work...\nthe loss is very large and generate meaningless image." }, "source": "meta" }