![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F206941851c1e8bd7caa6a1e9566ddc3b%2Fdcgan.png?generation=1562478320280603&alt=media\")
Hi,
\n\nI am very new to GANs and kaggle competitions in general. I want to at least generate something that looks somewhat like a dog but all I get are similar pixelated pictures for every epoch. There is probably a lot missing from my code that I didn't do properly but if anyone could take a look at my kernel and start pointing me in the correct direction for some fixes that would helpful.
\n\nI set it to 2 epoch so I can commit, but changing it to 50 will still give same results for the images.
\n\nThanks!
\n\n", "messages": [ { "id": "569637", "postDate": "07/07/2019 04:30:03", "content": "Hi,
\n\nI am very new to GANs and kaggle competitions in general. I want to at least generate something that looks somewhat like a dog but all I get are similar pixelated pictures for every epoch. There is probably a lot missing from my code that I didn't do properly but if anyone could take a look at my kernel and start pointing me in the correct direction for some fixes that would helpful.
\n\nI set it to 2 epoch so I can commit, but changing it to 50 will still give same results for the images.
\n\nThanks!
\n\n", "rawMarkdown": "Hi,\n\nI am very new to GANs and kaggle competitions in general. I want to at least generate something that looks somewhat like a dog but all I get are similar pixelated pictures for every epoch. There is probably a lot missing from my code that I didn't do properly but if anyone could take a look at my kernel and start pointing me in the correct direction for some fixes that would helpful.\n\nI set it to 2 epoch so I can commit, but changing it to 50 will still give same results for the images.\n\nThanks!\n\n[my kernel](https://www.kaggle.com/cmalla94/dcgan-generating-dog-images-with-tensorflow?scriptVersionId=16836469)", "votes": null }, { "id": "569646", "postDate": "07/07/2019 05:09:11", "content": "Did you try for 50 epochs ?
", "rawMarkdown": "Did you try for 50 epochs ?", "votes": null }, { "id": "569653", "postDate": "07/07/2019 05:28:52", "content": "Yeah I set it to 50 epochs, let it run but the images end up looking the same.
", "rawMarkdown": "Yeah I set it to 50 epochs, let it run but the images end up looking the same.", "votes": null }, { "id": "569661", "postDate": "07/07/2019 05:47:33", "content": "hi Chad ,
\n\nI am also a newbee , but learning from the existing Kernels from experts
\n\n
Please refer to this architecture for DCGAN and validate
\n\nNormally ReLU can be used in Generator .
\n\nPlease verifiy the output activation for discriminator , can it be changed to sigmoid ?
", "rawMarkdown": "hi Chad ,\n\nI am also a newbee , but learning from the existing Kernels from experts \n\n\n\nPlease refer to this architecture for DCGAN and validate \n\nNormally ReLU can be used in Generator .\n\nPlease verifiy the output activation for discriminator , can it be changed to sigmoid ?", "votes": null }, { "id": "569676", "postDate": "07/07/2019 06:20:22", "content": "I tried both activations, sigmoid and tanh but same results. I believe the functions are just inverses (9 for sigmoid gives 1 for tanh and vice versa).
\n\nThe actual networks I followed closely to @jesucristo 's kernal but I feel the problem lies with my training loop.
\n\nI just tried training now and I get this error after 14 epoch:
\n\nResourceExhaustedError: OOM when allocating tensor with shape[64,32,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:FusedBatchNormGrad]
", "rawMarkdown": "I tried both activations, sigmoid and tanh but same results. I believe the functions are just inverses (9 for sigmoid gives 1 for tanh and vice versa). \n\nThe actual networks I followed closely to @jesucristo 's kernal but I feel the problem lies with my training loop. \n\nI just tried training now and I get this error after 14 epoch: \n\nResourceExhaustedError: OOM when allocating tensor with shape[64,32,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:FusedBatchNormGrad]", "votes": null }, { "id": "569688", "postDate": "07/07/2019 06:56:03", "content": "I forked your kernel and trying to do modification and run. \nI will get back if I am able to crack . I also follow Nanashi kernels.\nIf you find a solution before me ,please let me know .
", "rawMarkdown": "I forked your kernel and trying to do modification and run. \nI will get back if I am able to crack . I also follow Nanashi kernels.\nIf you find a solution before me ,please let me know .", "votes": null }, { "id": "569696", "postDate": "07/07/2019 07:19:53", "content": "Thank you Nirjhar and yes I will let you know
", "rawMarkdown": "Thank you Nirjhar and yes I will let you know", "votes": null }, { "id": "569825", "postDate": "07/07/2019 12:10:52", "content": "Hi Chad, my first observation is that your generated images take on pixel values between -1 and 1 because your Generator outputs with tanh and the original images are normalized as image /= 255.0 which take on pixel values between 0 and 1. Therefore the job of your Discriminator is too easy. Your Discriminator will simply say If I see a pixel less than 0 then it is fake. Try normalizing your images to image = (image - 127.5) / 127.5 like the TensorFlow tutorial here. (Then change your image shows to plt.imshow((images[image]+1.)/2.) )
Next, the TensorFlow tutorial has the Discriminator output with a linear neuron. When they write model.add(layers.Dense(1)). The absence of an activation causes the neuron to have a linear neuron. This lets your Discriminator output any negative and positive value. I suspect this helps the cost function have better derivatives. Try changing the last line of your Discriminator to tf.keras.layers.Dense(1, activation='linear')
The reason that your Discriminator should output linear is because your loss is defined as cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True). The parameter from_logits=True expects the Discriminator output to be all real numbers (it expects logits instead of probs) instead of just the limited numbers from a sigmoid or tanh.
(Perhaps if you set from_logits=False you can use a sigmoid output to your Discriminator).
Hi Chris,
\n\nThanks for the help so far. I changed the normalization but I am wondering why 127.5 and not 255.0? I will try using linear activation but if it doesn't improve will change to sigmoid the way you said.
\n\nAlso I think I didn't batch my dataset correctly so I'm getting the ResourceExhausted. I will debug that and see if anything changes.
", "rawMarkdown": "Hi Chris, \n\nThanks for the help so far. I changed the normalization but I am wondering why 127.5 and not 255.0? I will try using linear activation but if it doesn't improve will change to sigmoid the way you said. \n\nAlso I think I didn't batch my dataset correctly so I'm getting the ResourceExhausted. I will debug that and see if anything changes.", "votes": null }, { "id": "570034", "postDate": "07/07/2019 17:36:31", "content": "The original images have pixel values of 0, 1, 2, ..., 253, 254, 255. After applying the formula image = (image - 127.5) / 127.5, the values approximately become -1.0, -0.99, -0.98,... 0.98, 0.99, 1.0. So that formula maps the values from [0,255] to [-1,1]. And tanh outputs values from [-1,1]. Therefore the original images and the generator images will be using the same pixel values.
I've been messing with your code. I've gotten it to make blobs of color in various shapes but I haven't got the dog faces to show up yet. I'm currently working with a slightly simpler generator and discriminator architecture.
", "rawMarkdown": "The original images have pixel values of `0, 1, 2, ..., 253, 254, 255`. After applying the formula `image = (image - 127.5) / 127.5`, the values approximately become `-1.0, -0.99, -0.98,... 0.98, 0.99, 1.0`. So that formula maps the values from [0,255] to [-1,1]. And `tanh` outputs values from [-1,1]. Therefore the original images and the generator images will be using the same pixel values.\n\nI've been messing with your code. I've gotten it to make blobs of color in various shapes but I haven't got the dog faces to show up yet. I'm currently working with a slightly simpler generator and discriminator architecture.", "votes": null }, { "id": "570036", "postDate": "07/07/2019 17:42:33", "content": "What changes did you make to the code and what epoch were you able to run?
", "rawMarkdown": "What changes did you make to the code and what epoch were you able to run?", "votes": null }, { "id": "570047", "postDate": "07/07/2019 17:56:09", "content": "I'm running 50 epochs. I'm not getting anything great yet, but it is better than blue and yellow dots/lines like you had. I'm getting blobs of yellow and blue shapes now. The two big changes are adding image = (image - 127.5) / 127.5 in preprocessing and activation='linear' for Discriminator.
Next I crop the images using the bounding boxes to just dogs and use antialiasing (from PIL package, can't find antialiasing in tf 1.13.1) to resize to 64x64. This makes it easier for the GAN to generalize. (I think). Next I'm using
def make_generator():\nmodel = tf.keras.Sequential()\nmodel.add(layers.Dense(4*4*512, use_bias=False, input_shape=(100,)))\nmodel.add(layers.Reshape((4, 4, 512)))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(256, (5, 5), strides=(2, 2), padding='same', use_bias=False))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(128, (5, 5), strides=(2, 2), padding='same', use_bias=False))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(3, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))\n\nreturn model\nand
\n\ndef make_discriminator():\nmodel = tf.keras.Sequential()\nmodel.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same',\n input_shape=[64, 64, 3]))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Conv2D(256, (5, 5), strides=(2, 2), padding='same'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Conv2D(512, (5, 5), strides=(2, 2), padding='same'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(1, activation='linear')) #output logits\n\nreturn model\nIt still needs more work to make anything resembling dogs. I removed the autotune. That was causing the kernel to error. Now I'm experimenting with adjusting batch_sizes, learning rates, adjusting ratio of D updates versus G updates, and using a Gaussian seed instead of uniform, etc.
\n\nI'm new to GANs myself, so I'm not sure exactly what needs to be changed so it learns to draw dogs from scratch quickly.
", "rawMarkdown": "I'm running 50 epochs. I'm not getting anything great yet, but it is better than blue and yellow dots/lines like you had. I'm getting blobs of yellow and blue shapes now. The two big changes are adding `image = (image - 127.5) / 127.5` in preprocessing and `activation='linear'` for Discriminator. \n\nNext I crop the images using the bounding boxes to just dogs and use `antialiasing` (from PIL package, can't find antialiasing in tf 1.13.1) to resize to 64x64. This makes it easier for the GAN to generalize. (I think). Next I'm using\n\n def make_generator():\n model = tf.keras.Sequential()\n model.add(layers.Dense(4*4*512, use_bias=False, input_shape=(100,)))\n model.add(layers.Reshape((4, 4, 512)))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n\n model.add(layers.Conv2DTranspose(256, (5, 5), strides=(2, 2), padding='same', use_bias=False))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n\n model.add(layers.Conv2DTranspose(128, (5, 5), strides=(2, 2), padding='same', use_bias=False))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n \n model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n \n model.add(layers.Conv2DTranspose(3, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))\n\n return model\n\n\nand \n\n def make_discriminator():\n model = tf.keras.Sequential()\n model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same',\n input_shape=[64, 64, 3]))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n model.add(layers.Dropout(0.5))\n\n model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n model.add(layers.Dropout(0.5))\n \n model.add(layers.Conv2D(256, (5, 5), strides=(2, 2), padding='same'))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n model.add(layers.Dropout(0.5))\n \n model.add(layers.Conv2D(512, (5, 5), strides=(2, 2), padding='same'))\n model.add(layers.BatchNormalization())\n model.add(layers.LeakyReLU())\n model.add(layers.Dropout(0.5))\n\n model.add(layers.Flatten())\n model.add(layers.Dense(1, activation='linear')) #output logits\n\n return model\n\nIt still needs more work to make anything resembling dogs. I removed the autotune. That was causing the kernel to error. Now I'm experimenting with adjusting batch_sizes, learning rates, adjusting ratio of D updates versus G updates, and using a Gaussian seed instead of uniform, etc.\n\nI'm new to GANs myself, so I'm not sure exactly what needs to be changed so it learns to draw dogs from scratch quickly.", "votes": null }, { "id": "570186", "postDate": "07/08/2019 01:05:10", "content": "I've made more progress. The autotune is causing your kernel to crash ds = ds.prefetch(buffer_size=AUTOTUNE). Next, your kernel only plots the Red layer of an image with plt.imshow( (generated_image[0, :, :, 0]+1.)/2. ) and this causes matplotlib to display a blue yellow map. You need to turn that last 0 into a colon as plt.imshow( (generated_image[0, :, :, :]+1.)/2. ) then you will see RGB images. Finally you should plot the disc_loss and gen_loss and make sure they stay equal all the time (by adjusting learning rates). I've found that setting the Generator's Adam optimizer stronger helps keep the two in sync.
Oh my, wow thank you so much for your help Chris.
\n\nI am currently changing my code to use your bounding code from your kernel . Will make the other changes as well and let you know what has changed.
\n\nThank you again, really helpful and motivated me to not to give up on my first competition.
", "rawMarkdown": "Oh my, wow thank you so much for your help Chris. \n\nI am currently changing my code to use your bounding code from your [kernel](https://www.kaggle.com/cdeotte/supervised-generative-dog-net) . Will make the other changes as well and let you know what has changed. \n\nThank you again, really helpful and motivated me to not to give up on my first competition.", "votes": null }, { "id": "570246", "postDate": "07/08/2019 03:44:54", "content": "Here it is after 50 epochs (31000 batches) (1 hour run). In addition to what we discussed above, I added all the GAN hacks from here. I'll post images after 9 hours of running tomorrow. \n![\"image\"](\"http://playagricola.com/Kaggle/dogs3.png\")
\n(Each epoch is approx 625 batches of size 32. The 50th epoch has mean gen_loss = 1.39 and std gen_loss = 0.553 per batch. And mean disc_loss = 0.98 and std disc_loss = 0.208 per batch).
Hey Chris , so I have removed the line for prefetching but I am still getting the resource exhausted error.
\n\n'OOM when allocating tensor with shape[64,64,64,64]'
\n\nThat was after 10 epoch, now I am getting shapes so that's good. Is there anything I have configure for the kaggle kernel to manage resources?
\n\nThis is my updated kernel
", "rawMarkdown": "Hey Chris , so I have removed the line for prefetching but I am still getting the resource exhausted error.\n\n'OOM when allocating tensor with shape[64,64,64,64]'\n\nThat was after 10 epoch, now I am getting shapes so that's good. Is there anything I have configure for the kaggle kernel to manage resources?\n\nThis is my updated [kernel](https://www.kaggle.com/cmalla94/dcgan-generating-dog-images-with-tensorflow?scriptVersionId=16877018)", "votes": null }, { "id": "570552", "postDate": "07/08/2019 12:50:30", "content": "I'm not sure. Let me think. One thing I noticed is that you are including some blank images. After using my dog crop code only the first idxIn elements are images so you should use imagesIn = tf.cast(imagesIn[:idxIn,:,:,:], 'float32').
It's exciting that you're getting shapes, that's a great improvement.
", "rawMarkdown": "I'm not sure. Let me think. One thing I noticed is that you are including some blank images. After using my dog crop code only the first `idxIn` elements are images so you should use `imagesIn = tf.cast(imagesIn[:idxIn,:,:,:], 'float32')`. \n\nIt's exciting that you're getting shapes, that's a great improvement.", "votes": null }, { "id": "570568", "postDate": "07/08/2019 13:24:02", "content": "I have a feeling it is because of tf.enable_eager_execution(). If I don't enable eager execution then when it gets to the code block for
imagesIn = tf.cast(imagesIn[:idxIn, : , : , :], 'float32')or \nds = tf.data.Dataset.from_tensor_slices(imagesIn).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\nI get the error ValueError: Cannot create a tensor proto whose content is larger than 2GB. . Is there workaround for this to cast and make batches from the imagesIn without using tf.enable_eager_execution()?
Good question, let me experiment.
", "rawMarkdown": "Good question, let me experiment.", "votes": null }, { "id": "570597", "postDate": "07/08/2019 14:16:34", "content": "I found the problem. Inside your function def train_step(images), you have the line fake_pictures.append(generated_images) which saves fake images to the global variable fake_pictures. Each epoch it saves 1GB of data to RAM. You need to remove or change that.
If you remove the two instances of display.clear_output(wait=True) inside your train() function then images will get drawn to the screen (without overwriting each other) and you won't need to save them all to see them.
Ahh, okay I removed that line and running it now. By the way, did the 9 hour training finish for you?
", "rawMarkdown": "Ahh, okay I removed that line and running it now. By the way, did the 9 hour training finish for you?", "votes": null }, { "id": "570705", "postDate": "07/08/2019 17:14:21", "content": "Yes. I ran my code for 7 hours. It did 300 epochs (approx 200000 batches). Here's the output:\n![\"image\"](\"http://playagricola.com/Kaggle/dogs7819.png\")
\nThe means of Gen_loss and Disc_loss stayed near 1.0 the whole time, that's good. So I could probably train it longer if I can speed up the code (to finish under 9 hours).
Great results! I am at 40 epoch now without any errors so that line was the culprit. Thanks very much for catching. Next I will apply the previously mentioned optimizations and from that the kernel you mentioned.
", "rawMarkdown": "Great results! I am at 40 epoch now without any errors so that line was the culprit. Thanks very much for catching. Next I will apply the previously mentioned optimizations and from that the kernel you mentioned.", "votes": null }, { "id": "570713", "postDate": "07/08/2019 17:24:01", "content": "@cdeotte Did you use the same architecture as posted in the thread ? I read in some place to add dropout only to generator . I had some weird result in adding dropout in discriminator .
", "rawMarkdown": "cdeotte Did you use the same architecture as posted in the thread ? I read in some place to add dropout only to generator . I had some weird result in adding dropout in discriminator .", "votes": null }, { "id": "570714", "postDate": "07/08/2019 17:25:58", "content": "Yes, those pictures come from the above architectures. Currently my generator has no dropout. I also heard about the recommendation to add dropout to the generator both during training and image generation. I'll try that.
", "rawMarkdown": "Yes, those pictures come from the above architectures. Currently my generator has no dropout. I also heard about the recommendation to add dropout to the generator both during training and image generation. I'll try that.", "votes": null }, { "id": "570717", "postDate": "07/08/2019 17:30:23", "content": "I am currently running one with Dropout(0.5) in generator . Without Dropout it gave 100 FID . Lets see if it improves , or cracks ...
", "rawMarkdown": "I am currently running one with Dropout(0.5) in generator . Without Dropout it gave 100 FID . Lets see if it improves , or cracks ...", "votes": null }, { "id": "571084", "postDate": "07/09/2019 06:25:59", "content": "Which lr are you using on Generator and Discriminator?
", "rawMarkdown": "Which lr are you using on Generator and Discriminator?", "votes": null }, { "id": "571103", "postDate": "07/09/2019 07:00:38", "content": "Unfortunately it didnt give better results with dropout . But i am not completely discarding it . Might be few other reasons as well
", "rawMarkdown": "Unfortunately it didnt give better results with dropout . But i am not completely discarding it . Might be few other reasons as well", "votes": null }, { "id": "571256", "postDate": "07/09/2019 11:40:55", "content": "> Which lr are you using on Generator and Discriminator?
\n\nI'm using Adam with lr=0.0002 and beta1=0.5 in both G and D with smoothed positive and negative labels. This and other tricks are recommended from GAN hacks here. (And brought to my attention by Luca in this thread).
Did you try for 50 epochs ?
", "votes": null, "replies": [ { "id": 569653, "author_name": "cmalla94", "author_url": "", "post_date": "07/07/2019 05:28:52", "content": "Yeah I set it to 50 epochs, let it run but the images end up looking the same.
", "votes": null, "replies": [] } ] }, { "id": 569661, "author_name": "phoenix9032", "author_url": "", "post_date": "07/07/2019 05:47:33", "content": "hi Chad ,
\n\nI am also a newbee , but learning from the existing Kernels from experts
\n\n
Please refer to this architecture for DCGAN and validate
\n\nNormally ReLU can be used in Generator .
\n\nPlease verifiy the output activation for discriminator , can it be changed to sigmoid ?
", "votes": null, "replies": [ { "id": 569676, "author_name": "cmalla94", "author_url": "", "post_date": "07/07/2019 06:20:22", "content": "I tried both activations, sigmoid and tanh but same results. I believe the functions are just inverses (9 for sigmoid gives 1 for tanh and vice versa).
\n\nThe actual networks I followed closely to @jesucristo 's kernal but I feel the problem lies with my training loop.
\n\nI just tried training now and I get this error after 14 epoch:
\n\nResourceExhaustedError: OOM when allocating tensor with shape[64,32,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:FusedBatchNormGrad]
", "votes": null, "replies": [] }, { "id": 569688, "author_name": "phoenix9032", "author_url": "", "post_date": "07/07/2019 06:56:03", "content": "I forked your kernel and trying to do modification and run. \nI will get back if I am able to crack . I also follow Nanashi kernels.\nIf you find a solution before me ,please let me know .
", "votes": null, "replies": [] }, { "id": 569696, "author_name": "cmalla94", "author_url": "", "post_date": "07/07/2019 07:19:53", "content": "Thank you Nirjhar and yes I will let you know
", "votes": null, "replies": [] } ] }, { "id": 569825, "author_name": "cdeotte", "author_url": "", "post_date": "07/07/2019 12:10:52", "content": "Hi Chad, my first observation is that your generated images take on pixel values between -1 and 1 because your Generator outputs with tanh and the original images are normalized as image /= 255.0 which take on pixel values between 0 and 1. Therefore the job of your Discriminator is too easy. Your Discriminator will simply say If I see a pixel less than 0 then it is fake. Try normalizing your images to image = (image - 127.5) / 127.5 like the TensorFlow tutorial here. (Then change your image shows to plt.imshow((images[image]+1.)/2.) )
Next, the TensorFlow tutorial has the Discriminator output with a linear neuron. When they write model.add(layers.Dense(1)). The absence of an activation causes the neuron to have a linear neuron. This lets your Discriminator output any negative and positive value. I suspect this helps the cost function have better derivatives. Try changing the last line of your Discriminator to tf.keras.layers.Dense(1, activation='linear')
The reason that your Discriminator should output linear is because your loss is defined as cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True). The parameter from_logits=True expects the Discriminator output to be all real numbers (it expects logits instead of probs) instead of just the limited numbers from a sigmoid or tanh.
(Perhaps if you set from_logits=False you can use a sigmoid output to your Discriminator).
Hi Chris,
\n\nThanks for the help so far. I changed the normalization but I am wondering why 127.5 and not 255.0? I will try using linear activation but if it doesn't improve will change to sigmoid the way you said.
\n\nAlso I think I didn't batch my dataset correctly so I'm getting the ResourceExhausted. I will debug that and see if anything changes.
", "votes": null, "replies": [] }, { "id": 570034, "author_name": "cdeotte", "author_url": "", "post_date": "07/07/2019 17:36:31", "content": "The original images have pixel values of 0, 1, 2, ..., 253, 254, 255. After applying the formula image = (image - 127.5) / 127.5, the values approximately become -1.0, -0.99, -0.98,... 0.98, 0.99, 1.0. So that formula maps the values from [0,255] to [-1,1]. And tanh outputs values from [-1,1]. Therefore the original images and the generator images will be using the same pixel values.
I've been messing with your code. I've gotten it to make blobs of color in various shapes but I haven't got the dog faces to show up yet. I'm currently working with a slightly simpler generator and discriminator architecture.
", "votes": null, "replies": [] }, { "id": 570036, "author_name": "cmalla94", "author_url": "", "post_date": "07/07/2019 17:42:33", "content": "What changes did you make to the code and what epoch were you able to run?
", "votes": null, "replies": [] }, { "id": 570047, "author_name": "cdeotte", "author_url": "", "post_date": "07/07/2019 17:56:09", "content": "I'm running 50 epochs. I'm not getting anything great yet, but it is better than blue and yellow dots/lines like you had. I'm getting blobs of yellow and blue shapes now. The two big changes are adding image = (image - 127.5) / 127.5 in preprocessing and activation='linear' for Discriminator.
Next I crop the images using the bounding boxes to just dogs and use antialiasing (from PIL package, can't find antialiasing in tf 1.13.1) to resize to 64x64. This makes it easier for the GAN to generalize. (I think). Next I'm using
def make_generator():\nmodel = tf.keras.Sequential()\nmodel.add(layers.Dense(4*4*512, use_bias=False, input_shape=(100,)))\nmodel.add(layers.Reshape((4, 4, 512)))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(256, (5, 5), strides=(2, 2), padding='same', use_bias=False))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(128, (5, 5), strides=(2, 2), padding='same', use_bias=False))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\n\nmodel.add(layers.Conv2DTranspose(3, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))\n\nreturn model\nand
\n\ndef make_discriminator():\nmodel = tf.keras.Sequential()\nmodel.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same',\n input_shape=[64, 64, 3]))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Conv2D(256, (5, 5), strides=(2, 2), padding='same'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Conv2D(512, (5, 5), strides=(2, 2), padding='same'))\nmodel.add(layers.BatchNormalization())\nmodel.add(layers.LeakyReLU())\nmodel.add(layers.Dropout(0.5))\n\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(1, activation='linear')) #output logits\n\nreturn model\nIt still needs more work to make anything resembling dogs. I removed the autotune. That was causing the kernel to error. Now I'm experimenting with adjusting batch_sizes, learning rates, adjusting ratio of D updates versus G updates, and using a Gaussian seed instead of uniform, etc.
\n\nI'm new to GANs myself, so I'm not sure exactly what needs to be changed so it learns to draw dogs from scratch quickly.
", "votes": null, "replies": [] }, { "id": 570186, "author_name": "cdeotte", "author_url": "", "post_date": "07/08/2019 01:05:10", "content": "I've made more progress. The autotune is causing your kernel to crash ds = ds.prefetch(buffer_size=AUTOTUNE). Next, your kernel only plots the Red layer of an image with plt.imshow( (generated_image[0, :, :, 0]+1.)/2. ) and this causes matplotlib to display a blue yellow map. You need to turn that last 0 into a colon as plt.imshow( (generated_image[0, :, :, :]+1.)/2. ) then you will see RGB images. Finally you should plot the disc_loss and gen_loss and make sure they stay equal all the time (by adjusting learning rates). I've found that setting the Generator's Adam optimizer stronger helps keep the two in sync.
Oh my, wow thank you so much for your help Chris.
\n\nI am currently changing my code to use your bounding code from your kernel . Will make the other changes as well and let you know what has changed.
\n\nThank you again, really helpful and motivated me to not to give up on my first competition.
", "votes": null, "replies": [] }, { "id": 570246, "author_name": "cdeotte", "author_url": "", "post_date": "07/08/2019 03:44:54", "content": "Here it is after 50 epochs (31000 batches) (1 hour run). In addition to what we discussed above, I added all the GAN hacks from here. I'll post images after 9 hours of running tomorrow. \n\n(Each epoch is approx 625 batches of size 32. The 50th epoch has mean gen_loss = 1.39 and std gen_loss = 0.553 per batch. And mean disc_loss = 0.98 and std disc_loss = 0.208 per batch).
Hey Chris , so I have removed the line for prefetching but I am still getting the resource exhausted error.
\n\n'OOM when allocating tensor with shape[64,64,64,64]'
\n\nThat was after 10 epoch, now I am getting shapes so that's good. Is there anything I have configure for the kaggle kernel to manage resources?
\n\nThis is my updated kernel
", "votes": null, "replies": [] }, { "id": 570552, "author_name": "cdeotte", "author_url": "", "post_date": "07/08/2019 12:50:30", "content": "I'm not sure. Let me think. One thing I noticed is that you are including some blank images. After using my dog crop code only the first idxIn elements are images so you should use imagesIn = tf.cast(imagesIn[:idxIn,:,:,:], 'float32').
It's exciting that you're getting shapes, that's a great improvement.
", "votes": null, "replies": [] }, { "id": 570568, "author_name": "cmalla94", "author_url": "", "post_date": "07/08/2019 13:24:02", "content": "I have a feeling it is because of tf.enable_eager_execution(). If I don't enable eager execution then when it gets to the code block for
imagesIn = tf.cast(imagesIn[:idxIn, : , : , :], 'float32')or \nds = tf.data.Dataset.from_tensor_slices(imagesIn).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\nI get the error ValueError: Cannot create a tensor proto whose content is larger than 2GB. . Is there workaround for this to cast and make batches from the imagesIn without using tf.enable_eager_execution()?
Good question, let me experiment.
", "votes": null, "replies": [] }, { "id": 570597, "author_name": "cdeotte", "author_url": "", "post_date": "07/08/2019 14:16:34", "content": "I found the problem. Inside your function def train_step(images), you have the line fake_pictures.append(generated_images) which saves fake images to the global variable fake_pictures. Each epoch it saves 1GB of data to RAM. You need to remove or change that.
If you remove the two instances of display.clear_output(wait=True) inside your train() function then images will get drawn to the screen (without overwriting each other) and you won't need to save them all to see them.
Ahh, okay I removed that line and running it now. By the way, did the 9 hour training finish for you?
", "votes": null, "replies": [] }, { "id": 570705, "author_name": "cdeotte", "author_url": "", "post_date": "07/08/2019 17:14:21", "content": "Yes. I ran my code for 7 hours. It did 300 epochs (approx 200000 batches). Here's the output:\n\nThe means of Gen_loss and Disc_loss stayed near 1.0 the whole time, that's good. So I could probably train it longer if I can speed up the code (to finish under 9 hours).
Great results! I am at 40 epoch now without any errors so that line was the culprit. Thanks very much for catching. Next I will apply the previously mentioned optimizations and from that the kernel you mentioned.
", "votes": null, "replies": [] }, { "id": 570713, "author_name": "phoenix9032", "author_url": "", "post_date": "07/08/2019 17:24:01", "content": "@cdeotte Did you use the same architecture as posted in the thread ? I read in some place to add dropout only to generator . I had some weird result in adding dropout in discriminator .
", "votes": null, "replies": [] }, { "id": 570714, "author_name": "cdeotte", "author_url": "", "post_date": "07/08/2019 17:25:58", "content": "Yes, those pictures come from the above architectures. Currently my generator has no dropout. I also heard about the recommendation to add dropout to the generator both during training and image generation. I'll try that.
", "votes": null, "replies": [] }, { "id": 570717, "author_name": "phoenix9032", "author_url": "", "post_date": "07/08/2019 17:30:23", "content": "I am currently running one with Dropout(0.5) in generator . Without Dropout it gave 100 FID . Lets see if it improves , or cracks ...
", "votes": null, "replies": [] }, { "id": 571084, "author_name": "enric1296", "author_url": "", "post_date": "07/09/2019 06:25:59", "content": "Which lr are you using on Generator and Discriminator?
", "votes": null, "replies": [] }, { "id": 571103, "author_name": "phoenix9032", "author_url": "", "post_date": "07/09/2019 07:00:38", "content": "Unfortunately it didnt give better results with dropout . But i am not completely discarding it . Might be few other reasons as well
", "votes": null, "replies": [] }, { "id": 571256, "author_name": "cdeotte", "author_url": "", "post_date": "07/09/2019 11:40:55", "content": "> Which lr are you using on Generator and Discriminator?
\n\nI'm using Adam with lr=0.0002 and beta1=0.5 in both G and D with smoothed positive and negative labels. This and other tricks are recommended from GAN hacks here. (And brought to my attention by Luca in this thread).