{
  "id": 97989,
  "title": "Introducing DCGANs",
  "url": "/competitions/generative-dog-images/discussion/97989",
  "author_name": "",
  "post_date": "2019-06-30T12:47:20.339183700Z",
  "votes": 21,
  "comment_count": 10,
  "views": 0,
  "content": "<h2>My kernel: <a href=\"https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images\">Introducing DCGAN Dogs Images</a></h2>\n\n<p>Check the <a href=\"https://arxiv.org/abs/1511.06434\">Original Paper</a></p>\n\n<p>DCGANs = Deep Convolutional Generative Adversarial Networks</p>\n\n<p>GAN is known for its difficulty in learning, the paper proposed by Radford. introduces various techniques for successful learning:</p>\n\n<ul>\n<li>Convert max-pooling layers to convolution layers</li>\n<li>Convert fully connected layers to global average pooling layers in the discriminator</li>\n<li>Use batch normalization layers in the generator and the discriminator</li>\n<li>Use leaky ReLU activation functions in the discriminator</li>\n</ul>\n\n<p>DCGAN is one of the popular and successful network design for GAN. It mainly composes of convolution layers without max pooling or fully connected layers. It uses convolutional stride and transposed convolution for the downsampling and the upsampling. The figure below is the network design for the generator.</p>\n\n<p></p>\n\n<p>This is the DCGAN generator presented in the LSUN scene modeling paper. This network takes in a 100x1 noise vector, denoted z, and maps it into the G(Z) output which is 64x64x3.</p>\n\n<p>This architecture is especially interesting the way the first layer expands the random noise. The network goes from 100x1 to 1024x4x4! This layer is denoted ‘project and reshape’.</p>\n\n<p>We see that following this layer, classical convolutional layers are applied which reshape the network with the (N+P — F)/S + 1 equation classically taught with convolutional layers. In the diagram above we can see that the N parameter, (Height/Width), goes from 4 to 8 to 16 to 32, it doesn’t appear that there is any padding, the kernel filter parameter F is 5x5, and the stride is 2. You may find this equation to be useful for designing your own convolutional layers for customized output sizes.</p>\n\n<p>We see the network goes from:</p>\n\n<p><code>100x1 → 1024x4x4 → 512x8x8 → 256x16x16 → 128x32x32 → 64x64x3</code></p>\n\n<h3>Summary of DCGAN:</h3>\n\n<ul>\n<li>Replace all max pooling with convolutional stride</li>\n<li>Use transposed convolution for upsampling.</li>\n<li>Eliminate fully connected layers.</li>\n<li>Use Batch normalization except the output layer for the generator and the input layer of the discriminator.</li>\n<li>Use ReLU in the generator except for the output which uses tanh.</li>\n<li>Use LeakyReLU in the discriminator.</li>\n</ul>\n\n<h3>Tuning tips quote directly from the paper.</h3>\n\n<p>&gt; All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. In the LeakyReLU, the slope of the leak was set to 0.2 in all models. While previous GAN work has used momentum to accelerate training, we used the Adam optimizer with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. Additionally, we found leaving the momentum term β1 at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.</p>\n\n<h3>Some generated images</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2779868%2F14a76f32881bd7c8cc75cb940f601c47%2Fdogs1%20(1\" alt=\"\">.png?generation=1562804966674757&amp;alt=media)</p>\n\n<h3>References:</h3>\n\n<ul>\n<li><a href=\"https://arxiv.org/abs/1511.06434\">Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks</a></li>\n<li><a href=\"https://towardsdatascience.com/dcgans-deep-convolutional-generative-adversarial-networks-c7f392c2c8f8\">DCGANs (Deep Convolutional Generative Adversarial Networks)</a></li>\n<li><a href=\"https://github.com/chainer/chainer/tree/master/examples/dcgan\">DCGAN Chainer</a></li>\n<li><a href=\"https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b\">DCGAN: Generate the images with Deep Convolutional GAN</a></li>\n</ul>\n\n<p><a href=\"/jazivxt\">@jazivxt</a> propose another tensorflow version  <a href=\"https://www.kaggle.com/c/generative-dog-images/discussion/97983#latest-565096\">here</a></p>\n\n<p>Hope this helps.</p>",
  "messages": [
    {
      "id": "565105",
      "postDate": "06/30/2019 12:47:20",
      "content": "<h2>My kernel: <a href=\"https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images\">Introducing DCGAN Dogs Images</a></h2>\n\n<p>Check the <a href=\"https://arxiv.org/abs/1511.06434\">Original Paper</a></p>\n\n<p>DCGANs = Deep Convolutional Generative Adversarial Networks</p>\n\n<p>GAN is known for its difficulty in learning, the paper proposed by Radford. introduces various techniques for successful learning:</p>\n\n<ul>\n<li>Convert max-pooling layers to convolution layers</li>\n<li>Convert fully connected layers to global average pooling layers in the discriminator</li>\n<li>Use batch normalization layers in the generator and the discriminator</li>\n<li>Use leaky ReLU activation functions in the discriminator</li>\n</ul>\n\n<p>DCGAN is one of the popular and successful network design for GAN. It mainly composes of convolution layers without max pooling or fully connected layers. It uses convolutional stride and transposed convolution for the downsampling and the upsampling. The figure below is the network design for the generator.</p>\n\n<p></p>\n\n<p>This is the DCGAN generator presented in the LSUN scene modeling paper. This network takes in a 100x1 noise vector, denoted z, and maps it into the G(Z) output which is 64x64x3.</p>\n\n<p>This architecture is especially interesting the way the first layer expands the random noise. The network goes from 100x1 to 1024x4x4! This layer is denoted ‘project and reshape’.</p>\n\n<p>We see that following this layer, classical convolutional layers are applied which reshape the network with the (N+P — F)/S + 1 equation classically taught with convolutional layers. In the diagram above we can see that the N parameter, (Height/Width), goes from 4 to 8 to 16 to 32, it doesn’t appear that there is any padding, the kernel filter parameter F is 5x5, and the stride is 2. You may find this equation to be useful for designing your own convolutional layers for customized output sizes.</p>\n\n<p>We see the network goes from:</p>\n\n<p><code>100x1 → 1024x4x4 → 512x8x8 → 256x16x16 → 128x32x32 → 64x64x3</code></p>\n\n<h3>Summary of DCGAN:</h3>\n\n<ul>\n<li>Replace all max pooling with convolutional stride</li>\n<li>Use transposed convolution for upsampling.</li>\n<li>Eliminate fully connected layers.</li>\n<li>Use Batch normalization except the output layer for the generator and the input layer of the discriminator.</li>\n<li>Use ReLU in the generator except for the output which uses tanh.</li>\n<li>Use LeakyReLU in the discriminator.</li>\n</ul>\n\n<h3>Tuning tips quote directly from the paper.</h3>\n\n<p>&gt; All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. In the LeakyReLU, the slope of the leak was set to 0.2 in all models. While previous GAN work has used momentum to accelerate training, we used the Adam optimizer with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. Additionally, we found leaving the momentum term β1 at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.</p>\n\n<h3>Some generated images</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2779868%2F14a76f32881bd7c8cc75cb940f601c47%2Fdogs1%20(1\" alt=\"\">.png?generation=1562804966674757&amp;alt=media)</p>\n\n<h3>References:</h3>\n\n<ul>\n<li><a href=\"https://arxiv.org/abs/1511.06434\">Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks</a></li>\n<li><a href=\"https://towardsdatascience.com/dcgans-deep-convolutional-generative-adversarial-networks-c7f392c2c8f8\">DCGANs (Deep Convolutional Generative Adversarial Networks)</a></li>\n<li><a href=\"https://github.com/chainer/chainer/tree/master/examples/dcgan\">DCGAN Chainer</a></li>\n<li><a href=\"https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b\">DCGAN: Generate the images with Deep Convolutional GAN</a></li>\n</ul>\n\n<p><a href=\"/jazivxt\">@jazivxt</a> propose another tensorflow version  <a href=\"https://www.kaggle.com/c/generative-dog-images/discussion/97983#latest-565096\">here</a></p>\n\n<p>Hope this helps.</p>",
      "rawMarkdown": "## My kernel: [Introducing DCGAN Dogs Images](https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images)\n\nCheck the [Original Paper](https://arxiv.org/abs/1511.06434)\n\nDCGANs = Deep Convolutional Generative Adversarial Networks\n\nGAN is known for its difficulty in learning, the paper proposed by Radford. introduces various techniques for successful learning:\n\n- Convert max-pooling layers to convolution layers\n- Convert fully connected layers to global average pooling layers in the discriminator\n- Use batch normalization layers in the generator and the discriminator\n- Use leaky ReLU activation functions in the discriminator\n\nDCGAN is one of the popular and successful network design for GAN. It mainly composes of convolution layers without max pooling or fully connected layers. It uses convolutional stride and transposed convolution for the downsampling and the upsampling. The figure below is the network design for the generator.\n\n![](https://cdn-images-1.medium.com/max/800/1*rdXKdyfNjorzP10ZA3yNmQ.png)\n\nThis is the DCGAN generator presented in the LSUN scene modeling paper. This network takes in a 100x1 noise vector, denoted z, and maps it into the G(Z) output which is 64x64x3.\n\nThis architecture is especially interesting the way the first layer expands the random noise. The network goes from 100x1 to 1024x4x4! This layer is denoted ‘project and reshape’.\n\nWe see that following this layer, classical convolutional layers are applied which reshape the network with the (N+P — F)/S + 1 equation classically taught with convolutional layers. In the diagram above we can see that the N parameter, (Height/Width), goes from 4 to 8 to 16 to 32, it doesn’t appear that there is any padding, the kernel filter parameter F is 5x5, and the stride is 2. You may find this equation to be useful for designing your own convolutional layers for customized output sizes.\n\nWe see the network goes from:\n\n```100x1 → 1024x4x4 → 512x8x8 → 256x16x16 → 128x32x32 → 64x64x3```\n\n\n### Summary of DCGAN:\n\n- Replace all max pooling with convolutional stride\n- Use transposed convolution for upsampling.\n- Eliminate fully connected layers.\n- Use Batch normalization except the output layer for the generator and the input layer of the discriminator.\n- Use ReLU in the generator except for the output which uses tanh.\n- Use LeakyReLU in the discriminator.\n\n### Tuning tips quote directly from the paper.\n\n&gt; All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. In the LeakyReLU, the slope of the leak was set to 0.2 in all models. While previous GAN work has used momentum to accelerate training, we used the Adam optimizer with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. Additionally, we found leaving the momentum term β1 at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.\n\n### Some generated images\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2779868%2F14a76f32881bd7c8cc75cb940f601c47%2Fdogs1%20(1).png?generation=1562804966674757&amp;alt=media)\n\n### References:\n\n- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://arxiv.org/abs/1511.06434)\n- [DCGANs (Deep Convolutional Generative Adversarial Networks)](https://towardsdatascience.com/dcgans-deep-convolutional-generative-adversarial-networks-c7f392c2c8f8)\n- [DCGAN Chainer](https://github.com/chainer/chainer/tree/master/examples/dcgan)\n- [DCGAN: Generate the images with Deep Convolutional GAN](https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b)\n\n@jazivxt propose another tensorflow version  [here](https://www.kaggle.com/c/generative-dog-images/discussion/97983#latest-565096)\n\nHope this helps.",
      "votes": null
    },
    {
      "id": "565188",
      "postDate": "06/30/2019 14:55:28",
      "content": "<p>Good summary discussion!\nThanks a lot!</p>",
      "rawMarkdown": "Good summary discussion!\nThanks a lot!",
      "votes": null
    },
    {
      "id": "566409",
      "postDate": "07/02/2019 06:20:56",
      "content": "<p>thx!</p>",
      "rawMarkdown": "thx!",
      "votes": null
    },
    {
      "id": "566807",
      "postDate": "07/02/2019 16:06:10",
      "content": "<p>Thanks for posting. Great paper. How is the 100x1 connected to the 1024x4x4, is it fully connected?</p>",
      "rawMarkdown": "Thanks for posting. Great paper. How is the 100x1 connected to the 1024x4x4, is it fully connected?",
      "votes": null
    },
    {
      "id": "567397",
      "postDate": "07/03/2019 13:01:25",
      "content": "<p>sorry for the late reply <a href=\"/cdeotte\">@cdeotte</a> ,yes, I've seen implementations where it's fully connected. I'm actually fixing and improving my DCGAN kernel :)</p>",
      "rawMarkdown": "sorry for the late reply @cdeotte ,yes, I've seen implementations where it's fully connected. I'm actually fixing and improving my DCGAN kernel :)",
      "votes": null
    },
    {
      "id": "567432",
      "postDate": "07/03/2019 14:11:49",
      "content": "<p>Thanks, I fully connected my Z to a small set of maps and then up-convoluted to the first big set. I posted my first generative network <a href=\"https://www.kaggle.com/cdeotte/supervised-generative-dog-net\">here</a>. It uses following architecture</p>\n\n<pre><code>Z -&gt; 8x8x32 -&gt; 16x16x128 -&gt; 32x32x64 -&gt; 64x64x32 -&gt; 64x64x3\n</code></pre>\n\n<p>Now I need to add a discriminative network and make some fun dog pics!</p>",
      "rawMarkdown": "Thanks, I fully connected my Z to a small set of maps and then up-convoluted to the first big set. I posted my first generative network [here][1]. It uses following architecture\n\n    Z -&gt; 8x8x32 -&gt; 16x16x128 -&gt; 32x32x64 -&gt; 64x64x32 -&gt; 64x64x3\n\nNow I need to add a discriminative network and make some fun dog pics!\n\n[1]: https://www.kaggle.com/cdeotte/supervised-generative-dog-net",
      "votes": null
    },
    {
      "id": "569814",
      "postDate": "07/07/2019 11:53:25",
      "content": "<p>Great!</p>",
      "rawMarkdown": "Great!",
      "votes": null
    },
    {
      "id": "572455",
      "postDate": "07/11/2019 00:46:31",
      "content": "<p>Kernel updated!</p>\n\n<p>Now I use tricks from <a href=\"https://www.kaggle.com/c/generative-dog-images/discussion/98595#latest-570614\">All you need is GAN Hacks</a> like</p>\n\n<ul>\n<li><p>Use LeakyReLU = Avoid gradient vanishing. As you can see at the generator and discriminator\n<code>tf.nn.leaky_relu</code> it uses default = 0.2 </p></li>\n<li><p>Use Batch Normalization</p></li>\n</ul>\n\n<p><code>tf.layers.batch_normalization</code></p>\n\n<ul>\n<li>Use Label Smoothing</li>\n</ul>\n\n<p><code>d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real,\n                                                                         labels=tf.ones_like(d_model_real)* random.uniform(0.9, 1.0)))</code></p>\n\n<p>Note <code>random.uniform(0.9, 1.0)))</code> instead of 1</p>\n\n<ul>\n<li>Scale Images to the Range [-1,1]</li>\n</ul>\n\n<p><code>batch_z = np.random.uniform(-1, 1, size=(BATCH_SIZE, NOISE_SIZE))</code></p>",
      "rawMarkdown": "Kernel updated!\n\nNow I use tricks from [All you need is GAN Hacks](https://www.kaggle.com/c/generative-dog-images/discussion/98595#latest-570614) like\n\n- Use LeakyReLU = Avoid gradient vanishing. As you can see at the generator and discriminator\n```tf.nn.leaky_relu``` it uses default = 0.2 \n\n- Use Batch Normalization\n\n```tf.layers.batch_normalization```\n\n- Use Label Smoothing\n\n```d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real,\n                                                                         labels=tf.ones_like(d_model_real)* random.uniform(0.9, 1.0)))```\n\nNote ```random.uniform(0.9, 1.0)))``` instead of 1\n\n- Scale Images to the Range [-1,1]\n\n```batch_z = np.random.uniform(-1, 1, size=(BATCH_SIZE, NOISE_SIZE))```",
      "votes": null
    },
    {
      "id": "572458",
      "postDate": "07/11/2019 00:55:04",
      "content": "<p>Nice. Your dog images look great</p>",
      "rawMarkdown": "Nice. Your dog images look great",
      "votes": null
    },
    {
      "id": "572635",
      "postDate": "07/11/2019 07:43:38",
      "content": "<p>thanks for sharing bro!</p>",
      "rawMarkdown": "thanks for sharing bro!",
      "votes": null
    },
    {
      "id": "580119",
      "postDate": "07/19/2019 17:44:16",
      "content": "<p>great!</p>",
      "rawMarkdown": "great!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 565188,
      "author_name": "snakayama",
      "author_url": "",
      "post_date": "06/30/2019 14:55:28",
      "content": "<p>Good summary discussion!\nThanks a lot!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 566409,
      "author_name": "xiaoaerlaigeid",
      "author_url": "",
      "post_date": "07/02/2019 06:20:56",
      "content": "<p>thx!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 566807,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "07/02/2019 16:06:10",
      "content": "<p>Thanks for posting. Great paper. How is the 100x1 connected to the 1024x4x4, is it fully connected?</p>",
      "votes": null,
      "replies": [
        {
          "id": 567397,
          "author_name": "jesucristo",
          "author_url": "",
          "post_date": "07/03/2019 13:01:25",
          "content": "<p>sorry for the late reply <a href=\"/cdeotte\">@cdeotte</a> ,yes, I've seen implementations where it's fully connected. I'm actually fixing and improving my DCGAN kernel :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 567432,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "07/03/2019 14:11:49",
          "content": "<p>Thanks, I fully connected my Z to a small set of maps and then up-convoluted to the first big set. I posted my first generative network <a href=\"https://www.kaggle.com/cdeotte/supervised-generative-dog-net\">here</a>. It uses following architecture</p>\n\n<pre><code>Z -&gt; 8x8x32 -&gt; 16x16x128 -&gt; 32x32x64 -&gt; 64x64x32 -&gt; 64x64x3\n</code></pre>\n\n<p>Now I need to add a discriminative network and make some fun dog pics!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 569814,
      "author_name": "ameypatil",
      "author_url": "",
      "post_date": "07/07/2019 11:53:25",
      "content": "<p>Great!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 572455,
      "author_name": "jesucristo",
      "author_url": "",
      "post_date": "07/11/2019 00:46:31",
      "content": "<p>Kernel updated!</p>\n\n<p>Now I use tricks from <a href=\"https://www.kaggle.com/c/generative-dog-images/discussion/98595#latest-570614\">All you need is GAN Hacks</a> like</p>\n\n<ul>\n<li><p>Use LeakyReLU = Avoid gradient vanishing. As you can see at the generator and discriminator\n<code>tf.nn.leaky_relu</code> it uses default = 0.2 </p></li>\n<li><p>Use Batch Normalization</p></li>\n</ul>\n\n<p><code>tf.layers.batch_normalization</code></p>\n\n<ul>\n<li>Use Label Smoothing</li>\n</ul>\n\n<p><code>d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real,\n                                                                         labels=tf.ones_like(d_model_real)* random.uniform(0.9, 1.0)))</code></p>\n\n<p>Note <code>random.uniform(0.9, 1.0)))</code> instead of 1</p>\n\n<ul>\n<li>Scale Images to the Range [-1,1]</li>\n</ul>\n\n<p><code>batch_z = np.random.uniform(-1, 1, size=(BATCH_SIZE, NOISE_SIZE))</code></p>",
      "votes": null,
      "replies": [
        {
          "id": 572458,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "07/11/2019 00:55:04",
          "content": "<p>Nice. Your dog images look great</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 572635,
      "author_name": "krishnakatyal",
      "author_url": "",
      "post_date": "07/11/2019 07:43:38",
      "content": "<p>thanks for sharing bro!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 580119,
      "author_name": "krishnakatyal",
      "author_url": "",
      "post_date": "07/19/2019 17:44:16",
      "content": "<p>great!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "565105": "## My kernel: [Introducing DCGAN Dogs Images](https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images)\n\nCheck the [Original Paper](https://arxiv.org/abs/1511.06434)\n\nDCGANs = Deep Convolutional Generative Adversarial Networks\n\nGAN is known for its difficulty in learning, the paper proposed by Radford. introduces various techniques for successful learning:\n\n- Convert max-pooling layers to convolution layers\n- Convert fully connected layers to global average pooling layers in the discriminator\n- Use batch normalization layers in the generator and the discriminator\n- Use leaky ReLU activation functions in the discriminator\n\nDCGAN is one of the popular and successful network design for GAN. It mainly composes of convolution layers without max pooling or fully connected layers. It uses convolutional stride and transposed convolution for the downsampling and the upsampling. The figure below is the network design for the generator.\n\n![](https://cdn-images-1.medium.com/max/800/1*rdXKdyfNjorzP10ZA3yNmQ.png)\n\nThis is the DCGAN generator presented in the LSUN scene modeling paper. This network takes in a 100x1 noise vector, denoted z, and maps it into the G(Z) output which is 64x64x3.\n\nThis architecture is especially interesting the way the first layer expands the random noise. The network goes from 100x1 to 1024x4x4! This layer is denoted ‘project and reshape’.\n\nWe see that following this layer, classical convolutional layers are applied which reshape the network with the (N+P — F)/S + 1 equation classically taught with convolutional layers. In the diagram above we can see that the N parameter, (Height/Width), goes from 4 to 8 to 16 to 32, it doesn’t appear that there is any padding, the kernel filter parameter F is 5x5, and the stride is 2. You may find this equation to be useful for designing your own convolutional layers for customized output sizes.\n\nWe see the network goes from:\n\n```100x1 → 1024x4x4 → 512x8x8 → 256x16x16 → 128x32x32 → 64x64x3```\n\n\n### Summary of DCGAN:\n\n- Replace all max pooling with convolutional stride\n- Use transposed convolution for upsampling.\n- Eliminate fully connected layers.\n- Use Batch normalization except the output layer for the generator and the input layer of the discriminator.\n- Use ReLU in the generator except for the output which uses tanh.\n- Use LeakyReLU in the discriminator.\n\n### Tuning tips quote directly from the paper.\n\n&gt; All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. In the LeakyReLU, the slope of the leak was set to 0.2 in all models. While previous GAN work has used momentum to accelerate training, we used the Adam optimizer with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. Additionally, we found leaving the momentum term β1 at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.\n\n### Some generated images\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2779868%2F14a76f32881bd7c8cc75cb940f601c47%2Fdogs1%20(1).png?generation=1562804966674757&amp;alt=media)\n\n### References:\n\n- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://arxiv.org/abs/1511.06434)\n- [DCGANs (Deep Convolutional Generative Adversarial Networks)](https://towardsdatascience.com/dcgans-deep-convolutional-generative-adversarial-networks-c7f392c2c8f8)\n- [DCGAN Chainer](https://github.com/chainer/chainer/tree/master/examples/dcgan)\n- [DCGAN: Generate the images with Deep Convolutional GAN](https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b)\n\n@jazivxt propose another tensorflow version  [here](https://www.kaggle.com/c/generative-dog-images/discussion/97983#latest-565096)\n\nHope this helps.",
    "565188": "Good summary discussion!\nThanks a lot!",
    "566409": "thx!",
    "566807": "Thanks for posting. Great paper. How is the 100x1 connected to the 1024x4x4, is it fully connected?",
    "567397": "sorry for the late reply @cdeotte ,yes, I've seen implementations where it's fully connected. I'm actually fixing and improving my DCGAN kernel :)",
    "567432": "Thanks, I fully connected my Z to a small set of maps and then up-convoluted to the first big set. I posted my first generative network [here][1]. It uses following architecture\n\n    Z -&gt; 8x8x32 -&gt; 16x16x128 -&gt; 32x32x64 -&gt; 64x64x32 -&gt; 64x64x3\n\nNow I need to add a discriminative network and make some fun dog pics!\n\n[1]: https://www.kaggle.com/cdeotte/supervised-generative-dog-net",
    "569814": "Great!",
    "572455": "Kernel updated!\n\nNow I use tricks from [All you need is GAN Hacks](https://www.kaggle.com/c/generative-dog-images/discussion/98595#latest-570614) like\n\n- Use LeakyReLU = Avoid gradient vanishing. As you can see at the generator and discriminator\n```tf.nn.leaky_relu``` it uses default = 0.2 \n\n- Use Batch Normalization\n\n```tf.layers.batch_normalization```\n\n- Use Label Smoothing\n\n```d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real,\n                                                                         labels=tf.ones_like(d_model_real)* random.uniform(0.9, 1.0)))```\n\nNote ```random.uniform(0.9, 1.0)))``` instead of 1\n\n- Scale Images to the Range [-1,1]\n\n```batch_z = np.random.uniform(-1, 1, size=(BATCH_SIZE, NOISE_SIZE))```",
    "572458": "Nice. Your dog images look great",
    "572635": "thanks for sharing bro!",
    "580119": "great!"
  },
  "source": "meta"
}