{
  "id": 104211,
  "title": "Public LB 15.81605 solution",
  "url": "/competitions/generative-dog-images/writeups/kaggler-ja-yabea-public-lb-15-81605-solution",
  "author_name": "",
  "post_date": "2019-08-15T08:49:39.136734500Z",
  "votes": 39,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Here is one of my final submitted kernels (I selected quite similar two kernels for final submission). </p>\n\n<p><a href=\"https://www.kaggle.com/yukia18/sub-rals-ac-biggan-with-minibatchstddev\">[Sub] RaLS AC-BigGAN with MinibatchStddev</a>\nVersion 3 got public LB 15.81605.</p>\n\n<p>FID emphasizes sample diversity too much. My images generated from roose threshold (2~) truncated latents are not good as for quality but get good score. Once you download images.zip from my kernel, you can see that my generated images are crappy quality but diversed.</p>\n\n<p><strong>Key points:</strong></p>\n\n<ul>\n<li>Model: based on BigGAN\n<ul><li>shared embedding of dog breed labels</li>\n<li>hierarchical latent noise</li>\n<li>projection</li>\n<li>auxiliary classifier (ACGAN)</li></ul></li>\n<li>Loss: RaLS with weighted auxiliary classification loss</li>\n<li>Batch size: 128</li>\n<li>Exponential Moving Average of generator weights </li>\n<li>Uniform input noise on real images</li>\n</ul>",
  "messages": [
    {
      "id": "599685",
      "postDate": "08/15/2019 08:49:39",
      "content": "<p>Here is one of my final submitted kernels (I selected quite similar two kernels for final submission). </p>\n\n<p><a href=\"https://www.kaggle.com/yukia18/sub-rals-ac-biggan-with-minibatchstddev\">[Sub] RaLS AC-BigGAN with MinibatchStddev</a>\nVersion 3 got public LB 15.81605.</p>\n\n<p>FID emphasizes sample diversity too much. My images generated from roose threshold (2~) truncated latents are not good as for quality but get good score. Once you download images.zip from my kernel, you can see that my generated images are crappy quality but diversed.</p>\n\n<p><strong>Key points:</strong></p>\n\n<ul>\n<li>Model: based on BigGAN\n<ul><li>shared embedding of dog breed labels</li>\n<li>hierarchical latent noise</li>\n<li>projection</li>\n<li>auxiliary classifier (ACGAN)</li></ul></li>\n<li>Loss: RaLS with weighted auxiliary classification loss</li>\n<li>Batch size: 128</li>\n<li>Exponential Moving Average of generator weights </li>\n<li>Uniform input noise on real images</li>\n</ul>",
      "rawMarkdown": "Here is one of my final submitted kernels (I selected quite similar two kernels for final submission). \n\n[[Sub] RaLS AC-BigGAN with MinibatchStddev](https://www.kaggle.com/yukia18/sub-rals-ac-biggan-with-minibatchstddev)\nVersion 3 got public LB 15.81605.\n\nFID emphasizes sample diversity too much. My images generated from roose threshold (2~) truncated latents are not good as for quality but get good score. Once you download images.zip from my kernel, you can see that my generated images are crappy quality but diversed.\n\n**Key points:**\n\n- Model: based on BigGAN\n    - shared embedding of dog breed labels\n    - hierarchical latent noise\n    - projection\n    - auxiliary classifier (ACGAN)\n- Loss: RaLS with weighted auxiliary classification loss\n- Batch size: 128\n- Exponential Moving Average of generator weights \n- Uniform input noise on real images",
      "votes": null
    },
    {
      "id": "599737",
      "postDate": "08/15/2019 09:54:40",
      "content": "<p>wow amazing work! <a href=\"/yukia18\">@yukia18</a> I never thought of anything like this...</p>",
      "rawMarkdown": "wow amazing work! @yukia18 I never thought of anything like this...",
      "votes": null
    },
    {
      "id": "599757",
      "postDate": "08/15/2019 10:17:01",
      "content": "<p>Thanks. I used BigGAN too, but with less modifications and my settings are quite different.\nHigher batch sizes never worked for my setup, so I used 32 most of the time. And use of relativistic loss didn't bring any improvements (I tried only RaHinge variant), but forced me to use equal lr for D and G, otherwise I would get mode collapse after ~1 hour of training. So I opted to use standard Hinge loss.\nAlso I am using twice less channels in G (maybe afterall it's not a great idea :D)</p>\n\n<p>Nice pictures BTW. They are not as noisy as mine :)</p>",
      "rawMarkdown": "Thanks. I used BigGAN too, but with less modifications and my settings are quite different.\nHigher batch sizes never worked for my setup, so I used 32 most of the time. And use of relativistic loss didn't bring any improvements (I tried only RaHinge variant), but forced me to use equal lr for D and G, otherwise I would get mode collapse after ~1 hour of training. So I opted to use standard Hinge loss.\nAlso I am using twice less channels in G (maybe afterall it's not a great idea :D)\n\nNice pictures BTW. They are not as noisy as mine :)",
      "votes": null
    },
    {
      "id": "599853",
      "postDate": "08/15/2019 11:53:27",
      "content": "<p>Thanks. I also changed the original BigGAN only a little (shared embedding, hierarchical latent noise, projection, and exponential moving average are explained in <a href=\"https://arxiv.org/pdf/1809.11096.pdf\">paper</a>). Key points that I listed improved my simple DCGAN, so I think they are key points.</p>\n\n<p>Here is supplementary matters of my settings.\n- Batch size 64 and 128 worked, but 32 and 256 didn't. \n- I tried hinge loss, but it didn't work.\n- In original BigGAN, learning rates of generator and discriminator are 5e-5 and 1e-4 respectively, but they were too small on 9 hours training.\n- Channel sizes of generator and discriminator are less than half of originals. I tried imbalanced channel size GANs but didn't work.</p>\n\n<p>We used same architecture basically, but settings are quite different. It is interesting.</p>",
      "rawMarkdown": "Thanks. I also changed the original BigGAN only a little (shared embedding, hierarchical latent noise, projection, and exponential moving average are explained in [paper](https://arxiv.org/pdf/1809.11096.pdf)\\). Key points that I listed improved my simple DCGAN, so I think they are key points.\n  \nHere is supplementary matters of my settings.\n- Batch size 64 and 128 worked, but 32 and 256 didn't. \n- I tried hinge loss, but it didn't work.\n- In original BigGAN, learning rates of generator and discriminator are 5e-5 and 1e-4 respectively, but they were too small on 9 hours training.\n- Channel sizes of generator and discriminator are less than half of originals. I tried imbalanced channel size GANs but didn't work.\n  \nWe used same architecture basically, but settings are quite different. It is interesting.",
      "votes": null
    },
    {
      "id": "599943",
      "postDate": "08/15/2019 13:28:40",
      "content": "<p>amazing work!</p>",
      "rawMarkdown": "amazing work!",
      "votes": null
    },
    {
      "id": "600241",
      "postDate": "08/15/2019 20:37:43",
      "content": "<p>Congratulations Yabea. Your GAN is awesome. I also noticed that public FID favored diversity over fidelity. None-the-less, your dogs look great.  Below are examples of Yabea's dogs:\n<img src=\"http://playagricola.com/Kaggle/dogs581419.png\" alt=\"image\"></p>",
      "rawMarkdown": "Congratulations Yabea. Your GAN is awesome. I also noticed that public FID favored diversity over fidelity. None-the-less, your dogs look great.  Below are examples of Yabea's dogs:\n![image](http://playagricola.com/Kaggle/dogs581419.png)",
      "votes": null
    },
    {
      "id": "600408",
      "postDate": "08/16/2019 04:40:06",
      "content": "<p>Impressive Work.....!! Thanks for Sharing <a href=\"/yukia18\">@yukia18</a> </p>",
      "rawMarkdown": "Impressive Work.....!! Thanks for Sharing @yukia18",
      "votes": null
    },
    {
      "id": "601531",
      "postDate": "08/17/2019 18:46:19",
      "content": "<p>Great job! Good luck! </p>",
      "rawMarkdown": "Great job! Good luck!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 599737,
      "author_name": "jesucristo",
      "author_url": "",
      "post_date": "08/15/2019 09:54:40",
      "content": "<p>wow amazing work! <a href=\"/yukia18\">@yukia18</a> I never thought of anything like this...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 599757,
      "author_name": "dvorobiev",
      "author_url": "",
      "post_date": "08/15/2019 10:17:01",
      "content": "<p>Thanks. I used BigGAN too, but with less modifications and my settings are quite different.\nHigher batch sizes never worked for my setup, so I used 32 most of the time. And use of relativistic loss didn't bring any improvements (I tried only RaHinge variant), but forced me to use equal lr for D and G, otherwise I would get mode collapse after ~1 hour of training. So I opted to use standard Hinge loss.\nAlso I am using twice less channels in G (maybe afterall it's not a great idea :D)</p>\n\n<p>Nice pictures BTW. They are not as noisy as mine :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 599853,
          "author_name": "yukia18",
          "author_url": "",
          "post_date": "08/15/2019 11:53:27",
          "content": "<p>Thanks. I also changed the original BigGAN only a little (shared embedding, hierarchical latent noise, projection, and exponential moving average are explained in <a href=\"https://arxiv.org/pdf/1809.11096.pdf\">paper</a>). Key points that I listed improved my simple DCGAN, so I think they are key points.</p>\n\n<p>Here is supplementary matters of my settings.\n- Batch size 64 and 128 worked, but 32 and 256 didn't. \n- I tried hinge loss, but it didn't work.\n- In original BigGAN, learning rates of generator and discriminator are 5e-5 and 1e-4 respectively, but they were too small on 9 hours training.\n- Channel sizes of generator and discriminator are less than half of originals. I tried imbalanced channel size GANs but didn't work.</p>\n\n<p>We used same architecture basically, but settings are quite different. It is interesting.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 599943,
      "author_name": "meistermorxrc",
      "author_url": "",
      "post_date": "08/15/2019 13:28:40",
      "content": "<p>amazing work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 600241,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "08/15/2019 20:37:43",
      "content": "<p>Congratulations Yabea. Your GAN is awesome. I also noticed that public FID favored diversity over fidelity. None-the-less, your dogs look great.  Below are examples of Yabea's dogs:\n<img src=\"http://playagricola.com/Kaggle/dogs581419.png\" alt=\"image\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 600408,
      "author_name": "veeralakrishna",
      "author_url": "",
      "post_date": "08/16/2019 04:40:06",
      "content": "<p>Impressive Work.....!! Thanks for Sharing <a href=\"/yukia18\">@yukia18</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 601531,
      "author_name": "muhakabartay",
      "author_url": "",
      "post_date": "08/17/2019 18:46:19",
      "content": "<p>Great job! Good luck! </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "599685": "Here is one of my final submitted kernels (I selected quite similar two kernels for final submission). \n\n[[Sub] RaLS AC-BigGAN with MinibatchStddev](https://www.kaggle.com/yukia18/sub-rals-ac-biggan-with-minibatchstddev)\nVersion 3 got public LB 15.81605.\n\nFID emphasizes sample diversity too much. My images generated from roose threshold (2~) truncated latents are not good as for quality but get good score. Once you download images.zip from my kernel, you can see that my generated images are crappy quality but diversed.\n\n**Key points:**\n\n- Model: based on BigGAN\n    - shared embedding of dog breed labels\n    - hierarchical latent noise\n    - projection\n    - auxiliary classifier (ACGAN)\n- Loss: RaLS with weighted auxiliary classification loss\n- Batch size: 128\n- Exponential Moving Average of generator weights \n- Uniform input noise on real images",
    "599737": "wow amazing work! @yukia18 I never thought of anything like this...",
    "599757": "Thanks. I used BigGAN too, but with less modifications and my settings are quite different.\nHigher batch sizes never worked for my setup, so I used 32 most of the time. And use of relativistic loss didn't bring any improvements (I tried only RaHinge variant), but forced me to use equal lr for D and G, otherwise I would get mode collapse after ~1 hour of training. So I opted to use standard Hinge loss.\nAlso I am using twice less channels in G (maybe afterall it's not a great idea :D)\n\nNice pictures BTW. They are not as noisy as mine :)",
    "599853": "Thanks. I also changed the original BigGAN only a little (shared embedding, hierarchical latent noise, projection, and exponential moving average are explained in [paper](https://arxiv.org/pdf/1809.11096.pdf)\\). Key points that I listed improved my simple DCGAN, so I think they are key points.\n  \nHere is supplementary matters of my settings.\n- Batch size 64 and 128 worked, but 32 and 256 didn't. \n- I tried hinge loss, but it didn't work.\n- In original BigGAN, learning rates of generator and discriminator are 5e-5 and 1e-4 respectively, but they were too small on 9 hours training.\n- Channel sizes of generator and discriminator are less than half of originals. I tried imbalanced channel size GANs but didn't work.\n  \nWe used same architecture basically, but settings are quite different. It is interesting.",
    "599943": "amazing work!",
    "600241": "Congratulations Yabea. Your GAN is awesome. I also noticed that public FID favored diversity over fidelity. None-the-less, your dogs look great.  Below are examples of Yabea's dogs:\n![image](http://playagricola.com/Kaggle/dogs581419.png)",
    "600408": "Impressive Work.....!! Thanks for Sharing @yukia18",
    "601531": "Great job! Good luck!"
  },
  "source": "meta"
}