{
  "id": 100723,
  "title": "could we use mixed precision for training gan?",
  "url": "/competitions/generative-dog-images/discussion/100723",
  "author_name": "",
  "post_date": "2019-07-20T12:59:53.573029600Z",
  "votes": 5,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi, I'm new with GAN and I'm qurious whether we could use mixed precision in training gan.</p>\n\n<p>I can see that public kernels are training with batch_size of 32 which is very small.</p>\n\n<p>what if I use amp and increase the batch size?</p>\n\n<p>I think this could stabilize the training. am I wrong?</p>\n\n<p>will increasing batch size help boost the score?</p>",
  "messages": [
    {
      "id": "580608",
      "postDate": "07/20/2019 12:59:53",
      "content": "<p>Hi, I'm new with GAN and I'm qurious whether we could use mixed precision in training gan.</p>\n\n<p>I can see that public kernels are training with batch_size of 32 which is very small.</p>\n\n<p>what if I use amp and increase the batch size?</p>\n\n<p>I think this could stabilize the training. am I wrong?</p>\n\n<p>will increasing batch size help boost the score?</p>",
      "rawMarkdown": "Hi, I'm new with GAN and I'm qurious whether we could use mixed precision in training gan.\n\nI can see that public kernels are training with batch_size of 32 which is very small.\n\nwhat if I use amp and increase the batch size?\n\nI think this could stabilize the training. am I wrong?\n\nwill increasing batch size help boost the score?",
      "votes": null
    },
    {
      "id": "581152",
      "postDate": "07/21/2019 14:21:32",
      "content": "<p>any reason for downvoting?</p>",
      "rawMarkdown": "any reason for downvoting?",
      "votes": null
    },
    {
      "id": "581640",
      "postDate": "07/22/2019 07:44:52",
      "content": "<p>I think it's not worth it. GANs are already difficult to train, and casting gradients to fp16 can introduce a new set of problems. And since kaggle GPUs are P100s, which belong to Pascal series, they are not so perfomant in half-precision operations (as Volta or RTX).</p>\n\n<p>However, you may consider mixed precision if your model converges fast, and you can trade some computation speed to fit more samples in GPUs memory.</p>",
      "rawMarkdown": "I think it's not worth it. GANs are already difficult to train, and casting gradients to fp16 can introduce a new set of problems. And since kaggle GPUs are P100s, which belong to Pascal series, they are not so perfomant in half-precision operations (as Volta or RTX).\n\nHowever, you may consider mixed precision if your model converges fast, and you can trade some computation speed to fit more samples in GPUs memory.",
      "votes": null
    },
    {
      "id": "581657",
      "postDate": "07/22/2019 08:21:38",
      "content": "<p>Hi Dmitry, thanks for you valuable comment. I can see that GAN is really really difficult to train.. I see what you mean :) mixed precision could generate other problems.\nI'm new to GAN, so I'm wondering if batch size affects the training of GANs. have you experimented with different batch sizes? so you think fitting more samples in the mini batches will help right? or not sure?</p>",
      "rawMarkdown": "Hi Dmitry, thanks for you valuable comment. I can see that GAN is really really difficult to train.. I see what you mean :) mixed precision could generate other problems.\nI'm new to GAN, so I'm wondering if batch size affects the training of GANs. have you experimented with different batch sizes? so you think fitting more samples in the mini batches will help right? or not sure?",
      "votes": null
    },
    {
      "id": "581707",
      "postDate": "07/22/2019 09:28:39",
      "content": "<p>I tested 32 and 64 mostly. The results were similar, so I am not sure which is better. I tried 128 once, and it performed poorly. And I briefly tested 24. It turned out to be worse than 32, but I didn't run it for long.</p>",
      "rawMarkdown": "I tested 32 and 64 mostly. The results were similar, so I am not sure which is better. I tried 128 once, and it performed poorly. And I briefly tested 24. It turned out to be worse than 32, but I didn't run it for long.",
      "votes": null
    },
    {
      "id": "581712",
      "postDate": "07/22/2019 09:33:38",
      "content": "<p>oh I see that you have extra local gpus. thanks for sharing your experiments :) have a nice day.</p>",
      "rawMarkdown": "oh I see that you have extra local gpus. thanks for sharing your experiments :) have a nice day.",
      "votes": null
    },
    {
      "id": "581719",
      "postDate": "07/22/2019 09:40:11",
      "content": "<p>I didn't use my local gpu, all test were done in kernels.</p>",
      "rawMarkdown": "I didn't use my local gpu, all test were done in kernels.",
      "votes": null
    },
    {
      "id": "581720",
      "postDate": "07/22/2019 09:41:09",
      "content": "<p>really? I thought batch size of 32 was maximum :) never mind</p>",
      "rawMarkdown": "really? I thought batch size of 32 was maximum :) never mind",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 581152,
      "author_name": "yangsaewon",
      "author_url": "",
      "post_date": "07/21/2019 14:21:32",
      "content": "<p>any reason for downvoting?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 581640,
      "author_name": "dvorobiev",
      "author_url": "",
      "post_date": "07/22/2019 07:44:52",
      "content": "<p>I think it's not worth it. GANs are already difficult to train, and casting gradients to fp16 can introduce a new set of problems. And since kaggle GPUs are P100s, which belong to Pascal series, they are not so perfomant in half-precision operations (as Volta or RTX).</p>\n\n<p>However, you may consider mixed precision if your model converges fast, and you can trade some computation speed to fit more samples in GPUs memory.</p>",
      "votes": null,
      "replies": [
        {
          "id": 581657,
          "author_name": "yangsaewon",
          "author_url": "",
          "post_date": "07/22/2019 08:21:38",
          "content": "<p>Hi Dmitry, thanks for you valuable comment. I can see that GAN is really really difficult to train.. I see what you mean :) mixed precision could generate other problems.\nI'm new to GAN, so I'm wondering if batch size affects the training of GANs. have you experimented with different batch sizes? so you think fitting more samples in the mini batches will help right? or not sure?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 581707,
          "author_name": "dvorobiev",
          "author_url": "",
          "post_date": "07/22/2019 09:28:39",
          "content": "<p>I tested 32 and 64 mostly. The results were similar, so I am not sure which is better. I tried 128 once, and it performed poorly. And I briefly tested 24. It turned out to be worse than 32, but I didn't run it for long.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 581712,
          "author_name": "yangsaewon",
          "author_url": "",
          "post_date": "07/22/2019 09:33:38",
          "content": "<p>oh I see that you have extra local gpus. thanks for sharing your experiments :) have a nice day.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 581719,
          "author_name": "dvorobiev",
          "author_url": "",
          "post_date": "07/22/2019 09:40:11",
          "content": "<p>I didn't use my local gpu, all test were done in kernels.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 581720,
          "author_name": "yangsaewon",
          "author_url": "",
          "post_date": "07/22/2019 09:41:09",
          "content": "<p>really? I thought batch size of 32 was maximum :) never mind</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "580608": "Hi, I'm new with GAN and I'm qurious whether we could use mixed precision in training gan.\n\nI can see that public kernels are training with batch_size of 32 which is very small.\n\nwhat if I use amp and increase the batch size?\n\nI think this could stabilize the training. am I wrong?\n\nwill increasing batch size help boost the score?",
    "581152": "any reason for downvoting?",
    "581640": "I think it's not worth it. GANs are already difficult to train, and casting gradients to fp16 can introduce a new set of problems. And since kaggle GPUs are P100s, which belong to Pascal series, they are not so perfomant in half-precision operations (as Volta or RTX).\n\nHowever, you may consider mixed precision if your model converges fast, and you can trade some computation speed to fit more samples in GPUs memory.",
    "581657": "Hi Dmitry, thanks for you valuable comment. I can see that GAN is really really difficult to train.. I see what you mean :) mixed precision could generate other problems.\nI'm new to GAN, so I'm wondering if batch size affects the training of GANs. have you experimented with different batch sizes? so you think fitting more samples in the mini batches will help right? or not sure?",
    "581707": "I tested 32 and 64 mostly. The results were similar, so I am not sure which is better. I tried 128 once, and it performed poorly. And I briefly tested 24. It turned out to be worse than 32, but I didn't run it for long.",
    "581712": "oh I see that you have extra local gpus. thanks for sharing your experiments :) have a nice day.",
    "581719": "I didn't use my local gpu, all test were done in kernels.",
    "581720": "really? I thought batch size of 32 was maximum :) never mind"
  },
  "source": "meta"
}