![\"image\"](\"http://playagricola.com/Kaggle/score71419.png\")
When we use GANs, it is the problem that how we select the best generator in training loops.
\n\nIn this competition, the final score is MiFID score by the secret model trained on the secret dataset, so we cannot use this metric to select the best generator.\nHow do you select the generator? Followings are my candidates : \n- Select an epoch when Generator's loss is the lowest.\n- Use ImageNet-pretrained Inception to calculate MiFID, and then select the best generator. (But is it allowed to use ImageNet-pretrained in this competition?\n- Use some randomly-initialized models to calculate MiFID, and then select the best generator.\n- Use the latest generator in the training loop.
\n\n---- Edit ----\nHere is a similar question and answer from the Kaggle : https://www.kaggle.com/c/generative-dog-images/discussion/97780#latest-564399
", "messages": [ { "id": "594082", "postDate": "08/07/2019 14:39:24", "content": "When we use GANs, it is the problem that how we select the best generator in training loops.
\n\nIn this competition, the final score is MiFID score by the secret model trained on the secret dataset, so we cannot use this metric to select the best generator.\nHow do you select the generator? Followings are my candidates : \n- Select an epoch when Generator's loss is the lowest.\n- Use ImageNet-pretrained Inception to calculate MiFID, and then select the best generator. (But is it allowed to use ImageNet-pretrained in this competition?\n- Use some randomly-initialized models to calculate MiFID, and then select the best generator.\n- Use the latest generator in the training loop.
\n\n---- Edit ----\nHere is a similar question and answer from the Kaggle : https://www.kaggle.com/c/generative-dog-images/discussion/97780#latest-564399
", "rawMarkdown": "When we use GANs, it is the problem that how we select the best generator in training loops. \n\nIn this competition, the final score is MiFID score by the secret model trained on the secret dataset, so we cannot use this metric to select the best generator.\nHow do you select the generator? Followings are my candidates : \n- Select an epoch when Generator's loss is the lowest.\n- Use ImageNet-pretrained Inception to calculate MiFID, and then select the best generator. (But is it allowed to use ImageNet-pretrained in this competition?\n- Use some randomly-initialized models to calculate MiFID, and then select the best generator.\n- Use the latest generator in the training loop.\n\n---- Edit ----\nHere is a similar question and answer from the Kaggle : https://www.kaggle.com/c/generative-dog-images/discussion/97780#latest-564399", "votes": null }, { "id": "594083", "postDate": "08/07/2019 14:45:42", "content": "Select an epoch when Generator's loss is the lowest. - Nope . For GAN it is very erratic . We cant pin on this .\nI have experimented this : For one kernel goes one of the minimal loss at the epoch 547 . I stopped there . MFiD = 78\nI executed for more time . at 800 epoch loss is more - MfID -66
", "rawMarkdown": "Select an epoch when Generator's loss is the lowest. - Nope . For GAN it is very erratic . We cant pin on this .\nI have experimented this : For one kernel goes one of the minimal loss at the epoch 547 . I stopped there . MFiD = 78\nI executed for more time . at 800 epoch loss is more - MfID -66", "votes": null }, { "id": "594087", "postDate": "08/07/2019 14:55:59", "content": "Thanks for sharing your result.\nYeah, it seems difficult to select the generator only using generator's loss. If Discriminator is not strong enough, generator's loss goes down. \nAn alternative is to focus on (Discriminator's loss + Generator's loss) ... ?
", "rawMarkdown": "Thanks for sharing your result.\nYeah, it seems difficult to select the generator only using generator's loss. If Discriminator is not strong enough, generator's loss goes down. \nAn alternative is to focus on (Discriminator's loss + Generator's loss) ... ?", "votes": null }, { "id": "594091", "postDate": "08/07/2019 14:59:30", "content": "Knowing the variable nature of GAN losses , I would not go for stopping based on loss . I use loss diagrams just to see if training is stabilizing or not . But I have very little knowledge (about 4 weeks) . probably someone with more knowledge can confirm .
", "rawMarkdown": "Knowing the variable nature of GAN losses , I would not go for stopping based on loss . I use loss diagrams just to see if training is stabilizing or not . But I have very little knowledge (about 4 weeks) . probably someone with more knowledge can confirm .", "votes": null }, { "id": "594962", "postDate": "08/08/2019 17:26:46", "content": "Since the training process is a min-max game and its very unlikely that we'll find an equilibrium, its better to run the training as long as possible. Maybe it'll help to have a check for mode collapse.
", "rawMarkdown": "Since the training process is a min-max game and its very unlikely that we'll find an equilibrium, its better to run the training as long as possible. Maybe it'll help to have a check for mode collapse.", "votes": null }, { "id": "595040", "postDate": "08/08/2019 19:25:36", "content": "I thought the same but I don't understand why (in most of the cases) my FID is a lot better at ~100-200 epochs before infinity (max kernel time). I am still not sure how am I supposed to choose my final submission (trained until just the best public lb epoch or infinity)
", "rawMarkdown": "I thought the same but I don't understand why (in most of the cases) my FID is a lot better at ~100-200 epochs before infinity (max kernel time). I am still not sure how am I supposed to choose my final submission (trained until just the best public lb epoch or infinity)", "votes": null }, { "id": "595047", "postDate": "08/08/2019 19:31:28", "content": "Oh interesting! I should try a submission with fewer epochs, thanks. GANs are so random 🙄
", "rawMarkdown": "Oh interesting! I should try a submission with fewer epochs, thanks. GANs are so random 🙄", "votes": null }, { "id": "595048", "postDate": "08/08/2019 19:34:41", "content": "You should definitely try that. \nFor me, the difference was ~4 fid between the best and last epoch.\nBut the doubt still stands, is this gain only for the public LB? I am not sure what will happen on the actual private set.
", "rawMarkdown": "You should definitely try that. \nFor me, the difference was ~4 fid between the best and last epoch.\nBut the doubt still stands, is this gain only for the public LB? I am not sure what will happen on the actual private set.", "votes": null }, { "id": "595066", "postDate": "08/08/2019 19:56:02", "content": "The GAN and its evaluation is so complex, I think you should just submit the scores that performs the best on the public LB. At least thats what im going to do.
", "rawMarkdown": "The GAN and its evaluation is so complex, I think you should just submit the scores that performs the best on the public LB. At least thats what im going to do.", "votes": null }, { "id": "595345", "postDate": "08/09/2019 05:51:26", "content": "I just calculated the score for every 100 epochs :)
", "rawMarkdown": "I just calculated the score for every 100 epochs :)", "votes": null }, { "id": "595585", "postDate": "08/09/2019 12:08:39", "content": "In one kernel, I use MiFID code to calculate and plot MiFID versus epoch. Using that plot, I choose the best epoch. In a second kernel, I run my GAN for that many epochs and submit. Below we see that my RaLSGAN can run the full 200 epochs. Some GANs start to increase their MiFID after a certain epoch count, so for those GANs, I'll need to choose a number that is not 200.
\n\n
That looks like a score below ~40 which is pretty awesome!
", "rawMarkdown": "That looks like a score below ~40 which is pretty awesome!", "votes": null }, { "id": "595727", "postDate": "08/09/2019 15:26:11", "content": "Thanks :)\nHow do you calculate MiFID? You use ImageNet pretrained Inception in your code? Is it allowed to use external dataset (ImageNet) in this competition?
", "rawMarkdown": "Thanks :)\nHow do you calculate MiFID? You use ImageNet pretrained Inception in your code? Is it allowed to use external dataset (ImageNet) in this competition?", "votes": null }, { "id": "595730", "postDate": "08/09/2019 15:27:58", "content": "Thanks, I also think that to train as long as possible may be a promising method.
", "rawMarkdown": "Thanks, I also think that to train as long as possible may be a promising method.", "votes": null }, { "id": "595734", "postDate": "08/09/2019 15:34:11", "content": "Thank you for sharing interesting result.\nFrom the figure, MiFID seems to decrease as training using ImageNet pretrained Inception.\nWe need to hope that the secret dataset and the secret model in the private testing are the similar to public ones...
", "rawMarkdown": "Thank you for sharing interesting result.\nFrom the figure, MiFID seems to decrease as training using ImageNet pretrained Inception.\nWe need to hope that the secret dataset and the secret model in the private testing are the similar to public ones...", "votes": null }, { "id": "595743", "postDate": "08/09/2019 15:46:45", "content": "I wish! That GAN only gets 65.
", "rawMarkdown": "I wish! That GAN only gets 65.", "votes": null }, { "id": "596012", "postDate": "08/10/2019 02:22:56", "content": "I've been running experiments on an AWS box and calculating FID every 10 epochs using this library: https://github.com/mseitzer/pytorch-fid. I've found the FID score corresponds pretty well with the MiFID score, just normally about 20 points higher.
\n\nI don't think using external models would be allowed for your submission kernel but I guess you could always use it in a separate kernel to find the optimum epoch and then just hardcode that for your submission.
", "rawMarkdown": "I've been running experiments on an AWS box and calculating FID every 10 epochs using this library: https://github.com/mseitzer/pytorch-fid. I've found the FID score corresponds pretty well with the MiFID score, just normally about 20 points higher. \n\nI don't think using external models would be allowed for your submission kernel but I guess you could always use it in a separate kernel to find the optimum epoch and then just hardcode that for your submission.", "votes": null }, { "id": "598194", "postDate": "08/13/2019 08:53:50", "content": "yes I use it in the separate kernel not for submission :)
", "rawMarkdown": "yes I use it in the separate kernel not for submission :)", "votes": null } ], "comments": [ { "id": 594083, "author_name": "phoenix9032", "author_url": "", "post_date": "08/07/2019 14:45:42", "content": "Select an epoch when Generator's loss is the lowest. - Nope . For GAN it is very erratic . We cant pin on this .\nI have experimented this : For one kernel goes one of the minimal loss at the epoch 547 . I stopped there . MFiD = 78\nI executed for more time . at 800 epoch loss is more - MfID -66
", "votes": null, "replies": [ { "id": 594087, "author_name": "tanakin", "author_url": "", "post_date": "08/07/2019 14:55:59", "content": "Thanks for sharing your result.\nYeah, it seems difficult to select the generator only using generator's loss. If Discriminator is not strong enough, generator's loss goes down. \nAn alternative is to focus on (Discriminator's loss + Generator's loss) ... ?
", "votes": null, "replies": [] }, { "id": 594091, "author_name": "phoenix9032", "author_url": "", "post_date": "08/07/2019 14:59:30", "content": "Knowing the variable nature of GAN losses , I would not go for stopping based on loss . I use loss diagrams just to see if training is stabilizing or not . But I have very little knowledge (about 4 weeks) . probably someone with more knowledge can confirm .
", "votes": null, "replies": [] } ] }, { "id": 594962, "author_name": "yk1598", "author_url": "", "post_date": "08/08/2019 17:26:46", "content": "Since the training process is a min-max game and its very unlikely that we'll find an equilibrium, its better to run the training as long as possible. Maybe it'll help to have a check for mode collapse.
", "votes": null, "replies": [ { "id": 595040, "author_name": "timetraveller98", "author_url": "", "post_date": "08/08/2019 19:25:36", "content": "I thought the same but I don't understand why (in most of the cases) my FID is a lot better at ~100-200 epochs before infinity (max kernel time). I am still not sure how am I supposed to choose my final submission (trained until just the best public lb epoch or infinity)
", "votes": null, "replies": [] }, { "id": 595047, "author_name": "yk1598", "author_url": "", "post_date": "08/08/2019 19:31:28", "content": "Oh interesting! I should try a submission with fewer epochs, thanks. GANs are so random 🙄
", "votes": null, "replies": [] }, { "id": 595048, "author_name": "timetraveller98", "author_url": "", "post_date": "08/08/2019 19:34:41", "content": "You should definitely try that. \nFor me, the difference was ~4 fid between the best and last epoch.\nBut the doubt still stands, is this gain only for the public LB? I am not sure what will happen on the actual private set.
", "votes": null, "replies": [] }, { "id": 595066, "author_name": "shaygu", "author_url": "", "post_date": "08/08/2019 19:56:02", "content": "The GAN and its evaluation is so complex, I think you should just submit the scores that performs the best on the public LB. At least thats what im going to do.
", "votes": null, "replies": [] }, { "id": 595730, "author_name": "tanakin", "author_url": "", "post_date": "08/09/2019 15:27:58", "content": "Thanks, I also think that to train as long as possible may be a promising method.
", "votes": null, "replies": [] } ] }, { "id": 595345, "author_name": "yangsaewon", "author_url": "", "post_date": "08/09/2019 05:51:26", "content": "I just calculated the score for every 100 epochs :)
", "votes": null, "replies": [ { "id": 595727, "author_name": "tanakin", "author_url": "", "post_date": "08/09/2019 15:26:11", "content": "Thanks :)\nHow do you calculate MiFID? You use ImageNet pretrained Inception in your code? Is it allowed to use external dataset (ImageNet) in this competition?
", "votes": null, "replies": [] }, { "id": 596012, "author_name": "weoflnwef09i32s", "author_url": "", "post_date": "08/10/2019 02:22:56", "content": "I've been running experiments on an AWS box and calculating FID every 10 epochs using this library: https://github.com/mseitzer/pytorch-fid. I've found the FID score corresponds pretty well with the MiFID score, just normally about 20 points higher.
\n\nI don't think using external models would be allowed for your submission kernel but I guess you could always use it in a separate kernel to find the optimum epoch and then just hardcode that for your submission.
", "votes": null, "replies": [] }, { "id": 598194, "author_name": "yangsaewon", "author_url": "", "post_date": "08/13/2019 08:53:50", "content": "yes I use it in the separate kernel not for submission :)
", "votes": null, "replies": [] } ] }, { "id": 595585, "author_name": "cdeotte", "author_url": "", "post_date": "08/09/2019 12:08:39", "content": "In one kernel, I use MiFID code to calculate and plot MiFID versus epoch. Using that plot, I choose the best epoch. In a second kernel, I run my GAN for that many epochs and submit. Below we see that my RaLSGAN can run the full 200 epochs. Some GANs start to increase their MiFID after a certain epoch count, so for those GANs, I'll need to choose a number that is not 200.
\n\n
That looks like a score below ~40 which is pretty awesome!
", "votes": null, "replies": [] }, { "id": 595734, "author_name": "tanakin", "author_url": "", "post_date": "08/09/2019 15:34:11", "content": "Thank you for sharing interesting result.\nFrom the figure, MiFID seems to decrease as training using ImageNet pretrained Inception.\nWe need to hope that the secret dataset and the secret model in the private testing are the similar to public ones...
", "votes": null, "replies": [] }, { "id": 595743, "author_name": "cdeotte", "author_url": "", "post_date": "08/09/2019 15:46:45", "content": "I wish! That GAN only gets 65.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "594082": "When we use GANs, it is the problem that how we select the best generator in training loops. \n\nIn this competition, the final score is MiFID score by the secret model trained on the secret dataset, so we cannot use this metric to select the best generator.\nHow do you select the generator? Followings are my candidates : \n- Select an epoch when Generator's loss is the lowest.\n- Use ImageNet-pretrained Inception to calculate MiFID, and then select the best generator. (But is it allowed to use ImageNet-pretrained in this competition?\n- Use some randomly-initialized models to calculate MiFID, and then select the best generator.\n- Use the latest generator in the training loop.\n\n---- Edit ----\nHere is a similar question and answer from the Kaggle : https://www.kaggle.com/c/generative-dog-images/discussion/97780#latest-564399", "594083": "Select an epoch when Generator's loss is the lowest. - Nope . For GAN it is very erratic . We cant pin on this .\nI have experimented this : For one kernel goes one of the minimal loss at the epoch 547 . I stopped there . MFiD = 78\nI executed for more time . at 800 epoch loss is more - MfID -66", "594087": "Thanks for sharing your result.\nYeah, it seems difficult to select the generator only using generator's loss. If Discriminator is not strong enough, generator's loss goes down. \nAn alternative is to focus on (Discriminator's loss + Generator's loss) ... ?", "594091": "Knowing the variable nature of GAN losses , I would not go for stopping based on loss . I use loss diagrams just to see if training is stabilizing or not . But I have very little knowledge (about 4 weeks) . probably someone with more knowledge can confirm .", "594962": "Since the training process is a min-max game and its very unlikely that we'll find an equilibrium, its better to run the training as long as possible. Maybe it'll help to have a check for mode collapse.", "595040": "I thought the same but I don't understand why (in most of the cases) my FID is a lot better at ~100-200 epochs before infinity (max kernel time). I am still not sure how am I supposed to choose my final submission (trained until just the best public lb epoch or infinity)", "595047": "Oh interesting! I should try a submission with fewer epochs, thanks. GANs are so random 🙄", "595048": "You should definitely try that. \nFor me, the difference was ~4 fid between the best and last epoch.\nBut the doubt still stands, is this gain only for the public LB? I am not sure what will happen on the actual private set.", "595066": "The GAN and its evaluation is so complex, I think you should just submit the scores that performs the best on the public LB. At least thats what im going to do.", "595345": "I just calculated the score for every 100 epochs :)", "595585": "In one kernel, I use MiFID code to calculate and plot MiFID versus epoch. Using that plot, I choose the best epoch. In a second kernel, I run my GAN for that many epochs and submit. Below we see that my RaLSGAN can run the full 200 epochs. Some GANs start to increase their MiFID after a certain epoch count, so for those GANs, I'll need to choose a number that is not 200.\n\n", "595626": "That looks like a score below ~40 which is pretty awesome!", "595727": "Thanks :)\nHow do you calculate MiFID? You use ImageNet pretrained Inception in your code? Is it allowed to use external dataset (ImageNet) in this competition?", "595730": "Thanks, I also think that to train as long as possible may be a promising method.", "595734": "Thank you for sharing interesting result.\nFrom the figure, MiFID seems to decrease as training using ImageNet pretrained Inception.\nWe need to hope that the secret dataset and the secret model in the private testing are the similar to public ones...", "595743": "I wish! That GAN only gets 65.", "596012": "I've been running experiments on an AWS box and calculating FID every 10 epochs using this library: https://github.com/mseitzer/pytorch-fid. I've found the FID score corresponds pretty well with the MiFID score, just normally about 20 points higher. \n\nI don't think using external models would be allowed for your submission kernel but I guess you could always use it in a separate kernel to find the optimum epoch and then just hardcode that for your submission.", "598194": "yes I use it in the separate kernel not for submission :)" }, "source": "meta" }