![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19298%2Fbfd982134bcc8b552cbf7d384ca18f1e%2F__results___16_0.png?generation=1563658186093083&alt=media\")
In our experiments we are noticing that we have better images halfway the training (at about 300 epochs out of 650 possible in 8 hours). We would like to know if you are having the same experience and if you ideas about this (could it be overfitting)?
", "messages": [ { "id": "580366", "postDate": "07/20/2019 03:45:50", "content": "In our experiments we are noticing that we have better images halfway the training (at about 300 epochs out of 650 possible in 8 hours). We would like to know if you are having the same experience and if you ideas about this (could it be overfitting)?
", "rawMarkdown": "In our experiments we are noticing that we have better images halfway the training (at about 300 epochs out of 650 possible in 8 hours). We would like to know if you are having the same experience and if you ideas about this (could it be overfitting)?", "votes": null }, { "id": "580555", "postDate": "07/20/2019 10:58:42", "content": "Yes, I've noticed the same thing. At some point the model starts to overshoot, and the FID value can repeatedly bounce high and low. Maybe some sort of lr scheduling could help, but I haven't tried yet
", "rawMarkdown": "Yes, I've noticed the same thing. At some point the model starts to overshoot, and the FID value can repeatedly bounce high and low. Maybe some sort of lr scheduling could help, but I haven't tried yet", "votes": null }, { "id": "580564", "postDate": "07/20/2019 11:19:53", "content": "I guess your network has converged, you can try going deeper, or reducing your learning rate.
", "rawMarkdown": "I guess your network has converged, you can try going deeper, or reducing your learning rate.", "votes": null }, { "id": "580729", "postDate": "07/20/2019 16:20:44", "content": "how are you guys deciding when to stop training? looking at the loss convergence? or just submit and see the lb score?
", "rawMarkdown": "how are you guys deciding when to stop training? looking at the loss convergence? or just submit and see the lb score?", "votes": null }, { "id": "580803", "postDate": "07/20/2019 19:39:46", "content": "\nif time.perf_counter() - kernel_start_time > 32000:\n print(\"Time limit reached! Stopping kernel!\") \n break\n\nThis is how I stop my training. I train for the maximum kernel time.
@dvorobiev @theoviel Thank you for your suggestions, the solution could be early stopping or a decreasing planned learning rate. I am wondering if to use the generator smoothed rate of change as a possible indicator: clearly after 300 epochs the error series becomes stationary:
\n\n
thanks for the code! so you are training for the maximum time limit because it gives you the best score?
", "rawMarkdown": "thanks for the code! so you are training for the maximum time limit because it gives you the best score?", "votes": null }, { "id": "581813", "postDate": "07/22/2019 12:38:43", "content": "I have done an LR scheduling, as it was used in memorizer GAN in pytorch : with a little tweak based on my Epochs (800 Epochs ) . However I still found the score (65 using starter code ) got worst in 1400 epoch (76) .
\n\n\nlr_schedulerG = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizerG,\n T_0=epochs//200, eta_min=0.00005)\nlr_schedulerD = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizerD,\n T_0=epochs//200, eta_min=0.00005)\n
1400 seems a lot for me :) Maybe you've already reached your network capacity with 800 epochs. In this case I would try adding more layers or increasing number of filters first.
", "rawMarkdown": "1400 seems a lot for me :) Maybe you've already reached your network capacity with 800 epochs. In this case I would try adding more layers or increasing number of filters first.", "votes": null }, { "id": "582197", "postDate": "07/22/2019 22:07:09", "content": "I haven't actually tested if it gives the best score for my RaLSGAN but I can assure that more epochs give a better score for the normal GAN (best for maximum epochs) derived from the starter code. I will verify and post my finding ASAP.
", "rawMarkdown": "I haven't actually tested if it gives the best score for my RaLSGAN but I can assure that more epochs give a better score for the normal GAN (best for maximum epochs) derived from the starter code. I will verify and post my finding ASAP.", "votes": null } ], "comments": [ { "id": 580555, "author_name": "dvorobiev", "author_url": "", "post_date": "07/20/2019 10:58:42", "content": "Yes, I've noticed the same thing. At some point the model starts to overshoot, and the FID value can repeatedly bounce high and low. Maybe some sort of lr scheduling could help, but I haven't tried yet
", "votes": null, "replies": [ { "id": 581813, "author_name": "phoenix9032", "author_url": "", "post_date": "07/22/2019 12:38:43", "content": "I have done an LR scheduling, as it was used in memorizer GAN in pytorch : with a little tweak based on my Epochs (800 Epochs ) . However I still found the score (65 using starter code ) got worst in 1400 epoch (76) .
\n\n\nlr_schedulerG = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizerG,\n T_0=epochs//200, eta_min=0.00005)\nlr_schedulerD = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizerD,\n T_0=epochs//200, eta_min=0.00005)\n
1400 seems a lot for me :) Maybe you've already reached your network capacity with 800 epochs. In this case I would try adding more layers or increasing number of filters first.
", "votes": null, "replies": [] } ] }, { "id": 580564, "author_name": "theoviel", "author_url": "", "post_date": "07/20/2019 11:19:53", "content": "I guess your network has converged, you can try going deeper, or reducing your learning rate.
", "votes": null, "replies": [] }, { "id": 580729, "author_name": "yangsaewon", "author_url": "", "post_date": "07/20/2019 16:20:44", "content": "how are you guys deciding when to stop training? looking at the loss convergence? or just submit and see the lb score?
", "votes": null, "replies": [ { "id": 580803, "author_name": "timetraveller98", "author_url": "", "post_date": "07/20/2019 19:39:46", "content": "\nif time.perf_counter() - kernel_start_time > 32000:\n print(\"Time limit reached! Stopping kernel!\") \n break\n\nThis is how I stop my training. I train for the maximum kernel time.
thanks for the code! so you are training for the maximum time limit because it gives you the best score?
", "votes": null, "replies": [] }, { "id": 582197, "author_name": "timetraveller98", "author_url": "", "post_date": "07/22/2019 22:07:09", "content": "I haven't actually tested if it gives the best score for my RaLSGAN but I can assure that more epochs give a better score for the normal GAN (best for maximum epochs) derived from the starter code. I will verify and post my finding ASAP.
", "votes": null, "replies": [] } ] }, { "id": 580840, "author_name": "lucamassaron", "author_url": "", "post_date": "07/20/2019 21:29:53", "content": "@dvorobiev @theoviel Thank you for your suggestions, the solution could be early stopping or a decreasing planned learning rate. I am wondering if to use the generator smoothed rate of change as a possible indicator: clearly after 300 epochs the error series becomes stationary:
\n\n