![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3924325%2F3fd056b48325bb84d52dbd9fd3c8c8b6%2FtriangularDiag.png?generation=1596180577431828&alt=media\")
Related paper: https://arxiv.org/pdf/1712.07628.pdf\n[update] Thanks @kirderf for pointing out CyclicLR, here is the paper about it!
", "messages": [ { "id": "949701", "postDate": "07/28/2020 20:23:44", "content": "Related paper: https://arxiv.org/pdf/1712.07628.pdf\n[update] Thanks @kirderf for pointing out CyclicLR, here is the paper about it!
", "rawMarkdown": "Related paper: https://arxiv.org/pdf/1712.07628.pdf\n**[update]** Thanks @kirderf for pointing out *CyclicLR*, [here](https://arxiv.org/abs/1506.01186) is the paper about it!", "votes": null }, { "id": "949770", "postDate": "07/28/2020 23:01:45", "content": "Thanks for sharing ! But in my case using SGD with EfficientDet gave me worse score ..
", "rawMarkdown": "Thanks for sharing ! But in my case using SGD with EfficientDet gave me worse score ..", "votes": null }, { "id": "950247", "postDate": "07/29/2020 09:42:40", "content": "Have you tried different LR? With SGD you should try to start with a large learning rate vs with Adam. You can also try different combinations of decay, momentum, and also with and without nesterov. I usually start with a decay of (1/steps per epoch) and LR=batchsize*0.001 or 0.01.
", "rawMarkdown": "Have you tried different LR? With SGD you should try to start with a large learning rate vs with Adam. You can also try different combinations of decay, momentum, and also with and without nesterov. I usually start with a decay of (1/steps per epoch) and LR=batchsize*0.001 or 0.01.", "votes": null }, { "id": "950349", "postDate": "07/29/2020 10:57:38", "content": "Exactly!
", "rawMarkdown": "Exactly!", "votes": null }, { "id": "950372", "postDate": "07/29/2020 11:08:52", "content": "Does nesterov momentum work?
", "rawMarkdown": "Does nesterov momentum work?", "votes": null }, { "id": "950383", "postDate": "07/29/2020 11:16:21", "content": "Sometimes, for example with OneCyle Scheduler,there is a research paper about it. I'm not a pro in that area.
", "rawMarkdown": "Sometimes, for example with OneCyle Scheduler,there is a research paper about it. I'm not a pro in that area.", "votes": null }, { "id": "950393", "postDate": "07/29/2020 11:19:50", "content": "🙏 thanks
", "rawMarkdown": "🙏 thanks", "votes": null }, { "id": "950653", "postDate": "07/29/2020 14:20:44", "content": "SGD is better but slower. It's slow to converge in my case
", "rawMarkdown": "SGD is better but slower. It's slow to converge in my case", "votes": null }, { "id": "950660", "postDate": "07/29/2020 14:25:38", "content": "Yes it's slow but give it a try together with torch.optim.lr_scheduler.CyclicLR, better and faster in some cases.
", "rawMarkdown": "Yes it's slow but give it a try together with torch.optim.lr_scheduler.CyclicLR, better and faster in some cases.", "votes": null }, { "id": "950932", "postDate": "07/29/2020 18:26:20", "content": "I had been using Adam with ReduceLrOnPlateau but for some unknown reason, ReduceLrOnPlateau decreased the learning rate by a lot(even though I set the factor a minimal 0.1) after the second epoch. I realised this while running it in verbose mode. This used to slow down training a lot. So I switched to SGD and no lr scheduler. This worked well and improved my score! However, it still can be improved, I am trying out different schedulers .
", "rawMarkdown": "I had been using Adam with ReduceLrOnPlateau but for some unknown reason, ReduceLrOnPlateau decreased the learning rate by a lot(even though I set the factor a minimal 0.1) after the second epoch. I realised this while running it in verbose mode. This used to slow down training a lot. So I switched to SGD and no lr scheduler. This worked well and improved my score! However, it still can be improved, I am trying out different schedulers .", "votes": null }, { "id": "950937", "postDate": "07/29/2020 18:29:02", "content": "Glad to hear about that! What learning rate did you choose? Is it with momentum?
", "rawMarkdown": "Glad to hear about that! What learning rate did you choose? Is it with momentum?", "votes": null }, { "id": "950939", "postDate": "07/29/2020 18:29:58", "content": "@kirderf I tried 0.01 lr for momentum I tried 0.937..\nI ll try what you said thanks !
", "rawMarkdown": "kirderf I tried 0.01 lr for momentum I tried 0.937..\nI ll try what you said thanks !", "votes": null }, { "id": "950942", "postDate": "07/29/2020 18:32:58", "content": "Seems it's same with YOLO5 LOL
", "rawMarkdown": "Seems it's same with YOLO5 LOL", "votes": null }, { "id": "950944", "postDate": "07/29/2020 18:34:49", "content": "btw, I trained with AdamW for 20 epochs and then switch to SGD to finetune.
", "rawMarkdown": "btw, I trained with AdamW for 20 epochs and then switch to SGD to finetune.", "votes": null }, { "id": "950956", "postDate": "07/29/2020 18:57:32", "content": "I chose a learning rate of 0.01 since I'm using a batch size of 2(anything larger than runs out of memory), I haven't tried adding momentum yet but it's on one of my to-dos!
", "rawMarkdown": "I chose a learning rate of 0.01 since I'm using a batch size of 2(anything larger than runs out of memory), I haven't tried adding momentum yet but it's on one of my to-dos!", "votes": null }, { "id": "950958", "postDate": "07/29/2020 18:57:38", "content": "If one uses torch.optim.lr scheduler.CyclicLR with SGD, try for 12 epochs for 1 cycle (step_size_up = steps x 6 epoch) and run 1-3 full cycles then lower the LR and do it again and again, see if it helps, I used that setting in another competition to speed up the SGD learning. Similar setting was in the research paper for this scheduler optim.lr scheduler.CyclicLR if I remember correctly. If it not works here it can be a something for the next to try, the scheduler works well with SGD anyway.
", "rawMarkdown": "If one uses torch.optim.lr scheduler.CyclicLR with SGD, try for 12 epochs for 1 cycle (step_size_up = steps x 6 epoch) and run 1-3 full cycles then lower the LR and do it again and again, see if it helps, I used that setting in another competition to speed up the SGD learning. Similar setting was in the research paper for this scheduler optim.lr scheduler.CyclicLR if I remember correctly. If it not works here it can be a something for the next to try, the scheduler works well with SGD anyway.", "votes": null }, { "id": "950970", "postDate": "07/29/2020 19:07:03", "content": "I'll try it out
", "rawMarkdown": "I'll try it out", "votes": null }, { "id": "950990", "postDate": "07/29/2020 19:29:06", "content": "as regards for Adam, it can work well with CosineAnnealingLR and GradualWarmupScheduler, saw it in past competitions. I have a training in progress now to see if it works well for this task.
", "rawMarkdown": "as regards for Adam, it can work well with CosineAnnealingLR and GradualWarmupScheduler, saw it in past competitions. I have a training in progress now to see if it works well for this task.", "votes": null }, { "id": "950998", "postDate": "07/29/2020 19:34:14", "content": "Thanks for sharing. appreciate it!
", "rawMarkdown": "Thanks for sharing. appreciate it!", "votes": null }, { "id": "951036", "postDate": "07/29/2020 20:34:32", "content": "I just read the paper. According to the paper, we need to find *base_lr* and *max_lr* by increasing the learning rate linearly and plot the accuracy curve. But I'm confused whether it increases after each batch or epoch? Any explanation is really appreciated!
", "rawMarkdown": "I just read the paper. According to the paper, we need to find *base_lr* and *max_lr* by increasing the learning rate linearly and plot the accuracy curve. But I'm confused whether it increases after each batch or epoch? Any explanation is really appreciated!", "votes": null }, { "id": "951107", "postDate": "07/29/2020 22:06:26", "content": "The LR is handle through the cycle, in the paper the baselr and maxlr are close to each other like 0.001 and 0.005 but I tried bigger distance like 0.001 and 0.00001 with better result. The scheduler is stable so it's more controlled by number of steps than the baselr. If the baselr is far from the maxlr it only use that value the last epoch, which I like, a stable symmetrical A / V formed LR curve but with a last touch of a much lower LR in the last epoch before the cycle starts again.
", "rawMarkdown": "The LR is handle through the cycle, in the paper the baselr and maxlr are close to each other like 0.001 and 0.005 but I tried bigger distance like 0.001 and 0.00001 with better result. The scheduler is stable so it's more controlled by number of steps than the baselr. If the baselr is far from the maxlr it only use that value the last epoch, which I like, a stable symmetrical A / V formed LR curve but with a last touch of a much lower LR in the last epoch before the cycle starts again.", "votes": null }, { "id": "952145", "postDate": "07/30/2020 16:43:18", "content": "Hi, I did try CyclicLR with base_lr=0.002, max_lr=0.035, mode='traiangular', bs=2, no momentum, no weight_decay. At first 20 epochs the training loss going down pretty fast. However, it got stuck at around 0.48. Am I doing anything wrong? Is the lr too large to converge?
", "rawMarkdown": "Hi, I did try CyclicLR with base_lr=0.002, max_lr=0.035, mode='traiangular', bs=2, no momentum, no weight_decay. At first 20 epochs the training loss going down pretty fast. However, it got stuck at around 0.48. Am I doing anything wrong? Is the lr too large to converge?", "votes": null }, { "id": "952168", "postDate": "07/30/2020 17:11:36", "content": "What's the stepsizeup, 10 epochs,? In the paper they say that the best accuracy is at the end of a full cycle and it's also the best time/epoch to switch to a lower LR, after a few cycles. I usually starts with a full cycle of 12 epochs and with baselr=0.0001 and maxlr=0.01(or batchsize*0.001.), then change maxlr to 0.001 and train two cycles and see how it goes, before test some other settings/standard settings. If it not goes well after some testing, I switch to another scheduler and/or optimizer. Is unfortunately different mix of optimizers and schudulers the works for different tasks but one can find a good generic to have in the bag to try every time, like this one :)
", "rawMarkdown": "What's the stepsizeup, 10 epochs,? In the paper they say that the best accuracy is at the end of a full cycle and it's also the best time/epoch to switch to a lower LR, after a few cycles. I usually starts with a full cycle of 12 epochs and with baselr=0.0001 and maxlr=0.01(or batchsize*0.001.), then change maxlr to 0.001 and train two cycles and see how it goes, before test some other settings/standard settings. If it not goes well after some testing, I switch to another scheduler and/or optimizer. Is unfortunately different mix of optimizers and schudulers the works for different tasks but one can find a good generic to have in the bag to try every time, like this one :)", "votes": null }, { "id": "952174", "postDate": "07/30/2020 17:24:01", "content": "Thanks. I chose step_up=6. Maybe the problem is I didn't reduce the max_lr after several cycles.
", "rawMarkdown": "Thanks. I chose step_up=6. Maybe the problem is I didn't reduce the max_lr after several cycles.", "votes": null }, { "id": "952177", "postDate": "07/30/2020 17:26:24", "content": "Also, I found Pytorch only has a 'triangular2' mode which half the range after a cycle. Seems the reduce policy you described is hard to implement...
", "rawMarkdown": "Also, I found Pytorch only has a 'triangular2' mode which half the range after a cycle. Seems the reduce policy you described is hard to implement...", "votes": null }, { "id": "952753", "postDate": "07/31/2020 07:30:26", "content": "In Pytorch you can use all three. I usually use the standard triangular setting,
\n\n
Below settings I used in other competition and with Adam, just to show an example and that cycle_momentum must be set to False to work without SGD.\nEach epoch had 1606 steps so below has a full cycle of 12 epochs.\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-6, max_lr=1e-4,step_size_up=9636, cycle_momentum=False)
", "rawMarkdown": "In Pytorch you can use all three. I usually use the standard triangular setting,\n\n\n\nBelow settings I used in other competition and with Adam, just to show an example and that cycle_momentum must be set to False to work without SGD.\nEach epoch had 1606 steps so below has a full cycle of 12 epochs.\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-6, max_lr=1e-4,step_size_up=9636, cycle_momentum=False)", "votes": null }, { "id": "952803", "postDate": "07/31/2020 08:27:44", "content": "sophisticated explanation. Thanks a lot!
", "rawMarkdown": "sophisticated explanation. Thanks a lot!", "votes": null }, { "id": "953160", "postDate": "07/31/2020 15:18:56", "content": "Here is another research paper that describes how SGD's tackles batch training generalization.\nMaybe that's why SGD works well in this task/problem due to apex and bigger batch sizes?\nhttps://arxiv.org/pdf/1705.08741.pdf
", "rawMarkdown": "Here is another research paper that describes how SGD's tackles batch training generalization.\nMaybe that's why SGD works well in this task/problem due to apex and bigger batch sizes?\nhttps://arxiv.org/pdf/1705.08741.pdf", "votes": null } ], "comments": [ { "id": 949770, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/28/2020 23:01:45", "content": "Thanks for sharing ! But in my case using SGD with EfficientDet gave me worse score ..
", "votes": null, "replies": [ { "id": 950247, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 09:42:40", "content": "Have you tried different LR? With SGD you should try to start with a large learning rate vs with Adam. You can also try different combinations of decay, momentum, and also with and without nesterov. I usually start with a decay of (1/steps per epoch) and LR=batchsize*0.001 or 0.01.
", "votes": null, "replies": [] }, { "id": 950349, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 10:57:38", "content": "Exactly!
", "votes": null, "replies": [] }, { "id": 950372, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 11:08:52", "content": "Does nesterov momentum work?
", "votes": null, "replies": [] }, { "id": 950383, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 11:16:21", "content": "Sometimes, for example with OneCyle Scheduler,there is a research paper about it. I'm not a pro in that area.
", "votes": null, "replies": [] }, { "id": 950393, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 11:19:50", "content": "🙏 thanks
", "votes": null, "replies": [] }, { "id": 950939, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/29/2020 18:29:58", "content": "@kirderf I tried 0.01 lr for momentum I tried 0.937..\nI ll try what you said thanks !
", "votes": null, "replies": [] }, { "id": 950942, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 18:32:58", "content": "Seems it's same with YOLO5 LOL
", "votes": null, "replies": [] } ] }, { "id": 950653, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/29/2020 14:20:44", "content": "SGD is better but slower. It's slow to converge in my case
", "votes": null, "replies": [ { "id": 950660, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 14:25:38", "content": "Yes it's slow but give it a try together with torch.optim.lr_scheduler.CyclicLR, better and faster in some cases.
", "votes": null, "replies": [] }, { "id": 950958, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 18:57:38", "content": "If one uses torch.optim.lr scheduler.CyclicLR with SGD, try for 12 epochs for 1 cycle (step_size_up = steps x 6 epoch) and run 1-3 full cycles then lower the LR and do it again and again, see if it helps, I used that setting in another competition to speed up the SGD learning. Similar setting was in the research paper for this scheduler optim.lr scheduler.CyclicLR if I remember correctly. If it not works here it can be a something for the next to try, the scheduler works well with SGD anyway.
", "votes": null, "replies": [] }, { "id": 950970, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 19:07:03", "content": "I'll try it out
", "votes": null, "replies": [] }, { "id": 951036, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 20:34:32", "content": "I just read the paper. According to the paper, we need to find *base_lr* and *max_lr* by increasing the learning rate linearly and plot the accuracy curve. But I'm confused whether it increases after each batch or epoch? Any explanation is really appreciated!
", "votes": null, "replies": [] }, { "id": 951107, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 22:06:26", "content": "The LR is handle through the cycle, in the paper the baselr and maxlr are close to each other like 0.001 and 0.005 but I tried bigger distance like 0.001 and 0.00001 with better result. The scheduler is stable so it's more controlled by number of steps than the baselr. If the baselr is far from the maxlr it only use that value the last epoch, which I like, a stable symmetrical A / V formed LR curve but with a last touch of a much lower LR in the last epoch before the cycle starts again.
", "votes": null, "replies": [] }, { "id": 952145, "author_name": "zacchaeus", "author_url": "", "post_date": "07/30/2020 16:43:18", "content": "Hi, I did try CyclicLR with base_lr=0.002, max_lr=0.035, mode='traiangular', bs=2, no momentum, no weight_decay. At first 20 epochs the training loss going down pretty fast. However, it got stuck at around 0.48. Am I doing anything wrong? Is the lr too large to converge?
", "votes": null, "replies": [] }, { "id": 952168, "author_name": "kirderf", "author_url": "", "post_date": "07/30/2020 17:11:36", "content": "What's the stepsizeup, 10 epochs,? In the paper they say that the best accuracy is at the end of a full cycle and it's also the best time/epoch to switch to a lower LR, after a few cycles. I usually starts with a full cycle of 12 epochs and with baselr=0.0001 and maxlr=0.01(or batchsize*0.001.), then change maxlr to 0.001 and train two cycles and see how it goes, before test some other settings/standard settings. If it not goes well after some testing, I switch to another scheduler and/or optimizer. Is unfortunately different mix of optimizers and schudulers the works for different tasks but one can find a good generic to have in the bag to try every time, like this one :)
", "votes": null, "replies": [] }, { "id": 952174, "author_name": "zacchaeus", "author_url": "", "post_date": "07/30/2020 17:24:01", "content": "Thanks. I chose step_up=6. Maybe the problem is I didn't reduce the max_lr after several cycles.
", "votes": null, "replies": [] }, { "id": 952177, "author_name": "zacchaeus", "author_url": "", "post_date": "07/30/2020 17:26:24", "content": "Also, I found Pytorch only has a 'triangular2' mode which half the range after a cycle. Seems the reduce policy you described is hard to implement...
", "votes": null, "replies": [] }, { "id": 952753, "author_name": "kirderf", "author_url": "", "post_date": "07/31/2020 07:30:26", "content": "In Pytorch you can use all three. I usually use the standard triangular setting,
\n\n
Below settings I used in other competition and with Adam, just to show an example and that cycle_momentum must be set to False to work without SGD.\nEach epoch had 1606 steps so below has a full cycle of 12 epochs.\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-6, max_lr=1e-4,step_size_up=9636, cycle_momentum=False)
", "votes": null, "replies": [] }, { "id": 952803, "author_name": "zacchaeus", "author_url": "", "post_date": "07/31/2020 08:27:44", "content": "sophisticated explanation. Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 950932, "author_name": "daenys2000", "author_url": "", "post_date": "07/29/2020 18:26:20", "content": "I had been using Adam with ReduceLrOnPlateau but for some unknown reason, ReduceLrOnPlateau decreased the learning rate by a lot(even though I set the factor a minimal 0.1) after the second epoch. I realised this while running it in verbose mode. This used to slow down training a lot. So I switched to SGD and no lr scheduler. This worked well and improved my score! However, it still can be improved, I am trying out different schedulers .
", "votes": null, "replies": [ { "id": 950937, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 18:29:02", "content": "Glad to hear about that! What learning rate did you choose? Is it with momentum?
", "votes": null, "replies": [] }, { "id": 950956, "author_name": "daenys2000", "author_url": "", "post_date": "07/29/2020 18:57:32", "content": "I chose a learning rate of 0.01 since I'm using a batch size of 2(anything larger than runs out of memory), I haven't tried adding momentum yet but it's on one of my to-dos!
", "votes": null, "replies": [] } ] }, { "id": 950944, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 18:34:49", "content": "btw, I trained with AdamW for 20 epochs and then switch to SGD to finetune.
", "votes": null, "replies": [] }, { "id": 950990, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 19:29:06", "content": "as regards for Adam, it can work well with CosineAnnealingLR and GradualWarmupScheduler, saw it in past competitions. I have a training in progress now to see if it works well for this task.
", "votes": null, "replies": [ { "id": 950998, "author_name": "zacchaeus", "author_url": "", "post_date": "07/29/2020 19:34:14", "content": "Thanks for sharing. appreciate it!
", "votes": null, "replies": [] } ] }, { "id": 953160, "author_name": "kirderf", "author_url": "", "post_date": "07/31/2020 15:18:56", "content": "Here is another research paper that describes how SGD's tackles batch training generalization.\nMaybe that's why SGD works well in this task/problem due to apex and bigger batch sizes?\nhttps://arxiv.org/pdf/1705.08741.pdf
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "949701": "Related paper: https://arxiv.org/pdf/1712.07628.pdf\n**[update]** Thanks @kirderf for pointing out *CyclicLR*, [here](https://arxiv.org/abs/1506.01186) is the paper about it!", "949770": "Thanks for sharing ! But in my case using SGD with EfficientDet gave me worse score ..", "950247": "Have you tried different LR? With SGD you should try to start with a large learning rate vs with Adam. You can also try different combinations of decay, momentum, and also with and without nesterov. I usually start with a decay of (1/steps per epoch) and LR=batchsize*0.001 or 0.01.", "950349": "Exactly!", "950372": "Does nesterov momentum work?", "950383": "Sometimes, for example with OneCyle Scheduler,there is a research paper about it. I'm not a pro in that area.", "950393": "🙏 thanks", "950653": "SGD is better but slower. It's slow to converge in my case", "950660": "Yes it's slow but give it a try together with torch.optim.lr_scheduler.CyclicLR, better and faster in some cases.", "950932": "I had been using Adam with ReduceLrOnPlateau but for some unknown reason, ReduceLrOnPlateau decreased the learning rate by a lot(even though I set the factor a minimal 0.1) after the second epoch. I realised this while running it in verbose mode. This used to slow down training a lot. So I switched to SGD and no lr scheduler. This worked well and improved my score! However, it still can be improved, I am trying out different schedulers .", "950937": "Glad to hear about that! What learning rate did you choose? Is it with momentum?", "950939": "kirderf I tried 0.01 lr for momentum I tried 0.937..\nI ll try what you said thanks !", "950942": "Seems it's same with YOLO5 LOL", "950944": "btw, I trained with AdamW for 20 epochs and then switch to SGD to finetune.", "950956": "I chose a learning rate of 0.01 since I'm using a batch size of 2(anything larger than runs out of memory), I haven't tried adding momentum yet but it's on one of my to-dos!", "950958": "If one uses torch.optim.lr scheduler.CyclicLR with SGD, try for 12 epochs for 1 cycle (step_size_up = steps x 6 epoch) and run 1-3 full cycles then lower the LR and do it again and again, see if it helps, I used that setting in another competition to speed up the SGD learning. Similar setting was in the research paper for this scheduler optim.lr scheduler.CyclicLR if I remember correctly. If it not works here it can be a something for the next to try, the scheduler works well with SGD anyway.", "950970": "I'll try it out", "950990": "as regards for Adam, it can work well with CosineAnnealingLR and GradualWarmupScheduler, saw it in past competitions. I have a training in progress now to see if it works well for this task.", "950998": "Thanks for sharing. appreciate it!", "951036": "I just read the paper. According to the paper, we need to find *base_lr* and *max_lr* by increasing the learning rate linearly and plot the accuracy curve. But I'm confused whether it increases after each batch or epoch? Any explanation is really appreciated!", "951107": "The LR is handle through the cycle, in the paper the baselr and maxlr are close to each other like 0.001 and 0.005 but I tried bigger distance like 0.001 and 0.00001 with better result. The scheduler is stable so it's more controlled by number of steps than the baselr. If the baselr is far from the maxlr it only use that value the last epoch, which I like, a stable symmetrical A / V formed LR curve but with a last touch of a much lower LR in the last epoch before the cycle starts again.", "952145": "Hi, I did try CyclicLR with base_lr=0.002, max_lr=0.035, mode='traiangular', bs=2, no momentum, no weight_decay. At first 20 epochs the training loss going down pretty fast. However, it got stuck at around 0.48. Am I doing anything wrong? Is the lr too large to converge?", "952168": "What's the stepsizeup, 10 epochs,? In the paper they say that the best accuracy is at the end of a full cycle and it's also the best time/epoch to switch to a lower LR, after a few cycles. I usually starts with a full cycle of 12 epochs and with baselr=0.0001 and maxlr=0.01(or batchsize*0.001.), then change maxlr to 0.001 and train two cycles and see how it goes, before test some other settings/standard settings. If it not goes well after some testing, I switch to another scheduler and/or optimizer. Is unfortunately different mix of optimizers and schudulers the works for different tasks but one can find a good generic to have in the bag to try every time, like this one :)", "952174": "Thanks. I chose step_up=6. Maybe the problem is I didn't reduce the max_lr after several cycles.", "952177": "Also, I found Pytorch only has a 'triangular2' mode which half the range after a cycle. Seems the reduce policy you described is hard to implement...", "952753": "In Pytorch you can use all three. I usually use the standard triangular setting,\n\n\n\nBelow settings I used in other competition and with Adam, just to show an example and that cycle_momentum must be set to False to work without SGD.\nEach epoch had 1606 steps so below has a full cycle of 12 epochs.\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-6, max_lr=1e-4,step_size_up=9636, cycle_momentum=False)", "952803": "sophisticated explanation. Thanks a lot!", "953160": "Here is another research paper that describes how SGD's tackles batch training generalization.\nMaybe that's why SGD works well in this task/problem due to apex and bigger batch sizes?\nhttps://arxiv.org/pdf/1705.08741.pdf" }, "source": "meta" }