A reflection if one just want to use Pytorch built-in features. Combination of AdamW and torch.optim.lr_scheduler.ExponentialLR with gamma=0.75 seems to have quicker learning curve vs other schedulers I tried. Not significantly better as regards for accuracy, but faster. It reduces the LR for every epoch and with decays of LR * .75, so quite steep, but it seems to work well. Gamma=0.85 can also be a good option and works with other optimizers as well, and when the val-loss is losing momentum then switch to gamma=0.75.
", "messages": [ { "id": "957656", "postDate": "08/04/2020 13:25:33", "content": "A reflection if one just want to use Pytorch built-in features. Combination of AdamW and torch.optim.lr_scheduler.ExponentialLR with gamma=0.75 seems to have quicker learning curve vs other schedulers I tried. Not significantly better as regards for accuracy, but faster. It reduces the LR for every epoch and with decays of LR * .75, so quite steep, but it seems to work well. Gamma=0.85 can also be a good option and works with other optimizers as well, and when the val-loss is losing momentum then switch to gamma=0.75.
", "rawMarkdown": "A reflection if one just want to use Pytorch built-in features. Combination of AdamW and torch.optim.lr_scheduler.ExponentialLR with gamma=0.75 seems to have quicker learning curve vs other schedulers I tried. Not significantly better as regards for accuracy, but faster. It reduces the LR for every epoch and with decays of LR * .75, so quite steep, but it seems to work well. Gamma=0.85 can also be a good option and works with other optimizers as well, and when the val-loss is losing momentum then switch to gamma=0.75.", "votes": null }, { "id": "957984", "postDate": "08/04/2020 17:07:22", "content": "Too late to try it in this competition. I shall do it after that.
", "rawMarkdown": "Too late to try it in this competition. I shall do it after that.", "votes": null }, { "id": "958075", "postDate": "08/04/2020 18:45:09", "content": "Talking about Effdet models, we never faced any issues with ReduceLRonPlateau scheduler with AdamW.\nIt gave us great validation losses which produced as good as 0.74 on 5-fold local CV (OOF score) but those models seemed to had overfit the train set.\nWe noticed that beyond a certain OOF score for our models the public LB score only reduced. Thats why we stopped training after reaching certain Val loss for every one of our models.
\n\nI'm unsure at this point whether or not this 'overfitting' was due to noisy labels (too big/small bboxes etc.) or just actual overfit to the train set.
\n\nIt's too late for this competition now but for future references, what are your thoughts on improvements for Checkpoint saving metric for such competitions? As, unless we have a good checkpoint selection method reducing losses seem to be a gamble. \nBest Val loss proved to be unreliable (based on public scores) , atleast for my experiments. \nYoloV5 checkpoint saving metric is much better though.
", "rawMarkdown": "Talking about Effdet models, we never faced any issues with ReduceLRonPlateau scheduler with AdamW.\nIt gave us great validation losses which produced as good as 0.74 on 5-fold local CV (OOF score) but those models seemed to had overfit the train set.\nWe noticed that beyond a certain OOF score for our models the public LB score only reduced. Thats why we stopped training after reaching certain Val loss for every one of our models.\n\nI'm unsure at this point whether or not this 'overfitting' was due to noisy labels (too big/small bboxes etc.) or just actual overfit to the train set.\n\nIt's too late for this competition now but for future references, what are your thoughts on improvements for Checkpoint saving metric for such competitions? As, unless we have a good checkpoint selection method reducing losses seem to be a gamble. \nBest Val loss proved to be unreliable (based on public scores) , atleast for my experiments. \nYoloV5 checkpoint saving metric is much better though.", "votes": null } ], "comments": [ { "id": 957984, "author_name": "vadimtimakin", "author_url": "", "post_date": "08/04/2020 17:07:22", "content": "Too late to try it in this competition. I shall do it after that.
", "votes": null, "replies": [] }, { "id": 958075, "author_name": "ii5m0k3ii", "author_url": "", "post_date": "08/04/2020 18:45:09", "content": "Talking about Effdet models, we never faced any issues with ReduceLRonPlateau scheduler with AdamW.\nIt gave us great validation losses which produced as good as 0.74 on 5-fold local CV (OOF score) but those models seemed to had overfit the train set.\nWe noticed that beyond a certain OOF score for our models the public LB score only reduced. Thats why we stopped training after reaching certain Val loss for every one of our models.
\n\nI'm unsure at this point whether or not this 'overfitting' was due to noisy labels (too big/small bboxes etc.) or just actual overfit to the train set.
\n\nIt's too late for this competition now but for future references, what are your thoughts on improvements for Checkpoint saving metric for such competitions? As, unless we have a good checkpoint selection method reducing losses seem to be a gamble. \nBest Val loss proved to be unreliable (based on public scores) , atleast for my experiments. \nYoloV5 checkpoint saving metric is much better though.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "957656": "A reflection if one just want to use Pytorch built-in features. Combination of AdamW and torch.optim.lr_scheduler.ExponentialLR with gamma=0.75 seems to have quicker learning curve vs other schedulers I tried. Not significantly better as regards for accuracy, but faster. It reduces the LR for every epoch and with decays of LR * .75, so quite steep, but it seems to work well. Gamma=0.85 can also be a good option and works with other optimizers as well, and when the val-loss is losing momentum then switch to gamma=0.75.", "957984": "Too late to try it in this competition. I shall do it after that.", "958075": "Talking about Effdet models, we never faced any issues with ReduceLRonPlateau scheduler with AdamW.\nIt gave us great validation losses which produced as good as 0.74 on 5-fold local CV (OOF score) but those models seemed to had overfit the train set.\nWe noticed that beyond a certain OOF score for our models the public LB score only reduced. Thats why we stopped training after reaching certain Val loss for every one of our models.\n\nI'm unsure at this point whether or not this 'overfitting' was due to noisy labels (too big/small bboxes etc.) or just actual overfit to the train set.\n\nIt's too late for this competition now but for future references, what are your thoughts on improvements for Checkpoint saving metric for such competitions? As, unless we have a good checkpoint selection method reducing losses seem to be a gamble. \nBest Val loss proved to be unreliable (based on public scores) , atleast for my experiments. \nYoloV5 checkpoint saving metric is much better though." }, "source": "meta" }