{ "id": 169692, "title": "What lr should choose?", "url": "/competitions/global-wheat-detection/discussion/169692", "author_name": "Manh Lab", "post_date": "2020-07-24T18:36:14.881000", "votes": 1, "comment_count": 9, "views": 0, "content": "

I actuly use lr=0.005 ----> got max loss: 0.422 after 30epochs\nlr = 0.00015 ---> max loss: 0.45 after 40 epochs\nseem like not coverge

", "messages": [ { "id": 944435, "postDate": "2020-07-25T05:46:16.310Z", "content": "

0.422 could be close to converging. I generally try to set the learning rate based on the loss curves and not the absolute numeric value of the loss. For some problems/datasets/loss functions the loss won't be close to 0 even when the network has converged.

\n\n

\"\"\nimage credit

", "rawMarkdown": "0.422 could be close to converging. I generally try to set the learning rate based on the loss curves and not the absolute numeric value of the loss. For some problems/datasets/loss functions the loss won't be close to 0 even when the network has converged.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1202828%2F0d4a0ba9dafd31b55975e0b0ff8ab44b%2Flr_graph.png?generation=1595655633857057&alt=media)\n[image credit](https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10)", "votes": 1, "replies": [ { "id": 944631, "postDate": "2020-07-25T08:12:10.103Z", "content": "

I continue train 40 epochs and got 0.4100

", "rawMarkdown": "I continue train 40 epochs and got 0.4100" } ] }, { "id": 944211, "postDate": "2020-07-24T23:38:53.503Z", "content": "

Assuming you're using the FasterRCNN model you can start with 0.005 LR with SGD (like in the paper) and use some kind of LR scheduler to reduce the learning rate to get better convergence.

", "rawMarkdown": "Assuming you're using the FasterRCNN model you can start with 0.005 LR with SGD (like in the paper) and use some kind of LR scheduler to reduce the learning rate to get better convergence.", "votes": 1 }, { "id": 944019, "postDate": "2020-07-24T18:36:14.883Z", "content": "

I actuly use lr=0.005 ----> got max loss: 0.422 after 30epochs\nlr = 0.00015 ---> max loss: 0.45 after 40 epochs\nseem like not coverge

", "rawMarkdown": "I actuly use lr=0.005 ----> got max loss: 0.422 after 30epochs\nlr = 0.00015 ---> max loss: 0.45 after 40 epochs\nseem like not coverge", "votes": 1 }, { "id": 945636, "postDate": "2020-07-26T03:36:00.863Z", "content": "

you should try schedule learning rate like this.\n\"\"

", "rawMarkdown": "you should try schedule learning rate like this.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4196226%2Fea353651599b30932f113c3548fffb55%2Ftr.png?generation=1595734485641363&alt=media)\n", "replies": [ { "id": 945681, "postDate": "2020-07-26T04:42:08.553Z", "content": "

which scheduler you use

", "rawMarkdown": "which scheduler you use" }, { "id": 945813, "postDate": "2020-07-26T06:34:14.020Z", "content": "

A CUSTOM LEARNING SCHEUDLE

", "rawMarkdown": "A CUSTOM LEARNING SCHEUDLE" } ] }, { "id": 945630, "postDate": "2020-07-26T03:30:08.517Z", "content": "

I am assuming you are using Effdet-5 I tried 0.002 and got to val_loss to 0.4 by epoch-20

", "rawMarkdown": "I am assuming you are using Effdet-5 I tried 0.002 and got to val_loss to 0.4 by epoch-20\n" }, { "id": 944031, "postDate": "2020-07-24T18:55:38.797Z", "content": "

Have you tried higher values instead? have you tried with a step down policy? :)

", "rawMarkdown": "Have you tried higher values instead? have you tried with a step down policy? :)" }, { "id": 944030, "postDate": "2020-07-24T18:54:21.340Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 944435, "author_name": "Ryan Epp", "author_url": "", "post_date": "2020-07-25T05:46:16.310000", "content": "

0.422 could be close to converging. I generally try to set the learning rate based on the loss curves and not the absolute numeric value of the loss. For some problems/datasets/loss functions the loss won't be close to 0 even when the network has converged.

\n\n

\"\"\nimage credit

", "votes": 1, "replies": [ { "id": 944631, "author_name": "Manh Lab", "author_url": "", "post_date": "2020-07-25T08:12:10.103000", "content": "

I continue train 40 epochs and got 0.4100

", "votes": 0, "replies": [] } ] }, { "id": 944211, "author_name": "johnny", "author_url": "", "post_date": "2020-07-24T23:38:53.503000", "content": "

Assuming you're using the FasterRCNN model you can start with 0.005 LR with SGD (like in the paper) and use some kind of LR scheduler to reduce the learning rate to get better convergence.

", "votes": 1, "replies": [] }, { "id": 945636, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "2020-07-26T03:36:00.863000", "content": "

you should try schedule learning rate like this.\n\"\"

", "votes": 0, "replies": [ { "id": 945681, "author_name": "Manh Lab", "author_url": "", "post_date": "2020-07-26T04:42:08.553000", "content": "

which scheduler you use

", "votes": 0, "replies": [] }, { "id": 945813, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "2020-07-26T06:34:14.020000", "content": "

A CUSTOM LEARNING SCHEUDLE

", "votes": 0, "replies": [] } ] }, { "id": 945630, "author_name": "Nitin Datta", "author_url": "", "post_date": "2020-07-26T03:30:08.517000", "content": "

I am assuming you are using Effdet-5 I tried 0.002 and got to val_loss to 0.4 by epoch-20

", "votes": 0, "replies": [] }, { "id": 944031, "author_name": "Alberto Maria Falletta", "author_url": "", "post_date": "2020-07-24T18:55:38.797000", "content": "

Have you tried higher values instead? have you tried with a step down policy? :)

", "votes": 0, "replies": [] }, { "id": 944030, "author_name": "", "author_url": "", "post_date": "2020-07-24T18:54:21.340000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "944435": "0.422 could be close to converging. I generally try to set the learning rate based on the loss curves and not the absolute numeric value of the loss. For some problems/datasets/loss functions the loss won't be close to 0 even when the network has converged.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1202828%2F0d4a0ba9dafd31b55975e0b0ff8ab44b%2Flr_graph.png?generation=1595655633857057&alt=media)\n[image credit](https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10)", "944211": "Assuming you're using the FasterRCNN model you can start with 0.005 LR with SGD (like in the paper) and use some kind of LR scheduler to reduce the learning rate to get better convergence.", "944019": "I actuly use lr=0.005 ----> got max loss: 0.422 after 30epochs\nlr = 0.00015 ---> max loss: 0.45 after 40 epochs\nseem like not coverge", "945636": "you should try schedule learning rate like this.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4196226%2Fea353651599b30932f113c3548fffb55%2Ftr.png?generation=1595734485641363&alt=media)\n", "945630": "I am assuming you are using Effdet-5 I tried 0.002 and got to val_loss to 0.4 by epoch-20\n", "944031": "Have you tried higher values instead? have you tried with a step down policy? :)", "944030": "" } }