{
  "id": 95179,
  "title": "Goldilocks zone for learning rate",
  "url": "/competitions/champs-scalar-coupling/discussion/95179",
  "author_name": "",
  "post_date": "2019-06-10T14:57:15.512212600Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>While experimenting with hyper parameters I came across a very efficient way of setting learning rate. Value of learning rate could be crucial from time it takes to converge point of view. </p>\n\n<p>To know more about goldilocks read this paper <a href=\"https://arxiv.org/pdf/1807.02581.pdf\">https://arxiv.org/pdf/1807.02581.pdf</a></p>\n\n<p><img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/c24580d4d1e1a3c04f6a2f3593a2198a1764b922/1-Figure1-1.png\" alt=\"\"></p>\n\n<p><img src=\"https://www.jeremyjordan.me/content/images/2018/02/lr_finder.png\" alt=\"\"></p>\n\n<p>EDIT 1: It is written for NNs but ideas can be drawn for GBDT. I will share the range of values once I have tested all possible scenarios.</p>",
  "messages": [
    {
      "id": "549329",
      "postDate": "06/10/2019 14:57:15",
      "content": "<p>While experimenting with hyper parameters I came across a very efficient way of setting learning rate. Value of learning rate could be crucial from time it takes to converge point of view. </p>\n\n<p>To know more about goldilocks read this paper <a href=\"https://arxiv.org/pdf/1807.02581.pdf\">https://arxiv.org/pdf/1807.02581.pdf</a></p>\n\n<p><img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/c24580d4d1e1a3c04f6a2f3593a2198a1764b922/1-Figure1-1.png\" alt=\"\"></p>\n\n<p><img src=\"https://www.jeremyjordan.me/content/images/2018/02/lr_finder.png\" alt=\"\"></p>\n\n<p>EDIT 1: It is written for NNs but ideas can be drawn for GBDT. I will share the range of values once I have tested all possible scenarios.</p>",
      "rawMarkdown": "While experimenting with hyper parameters I came across a very efficient way of setting learning rate. Value of learning rate could be crucial from time it takes to converge point of view. \n\nTo know more about goldilocks read this paper https://arxiv.org/pdf/1807.02581.pdf\n\n![](https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/c24580d4d1e1a3c04f6a2f3593a2198a1764b922/1-Figure1-1.png)\n\n\n![](https://www.jeremyjordan.me/content/images/2018/02/lr_finder.png)\n\nEDIT 1: It is written for NNs but ideas can be drawn for GBDT. I will share the range of values once I have tested all possible scenarios.",
      "votes": null
    },
    {
      "id": "549473",
      "postDate": "06/10/2019 17:32:58",
      "content": "<p>Question: isn’t that the goal of choosing learning rate to minimize the lost function? It looks like the r ~ 0.01 reaches the minimum, and can we fix the learning rate here and use momentum to further improve the model?</p>",
      "rawMarkdown": "Question: isn’t that the goal of choosing learning rate to minimize the lost function? It looks like the r ~ 0.01 reaches the minimum, and can we fix the learning rate here and use momentum to further improve the model?",
      "votes": null
    },
    {
      "id": "551769",
      "postDate": "06/13/2019 05:16:49",
      "content": "<p>As far as I know, this technique was developed by Dr. Leslie Smith in 2015 (<a href=\"https://arxiv.org/pdf/1506.01186.pdf\">https://arxiv.org/pdf/1506.01186.pdf</a>). It is also already part of the fastai library. Also, a quick glance of the linked paper doesn't seem to show any information regarding this learning rate test. Which section is that information?</p>",
      "rawMarkdown": "As far as I know, this technique was developed by Dr. Leslie Smith in 2015 (https://arxiv.org/pdf/1506.01186.pdf). It is also already part of the fastai library. Also, a quick glance of the linked paper doesn't seem to show any information regarding this learning rate test. Which section is that information?",
      "votes": null
    },
    {
      "id": "803200",
      "postDate": "04/10/2020 08:32:10",
      "content": "<p>how can i apply goldilock's rule to decide learning rate of my model. any tool?</p>",
      "rawMarkdown": "how can i apply goldilock's rule to decide learning rate of my model. any tool?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 549473,
      "author_name": "b0wang",
      "author_url": "",
      "post_date": "06/10/2019 17:32:58",
      "content": "<p>Question: isn’t that the goal of choosing learning rate to minimize the lost function? It looks like the r ~ 0.01 reaches the minimum, and can we fix the learning rate here and use momentum to further improve the model?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 551769,
      "author_name": "tanlikesmath",
      "author_url": "",
      "post_date": "06/13/2019 05:16:49",
      "content": "<p>As far as I know, this technique was developed by Dr. Leslie Smith in 2015 (<a href=\"https://arxiv.org/pdf/1506.01186.pdf\">https://arxiv.org/pdf/1506.01186.pdf</a>). It is also already part of the fastai library. Also, a quick glance of the linked paper doesn't seem to show any information regarding this learning rate test. Which section is that information?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 803200,
      "author_name": "utagesonali",
      "author_url": "",
      "post_date": "04/10/2020 08:32:10",
      "content": "<p>how can i apply goldilock's rule to decide learning rate of my model. any tool?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "549329": "While experimenting with hyper parameters I came across a very efficient way of setting learning rate. Value of learning rate could be crucial from time it takes to converge point of view. \n\nTo know more about goldilocks read this paper https://arxiv.org/pdf/1807.02581.pdf\n\n![](https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/c24580d4d1e1a3c04f6a2f3593a2198a1764b922/1-Figure1-1.png)\n\n\n![](https://www.jeremyjordan.me/content/images/2018/02/lr_finder.png)\n\nEDIT 1: It is written for NNs but ideas can be drawn for GBDT. I will share the range of values once I have tested all possible scenarios.",
    "549473": "Question: isn’t that the goal of choosing learning rate to minimize the lost function? It looks like the r ~ 0.01 reaches the minimum, and can we fix the learning rate here and use momentum to further improve the model?",
    "551769": "As far as I know, this technique was developed by Dr. Leslie Smith in 2015 (https://arxiv.org/pdf/1506.01186.pdf). It is also already part of the fastai library. Also, a quick glance of the linked paper doesn't seem to show any information regarding this learning rate test. Which section is that information?",
    "803200": "how can i apply goldilock's rule to decide learning rate of my model. any tool?"
  },
  "source": "meta"
}