{
  "id": 442249,
  "title": "smaller n-estimator is better?",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/442249",
  "author_name": "",
  "post_date": "2023-09-21T19:23:23.547997Z",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi</p>\n<p>I am using a LGBM model and stratified 3 fold, with early stopping on. But recently I found out that my CV increases a lot when I set a smaller n-estimator(way before the n-estimator that early stop stops at), which is completely not what I have expected. I have not yet submitted to see the LB score, but I am wondering why this happens. Is it because the stratified is not a good cv for this competition(and I should try group k fold) or any other reason.</p>\n<p>Thanks</p>\n<p><em>EDIT: After submitting, the LB is still higher for the one trained with early stopping on, so it is definitely the problem with my CV strategy.</em></p>",
  "messages": [
    {
      "id": "2450370",
      "postDate": "09/21/2023 19:23:23",
      "content": "<p>Hi</p>\n<p>I am using a LGBM model and stratified 3 fold, with early stopping on. But recently I found out that my CV increases a lot when I set a smaller n-estimator(way before the n-estimator that early stop stops at), which is completely not what I have expected. I have not yet submitted to see the LB score, but I am wondering why this happens. Is it because the stratified is not a good cv for this competition(and I should try group k fold) or any other reason.</p>\n<p>Thanks</p>\n<p><em>EDIT: After submitting, the LB is still higher for the one trained with early stopping on, so it is definitely the problem with my CV strategy.</em></p>",
      "rawMarkdown": "Hi\n\nI am using a LGBM model and stratified 3 fold, with early stopping on. But recently I found out that my CV increases a lot when I set a smaller n-estimator(way before the n-estimator that early stop stops at), which is completely not what I have expected. I have not yet submitted to see the LB score, but I am wondering why this happens. Is it because the stratified is not a good cv for this competition(and I should try group k fold) or any other reason.\n\nThanks\n\n*EDIT: After submitting, the LB is still higher for the one trained with early stopping on, so it is definitely the problem with my CV strategy.*",
      "votes": null
    },
    {
      "id": "2450388",
      "postDate": "09/21/2023 19:46:05",
      "content": "<p>One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.</p>\n<p>Anyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.</p>",
      "rawMarkdown": "One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.\n\nAnyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.",
      "votes": null
    },
    {
      "id": "2450402",
      "postDate": "09/21/2023 20:10:35",
      "content": "<blockquote>\n  <p>One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.</p>\n</blockquote>\n<p>I've not tried Time Series Split CV before, and it sounds interesting. I will definitely give it a shot, as I am really doubting whether stratified fold is a good cv for this competition.</p>\n<blockquote>\n  <p>Anyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.</p>\n</blockquote>\n<p>Oh, I've tuned my lr a lot, so the best number of estimators might be different from yours, I will do more test to find the optimal one. But thanks a lot for sharing!</p>",
      "rawMarkdown": "> One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.\n\nI've not tried Time Series Split CV before, and it sounds interesting. I will definitely give it a shot, as I am really doubting whether stratified fold is a good cv for this competition.\n\n> Anyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.\n\nOh, I've tuned my lr a lot, so the best number of estimators might be different from yours, I will do more test to find the optimal one. But thanks a lot for sharing!",
      "votes": null
    },
    {
      "id": "2450505",
      "postDate": "09/21/2023 23:33:58",
      "content": "<p>Sure thing. For each series, you can take the last 24h as the validation set or something like that.</p>\n<p>I haven't tuned my LR so yes I think it's a great idea to tune it.</p>",
      "rawMarkdown": "Sure thing. For each series, you can take the last 24h as the validation set or something like that.\n\nI haven't tuned my LR so yes I think it's a great idea to tune it.",
      "votes": null
    },
    {
      "id": "2450644",
      "postDate": "09/22/2023 03:26:26",
      "content": "<p>I think a better CV strategy would be groupkfold by series-id. </p>",
      "rawMarkdown": "I think a better CV strategy would be groupkfold by series-id.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2450388,
      "author_name": "rimbax",
      "author_url": "",
      "post_date": "09/21/2023 19:46:05",
      "content": "<p>One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.</p>\n<p>Anyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2450402,
          "author_name": "yuchen2066",
          "author_url": "",
          "post_date": "09/21/2023 20:10:35",
          "content": "<blockquote>\n  <p>One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.</p>\n</blockquote>\n<p>I've not tried Time Series Split CV before, and it sounds interesting. I will definitely give it a shot, as I am really doubting whether stratified fold is a good cv for this competition.</p>\n<blockquote>\n  <p>Anyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.</p>\n</blockquote>\n<p>Oh, I've tuned my lr a lot, so the best number of estimators might be different from yours, I will do more test to find the optimal one. But thanks a lot for sharing!</p>",
          "votes": null,
          "replies": [
            {
              "id": 2450505,
              "author_name": "rimbax",
              "author_url": "",
              "post_date": "09/21/2023 23:33:58",
              "content": "<p>Sure thing. For each series, you can take the last 24h as the validation set or something like that.</p>\n<p>I haven't tuned my LR so yes I think it's a great idea to tune it.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2450644,
      "author_name": "awqatak",
      "author_url": "",
      "post_date": "09/22/2023 03:26:26",
      "content": "<p>I think a better CV strategy would be groupkfold by series-id. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2450370": "Hi\n\nI am using a LGBM model and stratified 3 fold, with early stopping on. But recently I found out that my CV increases a lot when I set a smaller n-estimator(way before the n-estimator that early stop stops at), which is completely not what I have expected. I have not yet submitted to see the LB score, but I am wondering why this happens. Is it because the stratified is not a good cv for this competition(and I should try group k fold) or any other reason.\n\nThanks\n\n*EDIT: After submitting, the LB is still higher for the one trained with early stopping on, so it is definitely the problem with my CV strategy.*",
    "2450388": "One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.\n\nAnyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.",
    "2450402": "> One thing, have you tried to use Time Series Split CV, since essentially the data is time series itself? I am kinda convinced that it represents the leaderboard score better, despite I haven't try to use it though.\n\nI've not tried Time Series Split CV before, and it sounds interesting. I will definitely give it a shot, as I am really doubting whether stratified fold is a good cv for this competition.\n\n> Anyway, 100 estimators work best for my case. I have tried to use 50 and 200 but the LB score is less.\n\nOh, I've tuned my lr a lot, so the best number of estimators might be different from yours, I will do more test to find the optimal one. But thanks a lot for sharing!",
    "2450505": "Sure thing. For each series, you can take the last 24h as the validation set or something like that.\n\nI haven't tuned my LR so yes I think it's a great idea to tune it.",
    "2450644": "I think a better CV strategy would be groupkfold by series-id."
  },
  "source": "meta"
}