{
  "id": 442552,
  "title": "CV vs LB difference",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/442552",
  "author_name": "",
  "post_date": "2023-09-23T07:02:28.665934900Z",
  "votes": 10,
  "comment_count": 11,
  "views": 0,
  "content": "<p>Hello everyone,</p>\n<p>What is your cv, lb score difference? I am observing a difference of ~0.04</p>",
  "messages": [
    {
      "id": "2452221",
      "postDate": "09/23/2023 07:02:28",
      "content": "<p>Hello everyone,</p>\n<p>What is your cv, lb score difference? I am observing a difference of ~0.04</p>",
      "rawMarkdown": "Hello everyone,\n\nWhat is your cv, lb score difference? I am observing a difference of ~0.04",
      "votes": null
    },
    {
      "id": "2452376",
      "postDate": "09/23/2023 08:43:38",
      "content": "<p>HOW DO YOU CHECK YOUR MODEL</p>",
      "rawMarkdown": "HOW DO YOU CHECK YOUR MODEL",
      "votes": null
    },
    {
      "id": "2452629",
      "postDate": "09/23/2023 13:14:19",
      "content": "<p>If you are asking about how to use the competition metric, you can check this <a href=\"https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric\" target=\"_blank\">zzzs-playing-with-the-competition-metric</a> by <a href=\"https://www.kaggle.com/carlmcbrideellis\" target=\"_blank\">@carlmcbrideellis</a> </p>",
      "rawMarkdown": "If you are asking about how to use the competition metric, you can check this [zzzs-playing-with-the-competition-metric](https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric) by @carlmcbrideellis",
      "votes": null
    },
    {
      "id": "2453035",
      "postDate": "09/23/2023 18:41:00",
      "content": "<p>Hi, may I ask how you are validating for your CV (what's your splitting strategy), and did you actually validate on the entire train's prediction (cause it is taking me a long time) or just a portion of train series?</p>",
      "rawMarkdown": "Hi, may I ask how you are validating for your CV (what's your splitting strategy), and did you actually validate on the entire train's prediction (cause it is taking me a long time) or just a portion of train series?",
      "votes": null
    },
    {
      "id": "2453503",
      "postDate": "09/24/2023 05:20:59",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/yuchen2066\" target=\"_blank\">@yuchen2066</a>, I am using StratifiedGroupKFold for cv and combining all the out of fold results and evaluating them.</p>",
      "rawMarkdown": "Hi @yuchen2066, I am using StratifiedGroupKFold for cv and combining all the out of fold results and evaluating them.",
      "votes": null
    },
    {
      "id": "2475970",
      "postDate": "10/10/2023 09:04:25",
      "content": "<p>My public LB is about 0.04~0.06 worse than CV in all subs so far. </p>\n<p>I would like to know the LB-CV gap of others.</p>",
      "rawMarkdown": "My public LB is about 0.04~0.06 worse than CV in all subs so far. \n\nI would like to know the LB-CV gap of others.",
      "votes": null
    },
    {
      "id": "2476751",
      "postDate": "10/10/2023 23:05:35",
      "content": "<p>CV method: 5fold-GroupKFold(by series_id)<br>\nCV score is better than LB score, my score gap between CV and LB is 0.03 ~ 0.05.  </p>",
      "rawMarkdown": "CV method: 5fold-GroupKFold(by series_id)\nCV score is better than LB score, my score gap between CV and LB is 0.03 ~ 0.05.",
      "votes": null
    },
    {
      "id": "2476806",
      "postDate": "10/11/2023 00:37:10",
      "content": "<p>model: LSTM<br>\ncv: 5fold-GroupKFold(by series_id)<br>\nlb is about 0.03 worse than CV.</p>",
      "rawMarkdown": "model: LSTM\ncv: 5fold-GroupKFold(by series_id)\nlb is about 0.03 worse than CV.",
      "votes": null
    },
    {
      "id": "2482945",
      "postDate": "10/15/2023 11:41:34",
      "content": "<p>Which dataset are yall using to do CV ? The Carl McBride's <a href=\"https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target\" target=\"_blank\">lightweight dataset</a> or the entire train set (which is a lot bigger) ?</p>\n<p>If the lightweight dataset is used, in particular which one, Zzzs_train.parquet or Zzzs_train_multi.parquet ?</p>",
      "rawMarkdown": "Which dataset are yall using to do CV ? The Carl McBride's [lightweight dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) or the entire train set (which is a lot bigger) ?\n\nIf the lightweight dataset is used, in particular which one, Zzzs_train.parquet or Zzzs_train_multi.parquet ?",
      "votes": null
    },
    {
      "id": "2483101",
      "postDate": "10/15/2023 13:29:15",
      "content": "<p>I'm using the entire train set.</p>",
      "rawMarkdown": "I'm using the entire train set.",
      "votes": null
    },
    {
      "id": "2485941",
      "postDate": "10/17/2023 14:48:03",
      "content": "<p>NNmodel<br>\n5fold-GroupKFold(by series_id)<br>\nlb is 0.02 ~ 0.03 worse than CV</p>",
      "rawMarkdown": "NNmodel\n5fold-GroupKFold(by series_id)\nlb is 0.02 ~ 0.03 worse than CV",
      "votes": null
    },
    {
      "id": "2486057",
      "postDate": "10/17/2023 16:47:53",
      "content": "<p>lightgbm<br>\n5fold-GroupKFold(by series_id)<br>\n(CV-LB)=(0.026~0.043)</p>",
      "rawMarkdown": "lightgbm\n5fold-GroupKFold(by series_id)\n(CV-LB)=(0.026~0.043)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2452376,
      "author_name": "markdjadchenko",
      "author_url": "",
      "post_date": "09/23/2023 08:43:38",
      "content": "<p>HOW DO YOU CHECK YOUR MODEL</p>",
      "votes": null,
      "replies": [
        {
          "id": 2452629,
          "author_name": "anilreddyvv",
          "author_url": "",
          "post_date": "09/23/2023 13:14:19",
          "content": "<p>If you are asking about how to use the competition metric, you can check this <a href=\"https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric\" target=\"_blank\">zzzs-playing-with-the-competition-metric</a> by <a href=\"https://www.kaggle.com/carlmcbrideellis\" target=\"_blank\">@carlmcbrideellis</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2453035,
      "author_name": "yuchen2066",
      "author_url": "",
      "post_date": "09/23/2023 18:41:00",
      "content": "<p>Hi, may I ask how you are validating for your CV (what's your splitting strategy), and did you actually validate on the entire train's prediction (cause it is taking me a long time) or just a portion of train series?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2453503,
          "author_name": "anilreddyvv",
          "author_url": "",
          "post_date": "09/24/2023 05:20:59",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/yuchen2066\" target=\"_blank\">@yuchen2066</a>, I am using StratifiedGroupKFold for cv and combining all the out of fold results and evaluating them.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2475970,
      "author_name": "ryotayoshinobu",
      "author_url": "",
      "post_date": "10/10/2023 09:04:25",
      "content": "<p>My public LB is about 0.04~0.06 worse than CV in all subs so far. </p>\n<p>I would like to know the LB-CV gap of others.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2476751,
      "author_name": "kuto0633",
      "author_url": "",
      "post_date": "10/10/2023 23:05:35",
      "content": "<p>CV method: 5fold-GroupKFold(by series_id)<br>\nCV score is better than LB score, my score gap between CV and LB is 0.03 ~ 0.05.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2476806,
      "author_name": "tanakar",
      "author_url": "",
      "post_date": "10/11/2023 00:37:10",
      "content": "<p>model: LSTM<br>\ncv: 5fold-GroupKFold(by series_id)<br>\nlb is about 0.03 worse than CV.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2482945,
      "author_name": "yeoyunsianggeremie",
      "author_url": "",
      "post_date": "10/15/2023 11:41:34",
      "content": "<p>Which dataset are yall using to do CV ? The Carl McBride's <a href=\"https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target\" target=\"_blank\">lightweight dataset</a> or the entire train set (which is a lot bigger) ?</p>\n<p>If the lightweight dataset is used, in particular which one, Zzzs_train.parquet or Zzzs_train_multi.parquet ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2483101,
          "author_name": "ryotayoshinobu",
          "author_url": "",
          "post_date": "10/15/2023 13:29:15",
          "content": "<p>I'm using the entire train set.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2485941,
      "author_name": "stgkrtua",
      "author_url": "",
      "post_date": "10/17/2023 14:48:03",
      "content": "<p>NNmodel<br>\n5fold-GroupKFold(by series_id)<br>\nlb is 0.02 ~ 0.03 worse than CV</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2486057,
      "author_name": "w5833946",
      "author_url": "",
      "post_date": "10/17/2023 16:47:53",
      "content": "<p>lightgbm<br>\n5fold-GroupKFold(by series_id)<br>\n(CV-LB)=(0.026~0.043)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2452221": "Hello everyone,\n\nWhat is your cv, lb score difference? I am observing a difference of ~0.04",
    "2452376": "HOW DO YOU CHECK YOUR MODEL",
    "2452629": "If you are asking about how to use the competition metric, you can check this [zzzs-playing-with-the-competition-metric](https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric) by @carlmcbrideellis",
    "2453035": "Hi, may I ask how you are validating for your CV (what's your splitting strategy), and did you actually validate on the entire train's prediction (cause it is taking me a long time) or just a portion of train series?",
    "2453503": "Hi @yuchen2066, I am using StratifiedGroupKFold for cv and combining all the out of fold results and evaluating them.",
    "2475970": "My public LB is about 0.04~0.06 worse than CV in all subs so far. \n\nI would like to know the LB-CV gap of others.",
    "2476751": "CV method: 5fold-GroupKFold(by series_id)\nCV score is better than LB score, my score gap between CV and LB is 0.03 ~ 0.05.",
    "2476806": "model: LSTM\ncv: 5fold-GroupKFold(by series_id)\nlb is about 0.03 worse than CV.",
    "2482945": "Which dataset are yall using to do CV ? The Carl McBride's [lightweight dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) or the entire train set (which is a lot bigger) ?\n\nIf the lightweight dataset is used, in particular which one, Zzzs_train.parquet or Zzzs_train_multi.parquet ?",
    "2483101": "I'm using the entire train set.",
    "2485941": "NNmodel\n5fold-GroupKFold(by series_id)\nlb is 0.02 ~ 0.03 worse than CV",
    "2486057": "lightgbm\n5fold-GroupKFold(by series_id)\n(CV-LB)=(0.026~0.043)"
  },
  "source": "meta"
}