{
  "id": 152569,
  "title": "How To Prepare A Stratified Split",
  "url": "/competitions/global-wheat-detection/discussion/152569",
  "author_name": "",
  "post_date": "2020-05-20T13:20:40.095088400Z",
  "votes": 6,
  "comment_count": 2,
  "views": 0,
  "content": "<p>For anyone who's relatively new:</p>\n\n<p>I've seen many public kernels doing a very simple split of the training data.</p>\n\n<p>But because we are dealing with data from different sources, we might want to make sure that each source is represented evenly in our training and validation data.</p>\n\n<p>So for that, we need a \"<strong>stratified split</strong>\". I've pasted a code snippet <a href=\"https://www.kaggle.com/alexandersoare/how-to-prepare-a-stratified-split\">here</a></p>",
  "messages": [
    {
      "id": "854950",
      "postDate": "05/20/2020 13:20:40",
      "content": "<p>For anyone who's relatively new:</p>\n\n<p>I've seen many public kernels doing a very simple split of the training data.</p>\n\n<p>But because we are dealing with data from different sources, we might want to make sure that each source is represented evenly in our training and validation data.</p>\n\n<p>So for that, we need a \"<strong>stratified split</strong>\". I've pasted a code snippet <a href=\"https://www.kaggle.com/alexandersoare/how-to-prepare-a-stratified-split\">here</a></p>",
      "rawMarkdown": "For anyone who's relatively new:\n\nI've seen many public kernels doing a very simple split of the training data.\n\nBut because we are dealing with data from different sources, we might want to make sure that each source is represented evenly in our training and validation data.\n\nSo for that, we need a \"**stratified split**\". I've pasted a code snippet [here](https://www.kaggle.com/alexandersoare/how-to-prepare-a-stratified-split)",
      "votes": null
    },
    {
      "id": "855393",
      "postDate": "05/20/2020 22:00:52",
      "content": "<p>What about splitting your data by source such that each fold uses one source for validation and the rest for training? I feel like this is the best approach since the LB is testing our ability to adapt to new data sources. If you do a stratified split, your CV score won't correlate as well with the LB and will be \"artificially\" higher. Your thoughts?</p>",
      "rawMarkdown": "What about splitting your data by source such that each fold uses one source for validation and the rest for training? I feel like this is the best approach since the LB is testing our ability to adapt to new data sources. If you do a stratified split, your CV score won't correlate as well with the LB and will be \"artificially\" higher. Your thoughts?",
      "votes": null
    },
    {
      "id": "855752",
      "postDate": "05/21/2020 07:10:07",
      "content": "<p>Makes sense, although just a couple of caveats IMO:\n1) Then your evaluation would depend a lot on which source you decided to hold out (furthermore the source dist is uneven), and that won't necessarily correlate with the results you would get under a test set.\n2) In the final evaluation it's wisest to use all the sources in training anyway. With a stratified split you get to validate during training, and then evaluate that same model on LB without needing to retrain with 100% of the data.</p>\n\n<p>In saying that, I believe I have read papers where the authors opted to do their CV in the way you suggested. Although I'm not sure they had any access at all to probing the final test distribution, so they had to. It's the fact that we can probe the LB 5 times a day that makes my thinking different.</p>",
      "rawMarkdown": "Makes sense, although just a couple of caveats IMO:\n1) Then your evaluation would depend a lot on which source you decided to hold out (furthermore the source dist is uneven), and that won't necessarily correlate with the results you would get under a test set.\n2) In the final evaluation it's wisest to use all the sources in training anyway. With a stratified split you get to validate during training, and then evaluate that same model on LB without needing to retrain with 100% of the data.\n\nIn saying that, I believe I have read papers where the authors opted to do their CV in the way you suggested. Although I'm not sure they had any access at all to probing the final test distribution, so they had to. It's the fact that we can probe the LB 5 times a day that makes my thinking different.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 855393,
      "author_name": "matthewmasters",
      "author_url": "",
      "post_date": "05/20/2020 22:00:52",
      "content": "<p>What about splitting your data by source such that each fold uses one source for validation and the rest for training? I feel like this is the best approach since the LB is testing our ability to adapt to new data sources. If you do a stratified split, your CV score won't correlate as well with the LB and will be \"artificially\" higher. Your thoughts?</p>",
      "votes": null,
      "replies": [
        {
          "id": 855752,
          "author_name": "alexandersoare",
          "author_url": "",
          "post_date": "05/21/2020 07:10:07",
          "content": "<p>Makes sense, although just a couple of caveats IMO:\n1) Then your evaluation would depend a lot on which source you decided to hold out (furthermore the source dist is uneven), and that won't necessarily correlate with the results you would get under a test set.\n2) In the final evaluation it's wisest to use all the sources in training anyway. With a stratified split you get to validate during training, and then evaluate that same model on LB without needing to retrain with 100% of the data.</p>\n\n<p>In saying that, I believe I have read papers where the authors opted to do their CV in the way you suggested. Although I'm not sure they had any access at all to probing the final test distribution, so they had to. It's the fact that we can probe the LB 5 times a day that makes my thinking different.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "854950": "For anyone who's relatively new:\n\nI've seen many public kernels doing a very simple split of the training data.\n\nBut because we are dealing with data from different sources, we might want to make sure that each source is represented evenly in our training and validation data.\n\nSo for that, we need a \"**stratified split**\". I've pasted a code snippet [here](https://www.kaggle.com/alexandersoare/how-to-prepare-a-stratified-split)",
    "855393": "What about splitting your data by source such that each fold uses one source for validation and the rest for training? I feel like this is the best approach since the LB is testing our ability to adapt to new data sources. If you do a stratified split, your CV score won't correlate as well with the LB and will be \"artificially\" higher. Your thoughts?",
    "855752": "Makes sense, although just a couple of caveats IMO:\n1) Then your evaluation would depend a lot on which source you decided to hold out (furthermore the source dist is uneven), and that won't necessarily correlate with the results you would get under a test set.\n2) In the final evaluation it's wisest to use all the sources in training anyway. With a stratified split you get to validate during training, and then evaluate that same model on LB without needing to retrain with 100% of the data.\n\nIn saying that, I believe I have read papers where the authors opted to do their CV in the way you suggested. Although I'm not sure they had any access at all to probing the final test distribution, so they had to. It's the fact that we can probe the LB 5 times a day that makes my thinking different."
  },
  "source": "meta"
}