{
  "id": 12567,
  "title": "How to choose validation set?",
  "url": "/competitions/datasciencebowl/discussion/12567",
  "author_name": "",
  "post_date": "2015-02-19T10:51:41.923Z",
  "votes": null,
  "comment_count": 3,
  "views": 1170,
  "content": "<p>Up until now, I'm basically setting aside a random 25% of training examples (same random pick each time) to function as a validation set. Since the classes are extremely unbalanced, I'm wondering whether this is the optimal strategy. Perhaps choosing a validation set such that the distribution is equal to the full training set (especially at least 1 example of each of the extremely infrequent classes?) would be wiser?&nbsp;Does anyone have an insight on this issue?</p>",
  "messages": [
    {
      "id": "64568",
      "postDate": "02/19/2015 10:51:41",
      "content": "<p>Up until now, I'm basically setting aside a random 25% of training examples (same random pick each time) to function as a validation set. Since the classes are extremely unbalanced, I'm wondering whether this is the optimal strategy. Perhaps choosing a validation set such that the distribution is equal to the full training set (especially at least 1 example of each of the extremely infrequent classes?) would be wiser?&nbsp;Does anyone have an insight on this issue?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64591",
      "postDate": "02/19/2015 19:26:48",
      "content": "<p>I found a stratified sample a lot more reliable. sklearn has an implementation of it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64620",
      "postDate": "02/20/2015 03:24:26",
      "content": "<p>I believe the tutorial uses stratified KFold. The implementation using sklearn is very easy in python.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64638",
      "postDate": "02/20/2015 11:10:09",
      "content": "<p>Thanks for the comments! I will definitely switch to a stratified sample.&nbsp;Probably I can then also use a bit more of the&nbsp;labelled examples for training.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 64591,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "02/19/2015 19:26:48",
      "content": "<p>I found a stratified sample a lot more reliable. sklearn has an implementation of it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64620,
      "author_name": "thekannman",
      "author_url": "",
      "post_date": "02/20/2015 03:24:26",
      "content": "<p>I believe the tutorial uses stratified KFold. The implementation using sklearn is very easy in python.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64638,
      "author_name": "eelkespaak",
      "author_url": "",
      "post_date": "02/20/2015 11:10:09",
      "content": "<p>Thanks for the comments! I will definitely switch to a stratified sample.&nbsp;Probably I can then also use a bit more of the&nbsp;labelled examples for training.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "64568": "",
    "64591": "",
    "64620": "",
    "64638": ""
  },
  "source": "meta"
}