{
  "id": 21720,
  "title": "Train/Val split Suggestion",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21720",
  "author_name": "",
  "post_date": "2016-06-16T09:01:35.093Z",
  "votes": null,
  "comment_count": 5,
  "views": 538,
  "content": "<p>I currently work on grid-knn based solution.   I saw some discussion about cv score is lower than LB score and I do get this problem in my validation frame work</p>\n\n<p>My validation framework in simply randomly split train data set by KFold(n_fold=3). I got CV score around 0.545 while 0.57 on LB.  Any suggestion  or idea why does it happened? </p>\n\n<p>Thanks a lot</p>",
  "messages": [
    {
      "id": "124210",
      "postDate": "06/16/2016 09:01:35",
      "content": "<p>I currently work on grid-knn based solution.   I saw some discussion about cv score is lower than LB score and I do get this problem in my validation frame work</p>\n\n<p>My validation framework in simply randomly split train data set by KFold(n_fold=3). I got CV score around 0.545 while 0.57 on LB.  Any suggestion  or idea why does it happened? </p>\n\n<p>Thanks a lot</p>",
      "rawMarkdown": "I currently work on grid-knn based solution.   I saw some discussion about cv score is lower than LB score and I do get this problem in my validation frame work\r\n\r\nMy validation framework in simply randomly split train data set by KFold(n_fold=3). I got CV score around 0.545 while 0.57 on LB.  Any suggestion  or idea why does it happened? \r\n\r\nThanks a lot",
      "votes": null
    },
    {
      "id": "124219",
      "postDate": "06/16/2016 10:08:35",
      "content": "<p>Hi,</p>\n\n<p>Maybe your fold number is to low so the partitions are to big and you are leaving to many examples out of the training sets. Anyway, as it has been said in other forum posts before, I would recommend you splitting your data by time and not randomly. It is important to replicate the problem characteristics in your local tests as much as possible.</p>",
      "rawMarkdown": "Hi,\r\n\r\nMaybe your fold number is to low so the partitions are to big and you are leaving to many examples out of the training sets. Anyway, as it has been said in other forum posts before, I would recommend you splitting your data by time and not randomly. It is important to replicate the problem characteristics in your local tests as much as possible.",
      "votes": null
    },
    {
      "id": "124253",
      "postDate": "06/16/2016 14:10:26",
      "content": "<p>So maybe use rows with time from 0 t o100000 for train and 100001 to 150000 for validation, could that work? (just an example for the numbers for the time/time-range).\nOr do you mean another way for splitting the data by time? only take data from one specific minute, or sth. like that .. (for example: time%1440 == 1) ?</p>",
      "rawMarkdown": "So maybe use rows with time from 0 t o100000 for train and 100001 to 150000 for validation, could that work? (just an example for the numbers for the time/time-range).\r\nOr do you mean another way for splitting the data by time? only take data from one specific minute, or sth. like that .. (for example: time%1440 == 1) ?",
      "votes": null
    },
    {
      "id": "124259",
      "postDate": "06/16/2016 14:48:24",
      "content": "<p>@shdgrao</p>\n\n<p>Thank for your suggestion. I will try it:)</p>\n\n<p>@AndreSe</p>\n\n<p>oh. I got it!!.  No I just randomly take sample from data, not based on specific time.  Thank for your advice!</p>",
      "rawMarkdown": "shdgrao\r\n\r\nThank for your suggestion. I will try it:)\r\n\r\n@AndreSe\r\n\r\noh. I got it!!.  No I just randomly take sample from data, not based on specific time.  Thank for your advice!",
      "votes": null
    },
    {
      "id": "124268",
      "postDate": "06/16/2016 16:04:22",
      "content": "<p>Even if you split by time, you will still get lower validation score than your LB score. It's because some place_id's only appear in the validation set after you split the training set by time. Then there is no way you would predict those correctly.</p>\n\n<p>In the generation of the test data, they first split data set by time, then removed those place_id's that are not in the training set.</p>",
      "rawMarkdown": "Even if you split by time, you will still get lower validation score than your LB score. It's because some place_id's only appear in the validation set after you split the training set by time. Then there is no way you would predict those correctly.\r\n\r\nIn the generation of the test data, they first split data set by time, then removed those place_id's that are not in the training set.",
      "votes": null
    },
    {
      "id": "124339",
      "postDate": "06/17/2016 09:36:46",
      "content": "<p>@Leon</p>\n\n<p>Thank for your suggestion. I finally get it right!</p>",
      "rawMarkdown": "Leon\r\n\r\nThank for your suggestion. I finally get it right!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 124219,
      "author_name": "shdgrao",
      "author_url": "",
      "post_date": "06/16/2016 10:08:35",
      "content": "<p>Hi,</p>\n\n<p>Maybe your fold number is to low so the partitions are to big and you are leaving to many examples out of the training sets. Anyway, as it has been said in other forum posts before, I would recommend you splitting your data by time and not randomly. It is important to replicate the problem characteristics in your local tests as much as possible.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124253,
      "author_name": "antsek",
      "author_url": "",
      "post_date": "06/16/2016 14:10:26",
      "content": "<p>So maybe use rows with time from 0 t o100000 for train and 100001 to 150000 for validation, could that work? (just an example for the numbers for the time/time-range).\nOr do you mean another way for splitting the data by time? only take data from one specific minute, or sth. like that .. (for example: time%1440 == 1) ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124259,
      "author_name": "kuanchen",
      "author_url": "",
      "post_date": "06/16/2016 14:48:24",
      "content": "<p>@shdgrao</p>\n\n<p>Thank for your suggestion. I will try it:)</p>\n\n<p>@AndreSe</p>\n\n<p>oh. I got it!!.  No I just randomly take sample from data, not based on specific time.  Thank for your advice!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124268,
      "author_name": "leonlu",
      "author_url": "",
      "post_date": "06/16/2016 16:04:22",
      "content": "<p>Even if you split by time, you will still get lower validation score than your LB score. It's because some place_id's only appear in the validation set after you split the training set by time. Then there is no way you would predict those correctly.</p>\n\n<p>In the generation of the test data, they first split data set by time, then removed those place_id's that are not in the training set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124339,
      "author_name": "kuanchen",
      "author_url": "",
      "post_date": "06/17/2016 09:36:46",
      "content": "<p>@Leon</p>\n\n<p>Thank for your suggestion. I finally get it right!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "124210": "I currently work on grid-knn based solution.   I saw some discussion about cv score is lower than LB score and I do get this problem in my validation frame work\r\n\r\nMy validation framework in simply randomly split train data set by KFold(n_fold=3). I got CV score around 0.545 while 0.57 on LB.  Any suggestion  or idea why does it happened? \r\n\r\nThanks a lot",
    "124219": "Hi,\r\n\r\nMaybe your fold number is to low so the partitions are to big and you are leaving to many examples out of the training sets. Anyway, as it has been said in other forum posts before, I would recommend you splitting your data by time and not randomly. It is important to replicate the problem characteristics in your local tests as much as possible.",
    "124253": "So maybe use rows with time from 0 t o100000 for train and 100001 to 150000 for validation, could that work? (just an example for the numbers for the time/time-range).\r\nOr do you mean another way for splitting the data by time? only take data from one specific minute, or sth. like that .. (for example: time%1440 == 1) ?",
    "124259": "shdgrao\r\n\r\nThank for your suggestion. I will try it:)\r\n\r\n@AndreSe\r\n\r\noh. I got it!!.  No I just randomly take sample from data, not based on specific time.  Thank for your advice!",
    "124268": "Even if you split by time, you will still get lower validation score than your LB score. It's because some place_id's only appear in the validation set after you split the training set by time. Then there is no way you would predict those correctly.\r\n\r\nIn the generation of the test data, they first split data set by time, then removed those place_id's that are not in the training set.",
    "124339": "Leon\r\n\r\nThank for your suggestion. I finally get it right!"
  },
  "source": "meta"
}