{
  "id": 21547,
  "title": "Let's discuss validation strategies...!",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21547",
  "author_name": "",
  "post_date": "2016-06-09T16:18:51.433Z",
  "votes": 2,
  "comment_count": 22,
  "views": 3522,
  "content": "<p>Hi guys,</p>\n\n<p>as we all know it will be crucial for this challenge to have a <strong>good validation strategy</strong>.\nSo let's start talking about it.</p>\n\n<p>I'm currently splitting the training data in a train and validation set by weeks. I use week 1-7 to train and evaluate  on week 8+9.\nIs there any better idea?</p>\n\n<p>My current model gives me locally a score of 0.65667.\nBut when I submit with this model I get a LB-Score of 1.07689. </p>\n\n<p>So it looks like I'm overfitting (I don't see how) or make some other mistake. \nHas anyone an idea what's happening here?</p>\n\n<p>Best,\nSterby</p>",
  "messages": [
    {
      "id": "123083",
      "postDate": "06/09/2016 16:18:51",
      "content": "<p>Hi guys,</p>\n\n<p>as we all know it will be crucial for this challenge to have a <strong>good validation strategy</strong>.\nSo let's start talking about it.</p>\n\n<p>I'm currently splitting the training data in a train and validation set by weeks. I use week 1-7 to train and evaluate  on week 8+9.\nIs there any better idea?</p>\n\n<p>My current model gives me locally a score of 0.65667.\nBut when I submit with this model I get a LB-Score of 1.07689. </p>\n\n<p>So it looks like I'm overfitting (I don't see how) or make some other mistake. \nHas anyone an idea what's happening here?</p>\n\n<p>Best,\nSterby</p>",
      "rawMarkdown": "Hi guys,\r\n\r\nas we all know it will be crucial for this challenge to have a **good validation strategy**.\r\nSo let's start talking about it.\r\n\r\nI'm currently splitting the training data in a train and validation set by weeks. I use week 1-7 to train and evaluate  on week 8+9.\r\nIs there any better idea?\r\n\r\nMy current model gives me locally a score of 0.65667.\r\nBut when I submit with this model I get a LB-Score of 1.07689. \r\n\r\nSo it looks like I'm overfitting (I don't see how) or make some other mistake. \r\nHas anyone an idea what's happening here?\r\n\r\nBest,\r\nSterby",
      "votes": null
    },
    {
      "id": "123113",
      "postDate": "06/09/2016 19:36:45",
      "content": "<p>Your CV strategy seams sound. Not knowing anything about your implementation or models, it's rather difficult to find a mistake. \nOn a hunch, i would tell you to double check that the model you are using for test predictions is trained on the newest available data. It's OK to train on weeks 1-7 to validate, but you should retrain on either all weeks or on weeks 3-9 to make the test predictions. </p>",
      "rawMarkdown": "Your CV strategy seams sound. Not knowing anything about your implementation or models, it's rather difficult to find a mistake. \r\nOn a hunch, i would tell you to double check that the model you are using for test predictions is trained on the newest available data. It's OK to train on weeks 1-7 to validate, but you should retrain on either all weeks or on weeks 3-9 to make the test predictions.",
      "votes": null
    },
    {
      "id": "123161",
      "postDate": "06/10/2016 02:15:26",
      "content": "<p>Hey Sterby, just curious on this, how did you handle items that did not exist in weeks 1-7? Could that be affecting your score since there can be new products released? I personally have downloaded the data but haven't had time to get started on it yet but that's my hunch on this. Cheers!</p>",
      "rawMarkdown": "Hey Sterby, just curious on this, how did you handle items that did not exist in weeks 1-7? Could that be affecting your score since there can be new products released? I personally have downloaded the data but haven't had time to get started on it yet but that's my hunch on this. Cheers!",
      "votes": null
    },
    {
      "id": "123198",
      "postDate": "06/10/2016 08:58:27",
      "content": "<p>Hi Sterby</p>\n\n<p>I am not able to see week 1, 2 in the training data. May I know where you got that data from?</p>\n\n<p>Ian</p>",
      "rawMarkdown": "Hi Sterby\r\n\r\nI am not able to see week 1, 2 in the training data. May I know where you got that data from?\r\n\r\nIan",
      "votes": null
    },
    {
      "id": "123200",
      "postDate": "06/10/2016 09:12:56",
      "content": "<p>I didn't checked if weeks 1 + 2 exists. Just assumed it ;) I splited &lt;8 / &gt;=8</p>",
      "rawMarkdown": "I didn't checked if weeks 1 + 2 exists. Just assumed it ;) I splited <8 / >=8",
      "votes": null
    },
    {
      "id": "125607",
      "postDate": "06/30/2016 20:21:09",
      "content": "<p>Did you use the test set?</p>",
      "rawMarkdown": "Did you use the test set?",
      "votes": null
    },
    {
      "id": "125660",
      "postDate": "07/01/2016 11:15:10",
      "content": "<p>Same  thing   here, tested a model    that  gives 0.51 on    holdout data sets  and on LB   gives 0.58.\nDid you guys   notice   sumething unusual   at  final distribution.\nBeing  Pareto in the  training  frame could not  hold   because if there  is increseang  behaviour  in  test  semana this  could also  mean those 2 weks  could come from some  local  holidays  or Christmas.\nWith this talled what  if    found   out final distribution is  gaussian?\nIt would  be interesting if other competitors  could    give   feedback on this.</p>",
      "rawMarkdown": "Same  thing   here, tested a model    that  gives 0.51 on    holdout data sets  and on LB   gives 0.58.\r\nDid you guys   notice   sumething unusual   at  final distribution.\r\nBeing  Pareto in the  training  frame could not  hold   because if there  is increseang  behaviour  in  test  semana this  could also  mean those 2 weks  could come from some  local  holidays  or Christmas.\r\nWith this talled what  if    found   out final distribution is  gaussian?\r\nIt would  be interesting if other competitors  could    give   feedback on this.",
      "votes": null
    },
    {
      "id": "126005",
      "postDate": "07/05/2016 13:20:05",
      "content": "<p>Did you re-train your model, including weeks 8+9, AFTER your local validation?</p>",
      "rawMarkdown": "Did you re-train your model, including weeks 8+9, AFTER your local validation?",
      "votes": null
    },
    {
      "id": "126007",
      "postDate": "07/05/2016 13:33:54",
      "content": "<p>I got .32 in local CV but .503 on LB :(</p>",
      "rawMarkdown": "I got .32 in local CV but .503 on LB :(",
      "votes": null
    },
    {
      "id": "126012",
      "postDate": "07/05/2016 14:00:51",
      "content": "<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>",
      "rawMarkdown": "I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...",
      "votes": null
    },
    {
      "id": "126026",
      "postDate": "07/05/2016 16:38:41",
      "content": "<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>",
      "rawMarkdown": "[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?",
      "votes": null
    },
    {
      "id": "126033",
      "postDate": "07/05/2016 17:54:44",
      "content": "<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>",
      "rawMarkdown": "[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)",
      "votes": null
    },
    {
      "id": "126035",
      "postDate": "07/05/2016 17:56:24",
      "content": "<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>",
      "rawMarkdown": "[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?",
      "votes": null
    },
    {
      "id": "126037",
      "postDate": "07/05/2016 18:14:07",
      "content": "<p>[quote=SteveKane;126035]</p>\n\n<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>\n\n<p>[/quote]</p>\n\n<p>I don't use CV, but a holdout set by weeks. As I described.\nI use the university cluster ;)\nRight now I'm using an ensemble of xgboost models and randomforests.</p>\n\n<p>Any suggestions?</p>",
      "rawMarkdown": "[quote=SteveKane;126035]\r\n\r\n[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?\r\n\r\n[/quote]\r\n\r\nI don't use CV, but a holdout set by weeks. As I described.\r\nI use the university cluster ;)\r\nRight now I'm using an ensemble of xgboost models and randomforests.\r\n\r\nAny suggestions?",
      "votes": null
    },
    {
      "id": "126038",
      "postDate": "07/05/2016 18:18:12",
      "content": "<p>[quote=Sterby;126037]</p>\n\n<p>[quote=SteveKane;126035]</p>\n\n<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>\n\n<p>[/quote]</p>\n\n<p>I don't use CV, but a holdout set by weeks. As I described.\nI use the university cluster ;)\nRight now I'm using an ensemble of xgboost models and randomforests.</p>\n\n<p>Any suggestions?</p>\n\n<p>[/quote]</p>\n\n<p>lucky...I'm using my laptop and I let it run when I'm at work or sleeping haha</p>",
      "rawMarkdown": "[quote=Sterby;126037]\r\n\r\n[quote=SteveKane;126035]\r\n\r\n[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?\r\n\r\n[/quote]\r\n\r\nI don't use CV, but a holdout set by weeks. As I described.\r\nI use the university cluster ;)\r\nRight now I'm using an ensemble of xgboost models and randomforests.\r\n\r\nAny suggestions?\r\n\r\n\r\n[/quote]\r\n\r\nlucky...I'm using my laptop and I let it run when I'm at work or sleeping haha",
      "votes": null
    },
    {
      "id": "126040",
      "postDate": "07/05/2016 18:56:49",
      "content": "<p>What is the cluster config? </p>\n\n<p>[quote=Sterby;126037]</p>\n\n<p>[quote=SteveKane;126035]</p>\n\n<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>\n\n<p>[/quote]</p>\n\n<p>I don't use CV, but a holdout set by weeks. As I described.\nI use the university cluster ;)\nRight now I'm using an ensemble of xgboost models and randomforests.</p>\n\n<p>Any suggestions?</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "What is the cluster config? \r\n\r\n[quote=Sterby;126037]\r\n\r\n[quote=SteveKane;126035]\r\n\r\n[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?\r\n\r\n[/quote]\r\n\r\nI don't use CV, but a holdout set by weeks. As I described.\r\nI use the university cluster ;)\r\nRight now I'm using an ensemble of xgboost models and randomforests.\r\n\r\nAny suggestions?\r\n\r\n\r\n[/quote]",
      "votes": null
    },
    {
      "id": "126263",
      "postDate": "07/07/2016 07:20:34",
      "content": "<p>It is key to have the right cv framework.  That's what I learned the hard way in the few Kaggle competitions I entered recently.</p>\n\n<p>For this one, it is possible to have a fairly reliable cv.  With mine I currently have a local cv of 0.457 and a public LB of 0.453.</p>\n\n<p>It took me a while (much too long) to find it, and I suspect current leaders found something similar or better much more rapidly than me.  </p>\n\n<p>Without reliable CV framework you are basically tuning randomly, or, worse, overtunning to the public LB.  Keep looking for a good CV setting if you do not have one.</p>",
      "rawMarkdown": "It is key to have the right cv framework.  That's what I learned the hard way in the few Kaggle competitions I entered recently.\r\n\r\nFor this one, it is possible to have a fairly reliable cv.  With mine I currently have a local cv of 0.457 and a public LB of 0.453.\r\n\r\nIt took me a while (much too long) to find it, and I suspect current leaders found something similar or better much more rapidly than me.  \r\n\r\nWithout reliable CV framework you are basically tuning randomly, or, worse, overtunning to the public LB.  Keep looking for a good CV setting if you do not have one.",
      "votes": null
    },
    {
      "id": "126265",
      "postDate": "07/07/2016 07:37:34",
      "content": "<p>Would you mind sharing your idea how to perform this validation? :)</p>",
      "rawMarkdown": "Would you mind sharing your idea how to perform this validation? :)",
      "votes": null
    },
    {
      "id": "126271",
      "postDate": "07/07/2016 08:16:21",
      "content": "<p>I'll share after the competition ends.   Here is a hint: read Gert's interview after he won a sales forecast competition: <a href=\"http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/\">http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/</a></p>\n\n<p>I tried to apply his advice to the current competition.</p>",
      "rawMarkdown": "I'll share after the competition ends.   Here is a hint: read Gert's interview after he won a sales forecast competition: http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/\r\n\r\nI tried to apply his advice to the current competition.",
      "votes": null
    },
    {
      "id": "126280",
      "postDate": "07/07/2016 10:13:00",
      "content": "<p>Nice, I did that too :)\nSo no more secret sauce ;)</p>",
      "rawMarkdown": "Nice, I did that too :)\r\nSo no more secret sauce ;)",
      "votes": null
    },
    {
      "id": "128372",
      "postDate": "07/19/2016 17:26:55",
      "content": "<p>Is there any evidence, that time plays a crucial roll here?</p>",
      "rawMarkdown": "Is there any evidence, that time plays a crucial roll here?",
      "votes": null
    },
    {
      "id": "128479",
      "postDate": "07/20/2016 12:38:34",
      "content": "<p><img src=\"https://i.imgflip.com/17oh2g.jpg\" alt=\"enter image description here\" title></p>",
      "rawMarkdown": "![enter image description here][1]\r\n\r\n\r\n  [1]: https://i.imgflip.com/17oh2g.jpg",
      "votes": null
    },
    {
      "id": "128564",
      "postDate": "07/21/2016 08:22:28",
      "content": "<p>Aw man, not a single vote on my joke... I'm never making grand master...</p>",
      "rawMarkdown": "Aw man, not a single vote on my joke... I'm never making grand master...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 123113,
      "author_name": "nickel",
      "author_url": "",
      "post_date": "06/09/2016 19:36:45",
      "content": "<p>Your CV strategy seams sound. Not knowing anything about your implementation or models, it's rather difficult to find a mistake. \nOn a hunch, i would tell you to double check that the model you are using for test predictions is trained on the newest available data. It's OK to train on weeks 1-7 to validate, but you should retrain on either all weeks or on weeks 3-9 to make the test predictions. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 123161,
      "author_name": "alexgyl",
      "author_url": "",
      "post_date": "06/10/2016 02:15:26",
      "content": "<p>Hey Sterby, just curious on this, how did you handle items that did not exist in weeks 1-7? Could that be affecting your score since there can be new products released? I personally have downloaded the data but haven't had time to get started on it yet but that's my hunch on this. Cheers!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 123198,
      "author_name": "ianlokh",
      "author_url": "",
      "post_date": "06/10/2016 08:58:27",
      "content": "<p>Hi Sterby</p>\n\n<p>I am not able to see week 1, 2 in the training data. May I know where you got that data from?</p>\n\n<p>Ian</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 123200,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "06/10/2016 09:12:56",
      "content": "<p>I didn't checked if weeks 1 + 2 exists. Just assumed it ;) I splited &lt;8 / &gt;=8</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125607,
      "author_name": "ermc23",
      "author_url": "",
      "post_date": "06/30/2016 20:21:09",
      "content": "<p>Did you use the test set?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125660,
      "author_name": "",
      "author_url": "",
      "post_date": "07/01/2016 11:15:10",
      "content": "<p>Same  thing   here, tested a model    that  gives 0.51 on    holdout data sets  and on LB   gives 0.58.\nDid you guys   notice   sumething unusual   at  final distribution.\nBeing  Pareto in the  training  frame could not  hold   because if there  is increseang  behaviour  in  test  semana this  could also  mean those 2 weks  could come from some  local  holidays  or Christmas.\nWith this talled what  if    found   out final distribution is  gaussian?\nIt would  be interesting if other competitors  could    give   feedback on this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126005,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "07/05/2016 13:20:05",
      "content": "<p>Did you re-train your model, including weeks 8+9, AFTER your local validation?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126007,
      "author_name": "abednadir",
      "author_url": "",
      "post_date": "07/05/2016 13:33:54",
      "content": "<p>I got .32 in local CV but .503 on LB :(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126012,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "07/05/2016 14:00:51",
      "content": "<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126026,
      "author_name": "abednadir",
      "author_url": "",
      "post_date": "07/05/2016 16:38:41",
      "content": "<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126033,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "07/05/2016 17:54:44",
      "content": "<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126035,
      "author_name": "abednadir",
      "author_url": "",
      "post_date": "07/05/2016 17:56:24",
      "content": "<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126037,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "07/05/2016 18:14:07",
      "content": "<p>[quote=SteveKane;126035]</p>\n\n<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>\n\n<p>[/quote]</p>\n\n<p>I don't use CV, but a holdout set by weeks. As I described.\nI use the university cluster ;)\nRight now I'm using an ensemble of xgboost models and randomforests.</p>\n\n<p>Any suggestions?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126038,
      "author_name": "abednadir",
      "author_url": "",
      "post_date": "07/05/2016 18:18:12",
      "content": "<p>[quote=Sterby;126037]</p>\n\n<p>[quote=SteveKane;126035]</p>\n\n<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>\n\n<p>[/quote]</p>\n\n<p>I don't use CV, but a holdout set by weeks. As I described.\nI use the university cluster ;)\nRight now I'm using an ensemble of xgboost models and randomforests.</p>\n\n<p>Any suggestions?</p>\n\n<p>[/quote]</p>\n\n<p>lucky...I'm using my laptop and I let it run when I'm at work or sleeping haha</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126040,
      "author_name": "chandrs",
      "author_url": "",
      "post_date": "07/05/2016 18:56:49",
      "content": "<p>What is the cluster config? </p>\n\n<p>[quote=Sterby;126037]</p>\n\n<p>[quote=SteveKane;126035]</p>\n\n<p>[quote=Sterby;126033]</p>\n\n<p>[quote=SteveKane;126026]</p>\n\n<p>[quote=Sterby;126012]</p>\n\n<p>I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\nThis results in a LB-Score of 0.4663. \nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...</p>\n\n<p>[/quote]</p>\n\n<p>are you saying that you used 95% of the training set to train your model?</p>\n\n<p>so you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?</p>\n\n<p>[/quote]\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \n=&gt; faster results\nOf course, for submission I used the 100% set ;)</p>\n\n<p>[/quote]</p>\n\n<p>ok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?</p>\n\n<p>if you don't mind saying, what kind of model is it?</p>\n\n<p>[/quote]</p>\n\n<p>I don't use CV, but a holdout set by weeks. As I described.\nI use the university cluster ;)\nRight now I'm using an ensemble of xgboost models and randomforests.</p>\n\n<p>Any suggestions?</p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126263,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/07/2016 07:20:34",
      "content": "<p>It is key to have the right cv framework.  That's what I learned the hard way in the few Kaggle competitions I entered recently.</p>\n\n<p>For this one, it is possible to have a fairly reliable cv.  With mine I currently have a local cv of 0.457 and a public LB of 0.453.</p>\n\n<p>It took me a while (much too long) to find it, and I suspect current leaders found something similar or better much more rapidly than me.  </p>\n\n<p>Without reliable CV framework you are basically tuning randomly, or, worse, overtunning to the public LB.  Keep looking for a good CV setting if you do not have one.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126265,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "07/07/2016 07:37:34",
      "content": "<p>Would you mind sharing your idea how to perform this validation? :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126271,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/07/2016 08:16:21",
      "content": "<p>I'll share after the competition ends.   Here is a hint: read Gert's interview after he won a sales forecast competition: <a href=\"http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/\">http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/</a></p>\n\n<p>I tried to apply his advice to the current competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126280,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "07/07/2016 10:13:00",
      "content": "<p>Nice, I did that too :)\nSo no more secret sauce ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 128372,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "07/19/2016 17:26:55",
      "content": "<p>Is there any evidence, that time plays a crucial roll here?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 128479,
      "author_name": "crimeprophet",
      "author_url": "",
      "post_date": "07/20/2016 12:38:34",
      "content": "<p><img src=\"https://i.imgflip.com/17oh2g.jpg\" alt=\"enter image description here\" title></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 128564,
      "author_name": "crimeprophet",
      "author_url": "",
      "post_date": "07/21/2016 08:22:28",
      "content": "<p>Aw man, not a single vote on my joke... I'm never making grand master...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "123083": "Hi guys,\r\n\r\nas we all know it will be crucial for this challenge to have a **good validation strategy**.\r\nSo let's start talking about it.\r\n\r\nI'm currently splitting the training data in a train and validation set by weeks. I use week 1-7 to train and evaluate  on week 8+9.\r\nIs there any better idea?\r\n\r\nMy current model gives me locally a score of 0.65667.\r\nBut when I submit with this model I get a LB-Score of 1.07689. \r\n\r\nSo it looks like I'm overfitting (I don't see how) or make some other mistake. \r\nHas anyone an idea what's happening here?\r\n\r\nBest,\r\nSterby",
    "123113": "Your CV strategy seams sound. Not knowing anything about your implementation or models, it's rather difficult to find a mistake. \r\nOn a hunch, i would tell you to double check that the model you are using for test predictions is trained on the newest available data. It's OK to train on weeks 1-7 to validate, but you should retrain on either all weeks or on weeks 3-9 to make the test predictions.",
    "123161": "Hey Sterby, just curious on this, how did you handle items that did not exist in weeks 1-7? Could that be affecting your score since there can be new products released? I personally have downloaded the data but haven't had time to get started on it yet but that's my hunch on this. Cheers!",
    "123198": "Hi Sterby\r\n\r\nI am not able to see week 1, 2 in the training data. May I know where you got that data from?\r\n\r\nIan",
    "123200": "I didn't checked if weeks 1 + 2 exists. Just assumed it ;) I splited <8 / >=8",
    "125607": "Did you use the test set?",
    "125660": "Same  thing   here, tested a model    that  gives 0.51 on    holdout data sets  and on LB   gives 0.58.\r\nDid you guys   notice   sumething unusual   at  final distribution.\r\nBeing  Pareto in the  training  frame could not  hold   because if there  is increseang  behaviour  in  test  semana this  could also  mean those 2 weks  could come from some  local  holidays  or Christmas.\r\nWith this talled what  if    found   out final distribution is  gaussian?\r\nIt would  be interesting if other competitors  could    give   feedback on this.",
    "126005": "Did you re-train your model, including weeks 8+9, AFTER your local validation?",
    "126007": "I got .32 in local CV but .503 on LB :(",
    "126012": "I build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...",
    "126026": "[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?",
    "126033": "[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)",
    "126035": "[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?",
    "126037": "[quote=SteveKane;126035]\r\n\r\n[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?\r\n\r\n[/quote]\r\n\r\nI don't use CV, but a holdout set by weeks. As I described.\r\nI use the university cluster ;)\r\nRight now I'm using an ensemble of xgboost models and randomforests.\r\n\r\nAny suggestions?",
    "126038": "[quote=Sterby;126037]\r\n\r\n[quote=SteveKane;126035]\r\n\r\n[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?\r\n\r\n[/quote]\r\n\r\nI don't use CV, but a holdout set by weeks. As I described.\r\nI use the university cluster ;)\r\nRight now I'm using an ensemble of xgboost models and randomforests.\r\n\r\nAny suggestions?\r\n\r\n\r\n[/quote]\r\n\r\nlucky...I'm using my laptop and I let it run when I'm at work or sleeping haha",
    "126040": "What is the cluster config? \r\n\r\n[quote=Sterby;126037]\r\n\r\n[quote=SteveKane;126035]\r\n\r\n[quote=Sterby;126033]\r\n\r\n[quote=SteveKane;126026]\r\n\r\n[quote=Sterby;126012]\r\n\r\nI build features on week 3-5, trained the model on 6+7 and validated on 8+9.\r\nOn my local validation with a 5% random sample this leads to a score of 0.5779.\r\nThen I use week 3-7 of the whole set to build the features and train on week 8+9.\r\nThis results in a LB-Score of 0.4663. \r\nNote that the validation score gets better if I use the whole dataset for validation. But with 5% its much faster and makes no difference...\r\n\t\r\n\r\n[/quote]\r\n\r\nare you saying that you used 95% of the training set to train your model?\r\n\r\nso you used 95% * 78 million rows to train and then 5% * 78 millions rows for validation?\r\n\r\n\r\n[/quote]\r\nNo, I tried to say, that I only do the whole validation (and tuning) process on 5% of the data. All what I was saying about splitting and so on was performed on only 5% of the training data. \r\n=> faster results\r\nOf course, for submission I used the 100% set ;)\r\n\r\n[/quote]\r\n\r\nok, so you mean that once you've tuned it using CV with 5% of the data, you then train your model on 100% of the data before using it to predict on the test set??? What kind of hardware are you working with?\r\n\r\nif you don't mind saying, what kind of model is it?\r\n\r\n[/quote]\r\n\r\nI don't use CV, but a holdout set by weeks. As I described.\r\nI use the university cluster ;)\r\nRight now I'm using an ensemble of xgboost models and randomforests.\r\n\r\nAny suggestions?\r\n\r\n\r\n[/quote]",
    "126263": "It is key to have the right cv framework.  That's what I learned the hard way in the few Kaggle competitions I entered recently.\r\n\r\nFor this one, it is possible to have a fairly reliable cv.  With mine I currently have a local cv of 0.457 and a public LB of 0.453.\r\n\r\nIt took me a while (much too long) to find it, and I suspect current leaders found something similar or better much more rapidly than me.  \r\n\r\nWithout reliable CV framework you are basically tuning randomly, or, worse, overtunning to the public LB.  Keep looking for a good CV setting if you do not have one.",
    "126265": "Would you mind sharing your idea how to perform this validation? :)",
    "126271": "I'll share after the competition ends.   Here is a hint: read Gert's interview after he won a sales forecast competition: http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/\r\n\r\nI tried to apply his advice to the current competition.",
    "126280": "Nice, I did that too :)\r\nSo no more secret sauce ;)",
    "128372": "Is there any evidence, that time plays a crucial roll here?",
    "128479": "![enter image description here][1]\r\n\r\n\r\n  [1]: https://i.imgflip.com/17oh2g.jpg",
    "128564": "Aw man, not a single vote on my joke... I'm never making grand master..."
  },
  "source": "meta"
}