{
  "id": 68398,
  "title": "CV and Tuning question",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/68398",
  "author_name": "",
  "post_date": "2018-10-12T09:53:19.686819500Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi,\nI have an issue currently that feels a bit odd and i don't undertand it.</p>\n\n<p>I have been using LGBM  on session level, and then another one on user level using predictions from the user level.\nMy CV-LB difference has been almost constant!!</p>\n\n<p>I'm using a groupKfold of 5 and use OOF predictions to regularize.</p>\n\n<p>I started tuning my models and tried going with lower number of leaves.\nmy CV score got significantly better!!\nBUT - my LB score stopped correlating, and the improvement was a lot milder or sometimes even score got worse.</p>\n\n<p>Why does this happen suddenly?\nwhy going down from 20 leaves to 10 gets such better CV and changes the distance with LB, where it was really constant  for so long?\nI sometimes even got worse LB score after simplifying the model with lower leaves and gaining better CV.</p>\n\n<p>I would love to hear some insight as to why does it happen, and when i should not simplify the model so much even if the CV seems to go better.\nThanks!</p>",
  "messages": [
    {
      "id": "402776",
      "postDate": "10/12/2018 09:53:19",
      "content": "<p>Hi,\nI have an issue currently that feels a bit odd and i don't undertand it.</p>\n\n<p>I have been using LGBM  on session level, and then another one on user level using predictions from the user level.\nMy CV-LB difference has been almost constant!!</p>\n\n<p>I'm using a groupKfold of 5 and use OOF predictions to regularize.</p>\n\n<p>I started tuning my models and tried going with lower number of leaves.\nmy CV score got significantly better!!\nBUT - my LB score stopped correlating, and the improvement was a lot milder or sometimes even score got worse.</p>\n\n<p>Why does this happen suddenly?\nwhy going down from 20 leaves to 10 gets such better CV and changes the distance with LB, where it was really constant  for so long?\nI sometimes even got worse LB score after simplifying the model with lower leaves and gaining better CV.</p>\n\n<p>I would love to hear some insight as to why does it happen, and when i should not simplify the model so much even if the CV seems to go better.\nThanks!</p>",
      "rawMarkdown": "Hi,\nI have an issue currently that feels a bit odd and i don't undertand it.\n\nI have been using LGBM  on session level, and then another one on user level using predictions from the user level.\nMy CV-LB difference has been almost constant!!\n\nI'm using a groupKfold of 5 and use OOF predictions to regularize.\n\nI started tuning my models and tried going with lower number of leaves.\nmy CV score got significantly better!!\nBUT - my LB score stopped correlating, and the improvement was a lot milder or sometimes even score got worse.\n\nWhy does this happen suddenly?\nwhy going down from 20 leaves to 10 gets such better CV and changes the distance with LB, where it was really constant  for so long?\nI sometimes even got worse LB score after simplifying the model with lower leaves and gaining better CV.\n\nI would love to hear some insight as to why does it happen, and when i should not simplify the model so much even if the CV seems to go better.\nThanks!",
      "votes": null
    },
    {
      "id": "402789",
      "postDate": "10/12/2018 10:20:14",
      "content": "<p>I am experiencing the same <a href=\"/tpthegreat\">@tpthegreat</a> (AmirH).  My recent CV was too good - 1.52 and LB at worst 1.65 ish ... </p>\n\n<p>I found that CV from the ensemble models is unstable than that of a single LGBM model. </p>",
      "rawMarkdown": "I am experiencing the same @tpthegreat (AmirH).  My recent CV was too good - 1.52 and LB at worst 1.65 ish ... \n\nI found that CV from the ensemble models is unstable than that of a single LGBM model.",
      "votes": null
    },
    {
      "id": "402795",
      "postDate": "10/12/2018 10:28:54",
      "content": "<p>That's so sad =(\nDo you regularize by predicting on OOF data before ensemble?</p>",
      "rawMarkdown": "That's so sad =(\nDo you regularize by predicting on OOF data before ensemble?",
      "votes": null
    },
    {
      "id": "402798",
      "postDate": "10/12/2018 10:34:45",
      "content": "<p>Nope. I had that later in the pipeline. Currently, I was just improving my single model scores. Alas, THE LEAK happened :( </p>",
      "rawMarkdown": "Nope. I had that later in the pipeline. Currently, I was just improving my single model scores. Alas, THE LEAK happened :(",
      "votes": null
    },
    {
      "id": "402805",
      "postDate": "10/12/2018 10:51:13",
      "content": "<p>Hi Amir,</p>\n\n<p>I experienced same, however, I guess if you have high colsample_by_tree and colsample_by_level, then I think the trees would more use the strongest variables as you have less deep tree and it might happen that one or couple of your strongest variables have different distributions in train and test and thus this phenomenon may take place. However, lowering these two parameters helped me in my case, but not sure if it would be helpful to you as well. I also believe that this situation can be leveraged for feature selection as well. For example - browser </p>",
      "rawMarkdown": "Hi Amir,\n\nI experienced same, however, I guess if you have high colsample_by_tree and colsample_by_level, then I think the trees would more use the strongest variables as you have less deep tree and it might happen that one or couple of your strongest variables have different distributions in train and test and thus this phenomenon may take place. However, lowering these two parameters helped me in my case, but not sure if it would be helpful to you as well. I also believe that this situation can be leveraged for feature selection as well. For example - browser",
      "votes": null
    },
    {
      "id": "402809",
      "postDate": "10/12/2018 11:12:27",
      "content": "<p>Hi Amir,\nI also have the same problem as most of us have. I have seen following discussion that refers to this problem in the second note out of the three. That one and some comments under this discussion may help you. I haven't tried things that are suggested though. I just wanted let you know in case you missed.</p>\n\n<p><a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67767\">https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67767</a></p>",
      "rawMarkdown": "Hi Amir,\nI also have the same problem as most of us have. I have seen following discussion that refers to this problem in the second note out of the three. That one and some comments under this discussion may help you. I haven't tried things that are suggested though. I just wanted let you know in case you missed.\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67767",
      "votes": null
    },
    {
      "id": "403634",
      "postDate": "10/14/2018 06:54:26",
      "content": "<p>This is really odd!!\nFor testing - i trained a session level model with 20 leaves, and with 10 leaves. where the 10 leaves has 0.003 better LB score</p>\n\n<p>when using them separately in a 6 leaves USER level model, that uses these session level predictions. the better, 10 leaves session prediction produces WORSE LB score by 0.003.</p>\n\n<p>Why does this happen?\nsame infastructure of user based model, produced worse LB when working with better session level predictions..</p>\n\n<p>I would expect that when using simpler model (6 leaves instead of 20 leaves), i willl get LESS overfitting, but this seems to be the other way around.</p>",
      "rawMarkdown": "This is really odd!!\nFor testing - i trained a session level model with 20 leaves, and with 10 leaves. where the 10 leaves has 0.003 better LB score\n\nwhen using them separately in a 6 leaves USER level model, that uses these session level predictions. the better, 10 leaves session prediction produces WORSE LB score by 0.003.\n\nWhy does this happen?\nsame infastructure of user based model, produced worse LB when working with better session level predictions..\n\nI would expect that when using simpler model (6 leaves instead of 20 leaves), i willl get LESS overfitting, but this seems to be the other way around.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 402789,
      "author_name": "shaz13",
      "author_url": "",
      "post_date": "10/12/2018 10:20:14",
      "content": "<p>I am experiencing the same <a href=\"/tpthegreat\">@tpthegreat</a> (AmirH).  My recent CV was too good - 1.52 and LB at worst 1.65 ish ... </p>\n\n<p>I found that CV from the ensemble models is unstable than that of a single LGBM model. </p>",
      "votes": null,
      "replies": [
        {
          "id": 402795,
          "author_name": "tpthegreat",
          "author_url": "",
          "post_date": "10/12/2018 10:28:54",
          "content": "<p>That's so sad =(\nDo you regularize by predicting on OOF data before ensemble?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 402798,
          "author_name": "shaz13",
          "author_url": "",
          "post_date": "10/12/2018 10:34:45",
          "content": "<p>Nope. I had that later in the pipeline. Currently, I was just improving my single model scores. Alas, THE LEAK happened :( </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 402805,
      "author_name": "indranilbhattacharya",
      "author_url": "",
      "post_date": "10/12/2018 10:51:13",
      "content": "<p>Hi Amir,</p>\n\n<p>I experienced same, however, I guess if you have high colsample_by_tree and colsample_by_level, then I think the trees would more use the strongest variables as you have less deep tree and it might happen that one or couple of your strongest variables have different distributions in train and test and thus this phenomenon may take place. However, lowering these two parameters helped me in my case, but not sure if it would be helpful to you as well. I also believe that this situation can be leveraged for feature selection as well. For example - browser </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 402809,
      "author_name": "lacino",
      "author_url": "",
      "post_date": "10/12/2018 11:12:27",
      "content": "<p>Hi Amir,\nI also have the same problem as most of us have. I have seen following discussion that refers to this problem in the second note out of the three. That one and some comments under this discussion may help you. I haven't tried things that are suggested though. I just wanted let you know in case you missed.</p>\n\n<p><a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67767\">https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67767</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 403634,
      "author_name": "tpthegreat",
      "author_url": "",
      "post_date": "10/14/2018 06:54:26",
      "content": "<p>This is really odd!!\nFor testing - i trained a session level model with 20 leaves, and with 10 leaves. where the 10 leaves has 0.003 better LB score</p>\n\n<p>when using them separately in a 6 leaves USER level model, that uses these session level predictions. the better, 10 leaves session prediction produces WORSE LB score by 0.003.</p>\n\n<p>Why does this happen?\nsame infastructure of user based model, produced worse LB when working with better session level predictions..</p>\n\n<p>I would expect that when using simpler model (6 leaves instead of 20 leaves), i willl get LESS overfitting, but this seems to be the other way around.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "402776": "Hi,\nI have an issue currently that feels a bit odd and i don't undertand it.\n\nI have been using LGBM  on session level, and then another one on user level using predictions from the user level.\nMy CV-LB difference has been almost constant!!\n\nI'm using a groupKfold of 5 and use OOF predictions to regularize.\n\nI started tuning my models and tried going with lower number of leaves.\nmy CV score got significantly better!!\nBUT - my LB score stopped correlating, and the improvement was a lot milder or sometimes even score got worse.\n\nWhy does this happen suddenly?\nwhy going down from 20 leaves to 10 gets such better CV and changes the distance with LB, where it was really constant  for so long?\nI sometimes even got worse LB score after simplifying the model with lower leaves and gaining better CV.\n\nI would love to hear some insight as to why does it happen, and when i should not simplify the model so much even if the CV seems to go better.\nThanks!",
    "402789": "I am experiencing the same @tpthegreat (AmirH).  My recent CV was too good - 1.52 and LB at worst 1.65 ish ... \n\nI found that CV from the ensemble models is unstable than that of a single LGBM model.",
    "402795": "That's so sad =(\nDo you regularize by predicting on OOF data before ensemble?",
    "402798": "Nope. I had that later in the pipeline. Currently, I was just improving my single model scores. Alas, THE LEAK happened :(",
    "402805": "Hi Amir,\n\nI experienced same, however, I guess if you have high colsample_by_tree and colsample_by_level, then I think the trees would more use the strongest variables as you have less deep tree and it might happen that one or couple of your strongest variables have different distributions in train and test and thus this phenomenon may take place. However, lowering these two parameters helped me in my case, but not sure if it would be helpful to you as well. I also believe that this situation can be leveraged for feature selection as well. For example - browser",
    "402809": "Hi Amir,\nI also have the same problem as most of us have. I have seen following discussion that refers to this problem in the second note out of the three. That one and some comments under this discussion may help you. I haven't tried things that are suggested though. I just wanted let you know in case you missed.\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67767",
    "403634": "This is really odd!!\nFor testing - i trained a session level model with 20 leaves, and with 10 leaves. where the 10 leaves has 0.003 better LB score\n\nwhen using them separately in a 6 leaves USER level model, that uses these session level predictions. the better, 10 leaves session prediction produces WORSE LB score by 0.003.\n\nWhy does this happen?\nsame infastructure of user based model, produced worse LB when working with better session level predictions..\n\nI would expect that when using simpler model (6 leaves instead of 20 leaves), i willl get LESS overfitting, but this seems to be the other way around."
  },
  "source": "meta"
}