{
  "id": 68746,
  "title": "Different methods of predicting Test Data",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/68746",
  "author_name": "",
  "post_date": "2018-10-16T17:59:16.712100500Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>After doing some tests, i have a question:</p>\n\n<p>I saw Olivier predicting Test Data using the folds' train sets, and then averaging all the folds' predictions.</p>\n\n<p>e.g take fold 1 for validation, train folds 2-5, predict on test.. and so on...</p>\n\n<p>meanwhile - in most places i read that it's best to use entire train set to predict Test.</p>\n\n<p>these 2 methods gave me different results in different cases in this competition.</p>\n\n<p>in one session based LGBM, the method of using folds' training to predict test and then averaging them, gave much better LB score than using the entire train set with lgb.cv to find the best iteration and then train to this iteration.</p>\n\n<p>1.4257 full train VS 1.4238 using folds</p>\n\n<p>in another LGBM that uses prior 0/1 classification , i had 0.0003 better result with entire train. this also gave me better consistency when using the prediction in User Level LGBM afterwards.</p>\n\n<p>What is your take on this?\nDo you use both methods or stick to one of them?</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "404999",
      "postDate": "10/16/2018 17:59:16",
      "content": "<p>Hi,</p>\n\n<p>After doing some tests, i have a question:</p>\n\n<p>I saw Olivier predicting Test Data using the folds' train sets, and then averaging all the folds' predictions.</p>\n\n<p>e.g take fold 1 for validation, train folds 2-5, predict on test.. and so on...</p>\n\n<p>meanwhile - in most places i read that it's best to use entire train set to predict Test.</p>\n\n<p>these 2 methods gave me different results in different cases in this competition.</p>\n\n<p>in one session based LGBM, the method of using folds' training to predict test and then averaging them, gave much better LB score than using the entire train set with lgb.cv to find the best iteration and then train to this iteration.</p>\n\n<p>1.4257 full train VS 1.4238 using folds</p>\n\n<p>in another LGBM that uses prior 0/1 classification , i had 0.0003 better result with entire train. this also gave me better consistency when using the prediction in User Level LGBM afterwards.</p>\n\n<p>What is your take on this?\nDo you use both methods or stick to one of them?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "Hi,\n\nAfter doing some tests, i have a question:\n\nI saw Olivier predicting Test Data using the folds' train sets, and then averaging all the folds' predictions.\n\ne.g take fold 1 for validation, train folds 2-5, predict on test.. and so on...\n\nmeanwhile - in most places i read that it's best to use entire train set to predict Test.\n\nthese 2 methods gave me different results in different cases in this competition.\n\nin one session based LGBM, the method of using folds' training to predict test and then averaging them, gave much better LB score than using the entire train set with lgb.cv to find the best iteration and then train to this iteration.\n\n1.4257 full train VS 1.4238 using folds\n\nin another LGBM that uses prior 0/1 classification , i had 0.0003 better result with entire train. this also gave me better consistency when using the prediction in User Level LGBM afterwards.\n\n\nWhat is your take on this?\nDo you use both methods or stick to one of them?\n\nThanks!",
      "votes": null
    },
    {
      "id": "406367",
      "postDate": "10/19/2018 06:15:10",
      "content": "<p>In some previous competitions I have seen the full retrain score clearly better. In others the average of prediction per fold has scored better. Sometimes the difference between them is small especially when compared to the standard deviation between folds. Try both ways and see which works better.</p>",
      "rawMarkdown": "In some previous competitions I have seen the full retrain score clearly better. In others the average of prediction per fold has scored better. Sometimes the difference between them is small especially when compared to the standard deviation between folds. Try both ways and see which works better.",
      "votes": null
    },
    {
      "id": "406643",
      "postDate": "10/19/2018 15:05:24",
      "content": "<p>Consider that when you are averaging over 5 folds, that is 5 different Booster models that may capture the signal differently and therefore the average of them is a blend of different models, but with just <code>lgbm.cv</code> you have only one Booster model. It's an unfair comparison! Try training five <code>lgbm.cv</code>'s with slightly different hyperparameters and then average those results.</p>",
      "rawMarkdown": "Consider that when you are averaging over 5 folds, that is 5 different Booster models that may capture the signal differently and therefore the average of them is a blend of different models, but with just `lgbm.cv` you have only one Booster model. It's an unfair comparison! Try training five `lgbm.cv`'s with slightly different hyperparameters and then average those results.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 406367,
      "author_name": "datahobbit",
      "author_url": "",
      "post_date": "10/19/2018 06:15:10",
      "content": "<p>In some previous competitions I have seen the full retrain score clearly better. In others the average of prediction per fold has scored better. Sometimes the difference between them is small especially when compared to the standard deviation between folds. Try both ways and see which works better.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 406643,
      "author_name": "returnofsputnik",
      "author_url": "",
      "post_date": "10/19/2018 15:05:24",
      "content": "<p>Consider that when you are averaging over 5 folds, that is 5 different Booster models that may capture the signal differently and therefore the average of them is a blend of different models, but with just <code>lgbm.cv</code> you have only one Booster model. It's an unfair comparison! Try training five <code>lgbm.cv</code>'s with slightly different hyperparameters and then average those results.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "404999": "Hi,\n\nAfter doing some tests, i have a question:\n\nI saw Olivier predicting Test Data using the folds' train sets, and then averaging all the folds' predictions.\n\ne.g take fold 1 for validation, train folds 2-5, predict on test.. and so on...\n\nmeanwhile - in most places i read that it's best to use entire train set to predict Test.\n\nthese 2 methods gave me different results in different cases in this competition.\n\nin one session based LGBM, the method of using folds' training to predict test and then averaging them, gave much better LB score than using the entire train set with lgb.cv to find the best iteration and then train to this iteration.\n\n1.4257 full train VS 1.4238 using folds\n\nin another LGBM that uses prior 0/1 classification , i had 0.0003 better result with entire train. this also gave me better consistency when using the prediction in User Level LGBM afterwards.\n\n\nWhat is your take on this?\nDo you use both methods or stick to one of them?\n\nThanks!",
    "406367": "In some previous competitions I have seen the full retrain score clearly better. In others the average of prediction per fold has scored better. Sometimes the difference between them is small especially when compared to the standard deviation between folds. Try both ways and see which works better.",
    "406643": "Consider that when you are averaging over 5 folds, that is 5 different Booster models that may capture the signal differently and therefore the average of them is a blend of different models, but with just `lgbm.cv` you have only one Booster model. It's an unfair comparison! Try training five `lgbm.cv`'s with slightly different hyperparameters and then average those results."
  },
  "source": "meta"
}