{
  "id": 21929,
  "title": "does xgboost work in this competition",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21929",
  "author_name": "",
  "post_date": "2016-06-28T13:23:50.157Z",
  "votes": null,
  "comment_count": 16,
  "views": 1587,
  "content": "<p>hello all, i have been trying random forest and knn as most of you. when i started trying xgboost, i got really bad result around 0.3 range and the training time is also long. i am wondering if anyone have any success with xgboost?  i am using multi:softprob and pick the top 3 probabilities. any feedback or thoughts are truly appreciated. </p>",
  "messages": [
    {
      "id": "125295",
      "postDate": "06/28/2016 13:23:50",
      "content": "<p>hello all, i have been trying random forest and knn as most of you. when i started trying xgboost, i got really bad result around 0.3 range and the training time is also long. i am wondering if anyone have any success with xgboost?  i am using multi:softprob and pick the top 3 probabilities. any feedback or thoughts are truly appreciated. </p>",
      "rawMarkdown": "hello all, i have been trying random forest and knn as most of you. when i started trying xgboost, i got really bad result around 0.3 range and the training time is also long. i am wondering if anyone have any success with xgboost?  i am using multi:softprob and pick the top 3 probabilities. any feedback or thoughts are truly appreciated.",
      "votes": null
    },
    {
      "id": "125302",
      "postDate": "06/28/2016 13:58:10",
      "content": "<p>Short answer is yes. Basically if you can get RF to work  then xgboost should work as well just better.\nIn my script I can use RF as a drop in replacement for xgboost and xgboost gives me slightly better results.</p>\n\n<p>It is hard to give you any feedback without any code to comment on.</p>",
      "rawMarkdown": "Short answer is yes. Basically if you can get RF to work  then xgboost should work as well just better.\r\nIn my script I can use RF as a drop in replacement for xgboost and xgboost gives me slightly better results.\r\n\r\nIt is hard to give you any feedback without any code to comment on.",
      "votes": null
    },
    {
      "id": "125365",
      "postDate": "06/29/2016 00:48:52",
      "content": "<p>@wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...</p>",
      "rawMarkdown": "wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...",
      "votes": null
    },
    {
      "id": "125386",
      "postDate": "06/29/2016 03:10:29",
      "content": "<p>[quote=Shubin;125365]</p>\n\n<p>@wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...</p>\n\n<p>[/quote]\nuse  spot pricing</p>",
      "rawMarkdown": "[quote=Shubin;125365]\r\n\r\n@wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...\r\n\r\n[/quote]\r\nuse  spot pricing",
      "votes": null
    },
    {
      "id": "125406",
      "postDate": "06/29/2016 07:27:28",
      "content": "<p>If i were you I would experiment with much lower values of ETA as well.</p>",
      "rawMarkdown": "If i were you I would experiment with much lower values of ETA as well.",
      "votes": null
    },
    {
      "id": "125410",
      "postDate": "06/29/2016 08:14:13",
      "content": "<p>changing &quot;nthread&quot; parameter may speed up the xgb training.   In my script, I use nthread=4 which gives the best speed up.</p>",
      "rawMarkdown": "changing \"nthread\" parameter may speed up the xgb training.   In my script, I use nthread=4 which gives the best speed up.",
      "votes": null
    },
    {
      "id": "125589",
      "postDate": "06/30/2016 16:54:51",
      "content": "<p>I use naive knn, and can get 0.47 .</p>\n\n<p>however, it is still disappointing...</p>\n\n<p>I believe the virtual world is so small, so KNN is not a good method</p>",
      "rawMarkdown": "I use naive knn, and can get 0.47 .\r\n\r\nhowever, it is still disappointing...\r\n\r\nI believe the virtual world is so small, so KNN is not a good method",
      "votes": null
    },
    {
      "id": "125608",
      "postDate": "06/30/2016 20:23:12",
      "content": "<p>you may wand to rescale and weigh each feature if KNN is applied. </p>\n\n<p>this script may help you.\n<a href=\"https://www.kaggle.com/aldente/facebook-v-predicting-check-ins/mad-scripts-battle-zzz/code\">https://www.kaggle.com/aldente/facebook-v-predicting-check-ins/mad-scripts-battle-zzz/code</a></p>",
      "rawMarkdown": "you may wand to rescale and weigh each feature if KNN is applied. \r\n\r\nthis script may help you.\r\nhttps://www.kaggle.com/aldente/facebook-v-predicting-check-ins/mad-scripts-battle-zzz/code",
      "votes": null
    },
    {
      "id": "125609",
      "postDate": "06/30/2016 20:23:36",
      "content": "<p>@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience</p>\n\n<p>@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?</p>",
      "rawMarkdown": "w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience\r\n\r\n@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?",
      "votes": null
    },
    {
      "id": "125610",
      "postDate": "06/30/2016 20:28:07",
      "content": "<p>[quote=Shubin;125609]</p>\n\n<p>@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience</p>\n\n<p>@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?</p>\n\n<p>[/quote]\nYou may not want to rely on n_jobs = -1, one could put different grid to different processes or thread, and save the partial result.</p>",
      "rawMarkdown": "[quote=Shubin;125609]\r\n\r\n@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience\r\n\r\n@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?\r\n\r\n[/quote]\r\nYou may not want to rely on n_jobs = -1, one could put different grid to different processes or thread, and save the partial result.",
      "votes": null
    },
    {
      "id": "125611",
      "postDate": "06/30/2016 20:31:48",
      "content": "<p>[quote=Shubin;125609]</p>\n\n<p>@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience</p>\n\n<p>@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?</p>\n\n<p>[/quote]\nForgot to mention, look the historical prices and some of the instances are pretty low at night. If you \nparallelize by grid number, half a night is a good bound for the running time.</p>",
      "rawMarkdown": "[quote=Shubin;125609]\r\n\r\n@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience\r\n\r\n@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?\r\n\r\n[/quote]\r\nForgot to mention, look the historical prices and some of the instances are pretty low at night. If you \r\nparallelize by grid number, half a night is a good bound for the running time.",
      "votes": null
    },
    {
      "id": "125612",
      "postDate": "06/30/2016 20:45:58",
      "content": "<p>Yes. In my script, n_threads=8 can reduce the running by half.</p>",
      "rawMarkdown": "Yes. In my script, n_threads=8 can reduce the running by half.",
      "votes": null
    },
    {
      "id": "125616",
      "postDate": "06/30/2016 21:41:35",
      "content": "<p>@ w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...</p>\n\n<p>@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?</p>",
      "rawMarkdown": "w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...\r\n\r\n\r\n\r\n\r\n@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?",
      "votes": null
    },
    {
      "id": "125620",
      "postDate": "06/30/2016 22:39:47",
      "content": "<p>[quote=Shubin;125616]</p>\n\n<p>@ w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...</p>\n\n<p>@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?</p>\n\n<p>[/quote]\nabout 4 hours on a 8xlarge</p>",
      "rawMarkdown": "[quote=Shubin;125616]\r\n\r\n@ w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...\r\n\r\n\r\n\r\n\r\n@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?\r\n\r\n\r\n\r\n\r\n\r\n[/quote]\r\nabout 4 hours on a 8xlarge",
      "votes": null
    },
    {
      "id": "125667",
      "postDate": "07/01/2016 13:39:21",
      "content": "<p>4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.</p>",
      "rawMarkdown": "4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.",
      "votes": null
    },
    {
      "id": "125684",
      "postDate": "07/01/2016 15:19:13",
      "content": "<p>[quote=DennisSakva;125667]</p>\n\n<p>4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.</p>\n\n<p>[/quote]\nEasiest way is split by grid number if you use the grid approach.</p>",
      "rawMarkdown": "[quote=DennisSakva;125667]\r\n\r\n4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.\r\n\r\n[/quote]\r\nEasiest way is split by grid number if you use the grid approach.",
      "votes": null
    },
    {
      "id": "125728",
      "postDate": "07/01/2016 23:13:51",
      "content": "<p>very time consuming. xgb takes about one day.</p>",
      "rawMarkdown": "very time consuming. xgb takes about one day.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 125302,
      "author_name": "mwilam",
      "author_url": "",
      "post_date": "06/28/2016 13:58:10",
      "content": "<p>Short answer is yes. Basically if you can get RF to work  then xgboost should work as well just better.\nIn my script I can use RF as a drop in replacement for xgboost and xgboost gives me slightly better results.</p>\n\n<p>It is hard to give you any feedback without any code to comment on.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125365,
      "author_name": "lishubin",
      "author_url": "",
      "post_date": "06/29/2016 00:48:52",
      "content": "<p>@wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125386,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "06/29/2016 03:10:29",
      "content": "<p>[quote=Shubin;125365]</p>\n\n<p>@wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...</p>\n\n<p>[/quote]\nuse  spot pricing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125406,
      "author_name": "mwilam",
      "author_url": "",
      "post_date": "06/29/2016 07:27:28",
      "content": "<p>If i were you I would experiment with much lower values of ETA as well.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125410,
      "author_name": "chu1990",
      "author_url": "",
      "post_date": "06/29/2016 08:14:13",
      "content": "<p>changing &quot;nthread&quot; parameter may speed up the xgb training.   In my script, I use nthread=4 which gives the best speed up.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125589,
      "author_name": "china1000",
      "author_url": "",
      "post_date": "06/30/2016 16:54:51",
      "content": "<p>I use naive knn, and can get 0.47 .</p>\n\n<p>however, it is still disappointing...</p>\n\n<p>I believe the virtual world is so small, so KNN is not a good method</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125608,
      "author_name": "chu1990",
      "author_url": "",
      "post_date": "06/30/2016 20:23:12",
      "content": "<p>you may wand to rescale and weigh each feature if KNN is applied. </p>\n\n<p>this script may help you.\n<a href=\"https://www.kaggle.com/aldente/facebook-v-predicting-check-ins/mad-scripts-battle-zzz/code\">https://www.kaggle.com/aldente/facebook-v-predicting-check-ins/mad-scripts-battle-zzz/code</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125609,
      "author_name": "lishubin",
      "author_url": "",
      "post_date": "06/30/2016 20:23:36",
      "content": "<p>@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience</p>\n\n<p>@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125610,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "06/30/2016 20:28:07",
      "content": "<p>[quote=Shubin;125609]</p>\n\n<p>@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience</p>\n\n<p>@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?</p>\n\n<p>[/quote]\nYou may not want to rely on n_jobs = -1, one could put different grid to different processes or thread, and save the partial result.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125611,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "06/30/2016 20:31:48",
      "content": "<p>[quote=Shubin;125609]</p>\n\n<p>@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience</p>\n\n<p>@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?</p>\n\n<p>[/quote]\nForgot to mention, look the historical prices and some of the instances are pretty low at night. If you \nparallelize by grid number, half a night is a good bound for the running time.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125612,
      "author_name": "chu1990",
      "author_url": "",
      "post_date": "06/30/2016 20:45:58",
      "content": "<p>Yes. In my script, n_threads=8 can reduce the running by half.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125616,
      "author_name": "lishubin",
      "author_url": "",
      "post_date": "06/30/2016 21:41:35",
      "content": "<p>@ w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...</p>\n\n<p>@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125620,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "06/30/2016 22:39:47",
      "content": "<p>[quote=Shubin;125616]</p>\n\n<p>@ w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...</p>\n\n<p>@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?</p>\n\n<p>[/quote]\nabout 4 hours on a 8xlarge</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125667,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "07/01/2016 13:39:21",
      "content": "<p>4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125684,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "07/01/2016 15:19:13",
      "content": "<p>[quote=DennisSakva;125667]</p>\n\n<p>4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.</p>\n\n<p>[/quote]\nEasiest way is split by grid number if you use the grid approach.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125728,
      "author_name": "chu1990",
      "author_url": "",
      "post_date": "07/01/2016 23:13:51",
      "content": "<p>very time consuming. xgb takes about one day.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "125295": "hello all, i have been trying random forest and knn as most of you. when i started trying xgboost, i got really bad result around 0.3 range and the training time is also long. i am wondering if anyone have any success with xgboost?  i am using multi:softprob and pick the top 3 probabilities. any feedback or thoughts are truly appreciated.",
    "125302": "Short answer is yes. Basically if you can get RF to work  then xgboost should work as well just better.\r\nIn my script I can use RF as a drop in replacement for xgboost and xgboost gives me slightly better results.\r\n\r\nIt is hard to give you any feedback without any code to comment on.",
    "125365": "wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...",
    "125386": "[quote=Shubin;125365]\r\n\r\n@wilam, thanks a lot for reply and i just managed to get xgboost work with eta = 1 for 50 rounds and another 50 round with 0.3. but this still takes 2 mins to run for a grid and i have 1000 grid so 2000 minutes. my aws bill is  going to be crazy...\r\n\r\n[/quote]\r\nuse  spot pricing",
    "125406": "If i were you I would experiment with much lower values of ETA as well.",
    "125410": "changing \"nthread\" parameter may speed up the xgb training.   In my script, I use nthread=4 which gives the best speed up.",
    "125589": "I use naive knn, and can get 0.47 .\r\n\r\nhowever, it is still disappointing...\r\n\r\nI believe the virtual world is so small, so KNN is not a good method",
    "125608": "you may wand to rescale and weigh each feature if KNN is applied. \r\n\r\nthis script may help you.\r\nhttps://www.kaggle.com/aldente/facebook-v-predicting-check-ins/mad-scripts-battle-zzz/code",
    "125609": "w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience\r\n\r\n@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?",
    "125610": "[quote=Shubin;125609]\r\n\r\n@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience\r\n\r\n@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?\r\n\r\n[/quote]\r\nYou may not want to rely on n_jobs = -1, one could put different grid to different processes or thread, and save the partial result.",
    "125611": "[quote=Shubin;125609]\r\n\r\n@ w_________g   i use spot machine and after running 10 hours, the price spikes and i lost all the info. i think i need to save the result to s3 while running. sad experience\r\n\r\n@chu_cs_nmsu i set n_jobs = -1 so i use all the cores, in my case is 8. i assume it is faster?\r\n\r\n[/quote]\r\nForgot to mention, look the historical prices and some of the instances are pretty low at night. If you \r\nparallelize by grid number, half a night is a good bound for the running time.",
    "125612": "Yes. In my script, n_threads=8 can reduce the running by half.",
    "125616": "w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...\r\n\r\n\r\n\r\n\r\n@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?",
    "125620": "[quote=Shubin;125616]\r\n\r\n@ w_________g  , you are right, but i think having a script saving intermediate step is also critical. here is a chart showing the pricing of an spot instance. it goes from 0.1 to 7$ per hour for same instance in the past 24 hours...\r\n\r\n\r\n\r\n\r\n@chu_cs_nmsu, if you do not mind, how long does your xgboost model run?\r\n\r\n\r\n\r\n\r\n\r\n[/quote]\r\nabout 4 hours on a 8xlarge",
    "125667": "4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.",
    "125684": "[quote=DennisSakva;125667]\r\n\r\n4 hours is awesome for XGBOOST model. Do you split your models between processes? My single XGBoost model never manages to utilize anything close to 8 processors.\r\n\r\n[/quote]\r\nEasiest way is split by grid number if you use the grid approach.",
    "125728": "very time consuming. xgb takes about one day."
  },
  "source": "meta"
}