{
  "id": 22322,
  "title": "XGB external memory?",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/22322",
  "author_name": "",
  "post_date": "2016-07-16T17:34:39.377Z",
  "votes": null,
  "comment_count": 1,
  "views": 357,
  "content": "<p>Dear all:</p>\n\n<p>I am a newbie when it comes to competition with large dataset (which I think this competition qualifies)</p>\n\n<p>While learning from all the amazing &quot;product mean/median&quot; scripts and Mr. Landry's wonderful &quot;h2o GBM&quot; script,  I am also trying my hand to use XGB (as always!) to build some models. </p>\n\n<p>and I find that I keep encountering memory issue!  with my AWS R3 instance with 64G of RAM, I get &quot;std::bad_alloc&quot; error when I try to compute using only week 9's data as training data (just above 1M records). A bit of investigation indicate that this is to do with XGB requiring  more memory than what is available (see <a href=\"https://github.com/dmlc/xgboost/issues/108\">link</a>)</p>\n\n<p>It seems that the only realistic option for using XGB is to use the <a href=\"http://xgboost.readthedocs.io/en/latest/how_to/external_memory.html\">external memory</a> version which is still in beta mode.</p>\n\n<p>So here is  my question, for those of you who are using XGB already, are you using this external memory version?  or are you applying specific methods to handle the data so that you can perform &quot;in-memory&quot; computation with XGB efficiently?</p>\n\n<p>P.S. I also found that even the prediction function is very slow (as compare to H2o), while using only 10,000 rows of data as training data (only for experiment), I can perform the training just like small dataset, but now the time that takes to predict for all the testing data  lasted more than 20 minutes.</p>\n\n<p>Best Regards</p>\n\n<p>Yifan </p>",
  "messages": [
    {
      "id": "127973",
      "postDate": "07/16/2016 17:34:39",
      "content": "<p>Dear all:</p>\n\n<p>I am a newbie when it comes to competition with large dataset (which I think this competition qualifies)</p>\n\n<p>While learning from all the amazing &quot;product mean/median&quot; scripts and Mr. Landry's wonderful &quot;h2o GBM&quot; script,  I am also trying my hand to use XGB (as always!) to build some models. </p>\n\n<p>and I find that I keep encountering memory issue!  with my AWS R3 instance with 64G of RAM, I get &quot;std::bad_alloc&quot; error when I try to compute using only week 9's data as training data (just above 1M records). A bit of investigation indicate that this is to do with XGB requiring  more memory than what is available (see <a href=\"https://github.com/dmlc/xgboost/issues/108\">link</a>)</p>\n\n<p>It seems that the only realistic option for using XGB is to use the <a href=\"http://xgboost.readthedocs.io/en/latest/how_to/external_memory.html\">external memory</a> version which is still in beta mode.</p>\n\n<p>So here is  my question, for those of you who are using XGB already, are you using this external memory version?  or are you applying specific methods to handle the data so that you can perform &quot;in-memory&quot; computation with XGB efficiently?</p>\n\n<p>P.S. I also found that even the prediction function is very slow (as compare to H2o), while using only 10,000 rows of data as training data (only for experiment), I can perform the training just like small dataset, but now the time that takes to predict for all the testing data  lasted more than 20 minutes.</p>\n\n<p>Best Regards</p>\n\n<p>Yifan </p>",
      "rawMarkdown": "Dear all:\r\n\r\nI am a newbie when it comes to competition with large dataset (which I think this competition qualifies)\r\n\r\nWhile learning from all the amazing \"product mean/median\" scripts and Mr. Landry's wonderful \"h2o GBM\" script,  I am also trying my hand to use XGB (as always!) to build some models. \r\n\r\nand I find that I keep encountering memory issue!  with my AWS R3 instance with 64G of RAM, I get \"std::bad_alloc\" error when I try to compute using only week 9's data as training data (just above 1M records). A bit of investigation indicate that this is to do with XGB requiring  more memory than what is available (see [link][1])\r\n\r\nIt seems that the only realistic option for using XGB is to use the [external memory][2] version which is still in beta mode.\r\n\r\nSo here is  my question, for those of you who are using XGB already, are you using this external memory version?  or are you applying specific methods to handle the data so that you can perform \"in-memory\" computation with XGB efficiently?\r\n\r\nP.S. I also found that even the prediction function is very slow (as compare to H2o), while using only 10,000 rows of data as training data (only for experiment), I can perform the training just like small dataset, but now the time that takes to predict for all the testing data  lasted more than 20 minutes.\r\n\r\n\r\nBest Regards\r\n\r\nYifan \r\n\r\n\r\n  [1]: https://github.com/dmlc/xgboost/issues/108\r\n  [2]: http://xgboost.readthedocs.io/en/latest/how_to/external_memory.html",
      "votes": null
    },
    {
      "id": "127990",
      "postDate": "07/16/2016 19:27:07",
      "content": "<p>School boy error I made earlier and now it is solved :)</p>\n\n<p><strong>solution: use XGB.Regressor</strong>  (instead of XGB.Classifier)</p>",
      "rawMarkdown": "School boy error I made earlier and now it is solved :)\r\n\r\n**solution: use XGB.Regressor**  (instead of XGB.Classifier)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 127990,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "07/16/2016 19:27:07",
      "content": "<p>School boy error I made earlier and now it is solved :)</p>\n\n<p><strong>solution: use XGB.Regressor</strong>  (instead of XGB.Classifier)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "127973": "Dear all:\r\n\r\nI am a newbie when it comes to competition with large dataset (which I think this competition qualifies)\r\n\r\nWhile learning from all the amazing \"product mean/median\" scripts and Mr. Landry's wonderful \"h2o GBM\" script,  I am also trying my hand to use XGB (as always!) to build some models. \r\n\r\nand I find that I keep encountering memory issue!  with my AWS R3 instance with 64G of RAM, I get \"std::bad_alloc\" error when I try to compute using only week 9's data as training data (just above 1M records). A bit of investigation indicate that this is to do with XGB requiring  more memory than what is available (see [link][1])\r\n\r\nIt seems that the only realistic option for using XGB is to use the [external memory][2] version which is still in beta mode.\r\n\r\nSo here is  my question, for those of you who are using XGB already, are you using this external memory version?  or are you applying specific methods to handle the data so that you can perform \"in-memory\" computation with XGB efficiently?\r\n\r\nP.S. I also found that even the prediction function is very slow (as compare to H2o), while using only 10,000 rows of data as training data (only for experiment), I can perform the training just like small dataset, but now the time that takes to predict for all the testing data  lasted more than 20 minutes.\r\n\r\n\r\nBest Regards\r\n\r\nYifan \r\n\r\n\r\n  [1]: https://github.com/dmlc/xgboost/issues/108\r\n  [2]: http://xgboost.readthedocs.io/en/latest/how_to/external_memory.html",
    "127990": "School boy error I made earlier and now it is solved :)\r\n\r\n**solution: use XGB.Regressor**  (instead of XGB.Classifier)"
  },
  "source": "meta"
}