Hello guys, I am newbie at kaggle and im trying xgboost for Bimbo. \nThe problem is: when i load the train data and i transfer as_matrix() it takes so much memory that my system( 8gb ram) cant handle. I dont speak about adding some features .. Any advice of applying XGboost to such a big training set ? Thank you in advance.
", "messages": [ { "id": "127972", "postDate": "07/16/2016 17:25:28", "content": "Hello guys, I am newbie at kaggle and im trying xgboost for Bimbo. \nThe problem is: when i load the train data and i transfer as_matrix() it takes so much memory that my system( 8gb ram) cant handle. I dont speak about adding some features .. Any advice of applying XGboost to such a big training set ? Thank you in advance.
", "rawMarkdown": "Hello guys, I am newbie at kaggle and im trying xgboost for Bimbo. \r\nThe problem is: when i load the train data and i transfer as_matrix() it takes so much memory that my system( 8gb ram) cant handle. I dont speak about adding some features .. Any advice of applying XGboost to such a big training set ? Thank you in advance.", "votes": null }, { "id": "127974", "postDate": "07/16/2016 17:36:10", "content": "just asked exactly the same question as you do, so I hope some of the advanced competitors here can help us out :)
", "rawMarkdown": "just asked exactly the same question as you do, so I hope some of the advanced competitors here can help us out :)", "votes": null }, { "id": "127983", "postDate": "07/16/2016 18:45:10", "content": "Two ways to go about this:
\n\nYou can also (try to) convert your data to libsvm format and run xgboost as a command line app. An example of doing that is here:
\n\nhttps://github.com/dmlc/xgboost/tree/master/demo/regression
\n\nWhen you use a scripting language to create a matrix like Python and create a DMatrix, it's copying the data once, so at that time you have two copies of the same data in memory. With the cmdline app, it just loads the data once and then uses it, so peak memory is lower.
\n\nOn the full train dataset, it uses about 1G per feature. So looks like you can probably work with 8 features before you hit that boundary.
\n\nYou can also look into online learners like vowpal wabbit, those never load the entire trainset into memory, but work on it line by line. Or use a neural network approach with training in batches.
\n\nYou can work with data that is too large for your actual memory by using memory mapped files. Numpy has a module that allows you to index into an array that way. I do recommend that you read sequentially and not through some complex index, because it has to page memory in and out and if you have your rows spread throughout memory, it will take a significant longer amount of time.
", "rawMarkdown": "You can also (try to) convert your data to libsvm format and run xgboost as a command line app. An example of doing that is here:\r\n\r\nhttps://github.com/dmlc/xgboost/tree/master/demo/regression\r\n\r\nWhen you use a scripting language to create a matrix like Python and create a DMatrix, it's copying the data once, so at that time you have two copies of the same data in memory. With the cmdline app, it just loads the data once and then uses it, so peak memory is lower.\r\n\r\nOn the full train dataset, it uses about 1G per feature. So looks like you can probably work with 8 features before you hit that boundary. \r\n\r\nYou can also look into online learners like vowpal wabbit, those never load the entire trainset into memory, but work on it line by line. Or use a neural network approach with training in batches.\r\n\r\nYou can work with data that is too large for your actual memory by using memory mapped files. Numpy has a module that allows you to index into an array that way. I do recommend that you read sequentially and not through some complex index, because it has to page memory in and out and if you have your rows spread throughout memory, it will take a significant longer amount of time.", "votes": null }, { "id": "127989", "postDate": "07/16/2016 19:25:59", "content": "@Konrad, thanks for your suggestion, bagging would definitely be a good idea that I would probably get onto in the later stage of this competition.
\n\nand I have just found out what was causing the problem for myself.
\n\nTurned out I was using the wrong XGB sklearn API: I was calling XGB.Classifier for my computation, and although it could still, in theory, perform the computation as required, the required memory and computation time are many time higher than what is required for regression. It was a school boy error :p
\n\nSo, solution: use XGB.Regressor :)
\n\n@salenator, perhaps this is the same issue for you?
", "rawMarkdown": "Konrad, thanks for your suggestion, bagging would definitely be a good idea that I would probably get onto in the later stage of this competition.\r\n\r\nand I have just found out what was causing the problem for myself.\r\n\r\nTurned out I was using the wrong XGB sklearn API: I was calling XGB.Classifier for my computation, and although it could still, in theory, perform the computation as required, the required memory and computation time are many time higher than what is required for regression. It was a school boy error :p\r\n\r\nSo, **solution: use XGB.Regressor :)**\r\n\r\n@salenator, perhaps this is the same issue for you?", "votes": null }, { "id": "127991", "postDate": "07/16/2016 19:31:16", "content": "@Gerard Toonstra, missed your post since I was writing at the same time.\nThanks for your suggestion, I keep this in mind and probably will try out Vowpal Wabbit on a later date
\n\nAs I explained above, at least for my case, it was a simple error of using the wrong learner API. \nFor now I am back to "happily XGBing" mode :)
", "rawMarkdown": "Gerard Toonstra, missed your post since I was writing at the same time.\r\nThanks for your suggestion, I keep this in mind and probably will try out Vowpal Wabbit on a later date \r\n\r\nAs I explained above, at least for my case, it was a simple error of using the wrong learner API. \r\nFor now I am back to \"happily XGBing\" mode :)", "votes": null }, { "id": "127992", "postDate": "07/16/2016 19:36:40", "content": "@Yifian Xie: Actually i dont use xgb from sklearn i use it from xgboost library. But i will try your suggestion and give you info about it. Did u try XGB.Regressor ? Is it working properly?
", "rawMarkdown": "Yifian Xie: Actually i dont use xgb from sklearn i use it from xgboost library. But i will try your suggestion and give you info about it. Did u try XGB.Regressor ? Is it working properly?", "votes": null }, { "id": "127993", "postDate": "07/16/2016 19:39:16", "content": "@salenator, Yes, I am using XGB.Regressor and it solved my problem.
", "rawMarkdown": "salenator, Yes, I am using XGB.Regressor and it solved my problem.", "votes": null }, { "id": "128000", "postDate": "07/16/2016 20:46:51", "content": "I tried this and it doesnt work
", "rawMarkdown": "I tried this and it doesnt work", "votes": null }, { "id": "128004", "postDate": "07/16/2016 21:09:41", "content": "@salenator, without seeing some of your code, or your error message, it is hard to get to the bottom. So here is just some general usage.
\n\nI use XGB's sklearn API, as described here. The one I am using is called xgb.XGBRegressor\nSo, to configure the model, you do something like:
\n\ngeneric_model=xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\n objective="reg:linear", subsample=0.5,\n colsample_bytree=0.8, seed=1234)\nTo fit the model, you do something like the following:
\n\nmodel.fit(train.as_matrix(), target.as_matrix())\nHere, train is the train data imported as pandas data frame with the "Demanda_uni_equil" column removed, and target is just identical to the "Demanda_uni_equil" column using as label
\n\nTo predict, you do something like:
\n\npred=model.predict(test.as_matrix())
", "rawMarkdown": "salenator, without seeing some of your code, or your error message, it is hard to get to the bottom. So here is just some general usage.\r\n\r\nI use XGB's sklearn API, as described [here][1]. The one I am using is called xgb.XGBRegressor\r\nSo, to configure the model, you do something like:\r\n\r\n generic_model=xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\r\n objective=\"reg:linear\", subsample=0.5,\r\n colsample_bytree=0.8, seed=1234)\r\n\r\n\r\nTo fit the model, you do something like the following:\r\n\r\n model.fit(train.as_matrix(), target.as_matrix())\r\n\r\nHere, ***train*** is the train data imported as pandas data frame with the \"Demanda_uni_equil\" column removed, and ***target*** is just identical to the \"Demanda_uni_equil\" column using as label\r\n\r\nTo predict, you do something like:\r\n\r\npred=model.predict(test.as_matrix())\r\n\r\n\r\n\r\n [1]: http://xgboost.readthedocs.io/en/latest/python/python_api.html", "votes": null }, { "id": "128005", "postDate": "07/16/2016 21:11:35", "content": "Missed your last message, when you say it didn't work, do you get some error message?
", "rawMarkdown": "Missed your last message, when you say it didn't work, do you get some error message?", "votes": null }, { "id": "128006", "postDate": "07/16/2016 21:15:36", "content": "No, 8GB / 8GB full ram, i will try to work my problem out.. I am grateful for the help!
", "rawMarkdown": "No, 8GB / 8GB full ram, i will try to work my problem out.. I am grateful for the help!", "votes": null }, { "id": "128008", "postDate": "07/16/2016 21:20:55", "content": "ok, here is something that might help:\nTry using this library: paratext
\n\nIt should help you to import large csv file very efficiently: both in terms of loading time and RAM usage.\nIMHO, 8GB for this competition is a bit tight, my local machine has 16GB RAM and I still bust it quite often.
\n\nAnyhow, try paratext, it should help.
", "rawMarkdown": "ok, here is something that might help:\r\nTry using this library: [paratext][1]\r\n\r\nIt should help you to import large csv file very efficiently: both in terms of loading time and RAM usage.\r\nIMHO, 8GB for this competition is a bit tight, my local machine has 16GB RAM and I still bust it quite often. \r\n\r\nAnyhow, try paratext, it should help.\r\n\r\n [1]: https://github.com/wiseio/paratext", "votes": null }, { "id": "128022", "postDate": "07/17/2016 04:52:23", "content": "For this dataset, try using Apache Spark. Apache Spark is a big data tool, it will convert the input data into RDD (Resilient Distributed Datasets). Most of the common ML algorithms are implemented in Spark in the MLlib library (http://spark.apache.org/mllib/) . Python API of the Spark is also there.
\n\nYou do not need multiple machines (or nodes) to set up Spark. It can be installed in standalone mode also on your single laptop. And although I have not tested Spark on this dataset, but I think you will not get memory error on your laptop if you use Spark as like XGB it will not load the whole dataset in the memory. It will convert them in RDDs and at a time the algorithm will run on that particular RDD and not whole dataset. Common ML algorithms are already implemented in Spark. Moreover XGBoost has also provided integration with Spark (https://github.com/dmlc/xgboost/tree/master/jvm-packages) through XGBoost4j package. But its Scala & Java API are only available & not python. So if you want to use Xgboost with Spark, you will need to use Scala/Java.
", "rawMarkdown": "For this dataset, try using **Apache Spark**. Apache Spark is a big data tool, it will convert the input data into RDD (Resilient Distributed Datasets). Most of the common ML algorithms are implemented in Spark in the **MLlib** library (http://spark.apache.org/mllib/) . Python API of the Spark is also there. \r\n\r\nYou do not need multiple machines (or nodes) to set up Spark. It can be installed in standalone mode also on your single laptop. And although I have not tested Spark on this dataset, but I think you will not get memory error on your laptop if you use Spark as like XGB it will not load the whole dataset in the memory. It will convert them in RDDs and at a time the algorithm will run on that particular RDD and not whole dataset. Common ML algorithms are already implemented in Spark. Moreover XGBoost has also provided integration with Spark (https://github.com/dmlc/xgboost/tree/master/jvm-packages) through XGBoost4j package. But its Scala & Java API are only available & not python. So if you want to use Xgboost with Spark, you will need to use Scala/Java.", "votes": null }, { "id": "141130", "postDate": "10/25/2016 10:07:35", "content": "I am using latest version of R 3.3.1 in 64 bit windows 10.\nMy training data in csv format is about 860MB.\nI am trying to create xgb.Dmatrix in xgboost package.\nMy RAM is 8GB.
\n\nThe command i used is as follows,
\n\ndtrain <- xgb.DMatrix(as.matrix(sapply(train, as.numeric)), label=train_labels[,1])
\n\nBut I am getting an error as follows,
\n\nCannot allocate additional 703MB.
\n\nCan somebody tell me how to solve this issue. I am extremely sorry if this question is repetitive.
", "rawMarkdown": "I am using latest version of R 3.3.1 in 64 bit windows 10.\r\nMy training data in csv format is about 860MB.\r\nI am trying to create xgb.Dmatrix in xgboost package.\r\nMy RAM is 8GB.\r\n\r\nThe command i used is as follows,\r\n\r\ndtrain <- xgb.DMatrix(as.matrix(sapply(train, as.numeric)), label=train_labels[,1])\r\n\r\nBut I am getting an error as follows,\r\n\r\nCannot allocate additional 703MB.\r\n\r\nCan somebody tell me how to solve this issue. I am extremely sorry if this question is repetitive.", "votes": null }, { "id": "180512", "postDate": "05/05/2017 18:25:25", "content": "Thank you a lot for these two solutions. Both of them are more than helpful!!!
", "rawMarkdown": "Thank you a lot for these two solutions. Both of them are more than helpful!!!", "votes": null } ], "comments": [ { "id": 127974, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 17:36:10", "content": "just asked exactly the same question as you do, so I hope some of the advanced competitors here can help us out :)
", "votes": null, "replies": [] }, { "id": 127983, "author_name": "konradb", "author_url": "", "post_date": "07/16/2016 18:45:10", "content": "Two ways to go about this:
\n\nThank you a lot for these two solutions. Both of them are more than helpful!!!
", "votes": null, "replies": [] } ] }, { "id": 127987, "author_name": "remap1", "author_url": "", "post_date": "07/16/2016 19:17:22", "content": "You can also (try to) convert your data to libsvm format and run xgboost as a command line app. An example of doing that is here:
\n\nhttps://github.com/dmlc/xgboost/tree/master/demo/regression
\n\nWhen you use a scripting language to create a matrix like Python and create a DMatrix, it's copying the data once, so at that time you have two copies of the same data in memory. With the cmdline app, it just loads the data once and then uses it, so peak memory is lower.
\n\nOn the full train dataset, it uses about 1G per feature. So looks like you can probably work with 8 features before you hit that boundary.
\n\nYou can also look into online learners like vowpal wabbit, those never load the entire trainset into memory, but work on it line by line. Or use a neural network approach with training in batches.
\n\nYou can work with data that is too large for your actual memory by using memory mapped files. Numpy has a module that allows you to index into an array that way. I do recommend that you read sequentially and not through some complex index, because it has to page memory in and out and if you have your rows spread throughout memory, it will take a significant longer amount of time.
", "votes": null, "replies": [] }, { "id": 127989, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 19:25:59", "content": "@Konrad, thanks for your suggestion, bagging would definitely be a good idea that I would probably get onto in the later stage of this competition.
\n\nand I have just found out what was causing the problem for myself.
\n\nTurned out I was using the wrong XGB sklearn API: I was calling XGB.Classifier for my computation, and although it could still, in theory, perform the computation as required, the required memory and computation time are many time higher than what is required for regression. It was a school boy error :p
\n\nSo, solution: use XGB.Regressor :)
\n\n@salenator, perhaps this is the same issue for you?
", "votes": null, "replies": [] }, { "id": 127991, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 19:31:16", "content": "@Gerard Toonstra, missed your post since I was writing at the same time.\nThanks for your suggestion, I keep this in mind and probably will try out Vowpal Wabbit on a later date
\n\nAs I explained above, at least for my case, it was a simple error of using the wrong learner API. \nFor now I am back to "happily XGBing" mode :)
", "votes": null, "replies": [] }, { "id": 127992, "author_name": "", "author_url": "", "post_date": "07/16/2016 19:36:40", "content": "@Yifian Xie: Actually i dont use xgb from sklearn i use it from xgboost library. But i will try your suggestion and give you info about it. Did u try XGB.Regressor ? Is it working properly?
", "votes": null, "replies": [] }, { "id": 127993, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 19:39:16", "content": "@salenator, Yes, I am using XGB.Regressor and it solved my problem.
", "votes": null, "replies": [] }, { "id": 128000, "author_name": "", "author_url": "", "post_date": "07/16/2016 20:46:51", "content": "I tried this and it doesnt work
", "votes": null, "replies": [] }, { "id": 128004, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 21:09:41", "content": "@salenator, without seeing some of your code, or your error message, it is hard to get to the bottom. So here is just some general usage.
\n\nI use XGB's sklearn API, as described here. The one I am using is called xgb.XGBRegressor\nSo, to configure the model, you do something like:
\n\ngeneric_model=xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\n objective="reg:linear", subsample=0.5,\n colsample_bytree=0.8, seed=1234)\nTo fit the model, you do something like the following:
\n\nmodel.fit(train.as_matrix(), target.as_matrix())\nHere, train is the train data imported as pandas data frame with the "Demanda_uni_equil" column removed, and target is just identical to the "Demanda_uni_equil" column using as label
\n\nTo predict, you do something like:
\n\npred=model.predict(test.as_matrix())
", "votes": null, "replies": [] }, { "id": 128005, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 21:11:35", "content": "Missed your last message, when you say it didn't work, do you get some error message?
", "votes": null, "replies": [] }, { "id": 128006, "author_name": "", "author_url": "", "post_date": "07/16/2016 21:15:36", "content": "No, 8GB / 8GB full ram, i will try to work my problem out.. I am grateful for the help!
", "votes": null, "replies": [] }, { "id": 128008, "author_name": "yifanxie", "author_url": "", "post_date": "07/16/2016 21:20:55", "content": "ok, here is something that might help:\nTry using this library: paratext
\n\nIt should help you to import large csv file very efficiently: both in terms of loading time and RAM usage.\nIMHO, 8GB for this competition is a bit tight, my local machine has 16GB RAM and I still bust it quite often.
\n\nAnyhow, try paratext, it should help.
", "votes": null, "replies": [] }, { "id": 128022, "author_name": "abhilashawasthi", "author_url": "", "post_date": "07/17/2016 04:52:23", "content": "For this dataset, try using Apache Spark. Apache Spark is a big data tool, it will convert the input data into RDD (Resilient Distributed Datasets). Most of the common ML algorithms are implemented in Spark in the MLlib library (http://spark.apache.org/mllib/) . Python API of the Spark is also there.
\n\nYou do not need multiple machines (or nodes) to set up Spark. It can be installed in standalone mode also on your single laptop. And although I have not tested Spark on this dataset, but I think you will not get memory error on your laptop if you use Spark as like XGB it will not load the whole dataset in the memory. It will convert them in RDDs and at a time the algorithm will run on that particular RDD and not whole dataset. Common ML algorithms are already implemented in Spark. Moreover XGBoost has also provided integration with Spark (https://github.com/dmlc/xgboost/tree/master/jvm-packages) through XGBoost4j package. But its Scala & Java API are only available & not python. So if you want to use Xgboost with Spark, you will need to use Scala/Java.
", "votes": null, "replies": [] }, { "id": 141130, "author_name": "kiragod", "author_url": "", "post_date": "10/25/2016 10:07:35", "content": "I am using latest version of R 3.3.1 in 64 bit windows 10.\nMy training data in csv format is about 860MB.\nI am trying to create xgb.Dmatrix in xgboost package.\nMy RAM is 8GB.
\n\nThe command i used is as follows,
\n\ndtrain <- xgb.DMatrix(as.matrix(sapply(train, as.numeric)), label=train_labels[,1])
\n\nBut I am getting an error as follows,
\n\nCannot allocate additional 703MB.
\n\nCan somebody tell me how to solve this issue. I am extremely sorry if this question is repetitive.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "127972": "Hello guys, I am newbie at kaggle and im trying xgboost for Bimbo. \r\nThe problem is: when i load the train data and i transfer as_matrix() it takes so much memory that my system( 8gb ram) cant handle. I dont speak about adding some features .. Any advice of applying XGboost to such a big training set ? Thank you in advance.", "127974": "just asked exactly the same question as you do, so I hope some of the advanced competitors here can help us out :)", "127983": "Two ways to go about this: \r\n\r\n 1. use the external memory version http://xgboost.readthedocs.io/en/latest/how_to/external_memory.html\r\n 2. Try bagging: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html\r\n(the implementation given in the link won't help you as this still requires loading into memory, but should give you a general idea). Essentially, it's about subsampling from the dataset, fitting a model (for instance xgboost) on each subset, and then averaging the predictions. It's not ideal, but should work with much lower memory footprint.", "127987": "You can also (try to) convert your data to libsvm format and run xgboost as a command line app. An example of doing that is here:\r\n\r\nhttps://github.com/dmlc/xgboost/tree/master/demo/regression\r\n\r\nWhen you use a scripting language to create a matrix like Python and create a DMatrix, it's copying the data once, so at that time you have two copies of the same data in memory. With the cmdline app, it just loads the data once and then uses it, so peak memory is lower.\r\n\r\nOn the full train dataset, it uses about 1G per feature. So looks like you can probably work with 8 features before you hit that boundary. \r\n\r\nYou can also look into online learners like vowpal wabbit, those never load the entire trainset into memory, but work on it line by line. Or use a neural network approach with training in batches.\r\n\r\nYou can work with data that is too large for your actual memory by using memory mapped files. Numpy has a module that allows you to index into an array that way. I do recommend that you read sequentially and not through some complex index, because it has to page memory in and out and if you have your rows spread throughout memory, it will take a significant longer amount of time.", "127989": "Konrad, thanks for your suggestion, bagging would definitely be a good idea that I would probably get onto in the later stage of this competition.\r\n\r\nand I have just found out what was causing the problem for myself.\r\n\r\nTurned out I was using the wrong XGB sklearn API: I was calling XGB.Classifier for my computation, and although it could still, in theory, perform the computation as required, the required memory and computation time are many time higher than what is required for regression. It was a school boy error :p\r\n\r\nSo, **solution: use XGB.Regressor :)**\r\n\r\n@salenator, perhaps this is the same issue for you?", "127991": "Gerard Toonstra, missed your post since I was writing at the same time.\r\nThanks for your suggestion, I keep this in mind and probably will try out Vowpal Wabbit on a later date \r\n\r\nAs I explained above, at least for my case, it was a simple error of using the wrong learner API. \r\nFor now I am back to \"happily XGBing\" mode :)", "127992": "Yifian Xie: Actually i dont use xgb from sklearn i use it from xgboost library. But i will try your suggestion and give you info about it. Did u try XGB.Regressor ? Is it working properly?", "127993": "salenator, Yes, I am using XGB.Regressor and it solved my problem.", "128000": "I tried this and it doesnt work", "128004": "salenator, without seeing some of your code, or your error message, it is hard to get to the bottom. So here is just some general usage.\r\n\r\nI use XGB's sklearn API, as described [here][1]. The one I am using is called xgb.XGBRegressor\r\nSo, to configure the model, you do something like:\r\n\r\n generic_model=xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\r\n objective=\"reg:linear\", subsample=0.5,\r\n colsample_bytree=0.8, seed=1234)\r\n\r\n\r\nTo fit the model, you do something like the following:\r\n\r\n model.fit(train.as_matrix(), target.as_matrix())\r\n\r\nHere, ***train*** is the train data imported as pandas data frame with the \"Demanda_uni_equil\" column removed, and ***target*** is just identical to the \"Demanda_uni_equil\" column using as label\r\n\r\nTo predict, you do something like:\r\n\r\npred=model.predict(test.as_matrix())\r\n\r\n\r\n\r\n [1]: http://xgboost.readthedocs.io/en/latest/python/python_api.html", "128005": "Missed your last message, when you say it didn't work, do you get some error message?", "128006": "No, 8GB / 8GB full ram, i will try to work my problem out.. I am grateful for the help!", "128008": "ok, here is something that might help:\r\nTry using this library: [paratext][1]\r\n\r\nIt should help you to import large csv file very efficiently: both in terms of loading time and RAM usage.\r\nIMHO, 8GB for this competition is a bit tight, my local machine has 16GB RAM and I still bust it quite often. \r\n\r\nAnyhow, try paratext, it should help.\r\n\r\n [1]: https://github.com/wiseio/paratext", "128022": "For this dataset, try using **Apache Spark**. Apache Spark is a big data tool, it will convert the input data into RDD (Resilient Distributed Datasets). Most of the common ML algorithms are implemented in Spark in the **MLlib** library (http://spark.apache.org/mllib/) . Python API of the Spark is also there. \r\n\r\nYou do not need multiple machines (or nodes) to set up Spark. It can be installed in standalone mode also on your single laptop. And although I have not tested Spark on this dataset, but I think you will not get memory error on your laptop if you use Spark as like XGB it will not load the whole dataset in the memory. It will convert them in RDDs and at a time the algorithm will run on that particular RDD and not whole dataset. Common ML algorithms are already implemented in Spark. Moreover XGBoost has also provided integration with Spark (https://github.com/dmlc/xgboost/tree/master/jvm-packages) through XGBoost4j package. But its Scala & Java API are only available & not python. So if you want to use Xgboost with Spark, you will need to use Scala/Java.", "141130": "I am using latest version of R 3.3.1 in 64 bit windows 10.\r\nMy training data in csv format is about 860MB.\r\nI am trying to create xgb.Dmatrix in xgboost package.\r\nMy RAM is 8GB.\r\n\r\nThe command i used is as follows,\r\n\r\ndtrain <- xgb.DMatrix(as.matrix(sapply(train, as.numeric)), label=train_labels[,1])\r\n\r\nBut I am getting an error as follows,\r\n\r\nCannot allocate additional 703MB.\r\n\r\nCan somebody tell me how to solve this issue. I am extremely sorry if this question is repetitive.", "180512": "Thank you a lot for these two solutions. Both of them are more than helpful!!!" }, "source": "meta" }