train.csv is too large to handle entirely in my computer, so I have split it into a sample of 500,000 random rows (I know I could split using other logic, like using only the last weeks input, for example).
\n\nMy question is, how will I fit my models if my computer can't handle the entire dataset? Is there a function in Sklearn to fit data in increments?
", "messages": [ { "id": 124937, "postDate": "2016-06-23T16:52:38.627Z", "content": "You can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in read_csv(). For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.
Using dtypes is indeed a good idea. Just beware that customer id will still require 64 bits. Here is the code I use FYI:
\n\ntrain = pd.read_csv('Data/train.csv', \n dtype = {'Semana' : 'int32',\n 'Agencia_ID' :'int32',\n 'Canal_ID' : 'int32',\n 'Ruta_SAK' : 'int32',\n 'Cliente-ID' : 'int64',\n 'Producto_ID':'int32',\n 'Venta_hoy':'float32',\n 'Venta_uni_hoy': 'int32',\n 'Dev_uni_proxima':'int32',\n 'Dev_proxima':'float32',\n 'Demanda_uni_equil':'int32'})\nAs said above, you can certianly do even better by using int8 in some places instead of int32.
", "rawMarkdown": "Using dtypes is indeed a good idea. Just beware that customer id will still require 64 bits. Here is the code I use FYI:\r\n\r\n train = pd.read_csv('Data/train.csv', \r\n dtype = {'Semana' : 'int32',\r\n 'Agencia_ID' :'int32',\r\n 'Canal_ID' : 'int32',\r\n 'Ruta_SAK' : 'int32',\r\n 'Cliente-ID' : 'int64',\r\n 'Producto_ID':'int32',\r\n 'Venta_hoy':'float32',\r\n 'Venta_uni_hoy': 'int32',\r\n 'Dev_uni_proxima':'int32',\r\n 'Dev_proxima':'float32',\r\n 'Demanda_uni_equil':'int32'})\r\n\r\nAs said above, you can certianly do even better by using int8 in some places instead of int32.\r\n", "votes": 5 }, { "id": 124964, "postDate": "2016-06-23T23:27:55.010Z", "content": "You can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
@scirpus. I was thinking the same, libffm is next on my list.
", "rawMarkdown": "@scirpus. I was thinking the same, libffm is next on my list.", "votes": 1 }, { "id": 125443, "postDate": "2016-06-29T13:56:48.663Z", "content": "[quote=leexiongan;125441]
\n\n[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\nI can't seem it :(
\n\n[/quote]
\n\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)
", "rawMarkdown": "[quote=leexiongan;125441]\r\n\r\n[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\n\r\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\r\nI can't seem it :(\r\n\r\n[/quote]\r\n\r\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)\r\n", "votes": 1 }, { "id": 125091, "postDate": "2016-06-25T16:58:54.893Z", "content": "Are you sure Cliente_Id requires 64 bits? My largest seems to be 2,015,152,015, which is smaller than largest int32 (2,147,483,647).
\n\n/mattias
", "rawMarkdown": "Are you sure Cliente_Id requires 64 bits? My largest seems to be 2,015,152,015, which is smaller than largest int32 (2,147,483,647).\r\n\r\n/mattias", "votes": 1 }, { "id": 124865, "postDate": "2016-06-22T20:29:14.747Z", "content": "[quote=Cookie;124598]
\n\nHi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set.
\n\nAppreciate any advise offered. Thanx
\n\n[/quote]
\n\nWhat we did in another competition was calculate the relevant metric per ID (so in this case, mean returns per client id) and the substitute the id for this value. When you create dummy variables for the levels you are basically working under the assumptions that the return rate differs between clients. This is more or less a shortcut (with some information loss) but better than not including it at all.
\n\nYou should do this separately for your training and test set as to not include information you would not normally have. You also need to be able to deal with unseen client ids. we simply imputed the median in those cases.
", "rawMarkdown": "[quote=Cookie;124598]\r\n\r\nHi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set. \r\n\r\nAppreciate any advise offered. Thanx\r\n\r\n[/quote]\r\n\r\nWhat we did in another competition was calculate the relevant metric per ID (so in this case, mean returns per client id) and the substitute the id for this value. When you create dummy variables for the levels you are basically working under the assumptions that the return rate differs between clients. This is more or less a shortcut (with some information loss) but better than not including it at all.\r\n \r\nYou should do this separately for your training and test set as to not include information you would not normally have. You also need to be able to deal with unseen client ids. we simply imputed the median in those cases.", "votes": 1 }, { "id": 124557, "postDate": "2016-06-19T22:53:16.237Z", "content": "You should try vowpal wabbit..
", "rawMarkdown": "You should try vowpal wabbit.. ", "votes": 2 }, { "id": 124555, "postDate": "2016-06-19T22:37:39.100Z", "content": "There are some algorithms available for incremental/online-learning in sklearn: docs.
", "rawMarkdown": "There are some algorithms available for incremental/online-learning in sklearn: [docs][1].\r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning", "votes": 2 }, { "id": 125482, "postDate": "2016-06-29T21:28:58.647Z", "content": "is there some "dtype" for R language? or some way to optimize factors?
", "rawMarkdown": "is there some \"dtype\" for R language? or some way to optimize factors?" }, { "id": 125472, "postDate": "2016-06-29T19:23:05.407Z", "content": "I will be surprised if the leaders aren't using LibFFM - this requires a huge amount of memory usually - idle_speculation used it brilliantly in the Expedia competition and I quote - \n"I built the model on a dual socket E5-2699v3 with ~700GB ram and was only able to use about half the bookings after '2014-07-01' due to memory limitations."
\n\nGood luck!!
", "rawMarkdown": "I will be surprised if the leaders aren't using LibFFM - this requires a huge amount of memory usually - idle_speculation used it brilliantly in the Expedia competition and I quote - \r\n\"I built the model on a dual socket E5-2699v3 with ~700GB ram and was only able to use about half the bookings after '2014-07-01' due to memory limitations.\"\r\n\r\nGood luck!! " }, { "id": 125464, "postDate": "2016-06-29T17:30:00.600Z", "content": "I run the bare bone xgboost script for the whole data set. The memory requirement is only 30G. So don't let memory scare you out of this competition
", "rawMarkdown": "I run the bare bone xgboost script for the whole data set. The memory requirement is only 30G. So don't let memory scare you out of this competition" }, { "id": 125451, "postDate": "2016-06-29T15:28:35.703Z", "content": "xgboost has an out of core option, see
\n\nhttps://xgboost.readthedocs.io/en/latest//how_to/external_memory.html
\n\nI haven't used it
", "rawMarkdown": "xgboost has an out of core option, see \r\n\r\nhttps://xgboost.readthedocs.io/en/latest//how_to/external_memory.html\r\n\r\nI haven't used it" }, { "id": 125444, "postDate": "2016-06-29T14:12:13.560Z", "content": "[quote=EZ;125443]
\n\n[quote=leexiongan;125441]
\n\n[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\nI can't seem it :(
\n\n[/quote]
\n\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)
\n\n[/quote]
\n\nThink you can call xgboost's train with parameter xgb_model. Does that do the job?
", "rawMarkdown": "[quote=EZ;125443]\r\n\r\n[quote=leexiongan;125441]\r\n\r\n[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\n\r\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\r\nI can't seem it :(\r\n\r\n[/quote]\r\n\r\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)\r\n\r\n\r\n[/quote]\r\n\r\nThink you can call xgboost's train with parameter xgb_model. Does that do the job?" }, { "id": 125441, "postDate": "2016-06-29T13:47:50.817Z", "content": "[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\nI can't seem it :(
", "rawMarkdown": "[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\n\r\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\r\nI can't seem it :(" }, { "id": 125094, "postDate": "2016-06-25T19:29:52.870Z", "content": "Mattias, you are right :)
", "rawMarkdown": "Mattias, you are right :)" }, { "id": 124966, "postDate": "2016-06-23T23:30:50.850Z", "content": "[quote=Kevin Kirchhoff;124937]
\n\nYou can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in read_csv(). For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.
[/quote]
\n\nGreat idea! Thank you
", "rawMarkdown": "[quote=Kevin Kirchhoff;124937]\r\n\r\nYou can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in `read_csv()`. For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.\r\n\r\n[/quote]\r\n\r\nGreat idea! Thank you" }, { "id": 124965, "postDate": "2016-06-23T23:29:15.063Z", "content": "[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nThanks! Yes, I just learned this earlier in the week. It does in fact make a huge difference!
", "rawMarkdown": "[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\nThanks! Yes, I just learned this earlier in the week. It does in fact make a huge difference!" }, { "id": 124925, "postDate": "2016-06-23T14:45:37.533Z", "content": "Hi Valentine. Thanks for sharing your experiences. Am working on your idea now :-)
", "rawMarkdown": "Hi Valentine. Thanks for sharing your experiences. Am working on your idea now :-)" }, { "id": 124598, "postDate": "2016-06-20T08:22:18.223Z", "content": "Hi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set.
\n\nAppreciate any advise offered. Thanx
", "rawMarkdown": "Hi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set. \r\n\r\nAppreciate any advise offered. Thanx" }, { "id": 124578, "postDate": "2016-06-20T03:54:05.543Z", "content": "[quote=sschnug;124555]
\n\nThere are some algorithms available for incremental/online-learning in sklearn: docs.
\n\n[/quote]
\n\nThis is very helpful, thanks!
", "rawMarkdown": "[quote=sschnug;124555]\r\n\r\nThere are some algorithms available for incremental/online-learning in sklearn: [docs][1].\r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning\r\n\r\n[/quote]\r\n\r\nThis is very helpful, thanks!" }, { "id": 124577, "postDate": "2016-06-20T03:53:03.803Z", "content": "[quote=FredR;124554]
\n\nI don't believe it's possible. The best you could do is do some feature engineering and reduce the content of your train set.
\n\nSo maybe you only pull one or two labels and discard the rest.
\n\nI had the same question and googling left me with "calling fit multiple times will wipe out previous fit calls."
\n\nHowever you can get pretty good results without even using scikitlearn or any machine learning algorithms at all on this data set. At the moment I'm 219 out of 572 and I haven't used anything from scikit learn or xgboost.
\n\n[/quote]
\n\nThat's good to know! I'm still doing some EDA but was already having trouble with the size of this dataset.
", "rawMarkdown": "[quote=FredR;124554]\r\n\r\nI don't believe it's possible. The best you could do is do some feature engineering and reduce the content of your train set. \r\n\r\nSo maybe you only pull one or two labels and discard the rest.\r\n\r\nI had the same question and googling left me with \"calling fit multiple times will wipe out previous fit calls.\"\r\n\r\n\r\nHowever you can get pretty good results without even using scikitlearn or any machine learning algorithms at all on this data set. At the moment I'm 219 out of 572 and I haven't used anything from scikit learn or xgboost.\r\n\r\n[/quote]\r\n\r\nThat's good to know! I'm still doing some EDA but was already having trouble with the size of this dataset." }, { "id": 124544, "postDate": "2016-06-19T20:33:31.017Z", "content": "train.csv is too large to handle entirely in my computer, so I have split it into a sample of 500,000 random rows (I know I could split using other logic, like using only the last weeks input, for example).
\n\nMy question is, how will I fit my models if my computer can't handle the entire dataset? Is there a function in Sklearn to fit data in increments?
", "rawMarkdown": "train.csv is too large to handle entirely in my computer, so I have split it into a sample of 500,000 random rows (I know I could split using other logic, like using only the last weeks input, for example).\r\n\r\nMy question is, how will I fit my models if my computer can't handle the entire dataset? Is there a function in Sklearn to fit data in increments?" }, { "id": 125138, "postDate": "2016-06-26T17:47:06.003Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 124556, "postDate": "2016-06-19T22:49:26.240Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 124554, "postDate": "2016-06-19T22:09:15.830Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 124937, "author_name": "Kevin Kirchhoff", "author_url": "", "post_date": "2016-06-23T16:52:38.627000", "content": "You can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in read_csv(). For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.
Using dtypes is indeed a good idea. Just beware that customer id will still require 64 bits. Here is the code I use FYI:
\n\ntrain = pd.read_csv('Data/train.csv', \n dtype = {'Semana' : 'int32',\n 'Agencia_ID' :'int32',\n 'Canal_ID' : 'int32',\n 'Ruta_SAK' : 'int32',\n 'Cliente-ID' : 'int64',\n 'Producto_ID':'int32',\n 'Venta_hoy':'float32',\n 'Venta_uni_hoy': 'int32',\n 'Dev_uni_proxima':'int32',\n 'Dev_proxima':'float32',\n 'Demanda_uni_equil':'int32'})\nAs said above, you can certianly do even better by using int8 in some places instead of int32.
", "votes": 5, "replies": [] }, { "id": 124964, "author_name": "EZ", "author_url": "", "post_date": "2016-06-23T23:27:55.010000", "content": "You can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
@scirpus. I was thinking the same, libffm is next on my list.
", "votes": 1, "replies": [] }, { "id": 125443, "author_name": "EZ", "author_url": "", "post_date": "2016-06-29T13:56:48.663000", "content": "[quote=leexiongan;125441]
\n\n[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\nI can't seem it :(
\n\n[/quote]
\n\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)
", "votes": 1, "replies": [] }, { "id": 125091, "author_name": "Mattias Fagerlund", "author_url": "", "post_date": "2016-06-25T16:58:54.893000", "content": "Are you sure Cliente_Id requires 64 bits? My largest seems to be 2,015,152,015, which is smaller than largest int32 (2,147,483,647).
\n\n/mattias
", "votes": 1, "replies": [] }, { "id": 124865, "author_name": "Valentin Zambelli", "author_url": "", "post_date": "2016-06-22T20:29:14.747000", "content": "[quote=Cookie;124598]
\n\nHi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set.
\n\nAppreciate any advise offered. Thanx
\n\n[/quote]
\n\nWhat we did in another competition was calculate the relevant metric per ID (so in this case, mean returns per client id) and the substitute the id for this value. When you create dummy variables for the levels you are basically working under the assumptions that the return rate differs between clients. This is more or less a shortcut (with some information loss) but better than not including it at all.
\n\nYou should do this separately for your training and test set as to not include information you would not normally have. You also need to be able to deal with unseen client ids. we simply imputed the median in those cases.
", "votes": 1, "replies": [] }, { "id": 124557, "author_name": "Shahnawaz Akhtar", "author_url": "", "post_date": "2016-06-19T22:53:16.237000", "content": "You should try vowpal wabbit..
", "votes": 2, "replies": [] }, { "id": 124555, "author_name": "sschnug", "author_url": "", "post_date": "2016-06-19T22:37:39.100000", "content": "There are some algorithms available for incremental/online-learning in sklearn: docs.
", "votes": 2, "replies": [] }, { "id": 125482, "author_name": "Antonio Flores", "author_url": "", "post_date": "2016-06-29T21:28:58.647000", "content": "is there some "dtype" for R language? or some way to optimize factors?
", "votes": 0, "replies": [] }, { "id": 125472, "author_name": "Scirpus", "author_url": "", "post_date": "2016-06-29T19:23:05.407000", "content": "I will be surprised if the leaders aren't using LibFFM - this requires a huge amount of memory usually - idle_speculation used it brilliantly in the Expedia competition and I quote - \n"I built the model on a dual socket E5-2699v3 with ~700GB ram and was only able to use about half the bookings after '2014-07-01' due to memory limitations."
\n\nGood luck!!
", "votes": 0, "replies": [] }, { "id": 125464, "author_name": "Jialun He", "author_url": "", "post_date": "2016-06-29T17:30:00.600000", "content": "I run the bare bone xgboost script for the whole data set. The memory requirement is only 30G. So don't let memory scare you out of this competition
", "votes": 0, "replies": [] }, { "id": 125451, "author_name": "CPMP", "author_url": "", "post_date": "2016-06-29T15:28:35.703000", "content": "xgboost has an out of core option, see
\n\nhttps://xgboost.readthedocs.io/en/latest//how_to/external_memory.html
\n\nI haven't used it
", "votes": 0, "replies": [] }, { "id": 125444, "author_name": "Sangxia", "author_url": "", "post_date": "2016-06-29T14:12:13.560000", "content": "[quote=EZ;125443]
\n\n[quote=leexiongan;125441]
\n\n[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\nI can't seem it :(
\n\n[/quote]
\n\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)
\n\n[/quote]
\n\nThink you can call xgboost's train with parameter xgb_model. Does that do the job?
", "votes": 0, "replies": [] }, { "id": 125441, "author_name": "leexa", "author_url": "", "post_date": "2016-06-29T13:47:50.817000", "content": "[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\nI can't seem it :(
", "votes": 0, "replies": [] }, { "id": 125094, "author_name": "CPMP", "author_url": "", "post_date": "2016-06-25T19:29:52.870000", "content": "Mattias, you are right :)
", "votes": 0, "replies": [] }, { "id": 124966, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-23T23:30:50.850000", "content": "[quote=Kevin Kirchhoff;124937]
\n\nYou can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in read_csv(). For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.
[/quote]
\n\nGreat idea! Thank you
", "votes": 0, "replies": [] }, { "id": 124965, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-23T23:29:15.063000", "content": "[quote=EZ;124964]
\n\nYou can definitely use pandas and scikit learn or xgboost for huge data.
\n\nYou can use the chunksize argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the warmstart option in sklearn. I think xgboost will do a warmstart if you repeatedly call fit
[/quote]
\n\nThanks! Yes, I just learned this earlier in the week. It does in fact make a huge difference!
", "votes": 0, "replies": [] }, { "id": 124925, "author_name": "Cookie", "author_url": "", "post_date": "2016-06-23T14:45:37.533000", "content": "Hi Valentine. Thanks for sharing your experiences. Am working on your idea now :-)
", "votes": 0, "replies": [] }, { "id": 124598, "author_name": "Cookie", "author_url": "", "post_date": "2016-06-20T08:22:18.223000", "content": "Hi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set.
\n\nAppreciate any advise offered. Thanx
", "votes": 0, "replies": [] }, { "id": 124578, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-20T03:54:05.543000", "content": "[quote=sschnug;124555]
\n\nThere are some algorithms available for incremental/online-learning in sklearn: docs.
\n\n[/quote]
\n\nThis is very helpful, thanks!
", "votes": 0, "replies": [] }, { "id": 124577, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-20T03:53:03.803000", "content": "[quote=FredR;124554]
\n\nI don't believe it's possible. The best you could do is do some feature engineering and reduce the content of your train set.
\n\nSo maybe you only pull one or two labels and discard the rest.
\n\nI had the same question and googling left me with "calling fit multiple times will wipe out previous fit calls."
\n\nHowever you can get pretty good results without even using scikitlearn or any machine learning algorithms at all on this data set. At the moment I'm 219 out of 572 and I haven't used anything from scikit learn or xgboost.
\n\n[/quote]
\n\nThat's good to know! I'm still doing some EDA but was already having trouble with the size of this dataset.
", "votes": 0, "replies": [] }, { "id": 125138, "author_name": "", "author_url": "", "post_date": "2016-06-26T17:47:06.003000", "content": "", "votes": 2, "replies": [] }, { "id": 124556, "author_name": "", "author_url": "", "post_date": "2016-06-19T22:49:26.240000", "content": "", "votes": 2, "replies": [] }, { "id": 124554, "author_name": "", "author_url": "", "post_date": "2016-06-19T22:09:15.830000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "124937": "You can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in `read_csv()`. For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.", "125065": "Using dtypes is indeed a good idea. Just beware that customer id will still require 64 bits. Here is the code I use FYI:\r\n\r\n train = pd.read_csv('Data/train.csv', \r\n dtype = {'Semana' : 'int32',\r\n 'Agencia_ID' :'int32',\r\n 'Canal_ID' : 'int32',\r\n 'Ruta_SAK' : 'int32',\r\n 'Cliente-ID' : 'int64',\r\n 'Producto_ID':'int32',\r\n 'Venta_hoy':'float32',\r\n 'Venta_uni_hoy': 'int32',\r\n 'Dev_uni_proxima':'int32',\r\n 'Dev_proxima':'float32',\r\n 'Demanda_uni_equil':'int32'})\r\n\r\nAs said above, you can certianly do even better by using int8 in some places instead of int32.\r\n", "124964": "You can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`", "125486": "@scirpus. I was thinking the same, libffm is next on my list.", "125443": "[quote=leexiongan;125441]\r\n\r\n[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\n\r\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\r\nI can't seem it :(\r\n\r\n[/quote]\r\n\r\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)\r\n", "125091": "Are you sure Cliente_Id requires 64 bits? My largest seems to be 2,015,152,015, which is smaller than largest int32 (2,147,483,647).\r\n\r\n/mattias", "124865": "[quote=Cookie;124598]\r\n\r\nHi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set. \r\n\r\nAppreciate any advise offered. Thanx\r\n\r\n[/quote]\r\n\r\nWhat we did in another competition was calculate the relevant metric per ID (so in this case, mean returns per client id) and the substitute the id for this value. When you create dummy variables for the levels you are basically working under the assumptions that the return rate differs between clients. This is more or less a shortcut (with some information loss) but better than not including it at all.\r\n \r\nYou should do this separately for your training and test set as to not include information you would not normally have. You also need to be able to deal with unseen client ids. we simply imputed the median in those cases.", "124557": "You should try vowpal wabbit.. ", "124555": "There are some algorithms available for incremental/online-learning in sklearn: [docs][1].\r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning", "125482": "is there some \"dtype\" for R language? or some way to optimize factors?", "125472": "I will be surprised if the leaders aren't using LibFFM - this requires a huge amount of memory usually - idle_speculation used it brilliantly in the Expedia competition and I quote - \r\n\"I built the model on a dual socket E5-2699v3 with ~700GB ram and was only able to use about half the bookings after '2014-07-01' due to memory limitations.\"\r\n\r\nGood luck!! ", "125464": "I run the bare bone xgboost script for the whole data set. The memory requirement is only 30G. So don't let memory scare you out of this competition", "125451": "xgboost has an out of core option, see \r\n\r\nhttps://xgboost.readthedocs.io/en/latest//how_to/external_memory.html\r\n\r\nI haven't used it", "125444": "[quote=EZ;125443]\r\n\r\n[quote=leexiongan;125441]\r\n\r\n[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\n\r\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\r\nI can't seem it :(\r\n\r\n[/quote]\r\n\r\nNo, Sorry, I just looked again, XGBoost doesn't support that. If you need that you'll need to use an algorithm that supports partial fitting. Many of the SKLearn models do support it as does most Neural networks ;-)\r\n\r\n\r\n[/quote]\r\n\r\nThink you can call xgboost's train with parameter xgb_model. Does that do the job?", "125441": "[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\n\r\nDo you know anywhere that explains how to use xgboost in chunks or batch data?\r\nI can't seem it :(", "125094": "Mattias, you are right :)", "124966": "[quote=Kevin Kirchhoff;124937]\r\n\r\nYou can dramatically reduce the memory usage if you are using pandas dataframes by using the dtype parameter in `read_csv()`. For example you can use np.int8 for the week values (Semana), since the maximum value is 9. I believe that pandas automatically sets each data type to np.float64. All of the columns can be set to np.int32, np.float32, or smaller. If you have at leas 8gb of memory, the entire training set should fit with the proper type assignment.\r\n\r\n[/quote]\r\n\r\nGreat idea! Thank you", "124965": "[quote=EZ;124964]\r\n\r\nYou can definitely use pandas and scikit learn or xgboost for huge data.\r\n\r\nYou can use the `chunksize` argument in pandas.read_csv, it gives you back an iterator, then you can process that chunk and use the `warmstart` option in sklearn. I think xgboost will do a warmstart if you repeatedly call `fit`\r\n\r\n[/quote]\r\n\r\nThanks! Yes, I just learned this earlier in the week. It does in fact make a huge difference!", "124925": "Hi Valentine. Thanks for sharing your experiences. Am working on your idea now :-)", "124598": "Hi. First time I'm handling such a big dataset in R. There are close to 1M ClientsIDs (factors). How do you guys model that many factors? 1-hot encode of a million ClientIDs seems unbelievable... I may downsample the train set but I definitely need all from the test set. \r\n\r\nAppreciate any advise offered. Thanx", "124578": "[quote=sschnug;124555]\r\n\r\nThere are some algorithms available for incremental/online-learning in sklearn: [docs][1].\r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning\r\n\r\n[/quote]\r\n\r\nThis is very helpful, thanks!", "124577": "[quote=FredR;124554]\r\n\r\nI don't believe it's possible. The best you could do is do some feature engineering and reduce the content of your train set. \r\n\r\nSo maybe you only pull one or two labels and discard the rest.\r\n\r\nI had the same question and googling left me with \"calling fit multiple times will wipe out previous fit calls.\"\r\n\r\n\r\nHowever you can get pretty good results without even using scikitlearn or any machine learning algorithms at all on this data set. At the moment I'm 219 out of 572 and I haven't used anything from scikit learn or xgboost.\r\n\r\n[/quote]\r\n\r\nThat's good to know! I'm still doing some EDA but was already having trouble with the size of this dataset.", "124544": "train.csv is too large to handle entirely in my computer, so I have split it into a sample of 500,000 random rows (I know I could split using other logic, like using only the last weeks input, for example).\r\n\r\nMy question is, how will I fit my models if my computer can't handle the entire dataset? Is there a function in Sklearn to fit data in increments?", "125138": "", "124556": "", "124554": "" } }