![\"Takes](\"https://pbs.twimg.com/media/DMxThzDW0AItRpX.jpg:large\" "\\"\\"")
So, simply preserving pandas train df in RAM consumes around 10GB. Even 128 GB may be not comfortable enough for this competition. So what are the best practicies to deal with such big dataset.
\n\nIs it a valid solution for xgboost to use batch generator? So split train data into batches (like you do in DL) and train xgboost on each batch, using weight from the previous batch.
\n\nWhat else?
\n\nUPDATE. Thanks for the comments! I've added kernel for reading data with low memory consumption https://www.kaggle.com/heyt0ny/read-data-for-low-memory-usage (python.)
", "messages": [ { "id": "233497", "postDate": "10/20/2017 11:58:30", "content": "So, simply preserving pandas train df in RAM consumes around 10GB. Even 128 GB may be not comfortable enough for this competition. So what are the best practicies to deal with such big dataset.
\n\nIs it a valid solution for xgboost to use batch generator? So split train data into batches (like you do in DL) and train xgboost on each batch, using weight from the previous batch.
\n\nWhat else?
\n\nUPDATE. Thanks for the comments! I've added kernel for reading data with low memory consumption https://www.kaggle.com/heyt0ny/read-data-for-low-memory-usage (python.)
", "rawMarkdown": "So, simply preserving pandas train df in RAM consumes around 10GB. Even 128 GB may be not comfortable enough for this competition. So what are the best practicies to deal with such big dataset. \n\nIs it a valid solution for xgboost to use batch generator? So split train data into batches (like you do in DL) and train xgboost on each batch, using weight from the previous batch. \n\nWhat else?\n\nUPDATE. Thanks for the comments! I've added kernel for reading data with low memory consumption https://www.kaggle.com/heyt0ny/read-data-for-low-memory-usage (python.)", "votes": null }, { "id": "233519", "postDate": "10/20/2017 12:52:57", "content": "A \"get started quickly\" solution would be to only use, e.g., the previous 2 months of training data, instead of the 55 months provided.
", "rawMarkdown": "A \"get started quickly\" solution would be to only use, e.g., the previous 2 months of training data, instead of the 55 months provided.", "votes": null }, { "id": "233567", "postDate": "10/20/2017 15:01:23", "content": "In R data.table seems rather efficient so far for processing - I produced both btb scripts on a laptop with 16gb ram...
", "rawMarkdown": "In R data.table seems rather efficient so far for processing - I produced both btb scripts on a laptop with 16gb ram...", "votes": null }, { "id": "233621", "postDate": "10/20/2017 17:56:35", "content": "You could use cloud services for running models if your personal workstation is not able to handle data.
", "rawMarkdown": "You could use cloud services for running models if your personal workstation is not able to handle data.", "votes": null }, { "id": "233648", "postDate": "10/20/2017 19:07:35", "content": "@feels_g00d_man i am using python!!(i5 with 16GB RAM) Initial Loading of train.csv takes 4.5+GB. i just changed the datatypes(float64 and int64) to {'id':np.int32,'store_nbr':np.int8, 'item_nbr':np.int32, 'unit_sales':np.float32} this will keep the data unchanged.\nThis brings down the size to 3.1+GB . Now just change the onpromotion column NaN values to any integer and map True and False to [1,0] and then convert to int8. this should bring the data to 2.6GB.\nHope this helps!.
", "rawMarkdown": "feels_g00d_man i am using python!!(i5 with 16GB RAM) Initial Loading of train.csv takes **4.5+GB**. i just changed the datatypes(float64 and int64) to {'id':np.int32,'store_nbr':np.int8, 'item_nbr':np.int32, 'unit_sales':np.float32} this will keep the data unchanged.\nThis brings down the size to **3.1+GB** . Now just change the onpromotion column NaN values to any integer and map True and False to [1,0] and then convert to int8. this should bring the data to **2.6GB**.\nHope this helps!.", "votes": null }, { "id": "233732", "postDate": "10/21/2017 01:03:35", "content": "Also using pandas DataFrame. It does take a while! Does anyone have a good feel for which type of solution is more robust /\"the norm\" (breaking into batches vs. casting data into different data types)?
", "rawMarkdown": "Also using pandas DataFrame. It does take a while! Does anyone have a good feel for which type of solution is more robust /\"the norm\" (breaking into batches vs. casting data into different data types)?", "votes": null }, { "id": "233736", "postDate": "10/21/2017 01:10:52", "content": "Try this and see how it works for you:
\n\ndef loaddata(filename, nrows=None):\n\n types = {'id': 'int64',\n 'item_nbr': 'int32',\n 'store_nbr': 'int16',\n 'unit_sales': 'float32',\n 'onpromotion': bool,\n }\n\n data = pd.read_csv(DATADIR + filename, parse_dates=['date'], \n dtype=types, nrows=nrows, infer_datetime_format=True)\n\n return data\nYou can load data with numpy also, that will be faster but then you wont have as much flexibility in operations as pandas provide. i suggest you load a sample of data in pandas using {'skiprows' = np.random.randint(range of index,size= data you dont want)}, a simple hack .
", "rawMarkdown": "You can load data with numpy also, that will be faster but then you wont have as much flexibility in operations as pandas provide. i suggest you load a sample of data in pandas using {'skiprows' = np.random.randint(range of index,size= data you dont want)}, a simple hack .", "votes": null }, { "id": "233786", "postDate": "10/21/2017 08:09:33", "content": "Yup. Quick note, id can be an int32 (i think max value is in 120 mil) and onpromotion can be made into int8 afters nans are replaced with -1. Total footprint with all of the data and meta data loaded was 2.8 GB.
\n\nedit: replacing timestamps with int16 that is days from 2013-01-01 shaves things down to 2.1 GB.
", "rawMarkdown": "Yup. Quick note, id can be an int32 (i think max value is in 120 mil) and onpromotion can be made into int8 afters nans are replaced with -1. Total footprint with all of the data and meta data loaded was 2.8 GB.\n\nedit: replacing timestamps with int16 that is days from 2013-01-01 shaves things down to 2.1 GB.", "votes": null }, { "id": "234503", "postDate": "10/23/2017 13:58:27", "content": "Sorry. But i dont understand, how you download data? I try split on path 1 million row, and unload to pandas but my comp (4 gb ddr) delay many time, about 3-4 minuts. How you change data types, when you dont download data?
", "rawMarkdown": "Sorry. But i dont understand, how you download data? I try split on path 1 million row, and unload to pandas but my comp (4 gb ddr) delay many time, about 3-4 minuts. How you change data types, when you dont download data?", "votes": null }, { "id": "234508", "postDate": "10/23/2017 14:04:22", "content": "I have a few questions here:
\n\nThe last question is most important for me since without the training set processed, there is no way I can build a model.
\n\nThanks,\nShivendra
", "rawMarkdown": "I have a few questions here:\n\n 1. How does R perform compared to pandas for the whole 4.7 GB file?\n 2. Taking into consideration the 70:30 standard rule where 70% needs to be the size of the training set and the rest as the test set, the training.csv file suggests that almost 97% of the data has been assigned to the training data here. Is this acceptable (the total size adding in the test set comes to be around 4933 MB or approx. 5 GB)?\n 3. Has anyone been able to process the training set on a MacBook with 8GB memory? I use a similar system and it felt like it would crash my laptop (using R 3.4.1).\n\nThe last question is most important for me since without the training set processed, there is no way I can build a model.\n\nThanks,\nShivendra", "votes": null }, { "id": "234534", "postDate": "10/23/2017 15:00:14", "content": "4GB Ram is probably not enough. I would suggest using the pandas nrows parameter in read_csv and see how big you can make it before you have trouble. Start with nrows=100000 and see. Once you have found a good size, you can use Pandas' chunk_size option to read a file bit by bit: (http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking)
2: The file named test.csv doesn't contain your validation data, it contains data without the number you have to predict. So you have to use the train.csv file and split that into your own train/test segments.
3: 8GB is probably too little for the entire dataset, you may have to process it it bit by bit, see my answer
", "rawMarkdown": "**2:** The file named `test.csv` doesn't contain your validation data, it contains data without the number you have to predict. So you have to use the `train.csv` file and split that into your own train/test segments.\n\n**3:** 8GB is probably too little for the entire dataset, you may have to process it it bit by bit, see [my answer](#234534)", "votes": null }, { "id": "234583", "postDate": "10/23/2017 16:22:02", "content": "Thanks @EZ. 8GB is certainly a small figure for computing now a days... Time to buy some memory.
\n\nEdit: Just rechecked, my MBP is 4GB and not 8GB.
", "rawMarkdown": "Thanks @EZ. 8GB is certainly a small figure for computing now a days... Time to buy some memory.\n\nEdit: Just rechecked, my MBP is 4GB and not 8GB.", "votes": null }, { "id": "234643", "postDate": "10/23/2017 18:02:23", "content": "True! R data.table is really a good option for such a large data set.
128 GB is definitely an overkill.
\n\nIt took me 1 minute to process the train.csv file with fread in R, and no time for other files. Without any additional calculations/training, and all files loaded, my RAM consumption is currently at ~8 GB.
\n\n16 GB should be enough for this competition, with good feature engineering I'm sure you don't even need to train on the whole train set. Personally, whenever I'm in trouble I just prefer to use virtual machines, the basic ones are quite cheap, you get free trial and most important for me, I can still run things normally on my laptop without having to train models overnight and stuff.
\n\nI'd recommend you to try LightGBM for training, it is several times faster than XGBoost and frequently provides similar, if not even better, results.
", "rawMarkdown": "128 GB is definitely an overkill.\n\nIt took me 1 minute to process the train.csv file with fread in R, and no time for other files. Without any additional calculations/training, and all files loaded, my RAM consumption is currently at ~8 GB. \n\n16 GB should be enough for this competition, with good feature engineering I'm sure you don't even need to train on the whole train set. Personally, whenever I'm in trouble I just prefer to use virtual machines, the basic ones are quite cheap, you get free trial and most important for me, I can still run things normally on my laptop without having to train models overnight and stuff.\n\nI'd recommend you to try LightGBM for training, it is several times faster than XGBoost and frequently provides similar, if not even better, results.", "votes": null }, { "id": "234696", "postDate": "10/23/2017 21:17:34", "content": "Thanks for the comment Aljaž.
\n\nI remember recent Instacart challenge with similar data scheme. The train.csv was around 500MB. But after some feature engineering, If I recall correctly - even 100 features were not fitting in 32gb ram. The guy from 2d place was using 300gb ram machine ))) So If train on whole dataset it may be that 128 gb is not an overkill.
\n\nI found it's harder to finetune lgb than xgb and also I'm using xgbfir - which is not available on lgb.
", "rawMarkdown": "Thanks for the comment Aljaž.\n\nI remember recent Instacart challenge with similar data scheme. The train.csv was around 500MB. But after some feature engineering, If I recall correctly - even 100 features were not fitting in 32gb ram. The guy from 2d place was using 300gb ram machine ))) So If train on whole dataset it may be that 128 gb is not an overkill.\n\nI found it's harder to finetune lgb than xgb and also I'm using xgbfir - which is not available on lgb.", "votes": null }, { "id": "234760", "postDate": "10/24/2017 01:14:53", "content": "You probably don't need all the data for training purpose. Some of the historical data may not provide much value - for the trade-off of model training performance vs speed.
", "rawMarkdown": "You probably don't need all the data for training purpose. Some of the historical data may not provide much value - for the trade-off of model training performance vs speed.", "votes": null }, { "id": "234799", "postDate": "10/24/2017 04:55:57", "content": "To be very honest, I’m not even sure if my system can support 300GB. It would be a good amount though, since that would allow me to run models on any dataset I deem worthwhile.
", "rawMarkdown": "To be very honest, I’m not even sure if my system can support 300GB. It would be a good amount though, since that would allow me to run models on any dataset I deem worthwhile.", "votes": null }, { "id": "234847", "postDate": "10/24/2017 07:33:06", "content": "feels_g00d_man, in Instacart I was able to squeeze ~65 features in 16 GB and then ~100 in 32 GB (can't remember if I didn't have more simply because they were inefficient or because they didn't fit).
\n\nI remember the guy, but he was rather an exception. However, it is true that I'm not really aiming for the gold, but silver should be more than accessible with 16 GB ;-)
", "rawMarkdown": "feels_g00d_man, in Instacart I was able to squeeze ~65 features in 16 GB and then ~100 in 32 GB (can't remember if I didn't have more simply because they were inefficient or because they didn't fit).\n\nI remember the guy, but he was rather an exception. However, it is true that I'm not really aiming for the gold, but silver should be more than accessible with 16 GB ;-)", "votes": null }, { "id": "234892", "postDate": "10/24/2017 09:09:55", "content": "Thats an impressive speed. Faster than read_csv() offered by readr.
I've made a simple map-reduce implementation to process this dataset in batches. It allows to filter or aggregate the dataset without memory overhead. You can find implementation with comments and the example of usage in my kernel https://www.kaggle.com/tsdaemon/mapreduce-for-batch-processing-of-a-big-dataset.
", "rawMarkdown": "I've made a simple map-reduce implementation to process this dataset in batches. It allows to filter or aggregate the dataset without memory overhead. You can find implementation with comments and the example of usage in my kernel https://www.kaggle.com/tsdaemon/mapreduce-for-batch-processing-of-a-big-dataset.", "votes": null }, { "id": "237759", "postDate": "10/30/2017 21:14:39", "content": "One alternative that I would prefer is to load my data into a local SQL Workbench and then query it by filtering a part of the data instead of loading the whole data into a dataframe (as that would consume less memory compared to latter) and then perform analysis. Not really sure if this would work but I'm definitely going to try it as I'm working on a 8 GB laptop.
", "rawMarkdown": "One alternative that I would prefer is to load my data into a local SQL Workbench and then query it by filtering a part of the data instead of loading the whole data into a dataframe (as that would consume less memory compared to latter) and then perform analysis. Not really sure if this would work but I'm definitely going to try it as I'm working on a 8 GB laptop.", "votes": null }, { "id": "238559", "postDate": "11/01/2017 15:19:18", "content": "Thanks! This helps me to reduce about 1/3 of data size
", "rawMarkdown": "Thanks! This helps me to reduce about 1/3 of data size", "votes": null }, { "id": "246019", "postDate": "11/20/2017 09:54:45", "content": "Hi . I have tried reading in train.csv with fread ...but unable to read in more than 3x 10^7 rows at once ...as it gives the error \"Reached total allocation of 3995Mb\".... any pointers ...so currently I am reading in the file in chunks ..
", "rawMarkdown": "Hi . I have tried reading in train.csv with fread ...but unable to read in more than 3x 10^7 rows at once ...as it gives the error \"Reached total allocation of 3995Mb\".... any pointers ...so currently I am reading in the file in chunks ..", "votes": null }, { "id": "252078", "postDate": "12/02/2017 07:28:42", "content": "Aljaž, how many memory do you have on your computer? I think 16GB is probably enough, but when I am joining the table in pandas, the usage of memory spike up and I get memory error. Any idea how to solve this problem? It seems that pandas need extra memory to do the join.
", "rawMarkdown": "Aljaž, how many memory do you have on your computer? I think 16GB is probably enough, but when I am joining the table in pandas, the usage of memory spike up and I get memory error. Any idea how to solve this problem? It seems that pandas need extra memory to do the join.", "votes": null }, { "id": "256893", "postDate": "12/12/2017 22:12:14", "content": "SQL worked for us actually in terms of initially data management but then integrating it with python and uploading to Kaggle we ran into problems. Actually is there a way to have Kernel's reference other submitted kernels (like ones that do intermediate steps?)
", "rawMarkdown": "SQL worked for us actually in terms of initially data management but then integrating it with python and uploading to Kaggle we ran into problems. Actually is there a way to have Kernel's reference other submitted kernels (like ones that do intermediate steps?)", "votes": null }, { "id": "256894", "postDate": "12/12/2017 22:14:29", "content": "Have you found this to be still true? Recently my team has been questioning this assumption, so I was hoping for some clarification.
", "rawMarkdown": "Have you found this to be still true? Recently my team has been questioning this assumption, so I was hoping for some clarification.", "votes": null }, { "id": "256897", "postDate": "12/12/2017 22:17:29", "content": "I did not use kernal to run my script...ran it locally instead...that would be a better option
", "rawMarkdown": "I did not use kernal to run my script...ran it locally instead...that would be a better option", "votes": null }, { "id": "257509", "postDate": "12/14/2017 11:30:46", "content": "Is this doable in R at all? I tried h2o & 16GB RAM (with memory.limit() = 100,000) and h2o?
", "rawMarkdown": "Is this doable in R at all? I tried h2o & 16GB RAM (with memory.limit() = 100,000) and h2o?", "votes": null } ], "comments": [ { "id": 233519, "author_name": "inversion", "author_url": "", "post_date": "10/20/2017 12:52:57", "content": "A \"get started quickly\" solution would be to only use, e.g., the previous 2 months of training data, instead of the 55 months provided.
", "votes": null, "replies": [] }, { "id": 233567, "author_name": "konradb", "author_url": "", "post_date": "10/20/2017 15:01:23", "content": "In R data.table seems rather efficient so far for processing - I produced both btb scripts on a laptop with 16gb ram...
", "votes": null, "replies": [ { "id": 234643, "author_name": "pranav84", "author_url": "", "post_date": "10/23/2017 18:02:23", "content": "True! R data.table is really a good option for such a large data set.
Thats an impressive speed. Faster than read_csv() offered by readr.
Hi . I have tried reading in train.csv with fread ...but unable to read in more than 3x 10^7 rows at once ...as it gives the error \"Reached total allocation of 3995Mb\".... any pointers ...so currently I am reading in the file in chunks ..
", "votes": null, "replies": [] } ] }, { "id": 233621, "author_name": "miteshyadav", "author_url": "", "post_date": "10/20/2017 17:56:35", "content": "You could use cloud services for running models if your personal workstation is not able to handle data.
", "votes": null, "replies": [] }, { "id": 233648, "author_name": "trekkerthemaker", "author_url": "", "post_date": "10/20/2017 19:07:35", "content": "@feels_g00d_man i am using python!!(i5 with 16GB RAM) Initial Loading of train.csv takes 4.5+GB. i just changed the datatypes(float64 and int64) to {'id':np.int32,'store_nbr':np.int8, 'item_nbr':np.int32, 'unit_sales':np.float32} this will keep the data unchanged.\nThis brings down the size to 3.1+GB . Now just change the onpromotion column NaN values to any integer and map True and False to [1,0] and then convert to int8. this should bring the data to 2.6GB.\nHope this helps!.
", "votes": null, "replies": [] }, { "id": 233732, "author_name": "lthorne", "author_url": "", "post_date": "10/21/2017 01:03:35", "content": "Also using pandas DataFrame. It does take a while! Does anyone have a good feel for which type of solution is more robust /\"the norm\" (breaking into batches vs. casting data into different data types)?
", "votes": null, "replies": [ { "id": 233776, "author_name": "trekkerthemaker", "author_url": "", "post_date": "10/21/2017 07:11:12", "content": "You can load data with numpy also, that will be faster but then you wont have as much flexibility in operations as pandas provide. i suggest you load a sample of data in pandas using {'skiprows' = np.random.randint(range of index,size= data you dont want)}, a simple hack .
", "votes": null, "replies": [] } ] }, { "id": 233736, "author_name": "ezietsman", "author_url": "", "post_date": "10/21/2017 01:10:52", "content": "Try this and see how it works for you:
\n\ndef loaddata(filename, nrows=None):\n\n types = {'id': 'int64',\n 'item_nbr': 'int32',\n 'store_nbr': 'int16',\n 'unit_sales': 'float32',\n 'onpromotion': bool,\n }\n\n data = pd.read_csv(DATADIR + filename, parse_dates=['date'], \n dtype=types, nrows=nrows, infer_datetime_format=True)\n\n return data\nYup. Quick note, id can be an int32 (i think max value is in 120 mil) and onpromotion can be made into int8 afters nans are replaced with -1. Total footprint with all of the data and meta data loaded was 2.8 GB.
\n\nedit: replacing timestamps with int16 that is days from 2013-01-01 shaves things down to 2.1 GB.
", "votes": null, "replies": [] }, { "id": 238559, "author_name": "gaogao", "author_url": "", "post_date": "11/01/2017 15:19:18", "content": "Thanks! This helps me to reduce about 1/3 of data size
", "votes": null, "replies": [] } ] }, { "id": 234503, "author_name": "koklushkin", "author_url": "", "post_date": "10/23/2017 13:58:27", "content": "Sorry. But i dont understand, how you download data? I try split on path 1 million row, and unload to pandas but my comp (4 gb ddr) delay many time, about 3-4 minuts. How you change data types, when you dont download data?
", "votes": null, "replies": [ { "id": 234534, "author_name": "ezietsman", "author_url": "", "post_date": "10/23/2017 15:00:14", "content": "4GB Ram is probably not enough. I would suggest using the pandas nrows parameter in read_csv and see how big you can make it before you have trouble. Start with nrows=100000 and see. Once you have found a good size, you can use Pandas' chunk_size option to read a file bit by bit: (http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking)
I have a few questions here:
\n\nThe last question is most important for me since without the training set processed, there is no way I can build a model.
\n\nThanks,\nShivendra
", "votes": null, "replies": [ { "id": 234538, "author_name": "ezietsman", "author_url": "", "post_date": "10/23/2017 15:03:38", "content": "2: The file named test.csv doesn't contain your validation data, it contains data without the number you have to predict. So you have to use the train.csv file and split that into your own train/test segments.
3: 8GB is probably too little for the entire dataset, you may have to process it it bit by bit, see my answer
", "votes": null, "replies": [] }, { "id": 234583, "author_name": "shivendra90", "author_url": "", "post_date": "10/23/2017 16:22:02", "content": "Thanks @EZ. 8GB is certainly a small figure for computing now a days... Time to buy some memory.
\n\nEdit: Just rechecked, my MBP is 4GB and not 8GB.
", "votes": null, "replies": [] } ] }, { "id": 234660, "author_name": "aljaz91", "author_url": "", "post_date": "10/23/2017 19:33:48", "content": "128 GB is definitely an overkill.
\n\nIt took me 1 minute to process the train.csv file with fread in R, and no time for other files. Without any additional calculations/training, and all files loaded, my RAM consumption is currently at ~8 GB.
\n\n16 GB should be enough for this competition, with good feature engineering I'm sure you don't even need to train on the whole train set. Personally, whenever I'm in trouble I just prefer to use virtual machines, the basic ones are quite cheap, you get free trial and most important for me, I can still run things normally on my laptop without having to train models overnight and stuff.
\n\nI'd recommend you to try LightGBM for training, it is several times faster than XGBoost and frequently provides similar, if not even better, results.
", "votes": null, "replies": [ { "id": 234696, "author_name": "heyt0ny", "author_url": "", "post_date": "10/23/2017 21:17:34", "content": "Thanks for the comment Aljaž.
\n\nI remember recent Instacart challenge with similar data scheme. The train.csv was around 500MB. But after some feature engineering, If I recall correctly - even 100 features were not fitting in 32gb ram. The guy from 2d place was using 300gb ram machine ))) So If train on whole dataset it may be that 128 gb is not an overkill.
\n\nI found it's harder to finetune lgb than xgb and also I'm using xgbfir - which is not available on lgb.
", "votes": null, "replies": [] }, { "id": 234799, "author_name": "shivendra90", "author_url": "", "post_date": "10/24/2017 04:55:57", "content": "To be very honest, I’m not even sure if my system can support 300GB. It would be a good amount though, since that would allow me to run models on any dataset I deem worthwhile.
", "votes": null, "replies": [] }, { "id": 234847, "author_name": "aljaz91", "author_url": "", "post_date": "10/24/2017 07:33:06", "content": "feels_g00d_man, in Instacart I was able to squeeze ~65 features in 16 GB and then ~100 in 32 GB (can't remember if I didn't have more simply because they were inefficient or because they didn't fit).
\n\nI remember the guy, but he was rather an exception. However, it is true that I'm not really aiming for the gold, but silver should be more than accessible with 16 GB ;-)
", "votes": null, "replies": [] }, { "id": 252078, "author_name": "noklamchan", "author_url": "", "post_date": "12/02/2017 07:28:42", "content": "Aljaž, how many memory do you have on your computer? I think 16GB is probably enough, but when I am joining the table in pandas, the usage of memory spike up and I get memory error. Any idea how to solve this problem? It seems that pandas need extra memory to do the join.
", "votes": null, "replies": [] } ] }, { "id": 234760, "author_name": "dongxu027", "author_url": "", "post_date": "10/24/2017 01:14:53", "content": "You probably don't need all the data for training purpose. Some of the historical data may not provide much value - for the trade-off of model training performance vs speed.
", "votes": null, "replies": [ { "id": 256894, "author_name": "skyrmion", "author_url": "", "post_date": "12/12/2017 22:14:29", "content": "Have you found this to be still true? Recently my team has been questioning this assumption, so I was hoping for some clarification.
", "votes": null, "replies": [] } ] }, { "id": 235420, "author_name": "tsdaemon", "author_url": "", "post_date": "10/25/2017 11:11:26", "content": "I've made a simple map-reduce implementation to process this dataset in batches. It allows to filter or aggregate the dataset without memory overhead. You can find implementation with comments and the example of usage in my kernel https://www.kaggle.com/tsdaemon/mapreduce-for-batch-processing-of-a-big-dataset.
", "votes": null, "replies": [] }, { "id": 237759, "author_name": "miteshyadav", "author_url": "", "post_date": "10/30/2017 21:14:39", "content": "One alternative that I would prefer is to load my data into a local SQL Workbench and then query it by filtering a part of the data instead of loading the whole data into a dataframe (as that would consume less memory compared to latter) and then perform analysis. Not really sure if this would work but I'm definitely going to try it as I'm working on a 8 GB laptop.
", "votes": null, "replies": [ { "id": 256893, "author_name": "skyrmion", "author_url": "", "post_date": "12/12/2017 22:12:14", "content": "SQL worked for us actually in terms of initially data management but then integrating it with python and uploading to Kaggle we ran into problems. Actually is there a way to have Kernel's reference other submitted kernels (like ones that do intermediate steps?)
", "votes": null, "replies": [] }, { "id": 256897, "author_name": "miteshyadav", "author_url": "", "post_date": "12/12/2017 22:17:29", "content": "I did not use kernal to run my script...ran it locally instead...that would be a better option
", "votes": null, "replies": [] } ] }, { "id": 257509, "author_name": "parijatkumar2003", "author_url": "", "post_date": "12/14/2017 11:30:46", "content": "Is this doable in R at all? I tried h2o & 16GB RAM (with memory.limit() = 100,000) and h2o?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "233497": "So, simply preserving pandas train df in RAM consumes around 10GB. Even 128 GB may be not comfortable enough for this competition. So what are the best practicies to deal with such big dataset. \n\nIs it a valid solution for xgboost to use batch generator? So split train data into batches (like you do in DL) and train xgboost on each batch, using weight from the previous batch. \n\nWhat else?\n\nUPDATE. Thanks for the comments! I've added kernel for reading data with low memory consumption https://www.kaggle.com/heyt0ny/read-data-for-low-memory-usage (python.)", "233519": "A \"get started quickly\" solution would be to only use, e.g., the previous 2 months of training data, instead of the 55 months provided.", "233567": "In R data.table seems rather efficient so far for processing - I produced both btb scripts on a laptop with 16gb ram...", "233621": "You could use cloud services for running models if your personal workstation is not able to handle data.", "233648": "feels_g00d_man i am using python!!(i5 with 16GB RAM) Initial Loading of train.csv takes **4.5+GB**. i just changed the datatypes(float64 and int64) to {'id':np.int32,'store_nbr':np.int8, 'item_nbr':np.int32, 'unit_sales':np.float32} this will keep the data unchanged.\nThis brings down the size to **3.1+GB** . Now just change the onpromotion column NaN values to any integer and map True and False to [1,0] and then convert to int8. this should bring the data to **2.6GB**.\nHope this helps!.", "233732": "Also using pandas DataFrame. It does take a while! Does anyone have a good feel for which type of solution is more robust /\"the norm\" (breaking into batches vs. casting data into different data types)?", "233736": "Try this and see how it works for you:\n\n\n def loaddata(filename, nrows=None):\n \n types = {'id': 'int64',\n 'item_nbr': 'int32',\n 'store_nbr': 'int16',\n 'unit_sales': 'float32',\n 'onpromotion': bool,\n }\n \n data = pd.read_csv(DATADIR + filename, parse_dates=['date'], \n dtype=types, nrows=nrows, infer_datetime_format=True)\n \n return data", "233776": "You can load data with numpy also, that will be faster but then you wont have as much flexibility in operations as pandas provide. i suggest you load a sample of data in pandas using {'skiprows' = np.random.randint(range of index,size= data you dont want)}, a simple hack .", "233786": "Yup. Quick note, id can be an int32 (i think max value is in 120 mil) and onpromotion can be made into int8 afters nans are replaced with -1. Total footprint with all of the data and meta data loaded was 2.8 GB.\n\nedit: replacing timestamps with int16 that is days from 2013-01-01 shaves things down to 2.1 GB.", "234503": "Sorry. But i dont understand, how you download data? I try split on path 1 million row, and unload to pandas but my comp (4 gb ddr) delay many time, about 3-4 minuts. How you change data types, when you dont download data?", "234508": "I have a few questions here:\n\n 1. How does R perform compared to pandas for the whole 4.7 GB file?\n 2. Taking into consideration the 70:30 standard rule where 70% needs to be the size of the training set and the rest as the test set, the training.csv file suggests that almost 97% of the data has been assigned to the training data here. Is this acceptable (the total size adding in the test set comes to be around 4933 MB or approx. 5 GB)?\n 3. Has anyone been able to process the training set on a MacBook with 8GB memory? I use a similar system and it felt like it would crash my laptop (using R 3.4.1).\n\nThe last question is most important for me since without the training set processed, there is no way I can build a model.\n\nThanks,\nShivendra", "234534": "4GB Ram is probably not enough. I would suggest using the pandas `nrows` parameter in `read_csv` and see how big you can make it before you have trouble. Start with `nrows=100000` and see. Once you have found a good size, you can use Pandas' chunk_size option to read a file bit by bit: (http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking)", "234538": "**2:** The file named `test.csv` doesn't contain your validation data, it contains data without the number you have to predict. So you have to use the `train.csv` file and split that into your own train/test segments.\n\n**3:** 8GB is probably too little for the entire dataset, you may have to process it it bit by bit, see [my answer](#234534)", "234583": "Thanks @EZ. 8GB is certainly a small figure for computing now a days... Time to buy some memory.\n\nEdit: Just rechecked, my MBP is 4GB and not 8GB.", "234643": "True! R data.table is really a good option for such a large data set. ![Takes only 54 secs to read 4.6 GB train.csv file][1]\n\n\n [1]: https://pbs.twimg.com/media/DMxThzDW0AItRpX.jpg:large", "234660": "128 GB is definitely an overkill.\n\nIt took me 1 minute to process the train.csv file with fread in R, and no time for other files. Without any additional calculations/training, and all files loaded, my RAM consumption is currently at ~8 GB. \n\n16 GB should be enough for this competition, with good feature engineering I'm sure you don't even need to train on the whole train set. Personally, whenever I'm in trouble I just prefer to use virtual machines, the basic ones are quite cheap, you get free trial and most important for me, I can still run things normally on my laptop without having to train models overnight and stuff.\n\nI'd recommend you to try LightGBM for training, it is several times faster than XGBoost and frequently provides similar, if not even better, results.", "234696": "Thanks for the comment Aljaž.\n\nI remember recent Instacart challenge with similar data scheme. The train.csv was around 500MB. But after some feature engineering, If I recall correctly - even 100 features were not fitting in 32gb ram. The guy from 2d place was using 300gb ram machine ))) So If train on whole dataset it may be that 128 gb is not an overkill.\n\nI found it's harder to finetune lgb than xgb and also I'm using xgbfir - which is not available on lgb.", "234760": "You probably don't need all the data for training purpose. Some of the historical data may not provide much value - for the trade-off of model training performance vs speed.", "234799": "To be very honest, I’m not even sure if my system can support 300GB. It would be a good amount though, since that would allow me to run models on any dataset I deem worthwhile.", "234847": "feels_g00d_man, in Instacart I was able to squeeze ~65 features in 16 GB and then ~100 in 32 GB (can't remember if I didn't have more simply because they were inefficient or because they didn't fit).\n\nI remember the guy, but he was rather an exception. However, it is true that I'm not really aiming for the gold, but silver should be more than accessible with 16 GB ;-)", "234892": "Thats an impressive speed. Faster than `read_csv()` offered by `readr`.", "235420": "I've made a simple map-reduce implementation to process this dataset in batches. It allows to filter or aggregate the dataset without memory overhead. You can find implementation with comments and the example of usage in my kernel https://www.kaggle.com/tsdaemon/mapreduce-for-batch-processing-of-a-big-dataset.", "237759": "One alternative that I would prefer is to load my data into a local SQL Workbench and then query it by filtering a part of the data instead of loading the whole data into a dataframe (as that would consume less memory compared to latter) and then perform analysis. Not really sure if this would work but I'm definitely going to try it as I'm working on a 8 GB laptop.", "238559": "Thanks! This helps me to reduce about 1/3 of data size", "246019": "Hi . I have tried reading in train.csv with fread ...but unable to read in more than 3x 10^7 rows at once ...as it gives the error \"Reached total allocation of 3995Mb\".... any pointers ...so currently I am reading in the file in chunks ..", "252078": "Aljaž, how many memory do you have on your computer? I think 16GB is probably enough, but when I am joining the table in pandas, the usage of memory spike up and I get memory error. Any idea how to solve this problem? It seems that pandas need extra memory to do the join.", "256893": "SQL worked for us actually in terms of initially data management but then integrating it with python and uploading to Kaggle we ran into problems. Actually is there a way to have Kernel's reference other submitted kernels (like ones that do intermediate steps?)", "256894": "Have you found this to be still true? Recently my team has been questioning this assumption, so I was hoping for some clarification.", "256897": "I did not use kernal to run my script...ran it locally instead...that would be a better option", "257509": "Is this doable in R at all? I tried h2o & 16GB RAM (with memory.limit() = 100,000) and h2o?" }, "source": "meta" }