Hi,\nI am not able to read CSVs in new dataset v2. The kernel seem to attempt to read it. It maximises available RAM, after some time it dies and restarts.\nAlso, previous version train.csv and test.csv were 1.5 GB and 1.3 GB, respectively. However, version 2 dataset seems quite large (train_v2.csv and test_v2.csv are 23.67 GB and 7.09 GB, respectively ). \nIt is quite something to think about as with available RAM being around 16 GB, reading both CSVs at a time doesn't seem feasible...\nAny solutions/suggestions ???
", "messages": [ { "id": "418272", "postDate": "11/09/2018 15:20:13", "content": "Hi,\nI am not able to read CSVs in new dataset v2. The kernel seem to attempt to read it. It maximises available RAM, after some time it dies and restarts.\nAlso, previous version train.csv and test.csv were 1.5 GB and 1.3 GB, respectively. However, version 2 dataset seems quite large (train_v2.csv and test_v2.csv are 23.67 GB and 7.09 GB, respectively ). \nIt is quite something to think about as with available RAM being around 16 GB, reading both CSVs at a time doesn't seem feasible...\nAny solutions/suggestions ???
", "rawMarkdown": "Hi,\nI am not able to read CSVs in new dataset v2. The kernel seem to attempt to read it. It maximises available RAM, after some time it dies and restarts.\nAlso, previous version train.csv and test.csv were 1.5 GB and 1.3 GB, respectively. However, version 2 dataset seems quite large (train_v2.csv and test_v2.csv are 23.67 GB and 7.09 GB, respectively ). \nIt is quite something to think about as with available RAM being around 16 GB, reading both CSVs at a time doesn't seem feasible...\nAny solutions/suggestions ???", "votes": null }, { "id": "418276", "postDate": "11/09/2018 15:27:01", "content": "Hello Pratik,
\n\nI raised the issue here : https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71042\nI'm still working on it but as Vincent suggest in the thread, if you don't read the \"hit\" column you can save 20gb... as Vincent says, it must be a critical feature... I think we need to think about what columns are really needed and drop the only NaNs ones and do some optimization on the dataframe (changing column type from float 64 to 32 etc...)
", "rawMarkdown": "Hello Pratik,\n\nI raised the issue here : https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71042\nI'm still working on it but as Vincent suggest in the thread, if you don't read the \"hit\" column you can save 20gb... as Vincent says, it must be a critical feature... I think we need to think about what columns are really needed and drop the only NaNs ones and do some optimization on the dataframe (changing column type from float 64 to 32 etc...)", "votes": null }, { "id": "418283", "postDate": "11/09/2018 15:36:00", "content": "Yeah.... But, even after reading it. We need to normalise columns with json type values which can create copies while merging as per method load_csv mentioned in most of kernels.
", "rawMarkdown": "Yeah.... But, even after reading it. We need to normalise columns with json type values which can create copies while merging as per method load_csv mentioned in most of kernels.", "votes": null }, { "id": "418308", "postDate": "11/09/2018 16:17:36", "content": "There is a misunderstood, you can save 20Gb if you don't read hits... (800Mb is the size of train_v2 without 'hits')
\n\nBut i think this feature must be important :-)
", "rawMarkdown": "There is a misunderstood, you can save 20Gb if you don't read hits... (800Mb is the size of train_v2 without 'hits')\n\nBut i think this feature must be important :-)", "votes": null }, { "id": "418312", "postDate": "11/09/2018 16:21:34", "content": "Edited :)
", "rawMarkdown": "Edited :)", "votes": null }, { "id": "418543", "postDate": "11/10/2018 04:43:27", "content": "Try this kernel, https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize
", "rawMarkdown": "Try this kernel, https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize", "votes": null }, { "id": "418575", "postDate": "11/10/2018 07:03:22", "content": "Here's a kernel without the hits column. You can import the output data into R or Python. Instructions included in the script.\nhttps://www.kaggle.com/burritodan/gstore-1-flatten
", "rawMarkdown": "Here's a kernel without the hits column. You can import the output data into R or Python. Instructions included in the script.\nhttps://www.kaggle.com/burritodan/gstore-1-flatten", "votes": null }, { "id": "418609", "postDate": "11/10/2018 09:09:14", "content": "Thank you... I will check it....
", "rawMarkdown": "Thank you... I will check it....", "votes": null }, { "id": "418610", "postDate": "11/10/2018 09:09:24", "content": "Yeah sure... Thank you...
", "rawMarkdown": "Yeah sure... Thank you...", "votes": null }, { "id": "418970", "postDate": "11/11/2018 01:41:29", "content": "I succeeded in analyzing the new data set and submitting my prediction result. However, since the new data set is very big, I could only do basic feature engineering. https://www.kaggle.com/yoshoku/gacrp-v2-starter-kit
", "rawMarkdown": "I succeeded in analyzing the new data set and submitting my prediction result. However, since the new data set is very big, I could only do basic feature engineering. https://www.kaggle.com/yoshoku/gacrp-v2-starter-kit", "votes": null }, { "id": "418977", "postDate": "11/11/2018 02:16:36", "content": "Hello Yoshoku,
\n\nThanks for your contribution. Unfortunately, you are missing the \"hits\" column which is the biggest part.. If you look at JSON COLUMNS, you only have ['device', 'geoNetwork', 'totals', 'trafficSource'], you should add hits to normalize this column. But for now, no one succeeded to normalize it properly..
", "rawMarkdown": "Hello Yoshoku, \n\nThanks for your contribution. Unfortunately, you are missing the \"hits\" column which is the biggest part.. If you look at JSON COLUMNS, you only have ['device', 'geoNetwork', 'totals', 'trafficSource'], you should add hits to normalize this column. But for now, no one succeeded to normalize it properly..", "votes": null }, { "id": "418980", "postDate": "11/11/2018 02:44:10", "content": "Thank you for your advice! I will try to normalize the hits column.
", "rawMarkdown": "Thank you for your advice! I will try to normalize the hits column.", "votes": null } ], "comments": [ { "id": 418276, "author_name": "xtelima", "author_url": "", "post_date": "11/09/2018 15:27:01", "content": "Hello Pratik,
\n\nI raised the issue here : https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71042\nI'm still working on it but as Vincent suggest in the thread, if you don't read the \"hit\" column you can save 20gb... as Vincent says, it must be a critical feature... I think we need to think about what columns are really needed and drop the only NaNs ones and do some optimization on the dataframe (changing column type from float 64 to 32 etc...)
", "votes": null, "replies": [ { "id": 418283, "author_name": "jeffysonar", "author_url": "", "post_date": "11/09/2018 15:36:00", "content": "Yeah.... But, even after reading it. We need to normalise columns with json type values which can create copies while merging as per method load_csv mentioned in most of kernels.
", "votes": null, "replies": [] }, { "id": 418308, "author_name": "vlarmet", "author_url": "", "post_date": "11/09/2018 16:17:36", "content": "There is a misunderstood, you can save 20Gb if you don't read hits... (800Mb is the size of train_v2 without 'hits')
\n\nBut i think this feature must be important :-)
", "votes": null, "replies": [] }, { "id": 418312, "author_name": "xtelima", "author_url": "", "post_date": "11/09/2018 16:21:34", "content": "Edited :)
", "votes": null, "replies": [] } ] }, { "id": 418543, "author_name": "codlife", "author_url": "", "post_date": "11/10/2018 04:43:27", "content": "Try this kernel, https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize
", "votes": null, "replies": [ { "id": 418609, "author_name": "jeffysonar", "author_url": "", "post_date": "11/10/2018 09:09:14", "content": "Thank you... I will check it....
", "votes": null, "replies": [] } ] }, { "id": 418575, "author_name": "burritodan", "author_url": "", "post_date": "11/10/2018 07:03:22", "content": "Here's a kernel without the hits column. You can import the output data into R or Python. Instructions included in the script.\nhttps://www.kaggle.com/burritodan/gstore-1-flatten
", "votes": null, "replies": [ { "id": 418610, "author_name": "jeffysonar", "author_url": "", "post_date": "11/10/2018 09:09:24", "content": "Yeah sure... Thank you...
", "votes": null, "replies": [] } ] }, { "id": 418970, "author_name": "yoshoku", "author_url": "", "post_date": "11/11/2018 01:41:29", "content": "I succeeded in analyzing the new data set and submitting my prediction result. However, since the new data set is very big, I could only do basic feature engineering. https://www.kaggle.com/yoshoku/gacrp-v2-starter-kit
", "votes": null, "replies": [ { "id": 418977, "author_name": "xtelima", "author_url": "", "post_date": "11/11/2018 02:16:36", "content": "Hello Yoshoku,
\n\nThanks for your contribution. Unfortunately, you are missing the \"hits\" column which is the biggest part.. If you look at JSON COLUMNS, you only have ['device', 'geoNetwork', 'totals', 'trafficSource'], you should add hits to normalize this column. But for now, no one succeeded to normalize it properly..
", "votes": null, "replies": [] }, { "id": 418980, "author_name": "yoshoku", "author_url": "", "post_date": "11/11/2018 02:44:10", "content": "Thank you for your advice! I will try to normalize the hits column.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "418272": "Hi,\nI am not able to read CSVs in new dataset v2. The kernel seem to attempt to read it. It maximises available RAM, after some time it dies and restarts.\nAlso, previous version train.csv and test.csv were 1.5 GB and 1.3 GB, respectively. However, version 2 dataset seems quite large (train_v2.csv and test_v2.csv are 23.67 GB and 7.09 GB, respectively ). \nIt is quite something to think about as with available RAM being around 16 GB, reading both CSVs at a time doesn't seem feasible...\nAny solutions/suggestions ???", "418276": "Hello Pratik,\n\nI raised the issue here : https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71042\nI'm still working on it but as Vincent suggest in the thread, if you don't read the \"hit\" column you can save 20gb... as Vincent says, it must be a critical feature... I think we need to think about what columns are really needed and drop the only NaNs ones and do some optimization on the dataframe (changing column type from float 64 to 32 etc...)", "418283": "Yeah.... But, even after reading it. We need to normalise columns with json type values which can create copies while merging as per method load_csv mentioned in most of kernels.", "418308": "There is a misunderstood, you can save 20Gb if you don't read hits... (800Mb is the size of train_v2 without 'hits')\n\nBut i think this feature must be important :-)", "418312": "Edited :)", "418543": "Try this kernel, https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize", "418575": "Here's a kernel without the hits column. You can import the output data into R or Python. Instructions included in the script.\nhttps://www.kaggle.com/burritodan/gstore-1-flatten", "418609": "Thank you... I will check it....", "418610": "Yeah sure... Thank you...", "418970": "I succeeded in analyzing the new data set and submitting my prediction result. However, since the new data set is very big, I could only do basic feature engineering. https://www.kaggle.com/yoshoku/gacrp-v2-starter-kit", "418977": "Hello Yoshoku, \n\nThanks for your contribution. Unfortunately, you are missing the \"hits\" column which is the biggest part.. If you look at JSON COLUMNS, you only have ['device', 'geoNetwork', 'totals', 'trafficSource'], you should add hits to normalize this column. But for now, no one succeeded to normalize it properly..", "418980": "Thank you for your advice! I will try to normalize the hits column." }, "source": "meta" }