Hello everyone,
\n\nI have seen a lot of people struggling lately about loading train_v2.csv and test_v2.csv. Let me help you out.
\n\nWhy you can't load these files ?
\n\nThe reason why you can't load these files in the Kaggle Kernel is because of the RAM limitations. Indeed, the Kaggle Kernel is \"limited\" to 17.2Gb RAM (you can see this information at the top-right of your kernel, in the session box). When you will try to load the data, you will see your ram quickly displaying 17.2/17.2. Why ? Because python is loading the data \"in-memory\", which means that it depends on the ram available.
\n\nHow to load these files ?
\n\nUnfortunatly, there is no magic solution to use the Kaggle Kernel in this case... But there is a solution, and it's \"free\" ! \nYou can use the \"free\" 300$ credit coupon offered by Google Cloud to use their VMs. To make this possible, i suggest that you read this very good tutorial : https://medium.com/@kn.maragatham09/installing-jupyter-notebook-on-google-cloud-11979e40cd10 . In \"Step 3 : Create a VM\", don't forget to allocate enough RAM to your VM (I use more than 40gb). At the end, you will have access to a jupyter notebook, just like a Kaggle Kernel :)
\n\nNow you are probably wondering how to access those files from your notebook ? This is where Kaggle API is magic.. Just follow this link : https://colab.research.google.com/drive/1DofKEdQYaXmDWBzuResXWWvxhLgDeVyl
\n\nAnd TADAM ! You will be able to begin the competition !
\n\nHope this helps.
", "messages": [ { "id": "420373", "postDate": "11/13/2018 14:10:42", "content": "Hello everyone,
\n\nI have seen a lot of people struggling lately about loading train_v2.csv and test_v2.csv. Let me help you out.
\n\nWhy you can't load these files ?
\n\nThe reason why you can't load these files in the Kaggle Kernel is because of the RAM limitations. Indeed, the Kaggle Kernel is \"limited\" to 17.2Gb RAM (you can see this information at the top-right of your kernel, in the session box). When you will try to load the data, you will see your ram quickly displaying 17.2/17.2. Why ? Because python is loading the data \"in-memory\", which means that it depends on the ram available.
\n\nHow to load these files ?
\n\nUnfortunatly, there is no magic solution to use the Kaggle Kernel in this case... But there is a solution, and it's \"free\" ! \nYou can use the \"free\" 300$ credit coupon offered by Google Cloud to use their VMs. To make this possible, i suggest that you read this very good tutorial : https://medium.com/@kn.maragatham09/installing-jupyter-notebook-on-google-cloud-11979e40cd10 . In \"Step 3 : Create a VM\", don't forget to allocate enough RAM to your VM (I use more than 40gb). At the end, you will have access to a jupyter notebook, just like a Kaggle Kernel :)
\n\nNow you are probably wondering how to access those files from your notebook ? This is where Kaggle API is magic.. Just follow this link : https://colab.research.google.com/drive/1DofKEdQYaXmDWBzuResXWWvxhLgDeVyl
\n\nAnd TADAM ! You will be able to begin the competition !
\n\nHope this helps.
", "rawMarkdown": "Hello everyone,\n\nI have seen a lot of people struggling lately about loading train_v2.csv and test_v2.csv. Let me help you out.\n\n**Why you can't load these files ?** \n\nThe reason why you can't load these files in the Kaggle Kernel is because of the RAM limitations. Indeed, the Kaggle Kernel is \"limited\" to 17.2Gb RAM (you can see this information at the top-right of your kernel, in the session box). When you will try to load the data, you will see your ram quickly displaying 17.2/17.2. *Why ?* Because python is loading the data \"in-memory\", which means that it depends on the ram available.\n\n**How to load these files ?**\n\nUnfortunatly, there is no magic solution to use the Kaggle Kernel in this case... But there is a solution, and it's \"free\" ! \nYou can use the \"free\" 300$ credit coupon offered by Google Cloud to use their VMs. To make this possible, i suggest that you read this very good tutorial : https://medium.com/@kn.maragatham09/installing-jupyter-notebook-on-google-cloud-11979e40cd10 . In \"Step 3 : Create a VM\", don't forget to allocate enough RAM to your VM (I use more than 40gb). At the end, you will have access to a jupyter notebook, just like a Kaggle Kernel :) \n\nNow you are probably wondering how to access those files from your notebook ? This is where Kaggle API is magic.. Just follow this link : https://colab.research.google.com/drive/1DofKEdQYaXmDWBzuResXWWvxhLgDeVyl\n\nAnd TADAM ! You will be able to begin the competition ! \n\nHope this helps.", "votes": null }, { "id": "420815", "postDate": "11/14/2018 06:51:19", "content": "A suggestion for those who're running Jupyter notebook on local PCs/laptops with not enough RAM is to use the nrows parameter for you to do your initial data exploration on the 1st 1000 rows for example.\nOnce you're done going thru the sample, you can then read your data in batches by using the skiprows parameter
", "rawMarkdown": "A suggestion for those who're running Jupyter notebook on local PCs/laptops with not enough RAM is to use the nrows parameter for you to do your initial data exploration on the 1st 1000 rows for example.\nOnce you're done going thru the sample, you can then read your data in batches by using the skiprows parameter", "votes": null }, { "id": "421123", "postDate": "11/14/2018 16:02:25", "content": "Thanks. I'm experiencing the same issue and it gets even worse once you flatten the JSON columns. Is anyone running this competition on \"free cloud notebooks\"?
", "rawMarkdown": "Thanks. I'm experiencing the same issue and it gets even worse once you flatten the JSON columns. Is anyone running this competition on \"free cloud notebooks\"?", "votes": null }, { "id": "424283", "postDate": "11/19/2018 20:45:04", "content": "Any idea how one can load the train data who are using R studio on Laptop? Any idea how to get rid of Memory error.?
", "rawMarkdown": "Any idea how one can load the train data who are using R studio on Laptop? Any idea how to get rid of Memory error.?", "votes": null }, { "id": "424304", "postDate": "11/19/2018 21:45:07", "content": "Here is on solution : read.csv(con, nrows=\"CHUNK SIZE\",...)
", "rawMarkdown": "Here is on solution : read.csv(con, nrows=\"CHUNK SIZE\",...)", "votes": null }, { "id": "426552", "postDate": "11/23/2018 12:40:55", "content": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd
", "rawMarkdown": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd", "votes": null }, { "id": "426627", "postDate": "11/23/2018 14:59:30", "content": "There is a lot of useless data, and a lot of categorical data that is stored as text. My solution was to load the test and train data in batches of about 10,000 rows. Then I would process each batch (converting text to categoricals, dropping useless data, etc) and then finally I would concatenate all the processed batches. The processing reduced the data size enough that I could easily load the train and test data at the same time.
", "rawMarkdown": "There is a lot of useless data, and a lot of categorical data that is stored as text. My solution was to load the test and train data in batches of about 10,000 rows. Then I would process each batch (converting text to categoricals, dropping useless data, etc) and then finally I would concatenate all the processed batches. The processing reduced the data size enough that I could easily load the train and test data at the same time.", "votes": null } ], "comments": [ { "id": 420815, "author_name": "teesoong", "author_url": "", "post_date": "11/14/2018 06:51:19", "content": "A suggestion for those who're running Jupyter notebook on local PCs/laptops with not enough RAM is to use the nrows parameter for you to do your initial data exploration on the 1st 1000 rows for example.\nOnce you're done going thru the sample, you can then read your data in batches by using the skiprows parameter
", "votes": null, "replies": [ { "id": 424283, "author_name": "jainik1392", "author_url": "", "post_date": "11/19/2018 20:45:04", "content": "Any idea how one can load the train data who are using R studio on Laptop? Any idea how to get rid of Memory error.?
", "votes": null, "replies": [] }, { "id": 424304, "author_name": "xtelima", "author_url": "", "post_date": "11/19/2018 21:45:07", "content": "Here is on solution : read.csv(con, nrows=\"CHUNK SIZE\",...)
", "votes": null, "replies": [] } ] }, { "id": 421123, "author_name": "luisflowers", "author_url": "", "post_date": "11/14/2018 16:02:25", "content": "Thanks. I'm experiencing the same issue and it gets even worse once you flatten the JSON columns. Is anyone running this competition on \"free cloud notebooks\"?
", "votes": null, "replies": [] }, { "id": 426552, "author_name": "mlisovyi", "author_url": "", "post_date": "11/23/2018 12:40:55", "content": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd
", "votes": null, "replies": [] }, { "id": 426627, "author_name": "afgonczol", "author_url": "", "post_date": "11/23/2018 14:59:30", "content": "There is a lot of useless data, and a lot of categorical data that is stored as text. My solution was to load the test and train data in batches of about 10,000 rows. Then I would process each batch (converting text to categoricals, dropping useless data, etc) and then finally I would concatenate all the processed batches. The processing reduced the data size enough that I could easily load the train and test data at the same time.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "420373": "Hello everyone,\n\nI have seen a lot of people struggling lately about loading train_v2.csv and test_v2.csv. Let me help you out.\n\n**Why you can't load these files ?** \n\nThe reason why you can't load these files in the Kaggle Kernel is because of the RAM limitations. Indeed, the Kaggle Kernel is \"limited\" to 17.2Gb RAM (you can see this information at the top-right of your kernel, in the session box). When you will try to load the data, you will see your ram quickly displaying 17.2/17.2. *Why ?* Because python is loading the data \"in-memory\", which means that it depends on the ram available.\n\n**How to load these files ?**\n\nUnfortunatly, there is no magic solution to use the Kaggle Kernel in this case... But there is a solution, and it's \"free\" ! \nYou can use the \"free\" 300$ credit coupon offered by Google Cloud to use their VMs. To make this possible, i suggest that you read this very good tutorial : https://medium.com/@kn.maragatham09/installing-jupyter-notebook-on-google-cloud-11979e40cd10 . In \"Step 3 : Create a VM\", don't forget to allocate enough RAM to your VM (I use more than 40gb). At the end, you will have access to a jupyter notebook, just like a Kaggle Kernel :) \n\nNow you are probably wondering how to access those files from your notebook ? This is where Kaggle API is magic.. Just follow this link : https://colab.research.google.com/drive/1DofKEdQYaXmDWBzuResXWWvxhLgDeVyl\n\nAnd TADAM ! You will be able to begin the competition ! \n\nHope this helps.", "420815": "A suggestion for those who're running Jupyter notebook on local PCs/laptops with not enough RAM is to use the nrows parameter for you to do your initial data exploration on the 1st 1000 rows for example.\nOnce you're done going thru the sample, you can then read your data in batches by using the skiprows parameter", "421123": "Thanks. I'm experiencing the same issue and it gets even worse once you flatten the JSON columns. Is anyone running this competition on \"free cloud notebooks\"?", "424283": "Any idea how one can load the train data who are using R studio on Laptop? Any idea how to get rid of Memory error.?", "424304": "Here is on solution : read.csv(con, nrows=\"CHUNK SIZE\",...)", "426552": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd", "426627": "There is a lot of useless data, and a lot of categorical data that is stored as text. My solution was to load the test and train data in batches of about 10,000 rows. Then I would process each batch (converting text to categoricals, dropping useless data, etc) and then finally I would concatenate all the processed batches. The processing reduced the data size enough that I could easily load the train and test data at the same time." }, "source": "meta" }