![\"\"](\"https://storage.googleapis.com/kagglesdsdata/datasets/1729238/2834486/Screenshot%20from%202021-11-20%2019-21-38.png?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=databundle-worker-v2%40kaggle-161607.iam.gserviceaccount.com%2F20211121%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20211121T032346Z&X-Goog-Expires=345599&X-Goog-SignedHeaders=host&X-Goog-Signature=77bcc2295106ee39e55a5792d154c6584ca77be0d1241f5080ffe32cfd8e3e344f8b7b0ed6d353178eb9f5b5bd478641f4da25e5d15c9fd121f59ba4f667deefa17b2e1213d9a2887bda3aa1ac7b664b6c4cb6fc8c81efbe1bd0340687693fcab778d6183e3db1b25259da14a5d4711ef7f87eb681b736022b463638db68e3b154c8faf2a2feb31030b8b0e5893eefa146f0f7c5cb7de400bf66aaf2c5cad2ff029864f04bf8d35e53d65b2a8c44ec9fae916ee1e141da610d69d81df19a557920ff8506af819e18b59901aa7acea38405178b1805d8a683f64cdd83cb78417f3a2669c4e9d951c0970fd8ca41165afe5f4a4e0ea103daa26a09bd22f62d6754\")
It took almost 15 minutes for my Kaggle notebook to load the data for this competition. This is precisely the reason why I don't participate on Kaggle nearly as much as I would like; I just sit here waiting for data to load. It happens every single time. There's nothing wrong with my MacBook Pro, and I'm not running superfluous applications in the background.
\nThe wait time on this is unacceptable, especially considering that what we are all doing is pro bono work.
\nDoes anyone have a remedy for this so that every time I want to load my notebook I need not bring a copy of \"War and Peace\" along with me to page thru while I wait?
\n-R
", "messages": [ { "id": "1587832", "postDate": "11/19/2021 01:00:06", "content": "It took almost 15 minutes for my Kaggle notebook to load the data for this competition. This is precisely the reason why I don't participate on Kaggle nearly as much as I would like; I just sit here waiting for data to load. It happens every single time. There's nothing wrong with my MacBook Pro, and I'm not running superfluous applications in the background.
\nThe wait time on this is unacceptable, especially considering that what we are all doing is pro bono work.
\nDoes anyone have a remedy for this so that every time I want to load my notebook I need not bring a copy of \"War and Peace\" along with me to page thru while I wait?
\n-R
", "rawMarkdown": "It took almost 15 minutes for my Kaggle notebook to load the data for this competition. This is precisely the reason why I don't participate on Kaggle nearly as much as I would like; I just sit here waiting for data to load. It happens every single time. There's nothing wrong with my MacBook Pro, and I'm not running superfluous applications in the background.\n\nThe wait time on this is unacceptable, especially considering that what we are all doing is pro bono work. \n\nDoes anyone have a remedy for this so that every time I want to load my notebook I need not bring a copy of \"War and Peace\" along with me to page thru while I wait?\n\n-R", "votes": null }, { "id": "1589997", "postDate": "11/20/2021 19:50:37", "content": "What do you mean by \"load data\" here? Do you mean reading the CSV (e.g. pd.read_csv) or adding the GResearch dataset to the notebook?
@brandenkmurray I mean just pd.read_csv. Right now, I'm trying to skip the first 20million rows and have set low_memory to false. It is still processing the pd.read_csv command on three files (training set, asset details, sample submission) after 20 minutes. If I omit the low memory option, I get my data frames, but as soon as I try to merge my small training data set to asset_details, it says i've \"Tried to allocate too much memory\" and quits. How can anyone do this competition or, frankly, anything on this site? I'd be fascinated to know, and that is a dead-serious question.
", "rawMarkdown": "brandenkmurray I mean just pd.read_csv. Right now, I'm trying to skip the first 20million rows and have set low_memory to false. It is still processing the pd.read_csv command on three files (training set, asset details, sample submission) after 20 minutes. If I omit the low memory option, I get my data frames, but as soon as I try to merge my small training data set to asset_details, it says i've \"Tried to allocate too much memory\" and quits. How can **anyone** do this competition or, frankly, **anything** on this site? I'd be fascinated to know, and that is a dead-serious question.", "votes": null }, { "id": "1590217", "postDate": "11/21/2021 01:55:07", "content": "Still not loaded.
", "rawMarkdown": "Still not loaded.", "votes": null }, { "id": "1590220", "postDate": "11/21/2021 01:59:09", "content": "@maggiemd This issue deserves your attention as soon as possible. I enjoy time series, I have an MS in Finance, but all of that is useless if your site cannot handle in-memory processes. Could you please help me out here? Kaggle is useless if everyone needs a mainframe to compete or an endless cloud budget for AWS Sagemaker.
", "rawMarkdown": "maggiemd This issue deserves your attention as soon as possible. I enjoy time series, I have an MS in Finance, but all of that is useless if your site cannot handle in-memory processes. Could you please help me out here? Kaggle is useless if everyone needs a mainframe to compete or an endless cloud budget for AWS Sagemaker.", "votes": null }, { "id": "1590249", "postDate": "11/21/2021 02:20:14", "content": "Just got the \"Are you still there?\" message, and the data still have not loaded! 😂😂😂 So, I'm going to go get loaded and watch some TV, until I hear back from @maggiemd -- or one of her fellow wizards -- because I am certain this affects others as well. It has been three years since I joined this site and every single time, this has been the issue stopping me from making a decent submission (all one of them) or multiple submissions. Asking for help has gotten me nowhere, much less searching your forums with posts whose most recent vintages are c. 2018. It's time to have a \"come to Jesus meeting\" on this.
", "rawMarkdown": "Just got the \"Are you still there?\" message, and the data still have not loaded! 😂😂😂 So, I'm going to go get loaded and watch some TV, until I hear back from @maggiemd -- or one of her fellow wizards -- because I am certain this affects others as well. It has been three years since I joined this site and **every single time**, this has been the issue stopping me from making a decent submission (all one of them) or multiple submissions. Asking for help has gotten me nowhere, much less searching your forums with posts whose most recent vintages are c. 2018. It's time to have a \"come to Jesus meeting\" on this.", "votes": null }, { "id": "1590252", "postDate": "11/21/2021 02:30:08", "content": "Hmm.. Sounds strange. It works fine in every notebook I've tried and shouldn't take more than a minute to load. Could you make your notebook Public so I or someone else can try to replicate it?
", "rawMarkdown": "Hmm.. Sounds strange. It works fine in every notebook I've tried and shouldn't take more than a minute to load. Could you make your notebook Public so I or someone else can try to replicate it?", "votes": null }, { "id": "1590254", "postDate": "11/21/2021 02:35:34", "content": "@brandenkmurray Here ya go! Thanks for talking to me about this! https://www.kaggle.com/rcd1693/cryptopython
\nCC: @maggiemd
Hmm.. no idea why it isn't working for you. Only takes ~47s to read the CSV for me
\n
@brandenkmurray WTF! What kind of machine are you on? I have a MacBook Pro and 16GB of memory. There must be a setting I can change.
", "rawMarkdown": "brandenkmurray WTF! What kind of machine are you on? I have a MacBook Pro and 16GB of memory. There must be a setting I can change.", "votes": null }, { "id": "1590295", "postDate": "11/21/2021 04:32:02", "content": "Notebooks run on Kaggle computing resources in the cloud, it shouldn't be using your computer at all.
", "rawMarkdown": "Notebooks run on Kaggle computing resources in the cloud, it shouldn't be using your computer at all.", "votes": null }, { "id": "1591131", "postDate": "11/22/2021 02:31:15", "content": "@brandenkmurray something could be up with my account? It seems to max out kaggle CPU and Memory really quick. After trying the merge step, I got a rather large error message saying
\nMemoryError: Unable to allocate 319. TiB for an array with shape (43804950216100,) and data type int64
\nI have no idea where the shape is coming from; it's a simple inner join.
\nCC: @maggiemd
", "rawMarkdown": "brandenkmurray something could be up with my account? It seems to max out kaggle CPU and Memory really quick. After trying the merge step, I got a rather large error message saying\n\nMemoryError: Unable to allocate 319. TiB for an array with shape (43804950216100,) and data type int64\n\nI have no idea where the shape is coming from; it's a simple inner join.\n\nCC: @maggiemd", "votes": null }, { "id": "1607310", "postDate": "12/05/2021 20:34:16", "content": "It's a problem of your local python and pandas for MacOS
", "rawMarkdown": "It's a problem of your local python and pandas for MacOS", "votes": null } ], "comments": [ { "id": 1589997, "author_name": "brandenkmurray", "author_url": "", "post_date": "11/20/2021 19:50:37", "content": "What do you mean by \"load data\" here? Do you mean reading the CSV (e.g. pd.read_csv) or adding the GResearch dataset to the notebook?
@brandenkmurray I mean just pd.read_csv. Right now, I'm trying to skip the first 20million rows and have set low_memory to false. It is still processing the pd.read_csv command on three files (training set, asset details, sample submission) after 20 minutes. If I omit the low memory option, I get my data frames, but as soon as I try to merge my small training data set to asset_details, it says i've \"Tried to allocate too much memory\" and quits. How can anyone do this competition or, frankly, anything on this site? I'd be fascinated to know, and that is a dead-serious question.
", "votes": null, "replies": [ { "id": 1590252, "author_name": "brandenkmurray", "author_url": "", "post_date": "11/21/2021 02:30:08", "content": "Hmm.. Sounds strange. It works fine in every notebook I've tried and shouldn't take more than a minute to load. Could you make your notebook Public so I or someone else can try to replicate it?
", "votes": null, "replies": [] }, { "id": 1590254, "author_name": "rcd1693", "author_url": "", "post_date": "11/21/2021 02:35:34", "content": "@brandenkmurray Here ya go! Thanks for talking to me about this! https://www.kaggle.com/rcd1693/cryptopython
\nCC: @maggiemd
Hmm.. no idea why it isn't working for you. Only takes ~47s to read the CSV for me
\n
@brandenkmurray WTF! What kind of machine are you on? I have a MacBook Pro and 16GB of memory. There must be a setting I can change.
", "votes": null, "replies": [] }, { "id": 1590295, "author_name": "brandenkmurray", "author_url": "", "post_date": "11/21/2021 04:32:02", "content": "Notebooks run on Kaggle computing resources in the cloud, it shouldn't be using your computer at all.
", "votes": null, "replies": [] }, { "id": 1591131, "author_name": "rcd1693", "author_url": "", "post_date": "11/22/2021 02:31:15", "content": "@brandenkmurray something could be up with my account? It seems to max out kaggle CPU and Memory really quick. After trying the merge step, I got a rather large error message saying
\nMemoryError: Unable to allocate 319. TiB for an array with shape (43804950216100,) and data type int64
\nI have no idea where the shape is coming from; it's a simple inner join.
\nCC: @maggiemd
", "votes": null, "replies": [] } ] }, { "id": 1590217, "author_name": "rcd1693", "author_url": "", "post_date": "11/21/2021 01:55:07", "content": "Still not loaded.
", "votes": null, "replies": [] }, { "id": 1590220, "author_name": "rcd1693", "author_url": "", "post_date": "11/21/2021 01:59:09", "content": "@maggiemd This issue deserves your attention as soon as possible. I enjoy time series, I have an MS in Finance, but all of that is useless if your site cannot handle in-memory processes. Could you please help me out here? Kaggle is useless if everyone needs a mainframe to compete or an endless cloud budget for AWS Sagemaker.
", "votes": null, "replies": [] }, { "id": 1590249, "author_name": "rcd1693", "author_url": "", "post_date": "11/21/2021 02:20:14", "content": "Just got the \"Are you still there?\" message, and the data still have not loaded! 😂😂😂 So, I'm going to go get loaded and watch some TV, until I hear back from @maggiemd -- or one of her fellow wizards -- because I am certain this affects others as well. It has been three years since I joined this site and every single time, this has been the issue stopping me from making a decent submission (all one of them) or multiple submissions. Asking for help has gotten me nowhere, much less searching your forums with posts whose most recent vintages are c. 2018. It's time to have a \"come to Jesus meeting\" on this.
", "votes": null, "replies": [] }, { "id": 1607310, "author_name": "vladimircape", "author_url": "", "post_date": "12/05/2021 20:34:16", "content": "It's a problem of your local python and pandas for MacOS
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1587832": "It took almost 15 minutes for my Kaggle notebook to load the data for this competition. This is precisely the reason why I don't participate on Kaggle nearly as much as I would like; I just sit here waiting for data to load. It happens every single time. There's nothing wrong with my MacBook Pro, and I'm not running superfluous applications in the background.\n\nThe wait time on this is unacceptable, especially considering that what we are all doing is pro bono work. \n\nDoes anyone have a remedy for this so that every time I want to load my notebook I need not bring a copy of \"War and Peace\" along with me to page thru while I wait?\n\n-R", "1589997": "What do you mean by \"load data\" here? Do you mean reading the CSV (e.g. `pd.read_csv`) or adding the GResearch dataset to the notebook?", "1590204": "brandenkmurray I mean just pd.read_csv. Right now, I'm trying to skip the first 20million rows and have set low_memory to false. It is still processing the pd.read_csv command on three files (training set, asset details, sample submission) after 20 minutes. If I omit the low memory option, I get my data frames, but as soon as I try to merge my small training data set to asset_details, it says i've \"Tried to allocate too much memory\" and quits. How can **anyone** do this competition or, frankly, **anything** on this site? I'd be fascinated to know, and that is a dead-serious question.", "1590217": "Still not loaded.", "1590220": "maggiemd This issue deserves your attention as soon as possible. I enjoy time series, I have an MS in Finance, but all of that is useless if your site cannot handle in-memory processes. Could you please help me out here? Kaggle is useless if everyone needs a mainframe to compete or an endless cloud budget for AWS Sagemaker.", "1590249": "Just got the \"Are you still there?\" message, and the data still have not loaded! 😂😂😂 So, I'm going to go get loaded and watch some TV, until I hear back from @maggiemd -- or one of her fellow wizards -- because I am certain this affects others as well. It has been three years since I joined this site and **every single time**, this has been the issue stopping me from making a decent submission (all one of them) or multiple submissions. Asking for help has gotten me nowhere, much less searching your forums with posts whose most recent vintages are c. 2018. It's time to have a \"come to Jesus meeting\" on this.", "1590252": "Hmm.. Sounds strange. It works fine in every notebook I've tried and shouldn't take more than a minute to load. Could you make your notebook Public so I or someone else can try to replicate it?", "1590254": "brandenkmurray Here ya go! Thanks for talking to me about this! https://www.kaggle.com/rcd1693/cryptopython\nCC: @maggiemd", "1590266": "Hmm.. no idea why it isn't working for you. Only takes ~47s to read the CSV for me\n\n", "1590279": "brandenkmurray WTF! What kind of machine are you on? I have a MacBook Pro and 16GB of memory. There must be a setting I can change.", "1590295": "Notebooks run on Kaggle computing resources in the cloud, it shouldn't be using your computer at all.", "1591131": "brandenkmurray something could be up with my account? It seems to max out kaggle CPU and Memory really quick. After trying the merge step, I got a rather large error message saying\n\nMemoryError: Unable to allocate 319. TiB for an array with shape (43804950216100,) and data type int64\n\nI have no idea where the shape is coming from; it's a simple inner join.\n\nCC: @maggiemd", "1607310": "It's a problem of your local python and pandas for MacOS" }, "source": "meta" }