{
  "id": 77710,
  "title": "how to solve memory error on AWS instance c4.4 large? ",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/77710",
  "author_name": "",
  "post_date": "2019-01-15T22:44:36.132376400Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I am running the exploratory analysis on jupyter notebook from AWS instance. I am using Deep learning Ubuntu Version 20 AMI and the instance is c4.4xlarge, which has 30 GB RAM. I am only doing some basic stuff on the dataframe, like dropping, merging yet. I have not conducted any machine learning or deep learning. But, it already shows memory error. Why is there a memory problem given that the AMI has 30 GB RAM? Does anyone have any suggestions on how to overcome the memory error except switching to a even more powerful instance type? I have tried decreasing the size by changing dtypes for all dataframes at the beginning already. I considered about dask, but did not use dask eventually since it does not have all functionality that pandas have. </p>\n\n<p>Also, can I ask how you guys managed to handle such big data file? I am new to Kaggle and this is the first big file I have handled so far. Thanks ahead!</p>",
  "messages": [
    {
      "id": "456479",
      "postDate": "01/15/2019 22:44:36",
      "content": "<p>I am running the exploratory analysis on jupyter notebook from AWS instance. I am using Deep learning Ubuntu Version 20 AMI and the instance is c4.4xlarge, which has 30 GB RAM. I am only doing some basic stuff on the dataframe, like dropping, merging yet. I have not conducted any machine learning or deep learning. But, it already shows memory error. Why is there a memory problem given that the AMI has 30 GB RAM? Does anyone have any suggestions on how to overcome the memory error except switching to a even more powerful instance type? I have tried decreasing the size by changing dtypes for all dataframes at the beginning already. I considered about dask, but did not use dask eventually since it does not have all functionality that pandas have. </p>\n\n<p>Also, can I ask how you guys managed to handle such big data file? I am new to Kaggle and this is the first big file I have handled so far. Thanks ahead!</p>",
      "rawMarkdown": "I am running the exploratory analysis on jupyter notebook from AWS instance. I am using Deep learning Ubuntu Version 20 AMI and the instance is c4.4xlarge, which has 30 GB RAM. I am only doing some basic stuff on the dataframe, like dropping, merging yet. I have not conducted any machine learning or deep learning. But, it already shows memory error. Why is there a memory problem given that the AMI has 30 GB RAM? Does anyone have any suggestions on how to overcome the memory error except switching to a even more powerful instance type? I have tried decreasing the size by changing dtypes for all dataframes at the beginning already. I considered about dask, but did not use dask eventually since it does not have all functionality that pandas have. \n\nAlso, can I ask how you guys managed to handle such big data file? I am new to Kaggle and this is the first big file I have handled so far. Thanks ahead!",
      "votes": null
    },
    {
      "id": "456927",
      "postDate": "01/16/2019 19:01:41",
      "content": "<p>hi, i too faced quite a bit memory error issues using elo data sets.  but i assumed it is because of my laptop memry of 8gb. im surprised u r facing out of memory in 30gb ram instance.</p>\n\n<p>i used the following techniques to reduce memory usage:</p>\n\n<ol>\n<li>reduced merges to very minimum. used concat instead</li>\n<li>stopped using operations such as info, describe, etc</li>\n<li>deleted temp variables by using del operation in python</li>\n<li>i did my operations in chunks . every time i came to a logical end, i would write the dataframe to disk. and my next operations would be on the new csv</li>\n</ol>\n\n<p>not sure if you came across this: <a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75713#452412\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75713#452412</a></p>\n\n<p>cheers! and all the best in the competition</p>",
      "rawMarkdown": "hi, i too faced quite a bit memory error issues using elo data sets.  but i assumed it is because of my laptop memry of 8gb. im surprised u r facing out of memory in 30gb ram instance.\n\ni used the following techniques to reduce memory usage:\n\n1.  reduced merges to very minimum. used concat instead\n2. stopped using operations such as info, describe, etc\n3. deleted temp variables by using del operation in python\n4. i did my operations in chunks . every time i came to a logical end, i would write the dataframe to disk. and my next operations would be on the new csv\n\n\nnot sure if you came across this: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75713#452412\n\ncheers! and all the best in the competition",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 456927,
      "author_name": "somaktukai",
      "author_url": "",
      "post_date": "01/16/2019 19:01:41",
      "content": "<p>hi, i too faced quite a bit memory error issues using elo data sets.  but i assumed it is because of my laptop memry of 8gb. im surprised u r facing out of memory in 30gb ram instance.</p>\n\n<p>i used the following techniques to reduce memory usage:</p>\n\n<ol>\n<li>reduced merges to very minimum. used concat instead</li>\n<li>stopped using operations such as info, describe, etc</li>\n<li>deleted temp variables by using del operation in python</li>\n<li>i did my operations in chunks . every time i came to a logical end, i would write the dataframe to disk. and my next operations would be on the new csv</li>\n</ol>\n\n<p>not sure if you came across this: <a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75713#452412\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75713#452412</a></p>\n\n<p>cheers! and all the best in the competition</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "456479": "I am running the exploratory analysis on jupyter notebook from AWS instance. I am using Deep learning Ubuntu Version 20 AMI and the instance is c4.4xlarge, which has 30 GB RAM. I am only doing some basic stuff on the dataframe, like dropping, merging yet. I have not conducted any machine learning or deep learning. But, it already shows memory error. Why is there a memory problem given that the AMI has 30 GB RAM? Does anyone have any suggestions on how to overcome the memory error except switching to a even more powerful instance type? I have tried decreasing the size by changing dtypes for all dataframes at the beginning already. I considered about dask, but did not use dask eventually since it does not have all functionality that pandas have. \n\nAlso, can I ask how you guys managed to handle such big data file? I am new to Kaggle and this is the first big file I have handled so far. Thanks ahead!",
    "456927": "hi, i too faced quite a bit memory error issues using elo data sets.  but i assumed it is because of my laptop memry of 8gb. im surprised u r facing out of memory in 30gb ram instance.\n\ni used the following techniques to reduce memory usage:\n\n1.  reduced merges to very minimum. used concat instead\n2. stopped using operations such as info, describe, etc\n3. deleted temp variables by using del operation in python\n4. i did my operations in chunks . every time i came to a logical end, i would write the dataframe to disk. and my next operations would be on the new csv\n\n\nnot sure if you came across this: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75713#452412\n\ncheers! and all the best in the competition"
  },
  "source": "meta"
}