{
  "id": 6077,
  "title": "Pandas memory error with 8 gb ram",
  "url": "/competitions/expedia-personalized-sort/discussion/6077",
  "author_name": "",
  "post_date": "2013-10-22T12:54:25.207Z",
  "votes": null,
  "comment_count": 6,
  "views": 4770,
  "content": "<p>Hi all,</p>\n<p>I run the benchmark python code and the read_csv method in the data_io file is either killing the process or returning a memory error.</p>\n<p>I have the last pandas version and a 8gb machine. I run on linux mint 15.</p>\n<p>&nbsp;</p>\n<p>Anybody facing the same problem? Any ideas?&nbsp;</p>\n<p>&nbsp;</p>\n<p>Thank you,</p>\n<p>Simon</p>",
  "messages": [
    {
      "id": "32524",
      "postDate": "10/22/2013 12:54:25",
      "content": "<p>Hi all,</p>\n<p>I run the benchmark python code and the read_csv method in the data_io file is either killing the process or returning a memory error.</p>\n<p>I have the last pandas version and a 8gb machine. I run on linux mint 15.</p>\n<p>&nbsp;</p>\n<p>Anybody facing the same problem? Any ideas?&nbsp;</p>\n<p>&nbsp;</p>\n<p>Thank you,</p>\n<p>Simon</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32526",
      "postDate": "10/22/2013 13:21:44",
      "content": "<p>I think you need more ram for this dataset. 8 is to low.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32527",
      "postDate": "10/22/2013 13:32:35",
      "content": "<p>Yes, I am also facing this problem. My laptop is also 8gb and I am running the code in lubuntu 12.10.&nbsp;</p>\n<p>&nbsp;</p>\n<p>The way I am doing is only reading parts of the training data by using &quot;nrows&quot; parameter or you can use &quot;chunksize&quot;. I only found SGDregression in scikit learn has online learning ability. However, the result seems not good. @_@</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32537",
      "postDate": "10/22/2013 16:08:17",
      "content": "<p>Wow 8 is too low? Maybe i should lend a 256gb amazon machine lol... I guess i'll train on a smaller data and export predictions chunk by chunk but it seems really weird not being able to load a 2gb csv file into 8gb...</p>\n<p>Ty very much for the answers anyway.</p>\n<p>Did anybody managed to load the whole training/testing set?</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32883",
      "postDate": "10/30/2013 09:35:27",
      "content": "<p>Do you have the most recent version of Pandas? There was a new C based CSV reader introduced a few months ago that may assist.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41833",
      "postDate": "04/13/2014 07:00:37",
      "content": "<p>I'm having this issue as well on Ubuntu 12.04. When I start the benchmark, I have more than 27GB RAM free out of 32GB. I watch the RAM usage go all the way up to the top and then get the MemoryError.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42741",
      "postDate": "04/22/2014 19:39:43",
      "content": "<p>I can report that my problem was related to pandas being out of date.</p>\n<p>I resolved this using easy_install -U. Other update methods did not correctly update the pandas version.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 32526,
      "author_name": "leustagos",
      "author_url": "",
      "post_date": "10/22/2013 13:21:44",
      "content": "<p>I think you need more ram for this dataset. 8 is to low.</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32527,
      "author_name": "ericwu0",
      "author_url": "",
      "post_date": "10/22/2013 13:32:35",
      "content": "<p>Yes, I am also facing this problem. My laptop is also 8gb and I am running the code in lubuntu 12.10.&nbsp;</p>\n<p>&nbsp;</p>\n<p>The way I am doing is only reading parts of the training data by using &quot;nrows&quot; parameter or you can use &quot;chunksize&quot;. I only found SGDregression in scikit learn has online learning ability. However, the result seems not good. @_@</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32537,
      "author_name": "simonmaby",
      "author_url": "",
      "post_date": "10/22/2013 16:08:17",
      "content": "<p>Wow 8 is too low? Maybe i should lend a 256gb amazon machine lol... I guess i'll train on a smaller data and export predictions chunk by chunk but it seems really weird not being able to load a 2gb csv file into 8gb...</p>\n<p>Ty very much for the answers anyway.</p>\n<p>Did anybody managed to load the whole training/testing set?</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32883,
      "author_name": "a1rplay",
      "author_url": "",
      "post_date": "10/30/2013 09:35:27",
      "content": "<p>Do you have the most recent version of Pandas? There was a new C based CSV reader introduced a few months ago that may assist.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41833,
      "author_name": "kleinsound",
      "author_url": "",
      "post_date": "04/13/2014 07:00:37",
      "content": "<p>I'm having this issue as well on Ubuntu 12.04. When I start the benchmark, I have more than 27GB RAM free out of 32GB. I watch the RAM usage go all the way up to the top and then get the MemoryError.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42741,
      "author_name": "kleinsound",
      "author_url": "",
      "post_date": "04/22/2014 19:39:43",
      "content": "<p>I can report that my problem was related to pandas being out of date.</p>\n<p>I resolved this using easy_install -U. Other update methods did not correctly update the pandas version.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "32524": "",
    "32526": "",
    "32527": "",
    "32537": "",
    "32883": "",
    "41833": "",
    "42741": ""
  },
  "source": "meta"
}