{
  "id": 5443,
  "title": "MemoryError on EC2",
  "url": "/competitions/accelerometer-biometric-competition/discussion/5443",
  "author_name": "",
  "post_date": "2013-08-19T09:43:13.703Z",
  "votes": null,
  "comment_count": 2,
  "views": 1439,
  "content": "<p>I am have having a hard time trying to load the data into a <strong>pandas</strong> DataFrame. I have been trying to use an EC2 instance. My team has a&nbsp;m2.xlarge instance and yet, we keep running into a MemoryError being thrown.</p>\n<p>Tried to `read_csv` in chunks followed by `concat` method in vain. Any tips on how to handle this? Should I get a different instance type?</p>",
  "messages": [
    {
      "id": "28935",
      "postDate": "08/19/2013 09:43:13",
      "content": "<p>I am have having a hard time trying to load the data into a <strong>pandas</strong> DataFrame. I have been trying to use an EC2 instance. My team has a&nbsp;m2.xlarge instance and yet, we keep running into a MemoryError being thrown.</p>\n<p>Tried to `read_csv` in chunks followed by `concat` method in vain. Any tips on how to handle this? Should I get a different instance type?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "28962",
      "postDate": "08/19/2013 17:40:05",
      "content": "<p>I'm a complete python noob, but this works for me on an m2.xlarge instance.</p>\n<p>https://gist.github.com/jlburkhead/6271870</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29103",
      "postDate": "08/20/2013 23:32:25",
      "content": "<p>&nbsp;</p>\n<p>Rohit,</p>\n<p>Not sure, but your problem could be due to ipython notebook.&nbsp;</p>\n<p>I am rewriting my code to read the train.csv and parse it out using process pools and it crashed the kernel in ipython notebook.&nbsp; When I tried the same code in spyder it ran fine with no kernel crash.&nbsp;</p>\n<p>Steve</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 28962,
      "author_name": "foobarbazquux",
      "author_url": "",
      "post_date": "08/19/2013 17:40:05",
      "content": "<p>I'm a complete python noob, but this works for me on an m2.xlarge instance.</p>\n<p>https://gist.github.com/jlburkhead/6271870</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29103,
      "author_name": "geringer",
      "author_url": "",
      "post_date": "08/20/2013 23:32:25",
      "content": "<p>&nbsp;</p>\n<p>Rohit,</p>\n<p>Not sure, but your problem could be due to ipython notebook.&nbsp;</p>\n<p>I am rewriting my code to read the train.csv and parse it out using process pools and it crashed the kernel in ipython notebook.&nbsp; When I tried the same code in spyder it ran fine with no kernel crash.&nbsp;</p>\n<p>Steve</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "28935": "",
    "28962": "",
    "29103": ""
  },
  "source": "meta"
}