{
  "id": 15307,
  "title": "Can't load file into Ipython",
  "url": "/competitions/grasp-and-lift-eeg-detection/discussion/15307",
  "author_name": "",
  "post_date": "2015-07-17T03:41:03.830Z",
  "votes": null,
  "comment_count": 3,
  "views": 1177,
  "content": "<p>Hi, </p>\n\n<p>I have hit an impasse. I have combined the subject files in the training set into one big csv file. But when I try to read it into a dataframe in Pandas using Ipython Notebook, it works for 45 minutes and then gives me a memory error. This has happened each time (several times). I am not sure what to do at this point. \nWould it be any different to try and put the data in a database and import it from there ?  Or use R (RStudio) ?  Or do I simply need to find a computer with more Ram ?</p>\n\n<p>The file is 2.5 GB big. \nMy current computer has 4 GB of Ram and a 64 bit Windows OS. \nI have an Ubuntu pc with 2.5 GB Ram that I haven't tried yet. </p>\n\n<p>Should I start trying to learn about Amazon cloud servers? (which I have never used)</p>\n\n<p>Am I effectively out of this competition ?</p>\n\n<p>Any thoughts on how to get past this would be greatly appreciated . </p>\n\n<p>Thanks,</p>\n\n<p>Chris  </p>",
  "messages": [
    {
      "id": "85756",
      "postDate": "07/17/2015 03:41:03",
      "content": "<p>Hi, </p>\n\n<p>I have hit an impasse. I have combined the subject files in the training set into one big csv file. But when I try to read it into a dataframe in Pandas using Ipython Notebook, it works for 45 minutes and then gives me a memory error. This has happened each time (several times). I am not sure what to do at this point. \nWould it be any different to try and put the data in a database and import it from there ?  Or use R (RStudio) ?  Or do I simply need to find a computer with more Ram ?</p>\n\n<p>The file is 2.5 GB big. \nMy current computer has 4 GB of Ram and a 64 bit Windows OS. \nI have an Ubuntu pc with 2.5 GB Ram that I haven't tried yet. </p>\n\n<p>Should I start trying to learn about Amazon cloud servers? (which I have never used)</p>\n\n<p>Am I effectively out of this competition ?</p>\n\n<p>Any thoughts on how to get past this would be greatly appreciated . </p>\n\n<p>Thanks,</p>\n\n<p>Chris  </p>",
      "rawMarkdown": "Hi, \r\n\r\nI have hit an impasse. I have combined the subject files in the training set into one big csv file. But when I try to read it into a dataframe in Pandas using Ipython Notebook, it works for 45 minutes and then gives me a memory error. This has happened each time (several times). I am not sure what to do at this point. \r\nWould it be any different to try and put the data in a database and import it from there ?  Or use R (RStudio) ?  Or do I simply need to find a computer with more Ram ?\r\n\r\nThe file is 2.5 GB big. \r\nMy current computer has 4 GB of Ram and a 64 bit Windows OS. \r\nI have an Ubuntu pc with 2.5 GB Ram that I haven't tried yet. \r\n\r\nShould I start trying to learn about Amazon cloud servers? (which I have never used)\r\n\r\nAm I effectively out of this competition ?\r\n\r\nAny thoughts on how to get past this would be greatly appreciated . \r\n\r\nThanks,\r\n\r\nChris",
      "votes": null
    },
    {
      "id": "85989",
      "postDate": "07/18/2015 23:23:15",
      "content": "<p>Ok, </p>\n\n<p>So I added 4 gb of Ram to my laptop to make it 8 gb total and I changed from 32 bit Firefox to 64 bit Chrome for running IPython Notebook and now I can at least load the file into a pandas dataframe. </p>\n\n<p>How are other people dealing with the files ? What are the specs of your PC Ram wise and what IDEs are you using if you are using Python ?</p>",
      "rawMarkdown": "Ok, \r\n\r\nSo I added 4 gb of Ram to my laptop to make it 8 gb total and I changed from 32 bit Firefox to 64 bit Chrome for running IPython Notebook and now I can at least load the file into a pandas dataframe. \r\n\r\nHow are other people dealing with the files ? What are the specs of your PC Ram wise and what IDEs are you using if you are using Python ?",
      "votes": null
    },
    {
      "id": "86114",
      "postDate": "07/20/2015 02:31:48",
      "content": "<p>I found  Blaze <a href=\"http://continuum.io/blog/blaze\">http://continuum.io/blog/blaze</a> to be a big help in this regard. It loads data in seconds that took 45 minutes before.  Still curious how other people are doing it. If you are reading this, please respond with how you loaded data into your system .  I know I am new, has anyone else been running into this as a problem ?</p>",
      "rawMarkdown": "I found  Blaze http://continuum.io/blog/blaze to be a big help in this regard. It loads data in seconds that took 45 minutes before.  Still curious how other people are doing it. If you are reading this, please respond with how you loaded data into your system .  I know I am new, has anyone else been running into this as a problem ?",
      "votes": null
    },
    {
      "id": "87104",
      "postDate": "07/26/2015 23:43:29",
      "content": "<p>Yes, I had some problems too, because I have only 2GB of RAM in one of my computers. The solution is to leave the files partitioned and load them only when they are needed, so you'll have much more free memory (for a cost of some disk I/O).</p>",
      "rawMarkdown": "Yes, I had some problems too, because I have only 2GB of RAM in one of my computers. The solution is to leave the files partitioned and load them only when they are needed, so you'll have much more free memory (for a cost of some disk I/O).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 85989,
      "author_name": "chrisfs",
      "author_url": "",
      "post_date": "07/18/2015 23:23:15",
      "content": "<p>Ok, </p>\n\n<p>So I added 4 gb of Ram to my laptop to make it 8 gb total and I changed from 32 bit Firefox to 64 bit Chrome for running IPython Notebook and now I can at least load the file into a pandas dataframe. </p>\n\n<p>How are other people dealing with the files ? What are the specs of your PC Ram wise and what IDEs are you using if you are using Python ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86114,
      "author_name": "chrisfs",
      "author_url": "",
      "post_date": "07/20/2015 02:31:48",
      "content": "<p>I found  Blaze <a href=\"http://continuum.io/blog/blaze\">http://continuum.io/blog/blaze</a> to be a big help in this regard. It loads data in seconds that took 45 minutes before.  Still curious how other people are doing it. If you are reading this, please respond with how you loaded data into your system .  I know I am new, has anyone else been running into this as a problem ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 87104,
      "author_name": "apaszke",
      "author_url": "",
      "post_date": "07/26/2015 23:43:29",
      "content": "<p>Yes, I had some problems too, because I have only 2GB of RAM in one of my computers. The solution is to leave the files partitioned and load them only when they are needed, so you'll have much more free memory (for a cost of some disk I/O).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "85756": "Hi, \r\n\r\nI have hit an impasse. I have combined the subject files in the training set into one big csv file. But when I try to read it into a dataframe in Pandas using Ipython Notebook, it works for 45 minutes and then gives me a memory error. This has happened each time (several times). I am not sure what to do at this point. \r\nWould it be any different to try and put the data in a database and import it from there ?  Or use R (RStudio) ?  Or do I simply need to find a computer with more Ram ?\r\n\r\nThe file is 2.5 GB big. \r\nMy current computer has 4 GB of Ram and a 64 bit Windows OS. \r\nI have an Ubuntu pc with 2.5 GB Ram that I haven't tried yet. \r\n\r\nShould I start trying to learn about Amazon cloud servers? (which I have never used)\r\n\r\nAm I effectively out of this competition ?\r\n\r\nAny thoughts on how to get past this would be greatly appreciated . \r\n\r\nThanks,\r\n\r\nChris",
    "85989": "Ok, \r\n\r\nSo I added 4 gb of Ram to my laptop to make it 8 gb total and I changed from 32 bit Firefox to 64 bit Chrome for running IPython Notebook and now I can at least load the file into a pandas dataframe. \r\n\r\nHow are other people dealing with the files ? What are the specs of your PC Ram wise and what IDEs are you using if you are using Python ?",
    "86114": "I found  Blaze http://continuum.io/blog/blaze to be a big help in this regard. It loads data in seconds that took 45 minutes before.  Still curious how other people are doing it. If you are reading this, please respond with how you loaded data into your system .  I know I am new, has anyone else been running into this as a problem ?",
    "87104": "Yes, I had some problems too, because I have only 2GB of RAM in one of my computers. The solution is to leave the files partitioned and load them only when they are needed, so you'll have much more free memory (for a cost of some disk I/O)."
  },
  "source": "meta"
}