{
  "id": 3662,
  "title": "Takes forever to read events.csv",
  "url": "/competitions/event-recommendation-engine-challenge/discussion/3662",
  "author_name": "",
  "post_date": "2013-01-22T13:17:57.620Z",
  "votes": null,
  "comment_count": 4,
  "views": 1523,
  "content": "<p>Hi people!</p>\r\n<p>I am wondering if some of you had any performance issues in reading event.csv since it is a very large file. Is it because I am using pandas-python package &nbsp;that it is taking a very long time to read the &quot;event.csv&quot; file and eventually freezes my computer?\r\n Or perhaps the amount of RAM I have which is only 4 GB?&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>Your helpful suggestions on how to cope with such large .csv files will be greatly appreciated!</p>\r\n<p>&nbsp;</p>\r\n<p>Thanks a lot!</p>",
  "messages": [
    {
      "id": "19583",
      "postDate": "01/22/2013 13:17:57",
      "content": "<p>Hi people!</p>\r\n<p>I am wondering if some of you had any performance issues in reading event.csv since it is a very large file. Is it because I am using pandas-python package &nbsp;that it is taking a very long time to read the &quot;event.csv&quot; file and eventually freezes my computer?\r\n Or perhaps the amount of RAM I have which is only 4 GB?&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>Your helpful suggestions on how to cope with such large .csv files will be greatly appreciated!</p>\r\n<p>&nbsp;</p>\r\n<p>Thanks a lot!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "19584",
      "postDate": "01/22/2013 13:36:49",
      "content": "<p>I'm also using Python and had some problems with the large files, and my solution was to insert the data into MySQL using the following sql code:&nbsp;</p>\r\n<p><span style=\"line-height:1.4em\">LOAD DATA LOCAL INFILE 'c:/Projekt/Big Data/events.csv' INTO TABLE database.table FIELDS TERMINATED BY ',' ENCLOSED BY '&quot;' LINES TERMINATED BY '\\n' IGNORE 1 LINES;</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "19589",
      "postDate": "01/22/2013 16:14:29",
      "content": "<p>@<a class=\"x_x_profilelink\" title=\"View Erik Nordeus's profile\" href=\"http://www.kaggle.com/users/69997/erik-nordeus\">Erik Nordeus</a>&nbsp;Thank you very much for the fast response, but by storing the data in MySQL are you able to directly process that data\r\n using Python? If so, please kindly give me some clues on how to do that! Thanks a lot! :)</p>\r\n<p>Best regards!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "19591",
      "postDate": "01/22/2013 16:44:10",
      "content": "<p>You should watch this video tutorial:&nbsp;<a title=\"Python MySQLdb Quickstart\" href=\"http://youtu.be/ooIB8ZJattw\">Python MySQLdb Quickstart</a></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "19594",
      "postDate": "01/22/2013 17:53:46",
      "content": "<p>Thanks! I'll watch it noW!! :) :) :)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 19584,
      "author_name": "eriknordeus",
      "author_url": "",
      "post_date": "01/22/2013 13:36:49",
      "content": "<p>I'm also using Python and had some problems with the large files, and my solution was to insert the data into MySQL using the following sql code:&nbsp;</p>\r\n<p><span style=\"line-height:1.4em\">LOAD DATA LOCAL INFILE 'c:/Projekt/Big Data/events.csv' INTO TABLE database.table FIELDS TERMINATED BY ',' ENCLOSED BY '&quot;' LINES TERMINATED BY '\\n' IGNORE 1 LINES;</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 19589,
      "author_name": "issamlaradji",
      "author_url": "",
      "post_date": "01/22/2013 16:14:29",
      "content": "<p>@<a class=\"x_x_profilelink\" title=\"View Erik Nordeus's profile\" href=\"http://www.kaggle.com/users/69997/erik-nordeus\">Erik Nordeus</a>&nbsp;Thank you very much for the fast response, but by storing the data in MySQL are you able to directly process that data\r\n using Python? If so, please kindly give me some clues on how to do that! Thanks a lot! :)</p>\r\n<p>Best regards!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 19591,
      "author_name": "eriknordeus",
      "author_url": "",
      "post_date": "01/22/2013 16:44:10",
      "content": "<p>You should watch this video tutorial:&nbsp;<a title=\"Python MySQLdb Quickstart\" href=\"http://youtu.be/ooIB8ZJattw\">Python MySQLdb Quickstart</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 19594,
      "author_name": "issamlaradji",
      "author_url": "",
      "post_date": "01/22/2013 17:53:46",
      "content": "<p>Thanks! I'll watch it noW!! :) :) :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "19583": "",
    "19584": "",
    "19589": "",
    "19591": "",
    "19594": ""
  },
  "source": "meta"
}