{
  "id": 6618,
  "title": "What DBMS are you people using?",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/6618",
  "author_name": "",
  "post_date": "2013-12-18T00:53:13.700Z",
  "votes": null,
  "comment_count": 7,
  "views": 2909,
  "content": "<p>Hi, I am new to kaggle. I have worked on this competition and am able to get some results. The only problem is the data being to large and for every computation my system takes a day.</p>\n<p>I am fetching data from csv and storing in csv which is not efficient. Please suggest me a database that is fast enough for this data.</p>",
  "messages": [
    {
      "id": "36276",
      "postDate": "12/18/2013 00:53:13",
      "content": "<p>Hi, I am new to kaggle. I have worked on this competition and am able to get some results. The only problem is the data being to large and for every computation my system takes a day.</p>\n<p>I am fetching data from csv and storing in csv which is not efficient. Please suggest me a database that is fast enough for this data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36278",
      "postDate": "12/18/2013 01:34:58",
      "content": "<p>Postgresql is free and supports the csv rfc (meaning it will read the carriage returns embedded in the files correctly). &nbsp;See the forum post on this:</p>\n<p>&nbsp;</p>\n<p>http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5606/sql-script-to-load-data-into-postgresql-db</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36280",
      "postDate": "12/18/2013 03:48:38",
      "content": "<p>Thank You James, I will try Postgresql, Due to less time, I asked for the suggestion rather then trying various DBMS. Its nice to hear from someone of your rank.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36284",
      "postDate": "12/18/2013 05:13:54",
      "content": "<p>It is not essential to use DB.</p>\n<p>I used csv modules in python, created some data structures which would be needed for prediction in 1-2 hours and saved them to disk by serializing(pickle).</p>\n<p>For prediction, I load the pickled file and it hardly takes 1-2 minutes to use them and make prediction.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36288",
      "postDate": "12/18/2013 08:39:44",
      "content": "<p>Thank you Raghavendra, but is the pickle more efficient or the DB? I was using the csv modules in python itself but was reading from csv and also writing into csv. That is taking very long. May be my code is not that efficient.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36290",
      "postDate": "12/18/2013 09:13:22",
      "content": "<p>For this task, I felt a disk/DB reliant approach might not scale unless you have very good hardware/cluster (may be wrong, some people have been using postgres). Using DB would definitely keep your options flexible.</p>\n<p>&nbsp;</p>\n<p>Reading/writing CSV should not take much time, may be the computation that you are doing in between per row is compute intensive.</p>\n<p>&nbsp;</p>\n<p>Pickling is an option if you want to divide your task into two parts and want to manage everything in RAM.</p>\n<p>1. Preparation phase : Where you will clean the data, remove stop words and convert the data into multiple data-structures which could be used in prediction.&nbsp;</p>\n<p>This you might want to do only once. But depending on the structures you will store, it limits your options/logic for prediction.</p>\n<p>2. Prediction phase : You can come up with multiple approaches to use the saved structures to make prediction.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36293",
      "postDate": "12/18/2013 09:28:01",
      "content": "<p>I have done the preparation phase. I made a dictionary of train data after removing all irrelevant things. But when I am doing the prediction task, row by row comparison and computation is done which is taking too much time.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36304",
      "postDate": "12/18/2013 12:00:08",
      "content": "<p>I've used <a href=\"http://code.google.com/p/leveldb/\">leveldb</a>. It's pretty fast and awesome.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 36278,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "12/18/2013 01:34:58",
      "content": "<p>Postgresql is free and supports the csv rfc (meaning it will read the carriage returns embedded in the files correctly). &nbsp;See the forum post on this:</p>\n<p>&nbsp;</p>\n<p>http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5606/sql-script-to-load-data-into-postgresql-db</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36280,
      "author_name": "ashutoshsharma1",
      "author_url": "",
      "post_date": "12/18/2013 03:48:38",
      "content": "<p>Thank You James, I will try Postgresql, Due to less time, I asked for the suggestion rather then trying various DBMS. Its nice to hear from someone of your rank.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36284,
      "author_name": "raghud",
      "author_url": "",
      "post_date": "12/18/2013 05:13:54",
      "content": "<p>It is not essential to use DB.</p>\n<p>I used csv modules in python, created some data structures which would be needed for prediction in 1-2 hours and saved them to disk by serializing(pickle).</p>\n<p>For prediction, I load the pickled file and it hardly takes 1-2 minutes to use them and make prediction.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36288,
      "author_name": "ashutoshsharma1",
      "author_url": "",
      "post_date": "12/18/2013 08:39:44",
      "content": "<p>Thank you Raghavendra, but is the pickle more efficient or the DB? I was using the csv modules in python itself but was reading from csv and also writing into csv. That is taking very long. May be my code is not that efficient.</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36290,
      "author_name": "raghud",
      "author_url": "",
      "post_date": "12/18/2013 09:13:22",
      "content": "<p>For this task, I felt a disk/DB reliant approach might not scale unless you have very good hardware/cluster (may be wrong, some people have been using postgres). Using DB would definitely keep your options flexible.</p>\n<p>&nbsp;</p>\n<p>Reading/writing CSV should not take much time, may be the computation that you are doing in between per row is compute intensive.</p>\n<p>&nbsp;</p>\n<p>Pickling is an option if you want to divide your task into two parts and want to manage everything in RAM.</p>\n<p>1. Preparation phase : Where you will clean the data, remove stop words and convert the data into multiple data-structures which could be used in prediction.&nbsp;</p>\n<p>This you might want to do only once. But depending on the structures you will store, it limits your options/logic for prediction.</p>\n<p>2. Prediction phase : You can come up with multiple approaches to use the saved structures to make prediction.</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36293,
      "author_name": "ashutoshsharma1",
      "author_url": "",
      "post_date": "12/18/2013 09:28:01",
      "content": "<p>I have done the preparation phase. I made a dictionary of train data after removing all irrelevant things. But when I am doing the prediction task, row by row comparison and computation is done which is taking too much time.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36304,
      "author_name": "renz85",
      "author_url": "",
      "post_date": "12/18/2013 12:00:08",
      "content": "<p>I've used <a href=\"http://code.google.com/p/leveldb/\">leveldb</a>. It's pretty fast and awesome.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "36276": "",
    "36278": "",
    "36280": "",
    "36284": "",
    "36288": "",
    "36290": "",
    "36293": "",
    "36304": ""
  },
  "source": "meta"
}