{
  "id": 6014,
  "title": "Dealing with Large Data Set",
  "url": "/competitions/expedia-personalized-sort/discussion/6014",
  "author_name": "",
  "post_date": "2013-10-12T00:58:08.247Z",
  "votes": null,
  "comment_count": 3,
  "views": 2349,
  "content": "<p>I am wondering how some of you guys are dealing with the large data set. I try and open the train.csv file but my computer doesn't seem to be able to handle the size. I have a Macbook Pro 8GB RAM. Any suggestions?</p>\n<p>Total newb :) Thanks.</p>",
  "messages": [
    {
      "id": "32276",
      "postDate": "10/12/2013 00:58:08",
      "content": "<p>I am wondering how some of you guys are dealing with the large data set. I try and open the train.csv file but my computer doesn't seem to be able to handle the size. I have a Macbook Pro 8GB RAM. Any suggestions?</p>\n<p>Total newb :) Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32279",
      "postDate": "10/12/2013 03:29:07",
      "content": "<p>https://www.kaggle.com/c/expedia-personalized-sort/forums/t/5834/simplest-way-to-view-the-csv-files</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32373",
      "postDate": "10/16/2013 18:41:54",
      "content": "<p>My first competition I only had 8 Gigs of Ram. I had so many problems that maxed my machines Ram to 16 Gigs. I still only have 16 Gigs to work with, and it is one my constraint. I am fairly novice, so I think learning to deal with less ram is educational, but I wish I had more. Hint: taking many small bites is sometimes a solution. I would love to hear advice by someone who is not as novice as I am.</p>\n<p>Sorry, I misunderstood the question. It is better answered by the link before me. My answer was for training a model which also has memory constraints.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32374",
      "postDate": "10/16/2013 19:06:13",
      "content": "<p>Everyone's computing limitations and personal preferences are different so there isn't a single answer here. Since you can't even load the dataset into memory I'm assuming that you're using R, here are a few potential approches:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">Use packages that deal with large data sets better than base R, for example fread() and data tables will go a lot faster and require much less memory than read.csv() and data frames.&nbsp;</span></li>\n<li><span style=\"line-height: 1.4\">Use a language that handles memory better than R like Python, Java or C++.</span></li>\n<li><span style=\"line-height: 1.4\">Don't use (or even load) all of the training data. You aren't going to be able to run many types of models with millions of rows of data if you memory limitations anyway and the large test set can be loaded into memory in chunks if needed.</span></li>\n<li><span style=\"line-height: 1.4\">Rent a bigger machine through something like AWS.<br></span></li>\n</ul>\n<p>&nbsp;</p>\n<p>Clearly you can combine many of these options, along with other approaches, but hopefully that should give a few options to explore.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 32279,
      "author_name": "jay670790",
      "author_url": "",
      "post_date": "10/12/2013 03:29:07",
      "content": "<p>https://www.kaggle.com/c/expedia-personalized-sort/forums/t/5834/simplest-way-to-view-the-csv-files</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32373,
      "author_name": "joexjmmvhm",
      "author_url": "",
      "post_date": "10/16/2013 18:41:54",
      "content": "<p>My first competition I only had 8 Gigs of Ram. I had so many problems that maxed my machines Ram to 16 Gigs. I still only have 16 Gigs to work with, and it is one my constraint. I am fairly novice, so I think learning to deal with less ram is educational, but I wish I had more. Hint: taking many small bites is sometimes a solution. I would love to hear advice by someone who is not as novice as I am.</p>\n<p>Sorry, I misunderstood the question. It is better answered by the link before me. My answer was for training a model which also has memory constraints.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32374,
      "author_name": "dmcgarry",
      "author_url": "",
      "post_date": "10/16/2013 19:06:13",
      "content": "<p>Everyone's computing limitations and personal preferences are different so there isn't a single answer here. Since you can't even load the dataset into memory I'm assuming that you're using R, here are a few potential approches:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">Use packages that deal with large data sets better than base R, for example fread() and data tables will go a lot faster and require much less memory than read.csv() and data frames.&nbsp;</span></li>\n<li><span style=\"line-height: 1.4\">Use a language that handles memory better than R like Python, Java or C++.</span></li>\n<li><span style=\"line-height: 1.4\">Don't use (or even load) all of the training data. You aren't going to be able to run many types of models with millions of rows of data if you memory limitations anyway and the large test set can be loaded into memory in chunks if needed.</span></li>\n<li><span style=\"line-height: 1.4\">Rent a bigger machine through something like AWS.<br></span></li>\n</ul>\n<p>&nbsp;</p>\n<p>Clearly you can combine many of these options, along with other approaches, but hopefully that should give a few options to explore.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "32276": "",
    "32279": "",
    "32373": "",
    "32374": ""
  },
  "source": "meta"
}