{
  "id": 6244,
  "title": "I can't load data",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/6244",
  "author_name": "",
  "post_date": "2013-11-06T16:22:35.393Z",
  "votes": null,
  "comment_count": 11,
  "views": 2829,
  "content": "<p>I use GATE and I can't load the data. Neither in Excel, the file is too large. How can I load it?</p>\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "33324",
      "postDate": "11/06/2013 16:22:35",
      "content": "<p>I use GATE and I can't load the data. Neither in Excel, the file is too large. How can I load it?</p>\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33325",
      "postDate": "11/06/2013 16:24:22",
      "content": "<p>I'm in the same situation.&nbsp;</p>\n<p>Please help :/</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33333",
      "postDate": "11/06/2013 18:47:24",
      "content": "<p>I would look into using a Amazon EC2 instance (although I am still running into RAM issues training algorithms).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33422",
      "postDate": "11/08/2013 13:50:42",
      "content": "<p>Under Linux/MacOS/Cygwin type</p>\n<p style=\"padding-left: 30px\">head -100001 Train.csv &gt; TrainSmall.csv</p>\n<p>This will give you a subset of the first 100k questions, you can use that subset to prototype your algorithm.</p>\n<p>Otherwise, if you really have no clue how to handle such a big dataset, this competition is probably not the right one for you. I don't know about GATE, but most off-the-shelf software can not deal with a dataset that does not fit in memory. Most likely the winning entry will require some programming expertise.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33445",
      "postDate": "11/08/2013 22:20:42",
      "content": "<p>[quote=Maarten Bosma;33422]</p>\n<p><span style=\"line-height: 1.4\">Otherwise, if you really have no clue how to handle such a big dataset, this competition is probably not the right one for you. I don't know about GATE, but most off-the-shelf software can not deal with a dataset that does not fit in memory. Most likely the winning entry will require some programming expertise.</span></p>\n<p>[/quote]</p>\n<p>I understand what you're saying, but then again, I wouldn't want beginners to not try their hand on the dataset. Yes, it is a large dataset. Yes, even after loading the dataset there's a bigger problem of processing the data. But that should not discourage one to try working on this.&nbsp;</p>\n<p>Here are some more hints to add to Maarten's training on the head of the file idea:</p>\n<p>1. See if you can use a model that does not require you to load everything to the RAM.&nbsp;</p>\n<p>2. Look up Online Machine Learning - see if that leads you anywhere.</p>\n<p>3. Even if you decide to do batch learning, see if you can split the processing into parts. Do A and then save A(data) to disk. Do B on A(data) and then save B(.) to disk, so on..</p>\n<p>4. Use domain knowledge to see if there are ways in which you can write a smaller and memory friendlier model yourself. Remember: it doesn't need to hit the top of the leaderboard.</p>\n<p>All said and done, these are just a few of my thoughts. I am pretty much new to this as well. I learned how to handle large datasets only after trial and error. So people with more experience feel free to add/delete to/from my list.&nbsp;</p>\n<p>TL;DR Don't let the size of the data stop you from participating.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33461",
      "postDate": "11/09/2013 12:30:45",
      "content": "<p>Ok. Here are some tools, which might be helpful to handle large datasets:</p>\n<ol>\n<li>Apache Mamout</li>\n<li>Apache Lucene</li>\n<li>Vowpal Wabbit</li>\n<li>Sofia</li>\n<li><a href=\"http://scikit-learn.org/stable/auto_examples/applications/plot_out_of_core_classification.html\">Sci-kit learn</a></li>\n</ol>\n<p>Note, that the document term matrix should actually fit in memory on most computers/laptops.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33465",
      "postDate": "11/09/2013 15:43:05",
      "content": "<p>I suggest starting with just a few thousand observations, that is enough to produce a submission that beats the benchmarks. You will have to split the test set into pieces to score it though. I would not give up just because the dataset is big, it is a very interesting problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33541",
      "postDate": "11/11/2013 06:57:34",
      "content": "<p>how about using the pandas package in python for loading the data???</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33574",
      "postDate": "11/11/2013 17:02:02",
      "content": "<p>Pandas can read the entire file(although it throws errors for me if I try and take a subset), but it'll still potentially give memory problems.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35021",
      "postDate": "11/22/2013 00:45:38",
      "content": "<p>Hi Everyone,</p>\n<p>I am a student and a beginner in data science and started with this competition just now as it is our class project (Information Retrieval) . I am using mongoDB and loaded the data in it. Now I am thinking to use python NLTK for rest of the job. This is trial and error method but I am not sure if there is enough time left for it.</p>\n<p>Can you please guide me and tell me if I am going in the right direction?</p>\n<p>Should I go for Hadoop and Mahaout Instead?</p>\n<p>Or should I use Pandas?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35233",
      "postDate": "11/25/2013 21:52:18",
      "content": "<p>Postgresql is very good at handling big data if one does not have a great hardware. Give it a try .</p>\n<p>Then you can subset it if needed.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35295",
      "postDate": "11/26/2013 15:51:04",
      "content": "<p>Thank you to all! I read all and I am trying something..</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 33325,
      "author_name": "annavlk31",
      "author_url": "",
      "post_date": "11/06/2013 16:24:22",
      "content": "<p>I'm in the same situation.&nbsp;</p>\n<p>Please help :/</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33333,
      "author_name": "telser",
      "author_url": "",
      "post_date": "11/06/2013 18:47:24",
      "content": "<p>I would look into using a Amazon EC2 instance (although I am still running into RAM issues training algorithms).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33422,
      "author_name": "ma2rten",
      "author_url": "",
      "post_date": "11/08/2013 13:50:42",
      "content": "<p>Under Linux/MacOS/Cygwin type</p>\n<p style=\"padding-left: 30px\">head -100001 Train.csv &gt; TrainSmall.csv</p>\n<p>This will give you a subset of the first 100k questions, you can use that subset to prototype your algorithm.</p>\n<p>Otherwise, if you really have no clue how to handle such a big dataset, this competition is probably not the right one for you. I don't know about GATE, but most off-the-shelf software can not deal with a dataset that does not fit in memory. Most likely the winning entry will require some programming expertise.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33445,
      "author_name": "rohits",
      "author_url": "",
      "post_date": "11/08/2013 22:20:42",
      "content": "<p>[quote=Maarten Bosma;33422]</p>\n<p><span style=\"line-height: 1.4\">Otherwise, if you really have no clue how to handle such a big dataset, this competition is probably not the right one for you. I don't know about GATE, but most off-the-shelf software can not deal with a dataset that does not fit in memory. Most likely the winning entry will require some programming expertise.</span></p>\n<p>[/quote]</p>\n<p>I understand what you're saying, but then again, I wouldn't want beginners to not try their hand on the dataset. Yes, it is a large dataset. Yes, even after loading the dataset there's a bigger problem of processing the data. But that should not discourage one to try working on this.&nbsp;</p>\n<p>Here are some more hints to add to Maarten's training on the head of the file idea:</p>\n<p>1. See if you can use a model that does not require you to load everything to the RAM.&nbsp;</p>\n<p>2. Look up Online Machine Learning - see if that leads you anywhere.</p>\n<p>3. Even if you decide to do batch learning, see if you can split the processing into parts. Do A and then save A(data) to disk. Do B on A(data) and then save B(.) to disk, so on..</p>\n<p>4. Use domain knowledge to see if there are ways in which you can write a smaller and memory friendlier model yourself. Remember: it doesn't need to hit the top of the leaderboard.</p>\n<p>All said and done, these are just a few of my thoughts. I am pretty much new to this as well. I learned how to handle large datasets only after trial and error. So people with more experience feel free to add/delete to/from my list.&nbsp;</p>\n<p>TL;DR Don't let the size of the data stop you from participating.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33461,
      "author_name": "ma2rten",
      "author_url": "",
      "post_date": "11/09/2013 12:30:45",
      "content": "<p>Ok. Here are some tools, which might be helpful to handle large datasets:</p>\n<ol>\n<li>Apache Mamout</li>\n<li>Apache Lucene</li>\n<li>Vowpal Wabbit</li>\n<li>Sofia</li>\n<li><a href=\"http://scikit-learn.org/stable/auto_examples/applications/plot_out_of_core_classification.html\">Sci-kit learn</a></li>\n</ol>\n<p>Note, that the document term matrix should actually fit in memory on most computers/laptops.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33465,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "11/09/2013 15:43:05",
      "content": "<p>I suggest starting with just a few thousand observations, that is enough to produce a submission that beats the benchmarks. You will have to split the test set into pieces to score it though. I would not give up just because the dataset is big, it is a very interesting problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33541,
      "author_name": "sword19",
      "author_url": "",
      "post_date": "11/11/2013 06:57:34",
      "content": "<p>how about using the pandas package in python for loading the data???</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33574,
      "author_name": "telser",
      "author_url": "",
      "post_date": "11/11/2013 17:02:02",
      "content": "<p>Pandas can read the entire file(although it throws errors for me if I try and take a subset), but it'll still potentially give memory problems.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35021,
      "author_name": "ashutoshsharma1",
      "author_url": "",
      "post_date": "11/22/2013 00:45:38",
      "content": "<p>Hi Everyone,</p>\n<p>I am a student and a beginner in data science and started with this competition just now as it is our class project (Information Retrieval) . I am using mongoDB and loaded the data in it. Now I am thinking to use python NLTK for rest of the job. This is trial and error method but I am not sure if there is enough time left for it.</p>\n<p>Can you please guide me and tell me if I am going in the right direction?</p>\n<p>Should I go for Hadoop and Mahaout Instead?</p>\n<p>Or should I use Pandas?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35233,
      "author_name": "root3d",
      "author_url": "",
      "post_date": "11/25/2013 21:52:18",
      "content": "<p>Postgresql is very good at handling big data if one does not have a great hardware. Give it a try .</p>\n<p>Then you can subset it if needed.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35295,
      "author_name": "jorgeblasco",
      "author_url": "",
      "post_date": "11/26/2013 15:51:04",
      "content": "<p>Thank you to all! I read all and I am trying something..</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "33324": "",
    "33325": "",
    "33333": "",
    "33422": "",
    "33445": "",
    "33461": "",
    "33465": "",
    "33541": "",
    "33574": "",
    "35021": "",
    "35233": "",
    "35295": ""
  },
  "source": "meta"
}