{
  "id": 5599,
  "title": "Breaking Train data into 4 files",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5599",
  "author_name": "",
  "post_date": "2013-08-31T09:49:01.983Z",
  "votes": null,
  "comment_count": 3,
  "views": 2616,
  "content": "<p>Is it Possible for the Competition Host&nbsp;to break train data into 4 files so that it become easy to read data into R. I know colbycol package can be use to read particular percentage of data but it is causing differnce in number of rows read for each column</p>",
  "messages": [
    {
      "id": "29836",
      "postDate": "08/31/2013 09:49:01",
      "content": "<p>Is it Possible for the Competition Host&nbsp;to break train data into 4 files so that it become easy to read data into R. I know colbycol package can be use to read particular percentage of data but it is causing differnce in number of rows read for each column</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30062",
      "postDate": "09/03/2013 15:10:09",
      "content": "<p>Dealing with a large-ish data set is part of the challenge. &nbsp;We will not be spoon feeding it to you :-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30175",
      "postDate": "09/04/2013 18:36:11",
      "content": "<p>[quote=William Cukierski;30062]</p>\n<p>Dealing with a large-ish data set is part of the challenge. &nbsp;We will not be spoon feeding it to you :-)</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>Okie Dokie William</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32074",
      "postDate": "10/07/2013 07:31:48",
      "content": "<p>I don't know how it would work with R (I like and use R, but not for this challenge), but cutting the data file in peaces could be easily done with a script (e.g. Python). However, it is also possible to deal with it <em>without</em> cutting it: taking small steps, working with it line by line, word by word, tag by tag. I already made few approaches and finally was able to deal with it without cutting the file into peaces. Subsetting also slows down the process, because you have to read, analyze and write already big files. You could also think on decoding the data to some computer- and memory-friendly format, it helps. Think of it in a map-reduce manner. Try few approaches, experiment with it ;)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 30062,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "09/03/2013 15:10:09",
      "content": "<p>Dealing with a large-ish data set is part of the challenge. &nbsp;We will not be spoon feeding it to you :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30175,
      "author_name": "thakurrajanand",
      "author_url": "",
      "post_date": "09/04/2013 18:36:11",
      "content": "<p>[quote=William Cukierski;30062]</p>\n<p>Dealing with a large-ish data set is part of the challenge. &nbsp;We will not be spoon feeding it to you :-)</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>Okie Dokie William</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32074,
      "author_name": "tymekw",
      "author_url": "",
      "post_date": "10/07/2013 07:31:48",
      "content": "<p>I don't know how it would work with R (I like and use R, but not for this challenge), but cutting the data file in peaces could be easily done with a script (e.g. Python). However, it is also possible to deal with it <em>without</em> cutting it: taking small steps, working with it line by line, word by word, tag by tag. I already made few approaches and finally was able to deal with it without cutting the file into peaces. Subsetting also slows down the process, because you have to read, analyze and write already big files. You could also think on decoding the data to some computer- and memory-friendly format, it helps. Think of it in a map-reduce manner. Try few approaches, experiment with it ;)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29836": "",
    "30062": "",
    "30175": "",
    "32074": ""
  },
  "source": "meta"
}