{
  "id": 8258,
  "title": "fread function in R",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/8258",
  "author_name": "",
  "post_date": "2014-05-23T02:51:30.510Z",
  "votes": null,
  "comment_count": 5,
  "views": 2801,
  "content": "<p>Hi everyone.</p>\n<p>I've seen that many people in this forum suggest fread function to read transaction.csv. However, my computer couldn't even read. So I was wondering if any of you who successfully load the csv file could share how much ram that is setup in your computer. By the way, I am using 8gb ram so if anyone of you who has 8gb ram will not able to load transaction.csv in R, you might want to find other way to figure it out :( &nbsp;</p>",
  "messages": [
    {
      "id": "46104",
      "postDate": "05/23/2014 02:51:30",
      "content": "<p>Hi everyone.</p>\n<p>I've seen that many people in this forum suggest fread function to read transaction.csv. However, my computer couldn't even read. So I was wondering if any of you who successfully load the csv file could share how much ram that is setup in your computer. By the way, I am using 8gb ram so if anyone of you who has 8gb ram will not able to load transaction.csv in R, you might want to find other way to figure it out :( &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "46108",
      "postDate": "05/23/2014 03:21:06",
      "content": "<p>I believe the best way is to use the read.big.matrix bigmemory library in R but the challenge is it does not support strings. The alternate is to convert the date into julian and the product measurement to its key and then use the bigmemory for manipulation. I posted a python script to do the same but on windows platform thats was also getting slow, the best way I believe is to split it into 1GB file and then do the manipulation.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "47094",
      "postDate": "05/26/2014 16:18:42",
      "content": "<p>I'd suggest you use SQL server to import transaction data followed by using RODBC library to connect.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "47213",
      "postDate": "05/29/2014 02:24:10",
      "content": "<p>I used fread to read the file in 4 chunks. Use &nbsp;the skip argument of fread as appropriate. Filtering each chunk to remove customers not in say the train set reduced the size to one that can be loaded into memory.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "48882",
      "postDate": "06/09/2014 03:47:06",
      "content": "<p>I'm so glad I wasn't the only one having problems reading this in.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49102",
      "postDate": "06/13/2014 14:15:48",
      "content": "<p>fread works with 16GB ram, otherwise you have to break into chunks as Rokoson suggests</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 46108,
      "author_name": "vishwanathanraman",
      "author_url": "",
      "post_date": "05/23/2014 03:21:06",
      "content": "<p>I believe the best way is to use the read.big.matrix bigmemory library in R but the challenge is it does not support strings. The alternate is to convert the date into julian and the product measurement to its key and then use the bigmemory for manipulation. I posted a python script to do the same but on windows platform thats was also getting slow, the best way I believe is to split it into 1GB file and then do the manipulation.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 47094,
      "author_name": "namesj",
      "author_url": "",
      "post_date": "05/26/2014 16:18:42",
      "content": "<p>I'd suggest you use SQL server to import transaction data followed by using RODBC library to connect.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 47213,
      "author_name": "omoekan09",
      "author_url": "",
      "post_date": "05/29/2014 02:24:10",
      "content": "<p>I used fread to read the file in 4 chunks. Use &nbsp;the skip argument of fread as appropriate. Filtering each chunk to remove customers not in say the train set reduced the size to one that can be loaded into memory.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 48882,
      "author_name": "penguingirl",
      "author_url": "",
      "post_date": "06/09/2014 03:47:06",
      "content": "<p>I'm so glad I wasn't the only one having problems reading this in.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49102,
      "author_name": "numinary",
      "author_url": "",
      "post_date": "06/13/2014 14:15:48",
      "content": "<p>fread works with 16GB ram, otherwise you have to break into chunks as Rokoson suggests</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "46104": "",
    "46108": "",
    "47094": "",
    "47213": "",
    "48882": "",
    "49102": ""
  },
  "source": "meta"
}