{ "id": 7883, "title": "R FREAD for large files", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7883", "author_name": "", "post_date": "2014-04-26T14:44:30.910Z", "votes": 29, "comment_count": 10, "views": 25956, "content": "

Just discovered R package data.table for reading in large files. If read.csv is taking too long:

\n

install.packages("data.table")

\n

library(data.table)

\n

train <- fread("train.csv")

\n

train<-as.data.frame(train)

\n

Took about 5 minutes compared to 2 hours with read.csv.

", "messages": [ { "id": "43081", "postDate": "04/26/2014 14:44:30", "content": "

Just discovered R package data.table for reading in large files. If read.csv is taking too long:

\n

install.packages("data.table")

\n

library(data.table)

\n

train <- fread("train.csv")

\n

train<-as.data.frame(train)

\n

Took about 5 minutes compared to 2 hours with read.csv.

", "rawMarkdown": "", "votes": null }, { "id": "43096", "postDate": "04/26/2014 19:03:55", "content": "

fread is an amazing function-- it's actually quicker to fread() the raw .csv than to save() and load() the dataset!

", "rawMarkdown": "", "votes": null }, { "id": "44462", "postDate": "05/13/2014 14:41:32", "content": "

wow great. thank you for your help !

", "rawMarkdown": "", "votes": null }, { "id": "44689", "postDate": "05/17/2014 02:08:51", "content": "

Use uncompressed RDS files after preparing data.table

\n

> start.time <- proc.time()
> saveRDS(transactions, file="data/transactions_uncomp.Rds", compress=F)
> end.time <- proc.time()
> cat("Elapsed; ", end.time[3]-start.time[3], "seconds.\\n")
Elapsed; 99.28 seconds.

\n\n

> start.time <- proc.time()
> transactions <- readRDS(file="data/transactions_uncomp.Rds")
> end.time <- proc.time()
> cat("Elapsed; ", end.time[3]-start.time[3], "seconds.\\n")
Elapsed; 102.93 seconds.

\n\n

> str(transactions)
Classes ‘data.table’ and 'data.frame': 349655789 obs. of 11 variables:
$ id : Factor w/ 311541 levels "100007447","100010021",..: 309397 309397 309397 309397 309397 309397 309397 309397 309397 309397 ...
$ chain : int 205 205 205 205 205 205 205 205 205 205 ...
$ dept : int 7 63 97 25 55 97 99 59 9 73 ...
$ category : int 707 6319 9753 2509 5555 9753 9909 5907 921 7344 ...
$ company : Factor w/ 32773 levels "10000","1010000010",..: 22881 21689 3401 23605 21897 2430 10518 7758 184 15840 ...
$ brand : int 12564 17876 0 31373 32094 0 15343 2012 9209 20285 ...
$ date : Date, format: "2012-03-02" "2012-03-02" ...
$ productsize : num 12 64 1 16 16 1 16 16 4 8 ...
$ productmeasure : Factor w/ 12 levels "","1","CT","FT",..: 8 8 3 8 8 3 8 8 8 3 ...
$ purchasequantity: int 1 1 1 1 2 1 1 1 2 1 ...
$ purchaseamount : num 7.59 1.59 5.99 1.99 10.38 ...
- attr(*, ".internal.selfref")=<externalptr>

", "rawMarkdown": "", "votes": null }, { "id": "49267", "postDate": "06/18/2014 03:26:22", "content": "

I agree this is an amazing function! I did have to add another argument to avoid getting nonsense in some fields with very large integer values.

\n\n

fread("transactions.csv",integer64="character")

", "rawMarkdown": "", "votes": null }, { "id": "49976", "postDate": "07/02/2014 21:12:23", "content": "

Just require(bit64) and then you'll see the bit64::integer64 type properly displayed.

", "rawMarkdown": "", "votes": null }, { "id": "57285", "postDate": "11/04/2014 09:48:13", "content": "

Hi I am learning data analysis on this project, I know the competition is over but learning for knowledge. For fread how can I see 64 bit integers as it is displaying some junk data?

\n

how do i get bit64?

\n

can someone help in it?

", "rawMarkdown": "", "votes": null }, { "id": "57286", "postDate": "11/04/2014 10:34:14", "content": "

print(DT)  # displays "junk" for integer64 column

\n

install.packages("bit64")

\n

require(bit64)

\n

print(DT)   # now displays integer64 nicely

", "rawMarkdown": "", "votes": null }, { "id": "57288", "postDate": "11/04/2014 11:09:47", "content": "

Wow Thanks a lot that works...:)

", "rawMarkdown": "", "votes": null }, { "id": "71210", "postDate": "04/10/2015 14:18:52", "content": "

Hi All,

\n

i am using Windows 7 OS in my 64 bit machine and installed R 3.0.2 version. i am using 8 GB RAM. I have installed data.table package 1.9.4 version. i am using fread() to read a 3 GB file which has 476147 rows and 556 columns.
 
when i read using fread() it makes R session aborted and gives R fatal error. when i am using read.csv() it reads it successfully but it take around 20 mins. 
 
Any body knows why fread() is not working in this case?

\n

- Solomon

", "rawMarkdown": "", "votes": null }, { "id": "71396", "postDate": "04/13/2015 17:54:37", "content": "

Please install v1.9.5 from GitHub and try again.  Currently there's a good chance it is fixed already. If it still crashes then rerun with verbose=TRUE and include the output in a new issue on GitHub. If the trace output is not enough and we need the file to reproduce, we'll let you know. Or include the file, or subset of, up front can help.

\n

- Matt

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 43096, "author_name": "zachmayer", "author_url": "", "post_date": "04/26/2014 19:03:55", "content": "

fread is an amazing function-- it's actually quicker to fread() the raw .csv than to save() and load() the dataset!

", "votes": null, "replies": [] }, { "id": 44462, "author_name": "taekkyunshin", "author_url": "", "post_date": "05/13/2014 14:41:32", "content": "

wow great. thank you for your help !

", "votes": null, "replies": [] }, { "id": 44689, "author_name": "databystander", "author_url": "", "post_date": "05/17/2014 02:08:51", "content": "

Use uncompressed RDS files after preparing data.table

\n

> start.time <- proc.time()
> saveRDS(transactions, file="data/transactions_uncomp.Rds", compress=F)
> end.time <- proc.time()
> cat("Elapsed; ", end.time[3]-start.time[3], "seconds.\\n")
Elapsed; 99.28 seconds.

\n\n

> start.time <- proc.time()
> transactions <- readRDS(file="data/transactions_uncomp.Rds")
> end.time <- proc.time()
> cat("Elapsed; ", end.time[3]-start.time[3], "seconds.\\n")
Elapsed; 102.93 seconds.

\n\n

> str(transactions)
Classes ‘data.table’ and 'data.frame': 349655789 obs. of 11 variables:
$ id : Factor w/ 311541 levels "100007447","100010021",..: 309397 309397 309397 309397 309397 309397 309397 309397 309397 309397 ...
$ chain : int 205 205 205 205 205 205 205 205 205 205 ...
$ dept : int 7 63 97 25 55 97 99 59 9 73 ...
$ category : int 707 6319 9753 2509 5555 9753 9909 5907 921 7344 ...
$ company : Factor w/ 32773 levels "10000","1010000010",..: 22881 21689 3401 23605 21897 2430 10518 7758 184 15840 ...
$ brand : int 12564 17876 0 31373 32094 0 15343 2012 9209 20285 ...
$ date : Date, format: "2012-03-02" "2012-03-02" ...
$ productsize : num 12 64 1 16 16 1 16 16 4 8 ...
$ productmeasure : Factor w/ 12 levels "","1","CT","FT",..: 8 8 3 8 8 3 8 8 8 3 ...
$ purchasequantity: int 1 1 1 1 2 1 1 1 2 1 ...
$ purchaseamount : num 7.59 1.59 5.99 1.99 10.38 ...
- attr(*, ".internal.selfref")=<externalptr>

", "votes": null, "replies": [] }, { "id": 49267, "author_name": "swhardy", "author_url": "", "post_date": "06/18/2014 03:26:22", "content": "

I agree this is an amazing function! I did have to add another argument to avoid getting nonsense in some fields with very large integer values.

\n\n

fread("transactions.csv",integer64="character")

", "votes": null, "replies": [] }, { "id": 49976, "author_name": "mattdowle", "author_url": "", "post_date": "07/02/2014 21:12:23", "content": "

Just require(bit64) and then you'll see the bit64::integer64 type properly displayed.

", "votes": null, "replies": [] }, { "id": 57285, "author_name": "shainanhemrajani", "author_url": "", "post_date": "11/04/2014 09:48:13", "content": "

Hi I am learning data analysis on this project, I know the competition is over but learning for knowledge. For fread how can I see 64 bit integers as it is displaying some junk data?

\n

how do i get bit64?

\n

can someone help in it?

", "votes": null, "replies": [] }, { "id": 57286, "author_name": "mattdowle", "author_url": "", "post_date": "11/04/2014 10:34:14", "content": "

print(DT)  # displays "junk" for integer64 column

\n

install.packages("bit64")

\n

require(bit64)

\n

print(DT)   # now displays integer64 nicely

", "votes": null, "replies": [] }, { "id": 57288, "author_name": "shainanhemrajani", "author_url": "", "post_date": "11/04/2014 11:09:47", "content": "

Wow Thanks a lot that works...:)

", "votes": null, "replies": [] }, { "id": 71210, "author_name": "solomonpushparaj", "author_url": "", "post_date": "04/10/2015 14:18:52", "content": "

Hi All,

\n

i am using Windows 7 OS in my 64 bit machine and installed R 3.0.2 version. i am using 8 GB RAM. I have installed data.table package 1.9.4 version. i am using fread() to read a 3 GB file which has 476147 rows and 556 columns.
 
when i read using fread() it makes R session aborted and gives R fatal error. when i am using read.csv() it reads it successfully but it take around 20 mins. 
 
Any body knows why fread() is not working in this case?

\n

- Solomon

", "votes": null, "replies": [] }, { "id": 71396, "author_name": "mattdowle", "author_url": "", "post_date": "04/13/2015 17:54:37", "content": "

Please install v1.9.5 from GitHub and try again.  Currently there's a good chance it is fixed already. If it still crashes then rerun with verbose=TRUE and include the output in a new issue on GitHub. If the trace output is not enough and we need the file to reproduce, we'll let you know. Or include the file, or subset of, up front can help.

\n

- Matt

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "43081": "", "43096": "", "44462": "", "44689": "", "49267": "", "49976": "", "57285": "", "57286": "", "57288": "", "71210": "", "71396": "" }, "source": "meta" }