{ "id": 5673, "title": "How to import 2.19 GB data in R in low configuration machine ? ", "url": "/competitions/expedia-personalized-sort/discussion/5673", "author_name": "", "post_date": "2013-09-06T06:30:30.647Z", "votes": null, "comment_count": 4, "views": 5762, "content": "

Windows 7 , 4 - 8 GB RAM , 2.2 GHz, Dual Core  taking infinite to import .

\n

Any suggestion on how to manager the import in chunks or method to overcome if using low configuration machine. 

\n

Takes infinite for importing how much it will for predicting. :). 

\n

 

\n

Thanks !

", "messages": [ { "id": "30279", "postDate": "09/06/2013 06:30:30", "content": "

Windows 7 , 4 - 8 GB RAM , 2.2 GHz, Dual Core  taking infinite to import .

\n

Any suggestion on how to manager the import in chunks or method to overcome if using low configuration machine. 

\n

Takes infinite for importing how much it will for predicting. :). 

\n

 

\n

Thanks !

", "rawMarkdown": "", "votes": null }, { "id": "30281", "postDate": "09/06/2013 06:42:55", "content": "

Use fread() in the data.table package.

\n

 

\n

See benchmarks (and further alternatives) here: http://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r

", "rawMarkdown": "", "votes": null }, { "id": "30397", "postDate": "09/07/2013 02:00:11", "content": "

Yogesh: I used the following code:

\n


memory.limit(1E8)

\n

#I will igonore for the moment time of day

\n

dateconv=setClass("dateconv")

\n

setAs("character","dateconv", function(from) as.numeric(as.Date(from, format="%Y-%m-%d %H:%M:%S") ) )

\n

#NA coded as NULL . Read a little bit (200)

\n

train <- read.csv("train.csv", header=TRUE, nrows=200,sep = ",", quote="\\"", dec=".",na.strings="NULL")

\n

classes = sapply(train,class)

\n

# I change logical variables to integer -not needed

\n

classes[classes=="logical"]<- "integer"

\n

# train

\n

classes[c(5, 6, 10, 12, 13, 14, 16, 25, 26, 30, 33, 36, 39, 42, 45, 48, 51, 53)]="numeric"

\n

#test

\n

# classes[c(5, 6,10, 12, 13, 14, 15, 24, 25, 29, 32, 35, 38, 41, 44, 47, 50)]="numeric"

\n

classes[2] <- "dateconv"

\n

#Read all

\n

train <- read.csv("train.csv", header=TRUE, colClasses=classes,sep = ",", quote="\\"", dec=".",na.strings="NULL")

\n

object.size(train)

\n

# 2816585496 bytes

\n

dim(train)

\n

# 9917530 54

\n

 

\n

It takes some minutes to read it. After import and saving. my machine almost fainted. Be sure that you have enough virtual memory. If you save the session with train data, the size of file is ca, 193500KB

\n

I hope it Works for you too.

", "rawMarkdown": "", "votes": null }, { "id": "31062", "postDate": "09/17/2013 21:26:58", "content": "

If you don't want to overload your pc you can always use a postgres database for data storage and use the RODBC package in R to query parts of the data with SQL syntax.

\n

Below 2 sql files to import train.csv and test.csv into postgres.

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31316", "postDate": "09/21/2013 20:36:38", "content": "

[quote=Yogesh;30279]

\n

Windows 7 , 4 - 8 GB RAM , 2.2 GHz, Dual Core  taking infinite to import .

\n

Any suggestion on how to manager the import in chunks or method to overcome if using low configuration machine. 

\n

Takes infinite for importing how much it will for predicting. :). 

\n

 

\n

Thanks !

\n

[/quote]

\n

Hi below code I used to read 7 GB of Facebook competition data.

\n

 

\n

library(ff)

\n

train <- read.csv.ffdf(file="Train.csv",header=TRUE,VERBOSE=TRUE,first.rows=100000,next.rows=100000,colClasses=NA)

\n

 

\n

It took 37 min to read the file in R

\n

I used same code to read Expedia data also. I think it took 15-20 min (don't remember exactly). It won't consume more than 4 GB of RAM (may work in 3 GB of RAM also)

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30281, "author_name": "mm247601", "author_url": "", "post_date": "09/06/2013 06:42:55", "content": "

Use fread() in the data.table package.

\n

 

\n

See benchmarks (and further alternatives) here: http://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r

", "votes": null, "replies": [] }, { "id": 30397, "author_name": "gmobaz", "author_url": "", "post_date": "09/07/2013 02:00:11", "content": "

Yogesh: I used the following code:

\n


memory.limit(1E8)

\n

#I will igonore for the moment time of day

\n

dateconv=setClass("dateconv")

\n

setAs("character","dateconv", function(from) as.numeric(as.Date(from, format="%Y-%m-%d %H:%M:%S") ) )

\n

#NA coded as NULL . Read a little bit (200)

\n

train <- read.csv("train.csv", header=TRUE, nrows=200,sep = ",", quote="\\"", dec=".",na.strings="NULL")

\n

classes = sapply(train,class)

\n

# I change logical variables to integer -not needed

\n

classes[classes=="logical"]<- "integer"

\n

# train

\n

classes[c(5, 6, 10, 12, 13, 14, 16, 25, 26, 30, 33, 36, 39, 42, 45, 48, 51, 53)]="numeric"

\n

#test

\n

# classes[c(5, 6,10, 12, 13, 14, 15, 24, 25, 29, 32, 35, 38, 41, 44, 47, 50)]="numeric"

\n

classes[2] <- "dateconv"

\n

#Read all

\n

train <- read.csv("train.csv", header=TRUE, colClasses=classes,sep = ",", quote="\\"", dec=".",na.strings="NULL")

\n

object.size(train)

\n

# 2816585496 bytes

\n

dim(train)

\n

# 9917530 54

\n

 

\n

It takes some minutes to read it. After import and saving. my machine almost fainted. Be sure that you have enough virtual memory. If you save the session with train data, the size of file is ca, 193500KB

\n

I hope it Works for you too.

", "votes": null, "replies": [] }, { "id": 31062, "author_name": "mennotaanman", "author_url": "", "post_date": "09/17/2013 21:26:58", "content": "

If you don't want to overload your pc you can always use a postgres database for data storage and use the RODBC package in R to query parts of the data with SQL syntax.

\n

Below 2 sql files to import train.csv and test.csv into postgres.

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 31316, "author_name": "thakurrajanand", "author_url": "", "post_date": "09/21/2013 20:36:38", "content": "

[quote=Yogesh;30279]

\n

Windows 7 , 4 - 8 GB RAM , 2.2 GHz, Dual Core  taking infinite to import .

\n

Any suggestion on how to manager the import in chunks or method to overcome if using low configuration machine. 

\n

Takes infinite for importing how much it will for predicting. :). 

\n

 

\n

Thanks !

\n

[/quote]

\n

Hi below code I used to read 7 GB of Facebook competition data.

\n

 

\n

library(ff)

\n

train <- read.csv.ffdf(file="Train.csv",header=TRUE,VERBOSE=TRUE,first.rows=100000,next.rows=100000,colClasses=NA)

\n

 

\n

It took 37 min to read the file in R

\n

I used same code to read Expedia data also. I think it took 15-20 min (don't remember exactly). It won't consume more than 4 GB of RAM (may work in 3 GB of RAM also)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30279": "", "30281": "", "30397": "", "31062": "", "31316": "" }, "source": "meta" }