{ "id": 8298, "title": "Trouble loading 'transactions.csv' into R", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/8298", "author_name": "", "post_date": "2014-05-28T07:39:32.230Z", "votes": null, "comment_count": 5, "views": 2573, "content": "

Anyone know a way to load 'transactions.csv' into R on a 32-bit 3GB RAM machine?

\n

Is it even possible? If not, would a 64-bit 4GB or a 64-bit 8GB RAM machines be able to handle it?

\n\n

thanks

", "messages": [ { "id": "47165", "postDate": "05/28/2014 07:39:32", "content": "

Anyone know a way to load 'transactions.csv' into R on a 32-bit 3GB RAM machine?

\n

Is it even possible? If not, would a 64-bit 4GB or a 64-bit 8GB RAM machines be able to handle it?

\n\n

thanks

", "rawMarkdown": "", "votes": null }, { "id": "47177", "postDate": "05/28/2014 12:19:13", "content": "

Am facing the same issue . I'm running 32bit OS and using RODBC to connect to transaction file.

\n

However, I'm able to work on only 200,000 Rows at a time . I get memory limitation error if i use more than that.

\n

I think we just can't beat the benchmark without utilizing the transaction file.

", "rawMarkdown": "", "votes": null }, { "id": "47218", "postDate": "05/29/2014 03:30:50", "content": "

Check this thread: https://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8258/fread-function-in-r

", "rawMarkdown": "", "votes": null }, { "id": "47300", "postDate": "05/30/2014 02:04:16", "content": "

On your question about 64 bit machine - the unpacked file is about 20Gb. One of my machines is a 64bit running Win7Pro64 bit with 16Gb of RAM. R runs out of RAM loading the transactions.csv directly. You have to use other methods, some of them were mentioned on this forum already.

", "rawMarkdown": "", "votes": null }, { "id": "48546", "postDate": "06/03/2014 03:30:32", "content": "

You cannot load a 20g file in 4 or 8g memory. You could put the file in a database or read the files chunk by chunk and filter the no so needed transactions. However, the methods mentioned in this forum take some possibly useful temporal transactions out.

", "rawMarkdown": "", "votes": null }, { "id": "48577", "postDate": "06/03/2014 22:17:19", "content": "

You can load, filter and save the big transaction file in Chunks similar to the python code posted on the other thread with very small foot print. (I did it on a 4G Win 32bit testing machine, with R footprint less than 100M memory, I think. But whether you can do any meaningful modeling, is another story.) However, R is not good at these tasks, R code can be significantly slower than python for doing this kind of things.

\n

Here is my sample code to load and filter and save the transaction files from and into gzip'ed files:

\n

options(stringsAsFactors=F)
offers=read.csv("Data/offers.csv.gz")
allcats=unique(offers$category)

\n

trainHistory=read.csv("Data/trainHistory.csv.gz")
train_ids=unique(trainHistory$id)
testHistory=read.csv("Data/testHistory.csv.gz")
test_ids=unique(testHistory$id)
all_ids=unique(c(train_ids, test_ids))

\n

gzf = gzfile("Data/reduced_df_quicksave.csv.gz","w")
con=file("Data/transactions.csv.gz",open='r')
blocksize=500000
rownum=blocksize
readnum=0
totalrows=0
keptrows=0
begTime=Sys.time()
headers=read.csv(con, nrows=1, header=F)
while(rownum >= blocksize) {
        tempdf=read.csv(con, nrows=blocksize, header=F)
        rownum=nrow(tempdf)
        totalrows=totalrows+rownum
        colnames(tempdf)=headers
        tempdf=tempdf[tempdf$category %in% allcats & tempdf$id %in% all_ids,]
        if (nrow(tempdf) > 0) {
                if (keptrows == 0) {
                        write.table(tempdf, row.names=F, col.names=TRUE, sep=",", file=gzf)
                } else {
                        write.table(tempdf, row.names=F, col.names=FALSE, sep=",", file=gzf)
                }
                keptrows=keptrows+nrow(tempdf)
        }
        rm(tempdf);gc()
        readnum=readnum+1
        cat(paste("Read",readnum,"blocks, total rows read:",format(totalrows,scientific=F),
                        ", total rows kept:", format(keptrows,scientific=F),
                        ", time used:",format(Sys.time()-begTime),"\\n"))
        flush.console()
}
close(con)

\n

close(gzf)
cat(paste("   time used:",format(Sys.time()-begTime),"\\n"))

\n

It took 40+ minutes on my windows machine, with the file located on a local hard drive.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 47177, "author_name": "namesj", "author_url": "", "post_date": "05/28/2014 12:19:13", "content": "

Am facing the same issue . I'm running 32bit OS and using RODBC to connect to transaction file.

\n

However, I'm able to work on only 200,000 Rows at a time . I get memory limitation error if i use more than that.

\n

I think we just can't beat the benchmark without utilizing the transaction file.

", "votes": null, "replies": [] }, { "id": 47218, "author_name": "omoekan09", "author_url": "", "post_date": "05/29/2014 03:30:50", "content": "

Check this thread: https://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8258/fread-function-in-r

", "votes": null, "replies": [] }, { "id": 47300, "author_name": "suboperator", "author_url": "", "post_date": "05/30/2014 02:04:16", "content": "

On your question about 64 bit machine - the unpacked file is about 20Gb. One of my machines is a 64bit running Win7Pro64 bit with 16Gb of RAM. R runs out of RAM loading the transactions.csv directly. You have to use other methods, some of them were mentioned on this forum already.

", "votes": null, "replies": [] }, { "id": 48546, "author_name": "fenec8", "author_url": "", "post_date": "06/03/2014 03:30:32", "content": "

You cannot load a 20g file in 4 or 8g memory. You could put the file in a database or read the files chunk by chunk and filter the no so needed transactions. However, the methods mentioned in this forum take some possibly useful temporal transactions out.

", "votes": null, "replies": [] }, { "id": 48577, "author_name": "weiwunyc", "author_url": "", "post_date": "06/03/2014 22:17:19", "content": "

You can load, filter and save the big transaction file in Chunks similar to the python code posted on the other thread with very small foot print. (I did it on a 4G Win 32bit testing machine, with R footprint less than 100M memory, I think. But whether you can do any meaningful modeling, is another story.) However, R is not good at these tasks, R code can be significantly slower than python for doing this kind of things.

\n

Here is my sample code to load and filter and save the transaction files from and into gzip'ed files:

\n

options(stringsAsFactors=F)
offers=read.csv("Data/offers.csv.gz")
allcats=unique(offers$category)

\n

trainHistory=read.csv("Data/trainHistory.csv.gz")
train_ids=unique(trainHistory$id)
testHistory=read.csv("Data/testHistory.csv.gz")
test_ids=unique(testHistory$id)
all_ids=unique(c(train_ids, test_ids))

\n

gzf = gzfile("Data/reduced_df_quicksave.csv.gz","w")
con=file("Data/transactions.csv.gz",open='r')
blocksize=500000
rownum=blocksize
readnum=0
totalrows=0
keptrows=0
begTime=Sys.time()
headers=read.csv(con, nrows=1, header=F)
while(rownum >= blocksize) {
        tempdf=read.csv(con, nrows=blocksize, header=F)
        rownum=nrow(tempdf)
        totalrows=totalrows+rownum
        colnames(tempdf)=headers
        tempdf=tempdf[tempdf$category %in% allcats & tempdf$id %in% all_ids,]
        if (nrow(tempdf) > 0) {
                if (keptrows == 0) {
                        write.table(tempdf, row.names=F, col.names=TRUE, sep=",", file=gzf)
                } else {
                        write.table(tempdf, row.names=F, col.names=FALSE, sep=",", file=gzf)
                }
                keptrows=keptrows+nrow(tempdf)
        }
        rm(tempdf);gc()
        readnum=readnum+1
        cat(paste("Read",readnum,"blocks, total rows read:",format(totalrows,scientific=F),
                        ", total rows kept:", format(keptrows,scientific=F),
                        ", time used:",format(Sys.time()-begTime),"\\n"))
        flush.console()
}
close(con)

\n

close(gzf)
cat(paste("   time used:",format(Sys.time()-begTime),"\\n"))

\n

It took 40+ minutes on my windows machine, with the file located on a local hard drive.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "47165": "", "47177": "", "47218": "", "47300": "", "48546": "", "48577": "" }, "source": "meta" }