{ "id": 1195, "title": "reading the files to R", "url": "/competitions/WhatDoYouKnow/discussion/1195", "author_name": "", "post_date": "2011-12-28T08:36:12.280Z", "votes": null, "comment_count": 13, "views": 10151, "content": "

hi want to take a part in this competition but i dont know how to read those csv big files to R

\r\n

i use read.csv but R tells me that i have a memory problem 

\r\n

what can i do???

\r\n

 

\r\n

elad

", "messages": [ { "id": "7530", "postDate": "12/28/2011 08:36:12", "content": "

hi want to take a part in this competition but i dont know how to read those csv big files to R

\r\n

i use read.csv but R tells me that i have a memory problem 

\r\n

what can i do???

\r\n

 

\r\n

elad

", "rawMarkdown": "", "votes": null }, { "id": "7534", "postDate": "12/28/2011 16:41:58", "content": "

Elad,

\r\n

Unfortunately, this kind of problem is quite frequent in data mining; sampling and cleaning the data can take as much time as the algorithm/machine learning part. Here are several things to try:
\r\n1. Use the first n lines of the file. This will give you a sense of the data, although it's a bad way to construct a random sample.
\r\n2. Write a small program in perl (or other language that is more space efficient than R) to sample from the big file.
\r\n3. You can use the colClasses argument to read.csv to control whether the data in a column is read or not. Use "NULL" to skip that column. This will save some space.

\r\n

--Steve

", "rawMarkdown": "", "votes": null }, { "id": "7535", "postDate": "12/28/2011 16:57:28", "content": "

I've only been using R for a few weeks, so I'm a novice, but here are some observations...

\r\n

Reading the largest file (training.csv) using read.csv("training.csv") requires roughly 2gb of free memory on my 64-bit linux PC.  I assume that's true on Windows as well.  If you don't have that much there are ways to make read.csv a bit more memory efficient. \r\n You can specify the column datatypes using colClasses, for example, and you can also set a maximum number of rows with\r\nnrows.  Both of these things will help a little, but probably not enough.

\r\n

The next option is to subset the data by rows and/or columns.  For example, the "tag_string" column isn't useful in raw form, so perhaps you don't need to load that.  Maybe there are other columns you don't need right away as well.  You can exclude columns\r\n by setting the class to NULL using colClasses (take a look at the included lmer benchmark for an example).

\r\n

Subsetting by rows isn't as easy from within R - I'm fairly certain you'd have to write actual code to do that.  If you have access to a grep-like utility (availabye for every common OS) or you are familiar with a scripting/programming language, subsetting\r\n the data based on some logical criteria (group_name or track_name for example) is very simple.

", "rawMarkdown": "", "votes": null }, { "id": "7559", "postDate": "12/29/2011 18:19:24", "content": "

some one knows how many records training_validation and test_validation???

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "7561", "postDate": "12/29/2011 18:20:36", "content": "

Elad your "?" key appears to be stuck

", "rawMarkdown": "", "votes": null }, { "id": "7563", "postDate": "12/29/2011 18:30:43", "content": "

You can find out very easily with the "wc" command on any unix-like system...

\r\n

% wc -l valid*.csv
 80076 valid_test.csv
 4763363 valid_training.csv
 4843439 total\r\n
", "rawMarkdown": "", "votes": null }, { "id": "7564", "postDate": "12/29/2011 18:32:56", "content": "

it's not stuck i am regular with using a lot of smaple marks when i write a question(:

\r\n

and it will help me alot to know how many records we have on those files

", "rawMarkdown": "", "votes": null }, { "id": "7566", "postDate": "12/29/2011 18:44:27", "content": "

I suspect if you are unable to count the lines in a file then this "may" not be the right competition for you

", "rawMarkdown": "", "votes": null }, { "id": "7576", "postDate": "12/30/2011 16:02:51", "content": "

thank you jason for the warnings....it is very nice to hear that you really care

", "rawMarkdown": "", "votes": null }, { "id": "7721", "postDate": "01/06/2012 16:51:26", "content": "

Here's some R code I wrote to convert the csv files to rda files. If you have 4 GB of ram, it should run fine. 3gb and 2gb may also work, just change the memory limit command to 3000 or 2000 (this only applies to windows).

\r\n

It's pretty quick and dirty, but it'll give you rda files that are pretty much identical to the csv files, and will open faster. Note that the "training.rda" file still takes me about 40 seconds to open, so step 2 is probably taking a sample.

\r\n

You'll need to change the directory to wherever your data are.

\r\n
\r\n

#Setup
rm(list = ls(all = TRUE))
memory.limit(4000)
setwd("~/Kaggle/Grockit/Data")

#Function to convert csv to rda files
convert <- function(name) {
\tcsv <- paste(name,'.csv',sep='')
\trda <- paste(name,'.rda',sep='')
\tassign(name,read.csv(csv, stringsAsFactors = FALSE))
\tsave(list=name, file=rda)
\trm(name)
\tgc(reset=TRUE)
}

#Run function on each file
convert('category_labels')
convert('valid_test')
convert('test')
convert('valid_training')
convert('training')

\r\n
", "rawMarkdown": "", "votes": null }, { "id": "7749", "postDate": "01/07/2012 18:17:42", "content": "

When dealing with large data, I doubt I'd trust "read.csv" to pick the optimal data types. For this contest in particular, I assume it would default to a lot of numerics when integers would be much more memory effecient. I'd look into the colClasses argument.

", "rawMarkdown": "", "votes": null }, { "id": "7754", "postDate": "01/07/2012 21:42:19", "content": "

Hmmm, I'm pretty sure read.csv chose integers when applicable, but I need to check that. I know omitting colClasses takes longer, but I only ran the code once, then deleted the csv files.

\r\n

I'll look into it.

", "rawMarkdown": "", "votes": null }, { "id": "7759", "postDate": "01/08/2012 00:21:44", "content": "

It appears read.csv actually uses type.convert, which should catch the integers as integers. It might add a little overhead to scan the whole column and then decide what to convert it into however.

", "rawMarkdown": "", "votes": null }, { "id": "7764", "postDate": "01/08/2012 16:22:36", "content": "

[quote=Shea Parkes;7759]

\r\n

It appears read.csv actually uses type.convert, which should catch the integers as integers. It might add a little overhead to scan the whole column and then decide what to convert it into however.

\r\n

[/quote]

\r\n

It does add overhead, but since it's hopefully a 1 time operation, I decided not to spend time optimize it.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 7534, "author_name": "sbagley", "author_url": "", "post_date": "12/28/2011 16:41:58", "content": "

Elad,

\r\n

Unfortunately, this kind of problem is quite frequent in data mining; sampling and cleaning the data can take as much time as the algorithm/machine learning part. Here are several things to try:
\r\n1. Use the first n lines of the file. This will give you a sense of the data, although it's a bad way to construct a random sample.
\r\n2. Write a small program in perl (or other language that is more space efficient than R) to sample from the big file.
\r\n3. You can use the colClasses argument to read.csv to control whether the data in a column is read or not. Use "NULL" to skip that column. This will save some space.

\r\n

--Steve

", "votes": null, "replies": [] }, { "id": 7535, "author_name": "yetiman", "author_url": "", "post_date": "12/28/2011 16:57:28", "content": "

I've only been using R for a few weeks, so I'm a novice, but here are some observations...

\r\n

Reading the largest file (training.csv) using read.csv("training.csv") requires roughly 2gb of free memory on my 64-bit linux PC.  I assume that's true on Windows as well.  If you don't have that much there are ways to make read.csv a bit more memory efficient. \r\n You can specify the column datatypes using colClasses, for example, and you can also set a maximum number of rows with\r\nnrows.  Both of these things will help a little, but probably not enough.

\r\n

The next option is to subset the data by rows and/or columns.  For example, the "tag_string" column isn't useful in raw form, so perhaps you don't need to load that.  Maybe there are other columns you don't need right away as well.  You can exclude columns\r\n by setting the class to NULL using colClasses (take a look at the included lmer benchmark for an example).

\r\n

Subsetting by rows isn't as easy from within R - I'm fairly certain you'd have to write actual code to do that.  If you have access to a grep-like utility (availabye for every common OS) or you are familiar with a scripting/programming language, subsetting\r\n the data based on some logical criteria (group_name or track_name for example) is very simple.

", "votes": null, "replies": [] }, { "id": 7559, "author_name": "bensalel", "author_url": "", "post_date": "12/29/2011 18:19:24", "content": "

some one knows how many records training_validation and test_validation???

\r\n

 

", "votes": null, "replies": [] }, { "id": 7561, "author_name": "jasontigg", "author_url": "", "post_date": "12/29/2011 18:20:36", "content": "

Elad your "?" key appears to be stuck

", "votes": null, "replies": [] }, { "id": 7563, "author_name": "yetiman", "author_url": "", "post_date": "12/29/2011 18:30:43", "content": "

You can find out very easily with the "wc" command on any unix-like system...

\r\n

% wc -l valid*.csv
 80076 valid_test.csv
 4763363 valid_training.csv
 4843439 total\r\n
", "votes": null, "replies": [] }, { "id": 7564, "author_name": "bensalel", "author_url": "", "post_date": "12/29/2011 18:32:56", "content": "

it's not stuck i am regular with using a lot of smaple marks when i write a question(:

\r\n

and it will help me alot to know how many records we have on those files

", "votes": null, "replies": [] }, { "id": 7566, "author_name": "jasontigg", "author_url": "", "post_date": "12/29/2011 18:44:27", "content": "

I suspect if you are unable to count the lines in a file then this "may" not be the right competition for you

", "votes": null, "replies": [] }, { "id": 7576, "author_name": "bensalel", "author_url": "", "post_date": "12/30/2011 16:02:51", "content": "

thank you jason for the warnings....it is very nice to hear that you really care

", "votes": null, "replies": [] }, { "id": 7721, "author_name": "zachmayer", "author_url": "", "post_date": "01/06/2012 16:51:26", "content": "

Here's some R code I wrote to convert the csv files to rda files. If you have 4 GB of ram, it should run fine. 3gb and 2gb may also work, just change the memory limit command to 3000 or 2000 (this only applies to windows).

\r\n

It's pretty quick and dirty, but it'll give you rda files that are pretty much identical to the csv files, and will open faster. Note that the "training.rda" file still takes me about 40 seconds to open, so step 2 is probably taking a sample.

\r\n

You'll need to change the directory to wherever your data are.

\r\n
\r\n

#Setup
rm(list = ls(all = TRUE))
memory.limit(4000)
setwd("~/Kaggle/Grockit/Data")

#Function to convert csv to rda files
convert <- function(name) {
\tcsv <- paste(name,'.csv',sep='')
\trda <- paste(name,'.rda',sep='')
\tassign(name,read.csv(csv, stringsAsFactors = FALSE))
\tsave(list=name, file=rda)
\trm(name)
\tgc(reset=TRUE)
}

#Run function on each file
convert('category_labels')
convert('valid_test')
convert('test')
convert('valid_training')
convert('training')

\r\n
", "votes": null, "replies": [] }, { "id": 7749, "author_name": "biasvariance", "author_url": "", "post_date": "01/07/2012 18:17:42", "content": "

When dealing with large data, I doubt I'd trust "read.csv" to pick the optimal data types. For this contest in particular, I assume it would default to a lot of numerics when integers would be much more memory effecient. I'd look into the colClasses argument.

", "votes": null, "replies": [] }, { "id": 7754, "author_name": "zachmayer", "author_url": "", "post_date": "01/07/2012 21:42:19", "content": "

Hmmm, I'm pretty sure read.csv chose integers when applicable, but I need to check that. I know omitting colClasses takes longer, but I only ran the code once, then deleted the csv files.

\r\n

I'll look into it.

", "votes": null, "replies": [] }, { "id": 7759, "author_name": "biasvariance", "author_url": "", "post_date": "01/08/2012 00:21:44", "content": "

It appears read.csv actually uses type.convert, which should catch the integers as integers. It might add a little overhead to scan the whole column and then decide what to convert it into however.

", "votes": null, "replies": [] }, { "id": 7764, "author_name": "zachmayer", "author_url": "", "post_date": "01/08/2012 16:22:36", "content": "

[quote=Shea Parkes;7759]

\r\n

It appears read.csv actually uses type.convert, which should catch the integers as integers. It might add a little overhead to scan the whole column and then decide what to convert it into however.

\r\n

[/quote]

\r\n

It does add overhead, but since it's hopefully a 1 time operation, I decided not to spend time optimize it.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "7530": "", "7534": "", "7535": "", "7559": "", "7561": "", "7563": "", "7564": "", "7566": "", "7576": "", "7721": "", "7749": "", "7754": "", "7759": "", "7764": "" }, "source": "meta" }