{ "id": 5654, "title": "Code To read Train dataset in R", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5654", "author_name": "", "post_date": "2013-09-04T20:52:28.657Z", "votes": 4, "comment_count": 11, "views": 7136, "content": "

library(ff)

\n

train <- read.csv.ffdf(file="Train.csv",header=TRUE,VERBOSE=TRUE,first.rows=10000,next.rows=10000,colClasses=NA)

\n

 

\n

It took 37 min to read the file in R

\n

 

", "messages": [ { "id": "30188", "postDate": "09/04/2013 20:52:28", "content": "

library(ff)

\n

train <- read.csv.ffdf(file="Train.csv",header=TRUE,VERBOSE=TRUE,first.rows=10000,next.rows=10000,colClasses=NA)

\n

 

\n

It took 37 min to read the file in R

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "30525", "postDate": "09/09/2013 10:16:46", "content": "

There is no point of loading the entire dataset into R. You'll need some 16 GB RAM to perform the analysis. I'm also kind of confused in how to process the data since I have only 4GB RAM system. R is definitely not a good choice, may be RHadoop + AWS can be of some help. Still I'm not sure.

", "rawMarkdown": "", "votes": null }, { "id": "30656", "postDate": "09/10/2013 14:19:01", "content": "

I wonder if there is a way in R to get a sample from the train data without loading the whole set into memory?

", "rawMarkdown": "", "votes": null }, { "id": "30659", "postDate": "09/10/2013 14:36:22", "content": "

[quote=Ishitori;30656]

\n

I wonder if there is a way in R to get a sample from the train data without loading the whole set into memory?

\n

[/quote]

\n

 

\n

I think read.table command have option to do so. Just google it. If you can't find it let me know I will try and post the code.

", "rawMarkdown": "", "votes": null }, { "id": "30660", "postDate": "09/10/2013 14:39:12", "content": "

Ok, will google it. 

\n

I wonder is sampling a commonly used approach nowadays, when everybody seems to be using Hadoop and other tools for working with Big data? Or people tend to just upload everything to HDFS and do even exploration stuff in clusters?

", "rawMarkdown": "", "votes": null }, { "id": "30713", "postDate": "09/11/2013 02:22:24", "content": "

[quote=Ishitori;30660]

\n

Ok, will google it. 

\n

I wonder is sampling a commonly used approach nowadays, when everybody seems to be using Hadoop and other tools for working with Big data? Or people tend to just upload everything to HDFS and do even exploration stuff in clusters?

\n

[/quote]

\n

 

\n

That's not 100% true. Although we have hadoop clusters running in the office but we can't use office resources for personal work that too for a job. In such situations AWS + RHadoop looks to me more useful.

", "rawMarkdown": "", "votes": null }, { "id": "30723", "postDate": "09/11/2013 08:25:28", "content": "

The csv file is a text file. We can split the text file into smaller chunks (tail, less, split etc in linux and some splitter programs in windows). Then load one of the smaller file to see a view of the subset of data. Hope that helps!

", "rawMarkdown": "", "votes": null }, { "id": "30725", "postDate": "09/11/2013 09:17:50", "content": "

[quote=Ritesh Gupta;30723]

\n

The csv file is a text file. We can split the text file into smaller chunks (tail, less, split etc in linux and some splitter programs in windows). Then load one of the smaller file to see a view of the subset of data. Hope that helps!

\n

[/quote]

\n

To add to this, there was a discussion earlier about using shell commands. The data has newline characters which makes it slightly more complicated. Dmitrim has some code that cleans this problem. Once that is done, you could use something like:

\n

awk 'BEGIN { srand(systime()); } {if (rand() < 0.25) { print $0; } }' Train.csv > sampled_train.csv

\n

That should hopefully do the job.

\n

 

\n

PS: The code above is rendering incorrectly. I have a copy here.

", "rawMarkdown": "", "votes": null }, { "id": "30966", "postDate": "09/16/2013 11:44:26", "content": "

[quote=Ishitori;30656]

\n

I wonder if there is a way in R to get a sample from the train data without loading the whole set into memory?

\n

[/quote]

\n

ds <- read.csv(...., nrows=2e4)
ds <- rbind(ds, read.csv(...., skip=2e4, nrows=2e4))
ds <- rbind(ds, read.csv(...., skip=4e4, nrows=2e4))
ds <- rbind(ds, read.csv(...., skip=6e4))

\n

Hope this will help you to read data in chunks

", "rawMarkdown": "", "votes": null }, { "id": "31167", "postDate": "09/19/2013 10:04:24", "content": "

You can try with hadoop+mahout combinations to handle big file and also its scalable model.

\n

Thanks,

\n

Saravanan

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "32815", "postDate": "10/29/2013 07:05:06", "content": "

This was not meant to be optimized or simplified code since I needed something to work right away. The algorithm isn't efficient as it takes over 36 hours but if you're desperate for something to work right away, this should do the trick. At 500K lines, you'll end up with 291 files for the 6M+ records.  If you're not comfortable with R, I suggest reading the other posts on postgres as that would probably be more efficient, but I'm a fan of R.

\n

segment<-function()
{
train.file<-file("Train.csv",open="r")
curr.block<-readLines(train.file,n=500000, ok=TRUE)
curr.block<-curr.block[-1] #delete first line
file.suffix<-1
temp<-1
while(length(curr.block)>0)
{

id.title.question.tags<-data.frame(idnu=numeric(0),title=character(0),question=character(0),tags=character(0),stringsAsFactors=FALSE)
curr.row<-0
start.line.curr.obs<-0
curr.idnu<-NA
curr.title<-NA
curr.question<-NA
curr.tags<-NA
start.time<-Sys.time()
#loop over lines
for(iter.line in curr.block)
{
output<-grep('\\"[0-9]+\\",',iter.line,value=TRUE)
#found start line of an observation/row
if(length(output)>0)
{

curr.row<-curr.row+1
if(curr.row>1)
{
#this means that all the lines for the previous obs (curr.row-1) are processed
#so start adding values to data frame
id.title.question.tags[curr.row-1,"idnu"]<-as.numeric(curr.idnu)
id.title.question.tags[curr.row-1,"title"]<-as.character(curr.title)
id.title.question.tags[curr.row-1,"question"]<-as.character(curr.question)
id.title.question.tags[curr.row-1,"tags"]<-as.character(curr.tags)

curr.idnu<-NA
curr.title<-NA
curr.question<-NA
curr.tags<-NA

curr.time<-Sys.time()
elapsed.time<-round(as.numeric(difftime(curr.time,start.time,units="mins")),digits=2)
print(paste(file.suffix," : ",temp," : ",elapsed.time," mins",sep=""))
temp<-temp+1


}
start.line.curr.obs <- which(curr.block==iter.line)
split.line<-strsplit(iter.line,split='\\",\\"')[[1]]
#1st element should be idnu
#also remove the first quote
curr.idnu<-sub('\\"','',split.line[1])
#2nd element should be title
curr.title<-split.line[2]
#3rd element is start of question, this can go through multiple lines
#which is why we need a while loop following this line
curr.question<-split.line[3]
}
#so probably is a line of a multi-line question
#OR is the tag line
else
{
#check if a multi-line question or tag line
split.line<-strsplit(iter.line,split='\\",\\"')[[1]]
#this should be the tag line for the curr observation
if(length(split.line)==2)
{
#removing last leftover quote, these leftover quotes come from strsplit regex
curr.tags<-sub('\\"','',split.line[2])
}
else
{
#anything else needs to be added to multi-line
curr.question<-paste(curr.question,iter.line,sep="")
}

}
}
#need to process last observation
#no matter if all the fields are complete
if(!is.na(curr.tags))
{
id.title.question.tags[curr.row,"idnu"]<-as.numeric(curr.idnu)
id.title.question.tags[curr.row,"title"]<-as.character(curr.title)
id.title.question.tags[curr.row,"question"]<-as.character(curr.question)
id.title.question.tags[curr.row,"tags"]<-as.character(curr.tags)

}
else
{
#keep going to next line until tag line is reached
while(is.na(curr.tags))
{
iter.line<-readLines(train.file,n=1, ok=TRUE) #read next line
#check if a multi-line question or tag line
split.line<-strsplit(iter.line,split='\\",\\"')[[1]]
#this should be the tag line for the curr observation
if(length(split.line)==2)
{
#removing last leftover quote, these leftover quotes come from strsplit regex
curr.tags<-sub('\\"','',split.line[2])
id.title.question.tags[curr.row,"idnu"]<-as.numeric(curr.idnu)
id.title.question.tags[curr.row,"title"]<-as.character(curr.title)
id.title.question.tags[curr.row,"question"]<-as.character(curr.question)
id.title.question.tags[curr.row,"tags"]<-as.character(curr.tags)

\n


}
else
{
#anything else needs to be added to multi-line
curr.question<-paste(curr.question,iter.line,sep="")
}

}

}
filename<-paste("train",file.suffix,sep="")
save(file=filename,id.title.question.tags)
curr.time<-Sys.time()
elapsed.time<-round(as.numeric(difftime(curr.time,start.time,units="mins")),digits=2)
print(paste(elapsed.time," mins",sep=""))
rm(id.title.question.tags)
rm(curr.block)
curr.block<-readLines(train.file,n=500000, ok=TRUE) #read next 500K
file.suffix<-file.suffix+1
temp<-1
}
close(train.file)
noquote("done")
id.title.question.tags

}

", "rawMarkdown": "", "votes": null }, { "id": "35114", "postDate": "11/24/2013 16:03:59", "content": "

I found it really difficult to get the data in R, even though I have 8GB RAM. 

\n

Give a try to StatAce. It's a scalable R SaaS. Turned out to be useful in the actual training/prediction stage.

\n

 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30525, "author_name": "ankitksharma", "author_url": "", "post_date": "09/09/2013 10:16:46", "content": "

There is no point of loading the entire dataset into R. You'll need some 16 GB RAM to perform the analysis. I'm also kind of confused in how to process the data since I have only 4GB RAM system. R is definitely not a good choice, may be RHadoop + AWS can be of some help. Still I'm not sure.

", "votes": null, "replies": [] }, { "id": 30656, "author_name": "ishitori", "author_url": "", "post_date": "09/10/2013 14:19:01", "content": "

I wonder if there is a way in R to get a sample from the train data without loading the whole set into memory?

", "votes": null, "replies": [] }, { "id": 30659, "author_name": "thakurrajanand", "author_url": "", "post_date": "09/10/2013 14:36:22", "content": "

[quote=Ishitori;30656]

\n

I wonder if there is a way in R to get a sample from the train data without loading the whole set into memory?

\n

[/quote]

\n

 

\n

I think read.table command have option to do so. Just google it. If you can't find it let me know I will try and post the code.

", "votes": null, "replies": [] }, { "id": 30660, "author_name": "ishitori", "author_url": "", "post_date": "09/10/2013 14:39:12", "content": "

Ok, will google it. 

\n

I wonder is sampling a commonly used approach nowadays, when everybody seems to be using Hadoop and other tools for working with Big data? Or people tend to just upload everything to HDFS and do even exploration stuff in clusters?

", "votes": null, "replies": [] }, { "id": 30713, "author_name": "ankitksharma", "author_url": "", "post_date": "09/11/2013 02:22:24", "content": "

[quote=Ishitori;30660]

\n

Ok, will google it. 

\n

I wonder is sampling a commonly used approach nowadays, when everybody seems to be using Hadoop and other tools for working with Big data? Or people tend to just upload everything to HDFS and do even exploration stuff in clusters?

\n

[/quote]

\n

 

\n

That's not 100% true. Although we have hadoop clusters running in the office but we can't use office resources for personal work that too for a job. In such situations AWS + RHadoop looks to me more useful.

", "votes": null, "replies": [] }, { "id": 30723, "author_name": "gritesh77", "author_url": "", "post_date": "09/11/2013 08:25:28", "content": "

The csv file is a text file. We can split the text file into smaller chunks (tail, less, split etc in linux and some splitter programs in windows). Then load one of the smaller file to see a view of the subset of data. Hope that helps!

", "votes": null, "replies": [] }, { "id": 30725, "author_name": "rohits", "author_url": "", "post_date": "09/11/2013 09:17:50", "content": "

[quote=Ritesh Gupta;30723]

\n

The csv file is a text file. We can split the text file into smaller chunks (tail, less, split etc in linux and some splitter programs in windows). Then load one of the smaller file to see a view of the subset of data. Hope that helps!

\n

[/quote]

\n

To add to this, there was a discussion earlier about using shell commands. The data has newline characters which makes it slightly more complicated. Dmitrim has some code that cleans this problem. Once that is done, you could use something like:

\n

awk 'BEGIN { srand(systime()); } {if (rand() < 0.25) { print $0; } }' Train.csv > sampled_train.csv

\n

That should hopefully do the job.

\n

 

\n

PS: The code above is rendering incorrectly. I have a copy here.

", "votes": null, "replies": [] }, { "id": 30966, "author_name": "parthibangowthaman", "author_url": "", "post_date": "09/16/2013 11:44:26", "content": "

[quote=Ishitori;30656]

\n

I wonder if there is a way in R to get a sample from the train data without loading the whole set into memory?

\n

[/quote]

\n

ds <- read.csv(...., nrows=2e4)
ds <- rbind(ds, read.csv(...., skip=2e4, nrows=2e4))
ds <- rbind(ds, read.csv(...., skip=4e4, nrows=2e4))
ds <- rbind(ds, read.csv(...., skip=6e4))

\n

Hope this will help you to read data in chunks

", "votes": null, "replies": [] }, { "id": 31167, "author_name": "saravanan2", "author_url": "", "post_date": "09/19/2013 10:04:24", "content": "

You can try with hadoop+mahout combinations to handle big file and also its scalable model.

\n

Thanks,

\n

Saravanan

\n

 

", "votes": null, "replies": [] }, { "id": 32815, "author_name": "davidcho0", "author_url": "", "post_date": "10/29/2013 07:05:06", "content": "

This was not meant to be optimized or simplified code since I needed something to work right away. The algorithm isn't efficient as it takes over 36 hours but if you're desperate for something to work right away, this should do the trick. At 500K lines, you'll end up with 291 files for the 6M+ records.  If you're not comfortable with R, I suggest reading the other posts on postgres as that would probably be more efficient, but I'm a fan of R.

\n

segment<-function()
{
train.file<-file("Train.csv",open="r")
curr.block<-readLines(train.file,n=500000, ok=TRUE)
curr.block<-curr.block[-1] #delete first line
file.suffix<-1
temp<-1
while(length(curr.block)>0)
{

id.title.question.tags<-data.frame(idnu=numeric(0),title=character(0),question=character(0),tags=character(0),stringsAsFactors=FALSE)
curr.row<-0
start.line.curr.obs<-0
curr.idnu<-NA
curr.title<-NA
curr.question<-NA
curr.tags<-NA
start.time<-Sys.time()
#loop over lines
for(iter.line in curr.block)
{
output<-grep('\\"[0-9]+\\",',iter.line,value=TRUE)
#found start line of an observation/row
if(length(output)>0)
{

curr.row<-curr.row+1
if(curr.row>1)
{
#this means that all the lines for the previous obs (curr.row-1) are processed
#so start adding values to data frame
id.title.question.tags[curr.row-1,"idnu"]<-as.numeric(curr.idnu)
id.title.question.tags[curr.row-1,"title"]<-as.character(curr.title)
id.title.question.tags[curr.row-1,"question"]<-as.character(curr.question)
id.title.question.tags[curr.row-1,"tags"]<-as.character(curr.tags)

curr.idnu<-NA
curr.title<-NA
curr.question<-NA
curr.tags<-NA

curr.time<-Sys.time()
elapsed.time<-round(as.numeric(difftime(curr.time,start.time,units="mins")),digits=2)
print(paste(file.suffix," : ",temp," : ",elapsed.time," mins",sep=""))
temp<-temp+1


}
start.line.curr.obs <- which(curr.block==iter.line)
split.line<-strsplit(iter.line,split='\\",\\"')[[1]]
#1st element should be idnu
#also remove the first quote
curr.idnu<-sub('\\"','',split.line[1])
#2nd element should be title
curr.title<-split.line[2]
#3rd element is start of question, this can go through multiple lines
#which is why we need a while loop following this line
curr.question<-split.line[3]
}
#so probably is a line of a multi-line question
#OR is the tag line
else
{
#check if a multi-line question or tag line
split.line<-strsplit(iter.line,split='\\",\\"')[[1]]
#this should be the tag line for the curr observation
if(length(split.line)==2)
{
#removing last leftover quote, these leftover quotes come from strsplit regex
curr.tags<-sub('\\"','',split.line[2])
}
else
{
#anything else needs to be added to multi-line
curr.question<-paste(curr.question,iter.line,sep="")
}

}
}
#need to process last observation
#no matter if all the fields are complete
if(!is.na(curr.tags))
{
id.title.question.tags[curr.row,"idnu"]<-as.numeric(curr.idnu)
id.title.question.tags[curr.row,"title"]<-as.character(curr.title)
id.title.question.tags[curr.row,"question"]<-as.character(curr.question)
id.title.question.tags[curr.row,"tags"]<-as.character(curr.tags)

}
else
{
#keep going to next line until tag line is reached
while(is.na(curr.tags))
{
iter.line<-readLines(train.file,n=1, ok=TRUE) #read next line
#check if a multi-line question or tag line
split.line<-strsplit(iter.line,split='\\",\\"')[[1]]
#this should be the tag line for the curr observation
if(length(split.line)==2)
{
#removing last leftover quote, these leftover quotes come from strsplit regex
curr.tags<-sub('\\"','',split.line[2])
id.title.question.tags[curr.row,"idnu"]<-as.numeric(curr.idnu)
id.title.question.tags[curr.row,"title"]<-as.character(curr.title)
id.title.question.tags[curr.row,"question"]<-as.character(curr.question)
id.title.question.tags[curr.row,"tags"]<-as.character(curr.tags)

\n


}
else
{
#anything else needs to be added to multi-line
curr.question<-paste(curr.question,iter.line,sep="")
}

}

}
filename<-paste("train",file.suffix,sep="")
save(file=filename,id.title.question.tags)
curr.time<-Sys.time()
elapsed.time<-round(as.numeric(difftime(curr.time,start.time,units="mins")),digits=2)
print(paste(elapsed.time," mins",sep=""))
rm(id.title.question.tags)
rm(curr.block)
curr.block<-readLines(train.file,n=500000, ok=TRUE) #read next 500K
file.suffix<-file.suffix+1
temp<-1
}
close(train.file)
noquote("done")
id.title.question.tags

}

", "votes": null, "replies": [] }, { "id": 35114, "author_name": "todorkazakov", "author_url": "", "post_date": "11/24/2013 16:03:59", "content": "

I found it really difficult to get the data in R, even though I have 8GB RAM. 

\n

Give a try to StatAce. It's a scalable R SaaS. Turned out to be useful in the actual training/prediction stage.

\n

 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30188": "", "30525": "", "30656": "", "30659": "", "30660": "", "30713": "", "30723": "", "30725": "", "30966": "", "31167": "", "32815": "", "35114": "" }, "source": "meta" }