{ "id": 5631, "title": "data handling in R Vs Pig", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5631", "author_name": "", "post_date": "2013-09-03T06:52:16.780Z", "votes": null, "comment_count": 18, "views": 13366, "content": "

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked. 

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

", "messages": [ { "id": "30037", "postDate": "09/03/2013 06:52:16", "content": "

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked. 

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

", "rawMarkdown": "", "votes": null }, { "id": "30045", "postDate": "09/03/2013 11:21:49", "content": "

Even I am also wondering why Kaggle has given such a large dataset for individuals. We cannot process this in R, nor we have Hadoop clusters running for us and also it is not a good practice to use it on office systems. So, I think the main thing in this competition is how a person can handle this data effectively rather applying sophisticated algorithms.

", "rawMarkdown": "", "votes": null }, { "id": "30049", "postDate": "09/03/2013 11:57:50", "content": "

Probably been capable of handling "big data" is wishable in a facebook data scientist xD

", "rawMarkdown": "", "votes": null }, { "id": "30237", "postDate": "09/05/2013 17:35:10", "content": "

you can always use AWS to fire up a hadoop cluster :)

", "rawMarkdown": "", "votes": null }, { "id": "30240", "postDate": "09/05/2013 17:55:55", "content": "

And do you think AWS gives you 8GB space with free version? I am not sure about it. I need to verify it.

", "rawMarkdown": "", "votes": null }, { "id": "30245", "postDate": "09/05/2013 20:30:57", "content": "

During a statistics course at school, a teaching assistant of that course had demonstrated cloud computing aspects on AWS like EC2 instances & S3 storage where TA had rented 10 GB of the space in the cloud and it had costed around (< $10/month). I am wondering anyone has an experience using AWS  & info related to costs associated per 1 GB storage ...   

", "rawMarkdown": "", "votes": null }, { "id": "30250", "postDate": "09/05/2013 21:30:43", "content": "

they post prices for all services and have a pricing calculator. http://calculator.s3.amazonaws.com/calc5.html, http://aws.amazon.com/ec2/pricing/. Storage space is incredibly cheap. RAM is somewhat more expensive.

", "rawMarkdown": "", "votes": null }, { "id": "30252", "postDate": "09/05/2013 22:07:52", "content": "

[quote=Chit;30037]

\n

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked. 

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

\n

[/quote]

\n

 

\n

See this thread https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5654/code-to-read-train-dataset-in-r  you will be able to read full Train file in R with 4 GB RAM easily

", "rawMarkdown": "", "votes": null }, { "id": "30256", "postDate": "09/06/2013 00:09:12", "content": "

[quote=Thakur Raj Anand;30252]

\n

See this thread https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5654/code-to-read-train-dataset-in-r  you will be able to read full Train file in R with 4 GB RAM easily

\n

[/quote]

\n

Reading the data is not the problem. There is another elephant standing in the room.

", "rawMarkdown": "", "votes": null }, { "id": "31798", "postDate": "09/29/2013 20:28:07", "content": "

It is easy to write a parser for this data set in Python. I tried to use iopro from Continuum using regex option and it gave seg fault, so just wrote a parser.

", "rawMarkdown": "", "votes": null }, { "id": "31825", "postDate": "09/30/2013 12:41:51", "content": "

i think given data file is messy and i anticipate that this is close to a real life scenario..i have witnessed some random tags while skimming the csv, both predictinh them and keeping variance low is a hard task. (i am noob)

", "rawMarkdown": "", "votes": null }, { "id": "32075", "postDate": "10/07/2013 09:23:34", "content": "

[quote=Chit;30037]

\n

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked. 

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

\n

[/quote]

\n

I am wondering how R is able to detect the line and column separators properly but not Pig

", "rawMarkdown": "", "votes": null }, { "id": "32195", "postDate": "10/10/2013 00:59:03", "content": "

I am using python csv reader and have not had any problems accessing the data. I do not load the entire file into memory. My first step is to convert the data into a better format (but I lose some of the information by doing so maybe I should upgrade my ram... ):

\n

 input_file = open(input_filename,'r')

\n

reader = csv.reader( input_file )
if skip_first_line:
     reader.next()

\n

 

\n

def iterable(reader):
     for line in reader:
          yield line[1]

\n

#Use 0 for ID, 1 for Title, 2 for Question Text, 3 for Keywords

\n

#Alternately yield line to return a list. The list has 4 items as indicated above.

\n

 

\n

# Now you can just call the function:

\n

nextLine = iterable(reader)

\n

 

\n

This is known as a generator. I was pointed to this from an earlier post on this forum, but haven't kept track as to which one.

", "rawMarkdown": "", "votes": null }, { "id": "32224", "postDate": "10/10/2013 16:50:35", "content": "

Is using an iterable faster than running a for loop on the reader?

\n

I am using the latter and it takes too long to go through the data.

", "rawMarkdown": "", "votes": null }, { "id": "32251", "postDate": "10/11/2013 14:35:33", "content": "

[quote=Chit;32075]

\n

[quote=Chit;30037]

\n

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked.

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

\n

[/quote]

\n

I am wondering how R is able to detect the line and column separators properly but not Pig

\n

[/quote]

\n

After numerous edits to this post, I am still not sure how to load Train.zip or Train.csv into Pig directly apart from rolling out a custom jar. 

\n

The below works for me. First we parse the input file in Python to remove special white space characters, remove wrapping fields in " " and use \\t as our field delimiter (our file should no longer contain any \\t and the only \\n should now be between rows).

\n

import csv

\n

inputFile = "data/Train.csv"
outputFile = "single_whitespaces_only.csv"

\n

f = open(outputFile, 'wb')
out = csv.writer(f, delimiter='\\t')

\n

i = 0

\n

with open(inputFile) as csvfile:
    csvfile.readline() # removes header row
    for row in csv.reader(csvfile, delimiter=',', quotechar='"'):
        i += 1
        if i > 100: break
        else:
            out.writerow([" ".join(part.split()) for part in row])
f.close()

\n

And we can load it into Pig using PigStorage() like this:

\n

data = LOAD 'single_whitespaces_only.csv' using PigStorage('\\t') AS (ID:int, Title:chararray, Body:chararray, Tag:chararray) ;

\n

Now that I think of it we probably should have also replaced "" with " but I guess if we will be parsing the whole thing we will want to make modifications that go even deeper than that.

", "rawMarkdown": "", "votes": null }, { "id": "32252", "postDate": "10/11/2013 15:07:39", "content": "

[quote=Amol Desai;32224]

\n

Is using an iterable faster than running a for loop on the reader?

\n

I am using the latter and it takes too long to go through the data.

\n

[/quote]

\n

Both in principle will do the same thing. They will allow you to step over the file at each given point in time yielding you access to just a single line of data. That's why as you continue to read the file your memory foot print doesn't increase. 

\n

The fun begins as you think what to do next. Do you process the data line by line and write it out to a file? Do you perform some calculations on it? Do you store all / chunks of it into an in memory data structure? And if so, what data structure? If you do, think about the lookups, how quickly they will become expensive with that much data? 

\n

You might want to check out IPython magics %time and %timeit if you'd like to time your code and see where the bottlenecks are. It took me only 120 seconds to step over the entire file. If I were to venture a guess I would stipulate it is not the disk IO but rather the expensive lookups you are doing that consume CPU cycles.

\n

Figuring all this out can be really fun and educational. At the same time you might probably get equally as much mileage if not more from learning tools that are geared towards working with data at this scale. I guess this is more of a philosophical question and goes back to what your motivation for participating in this competition is.

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "32349", "postDate": "10/15/2013 17:38:14", "content": "

I cleaned the data using tr and sed commands. Now processing them using map-reduce.

", "rawMarkdown": "", "votes": null }, { "id": "32905", "postDate": "10/30/2013 22:37:35", "content": "

Using unix functions first enables to have datasets much more amenable to python-pandas R and PIG.

", "rawMarkdown": "", "votes": null }, { "id": "53115", "postDate": "09/04/2014 21:28:03", "content": "

Also there is a tool, MortarData which will allow you to cut pig jobs to run on a cluster.  I've been able to run through some pretty complicated stuff for ~10 bucks.  

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30045, "author_name": "ankitksharma", "author_url": "", "post_date": "09/03/2013 11:21:49", "content": "

Even I am also wondering why Kaggle has given such a large dataset for individuals. We cannot process this in R, nor we have Hadoop clusters running for us and also it is not a good practice to use it on office systems. So, I think the main thing in this competition is how a person can handle this data effectively rather applying sophisticated algorithms.

", "votes": null, "replies": [] }, { "id": 30049, "author_name": "mansilla", "author_url": "", "post_date": "09/03/2013 11:57:50", "content": "

Probably been capable of handling "big data" is wishable in a facebook data scientist xD

", "votes": null, "replies": [] }, { "id": 30237, "author_name": "foobarbazquux", "author_url": "", "post_date": "09/05/2013 17:35:10", "content": "

you can always use AWS to fire up a hadoop cluster :)

", "votes": null, "replies": [] }, { "id": 30240, "author_name": "ankitksharma", "author_url": "", "post_date": "09/05/2013 17:55:55", "content": "

And do you think AWS gives you 8GB space with free version? I am not sure about it. I need to verify it.

", "votes": null, "replies": [] }, { "id": 30245, "author_name": "chandrad", "author_url": "", "post_date": "09/05/2013 20:30:57", "content": "

During a statistics course at school, a teaching assistant of that course had demonstrated cloud computing aspects on AWS like EC2 instances & S3 storage where TA had rented 10 GB of the space in the cloud and it had costed around (< $10/month). I am wondering anyone has an experience using AWS  & info related to costs associated per 1 GB storage ...   

", "votes": null, "replies": [] }, { "id": 30250, "author_name": "foobarbazquux", "author_url": "", "post_date": "09/05/2013 21:30:43", "content": "

they post prices for all services and have a pricing calculator. http://calculator.s3.amazonaws.com/calc5.html, http://aws.amazon.com/ec2/pricing/. Storage space is incredibly cheap. RAM is somewhat more expensive.

", "votes": null, "replies": [] }, { "id": 30252, "author_name": "thakurrajanand", "author_url": "", "post_date": "09/05/2013 22:07:52", "content": "

[quote=Chit;30037]

\n

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked. 

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

\n

[/quote]

\n

 

\n

See this thread https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5654/code-to-read-train-dataset-in-r  you will be able to read full Train file in R with 4 GB RAM easily

", "votes": null, "replies": [] }, { "id": 30256, "author_name": "rohits", "author_url": "", "post_date": "09/06/2013 00:09:12", "content": "

[quote=Thakur Raj Anand;30252]

\n

See this thread https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5654/code-to-read-train-dataset-in-r  you will be able to read full Train file in R with 4 GB RAM easily

\n

[/quote]

\n

Reading the data is not the problem. There is another elephant standing in the room.

", "votes": null, "replies": [] }, { "id": 31798, "author_name": "krishnadubba", "author_url": "", "post_date": "09/29/2013 20:28:07", "content": "

It is easy to write a parser for this data set in Python. I tried to use iopro from Continuum using regex option and it gave seg fault, so just wrote a parser.

", "votes": null, "replies": [] }, { "id": 31825, "author_name": "flyingmadden", "author_url": "", "post_date": "09/30/2013 12:41:51", "content": "

i think given data file is messy and i anticipate that this is close to a real life scenario..i have witnessed some random tags while skimming the csv, both predictinh them and keeping variance low is a hard task. (i am noob)

", "votes": null, "replies": [] }, { "id": 32075, "author_name": "chitrasen", "author_url": "", "post_date": "10/07/2013 09:23:34", "content": "

[quote=Chit;30037]

\n

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked. 

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

\n

[/quote]

\n

I am wondering how R is able to detect the line and column separators properly but not Pig

", "votes": null, "replies": [] }, { "id": 32195, "author_name": "nikhilagarwal1", "author_url": "", "post_date": "10/10/2013 00:59:03", "content": "

I am using python csv reader and have not had any problems accessing the data. I do not load the entire file into memory. My first step is to convert the data into a better format (but I lose some of the information by doing so maybe I should upgrade my ram... ):

\n

 input_file = open(input_filename,'r')

\n

reader = csv.reader( input_file )
if skip_first_line:
     reader.next()

\n

 

\n

def iterable(reader):
     for line in reader:
          yield line[1]

\n

#Use 0 for ID, 1 for Title, 2 for Question Text, 3 for Keywords

\n

#Alternately yield line to return a list. The list has 4 items as indicated above.

\n

 

\n

# Now you can just call the function:

\n

nextLine = iterable(reader)

\n

 

\n

This is known as a generator. I was pointed to this from an earlier post on this forum, but haven't kept track as to which one.

", "votes": null, "replies": [] }, { "id": 32224, "author_name": "amoldesai", "author_url": "", "post_date": "10/10/2013 16:50:35", "content": "

Is using an iterable faster than running a for loop on the reader?

\n

I am using the latter and it takes too long to go through the data.

", "votes": null, "replies": [] }, { "id": 32251, "author_name": "radek1", "author_url": "", "post_date": "10/11/2013 14:35:33", "content": "

[quote=Chit;32075]

\n

[quote=Chit;30037]

\n

When I took a sample from Train.csv and loaded into pig using PigStorage(',') function, I am not the columns properly (miss placed ) as there are commas in the text message. However while loading the same data in R using read.csv function, it worked.

\n

The problem is R cannot handle such a big file, so I am trying to process using pig. Please suggest how you are handling the dataset.

\n

[/quote]

\n

I am wondering how R is able to detect the line and column separators properly but not Pig

\n

[/quote]

\n

After numerous edits to this post, I am still not sure how to load Train.zip or Train.csv into Pig directly apart from rolling out a custom jar. 

\n

The below works for me. First we parse the input file in Python to remove special white space characters, remove wrapping fields in " " and use \\t as our field delimiter (our file should no longer contain any \\t and the only \\n should now be between rows).

\n

import csv

\n

inputFile = "data/Train.csv"
outputFile = "single_whitespaces_only.csv"

\n

f = open(outputFile, 'wb')
out = csv.writer(f, delimiter='\\t')

\n

i = 0

\n

with open(inputFile) as csvfile:
    csvfile.readline() # removes header row
    for row in csv.reader(csvfile, delimiter=',', quotechar='"'):
        i += 1
        if i > 100: break
        else:
            out.writerow([" ".join(part.split()) for part in row])
f.close()

\n

And we can load it into Pig using PigStorage() like this:

\n

data = LOAD 'single_whitespaces_only.csv' using PigStorage('\\t') AS (ID:int, Title:chararray, Body:chararray, Tag:chararray) ;

\n

Now that I think of it we probably should have also replaced "" with " but I guess if we will be parsing the whole thing we will want to make modifications that go even deeper than that.

", "votes": null, "replies": [] }, { "id": 32252, "author_name": "radek1", "author_url": "", "post_date": "10/11/2013 15:07:39", "content": "

[quote=Amol Desai;32224]

\n

Is using an iterable faster than running a for loop on the reader?

\n

I am using the latter and it takes too long to go through the data.

\n

[/quote]

\n

Both in principle will do the same thing. They will allow you to step over the file at each given point in time yielding you access to just a single line of data. That's why as you continue to read the file your memory foot print doesn't increase. 

\n

The fun begins as you think what to do next. Do you process the data line by line and write it out to a file? Do you perform some calculations on it? Do you store all / chunks of it into an in memory data structure? And if so, what data structure? If you do, think about the lookups, how quickly they will become expensive with that much data? 

\n

You might want to check out IPython magics %time and %timeit if you'd like to time your code and see where the bottlenecks are. It took me only 120 seconds to step over the entire file. If I were to venture a guess I would stipulate it is not the disk IO but rather the expensive lookups you are doing that consume CPU cycles.

\n

Figuring all this out can be really fun and educational. At the same time you might probably get equally as much mileage if not more from learning tools that are geared towards working with data at this scale. I guess this is more of a philosophical question and goes back to what your motivation for participating in this competition is.

\n

 

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 32349, "author_name": "chitrasen", "author_url": "", "post_date": "10/15/2013 17:38:14", "content": "

I cleaned the data using tr and sed commands. Now processing them using map-reduce.

", "votes": null, "replies": [] }, { "id": 32905, "author_name": "louisdv", "author_url": "", "post_date": "10/30/2013 22:37:35", "content": "

Using unix functions first enables to have datasets much more amenable to python-pandas R and PIG.

", "votes": null, "replies": [] }, { "id": 53115, "author_name": "gersondave", "author_url": "", "post_date": "09/04/2014 21:28:03", "content": "

Also there is a tool, MortarData which will allow you to cut pig jobs to run on a cluster.  I've been able to run through some pretty complicated stuff for ~10 bucks.  

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30037": "", "30045": "", "30049": "", "30237": "", "30240": "", "30245": "", "30250": "", "30252": "", "30256": "", "31798": "", "31825": "", "32075": "", "32195": "", "32224": "", "32251": "", "32252": "", "32349": "", "32905": "", "53115": "" }, "source": "meta" }