{ "id": 14292, "title": "R, file size, and speed", "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14292", "author_name": "", "post_date": "2015-05-19T00:17:45.650Z", "votes": 3, "comment_count": 12, "views": 2154, "content": "

Hey everyone -

\n\n

I'm an R user relatively new to modeling and prediction. I love Kaggle for the exposure to new problems, data set and just general practice. However, I struggle when the sets get large. My R session just times out or just takes so long. 

\n\n

Are there techical tips that anyone can share to help working with large sets? I'm looking into resources like H2O and even server based solutions but I'm a little lost there.  

\n

I've managed to create one model (actually a couple leading to one submission) that's done reasonably well. I just want to continuing exploring different approaches. It's just very taxing on my machine apparently and I'm getting frustrated.

\n\n

Id love to hear about whatever tips anyone may have! Thanks!

", "messages": [ { "id": "79052", "postDate": "05/19/2015 00:17:45", "content": "

Hey everyone -

\n\n

I'm an R user relatively new to modeling and prediction. I love Kaggle for the exposure to new problems, data set and just general practice. However, I struggle when the sets get large. My R session just times out or just takes so long. 

\n\n

Are there techical tips that anyone can share to help working with large sets? I'm looking into resources like H2O and even server based solutions but I'm a little lost there.  

\n

I've managed to create one model (actually a couple leading to one submission) that's done reasonably well. I just want to continuing exploring different approaches. It's just very taxing on my machine apparently and I'm getting frustrated.

\n\n

Id love to hear about whatever tips anyone may have! Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "79169", "postDate": "05/19/2015 07:00:55", "content": "

Hi Andrew,

\n

I use Hadoop on Azure (HDInsight) to load the large data set in, then I use HIVE queries to extract the data I need. Not sure if it's the best solution, but it's relatively fast (1-2 mins or less per query), and HIVE queries are a lot like SQL, which I am fairly comfortable with.

\n

I am sure there are comparable solutions out there on all other cloud vendors - but Azure happens to come with my Visual Studio Ultimate and it has a 30-day trial too - just don't forget to shut down the cluster when done for the day :)

\n

Hope this helps a bit,

\n

Eugene

", "rawMarkdown": "", "votes": null }, { "id": "79203", "postDate": "05/19/2015 09:21:04", "content": "

I see two choices:

\n

First, you can use databases, and use SQL to manipulate your data.

\n

Second, use some programming language and you can do it all on your own. For this particular competition, you can sort the data by bidder_id, and then you can read the data row by row, compare the current row's bidder_id with last row's bidder_id. And then combine them if they have same bidder_id, create a new one otherwise

", "rawMarkdown": "", "votes": null }, { "id": "79206", "postDate": "05/19/2015 09:32:37", "content": "

Just for this case, I think using vectorization can solve lots of performance problems.

", "rawMarkdown": "", "votes": null }, { "id": "79234", "postDate": "05/19/2015 12:38:47", "content": "

@Jiming Ye, what kind of program are you using for this competition ?

", "rawMarkdown": "", "votes": null }, { "id": "79251", "postDate": "05/19/2015 14:02:47", "content": "

I work with R only on this competition. I keep the memory clean by using rm() and gc() constantly. Once model is built I remove all objects except necessary for prediction. 

\n

I use Linux 8GB RAM and Mac 8GB RAM.

", "rawMarkdown": "", "votes": null }, { "id": "79256", "postDate": "05/19/2015 14:31:33", "content": "

@Dzidas, In your opinion, what is the main factor for this competition ?

", "rawMarkdown": "", "votes": null }, { "id": "79261", "postDate": "05/19/2015 14:47:25", "content": "

Please define "main factor" - are you after main difficulty, feature, etc.

", "rawMarkdown": "", "votes": null }, { "id": "79316", "postDate": "05/19/2015 19:21:53", "content": "

You can read large datasets in R, and this one is not that large. Just use data.table package and its excellent fread function. On my laptop (8GB of RAM) I can read bids.csv file in 40 seconds.

", "rawMarkdown": "", "votes": null }, { "id": "79349", "postDate": "05/19/2015 21:09:35", "content": "

Thanks for the ideas and responses everyone. @vk.net. It's not so much the reading in that I'm struggling with. I'm doing it the way you suggest. It's super fast ... Especially compared to just read.csv(). 

\n

My system is struggling when building and running the models. I've tried some sampling strategies.... Sort of some custom bagging methods if you will and it just doesn't seem very efficient. 

\n

I've also tried storing in MySQL and just reading in just samples so I'm not storing as much in mem  I've gotten mixed results. 

", "rawMarkdown": "", "votes": null }, { "id": "79535", "postDate": "05/20/2015 22:20:31", "content": "

I am using a "poor man's approach" to this problem. I have a very old (but funcional) laptop (Dell D830) with 4 GB RAM.

\n

I created a SQLite database and loaded the three .csv data files into it, in different tables. I then created the necessary queries for feature extraction.

\n

Overall, the system is not incredibly fast, but I get my features matrix in about 5 minutes, including time employed by R in some internal calculations and more aggregations (there's only so much one can do with SQLite, but that's enough).

\n

I tried reading the whole file into memory, but this approach is much more manageable.

", "rawMarkdown": "", "votes": null }, { "id": "79541", "postDate": "05/20/2015 22:55:21", "content": "

That's why I use MATLAB. The DATASTORE object in it allows manipulation of parts of data as needed, keeping the memory light, and processing fast.

", "rawMarkdown": "", "votes": null }, { "id": "79669", "postDate": "05/21/2015 20:32:00", "content": "

I've been using dplyr to do data maniupulations, joins, etc. No problems at all on my older Windows laptop with 8GB RAM. The dplyr package works similar to SQL, and you don't have to setup any data connections.

\n

Good luck

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 79169, "author_name": "ivanko", "author_url": "", "post_date": "05/19/2015 07:00:55", "content": "

Hi Andrew,

\n

I use Hadoop on Azure (HDInsight) to load the large data set in, then I use HIVE queries to extract the data I need. Not sure if it's the best solution, but it's relatively fast (1-2 mins or less per query), and HIVE queries are a lot like SQL, which I am fairly comfortable with.

\n

I am sure there are comparable solutions out there on all other cloud vendors - but Azure happens to come with my Visual Studio Ultimate and it has a 30-day trial too - just don't forget to shut down the cluster when done for the day :)

\n

Hope this helps a bit,

\n

Eugene

", "votes": null, "replies": [] }, { "id": 79203, "author_name": "yejiming", "author_url": "", "post_date": "05/19/2015 09:21:04", "content": "

I see two choices:

\n

First, you can use databases, and use SQL to manipulate your data.

\n

Second, use some programming language and you can do it all on your own. For this particular competition, you can sort the data by bidder_id, and then you can read the data row by row, compare the current row's bidder_id with last row's bidder_id. And then combine them if they have same bidder_id, create a new one otherwise

", "votes": null, "replies": [] }, { "id": 79206, "author_name": "patriczhao", "author_url": "", "post_date": "05/19/2015 09:32:37", "content": "

Just for this case, I think using vectorization can solve lots of performance problems.

", "votes": null, "replies": [] }, { "id": 79234, "author_name": "mongolia", "author_url": "", "post_date": "05/19/2015 12:38:47", "content": "

@Jiming Ye, what kind of program are you using for this competition ?

", "votes": null, "replies": [] }, { "id": 79251, "author_name": "dzidas", "author_url": "", "post_date": "05/19/2015 14:02:47", "content": "

I work with R only on this competition. I keep the memory clean by using rm() and gc() constantly. Once model is built I remove all objects except necessary for prediction. 

\n

I use Linux 8GB RAM and Mac 8GB RAM.

", "votes": null, "replies": [] }, { "id": 79256, "author_name": "mongolia", "author_url": "", "post_date": "05/19/2015 14:31:33", "content": "

@Dzidas, In your opinion, what is the main factor for this competition ?

", "votes": null, "replies": [] }, { "id": 79261, "author_name": "dzidas", "author_url": "", "post_date": "05/19/2015 14:47:25", "content": "

Please define "main factor" - are you after main difficulty, feature, etc.

", "votes": null, "replies": [] }, { "id": 79316, "author_name": "vkuznet", "author_url": "", "post_date": "05/19/2015 19:21:53", "content": "

You can read large datasets in R, and this one is not that large. Just use data.table package and its excellent fread function. On my laptop (8GB of RAM) I can read bids.csv file in 40 seconds.

", "votes": null, "replies": [] }, { "id": 79349, "author_name": "andrewgeisler", "author_url": "", "post_date": "05/19/2015 21:09:35", "content": "

Thanks for the ideas and responses everyone. @vk.net. It's not so much the reading in that I'm struggling with. I'm doing it the way you suggest. It's super fast ... Especially compared to just read.csv(). 

\n

My system is struggling when building and running the models. I've tried some sampling strategies.... Sort of some custom bagging methods if you will and it just doesn't seem very efficient. 

\n

I've also tried storing in MySQL and just reading in just samples so I'm not storing as much in mem  I've gotten mixed results. 

", "votes": null, "replies": [] }, { "id": 79535, "author_name": "rinzewind", "author_url": "", "post_date": "05/20/2015 22:20:31", "content": "

I am using a "poor man's approach" to this problem. I have a very old (but funcional) laptop (Dell D830) with 4 GB RAM.

\n

I created a SQLite database and loaded the three .csv data files into it, in different tables. I then created the necessary queries for feature extraction.

\n

Overall, the system is not incredibly fast, but I get my features matrix in about 5 minutes, including time employed by R in some internal calculations and more aggregations (there's only so much one can do with SQLite, but that's enough).

\n

I tried reading the whole file into memory, but this approach is much more manageable.

", "votes": null, "replies": [] }, { "id": 79541, "author_name": "rawalc", "author_url": "", "post_date": "05/20/2015 22:55:21", "content": "

That's why I use MATLAB. The DATASTORE object in it allows manipulation of parts of data as needed, keeping the memory light, and processing fast.

", "votes": null, "replies": [] }, { "id": 79669, "author_name": "jeffhebert", "author_url": "", "post_date": "05/21/2015 20:32:00", "content": "

I've been using dplyr to do data maniupulations, joins, etc. No problems at all on my older Windows laptop with 8GB RAM. The dplyr package works similar to SQL, and you don't have to setup any data connections.

\n

Good luck

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "79052": "", "79169": "", "79203": "", "79206": "", "79234": "", "79251": "", "79256": "", "79261": "", "79316": "", "79349": "", "79535": "", "79541": "", "79669": "" }, "source": "meta" }