{ "id": 10933, "title": "Google BigQuery", "url": "/competitions/avazu-ctr-prediction/discussion/10933", "author_name": "", "post_date": "2014-11-15T14:08:16.517Z", "votes": null, "comment_count": 5, "views": 1885, "content": "

Hi,

\n

Is it against the rules to use Google BigQuery as 'database'?

\n

I do not have enough Ram in my computer so I need a remote option, but I am afraid Google BigQuery may not qualify since it's not open source. 

\n

If it's agains the rules, could you recommend how to proceed with any  other remote specific option?

\n

Thanks a lot,

", "messages": [ { "id": "58114", "postDate": "11/15/2014 14:08:16", "content": "

Hi,

\n

Is it against the rules to use Google BigQuery as 'database'?

\n

I do not have enough Ram in my computer so I need a remote option, but I am afraid Google BigQuery may not qualify since it's not open source. 

\n

If it's agains the rules, could you recommend how to proceed with any  other remote specific option?

\n

Thanks a lot,

", "rawMarkdown": "", "votes": null }, { "id": "58116", "postDate": "11/15/2014 16:48:28", "content": "

Many contestants use cloud resources all the time, especially ec2. The only downside is when your credit card bill comes. If you manage to win, porting your sql to an open source platform probably would not be a big deal.

", "rawMarkdown": "", "votes": null }, { "id": "58117", "postDate": "11/15/2014 18:04:48", "content": "

Thanks James.

\n

Any idea on how much money I would spend for this specific competition using ec2 or google big query?  Are we talking about around $100? or more like $500?

\n

I tried to do my homework by checking the prices on both, but it seems almost impossible to do a calculation in advance.

\n

Thank you,

", "rawMarkdown": "", "votes": null }, { "id": "58121", "postDate": "11/15/2014 20:41:17", "content": "

No, I don't have an idea because there are too many variables like what exactly you do and how much you do it. I will say that I am running an ec2 c3.8xlarge instance today (for epilepsy competition) with a spot price cap of 30 cents an hour, which is much better than on demand pricing of $1.68 per hour. However if the spot price goes above 30 cents Amazon will terminate my instance. Plus there are costs for storage and data transfer.

\n

If you really just want to write sql queries against the data then you can download  a free dbms like postgresql and run it on your own machine. It will obviously be slower than bigquery. If you want to build models on all the data without downsampling then vowpal wabbit or the tinrtgu code can do this with very little memory.

\n

If you want to do, say, a gbm then renting a cloud machine may be the only way to go.  On Amazon you can monitor spot prices on different types of machines and the availability zones so that you jump on the cloud machine when the price becomes attractive enough.

\n

Unfortunately for competitions with big datasets like Avazu and Criteo I do think those who are willing/able to spend money on Cloud resources, or otherwise have access to powerful machine, have an advantage. There has been some discussion on the forum about kaggle partnering with a cloud provider to provide discounts to high ranking participants, but don't know if it will happen.

", "rawMarkdown": "", "votes": null }, { "id": "58146", "postDate": "11/16/2014 19:10:35", "content": "

In my opinion it is much more cost effective to just lease a dedicated server. For similar specs you will pay 2-3 times less than ec2. To give you an example you can find 6 Core Server - Intel Xeon E5 1650 V2 + 128 GB RAM for less than $300 a month. Good luck if you wanted to lease this kind of spec from amazon.

\n

Also 50 million records is not more than a db like postgresql or mysql can handle. Just create reasonable indices and you're good to go. I think many people start to think about these Big Data tools a little bit too early.

", "rawMarkdown": "", "votes": null }, { "id": "58181", "postDate": "11/17/2014 05:54:26", "content": "

Never pay the on demand price on Amazon, it's much too high. Prices for spot instances are often very attractive; at this moment less than 13 cents an hour for r3.4xlarge which has 16 cpu and 122 GB of ram. 

\n

The downside is you're always at risk of instance termination if the spot price goes above your bid. And with Amazon everything is a la carte so you have storage, data transfer, AMIs, keypairs, and other things that you get nickeled and dimed for. And they won't be forgiving about mistakes, like forgetting to terminate an instance.

\n

The costs are not very transparent at all. So I can see how a fixed all in price for a dedicated server could be attractive.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 58116, "author_name": "jfkingiii", "author_url": "", "post_date": "11/15/2014 16:48:28", "content": "

Many contestants use cloud resources all the time, especially ec2. The only downside is when your credit card bill comes. If you manage to win, porting your sql to an open source platform probably would not be a big deal.

", "votes": null, "replies": [] }, { "id": 58117, "author_name": "fernandoconstantino", "author_url": "", "post_date": "11/15/2014 18:04:48", "content": "

Thanks James.

\n

Any idea on how much money I would spend for this specific competition using ec2 or google big query?  Are we talking about around $100? or more like $500?

\n

I tried to do my homework by checking the prices on both, but it seems almost impossible to do a calculation in advance.

\n

Thank you,

", "votes": null, "replies": [] }, { "id": 58121, "author_name": "jfkingiii", "author_url": "", "post_date": "11/15/2014 20:41:17", "content": "

No, I don't have an idea because there are too many variables like what exactly you do and how much you do it. I will say that I am running an ec2 c3.8xlarge instance today (for epilepsy competition) with a spot price cap of 30 cents an hour, which is much better than on demand pricing of $1.68 per hour. However if the spot price goes above 30 cents Amazon will terminate my instance. Plus there are costs for storage and data transfer.

\n

If you really just want to write sql queries against the data then you can download  a free dbms like postgresql and run it on your own machine. It will obviously be slower than bigquery. If you want to build models on all the data without downsampling then vowpal wabbit or the tinrtgu code can do this with very little memory.

\n

If you want to do, say, a gbm then renting a cloud machine may be the only way to go.  On Amazon you can monitor spot prices on different types of machines and the availability zones so that you jump on the cloud machine when the price becomes attractive enough.

\n

Unfortunately for competitions with big datasets like Avazu and Criteo I do think those who are willing/able to spend money on Cloud resources, or otherwise have access to powerful machine, have an advantage. There has been some discussion on the forum about kaggle partnering with a cloud provider to provide discounts to high ranking participants, but don't know if it will happen.

", "votes": null, "replies": [] }, { "id": 58146, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/16/2014 19:10:35", "content": "

In my opinion it is much more cost effective to just lease a dedicated server. For similar specs you will pay 2-3 times less than ec2. To give you an example you can find 6 Core Server - Intel Xeon E5 1650 V2 + 128 GB RAM for less than $300 a month. Good luck if you wanted to lease this kind of spec from amazon.

\n

Also 50 million records is not more than a db like postgresql or mysql can handle. Just create reasonable indices and you're good to go. I think many people start to think about these Big Data tools a little bit too early.

", "votes": null, "replies": [] }, { "id": 58181, "author_name": "jfkingiii", "author_url": "", "post_date": "11/17/2014 05:54:26", "content": "

Never pay the on demand price on Amazon, it's much too high. Prices for spot instances are often very attractive; at this moment less than 13 cents an hour for r3.4xlarge which has 16 cpu and 122 GB of ram. 

\n

The downside is you're always at risk of instance termination if the spot price goes above your bid. And with Amazon everything is a la carte so you have storage, data transfer, AMIs, keypairs, and other things that you get nickeled and dimed for. And they won't be forgiving about mistakes, like forgetting to terminate an instance.

\n

The costs are not very transparent at all. So I can see how a fixed all in price for a dedicated server could be attractive.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "58114": "", "58116": "", "58117": "", "58121": "", "58146": "", "58181": "" }, "source": "meta" }