What kind of hardware did you use for this competition? I am particularly interested if any of the top teams used AWS or something like it. I often see AWS recommended, but don't often see top Kagglers mention that they used AWS. If you did use AWS, how much did it cost?
", "messages": [ { "id": "144108", "postDate": "11/12/2016 11:21:46", "content": "What kind of hardware did you use for this competition? I am particularly interested if any of the top teams used AWS or something like it. I often see AWS recommended, but don't often see top Kagglers mention that they used AWS. If you did use AWS, how much did it cost?
", "rawMarkdown": "What kind of hardware did you use for this competition? I am particularly interested if any of the top teams used AWS or something like it. I often see AWS recommended, but don't often see top Kagglers mention that they used AWS. If you did use AWS, how much did it cost?", "votes": null }, { "id": "144111", "postDate": "11/12/2016 11:38:23", "content": "Personally, my own PC (16GB/8core) - and for faster modelling, the trial version of google cloud. I realized they currently have a great offer with free 300USD voucher for first three months, limited to 24 cores at any one time. AWS spot pricing was getting a little expensive, and the g-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data)....
", "rawMarkdown": "Personally, my own PC (16GB/8core) - and for faster modelling, the trial version of google cloud. I realized they currently have a great offer with free 300USD voucher for first three months, limited to 24 cores at any one time. AWS spot pricing was getting a little expensive, and the g-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data)....", "votes": null }, { "id": "144115", "postDate": "11/12/2016 12:13:18", "content": "I used my own laptop with 16GB and 4 threads, then later in the competition started to use AWS spot instances to speedup training or train multiple models at once (8 or 16 cores, usually costed 0.1-0.25 $/hr, exact instance type selected by current price history - spot prices are very unstable now and sometimes become crazy high).
", "rawMarkdown": "I used my own laptop with 16GB and 4 threads, then later in the competition started to use AWS spot instances to speedup training or train multiple models at once (8 or 16 cores, usually costed 0.1-0.25 $/hr, exact instance type selected by current price history - spot prices are very unstable now and sometimes become crazy high).", "votes": null }, { "id": "144119", "postDate": "11/12/2016 12:39:43", "content": "[quote=Darragh;144111]
\n\ng-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data)....
\n\n[/quote]
\n\nAgreeing with this. + they nearly never get shut down I noticed in the past.
\n\nAnd 32 core + 128GB RAM + 375GB SSD, with 24 hours to grab all your data if it gets shutdown, for only $45/month (4-6h/day) when you start a new instance every 24 hours, is hard to beat. Especially considering the price is fixed beforehand, unlike in AWS.
\n\nIf your instance gets shutdown within the first 10 minutes, you pay nothing and keep all your data for 24 hours. Over a year you would pay about $500 (for 32 CPU cores + 128 GB RAM, 4-6h/day), and have about 1 random shutdown every 2 weeks (according to one of my clients).
\n\nUS location is the way to go for the cheapest prices for Google. Those who are interested can fetch all the prices here: https://cloudpricingcalculator.appspot.com/static/data/pricelist.json (the price varies rarely much):
\n\n "CP-COMPUTEENGINE-CUSTOM-VM-CORE-PREEMPTIBLE": {\n "us": 0.00698,\n "europe": 0.00768,\n "asia": 0.00768,\n "asia-east": 0.00768,\n "asia-northeast": 0.00883\n },\n "CP-COMPUTEENGINE-CUSTOM-VM-RAM-PREEMPTIBLE": {\n "us": 0.00094,\n "europe": 0.00103,\n "asia": 0.00103,\n "asia-east": 0.00103,\n "asia-northeast": 0.001178\n },\nIf you shrink down your CPU/RAM requirements, you can get away with a full 24h running instance (which can still be terminated randomly - but restartable) for the price of a coffee a day (16 core, 60 CPUs, $28/month, 6h/day).
\n\nedit: missing 6h/day, Ctrl+F failure
", "rawMarkdown": "[quote=Darragh;144111]\r\n\r\ng-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data).... \r\n\r\n[/quote]\r\n\r\nAgreeing with this. + they nearly never get shut down I noticed in the past.\r\n\r\nAnd 32 core + 128GB RAM + 375GB SSD, with 24 hours to grab all your data if it gets shutdown, for only $45/month (4-6h/day) when you start a new instance every 24 hours, is hard to beat. Especially considering the price is fixed beforehand, unlike in AWS.\r\n\r\nIf your instance gets shutdown within the first 10 minutes, you pay nothing and keep all your data for 24 hours. Over a year you would pay about $500 (for 32 CPU cores + 128 GB RAM, 4-6h/day), and have about 1 random shutdown every 2 weeks (according to one of my clients).\r\n\r\nUS location is the way to go for the cheapest prices for Google. Those who are interested can fetch all the prices here: https://cloudpricingcalculator.appspot.com/static/data/pricelist.json (the price varies rarely much):\r\n\r\n \"CP-COMPUTEENGINE-CUSTOM-VM-CORE-PREEMPTIBLE\": {\r\n \"us\": 0.00698,\r\n \"europe\": 0.00768,\r\n \"asia\": 0.00768,\r\n \"asia-east\": 0.00768,\r\n \"asia-northeast\": 0.00883\r\n },\r\n \"CP-COMPUTEENGINE-CUSTOM-VM-RAM-PREEMPTIBLE\": {\r\n \"us\": 0.00094,\r\n \"europe\": 0.00103,\r\n \"asia\": 0.00103,\r\n \"asia-east\": 0.00103,\r\n \"asia-northeast\": 0.001178\r\n },\r\n\r\nIf you shrink down your CPU/RAM requirements, you can get away with a full 24h running instance (which can still be terminated randomly - but restartable) for the price of a coffee a day (16 core, 60 CPUs, $28/month, 6h/day).\r\n\r\nedit: missing 6h/day, Ctrl+F failure", "votes": null }, { "id": "144125", "postDate": "11/12/2016 13:20:32", "content": "I did use AWS, but I used proprietary database server software and a specialized PC client with a point and click GUI for developing and running algorithms. Unfortunately, I don't know anything about the cost since it was installed by one of my consulting clients. I know it is a relatively small Linux server with a single 2.3 GHz Intel XEON processor, 32 GB of RAM, and about 1.2 TB storage mostly attached via network links. The performance is pretty good, enabling me to process tables with hundreds of millions of rows in only a couple minutes.
\n\nThis is my first competition and I only had 20 or 30 hours to work on it over the past 2 weeks. I'm just trying to figure out how this all works, and I am looking forward to having a lot more fun with the next competition, hopefully also with better results. My question is if the use of proprietary software might disqualify me if I ever did qualify as one of the winners? Are we restricted at to the software we can use?
\n\nIn case you're interested, I'm attaching a screenshot of the PC client interface.
", "rawMarkdown": "I did use AWS, but I used proprietary database server software and a specialized PC client with a point and click GUI for developing and running algorithms. Unfortunately, I don't know anything about the cost since it was installed by one of my consulting clients. I know it is a relatively small Linux server with a single 2.3 GHz Intel XEON processor, 32 GB of RAM, and about 1.2 TB storage mostly attached via network links. The performance is pretty good, enabling me to process tables with hundreds of millions of rows in only a couple minutes.\r\n\r\nThis is my first competition and I only had 20 or 30 hours to work on it over the past 2 weeks. I'm just trying to figure out how this all works, and I am looking forward to having a lot more fun with the next competition, hopefully also with better results. My question is if the use of proprietary software might disqualify me if I ever did qualify as one of the winners? Are we restricted at to the software we can use? \r\n\r\nIn case you're interested, I'm attaching a screenshot of the PC client interface.", "votes": null }, { "id": "144126", "postDate": "11/12/2016 13:29:45", "content": "I used to use AWS instances, but I found it was costing a lot to use it regularly. So I had my 17 year old son build me a data-crunching PC. Since he assembled it himself, it cost about $1650. It has a 6-core (12 thread) processor and 64 GB RAM. He used www.pcpartpicker.com to help him pick parts that were compatible and find the best prices on those parts. He made the process look easy. If I'd known it was that easy, I would have built my own PC years ago. I think this is my specific configuration: http://pcpartpicker.com/list/fRxvgs (this configuration is months old and therefore is probably out of date, but it gives you an idea of what pcpartpicker does).
", "rawMarkdown": "I used to use AWS instances, but I found it was costing a lot to use it regularly. So I had my 17 year old son build me a data-crunching PC. Since he assembled it himself, it cost about $1650. It has a 6-core (12 thread) processor and 64 GB RAM. He used www.pcpartpicker.com to help him pick parts that were compatible and find the best prices on those parts. He made the process look easy. If I'd known it was that easy, I would have built my own PC years ago. I think this is my specific configuration: http://pcpartpicker.com/list/fRxvgs (this configuration is months old and therefore is probably out of date, but it gives you an idea of what pcpartpicker does).", "votes": null }, { "id": "144129", "postDate": "11/12/2016 13:41:46", "content": "16GB laptop with 8 cores the whole time. Mostly worked with the data in tall format.
", "rawMarkdown": "16GB laptop with 8 cores the whole time. Mostly worked with the data in tall format.", "votes": null }, { "id": "144130", "postDate": "11/12/2016 13:45:24", "content": "Just to echo @BreakfastPirate -- building your own computer is much easier than you might think. These days you can easily build a very good machine with 32+ GB RAM and a solid 4+ core processor for $1000. Computers are just adult legos. Many resources such as /r/buildapc are full of people willing to help you pick out what is compatible and appropriate. If you have the money, this is a nice alternative to eternal pay-by-the month
", "rawMarkdown": "Just to echo @BreakfastPirate -- building your own computer is **much** easier than you might think. These days you can easily build a very good machine with 32+ GB RAM and a solid 4+ core processor for $1000. Computers are just adult legos. Many resources such as [/r/buildapc][1] are full of people willing to help you pick out what is compatible and appropriate. If you have the money, this is a nice alternative to eternal pay-by-the month\r\n\r\n\r\n [1]: https://m.reddit.com/r/buildapc/", "votes": null }, { "id": "144131", "postDate": "11/12/2016 13:52:21", "content": "Hi\nMy personal laptop with i7 and 16gb of ram. \nFor large dataset. I use azure vm with 56gb of ram.
", "rawMarkdown": "Hi\r\nMy personal laptop with i7 and 16gb of ram. \r\nFor large dataset. I use azure vm with 56gb of ram.", "votes": null }, { "id": "144133", "postDate": "11/12/2016 13:59:34", "content": "I used GCP... personally I find it much easier (and cheaper) than AWS.
\n\nSpecifically, I worked on the 104GB-RAM and 16-core instance, all pre-emptible. This is the first time I didn't use my laptop at all for the entire competition. I used the Google Storage for accessing the raw files, RStudio server and Jupyter notebooks for codes on the remote instances and GitHub for version control.
\n\nThat instance costs $0.212 per hour, my aggregate bill was ~ $25 for 1.5 months, which equates to ~ 120 hours of usage, or ~ 2-3 hours of usage each day.
\n\nI don't think I had a choice. I have a 4GB MacBook Air :-|
", "rawMarkdown": "I used GCP... personally I find it much easier (and cheaper) than AWS.\r\n\r\nSpecifically, I worked on the 104GB-RAM and 16-core instance, all pre-emptible. This is the first time I didn't use my laptop at all for the entire competition. I used the Google Storage for accessing the raw files, RStudio server and Jupyter notebooks for codes on the remote instances and GitHub for version control.\r\n\r\nThat instance costs $0.212 per hour, my aggregate bill was ~ $25 for 1.5 months, which equates to ~ 120 hours of usage, or ~ 2-3 hours of usage each day.\r\n\r\nI don't think I had a choice. I have a 4GB MacBook Air :-|", "votes": null }, { "id": "144134", "postDate": "11/12/2016 14:05:05", "content": "I use my Asus Gamers Republic laptop with i7 and 24G. And also 48G swap file on SSD helps sometimes. Anyway I first prefer to compress data to more convinient formats, e.g. binary values to np.int8 and so on. This preparation greatly reduces memory requirements.
", "rawMarkdown": "I use my Asus Gamers Republic laptop with i7 and 24G. And also 48G swap file on SSD helps sometimes. Anyway I first prefer to compress data to more convinient formats, e.g. binary values to np.int8 and so on. This preparation greatly reduces memory requirements.", "votes": null }, { "id": "144151", "postDate": "11/12/2016 16:33:17", "content": "I used my macbookpro (16GB, 4 cores/8 threads)
\n\nIt was limiting in some ways, for instance I could not reliably computer moving averages as Darragh described. I had kernel shutdown most probably due to memory allocation failures in python or xgboost.
", "rawMarkdown": "I used my macbookpro (16GB, 4 cores/8 threads)\r\n\r\nIt was limiting in some ways, for instance I could not reliably computer moving averages as Darragh described. I had kernel shutdown most probably due to memory allocation failures in python or xgboost.", "votes": null }, { "id": "144298", "postDate": "11/13/2016 17:10:43", "content": "I also used my Macbook Pro (16gb, 4 cores/8 threads). It was just annoying enough to process data (as quite a bit was required for this) that I lost motivation to try new things after hitting the wall a few times as it takes something like 1 hour minimum just to test if a new idea works. A week ago I got a new desktop built and indeed the process was extremely easy. Being in Europe and not speaking the local language I had it custom built for 75 euros extra but in the US it's extremely easy now to build your own desktop.
", "rawMarkdown": "I also used my Macbook Pro (16gb, 4 cores/8 threads). It was just annoying enough to process data (as quite a bit was required for this) that I lost motivation to try new things after hitting the wall a few times as it takes something like 1 hour minimum just to test if a new idea works. A week ago I got a new desktop built and indeed the process was extremely easy. Being in Europe and not speaking the local language I had it custom built for 75 euros extra but in the US it's extremely easy now to build your own desktop.", "votes": null }, { "id": "144306", "postDate": "11/13/2016 17:46:24", "content": "I used my own laptop (16gb, 4 threads & nvidia graphic card). probably try aws next time.
", "rawMarkdown": "I used my own laptop (16gb, 4 threads & nvidia graphic card). probably try aws next time.", "votes": null } ], "comments": [ { "id": 144111, "author_name": "darraghdog", "author_url": "", "post_date": "11/12/2016 11:38:23", "content": "Personally, my own PC (16GB/8core) - and for faster modelling, the trial version of google cloud. I realized they currently have a great offer with free 300USD voucher for first three months, limited to 24 cores at any one time. AWS spot pricing was getting a little expensive, and the g-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data)....
", "votes": null, "replies": [] }, { "id": 144115, "author_name": "alexeynoskov", "author_url": "", "post_date": "11/12/2016 12:13:18", "content": "I used my own laptop with 16GB and 4 threads, then later in the competition started to use AWS spot instances to speedup training or train multiple models at once (8 or 16 cores, usually costed 0.1-0.25 $/hr, exact instance type selected by current price history - spot prices are very unstable now and sometimes become crazy high).
", "votes": null, "replies": [] }, { "id": 144119, "author_name": "laurae2", "author_url": "", "post_date": "11/12/2016 12:39:43", "content": "[quote=Darragh;144111]
\n\ng-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data)....
\n\n[/quote]
\n\nAgreeing with this. + they nearly never get shut down I noticed in the past.
\n\nAnd 32 core + 128GB RAM + 375GB SSD, with 24 hours to grab all your data if it gets shutdown, for only $45/month (4-6h/day) when you start a new instance every 24 hours, is hard to beat. Especially considering the price is fixed beforehand, unlike in AWS.
\n\nIf your instance gets shutdown within the first 10 minutes, you pay nothing and keep all your data for 24 hours. Over a year you would pay about $500 (for 32 CPU cores + 128 GB RAM, 4-6h/day), and have about 1 random shutdown every 2 weeks (according to one of my clients).
\n\nUS location is the way to go for the cheapest prices for Google. Those who are interested can fetch all the prices here: https://cloudpricingcalculator.appspot.com/static/data/pricelist.json (the price varies rarely much):
\n\n "CP-COMPUTEENGINE-CUSTOM-VM-CORE-PREEMPTIBLE": {\n "us": 0.00698,\n "europe": 0.00768,\n "asia": 0.00768,\n "asia-east": 0.00768,\n "asia-northeast": 0.00883\n },\n "CP-COMPUTEENGINE-CUSTOM-VM-RAM-PREEMPTIBLE": {\n "us": 0.00094,\n "europe": 0.00103,\n "asia": 0.00103,\n "asia-east": 0.00103,\n "asia-northeast": 0.001178\n },\nIf you shrink down your CPU/RAM requirements, you can get away with a full 24h running instance (which can still be terminated randomly - but restartable) for the price of a coffee a day (16 core, 60 CPUs, $28/month, 6h/day).
\n\nedit: missing 6h/day, Ctrl+F failure
", "votes": null, "replies": [] }, { "id": 144125, "author_name": "gdangelo", "author_url": "", "post_date": "11/12/2016 13:20:32", "content": "I did use AWS, but I used proprietary database server software and a specialized PC client with a point and click GUI for developing and running algorithms. Unfortunately, I don't know anything about the cost since it was installed by one of my consulting clients. I know it is a relatively small Linux server with a single 2.3 GHz Intel XEON processor, 32 GB of RAM, and about 1.2 TB storage mostly attached via network links. The performance is pretty good, enabling me to process tables with hundreds of millions of rows in only a couple minutes.
\n\nThis is my first competition and I only had 20 or 30 hours to work on it over the past 2 weeks. I'm just trying to figure out how this all works, and I am looking forward to having a lot more fun with the next competition, hopefully also with better results. My question is if the use of proprietary software might disqualify me if I ever did qualify as one of the winners? Are we restricted at to the software we can use?
\n\nIn case you're interested, I'm attaching a screenshot of the PC client interface.
", "votes": null, "replies": [] }, { "id": 144126, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/12/2016 13:29:45", "content": "I used to use AWS instances, but I found it was costing a lot to use it regularly. So I had my 17 year old son build me a data-crunching PC. Since he assembled it himself, it cost about $1650. It has a 6-core (12 thread) processor and 64 GB RAM. He used www.pcpartpicker.com to help him pick parts that were compatible and find the best prices on those parts. He made the process look easy. If I'd known it was that easy, I would have built my own PC years ago. I think this is my specific configuration: http://pcpartpicker.com/list/fRxvgs (this configuration is months old and therefore is probably out of date, but it gives you an idea of what pcpartpicker does).
", "votes": null, "replies": [] }, { "id": 144129, "author_name": "ben519", "author_url": "", "post_date": "11/12/2016 13:41:46", "content": "16GB laptop with 8 cores the whole time. Mostly worked with the data in tall format.
", "votes": null, "replies": [] }, { "id": 144130, "author_name": "apryor6", "author_url": "", "post_date": "11/12/2016 13:45:24", "content": "Just to echo @BreakfastPirate -- building your own computer is much easier than you might think. These days you can easily build a very good machine with 32+ GB RAM and a solid 4+ core processor for $1000. Computers are just adult legos. Many resources such as /r/buildapc are full of people willing to help you pick out what is compatible and appropriate. If you have the money, this is a nice alternative to eternal pay-by-the month
", "votes": null, "replies": [] }, { "id": 144131, "author_name": "vanausloos", "author_url": "", "post_date": "11/12/2016 13:52:21", "content": "Hi\nMy personal laptop with i7 and 16gb of ram. \nFor large dataset. I use azure vm with 56gb of ram.
", "votes": null, "replies": [] }, { "id": 144133, "author_name": "rohanrao", "author_url": "", "post_date": "11/12/2016 13:59:34", "content": "I used GCP... personally I find it much easier (and cheaper) than AWS.
\n\nSpecifically, I worked on the 104GB-RAM and 16-core instance, all pre-emptible. This is the first time I didn't use my laptop at all for the entire competition. I used the Google Storage for accessing the raw files, RStudio server and Jupyter notebooks for codes on the remote instances and GitHub for version control.
\n\nThat instance costs $0.212 per hour, my aggregate bill was ~ $25 for 1.5 months, which equates to ~ 120 hours of usage, or ~ 2-3 hours of usage each day.
\n\nI don't think I had a choice. I have a 4GB MacBook Air :-|
", "votes": null, "replies": [] }, { "id": 144134, "author_name": "nchervyakov", "author_url": "", "post_date": "11/12/2016 14:05:05", "content": "I use my Asus Gamers Republic laptop with i7 and 24G. And also 48G swap file on SSD helps sometimes. Anyway I first prefer to compress data to more convinient formats, e.g. binary values to np.int8 and so on. This preparation greatly reduces memory requirements.
", "votes": null, "replies": [] }, { "id": 144151, "author_name": "cpmpml", "author_url": "", "post_date": "11/12/2016 16:33:17", "content": "I used my macbookpro (16GB, 4 cores/8 threads)
\n\nIt was limiting in some ways, for instance I could not reliably computer moving averages as Darragh described. I had kernel shutdown most probably due to memory allocation failures in python or xgboost.
", "votes": null, "replies": [] }, { "id": 144298, "author_name": "qwang88", "author_url": "", "post_date": "11/13/2016 17:10:43", "content": "I also used my Macbook Pro (16gb, 4 cores/8 threads). It was just annoying enough to process data (as quite a bit was required for this) that I lost motivation to try new things after hitting the wall a few times as it takes something like 1 hour minimum just to test if a new idea works. A week ago I got a new desktop built and indeed the process was extremely easy. Being in Europe and not speaking the local language I had it custom built for 75 euros extra but in the US it's extremely easy now to build your own desktop.
", "votes": null, "replies": [] }, { "id": 144306, "author_name": "stgrmks", "author_url": "", "post_date": "11/13/2016 17:46:24", "content": "I used my own laptop (16gb, 4 threads & nvidia graphic card). probably try aws next time.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "144108": "What kind of hardware did you use for this competition? I am particularly interested if any of the top teams used AWS or something like it. I often see AWS recommended, but don't often see top Kagglers mention that they used AWS. If you did use AWS, how much did it cost?", "144111": "Personally, my own PC (16GB/8core) - and for faster modelling, the trial version of google cloud. I realized they currently have a great offer with free 300USD voucher for first three months, limited to 24 cores at any one time. AWS spot pricing was getting a little expensive, and the g-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data)....", "144115": "I used my own laptop with 16GB and 4 threads, then later in the competition started to use AWS spot instances to speedup training or train multiple models at once (8 or 16 cores, usually costed 0.1-0.25 $/hr, exact instance type selected by current price history - spot prices are very unstable now and sometimes become crazy high).", "144119": "[quote=Darragh;144111]\r\n\r\ng-cloud preemtible prices seem to beat it (~ 22c/hr for 16 core - limited to 24 hour runtime, but it can be restarted after 24 hrs without loss of data).... \r\n\r\n[/quote]\r\n\r\nAgreeing with this. + they nearly never get shut down I noticed in the past.\r\n\r\nAnd 32 core + 128GB RAM + 375GB SSD, with 24 hours to grab all your data if it gets shutdown, for only $45/month (4-6h/day) when you start a new instance every 24 hours, is hard to beat. Especially considering the price is fixed beforehand, unlike in AWS.\r\n\r\nIf your instance gets shutdown within the first 10 minutes, you pay nothing and keep all your data for 24 hours. Over a year you would pay about $500 (for 32 CPU cores + 128 GB RAM, 4-6h/day), and have about 1 random shutdown every 2 weeks (according to one of my clients).\r\n\r\nUS location is the way to go for the cheapest prices for Google. Those who are interested can fetch all the prices here: https://cloudpricingcalculator.appspot.com/static/data/pricelist.json (the price varies rarely much):\r\n\r\n \"CP-COMPUTEENGINE-CUSTOM-VM-CORE-PREEMPTIBLE\": {\r\n \"us\": 0.00698,\r\n \"europe\": 0.00768,\r\n \"asia\": 0.00768,\r\n \"asia-east\": 0.00768,\r\n \"asia-northeast\": 0.00883\r\n },\r\n \"CP-COMPUTEENGINE-CUSTOM-VM-RAM-PREEMPTIBLE\": {\r\n \"us\": 0.00094,\r\n \"europe\": 0.00103,\r\n \"asia\": 0.00103,\r\n \"asia-east\": 0.00103,\r\n \"asia-northeast\": 0.001178\r\n },\r\n\r\nIf you shrink down your CPU/RAM requirements, you can get away with a full 24h running instance (which can still be terminated randomly - but restartable) for the price of a coffee a day (16 core, 60 CPUs, $28/month, 6h/day).\r\n\r\nedit: missing 6h/day, Ctrl+F failure", "144125": "I did use AWS, but I used proprietary database server software and a specialized PC client with a point and click GUI for developing and running algorithms. Unfortunately, I don't know anything about the cost since it was installed by one of my consulting clients. I know it is a relatively small Linux server with a single 2.3 GHz Intel XEON processor, 32 GB of RAM, and about 1.2 TB storage mostly attached via network links. The performance is pretty good, enabling me to process tables with hundreds of millions of rows in only a couple minutes.\r\n\r\nThis is my first competition and I only had 20 or 30 hours to work on it over the past 2 weeks. I'm just trying to figure out how this all works, and I am looking forward to having a lot more fun with the next competition, hopefully also with better results. My question is if the use of proprietary software might disqualify me if I ever did qualify as one of the winners? Are we restricted at to the software we can use? \r\n\r\nIn case you're interested, I'm attaching a screenshot of the PC client interface.", "144126": "I used to use AWS instances, but I found it was costing a lot to use it regularly. So I had my 17 year old son build me a data-crunching PC. Since he assembled it himself, it cost about $1650. It has a 6-core (12 thread) processor and 64 GB RAM. He used www.pcpartpicker.com to help him pick parts that were compatible and find the best prices on those parts. He made the process look easy. If I'd known it was that easy, I would have built my own PC years ago. I think this is my specific configuration: http://pcpartpicker.com/list/fRxvgs (this configuration is months old and therefore is probably out of date, but it gives you an idea of what pcpartpicker does).", "144129": "16GB laptop with 8 cores the whole time. Mostly worked with the data in tall format.", "144130": "Just to echo @BreakfastPirate -- building your own computer is **much** easier than you might think. These days you can easily build a very good machine with 32+ GB RAM and a solid 4+ core processor for $1000. Computers are just adult legos. Many resources such as [/r/buildapc][1] are full of people willing to help you pick out what is compatible and appropriate. If you have the money, this is a nice alternative to eternal pay-by-the month\r\n\r\n\r\n [1]: https://m.reddit.com/r/buildapc/", "144131": "Hi\r\nMy personal laptop with i7 and 16gb of ram. \r\nFor large dataset. I use azure vm with 56gb of ram.", "144133": "I used GCP... personally I find it much easier (and cheaper) than AWS.\r\n\r\nSpecifically, I worked on the 104GB-RAM and 16-core instance, all pre-emptible. This is the first time I didn't use my laptop at all for the entire competition. I used the Google Storage for accessing the raw files, RStudio server and Jupyter notebooks for codes on the remote instances and GitHub for version control.\r\n\r\nThat instance costs $0.212 per hour, my aggregate bill was ~ $25 for 1.5 months, which equates to ~ 120 hours of usage, or ~ 2-3 hours of usage each day.\r\n\r\nI don't think I had a choice. I have a 4GB MacBook Air :-|", "144134": "I use my Asus Gamers Republic laptop with i7 and 24G. And also 48G swap file on SSD helps sometimes. Anyway I first prefer to compress data to more convinient formats, e.g. binary values to np.int8 and so on. This preparation greatly reduces memory requirements.", "144151": "I used my macbookpro (16GB, 4 cores/8 threads)\r\n\r\nIt was limiting in some ways, for instance I could not reliably computer moving averages as Darragh described. I had kernel shutdown most probably due to memory allocation failures in python or xgboost.", "144298": "I also used my Macbook Pro (16gb, 4 cores/8 threads). It was just annoying enough to process data (as quite a bit was required for this) that I lost motivation to try new things after hitting the wall a few times as it takes something like 1 hour minimum just to test if a new idea works. A week ago I got a new desktop built and indeed the process was extremely easy. Being in Europe and not speaking the local language I had it custom built for 75 euros extra but in the US it's extremely easy now to build your own desktop.", "144306": "I used my own laptop (16gb, 4 threads & nvidia graphic card). probably try aws next time." }, "source": "meta" }