{
  "id": 21003,
  "title": "AWS Usage",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21003",
  "author_name": "",
  "post_date": "2016-05-16T23:49:58.193Z",
  "votes": 1,
  "comment_count": 18,
  "views": 3532,
  "content": "<p>Does anyone have any experience using AWS for this type of problem? How expensive would it be in practice?</p>",
  "messages": [
    {
      "id": "120268",
      "postDate": "05/16/2016 23:49:58",
      "content": "<p>Does anyone have any experience using AWS for this type of problem? How expensive would it be in practice?</p>",
      "rawMarkdown": "Does anyone have any experience using AWS for this type of problem? How expensive would it be in practice?",
      "votes": null
    },
    {
      "id": "120269",
      "postDate": "05/17/2016 00:21:22",
      "content": "<p>How expensive it is will depend on what type of instance you select and the location of the servers you select (which they recommend are the ones closest to you). But for instances with the most RAM (~244GB) and CPU(32 cores) in Northern California (where I am) spot pricing about $0.33/hour, but sometimes it spikes up and it may kill your instance if your bid price isn't high enough. So it's at least ~$8/day but could easily go to $30-$40 if the prices go up. </p>\n\n<p>For fun I tried to run an xgboost with 100,000+ classes on my desktop and it tried to allocate a vector that was ~83,000 GB or something ridiculous like that, so I don't think AWS is going to help with that in case that was something you were considering.</p>",
      "rawMarkdown": "How expensive it is will depend on what type of instance you select and the location of the servers you select (which they recommend are the ones closest to you). But for instances with the most RAM (~244GB) and CPU(32 cores) in Northern California (where I am) spot pricing about $0.33/hour, but sometimes it spikes up and it may kill your instance if your bid price isn't high enough. So it's at least ~$8/day but could easily go to $30-$40 if the prices go up. \r\n\r\nFor fun I tried to run an xgboost with 100,000+ classes on my desktop and it tried to allocate a vector that was ~83,000 GB or something ridiculous like that, so I don't think AWS is going to help with that in case that was something you were considering.",
      "votes": null
    },
    {
      "id": "120271",
      "postDate": "05/17/2016 01:01:29",
      "content": "<p>Thanks for answering. My plan was to chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid. Local cv indicates this approach would get above 0.55, but on my laptop it would take a long time to run - around a week I think though I'm not sure exactly. Would it be cheaper to do something like this? I wouldn't need a good computer, I'd just need lots of cores.</p>",
      "rawMarkdown": "Thanks for answering. My plan was to chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid. Local cv indicates this approach would get above 0.55, but on my laptop it would take a long time to run - around a week I think though I'm not sure exactly. Would it be cheaper to do something like this? I wouldn't need a good computer, I'd just need lots of cores.",
      "votes": null
    },
    {
      "id": "120282",
      "postDate": "05/17/2016 03:03:54",
      "content": "<p>Lately, I've found better and cheaper alternatives than AWS. The VPS I'm using right now costs 25$ per quarter (16 GB memory and 2 cores and uptime of 99.9%).</p>\n\n<p>Another alternative is using Databricks if you know Spark. </p>\n\n<p>I'd recommend that for everything except GPU instances (i.e deep learning). Then it is still cheaper to buy a new GPU since AWS bills (or any other provider) for GPU instances can go up to 150$ easily in a month. </p>",
      "rawMarkdown": "Lately, I've found better and cheaper alternatives than AWS. The VPS I'm using right now costs 25$ per quarter (16 GB memory and 2 cores and uptime of 99.9%).\r\n\r\nAnother alternative is using Databricks if you know Spark. \r\n\r\nI'd recommend that for everything except GPU instances (i.e deep learning). Then it is still cheaper to buy a new GPU since AWS bills (or any other provider) for GPU instances can go up to 150$ easily in a month.",
      "votes": null
    },
    {
      "id": "120295",
      "postDate": "05/17/2016 06:58:55",
      "content": "<p>@Jared, AWS has like 30 different options so you'd really have to get in there yourself and check for prices and specs that will suit your needs. There's one instance type with 40 CPUs @ 2.4Ghz w/ 160Gb RAM and another with 36 CPUs @ 2.9Ghz w/ 60 Gb for about the same spot price as I mentioned above. There are cheaper instances with 16 and fewer CPUs at about half the spot price, but those might take twice a long to so you won't really be saving any money.</p>\n\n<p>@wacax, what VPS are you using? Do they have options with more CPUs and RAM?</p>",
      "rawMarkdown": "Jared, AWS has like 30 different options so you'd really have to get in there yourself and check for prices and specs that will suit your needs. There's one instance type with 40 CPUs @ 2.4Ghz w/ 160Gb RAM and another with 36 CPUs @ 2.9Ghz w/ 60 Gb for about the same spot price as I mentioned above. There are cheaper instances with 16 and fewer CPUs at about half the spot price, but those might take twice a long to so you won't really be saving any money.\r\n\r\n@wacax, what VPS are you using? Do they have options with more CPUs and RAM?",
      "votes": null
    },
    {
      "id": "120446",
      "postDate": "05/18/2016 10:24:12",
      "content": "<p>@wacax Databricks uses AWS EC2, so why is the pricing any cheaper than using it directly? </p>\n\n<p>Thanks,\nBen</p>",
      "rawMarkdown": "wacax Databricks uses AWS EC2, so why is the pricing any cheaper than using it directly? \r\n\r\nThanks,\r\nBen",
      "votes": null
    },
    {
      "id": "120658",
      "postDate": "05/19/2016 19:14:53",
      "content": "<p>Hi @Jared Turkewitz, I see that you are currently ranked as No.1 on public leaderboard - congratulations! Are you actually using the approach you mentioned above - &quot;chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid&quot;?</p>\n\n<p>Just curious - thanks a lot!</p>",
      "rawMarkdown": "Hi @Jared Turkewitz, I see that you are currently ranked as No.1 on public leaderboard - congratulations! Are you actually using the approach you mentioned above - \"chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid\"?\r\n\r\nJust curious - thanks a lot!",
      "votes": null
    },
    {
      "id": "120659",
      "postDate": "05/19/2016 19:21:39",
      "content": "<p>Yes I am. I deliberately made the method worse so it would run in about 15 hours on my 8 gb macbook pro. To make it optimum it would take over a week to run on my laptop, but the dumbed down version does well enough for now. Unfortunately now that this method is out there I expect people with more computing resources will start to dominate the competition. There is still a lot of room for improvement in this competition.</p>\n\n<p>Divide and conquer!</p>",
      "rawMarkdown": "Yes I am. I deliberately made the method worse so it would run in about 15 hours on my 8 gb macbook pro. To make it optimum it would take over a week to run on my laptop, but the dumbed down version does well enough for now. Unfortunately now that this method is out there I expect people with more computing resources will start to dominate the competition. There is still a lot of room for improvement in this competition.\r\n\r\nDivide and conquer!",
      "votes": null
    },
    {
      "id": "120711",
      "postDate": "05/20/2016 02:26:14",
      "content": "<p>Thank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your &quot;score per computational resources used&quot; is still better than mine.</p>\n\n<p>I used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.</p>\n\n<p>This competition would greatly benefit from &quot;variable eta&quot; whereby the first few round have high eta, then lower eta and lower eta for later rounds.</p>",
      "rawMarkdown": "Thank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your \"score per computational resources used\" is still better than mine.\r\n\r\nI used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.\r\n\r\nThis competition would greatly benefit from \"variable eta\" whereby the first few round have high eta, then lower eta and lower eta for later rounds.",
      "votes": null
    },
    {
      "id": "120872",
      "postDate": "05/21/2016 07:01:31",
      "content": "<p>Hi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. Right?</p>\n\n<p>Cheers</p>\n\n<p>Gerhard</p>",
      "rawMarkdown": "Hi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. Right?\r\n\r\nCheers\r\n\r\nGerhard",
      "votes": null
    },
    {
      "id": "120884",
      "postDate": "05/21/2016 10:58:08",
      "content": "<p>Hm, </p>\n\n<p>I have set up my xgboost now but I am getting warnings, a lot of them in CV.</p>\n\n<pre><code>In foldVector[which(y == dimnames(numInClass)$y[i])] &lt;- sample(seqVector) :\n  number of items to replace is not a multiple of replacement length\n</code></pre>\n\n<p>I already checked the dimensions and lengths of matrices and vectors and the correct number of classes. They seem ok.</p>\n\n<p>What causes this warning? And, can I ignore it? Probably not...</p>\n\n<p>Cheers</p>\n\n<p>Gerhard</p>",
      "rawMarkdown": "Hm, \r\n\r\nI have set up my xgboost now but I am getting warnings, a lot of them in CV.\r\n\r\n    In foldVector[which(y == dimnames(numInClass)$y[i])] <- sample(seqVector) :\r\n      number of items to replace is not a multiple of replacement length\r\n\r\nI already checked the dimensions and lengths of matrices and vectors and the correct number of classes. They seem ok.\r\n\r\nWhat causes this warning? And, can I ignore it? Probably not...\r\n\r\nCheers\r\n\r\nGerhard",
      "votes": null
    },
    {
      "id": "120893",
      "postDate": "05/21/2016 13:35:16",
      "content": "<p>[quote=MightyBird;120872]</p>\n\n<p>Hi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. </p>\n\n<p>[/quote]</p>\n\n<p>For starters, I ran validation on a handful of grid cells to find a &quot;fairly good&quot; set of parameters, and then I used those same parameters across all grid cells. The other extreme would be to optimize parameters for each individual grid cell, but I think that would be very computationally expensive.  Maybe there is some middle ground.</p>",
      "rawMarkdown": "[quote=MightyBird;120872]\r\n\r\nHi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. \r\n\r\n[/quote]\r\n\r\nFor starters, I ran validation on a handful of grid cells to find a \"fairly good\" set of parameters, and then I used those same parameters across all grid cells. The other extreme would be to optimize parameters for each individual grid cell, but I think that would be very computationally expensive.  Maybe there is some middle ground.",
      "votes": null
    },
    {
      "id": "120971",
      "postDate": "05/22/2016 09:40:24",
      "content": "<p>So, I have it all set up and submitted it to &quot;the cloud&quot;. I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. </p>\n\n<p>For those interested: 8 core / 10GB RAM / 50x50 grid / 6 features</p>\n\n<p>Hopefully the result is worth it.</p>\n\n<p>Cheers </p>\n\n<p>Gerhard</p>",
      "rawMarkdown": "So, I have it all set up and submitted it to \"the cloud\". I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. \r\n\r\nFor those interested: 8 core / 10GB RAM / 50x50 grid / 6 features\r\n\r\nHopefully the result is worth it.\r\n\r\nCheers \r\n\r\nGerhard",
      "votes": null
    },
    {
      "id": "120985",
      "postDate": "05/22/2016 12:51:32",
      "content": "<p>[quote=MightyBird;120971]</p>\n\n<p>So, I have it all set up and submitted it to &quot;the cloud&quot;. I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. </p>\n\n<p>For those interested: 8 core / 10GB RAM / 50x50 grid / 6 features</p>\n\n<p>Hopefully the result is worth it.</p>\n\n<p>Cheers </p>\n\n<p>Gerhard</p>\n\n<p>[/quote]</p>\n\n<p>Hello Gerhard, \nwhich cloud provider you are using?</p>",
      "rawMarkdown": "[quote=MightyBird;120971]\r\n\r\nSo, I have it all set up and submitted it to \"the cloud\". I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. \r\n\r\nFor those interested: 8 core / 10GB RAM / 50x50 grid / 6 features\r\n\r\nHopefully the result is worth it.\r\n\r\nCheers \r\n\r\nGerhard\r\n\r\n[/quote]\r\n\r\n\r\nHello Gerhard, \r\nwhich cloud provider you are using?",
      "votes": null
    },
    {
      "id": "120986",
      "postDate": "05/22/2016 12:55:56",
      "content": "<p>I find that when there are only a few features, XGBoost does not fully utilize all the CPUs.  This problem is one of those cases. So to utilize all the CPUs on my machine, I broke to grid cells into three groups and ran three XGBoosts simultaneously. I'm using R, and each instance of R is taking a little over 3 Gig of memory.  With 8 core and 10GB, you could cut your time in half, or maybe even to one third your planned time.</p>",
      "rawMarkdown": "I find that when there are only a few features, XGBoost does not fully utilize all the CPUs.  This problem is one of those cases. So to utilize all the CPUs on my machine, I broke to grid cells into three groups and ran three XGBoosts simultaneously. I'm using R, and each instance of R is taking a little over 3 Gig of memory.  With 8 core and 10GB, you could cut your time in half, or maybe even to one third your planned time.",
      "votes": null
    },
    {
      "id": "120988",
      "postDate": "05/22/2016 13:14:22",
      "content": "<p>Hi,</p>\n\n<p>I am using a German provider called Strato. www.strato.de. And I have a prepaid tariff. Cuts costs half - if I stay below a certain limit.</p>\n\n<p>Unfortunately, they don't provide more that 8 cores per instance - yet.</p>\n\n<p>@BreakfastPirate: My CPU load is way above 700%. So that's ok. </p>\n\n<p>Gerhard</p>",
      "rawMarkdown": "Hi,\r\n\r\nI am using a German provider called Strato. www.strato.de. And I have a prepaid tariff. Cuts costs half - if I stay below a certain limit.\r\n\r\nUnfortunately, they don't provide more that 8 cores per instance - yet.\r\n\r\n@BreakfastPirate: My CPU load is way above 700%. So that's ok. \r\n\r\nGerhard",
      "votes": null
    },
    {
      "id": "121841",
      "postDate": "05/30/2016 09:13:05",
      "content": "<p>So, another milestone. Script (xgb/2500grids) ran 2x as long as expected but resulted in 0.54386. In the end it ran a week. 5 weeks to go :-) Now I added some new features. See you in a week...</p>\n\n<p>Gerhard</p>",
      "rawMarkdown": "So, another milestone. Script (xgb/2500grids) ran 2x as long as expected but resulted in 0.54386. In the end it ran a week. 5 weeks to go :-) Now I added some new features. See you in a week...\r\n\r\nGerhard",
      "votes": null
    },
    {
      "id": "121849",
      "postDate": "05/30/2016 10:14:30",
      "content": "<p>Hi,</p>\n\n<p>This is my first kaggle competition. I have started a bit late but I am trying to catch up. The most I've got with the grid approach is to 0.576. That took about 6 hours in a 16core machine.\nI've been using basic time and space features. From now own I think I can not get better without spending to much computation time. Time to explore new features, new grid configurations an even new algorithms. I am looking forward to seeing how far people can get with this dataset...</p>",
      "rawMarkdown": "Hi,\r\n\r\nThis is my first kaggle competition. I have started a bit late but I am trying to catch up. The most I've got with the grid approach is to 0.576. That took about 6 hours in a 16core machine.\r\nI've been using basic time and space features. From now own I think I can not get better without spending to much computation time. Time to explore new features, new grid configurations an even new algorithms. I am looking forward to seeing how far people can get with this dataset...",
      "votes": null
    },
    {
      "id": "125168",
      "postDate": "06/27/2016 04:35:22",
      "content": "<p>[quote=BreakfastPirate;120711]</p>\n\n<p>Thank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your &quot;score per computational resources used&quot; is still better than mine.</p>\n\n<p>I used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.</p>\n\n<p>This competition would greatly benefit from &quot;variable eta&quot; whereby the first few round have high eta, then lower eta and lower eta for later rounds.</p>\n\n<p>[/quote]\nHow do you implement the &quot;variable eta&quot; with R? It seems that the eta parameter doesn't take a vector or is there a &quot;decay&quot; parameter?</p>",
      "rawMarkdown": "[quote=BreakfastPirate;120711]\r\n\r\nThank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your \"score per computational resources used\" is still better than mine.\r\n\r\nI used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.\r\n\r\nThis competition would greatly benefit from \"variable eta\" whereby the first few round have high eta, then lower eta and lower eta for later rounds.\r\n\r\n[/quote]\r\nHow do you implement the \"variable eta\" with R? It seems that the eta parameter doesn't take a vector or is there a \"decay\" parameter?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 120269,
      "author_name": "brandenkmurray",
      "author_url": "",
      "post_date": "05/17/2016 00:21:22",
      "content": "<p>How expensive it is will depend on what type of instance you select and the location of the servers you select (which they recommend are the ones closest to you). But for instances with the most RAM (~244GB) and CPU(32 cores) in Northern California (where I am) spot pricing about $0.33/hour, but sometimes it spikes up and it may kill your instance if your bid price isn't high enough. So it's at least ~$8/day but could easily go to $30-$40 if the prices go up. </p>\n\n<p>For fun I tried to run an xgboost with 100,000+ classes on my desktop and it tried to allocate a vector that was ~83,000 GB or something ridiculous like that, so I don't think AWS is going to help with that in case that was something you were considering.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120271,
      "author_name": "jturkewitz",
      "author_url": "",
      "post_date": "05/17/2016 01:01:29",
      "content": "<p>Thanks for answering. My plan was to chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid. Local cv indicates this approach would get above 0.55, but on my laptop it would take a long time to run - around a week I think though I'm not sure exactly. Would it be cheaper to do something like this? I wouldn't need a good computer, I'd just need lots of cores.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120282,
      "author_name": "wacaxx",
      "author_url": "",
      "post_date": "05/17/2016 03:03:54",
      "content": "<p>Lately, I've found better and cheaper alternatives than AWS. The VPS I'm using right now costs 25$ per quarter (16 GB memory and 2 cores and uptime of 99.9%).</p>\n\n<p>Another alternative is using Databricks if you know Spark. </p>\n\n<p>I'd recommend that for everything except GPU instances (i.e deep learning). Then it is still cheaper to buy a new GPU since AWS bills (or any other provider) for GPU instances can go up to 150$ easily in a month. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120295,
      "author_name": "brandenkmurray",
      "author_url": "",
      "post_date": "05/17/2016 06:58:55",
      "content": "<p>@Jared, AWS has like 30 different options so you'd really have to get in there yourself and check for prices and specs that will suit your needs. There's one instance type with 40 CPUs @ 2.4Ghz w/ 160Gb RAM and another with 36 CPUs @ 2.9Ghz w/ 60 Gb for about the same spot price as I mentioned above. There are cheaper instances with 16 and fewer CPUs at about half the spot price, but those might take twice a long to so you won't really be saving any money.</p>\n\n<p>@wacax, what VPS are you using? Do they have options with more CPUs and RAM?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120446,
      "author_name": "bentannenbaum",
      "author_url": "",
      "post_date": "05/18/2016 10:24:12",
      "content": "<p>@wacax Databricks uses AWS EC2, so why is the pricing any cheaper than using it directly? </p>\n\n<p>Thanks,\nBen</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120658,
      "author_name": "howardxie",
      "author_url": "",
      "post_date": "05/19/2016 19:14:53",
      "content": "<p>Hi @Jared Turkewitz, I see that you are currently ranked as No.1 on public leaderboard - congratulations! Are you actually using the approach you mentioned above - &quot;chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid&quot;?</p>\n\n<p>Just curious - thanks a lot!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120659,
      "author_name": "jturkewitz",
      "author_url": "",
      "post_date": "05/19/2016 19:21:39",
      "content": "<p>Yes I am. I deliberately made the method worse so it would run in about 15 hours on my 8 gb macbook pro. To make it optimum it would take over a week to run on my laptop, but the dumbed down version does well enough for now. Unfortunately now that this method is out there I expect people with more computing resources will start to dominate the competition. There is still a lot of room for improvement in this competition.</p>\n\n<p>Divide and conquer!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120711,
      "author_name": "breakfastpirate",
      "author_url": "",
      "post_date": "05/20/2016 02:26:14",
      "content": "<p>Thank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your &quot;score per computational resources used&quot; is still better than mine.</p>\n\n<p>I used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.</p>\n\n<p>This competition would greatly benefit from &quot;variable eta&quot; whereby the first few round have high eta, then lower eta and lower eta for later rounds.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120872,
      "author_name": "mightybird",
      "author_url": "",
      "post_date": "05/21/2016 07:01:31",
      "content": "<p>Hi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. Right?</p>\n\n<p>Cheers</p>\n\n<p>Gerhard</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120884,
      "author_name": "mightybird",
      "author_url": "",
      "post_date": "05/21/2016 10:58:08",
      "content": "<p>Hm, </p>\n\n<p>I have set up my xgboost now but I am getting warnings, a lot of them in CV.</p>\n\n<pre><code>In foldVector[which(y == dimnames(numInClass)$y[i])] &lt;- sample(seqVector) :\n  number of items to replace is not a multiple of replacement length\n</code></pre>\n\n<p>I already checked the dimensions and lengths of matrices and vectors and the correct number of classes. They seem ok.</p>\n\n<p>What causes this warning? And, can I ignore it? Probably not...</p>\n\n<p>Cheers</p>\n\n<p>Gerhard</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120893,
      "author_name": "breakfastpirate",
      "author_url": "",
      "post_date": "05/21/2016 13:35:16",
      "content": "<p>[quote=MightyBird;120872]</p>\n\n<p>Hi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. </p>\n\n<p>[/quote]</p>\n\n<p>For starters, I ran validation on a handful of grid cells to find a &quot;fairly good&quot; set of parameters, and then I used those same parameters across all grid cells. The other extreme would be to optimize parameters for each individual grid cell, but I think that would be very computationally expensive.  Maybe there is some middle ground.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120971,
      "author_name": "mightybird",
      "author_url": "",
      "post_date": "05/22/2016 09:40:24",
      "content": "<p>So, I have it all set up and submitted it to &quot;the cloud&quot;. I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. </p>\n\n<p>For those interested: 8 core / 10GB RAM / 50x50 grid / 6 features</p>\n\n<p>Hopefully the result is worth it.</p>\n\n<p>Cheers </p>\n\n<p>Gerhard</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120985,
      "author_name": "ascending",
      "author_url": "",
      "post_date": "05/22/2016 12:51:32",
      "content": "<p>[quote=MightyBird;120971]</p>\n\n<p>So, I have it all set up and submitted it to &quot;the cloud&quot;. I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. </p>\n\n<p>For those interested: 8 core / 10GB RAM / 50x50 grid / 6 features</p>\n\n<p>Hopefully the result is worth it.</p>\n\n<p>Cheers </p>\n\n<p>Gerhard</p>\n\n<p>[/quote]</p>\n\n<p>Hello Gerhard, \nwhich cloud provider you are using?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120986,
      "author_name": "breakfastpirate",
      "author_url": "",
      "post_date": "05/22/2016 12:55:56",
      "content": "<p>I find that when there are only a few features, XGBoost does not fully utilize all the CPUs.  This problem is one of those cases. So to utilize all the CPUs on my machine, I broke to grid cells into three groups and ran three XGBoosts simultaneously. I'm using R, and each instance of R is taking a little over 3 Gig of memory.  With 8 core and 10GB, you could cut your time in half, or maybe even to one third your planned time.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120988,
      "author_name": "mightybird",
      "author_url": "",
      "post_date": "05/22/2016 13:14:22",
      "content": "<p>Hi,</p>\n\n<p>I am using a German provider called Strato. www.strato.de. And I have a prepaid tariff. Cuts costs half - if I stay below a certain limit.</p>\n\n<p>Unfortunately, they don't provide more that 8 cores per instance - yet.</p>\n\n<p>@BreakfastPirate: My CPU load is way above 700%. So that's ok. </p>\n\n<p>Gerhard</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121841,
      "author_name": "mightybird",
      "author_url": "",
      "post_date": "05/30/2016 09:13:05",
      "content": "<p>So, another milestone. Script (xgb/2500grids) ran 2x as long as expected but resulted in 0.54386. In the end it ran a week. 5 weeks to go :-) Now I added some new features. See you in a week...</p>\n\n<p>Gerhard</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121849,
      "author_name": "shdgrao",
      "author_url": "",
      "post_date": "05/30/2016 10:14:30",
      "content": "<p>Hi,</p>\n\n<p>This is my first kaggle competition. I have started a bit late but I am trying to catch up. The most I've got with the grid approach is to 0.576. That took about 6 hours in a 16core machine.\nI've been using basic time and space features. From now own I think I can not get better without spending to much computation time. Time to explore new features, new grid configurations an even new algorithms. I am looking forward to seeing how far people can get with this dataset...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125168,
      "author_name": "gospursgo",
      "author_url": "",
      "post_date": "06/27/2016 04:35:22",
      "content": "<p>[quote=BreakfastPirate;120711]</p>\n\n<p>Thank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your &quot;score per computational resources used&quot; is still better than mine.</p>\n\n<p>I used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.</p>\n\n<p>This competition would greatly benefit from &quot;variable eta&quot; whereby the first few round have high eta, then lower eta and lower eta for later rounds.</p>\n\n<p>[/quote]\nHow do you implement the &quot;variable eta&quot; with R? It seems that the eta parameter doesn't take a vector or is there a &quot;decay&quot; parameter?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "120268": "Does anyone have any experience using AWS for this type of problem? How expensive would it be in practice?",
    "120269": "How expensive it is will depend on what type of instance you select and the location of the servers you select (which they recommend are the ones closest to you). But for instances with the most RAM (~244GB) and CPU(32 cores) in Northern California (where I am) spot pricing about $0.33/hour, but sometimes it spikes up and it may kill your instance if your bid price isn't high enough. So it's at least ~$8/day but could easily go to $30-$40 if the prices go up. \r\n\r\nFor fun I tried to run an xgboost with 100,000+ classes on my desktop and it tried to allocate a vector that was ~83,000 GB or something ridiculous like that, so I don't think AWS is going to help with that in case that was something you were considering.",
    "120271": "Thanks for answering. My plan was to chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid. Local cv indicates this approach would get above 0.55, but on my laptop it would take a long time to run - around a week I think though I'm not sure exactly. Would it be cheaper to do something like this? I wouldn't need a good computer, I'd just need lots of cores.",
    "120282": "Lately, I've found better and cheaper alternatives than AWS. The VPS I'm using right now costs 25$ per quarter (16 GB memory and 2 cores and uptime of 99.9%).\r\n\r\nAnother alternative is using Databricks if you know Spark. \r\n\r\nI'd recommend that for everything except GPU instances (i.e deep learning). Then it is still cheaper to buy a new GPU since AWS bills (or any other provider) for GPU instances can go up to 150$ easily in a month.",
    "120295": "Jared, AWS has like 30 different options so you'd really have to get in there yourself and check for prices and specs that will suit your needs. There's one instance type with 40 CPUs @ 2.4Ghz w/ 160Gb RAM and another with 36 CPUs @ 2.9Ghz w/ 60 Gb for about the same spot price as I mentioned above. There are cheaper instances with 16 and fewer CPUs at about half the spot price, but those might take twice a long to so you won't really be saving any money.\r\n\r\n@wacax, what VPS are you using? Do they have options with more CPUs and RAM?",
    "120446": "wacax Databricks uses AWS EC2, so why is the pricing any cheaper than using it directly? \r\n\r\nThanks,\r\nBen",
    "120658": "Hi @Jared Turkewitz, I see that you are currently ranked as No.1 on public leaderboard - congratulations! Are you actually using the approach you mentioned above - \"chunk the dataset into a 100 x 100 grid (or finer maybe) and run an independent xgb model on each grid\"?\r\n\r\nJust curious - thanks a lot!",
    "120659": "Yes I am. I deliberately made the method worse so it would run in about 15 hours on my 8 gb macbook pro. To make it optimum it would take over a week to run on my laptop, but the dumbed down version does well enough for now. Unfortunately now that this method is out there I expect people with more computing resources will start to dominate the competition. There is still a lot of room for improvement in this competition.\r\n\r\nDivide and conquer!",
    "120711": "Thank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your \"score per computational resources used\" is still better than mine.\r\n\r\nI used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.\r\n\r\nThis competition would greatly benefit from \"variable eta\" whereby the first few round have high eta, then lower eta and lower eta for later rounds.",
    "120872": "Hi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. Right?\r\n\r\nCheers\r\n\r\nGerhard",
    "120884": "Hm, \r\n\r\nI have set up my xgboost now but I am getting warnings, a lot of them in CV.\r\n\r\n    In foldVector[which(y == dimnames(numInClass)$y[i])] <- sample(seqVector) :\r\n      number of items to replace is not a multiple of replacement length\r\n\r\nI already checked the dimensions and lengths of matrices and vectors and the correct number of classes. They seem ok.\r\n\r\nWhat causes this warning? And, can I ignore it? Probably not...\r\n\r\nCheers\r\n\r\nGerhard",
    "120893": "[quote=MightyBird;120872]\r\n\r\nHi there XGBOOST / grid users. I was wondering if you are doing a cross validation per grid before building the model or will it be fine to do it only once? I suspect we have to do it each time because number of instances and probably more important the number of classes do vary. \r\n\r\n[/quote]\r\n\r\nFor starters, I ran validation on a handful of grid cells to find a \"fairly good\" set of parameters, and then I used those same parameters across all grid cells. The other extreme would be to optimize parameters for each individual grid cell, but I think that would be very computationally expensive.  Maybe there is some middle ground.",
    "120971": "So, I have it all set up and submitted it to \"the cloud\". I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. \r\n\r\nFor those interested: 8 core / 10GB RAM / 50x50 grid / 6 features\r\n\r\nHopefully the result is worth it.\r\n\r\nCheers \r\n\r\nGerhard",
    "120985": "[quote=MightyBird;120971]\r\n\r\nSo, I have it all set up and submitted it to \"the cloud\". I am supposed to receive my results in 3.61 days! Hooray! And it will cost me around $7.50. \r\n\r\nFor those interested: 8 core / 10GB RAM / 50x50 grid / 6 features\r\n\r\nHopefully the result is worth it.\r\n\r\nCheers \r\n\r\nGerhard\r\n\r\n[/quote]\r\n\r\n\r\nHello Gerhard, \r\nwhich cloud provider you are using?",
    "120986": "I find that when there are only a few features, XGBoost does not fully utilize all the CPUs.  This problem is one of those cases. So to utilize all the CPUs on my machine, I broke to grid cells into three groups and ran three XGBoosts simultaneously. I'm using R, and each instance of R is taking a little over 3 Gig of memory.  With 8 core and 10GB, you could cut your time in half, or maybe even to one third your planned time.",
    "120988": "Hi,\r\n\r\nI am using a German provider called Strato. www.strato.de. And I have a prepaid tariff. Cuts costs half - if I stay below a certain limit.\r\n\r\nUnfortunately, they don't provide more that 8 cores per instance - yet.\r\n\r\n@BreakfastPirate: My CPU load is way above 700%. So that's ok. \r\n\r\nGerhard",
    "121841": "So, another milestone. Script (xgb/2500grids) ran 2x as long as expected but resulted in 0.54386. In the end it ran a week. 5 weeks to go :-) Now I added some new features. See you in a week...\r\n\r\nGerhard",
    "121849": "Hi,\r\n\r\nThis is my first kaggle competition. I have started a bit late but I am trying to catch up. The most I've got with the grid approach is to 0.576. That took about 6 hours in a 16core machine.\r\nI've been using basic time and space features. From now own I think I can not get better without spending to much computation time. Time to explore new features, new grid configurations an even new algorithms. I am looking forward to seeing how far people can get with this dataset...",
    "125168": "[quote=BreakfastPirate;120711]\r\n\r\nThank you Jared. I set mine up such that it would run approximately 24 hours on a 6-core machine. So I think your \"score per computational resources used\" is still better than mine.\r\n\r\nI used a 50x50 grid so 2500 runs of XGBoost. I agree there is a lot of room for improvement. Yes, brute force with a lot of computing resources will be an advantage. But I think there are also some tricks for reducing compute time without much loss of accuracy, and then using the saved time to squeeze out better performance. The nice thing about this approach is that you can optimize on about 10% of the grid and get a pretty good idea of what your performance improvement will be on the other 90%.\r\n\r\nThis competition would greatly benefit from \"variable eta\" whereby the first few round have high eta, then lower eta and lower eta for later rounds.\r\n\r\n[/quote]\r\nHow do you implement the \"variable eta\" with R? It seems that the eta parameter doesn't take a vector or is there a \"decay\" parameter?"
  },
  "source": "meta"
}