{
  "id": 2905,
  "title": "Got uncorrelated models?",
  "url": "/competitions/MerckActivity/discussion/2905",
  "author_name": "",
  "post_date": "2012-10-17T01:59:42.437Z",
  "votes": null,
  "comment_count": 7,
  "views": 4204,
  "content": "<p>Now that we're wrapped up and waiting patiently for the final scoring, I decided to post the results of a model. &nbsp;It scored .467 public/.460 private and formed half of a final ensemble.&nbsp;If it is relatively uncorrelated to your models, we should talk about\r\n teaming up on a future competion to mutually benefit from diversity in our models.</p>",
  "messages": [
    {
      "id": "15762",
      "postDate": "10/17/2012 01:59:42",
      "content": "<p>Now that we're wrapped up and waiting patiently for the final scoring, I decided to post the results of a model. &nbsp;It scored .467 public/.460 private and formed half of a final ensemble.&nbsp;If it is relatively uncorrelated to your models, we should talk about\r\n teaming up on a future competion to mutually benefit from diversity in our models.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15774",
      "postDate": "10/17/2012 04:01:05",
      "content": "<p>What language do you use? <br>\r\nI use R predominantly and am looking for someone who uses a language other than R for future competitions like python or .net or java</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15785",
      "postDate": "10/17/2012 05:36:06",
      "content": "<p>I'm not interested in teaming up, though I am curious to what extent correlation enables you to do some forensics on the prediction. So I'll hazard a guess that your ensemble is a mixture of GBM and K Quantile Regression, or at least that's what gives the\r\n highest correlation score from the elements in my ensemble.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15789",
      "postDate": "10/17/2012 05:54:40",
      "content": "<p>My single gbm model with pub/prv scores 0.46379/0.46225:</p>\r\n<blockquote>\r\n<p>cor(mine$Activity, edge2$Activity)<br>\r\n[1] 0.9813869</p>\r\n</blockquote>\r\n<p>My final ensemble:</p>\r\n<blockquote>\r\n<p>cor(mybest$Activity, edge2$Activity)<br>\r\n[1] 0.983771</p>\r\n</blockquote>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15836",
      "postDate": "10/17/2012 17:01:22",
      "content": "<p>For comparison, this model has an overall correlation of 0.980 with the other half of the ensemble, but it has an average correlation of only 0.891 across activities (min: 0.818 activity 13, max 0.941: activity 5). &nbsp;The other half only achieved 0.429 public,\r\n but it still added &gt; 1% to the overall score. dmitrim, if you score an ensemble of our models, I'd love to hear how they do.</p>\r\n<p>Black Magic: All the work for this competition was done in R, but in real life I'm a c/c&#43;&#43;/c# developer.</p>\r\n<p>This model wasn't gbm or quantile regression. &nbsp;I'm curious too how much correlation comes from the type of model as opposed to feature selection/dim reduction, etc.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15842",
      "postDate": "10/17/2012 17:27:55",
      "content": "<p>Edge2, oh I didn't know you could submit after the deadline, thanks! <br>\r\nHere's my submissions 50/50 averaged with yours:<br>\r\nSingle model &#43; Edge2 model -- 0.48325 / 0.47953<br>\r\nMy final ensemble &#43; Edge2 model -- 0.48373 / 0.48105</p>\r\n<p>That's pretty cool and would place around 5th!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15846",
      "postDate": "10/17/2012 18:17:25",
      "content": "<p>Yeah, that's awesome. That's the power of diversity in an ensemble.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15912",
      "postDate": "10/18/2012 20:55:25",
      "content": "<p>Pearson Correlation across all activities:</p>\r\n<p>cor(preds.all.df$pred,preds.all.edge$Activity)<br>\r\n[1] 0.9802859</p>\r\n<p>50/50 Blend with one of our best ensembles:<br>\r\n(Public/Private)<br>\r\n(0.48476,0.48435)</p>\r\n<p>Activity specific Pearson correlations:<br>\r\nby(preds.all.cbind,act.flags,function(x) cor(x$pred,x$pred.edge))<br>\r\nact.flags: 1</p>\r\n<h2>[1] 0.9309281</h2>\r\n<p>act.flags: 2</p>\r\n<h2>[1] 0.884141</h2>\r\n<p>act.flags: 3</p>\r\n<h2>[1] 0.8437928</h2>\r\n<p>act.flags: 4</p>\r\n<h2>[1] 0.8817598</h2>\r\n<p>act.flags: 5</p>\r\n<h2>[1] 0.9653364</h2>\r\n<p>act.flags: 6</p>\r\n<h2>[1] 0.9482503</h2>\r\n<p>act.flags: 7</p>\r\n<h2>[1] 0.9312552</h2>\r\n<p>act.flags: 8</p>\r\n<h2>[1] 0.9073388</h2>\r\n<p>act.flags: 9</p>\r\n<h2>[1] 0.9309852</h2>\r\n<p>act.flags: 10</p>\r\n<h2>[1] 0.9327654</h2>\r\n<p>act.flags: 11</p>\r\n<h2>[1] 0.9201682</h2>\r\n<p>act.flags: 12</p>\r\n<h2>[1] 0.877405</h2>\r\n<p>act.flags: 13</p>\r\n<h2>[1] 0.8417436</h2>\r\n<p>act.flags: 14</p>\r\n<h2>[1] 0.9098139</h2>\r\n<p>act.flags: 15<br>\r\n[1] 0.9026113</p>\r\n<p>(Neil reviewed none of this work. I hope it's right.)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 15774,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/17/2012 04:01:05",
      "content": "<p>What language do you use? <br>\r\nI use R predominantly and am looking for someone who uses a language other than R for future competitions like python or .net or java</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15785,
      "author_name": "s445203",
      "author_url": "",
      "post_date": "10/17/2012 05:36:06",
      "content": "<p>I'm not interested in teaming up, though I am curious to what extent correlation enables you to do some forensics on the prediction. So I'll hazard a guess that your ensemble is a mixture of GBM and K Quantile Regression, or at least that's what gives the\r\n highest correlation score from the elements in my ensemble.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15789,
      "author_name": "dmitrim",
      "author_url": "",
      "post_date": "10/17/2012 05:54:40",
      "content": "<p>My single gbm model with pub/prv scores 0.46379/0.46225:</p>\r\n<blockquote>\r\n<p>cor(mine$Activity, edge2$Activity)<br>\r\n[1] 0.9813869</p>\r\n</blockquote>\r\n<p>My final ensemble:</p>\r\n<blockquote>\r\n<p>cor(mybest$Activity, edge2$Activity)<br>\r\n[1] 0.983771</p>\r\n</blockquote>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15836,
      "author_name": "edge219878",
      "author_url": "",
      "post_date": "10/17/2012 17:01:22",
      "content": "<p>For comparison, this model has an overall correlation of 0.980 with the other half of the ensemble, but it has an average correlation of only 0.891 across activities (min: 0.818 activity 13, max 0.941: activity 5). &nbsp;The other half only achieved 0.429 public,\r\n but it still added &gt; 1% to the overall score. dmitrim, if you score an ensemble of our models, I'd love to hear how they do.</p>\r\n<p>Black Magic: All the work for this competition was done in R, but in real life I'm a c/c&#43;&#43;/c# developer.</p>\r\n<p>This model wasn't gbm or quantile regression. &nbsp;I'm curious too how much correlation comes from the type of model as opposed to feature selection/dim reduction, etc.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15842,
      "author_name": "dmitrim",
      "author_url": "",
      "post_date": "10/17/2012 17:27:55",
      "content": "<p>Edge2, oh I didn't know you could submit after the deadline, thanks! <br>\r\nHere's my submissions 50/50 averaged with yours:<br>\r\nSingle model &#43; Edge2 model -- 0.48325 / 0.47953<br>\r\nMy final ensemble &#43; Edge2 model -- 0.48373 / 0.48105</p>\r\n<p>That's pretty cool and would place around 5th!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15846,
      "author_name": "edge219878",
      "author_url": "",
      "post_date": "10/17/2012 18:17:25",
      "content": "<p>Yeah, that's awesome. That's the power of diversity in an ensemble.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15912,
      "author_name": "biasvariance",
      "author_url": "",
      "post_date": "10/18/2012 20:55:25",
      "content": "<p>Pearson Correlation across all activities:</p>\r\n<p>cor(preds.all.df$pred,preds.all.edge$Activity)<br>\r\n[1] 0.9802859</p>\r\n<p>50/50 Blend with one of our best ensembles:<br>\r\n(Public/Private)<br>\r\n(0.48476,0.48435)</p>\r\n<p>Activity specific Pearson correlations:<br>\r\nby(preds.all.cbind,act.flags,function(x) cor(x$pred,x$pred.edge))<br>\r\nact.flags: 1</p>\r\n<h2>[1] 0.9309281</h2>\r\n<p>act.flags: 2</p>\r\n<h2>[1] 0.884141</h2>\r\n<p>act.flags: 3</p>\r\n<h2>[1] 0.8437928</h2>\r\n<p>act.flags: 4</p>\r\n<h2>[1] 0.8817598</h2>\r\n<p>act.flags: 5</p>\r\n<h2>[1] 0.9653364</h2>\r\n<p>act.flags: 6</p>\r\n<h2>[1] 0.9482503</h2>\r\n<p>act.flags: 7</p>\r\n<h2>[1] 0.9312552</h2>\r\n<p>act.flags: 8</p>\r\n<h2>[1] 0.9073388</h2>\r\n<p>act.flags: 9</p>\r\n<h2>[1] 0.9309852</h2>\r\n<p>act.flags: 10</p>\r\n<h2>[1] 0.9327654</h2>\r\n<p>act.flags: 11</p>\r\n<h2>[1] 0.9201682</h2>\r\n<p>act.flags: 12</p>\r\n<h2>[1] 0.877405</h2>\r\n<p>act.flags: 13</p>\r\n<h2>[1] 0.8417436</h2>\r\n<p>act.flags: 14</p>\r\n<h2>[1] 0.9098139</h2>\r\n<p>act.flags: 15<br>\r\n[1] 0.9026113</p>\r\n<p>(Neil reviewed none of this work. I hope it's right.)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "15762": "",
    "15774": "",
    "15785": "",
    "15789": "",
    "15836": "",
    "15842": "",
    "15846": "",
    "15912": ""
  },
  "source": "meta"
}