{
  "id": 14394,
  "title": "A basic look at LB scores",
  "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14394",
  "author_name": "",
  "post_date": "2015-05-27T03:35:06.223Z",
  "votes": 48,
  "comment_count": 5,
  "views": 2596,
  "content": "<p><em>admins please remove if this violates anything..I know the rules are a bit stricter on these comps.</em></p>\n<p>TL;DR see the plot</p>\n\n<p>For those of you out there struggling a bit with wanting to be atop the public leader-board, and getting frustrated with your awesome models that seemingly don't score well, or perplexed by the lack of continuity and lack of positive feedback from the public leader-board, I thought I would post this to help you along a bit.</p>\n<p>Lets say I have a model that I have constructed carefully, and being the solid modeler that I am, I have in my possession, the out-of-fold predictions for this model on the all the<strong> training data</strong>. &nbsp; (This is our best guess at how the test data would score if I predicted it with my model. Some people call it local validation. Its not the 'Bible', but its often the best we can do. &nbsp;For those of you just getting into this, K-fold cross validation is a necessity when the competition has smallish or unstable data. We have to estimate what the heck we are doing,else we are flying blind. Further, we have to always ensure that our out-of-fold predictions are <em><strong>in fact out-of-fold</strong></em>. &nbsp;This involves building K different models with different subsets of the data. &nbsp;Sometimes this is tricky to get unbiased local validation)</p>\n<p>I would like to experiment and see visually and empirically how much my public AUC could deviate from my local AUC. &nbsp;This assumes my local AUC is reliable which may or may not be true.. but we proceed undaunted.</p>\n<p>I am going to create 2 vectors from my ~2000 out-of-fold training predictions. 4700 randomly selected (with replacement) pairs of <strong>predictions</strong> and<strong> y</strong> (ground truth). &nbsp;This expands my data to the size of the test data. This won't change much statistically, given these sizes, and can be verified easily. But we will build a 4700 length pseudo-test set to try this out and keep the sizing proper. For control, I score my full test vector (all 4700), and receive a <strong>.918 AUC</strong>. &nbsp;<strong>This is the Bible</strong> as it relates to our experiment. &nbsp;I have synthetically made a set that I can be certain is a .918 AUC that should be pretty structurally similar to what we might expect the test predictions to look like. I know this because I have created the predictions and the truth in my hand to run this little test.</p>\n<ul>\n<li><strong><span style=\"line-height: 1.4\">The experiment.</span></strong></li>\n</ul>\n<p style=\"padding-left: 30px\">I am going to randomly select 30% of my 4700 pseudo-test predictions (this is what the leader-board does) and score the AUC.&nbsp;</p>\n<p style=\"padding-left: 30px\">I am going to save my score and do this another 100,000 times for good measure.</p>\n<p style=\"padding-left: 30px\">This gives me an idea of how volatile the LB is with respect to my psuedo-test set. Remember, I know what the truth is for this experiment, I am just curious how much my estimate of this could swing if I only score a randomly selected 30%..</p>\n<p style=\"padding-left: 30px\"><em>I tried to simulate this with as close to possible real world conditions for this competition. There probably exists some closed-form estimate for this but that math is above my pay grade. So we are gonna do it empirically, and see it with our own eyes.</em></p>\n<p>The results:</p>\n<p>the &nbsp;plot kind of tells the tale.</p>\n<p>On average the LB will score right on the money. But with a SD of .0204, we really cannot rely on the public leader-board score.</p>\n<p>Q: Did your model get a favorable or unfavorable split?</p>\n<p>A:We have no way of knowing.</p>\n<p>Q: What about if I run two different models and score them both? Is one better because it scored .01 higher?<strong>&nbsp;That would jump me 100 places up the leaderboard!!!</strong></p>\n<p>A:We have no way of knowing.&nbsp;</p>\n<p>All we can do is create as reliable a local process as possible. It is very tempting to get falsely encouraged or discouraged by the leader-board. &nbsp;But the leader-board is like my ex-gf from college, cannot really be trusted all the time..</p>\n<p>Hope this helps anyone just getting going..</p>",
  "messages": [
    {
      "id": "80090",
      "postDate": "05/27/2015 03:35:06",
      "content": "<p><em>admins please remove if this violates anything..I know the rules are a bit stricter on these comps.</em></p>\n<p>TL;DR see the plot</p>\n\n<p>For those of you out there struggling a bit with wanting to be atop the public leader-board, and getting frustrated with your awesome models that seemingly don't score well, or perplexed by the lack of continuity and lack of positive feedback from the public leader-board, I thought I would post this to help you along a bit.</p>\n<p>Lets say I have a model that I have constructed carefully, and being the solid modeler that I am, I have in my possession, the out-of-fold predictions for this model on the all the<strong> training data</strong>. &nbsp; (This is our best guess at how the test data would score if I predicted it with my model. Some people call it local validation. Its not the 'Bible', but its often the best we can do. &nbsp;For those of you just getting into this, K-fold cross validation is a necessity when the competition has smallish or unstable data. We have to estimate what the heck we are doing,else we are flying blind. Further, we have to always ensure that our out-of-fold predictions are <em><strong>in fact out-of-fold</strong></em>. &nbsp;This involves building K different models with different subsets of the data. &nbsp;Sometimes this is tricky to get unbiased local validation)</p>\n<p>I would like to experiment and see visually and empirically how much my public AUC could deviate from my local AUC. &nbsp;This assumes my local AUC is reliable which may or may not be true.. but we proceed undaunted.</p>\n<p>I am going to create 2 vectors from my ~2000 out-of-fold training predictions. 4700 randomly selected (with replacement) pairs of <strong>predictions</strong> and<strong> y</strong> (ground truth). &nbsp;This expands my data to the size of the test data. This won't change much statistically, given these sizes, and can be verified easily. But we will build a 4700 length pseudo-test set to try this out and keep the sizing proper. For control, I score my full test vector (all 4700), and receive a <strong>.918 AUC</strong>. &nbsp;<strong>This is the Bible</strong> as it relates to our experiment. &nbsp;I have synthetically made a set that I can be certain is a .918 AUC that should be pretty structurally similar to what we might expect the test predictions to look like. I know this because I have created the predictions and the truth in my hand to run this little test.</p>\n<ul>\n<li><strong><span style=\"line-height: 1.4\">The experiment.</span></strong></li>\n</ul>\n<p style=\"padding-left: 30px\">I am going to randomly select 30% of my 4700 pseudo-test predictions (this is what the leader-board does) and score the AUC.&nbsp;</p>\n<p style=\"padding-left: 30px\">I am going to save my score and do this another 100,000 times for good measure.</p>\n<p style=\"padding-left: 30px\">This gives me an idea of how volatile the LB is with respect to my psuedo-test set. Remember, I know what the truth is for this experiment, I am just curious how much my estimate of this could swing if I only score a randomly selected 30%..</p>\n<p style=\"padding-left: 30px\"><em>I tried to simulate this with as close to possible real world conditions for this competition. There probably exists some closed-form estimate for this but that math is above my pay grade. So we are gonna do it empirically, and see it with our own eyes.</em></p>\n<p>The results:</p>\n<p>the &nbsp;plot kind of tells the tale.</p>\n<p>On average the LB will score right on the money. But with a SD of .0204, we really cannot rely on the public leader-board score.</p>\n<p>Q: Did your model get a favorable or unfavorable split?</p>\n<p>A:We have no way of knowing.</p>\n<p>Q: What about if I run two different models and score them both? Is one better because it scored .01 higher?<strong>&nbsp;That would jump me 100 places up the leaderboard!!!</strong></p>\n<p>A:We have no way of knowing.&nbsp;</p>\n<p>All we can do is create as reliable a local process as possible. It is very tempting to get falsely encouraged or discouraged by the leader-board. &nbsp;But the leader-board is like my ex-gf from college, cannot really be trusted all the time..</p>\n<p>Hope this helps anyone just getting going..</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80123",
      "postDate": "05/27/2015 11:13:22",
      "content": "<p>That's an interesting experiment, thanks for sharing. I agree that it seems that one should be able to compute this distribution in closed form (though the use of AUC probably makes it somewhat more complicated than if we'd care only about accuracy).</p>\n<p>I have a question about your terminology, though. I know what k-fold cross validation is, but what are &quot;out-of-fold predictions&quot;? Google doesn't seem to know much about that.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80132",
      "postDate": "05/27/2015 13:34:22",
      "content": "<p>As it turns out, there is a paper which deals with the mean and variance of the ROC-AUC metric, as well as confidence intervals for it:</p>\n\n<p style=\"padding-left: 30px\">Cortes, Corinna, and Mehryar Mohri. &quot;Confidence Intervals for the Area Under the ROC Curve.&quot; Advances in Neural Information Processing Systems. 2005. <a href=\"http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2005_197.pdf\">[pdf]</a></p>\n<p style=\"padding-left: 30px\">&nbsp;</p>\n<p>It would be interesting to compare the formulas obtained in this paper to your experimental results.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80158",
      "postDate": "05/27/2015 16:29:08",
      "content": "<p>[quote=cfh;80123]</p>\n<p>I have a question about your terminology, though. I know what k-fold cross validation is, but what are &quot;out-of-fold predictions&quot;? Google doesn't seem to know much about that.</p>\n<p>[/quote]</p>\n<p>If I split my training data into 4 - folds &nbsp;called <strong>A,B,C,D</strong></p>\n<p>I then train one model on folds &nbsp;[&nbsp;<strong>A,B and C] </strong><em>(all together one training set)</em>&nbsp;<strong>. Model is called M1</strong></p>\n<p>I then predict with<strong> M1</strong> on fold <strong>D</strong>.&nbsp;</p>\n<p>These predictions are considered out-of-fold predictions. &nbsp;(this seems to be reasonably common terminology, just a variation on k-fold )</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81291",
      "postDate": "06/09/2015 00:36:51",
      "content": "<p>&gt; shakeup('https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.06958167</p>\n<p>$shakeup.all<br>[1] 0.0479417</p>\n<p>You can compare this to:</p>\n<p>&gt; shakeup('https://www.kaggle.com/c/otto-group-product-classification-challenge/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.002651178</p>\n<p>$shakeup.all<br>[1] 0.00391377</p>\n<p>The script is from:<br>https://www.kaggle.com/c/liberty-mutual-fire-peril/forums/t/10187/quantifying-leaderboard-shake-up/52910</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81293",
      "postDate": "06/09/2015 00:47:25",
      "content": "<p>I think the shakeup is mainly because of the size of dataset.</p>\n<p>But I am wondering if the correlation between ROC on 30% &nbsp;and ROC on 100%&nbsp;</p>\n<p>is as strong as logloss on 30% is correlated to logloss on 100%</p>\n<p>[quote=Mike Kim;81291]</p>\n<p>&gt; shakeup('https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.06958167</p>\n<p>$shakeup.all<br>[1] 0.0479417</p>\n<p>You can compare this to:</p>\n<p>&gt; shakeup('https://www.kaggle.com/c/otto-group-product-classification-challenge/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.002651178</p>\n<p>$shakeup.all<br>[1] 0.00391377</p>\n<p>The script is from:<br>https://www.kaggle.com/c/liberty-mutual-fire-peril/forums/t/10187/quantifying-leaderboard-shake-up/52910</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 80123,
      "author_name": "cfhlearns",
      "author_url": "",
      "post_date": "05/27/2015 11:13:22",
      "content": "<p>That's an interesting experiment, thanks for sharing. I agree that it seems that one should be able to compute this distribution in closed form (though the use of AUC probably makes it somewhat more complicated than if we'd care only about accuracy).</p>\n<p>I have a question about your terminology, though. I know what k-fold cross validation is, but what are &quot;out-of-fold predictions&quot;? Google doesn't seem to know much about that.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80132,
      "author_name": "cfhlearns",
      "author_url": "",
      "post_date": "05/27/2015 13:34:22",
      "content": "<p>As it turns out, there is a paper which deals with the mean and variance of the ROC-AUC metric, as well as confidence intervals for it:</p>\n\n<p style=\"padding-left: 30px\">Cortes, Corinna, and Mehryar Mohri. &quot;Confidence Intervals for the Area Under the ROC Curve.&quot; Advances in Neural Information Processing Systems. 2005. <a href=\"http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2005_197.pdf\">[pdf]</a></p>\n<p style=\"padding-left: 30px\">&nbsp;</p>\n<p>It would be interesting to compare the formulas obtained in this paper to your experimental results.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80158,
      "author_name": "scharf",
      "author_url": "",
      "post_date": "05/27/2015 16:29:08",
      "content": "<p>[quote=cfh;80123]</p>\n<p>I have a question about your terminology, though. I know what k-fold cross validation is, but what are &quot;out-of-fold predictions&quot;? Google doesn't seem to know much about that.</p>\n<p>[/quote]</p>\n<p>If I split my training data into 4 - folds &nbsp;called <strong>A,B,C,D</strong></p>\n<p>I then train one model on folds &nbsp;[&nbsp;<strong>A,B and C] </strong><em>(all together one training set)</em>&nbsp;<strong>. Model is called M1</strong></p>\n<p>I then predict with<strong> M1</strong> on fold <strong>D</strong>.&nbsp;</p>\n<p>These predictions are considered out-of-fold predictions. &nbsp;(this seems to be reasonably common terminology, just a variation on k-fold )</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81291,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "06/09/2015 00:36:51",
      "content": "<p>&gt; shakeup('https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.06958167</p>\n<p>$shakeup.all<br>[1] 0.0479417</p>\n<p>You can compare this to:</p>\n<p>&gt; shakeup('https://www.kaggle.com/c/otto-group-product-classification-challenge/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.002651178</p>\n<p>$shakeup.all<br>[1] 0.00391377</p>\n<p>The script is from:<br>https://www.kaggle.com/c/liberty-mutual-fire-peril/forums/t/10187/quantifying-leaderboard-shake-up/52910</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81293,
      "author_name": "skylibrary",
      "author_url": "",
      "post_date": "06/09/2015 00:47:25",
      "content": "<p>I think the shakeup is mainly because of the size of dataset.</p>\n<p>But I am wondering if the correlation between ROC on 30% &nbsp;and ROC on 100%&nbsp;</p>\n<p>is as strong as logloss on 30% is correlated to logloss on 100%</p>\n<p>[quote=Mike Kim;81291]</p>\n<p>&gt; shakeup('https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.06958167</p>\n<p>$shakeup.all<br>[1] 0.0479417</p>\n<p>You can compare this to:</p>\n<p>&gt; shakeup('https://www.kaggle.com/c/otto-group-product-classification-challenge/leaderboard')<br>Joining by: id<br>$shakeup.top<br>[1] 0.002651178</p>\n<p>$shakeup.all<br>[1] 0.00391377</p>\n<p>The script is from:<br>https://www.kaggle.com/c/liberty-mutual-fire-peril/forums/t/10187/quantifying-leaderboard-shake-up/52910</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "80090": "",
    "80123": "",
    "80132": "",
    "80158": "",
    "81291": "",
    "81293": ""
  },
  "source": "meta"
}