{
  "id": 14133,
  "title": "Validation methodology",
  "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14133",
  "author_name": "",
  "post_date": "2015-05-10T04:45:16.427Z",
  "votes": null,
  "comment_count": 6,
  "views": 1443,
  "content": "<p>Hello Everyone,</p>\n\n<p>I am having difficulties establishing a local validation methodology matching with my submissions.</p>\n<p>I am using scikit learn's roc_auc_score , however, it over estimates the submission's score greatly:</p>\n<p>Local score&nbsp;estimated by&nbsp;roc_auc_score: 0.88&nbsp;</p>\n<p>Submission score : 0.51</p>\n<p>Anyone having a similar problem? Or seeing an obvious mistake in using the&nbsp;roc_auc_score?</p>",
  "messages": [
    {
      "id": "77739",
      "postDate": "05/10/2015 04:45:16",
      "content": "<p>Hello Everyone,</p>\n\n<p>I am having difficulties establishing a local validation methodology matching with my submissions.</p>\n<p>I am using scikit learn's roc_auc_score , however, it over estimates the submission's score greatly:</p>\n<p>Local score&nbsp;estimated by&nbsp;roc_auc_score: 0.88&nbsp;</p>\n<p>Submission score : 0.51</p>\n<p>Anyone having a similar problem? Or seeing an obvious mistake in using the&nbsp;roc_auc_score?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "77740",
      "postDate": "05/10/2015 05:07:00",
      "content": "<p>My local CV and leaderboard scores are pretty close. You must be doing something wrong.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "77741",
      "postDate": "05/10/2015 06:02:57",
      "content": "<p>I am not sure if you are already doing it yet,</p>\n<p>But you need to use Stratified KFold from scikit learn : The mean scores across multiple folds using this method will provide a very close estimate to your Kaggle roc score.</p>\n<p>http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedKFold.html</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "77785",
      "postDate": "05/10/2015 17:36:18",
      "content": "<p>I have the same problem. Or maybe I would not call it a problem but rather a feature - I am not using the sklearn cross_validation but pick sub samples myself. I guess I have to aggregate the results properly before calculating the roc score, otherwise it will not have much in common with the leaderboard score.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "77789",
      "postDate": "05/10/2015 19:11:21",
      "content": "<p>Hi Ugur,&nbsp;</p>\n\n<p>Maybe try adding scoring = 'roc_auc' parmeter to your GridSearchCV. I found that this helps more accurately predict the ROC area and also find a better model over the parameter schmoo than using the default prediction accuracy.&nbsp;</p>\n<p>http://stackoverflow.com/questions/14955458/does-gridsearchcv-use-predict-or-predict-proba-when-using-auc-score-as-score-fu</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "77825",
      "postDate": "05/11/2015 00:54:41",
      "content": "<p>Hello Everyone,</p>\n<p>Thank you very much for your responses. I figured out my problem. Basically I was making a wrong join in pandas for submission. Therefore, all my submissions was worth nothing and got the scores totally out of luck :).</p>\n<p>I fixed it and now validation and submission scores are pretty close.&nbsp;</p>\n\n<p>@Kushankr&nbsp;and @VijaySathish thank you, I am trying&nbsp;the methods you proposed!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80171",
      "postDate": "05/27/2015 18:52:42",
      "content": "<p>[quote=U&#287;ur Akyol;77825]</p>\n<p>Hello Everyone,</p>\n<p>Thank you very much for your responses. I figured out my problem. Basically I was making a wrong join in pandas for submission. Therefore, all my submissions was worth nothing and got the scores totally out of luck :).</p>\n<p>I fixed it and now validation and submission scores are pretty close.&nbsp;</p>\n<p>@Kushankr&nbsp;and @VijaySathish thank you, I am trying&nbsp;the methods you proposed!</p>\n<p>[/quote]</p>\n\n<p>Can you share the specifics of the bug. &nbsp;I am having the same problem and am not able to find the bug. Thanks in advance.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 77740,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "05/10/2015 05:07:00",
      "content": "<p>My local CV and leaderboard scores are pretty close. You must be doing something wrong.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 77741,
      "author_name": "",
      "author_url": "",
      "post_date": "05/10/2015 06:02:57",
      "content": "<p>I am not sure if you are already doing it yet,</p>\n<p>But you need to use Stratified KFold from scikit learn : The mean scores across multiple folds using this method will provide a very close estimate to your Kaggle roc score.</p>\n<p>http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedKFold.html</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 77785,
      "author_name": "mpekalski",
      "author_url": "",
      "post_date": "05/10/2015 17:36:18",
      "content": "<p>I have the same problem. Or maybe I would not call it a problem but rather a feature - I am not using the sklearn cross_validation but pick sub samples myself. I guess I have to aggregate the results properly before calculating the roc score, otherwise it will not have much in common with the leaderboard score.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 77789,
      "author_name": "vijaysathish",
      "author_url": "",
      "post_date": "05/10/2015 19:11:21",
      "content": "<p>Hi Ugur,&nbsp;</p>\n\n<p>Maybe try adding scoring = 'roc_auc' parmeter to your GridSearchCV. I found that this helps more accurately predict the ROC area and also find a better model over the parameter schmoo than using the default prediction accuracy.&nbsp;</p>\n<p>http://stackoverflow.com/questions/14955458/does-gridsearchcv-use-predict-or-predict-proba-when-using-auc-score-as-score-fu</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 77825,
      "author_name": "ugur47",
      "author_url": "",
      "post_date": "05/11/2015 00:54:41",
      "content": "<p>Hello Everyone,</p>\n<p>Thank you very much for your responses. I figured out my problem. Basically I was making a wrong join in pandas for submission. Therefore, all my submissions was worth nothing and got the scores totally out of luck :).</p>\n<p>I fixed it and now validation and submission scores are pretty close.&nbsp;</p>\n\n<p>@Kushankr&nbsp;and @VijaySathish thank you, I am trying&nbsp;the methods you proposed!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80171,
      "author_name": "saquibk",
      "author_url": "",
      "post_date": "05/27/2015 18:52:42",
      "content": "<p>[quote=U&#287;ur Akyol;77825]</p>\n<p>Hello Everyone,</p>\n<p>Thank you very much for your responses. I figured out my problem. Basically I was making a wrong join in pandas for submission. Therefore, all my submissions was worth nothing and got the scores totally out of luck :).</p>\n<p>I fixed it and now validation and submission scores are pretty close.&nbsp;</p>\n<p>@Kushankr&nbsp;and @VijaySathish thank you, I am trying&nbsp;the methods you proposed!</p>\n<p>[/quote]</p>\n\n<p>Can you share the specifics of the bug. &nbsp;I am having the same problem and am not able to find the bug. Thanks in advance.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "77739": "",
    "77740": "",
    "77741": "",
    "77785": "",
    "77789": "",
    "77825": "",
    "80171": ""
  },
  "source": "meta"
}