{
  "id": 124844,
  "title": "Drop in AUC Expected?",
  "url": "/competitions/cat-in-the-dat-ii/discussion/124844",
  "author_name": "",
  "post_date": "2020-01-07T02:17:56.077354900Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hey!</p>\n\n<p>I'm getting approximately a 0.005 drop in AUC from my 5-fold cross validation to the Leaderboard.</p>\n\n<p>This is my first Kaggle comp, so I'm wondering if this decrease is expected? None of my folds dropped nearly that much.</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "712263",
      "postDate": "01/07/2020 02:17:56",
      "content": "<p>Hey!</p>\n\n<p>I'm getting approximately a 0.005 drop in AUC from my 5-fold cross validation to the Leaderboard.</p>\n\n<p>This is my first Kaggle comp, so I'm wondering if this decrease is expected? None of my folds dropped nearly that much.</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "Hey!\n\nI'm getting approximately a 0.005 drop in AUC from my 5-fold cross validation to the Leaderboard.\n\nThis is my first Kaggle comp, so I'm wondering if this decrease is expected? None of my folds dropped nearly that much.\n\nThanks!",
      "votes": null
    },
    {
      "id": "712795",
      "postDate": "01/07/2020 15:36:54",
      "content": "<p>My auc :\nXGB, 5-fold cross validation,target_encoded noms\nCV:[1597]train-auc:0.789616+0.000237    test-auc:0.788510+0.000955\nlb:0.78586</p>",
      "rawMarkdown": "My auc :\nXGB, 5-fold cross validation,target_encoded noms\nCV:[1597]train-auc:0.789616+0.000237\ttest-auc:0.788510+0.000955\nlb:0.78586",
      "votes": null
    },
    {
      "id": "731914",
      "postDate": "01/29/2020 08:36:12",
      "content": "<p>It's the same with me. This is probably because there are small systematic differences between the training and test data set.</p>",
      "rawMarkdown": "It's the same with me. This is probably because there are small systematic differences between the training and test data set.",
      "votes": null
    },
    {
      "id": "732383",
      "postDate": "01/29/2020 18:46:29",
      "content": "<p>It is must likely just 'natural' variation as a result of different feature values between the different data sets. </p>\n\n<p>Print the five different AUC's from your cross validation and see how they vary.</p>",
      "rawMarkdown": "It is must likely just 'natural' variation as a result of different feature values between the different data sets. \n\nPrint the five different AUC's from your cross validation and see how they vary.",
      "votes": null
    },
    {
      "id": "738973",
      "postDate": "02/07/2020 08:33:59",
      "content": "<p>What function do you use ? I use <code>roc_auc_score</code> of scikit.</p>",
      "rawMarkdown": "What function do you use ? I use `roc_auc_score` of scikit.",
      "votes": null
    },
    {
      "id": "743435",
      "postDate": "02/12/2020 04:00:28",
      "content": "<p>Is there a big difference between 0.786 and 0.781? or does it boil down to specific case by case probing? Would augmenting probabilities and reducing uncertainty improve AUC?</p>",
      "rawMarkdown": "Is there a big difference between 0.786 and 0.781? or does it boil down to specific case by case probing? Would augmenting probabilities and reducing uncertainty improve AUC?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 712795,
      "author_name": "kailai",
      "author_url": "",
      "post_date": "01/07/2020 15:36:54",
      "content": "<p>My auc :\nXGB, 5-fold cross validation,target_encoded noms\nCV:[1597]train-auc:0.789616+0.000237    test-auc:0.788510+0.000955\nlb:0.78586</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 731914,
      "author_name": "frankmollard",
      "author_url": "",
      "post_date": "01/29/2020 08:36:12",
      "content": "<p>It's the same with me. This is probably because there are small systematic differences between the training and test data set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 732383,
      "author_name": "fchmiel",
      "author_url": "",
      "post_date": "01/29/2020 18:46:29",
      "content": "<p>It is must likely just 'natural' variation as a result of different feature values between the different data sets. </p>\n\n<p>Print the five different AUC's from your cross validation and see how they vary.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 738973,
      "author_name": "catadanna",
      "author_url": "",
      "post_date": "02/07/2020 08:33:59",
      "content": "<p>What function do you use ? I use <code>roc_auc_score</code> of scikit.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 743435,
      "author_name": "rkotwani",
      "author_url": "",
      "post_date": "02/12/2020 04:00:28",
      "content": "<p>Is there a big difference between 0.786 and 0.781? or does it boil down to specific case by case probing? Would augmenting probabilities and reducing uncertainty improve AUC?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "712263": "Hey!\n\nI'm getting approximately a 0.005 drop in AUC from my 5-fold cross validation to the Leaderboard.\n\nThis is my first Kaggle comp, so I'm wondering if this decrease is expected? None of my folds dropped nearly that much.\n\nThanks!",
    "712795": "My auc :\nXGB, 5-fold cross validation,target_encoded noms\nCV:[1597]train-auc:0.789616+0.000237\ttest-auc:0.788510+0.000955\nlb:0.78586",
    "731914": "It's the same with me. This is probably because there are small systematic differences between the training and test data set.",
    "732383": "It is must likely just 'natural' variation as a result of different feature values between the different data sets. \n\nPrint the five different AUC's from your cross validation and see how they vary.",
    "738973": "What function do you use ? I use `roc_auc_score` of scikit.",
    "743435": "Is there a big difference between 0.786 and 0.781? or does it boil down to specific case by case probing? Would augmenting probabilities and reducing uncertainty improve AUC?"
  },
  "source": "meta"
}