{
  "id": 73156,
  "title": "CV/LB difference and ...Reason maybe?",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/73156",
  "author_name": "",
  "post_date": "2018-11-30T10:10:53.957334600Z",
  "votes": 14,
  "comment_count": 6,
  "views": 0,
  "content": "<p>It's strange that naive(only train.csv) gives 3.92 cv and 3.92LB but When I add the info from hist/new , got 3.71 CV but 3.94LB.</p>\n\n<p>Moreover, when I remove the 'likely outlier' ( target &lt; -20, you guys may notice that there's a small part of the target value below -30) I got local CV RMSE 1.5X, but LB still. </p>\n\n<p>My guess is the nature of the metrics RMSE,  these 'outliers' are SO FAR from the data main cluster that they stretch the RMSE so large. But they are the actuall part of the data(especially for the TEST set). </p>\n\n<p>And sometimes the model is not sensitive to these 'outlier' , or it's difficult to correctly predict these value. </p>\n\n<p>I wonder if we can do something?</p>\n\n<p>(1) maybe weight more to these ' outliers' (might be tricky and/or extremely stupid)\n(2) EDA more on these part to find out something？</p>\n\n<p>And...ofc there could be other reason for the CV/LB gap, any criticize is welcomed! </p>",
  "messages": [
    {
      "id": "430392",
      "postDate": "11/30/2018 10:10:53",
      "content": "<p>It's strange that naive(only train.csv) gives 3.92 cv and 3.92LB but When I add the info from hist/new , got 3.71 CV but 3.94LB.</p>\n\n<p>Moreover, when I remove the 'likely outlier' ( target &lt; -20, you guys may notice that there's a small part of the target value below -30) I got local CV RMSE 1.5X, but LB still. </p>\n\n<p>My guess is the nature of the metrics RMSE,  these 'outliers' are SO FAR from the data main cluster that they stretch the RMSE so large. But they are the actuall part of the data(especially for the TEST set). </p>\n\n<p>And sometimes the model is not sensitive to these 'outlier' , or it's difficult to correctly predict these value. </p>\n\n<p>I wonder if we can do something?</p>\n\n<p>(1) maybe weight more to these ' outliers' (might be tricky and/or extremely stupid)\n(2) EDA more on these part to find out something？</p>\n\n<p>And...ofc there could be other reason for the CV/LB gap, any criticize is welcomed! </p>",
      "rawMarkdown": "It's strange that naive(only train.csv) gives 3.92 cv and 3.92LB but When I add the info from hist/new , got 3.71 CV but 3.94LB.\n\nMoreover, when I remove the 'likely outlier' ( target &lt; -20, you guys may notice that there's a small part of the target value below -30) I got local CV RMSE 1.5X, but LB still. \n\nMy guess is the nature of the metrics RMSE,  these 'outliers' are SO FAR from the data main cluster that they stretch the RMSE so large. But they are the actuall part of the data(especially for the TEST set). \n\nAnd sometimes the model is not sensitive to these 'outlier' , or it's difficult to correctly predict these value. \n\nI wonder if we can do something?\n\n(1) maybe weight more to these ' outliers' (might be tricky and/or extremely stupid)\n(2) EDA more on these part to find out something？\n\nAnd...ofc there could be other reason for the CV/LB gap, any criticize is welcomed!",
      "votes": null
    },
    {
      "id": "430430",
      "postDate": "11/30/2018 11:20:53",
      "content": "<p>My guess is, the catch in this competition is in those outliers. So I will focus on classifying them first.  That is I will form a new binary target, outlier or not and   do some classification of it. </p>",
      "rawMarkdown": "My guess is, the catch in this competition is in those outliers. So I will focus on classifying them first.  That is I will form a new binary target, outlier or not and   do some classification of it.",
      "votes": null
    },
    {
      "id": "430599",
      "postDate": "11/30/2018 17:17:23",
      "content": "<p>I encountered the same for the Outliers. But because LB is so different this means the outliers are a key aspect of this challenge.</p>",
      "rawMarkdown": "I encountered the same for the Outliers. But because LB is so different this means the outliers are a key aspect of this challenge.",
      "votes": null
    },
    {
      "id": "430632",
      "postDate": "11/30/2018 18:44:10",
      "content": "<p>I did a quick spot check: if you exclude the -33.xx values from the validation set, the error drops to ~ 1.5 =&gt; those 2000+ points are responsible for over half of the error...</p>",
      "rawMarkdown": "I did a quick spot check: if you exclude the -33.xx values from the validation set, the error drops to ~ 1.5 =&gt; those 2000+ points are responsible for over half of the error...",
      "votes": null
    },
    {
      "id": "431008",
      "postDate": "12/01/2018 12:16:20",
      "content": "<blockquote>\n  <p>\"When I add the info from hist/new , got 3.71 CV but 3.94LB.\"</p>\n</blockquote>\n\n<p>Maybe there's a bug in your code around here and the test data is not aligned right....that's what happened to me</p>",
      "rawMarkdown": "&gt; \"When I add the info from hist/new , got 3.71 CV but 3.94LB.\"\n\nMaybe there's a bug in your code around here and the test data is not aligned right....that's what happened to me",
      "votes": null
    },
    {
      "id": "431228",
      "postDate": "12/01/2018 22:05:31",
      "content": "<p>FWIW: I tried removing the outliers (same overfit to training as you experienced), as well as weighing the outliers extra - made both the cv and lb worse. My current best bet: predicting which observations are below -20, but it's an extremely small proportion.</p>",
      "rawMarkdown": "FWIW: I tried removing the outliers (same overfit to training as you experienced), as well as weighing the outliers extra - made both the cv and lb worse. My current best bet: predicting which observations are below -20, but it's an extremely small proportion.",
      "votes": null
    },
    {
      "id": "431233",
      "postDate": "12/01/2018 22:39:50",
      "content": "<p>1% is not extremely small though.</p>",
      "rawMarkdown": "1% is not extremely small though.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 430430,
      "author_name": "chambox",
      "author_url": "",
      "post_date": "11/30/2018 11:20:53",
      "content": "<p>My guess is, the catch in this competition is in those outliers. So I will focus on classifying them first.  That is I will form a new binary target, outlier or not and   do some classification of it. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430599,
      "author_name": "rdoume",
      "author_url": "",
      "post_date": "11/30/2018 17:17:23",
      "content": "<p>I encountered the same for the Outliers. But because LB is so different this means the outliers are a key aspect of this challenge.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430632,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/30/2018 18:44:10",
      "content": "<p>I did a quick spot check: if you exclude the -33.xx values from the validation set, the error drops to ~ 1.5 =&gt; those 2000+ points are responsible for over half of the error...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 431008,
      "author_name": "grmunjal",
      "author_url": "",
      "post_date": "12/01/2018 12:16:20",
      "content": "<blockquote>\n  <p>\"When I add the info from hist/new , got 3.71 CV but 3.94LB.\"</p>\n</blockquote>\n\n<p>Maybe there's a bug in your code around here and the test data is not aligned right....that's what happened to me</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 431228,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "12/01/2018 22:05:31",
      "content": "<p>FWIW: I tried removing the outliers (same overfit to training as you experienced), as well as weighing the outliers extra - made both the cv and lb worse. My current best bet: predicting which observations are below -20, but it's an extremely small proportion.</p>",
      "votes": null,
      "replies": [
        {
          "id": 431233,
          "author_name": "chambox",
          "author_url": "",
          "post_date": "12/01/2018 22:39:50",
          "content": "<p>1% is not extremely small though.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "430392": "It's strange that naive(only train.csv) gives 3.92 cv and 3.92LB but When I add the info from hist/new , got 3.71 CV but 3.94LB.\n\nMoreover, when I remove the 'likely outlier' ( target &lt; -20, you guys may notice that there's a small part of the target value below -30) I got local CV RMSE 1.5X, but LB still. \n\nMy guess is the nature of the metrics RMSE,  these 'outliers' are SO FAR from the data main cluster that they stretch the RMSE so large. But they are the actuall part of the data(especially for the TEST set). \n\nAnd sometimes the model is not sensitive to these 'outlier' , or it's difficult to correctly predict these value. \n\nI wonder if we can do something?\n\n(1) maybe weight more to these ' outliers' (might be tricky and/or extremely stupid)\n(2) EDA more on these part to find out something？\n\nAnd...ofc there could be other reason for the CV/LB gap, any criticize is welcomed!",
    "430430": "My guess is, the catch in this competition is in those outliers. So I will focus on classifying them first.  That is I will form a new binary target, outlier or not and   do some classification of it.",
    "430599": "I encountered the same for the Outliers. But because LB is so different this means the outliers are a key aspect of this challenge.",
    "430632": "I did a quick spot check: if you exclude the -33.xx values from the validation set, the error drops to ~ 1.5 =&gt; those 2000+ points are responsible for over half of the error...",
    "431008": "&gt; \"When I add the info from hist/new , got 3.71 CV but 3.94LB.\"\n\nMaybe there's a bug in your code around here and the test data is not aligned right....that's what happened to me",
    "431228": "FWIW: I tried removing the outliers (same overfit to training as you experienced), as well as weighing the outliers extra - made both the cv and lb worse. My current best bet: predicting which observations are below -20, but it's an extremely small proportion.",
    "431233": "1% is not extremely small though."
  },
  "source": "meta"
}