{
  "id": 81583,
  "title": "Hypothesis: there are more outliers in public than in private testing dataset by proportion",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/81583",
  "author_name": "",
  "post_date": "2019-02-22T15:46:03.287743Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>We all notice that the LB RMSE is typically higher than that in our local CV. One reason I can think of is that the distribution between the training dataset and the public testing dataset are different. In particular, distribution here means especially the number of outliers among other things. </p>\n\n<p>If my hypothesis is true, it means there are less outliers on proportion in the private dataset. For example, there are 1000 points in both training and testing dataset. And in each training/testing dataset, there will be 100 outlier points. For the 30% of the public testing dataset, there can be 40 outlier points (so 40/300 = 13.33% of the public testing dataset) whereas in the 70% of the private testing dataset, there can be 60 outlier points (8.57% of the private testing dataset).</p>\n\n<p>Maybe some of you realize this already. Let me know your thoughts. Feel free to prove me I am wrong.</p>",
  "messages": [
    {
      "id": "476725",
      "postDate": "02/22/2019 15:46:03",
      "content": "<p>We all notice that the LB RMSE is typically higher than that in our local CV. One reason I can think of is that the distribution between the training dataset and the public testing dataset are different. In particular, distribution here means especially the number of outliers among other things. </p>\n\n<p>If my hypothesis is true, it means there are less outliers on proportion in the private dataset. For example, there are 1000 points in both training and testing dataset. And in each training/testing dataset, there will be 100 outlier points. For the 30% of the public testing dataset, there can be 40 outlier points (so 40/300 = 13.33% of the public testing dataset) whereas in the 70% of the private testing dataset, there can be 60 outlier points (8.57% of the private testing dataset).</p>\n\n<p>Maybe some of you realize this already. Let me know your thoughts. Feel free to prove me I am wrong.</p>",
      "rawMarkdown": "We all notice that the LB RMSE is typically higher than that in our local CV. One reason I can think of is that the distribution between the training dataset and the public testing dataset are different. In particular, distribution here means especially the number of outliers among other things. \n\nIf my hypothesis is true, it means there are less outliers on proportion in the private dataset. For example, there are 1000 points in both training and testing dataset. And in each training/testing dataset, there will be 100 outlier points. For the 30% of the public testing dataset, there can be 40 outlier points (so 40/300 = 13.33% of the public testing dataset) whereas in the 70% of the private testing dataset, there can be 60 outlier points (8.57% of the private testing dataset).\n\nMaybe some of you realize this already. Let me know your thoughts. Feel free to prove me I am wrong.",
      "votes": null
    },
    {
      "id": "476740",
      "postDate": "02/22/2019 16:23:19",
      "content": "<p>Interesting. But I guess that there is also the option that there are less outliers in the training and public and private LB have the same proportion isn't ?\nFrancois</p>",
      "rawMarkdown": "Interesting. But I guess that there is also the option that there are less outliers in the training and public and private LB have the same proportion isn't ?\nFrancois",
      "votes": null
    },
    {
      "id": "476744",
      "postDate": "02/22/2019 16:30:02",
      "content": "<p>True. Another of my thoughts is that the training and testing datasets are not necessarily similar. So many possibilities. </p>",
      "rawMarkdown": "True. Another of my thoughts is that the training and testing datasets are not necessarily similar. So many possibilities.",
      "votes": null
    },
    {
      "id": "476746",
      "postDate": "02/22/2019 16:36:08",
      "content": "<p>Yes. I guess that's the problem in most of the projects. It what make them interesting :) </p>",
      "rawMarkdown": "Yes. I guess that's the problem in most of the projects. It what make them interesting :)",
      "votes": null
    },
    {
      "id": "477712",
      "postDate": "02/25/2019 06:30:45",
      "content": "<p>we're going to see it asap buddy :)</p>",
      "rawMarkdown": "we're going to see it asap buddy :)",
      "votes": null
    },
    {
      "id": "477808",
      "postDate": "02/25/2019 09:36:13",
      "content": "<p>You have two submissions ;) </p>",
      "rawMarkdown": "You have two submissions ;)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 476740,
      "author_name": "francoisdubois",
      "author_url": "",
      "post_date": "02/22/2019 16:23:19",
      "content": "<p>Interesting. But I guess that there is also the option that there are less outliers in the training and public and private LB have the same proportion isn't ?\nFrancois</p>",
      "votes": null,
      "replies": [
        {
          "id": 476744,
          "author_name": "pukkinming",
          "author_url": "",
          "post_date": "02/22/2019 16:30:02",
          "content": "<p>True. Another of my thoughts is that the training and testing datasets are not necessarily similar. So many possibilities. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 476746,
          "author_name": "francoisdubois",
          "author_url": "",
          "post_date": "02/22/2019 16:36:08",
          "content": "<p>Yes. I guess that's the problem in most of the projects. It what make them interesting :) </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 477712,
      "author_name": "frtgnn",
      "author_url": "",
      "post_date": "02/25/2019 06:30:45",
      "content": "<p>we're going to see it asap buddy :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 477808,
      "author_name": "oleg90777",
      "author_url": "",
      "post_date": "02/25/2019 09:36:13",
      "content": "<p>You have two submissions ;) </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "476725": "We all notice that the LB RMSE is typically higher than that in our local CV. One reason I can think of is that the distribution between the training dataset and the public testing dataset are different. In particular, distribution here means especially the number of outliers among other things. \n\nIf my hypothesis is true, it means there are less outliers on proportion in the private dataset. For example, there are 1000 points in both training and testing dataset. And in each training/testing dataset, there will be 100 outlier points. For the 30% of the public testing dataset, there can be 40 outlier points (so 40/300 = 13.33% of the public testing dataset) whereas in the 70% of the private testing dataset, there can be 60 outlier points (8.57% of the private testing dataset).\n\nMaybe some of you realize this already. Let me know your thoughts. Feel free to prove me I am wrong.",
    "476740": "Interesting. But I guess that there is also the option that there are less outliers in the training and public and private LB have the same proportion isn't ?\nFrancois",
    "476744": "True. Another of my thoughts is that the training and testing datasets are not necessarily similar. So many possibilities.",
    "476746": "Yes. I guess that's the problem in most of the projects. It what make them interesting :)",
    "477712": "we're going to see it asap buddy :)",
    "477808": "You have two submissions ;)"
  },
  "source": "meta"
}