{
  "id": 82046,
  "title": "CV, public LB and private LB",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/82046",
  "author_name": "YoonSoo",
  "post_date": "2019-02-27T01:38:14.135000",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>First of all, congratulations to winners!\nLikely due to outliers, private leaderboard had huge shakeups. It struck me too, and I slipped about -950 on the final rankings.\nAmong two submissions, I chose one with the highest CV score, and the other with the highest public LB score. But neither did well on the private leaderboard.</p>\n\n<p>Did trusting CV work for you guys? Or is there any strategy that made robust submissions?\nWhat were your CV, public LB, and private LB scores?</p>\n\n<p>After post processing, mine was as follows:\nModel with best CV - CV: 3.62246, public LB: 3.686, private LB: 3.619\nModel with best public LB - CV: ??(I didn't record it), public LB: 3.668, private LB: 3.616</p>\n\n<p>Thanks.</p>",
  "messages": [
    {
      "id": 479212,
      "postDate": "2019-02-27T02:50:24.470Z",
      "content": "<p>In this competition, I've selected top 2 lb score submission and it also struck me out. </p>\n\n<p>From the leader board proving, I concluded Test set has more so called \"outlier\" than Train set. From the outlier ratio calculated from probing, I modified the ratio of outlier of each validation fold for every cross validation set. By doing it, the cv and lb showed not that huge difference(about 0.02) and also they ar align. Our teams mistake was not to trust our cv and select highest lb score submission sets.</p>\n\n<p>My best cv score was 3.716 public 3.698 private 3.609 without any post processing. And by ensembling with public kernels, we lose track cv; ? public 3.677 private 3.622. </p>\n\n<p>The most valuable thing that I learn from this competition is always keep track of everything what I did and trust cv.</p>",
      "rawMarkdown": "In this competition, I've selected top 2 lb score submission and it also struck me out. \n\nFrom the leader board proving, I concluded Test set has more so called \"outlier\" than Train set. From the outlier ratio calculated from probing, I modified the ratio of outlier of each validation fold for every cross validation set. By doing it, the cv and lb showed not that huge difference(about 0.02) and also they ar align. Our teams mistake was not to trust our cv and select highest lb score submission sets.\n\nMy best cv score was 3.716 public 3.698 private 3.609 without any post processing. And by ensembling with public kernels, we lose track cv; ? public 3.677 private 3.622. \n\nThe most valuable thing that I learn from this competition is always keep track of everything what I did and trust cv.",
      "votes": 1,
      "replies": [
        {
          "id": 479219,
          "postDate": "2019-02-27T02:57:48.060Z",
          "content": "<p>Thank you for sharing. Like you said, maybe I should have modified outlier ratio when generating folds to make cv more robust.</p>",
          "rawMarkdown": "Thank you for sharing. Like you said, maybe I should have modified outlier ratio when generating folds to make cv more robust."
        },
        {
          "id": 479240,
          "postDate": "2019-02-27T03:07:01.283Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 479140,
      "postDate": "2019-02-27T01:38:14.137Z",
      "content": "<p>First of all, congratulations to winners!\nLikely due to outliers, private leaderboard had huge shakeups. It struck me too, and I slipped about -950 on the final rankings.\nAmong two submissions, I chose one with the highest CV score, and the other with the highest public LB score. But neither did well on the private leaderboard.</p>\n\n<p>Did trusting CV work for you guys? Or is there any strategy that made robust submissions?\nWhat were your CV, public LB, and private LB scores?</p>\n\n<p>After post processing, mine was as follows:\nModel with best CV - CV: 3.62246, public LB: 3.686, private LB: 3.619\nModel with best public LB - CV: ??(I didn't record it), public LB: 3.668, private LB: 3.616</p>\n\n<p>Thanks.</p>",
      "rawMarkdown": "First of all, congratulations to winners!\nLikely due to outliers, private leaderboard had huge shakeups. It struck me too, and I slipped about -950 on the final rankings.\nAmong two submissions, I chose one with the highest CV score, and the other with the highest public LB score. But neither did well on the private leaderboard.\n\nDid trusting CV work for you guys? Or is there any strategy that made robust submissions?\nWhat were your CV, public LB, and private LB scores?\n\nAfter post processing, mine was as follows:\nModel with best CV - CV: 3.62246, public LB: 3.686, private LB: 3.619\nModel with best public LB - CV: ??(I didn't record it), public LB: 3.668, private LB: 3.616\n\nThanks.",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 479212,
      "author_name": "giba.kim",
      "author_url": "",
      "post_date": "2019-02-27T02:50:24.470000",
      "content": "<p>In this competition, I've selected top 2 lb score submission and it also struck me out. </p>\n\n<p>From the leader board proving, I concluded Test set has more so called \"outlier\" than Train set. From the outlier ratio calculated from probing, I modified the ratio of outlier of each validation fold for every cross validation set. By doing it, the cv and lb showed not that huge difference(about 0.02) and also they ar align. Our teams mistake was not to trust our cv and select highest lb score submission sets.</p>\n\n<p>My best cv score was 3.716 public 3.698 private 3.609 without any post processing. And by ensembling with public kernels, we lose track cv; ? public 3.677 private 3.622. </p>\n\n<p>The most valuable thing that I learn from this competition is always keep track of everything what I did and trust cv.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 479219,
          "author_name": "YoonSoo",
          "author_url": "",
          "post_date": "2019-02-27T02:57:48.060000",
          "content": "<p>Thank you for sharing. Like you said, maybe I should have modified outlier ratio when generating folds to make cv more robust.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 479240,
          "author_name": "",
          "author_url": "",
          "post_date": "2019-02-27T03:07:01.283000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "479212": "In this competition, I've selected top 2 lb score submission and it also struck me out. \n\nFrom the leader board proving, I concluded Test set has more so called \"outlier\" than Train set. From the outlier ratio calculated from probing, I modified the ratio of outlier of each validation fold for every cross validation set. By doing it, the cv and lb showed not that huge difference(about 0.02) and also they ar align. Our teams mistake was not to trust our cv and select highest lb score submission sets.\n\nMy best cv score was 3.716 public 3.698 private 3.609 without any post processing. And by ensembling with public kernels, we lose track cv; ? public 3.677 private 3.622. \n\nThe most valuable thing that I learn from this competition is always keep track of everything what I did and trust cv.",
    "479140": "First of all, congratulations to winners!\nLikely due to outliers, private leaderboard had huge shakeups. It struck me too, and I slipped about -950 on the final rankings.\nAmong two submissions, I chose one with the highest CV score, and the other with the highest public LB score. But neither did well on the private leaderboard.\n\nDid trusting CV work for you guys? Or is there any strategy that made robust submissions?\nWhat were your CV, public LB, and private LB scores?\n\nAfter post processing, mine was as follows:\nModel with best CV - CV: 3.62246, public LB: 3.686, private LB: 3.619\nModel with best public LB - CV: ??(I didn't record it), public LB: 3.668, private LB: 3.616\n\nThanks."
  }
}