{
  "id": 81975,
  "title": "Not picking the best model as a submission...",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/81975",
  "author_name": "raddar",
  "post_date": "2019-02-26T14:39:04.608000",
  "votes": 29,
  "comment_count": 23,
  "views": 0,
  "content": "<p>We have been choosing between 3 submissions with LB scores of 3.639, 3.650 and 3.662. \nAnd guess what - we do not believe in our best public LB model!</p>\n\n<p>This is indeed one of the toughest choices in Kaggle to make...\nHow do you feel ?</p>",
  "messages": [
    {
      "id": 478724,
      "postDate": "2019-02-26T14:39:04.610Z",
      "content": "<p>We have been choosing between 3 submissions with LB scores of 3.639, 3.650 and 3.662. \nAnd guess what - we do not believe in our best public LB model!</p>\n\n<p>This is indeed one of the toughest choices in Kaggle to make...\nHow do you feel ?</p>",
      "rawMarkdown": "We have been choosing between 3 submissions with LB scores of 3.639, 3.650 and 3.662. \nAnd guess what - we do not believe in our best public LB model!\n\nThis is indeed one of the toughest choices in Kaggle to make...\nHow do you feel ?",
      "votes": 29
    },
    {
      "id": 479425,
      "postDate": "2019-02-27T05:59:32.173Z",
      "content": "<p>the best CV model with no overrides (3.662 public) was the best submission for us.</p>\n\n<p>screenshot provided!</p>",
      "rawMarkdown": "the best CV model with no overrides (3.662 public) was the best submission for us.\n\nscreenshot provided!",
      "votes": 1
    },
    {
      "id": 478738,
      "postDate": "2019-02-26T14:50:56.403Z",
      "content": "<p>How do I feel?\nAwful and curious in the face of ambiguity =D</p>\n\n<p>In the previous santander competition, I was first faced with the issue of CV vs LB.\nmy best Private LB submission was submitted 2 weeks before end of competition, and because of bad public LB score i went on a completely different path of good overfitted public LB .\nI selected best LB submissions and slid down private LB when competition ended.\nThat left some scars.</p>\n\n<p>Now the choice is a bit simpler, i strongly believe our CV score, as it correlates well with LB, but unlike you, we don't have sub 3.66 submissions =)\nso the choice is easier.\nNot sure yet how to use the second submission yet, any last minute tips for trying post processing would be nice =P\nHaven't yet found a post processing that improved CV in order to even try to do it on submission.</p>\n\n<p>You've done an awesome job in this competition, robin hood of demystifications.\nYou deserve the 1st position.</p>",
      "rawMarkdown": "How do I feel?\nAwful and curious in the face of ambiguity =D\n\nIn the previous santander competition, I was first faced with the issue of CV vs LB.\nmy best Private LB submission was submitted 2 weeks before end of competition, and because of bad public LB score i went on a completely different path of good overfitted public LB .\nI selected best LB submissions and slid down private LB when competition ended.\nThat left some scars.\n\nNow the choice is a bit simpler, i strongly believe our CV score, as it correlates well with LB, but unlike you, we don't have sub 3.66 submissions =)\nso the choice is easier.\nNot sure yet how to use the second submission yet, any last minute tips for trying post processing would be nice =P\nHaven't yet found a post processing that improved CV in order to even try to do it on submission.\n\nYou've done an awesome job in this competition, robin hood of demystifications.\nYou deserve the 1st position.",
      "votes": 1
    },
    {
      "id": 478748,
      "postDate": "2019-02-26T15:19:46.483Z",
      "content": "<p>I think at this very moment many teams have chance to win the game or gold medal,but final submissions maybe make you shakedown/up,  luck is needed for success.I prefer to trust cv.</p>",
      "rawMarkdown": "I think at this very moment many teams have chance to win the game or gold medal,but final submissions maybe make you shakedown/up,  luck is needed for success.I prefer to trust cv.",
      "votes": 2
    },
    {
      "id": 478737,
      "postDate": "2019-02-26T14:50:37.787Z",
      "content": "<p>in my opinion, if your worries become true after final scoring, it is the fault of ELO. they should prepared train/test datasets with proper distributions. in this case, outlier values -33</p>",
      "rawMarkdown": "in my opinion, if your worries become true after final scoring, it is the fault of ELO. they should prepared train/test datasets with proper distributions. in this case, outlier values -33",
      "votes": 2
    },
    {
      "id": 478726,
      "postDate": "2019-02-26T14:44:23.007Z",
      "content": "<p>Same.</p>",
      "rawMarkdown": "Same.",
      "votes": 2,
      "replies": [
        {
          "id": 478732,
          "postDate": "2019-02-26T14:48:46.813Z",
          "content": "<p>same22222</p>",
          "rawMarkdown": "same22222"
        }
      ]
    },
    {
      "id": 479266,
      "postDate": "2019-02-27T03:23:43.330Z",
      "content": "<p>Congrats <a href=\"/raddar\">@raddar</a> and team. Thanks Raddar for your insights on this data. It really helped me in figuring out how to create more features.</p>\n\n<p>Here is my post on feature selection techniques\n<a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052</a></p>",
      "rawMarkdown": "Congrats @raddar and team. Thanks Raddar for your insights on this data. It really helped me in figuring out how to create more features.\n\nHere is my post on feature selection techniques\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052",
      "votes": -3
    },
    {
      "id": 479460,
      "postDate": "2019-02-27T06:45:43.277Z",
      "content": "<p>Congratulations, and special thanks to your inspring and genius kenel/post.</p>",
      "rawMarkdown": "Congratulations, and special thanks to your inspring and genius kenel/post."
    },
    {
      "id": 479129,
      "postDate": "2019-02-27T01:28:59.407Z",
      "content": "<p>These are such anxious moments. In the Quora Challenge we had two kernels I thought would do well. The first one got us 22nd. The other wouldve gotten us somewhere in the 200's</p>",
      "rawMarkdown": "These are such anxious moments. In the Quora Challenge we had two kernels I thought would do well. The first one got us 22nd. The other wouldve gotten us somewhere in the 200's"
    },
    {
      "id": 479119,
      "postDate": "2019-02-27T01:18:50.147Z",
      "content": "<p>Did it help you eventually, can you confirm ?</p>",
      "rawMarkdown": "Did it help you eventually, can you confirm ?"
    },
    {
      "id": 479036,
      "postDate": "2019-02-27T00:07:23.377Z",
      "content": "<p>Were you right?</p>",
      "rawMarkdown": "Were you right?"
    },
    {
      "id": 479022,
      "postDate": "2019-02-26T23:32:18.817Z",
      "content": "<p>So how do you choose then ? Do you trust more your CV than your LB ?</p>",
      "rawMarkdown": "So how do you choose then ? Do you trust more your CV than your LB ?"
    },
    {
      "id": 478910,
      "postDate": "2019-02-26T19:34:26.333Z",
      "content": "<p>Same here. I will take the risk to use the best LB submission though... The other one will be my best CV with postprocessing....</p>",
      "rawMarkdown": "Same here. I will take the risk to use the best LB submission though... The other one will be my best CV with postprocessing...."
    },
    {
      "id": 478833,
      "postDate": "2019-02-26T17:33:34.237Z",
      "content": "<p>Same +1.</p>",
      "rawMarkdown": "Same +1."
    },
    {
      "id": 478825,
      "postDate": "2019-02-26T17:17:00.387Z",
      "content": "<p>That's gutsy, good luck!\nAfter unsuccessfully trying to build validation schemas for the outlier detection threshold I have decided to go with best CV (3.6371) with two very different numbers of clipped predictions, 34 and 58.\nHow many samples are you clipping in the final subs?</p>",
      "rawMarkdown": "That's gutsy, good luck!\nAfter unsuccessfully trying to build validation schemas for the outlier detection threshold I have decided to go with best CV (3.6371) with two very different numbers of clipped predictions, 34 and 58.\nHow many samples are you clipping in the final subs?",
      "replies": [
        {
          "id": 478844,
          "postDate": "2019-02-26T17:41:12.397Z",
          "content": "<p><a href=\"/alsaco\">@alsaco</a> what do you mean by clipping?</p>",
          "rawMarkdown": "@alsaco what do you mean by clipping?",
          "votes": 1
        },
        {
          "id": 478846,
          "postDate": "2019-02-26T17:46:44.353Z",
          "content": "<p>Mapping certain predictions to -33.2</p>",
          "rawMarkdown": "Mapping certain predictions to -33.2",
          "votes": 2
        },
        {
          "id": 479322,
          "postDate": "2019-02-27T04:19:31.240Z",
          "content": "<p>I clipped 75, and it indeed improved my Public LB but what a surprise in Private LB. Now I checked my cv without outlier classification, It perform way better with only regression then doing all classification and clipping. ( Clipping with probability less then 50% is playing with fire! )</p>",
          "rawMarkdown": "I clipped 75, and it indeed improved my Public LB but what a surprise in Private LB. Now I checked my cv without outlier classification, It perform way better with only regression then doing all classification and clipping. ( Clipping with probability less then 50% is playing with fire! )"
        }
      ]
    },
    {
      "id": 478822,
      "postDate": "2019-02-26T17:12:23.353Z",
      "content": "<p>To Choose is To Renounce...</p>",
      "rawMarkdown": "To Choose is To Renounce..."
    },
    {
      "id": 478794,
      "postDate": "2019-02-26T16:45:38.593Z",
      "content": "<p>probably dice with death for me lol. </p>",
      "rawMarkdown": "probably dice with death for me lol. "
    },
    {
      "id": 478756,
      "postDate": "2019-02-26T15:31:47.923Z",
      "content": "<p>The same choice for me but the price is much lower</p>",
      "rawMarkdown": "The same choice for me but the price is much lower"
    },
    {
      "id": 478750,
      "postDate": "2019-02-26T15:20:47.057Z",
      "content": "<p>very tough call. But best LB may be part of submission according to me considering there are quite few outliers in test set.</p>",
      "rawMarkdown": "very tough call. But best LB may be part of submission according to me considering there are quite few outliers in test set."
    },
    {
      "id": 478821,
      "postDate": "2019-02-26T17:12:21.820Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 479425,
      "author_name": "raddar",
      "author_url": "",
      "post_date": "2019-02-27T05:59:32.173000",
      "content": "<p>the best CV model with no overrides (3.662 public) was the best submission for us.</p>\n\n<p>screenshot provided!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 478738,
      "author_name": "AmirH",
      "author_url": "",
      "post_date": "2019-02-26T14:50:56.403000",
      "content": "<p>How do I feel?\nAwful and curious in the face of ambiguity =D</p>\n\n<p>In the previous santander competition, I was first faced with the issue of CV vs LB.\nmy best Private LB submission was submitted 2 weeks before end of competition, and because of bad public LB score i went on a completely different path of good overfitted public LB .\nI selected best LB submissions and slid down private LB when competition ended.\nThat left some scars.</p>\n\n<p>Now the choice is a bit simpler, i strongly believe our CV score, as it correlates well with LB, but unlike you, we don't have sub 3.66 submissions =)\nso the choice is easier.\nNot sure yet how to use the second submission yet, any last minute tips for trying post processing would be nice =P\nHaven't yet found a post processing that improved CV in order to even try to do it on submission.</p>\n\n<p>You've done an awesome job in this competition, robin hood of demystifications.\nYou deserve the 1st position.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 478748,
      "author_name": "senkin13",
      "author_url": "",
      "post_date": "2019-02-26T15:19:46.483000",
      "content": "<p>I think at this very moment many teams have chance to win the game or gold medal,but final submissions maybe make you shakedown/up,  luck is needed for success.I prefer to trust cv.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 478737,
      "author_name": "dgk1",
      "author_url": "",
      "post_date": "2019-02-26T14:50:37.787000",
      "content": "<p>in my opinion, if your worries become true after final scoring, it is the fault of ELO. they should prepared train/test datasets with proper distributions. in this case, outlier values -33</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 478726,
      "author_name": "Fatih Öztürk",
      "author_url": "",
      "post_date": "2019-02-26T14:44:23.007000",
      "content": "<p>Same.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 478732,
          "author_name": "choco",
          "author_url": "",
          "post_date": "2019-02-26T14:48:46.813000",
          "content": "<p>same22222</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 479266,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2019-02-27T03:23:43.330000",
      "content": "<p>Congrats <a href=\"/raddar\">@raddar</a> and team. Thanks Raddar for your insights on this data. It really helped me in figuring out how to create more features.</p>\n\n<p>Here is my post on feature selection techniques\n<a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052</a></p>",
      "votes": -3,
      "replies": []
    },
    {
      "id": 479460,
      "author_name": "Manchoi",
      "author_url": "",
      "post_date": "2019-02-27T06:45:43.277000",
      "content": "<p>Congratulations, and special thanks to your inspring and genius kenel/post.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 479129,
      "author_name": "ryches",
      "author_url": "",
      "post_date": "2019-02-27T01:28:59.407000",
      "content": "<p>These are such anxious moments. In the Quora Challenge we had two kernels I thought would do well. The first one got us 22nd. The other wouldve gotten us somewhere in the 200's</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 479119,
      "author_name": "Vishy",
      "author_url": "",
      "post_date": "2019-02-27T01:18:50.147000",
      "content": "<p>Did it help you eventually, can you confirm ?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 479036,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2019-02-27T00:07:23.377000",
      "content": "<p>Were you right?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 479022,
      "author_name": "Ben Farcy",
      "author_url": "",
      "post_date": "2019-02-26T23:32:18.817000",
      "content": "<p>So how do you choose then ? Do you trust more your CV than your LB ?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478910,
      "author_name": "Dimosthenis Karaflos",
      "author_url": "",
      "post_date": "2019-02-26T19:34:26.333000",
      "content": "<p>Same here. I will take the risk to use the best LB submission though... The other one will be my best CV with postprocessing....</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478833,
      "author_name": "Yinghui Jiang",
      "author_url": "",
      "post_date": "2019-02-26T17:33:34.237000",
      "content": "<p>Same +1.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478825,
      "author_name": "Alejandro Sáez",
      "author_url": "",
      "post_date": "2019-02-26T17:17:00.387000",
      "content": "<p>That's gutsy, good luck!\nAfter unsuccessfully trying to build validation schemas for the outlier detection threshold I have decided to go with best CV (3.6371) with two very different numbers of clipped predictions, 34 and 58.\nHow many samples are you clipping in the final subs?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 478844,
          "author_name": "AmirH",
          "author_url": "",
          "post_date": "2019-02-26T17:41:12.397000",
          "content": "<p><a href=\"/alsaco\">@alsaco</a> what do you mean by clipping?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 478846,
          "author_name": "Alejandro Sáez",
          "author_url": "",
          "post_date": "2019-02-26T17:46:44.353000",
          "content": "<p>Mapping certain predictions to -33.2</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 479322,
          "author_name": "Rraj",
          "author_url": "",
          "post_date": "2019-02-27T04:19:31.240000",
          "content": "<p>I clipped 75, and it indeed improved my Public LB but what a surprise in Private LB. Now I checked my cv without outlier classification, It perform way better with only regression then doing all classification and clipping. ( Clipping with probability less then 50% is playing with fire! )</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 478822,
      "author_name": "Eric Vos",
      "author_url": "",
      "post_date": "2019-02-26T17:12:23.353000",
      "content": "<p>To Choose is To Renounce...</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478794,
      "author_name": "silverstone",
      "author_url": "",
      "post_date": "2019-02-26T16:45:38.593000",
      "content": "<p>probably dice with death for me lol. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478756,
      "author_name": "kss",
      "author_url": "",
      "post_date": "2019-02-26T15:31:47.923000",
      "content": "<p>The same choice for me but the price is much lower</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478750,
      "author_name": "SubikashPal",
      "author_url": "",
      "post_date": "2019-02-26T15:20:47.057000",
      "content": "<p>very tough call. But best LB may be part of submission according to me considering there are quite few outliers in test set.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 478821,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-02-26T17:12:21.820000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "478724": "We have been choosing between 3 submissions with LB scores of 3.639, 3.650 and 3.662. \nAnd guess what - we do not believe in our best public LB model!\n\nThis is indeed one of the toughest choices in Kaggle to make...\nHow do you feel ?",
    "479425": "the best CV model with no overrides (3.662 public) was the best submission for us.\n\nscreenshot provided!",
    "478738": "How do I feel?\nAwful and curious in the face of ambiguity =D\n\nIn the previous santander competition, I was first faced with the issue of CV vs LB.\nmy best Private LB submission was submitted 2 weeks before end of competition, and because of bad public LB score i went on a completely different path of good overfitted public LB .\nI selected best LB submissions and slid down private LB when competition ended.\nThat left some scars.\n\nNow the choice is a bit simpler, i strongly believe our CV score, as it correlates well with LB, but unlike you, we don't have sub 3.66 submissions =)\nso the choice is easier.\nNot sure yet how to use the second submission yet, any last minute tips for trying post processing would be nice =P\nHaven't yet found a post processing that improved CV in order to even try to do it on submission.\n\nYou've done an awesome job in this competition, robin hood of demystifications.\nYou deserve the 1st position.",
    "478748": "I think at this very moment many teams have chance to win the game or gold medal,but final submissions maybe make you shakedown/up,  luck is needed for success.I prefer to trust cv.",
    "478737": "in my opinion, if your worries become true after final scoring, it is the fault of ELO. they should prepared train/test datasets with proper distributions. in this case, outlier values -33",
    "478726": "Same.",
    "479266": "Congrats @raddar and team. Thanks Raddar for your insights on this data. It really helped me in figuring out how to create more features.\n\nHere is my post on feature selection techniques\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052",
    "479460": "Congratulations, and special thanks to your inspring and genius kenel/post.",
    "479129": "These are such anxious moments. In the Quora Challenge we had two kernels I thought would do well. The first one got us 22nd. The other wouldve gotten us somewhere in the 200's",
    "479119": "Did it help you eventually, can you confirm ?",
    "479036": "Were you right?",
    "479022": "So how do you choose then ? Do you trust more your CV than your LB ?",
    "478910": "Same here. I will take the risk to use the best LB submission though... The other one will be my best CV with postprocessing....",
    "478833": "Same +1.",
    "478825": "That's gutsy, good luck!\nAfter unsuccessfully trying to build validation schemas for the outlier detection threshold I have decided to go with best CV (3.6371) with two very different numbers of clipped predictions, 34 and 58.\nHow many samples are you clipping in the final subs?",
    "478822": "To Choose is To Renounce...",
    "478794": "probably dice with death for me lol. ",
    "478756": "The same choice for me but the price is much lower",
    "478750": "very tough call. But best LB may be part of submission according to me considering there are quite few outliers in test set.",
    "478821": ""
  }
}