My best stacking model was 3.687, but with different versions of kernel submitted to the competition, generated the Private LB Score of 3.611, 3.612 and 3.613.
\n\nHow is this possible? Also I had to select my best submission so I had selected 3.687 Public LB that generated a Private LB of 3.613, and had I selected the one with Private LB 3.611, I would have made it in top 5% (Silver Medal)
\n\nMy question is, how should i select my best scores in Public LB, if the score of Public LB generates three different private LB's
", "messages": [ { "id": "479913", "postDate": "02/27/2019 15:09:47", "content": "My best stacking model was 3.687, but with different versions of kernel submitted to the competition, generated the Private LB Score of 3.611, 3.612 and 3.613.
\n\nHow is this possible? Also I had to select my best submission so I had selected 3.687 Public LB that generated a Private LB of 3.613, and had I selected the one with Private LB 3.611, I would have made it in top 5% (Silver Medal)
\n\nMy question is, how should i select my best scores in Public LB, if the score of Public LB generates three different private LB's
", "rawMarkdown": "My best stacking model was 3.687, but with different versions of kernel submitted to the competition, generated the Private LB Score of 3.611, 3.612 and 3.613.\n\nHow is this possible? Also I had to select my best submission so I had selected 3.687 Public LB that generated a Private LB of 3.613, and had I selected the one with Private LB 3.611, I would have made it in top 5% (Silver Medal)\n\nMy question is, how should i select my best scores in Public LB, if the score of Public LB generates three different private LB's", "votes": null }, { "id": "479962", "postDate": "02/27/2019 16:13:37", "content": "Choosing one of the final submissions is always a tricky part of the process, and in the end there is no so full proof method.
\n\nI am not sure of what you mean by different versions getting different results - was that not to be expected?\nIf you mean the exact same Kernel, just re-running it, it might have been due to not setting random seeds at the beginning?
\n\nIn the end, you would want to pick whatever models you think are least likely to have over-fitted to the public LB - a constant watch and questioning over if what you're doing might be overfiting is the best solution to this, along with stacking different models with different assumptions.
\n\nHere are some tips from this article based on what I said:
\n\nTrust your CV score, and not LB score. The leaderboard score is scored only on a small percentage of the full test set. In some cases, it’s only a few hundred test cases. Your cross-validation score will be much more reliable in general.
\n\nFor the final 2 models, pick very different models. Picking two very similar solutions means that your solutions either fail together or win together, effectively meaning that you only pick one model. You should reduce your risk by picking two confident but very different models. You should not depend on the leaderboard score at all.
", "rawMarkdown": "Choosing one of the final submissions is always a tricky part of the process, and in the end there is no so full proof method.\n\nI am not sure of what you mean by different versions getting different results - was that not to be expected?\nIf you mean the exact same Kernel, just re-running it, it might have been due to not setting random seeds at the beginning?\n\nIn the end, you would want to pick whatever models you think are least likely to have over-fitted to the public LB - a constant watch and questioning over if what you're doing might be overfiting is the best solution to this, along with stacking different models with different assumptions. \n\nHere are some tips from [this article][1] based on what I said:\n\nTrust your CV score, and not LB score. The leaderboard score is scored only on a small percentage of the full test set. In some cases, it’s only a few hundred test cases. Your cross-validation score will be much more reliable in general.\n\nFor the final 2 models, pick very different models. Picking two very similar solutions means that your solutions either fail together or win together, effectively meaning that you only pick one model. You should reduce your risk by picking two confident but very different models. You should not depend on the leaderboard score at all.\n\n\n [1]: http://www.chioka.in/how-to-select-your-final-models-in-a-kaggle-competitio/", "votes": null }, { "id": "479965", "postDate": "02/27/2019 16:20:09", "content": "what i meant with different versions of the kernel is that i made changes in feature selection techniques in my kernel. and republished it multiple times with multiple outputs.
", "rawMarkdown": "what i meant with different versions of the kernel is that i made changes in feature selection techniques in my kernel. and republished it multiple times with multiple outputs.", "votes": null }, { "id": "479967", "postDate": "02/27/2019 16:20:59", "content": "Thats a great suggestion, I will keep this in mind for the future
", "rawMarkdown": "Thats a great suggestion, I will keep this in mind for the future", "votes": null }, { "id": "479970", "postDate": "02/27/2019 16:22:48", "content": "Feeling bad on missing out the silver medal, but this is a lesson learnt.
", "rawMarkdown": "Feeling bad on missing out the silver medal, but this is a lesson learnt.", "votes": null }, { "id": "479972", "postDate": "02/27/2019 16:25:17", "content": "Well, different techniques would give you different results, I would be surprised if this hadn't happened!\nThey will each run for different times and make different assumptions, so they aren't guaranteed to converge on the set of best features.
\n\nI think everyone goes through this lesson once on Kaggle, better to learn :)
", "rawMarkdown": "Well, different techniques would give you different results, I would be surprised if this hadn't happened!\nThey will each run for different times and make different assumptions, so they aren't guaranteed to converge on the set of best features.\n\nI think everyone goes through this lesson once on Kaggle, better to learn :)", "votes": null }, { "id": "479980", "postDate": "02/27/2019 16:36:50", "content": "It would be easier if you obtain the train RMSE (CV) of your blended/stacked model. And then pick the blending with the best CV.
", "rawMarkdown": "It would be easier if you obtain the train RMSE (CV) of your blended/stacked model. And then pick the blending with the best CV.", "votes": null }, { "id": "479997", "postDate": "02/27/2019 16:58:43", "content": "Thanks for the suggestion. I was using the stacking approach, and I am happy that i didn't include the public blending kernels in my final score. They have a bad Private LB Score
\n\nPrivate LB with 3.613\nhttps://www.kaggle.com/saurabh502/why-no-simple-blend-plb-3-687?scriptVersionId=10030814
Private LB with 3.614\nhttps://www.kaggle.com/ashishpatel26/lb-3-687-truncated-mean
Private LB with 3.614\nhttps://www.kaggle.com/vanshjatana/simple-blend-3-683
So should one avoid using Blending approach for the Final Selection?
", "rawMarkdown": "Thanks for the suggestion. I was using the stacking approach, and I am happy that i didn't include the public blending kernels in my final score. They have a bad Private LB Score\n\n1. Private LB with 3.613\nhttps://www.kaggle.com/saurabh502/why-no-simple-blend-plb-3-687?scriptVersionId=10030814\n\n2. Private LB with 3.614\nhttps://www.kaggle.com/ashishpatel26/lb-3-687-truncated-mean\n\n3. Private LB with 3.614\nhttps://www.kaggle.com/vanshjatana/simple-blend-3-683\n\nSo should one avoid using Blending approach for the Final Selection?", "votes": null }, { "id": "480009", "postDate": "02/27/2019 17:13:07", "content": "Well, that's what machine learning is about: build (train) models that perform well on unseen data. Private LB score is close to that, except that you can see the data (but not the target).
\n\nBeing able to select models that will yield best LB score is the best you can learn on Kaggle. it is what will make you a useful datascientist in real world.
", "rawMarkdown": "Well, that's what machine learning is about: build (train) models that perform well on unseen data. Private LB score is close to that, except that you can see the data (but not the target).\n\nBeing able to select models that will yield best LB score is the best you can learn on Kaggle. it is what will make you a useful datascientist in real world.", "votes": null } ], "comments": [ { "id": 479962, "author_name": "brunogc", "author_url": "", "post_date": "02/27/2019 16:13:37", "content": "Choosing one of the final submissions is always a tricky part of the process, and in the end there is no so full proof method.
\n\nI am not sure of what you mean by different versions getting different results - was that not to be expected?\nIf you mean the exact same Kernel, just re-running it, it might have been due to not setting random seeds at the beginning?
\n\nIn the end, you would want to pick whatever models you think are least likely to have over-fitted to the public LB - a constant watch and questioning over if what you're doing might be overfiting is the best solution to this, along with stacking different models with different assumptions.
\n\nHere are some tips from this article based on what I said:
\n\nTrust your CV score, and not LB score. The leaderboard score is scored only on a small percentage of the full test set. In some cases, it’s only a few hundred test cases. Your cross-validation score will be much more reliable in general.
\n\nFor the final 2 models, pick very different models. Picking two very similar solutions means that your solutions either fail together or win together, effectively meaning that you only pick one model. You should reduce your risk by picking two confident but very different models. You should not depend on the leaderboard score at all.
", "votes": null, "replies": [ { "id": 479965, "author_name": "roydatascience", "author_url": "", "post_date": "02/27/2019 16:20:09", "content": "what i meant with different versions of the kernel is that i made changes in feature selection techniques in my kernel. and republished it multiple times with multiple outputs.
", "votes": null, "replies": [] }, { "id": 479967, "author_name": "roydatascience", "author_url": "", "post_date": "02/27/2019 16:20:59", "content": "Thats a great suggestion, I will keep this in mind for the future
", "votes": null, "replies": [] }, { "id": 479970, "author_name": "roydatascience", "author_url": "", "post_date": "02/27/2019 16:22:48", "content": "Feeling bad on missing out the silver medal, but this is a lesson learnt.
", "votes": null, "replies": [] }, { "id": 479972, "author_name": "brunogc", "author_url": "", "post_date": "02/27/2019 16:25:17", "content": "Well, different techniques would give you different results, I would be surprised if this hadn't happened!\nThey will each run for different times and make different assumptions, so they aren't guaranteed to converge on the set of best features.
\n\nI think everyone goes through this lesson once on Kaggle, better to learn :)
", "votes": null, "replies": [] } ] }, { "id": 479980, "author_name": "kingychiu", "author_url": "", "post_date": "02/27/2019 16:36:50", "content": "It would be easier if you obtain the train RMSE (CV) of your blended/stacked model. And then pick the blending with the best CV.
", "votes": null, "replies": [ { "id": 479997, "author_name": "roydatascience", "author_url": "", "post_date": "02/27/2019 16:58:43", "content": "Thanks for the suggestion. I was using the stacking approach, and I am happy that i didn't include the public blending kernels in my final score. They have a bad Private LB Score
\n\nPrivate LB with 3.613\nhttps://www.kaggle.com/saurabh502/why-no-simple-blend-plb-3-687?scriptVersionId=10030814
Private LB with 3.614\nhttps://www.kaggle.com/ashishpatel26/lb-3-687-truncated-mean
Private LB with 3.614\nhttps://www.kaggle.com/vanshjatana/simple-blend-3-683
So should one avoid using Blending approach for the Final Selection?
", "votes": null, "replies": [] } ] }, { "id": 480009, "author_name": "cpmpml", "author_url": "", "post_date": "02/27/2019 17:13:07", "content": "Well, that's what machine learning is about: build (train) models that perform well on unseen data. Private LB score is close to that, except that you can see the data (but not the target).
\n\nBeing able to select models that will yield best LB score is the best you can learn on Kaggle. it is what will make you a useful datascientist in real world.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "479913": "My best stacking model was 3.687, but with different versions of kernel submitted to the competition, generated the Private LB Score of 3.611, 3.612 and 3.613.\n\nHow is this possible? Also I had to select my best submission so I had selected 3.687 Public LB that generated a Private LB of 3.613, and had I selected the one with Private LB 3.611, I would have made it in top 5% (Silver Medal)\n\nMy question is, how should i select my best scores in Public LB, if the score of Public LB generates three different private LB's", "479962": "Choosing one of the final submissions is always a tricky part of the process, and in the end there is no so full proof method.\n\nI am not sure of what you mean by different versions getting different results - was that not to be expected?\nIf you mean the exact same Kernel, just re-running it, it might have been due to not setting random seeds at the beginning?\n\nIn the end, you would want to pick whatever models you think are least likely to have over-fitted to the public LB - a constant watch and questioning over if what you're doing might be overfiting is the best solution to this, along with stacking different models with different assumptions. \n\nHere are some tips from [this article][1] based on what I said:\n\nTrust your CV score, and not LB score. The leaderboard score is scored only on a small percentage of the full test set. In some cases, it’s only a few hundred test cases. Your cross-validation score will be much more reliable in general.\n\nFor the final 2 models, pick very different models. Picking two very similar solutions means that your solutions either fail together or win together, effectively meaning that you only pick one model. You should reduce your risk by picking two confident but very different models. You should not depend on the leaderboard score at all.\n\n\n [1]: http://www.chioka.in/how-to-select-your-final-models-in-a-kaggle-competitio/", "479965": "what i meant with different versions of the kernel is that i made changes in feature selection techniques in my kernel. and republished it multiple times with multiple outputs.", "479967": "Thats a great suggestion, I will keep this in mind for the future", "479970": "Feeling bad on missing out the silver medal, but this is a lesson learnt.", "479972": "Well, different techniques would give you different results, I would be surprised if this hadn't happened!\nThey will each run for different times and make different assumptions, so they aren't guaranteed to converge on the set of best features.\n\nI think everyone goes through this lesson once on Kaggle, better to learn :)", "479980": "It would be easier if you obtain the train RMSE (CV) of your blended/stacked model. And then pick the blending with the best CV.", "479997": "Thanks for the suggestion. I was using the stacking approach, and I am happy that i didn't include the public blending kernels in my final score. They have a bad Private LB Score\n\n1. Private LB with 3.613\nhttps://www.kaggle.com/saurabh502/why-no-simple-blend-plb-3-687?scriptVersionId=10030814\n\n2. Private LB with 3.614\nhttps://www.kaggle.com/ashishpatel26/lb-3-687-truncated-mean\n\n3. Private LB with 3.614\nhttps://www.kaggle.com/vanshjatana/simple-blend-3-683\n\nSo should one avoid using Blending approach for the Final Selection?", "480009": "Well, that's what machine learning is about: build (train) models that perform well on unseen data. Private LB score is close to that, except that you can see the data (but not the target).\n\nBeing able to select models that will yield best LB score is the best you can learn on Kaggle. it is what will make you a useful datascientist in real world." }, "source": "meta" }