![\"Keep](\"https://i.imgur.com/SbgMlig.png\")
Some people have despaired that this competition may be hopeless, with small changes in seeds moving LB scores by a lot, it being hard to make progress on the LB, and outlier targets being a huge pain.
\n\nI agree that this competition is weird in many ways (what even is the target? loyalty to what?), but I disagree that it is hopeless.
\n\nTrust Your CV
\n\nA lot of people here are a bit freaked out about their Public Leaderboard score... but what if I told you that it doesn't actually matter that much?
\n\nThink about when you make your 5-fold model... you have scores on some folds that are much different than scores on other folds. You judge your model based on the 5-fold average, not one individual fold chosen at random, because you get more data and it smooths out the noise/randomness/luck. Right?
\n\nWell, judging based on the public leaderboard is a lot like judging based on one particular fold instead of the five fold average.
\n\nJust look at the numbers. As we know, the public leaderboard is mostly for show and the real rankings and juicy medals are chosen based on a Private Leaderboard that we don't know our score on until the end of the competition. The Public Leaderboard is only 37,087 rows, whereas the private Leaderboard will score with the remaining 86,536 rows of the test set -- more than twice as much!
\n\nThe train set has 201,917 rows, which means a five-fold CV will be judged on 201,917 rows, and a single fold will be judged on 40,383 records. That means the Public Leaderboard and a single fold are roughly the same size and should have roughly the same noise-level.
\n\nTherefore, we trust our CV.
\n\n
Mind the Gap
\n\nThat being said, there are some times you should defer to the Public Leaderboard... for example, if there are meaningful differences between train and test (such as time stratification, other non-random sampling, or something weird going on). We can know from adversarial validation that there aren't really any particular reasons to be wary of this test set, but we still should be mindful of the LB-CV gap. After all, certain bugs can happen all the time where we break our test set predictions (but not our CV predictions) and do badly.
\n\nThe way to look at this is to try to be roughly in line with the gaps of others, but don't fret too much about whether your gap is precisely the same with every submission.
\n\nTo do this, I made a table tracking a lot of the gaps in LB and CV score for various well-scoring kernels and reports in the \"Best Single Model\" thread:
\n\n| Model | CV | LB | Diff |\n| --- | --- | --- | --- |\n| Chau II | 3.650 | 3.697 | 0.047 |\n| Chau | 3.653 | 3.699 | 0.046 |\n| Grand Rookie | 3.6581 | 3.697 | 0.0389 |\n| Namit | 3.6611 | 3.704 | 0.0429 |\n| Dimitrios II | 3.66377 | 3.708 | 0.04423 |\n| Elo World #25 | 3.66638 | 3.709 | 0.04262 |\n| Dimitrios I | 3.66643 | 3.705 | 0.03857 |\n| Frederick | 3.66740 | 3.706 | 0.03860 |\n| Konrad | 3.6678 | 3.709 | 0.0412 |\n| LGB + FE | 3.6703 | 3.707 | 0.0367 |\n| Tidy LGB | 3.67036 | 3.711 | 0.04064 |\n| Move Up LB | 3.67208 | 3.703 | 0.03092 |\n| Aleksandr | 3.6724 | 3.700 | 0.028 |\n| Elo World #22 | 3.67872 | 3.720 | 0.041 |\n| Elo World #24 | 3.67942 | 3.721 | 0.04158 |\n| Elo World #19 | 3.67944 | 3.719 | 0.03956 |\n| Elo World #20 | 3.67987 | 3.721 | 0.04113 |\n| YGTWMC | 3.68954 | 3.737 | 0.04746 |\n| Elo World #15 | 3.68989 | 3.738 | 0.04811 |
\n\nWhen I do statistics on this, I find the average gap between CV and LB is 0.041 with a SD of 0.005. Based on this, I'm not alarmed as much by differences in my CV-LB gap, though I have been rejecting submissions with a gap larger than 0.05 (as has happened to me a few times) as something being potentially wrong with them.
\n\nIn the end, the best we can do is trust our CV, and maybe use our two submissions to do best CV and best LB, and hope that the Private Leaderboard will sort things out.
\n\nGood luck!
", "messages": [ { "id": "434711", "postDate": "12/06/2018 20:43:08", "content": "Some people have despaired that this competition may be hopeless, with small changes in seeds moving LB scores by a lot, it being hard to make progress on the LB, and outlier targets being a huge pain.
\n\nI agree that this competition is weird in many ways (what even is the target? loyalty to what?), but I disagree that it is hopeless.
\n\nTrust Your CV
\n\nA lot of people here are a bit freaked out about their Public Leaderboard score... but what if I told you that it doesn't actually matter that much?
\n\nThink about when you make your 5-fold model... you have scores on some folds that are much different than scores on other folds. You judge your model based on the 5-fold average, not one individual fold chosen at random, because you get more data and it smooths out the noise/randomness/luck. Right?
\n\nWell, judging based on the public leaderboard is a lot like judging based on one particular fold instead of the five fold average.
\n\nJust look at the numbers. As we know, the public leaderboard is mostly for show and the real rankings and juicy medals are chosen based on a Private Leaderboard that we don't know our score on until the end of the competition. The Public Leaderboard is only 37,087 rows, whereas the private Leaderboard will score with the remaining 86,536 rows of the test set -- more than twice as much!
\n\nThe train set has 201,917 rows, which means a five-fold CV will be judged on 201,917 rows, and a single fold will be judged on 40,383 records. That means the Public Leaderboard and a single fold are roughly the same size and should have roughly the same noise-level.
\n\nTherefore, we trust our CV.
\n\n
Mind the Gap
\n\nThat being said, there are some times you should defer to the Public Leaderboard... for example, if there are meaningful differences between train and test (such as time stratification, other non-random sampling, or something weird going on). We can know from adversarial validation that there aren't really any particular reasons to be wary of this test set, but we still should be mindful of the LB-CV gap. After all, certain bugs can happen all the time where we break our test set predictions (but not our CV predictions) and do badly.
\n\nThe way to look at this is to try to be roughly in line with the gaps of others, but don't fret too much about whether your gap is precisely the same with every submission.
\n\nTo do this, I made a table tracking a lot of the gaps in LB and CV score for various well-scoring kernels and reports in the \"Best Single Model\" thread:
\n\n| Model | CV | LB | Diff |\n| --- | --- | --- | --- |\n| Chau II | 3.650 | 3.697 | 0.047 |\n| Chau | 3.653 | 3.699 | 0.046 |\n| Grand Rookie | 3.6581 | 3.697 | 0.0389 |\n| Namit | 3.6611 | 3.704 | 0.0429 |\n| Dimitrios II | 3.66377 | 3.708 | 0.04423 |\n| Elo World #25 | 3.66638 | 3.709 | 0.04262 |\n| Dimitrios I | 3.66643 | 3.705 | 0.03857 |\n| Frederick | 3.66740 | 3.706 | 0.03860 |\n| Konrad | 3.6678 | 3.709 | 0.0412 |\n| LGB + FE | 3.6703 | 3.707 | 0.0367 |\n| Tidy LGB | 3.67036 | 3.711 | 0.04064 |\n| Move Up LB | 3.67208 | 3.703 | 0.03092 |\n| Aleksandr | 3.6724 | 3.700 | 0.028 |\n| Elo World #22 | 3.67872 | 3.720 | 0.041 |\n| Elo World #24 | 3.67942 | 3.721 | 0.04158 |\n| Elo World #19 | 3.67944 | 3.719 | 0.03956 |\n| Elo World #20 | 3.67987 | 3.721 | 0.04113 |\n| YGTWMC | 3.68954 | 3.737 | 0.04746 |\n| Elo World #15 | 3.68989 | 3.738 | 0.04811 |
\n\nWhen I do statistics on this, I find the average gap between CV and LB is 0.041 with a SD of 0.005. Based on this, I'm not alarmed as much by differences in my CV-LB gap, though I have been rejecting submissions with a gap larger than 0.05 (as has happened to me a few times) as something being potentially wrong with them.
\n\nIn the end, the best we can do is trust our CV, and maybe use our two submissions to do best CV and best LB, and hope that the Private Leaderboard will sort things out.
\n\nGood luck!
", "rawMarkdown": "Some people have despaired that this competition may be hopeless, [with small changes in seeds moving LB scores by a lot](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73887), [it being hard to make progress on the LB](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73497), and [outlier targets being a huge pain](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73425).\n\nI agree that this competition is weird in many ways ([what even is the target? loyalty to what?](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73634)), but I disagree that it is hopeless.\n\n**Trust Your CV**\n\nA lot of people here are a bit freaked out about their Public Leaderboard score... but what if I told you that it doesn't actually matter that much?\n\nThink about when you make your 5-fold model... you have scores on some folds that are much different than scores on other folds. You judge your model based on the 5-fold average, not one individual fold chosen at random, because you get more data and it smooths out the noise/randomness/luck. Right?\n\nWell, judging based on the public leaderboard is a lot like judging based on one particular fold instead of the five fold average.\n\nJust look at the numbers. As we know, the public leaderboard is mostly for show and the real rankings and juicy medals are chosen based on a Private Leaderboard that we don't know our score on until the end of the competition. The Public Leaderboard is only 37,087 rows, whereas the private Leaderboard will score with the remaining 86,536 rows of the test set -- more than twice as much!\n\nThe train set has 201,917 rows, which means a five-fold CV will be judged on 201,917 rows, and a single fold will be judged on 40,383 records. That means the Public Leaderboard and a single fold are roughly the same size and should have roughly the same noise-level.\n\nTherefore, we trust our CV.\n\n![Keep calm and trust your CV][1]\n\n\n**Mind the Gap**\n\nThat being said, there are *some times* you should defer to the Public Leaderboard... for example, if there are meaningful differences between train and test (such as time stratification, other non-random sampling, or something weird going on). We can know from [adversarial validation](https://www.kaggle.com/tunguz/elo-adversarial-validation) that there aren't really any particular reasons to be wary of this test set, but we still should be mindful of the LB-CV gap. After all, certain bugs can happen all the time where we break our test set predictions (but not our CV predictions) and do badly.\n\nThe way to look at this is to try to be roughly in line with the gaps of others, but don't fret too much about whether your gap is precisely the same with every submission.\n\nTo do this, I made a table tracking a lot of the gaps in LB and CV score for various well-scoring kernels and [reports in the \"Best Single Model\" thread](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517):\n\n| **Model** | **CV** | **LB** | **Diff** |\n| --- | --- | --- | --- |\n| [Chau II](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#433374) | 3.650 | 3.697 | 0.047 |\n| [Chau](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73497#432584) | 3.653 | 3.699 | 0.046 |\n| [Grand Rookie](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432508) | 3.6581 | 3.697 | 0.0389 |\n| [Namit](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#434309) | 3.6611 | 3.704 | 0.0429 |\n| [Dimitrios II](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432343) | 3.66377 | 3.708 | 0.04423 |\n| [Elo World #25](https://www.kaggle.com/fabiendaniel/elo-world/notebook) | 3.66638 | 3.709 | 0.04262 |\n| [Dimitrios I](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432343) | 3.66643 | 3.705 | 0.03857 |\n| [Frederick](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#433195) | 3.66740 | 3.706 | 0.03860 |\n| [Konrad](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432711) | 3.6678 | 3.709 | 0.0412 |\n| [LGB + FE](https://www.kaggle.com/peterhurford/lgb-fe-lb-3-707?scriptVersionId=8092015) | 3.6703 | 3.707 | 0.0367 |\n| [Tidy LGB](https://www.kaggle.com/cyones77/tidy-elo-starter-with-lightgbm-and-cv-3-711/output) | 3.67036 | 3.711 | 0.04064 |\n| [Move Up LB](https://www.kaggle.com/akshay00/let-s-move-up-on-the-lb-3-703-please-upvote/notebook?scriptVersionId=8123273) | 3.67208 | 3.703 | 0.03092 |\n| [Aleksandr](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#434439) | 3.6724 | 3.700 | 0.028 |\n| [Elo World #22](https://www.kaggle.com/fabiendaniel/elo-world?scriptVersionId=8048217) | 3.67872 | 3.720 | 0.041 |\n| [Elo World #24](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=8048927) | 3.67942 | 3.721 | 0.04158 |\n| [Elo World #19](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=7992489) | 3.67944 | 3.719 | 0.03956 |\n| [Elo World #20](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=8007845) | 3.67987 | 3.721 | 0.04113 |\n| [YGTWMC](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737/notebook?scriptVersionId=7873481) | 3.68954 | 3.737 | 0.04746 |\n| [Elo World #15](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=7922421) | 3.68989 | 3.738 | 0.04811 |\n\nWhen I do statistics on this, I find the average gap between CV and LB is 0.041 with a SD of 0.005. Based on this, I'm not alarmed as much by differences in my CV-LB gap, though I have been rejecting submissions with a gap larger than 0.05 (as has happened to me a few times) as something being potentially wrong with them.\n\nIn the end, the best we can do is trust our CV, and maybe use our two submissions to do best CV and best LB, and hope that the Private Leaderboard will sort things out.\n\nGood luck!\n\n [1]: https://i.imgur.com/SbgMlig.png", "votes": null }, { "id": "435313", "postDate": "12/07/2018 20:45:35", "content": "good summary of CV/leaderboard score management
", "rawMarkdown": "good summary of CV/leaderboard score management", "votes": null }, { "id": "435351", "postDate": "12/07/2018 22:17:18", "content": "my best score now - CV: 3.6522 and LB: 3.697
", "rawMarkdown": "my best score now - CV: 3.6522 and LB: 3.697", "votes": null }, { "id": "435354", "postDate": "12/07/2018 22:26:42", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "435364", "postDate": "12/07/2018 23:01:18", "content": "@Peter Hurford: \nI guess I am confused here as to why are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different. And to your analogy of Public Leaderboard result being same as result on 1 fold; I feel results based on 37,087 rows is like take small samples of data and running across different folds. Also for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. For example: In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values. \nYour thoughts are very much welcome :)
", "rawMarkdown": "Peter Hurford: \nI guess I am confused here as to why are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different. And to your analogy of Public Leaderboard result being same as result on 1 fold; I feel results based on **37,087** rows is like take small samples of data and running across different folds. Also for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. For example: In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values. \nYour thoughts are very much welcome :)", "votes": null }, { "id": "435381", "postDate": "12/07/2018 23:44:15", "content": "\n\n\nwhy are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different
\n
Random noise... the same reason why each fold of an unstratified KFold are different.
\n\n\n\n\nAlso for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values.
\n
You're correct that I'm making that assumption. I'm not 100% sure, of course, but given that we can predict the outlier with ~0.9 AUC but cannot predict the train/test set >0.5 AUC, I'm not that concerned about the potential for these differences.
", "rawMarkdown": "> why are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different\n\nRandom noise... the same reason why each fold of an unstratified KFold are different.\n\n> Also for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values.\n\nYou're correct that I'm making that assumption. I'm not 100% sure, of course, but given that we can predict the outlier with ~0.9 AUC but cannot predict the train/test set >0.5 AUC, I'm not that concerned about the potential for these differences.", "votes": null }, { "id": "440133", "postDate": "12/17/2018 05:35:27", "content": "What about a gap more than 0.3? I think it is mainly caused by outliers.
", "rawMarkdown": "What about a gap more than 0.3? I think it is mainly caused by outliers.", "votes": null }, { "id": "440188", "postDate": "12/17/2018 07:34:13", "content": "That definitely sounds to me like you're making a mistake in your code somewhere.
", "rawMarkdown": "That definitely sounds to me like you're making a mistake in your code somewhere.", "votes": null }, { "id": "440314", "postDate": "12/17/2018 11:46:37", "content": "My CV:3.6519 and LB:3.688, in this competition, i feel CV result is consistent with LB score sometimes, but sometimes not.
", "rawMarkdown": "My CV:3.6519 and LB:3.688, in this competition, i feel CV result is consistent with LB score sometimes, but sometimes not.", "votes": null }, { "id": "440935", "postDate": "12/18/2018 05:37:05", "content": "But I only add some features about outliers which made my local CV decreased from 3.658 to 3.432，however LB from 3.698 to 3.8.
", "rawMarkdown": "But I only add some features about outliers which made my local CV decreased from 3.658 to 3.432，however LB from 3.698 to 3.8.", "votes": null }, { "id": "440977", "postDate": "12/18/2018 06:24:27", "content": "That definitely sounds like overfitting/leakage to me. Your validation set likely has information about the outliers that it shouldn't have, which makes it unrepresentative of the Public LB. Try just generating whatever outliers you're generating within your 5CV folds.
", "rawMarkdown": "That definitely sounds like overfitting/leakage to me. Your validation set likely has information about the outliers that it shouldn't have, which makes it unrepresentative of the Public LB. Try just generating whatever outliers you're generating within your 5CV folds.", "votes": null }, { "id": "440997", "postDate": "12/18/2018 07:00:18", "content": "Thank! I will try
", "rawMarkdown": "Thank! I will try", "votes": null }, { "id": "443887", "postDate": "12/22/2018 16:58:10", "content": "Thanks !
", "rawMarkdown": "Thanks !", "votes": null } ], "comments": [ { "id": 435313, "author_name": "hampsonw", "author_url": "", "post_date": "12/07/2018 20:45:35", "content": "good summary of CV/leaderboard score management
", "votes": null, "replies": [] }, { "id": 435351, "author_name": "jingobox", "author_url": "", "post_date": "12/07/2018 22:17:18", "content": "my best score now - CV: 3.6522 and LB: 3.697
", "votes": null, "replies": [] }, { "id": 435354, "author_name": "adhish1", "author_url": "", "post_date": "12/07/2018 22:26:42", "content": "Thanks!
", "votes": null, "replies": [] }, { "id": 435364, "author_name": "akshay00", "author_url": "", "post_date": "12/07/2018 23:01:18", "content": "@Peter Hurford: \nI guess I am confused here as to why are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different. And to your analogy of Public Leaderboard result being same as result on 1 fold; I feel results based on 37,087 rows is like take small samples of data and running across different folds. Also for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. For example: In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values. \nYour thoughts are very much welcome :)
", "votes": null, "replies": [ { "id": 435381, "author_name": "peterhurford", "author_url": "", "post_date": "12/07/2018 23:44:15", "content": "\n\n\nwhy are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different
\n
Random noise... the same reason why each fold of an unstratified KFold are different.
\n\n\n\n\nAlso for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values.
\n
You're correct that I'm making that assumption. I'm not 100% sure, of course, but given that we can predict the outlier with ~0.9 AUC but cannot predict the train/test set >0.5 AUC, I'm not that concerned about the potential for these differences.
", "votes": null, "replies": [] } ] }, { "id": 440133, "author_name": "txfinbuaa", "author_url": "", "post_date": "12/17/2018 05:35:27", "content": "What about a gap more than 0.3? I think it is mainly caused by outliers.
", "votes": null, "replies": [ { "id": 440188, "author_name": "peterhurford", "author_url": "", "post_date": "12/17/2018 07:34:13", "content": "That definitely sounds to me like you're making a mistake in your code somewhere.
", "votes": null, "replies": [] }, { "id": 440935, "author_name": "txfinbuaa", "author_url": "", "post_date": "12/18/2018 05:37:05", "content": "But I only add some features about outliers which made my local CV decreased from 3.658 to 3.432，however LB from 3.698 to 3.8.
", "votes": null, "replies": [] }, { "id": 440977, "author_name": "peterhurford", "author_url": "", "post_date": "12/18/2018 06:24:27", "content": "That definitely sounds like overfitting/leakage to me. Your validation set likely has information about the outliers that it shouldn't have, which makes it unrepresentative of the Public LB. Try just generating whatever outliers you're generating within your 5CV folds.
", "votes": null, "replies": [] }, { "id": 440997, "author_name": "txfinbuaa", "author_url": "", "post_date": "12/18/2018 07:00:18", "content": "Thank! I will try
", "votes": null, "replies": [] } ] }, { "id": 440314, "author_name": "codlife", "author_url": "", "post_date": "12/17/2018 11:46:37", "content": "My CV:3.6519 and LB:3.688, in this competition, i feel CV result is consistent with LB score sometimes, but sometimes not.
", "votes": null, "replies": [] }, { "id": 443887, "author_name": "kartikey21", "author_url": "", "post_date": "12/22/2018 16:58:10", "content": "Thanks !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "434711": "Some people have despaired that this competition may be hopeless, [with small changes in seeds moving LB scores by a lot](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73887), [it being hard to make progress on the LB](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73497), and [outlier targets being a huge pain](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73425).\n\nI agree that this competition is weird in many ways ([what even is the target? loyalty to what?](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73634)), but I disagree that it is hopeless.\n\n**Trust Your CV**\n\nA lot of people here are a bit freaked out about their Public Leaderboard score... but what if I told you that it doesn't actually matter that much?\n\nThink about when you make your 5-fold model... you have scores on some folds that are much different than scores on other folds. You judge your model based on the 5-fold average, not one individual fold chosen at random, because you get more data and it smooths out the noise/randomness/luck. Right?\n\nWell, judging based on the public leaderboard is a lot like judging based on one particular fold instead of the five fold average.\n\nJust look at the numbers. As we know, the public leaderboard is mostly for show and the real rankings and juicy medals are chosen based on a Private Leaderboard that we don't know our score on until the end of the competition. The Public Leaderboard is only 37,087 rows, whereas the private Leaderboard will score with the remaining 86,536 rows of the test set -- more than twice as much!\n\nThe train set has 201,917 rows, which means a five-fold CV will be judged on 201,917 rows, and a single fold will be judged on 40,383 records. That means the Public Leaderboard and a single fold are roughly the same size and should have roughly the same noise-level.\n\nTherefore, we trust our CV.\n\n![Keep calm and trust your CV][1]\n\n\n**Mind the Gap**\n\nThat being said, there are *some times* you should defer to the Public Leaderboard... for example, if there are meaningful differences between train and test (such as time stratification, other non-random sampling, or something weird going on). We can know from [adversarial validation](https://www.kaggle.com/tunguz/elo-adversarial-validation) that there aren't really any particular reasons to be wary of this test set, but we still should be mindful of the LB-CV gap. After all, certain bugs can happen all the time where we break our test set predictions (but not our CV predictions) and do badly.\n\nThe way to look at this is to try to be roughly in line with the gaps of others, but don't fret too much about whether your gap is precisely the same with every submission.\n\nTo do this, I made a table tracking a lot of the gaps in LB and CV score for various well-scoring kernels and [reports in the \"Best Single Model\" thread](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517):\n\n| **Model** | **CV** | **LB** | **Diff** |\n| --- | --- | --- | --- |\n| [Chau II](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#433374) | 3.650 | 3.697 | 0.047 |\n| [Chau](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73497#432584) | 3.653 | 3.699 | 0.046 |\n| [Grand Rookie](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432508) | 3.6581 | 3.697 | 0.0389 |\n| [Namit](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#434309) | 3.6611 | 3.704 | 0.0429 |\n| [Dimitrios II](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432343) | 3.66377 | 3.708 | 0.04423 |\n| [Elo World #25](https://www.kaggle.com/fabiendaniel/elo-world/notebook) | 3.66638 | 3.709 | 0.04262 |\n| [Dimitrios I](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432343) | 3.66643 | 3.705 | 0.03857 |\n| [Frederick](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#433195) | 3.66740 | 3.706 | 0.03860 |\n| [Konrad](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#432711) | 3.6678 | 3.709 | 0.0412 |\n| [LGB + FE](https://www.kaggle.com/peterhurford/lgb-fe-lb-3-707?scriptVersionId=8092015) | 3.6703 | 3.707 | 0.0367 |\n| [Tidy LGB](https://www.kaggle.com/cyones77/tidy-elo-starter-with-lightgbm-and-cv-3-711/output) | 3.67036 | 3.711 | 0.04064 |\n| [Move Up LB](https://www.kaggle.com/akshay00/let-s-move-up-on-the-lb-3-703-please-upvote/notebook?scriptVersionId=8123273) | 3.67208 | 3.703 | 0.03092 |\n| [Aleksandr](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73517#434439) | 3.6724 | 3.700 | 0.028 |\n| [Elo World #22](https://www.kaggle.com/fabiendaniel/elo-world?scriptVersionId=8048217) | 3.67872 | 3.720 | 0.041 |\n| [Elo World #24](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=8048927) | 3.67942 | 3.721 | 0.04158 |\n| [Elo World #19](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=7992489) | 3.67944 | 3.719 | 0.03956 |\n| [Elo World #20](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=8007845) | 3.67987 | 3.721 | 0.04113 |\n| [YGTWMC](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737/notebook?scriptVersionId=7873481) | 3.68954 | 3.737 | 0.04746 |\n| [Elo World #15](https://www.kaggle.com/fabiendaniel/elo-world/notebook?scriptVersionId=7922421) | 3.68989 | 3.738 | 0.04811 |\n\nWhen I do statistics on this, I find the average gap between CV and LB is 0.041 with a SD of 0.005. Based on this, I'm not alarmed as much by differences in my CV-LB gap, though I have been rejecting submissions with a gap larger than 0.05 (as has happened to me a few times) as something being potentially wrong with them.\n\nIn the end, the best we can do is trust our CV, and maybe use our two submissions to do best CV and best LB, and hope that the Private Leaderboard will sort things out.\n\nGood luck!\n\n [1]: https://i.imgur.com/SbgMlig.png", "435313": "good summary of CV/leaderboard score management", "435351": "my best score now - CV: 3.6522 and LB: 3.697", "435354": "Thanks!", "435364": "Peter Hurford: \nI guess I am confused here as to why are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different. And to your analogy of Public Leaderboard result being same as result on 1 fold; I feel results based on **37,087** rows is like take small samples of data and running across different folds. Also for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. For example: In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values. \nYour thoughts are very much welcome :)", "435381": "> why are you assuming that the distribution of Public Leaderboard and Private LeaderBoard going to be different\n\nRandom noise... the same reason why each fold of an unstratified KFold are different.\n\n> Also for your CV/LB assumption, you are making a big assumption that distribution in test data and train data is exactly the same. In your training set you have almost 2000 outlier values, what if we actually have 3000-4000 outlier values in the testing dataset and this is based on one of the assumption that our score is getting affected a-lot by the outlier values.\n\nYou're correct that I'm making that assumption. I'm not 100% sure, of course, but given that we can predict the outlier with ~0.9 AUC but cannot predict the train/test set >0.5 AUC, I'm not that concerned about the potential for these differences.", "440133": "What about a gap more than 0.3? I think it is mainly caused by outliers.", "440188": "That definitely sounds to me like you're making a mistake in your code somewhere.", "440314": "My CV:3.6519 and LB:3.688, in this competition, i feel CV result is consistent with LB score sometimes, but sometimes not.", "440935": "But I only add some features about outliers which made my local CV decreased from 3.658 to 3.432，however LB from 3.698 to 3.8.", "440977": "That definitely sounds like overfitting/leakage to me. Your validation set likely has information about the outliers that it shouldn't have, which makes it unrepresentative of the Public LB. Try just generating whatever outliers you're generating within your 5CV folds.", "440997": "Thank! I will try", "443887": "Thanks !" }, "source": "meta" }