First of all, thanks to my teammates @applehph, @fergusoci, @m7catsue, and @mahluo for all of their hard work over the past month. We had high hopes finishing the competition in gold contention with our public LB position of 11th place, however it didn't work out that way in the end. We still ended it with a silver medal, so I'm still very proud of how our team performed!
\n\nLike all of these competitions, you come away with something new to remember and apply to future competitions. For our team, I feel there are two distinct lessons that we learned:
\n\n1) Never underestimate ensembling or stacking diverse models\nOur final two submissions we chose were single LGB models. We made that decision mostly because we never saw good effects in our CV and LB when ensembling a variety of models (or even multiple different LGB models). However, the ensemble solutions we didn't pick actually turned out to perform best on the private dataset. This points to an important truth of simply relying on good practices. And ensembling is a good practice.
\n\n2) A correlated CV and LB is crucial before any more work should be done\nThat leads me to point number 2. It is very difficult to make good decisions on features and best model parameters if you don't have a correlated LB and CV. We were unable to make a good decision on ensembling because we had trouble in this area. We don't need them to be exactly mirror images of each other, but when CV goes up we should expect to see LB go up. And if they don't, then figure it out...or rely on CV!
\n\nAnyways, we learned a lot and I'm excited to bring this knowledge into the next competition. Thank you Kaggle, congrats to everyone, and job well done!
", "comments": [ { "id": 728174, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2020-01-24T13:37:53.633000", "content": "Hmmm... It's really interesting to read about negative experience too. Thanks!
\n\nSometimes we can't reach the strong correlation between CV and LB even with a great effort. We got an evidences from probing (https://www.kaggle.com/c/data-science-bowl-2019/discussion/126163) that public and private have much more in common in comparision with train-public correlation. So the best CV model is not always the best overall model. Sometimes the best model is one that generalizes better on new (and slightly different) data. Stability and independence from random factors such as rearrangement of columns, change of seed is also very important. Instability of public score is a sign that the model has learned something train-specific or random maybe.
\n\nI'm sorry, it's just my thoughts, I admit that I could be completely wrong.
", "votes": 3, "replies": [ { "id": 728189, "author_name": "TJ Klein", "author_url": "", "post_date": "2020-01-24T13:46:10.817000", "content": "Thanks for your thoughts @sergeifironov. I think you hit on a key point: Generalization is the key and is not always indicated by best CV. One other thing that causes models to not be robust enough to generalize well is having too many features. We tried to reduce our feature set, but had a hard time maintaining a good CV by doing it. In hindsight, it would have been better to lean into feature reduction and not be scared off by a dropping CV.
", "votes": 0, "replies": [] } ] }, { "id": 735007, "author_name": "NorwayPing", "author_url": "", "post_date": "2020-02-02T11:06:54.337000", "content": "I think the most important failure was we didn't spend a good amount of time identifying the validation scheme. Instead we followed the intuition to trust public test set at the end. \nIt would be interesting to learn how others deal with it and how we can avoid the same pitfalls.
", "votes": 1, "replies": [] }, { "id": 727743, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-24T02:16:41.623000", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @teejmahal20
", "votes": 0, "replies": [] }, { "id": 728144, "author_name": "Miyabon", "author_url": "", "post_date": "2020-01-24T12:57:50.487000", "content": "Thanks for sharing yout insights😄 👍
", "votes": 1, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 727595, "postDate": "2020-01-23T21:16:03.187Z", "content": "First of all, thanks to my teammates @applehph, @fergusoci, @m7catsue, and @mahluo for all of their hard work over the past month. We had high hopes finishing the competition in gold contention with our public LB position of 11th place, however it didn't work out that way in the end. We still ended it with a silver medal, so I'm still very proud of how our team performed!
\n\nLike all of these competitions, you come away with something new to remember and apply to future competitions. For our team, I feel there are two distinct lessons that we learned:
\n\n1) Never underestimate ensembling or stacking diverse models\nOur final two submissions we chose were single LGB models. We made that decision mostly because we never saw good effects in our CV and LB when ensembling a variety of models (or even multiple different LGB models). However, the ensemble solutions we didn't pick actually turned out to perform best on the private dataset. This points to an important truth of simply relying on good practices. And ensembling is a good practice.
\n\n2) A correlated CV and LB is crucial before any more work should be done\nThat leads me to point number 2. It is very difficult to make good decisions on features and best model parameters if you don't have a correlated LB and CV. We were unable to make a good decision on ensembling because we had trouble in this area. We don't need them to be exactly mirror images of each other, but when CV goes up we should expect to see LB go up. And if they don't, then figure it out...or rely on CV!
\n\nAnyways, we learned a lot and I'm excited to bring this knowledge into the next competition. Thank you Kaggle, congrats to everyone, and job well done!
", "rawMarkdown": "First of all, thanks to my teammates @applehph, @fergusoci, @m7catsue, and @mahluo for all of their hard work over the past month. We had high hopes finishing the competition in gold contention with our public LB position of 11th place, however it didn't work out that way in the end. We still ended it with a silver medal, so I'm still very proud of how our team performed!\n\nLike all of these competitions, you come away with something new to remember and apply to future competitions. For our team, I feel there are two distinct lessons that we learned: \n\n**1) Never underestimate ensembling or stacking diverse models**\nOur final two submissions we chose were single LGB models. We made that decision mostly because we never saw good effects in our CV and LB when ensembling a variety of models (or even multiple different LGB models). However, the ensemble solutions we didn't pick actually turned out to perform best on the private dataset. This points to an important truth of simply relying on good practices. And ensembling is a good practice.\n\n**2) A correlated CV and LB is crucial before any more work should be done**\nThat leads me to point number 2. It is very difficult to make good decisions on features and best model parameters if you don't have a correlated LB and CV. We were unable to make a good decision on ensembling because we had trouble in this area. We don't need them to be exactly mirror images of each other, but when CV goes up we should expect to see LB go up. And if they don't, then figure it out...or rely on CV! \n\nAnyways, we learned a lot and I'm excited to bring this knowledge into the next competition. Thank you Kaggle, congrats to everyone, and job well done!", "votes": 27 }, { "id": 728174, "postDate": "2020-01-24T13:37:53.633Z", "content": "Hmmm... It's really interesting to read about negative experience too. Thanks!
\n\nSometimes we can't reach the strong correlation between CV and LB even with a great effort. We got an evidences from probing (https://www.kaggle.com/c/data-science-bowl-2019/discussion/126163) that public and private have much more in common in comparision with train-public correlation. So the best CV model is not always the best overall model. Sometimes the best model is one that generalizes better on new (and slightly different) data. Stability and independence from random factors such as rearrangement of columns, change of seed is also very important. Instability of public score is a sign that the model has learned something train-specific or random maybe.
\n\nI'm sorry, it's just my thoughts, I admit that I could be completely wrong.
", "rawMarkdown": "Hmmm... It's really interesting to read about negative experience too. Thanks!\n\nSometimes we can't reach the strong correlation between CV and LB even with a great effort. We got an evidences from probing (https://www.kaggle.com/c/data-science-bowl-2019/discussion/126163) that public and private have much more in common in comparision with train-public correlation. So the best CV model is not always the best overall model. Sometimes the best model is one that generalizes better on new (and slightly different) data. Stability and independence from random factors such as rearrangement of columns, change of seed is also very important. Instability of public score is a sign that the model has learned something train-specific or random maybe.\n\nI'm sorry, it's just my thoughts, I admit that I could be completely wrong.", "votes": 3, "replies": [ { "id": 728189, "postDate": "2020-01-24T13:46:10.817Z", "content": "Thanks for your thoughts @sergeifironov. I think you hit on a key point: Generalization is the key and is not always indicated by best CV. One other thing that causes models to not be robust enough to generalize well is having too many features. We tried to reduce our feature set, but had a hard time maintaining a good CV by doing it. In hindsight, it would have been better to lean into feature reduction and not be scared off by a dropping CV.
", "rawMarkdown": "Thanks for your thoughts @sergeifironov. I think you hit on a key point: Generalization is the key and is not always indicated by best CV. One other thing that causes models to not be robust enough to generalize well is having too many features. We tried to reduce our feature set, but had a hard time maintaining a good CV by doing it. In hindsight, it would have been better to lean into feature reduction and not be scared off by a dropping CV." } ] }, { "id": 735007, "postDate": "2020-02-02T11:06:54.337Z", "content": "I think the most important failure was we didn't spend a good amount of time identifying the validation scheme. Instead we followed the intuition to trust public test set at the end. \nIt would be interesting to learn how others deal with it and how we can avoid the same pitfalls.
", "rawMarkdown": "I think the most important failure was we didn't spend a good amount of time identifying the validation scheme. Instead we followed the intuition to trust public test set at the end. \nIt would be interesting to learn how others deal with it and how we can avoid the same pitfalls. ", "votes": 1 }, { "id": 727743, "postDate": "2020-01-24T02:16:41.623Z", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @teejmahal20
", "rawMarkdown": "Congratulations\nThanks for Sharing your Approach & Insights!! @teejmahal20 " }, { "id": 728144, "postDate": "2020-01-24T12:57:50.487Z", "content": "Thanks for sharing yout insights😄 👍
", "rawMarkdown": "Thanks for sharing yout insights😄 👍 ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T06:54:10Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3855598, "solution_url": "/competitions/data-science-bowl-2019/writeups/team-cv-lessons-learned-from-a-95-position-drop-pu" }