A journey to 4th place.
\n\nAt first I would like to thank you all for I’ve learned a lot! I guess I’m standing now where Zach stood in the first Don’t Overfit contest. This was my first competition, and first serious contact with machine learning at all.
\n\nUnfortunately, I couldn’t rely on any metrics. I didn’t manage to get a reliable CV scoring throughout the competition. The only way to know if I was doing any better, was to check a Public LB score, so…\nI figured I needed another ranking system, that will indicate me whether I’m going off or not. As Public LB scores was all I could rely on, I developed a simple tool that:\n * loads all previous models I submitted together with Public LB score,\n * loads a new model that I want predict a score for,\n * performs rank and MinMaxScale on all of those models \n * calculates mse, mae, cos, dist, r2 and cor between a new model and all previous models I’ve submitted, searching for submitted model that was ‚closest’ to the one I’m predicting the score for.
\n\nThis ranker improved over time, as I added more and more submissions to its database. \nWhile not very complex, it allowed me to reject non-promising ideas at an early stage, without losing tons of submissions.
\n\nI believe many of you have already seen the kernel Robust, Lasso, Patches with RFE & GS I will refer to it as a LassoKernel from now on. I originally posted it with heuristics yielding 0.868 on Public LB. The heuristics were carefully crafted, as touching any of the parameters would result in a significant drop of Public LB score. How did I come to that? When working on that kernel, at first I did not use any random seed and there was one lucky shot of 0.867 LB that I struggled to reproduce. I fed that one lucky shot into my ranking system described above and started a loop with different random seeds in search of a model as close to 0.867 lucky submission as I could get. That is how it’s gotten to this specific random seed that yields 0.868. This was my base model. Mind that it did not take advantage of any LB probing, as training was done on training samples only, but the process of developing it was strongly dependent on Public LB scores of all my previous submissions, so there definitely is a data leak.
\n\nReading discussions and other kernels, I figured that many people favoured LogisticRegression. So I replaced kernels’ Lasso with LogisticRegression and did many experiments to get the best score I could, hoping to beat a result I got from Lasso. Results can be seen in my second kernel Robust, Patches with RFE & GS and LogReg (LogregKernel).
\n\nThe next step was to stack those models. Simple mean of submissions from the two above kernels yields a 0.872 Public LB (and mean rank 0.873 if I recall correctly). Each of the kernels is a mean of multiple models as well, so I decided to sacrifice a bunch of submissions to get the LB score of different components and some combinations of means or mean ranks of them. That’s how I got to know that eliminating last component of my Lasso kernel yielded 0.872 LB alone, and averaged with several components from my Logreg kernel, yielded a 0.875 (mean rank of 0.876)
\n\nLassoKernel = 0.872\nLogregKernel = 0.868\nBestSoFar = meanRank((LassoKernel, LogregKernel*), weights=[2,1]) = 0.876
\n\n*Precisely it was rank of some components from LogregKernel, but I have a difficulty retrospecting which ones, sorry!
\n\nThen there came a few weeks of stagnation, until @nightwolfbrooks kernel Hyper parameter tuned SVM model (NightwolfsSVMKernel ) came out. I think this kernel deserves much more attention than it seems to have gotten. It is very well crafted, and playing around with it I couldn’t improve it’s score, so I took the submission as is. It turned out that it introduced enough diversity into my BestSoFar, to get to the 0.881 Public LB.
\n\nNightwolfsSVMKernel = 0.861\nTotalBest = meanRank((BestSoFar, NightwolfsSVMKernel), weights=[2,1]) = 0.881
\n\nAs I mentioned above, I couldn’t rely on CV at all, so both GridSearchCV and RFECV in my Kernels were not reliable, and the overall performance of my kernels is a combination of pure chance and my determination to advance on Public LB. \nAs to the noise introduced in my Lasso kernel - it only affects RFECV in some mysterious way, so that it chooses ‚optimal’ feature set. With noise removed (or with different random_state) other features get selected, and the LB score goes down. With noise introduced only to data fed to feature selector, but not to the GridSearchCV the 0.868 LB score still holds.
\n\nI find this a bit disappointing, but I hope you learn at least a bit from my approach.
\n\nThank you all!
", "comments": [ { "id": 529290, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-09T15:43:49.507000", "content": "Congrats feature -blind ! Your kernel is impressive and it is what I started with 20 days ago. I downloaded your submission.csv and extracted the hyperplane directly from your test.csv predictions. For the next week, I randomly rotated your hyperplane 5 degrees and resubmitted (5 times each day). I got your model up to LB 0.875.
Then Nicholas Brooks posted his LB 0.861 SVM kernel, here. The next day, my rank dropped 10 places and 10 new people had LB 0.876 and higher. I suspected the secret was ensembling with Nick's kernel. (Thanks for confirming this). I was about to do that myself, but then I said, I can't keep making small slow updates, I need a more efficient LB-probing scheme.
\n\nThe next day, I brainstormed and devised my current efficient LB-probing scheme here. I was so excited because it gained LB and Private score so quickly in all my simulations using synthetic data.
", "votes": 3, "replies": [ { "id": 529376, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-09T18:49:08.567000", "content": "Thank you! I see you're full of great ideas! I would love to see your code for random rotations of that hyperplane some day :) EDIT: I should read your materials more carefully! I see you've already posted that code. Thanks!
", "votes": 1, "replies": [] }, { "id": 529838, "author_name": "Eric R", "author_url": "", "post_date": "2019-05-10T23:13:20.360000", "content": "I've never ensembled public kernels. Does ensembling two methods with similar performances almost always yield better performance?
", "votes": 0, "replies": [] }, { "id": 529944, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-11T08:03:52.183000", "content": "I've tried ensembling every public kernel that scored higher than 0.84 LB, and found they didn't introduce anything new to what I've already had, until NightwolfsSVMKernel. Turns out, that kernel used two features that my models didn't, and that is what I think made them work together well.
\n\nAs a side note: I'm under impression that usually you don't just take the kernel as is. You modify it to generate 'stacked predictions', or 'OOB predictions' and then use these to train meta-estimator (see here). But, with so small sample set, I couldn't get it to work better than simple mean or rankMean.
", "votes": 1, "replies": [] }, { "id": 530005, "author_name": "Eric R", "author_url": "", "post_date": "2019-05-11T12:58:10.830000", "content": "I see... sounds good to training the meta-estimator using CV. After a few competitions, I realized ensembling is really important for winning kaggle competition, but not so much in industry.
", "votes": 2, "replies": [] } ] }, { "id": 529625, "author_name": "AleksanderMendyk", "author_url": "", "post_date": "2019-05-10T11:32:16.130000", "content": "Congratulations on the achievement! Your kernel was something that jumped me into the Python world. Thanks and wish you all the best for future endeavors - you'll for sure excel in this field
", "votes": 1, "replies": [ { "id": 529652, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T12:23:37.670000", "content": "Dzięki ;P
", "votes": 0, "replies": [] } ] }, { "id": 726220, "author_name": "JoO", "author_url": "", "post_date": "2020-01-22T23:55:59.440000", "content": "It looks like there is a hidden LB probing here and the lucky random seed play some role here?
", "votes": 0, "replies": [] }, { "id": 532859, "author_name": "huguli maguli", "author_url": "", "post_date": "2019-05-17T22:39:03.877000", "content": "Thanks for sharing your approach / I also enjoyed looking into your kernel. \nMy question is how your manage your code - what you described, e.g. keeping track of all your previous submission and making sure you can find the one you need even after a subtle change (e.g. random_seed) I wonder how do you keep track of it ? do you have a smart way of naming your result files ? or you write a paragraph in your internal git ?
", "votes": 0, "replies": [] }, { "id": 529582, "author_name": "WajihullahBaig", "author_url": "", "post_date": "2019-05-10T09:16:52.367000", "content": "I am not fond of lb probing, that is why I used your lasso kernel extensively. Used it figure out important features. Therefore I would put into the best kernel category.
", "votes": 0, "replies": [ { "id": 529586, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T09:22:26.527000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 529570, "author_name": "Akhilesh", "author_url": "", "post_date": "2019-05-10T08:51:50.310000", "content": "Congratulations on then 4th place! Great Job
", "votes": 0, "replies": [ { "id": 529587, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T09:22:41.300000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 529542, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-10T06:52:17.347000", "content": "Great job! I am pleased to see LB probing was not the only way to go.
\n\nWhy do you find it disappointing that adding noise helps?
", "votes": 0, "replies": [ { "id": 529585, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T09:21:52.870000", "content": "Maybe I wasn't clear enough. I feel a bit disappointed about the solution overall, as it is fairly simple. I guess I would've expected top solutions, to be somewhat clever.
", "votes": 1, "replies": [] }, { "id": 529591, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-10T09:29:48.687000", "content": "Overfitting often comes from too complex models. Simplicity is a quality when you have little data as here.
", "votes": 4, "replies": [] } ] }, { "id": 529504, "author_name": "StudentT", "author_url": "", "post_date": "2019-05-10T04:06:59.440000", "content": "Congratulations with fourth place and thank you for sharing your approach! Much to learn here!
\n\nJust for my own education, why isn't this considered as an LB-probing approach? According to the text the solution: \"loads all previous models submitted together with Public LB score\". Although, perhaps a bit more indirect than other approaches, it still uses the LB-score to optimize the prediction, which could be considered as probing (?). Or am i missing the point here...?
", "votes": 0, "replies": [ { "id": 529583, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T09:17:04.520000", "content": "I guess it depends on where you draw the line.\nThe way I look at it, LB-probing is when you use samples from LB in training. I used LB only for validation. Well, that's what the LB is for, right? I allows you to track your progress in comparison to the others, and you simply use that information to see if you've chosen the right direction.
", "votes": 1, "replies": [] }, { "id": 529896, "author_name": "StudentT", "author_url": "", "post_date": "2019-05-11T05:17:06.693000", "content": "Makes sense, thanks for clarifying.
\n\nIts actually quite interesting point you raise here, fundamentally, and using the most extremist interpretation probing here, everyone submitting more than once are in essence LB probing - provided that they have peeked at the LB score...
", "votes": 1, "replies": [] }, { "id": 529945, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-11T08:04:49.903000", "content": "Exactly!
", "votes": 0, "replies": [] } ] }, { "id": 529444, "author_name": "Mark F", "author_url": "", "post_date": "2019-05-09T22:53:31.637000", "content": "I also had a similar experience with poor CV metrics. I tried using leave one out CV, and for some reason, it didn’t seem to do very well at predicting how well a model would do.
\n\nWhy is this, I was under the impression CV should work better than it did.
", "votes": 0, "replies": [] }, { "id": 529408, "author_name": "Sani Kamal", "author_url": "", "post_date": "2019-05-09T20:19:30.547000", "content": "Congrats feature -blind !
", "votes": 0, "replies": [ { "id": 529588, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T09:23:06.750000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 529210, "author_name": "vroom", "author_url": "", "post_date": "2019-05-09T12:48:17.730000", "content": "It's very innovative and impressive, if the noise level to generate the data is higher, you could definitely get into top 3.
", "votes": 0, "replies": [ { "id": 529590, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-10T09:26:19.267000", "content": "You may be right that higher noise could make LB-probing strategies a bit less efficient, but I'm sure the guys at the top would figure out a smart way to deal with it. Nevertheless I'm flattered :) Thanks!
", "votes": 0, "replies": [] } ] } ], "comment_count": 25, "messages": [ { "id": 529114, "postDate": "2019-05-09T08:13:05.207Z", "content": "A journey to 4th place.
\n\nAt first I would like to thank you all for I’ve learned a lot! I guess I’m standing now where Zach stood in the first Don’t Overfit contest. This was my first competition, and first serious contact with machine learning at all.
\n\nUnfortunately, I couldn’t rely on any metrics. I didn’t manage to get a reliable CV scoring throughout the competition. The only way to know if I was doing any better, was to check a Public LB score, so…\nI figured I needed another ranking system, that will indicate me whether I’m going off or not. As Public LB scores was all I could rely on, I developed a simple tool that:\n * loads all previous models I submitted together with Public LB score,\n * loads a new model that I want predict a score for,\n * performs rank and MinMaxScale on all of those models \n * calculates mse, mae, cos, dist, r2 and cor between a new model and all previous models I’ve submitted, searching for submitted model that was ‚closest’ to the one I’m predicting the score for.
\n\nThis ranker improved over time, as I added more and more submissions to its database. \nWhile not very complex, it allowed me to reject non-promising ideas at an early stage, without losing tons of submissions.
\n\nI believe many of you have already seen the kernel Robust, Lasso, Patches with RFE & GS I will refer to it as a LassoKernel from now on. I originally posted it with heuristics yielding 0.868 on Public LB. The heuristics were carefully crafted, as touching any of the parameters would result in a significant drop of Public LB score. How did I come to that? When working on that kernel, at first I did not use any random seed and there was one lucky shot of 0.867 LB that I struggled to reproduce. I fed that one lucky shot into my ranking system described above and started a loop with different random seeds in search of a model as close to 0.867 lucky submission as I could get. That is how it’s gotten to this specific random seed that yields 0.868. This was my base model. Mind that it did not take advantage of any LB probing, as training was done on training samples only, but the process of developing it was strongly dependent on Public LB scores of all my previous submissions, so there definitely is a data leak.
\n\nReading discussions and other kernels, I figured that many people favoured LogisticRegression. So I replaced kernels’ Lasso with LogisticRegression and did many experiments to get the best score I could, hoping to beat a result I got from Lasso. Results can be seen in my second kernel Robust, Patches with RFE & GS and LogReg (LogregKernel).
\n\nThe next step was to stack those models. Simple mean of submissions from the two above kernels yields a 0.872 Public LB (and mean rank 0.873 if I recall correctly). Each of the kernels is a mean of multiple models as well, so I decided to sacrifice a bunch of submissions to get the LB score of different components and some combinations of means or mean ranks of them. That’s how I got to know that eliminating last component of my Lasso kernel yielded 0.872 LB alone, and averaged with several components from my Logreg kernel, yielded a 0.875 (mean rank of 0.876)
\n\nLassoKernel = 0.872\nLogregKernel = 0.868\nBestSoFar = meanRank((LassoKernel, LogregKernel*), weights=[2,1]) = 0.876
\n\n*Precisely it was rank of some components from LogregKernel, but I have a difficulty retrospecting which ones, sorry!
\n\nThen there came a few weeks of stagnation, until @nightwolfbrooks kernel Hyper parameter tuned SVM model (NightwolfsSVMKernel ) came out. I think this kernel deserves much more attention than it seems to have gotten. It is very well crafted, and playing around with it I couldn’t improve it’s score, so I took the submission as is. It turned out that it introduced enough diversity into my BestSoFar, to get to the 0.881 Public LB.
\n\nNightwolfsSVMKernel = 0.861\nTotalBest = meanRank((BestSoFar, NightwolfsSVMKernel), weights=[2,1]) = 0.881
\n\nAs I mentioned above, I couldn’t rely on CV at all, so both GridSearchCV and RFECV in my Kernels were not reliable, and the overall performance of my kernels is a combination of pure chance and my determination to advance on Public LB. \nAs to the noise introduced in my Lasso kernel - it only affects RFECV in some mysterious way, so that it chooses ‚optimal’ feature set. With noise removed (or with different random_state) other features get selected, and the LB score goes down. With noise introduced only to data fed to feature selector, but not to the GridSearchCV the 0.868 LB score still holds.
\n\nI find this a bit disappointing, but I hope you learn at least a bit from my approach.
\n\nThank you all!
", "rawMarkdown": "A journey to 4th place.\n\nAt first I would like to thank you all for I’ve learned a lot! I guess I’m standing now where Zach stood in the first Don’t Overfit contest. This was my first competition, and first serious contact with machine learning at all.\n\nUnfortunately, I couldn’t rely on any metrics. I didn’t manage to get a reliable CV scoring throughout the competition. The only way to know if I was doing any better, was to check a Public LB score, so…\nI figured I needed another ranking system, that will indicate me whether I’m going off or not. As Public LB scores was all I could rely on, I developed a simple tool that:\n * loads all previous models I submitted together with Public LB score,\n * loads a new model that I want predict a score for,\n * performs rank and MinMaxScale on all of those models \n * calculates mse, mae, cos, dist, r2 and cor between a new model and all previous models I’ve submitted, searching for submitted model that was ‚closest’ to the one I’m predicting the score for.\n\nThis ranker improved over time, as I added more and more submissions to its database. \nWhile not very complex, it allowed me to reject non-promising ideas at an early stage, without losing tons of submissions. \n\n\nI believe many of you have already seen the kernel [Robust, Lasso, Patches with RFE & GS](https://www.kaggle.com/featureblind/robust-lasso-patches-with-rfe-gs,) I will refer to it as a LassoKernel from now on. I originally posted it with heuristics yielding 0.868 on Public LB. The heuristics were carefully crafted, as touching any of the parameters would result in a significant drop of Public LB score. How did I come to that? When working on that kernel, at first I did not use any random seed and there was one lucky shot of 0.867 LB that I struggled to reproduce. I fed that one lucky shot into my ranking system described above and started a loop with different random seeds in search of a model as close to 0.867 lucky submission as I could get. That is how it’s gotten to this specific random seed that yields 0.868. This was my base model. Mind that it did not take advantage of any LB probing, as training was done on training samples only, but the process of developing it was strongly dependent on Public LB scores of all my previous submissions, so there definitely is a data leak.\n\n\nReading discussions and other kernels, I figured that many people favoured LogisticRegression. So I replaced kernels’ Lasso with LogisticRegression and did many experiments to get the best score I could, hoping to beat a result I got from Lasso. Results can be seen in my second kernel [ Robust, Patches with RFE & GS and LogReg](https://www.kaggle.com/featureblind/robust-patches-with-rfe-gs-and-logreg) (LogregKernel).\n\nThe next step was to stack those models. Simple mean of submissions from the two above kernels yields a 0.872 Public LB (and mean rank 0.873 if I recall correctly). Each of the kernels is a mean of multiple models as well, so I decided to sacrifice a bunch of submissions to get the LB score of different components and some combinations of means or mean ranks of them. That’s how I got to know that eliminating last component of my Lasso kernel yielded 0.872 LB alone, and averaged with several components from my Logreg kernel, yielded a 0.875 (mean rank of 0.876)\n\nLassoKernel = 0.872\nLogregKernel = 0.868\nBestSoFar = meanRank((LassoKernel, LogregKernel*), weights=[2,1]) = 0.876\n\n*Precisely it was rank of some components from LogregKernel, but I have a difficulty retrospecting which ones, sorry!\n\nThen there came a few weeks of stagnation, until @nightwolfbrooks kernel [Hyper parameter tuned SVM model](https://www.kaggle.com/nightwolfbrooks/hyper-parameter-tuned-svm-model) (NightwolfsSVMKernel ) came out. I think this kernel deserves much more attention than it seems to have gotten. It is very well crafted, and playing around with it I couldn’t improve it’s score, so I took the submission as is. It turned out that it introduced enough diversity into my BestSoFar, to get to the 0.881 Public LB.\n\nNightwolfsSVMKernel = 0.861\nTotalBest = meanRank((BestSoFar, NightwolfsSVMKernel), weights=[2,1]) = 0.881\n\nAs I mentioned above, I couldn’t rely on CV at all, so both GridSearchCV and RFECV in my Kernels were not reliable, and the overall performance of my kernels is a combination of pure chance and my determination to advance on Public LB. \nAs to the noise introduced in my Lasso kernel - it only affects RFECV in some mysterious way, so that it chooses ‚optimal’ feature set. With noise removed (or with different random_state) other features get selected, and the LB score goes down. With noise introduced only to data fed to feature selector, but not to the GridSearchCV the 0.868 LB score still holds.\n\nI find this a bit disappointing, but I hope you learn at least a bit from my approach.\n\nThank you all!\n", "votes": 31 }, { "id": 529290, "postDate": "2019-05-09T15:43:49.507Z", "content": "Congrats feature -blind ! Your kernel is impressive and it is what I started with 20 days ago. I downloaded your submission.csv and extracted the hyperplane directly from your test.csv predictions. For the next week, I randomly rotated your hyperplane 5 degrees and resubmitted (5 times each day). I got your model up to LB 0.875.
Then Nicholas Brooks posted his LB 0.861 SVM kernel, here. The next day, my rank dropped 10 places and 10 new people had LB 0.876 and higher. I suspected the secret was ensembling with Nick's kernel. (Thanks for confirming this). I was about to do that myself, but then I said, I can't keep making small slow updates, I need a more efficient LB-probing scheme.
\n\nThe next day, I brainstormed and devised my current efficient LB-probing scheme here. I was so excited because it gained LB and Private score so quickly in all my simulations using synthetic data.
", "rawMarkdown": "Congrats feature -blind ! Your [kernel][1] is impressive and it is what I started with 20 days ago. I downloaded your `submission.csv` and extracted the hyperplane directly from your `test.csv` predictions. For the next week, I randomly rotated your hyperplane 5 degrees and resubmitted (5 times each day). I got your model up to LB 0.875. \n \nThen Nicholas Brooks posted his LB 0.861 SVM kernel, [here][2]. The next day, my rank dropped 10 places and 10 new people had LB 0.876 and higher. I suspected the secret was ensembling with Nick's kernel. (Thanks for confirming this). I was about to do that myself, but then I said, I can't keep making small slow updates, I need a more efficient LB-probing scheme. \n \nThe next day, I brainstormed and devised my current efficient LB-probing scheme [here][3]. I was so excited because it gained LB and Private score so quickly in all my simulations using synthetic data. \n \n[1]: https://www.kaggle.com/featureblind/robust-lasso-patches-with-rfe-gs\n[2]: https://www.kaggle.com/nightwolfbrooks/hyper-parameter-tuned-svm-model\n[3]: https://www.kaggle.com/cdeotte/lb-probing-strategies-0-890-2nd-place", "votes": 3, "replies": [ { "id": 529376, "postDate": "2019-05-09T18:49:08.567Z", "content": "Thank you! I see you're full of great ideas! I would love to see your code for random rotations of that hyperplane some day :) EDIT: I should read your materials more carefully! I see you've already posted that code. Thanks!
", "rawMarkdown": "Thank you! I see you're full of great ideas! I would love to see your code for random rotations of that hyperplane some day :) EDIT: I should read your materials more carefully! I see you've already posted that code. Thanks!", "votes": 1 }, { "id": 529838, "postDate": "2019-05-10T23:13:20.360Z", "content": "I've never ensembled public kernels. Does ensembling two methods with similar performances almost always yield better performance?
", "rawMarkdown": "I've never ensembled public kernels. Does ensembling two methods with similar performances almost always yield better performance?" }, { "id": 529944, "postDate": "2019-05-11T08:03:52.183Z", "content": "I've tried ensembling every public kernel that scored higher than 0.84 LB, and found they didn't introduce anything new to what I've already had, until NightwolfsSVMKernel. Turns out, that kernel used two features that my models didn't, and that is what I think made them work together well.
\n\nAs a side note: I'm under impression that usually you don't just take the kernel as is. You modify it to generate 'stacked predictions', or 'OOB predictions' and then use these to train meta-estimator (see here). But, with so small sample set, I couldn't get it to work better than simple mean or rankMean.
", "rawMarkdown": "I've tried ensembling every public kernel that scored higher than 0.84 LB, and found they didn't introduce anything new to what I've already had, until [NightwolfsSVMKernel](https://www.kaggle.com/nightwolfbrooks/hyper-parameter-tuned-svm-model). Turns out, that kernel used two features that my models didn't, and that is what I think made them work together well. \n\nAs a side note: I'm under impression that usually you don't just take the kernel as is. You modify it to generate 'stacked predictions', or 'OOB predictions' and then use these to train meta-estimator (see [here](https://www.kaggle.com/c/dont-overfit-ii/discussion/83183#latest-494996)). But, with so small sample set, I couldn't get it to work better than simple mean or rankMean.", "votes": 1 }, { "id": 530005, "postDate": "2019-05-11T12:58:10.830Z", "content": "I see... sounds good to training the meta-estimator using CV. After a few competitions, I realized ensembling is really important for winning kaggle competition, but not so much in industry.
", "rawMarkdown": "I see... sounds good to training the meta-estimator using CV. After a few competitions, I realized ensembling is really important for winning kaggle competition, but not so much in industry.", "votes": 2 } ] }, { "id": 529625, "postDate": "2019-05-10T11:32:16.130Z", "content": "Congratulations on the achievement! Your kernel was something that jumped me into the Python world. Thanks and wish you all the best for future endeavors - you'll for sure excel in this field
", "rawMarkdown": "Congratulations on the achievement! Your kernel was something that jumped me into the Python world. Thanks and wish you all the best for future endeavors - you'll for sure excel in this field", "votes": 1, "replies": [ { "id": 529652, "postDate": "2019-05-10T12:23:37.670Z", "content": "Dzięki ;P
", "rawMarkdown": "Dzięki ;P" } ] }, { "id": 726220, "postDate": "2020-01-22T23:55:59.440Z", "content": "It looks like there is a hidden LB probing here and the lucky random seed play some role here?
", "rawMarkdown": "It looks like there is a hidden LB probing here and the lucky random seed play some role here? " }, { "id": 532859, "postDate": "2019-05-17T22:39:03.877Z", "content": "Thanks for sharing your approach / I also enjoyed looking into your kernel. \nMy question is how your manage your code - what you described, e.g. keeping track of all your previous submission and making sure you can find the one you need even after a subtle change (e.g. random_seed) I wonder how do you keep track of it ? do you have a smart way of naming your result files ? or you write a paragraph in your internal git ?
", "rawMarkdown": "Thanks for sharing your approach / I also enjoyed looking into your kernel. \nMy question is how your manage your code - what you described, e.g. keeping track of all your previous submission and making sure you can find the one you need even after a subtle change (e.g. random_seed) I wonder how do you keep track of it ? do you have a smart way of naming your result files ? or you write a paragraph in your internal git ?" }, { "id": 529582, "postDate": "2019-05-10T09:16:52.367Z", "content": "I am not fond of lb probing, that is why I used your lasso kernel extensively. Used it figure out important features. Therefore I would put into the best kernel category.
", "rawMarkdown": "I am not fond of lb probing, that is why I used your lasso kernel extensively. Used it figure out important features. Therefore I would put into the best kernel category.", "replies": [ { "id": 529586, "postDate": "2019-05-10T09:22:26.527Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 529570, "postDate": "2019-05-10T08:51:50.310Z", "content": "Congratulations on then 4th place! Great Job
", "rawMarkdown": "Congratulations on then 4th place! Great Job\n", "replies": [ { "id": 529587, "postDate": "2019-05-10T09:22:41.300Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 529542, "postDate": "2019-05-10T06:52:17.347Z", "content": "Great job! I am pleased to see LB probing was not the only way to go.
\n\nWhy do you find it disappointing that adding noise helps?
", "rawMarkdown": "Great job! I am pleased to see LB probing was not the only way to go.\n\nWhy do you find it disappointing that adding noise helps?", "replies": [ { "id": 529585, "postDate": "2019-05-10T09:21:52.870Z", "content": "Maybe I wasn't clear enough. I feel a bit disappointed about the solution overall, as it is fairly simple. I guess I would've expected top solutions, to be somewhat clever.
", "rawMarkdown": "Maybe I wasn't clear enough. I feel a bit disappointed about the solution overall, as it is fairly simple. I guess I would've expected top solutions, to be somewhat clever.", "votes": 1 }, { "id": 529591, "postDate": "2019-05-10T09:29:48.687Z", "content": "Overfitting often comes from too complex models. Simplicity is a quality when you have little data as here.
", "rawMarkdown": "Overfitting often comes from too complex models. Simplicity is a quality when you have little data as here.", "votes": 4 } ] }, { "id": 529504, "postDate": "2019-05-10T04:06:59.440Z", "content": "Congratulations with fourth place and thank you for sharing your approach! Much to learn here!
\n\nJust for my own education, why isn't this considered as an LB-probing approach? According to the text the solution: \"loads all previous models submitted together with Public LB score\". Although, perhaps a bit more indirect than other approaches, it still uses the LB-score to optimize the prediction, which could be considered as probing (?). Or am i missing the point here...?
", "rawMarkdown": "Congratulations with fourth place and thank you for sharing your approach! Much to learn here!\n\nJust for my own education, why isn't this considered as an LB-probing approach? According to the text the solution: \"loads all previous models submitted together with Public LB score\". Although, perhaps a bit more indirect than other approaches, it still uses the LB-score to optimize the prediction, which could be considered as probing (?). Or am i missing the point here...? ", "replies": [ { "id": 529583, "postDate": "2019-05-10T09:17:04.520Z", "content": "I guess it depends on where you draw the line.\nThe way I look at it, LB-probing is when you use samples from LB in training. I used LB only for validation. Well, that's what the LB is for, right? I allows you to track your progress in comparison to the others, and you simply use that information to see if you've chosen the right direction.
", "rawMarkdown": "I guess it depends on where you draw the line.\nThe way I look at it, LB-probing is when you use samples from LB in training. I used LB only for validation. Well, that's what the LB is for, right? I allows you to track your progress in comparison to the others, and you simply use that information to see if you've chosen the right direction.", "votes": 1 }, { "id": 529896, "postDate": "2019-05-11T05:17:06.693Z", "content": "Makes sense, thanks for clarifying.
\n\nIts actually quite interesting point you raise here, fundamentally, and using the most extremist interpretation probing here, everyone submitting more than once are in essence LB probing - provided that they have peeked at the LB score...
", "rawMarkdown": "Makes sense, thanks for clarifying. \n\nIts actually quite interesting point you raise here, fundamentally, and using the most extremist interpretation probing here, everyone submitting more than once are in essence LB probing - provided that they have peeked at the LB score...", "votes": 1 }, { "id": 529945, "postDate": "2019-05-11T08:04:49.903Z", "content": "Exactly!
", "rawMarkdown": "Exactly!" } ] }, { "id": 529444, "postDate": "2019-05-09T22:53:31.637Z", "content": "I also had a similar experience with poor CV metrics. I tried using leave one out CV, and for some reason, it didn’t seem to do very well at predicting how well a model would do.
\n\nWhy is this, I was under the impression CV should work better than it did.
", "rawMarkdown": "I also had a similar experience with poor CV metrics. I tried using leave one out CV, and for some reason, it didn’t seem to do very well at predicting how well a model would do.\n\nWhy is this, I was under the impression CV should work better than it did." }, { "id": 529408, "postDate": "2019-05-09T20:19:30.547Z", "content": "Congrats feature -blind !
", "rawMarkdown": "Congrats feature -blind ! ", "replies": [ { "id": 529588, "postDate": "2019-05-10T09:23:06.750Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 529210, "postDate": "2019-05-09T12:48:17.730Z", "content": "It's very innovative and impressive, if the noise level to generate the data is higher, you could definitely get into top 3.
", "rawMarkdown": "It's very innovative and impressive, if the noise level to generate the data is higher, you could definitely get into top 3.", "replies": [ { "id": 529590, "postDate": "2019-05-10T09:26:19.267Z", "content": "You may be right that higher noise could make LB-probing strategies a bit less efficient, but I'm sure the guys at the top would figure out a smart way to deal with it. Nevertheless I'm flattered :) Thanks!
", "rawMarkdown": "You may be right that higher noise could make LB-probing strategies a bit less efficient, but I'm sure the guys at the top would figure out a smart way to deal with it. Nevertheless I'm flattered :) Thanks!" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:13:02Z", "competition_id": 12896, "competition_slug": "dont-overfit-ii", "team_id": 2804002, "solution_url": "/competitions/dont-overfit-ii/writeups/feature-blind-4th-place-solution-no-lb-probing" }