15th place solution
\nFirst of all, I would like to thank the Kaggle community for sharing great ideas and engaging discussions. I would also like to thank the hosts for organizing this interesting task competition.
\nI used public note transform ( Kaplan , Nelson etc. ) and Rank Gauss transform.
\nBecause the number of unique values of the target decreases by using Kaplan and so on , I concerned that amount of information decreases.
\nTherefore I used Rank Gauss transform as not decreasing uniques , then LB was improved slightly.
[simple xgb result]
\n| \n | CV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| Kaplan | \n0.675 | \n\n | \n |
| Nelson | \n0.680 | \n0.685 | \n0.688 | \n
| Rank Gauss | \n0.680 | \n0.686 | \n0.689 | \n
I implemented stacking 30 models twice.
\nWhen n time stacking , I use n-1 time OOF as feature.
30 models = GBDTs(XGB,CAT,LGB) : 3 * Various targets and settings : 10
\n[ensemble result(detail below)]
\n| \n | CV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| No stack and below ensemble | \n0.687 | \n0.689 | \n0.693 | \n
| 1st stack and below ensemble | \n0.690 | \n\n | \n |
| 2nd stack and below ensemble(final submission) | \n0.694 | \n0.691 | \n0.695 | \n
I implemented ensembling as follows using Rank Gauss transform target.
\ny_pred.loc[oof['pred']<threshold,'prediction'] = oof['pred']I consider that the competition score is calculated from two part. One part (a) is to compare each efs=1 prediction ,other (b) is to compare efs=0 prediction against efs=1 prediction.
\nBy training only efs=1 data , ‘a’ part score will be very high.
\n‘a’ part score can be improved by ensemble 1.
\n‘b’ part score can be improved by ensemble 2.
[2nd stack result]
\n| \n | 'a' part (efs=1) CV | \nCV(a+b part) | \n
|---|---|---|
| efs=1 data model | \n0.745 | \n0.561 | \n
| All data model | \n0.597 | \n0.686 | \n
| Ensemble 1 | \n0.684 | \n0.669 | \n
| Ensemble 2(final submission) | \n0.680 | \n0.694 | \n
・Target encoding
\n・Umap/t-SNE as a feature
@tamryow
\nCongratulations on the gold medal! I also improved my LB and CV by stacking in the same way.
\nI didn't try stacking a second time, so your solution is helpful.
\nThank you for sharing!
Thanks!
\nI'm glad I could help you.
Also i did not get the number 2 very well: \"Substituting all data trained model prediction for above prediction in case fallling below the threshold ageinst all data trained model prediction.
\ny_pred.loc[oof['pred']<threshold,'prediction'] = oof['pred']\" what is this threshold?
The threshold is adjusted to optimize CV.
", "votes": 0, "replies": [] } ] }, { "id": 3143930, "author_name": "Vasilis", "author_url": "", "post_date": "2025-03-07T18:22:34.020000", "content": "Hey congrats on the gold! How do you manage to do all this stacking without overfitting? How do you hyperparam optimize to avoid overfitting? Do you use eval set to early stop or not?
", "votes": 1, "replies": [ { "id": 3144200, "author_name": "tamryow", "author_url": "", "post_date": "2025-03-08T04:52:28.267000", "content": "Thank you!
\nI happy to get a gold medal although I think also I have a luck.
\n
\n\nHow do you manage to do all this stacking without overfitting?
\n
I confirmed CV and LB by simple XGB every stacking.
\nBy the second stack, both cv/lb had improved, so i thought there was no overfitting and decided to stack twice.
\n
\n\nHow do you hyperparam optimize to avoid overfitting?
\n
I did not make any fine adjustments to the hyperparameters other than slightly changing the values in the public notebook.
\nThe cv/lb values are not very different both the public notebook and my simple xgb , so I thought it don't have overfitting.
\n
\n\nDo you use eval set to early stop or not?
\n
Yes,i did.
\n
\nIn terms of overfitting, I personally think it was important to avoid overfitting the score elements (a and b parts).
It may seem like hindsight, but I think the reason I was able to win the gold medal this time was because I avoided overfitting score elements(‘a’,’b’part) in CV.
\nIn a typical model in the public notes, the CV and LB for each part are as follows.
\nThe model in the public notes neglects the prediction of 'a' part, and it seemed that the prediction results were not reflected in Public LB.
\n(There were also some other unreliable CV/LB relationships in Public LB (probably the distribution/trend of one of the races was different from CV, etc.) in my submissions, so I don't think it was just that relationship.)
\nIn any case, I considered Public LB to be unreliable based on the above, and I worked on it assuming that Private LB, which has more data, would approach the score of CV.
\n| \n | 'a' part (efs=1) CV | \nCV(a+b part) | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|
| Public note model | \n0.592 | \n0.681 | \n0.691 | \n0.691 | \n
| My final submit model | \n0.680 | \n0.694 | \n0.691 | \n0.695 | \n
15th place solution
\nFirst of all, I would like to thank the Kaggle community for sharing great ideas and engaging discussions. I would also like to thank the hosts for organizing this interesting task competition.
\nI used public note transform ( Kaplan , Nelson etc. ) and Rank Gauss transform.
\nBecause the number of unique values of the target decreases by using Kaplan and so on , I concerned that amount of information decreases.
\nTherefore I used Rank Gauss transform as not decreasing uniques , then LB was improved slightly.
[simple xgb result]
\n| \n | CV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| Kaplan | \n0.675 | \n\n | \n |
| Nelson | \n0.680 | \n0.685 | \n0.688 | \n
| Rank Gauss | \n0.680 | \n0.686 | \n0.689 | \n
I implemented stacking 30 models twice.
\nWhen n time stacking , I use n-1 time OOF as feature.
30 models = GBDTs(XGB,CAT,LGB) : 3 * Various targets and settings : 10
\n[ensemble result(detail below)]
\n| \n | CV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| No stack and below ensemble | \n0.687 | \n0.689 | \n0.693 | \n
| 1st stack and below ensemble | \n0.690 | \n\n | \n |
| 2nd stack and below ensemble(final submission) | \n0.694 | \n0.691 | \n0.695 | \n
I implemented ensembling as follows using Rank Gauss transform target.
\ny_pred.loc[oof['pred']<threshold,'prediction'] = oof['pred']I consider that the competition score is calculated from two part. One part (a) is to compare each efs=1 prediction ,other (b) is to compare efs=0 prediction against efs=1 prediction.
\nBy training only efs=1 data , ‘a’ part score will be very high.
\n‘a’ part score can be improved by ensemble 1.
\n‘b’ part score can be improved by ensemble 2.
[2nd stack result]
\n| \n | 'a' part (efs=1) CV | \nCV(a+b part) | \n
|---|---|---|
| efs=1 data model | \n0.745 | \n0.561 | \n
| All data model | \n0.597 | \n0.686 | \n
| Ensemble 1 | \n0.684 | \n0.669 | \n
| Ensemble 2(final submission) | \n0.680 | \n0.694 | \n
・Target encoding
\n・Umap/t-SNE as a feature
Thanks!
\nI'm glad I could help you.
Also i did not get the number 2 very well: \"Substituting all data trained model prediction for above prediction in case fallling below the threshold ageinst all data trained model prediction.
\ny_pred.loc[oof['pred']<threshold,'prediction'] = oof['pred']\" what is this threshold?
Hey congrats on the gold! How do you manage to do all this stacking without overfitting? How do you hyperparam optimize to avoid overfitting? Do you use eval set to early stop or not?
", "rawMarkdown": "Hey congrats on the gold! How do you manage to do all this stacking without overfitting? How do you hyperparam optimize to avoid overfitting? Do you use eval set to early stop or not?", "votes": 1, "replies": [ { "id": 3144200, "postDate": "2025-03-08T04:52:28.267Z", "content": "Thank you!
\nI happy to get a gold medal although I think also I have a luck.
\n
\n\nHow do you manage to do all this stacking without overfitting?
\n
I confirmed CV and LB by simple XGB every stacking.
\nBy the second stack, both cv/lb had improved, so i thought there was no overfitting and decided to stack twice.
\n
\n\nHow do you hyperparam optimize to avoid overfitting?
\n
I did not make any fine adjustments to the hyperparameters other than slightly changing the values in the public notebook.
\nThe cv/lb values are not very different both the public notebook and my simple xgb , so I thought it don't have overfitting.
\n
\n\nDo you use eval set to early stop or not?
\n
Yes,i did.
\n
\nIn terms of overfitting, I personally think it was important to avoid overfitting the score elements (a and b parts).
It may seem like hindsight, but I think the reason I was able to win the gold medal this time was because I avoided overfitting score elements(‘a’,’b’part) in CV.
\nIn a typical model in the public notes, the CV and LB for each part are as follows.
\nThe model in the public notes neglects the prediction of 'a' part, and it seemed that the prediction results were not reflected in Public LB.
\n(There were also some other unreliable CV/LB relationships in Public LB (probably the distribution/trend of one of the races was different from CV, etc.) in my submissions, so I don't think it was just that relationship.)
\nIn any case, I considered Public LB to be unreliable based on the above, and I worked on it assuming that Private LB, which has more data, would approach the score of CV.
\n| \n | 'a' part (efs=1) CV | \nCV(a+b part) | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|
| Public note model | \n0.592 | \n0.681 | \n0.691 | \n0.691 | \n
| My final submit model | \n0.680 | \n0.694 | \n0.691 | \n0.695 | \n