Congratulations to all the winners! and everyone who avoided shakeup on private LB!
\nI've decided to share my solution because it achieves top 50 without post processing and contains some interesting ideas! For post processing, I purposely degraded (i.e. add noise to) my Asian predictions. This improved CV score (+0.0005) and improves public LB score, but hurt private LB score (-0.0015) UGH! My final solution achieves top 50th private LB without PP and 854th private LB with PP :-(
I quickly jumped to 1st place public LB for the first month of the competition by stacking Survival:Cox loss model over KaplanMeierFitter, AFT, and EFS target models. First I trained 6 nested fold models. Using 10 folds inside 10 folds to prevent leakage, I trained 6 models:
I used the nested OOF predictions from the 6 models above and then trained models using these OOF and the original features:
\nThe blend of these two models achieved 1st place Public LB for the first half of the competition
After the first half of the competition, many teams started achieving high public LB scores and I would wake up each morning to see my public LB ranking slowly decreasing. So, slowly I brainstormed and trained more diverse models to add to my ensemble:
\nAdding these diverse models to my ensemble improved my CV score 0.686 -> 0.687 -> 0.688 -> 0.689 -> 0.690 -> 0.691. And my public LB improved 0.691 -> 0.692 -> 0.693 -> 0.694 -> 0.695 -> 0.696!
To be honest, I was very surprised to drop to 854th rank. I thought there would be some shakeup, but I expected to finish in the top50. All during the competition, everything I did improved both my CV score and my public LB score. So I assumed my private LB score would be good.
\nIn retrospect, what hurt me the most was my post process (to degrade Asian predictions). Additionally, my full ensemble did not receive as much private LB boost as other teams. When reading top team solutions, many teams received a +0.007 boost comparing private LB score versus CV score. When I remove my post process, my full ensemble only achieves a +0.002 boost comparing private LB versus CV score.
In the middle of the competition, I discovered that if I degrade my Asian predictions, it improved CV score by +0.0005 and improved public LB too. Unfortunately after the competition ended, I discovered that this hurt the private LB score by -0.0015 UGH!
My favorite model in my ensemble was my CatBoost pairwise ranker. I created a new dataset with 20 million rows! I randomly picked 5 million pairs from efs=1 and I randomly picked 5 million valid pairs from efs=1 and efs=0 (pick one efs=1 with efs_time smaller than efs=0 censored). I then trained a CatBoost ranking model using these 10 million pairs. (Note that a ranker model does not require any target transformation because each valid pair is ranked by efs_time).
I stacked this ranker model as a level 2 model over my level 1 models' OOF (described above). And I added 180 target encoded features made from combinations of columns using targets efs and efs_time using statistics mean, max, min, nunique. (See code here for specific TE engineered features in code cell #27. I found these beneficial TE features using a random search with RAPIDS cuDF cuML).
= {\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n}\nEach consecutive row were pairs and I trained with
\n = Pool(\n , \n label,\n group_id=[i \n cat_features\n)\nWhen I remove my post process which degrades Asians predictions, my final submission achieves CV 0.692 - Public LB 0.693 - Private LB 0.694 and achieves Top50 on private LB hooray!
Unfortunately during the competition I submitted both final submission with my post process and achieved CV 0.693 - Public LB 0.693 - Private LB 0.692 UGH :-(
I share a simple version of my final submission without post processing here. Enjoy!
", "comments": [ { "id": 3146736, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2025-03-11T07:51:31.733000", "content": "\n\nI discovered that if I degrade my Asian predictions, it improved CV score by +0.0005 and improved public LB too.
\n
We also discovered this but the improvement was not stable across all the folds, therefore we didn't consider using it. This was one of such competitions where it was easy to overfit to CV.
\nI just went down from 42th place to 1329th place in Jane Street competition, so I feel your pain.
", "votes": 6, "replies": [ { "id": 3146800, "author_name": "Chris Deotte", "author_url": "", "post_date": "2025-03-11T09:01:13.440000", "content": "Congrats Ahmet and team achieving 2nd place cash gold!
\n\n\nWe also discovered this but the improvement was not stable across all the folds
\n
This is a great way to prevent overfitting by checking if ideas consistently improve all folds. I will use this technique in the future.
", "votes": 1, "replies": [] } ] }, { "id": 3146774, "author_name": "hyd", "author_url": "", "post_date": "2025-03-11T08:27:04.963000", "content": "My sub which degrades Asians predictions also imporve CV by +0.0005, hurt the private LB score by -0.0017~ but i also choosed one sub without post processing.
\nSo my final CV 0.69285 - Public LB 0.69288 - Private LB 0.69722
\nIt seems like we need some luck to win gold medal~
Congratulations @hydantess Amazing job achieving solo gold 10th in 1 week with only 5 subs! What does your final solution look like?
", "votes": 1, "replies": [ { "id": 3146809, "author_name": "hyd", "author_url": "", "post_date": "2025-03-11T09:20:12.693000", "content": "A classfication model and a regression model on efs=1, just like other top teams, but i only used lightgbm and NN, didn't try ranking model.
", "votes": 5, "replies": [] } ] } ] }, { "id": 3146746, "author_name": "Tom", "author_url": "", "post_date": "2025-03-11T07:56:34.393000", "content": "When I first handle post processing part I have a strong feeling this might destroys everything. And it actually happened. But it did well if we just apply this processing on prl nn.
", "votes": 2, "replies": [] }, { "id": 3146562, "author_name": "Robert Hatch", "author_url": "", "post_date": "2025-03-11T02:49:40.007000", "content": "Getting a surprisingly bad public LB score (that appears to be just random variation after the fact) makes for difficult decisions and second guessing everything. But the worst is just getting great public LB score… that after the fact appears in part to be random variation the other direction.
\nRough. No mixed signals, no reason to suspect any issue.
\nNew dataset AND pair ranking sounds really impressive, and I love the idea of Cox model stacker with a variety of bar model inputs. Really well done!
\nFirst and second, especially, did so much amazing work, whereas I was only barely scratching the surface of the winning split probabilities and multiply. But one idea I really wanted to try in the final days but ran out of time was stacking diverse models with diverse targets as inputs into the split models as Stacker models on top of that. Your initial stacker model reminds me of that. The different target ideas throughout the competition really add a dimension to ensembles and stacking.
", "votes": 2, "replies": [ { "id": 3146802, "author_name": "Chris Deotte", "author_url": "", "post_date": "2025-03-11T09:07:42.683000", "content": "Thanks Robert. Congrats achieving solo cash gold!
\n\n\nNew dataset AND pair ranking sounds really impressive
\n
I enjoyed building a ranker model because it does not require any target transformation and this made it feel more pure and natural. Each pair of rows gets a target of 0 and 1 based on efs_time. And we only make valid pairs (which are efs=1 paired with efs=1, or efs=1 paired with efs=0 where efs=1's efs_time is less than efs=0's efs_time).
tmp_train[] = np.repeat( np.arange(len(tmp_train)\ntmp_train[] = tmp_train.groupby()[]\\\n .rank(=''). - 1\nand then
\nX_train = tmp_train[FEATURES]\ny_train = tmp_train\n\ntrain_pool = Pool(\n data=X_train, \n label=y_train,\n group_id=[i // i range((X_train))],\n cat_features=CATS\n)\nCongratulations to all the winners! and everyone who avoided shakeup on private LB!
\nI've decided to share my solution because it achieves top 50 without post processing and contains some interesting ideas! For post processing, I purposely degraded (i.e. add noise to) my Asian predictions. This improved CV score (+0.0005) and improves public LB score, but hurt private LB score (-0.0015) UGH! My final solution achieves top 50th private LB without PP and 854th private LB with PP :-(
I quickly jumped to 1st place public LB for the first month of the competition by stacking Survival:Cox loss model over KaplanMeierFitter, AFT, and EFS target models. First I trained 6 nested fold models. Using 10 folds inside 10 folds to prevent leakage, I trained 6 models:
I used the nested OOF predictions from the 6 models above and then trained models using these OOF and the original features:
\nThe blend of these two models achieved 1st place Public LB for the first half of the competition
After the first half of the competition, many teams started achieving high public LB scores and I would wake up each morning to see my public LB ranking slowly decreasing. So, slowly I brainstormed and trained more diverse models to add to my ensemble:
\nAdding these diverse models to my ensemble improved my CV score 0.686 -> 0.687 -> 0.688 -> 0.689 -> 0.690 -> 0.691. And my public LB improved 0.691 -> 0.692 -> 0.693 -> 0.694 -> 0.695 -> 0.696!
To be honest, I was very surprised to drop to 854th rank. I thought there would be some shakeup, but I expected to finish in the top50. All during the competition, everything I did improved both my CV score and my public LB score. So I assumed my private LB score would be good.
\nIn retrospect, what hurt me the most was my post process (to degrade Asian predictions). Additionally, my full ensemble did not receive as much private LB boost as other teams. When reading top team solutions, many teams received a +0.007 boost comparing private LB score versus CV score. When I remove my post process, my full ensemble only achieves a +0.002 boost comparing private LB versus CV score.
In the middle of the competition, I discovered that if I degrade my Asian predictions, it improved CV score by +0.0005 and improved public LB too. Unfortunately after the competition ended, I discovered that this hurt the private LB score by -0.0015 UGH!
My favorite model in my ensemble was my CatBoost pairwise ranker. I created a new dataset with 20 million rows! I randomly picked 5 million pairs from efs=1 and I randomly picked 5 million valid pairs from efs=1 and efs=0 (pick one efs=1 with efs_time smaller than efs=0 censored). I then trained a CatBoost ranking model using these 10 million pairs. (Note that a ranker model does not require any target transformation because each valid pair is ranked by efs_time).
I stacked this ranker model as a level 2 model over my level 1 models' OOF (described above). And I added 180 target encoded features made from combinations of columns using targets efs and efs_time using statistics mean, max, min, nunique. (See code here for specific TE engineered features in code cell #27. I found these beneficial TE features using a random search with RAPIDS cuDF cuML).
= {\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n}\nEach consecutive row were pairs and I trained with
\n = Pool(\n , \n label,\n group_id=[i \n cat_features\n)\nWhen I remove my post process which degrades Asians predictions, my final submission achieves CV 0.692 - Public LB 0.693 - Private LB 0.694 and achieves Top50 on private LB hooray!
Unfortunately during the competition I submitted both final submission with my post process and achieved CV 0.693 - Public LB 0.693 - Private LB 0.692 UGH :-(
I share a simple version of my final submission without post processing here. Enjoy!
", "rawMarkdown": "Congratulations to all the winners! and everyone who avoided shakeup on private LB! \n\nI've decided to share my solution because it achieves `top 50 without post processing` and contains some interesting ideas! For post processing, I purposely degraded (i.e. add noise to) my Asian predictions. This improved CV score (`+0.0005`) and improves public LB score, but hurt private LB score (`-0.0015`) **UGH!** My final solution achieves top **50th** private LB **without** PP and **854th** private LB **with** PP :-(\n\n# Stacking Models - Level 1\nI quickly jumped to 1st place public LB for the first month of the competition by stacking `Survival:Cox loss` model over `KaplanMeierFitter, AFT, and EFS` target models. First I trained 6 nested fold models. Using 10 folds inside 10 folds to prevent leakage, I trained 6 models:\n* XGBoost KaplanMeierFitter\n* CatBoost KaplanMeierFitter\n* CatBoost AFT\n* CoxPHFitter with TargetEncoded categorical\n* CatBoost classify EFS\n* XGBoost classify EFS\n\n# Stacking Models - Level 2\nI used the nested OOF predictions from the 6 models above and then trained models using these OOF and the original features:\n* XGBoost Survivial:Cox\n* CatBoost Survival:Cox\n\nThe blend of these two models achieved `1st place Public LB` for the first half of the competition\n\n# Diversity Models\nAfter the first half of the competition, many teams started achieving high public LB scores and I would wake up each morning to see my public LB ranking slowly decreasing. So, slowly I brainstormed and trained more diverse models to add to my ensemble:\n* AutoGluon\n* TabM\n* NN Pairwise loss\n* GBDT Pairwise loss\n* GBDT with TargetEncoding\n\nAdding these diverse models to my ensemble improved my `CV score 0.686` -> `0.687` -> `0.688` -> `0.689` -> `0.690` -> `0.691`. And my public `LB improved 0.691` -> `0.692` -> `0.693` -> `0.694` -> `0.695` -> `0.696`!\n\n# The ShakeUp Was a Surprise!\nTo be honest, I was very surprised to drop to 854th rank. I thought there would be some shakeup, but I expected to finish in the top50. All during the competition, everything I did improved both my CV score and my public LB score. So I assumed my private LB score would be good.\n\nIn retrospect, what hurt me the most was my post process (to degrade Asian predictions). Additionally, my full ensemble did not receive as much private LB boost as other teams. When reading top team solutions, many teams received a `+0.007` boost comparing private LB score versus CV score. When I remove my post process, my full ensemble only achieves a `+0.002` boost comparing private LB versus CV score.\n\n# Post Process\nIn the middle of the competition, I discovered that if I degrade my Asian predictions, it improved CV score by `+0.0005` and improved public LB too. Unfortunately after the competition ended, I discovered that this hurt the private LB score by `-0.0015` UGH!\n\n# CatBoost Pairwise Ranker \n\nMy favorite model in my ensemble was my CatBoost pairwise ranker. I created a new dataset with 20 million rows! I randomly picked 5 million pairs from `efs=1` and I randomly picked 5 million valid pairs from `efs=1 and efs=0` (pick one `efs=1` with `efs_time` smaller than `efs=0` censored). I then trained a CatBoost ranking model using these 10 million pairs. (Note that a ranker model does not require any target transformation because each valid pair is ranked by `efs_time`).\n\nI stacked this ranker model as a level 2 model over my level 1 models' OOF (described above). And I added 180 target encoded features made from combinations of columns using targets `efs` and `efs_time` using statistics `mean`, `max`, `min`, `nunique`. (See code [here][1] for specific TE engineered features in code cell #27. I found these beneficial TE features using a random search with RAPIDS cuDF cuML).\n\n params = {\n 'loss_function': 'PairLogitPairwise',\n 'depth': 5,\n 'colsample_bylevel': 0.5,\n 'bootstrap_type': 'Bernoulli',\n 'subsample': 0.8,\n 'learning_rate': 0.03,\n 'eval_metric': 'NDCG:top=2',\n 'task_type': 'GPU',\n }\n\nEach consecutive row were pairs and I trained with\n\n train_pool = Pool(\n data=X_train, \n label=y_train,\n group_id=[i // 2 for i in range(len(X_train))],\n cat_features=CATS\n )\n\n# Fantasy - Final Submission - CV 0.692 - Public LB 0.693 - Private LB 0.694\nWhen I remove my post process which degrades Asians predictions, my final submission achieves `CV 0.692` - `Public LB 0.693` - `Private LB 0.694` and achieves `Top50 on private LB` hooray! \n\n# Reality - Final Submission - CV 0.693 - Public LB 0.693 - Private LB 0.692\nUnfortunately during the competition I submitted both final submission with my post process and achieved `CV 0.693` - `Public LB 0.693` - `Private LB 0.692` UGH :-(\n\n# Final Submission Code\nI share a simple version of my final submission without post processing [here][1]. Enjoy!\n\n[1]: https://www.kaggle.com/code/cdeotte/stacking-model-top-50th-lb-0-694/\n", "votes": 36 }, { "id": 3146736, "postDate": "2025-03-11T07:51:31.733Z", "content": "\n\nI discovered that if I degrade my Asian predictions, it improved CV score by +0.0005 and improved public LB too.
\n
We also discovered this but the improvement was not stable across all the folds, therefore we didn't consider using it. This was one of such competitions where it was easy to overfit to CV.
\nI just went down from 42th place to 1329th place in Jane Street competition, so I feel your pain.
", "rawMarkdown": "> I discovered that if I degrade my Asian predictions, it improved CV score by +0.0005 and improved public LB too.\n\nWe also discovered this but the improvement was not stable across all the folds, therefore we didn't consider using it. This was one of such competitions where it was easy to overfit to CV.\n\nI just went down from 42th place to 1329th place in Jane Street competition, so I feel your pain.", "votes": 6, "replies": [ { "id": 3146800, "postDate": "2025-03-11T09:01:13.440Z", "content": "Congrats Ahmet and team achieving 2nd place cash gold!
\n\n\nWe also discovered this but the improvement was not stable across all the folds
\n
This is a great way to prevent overfitting by checking if ideas consistently improve all folds. I will use this technique in the future.
", "rawMarkdown": "Congrats Ahmet and team achieving 2nd place cash gold!\n\n>We also discovered this but the improvement was not stable across all the folds\n\nThis is a great way to prevent overfitting by checking if ideas consistently improve all folds. I will use this technique in the future.", "votes": 1 } ] }, { "id": 3146774, "postDate": "2025-03-11T08:27:04.963Z", "content": "My sub which degrades Asians predictions also imporve CV by +0.0005, hurt the private LB score by -0.0017~ but i also choosed one sub without post processing.
\nSo my final CV 0.69285 - Public LB 0.69288 - Private LB 0.69722
\nIt seems like we need some luck to win gold medal~
Congratulations @hydantess Amazing job achieving solo gold 10th in 1 week with only 5 subs! What does your final solution look like?
", "rawMarkdown": "Congratulations @hydantess Amazing job achieving solo gold 10th in 1 week with only 5 subs! What does your final solution look like?", "votes": 1, "replies": [ { "id": 3146809, "postDate": "2025-03-11T09:20:12.693Z", "content": "A classfication model and a regression model on efs=1, just like other top teams, but i only used lightgbm and NN, didn't try ranking model.
", "rawMarkdown": "A classfication model and a regression model on efs=1, just like other top teams, but i only used lightgbm and NN, didn't try ranking model.", "votes": 5 } ] } ] }, { "id": 3146746, "postDate": "2025-03-11T07:56:34.393Z", "content": "When I first handle post processing part I have a strong feeling this might destroys everything. And it actually happened. But it did well if we just apply this processing on prl nn.
", "rawMarkdown": "When I first handle post processing part I have a strong feeling this might destroys everything. And it actually happened. But it did well if we just apply this processing on prl nn.", "votes": 2 }, { "id": 3146562, "postDate": "2025-03-11T02:49:40.007Z", "content": "Getting a surprisingly bad public LB score (that appears to be just random variation after the fact) makes for difficult decisions and second guessing everything. But the worst is just getting great public LB score… that after the fact appears in part to be random variation the other direction.
\nRough. No mixed signals, no reason to suspect any issue.
\nNew dataset AND pair ranking sounds really impressive, and I love the idea of Cox model stacker with a variety of bar model inputs. Really well done!
\nFirst and second, especially, did so much amazing work, whereas I was only barely scratching the surface of the winning split probabilities and multiply. But one idea I really wanted to try in the final days but ran out of time was stacking diverse models with diverse targets as inputs into the split models as Stacker models on top of that. Your initial stacker model reminds me of that. The different target ideas throughout the competition really add a dimension to ensembles and stacking.
", "rawMarkdown": "Getting a surprisingly bad public LB score (that appears to be just random variation after the fact) makes for difficult decisions and second guessing everything. But the worst is just getting great public LB score... that after the fact appears in part to be random variation the other direction. \n\nRough. No mixed signals, no reason to suspect any issue. \n\nNew dataset AND pair ranking sounds really impressive, and I love the idea of Cox model stacker with a variety of bar model inputs. Really well done!\n\nFirst and second, especially, did so much amazing work, whereas I was only barely scratching the surface of the winning split probabilities and multiply. But one idea I really wanted to try in the final days but ran out of time was stacking diverse models with diverse targets as inputs into the split models as Stacker models on top of that. Your initial stacker model reminds me of that. The different target ideas throughout the competition really add a dimension to ensembles and stacking. ", "votes": 2, "replies": [ { "id": 3146802, "postDate": "2025-03-11T09:07:42.683Z", "content": "Thanks Robert. Congrats achieving solo cash gold!
\n\n\nNew dataset AND pair ranking sounds really impressive
\n
I enjoyed building a ranker model because it does not require any target transformation and this made it feel more pure and natural. Each pair of rows gets a target of 0 and 1 based on efs_time. And we only make valid pairs (which are efs=1 paired with efs=1, or efs=1 paired with efs=0 where efs=1's efs_time is less than efs=0's efs_time).
tmp_train[] = np.repeat( np.arange(len(tmp_train)\ntmp_train[] = tmp_train.groupby()[]\\\n .rank(=''). - 1\nand then
\nX_train = tmp_train[FEATURES]\ny_train = tmp_train\n\ntrain_pool = Pool(\n data=X_train, \n label=y_train,\n group_id=[i // i range((X_train))],\n cat_features=CATS\n)\n