{
  "id": 566550,
  "title": "1st Place Solution-2 Targets and Ensemble",
  "content": "<p>First, congratulations to the teams that finally \"survived\" in this competition, and thank you to the participants who shared their experiences and provided help in the forum.</p>\n<p>I also trained classifier and regressor models independently and then combined them together with a magic function, just like 2nd and 4th place solution. The main idea is that when a patient has a high probability of <code>efs == 0</code>, we should give them a high score. Otherwise, we should give them a relatively lower score and pay more attention to the rank of <code>efs_time</code>. So the regressor just needs to focus on the ranking task where <code>efs == 1</code>.</p>\n<p>Here is the simplified <a href=\"https://www.kaggle.com/code/minerppdy/1st-place-simplified-private-lb-0-700\" target=\"_blank\">code</a>.</p>\n<h2>1. Feature Engineering</h2>\n<ol>\n<li>Original features  </li>\n<li>One-hot encoding for all categorical features(but still keep the original category features)  </li>\n<li>Copy continuous features as categorical features  </li>\n</ol>\n<h2>2. CV Strategy</h2>\n<ol>\n<li>The CV is very unstable with different splits Strategy. To make the results comparable, I used 10-fold splits with same seeds for all classifier and regressor model training.</li>\n</ol>\n<pre><code>y_combine = data[].astype()++data[][].astype()\nskf = StratifiedKFold(n_splits=, shuffle=, random_state=)\nskf.split(X,y_combine)\n</code></pre>\n<ol>\n<li>without use early stoping to reduce the risk of overfitting, especially when I use a large number of folds.</li>\n</ol>\n<h2>3. Classifier</h2>\n<h3>Target: <code>P(efs=0)</code></h3>\n<h3>Models Used</h3>\n<ol>\n<li>XGBoost, LightGBM, CatBoost  </li>\n<li>NN, TabM</li>\n<li>GNN  </li>\n<li>NN/TabNet/GNN with pairwise-rank-loss</li>\n</ol>\n<h3>tricks</h3>\n<ol>\n<li>Gnn method. I use KNN to find nearest 25 nodes and  create edges(use Euclidean Distance),then use graphsage to fit the targets.</li>\n<li>when use rank loss, the predition would shift over different fold because rank loss is not sensitive to shift, which may cause lower CV when calculate metric in the entire set. So you need to fix these shifts to keep same mean/mid prediction value over models trained in differece fold.</li>\n<li>GNN also have this problem even if I used logloss，I still don't know why.</li>\n</ol>\n<h3>auc metrics</h3>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>auc</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>lightgbm</td>\n<td>0.7603</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>0.7617</td>\n</tr>\n<tr>\n<td>xgboost</td>\n<td>0.7606</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>0.75902</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>0.7596</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>0.7591</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>0.7598</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>0.7597</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>0.7586</td>\n</tr>\n</tbody>\n</table>\n<h3>interesting finding</h3>\n<ol>\n<li>The best max depth for LGB and XGB is 2 (for CatBoost, it's 6, and I still don't know why). This is nearly unheard of in my experience. It means there is little valuable feature interaction information, and the task is easy to fit. Maybe that's why NN like models also work well in this tabular data.</li>\n</ol>\n<h2>4. Regressor</h2>\n<h3>Target</h3>\n<p>Grouped and normalized rank by <code>efs</code>  </p>\n<pre><code>efs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  \nefs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  \n</code></pre>\n<h3>Models used</h3>\n<p>XGBoost lightgbm catboost( NN like model didn't work well for me, if any one succeed in NN like model, please let me know.)</p>\n<h3>Tricks</h3>\n<ol>\n<li>Add efs as a feature when training and focus on the performance at samples where <code>efs==1</code>. Set <code>efs=1</code> when making inference on LB data. </li>\n<li>apply sample weight on efs==1 and efs==0 with 0.6:0.4</li>\n<li>Here's why this trick is used. Initially, I trained the regressor only on samples where <code>efs==1</code>. When using this model to make inference on samples where <code>efs==0</code>, it still showed an obvious correlation between prediction and ground truth. This is strange because efs_time for <code>efs==0</code> is meaningless due to various reasons could cause data censoring. I guess it's because of the SurvalGAN algorithm. By the way, it turns out that there are similar patterns between samples where <code>efs==0</code> and <code>efs==1</code>. So, by adding samples where <code>efs==0</code>, the regressor's performance on <code>efs==1</code> improved significantly!</li>\n</ol>\n<h3>Raw Concordance Index on efs==1</h3>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>c-index</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>xgboost</td>\n<td>0.770229</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>0.767615</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>0.769340</td>\n</tr>\n</tbody>\n</table>\n<h2>5.Merge Function</h2>\n<pre><code> ():\n    \n    y_fun = (Y_HAT_REG&gt;)*c*(np.(Y_HAT_REG))**(b)\n    x_fun =(Y_HAT_CLS&gt;)*(np.(Y_HAT_CLS))**(a)\n    res = (-y_fun)*x_fun+y_fun\n    res = pd.Series(res).rank()/(res)\n     res\n</code></pre>\n<p>This is what the merge function looks like(without rank transform), where x is the probablity of efs==0 and y is predicted efs_time(scaled to 0~1) <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F24200812%2F9121e2fc16b8f0cc18f16c9d8e363608%2Fmerge_function.jpg?generation=1741316566114375&amp;alt=media\" alt=\"\"></p>\n<h2>6 Ensembling Method</h2>\n<ol>\n<li><p>Create combinations between classifiers and regressors, then find the best (a, b, c) for the merge function by with 5-folds and then get the merged predictions.</p>\n<p>Here is Stratified Concordance Index of combinations. Ignoring the rank transform in merge function, pretty sure there is no leakage</p>\n<table>\n<thead>\n<tr>\n<th>cls</th>\n<th>reg</th>\n<th>CV Stratified c-index</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>xgboost</td>\n<td>xgboost</td>\n<td>0.6938821198444668</td>\n</tr>\n<tr>\n<td>xgboost</td>\n<td>catboost</td>\n<td>0.693390761259049</td>\n</tr>\n<tr>\n<td>xgboost</td>\n<td>lightgbm</td>\n<td>0.6920553689042972</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>xgboost</td>\n<td>0.6946771227057665</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>catboost</td>\n<td>0.6942415329389815</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>lightgbm</td>\n<td>0.6928675187239672</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>xgboost</td>\n<td>0.6935306031275579</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>catboost</td>\n<td>0.693123814234674</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>lightgbm</td>\n<td>0.6917244174625249</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>xgboost</td>\n<td>0.6943607674628444</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>catboost</td>\n<td>0.6936616324722884</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>lightgbm</td>\n<td>0.6922428015362854</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>xgboost</td>\n<td>0.6939542378441973</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>catboost</td>\n<td>0.6932931191267933</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>lightgbm</td>\n<td>0.6920700218390199</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>xgboost</td>\n<td>0.6940062476514388</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>catboost</td>\n<td>0.6932012736212297</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>lightgbm</td>\n<td>0.6921340326054011</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>xgboost</td>\n<td>0.693141398733159</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>catboost</td>\n<td>0.6927626179572159</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>lightgbm</td>\n<td>0.6912730733179469</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>xgboost</td>\n<td>0.6944776101013886</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>catboost</td>\n<td>0.6936542865352446</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>lightgbm</td>\n<td>0.6924528052605144</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>xgboost</td>\n<td>0.6934854451860402</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>catboost</td>\n<td>0.692783734834767</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>lightgbm</td>\n<td>0.6916329456191074</td>\n</tr>\n</tbody>\n</table></li>\n<li><p>Ensemble the merged predictions using a weighted average with Optuna and 5-folds.</p></li>\n<li><p>Since there are 9 classifiers × 3 regressors = 27 combinations, there is a risk of overfitting. Therefore, I set the Optuna search range for weights to be between 0.1 and 1, rather than 0 and 1. I think this is a form of regularization.</p></li>\n<li><p>The  final CV Stratified c-index is round 0.6965</p></li>\n<li><p>Adding noise to some race groups significantly improved cross-validation performance, but it didn't work on either the public or private LB.</p></li>\n</ol>",
  "comments": [
    {
      "id": 3142300,
      "author_name": "hyd",
      "author_url": "",
      "post_date": "2025-03-06T09:25:11.360000",
      "content": "<p>Great workl! <br>\nWe have similiar auc score, but your regression models are much better than mine, my final cv is around 0.693~</p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 3142003,
      "author_name": "klog",
      "author_url": "",
      "post_date": "2025-03-06T03:12:10.550000",
      "content": "<p>Congratulations! 🎉 You were leading at the top from start to finish, and it was truly amazing!</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 3149530,
      "author_name": "Andrey Chankin",
      "author_url": "",
      "post_date": "2025-03-14T10:41:28.767000",
      "content": "<p><a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> in one of the old posts you mentioned feature engineering, did you keep them in your final sub?<br>\nAnd, do you mind sharing your background? - Your acc is fresh and you were top1 since the very beginning, just curious</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3149563,
          "author_name": "minerppdy",
          "author_url": "",
          "post_date": "2025-03-14T11:37:45.990000",
          "content": "<p>I have posted the simplified <a href=\"https://www.kaggle.com/code/minerppdy/1st-place-simplified-private-lb-0-700\" target=\"_blank\">code</a> and the feature engineering is almost the same as in my final submission. I registered on Kaggle years ago, but recently when I returned to Kaggle, I couldn't verify my account with Persona, so I had to delete the account and register again…</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3143819,
      "author_name": "Zhong",
      "author_url": "",
      "post_date": "2025-03-07T16:29:18.060000",
      "content": "<p>Congrats! Very nice approach</p>",
      "votes": 2,
      "replies": [
        {
          "id": 3145130,
          "author_name": "Fatima",
          "author_url": "",
          "post_date": "2025-03-09T11:49:13.680000",
          "content": "<p>Yes, i think the same, it is reale good</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3142732,
      "author_name": "Gunes Evitan",
      "author_url": "",
      "post_date": "2025-03-06T16:26:29.163000",
      "content": "<p>Congrats on strong finish! I think you added a decent amount of diversity by ensembling combination of classifiers and regressors. I've never thought doing such thing because I was focused on finalizing the classifier part first. This made me go into tunnel vision and I  used only the same classifier blend with my regressors. I think you might push your score a little bit more by adding 3 more combinations of classifier blend x lightgbm, classifier blend x xgboost and classifier blend x catboost.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 3142790,
          "author_name": "minerppdy",
          "author_url": "",
          "post_date": "2025-03-06T17:10:09.853000",
          "content": "<p>i also tried blend cls and reg first, then apply the merge function, hard to say which is better</p>",
          "votes": 0,
          "replies": [
            {
              "id": 3142829,
              "author_name": "Robert Hatch",
              "author_url": "",
              "post_date": "2025-03-06T17:32:26.607000",
              "content": "<p>I had some strong hints that optimizing all together at once could be best. Or maybe better to say that I knew blend first couldn't search the entire optimization space. It was clear it was further from optimal then normal for effectively optimizing against this formula. </p>\n<p>Got clear evidence of that by stumbling upon an unusually amazing CV score that my normal ensembling method couldn't come close to finding.</p>\n<p>However, I didn't trust my individual model's CV and using a new ensembling method quite enough since I was still trying to hedge against a fear of overfitting CV. </p>\n<p>With the hundreds of base models I used, it could've been interesting to try the greedy ensemble selection I used to pare down the number, but then restart ensembling on the selected models with your Optuna method or something similar. </p>",
              "votes": -1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3142717,
      "author_name": "Sergey Bryansky",
      "author_url": "",
      "post_date": "2025-03-06T16:18:24.033000",
      "content": "<p>Same approach to solve tasks independently then combined it, rank is all u need for ranking task. Cool find with addition of efs as feature, very very smart! I also experimented with some sort of stacking and target recreation using similar approach with probabilities of efs, but always it was overfitted! Congratz! </p>",
      "votes": 2,
      "replies": [
        {
          "id": 3142927,
          "author_name": "Robert Hatch",
          "author_url": "",
          "post_date": "2025-03-06T18:54:37.587000",
          "content": "<p>Something interesting and surprising I found was a (probably inferior) variant of efs as feature: because I tried the trick late and was lazy, I was predicting the smaller train set with hyperparameters tuned for a larger train set. Because of that and/or other unclear reasons, adding rest of train data with sample weight <em>0</em> was a big help to my model. I'm unsure the overall impact, which hyperparameters are affected by the data, or exactly why/how it helped. But it did. On both LGB and Catboost. </p>",
          "votes": -1,
          "replies": []
        }
      ]
    },
    {
      "id": 3141992,
      "author_name": "Robert Hatch",
      "author_url": "",
      "post_date": "2025-03-06T02:49:30.907000",
      "content": "<p>Huge congrats! (1st during almost the whole competition too, was it?) Can you give more insight into how you tuned/trained the a,b,c constants of the merge function? Did you just find what maximized CV score? Were they part of the Optuna search? Or…?</p>\n<p>(I was really worried about overfitting if I just took the constants that maximized CV with my VERY similar function and postprocessing 'tricks'. So even on my winning gamble sub I went a little under the CV optimal constants)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3142035,
          "author_name": "minerppdy",
          "author_url": "",
          "post_date": "2025-03-06T03:39:34.957000",
          "content": "<p><a href=\"https://www.kaggle.com/roberthatch\" target=\"_blank\">@roberthatch</a>  1. I used 5-fold validation to get  the reliable CV score   2. use optuna to find the best (a,b,c) to maximize C-index of train_set,  then get oof result and LB result with best (a,b,c), do this by 5 loops, then calucate the oof C-index   3. how ever this cant prevent overfitting if the form of merge function is not appropriate, just make sure you CV is reliable</p>",
          "votes": 0,
          "replies": [
            {
              "id": 3142096,
              "author_name": "Robert Hatch",
              "author_url": "",
              "post_date": "2025-03-06T05:01:20.673000",
              "content": "<p>Thanks, that's about how I would've done it if not trying to be more careful. My formulation was a little different and I was worried the constants wouldn't translate to LB and would drastically increase any partial over fitting I already had. In part because my LB score was always a bit lower than expected from the first day I tried the two completely separate model trick. (I think my simple LGB two models I had a two targets blend with 0.682 to 0.693 CV  and around 0.688 LB to low 689 LB. With the trick my CV jumped to 0.685 to 686, but LB 0.687 to 0.689. It got much better with Cat boost and with more models, but there was still a concern whether LB score was noise or some partial inability to generalize. Impressive how perfectly your tuning methodology worked!</p>",
              "votes": -1,
              "replies": []
            },
            {
              "id": 3142097,
              "author_name": "Robert Hatch",
              "author_url": "",
              "post_date": "2025-03-06T05:03:08.600000",
              "content": "<p>My 5th place winning submission with postprocessing was only (low) 0.693 on public LB!</p>",
              "votes": -1,
              "replies": []
            }
          ]
        },
        {
          "id": 3142040,
          "author_name": "minerppdy",
          "author_url": "",
          "post_date": "2025-03-06T03:45:20.747000",
          "content": "<p>I also tried NN with pair-wise-loss to combine all the classifiers and regressors, but it didn't work, the CV score is bad </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3175505,
      "author_name": "Focus",
      "author_url": "",
      "post_date": "2025-04-10T08:09:58.850000",
      "content": "<p>I am very thankful if <a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> answer this.<br>\nIn initBTree method there are following statements.</p>\n<p>last_full_row = int(np.log2(len(values) + 1) - 1)<br>\nlen_ragged_row = len(values) - (2 ** (last_full_row + 1) - 1)</p>\n<p>Where does these come from ?<br>\nWhy log2?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3167162,
      "author_name": "Akshat Singhal",
      "author_url": "",
      "post_date": "2025-04-01T10:14:06.877000",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/minerpddy\" target=\"_blank\">@minerpddy</a> congratulations achieving the first place.<br>\nCan you please explain your choice for the magic function. How have you chosen this exact form and not any other? Especially this line here <code>res = (1-y_fun)*x_fun+y_fun</code></p>",
      "votes": 0,
      "replies": [
        {
          "id": 3167165,
          "author_name": "",
          "author_url": "",
          "post_date": "2025-04-01T10:15:53.833000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 3168395,
          "author_name": "LearNerZhou99",
          "author_url": "",
          "post_date": "2025-04-02T11:44:24.420000",
          "content": "<p>yeap, thats really amazing and magic</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3153161,
      "author_name": "Benjamin Richards",
      "author_url": "",
      "post_date": "2025-03-18T13:21:26.637000",
      "content": "<p>Nice work, very interesting. I'd love to learn more about merge functions for blending classifier and regressors. I've seen many blending techniques for models of the same type, but here you are blending models of different types. Just wondering where you first learnt about the merge function? Or if you know of any others? Thanks in advance</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3152420,
      "author_name": "Igor Kostenko",
      "author_url": "",
      "post_date": "2025-03-17T19:31:04.287000",
      "content": "<p>Thanks. I've learned a lot from your solution. Two targets, model combining, amazing</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3150352,
      "author_name": "NEElen",
      "author_url": "",
      "post_date": "2025-03-15T11:49:16.317000",
      "content": "<p>Excellent result. Thank you for sharing your knowledge.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3149398,
      "author_name": "RIYADH HAITHAM",
      "author_url": "",
      "post_date": "2025-03-14T06:24:45.813000",
      "content": "<p>great work did you use hurdle modeling here</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3149566,
          "author_name": "minerppdy",
          "author_url": "",
          "post_date": "2025-03-14T11:39:42.663000",
          "content": "<p>No, but I think the idea is similar.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 3149781,
              "author_name": "RIYADH HAITHAM",
              "author_url": "",
              "post_date": "2025-03-14T17:10:24.127000",
              "content": "<p>thank you if you donot mind i have one question i read all your notebook i understanded most of it just one thing  which is the magic function how you reached it i tried using just simple way like Y_hat_reg * a + Y_hat_clas * b the results was worse did you find it by chance or is it related  with the dataset it self and is it really necessary to pass efs why not just passing all the data without efs column or you want to make conditional training  for your regressor </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3150076,
              "author_name": "minerppdy",
              "author_url": "",
              "post_date": "2025-03-15T03:49:40.540000",
              "content": "<ul>\n<li>The merge function is based on the understanding of C-index metric.</li>\n</ul>\n<blockquote>\n  <p>The main idea is that when a patient has a high probability of efs == 0, we should give them a high score. Otherwise, we should give them a relatively lower score and pay more attention to the rank of efs_time. So the regressor just needs to focus on the ranking task where efs == 1.  </p>\n</blockquote>\n<ul>\n<li>Since the regressor only needs to focus on the ranking task for samples where efs == 1, you can train it using just the efs == 1 samples. Additionally, you can train the model using both efs == 1 and efs == 0 samples to achieve better performance. However, in this case, it is necessary to add efs as an extra feature to prevent the model from getting confused by mixing the two types of samples together.</li>\n</ul>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3147747,
      "author_name": "Arjun Singh",
      "author_url": "",
      "post_date": "2025-03-12T11:49:07.810000",
      "content": "<p>First thing, Congrats! Its my lack of knowledge that I never thought of such approach. Well, it's obvious since I'm still learning but reading your solution gave somewhat insight on how I should've framed the problem. Thanks for sharing the solutions.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3147059,
      "author_name": "DeStee",
      "author_url": "",
      "post_date": "2025-03-11T15:34:23.940000",
      "content": "<p>Very good aproach</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3146947,
      "author_name": "Samay Ashar",
      "author_url": "",
      "post_date": "2025-03-11T13:14:10.810000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a>, solid approach. 👏\nClassifier + Regressor as two separate models is something I never thought of.</p>\n<p>I think the depth 6 of CatBoost might be because of the way it handles categorical features via target encoding unlike LGB/XGB which use one-hot encoding. It can lead to deeper trees for understanding the feature interaction.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3146453,
      "author_name": "Luana Belo",
      "author_url": "",
      "post_date": "2025-03-10T22:25:10.783000",
      "content": "<p>Congrats! Really cool to see your thought process behind the solution, super helpful for juniors like me to understand how to approach these challenges :)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3146235,
      "author_name": "Bhargav Chirumamilla",
      "author_url": "",
      "post_date": "2025-03-10T17:00:59.850000",
      "content": "<p>Congrats! Very nice approach</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3146157,
      "author_name": "Samith Chimminiyan",
      "author_url": "",
      "post_date": "2025-03-10T15:17:12.943000",
      "content": "<p>Congrats !!! Very nice approach. Good for Learning. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3145638,
      "author_name": "Palbha",
      "author_url": "",
      "post_date": "2025-03-10T04:33:06.093000",
      "content": "<p>Hi Omar .. I am trying to understand the code &amp; I saw you trained model with pca &amp; tsne features &amp; then use the best params to find the optimal parameters again with an optuna study &amp;  train the model with best params the training data without pca &amp; tsne .. I mean why not directly find optimal parameters on train data without tsne &amp; pca ? </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3144468,
      "author_name": "Rajwa Abisena",
      "author_url": "",
      "post_date": "2025-03-08T12:05:03.543000",
      "content": "<p>Interesting approach</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3146100,
          "author_name": "Ishraq Tanvir",
          "author_url": "",
          "post_date": "2025-03-10T14:13:13.353000",
          "content": "<p>Yes u are right</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3143993,
      "author_name": "Palbha",
      "author_url": "",
      "post_date": "2025-03-07T19:25:07.953000",
      "content": "<p>Congratulations !! Thanks for sharing the approach. Would be more helpful if you could share the code so we could understand better in detail</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3144004,
          "author_name": "",
          "author_url": "",
          "post_date": "2025-03-07T19:50:10.830000",
          "content": "<h1>Función para optimizar el uso de memoria</h1>\n<p>def optimize_dtypes(df):<br>\n    df_copy = df.copy()<br>\n    for col in df_copy.columns:<br>\n        if df_copy[col].dtype == 'object':<br>\n            df_copy[col] = df_copy[col].astype('category')<br>\n        elif df_copy[col].dtype == 'float64':<br>\n            df_copy[col] = df_copy[col].astype('float32')<br>\n        elif df_copy[col].dtype == 'int64':<br>\n            df_copy[col] = df_copy[col].astype('int32')<br>\n    return df_copy</p>\n<h1>Función para reemplazar nulos con valores por defecto</h1>\n<p>def replace_nulls_with_default(df, float_default=-1.0, object_default=\"Missing\"):<br>\n    df_copy = df.copy()<br>\n    for col in df_copy.select_dtypes(include=['float', 'int']).columns:<br>\n        df_copy.loc[:, col] = df_copy[col].replace(\"\", float_default).fillna(float_default)</p>\n<pre><code> col  df_copy(include=):\n     df_copy == :\n         object_default not  df_copy:\n            df_copy = df_copy(object_default)\n    df_copy = df_copy(, object_default)(object_default)\n\nreturn df_copy\n</code></pre>\n<h1>Función para calcular la probabilidad de supervivencia</h1>\n<p>def transform_survival_probability(df, time_col='efs_time', event_col='efs'):<br>\n    kmf = KaplanMeierFitter()<br>\n    kmf.fit(df[time_col], event_observed=df[event_col])<br>\n    survival_probabilities = kmf.survival_function_at_times(df[time_col]).values.flatten()<br>\n    return survival_probabilities</p>\n<h1>Función para calcular el índice C estratificado (corregida)</h1>\n<p>def stratified_c_index(y_true, y_pred, groups):<br>\n    unique_groups = np.unique(groups)<br>\n    c_indices = []</p>\n<pre><code> group  unique_groups:\n    mask = groups == group\n    y_true_group = y_true[mask]\n    y_pred_group = y_pred[mask]\n\n     len(y_true_group) &lt; :\n        continue\n\n    pairs = .(list(combinations((len(y_true_group)), )))\n    y_true_pairs = y_true_group[pairs]\n    y_pred_pairs = y_pred_group[pairs]\n\n    true_diff = y_true_pairs[:, ] - y_true_pairs[:, ]\n    pred_diff = y_pred_pairs[:, ] - y_pred_pairs[:, ]\n\n    permissible = true_diff != \n    concordant = (true_diff * pred_diff) &gt; \n\n    c_index = .(concordant &amp; permissible) / .(permissible)\n    c_indices.(c_index)\n\n .(c_indices) - .(c_indices)\n</code></pre>\n<p>def custom_scorer(estimator, X, y):<br>\n    y_pred = estimator.predict(X)<br>\n    return stratified_c_index(y.values, y_pred, race_groups.values[X.index])</p>\n<h1>Optimizar el uso de memoria</h1>\n<p>df_tr = optimize_dtypes(df_tr)<br>\ndf_ts = optimize_dtypes(df_ts)</p>\n<h1>Reemplazar nulos con valores por defecto</h1>\n<p>df_tr = replace_nulls_with_default(df_tr)<br>\ndf_ts = replace_nulls_with_default(df_ts)</p>\n<h1>Calcular y añadir la columna objetivo</h1>\n<p>df_tr[\"target\"] = np.sqrt(transform_survival_probability(df_tr, time_col='efs_time', event_col='efs'))</p>\n<h1>Eliminar columnas irrelevantes</h1>\n<p>drop_cols = [\"ID\", 'efs', 'efs_time']<br>\ndf_tr = df_tr.drop(columns=[col for col in drop_cols if col in df_tr.columns])<br>\ndf_ts = df_ts.drop(columns=[col for col in drop_cols if col in df_ts.columns])</p>\n<h1>Preparar los datos para el modelo</h1>\n<p>X = df_tr.drop(columns=['target'], axis=1)<br>\ny = df_tr['target']<br>\nrace_groups = df_tr['race_group']</p>\n<h1>Identificar columnas categóricas</h1>\n<p>cat_features = list(X.select_dtypes(include=['object', 'category']).columns)</p>\n<h1>Convertir columnas categóricas a tipo string</h1>\n<p>for col in cat_features:<br>\n    X[col] = X[col].astype(str)<br>\n    df_ts[col] = df_ts[col].astype(str)</p>\n<h1>Aplicar PCA</h1>\n<p>pca = PCA(n_components=10)<br>\nX_pca = pca.fit_transform(X.select_dtypes(include=['float32', 'int32']))</p>\n<h1>Aplicar t-SNE</h1>\n<p>tsne = TSNE(n_components=2, random_state=42)<br>\nX_tsne = tsne.fit_transform(X.select_dtypes(include=['float32', 'int32']))</p>\n<h1>Añadir las nuevas características al DataFrame original</h1>\n<p>X_new = pd.concat([X, <br>\n                   pd.DataFrame(X_pca, columns=[f'PCA_{i}' for i in range(10)], index=X.index),<br>\n                   pd.DataFrame(X_tsne, columns=['TSNE_1', 'TSNE_2'], index=X.index)], <br>\n                   axis=1)</p>\n<h1>Actualizar la lista de características categóricas</h1>\n<p>cat_features = list(X_new.select_dtypes(include=['object', 'category']).columns)</p>\n<h1>Configurar validación cruzada con KFold</h1>\n<p>kf = KFold(n_splits=5, shuffle=True, random_state=42)</p>\n<h1>Función objetivo para la búsqueda amplia (más agresiva)</h1>\n<p>def objective_wide(trial):<br>\n    params = {<br>\n        'iterations': trial.suggest_int('iterations', 100, 10000),<br>\n        'learning_rate': trial.suggest_float('learning_rate', 1e-5, 1.0, log=True),<br>\n        'depth': trial.suggest_int('depth', 2, 16),<br>\n        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1e-10, 100.0, log=True),<br>\n        'border_count': trial.suggest_categorical('border_count', [32, 64, 128, 255, 512]),<br>\n        'bagging_temperature': trial.suggest_float('bagging_temperature', 0, 2),<br>\n        'random_strength': trial.suggest_float('random_strength', 1e-10, 100.0, log=True),<br>\n        'loss_function': 'RMSE',<br>\n        'eval_metric': 'RMSE',<br>\n        'random_seed': 42,<br>\n        'task_type': 'GPU',<br>\n        'devices': '0,1',<br>\n        'verbose': False,<br>\n        'cat_features': cat_features,<br>\n        'used_ram_limit': '10gb',<br>\n        'allow_writing_files': False<br>\n    }</p>\n<pre><code>model = CatBoostRegressor(**params)\n\n:\n    score = cross_val_score(model, X_new, y, cv=kf, scoring=custom_scorer, error_score=).mean()\n    ()\n     score\n Exception  e:\n    ()\n     ()\n</code></pre>\n<h1>Crear el estudio de Optuna y optimizar (búsqueda amplia)</h1>\n<p>study_wide = optuna.create_study(direction='maximize')</p>\n<h1>Usar parallel_backend aquí</h1>\n<p>with parallel_backend('loky', n_jobs=-1):<br>\n    study_wide.optimize(objective_wide, n_trials=1000, timeout=14400)  # 4 horas de tiempo límite</p>\n<h1>Mejores parámetros y puntuación de la búsqueda amplia</h1>\n<p>best_params_wide = study_wide.best_params<br>\nprint('Mejores parámetros (búsqueda amplia):', best_params_wide)<br>\nprint('Mejor puntuación (búsqueda amplia):', study_wide.best_value)</p>\n<h1>Guardar los mejores parámetros de la búsqueda amplia</h1>\n<p>np.save('best_params_wide.npy', best_params_wide)</p>\n<h1>Mejores parámetros y puntuación de la búsqueda amplia</h1>\n<p>best_params_wide = study_wide.best_params<br>\nprint('Mejores parámetros (búsqueda amplia):', best_params_wide)<br>\nprint('Mejor puntuación (búsqueda amplia):', study_wide.best_value)</p>\n<h1>Definir el espacio de búsqueda fina (más agresiva)</h1>\n<p>def objective_fine(trial):<br>\n    params = {<br>\n        'iterations': trial.suggest_int('iterations', max(100, int(best_params_wide['iterations']<em>0.5)), min(15000, int(best_params_wide['iterations']</em>1.5))),<br>\n        'learning_rate': trial.suggest_float('learning_rate', max(1e-6, best_params_wide['learning_rate']<em>0.5), min(1.0, best_params_wide['learning_rate']</em>1.5), log=True),<br>\n        'depth': trial.suggest_int('depth', max(1, best_params_wide['depth']-3), min(20, best_params_wide['depth']+3)),<br>\n        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', max(1e-11, best_params_wide['l2_leaf_reg']<em>0.5), min(200.0, best_params_wide['l2_leaf_reg']</em>1.5), log=True),<br>\n        'border_count': trial.suggest_categorical('border_count', [32, 64, 128, 255, 512]),<br>\n        'bagging_temperature': trial.suggest_float('bagging_temperature', max(0, best_params_wide['bagging_temperature']<em>0.5), min(3, best_params_wide['bagging_temperature']</em>1.5)),<br>\n        'random_strength': trial.suggest_float('random_strength', max(1e-11, best_params_wide['random_strength']<em>0.5), min(200.0, best_params_wide['random_strength']</em>1.5), log=True),<br>\n        'loss_function': 'RMSE',<br>\n        'eval_metric': 'RMSE',<br>\n        'random_seed': 42,<br>\n        'task_type': 'GPU',<br>\n        'devices': '0,1',<br>\n        'verbose': False,<br>\n        'cat_features': cat_features,<br>\n        'used_ram_limit': '10gb',<br>\n        'allow_writing_files': False<br>\n    }</p>\n<pre><code>model = CatBoostRegressor(**params)\n\n:\n    score = cross_val_score(model, X_new, y, cv=kf, scoring=custom_scorer, error_score=).mean()\n    ()\n     score\n Exception  e:\n    ()\n     ()\n</code></pre>\n<h1>Crear el estudio de Optuna y optimizar (búsqueda fina)</h1>\n<p>study_fine = optuna.create_study(direction='maximize')</p>\n<h1>Usar parallel_backend aquí</h1>\n<p>with parallel_backend('loky', n_jobs=-1):<br>\n    study_fine.optimize(objective_fine, n_trials=400, timeout=10400)  # 3 hora de tiempo límite</p>\n<h1>Mejores parámetros y puntuación de la búsqueda fina</h1>\n<p>best_params_fine = study_fine.best_params<br>\nprint('Mejores parámetros (búsqueda fina):', best_params_fine)<br>\nprint('Mejor puntuación (búsqueda fina):', study_fine.best_value)</p>\n<h1>Comparar resultados</h1>\n<p>print('Mejora en la puntuación:', study_fine.best_value - study_wide.best_value)</p>\n<h1>Comparar resultados y seleccionar los mejores parámetros</h1>\n<p>if study_fine.best_value &gt; study_wide.best_value:<br>\n    print(\"La búsqueda fina mejoró los resultados.\")<br>\n    best_params = best_params_fine<br>\nelse:<br>\n    print(\"La búsqueda amplia dio mejores resultados.\")<br>\n    best_params = best_params_wide</p>\n<h1>Guardar los mejores parámetros finales</h1>\n<p>np.save('best_params.npy', best_params)</p>\n<p>print('Mejores parámetros finales:', best_params)<br>\nprint('Mejor puntuación final:', max(study_fine.best_value, study_wide.best_value))</p>\n<h1>Intentar cargar los mejores parámetros, si el archivo existe</h1>\n<p>if os.path.exists('best_params.npy'):<br>\n    best_params = np.load('best_params.npy', allow_pickle=True).item()<br>\nelse:<br>\n    print(\"Advertencia: No se encontró el archivo 'best_params.npy'. Usando los parámetros en memoria.\")</p>\n<h1>Initialize variables for predictions and scores</h1>\n<p>final_predictions = np.zeros(len(X))<br>\nfold_scores = []</p>\n<p>for fold, (train_idx, val_idx) in enumerate(kf.split(X, race_groups)):<br>\n    print(f\"Fold {fold + 1}\")<br>\n    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]<br>\n    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]<br>\n    race_val = race_groups.iloc[val_idx]</p>\n<pre><code>\nmodel = CatBoostRegressor(\n    *st_params,\n    =42,\n    =,\n    =,  # Usa GPUs en serie\n    =,\n    =cat_features,\n    =0.8,\n    =\n)\n\n\nmodel.fit(\n    X_train, y_train,\n    eval_set=(X_val, y_val),\n    =50,\n    =\n)\n\n\ny_val_pred = model.predict(X_val)\nfold_score = stratified_c_index(y_val.values, y_val_pred, race_val.values)\nfold_scores.append(fold_score)\n(f)\n\nfinal_predictions[val_idx] = y_val_pred\n</code></pre>\n<h1>Overall Stratified Concordance Index</h1>\n<p>overall_score = stratified_c_index(y.values, final_predictions, race_groups.values)<br>\nprint(f\"Overall Stratified C-Index: {overall_score}\")</p>\n<h1>Calculate standard C-index</h1>\n<p>c_index = concordance_index(y, final_predictions)<br>\nprint(f\"Standard C-Index: {c_index}\")</p>\n<h1>Plot the scores per fold</h1>\n<p>plt.figure(figsize=(10, 6))<br>\nplt.plot(range(1, len(fold_scores) + 1), fold_scores, marker='o', linestyle='-', color='b')<br>\nplt.title('Stratified C-Index by Fold')<br>\nplt.xlabel('Fold')<br>\nplt.ylabel('Stratified C-Index')<br>\nplt.xticks(range(1, len(fold_scores) + 1))<br>\nplt.grid()<br>\nplt.show()</p>\n<h1>Plot predicted vs actual values</h1>\n<p>plt.figure(figsize=(10, 6))<br>\nplt.scatter(y, final_predictions, alpha=0.5)<br>\nplt.plot([y.min(), y.max()], [y.min(), y.max()], 'r--', lw=2)<br>\nplt.xlabel('Actual Values')<br>\nplt.ylabel('Predicted Values')<br>\nplt.title('Predicted vs Actual Values')<br>\nplt.tight_layout()<br>\nplt.show()</p>",
          "votes": -6,
          "replies": [
            {
              "id": 3144755,
              "author_name": "Palbha",
              "author_url": "",
              "post_date": "2025-03-08T20:18:57.187000",
              "content": "<p><a href=\"https://www.kaggle.com/code/palbha/cibmtr-replicate-1st-prize-solution\" target=\"_blank\">https://www.kaggle.com/code/palbha/cibmtr-replicate-1st-prize-solution</a> - I have created this notebook as per what you shared - patiently waiting for it to run now :) .. Thanks a lot for sharing !! </p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3143820,
      "author_name": "minerppdy",
      "author_url": "",
      "post_date": "2025-03-07T16:30:19.837000",
      "content": "<p><a href=\"https://www.kaggle.com/omararielvallejos\" target=\"_blank\">@omararielvallejos</a> i saw your posted code, what's 0.74 mean? and how it comes?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3143809,
      "author_name": "Carlos Musungu",
      "author_url": "",
      "post_date": "2025-03-07T16:16:39.230000",
      "content": "<p>Wow, great work!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3142830,
      "author_name": "Andreas Bisiadis",
      "author_url": "",
      "post_date": "2025-03-06T17:34:30.647000",
      "content": "<blockquote>\n  <p>Add efs as a feature when training and focus on the performance at samples where efs==1. Set efs=1 when making inference on LB data. </p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> Do you use the actual <code>efs</code> target (retaining both 0 and 1 values) as feature for training and set efs=1 for all rows during inference?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3142845,
          "author_name": "minerppdy",
          "author_url": "",
          "post_date": "2025-03-06T17:51:30.210000",
          "content": "<p><a href=\"https://www.kaggle.com/andreasbis\" target=\"_blank\">@andreasbis</a> yes, the purpose is to improve the performance on efs=1 using the potencial information of efs=0, and meanwhile make sure the model wouldn't confused by mixing different samples together.  without this feature, the result is really bad</p>",
          "votes": 1,
          "replies": [
            {
              "id": 3142961,
              "author_name": "Robert Hatch",
              "author_url": "",
              "post_date": "2025-03-06T19:35:31.300000",
              "content": "<p>Huh, and from code you shared you had efs==0 as it's own separate uniform 0-1 ranking? Interesting. If that even mattered I wonder why it works. I suspect there are competing nuances to the impact of your exact implementation, but I don't think I have fully grasped it at a theoretical level yet. </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3143178,
              "author_name": "minerppdy",
              "author_url": "",
              "post_date": "2025-03-07T02:24:32.910000",
              "content": "<p><a href=\"https://www.kaggle.com/roberthatch\" target=\"_blank\">@roberthatch</a> I have tried several kinds of transform,but didn't find impressive difference, for example:</p>\n<pre><code>efs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  \nefs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  +\n</code></pre>\n<pre><code>efs_time_norm = efs_time.rank() / (efs)\n</code></pre>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3142778,
      "author_name": "Fahim Sarker",
      "author_url": "",
      "post_date": "2025-03-06T16:54:54.977000",
      "content": "<p>Congratulations,you did a great and amazing work</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3142556,
      "author_name": "LearNerZhou99",
      "author_url": "",
      "post_date": "2025-03-06T13:24:01.497000",
      "content": "<p>appreciate for ur share!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3142546,
      "author_name": "rai.two",
      "author_url": "",
      "post_date": "2025-03-06T13:17:03.380000",
      "content": "<p>Great solution, congratulations!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3142459,
      "author_name": "Joris",
      "author_url": "",
      "post_date": "2025-03-06T12:10:23.500000",
      "content": "<p>Nice, good job!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3142289,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-06T09:18:35.603000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3142027,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-06T03:30:44.580000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3143791,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-07T15:49:18.403000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 3143822,
          "author_name": "",
          "author_url": "",
          "post_date": "2025-03-07T16:32:31.680000",
          "content": "",
          "votes": 0,
          "replies": [
            {
              "id": 3146021,
              "author_name": "",
              "author_url": "",
              "post_date": "2025-03-10T13:17:55.640000",
              "content": "",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3142267,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-06T08:50:04.523000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3141952,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-06T01:55:42.903000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3141958,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-06T02:08:46.463000",
      "content": "",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3146478,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-10T23:40:27.923000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3145227,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-03-09T14:40:51.623000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 59,
  "messages": [
    {
      "id": 3141941,
      "postDate": "2025-03-06T01:48:07.920Z",
      "content": "<p>First, congratulations to the teams that finally \"survived\" in this competition, and thank you to the participants who shared their experiences and provided help in the forum.</p>\n<p>I also trained classifier and regressor models independently and then combined them together with a magic function, just like 2nd and 4th place solution. The main idea is that when a patient has a high probability of <code>efs == 0</code>, we should give them a high score. Otherwise, we should give them a relatively lower score and pay more attention to the rank of <code>efs_time</code>. So the regressor just needs to focus on the ranking task where <code>efs == 1</code>.</p>\n<p>Here is the simplified <a href=\"https://www.kaggle.com/code/minerppdy/1st-place-simplified-private-lb-0-700\" target=\"_blank\">code</a>.</p>\n<h2>1. Feature Engineering</h2>\n<ol>\n<li>Original features  </li>\n<li>One-hot encoding for all categorical features(but still keep the original category features)  </li>\n<li>Copy continuous features as categorical features  </li>\n</ol>\n<h2>2. CV Strategy</h2>\n<ol>\n<li>The CV is very unstable with different splits Strategy. To make the results comparable, I used 10-fold splits with same seeds for all classifier and regressor model training.</li>\n</ol>\n<pre><code>y_combine = data[].astype()++data[][].astype()\nskf = StratifiedKFold(n_splits=, shuffle=, random_state=)\nskf.split(X,y_combine)\n</code></pre>\n<ol>\n<li>without use early stoping to reduce the risk of overfitting, especially when I use a large number of folds.</li>\n</ol>\n<h2>3. Classifier</h2>\n<h3>Target: <code>P(efs=0)</code></h3>\n<h3>Models Used</h3>\n<ol>\n<li>XGBoost, LightGBM, CatBoost  </li>\n<li>NN, TabM</li>\n<li>GNN  </li>\n<li>NN/TabNet/GNN with pairwise-rank-loss</li>\n</ol>\n<h3>tricks</h3>\n<ol>\n<li>Gnn method. I use KNN to find nearest 25 nodes and  create edges(use Euclidean Distance),then use graphsage to fit the targets.</li>\n<li>when use rank loss, the predition would shift over different fold because rank loss is not sensitive to shift, which may cause lower CV when calculate metric in the entire set. So you need to fix these shifts to keep same mean/mid prediction value over models trained in differece fold.</li>\n<li>GNN also have this problem even if I used logloss，I still don't know why.</li>\n</ol>\n<h3>auc metrics</h3>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>auc</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>lightgbm</td>\n<td>0.7603</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>0.7617</td>\n</tr>\n<tr>\n<td>xgboost</td>\n<td>0.7606</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>0.75902</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>0.7596</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>0.7591</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>0.7598</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>0.7597</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>0.7586</td>\n</tr>\n</tbody>\n</table>\n<h3>interesting finding</h3>\n<ol>\n<li>The best max depth for LGB and XGB is 2 (for CatBoost, it's 6, and I still don't know why). This is nearly unheard of in my experience. It means there is little valuable feature interaction information, and the task is easy to fit. Maybe that's why NN like models also work well in this tabular data.</li>\n</ol>\n<h2>4. Regressor</h2>\n<h3>Target</h3>\n<p>Grouped and normalized rank by <code>efs</code>  </p>\n<pre><code>efs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  \nefs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  \n</code></pre>\n<h3>Models used</h3>\n<p>XGBoost lightgbm catboost( NN like model didn't work well for me, if any one succeed in NN like model, please let me know.)</p>\n<h3>Tricks</h3>\n<ol>\n<li>Add efs as a feature when training and focus on the performance at samples where <code>efs==1</code>. Set <code>efs=1</code> when making inference on LB data. </li>\n<li>apply sample weight on efs==1 and efs==0 with 0.6:0.4</li>\n<li>Here's why this trick is used. Initially, I trained the regressor only on samples where <code>efs==1</code>. When using this model to make inference on samples where <code>efs==0</code>, it still showed an obvious correlation between prediction and ground truth. This is strange because efs_time for <code>efs==0</code> is meaningless due to various reasons could cause data censoring. I guess it's because of the SurvalGAN algorithm. By the way, it turns out that there are similar patterns between samples where <code>efs==0</code> and <code>efs==1</code>. So, by adding samples where <code>efs==0</code>, the regressor's performance on <code>efs==1</code> improved significantly!</li>\n</ol>\n<h3>Raw Concordance Index on efs==1</h3>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>c-index</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>xgboost</td>\n<td>0.770229</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>0.767615</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>0.769340</td>\n</tr>\n</tbody>\n</table>\n<h2>5.Merge Function</h2>\n<pre><code> ():\n    \n    y_fun = (Y_HAT_REG&gt;)*c*(np.(Y_HAT_REG))**(b)\n    x_fun =(Y_HAT_CLS&gt;)*(np.(Y_HAT_CLS))**(a)\n    res = (-y_fun)*x_fun+y_fun\n    res = pd.Series(res).rank()/(res)\n     res\n</code></pre>\n<p>This is what the merge function looks like(without rank transform), where x is the probablity of efs==0 and y is predicted efs_time(scaled to 0~1) <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F24200812%2F9121e2fc16b8f0cc18f16c9d8e363608%2Fmerge_function.jpg?generation=1741316566114375&amp;alt=media\" alt=\"\"></p>\n<h2>6 Ensembling Method</h2>\n<ol>\n<li><p>Create combinations between classifiers and regressors, then find the best (a, b, c) for the merge function by with 5-folds and then get the merged predictions.</p>\n<p>Here is Stratified Concordance Index of combinations. Ignoring the rank transform in merge function, pretty sure there is no leakage</p>\n<table>\n<thead>\n<tr>\n<th>cls</th>\n<th>reg</th>\n<th>CV Stratified c-index</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>xgboost</td>\n<td>xgboost</td>\n<td>0.6938821198444668</td>\n</tr>\n<tr>\n<td>xgboost</td>\n<td>catboost</td>\n<td>0.693390761259049</td>\n</tr>\n<tr>\n<td>xgboost</td>\n<td>lightgbm</td>\n<td>0.6920553689042972</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>xgboost</td>\n<td>0.6946771227057665</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>catboost</td>\n<td>0.6942415329389815</td>\n</tr>\n<tr>\n<td>catboost</td>\n<td>lightgbm</td>\n<td>0.6928675187239672</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>xgboost</td>\n<td>0.6935306031275579</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>catboost</td>\n<td>0.693123814234674</td>\n</tr>\n<tr>\n<td>lightgbm</td>\n<td>lightgbm</td>\n<td>0.6917244174625249</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>xgboost</td>\n<td>0.6943607674628444</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>catboost</td>\n<td>0.6936616324722884</td>\n</tr>\n<tr>\n<td>tabm</td>\n<td>lightgbm</td>\n<td>0.6922428015362854</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>xgboost</td>\n<td>0.6939542378441973</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>catboost</td>\n<td>0.6932931191267933</td>\n</tr>\n<tr>\n<td>nn</td>\n<td>lightgbm</td>\n<td>0.6920700218390199</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>xgboost</td>\n<td>0.6940062476514388</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>catboost</td>\n<td>0.6932012736212297</td>\n</tr>\n<tr>\n<td>gnn</td>\n<td>lightgbm</td>\n<td>0.6921340326054011</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>xgboost</td>\n<td>0.693141398733159</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>catboost</td>\n<td>0.6927626179572159</td>\n</tr>\n<tr>\n<td>ranktabm</td>\n<td>lightgbm</td>\n<td>0.6912730733179469</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>xgboost</td>\n<td>0.6944776101013886</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>catboost</td>\n<td>0.6936542865352446</td>\n</tr>\n<tr>\n<td>ranknn</td>\n<td>lightgbm</td>\n<td>0.6924528052605144</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>xgboost</td>\n<td>0.6934854451860402</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>catboost</td>\n<td>0.692783734834767</td>\n</tr>\n<tr>\n<td>rankgnn</td>\n<td>lightgbm</td>\n<td>0.6916329456191074</td>\n</tr>\n</tbody>\n</table></li>\n<li><p>Ensemble the merged predictions using a weighted average with Optuna and 5-folds.</p></li>\n<li><p>Since there are 9 classifiers × 3 regressors = 27 combinations, there is a risk of overfitting. Therefore, I set the Optuna search range for weights to be between 0.1 and 1, rather than 0 and 1. I think this is a form of regularization.</p></li>\n<li><p>The  final CV Stratified c-index is round 0.6965</p></li>\n<li><p>Adding noise to some race groups significantly improved cross-validation performance, but it didn't work on either the public or private LB.</p></li>\n</ol>",
      "rawMarkdown": "First, congratulations to the teams that finally \"survived\" in this competition, and thank you to the participants who shared their experiences and provided help in the forum.\n\nI also trained classifier and regressor models independently and then combined them together with a magic function, just like 2nd and 4th place solution. The main idea is that when a patient has a high probability of `efs == 0`, we should give them a high score. Otherwise, we should give them a relatively lower score and pay more attention to the rank of `efs_time`. So the regressor just needs to focus on the ranking task where `efs == 1`.\n\nHere is the simplified [code](https://www.kaggle.com/code/minerppdy/1st-place-simplified-private-lb-0-700).\n\n## 1. Feature Engineering\n1. Original features  \n2. One-hot encoding for all categorical features(but still keep the original category features)  \n3. Copy continuous features as categorical features  \n\n## 2. CV Strategy\n1. The CV is very unstable with different splits Strategy. To make the results comparable, I used 10-fold splits with same seeds for all classifier and regressor model training.\n```python\ny_combine = data['efs'].astype('str')+'|'+data['X']['race_group'].astype('str')\nskf = StratifiedKFold(n_splits=10, shuffle=True, random_state=888)\nskf.split(X,y_combine)\n```\n2. without use early stoping to reduce the risk of overfitting, especially when I use a large number of folds.\n\n\n\n## 3. Classifier\n### Target: `P(efs=0)`\n### Models Used  \n1. XGBoost, LightGBM, CatBoost  \n2. NN, TabM\n3. GNN  \n4. NN/TabNet/GNN with pairwise-rank-loss\n### tricks\n1. Gnn method. I use KNN to find nearest 25 nodes and  create edges(use Euclidean Distance),then use graphsage to fit the targets.\n2. when use rank loss, the predition would shift over different fold because rank loss is not sensitive to shift, which may cause lower CV when calculate metric in the entire set. So you need to fix these shifts to keep same mean/mid prediction value over models trained in differece fold.\n3. GNN also have this problem even if I used logloss，I still don't know why.\n### auc metrics\n\n|  model| auc |\n| --- | --- |\n|lightgbm | 0.7603  |\n|catboost| 0.7617  |\n|xgboost|  0.7606  |\n|nn| 0.75902  |\n|tabm| 0.7596  |\n|gnn|  0.7591  |\n|ranknn| 0.7598  |\n|ranktabm| 0.7597  |\n|rankgnn| 0.7586  |\n### interesting finding\n1. The best max depth for LGB and XGB is 2 (for CatBoost, it's 6, and I still don't know why). This is nearly unheard of in my experience. It means there is little valuable feature interaction information, and the task is easy to fit. Maybe that's why NN like models also work well in this tabular data.\n## 4. Regressor\n### Target  \nGrouped and normalized rank by `efs`  \n```python\nefs_time_norm[efs == 1] = efs_time[efs == 1].rank() / sum(efs == 1)  \nefs_time_norm[efs == 0] = efs_time[efs == 0].rank() / sum(efs == 0)  \n```\n### Models used\nXGBoost lightgbm catboost( NN like model didn't work well for me, if any one succeed in NN like model, please let me know.)\n### Tricks\n1. Add efs as a feature when training and focus on the performance at samples where `efs==1`. Set `efs=1` when making inference on LB data. \n2. apply sample weight on efs==1 and efs==0 with 0.6:0.4\n3. Here's why this trick is used. Initially, I trained the regressor only on samples where `efs==1`. When using this model to make inference on samples where `efs==0`, it still showed an obvious correlation between prediction and ground truth. This is strange because efs_time for `efs==0` is meaningless due to various reasons could cause data censoring. I guess it's because of the SurvalGAN algorithm. By the way, it turns out that there are similar patterns between samples where `efs==0` and `efs==1`. So, by adding samples where `efs==0`, the regressor's performance on `efs==1` improved significantly!\n\n### Raw Concordance Index on efs==1\n|  model| c-index |\n| --- | --- |\n|xgboost|0.770229|\n|lightgbm|0.767615|\n|catboost|0.769340|\n\n\n## 5.Merge Function\n```python\ndef model_merge(Y_HAT_REG,Y_HAT_CLS,a=2.96,b=1.77,c=0.52):\n    '''\n    Y_HAT_REG and Y_HAT_CLS need be scaled to 0~1\n    a,b,c need to be tuned with optuna \n    '''\n    y_fun = (Y_HAT_REG>0)*c*(np.abs(Y_HAT_REG))**(b)\n    x_fun =(Y_HAT_CLS>0)*(np.abs(Y_HAT_CLS))**(a)\n    res = (1-y_fun)*x_fun+y_fun\n    res = pd.Series(res).rank()/len(res)\n    return res\n```\nThis is what the merge function looks like(without rank transform), where x is the probablity of efs==0 and y is predicted efs_time(scaled to 0~1) \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F24200812%2F9121e2fc16b8f0cc18f16c9d8e363608%2Fmerge_function.jpg?generation=1741316566114375&alt=media)\n\n## 6 Ensembling Method\n1. Create combinations between classifiers and regressors, then find the best (a, b, c) for the merge function by with 5-folds and then get the merged predictions.\n\n    Here is Stratified Concordance Index of combinations. Ignoring the rank transform in merge function, pretty sure there is no leakage\n\n    |cls|reg|CV Stratified c-index|\n    |---|---|---|\n    |xgboost|xgboost | 0.6938821198444668    |\n    |xgboost|catboost|0.693390761259049|\n    |xgboost|lightgbm|0.6920553689042972|\n    |catboost|xgboost|0.6946771227057665|\n    |catboost|catboost|0.6942415329389815|\n    |catboost|lightgbm|0.6928675187239672|\n    |lightgbm|xgboost|0.6935306031275579|\n    |lightgbm|catboost|0.693123814234674|\n    |lightgbm|lightgbm|0.6917244174625249|\n    |tabm|xgboost|0.6943607674628444|\n    |tabm|catboost|0.6936616324722884|\n    |tabm|lightgbm|0.6922428015362854|\n    |nn|xgboost|0.6939542378441973|\n    |nn|catboost|0.6932931191267933|\n    |nn|lightgbm|0.6920700218390199|\n    |gnn|xgboost|0.6940062476514388|\n    |gnn|catboost|0.6932012736212297|\n    |gnn|lightgbm|0.6921340326054011|\n    |ranktabm|xgboost|0.693141398733159|\n    |ranktabm|catboost|0.6927626179572159|\n    |ranktabm|lightgbm|0.6912730733179469|\n    |ranknn|xgboost|0.6944776101013886|\n    |ranknn|catboost|0.6936542865352446|\n    |ranknn|lightgbm|0.6924528052605144|\n    |rankgnn|xgboost|0.6934854451860402|\n    |rankgnn|catboost|0.692783734834767|\n    |rankgnn|lightgbm|0.6916329456191074|\n\n2. Ensemble the merged predictions using a weighted average with Optuna and 5-folds.\n3. Since there are 9 classifiers × 3 regressors = 27 combinations, there is a risk of overfitting. Therefore, I set the Optuna search range for weights to be between 0.1 and 1, rather than 0 and 1. I think this is a form of regularization.\n4.  The  final CV Stratified c-index is round 0.6965\n5.  Adding noise to some race groups significantly improved cross-validation performance, but it didn't work on either the public or private LB.\n\n\n\n\n",
      "votes": 186
    },
    {
      "id": 3142300,
      "postDate": "2025-03-06T09:25:11.360Z",
      "content": "<p>Great workl! <br>\nWe have similiar auc score, but your regression models are much better than mine, my final cv is around 0.693~</p>",
      "rawMarkdown": "Great workl! \nWe have similiar auc score, but your regression models are much better than mine, my final cv is around 0.693~",
      "votes": 5
    },
    {
      "id": 3142003,
      "postDate": "2025-03-06T03:12:10.550Z",
      "content": "<p>Congratulations! 🎉 You were leading at the top from start to finish, and it was truly amazing!</p>",
      "rawMarkdown": "Congratulations! 🎉 You were leading at the top from start to finish, and it was truly amazing!",
      "votes": 3
    },
    {
      "id": 3149530,
      "postDate": "2025-03-14T10:41:28.767Z",
      "content": "<p><a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> in one of the old posts you mentioned feature engineering, did you keep them in your final sub?<br>\nAnd, do you mind sharing your background? - Your acc is fresh and you were top1 since the very beginning, just curious</p>",
      "rawMarkdown": "@minerppdy in one of the old posts you mentioned feature engineering, did you keep them in your final sub?\nAnd, do you mind sharing your background? - Your acc is fresh and you were top1 since the very beginning, just curious",
      "votes": 1,
      "replies": [
        {
          "id": 3149563,
          "postDate": "2025-03-14T11:37:45.990Z",
          "content": "<p>I have posted the simplified <a href=\"https://www.kaggle.com/code/minerppdy/1st-place-simplified-private-lb-0-700\" target=\"_blank\">code</a> and the feature engineering is almost the same as in my final submission. I registered on Kaggle years ago, but recently when I returned to Kaggle, I couldn't verify my account with Persona, so I had to delete the account and register again…</p>",
          "rawMarkdown": "I have posted the simplified [code](https://www.kaggle.com/code/minerppdy/1st-place-simplified-private-lb-0-700) and the feature engineering is almost the same as in my final submission. I registered on Kaggle years ago, but recently when I returned to Kaggle, I couldn't verify my account with Persona, so I had to delete the account and register again...",
          "votes": 1
        }
      ]
    },
    {
      "id": 3143819,
      "postDate": "2025-03-07T16:29:18.060Z",
      "content": "<p>Congrats! Very nice approach</p>",
      "rawMarkdown": "Congrats! Very nice approach",
      "votes": 2,
      "replies": [
        {
          "id": 3145130,
          "postDate": "2025-03-09T11:49:13.680Z",
          "content": "<p>Yes, i think the same, it is reale good</p>",
          "rawMarkdown": "Yes, i think the same, it is reale good",
          "votes": 1
        }
      ]
    },
    {
      "id": 3142732,
      "postDate": "2025-03-06T16:26:29.163Z",
      "content": "<p>Congrats on strong finish! I think you added a decent amount of diversity by ensembling combination of classifiers and regressors. I've never thought doing such thing because I was focused on finalizing the classifier part first. This made me go into tunnel vision and I  used only the same classifier blend with my regressors. I think you might push your score a little bit more by adding 3 more combinations of classifier blend x lightgbm, classifier blend x xgboost and classifier blend x catboost.</p>",
      "rawMarkdown": "Congrats on strong finish! I think you added a decent amount of diversity by ensembling combination of classifiers and regressors. I've never thought doing such thing because I was focused on finalizing the classifier part first. This made me go into tunnel vision and I  used only the same classifier blend with my regressors. I think you might push your score a little bit more by adding 3 more combinations of classifier blend x lightgbm, classifier blend x xgboost and classifier blend x catboost.",
      "votes": 2,
      "replies": [
        {
          "id": 3142790,
          "postDate": "2025-03-06T17:10:09.853Z",
          "content": "<p>i also tried blend cls and reg first, then apply the merge function, hard to say which is better</p>",
          "rawMarkdown": "i also tried blend cls and reg first, then apply the merge function, hard to say which is better",
          "replies": [
            {
              "id": 3142829,
              "postDate": "2025-03-06T17:32:26.607Z",
              "content": "<p>I had some strong hints that optimizing all together at once could be best. Or maybe better to say that I knew blend first couldn't search the entire optimization space. It was clear it was further from optimal then normal for effectively optimizing against this formula. </p>\n<p>Got clear evidence of that by stumbling upon an unusually amazing CV score that my normal ensembling method couldn't come close to finding.</p>\n<p>However, I didn't trust my individual model's CV and using a new ensembling method quite enough since I was still trying to hedge against a fear of overfitting CV. </p>\n<p>With the hundreds of base models I used, it could've been interesting to try the greedy ensemble selection I used to pare down the number, but then restart ensembling on the selected models with your Optuna method or something similar. </p>",
              "rawMarkdown": "I had some strong hints that optimizing all together at once could be best. Or maybe better to say that I knew blend first couldn't search the entire optimization space. It was clear it was further from optimal then normal for effectively optimizing against this formula. \n\nGot clear evidence of that by stumbling upon an unusually amazing CV score that my normal ensembling method couldn't come close to finding.\n\nHowever, I didn't trust my individual model's CV and using a new ensembling method quite enough since I was still trying to hedge against a fear of overfitting CV. \n\nWith the hundreds of base models I used, it could've been interesting to try the greedy ensemble selection I used to pare down the number, but then restart ensembling on the selected models with your Optuna method or something similar. ",
              "votes": -1
            }
          ]
        }
      ]
    },
    {
      "id": 3142717,
      "postDate": "2025-03-06T16:18:24.033Z",
      "content": "<p>Same approach to solve tasks independently then combined it, rank is all u need for ranking task. Cool find with addition of efs as feature, very very smart! I also experimented with some sort of stacking and target recreation using similar approach with probabilities of efs, but always it was overfitted! Congratz! </p>",
      "rawMarkdown": "Same approach to solve tasks independently then combined it, rank is all u need for ranking task. Cool find with addition of efs as feature, very very smart! I also experimented with some sort of stacking and target recreation using similar approach with probabilities of efs, but always it was overfitted! Congratz! ",
      "votes": 2,
      "replies": [
        {
          "id": 3142927,
          "postDate": "2025-03-06T18:54:37.587Z",
          "content": "<p>Something interesting and surprising I found was a (probably inferior) variant of efs as feature: because I tried the trick late and was lazy, I was predicting the smaller train set with hyperparameters tuned for a larger train set. Because of that and/or other unclear reasons, adding rest of train data with sample weight <em>0</em> was a big help to my model. I'm unsure the overall impact, which hyperparameters are affected by the data, or exactly why/how it helped. But it did. On both LGB and Catboost. </p>",
          "rawMarkdown": "Something interesting and surprising I found was a (probably inferior) variant of efs as feature: because I tried the trick late and was lazy, I was predicting the smaller train set with hyperparameters tuned for a larger train set. Because of that and/or other unclear reasons, adding rest of train data with sample weight *0* was a big help to my model. I'm unsure the overall impact, which hyperparameters are affected by the data, or exactly why/how it helped. But it did. On both LGB and Catboost. ",
          "votes": -1
        }
      ]
    },
    {
      "id": 3141992,
      "postDate": "2025-03-06T02:49:30.907Z",
      "content": "<p>Huge congrats! (1st during almost the whole competition too, was it?) Can you give more insight into how you tuned/trained the a,b,c constants of the merge function? Did you just find what maximized CV score? Were they part of the Optuna search? Or…?</p>\n<p>(I was really worried about overfitting if I just took the constants that maximized CV with my VERY similar function and postprocessing 'tricks'. So even on my winning gamble sub I went a little under the CV optimal constants)</p>",
      "rawMarkdown": "Huge congrats! (1st during almost the whole competition too, was it?) Can you give more insight into how you tuned/trained the a,b,c constants of the merge function? Did you just find what maximized CV score? Were they part of the Optuna search? Or...?\n\n(I was really worried about overfitting if I just took the constants that maximized CV with my VERY similar function and postprocessing 'tricks'. So even on my winning gamble sub I went a little under the CV optimal constants)",
      "replies": [
        {
          "id": 3142035,
          "postDate": "2025-03-06T03:39:34.957Z",
          "content": "<p><a href=\"https://www.kaggle.com/roberthatch\" target=\"_blank\">@roberthatch</a>  1. I used 5-fold validation to get  the reliable CV score   2. use optuna to find the best (a,b,c) to maximize C-index of train_set,  then get oof result and LB result with best (a,b,c), do this by 5 loops, then calucate the oof C-index   3. how ever this cant prevent overfitting if the form of merge function is not appropriate, just make sure you CV is reliable</p>",
          "rawMarkdown": "@roberthatch  1. I used 5-fold validation to get  the reliable CV score   2. use optuna to find the best (a,b,c) to maximize C-index of train_set,  then get oof result and LB result with best (a,b,c), do this by 5 loops, then calucate the oof C-index   3. how ever this cant prevent overfitting if the form of merge function is not appropriate, just make sure you CV is reliable",
          "replies": [
            {
              "id": 3142096,
              "postDate": "2025-03-06T05:01:20.673Z",
              "content": "<p>Thanks, that's about how I would've done it if not trying to be more careful. My formulation was a little different and I was worried the constants wouldn't translate to LB and would drastically increase any partial over fitting I already had. In part because my LB score was always a bit lower than expected from the first day I tried the two completely separate model trick. (I think my simple LGB two models I had a two targets blend with 0.682 to 0.693 CV  and around 0.688 LB to low 689 LB. With the trick my CV jumped to 0.685 to 686, but LB 0.687 to 0.689. It got much better with Cat boost and with more models, but there was still a concern whether LB score was noise or some partial inability to generalize. Impressive how perfectly your tuning methodology worked!</p>",
              "rawMarkdown": "Thanks, that's about how I would've done it if not trying to be more careful. My formulation was a little different and I was worried the constants wouldn't translate to LB and would drastically increase any partial over fitting I already had. In part because my LB score was always a bit lower than expected from the first day I tried the two completely separate model trick. (I think my simple LGB two models I had a two targets blend with 0.682 to 0.693 CV  and around 0.688 LB to low 689 LB. With the trick my CV jumped to 0.685 to 686, but LB 0.687 to 0.689. It got much better with Cat boost and with more models, but there was still a concern whether LB score was noise or some partial inability to generalize. Impressive how perfectly your tuning methodology worked!",
              "votes": -1
            },
            {
              "id": 3142097,
              "postDate": "2025-03-06T05:03:08.600Z",
              "content": "<p>My 5th place winning submission with postprocessing was only (low) 0.693 on public LB!</p>",
              "rawMarkdown": "My 5th place winning submission with postprocessing was only (low) 0.693 on public LB!",
              "votes": -1
            }
          ]
        },
        {
          "id": 3142040,
          "postDate": "2025-03-06T03:45:20.747Z",
          "content": "<p>I also tried NN with pair-wise-loss to combine all the classifiers and regressors, but it didn't work, the CV score is bad </p>",
          "rawMarkdown": "I also tried NN with pair-wise-loss to combine all the classifiers and regressors, but it didn't work, the CV score is bad ",
          "votes": 1
        }
      ]
    },
    {
      "id": 3175505,
      "postDate": "2025-04-10T08:09:58.850Z",
      "content": "<p>I am very thankful if <a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> answer this.<br>\nIn initBTree method there are following statements.</p>\n<p>last_full_row = int(np.log2(len(values) + 1) - 1)<br>\nlen_ragged_row = len(values) - (2 ** (last_full_row + 1) - 1)</p>\n<p>Where does these come from ?<br>\nWhy log2?</p>",
      "rawMarkdown": "I am very thankful if @minerppdy answer this.\nIn initBTree method there are following statements.\n\nlast_full_row = int(np.log2(len(values) + 1) - 1)\nlen_ragged_row = len(values) - (2 ** (last_full_row + 1) - 1)\n\nWhere does these come from ?\nWhy log2?"
    },
    {
      "id": 3167162,
      "postDate": "2025-04-01T10:14:06.877Z",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/minerpddy\" target=\"_blank\">@minerpddy</a> congratulations achieving the first place.<br>\nCan you please explain your choice for the magic function. How have you chosen this exact form and not any other? Especially this line here <code>res = (1-y_fun)*x_fun+y_fun</code></p>",
      "rawMarkdown": "Hi @minerpddy congratulations achieving the first place.\nCan you please explain your choice for the magic function. How have you chosen this exact form and not any other? Especially this line here `res = (1-y_fun)*x_fun+y_fun`",
      "replies": [
        {
          "id": 3167165,
          "postDate": "2025-04-01T10:15:53.833Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 3168395,
          "postDate": "2025-04-02T11:44:24.420Z",
          "content": "<p>yeap, thats really amazing and magic</p>",
          "rawMarkdown": "yeap, thats really amazing and magic"
        }
      ]
    },
    {
      "id": 3153161,
      "postDate": "2025-03-18T13:21:26.637Z",
      "content": "<p>Nice work, very interesting. I'd love to learn more about merge functions for blending classifier and regressors. I've seen many blending techniques for models of the same type, but here you are blending models of different types. Just wondering where you first learnt about the merge function? Or if you know of any others? Thanks in advance</p>",
      "rawMarkdown": "Nice work, very interesting. I'd love to learn more about merge functions for blending classifier and regressors. I've seen many blending techniques for models of the same type, but here you are blending models of different types. Just wondering where you first learnt about the merge function? Or if you know of any others? Thanks in advance"
    },
    {
      "id": 3152420,
      "postDate": "2025-03-17T19:31:04.287Z",
      "content": "<p>Thanks. I've learned a lot from your solution. Two targets, model combining, amazing</p>",
      "rawMarkdown": "Thanks. I've learned a lot from your solution. Two targets, model combining, amazing"
    },
    {
      "id": 3150352,
      "postDate": "2025-03-15T11:49:16.317Z",
      "content": "<p>Excellent result. Thank you for sharing your knowledge.</p>",
      "rawMarkdown": "Excellent result. Thank you for sharing your knowledge."
    },
    {
      "id": 3149398,
      "postDate": "2025-03-14T06:24:45.813Z",
      "content": "<p>great work did you use hurdle modeling here</p>",
      "rawMarkdown": "great work did you use hurdle modeling here",
      "replies": [
        {
          "id": 3149566,
          "postDate": "2025-03-14T11:39:42.663Z",
          "content": "<p>No, but I think the idea is similar.</p>",
          "rawMarkdown": "No, but I think the idea is similar.",
          "votes": 1,
          "replies": [
            {
              "id": 3149781,
              "postDate": "2025-03-14T17:10:24.127Z",
              "content": "<p>thank you if you donot mind i have one question i read all your notebook i understanded most of it just one thing  which is the magic function how you reached it i tried using just simple way like Y_hat_reg * a + Y_hat_clas * b the results was worse did you find it by chance or is it related  with the dataset it self and is it really necessary to pass efs why not just passing all the data without efs column or you want to make conditional training  for your regressor </p>",
              "rawMarkdown": "thank you if you donot mind i have one question i read all your notebook i understanded most of it just one thing  which is the magic function how you reached it i tried using just simple way like Y_hat_reg * a + Y_hat_clas * b the results was worse did you find it by chance or is it related  with the dataset it self and is it really necessary to pass efs why not just passing all the data without efs column or you want to make conditional training  for your regressor "
            },
            {
              "id": 3150076,
              "postDate": "2025-03-15T03:49:40.540Z",
              "content": "<ul>\n<li>The merge function is based on the understanding of C-index metric.</li>\n</ul>\n<blockquote>\n  <p>The main idea is that when a patient has a high probability of efs == 0, we should give them a high score. Otherwise, we should give them a relatively lower score and pay more attention to the rank of efs_time. So the regressor just needs to focus on the ranking task where efs == 1.  </p>\n</blockquote>\n<ul>\n<li>Since the regressor only needs to focus on the ranking task for samples where efs == 1, you can train it using just the efs == 1 samples. Additionally, you can train the model using both efs == 1 and efs == 0 samples to achieve better performance. However, in this case, it is necessary to add efs as an extra feature to prevent the model from getting confused by mixing the two types of samples together.</li>\n</ul>",
              "rawMarkdown": "- The merge function is based on the understanding of C-index metric.\n>The main idea is that when a patient has a high probability of efs == 0, we should give them a high score. Otherwise, we should give them a relatively lower score and pay more attention to the rank of efs_time. So the regressor just needs to focus on the ranking task where efs == 1.  \n\n- Since the regressor only needs to focus on the ranking task for samples where efs == 1, you can train it using just the efs == 1 samples. Additionally, you can train the model using both efs == 1 and efs == 0 samples to achieve better performance. However, in this case, it is necessary to add efs as an extra feature to prevent the model from getting confused by mixing the two types of samples together."
            }
          ]
        }
      ]
    },
    {
      "id": 3147747,
      "postDate": "2025-03-12T11:49:07.810Z",
      "content": "<p>First thing, Congrats! Its my lack of knowledge that I never thought of such approach. Well, it's obvious since I'm still learning but reading your solution gave somewhat insight on how I should've framed the problem. Thanks for sharing the solutions.</p>",
      "rawMarkdown": "First thing, Congrats! Its my lack of knowledge that I never thought of such approach. Well, it's obvious since I'm still learning but reading your solution gave somewhat insight on how I should've framed the problem. Thanks for sharing the solutions."
    },
    {
      "id": 3147059,
      "postDate": "2025-03-11T15:34:23.940Z",
      "content": "<p>Very good aproach</p>",
      "rawMarkdown": "Very good aproach"
    },
    {
      "id": 3146947,
      "postDate": "2025-03-11T13:14:10.810Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a>, solid approach. 👏\nClassifier + Regressor as two separate models is something I never thought of.</p>\n<p>I think the depth 6 of CatBoost might be because of the way it handles categorical features via target encoding unlike LGB/XGB which use one-hot encoding. It can lead to deeper trees for understanding the feature interaction.</p>",
      "rawMarkdown": "Congrats @minerppdy, solid approach. 👏\nClassifier + Regressor as two separate models is something I never thought of.\n\nI think the depth 6 of CatBoost might be because of the way it handles categorical features via target encoding unlike LGB/XGB which use one-hot encoding. It can lead to deeper trees for understanding the feature interaction."
    },
    {
      "id": 3146453,
      "postDate": "2025-03-10T22:25:10.783Z",
      "content": "<p>Congrats! Really cool to see your thought process behind the solution, super helpful for juniors like me to understand how to approach these challenges :)</p>",
      "rawMarkdown": "Congrats! Really cool to see your thought process behind the solution, super helpful for juniors like me to understand how to approach these challenges :)"
    },
    {
      "id": 3146235,
      "postDate": "2025-03-10T17:00:59.850Z",
      "content": "<p>Congrats! Very nice approach</p>",
      "rawMarkdown": "Congrats! Very nice approach"
    },
    {
      "id": 3146157,
      "postDate": "2025-03-10T15:17:12.943Z",
      "content": "<p>Congrats !!! Very nice approach. Good for Learning. </p>",
      "rawMarkdown": "Congrats !!! Very nice approach. Good for Learning. "
    },
    {
      "id": 3145638,
      "postDate": "2025-03-10T04:33:06.093Z",
      "content": "<p>Hi Omar .. I am trying to understand the code &amp; I saw you trained model with pca &amp; tsne features &amp; then use the best params to find the optimal parameters again with an optuna study &amp;  train the model with best params the training data without pca &amp; tsne .. I mean why not directly find optimal parameters on train data without tsne &amp; pca ? </p>",
      "rawMarkdown": "Hi Omar .. I am trying to understand the code & I saw you trained model with pca & tsne features & then use the best params to find the optimal parameters again with an optuna study &  train the model with best params the training data without pca & tsne .. I mean why not directly find optimal parameters on train data without tsne & pca ? "
    },
    {
      "id": 3144468,
      "postDate": "2025-03-08T12:05:03.543Z",
      "content": "<p>Interesting approach</p>",
      "rawMarkdown": "Interesting approach",
      "replies": [
        {
          "id": 3146100,
          "postDate": "2025-03-10T14:13:13.353Z",
          "content": "<p>Yes u are right</p>",
          "rawMarkdown": "Yes u are right"
        }
      ]
    },
    {
      "id": 3143993,
      "postDate": "2025-03-07T19:25:07.953Z",
      "content": "<p>Congratulations !! Thanks for sharing the approach. Would be more helpful if you could share the code so we could understand better in detail</p>",
      "rawMarkdown": "Congratulations !! Thanks for sharing the approach. Would be more helpful if you could share the code so we could understand better in detail",
      "replies": [
        {
          "id": 3144004,
          "postDate": "2025-03-07T19:50:10.830Z",
          "content": "<h1>Función para optimizar el uso de memoria</h1>\n<p>def optimize_dtypes(df):<br>\n    df_copy = df.copy()<br>\n    for col in df_copy.columns:<br>\n        if df_copy[col].dtype == 'object':<br>\n            df_copy[col] = df_copy[col].astype('category')<br>\n        elif df_copy[col].dtype == 'float64':<br>\n            df_copy[col] = df_copy[col].astype('float32')<br>\n        elif df_copy[col].dtype == 'int64':<br>\n            df_copy[col] = df_copy[col].astype('int32')<br>\n    return df_copy</p>\n<h1>Función para reemplazar nulos con valores por defecto</h1>\n<p>def replace_nulls_with_default(df, float_default=-1.0, object_default=\"Missing\"):<br>\n    df_copy = df.copy()<br>\n    for col in df_copy.select_dtypes(include=['float', 'int']).columns:<br>\n        df_copy.loc[:, col] = df_copy[col].replace(\"\", float_default).fillna(float_default)</p>\n<pre><code> col  df_copy(include=):\n     df_copy == :\n         object_default not  df_copy:\n            df_copy = df_copy(object_default)\n    df_copy = df_copy(, object_default)(object_default)\n\nreturn df_copy\n</code></pre>\n<h1>Función para calcular la probabilidad de supervivencia</h1>\n<p>def transform_survival_probability(df, time_col='efs_time', event_col='efs'):<br>\n    kmf = KaplanMeierFitter()<br>\n    kmf.fit(df[time_col], event_observed=df[event_col])<br>\n    survival_probabilities = kmf.survival_function_at_times(df[time_col]).values.flatten()<br>\n    return survival_probabilities</p>\n<h1>Función para calcular el índice C estratificado (corregida)</h1>\n<p>def stratified_c_index(y_true, y_pred, groups):<br>\n    unique_groups = np.unique(groups)<br>\n    c_indices = []</p>\n<pre><code> group  unique_groups:\n    mask = groups == group\n    y_true_group = y_true[mask]\n    y_pred_group = y_pred[mask]\n\n     len(y_true_group) &lt; :\n        continue\n\n    pairs = .(list(combinations((len(y_true_group)), )))\n    y_true_pairs = y_true_group[pairs]\n    y_pred_pairs = y_pred_group[pairs]\n\n    true_diff = y_true_pairs[:, ] - y_true_pairs[:, ]\n    pred_diff = y_pred_pairs[:, ] - y_pred_pairs[:, ]\n\n    permissible = true_diff != \n    concordant = (true_diff * pred_diff) &gt; \n\n    c_index = .(concordant &amp; permissible) / .(permissible)\n    c_indices.(c_index)\n\n .(c_indices) - .(c_indices)\n</code></pre>\n<p>def custom_scorer(estimator, X, y):<br>\n    y_pred = estimator.predict(X)<br>\n    return stratified_c_index(y.values, y_pred, race_groups.values[X.index])</p>\n<h1>Optimizar el uso de memoria</h1>\n<p>df_tr = optimize_dtypes(df_tr)<br>\ndf_ts = optimize_dtypes(df_ts)</p>\n<h1>Reemplazar nulos con valores por defecto</h1>\n<p>df_tr = replace_nulls_with_default(df_tr)<br>\ndf_ts = replace_nulls_with_default(df_ts)</p>\n<h1>Calcular y añadir la columna objetivo</h1>\n<p>df_tr[\"target\"] = np.sqrt(transform_survival_probability(df_tr, time_col='efs_time', event_col='efs'))</p>\n<h1>Eliminar columnas irrelevantes</h1>\n<p>drop_cols = [\"ID\", 'efs', 'efs_time']<br>\ndf_tr = df_tr.drop(columns=[col for col in drop_cols if col in df_tr.columns])<br>\ndf_ts = df_ts.drop(columns=[col for col in drop_cols if col in df_ts.columns])</p>\n<h1>Preparar los datos para el modelo</h1>\n<p>X = df_tr.drop(columns=['target'], axis=1)<br>\ny = df_tr['target']<br>\nrace_groups = df_tr['race_group']</p>\n<h1>Identificar columnas categóricas</h1>\n<p>cat_features = list(X.select_dtypes(include=['object', 'category']).columns)</p>\n<h1>Convertir columnas categóricas a tipo string</h1>\n<p>for col in cat_features:<br>\n    X[col] = X[col].astype(str)<br>\n    df_ts[col] = df_ts[col].astype(str)</p>\n<h1>Aplicar PCA</h1>\n<p>pca = PCA(n_components=10)<br>\nX_pca = pca.fit_transform(X.select_dtypes(include=['float32', 'int32']))</p>\n<h1>Aplicar t-SNE</h1>\n<p>tsne = TSNE(n_components=2, random_state=42)<br>\nX_tsne = tsne.fit_transform(X.select_dtypes(include=['float32', 'int32']))</p>\n<h1>Añadir las nuevas características al DataFrame original</h1>\n<p>X_new = pd.concat([X, <br>\n                   pd.DataFrame(X_pca, columns=[f'PCA_{i}' for i in range(10)], index=X.index),<br>\n                   pd.DataFrame(X_tsne, columns=['TSNE_1', 'TSNE_2'], index=X.index)], <br>\n                   axis=1)</p>\n<h1>Actualizar la lista de características categóricas</h1>\n<p>cat_features = list(X_new.select_dtypes(include=['object', 'category']).columns)</p>\n<h1>Configurar validación cruzada con KFold</h1>\n<p>kf = KFold(n_splits=5, shuffle=True, random_state=42)</p>\n<h1>Función objetivo para la búsqueda amplia (más agresiva)</h1>\n<p>def objective_wide(trial):<br>\n    params = {<br>\n        'iterations': trial.suggest_int('iterations', 100, 10000),<br>\n        'learning_rate': trial.suggest_float('learning_rate', 1e-5, 1.0, log=True),<br>\n        'depth': trial.suggest_int('depth', 2, 16),<br>\n        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1e-10, 100.0, log=True),<br>\n        'border_count': trial.suggest_categorical('border_count', [32, 64, 128, 255, 512]),<br>\n        'bagging_temperature': trial.suggest_float('bagging_temperature', 0, 2),<br>\n        'random_strength': trial.suggest_float('random_strength', 1e-10, 100.0, log=True),<br>\n        'loss_function': 'RMSE',<br>\n        'eval_metric': 'RMSE',<br>\n        'random_seed': 42,<br>\n        'task_type': 'GPU',<br>\n        'devices': '0,1',<br>\n        'verbose': False,<br>\n        'cat_features': cat_features,<br>\n        'used_ram_limit': '10gb',<br>\n        'allow_writing_files': False<br>\n    }</p>\n<pre><code>model = CatBoostRegressor(**params)\n\n:\n    score = cross_val_score(model, X_new, y, cv=kf, scoring=custom_scorer, error_score=).mean()\n    ()\n     score\n Exception  e:\n    ()\n     ()\n</code></pre>\n<h1>Crear el estudio de Optuna y optimizar (búsqueda amplia)</h1>\n<p>study_wide = optuna.create_study(direction='maximize')</p>\n<h1>Usar parallel_backend aquí</h1>\n<p>with parallel_backend('loky', n_jobs=-1):<br>\n    study_wide.optimize(objective_wide, n_trials=1000, timeout=14400)  # 4 horas de tiempo límite</p>\n<h1>Mejores parámetros y puntuación de la búsqueda amplia</h1>\n<p>best_params_wide = study_wide.best_params<br>\nprint('Mejores parámetros (búsqueda amplia):', best_params_wide)<br>\nprint('Mejor puntuación (búsqueda amplia):', study_wide.best_value)</p>\n<h1>Guardar los mejores parámetros de la búsqueda amplia</h1>\n<p>np.save('best_params_wide.npy', best_params_wide)</p>\n<h1>Mejores parámetros y puntuación de la búsqueda amplia</h1>\n<p>best_params_wide = study_wide.best_params<br>\nprint('Mejores parámetros (búsqueda amplia):', best_params_wide)<br>\nprint('Mejor puntuación (búsqueda amplia):', study_wide.best_value)</p>\n<h1>Definir el espacio de búsqueda fina (más agresiva)</h1>\n<p>def objective_fine(trial):<br>\n    params = {<br>\n        'iterations': trial.suggest_int('iterations', max(100, int(best_params_wide['iterations']<em>0.5)), min(15000, int(best_params_wide['iterations']</em>1.5))),<br>\n        'learning_rate': trial.suggest_float('learning_rate', max(1e-6, best_params_wide['learning_rate']<em>0.5), min(1.0, best_params_wide['learning_rate']</em>1.5), log=True),<br>\n        'depth': trial.suggest_int('depth', max(1, best_params_wide['depth']-3), min(20, best_params_wide['depth']+3)),<br>\n        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', max(1e-11, best_params_wide['l2_leaf_reg']<em>0.5), min(200.0, best_params_wide['l2_leaf_reg']</em>1.5), log=True),<br>\n        'border_count': trial.suggest_categorical('border_count', [32, 64, 128, 255, 512]),<br>\n        'bagging_temperature': trial.suggest_float('bagging_temperature', max(0, best_params_wide['bagging_temperature']<em>0.5), min(3, best_params_wide['bagging_temperature']</em>1.5)),<br>\n        'random_strength': trial.suggest_float('random_strength', max(1e-11, best_params_wide['random_strength']<em>0.5), min(200.0, best_params_wide['random_strength']</em>1.5), log=True),<br>\n        'loss_function': 'RMSE',<br>\n        'eval_metric': 'RMSE',<br>\n        'random_seed': 42,<br>\n        'task_type': 'GPU',<br>\n        'devices': '0,1',<br>\n        'verbose': False,<br>\n        'cat_features': cat_features,<br>\n        'used_ram_limit': '10gb',<br>\n        'allow_writing_files': False<br>\n    }</p>\n<pre><code>model = CatBoostRegressor(**params)\n\n:\n    score = cross_val_score(model, X_new, y, cv=kf, scoring=custom_scorer, error_score=).mean()\n    ()\n     score\n Exception  e:\n    ()\n     ()\n</code></pre>\n<h1>Crear el estudio de Optuna y optimizar (búsqueda fina)</h1>\n<p>study_fine = optuna.create_study(direction='maximize')</p>\n<h1>Usar parallel_backend aquí</h1>\n<p>with parallel_backend('loky', n_jobs=-1):<br>\n    study_fine.optimize(objective_fine, n_trials=400, timeout=10400)  # 3 hora de tiempo límite</p>\n<h1>Mejores parámetros y puntuación de la búsqueda fina</h1>\n<p>best_params_fine = study_fine.best_params<br>\nprint('Mejores parámetros (búsqueda fina):', best_params_fine)<br>\nprint('Mejor puntuación (búsqueda fina):', study_fine.best_value)</p>\n<h1>Comparar resultados</h1>\n<p>print('Mejora en la puntuación:', study_fine.best_value - study_wide.best_value)</p>\n<h1>Comparar resultados y seleccionar los mejores parámetros</h1>\n<p>if study_fine.best_value &gt; study_wide.best_value:<br>\n    print(\"La búsqueda fina mejoró los resultados.\")<br>\n    best_params = best_params_fine<br>\nelse:<br>\n    print(\"La búsqueda amplia dio mejores resultados.\")<br>\n    best_params = best_params_wide</p>\n<h1>Guardar los mejores parámetros finales</h1>\n<p>np.save('best_params.npy', best_params)</p>\n<p>print('Mejores parámetros finales:', best_params)<br>\nprint('Mejor puntuación final:', max(study_fine.best_value, study_wide.best_value))</p>\n<h1>Intentar cargar los mejores parámetros, si el archivo existe</h1>\n<p>if os.path.exists('best_params.npy'):<br>\n    best_params = np.load('best_params.npy', allow_pickle=True).item()<br>\nelse:<br>\n    print(\"Advertencia: No se encontró el archivo 'best_params.npy'. Usando los parámetros en memoria.\")</p>\n<h1>Initialize variables for predictions and scores</h1>\n<p>final_predictions = np.zeros(len(X))<br>\nfold_scores = []</p>\n<p>for fold, (train_idx, val_idx) in enumerate(kf.split(X, race_groups)):<br>\n    print(f\"Fold {fold + 1}\")<br>\n    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]<br>\n    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]<br>\n    race_val = race_groups.iloc[val_idx]</p>\n<pre><code>\nmodel = CatBoostRegressor(\n    *st_params,\n    =42,\n    =,\n    =,  # Usa GPUs en serie\n    =,\n    =cat_features,\n    =0.8,\n    =\n)\n\n\nmodel.fit(\n    X_train, y_train,\n    eval_set=(X_val, y_val),\n    =50,\n    =\n)\n\n\ny_val_pred = model.predict(X_val)\nfold_score = stratified_c_index(y_val.values, y_val_pred, race_val.values)\nfold_scores.append(fold_score)\n(f)\n\nfinal_predictions[val_idx] = y_val_pred\n</code></pre>\n<h1>Overall Stratified Concordance Index</h1>\n<p>overall_score = stratified_c_index(y.values, final_predictions, race_groups.values)<br>\nprint(f\"Overall Stratified C-Index: {overall_score}\")</p>\n<h1>Calculate standard C-index</h1>\n<p>c_index = concordance_index(y, final_predictions)<br>\nprint(f\"Standard C-Index: {c_index}\")</p>\n<h1>Plot the scores per fold</h1>\n<p>plt.figure(figsize=(10, 6))<br>\nplt.plot(range(1, len(fold_scores) + 1), fold_scores, marker='o', linestyle='-', color='b')<br>\nplt.title('Stratified C-Index by Fold')<br>\nplt.xlabel('Fold')<br>\nplt.ylabel('Stratified C-Index')<br>\nplt.xticks(range(1, len(fold_scores) + 1))<br>\nplt.grid()<br>\nplt.show()</p>\n<h1>Plot predicted vs actual values</h1>\n<p>plt.figure(figsize=(10, 6))<br>\nplt.scatter(y, final_predictions, alpha=0.5)<br>\nplt.plot([y.min(), y.max()], [y.min(), y.max()], 'r--', lw=2)<br>\nplt.xlabel('Actual Values')<br>\nplt.ylabel('Predicted Values')<br>\nplt.title('Predicted vs Actual Values')<br>\nplt.tight_layout()<br>\nplt.show()</p>",
          "rawMarkdown": "# Función para optimizar el uso de memoria\ndef optimize_dtypes(df):\n    df_copy = df.copy()\n    for col in df_copy.columns:\n        if df_copy[col].dtype == 'object':\n            df_copy[col] = df_copy[col].astype('category')\n        elif df_copy[col].dtype == 'float64':\n            df_copy[col] = df_copy[col].astype('float32')\n        elif df_copy[col].dtype == 'int64':\n            df_copy[col] = df_copy[col].astype('int32')\n    return df_copy\n\n\n# Función para reemplazar nulos con valores por defecto\ndef replace_nulls_with_default(df, float_default=-1.0, object_default=\"Missing\"):\n    df_copy = df.copy()\n    for col in df_copy.select_dtypes(include=['float', 'int']).columns:\n        df_copy.loc[:, col] = df_copy[col].replace(\"\", float_default).fillna(float_default)\n    \n    for col in df_copy.select_dtypes(include=['object', 'category']).columns:\n        if df_copy[col].dtype.name == 'category':\n            if object_default not in df_copy[col].cat.categories:\n                df_copy[col] = df_copy[col].cat.add_categories(object_default)\n        df_copy.loc[:, col] = df_copy[col].replace(\"\", object_default).fillna(object_default)\n    \n    return df_copy\n\n\n# Función para calcular la probabilidad de supervivencia\ndef transform_survival_probability(df, time_col='efs_time', event_col='efs'):\n    kmf = KaplanMeierFitter()\n    kmf.fit(df[time_col], event_observed=df[event_col])\n    survival_probabilities = kmf.survival_function_at_times(df[time_col]).values.flatten()\n    return survival_probabilities\n\n# Función para calcular el índice C estratificado (corregida)\ndef stratified_c_index(y_true, y_pred, groups):\n    unique_groups = np.unique(groups)\n    c_indices = []\n\n    for group in unique_groups:\n        mask = groups == group\n        y_true_group = y_true[mask]\n        y_pred_group = y_pred[mask]\n        \n        if len(y_true_group) < 2:\n            continue\n        \n        pairs = np.array(list(combinations(range(len(y_true_group)), 2)))\n        y_true_pairs = y_true_group[pairs]\n        y_pred_pairs = y_pred_group[pairs]\n        \n        true_diff = y_true_pairs[:, 0] - y_true_pairs[:, 1]\n        pred_diff = y_pred_pairs[:, 0] - y_pred_pairs[:, 1]\n        \n        permissible = true_diff != 0\n        concordant = (true_diff * pred_diff) > 0\n        \n        c_index = np.sum(concordant & permissible) / np.sum(permissible)\n        c_indices.append(c_index)\n\n    return np.mean(c_indices) - np.std(c_indices)\n\ndef custom_scorer(estimator, X, y):\n    y_pred = estimator.predict(X)\n    return stratified_c_index(y.values, y_pred, race_groups.values[X.index])\n\n# Optimizar el uso de memoria\ndf_tr = optimize_dtypes(df_tr)\ndf_ts = optimize_dtypes(df_ts)\n\n\n# Reemplazar nulos con valores por defecto\ndf_tr = replace_nulls_with_default(df_tr)\ndf_ts = replace_nulls_with_default(df_ts)\n\n# Calcular y añadir la columna objetivo\ndf_tr[\"target\"] = np.sqrt(transform_survival_probability(df_tr, time_col='efs_time', event_col='efs'))\n\n# Eliminar columnas irrelevantes\ndrop_cols = [\"ID\", 'efs', 'efs_time']\ndf_tr = df_tr.drop(columns=[col for col in drop_cols if col in df_tr.columns])\ndf_ts = df_ts.drop(columns=[col for col in drop_cols if col in df_ts.columns])\n\n# Preparar los datos para el modelo\nX = df_tr.drop(columns=['target'], axis=1)\ny = df_tr['target']\nrace_groups = df_tr['race_group']\n\n# Identificar columnas categóricas\ncat_features = list(X.select_dtypes(include=['object', 'category']).columns)\n\n# Convertir columnas categóricas a tipo string\nfor col in cat_features:\n    X[col] = X[col].astype(str)\n    df_ts[col] = df_ts[col].astype(str)\n\n# Aplicar PCA\npca = PCA(n_components=10)\nX_pca = pca.fit_transform(X.select_dtypes(include=['float32', 'int32']))\n\n# Aplicar t-SNE\ntsne = TSNE(n_components=2, random_state=42)\nX_tsne = tsne.fit_transform(X.select_dtypes(include=['float32', 'int32']))\n\n# Añadir las nuevas características al DataFrame original\nX_new = pd.concat([X, \n                   pd.DataFrame(X_pca, columns=[f'PCA_{i}' for i in range(10)], index=X.index),\n                   pd.DataFrame(X_tsne, columns=['TSNE_1', 'TSNE_2'], index=X.index)], \n                   axis=1)\n\n# Actualizar la lista de características categóricas\ncat_features = list(X_new.select_dtypes(include=['object', 'category']).columns)\n\n\n# Configurar validación cruzada con KFold\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\n\n# Función objetivo para la búsqueda amplia (más agresiva)\ndef objective_wide(trial):\n    params = {\n        'iterations': trial.suggest_int('iterations', 100, 10000),\n        'learning_rate': trial.suggest_float('learning_rate', 1e-5, 1.0, log=True),\n        'depth': trial.suggest_int('depth', 2, 16),\n        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1e-10, 100.0, log=True),\n        'border_count': trial.suggest_categorical('border_count', [32, 64, 128, 255, 512]),\n        'bagging_temperature': trial.suggest_float('bagging_temperature', 0, 2),\n        'random_strength': trial.suggest_float('random_strength', 1e-10, 100.0, log=True),\n        'loss_function': 'RMSE',\n        'eval_metric': 'RMSE',\n        'random_seed': 42,\n        'task_type': 'GPU',\n        'devices': '0,1',\n        'verbose': False,\n        'cat_features': cat_features,\n        'used_ram_limit': '10gb',\n        'allow_writing_files': False\n    }\n\n    model = CatBoostRegressor(**params)\n\n    try:\n        score = cross_val_score(model, X_new, y, cv=kf, scoring=custom_scorer, error_score=\"raise\").mean()\n        print(f\"Trial completed with score: {score}\")\n        return score\n    except Exception as e:\n        print(f\"Exception encountered: {e}\")\n        return float('-inf')\n\n# Crear el estudio de Optuna y optimizar (búsqueda amplia)\nstudy_wide = optuna.create_study(direction='maximize')\n\n# Usar parallel_backend aquí\nwith parallel_backend('loky', n_jobs=-1):\n    study_wide.optimize(objective_wide, n_trials=1000, timeout=14400)  # 4 horas de tiempo límite\n\n# Mejores parámetros y puntuación de la búsqueda amplia\nbest_params_wide = study_wide.best_params\nprint('Mejores parámetros (búsqueda amplia):', best_params_wide)\nprint('Mejor puntuación (búsqueda amplia):', study_wide.best_value)\n\n# Guardar los mejores parámetros de la búsqueda amplia\nnp.save('best_params_wide.npy', best_params_wide)\n\n\n# Mejores parámetros y puntuación de la búsqueda amplia\nbest_params_wide = study_wide.best_params\nprint('Mejores parámetros (búsqueda amplia):', best_params_wide)\nprint('Mejor puntuación (búsqueda amplia):', study_wide.best_value)\n\n# Definir el espacio de búsqueda fina (más agresiva)\ndef objective_fine(trial):\n    params = {\n        'iterations': trial.suggest_int('iterations', max(100, int(best_params_wide['iterations']*0.5)), min(15000, int(best_params_wide['iterations']*1.5))),\n        'learning_rate': trial.suggest_float('learning_rate', max(1e-6, best_params_wide['learning_rate']*0.5), min(1.0, best_params_wide['learning_rate']*1.5), log=True),\n        'depth': trial.suggest_int('depth', max(1, best_params_wide['depth']-3), min(20, best_params_wide['depth']+3)),\n        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', max(1e-11, best_params_wide['l2_leaf_reg']*0.5), min(200.0, best_params_wide['l2_leaf_reg']*1.5), log=True),\n        'border_count': trial.suggest_categorical('border_count', [32, 64, 128, 255, 512]),\n        'bagging_temperature': trial.suggest_float('bagging_temperature', max(0, best_params_wide['bagging_temperature']*0.5), min(3, best_params_wide['bagging_temperature']*1.5)),\n        'random_strength': trial.suggest_float('random_strength', max(1e-11, best_params_wide['random_strength']*0.5), min(200.0, best_params_wide['random_strength']*1.5), log=True),\n        'loss_function': 'RMSE',\n        'eval_metric': 'RMSE',\n        'random_seed': 42,\n        'task_type': 'GPU',\n        'devices': '0,1',\n        'verbose': False,\n        'cat_features': cat_features,\n        'used_ram_limit': '10gb',\n        'allow_writing_files': False\n    }\n\n    model = CatBoostRegressor(**params)\n\n    try:\n        score = cross_val_score(model, X_new, y, cv=kf, scoring=custom_scorer, error_score=\"raise\").mean()\n        print(f\"Fine-tuning trial completed with score: {score}\")\n        return score\n    except Exception as e:\n        print(f\"Exception encountered in fine-tuning: {e}\")\n        return float('-inf')\n\n# Crear el estudio de Optuna y optimizar (búsqueda fina)\nstudy_fine = optuna.create_study(direction='maximize')\n\n# Usar parallel_backend aquí\nwith parallel_backend('loky', n_jobs=-1):\n    study_fine.optimize(objective_fine, n_trials=400, timeout=10400)  # 3 hora de tiempo límite\n\n# Mejores parámetros y puntuación de la búsqueda fina\nbest_params_fine = study_fine.best_params\nprint('Mejores parámetros (búsqueda fina):', best_params_fine)\nprint('Mejor puntuación (búsqueda fina):', study_fine.best_value)\n\n# Comparar resultados\nprint('Mejora en la puntuación:', study_fine.best_value - study_wide.best_value)\n\n# Comparar resultados y seleccionar los mejores parámetros\nif study_fine.best_value > study_wide.best_value:\n    print(\"La búsqueda fina mejoró los resultados.\")\n    best_params = best_params_fine\nelse:\n    print(\"La búsqueda amplia dio mejores resultados.\")\n    best_params = best_params_wide\n\n# Guardar los mejores parámetros finales\nnp.save('best_params.npy', best_params)\n\nprint('Mejores parámetros finales:', best_params)\nprint('Mejor puntuación final:', max(study_fine.best_value, study_wide.best_value))\n\n# Intentar cargar los mejores parámetros, si el archivo existe\nif os.path.exists('best_params.npy'):\n    best_params = np.load('best_params.npy', allow_pickle=True).item()\nelse:\n    print(\"Advertencia: No se encontró el archivo 'best_params.npy'. Usando los parámetros en memoria.\")\n \n\n# Initialize variables for predictions and scores\nfinal_predictions = np.zeros(len(X))\nfold_scores = []\n\n\nfor fold, (train_idx, val_idx) in enumerate(kf.split(X, race_groups)):\n    print(f\"Fold {fold + 1}\")\n    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]\n    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]\n    race_val = race_groups.iloc[val_idx]\n\n    # Set up CatBoost model with best parameters\n    model = CatBoostRegressor(\n        **best_params,\n        random_seed=42,\n        task_type='GPU',\n        devices='0,1',  # Usa GPUs en serie\n        verbose=False,\n        cat_features=cat_features,\n        gpu_ram_part=0.8,\n        allow_writing_files=False\n    )\n\n    # Train the model\n    model.fit(\n        X_train, y_train,\n        eval_set=(X_val, y_val),\n        early_stopping_rounds=50,\n        verbose=False\n    )\n\n    # Predict and compute metric\n    y_val_pred = model.predict(X_val)\n    fold_score = stratified_c_index(y_val.values, y_val_pred, race_val.values)\n    fold_scores.append(fold_score)\n    print(f\"Stratified C-Index for Fold {fold + 1}: {fold_score}\")\n\n    final_predictions[val_idx] = y_val_pred\n\n# Overall Stratified Concordance Index\noverall_score = stratified_c_index(y.values, final_predictions, race_groups.values)\nprint(f\"Overall Stratified C-Index: {overall_score}\")\n\n# Calculate standard C-index\nc_index = concordance_index(y, final_predictions)\nprint(f\"Standard C-Index: {c_index}\")\n\n# Plot the scores per fold\nplt.figure(figsize=(10, 6))\nplt.plot(range(1, len(fold_scores) + 1), fold_scores, marker='o', linestyle='-', color='b')\nplt.title('Stratified C-Index by Fold')\nplt.xlabel('Fold')\nplt.ylabel('Stratified C-Index')\nplt.xticks(range(1, len(fold_scores) + 1))\nplt.grid()\nplt.show()\n\n# Plot predicted vs actual values\nplt.figure(figsize=(10, 6))\nplt.scatter(y, final_predictions, alpha=0.5)\nplt.plot([y.min(), y.max()], [y.min(), y.max()], 'r--', lw=2)\nplt.xlabel('Actual Values')\nplt.ylabel('Predicted Values')\nplt.title('Predicted vs Actual Values')\nplt.tight_layout()\nplt.show()",
          "votes": -6,
          "replies": [
            {
              "id": 3144755,
              "postDate": "2025-03-08T20:18:57.187Z",
              "content": "<p><a href=\"https://www.kaggle.com/code/palbha/cibmtr-replicate-1st-prize-solution\" target=\"_blank\">https://www.kaggle.com/code/palbha/cibmtr-replicate-1st-prize-solution</a> - I have created this notebook as per what you shared - patiently waiting for it to run now :) .. Thanks a lot for sharing !! </p>",
              "rawMarkdown": "https://www.kaggle.com/code/palbha/cibmtr-replicate-1st-prize-solution - I have created this notebook as per what you shared - patiently waiting for it to run now :) .. Thanks a lot for sharing !! ",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 3143820,
      "postDate": "2025-03-07T16:30:19.837Z",
      "content": "<p><a href=\"https://www.kaggle.com/omararielvallejos\" target=\"_blank\">@omararielvallejos</a> i saw your posted code, what's 0.74 mean? and how it comes?</p>",
      "rawMarkdown": "@omararielvallejos i saw your posted code, what's 0.74 mean? and how it comes?"
    },
    {
      "id": 3143809,
      "postDate": "2025-03-07T16:16:39.230Z",
      "content": "<p>Wow, great work!</p>",
      "rawMarkdown": "Wow, great work!"
    },
    {
      "id": 3142830,
      "postDate": "2025-03-06T17:34:30.647Z",
      "content": "<blockquote>\n  <p>Add efs as a feature when training and focus on the performance at samples where efs==1. Set efs=1 when making inference on LB data. </p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> Do you use the actual <code>efs</code> target (retaining both 0 and 1 values) as feature for training and set efs=1 for all rows during inference?</p>",
      "rawMarkdown": "> Add efs as a feature when training and focus on the performance at samples where efs==1. Set efs=1 when making inference on LB data. \n\n@minerppdy Do you use the actual `efs` target (retaining both 0 and 1 values) as feature for training and set efs=1 for all rows during inference?",
      "replies": [
        {
          "id": 3142845,
          "postDate": "2025-03-06T17:51:30.210Z",
          "content": "<p><a href=\"https://www.kaggle.com/andreasbis\" target=\"_blank\">@andreasbis</a> yes, the purpose is to improve the performance on efs=1 using the potencial information of efs=0, and meanwhile make sure the model wouldn't confused by mixing different samples together.  without this feature, the result is really bad</p>",
          "rawMarkdown": "@andreasbis yes, the purpose is to improve the performance on efs=1 using the potencial information of efs=0, and meanwhile make sure the model wouldn't confused by mixing different samples together.  without this feature, the result is really bad",
          "votes": 1,
          "replies": [
            {
              "id": 3142961,
              "postDate": "2025-03-06T19:35:31.300Z",
              "content": "<p>Huh, and from code you shared you had efs==0 as it's own separate uniform 0-1 ranking? Interesting. If that even mattered I wonder why it works. I suspect there are competing nuances to the impact of your exact implementation, but I don't think I have fully grasped it at a theoretical level yet. </p>",
              "rawMarkdown": "Huh, and from code you shared you had efs==0 as it's own separate uniform 0-1 ranking? Interesting. If that even mattered I wonder why it works. I suspect there are competing nuances to the impact of your exact implementation, but I don't think I have fully grasped it at a theoretical level yet. "
            },
            {
              "id": 3143178,
              "postDate": "2025-03-07T02:24:32.910Z",
              "content": "<p><a href=\"https://www.kaggle.com/roberthatch\" target=\"_blank\">@roberthatch</a> I have tried several kinds of transform,but didn't find impressive difference, for example:</p>\n<pre><code>efs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  \nefs_time_norm[efs == ] = efs_time[efs == ].rank() / (efs == )  +\n</code></pre>\n<pre><code>efs_time_norm = efs_time.rank() / (efs)\n</code></pre>",
              "rawMarkdown": "@roberthatch I have tried several kinds of transform,but didn't find impressive difference, for example:\n```python\nefs_time_norm[efs == 1] = efs_time[efs == 1].rank() / sum(efs == 1)  \nefs_time_norm[efs == 0] = efs_time[efs == 0].rank() / sum(efs == 0)  +1\n```\n\n```python\nefs_time_norm = efs_time.rank() / len(efs)\n```"
            }
          ]
        }
      ]
    },
    {
      "id": 3142778,
      "postDate": "2025-03-06T16:54:54.977Z",
      "content": "<p>Congratulations,you did a great and amazing work</p>",
      "rawMarkdown": "Congratulations,you did a great and amazing work"
    },
    {
      "id": 3142556,
      "postDate": "2025-03-06T13:24:01.497Z",
      "content": "<p>appreciate for ur share!</p>",
      "rawMarkdown": "appreciate for ur share!"
    },
    {
      "id": 3142546,
      "postDate": "2025-03-06T13:17:03.380Z",
      "content": "<p>Great solution, congratulations!</p>",
      "rawMarkdown": "Great solution, congratulations!"
    },
    {
      "id": 3142459,
      "postDate": "2025-03-06T12:10:23.500Z",
      "content": "<p>Nice, good job!</p>",
      "rawMarkdown": "Nice, good job!\n"
    },
    {
      "id": 3142289,
      "postDate": "2025-03-06T09:18:35.603Z",
      "content": "<p>Congrats amazing and smart work!</p>",
      "rawMarkdown": "Congrats amazing and smart work!"
    },
    {
      "id": 3142027,
      "postDate": "2025-03-06T03:30:44.580Z",
      "content": "<p>Thank you. After I used up the last chance last night, it occurred to me that I once accidentally mixed efs into the training set before and got a cv score of almost 1, so it should be possible to use the predicted efs as feature input. But unfortunately, I have used up the last chance. You have verified that this idea is correct, although I have some regrets, but thank you!🥹</p>",
      "rawMarkdown": "Thank you. After I used up the last chance last night, it occurred to me that I once accidentally mixed efs into the training set before and got a cv score of almost 1, so it should be possible to use the predicted efs as feature input. But unfortunately, I have used up the last chance. You have verified that this idea is correct, although I have some regrets, but thank you!🥹"
    },
    {
      "id": 3143791,
      "postDate": "2025-03-07T15:49:18.403Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 3143822,
          "postDate": "2025-03-07T16:32:31.680Z",
          "content": "<p><a href=\"https://www.kaggle.com/cody11null\" target=\"_blank\">@cody11null</a> what do you mean lost signal?</p>",
          "rawMarkdown": "@cody11null what do you mean lost signal?",
          "replies": [
            {
              "id": 3146021,
              "postDate": "2025-03-10T13:17:55.640Z",
              "rawMarkdown": "",
              "isDeleted": true
            }
          ]
        }
      ]
    },
    {
      "id": 3142267,
      "postDate": "2025-03-06T08:50:04.523Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 3141952,
      "postDate": "2025-03-06T01:55:42.903Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 3141958,
      "postDate": "2025-03-06T02:08:46.463Z",
      "content": "<p><a href=\"https://www.kaggle.com/minerppdy\" target=\"_blank\">@minerppdy</a> Learned a lot! Thank you!</p>",
      "rawMarkdown": "@minerppdy Learned a lot! Thank you!",
      "votes": 1
    },
    {
      "id": 3146478,
      "postDate": "2025-03-10T23:40:27.923Z",
      "content": "<p>Congratulations </p>",
      "rawMarkdown": "Congratulations "
    },
    {
      "id": 3145227,
      "postDate": "2025-03-09T14:40:51.623Z",
      "content": "<p>Thanks for the nice approach!</p>",
      "rawMarkdown": "Thanks for the nice approach!"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:46:18Z",
  "competition_id": 70942,
  "competition_slug": "equity-post-HCT-survival-predictions",
  "team_id": 13111369,
  "solution_url": "/competitions/equity-post-HCT-survival-predictions/writeups/minerppdy-1st-place-solution-2-targets-and-ensembl"
}