{ "id": 566589, "title": "84th Place Solution", "content": "

Congratulations to Kaggle and CIBMTR for hosting this competition and all Kagglers who shared their work and progress! In particular, I want to thank @albansteff and @abdmental01 for the notebooks behind the:

\n\n

I used in my best ensembles.

\n
\n

For all my in-house models I did not do feature aggregation and used 5-fold KFold CV scheme. My best selected submission is an ensemble of:

\n\n
\n\n
\n

GBDT Models

\n

The GBDT models I used are based on the Kaplan-Meier target of this notebook. The targets I used were: Kaplan-Meier, Nelson-Aalen and the transformation suggested by @cdeotte which is suitable for the Cox loss function:

\n
 :\n\n     ():\n\n        .data = data\n        .cat_cols = cat_cols\n\n        ._threshold = \n        ._shift = \n\n     ():\n\n        data = .data.loc[(.data.efs_time < ._threshold) | (.data.efs == )]\n\n        km = KaplanMeierFitter()        \n        km.fit(durations=data[], event_observed=data[])\n\n        target = km.survival_function_at_times(.data[]).values        \n        target[.data.efs == ] = target[.data.efs == ].() - ._shift\n        target /= np.std(target)\n\n        .data[] = target\n\n         .data\n\n     ():\n\n        data = .data.loc[(.data.efs_time < ._threshold) | (.data.efs == )]\n\n        na = NelsonAalenFitter()    \n        na.fit(durations=data[], event_observed=data[])\n\n        target = -na.cumulative_hazard_at_times(.data[]).values \n        target[.data.efs == ] = target[.data.efs == ].() - ._shift\n        target /= np.std(target)\n\n        .data[] = target\n\n         .data\n\n     ():\n\n        .data[] = .data.efs_time.copy()\n        .data.loc[.data.efs == , ] *= -\n\n         .data\n
\n

For the Kaplan-Meier and Nelson-Aalen targets, I used: CatBoost, LightGBM, XGBoost
\nFor the Cox-loss target, I used two CatBoost models with different parameters.

\n

MLP Models

\n

The MLP models were useful only if I used the OOF predictions of GBDT models; in order to have some variety in predictions, I trained a different set of 14 GBDT models (without target post-processing as I do with the GBDT models above) and use their predictions as meta features.

\n

The targets I used were: CoxPH, Kaplan-Meier, Nelson-Aalen, Quantile transform. I also used two loss functions:

\n\n
    def (self, outputs, y, race_groups, alpha=):\n\n        unique_races = race_groups.()\n        group_losses = []\n\n        for race in unique_races:\n\n            race_mask = (race_groups.() == race)\n            race_mask = torch.(race_mask, device=outputs.device, dtype=torch.bool)\n\n            race_outputs = outputs[race_mask]\n            race_y = y[race_mask]\n\n            group_loss = torch.nn.functional.(race_outputs.(), race_y)\n            group_losses.(group_loss)\n\n        group_losses_tensor = torch.(group_losses)\n\n        return group_losses_tensor.() + alpha * group_losses_tensor.()\n
\n\n
\n

Unsuccessful Experiments

\n

Some of my unsuccessful (that did not improve both CV and LB) were:

\n
    \n

  1. Implementing GNN: while I implemented an architecture using pytorch (not torch_geometric) it generalizes well only on the local CV, as the ensemble which had GNN models dropped from 0.693 to0.692 (luckily I didn't choose this one as best-LB!)

    2. \n

  2. As per the following post, I had tried generating additional data using SurvivalGAN, but the resulting models did not perform well on the LB.

    3. \n

  3. Early in the competition, I had tried TTA (Test Time Augmentation) by adding guassian noise to the (only) two continuous features of the dataset (age_at_hct, donor_age) but the LB degraded.

    4. \n

  4. DeBERTa; my idea was to convert all categorical columns (and some numeric columns that describe patients' age and health) into one text column and feed this column into DeBERTa using custom Pooling layers. While the CV of these models did not exceed 0.654, I learned to use pytorch and huggingface to fine-tune models and can use this code implementation for future competitions.

    5. \n
", "comments": [ { "id": 3142953, "author_name": "Sheikh Muhammad Abdullah", "author_url": "", "post_date": "2025-03-06T19:24:39.917000", "content": "

Congrats, @andreasbis! Your hard work has paid off. Happy to see you reach the silver position! Wishing you Good Luck for Future Competitions.

", "votes": 1, "replies": [] }, { "id": 3142205, "author_name": "Robert Hatch", "author_url": "", "post_date": "2025-03-06T07:49:45.133000", "content": "

Congrats! Your public work was an inspiration and starting off point for a lot of my long discovery and gradual improvements effort. And while I focused on multidimensional shifting of nelsen Aalen for my manual efforts, taking your public notebook with no change except probably seed did lead to a notable little piece of the weighting pie going to the second Cox model. Oh, but I didn't rank my predictions, and the big breakthrough re Cox model was realizing that it was giving basically logit or logit esque predictions. So it ensembled really well when I converted it to probability via expit function.

", "votes": 1, "replies": [] }, { "id": 3142179, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-03-06T06:56:59.550000", "content": "

I also harbored doubts about BERT style models and had asked you accordingly on the forums. I am happy you learnt this new skill and am sure you will use it effectively going ahead @andreasbis

\n

All the best and congratulations for the silver medal!

", "votes": 1, "replies": [] } ], "comment_count": 3, "messages": [ { "id": 3142136, "postDate": "2025-03-06T05:51:50.790Z", "content": "

Congratulations to Kaggle and CIBMTR for hosting this competition and all Kagglers who shared their work and progress! In particular, I want to thank @albansteff and @abdmental01 for the notebooks behind the:

\n\n

I used in my best ensembles.

\n
\n

For all my in-house models I did not do feature aggregation and used 5-fold KFold CV scheme. My best selected submission is an ensemble of:

\n\n
\n\n
\n

GBDT Models

\n

The GBDT models I used are based on the Kaplan-Meier target of this notebook. The targets I used were: Kaplan-Meier, Nelson-Aalen and the transformation suggested by @cdeotte which is suitable for the Cox loss function:

\n
 :\n\n     ():\n\n        .data = data\n        .cat_cols = cat_cols\n\n        ._threshold = \n        ._shift = \n\n     ():\n\n        data = .data.loc[(.data.efs_time < ._threshold) | (.data.efs == )]\n\n        km = KaplanMeierFitter()        \n        km.fit(durations=data[], event_observed=data[])\n\n        target = km.survival_function_at_times(.data[]).values        \n        target[.data.efs == ] = target[.data.efs == ].() - ._shift\n        target /= np.std(target)\n\n        .data[] = target\n\n         .data\n\n     ():\n\n        data = .data.loc[(.data.efs_time < ._threshold) | (.data.efs == )]\n\n        na = NelsonAalenFitter()    \n        na.fit(durations=data[], event_observed=data[])\n\n        target = -na.cumulative_hazard_at_times(.data[]).values \n        target[.data.efs == ] = target[.data.efs == ].() - ._shift\n        target /= np.std(target)\n\n        .data[] = target\n\n         .data\n\n     ():\n\n        .data[] = .data.efs_time.copy()\n        .data.loc[.data.efs == , ] *= -\n\n         .data\n
\n

For the Kaplan-Meier and Nelson-Aalen targets, I used: CatBoost, LightGBM, XGBoost
\nFor the Cox-loss target, I used two CatBoost models with different parameters.

\n

MLP Models

\n

The MLP models were useful only if I used the OOF predictions of GBDT models; in order to have some variety in predictions, I trained a different set of 14 GBDT models (without target post-processing as I do with the GBDT models above) and use their predictions as meta features.

\n

The targets I used were: CoxPH, Kaplan-Meier, Nelson-Aalen, Quantile transform. I also used two loss functions:

\n\n
    def (self, outputs, y, race_groups, alpha=):\n\n        unique_races = race_groups.()\n        group_losses = []\n\n        for race in unique_races:\n\n            race_mask = (race_groups.() == race)\n            race_mask = torch.(race_mask, device=outputs.device, dtype=torch.bool)\n\n            race_outputs = outputs[race_mask]\n            race_y = y[race_mask]\n\n            group_loss = torch.nn.functional.(race_outputs.(), race_y)\n            group_losses.(group_loss)\n\n        group_losses_tensor = torch.(group_losses)\n\n        return group_losses_tensor.() + alpha * group_losses_tensor.()\n
\n\n
\n

Unsuccessful Experiments

\n

Some of my unsuccessful (that did not improve both CV and LB) were:

\n
    \n

  1. Implementing GNN: while I implemented an architecture using pytorch (not torch_geometric) it generalizes well only on the local CV, as the ensemble which had GNN models dropped from 0.693 to0.692 (luckily I didn't choose this one as best-LB!)

    2. \n

  2. As per the following post, I had tried generating additional data using SurvivalGAN, but the resulting models did not perform well on the LB.

    3. \n

  3. Early in the competition, I had tried TTA (Test Time Augmentation) by adding guassian noise to the (only) two continuous features of the dataset (age_at_hct, donor_age) but the LB degraded.

    4. \n

  4. DeBERTa; my idea was to convert all categorical columns (and some numeric columns that describe patients' age and health) into one text column and feed this column into DeBERTa using custom Pooling layers. While the CV of these models did not exceed 0.654, I learned to use pytorch and huggingface to fine-tune models and can use this code implementation for future competitions.

    5. \n
", "rawMarkdown": "Congratulations to Kaggle and CIBMTR for hosting this competition and all Kagglers who shared their work and progress! In particular, I want to thank @albansteff and @abdmental01 for the notebooks behind the:\n\n- target transformation ([notebook](https://www.kaggle.com/code/abdmental01/xgb-monotone-fork-cv-improvement))\n- Event Mask PRL-NN ([notebook](https://www.kaggle.com/code/albansteff/event-masked-prl-nn))\n- and handling missing values in HLA columns ([notebook](https://www.kaggle.com/code/albansteff/cibmtr-eda-ensemble-model-recalculate-hla))\n\nI used in my best ensembles.\n\n---\n\nFor all my in-house models I did not do feature aggregation and used 5-fold KFold CV scheme. My `best selected` submission is an ensemble of:\n\n- 16 in-house models (8x GBDT and 8x MLP), based on my public [baseline](https://www.kaggle.com/code/andreasbis/cibmtr-eda-ensemble-model)\n- 2 variations of the [Event-masked PRL-NN](https://www.kaggle.com/code/albansteff/event-masked-prl-nn) notebook: one with 5-fold and one with 10-fold CV (an ensemble of these 2 achieves a CV of 0.68415)\n\n---\n\n- My best overall solution (still 0.693) also used the [XGB Monotone_ Fork _CV_Improvement](https://www.kaggle.com/code/abdmental01/xgb-monotone-fork-cv-improvement) notebook; the CV of my ensemble improved to 0.68748, but because the public LB dropped by 0.001 (0.693 -> 0.692), I did not choose it.\n\n---\n\n# GBDT Models\n\nThe GBDT models I used are based on the Kaplan-Meier target of [this](https://www.kaggle.com/code/abdmental01/xgb-monotone-fork-cv-improvement) notebook. The targets I used were: Kaplan-Meier, Nelson-Aalen and the transformation suggested by @cdeotte which is suitable for the Cox loss function:\n\n```\nclass Targets:\n\n def __init__(self, data, cat_cols, n_splits):\n \n self.data = data\n self.cat_cols = cat_cols\n\n self._threshold = 24\n self._shift = 0.2\n\n def create_target1(self):\n\n data = self.data.loc[(self.data.efs_time < self._threshold) | (self.data.efs == 0)]\n\n km = KaplanMeierFitter() \n km.fit(durations=data['efs_time'], event_observed=data['efs'])\n \n target = km.survival_function_at_times(self.data['efs_time']).values \n target[self.data.efs == 0] = target[self.data.efs == 1].min() - self._shift\n target /= np.std(target)\n\n self.data['target1'] = target\n\n return self.data\n\n def create_target2(self):\n\n data = self.data.loc[(self.data.efs_time < self._threshold) | (self.data.efs == 0)]\n\n na = NelsonAalenFitter() \n na.fit(durations=data['efs_time'], event_observed=data['efs'])\n \n target = -na.cumulative_hazard_at_times(self.data['efs_time']).values \n target[self.data.efs == 0] = target[self.data.efs == 1].min() - self._shift\n target /= np.std(target)\n\n self.data['target2'] = target\n\n return self.data\n \n def create_target3(self):\n\n self.data['target3'] = self.data.efs_time.copy()\n self.data.loc[self.data.efs == 0, 'target3'] *= -1\n\n return self.data\n```\n\nFor the Kaplan-Meier and Nelson-Aalen targets, I used: CatBoost, LightGBM, XGBoost\nFor the Cox-loss target, I used two CatBoost models with different parameters.\n\n# MLP Models\n\nThe MLP models were useful **only** if I used the OOF predictions of GBDT models; in order to have some variety in predictions, I trained a different set of 14 GBDT models (without target post-processing as I do with the GBDT models above) and use their predictions as meta features. \n\nThe targets I used were: CoxPH, Kaplan-Meier, Nelson-Aalen, Quantile transform. I also used two loss functions:\n\n- stratified MSE loss, calculated as:\n\n```\n def _stratified_mse_loss(self, outputs, y, race_groups, alpha=0.2):\n \n unique_races = race_groups.unique()\n group_losses = []\n \n for race in unique_races:\n \n race_mask = (race_groups.to_numpy() == race)\n race_mask = torch.tensor(race_mask, device=outputs.device, dtype=torch.bool)\n \n race_outputs = outputs[race_mask]\n race_y = y[race_mask]\n \n group_loss = torch.nn.functional.mse_loss(race_outputs.squeeze(), race_y)\n group_losses.append(group_loss)\n \n group_losses_tensor = torch.stack(group_losses)\n \n return group_losses_tensor.mean() + alpha * group_losses_tensor.std()\n```\n\n- and Pairwise Ranking loss, which is calculated in the same fashion as the loss above.\n\n---\n\n# Unsuccessful Experiments\n\nSome of my unsuccessful (that did not improve both CV and LB) were:\n\n1. Implementing GNN: while I implemented an architecture using pytorch (not torch_geometric) it generalizes well only on the local CV, as the ensemble which had GNN models dropped from 0.693 to0.692 (luckily I didn't choose this one as best-LB!)\n\n2. As per the following [post](https://www.kaggle.com/competitions/equity-post-HCT-survival-predictions/discussion/555490), I had tried generating additional data using SurvivalGAN, but the resulting models did not perform well on the LB.\n\n3. Early in the competition, I had tried TTA (Test Time Augmentation) by adding guassian noise to the (only) two continuous features of the dataset (age_at_hct, donor_age) but the LB degraded.\n\n4. DeBERTa; my idea was to convert all categorical columns (and some numeric columns that describe patients' age and health) into one text column and feed this column into DeBERTa using custom Pooling layers. While the CV of these models did not exceed 0.654, I learned to use pytorch and huggingface to fine-tune models and can use this code implementation for future competitions. ", "votes": 8 }, { "id": 3142953, "postDate": "2025-03-06T19:24:39.917Z", "content": "

Congrats, @andreasbis! Your hard work has paid off. Happy to see you reach the silver position! Wishing you Good Luck for Future Competitions.

", "rawMarkdown": "Congrats, @andreasbis! Your hard work has paid off. Happy to see you reach the silver position! Wishing you Good Luck for Future Competitions.", "votes": 1 }, { "id": 3142205, "postDate": "2025-03-06T07:49:45.133Z", "content": "

Congrats! Your public work was an inspiration and starting off point for a lot of my long discovery and gradual improvements effort. And while I focused on multidimensional shifting of nelsen Aalen for my manual efforts, taking your public notebook with no change except probably seed did lead to a notable little piece of the weighting pie going to the second Cox model. Oh, but I didn't rank my predictions, and the big breakthrough re Cox model was realizing that it was giving basically logit or logit esque predictions. So it ensembled really well when I converted it to probability via expit function.

", "rawMarkdown": "Congrats! Your public work was an inspiration and starting off point for a lot of my long discovery and gradual improvements effort. And while I focused on multidimensional shifting of nelsen Aalen for my manual efforts, taking your public notebook with no change except probably seed did lead to a notable little piece of the weighting pie going to the second Cox model. Oh, but I didn't rank my predictions, and the big breakthrough re Cox model was realizing that it was giving basically logit or logit esque predictions. So it ensembled really well when I converted it to probability via expit function. ", "votes": 1 }, { "id": 3142179, "postDate": "2025-03-06T06:56:59.550Z", "content": "

I also harbored doubts about BERT style models and had asked you accordingly on the forums. I am happy you learnt this new skill and am sure you will use it effectively going ahead @andreasbis

\n

All the best and congratulations for the silver medal!

", "rawMarkdown": "I also harbored doubts about BERT style models and had asked you accordingly on the forums. I am happy you learnt this new skill and am sure you will use it effectively going ahead @andreasbis \n\nAll the best and congratulations for the silver medal!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:47:30Z", "competition_id": 70942, "competition_slug": "equity-post-HCT-survival-predictions", "team_id": 13054600, "solution_url": "/competitions/equity-post-HCT-survival-predictions/writeups/andrew-84th-place-solution" }