# Rank 120 approach - simplicity is the name of the game!

Hello all,
Firstly, I wish to extend sincere thanks to Kaggle and CIBMTR for a very accessible tabular competition. I also wish to thank the contributors in the public forums for such a generous contribution through Dec2024-Mar2025! In my 3 years of Kaggle competitions, I have witnessed very few competitions with such a great public sharing across the forums! Sincere thanks to all the contributors! Also, special thanks to
[@benjenkins96](https://www.kaggle.com/benjenkins96)
and
[@meetbabariya](https://www.kaggle.com/meetbabariya)
for the team-up, we have a silver here!
My approach for the competition focused on a simple blend of boosted tree and NN models with and without the classifier masking public kernel. I used selective public work from across the forums, carefully modifying the ideas propitiated with private work and associated adjustments as below-
Model pipeline architecture
My model pipeline for the assignment can be summarized as below-
![](assets/5c77ee4859c4e90450664bbd.png)
Preprocessing
Feature engineering
Almost negligible, competition features were sufficient in entirety
Used
row-na = df.isna().sum(axis=1)
as an extra feature
Failed Feature Engineering experiments
Target encoder 😀
MEstimate encoder
Bigram and trigram interactions
Converting all features to string/ text type (this is commonly done in playground competitions)
HLA feature adjustment (taken from public work)
PCA (adapted from Yunbase)
Age in years and age in days (separating integer and decimal from age)
Target variable choices
Kaplan Meier Fitter (most stable and offered the best CV-LB)
Nelson Aelen Fitter (fitted well with the best CV but scored lower on the LB)
BFH Fitter (moderate CV and LB scores, but scores were always in sync)
Log efs-time (used in public NN models)
Failed target experiments
Target scheme
Issues faced
Weibull fitter
* Very low single model CV and high standard deviation across folds
Log-logistic
* Very low single model CV for boosted trees and NN models
Log-normal
* Very low single model CV for boosted trees and NN models
CV-scheme
I used a variety of CV schemes to varying degrees of success, but decided to opt for 10-fold (simple K-fold)
cv = KFold(random_state = 42, n_splits = 10, shuffle = True)
Failed CV experiments
CV scheme
Issues faced
Stratified Kfold -
race group
* Single models had a greater CV than kfold and CV-LB was good, but ensemble CV-LB went awry
Stratified Kfold -
race group + year-hct
* Single model CV-LB were not in sync after a point
Stratified Kfold -
race group + year-hct + efs
* Single model CV-LB were not in sync, especially for the Autogluon model
Model pipeline
Given below are the CV scores across component models -
Model Label
CV score
no mask
LB score
no mask
CV score
mask
LB score
mask
LITNN
0.68269241
0.688
0.68351879
0.689
TABM
0.68104054
0.685
0.68151639
0.685
L1-AG
0.68051078
0.687
0.68136013
0.688
L1-ML
0.6684 - 0.6792
0.673 - 0.685
0.6704 - 0.6810
0.674 - 0.685
L2-ML
0.68422683
0.689
0.68554
0.689
L2-LAMA
0.67890052
0.688
0.681053
0.689
XGBR Sample Weights
0.68174912
0.68046331
0.686
0.685
0.68267643
0.686
0.686
Key notes
Early stopping worked for me across boosted trees. Not using this reduced the CV score for me across models
Training on CPU provided a better CV score than training on GPU, especially for Catboost
I used target separators as the ones below-
np.where( ytrain.efs == 1, ytrain["KMF"] + separation_factor, ytrain["KMF"])
Using no separation (separation = 0) yielded a lower CV score for me. Separation constant of
0.10 and 0.15
were most productive for me
The XGB classifier mask upgraded the CV across model components but failed to improve the leaderboard almost throughout
Almost all feature engineering ideas were futile - using the base features as is helped a lot
Using the starting features and OOF features from across boosted tree models in a separate boosted tree stacking L2 model helped a lot with the CV and a bit with the leaderboard as well
Stacking a NN model with boosted tree OOF features helped with the LB but not the CV (gap between CV and LB increased from 0.006 to 0.009)
Stacking a boosted tree model with TABM model OOF features also increased the gap between CV and LB and was precluded
Final ensemble and post-processing
I used the public XGBclassifier and masked the predictions as illustrated in the kernel (predicted efs == 1 cases and adjusted the predictions upward). Instead of using a single weight across all model choices, I resorted to tuning
each model by a suitable separator according to the optimal CV scores
I used a ridge model as a base for the ensemble (though this idea did not work on the leaderboard with scores 0.686-0.687)
I realized that using negative weights across models could be tested here - I tuned weights manually post the ridge and improved the CV score thereby with a suitable weight set
My alternative submission was the same model structure without the masking structure
Submission details
CV score
Public leaderboard
Private Leaderboard
Model structure without masking but positive weights across single models
0.685564
0.690
0.693
Model structure without masking and any weight across the blend -
final
0.686844
0.691
0.693
Model structure with masking and any weight across the blend -
final
0.6878442
0.689
0.692
Model structure with masking for boosted trees only and any weight across the blend
0.6878162
0.689
0.692
Excluded models from the final ensemble
Model choice
Exclusion reason
TABM RMSE loss function
Good CV-LB with single model but never fitted well in the ensemble - using it almost always reduced the CV score
TABM MAE loss function
Moderate CV-LB with single model but never fitted well in the ensemble - using it almost always reduced the CV score
MLP RMSE loss function
Low CV score and excluded from single models
Boosted trees with MAE metric and loss function optimization
Low CV score and excluded from single models
Final thoughts and remarks
Wishing you the best for the future and happy learning! All the best for the upcoming Playground episodes and featured competitions as well!
Best regards,
Ravi Ramakrishnan