# 2nd Place Solution: Rethinking the CV–LB Assumption (First Competition Writeup)

Team:
Cranstouns
Team Members:
Mohamed Niyaz M, Karthick V, Finley Saxon Lobo H, Gautham Srinivasan
Public LB:
0.70787 (Rank 5)
Private LB:
0.70608 (Rank 2)
Final LeaderBoard Twist
Like many Kaggle competitions, this one came down to tiny margins and one crucial decision.
We built multiple strong solutions - but in the final moments, we tried to chose the generalized one.
Here’s what actually happened:
Model Name
Description
Public LB
Private LB
LGBM Feature-Optimized + optuna
Clean feature engineering + stable LightGBM pipeline + Optuna hyper parameter tuning
0.70785
0.70615 🥇 (would be 1st)
Final Submission: Multi-Model Weighted Ensemble
Blend of LGBM + CatBoost + XGBoost (chosen final submission)
0.70747
0.70613 🥈 (actual rank: 2)
Optuna-Tuned Boosting Model
Aggressively tuned boosting model via Optuna
0.70757
0.70590
If we had submitted
LGBM Feature-Optimized + optuna
, we would have secured
1st place
.
Instead, we submitted a slightly "safer-looking" ensemble and finished
2nd
.
Lesson:
Kaggle is not just about building models - it's about trusting the right one at the right time.
We approached this competition with a simple belief:
In structured medical data, clarity beats complexity.
Instead of chasing overly complex architectures, we focused on:
Clean, meaningful features
Strong validation
Stable models
Controlled experimentation
Experiment Journey
Phase 1 - Baseline Stability: "Start simple, stay honest"
We began with a LightGBM baseline using minimal preprocessing:
Median / constant imputation
Stratified 5-Fold CV
Light tuning
-> Stable CV
-> Public LB closely matched validation
This gave us a trusted foundation to build on.
Phase 2 - Feature Power: "Features > Models"
This phase delivered the largest performance gain, especially after incorporating additional data sources:
Data Utilization
Supplemental Set -> enriched patient-level signals
hospital_reference -> hospital-level patterns
icd10_reference -> clinical code mapping
These helped inject domain-level context into the model.
Key Features
Missing value indicators (major boost)
Ratio features (clinically meaningful scaling)
Feature interactions
Leakage-safe target encoding
Insight:
Missingness and external references together captured hidden clinical structure
This phase produced our strongest model:
  LGBM Feature-Optimized + Optuna
Phase 3 - Model Expansion: "Diversity helps"
We introduced multiple models:
LightGBM (core)
CatBoost (categorical strength)
XGBoost (complementary patterns)
Each model captured different errors, setting up for ensembling.
Phase 4 - Optimization (Optuna): "Tuning for Stability"
We used Optuna to tune our LGBM Feature-Optimized + Optuna:
Learning rate vs depth
Regularization
Subsampling
Result:
Slight CV improvement
Maintained stability
Produced the LGBM Feature-Optimized + optuna, actually ended up being our best model on the private LB.
  We didn't choose it since this was comparitively lower than other submissions in public leaderboard.
Phase 5 - Ensembling:
We built the final:
Multi-Model Weighted Ensemble (Final Submission)
Combined LGBM + CatBoost + XGBoost
Optimized ensemble model weights using
scipy.optimize
Designed for robustness
More stable - Slightly diluted strongest signal
The Hidden Winner
This model (our
LGBM Feature-Optimized + optuna
version) was different.
It combined Optuna tuning with an extra layer of targeted features.
It wasn't our most complex ensemble.
But it had something better:
Perfect alignment between CV and real leaderboard behavior
Why it worked so well:
Clean feature pipeline combined with effective Optuna tuning
Minimal overfitting despite the hyperparameter optimization
Strong fold consistency
The irony:
While we were trying to "play it safe" and improve performance with robust ensembles:
We overlooked this highly optimized individual model
And ended up not selecting what would have been the 1st place submission
The Final Decision Mistake
At submission time, we had to choose:
Higher public score stability (ensemble)
Better CV consistency (single model)
We chose:
weighted-ensemble
Because:
It looked more "robust"
It reduced variance
It felt safer under leaderboard shake-up
But:
The simpler model -> Had already captured the true signal
What This Competition Taught Us
CV > Public LB
Public leaderboard can mislead
Your CV is your real compass
Trust Your Best Signal
We had the winning model
We just didn’t trust it enough
Tiny Margins Decide Everything
Reproducibility
5-Fold Stratified CV
Fixed random seeds
Fully deterministic pipelines
Models:
LightGBM
CatBoost
XGBoost
This was our first win in Kaggle competition ❤️