{
  "id": 686200,
  "title": "2nd Place Solution: Rethinking the CV–LB Assumption (First Competition Writeup)",
  "url": "/competitions/ai-emergency-medicine-turkiye/discussion/686200",
  "author_name": "Niyaz",
  "post_date": "2026-03-30T17:54:30.705000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p><strong>Team:</strong> Cranstouns <br>\n<strong>Team Members:</strong> Mohamed Niyaz M, Karthick V, Finley Saxon Lobo H, Gautham Srinivasan <br>\n<strong>Public LB:</strong> 0.70787 (Rank 5) <br>\n<strong>Private LB:</strong> 0.70608 (Rank 2) <br></p>\n<h2>Final LeaderBoard Twist</h2>\n<p>Like many Kaggle competitions, this one came down to tiny margins and one crucial decision. <br>\nWe built multiple strong solutions - but in the final moments, we tried to chose the generalized one. <br></p>\n<p>Here’s what actually happened:</p>\n<table>\n<thead>\n<tr>\n<th>Model Name</th>\n<th>Description</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>LGBM Feature-Optimized + optuna</strong></td>\n<td>Clean feature engineering + stable LightGBM pipeline + Optuna hyper parameter tuning</td>\n<td><strong>0.70785</strong></td>\n<td><strong>0.70615 🥇 (would be 1st)</strong></td>\n</tr>\n<tr>\n<td><strong>Final Submission: Multi-Model Weighted Ensemble</strong></td>\n<td>Blend of LGBM + CatBoost + XGBoost (chosen final submission)</td>\n<td><strong>0.70747</strong></td>\n<td><strong>0.70613 🥈 (actual rank: 2)</strong></td>\n</tr>\n<tr>\n<td><strong>Optuna-Tuned Boosting Model</strong></td>\n<td>Aggressively tuned boosting model via Optuna</td>\n<td>0.70757</td>\n<td>0.70590</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>If we had submitted <strong>LGBM Feature-Optimized + optuna</strong>, we would have secured <em>1st place</em>.</li>\n<li>Instead, we submitted a slightly \"safer-looking\" ensemble and finished <em>2nd</em>.</li>\n</ul>\n<p><strong>Lesson:</strong> Kaggle is not just about building models - it's about trusting the right one at the right time.</p>\n<p>We approached this competition with a simple belief:</p>\n<blockquote>\n  <p>In structured medical data, clarity beats complexity.</p>\n</blockquote>\n<p>Instead of chasing overly complex architectures, we focused on:</p>\n<ul>\n<li>Clean, meaningful features</li>\n<li>Strong validation</li>\n<li>Stable models</li>\n<li>Controlled experimentation</li>\n</ul>\n<h2>Experiment Journey</h2>\n<h3>Phase 1 - Baseline Stability: \"Start simple, stay honest\"</h3>\n<p>We began with a LightGBM baseline using minimal preprocessing:</p>\n<ul>\n<li>Median / constant imputation</li>\n<li>Stratified 5-Fold CV</li>\n<li>Light tuning</li>\n</ul>\n<p>-&gt; Stable CV <br>\n-&gt; Public LB closely matched validation <br></p>\n<blockquote>\n  <p>This gave us a trusted foundation to build on.</p>\n</blockquote>\n<h3>Phase 2 - Feature Power: \"Features &gt; Models\"</h3>\n<p>This phase delivered the largest performance gain, especially after incorporating additional data sources:</p>\n<h4>Data Utilization</h4>\n<ul>\n<li>Supplemental Set -&gt; enriched patient-level signals</li>\n<li>hospital_reference -&gt; hospital-level patterns</li>\n<li>icd10_reference -&gt; clinical code mapping\nThese helped inject domain-level context into the model.</li>\n</ul>\n<h4>Key Features</h4>\n<ul>\n<li>Missing value indicators (major boost)</li>\n<li>Ratio features (clinically meaningful scaling)</li>\n<li>Feature interactions</li>\n<li>Leakage-safe target encoding</li>\n</ul>\n<h4>Insight:</h4>\n<p>Missingness and external references together captured hidden clinical structure</p>\n<blockquote>\n  <p>This phase produced our strongest model:\n  LGBM Feature-Optimized + Optuna</p>\n</blockquote>\n<h3>Phase 3 - Model Expansion: \"Diversity helps\"</h3>\n<p>We introduced multiple models:</p>\n<ul>\n<li>LightGBM (core)</li>\n<li>CatBoost (categorical strength)</li>\n<li>XGBoost (complementary patterns)</li>\n</ul>\n<blockquote>\n  <p>Each model captured different errors, setting up for ensembling.</p>\n</blockquote>\n<h3>Phase 4 - Optimization (Optuna): \"Tuning for Stability\"</h3>\n<p>We used Optuna to tune our LGBM Feature-Optimized + Optuna:</p>\n<ul>\n<li>Learning rate vs depth</li>\n<li>Regularization</li>\n<li>Subsampling</li>\n</ul>\n<p>Result:</p>\n<ul>\n<li>Slight CV improvement</li>\n<li>Maintained stability</li>\n</ul>\n<blockquote>\n  <p>Produced the LGBM Feature-Optimized + optuna, actually ended up being our best model on the private LB.\n  We didn't choose it since this was comparitively lower than other submissions in public leaderboard. </p>\n</blockquote>\n<h3>Phase 5 - Ensembling:</h3>\n<p>We built the final:</p>\n<ul>\n<li>Multi-Model Weighted Ensemble (Final Submission)</li>\n<li>Combined LGBM + CatBoost + XGBoost</li>\n<li>Optimized ensemble model weights using <code>scipy.optimize</code></li>\n<li>Designed for robustness</li>\n</ul>\n<blockquote>\n  <p>More stable - Slightly diluted strongest signal</p>\n</blockquote>\n<h2>The Hidden Winner</h2>\n<p>This model (our <code>LGBM Feature-Optimized + optuna</code> version) was different.</p>\n<ul>\n<li>It combined Optuna tuning with an extra layer of targeted features.</li>\n<li>It wasn't our most complex ensemble.</li>\n</ul>\n<p>But it had something better:</p>\n<blockquote>\n  <p>Perfect alignment between CV and real leaderboard behavior</p>\n</blockquote>\n<p>Why it worked so well:</p>\n<ul>\n<li>Clean feature pipeline combined with effective Optuna tuning</li>\n<li>Minimal overfitting despite the hyperparameter optimization</li>\n<li>Strong fold consistency</li>\n</ul>\n<p>The irony:\nWhile we were trying to \"play it safe\" and improve performance with robust ensembles:</p>\n<ul>\n<li>We overlooked this highly optimized individual model</li>\n<li>And ended up not selecting what would have been the 1st place submission</li>\n</ul>\n<h2>The Final Decision Mistake</h2>\n<p>At submission time, we had to choose:</p>\n<ul>\n<li>Higher public score stability (ensemble)</li>\n<li>Better CV consistency (single model)</li>\n</ul>\n<p>We chose:</p>\n<blockquote>\n  <p>weighted-ensemble</p>\n</blockquote>\n<p>Because:</p>\n<ul>\n<li>It looked more \"robust\"</li>\n<li>It reduced variance</li>\n<li>It felt safer under leaderboard shake-up</li>\n</ul>\n<p>But:</p>\n<ul>\n<li>The simpler model -&gt; Had already captured the true signal</li>\n</ul>\n<p>What This Competition Taught Us</p>\n<ol>\n<li>CV &gt; Public LB<ul>\n<li>Public leaderboard can mislead</li>\n<li>Your CV is your real compass </li></ul></li>\n<li>Trust Your Best Signal<ul>\n<li>We had the winning model</li>\n<li>We just didn’t trust it enough </li></ul></li>\n<li>Tiny Margins Decide Everything</li>\n</ol>\n<h2>Reproducibility</h2>\n<ul>\n<li>5-Fold Stratified CV</li>\n<li>Fixed random seeds</li>\n<li>Fully deterministic pipelines</li>\n<li>Models:<ul>\n<li>LightGBM</li>\n<li>CatBoost</li>\n<li>XGBoost</li></ul></li>\n</ul>\n<p>This was our first win in Kaggle competition ❤️</p>",
  "messages": [
    {
      "id": 3431817,
      "postDate": "2026-03-30T17:54:30.707Z",
      "content": "<p><strong>Team:</strong> Cranstouns <br>\n<strong>Team Members:</strong> Mohamed Niyaz M, Karthick V, Finley Saxon Lobo H, Gautham Srinivasan <br>\n<strong>Public LB:</strong> 0.70787 (Rank 5) <br>\n<strong>Private LB:</strong> 0.70608 (Rank 2) <br></p>\n<h2>Final LeaderBoard Twist</h2>\n<p>Like many Kaggle competitions, this one came down to tiny margins and one crucial decision. <br>\nWe built multiple strong solutions - but in the final moments, we tried to chose the generalized one. <br></p>\n<p>Here’s what actually happened:</p>\n<table>\n<thead>\n<tr>\n<th>Model Name</th>\n<th>Description</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>LGBM Feature-Optimized + optuna</strong></td>\n<td>Clean feature engineering + stable LightGBM pipeline + Optuna hyper parameter tuning</td>\n<td><strong>0.70785</strong></td>\n<td><strong>0.70615 🥇 (would be 1st)</strong></td>\n</tr>\n<tr>\n<td><strong>Final Submission: Multi-Model Weighted Ensemble</strong></td>\n<td>Blend of LGBM + CatBoost + XGBoost (chosen final submission)</td>\n<td><strong>0.70747</strong></td>\n<td><strong>0.70613 🥈 (actual rank: 2)</strong></td>\n</tr>\n<tr>\n<td><strong>Optuna-Tuned Boosting Model</strong></td>\n<td>Aggressively tuned boosting model via Optuna</td>\n<td>0.70757</td>\n<td>0.70590</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>If we had submitted <strong>LGBM Feature-Optimized + optuna</strong>, we would have secured <em>1st place</em>.</li>\n<li>Instead, we submitted a slightly \"safer-looking\" ensemble and finished <em>2nd</em>.</li>\n</ul>\n<p><strong>Lesson:</strong> Kaggle is not just about building models - it's about trusting the right one at the right time.</p>\n<p>We approached this competition with a simple belief:</p>\n<blockquote>\n  <p>In structured medical data, clarity beats complexity.</p>\n</blockquote>\n<p>Instead of chasing overly complex architectures, we focused on:</p>\n<ul>\n<li>Clean, meaningful features</li>\n<li>Strong validation</li>\n<li>Stable models</li>\n<li>Controlled experimentation</li>\n</ul>\n<h2>Experiment Journey</h2>\n<h3>Phase 1 - Baseline Stability: \"Start simple, stay honest\"</h3>\n<p>We began with a LightGBM baseline using minimal preprocessing:</p>\n<ul>\n<li>Median / constant imputation</li>\n<li>Stratified 5-Fold CV</li>\n<li>Light tuning</li>\n</ul>\n<p>-&gt; Stable CV <br>\n-&gt; Public LB closely matched validation <br></p>\n<blockquote>\n  <p>This gave us a trusted foundation to build on.</p>\n</blockquote>\n<h3>Phase 2 - Feature Power: \"Features &gt; Models\"</h3>\n<p>This phase delivered the largest performance gain, especially after incorporating additional data sources:</p>\n<h4>Data Utilization</h4>\n<ul>\n<li>Supplemental Set -&gt; enriched patient-level signals</li>\n<li>hospital_reference -&gt; hospital-level patterns</li>\n<li>icd10_reference -&gt; clinical code mapping\nThese helped inject domain-level context into the model.</li>\n</ul>\n<h4>Key Features</h4>\n<ul>\n<li>Missing value indicators (major boost)</li>\n<li>Ratio features (clinically meaningful scaling)</li>\n<li>Feature interactions</li>\n<li>Leakage-safe target encoding</li>\n</ul>\n<h4>Insight:</h4>\n<p>Missingness and external references together captured hidden clinical structure</p>\n<blockquote>\n  <p>This phase produced our strongest model:\n  LGBM Feature-Optimized + Optuna</p>\n</blockquote>\n<h3>Phase 3 - Model Expansion: \"Diversity helps\"</h3>\n<p>We introduced multiple models:</p>\n<ul>\n<li>LightGBM (core)</li>\n<li>CatBoost (categorical strength)</li>\n<li>XGBoost (complementary patterns)</li>\n</ul>\n<blockquote>\n  <p>Each model captured different errors, setting up for ensembling.</p>\n</blockquote>\n<h3>Phase 4 - Optimization (Optuna): \"Tuning for Stability\"</h3>\n<p>We used Optuna to tune our LGBM Feature-Optimized + Optuna:</p>\n<ul>\n<li>Learning rate vs depth</li>\n<li>Regularization</li>\n<li>Subsampling</li>\n</ul>\n<p>Result:</p>\n<ul>\n<li>Slight CV improvement</li>\n<li>Maintained stability</li>\n</ul>\n<blockquote>\n  <p>Produced the LGBM Feature-Optimized + optuna, actually ended up being our best model on the private LB.\n  We didn't choose it since this was comparitively lower than other submissions in public leaderboard. </p>\n</blockquote>\n<h3>Phase 5 - Ensembling:</h3>\n<p>We built the final:</p>\n<ul>\n<li>Multi-Model Weighted Ensemble (Final Submission)</li>\n<li>Combined LGBM + CatBoost + XGBoost</li>\n<li>Optimized ensemble model weights using <code>scipy.optimize</code></li>\n<li>Designed for robustness</li>\n</ul>\n<blockquote>\n  <p>More stable - Slightly diluted strongest signal</p>\n</blockquote>\n<h2>The Hidden Winner</h2>\n<p>This model (our <code>LGBM Feature-Optimized + optuna</code> version) was different.</p>\n<ul>\n<li>It combined Optuna tuning with an extra layer of targeted features.</li>\n<li>It wasn't our most complex ensemble.</li>\n</ul>\n<p>But it had something better:</p>\n<blockquote>\n  <p>Perfect alignment between CV and real leaderboard behavior</p>\n</blockquote>\n<p>Why it worked so well:</p>\n<ul>\n<li>Clean feature pipeline combined with effective Optuna tuning</li>\n<li>Minimal overfitting despite the hyperparameter optimization</li>\n<li>Strong fold consistency</li>\n</ul>\n<p>The irony:\nWhile we were trying to \"play it safe\" and improve performance with robust ensembles:</p>\n<ul>\n<li>We overlooked this highly optimized individual model</li>\n<li>And ended up not selecting what would have been the 1st place submission</li>\n</ul>\n<h2>The Final Decision Mistake</h2>\n<p>At submission time, we had to choose:</p>\n<ul>\n<li>Higher public score stability (ensemble)</li>\n<li>Better CV consistency (single model)</li>\n</ul>\n<p>We chose:</p>\n<blockquote>\n  <p>weighted-ensemble</p>\n</blockquote>\n<p>Because:</p>\n<ul>\n<li>It looked more \"robust\"</li>\n<li>It reduced variance</li>\n<li>It felt safer under leaderboard shake-up</li>\n</ul>\n<p>But:</p>\n<ul>\n<li>The simpler model -&gt; Had already captured the true signal</li>\n</ul>\n<p>What This Competition Taught Us</p>\n<ol>\n<li>CV &gt; Public LB<ul>\n<li>Public leaderboard can mislead</li>\n<li>Your CV is your real compass </li></ul></li>\n<li>Trust Your Best Signal<ul>\n<li>We had the winning model</li>\n<li>We just didn’t trust it enough </li></ul></li>\n<li>Tiny Margins Decide Everything</li>\n</ol>\n<h2>Reproducibility</h2>\n<ul>\n<li>5-Fold Stratified CV</li>\n<li>Fixed random seeds</li>\n<li>Fully deterministic pipelines</li>\n<li>Models:<ul>\n<li>LightGBM</li>\n<li>CatBoost</li>\n<li>XGBoost</li></ul></li>\n</ul>\n<p>This was our first win in Kaggle competition ❤️</p>",
      "rawMarkdown": "**Team:** Cranstouns <br>\n**Team Members:** Mohamed Niyaz M, Karthick V, Finley Saxon Lobo H, Gautham Srinivasan <br>\n**Public LB:** 0.70787 (Rank 5) <br>\n**Private LB:** 0.70608 (Rank 2) <br>\n\n## Final LeaderBoard Twist\nLike many Kaggle competitions, this one came down to tiny margins and one crucial decision. <br>\nWe built multiple strong solutions - but in the final moments, we tried to chose the generalized one. <br>\n\nHere’s what actually happened:\n| Model Name | Description | Public LB | Private LB | \n| --- | --- | --- | --- |  \n| **LGBM Feature-Optimized + optuna** | Clean feature engineering + stable LightGBM pipeline + Optuna hyper parameter tuning | **0.70785** | **0.70615 🥇 (would be 1st)** |\n| **Final Submission: Multi-Model Weighted Ensemble** | Blend of LGBM + CatBoost + XGBoost (chosen final submission) | **0.70747** | **0.70613 🥈 (actual rank: 2)** |\n| **Optuna-Tuned Boosting Model** | Aggressively tuned boosting model via Optuna | 0.70757 | 0.70590 |\n\n\n- If we had submitted **LGBM Feature-Optimized + optuna**, we would have secured *1st place*.\n- Instead, we submitted a slightly \"safer-looking\" ensemble and finished *2nd*.\n\n**Lesson:** Kaggle is not just about building models - it's about trusting the right one at the right time.\n\nWe approached this competition with a simple belief:\n> In structured medical data, clarity beats complexity.\n\nInstead of chasing overly complex architectures, we focused on:\n- Clean, meaningful features\n- Strong validation\n- Stable models\n- Controlled experimentation\n\n## Experiment Journey\n\n### Phase 1 - Baseline Stability: \"Start simple, stay honest\"\nWe began with a LightGBM baseline using minimal preprocessing:\n- Median / constant imputation\n- Stratified 5-Fold CV\n- Light tuning\n\n-> Stable CV <br>\n-> Public LB closely matched validation <br>\n\n> This gave us a trusted foundation to build on.\n\n### Phase 2 - Feature Power: \"Features > Models\"\nThis phase delivered the largest performance gain, especially after incorporating additional data sources:\n\n#### Data Utilization\n- Supplemental Set -> enriched patient-level signals\n- hospital_reference -> hospital-level patterns\n- icd10_reference -> clinical code mapping\nThese helped inject domain-level context into the model.\n\n#### Key Features\n- Missing value indicators (major boost)\n- Ratio features (clinically meaningful scaling)\n- Feature interactions\n- Leakage-safe target encoding\n\n#### Insight:\nMissingness and external references together captured hidden clinical structure\n\n> This phase produced our strongest model:\nLGBM Feature-Optimized + Optuna\n\n### Phase 3 - Model Expansion: \"Diversity helps\"\nWe introduced multiple models:\n- LightGBM (core)\n- CatBoost (categorical strength)\n- XGBoost (complementary patterns)\n> Each model captured different errors, setting up for ensembling.\n\n### Phase 4 - Optimization (Optuna): \"Tuning for Stability\"\nWe used Optuna to tune our LGBM Feature-Optimized + Optuna:\n- Learning rate vs depth\n- Regularization\n- Subsampling\n\nResult:\n- Slight CV improvement\n- Maintained stability\n\n> Produced the LGBM Feature-Optimized + optuna, actually ended up being our best model on the private LB.\n> We didn't choose it since this was comparitively lower than other submissions in public leaderboard. \n\n### Phase 5 - Ensembling:\nWe built the final:\n\n- Multi-Model Weighted Ensemble (Final Submission)\n- Combined LGBM + CatBoost + XGBoost\n- Optimized ensemble model weights using `scipy.optimize`\n- Designed for robustness\n\n> More stable - Slightly diluted strongest signal\n\n## The Hidden Winner\nThis model (our `LGBM Feature-Optimized + optuna` version) was different.\n- It combined Optuna tuning with an extra layer of targeted features.\n- It wasn't our most complex ensemble.\n\nBut it had something better:\n> Perfect alignment between CV and real leaderboard behavior\n\nWhy it worked so well:\n- Clean feature pipeline combined with effective Optuna tuning\n- Minimal overfitting despite the hyperparameter optimization\n- Strong fold consistency\n\nThe irony:\nWhile we were trying to \"play it safe\" and improve performance with robust ensembles:\n- We overlooked this highly optimized individual model\n- And ended up not selecting what would have been the 1st place submission\n\n## The Final Decision Mistake\nAt submission time, we had to choose:\n- Higher public score stability (ensemble)\n- Better CV consistency (single model)\n\nWe chose:\n> weighted-ensemble\n\nBecause:\n- It looked more \"robust\"\n- It reduced variance\n- It felt safer under leaderboard shake-up\n\nBut:\n- The simpler model -> Had already captured the true signal\n\nWhat This Competition Taught Us\n1. CV > Public LB\n    - Public leaderboard can mislead\n    - Your CV is your real compass \n2. Trust Your Best Signal\n    - We had the winning model\n    - We just didn’t trust it enough \n3. Tiny Margins Decide Everything\n\n## Reproducibility\n- 5-Fold Stratified CV\n- Fixed random seeds\n- Fully deterministic pipelines\n- Models:\n    - LightGBM\n    - CatBoost\n    - XGBoost\n\nThis was our first win in Kaggle competition ❤️",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3431817": "**Team:** Cranstouns <br>\n**Team Members:** Mohamed Niyaz M, Karthick V, Finley Saxon Lobo H, Gautham Srinivasan <br>\n**Public LB:** 0.70787 (Rank 5) <br>\n**Private LB:** 0.70608 (Rank 2) <br>\n\n## Final LeaderBoard Twist\nLike many Kaggle competitions, this one came down to tiny margins and one crucial decision. <br>\nWe built multiple strong solutions - but in the final moments, we tried to chose the generalized one. <br>\n\nHere’s what actually happened:\n| Model Name | Description | Public LB | Private LB | \n| --- | --- | --- | --- |  \n| **LGBM Feature-Optimized + optuna** | Clean feature engineering + stable LightGBM pipeline + Optuna hyper parameter tuning | **0.70785** | **0.70615 🥇 (would be 1st)** |\n| **Final Submission: Multi-Model Weighted Ensemble** | Blend of LGBM + CatBoost + XGBoost (chosen final submission) | **0.70747** | **0.70613 🥈 (actual rank: 2)** |\n| **Optuna-Tuned Boosting Model** | Aggressively tuned boosting model via Optuna | 0.70757 | 0.70590 |\n\n\n- If we had submitted **LGBM Feature-Optimized + optuna**, we would have secured *1st place*.\n- Instead, we submitted a slightly \"safer-looking\" ensemble and finished *2nd*.\n\n**Lesson:** Kaggle is not just about building models - it's about trusting the right one at the right time.\n\nWe approached this competition with a simple belief:\n> In structured medical data, clarity beats complexity.\n\nInstead of chasing overly complex architectures, we focused on:\n- Clean, meaningful features\n- Strong validation\n- Stable models\n- Controlled experimentation\n\n## Experiment Journey\n\n### Phase 1 - Baseline Stability: \"Start simple, stay honest\"\nWe began with a LightGBM baseline using minimal preprocessing:\n- Median / constant imputation\n- Stratified 5-Fold CV\n- Light tuning\n\n-> Stable CV <br>\n-> Public LB closely matched validation <br>\n\n> This gave us a trusted foundation to build on.\n\n### Phase 2 - Feature Power: \"Features > Models\"\nThis phase delivered the largest performance gain, especially after incorporating additional data sources:\n\n#### Data Utilization\n- Supplemental Set -> enriched patient-level signals\n- hospital_reference -> hospital-level patterns\n- icd10_reference -> clinical code mapping\nThese helped inject domain-level context into the model.\n\n#### Key Features\n- Missing value indicators (major boost)\n- Ratio features (clinically meaningful scaling)\n- Feature interactions\n- Leakage-safe target encoding\n\n#### Insight:\nMissingness and external references together captured hidden clinical structure\n\n> This phase produced our strongest model:\nLGBM Feature-Optimized + Optuna\n\n### Phase 3 - Model Expansion: \"Diversity helps\"\nWe introduced multiple models:\n- LightGBM (core)\n- CatBoost (categorical strength)\n- XGBoost (complementary patterns)\n> Each model captured different errors, setting up for ensembling.\n\n### Phase 4 - Optimization (Optuna): \"Tuning for Stability\"\nWe used Optuna to tune our LGBM Feature-Optimized + Optuna:\n- Learning rate vs depth\n- Regularization\n- Subsampling\n\nResult:\n- Slight CV improvement\n- Maintained stability\n\n> Produced the LGBM Feature-Optimized + optuna, actually ended up being our best model on the private LB.\n> We didn't choose it since this was comparitively lower than other submissions in public leaderboard. \n\n### Phase 5 - Ensembling:\nWe built the final:\n\n- Multi-Model Weighted Ensemble (Final Submission)\n- Combined LGBM + CatBoost + XGBoost\n- Optimized ensemble model weights using `scipy.optimize`\n- Designed for robustness\n\n> More stable - Slightly diluted strongest signal\n\n## The Hidden Winner\nThis model (our `LGBM Feature-Optimized + optuna` version) was different.\n- It combined Optuna tuning with an extra layer of targeted features.\n- It wasn't our most complex ensemble.\n\nBut it had something better:\n> Perfect alignment between CV and real leaderboard behavior\n\nWhy it worked so well:\n- Clean feature pipeline combined with effective Optuna tuning\n- Minimal overfitting despite the hyperparameter optimization\n- Strong fold consistency\n\nThe irony:\nWhile we were trying to \"play it safe\" and improve performance with robust ensembles:\n- We overlooked this highly optimized individual model\n- And ended up not selecting what would have been the 1st place submission\n\n## The Final Decision Mistake\nAt submission time, we had to choose:\n- Higher public score stability (ensemble)\n- Better CV consistency (single model)\n\nWe chose:\n> weighted-ensemble\n\nBecause:\n- It looked more \"robust\"\n- It reduced variance\n- It felt safer under leaderboard shake-up\n\nBut:\n- The simpler model -> Had already captured the true signal\n\nWhat This Competition Taught Us\n1. CV > Public LB\n    - Public leaderboard can mislead\n    - Your CV is your real compass \n2. Trust Your Best Signal\n    - We had the winning model\n    - We just didn’t trust it enough \n3. Tiny Margins Decide Everything\n\n## Reproducibility\n- 5-Fold Stratified CV\n- Fixed random seeds\n- Fully deterministic pipelines\n- Models:\n    - LightGBM\n    - CatBoost\n    - XGBoost\n\nThis was our first win in Kaggle competition ❤️"
  }
}