{ "id": 615045, "title": "2nd Place Solution – Dig4Bio Raman Transfer Learning Challenge", "content": "

Notebook: final-submit-kaggle : \nhttps://www.kaggle.com/code/daosyduyminh/final-submit-kaggle\nCore idea: build two independent pipelines (one heavy feature-engineering pipeline, one PCA/tree pipeline), train/stack them separately, then do a weighted blend at the end.

\n
    \n
  1. Data & Cleaning\nUsed the official files:\ntransfer_plate.csv (train, 3 targets: Glucose, Sodium Acetate, Magnesium Sulfate)\n96_samples.csv (test)\nRenamed spectral columns to numeric (0…2047) and forward-filled sample_id.\nSpectral values sometimes came as strings like [ … ], so they were stripped and cast to numeric.\nMissing spectral points were interpolated row-wise → guarantees consistent input shape.
    2. \n
  2. Model 1 – Upgraded FE Pipeline\nGoal: make the spectra easier for tree models by generating several “views” of the same signal.\nSteps:\nAdvanced spectral preprocessing: baseline correction (airPLS), SNV/standardization, plus Savitzky–Golay derivatives (order 1–2) to highlight edges/peaks.\nFeature views built from the cleaned spectra:\nbaseline+SNV version\n1st and 2nd derivative\npeak features (count, summed intensity, avg width)\nstatistical features (mean, std, skew, kurtosis)\n→ combined into 3 main feature sets: Combined_All, Combined_Processed, Mean_Processed.\nModels: for each of the 3 targets, train tuned ExtraTreesRegressor with shallower depth, larger leaf size, bootstrap and small ccp_alpha to reduce overfitting (because we only have 96 training samples).\n5-fold CV to get OOF predictions.\nPer-target ensemble inside this pipeline: tried 6 methods (simple avg, performance-weighted, optimal weights, linear stacking, ridge stacking, RF stacking) and picked the best CV R² per target. For Glucose and Sodium Acetate, stacking was clearly better.\nThis pipeline alone reaches ~0.90+ CV depending on the target.
    3. \n
  3. Model 2 – PCA + Linear / Trees\nThis is a lighter, more “classical ML” view to complement Model 1.\nSteps:\nLighter preprocessing (baseline + SNV) to keep the signal shape but reduce device drift.\nRecreated similar feature sets (combined, derivative, stats, peaks) so the two pipelines stay comparable.\nAdded a PCA → LinearRegression branch to get a low-variance linear model.\nAlso trained ExtraTrees on the combined features, but with slightly different hyperparams than Model 1.\nAgain, 5-fold CV + per-target stacking (linear / ridge / RF).\nIn this notebook, this pipeline actually got slightly higher CV (~0.927), so we’ll give it a higher weight in the final blend.
    4. \n
  4. Final Blending\nAfter both pipelines finished, we had:\nmodel1_linear_stacking.csv\nmodel2_linear_stacking.csv\nBoth files:\nhave the same 96 IDs\nare already post-processed (non-negative, clipped to the 1–99% range of train)\nFinal submission:\nsubmission = 0.45 * model1 + 0.55 * model2\nModel 2 got 0.55 because its CV was slightly better and more stable, but Model 1 brings useful FE-based diversity.
    5. \n
  5. Tried but not used\nPLS / classical spectroscopy regressors → didn’t beat the tree+stacking setup on 5-fold CV.\nVery large ensembles → too easy to overfit on 96 samples.\nDL → dataset too small, tabular+FE wins here.\nThe solution wins mainly because of:\ncareful spectral cleaning,\nmultiple feature views → tree-friendly input,\nOOF-based stacking per target, and\na final 2-model weighted blend (0.45 / 0.55).
    6. \n
", "comments": [], "comment_count": 0, "messages": [ { "id": 3313011, "postDate": "2025-11-08T13:16:15.043Z", "content": "

Notebook: final-submit-kaggle : \nhttps://www.kaggle.com/code/daosyduyminh/final-submit-kaggle\nCore idea: build two independent pipelines (one heavy feature-engineering pipeline, one PCA/tree pipeline), train/stack them separately, then do a weighted blend at the end.

\n
    \n
  1. Data & Cleaning\nUsed the official files:\ntransfer_plate.csv (train, 3 targets: Glucose, Sodium Acetate, Magnesium Sulfate)\n96_samples.csv (test)\nRenamed spectral columns to numeric (0…2047) and forward-filled sample_id.\nSpectral values sometimes came as strings like [ … ], so they were stripped and cast to numeric.\nMissing spectral points were interpolated row-wise → guarantees consistent input shape.
    2. \n
  2. Model 1 – Upgraded FE Pipeline\nGoal: make the spectra easier for tree models by generating several “views” of the same signal.\nSteps:\nAdvanced spectral preprocessing: baseline correction (airPLS), SNV/standardization, plus Savitzky–Golay derivatives (order 1–2) to highlight edges/peaks.\nFeature views built from the cleaned spectra:\nbaseline+SNV version\n1st and 2nd derivative\npeak features (count, summed intensity, avg width)\nstatistical features (mean, std, skew, kurtosis)\n→ combined into 3 main feature sets: Combined_All, Combined_Processed, Mean_Processed.\nModels: for each of the 3 targets, train tuned ExtraTreesRegressor with shallower depth, larger leaf size, bootstrap and small ccp_alpha to reduce overfitting (because we only have 96 training samples).\n5-fold CV to get OOF predictions.\nPer-target ensemble inside this pipeline: tried 6 methods (simple avg, performance-weighted, optimal weights, linear stacking, ridge stacking, RF stacking) and picked the best CV R² per target. For Glucose and Sodium Acetate, stacking was clearly better.\nThis pipeline alone reaches ~0.90+ CV depending on the target.
    3. \n
  3. Model 2 – PCA + Linear / Trees\nThis is a lighter, more “classical ML” view to complement Model 1.\nSteps:\nLighter preprocessing (baseline + SNV) to keep the signal shape but reduce device drift.\nRecreated similar feature sets (combined, derivative, stats, peaks) so the two pipelines stay comparable.\nAdded a PCA → LinearRegression branch to get a low-variance linear model.\nAlso trained ExtraTrees on the combined features, but with slightly different hyperparams than Model 1.\nAgain, 5-fold CV + per-target stacking (linear / ridge / RF).\nIn this notebook, this pipeline actually got slightly higher CV (~0.927), so we’ll give it a higher weight in the final blend.
    4. \n
  4. Final Blending\nAfter both pipelines finished, we had:\nmodel1_linear_stacking.csv\nmodel2_linear_stacking.csv\nBoth files:\nhave the same 96 IDs\nare already post-processed (non-negative, clipped to the 1–99% range of train)\nFinal submission:\nsubmission = 0.45 * model1 + 0.55 * model2\nModel 2 got 0.55 because its CV was slightly better and more stable, but Model 1 brings useful FE-based diversity.
    5. \n
  5. Tried but not used\nPLS / classical spectroscopy regressors → didn’t beat the tree+stacking setup on 5-fold CV.\nVery large ensembles → too easy to overfit on 96 samples.\nDL → dataset too small, tabular+FE wins here.\nThe solution wins mainly because of:\ncareful spectral cleaning,\nmultiple feature views → tree-friendly input,\nOOF-based stacking per target, and\na final 2-model weighted blend (0.45 / 0.55).
    6. \n
", "rawMarkdown": "\nNotebook: final-submit-kaggle : \nhttps://www.kaggle.com/code/daosyduyminh/final-submit-kaggle\n\nCore idea: build two independent pipelines (one heavy feature-engineering pipeline, one PCA/tree pipeline), train/stack them separately, then do a weighted blend at the end.\n\n1. Data & Cleaning\nUsed the official files:\n\n\ntransfer_plate.csv (train, 3 targets: Glucose, Sodium Acetate, Magnesium Sulfate)\n\n\n96_samples.csv (test)\n\n\nRenamed spectral columns to numeric (0…2047) and forward-filled sample_id.\n\n\nSpectral values sometimes came as strings like [ ... ], so they were stripped and cast to numeric.\n\n\nMissing spectral points were interpolated row-wise → guarantees consistent input shape.\n\n\n\n2. Model 1 – Upgraded FE Pipeline\nGoal: make the spectra easier for tree models by generating several “views” of the same signal.\nSteps:\nAdvanced spectral preprocessing: baseline correction (airPLS), SNV/standardization, plus Savitzky–Golay derivatives (order 1–2) to highlight edges/peaks.\n\n\nFeature views built from the cleaned spectra:\n\n\nbaseline+SNV version\n\n\n1st and 2nd derivative\n\n\npeak features (count, summed intensity, avg width)\n\n\nstatistical features (mean, std, skew, kurtosis)\n → combined into 3 main feature sets: Combined_All, Combined_Processed, Mean_Processed.\n\n\nModels: for each of the 3 targets, train tuned ExtraTreesRegressor with shallower depth, larger leaf size, bootstrap and small ccp_alpha to reduce overfitting (because we only have 96 training samples).\n\n\n5-fold CV to get OOF predictions.\n\n\nPer-target ensemble inside this pipeline: tried 6 methods (simple avg, performance-weighted, optimal weights, linear stacking, ridge stacking, RF stacking) and picked the best CV R² per target. For Glucose and Sodium Acetate, stacking was clearly better.\n\n\nThis pipeline alone reaches ~0.90+ CV depending on the target.\n\n3. Model 2 – PCA + Linear / Trees\nThis is a lighter, more “classical ML” view to complement Model 1.\nSteps:\nLighter preprocessing (baseline + SNV) to keep the signal shape but reduce device drift.\n\n\nRecreated similar feature sets (combined, derivative, stats, peaks) so the two pipelines stay comparable.\n\n\nAdded a PCA → LinearRegression branch to get a low-variance linear model.\n\n\nAlso trained ExtraTrees on the combined features, but with slightly different hyperparams than Model 1.\n\n\nAgain, 5-fold CV + per-target stacking (linear / ridge / RF).\n\n\nIn this notebook, this pipeline actually got slightly higher CV (~0.927), so we’ll give it a higher weight in the final blend.\n\n\n\n4. Final Blending\nAfter both pipelines finished, we had:\nmodel1_linear_stacking.csv\n\n\nmodel2_linear_stacking.csv\n\n\nBoth files:\nhave the same 96 IDs\n\n\nare already post-processed (non-negative, clipped to the 1–99% range of train)\n\n\nFinal submission:\nsubmission = 0.45 * model1 + 0.55 * model2\n\nModel 2 got 0.55 because its CV was slightly better and more stable, but Model 1 brings useful FE-based diversity.\n\n5. Tried but not used\nPLS / classical spectroscopy regressors → didn’t beat the tree+stacking setup on 5-fold CV.\n\n\nVery large ensembles → too easy to overfit on 96 samples.\n\n\nDL → dataset too small, tabular+FE wins here.\n\n\n\n\n\nThe solution wins mainly because of:\ncareful spectral cleaning,\n\n\nmultiple feature views → tree-friendly input,\n\n\nOOF-based stacking per target, and\n\n\na final 2-model weighted blend (0.45 / 0.55).\n\n\n\n\n", "votes": 2 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T13:37:33Z", "competition_id": 105802, "competition_slug": "dig-4-bio-raman-transfer-learning-challenge", "team_id": 14274218, "solution_url": "/competitions/dig-4-bio-raman-transfer-learning-challenge/writeups/2nd-place-solution-dig4bio-raman-transfer-learni" }