First, I would like to thank the BS KIDS team, Kaggle team and all kagglers for this amazing competition.
\nIt was hard to handle qwk metric with so few test samples without fall in the public LB climbing!
\nHere https://www.kaggle.com/steubk/dsb2019-bs-26-lgb-auc-xgb you can find (100% kaggle kernel) solution.
\nThis is a summary of my solution:
\ny = (2*0.7*y_lgb + 2*0.3*y_xgb + 2*3*y_auc_solved + 3*y_first_try_success)/5
with y_lgb as lgb regression (rmse) , y_xgb as xgb regression (rmse), y_auc_solved as binary classification(auc) for solved assessments (0/123) and y_first_try_success as binary classifcation (auc) for first try success assessments (012/3)
\nData Augmentation\nthanks to @akasyanama13 and his kernel https://www.kaggle.com/akasyanama13/another-way-of-validation I added about 11% training samples (2.018/17.690) in public and many more (75% ?) in private.
Validation Strategy\nFor each learning model trained a 5 GroupKFold on installation_id, with fixed number of iterations (no early stopping), averaged with 5 seeds and validated with median of 5.000 shuffled truncated samples: this guaranteed the stability of the rmse and qwk on cv and qwk on private.
Feature enginering \nI generated more than thousand features (1.047), starting from public kernels (@artgor https://www.kaggle.com/artgor/quick-and-dirty-regression , @braquino https://www.kaggle.com/braquino/890-features and others)\nand then added features for:\ncurrent-assessment, last-game, last-activity, last-assessment, \"correct\" event_data and encoded timestamp hour in cyclical continuous features.
Feature selection \nI selected 128 features with RFE.\nI done first elimation steps removing correlate features (corrcoef > 0.9999) and features with minimal gain in lgb regression model. \nFor last step (from 148 to 128) I computed rmse and qwk for each single feature and removed features for which the gain for rmse AND qwk was negative regardless of the lgb feature importance.\nLast step gives an improvement for qwk in cv and private (cv: 0,5616 --> 0,5624, private: 0.545 --> 0.553)
From RMSE to QWK (aka threshold definition)\nI built a simple Bayesian optimization and validated with 5.000 shuffled truncated samples from train.
The Magic Revisited (Simple models and ensembling)\nThis was the score before blending:
| model | \ncv | \ncv std | \n(public) | \nprivate | \n
|---|---|---|---|---|
| xgb | \n0.5574 | \n0.008 | \n0.544 | \n0.550 | \n
| lgb | \n0.5638 | \n0.008 | \n0.538 | \n0.553 | \n
A simple blend (0.7*y_lgb + 0.3*y_xgb) of xgb and lgb model has given no imporvements on cv and public but some improvements in private (0.555).
In the last days I tried some binary classifications with auc:\nbinary classification for solved assessments (0/123) and binary classification for first try success (012/3)
\nThe ordering induced by auc and the small number of elements for qwk made me think that I could try to blend the auc models directly with the regression models: In fact the blend gives an improvement in cv (0.5680) and an improvement of 0.002 over the simple blend in private. (qwk final score:0.557)
\nThank you for reading !
", "comments": [ { "id": 745570, "author_name": "Kevin Liu", "author_url": "", "post_date": "2020-02-14T00:00:21.693000", "content": "Your way to blend in the binary classification models is very inspiring. Code is super neat. Thanks for sharing and congratulations on your silver medal. A bit more luck should have landed you in the gold zone.
", "votes": 0, "replies": [] }, { "id": 728025, "author_name": "Serigne ", "author_url": "", "post_date": "2020-01-24T10:49:23.963000", "content": "Big congrats @steubk
", "votes": 0, "replies": [ { "id": 728123, "author_name": "steubk", "author_url": "", "post_date": "2020-01-24T12:25:16.513000", "content": "Grazie @serigne !
", "votes": 0, "replies": [] } ] }, { "id": 727751, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-24T02:21:00.873000", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @steubk
", "votes": 0, "replies": [ { "id": 727884, "author_name": "steubk", "author_url": "", "post_date": "2020-01-24T06:46:03.760000", "content": "thank you @veeralakrishna !
", "votes": 0, "replies": [] } ] } ], "comment_count": 5, "messages": [ { "id": 727364, "postDate": "2020-01-23T16:50:16.603Z", "content": "First, I would like to thank the BS KIDS team, Kaggle team and all kagglers for this amazing competition.
\nIt was hard to handle qwk metric with so few test samples without fall in the public LB climbing!
\nHere https://www.kaggle.com/steubk/dsb2019-bs-26-lgb-auc-xgb you can find (100% kaggle kernel) solution.
\nThis is a summary of my solution:
\ny = (2*0.7*y_lgb + 2*0.3*y_xgb + 2*3*y_auc_solved + 3*y_first_try_success)/5
with y_lgb as lgb regression (rmse) , y_xgb as xgb regression (rmse), y_auc_solved as binary classification(auc) for solved assessments (0/123) and y_first_try_success as binary classifcation (auc) for first try success assessments (012/3)
\nData Augmentation\nthanks to @akasyanama13 and his kernel https://www.kaggle.com/akasyanama13/another-way-of-validation I added about 11% training samples (2.018/17.690) in public and many more (75% ?) in private.
Validation Strategy\nFor each learning model trained a 5 GroupKFold on installation_id, with fixed number of iterations (no early stopping), averaged with 5 seeds and validated with median of 5.000 shuffled truncated samples: this guaranteed the stability of the rmse and qwk on cv and qwk on private.
Feature enginering \nI generated more than thousand features (1.047), starting from public kernels (@artgor https://www.kaggle.com/artgor/quick-and-dirty-regression , @braquino https://www.kaggle.com/braquino/890-features and others)\nand then added features for:\ncurrent-assessment, last-game, last-activity, last-assessment, \"correct\" event_data and encoded timestamp hour in cyclical continuous features.
Feature selection \nI selected 128 features with RFE.\nI done first elimation steps removing correlate features (corrcoef > 0.9999) and features with minimal gain in lgb regression model. \nFor last step (from 148 to 128) I computed rmse and qwk for each single feature and removed features for which the gain for rmse AND qwk was negative regardless of the lgb feature importance.\nLast step gives an improvement for qwk in cv and private (cv: 0,5616 --> 0,5624, private: 0.545 --> 0.553)
From RMSE to QWK (aka threshold definition)\nI built a simple Bayesian optimization and validated with 5.000 shuffled truncated samples from train.
The Magic Revisited (Simple models and ensembling)\nThis was the score before blending:
| model | \ncv | \ncv std | \n(public) | \nprivate | \n
|---|---|---|---|---|
| xgb | \n0.5574 | \n0.008 | \n0.544 | \n0.550 | \n
| lgb | \n0.5638 | \n0.008 | \n0.538 | \n0.553 | \n
A simple blend (0.7*y_lgb + 0.3*y_xgb) of xgb and lgb model has given no imporvements on cv and public but some improvements in private (0.555).
In the last days I tried some binary classifications with auc:\nbinary classification for solved assessments (0/123) and binary classification for first try success (012/3)
\nThe ordering induced by auc and the small number of elements for qwk made me think that I could try to blend the auc models directly with the regression models: In fact the blend gives an improvement in cv (0.5680) and an improvement of 0.002 over the simple blend in private. (qwk final score:0.557)
\nThank you for reading !
", "rawMarkdown": "First, I would like to thank the BS KIDS team, Kaggle team and all kagglers for this amazing competition. \n\nIt was hard to handle qwk metric with so few test samples without fall in the public LB climbing! \n\nHere https://www.kaggle.com/steubk/dsb2019-bs-26-lgb-auc-xgb you can find (100% kaggle kernel) solution.\n\nThis is a summary of my solution:\n\n- The Magic\nFor those about to \"hey, where is the kaggle's magic ?\", this is my _magic blend_:\n\n```y = (2*0.7*y_lgb + 2*0.3*y_xgb + 2*3*y_auc_solved + 3*y_first_try_success)/5```\n\nwith y\\_lgb as lgb regression (rmse) , y\\_xgb as xgb regression (rmse), y\\_auc\\_solved as binary classification(auc) for solved assessments (0/123) and y\\_first\\_try\\_success as binary classifcation (auc) for first try success assessments (012/3) \n\n- Data Augmentation\nthanks to @akasyanama13 and his kernel https://www.kaggle.com/akasyanama13/another-way-of-validation I added about 11% training samples (2.018/17.690) in public and many more (75% ?) in private. \n\n- Validation Strategy\nFor each learning model trained a 5 GroupKFold on installation_id, with fixed number of iterations (no early stopping), averaged with 5 seeds and validated with median of 5.000 shuffled truncated samples: this guaranteed the stability of the rmse and qwk on cv and qwk on private. \n\n- Feature enginering \nI generated more than thousand features (1.047), starting from public kernels (@artgor https://www.kaggle.com/artgor/quick-and-dirty-regression , @braquino https://www.kaggle.com/braquino/890-features and others)\nand then added features for:\ncurrent-assessment, last-game, last-activity, last-assessment, \"correct\" event_data and encoded timestamp hour in cyclical continuous features.\n \n- Feature selection \nI selected 128 features with RFE.\nI done first elimation steps removing correlate features (corrcoef > 0.9999) and features with minimal gain in lgb regression model. \nFor last step (from 148 to 128) I computed rmse and qwk for each single feature and removed features for which the gain for rmse AND qwk was negative regardless of the lgb feature importance.\nLast step gives an improvement for qwk in cv and private (cv: 0,5616 --> 0,5624, private: 0.545 --> 0.553)\n\n- From RMSE to QWK (aka threshold definition)\nI built a simple Bayesian optimization and validated with 5.000 shuffled truncated samples from train.\n\n- The Magic Revisited (Simple models and ensembling)\nThis was the score before blending: \n\n| model | cv |cv std|(public)|private| \n| --- | --- |---|---|---|\n|xgb|0.5574|0.008|0.544|0.550|\n|lgb|0.5638|0.008|0.538|0.553|\n\nA simple blend ```(0.7*y_lgb + 0.3*y_xgb)``` of xgb and lgb model has given no imporvements on cv and public but some improvements in private (0.555).\n\nIn the last days I tried some binary classifications with auc:\nbinary classification for solved assessments (0/123) and binary classification for first try success (012/3)\n\nThe ordering induced by auc and the small number of elements for qwk made me think that I could try to blend the auc models directly with the regression models: In fact the blend gives an improvement in cv (0.5680) and an improvement of 0.002 over the simple blend in private. (qwk final score:0.557) \n\n\nThank you for reading !", "votes": 18 }, { "id": 745570, "postDate": "2020-02-14T00:00:21.693Z", "content": "Your way to blend in the binary classification models is very inspiring. Code is super neat. Thanks for sharing and congratulations on your silver medal. A bit more luck should have landed you in the gold zone.
", "rawMarkdown": "Your way to blend in the binary classification models is very inspiring. Code is super neat. Thanks for sharing and congratulations on your silver medal. A bit more luck should have landed you in the gold zone." }, { "id": 728025, "postDate": "2020-01-24T10:49:23.963Z", "content": "Big congrats @steubk
", "rawMarkdown": "Big congrats @steubk ", "replies": [ { "id": 728123, "postDate": "2020-01-24T12:25:16.513Z", "content": "Grazie @serigne !
", "rawMarkdown": "Grazie @serigne !" } ] }, { "id": 727751, "postDate": "2020-01-24T02:21:00.873Z", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @steubk
", "rawMarkdown": "Congratulations\nThanks for Sharing your Approach & Insights!! @steubk ", "replies": [ { "id": 727884, "postDate": "2020-01-24T06:46:03.760Z", "content": "thank you @veeralakrishna !
", "rawMarkdown": "thank you @veeralakrishna !" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T06:53:41Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 4180066, "solution_url": "/competitions/data-science-bowl-2019/writeups/steubk-19th-place-solution" }