Here's a simple solution that gives you a 29th place.
\n1) Estimate probability of EFS by fitting a CatBoostClassifier.
\n2) Estimate EFS_TIME for the subsample where EFS==1 using CatBoostRegressor.
\nEstimates in 1) and 2) are obtained OOF using 10 folds.
Combine the estimates in Stage 1 to create a metric that maximizes (EDIT: not CI but Competition Metric). One simple approach, which gives Competition Metric of around 0.69, is to take the ratio of EFS probability and EFS_TIME . A more elaborate approach which gives you a 29th place is to use Optuna to find the optimal coefficients $$a,b,c,d,e,f$$ that maximize the Competition Metric of the following ratio:
\n$$ \\frac{a*efs^3 + b*efs^2 + c*efs + 0.001}{d*efs\\_time^3 + e*efs\\_time^2 + f*efs\\_time + 0.001}$$
\nThe final solution uses the ensemble of 10 models from Stage 1 with their corresponding coefficients obtained in Stage 2.
", "comments": [ { "id": 3148712, "author_name": "kacha", "author_url": "", "post_date": "2025-03-13T13:09:38.700000", "content": "How did you come up with this custom metric ? Is that common in some field?
", "votes": 1, "replies": [ { "id": 3150181, "author_name": "Nikolay Slobodchuk", "author_url": "", "post_date": "2025-03-15T06:20:01.193000", "content": "It's just a heuristic based on the idea that the metric should be a decreasing function of efs_time and an increasing function of efs.
Thank you !
", "votes": 0, "replies": [] } ] } ] }, { "id": 3145519, "author_name": "Chris Deotte", "author_url": "", "post_date": "2025-03-09T23:37:29.597000", "content": "Congrats! I like your approach. What was your final submission's CV score, public LB score, and private LB score?
\nFor your (2) regression model, did you use the efs_time as is? or transform it first, like use ranking or something?
Hi! Thanks.
\nOne of my submissions scored 0.695 (public) and 0.694(private), the other one 0.694 and 0.692. I've lost my CV score information, unfortunately. And I mostly relied on the public leaderboard data for CV due to personal time constraints 🤭
\nThe regression model in stage 1 was fit on unmodified efs_time. In the second stage predicted efs_time was transformed by np.clip(efs_time, a_min=0).
\nCondolences for your leaderboard position shakeup. 🥲
Here's a simple solution that gives you a 29th place.
\n1) Estimate probability of EFS by fitting a CatBoostClassifier.
\n2) Estimate EFS_TIME for the subsample where EFS==1 using CatBoostRegressor.
\nEstimates in 1) and 2) are obtained OOF using 10 folds.
Combine the estimates in Stage 1 to create a metric that maximizes (EDIT: not CI but Competition Metric). One simple approach, which gives Competition Metric of around 0.69, is to take the ratio of EFS probability and EFS_TIME . A more elaborate approach which gives you a 29th place is to use Optuna to find the optimal coefficients $$a,b,c,d,e,f$$ that maximize the Competition Metric of the following ratio:
\n$$ \\frac{a*efs^3 + b*efs^2 + c*efs + 0.001}{d*efs\\_time^3 + e*efs\\_time^2 + f*efs\\_time + 0.001}$$
\nThe final solution uses the ensemble of 10 models from Stage 1 with their corresponding coefficients obtained in Stage 2.
", "rawMarkdown": "Here's a simple solution that gives you a 29th place.\n\n## Stage 1\n1) Estimate probability of EFS by fitting a CatBoostClassifier.\n2) Estimate EFS_TIME for the subsample where EFS==1 using CatBoostRegressor.\nEstimates in 1) and 2) are obtained OOF using 10 folds.\n\n## Stage 2\nCombine the estimates in Stage 1 to create a metric that maximizes ~~ Concordance Index~~ (EDIT: not CI but Competition Metric). One simple approach, which gives ~~CI~~ Competition Metric of around 0.69, is to take the ratio of EFS probability and EFS_TIME . A more elaborate approach which gives you a 29th place is to use Optuna to find the optimal coefficients $$a,b,c,d,e,f$$ that maximize the ~~CI ~~ Competition Metric of the following ratio: \n\n$$ \\frac{a\\*efs^3 + b\\*efs^2 + c\\*efs + 0.001}{d\\*efs\\\\_time^3 + e\\*efs\\\\_time^2 + f\\*efs\\\\_time + 0.001}$$\n\nThe final solution uses the ensemble of 10 models from Stage 1 with their corresponding coefficients obtained in Stage 2.", "votes": 10 }, { "id": 3148712, "postDate": "2025-03-13T13:09:38.700Z", "content": "How did you come up with this custom metric ? Is that common in some field?
", "rawMarkdown": "How did you come up with this custom metric ? Is that common in some field? ", "votes": 1, "replies": [ { "id": 3150181, "postDate": "2025-03-15T06:20:01.193Z", "content": "It's just a heuristic based on the idea that the metric should be a decreasing function of efs_time and an increasing function of efs.
Thank you !
", "rawMarkdown": "Thank you !" } ] } ] }, { "id": 3145519, "postDate": "2025-03-09T23:37:29.597Z", "content": "Congrats! I like your approach. What was your final submission's CV score, public LB score, and private LB score?
\nFor your (2) regression model, did you use the efs_time as is? or transform it first, like use ranking or something?
Hi! Thanks.
\nOne of my submissions scored 0.695 (public) and 0.694(private), the other one 0.694 and 0.692. I've lost my CV score information, unfortunately. And I mostly relied on the public leaderboard data for CV due to personal time constraints 🤭
\nThe regression model in stage 1 was fit on unmodified efs_time. In the second stage predicted efs_time was transformed by np.clip(efs_time, a_min=0).
\nCondolences for your leaderboard position shakeup. 🥲