{ "id": 596716, "title": "Is possible to reach LB score 0.85 with single model?", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/596716", "author_name": "XuKong Ji", "post_date": "2025-08-05T02:08:56.784000", "votes": 2, "comment_count": 25, "views": 0, "content": "

I am sure single model can reach 0.84, but what's the upper bundary of single model and When we need to embrace multi-models ensemble

", "messages": [ { "id": 3263448, "postDate": "2025-08-05T08:24:53.097Z", "content": "

Single model 0.86 is possible, I believe 0.87 is too. Ensemble has not been working so well on LB for me so far…

", "rawMarkdown": "Single model 0.86 is possible, I believe 0.87 is too. Ensemble has not been working so well on LB for me so far...", "votes": 8, "replies": [ { "id": 3263461, "postDate": "2025-08-05T08:43:15.147Z", "content": "

Cool, so you just use one model which use the imu + thm + tof features and achieve 0.86?

", "rawMarkdown": "Cool, so you just use one model which use the imu + thm + tof features and achieve 0.86? " }, { "id": 3263498, "postDate": "2025-08-05T09:36:37.710Z", "content": "

Wow!!! Thanks for your answer!

", "rawMarkdown": "Wow!!! Thanks for your answer!" }, { "id": 3269057, "postDate": "2025-08-13T21:44:26.433Z", "content": "

Still no luck with ensembling ? 🫡

", "rawMarkdown": "Still no luck with ensembling ? 🫡" } ] }, { "id": 3263552, "postDate": "2025-08-05T11:14:40.317Z", "content": "

what is your definition of single model? I believe many people's kfold is better than 0.84

", "rawMarkdown": "what is your definition of single model? I believe many people's kfold is better than 0.84", "votes": 1, "replies": [ { "id": 3263994, "postDate": "2025-08-06T02:29:21.447Z", "content": "

Kfold of course

", "rawMarkdown": "Kfold of course" } ] }, { "id": 3263272, "postDate": "2025-08-05T02:08:56.783Z", "content": "

I am sure single model can reach 0.84, but what's the upper bundary of single model and When we need to embrace multi-models ensemble

", "rawMarkdown": "I am sure single model can reach 0.84, but what's the upper bundary of single model and When we need to embrace multi-models ensemble", "votes": 2 }, { "id": 3265190, "postDate": "2025-08-07T06:44:41.387Z", "content": "

I haven’t tried any ensembling yet (aside from fold averaging, of course). It's performing around that level, but I believe it can be pushed further. I just haven’t figured out how yet. =)

", "rawMarkdown": "I haven’t tried any ensembling yet (aside from fold averaging, of course). It's performing around that level, but I believe it can be pushed further. I just haven’t figured out how yet. =)" }, { "id": 3264001, "postDate": "2025-08-06T02:46:43.010Z", "content": "

There are ~50% hidden test data with only imu part avaliable, so the final score is ~0.5IMU-only score + ~0.5full-data score. As there already 0.87 score in leadboard. So I think it is possible to achieve ~0.84 for IMU-only part and ~0.90 for full-data part.

", "rawMarkdown": "There are ~50% hidden test data with only imu part avaliable, so the final score is ~0.5*IMU-only score + ~0.5*full-data score. As there already 0.87 score in leadboard. So I think it is possible to achieve ~0.84 for IMU-only part and ~0.90 for full-data part.", "replies": [ { "id": 3264046, "postDate": "2025-08-06T04:42:54.863Z", "content": "

If it's single-mode K-fold, a score of 0.9 is impossible. In fact, I believe the highest cv score wouldn't even exceed 0.87-0.88. This would require all your fold CV scores to surpass 0.9, with some folds performing exceptionally well. However, such results would only be possible with extremely high-quality data, and clearly our training set isn't perfect.

\n

The high scores on the public leaderboard can be easily explained - they're either from ensemble methods or just happened to overfit a small portion of the public test data.

", "rawMarkdown": "If it's single-mode K-fold, a score of 0.9 is impossible. In fact, I believe the highest cv score wouldn't even exceed 0.87-0.88. This would require all your fold CV scores to surpass 0.9, with some folds performing exceptionally well. However, such results would only be possible with extremely high-quality data, and clearly our training set isn't perfect.\n\nThe high scores on the public leaderboard can be easily explained - they're either from ensemble methods or just happened to overfit a small portion of the public test data.", "replies": [ { "id": 3277590, "postDate": "2025-08-28T10:17:39.113Z", "content": "

I think single-mode K-fold score ~0.9 is possible (GroupStratifiedKfold). I have achieved it. My biggest weakness right now is that the IMU only score is relatively low, with a CV of approximately 0.82. I estimate that the LB score achievable with the current IMU only model is around 0.874. This means that if my IMU only model reaches a CV of around 0.84, the LB score could potentially boost to 0.884.

", "rawMarkdown": "I think single-mode K-fold score ~0.9 is possible (GroupStratifiedKfold). I have achieved it. My biggest weakness right now is that the IMU only score is relatively low, with a CV of approximately 0.82. I estimate that the LB score achievable with the current IMU only model is around 0.874. This means that if my IMU only model reaches a CV of around 0.84, the LB score could potentially boost to 0.884. " }, { "id": 3277672, "postDate": "2025-08-28T13:09:38.373Z", "content": "

(0.90+0.82)/2 = 0.86 what makes you think that LB should be 87.4 ? You mean you have 0.928 CV with all features?

", "rawMarkdown": "(0.90+0.82)/2 = 0.86 what makes you think that LB should be 87.4 ? You mean you have 0.928 CV with all features?" }, { "id": 3277682, "postDate": "2025-08-28T13:37:06.410Z", "content": "

I only have ~0.9 CV with all features.

", "rawMarkdown": "I only have ~0.9 CV with all features.\n" }, { "id": 3277684, "postDate": "2025-08-28T13:43:52.490Z", "content": "

There are several reasons why I can estimate that I can achieve an LB score of 0.874: firstly, my current LB score is not the best version with the existing set of models. I am retrying the IMU only ensemble method, and the new ensemble method dropped my LB score from 0.864 to 0.863. Today, I discovered a way to increase the score of the all features model, boosting it from 0.863 to 0.868. Therefore, if I revert to the old IMU only ensemble method, I can achieve at least a 0.869 LB score. Additionally, the technique to increase the all features score has not yet been fully applied to the ensemble models, and the method I found a few days ago to increase the IMU CV from 0.811 to 0.819 has only been applied to one model to experiment with the LB score.

", "rawMarkdown": "There are several reasons why I can estimate that I can achieve an LB score of 0.874: firstly, my current LB score is not the best version with the existing set of models. I am retrying the IMU only ensemble method, and the new ensemble method dropped my LB score from 0.864 to 0.863. Today, I discovered a way to increase the score of the all features model, boosting it from 0.863 to 0.868. Therefore, if I revert to the old IMU only ensemble method, I can achieve at least a 0.869 LB score. Additionally, the technique to increase the all features score has not yet been fully applied to the ensemble models, and the method I found a few days ago to increase the IMU CV from 0.811 to 0.819 has only been applied to one model to experiment with the LB score." }, { "id": 3277686, "postDate": "2025-08-28T13:47:29.800Z", "content": "

There is still one idea that I find feasible to increase the score of the IMU only model, but I have failed many times. I have tried many ideas but still haven't been able to boost the IMU only model. At least today, I found a new way to increase the score of the IMU only model. I hope that the previous idea can succeed, which would make it possible to increase the IMU only model to a CV of 0.83.

", "rawMarkdown": "There is still one idea that I find feasible to increase the score of the IMU only model, but I have failed many times. I have tried many ideas but still haven't been able to boost the IMU only model. At least today, I found a new way to increase the score of the IMU only model. I hope that the previous idea can succeed, which would make it possible to increase the IMU only model to a CV of 0.83." }, { "id": 3277708, "postDate": "2025-08-28T14:41:52.927Z", "content": "

Fair enough! Isn't there a double risk of LB overfitting? Best of luck! I haven't had a working idea myself for a long time! :)

", "rawMarkdown": "Fair enough! Isn't there a double risk of LB overfitting? Best of luck! I haven't had a working idea myself for a long time! :)" }, { "id": 3277724, "postDate": "2025-08-28T15:27:56.537Z", "content": "

are you getting good cv lb correlation?

", "rawMarkdown": "are you getting good cv lb correlation?" }, { "id": 3277758, "postDate": "2025-08-28T15:55:55.050Z", "content": "

@mohammad2012191 There are some improvements that increase CV but not LB, some improvements that increase both CV and LB, and more importantly, you think these improvements are reasonable.

", "rawMarkdown": "@mohammad2012191 There are some improvements that increase CV but not LB, some improvements that increase both CV and LB, and more importantly, you think these improvements are reasonable.", "votes": 1 }, { "id": 3277764, "postDate": "2025-08-28T16:02:45.033Z", "content": "

@optimo The variance between the my folds is relatively large, with the difference between the best fold and the worst fold reaching up to 0.06. But from what I have observed, it seems that there will be little luck involved here. Unless one intentionally finds folds that correlate with the LB or tunes the ensemble weights to overfit the LB.

", "rawMarkdown": "@optimo The variance between the my folds is relatively large, with the difference between the best fold and the worst fold reaching up to 0.06. But from what I have observed, it seems that there will be little luck involved here. Unless one intentionally finds folds that correlate with the LB or tunes the ensemble weights to overfit the LB." }, { "id": 3277771, "postDate": "2025-08-28T16:17:13.210Z", "content": "

isnt that variance a little concerning for shakeup

", "rawMarkdown": "isnt that variance a little concerning for shakeup" }, { "id": 3278087, "postDate": "2025-08-29T07:41:11.320Z", "content": "

For cases where the scores are very close together (<0.005) on the LB, there can still be shakeups. However, if the difference is large, such as 0.888 and 0.868, I think it is very unlikely to happen. 😂

", "rawMarkdown": "For cases where the scores are very close together (<0.005) on the LB, there can still be shakeups. However, if the difference is large, such as 0.888 and 0.868, I think it is very unlikely to happen. 😂" }, { "id": 3278650, "postDate": "2025-08-30T08:29:31.327Z", "content": "

My IMU only model has not made any progress and is still approximately at CV=0.82. Meanwhile, my LB score has reached 0.873. Therefore, I believe that if the IMU only model achieves a CV of 0.84, reaching 0.88 would be understandable.

", "rawMarkdown": "My IMU only model has not made any progress and is still approximately at CV=0.82. Meanwhile, my LB score has reached 0.873. Therefore, I believe that if the IMU only model achieves a CV of 0.84, reaching 0.88 would be understandable." }, { "id": 3279695, "postDate": "2025-09-01T08:28:33.913Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3279694, "postDate": "2025-09-01T08:28:17.943Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3263967, "postDate": "2025-08-06T00:31:03.110Z", "content": "

you mean imu-only to reach 0.85?

", "rawMarkdown": "you mean imu-only to reach 0.85?", "replies": [ { "id": 3263995, "postDate": "2025-08-06T02:29:51.623Z", "content": "

I mean full data(imu+thm+tof)

", "rawMarkdown": "I mean full data(imu+thm+tof)" } ] } ], "comments": [ { "id": 3263448, "author_name": "Optimo", "author_url": "", "post_date": "2025-08-05T08:24:53.097000", "content": "

Single model 0.86 is possible, I believe 0.87 is too. Ensemble has not been working so well on LB for me so far…

", "votes": 8, "replies": [ { "id": 3263461, "author_name": "Wenxuan Ye", "author_url": "", "post_date": "2025-08-05T08:43:15.147000", "content": "

Cool, so you just use one model which use the imu + thm + tof features and achieve 0.86?

", "votes": 0, "replies": [] }, { "id": 3263498, "author_name": "XuKong Ji", "author_url": "", "post_date": "2025-08-05T09:36:37.710000", "content": "

Wow!!! Thanks for your answer!

", "votes": 0, "replies": [] }, { "id": 3269057, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-13T21:44:26.433000", "content": "

Still no luck with ensembling ? 🫡

", "votes": 0, "replies": [] } ] }, { "id": 3263552, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-08-05T11:14:40.317000", "content": "

what is your definition of single model? I believe many people's kfold is better than 0.84

", "votes": 1, "replies": [ { "id": 3263994, "author_name": "XuKong Ji", "author_url": "", "post_date": "2025-08-06T02:29:21.447000", "content": "

Kfold of course

", "votes": 0, "replies": [] } ] }, { "id": 3265190, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2025-08-07T06:44:41.387000", "content": "

I haven’t tried any ensembling yet (aside from fold averaging, of course). It's performing around that level, but I believe it can be pushed further. I just haven’t figured out how yet. =)

", "votes": 0, "replies": [] }, { "id": 3264001, "author_name": "Starry", "author_url": "", "post_date": "2025-08-06T02:46:43.010000", "content": "

There are ~50% hidden test data with only imu part avaliable, so the final score is ~0.5IMU-only score + ~0.5full-data score. As there already 0.87 score in leadboard. So I think it is possible to achieve ~0.84 for IMU-only part and ~0.90 for full-data part.

", "votes": 0, "replies": [ { "id": 3264046, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-08-06T04:42:54.863000", "content": "

If it's single-mode K-fold, a score of 0.9 is impossible. In fact, I believe the highest cv score wouldn't even exceed 0.87-0.88. This would require all your fold CV scores to surpass 0.9, with some folds performing exceptionally well. However, such results would only be possible with extremely high-quality data, and clearly our training set isn't perfect.

\n

The high scores on the public leaderboard can be easily explained - they're either from ensemble methods or just happened to overfit a small portion of the public test data.

", "votes": 0, "replies": [ { "id": 3277590, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-28T10:17:39.113000", "content": "

I think single-mode K-fold score ~0.9 is possible (GroupStratifiedKfold). I have achieved it. My biggest weakness right now is that the IMU only score is relatively low, with a CV of approximately 0.82. I estimate that the LB score achievable with the current IMU only model is around 0.874. This means that if my IMU only model reaches a CV of around 0.84, the LB score could potentially boost to 0.884.

", "votes": 0, "replies": [] }, { "id": 3277672, "author_name": "Optimo", "author_url": "", "post_date": "2025-08-28T13:09:38.373000", "content": "

(0.90+0.82)/2 = 0.86 what makes you think that LB should be 87.4 ? You mean you have 0.928 CV with all features?

", "votes": 0, "replies": [] }, { "id": 3277682, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-28T13:37:06.410000", "content": "

I only have ~0.9 CV with all features.

", "votes": 0, "replies": [] }, { "id": 3277684, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-28T13:43:52.490000", "content": "

There are several reasons why I can estimate that I can achieve an LB score of 0.874: firstly, my current LB score is not the best version with the existing set of models. I am retrying the IMU only ensemble method, and the new ensemble method dropped my LB score from 0.864 to 0.863. Today, I discovered a way to increase the score of the all features model, boosting it from 0.863 to 0.868. Therefore, if I revert to the old IMU only ensemble method, I can achieve at least a 0.869 LB score. Additionally, the technique to increase the all features score has not yet been fully applied to the ensemble models, and the method I found a few days ago to increase the IMU CV from 0.811 to 0.819 has only been applied to one model to experiment with the LB score.

", "votes": 0, "replies": [] }, { "id": 3277686, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-28T13:47:29.800000", "content": "

There is still one idea that I find feasible to increase the score of the IMU only model, but I have failed many times. I have tried many ideas but still haven't been able to boost the IMU only model. At least today, I found a new way to increase the score of the IMU only model. I hope that the previous idea can succeed, which would make it possible to increase the IMU only model to a CV of 0.83.

", "votes": 0, "replies": [] }, { "id": 3277708, "author_name": "Optimo", "author_url": "", "post_date": "2025-08-28T14:41:52.927000", "content": "

Fair enough! Isn't there a double risk of LB overfitting? Best of luck! I haven't had a working idea myself for a long time! :)

", "votes": 0, "replies": [] }, { "id": 3277724, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2025-08-28T15:27:56.537000", "content": "

are you getting good cv lb correlation?

", "votes": 0, "replies": [] }, { "id": 3277758, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-28T15:55:55.050000", "content": "

@mohammad2012191 There are some improvements that increase CV but not LB, some improvements that increase both CV and LB, and more importantly, you think these improvements are reasonable.

", "votes": 1, "replies": [] }, { "id": 3277764, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-28T16:02:45.033000", "content": "

@optimo The variance between the my folds is relatively large, with the difference between the best fold and the worst fold reaching up to 0.06. But from what I have observed, it seems that there will be little luck involved here. Unless one intentionally finds folds that correlate with the LB or tunes the ensemble weights to overfit the LB.

", "votes": 0, "replies": [] }, { "id": 3277771, "author_name": "Ravon", "author_url": "", "post_date": "2025-08-28T16:17:13.210000", "content": "

isnt that variance a little concerning for shakeup

", "votes": 0, "replies": [] }, { "id": 3278087, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-29T07:41:11.320000", "content": "

For cases where the scores are very close together (<0.005) on the LB, there can still be shakeups. However, if the difference is large, such as 0.888 and 0.868, I think it is very unlikely to happen. 😂

", "votes": 0, "replies": [] }, { "id": 3278650, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-30T08:29:31.327000", "content": "

My IMU only model has not made any progress and is still approximately at CV=0.82. Meanwhile, my LB score has reached 0.873. Therefore, I believe that if the IMU only model achieves a CV of 0.84, reaching 0.88 would be understandable.

", "votes": 0, "replies": [] }, { "id": 3279695, "author_name": "", "author_url": "", "post_date": "2025-09-01T08:28:33.913000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3279694, "author_name": "", "author_url": "", "post_date": "2025-09-01T08:28:17.943000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3263967, "author_name": "Tom", "author_url": "", "post_date": "2025-08-06T00:31:03.110000", "content": "

you mean imu-only to reach 0.85?

", "votes": 0, "replies": [ { "id": 3263995, "author_name": "XuKong Ji", "author_url": "", "post_date": "2025-08-06T02:29:51.623000", "content": "

I mean full data(imu+thm+tof)

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3263448": "Single model 0.86 is possible, I believe 0.87 is too. Ensemble has not been working so well on LB for me so far...", "3263552": "what is your definition of single model? I believe many people's kfold is better than 0.84", "3263272": "I am sure single model can reach 0.84, but what's the upper bundary of single model and When we need to embrace multi-models ensemble", "3265190": "I haven’t tried any ensembling yet (aside from fold averaging, of course). It's performing around that level, but I believe it can be pushed further. I just haven’t figured out how yet. =)", "3264001": "There are ~50% hidden test data with only imu part avaliable, so the final score is ~0.5*IMU-only score + ~0.5*full-data score. As there already 0.87 score in leadboard. So I think it is possible to achieve ~0.84 for IMU-only part and ~0.90 for full-data part.", "3263967": "you mean imu-only to reach 0.85?" } }