For this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it.
\nexample
\n train.groupby().tail().reset_index(dropTrue)\nIn my case, I used CatBoost and obtained the following results.
\n5 kfold: StratifiedGroupKFold (train[\"sequence_id\"])
\n※ update note : group = \"sequence_id\" means leakage, group = \"subject\" is correct.
metric: https://www.kaggle.com/code/metric/cmi-2025
\ncv: 0.5798545 (binary f1: 0.8626, f1 macro: 0.297100)
\npublic lb: 0.58
The CV and public LB seem to align perfectly, but what does everyone else think?
\n※ Note : I just only IMU sensor features
", "messages": [ { "id": 3213877, "postDate": "2025-05-30T15:53:36.653Z", "content": "For this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it.
\nexample
\n train.groupby().tail().reset_index(dropTrue)\nIn my case, I used CatBoost and obtained the following results.
\n5 kfold: StratifiedGroupKFold (train[\"sequence_id\"])
\n※ update note : group = \"sequence_id\" means leakage, group = \"subject\" is correct.
metric: https://www.kaggle.com/code/metric/cmi-2025
\ncv: 0.5798545 (binary f1: 0.8626, f1 macro: 0.297100)
\npublic lb: 0.58
The CV and public LB seem to align perfectly, but what does everyone else think?
\n※ Note : I just only IMU sensor features
", "rawMarkdown": "For this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it. \n\nexample\n```\ntrain = train.groupby(\"sequence_id\").tail(1).reset_index(drop=True)\n```\n\nIn my case, I used CatBoost and obtained the following results.\n\n5 kfold: StratifiedGroupKFold (train[\"sequence_id\"])\n※ update note : group = \"sequence_id\" means leakage, group = \"subject\" is correct.\n\nmetric: https://www.kaggle.com/code/metric/cmi-2025\n\ncv: 0.5798545 (binary f1: 0.8626, f1 macro: 0.297100)\npublic lb: 0.58\n\nThe CV and public LB seem to align perfectly, but what does everyone else think?\n\n※ Note : I just only IMU sensor features", "votes": 69 }, { "id": 3268024, "postDate": "2025-08-12T09:10:21.813Z", "content": "CV:(sgkf on subject and gesture)
\nIMU 0.825~0.828
\nAll: 0.886
LB:
\nIMU:0.839~0.844
\n(0.5IMU+0.5All):0.868
Any top teams willing to share their scores?
", "rawMarkdown": "CV:(sgkf on subject and gesture)\nIMU 0.825~0.828\nAll: 0.886\n\nLB:\nIMU:0.839~0.844\n(0.5*IMU+0.5*All):0.868\n\nAny top teams willing to share their scores?", "votes": 8, "replies": [ { "id": 3268029, "postDate": "2025-08-12T09:22:20.007Z", "content": "LB score 0.852
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
\nCV (sgkf on subject and gesture)
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
Your IMU-only score is awesome.
", "rawMarkdown": "Your IMU-only score is awesome." }, { "id": 3268041, "postDate": "2025-08-12T09:40:15.460Z", "content": "TY for confirming IMU can be pushed so much! I assume it's a single model? Wondering how much it can be pushed more with ensembling~
", "rawMarkdown": "TY for confirming IMU can be pushed so much! I assume it's a single model? Wondering how much it can be pushed more with ensembling~", "replies": [ { "id": 3268043, "postDate": "2025-08-12T09:45:44.073Z", "content": "Yes, single model. I am wondering too, I don't have diversed work to ensemble currently.
", "rawMarkdown": "Yes, single model. I am wondering too, I don't have diversed work to ensemble currently." } ] }, { "id": 3268459, "postDate": "2025-08-13T01:43:43.033Z", "content": "My best imu only CV: 0.824
\nLB: 0.834
I'm still confused about all sensors cv…
", "rawMarkdown": "My best imu only CV: 0.824\nLB: 0.834\n\nI'm still confused about all sensors cv...", "votes": 1, "replies": [ { "id": 3274578, "postDate": "2025-08-25T03:48:47.503Z", "content": "2025.8.25 update:
\nall sensors cv: 0.863 lb:0.865
", "rawMarkdown": "2025.8.25 update: \n\nall sensors cv: 0.863 lb:0.865" } ] }, { "id": 3269417, "postDate": "2025-08-14T13:07:32.870Z", "content": "CV(sgkf on subject and gesture)
\nIMU: 0.8390
\nAll: 0.8861
LB:
\n0.857
--overfitting?
", "rawMarkdown": "CV(sgkf on subject and gesture)\nIMU: 0.8390\nAll: 0.8861\n\nLB:\n0.857\n\n--overfitting?", "votes": 1, "replies": [ { "id": 3269437, "postDate": "2025-08-14T13:37:31.647Z", "content": "Did you test IMU only LB score?
", "rawMarkdown": "Did you test IMU only LB score?" }, { "id": 3269514, "postDate": "2025-08-14T15:22:04.420Z", "content": "IMU only LB: 0.835
", "rawMarkdown": "IMU only LB: 0.835" } ] }, { "id": 3269561, "postDate": "2025-08-14T16:32:46.667Z", "content": "Thanks for sharing, this confirms my IMU model is quite weak -
\nCV 0.827 - LB 0.822
\n(sgkf on subject and gesture)
\nMy IMU+TOF+THM on the other end looks alright :
\nCV 0.887 - LB 0.859
Hello, did you first try to sgkf first on subject before doing it on subject and gesture?
\nIf so, did you notice a diffrence in your lb score when switching from per subject sgkf to per subject and gesture sgkf?
Did not try no. It should not make any difference imo
", "rawMarkdown": "Did not try no. It should not make any difference imo" }, { "id": 3271703, "postDate": "2025-08-19T13:25:07.627Z", "content": "okok thx you for your answer
", "rawMarkdown": "okok thx you for your answer" } ] }, { "id": 3269893, "postDate": "2025-08-15T09:16:54.573Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3273198, "postDate": "2025-08-22T10:41:25.633Z", "content": "LB:0.860
\nCV:(5folds: sgkf on subject and gesture)
\nIMU 0.807
\nAll: 0.863
\nmy cv is quiet low….
0.78
", "rawMarkdown": "## cv\n - IMU only: 0.723\n - IMU + TMP + TOF: 0.771\n\n## public lb\n0.78", "votes": 9, "replies": [ { "id": 3214786, "postDate": "2025-06-01T04:38:38.397Z", "content": "Could you please share how you calculated your CV and how you split the data?
", "rawMarkdown": "Could you please share how you calculated your CV and how you split the data?", "votes": 1, "replies": [ { "id": 3214794, "postDate": "2025-06-01T04:55:01.343Z", "content": "I split the data with StratifiedGroupKFold, using each subject as the group label.
\nIdeally, the split should also account for metadata such as age and gender, but I haven’t implemented that yet.
Following the competition’s evaluation metric, I predict the gesture for each sequence ID and then compute the average Binary F1 and Macro F1 scores.
", "rawMarkdown": "## How to split the data\nI split the data with StratifiedGroupKFold, using each subject as the group label. \nIdeally, the split should also account for metadata such as age and gender, but I haven’t implemented that yet.\n\n## How to calculate CV\nFollowing [the competition’s evaluation metric](https://www.kaggle.com/code/metric/cmi-2025), I predict the gesture for each sequence ID and then compute the average Binary F1 and Macro F1 scores.", "votes": 12 }, { "id": 3215041, "postDate": "2025-06-01T14:07:41.700Z", "content": "Thanks I did without strat and group and it was all over the place will try this .
", "rawMarkdown": "Thanks I did without strat and group and it was all over the place will try this .", "votes": 1 } ] }, { "id": 3215486, "postDate": "2025-06-02T09:55:00.657Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 3218328, "postDate": "2025-06-06T04:54:18.303Z", "content": "update:
\nThanks for sharing! Did you incorporate demographic data in your model?
", "rawMarkdown": "Thanks for sharing! Did you incorporate demographic data in your model?", "votes": 1 }, { "id": 3218602, "postDate": "2025-06-06T12:20:57.917Z", "content": "Yes, but I'd probably get about the same score without using it.
", "rawMarkdown": "Yes, but I'd probably get about the same score without using it.", "votes": 3 }, { "id": 3218608, "postDate": "2025-06-06T12:27:50.370Z", "content": "I have the same experimental results, leading me to suspect that demographic data may not provide sufficient information, as a single subject can correspond to various gesture sequences.
", "rawMarkdown": "I have the same experimental results, leading me to suspect that demographic data may not provide sufficient information, as a single subject can correspond to various gesture sequences.", "votes": 2 } ] } ] }, { "id": 3251295, "postDate": "2025-07-20T11:49:36.517Z", "content": "| Model | \nCV | \nCV Strategy | \nLB | \n
|---|---|---|---|
| 1D CNN (IMU Only) | \n0.78129 | \nStratifiedGroupKFold (5 Folds) | \n0.79 | \n
| LightGBM (IMU Only) | \n0.77183 | \nStratifiedGroupKFold (5 Folds) | \n\n |
| LightGBM (All Data) | \n0.84145 | \nStratifiedGroupKFold (5 Folds) | \n\n |
| 1D CNN + LSTM (IMU Only) | \n0.77332 | \nStratifiedGroupKFold (5 Folds) | \n\n |
| 1D CNN + LSTM (All Data) | \n0.83636 | \nStratifiedGroupKFold (5 Folds) | \n\n |
Thanks for sharing! May I aks is it your ensemble used? And it is amazing lgbm! I can't push lgbm very well…
", "rawMarkdown": "Thanks for sharing! May I aks is it your ensemble used? And it is amazing lgbm! I can't push lgbm very well...", "replies": [ { "id": 3251458, "postDate": "2025-07-20T20:43:22.360Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 3252301, "postDate": "2025-07-22T12:08:23.113Z", "content": "@sercanyesiloz Can you share the LB scores of these models as well?
", "rawMarkdown": "@sercanyesiloz Can you share the LB scores of these models as well?" } ] }, { "id": 3213969, "postDate": "2025-05-30T17:42:32.137Z", "content": "Update !
\nI use all sensors features :
\ncv: 0.73542811647 (binary f1: 0.93720089, f1 macro:0.53365533)
\nAnd for submission :
\nI switched the model by checking whether \"tof_5_v63\" is null or not.
public lb : 0.65
\nHere, I calculated the average with the cv for the only IMU sensor and all sensors.
\n(0.5798545 + 0.7354281) / 2 = 0.657639058
This seems to align perfectly for cv and public lb.
\nThis is the good competition.
thanks for posting! could you explain the rationale why using the end of the sequence is a good first step here?
", "rawMarkdown": "thanks for posting! could you explain the rationale why using the end of the sequence is a good first step here?", "votes": 1, "replies": [ { "id": 3214114, "postDate": "2025-05-31T01:00:48.687Z", "content": "@jonwibowo You can get the total sequence length from sequence_counter. But I think we need to build a model that takes the sequence as input, current approach uses summarized info to build a simple tabular predictor.
\nWe have five labels, it's interesting to figure out a good way handling label information.
\nOne possible new label development is:
sequence_type, ordientation, behavior, phase and gesture in the text.@tom99763 Right! Thank you for comment.
", "rawMarkdown": "@tom99763 Right! Thank you for comment." }, { "id": 3214194, "postDate": "2025-05-31T04:59:53.047Z", "content": "@jonwibowo I believe the very end of a sequence tends to capture some aggregation or summary of the preceding activity. For this type of tabular data, features from the end of the sequence are often utilized.
\nHowever, in this particular case, there isn't any single feature (like those from the end) that distinctly stands out.
\nUltimately, it might be best to use all the data and apply a time-series model.
0.82 CV - 0.80 LB (don’t know if high or low 0.80), using Stratified K fold, will test with Group Stratified K fold as well. IMU only
\nL.E.: based on other competitors reaching 0.8, I believe it's high 0.8, which is in line with 0.01-0.015 gap I keep experiencing with this split strategy.
Nice just imu that is great
", "rawMarkdown": "Nice just imu that is great " }, { "id": 3223835, "postDate": "2025-06-14T00:01:39.380Z", "content": "When your cv exceeds 0.8, does the correlation between lb and cv weaken? In my experiment, when it exceeds 0.8, I tried to increase cv to 0.81, but lb did decrease by 0.01.
\nOr maybe i make some data leakage…
May I ask how do you split your folds?
", "rawMarkdown": "May I ask how do you split your folds?" }, { "id": 3223915, "postDate": "2025-06-14T05:33:25.437Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3223917, "postDate": "2025-06-14T05:42:45.567Z", "content": "Stratified K fold，split by gesture
", "rawMarkdown": "Stratified K fold，split by gesture" }, { "id": 3224275, "postDate": "2025-06-14T16:02:48.640Z", "content": "For stratified k fold, split by gesture, I observe a constant 0.01-0.015 gap between cv and lb (cv being higher).
\nFor stratified group k fold, other way around.
", "rawMarkdown": "For stratified k fold, split by gesture, I observe a constant 0.01-0.015 gap between cv and lb (cv being higher).\n\nFor stratified group k fold, other way around.", "votes": 2 } ] } ] }, { "id": 3216995, "postDate": "2025-06-04T11:42:51.907Z", "content": "| Feature | \nCV | \nLB | \n
|---|---|---|
| IMU | \n0.734 | \n0.74 | \n
| IMU+THM+TOF | \n0.812 | \n0.73 | \n
| OverAll | \n0.773 | \n0.78 | \n
@ye11725 Thanks. That's amazing. So, for the IMU+THM+TOF Leaderboard (LB), you just submitted it as null, right?
", "rawMarkdown": "@ye11725 Thanks. That's amazing. So, for the IMU+THM+TOF Leaderboard (LB), you just submitted it as null, right?", "votes": 1, "replies": [ { "id": 3217355, "postDate": "2025-06-04T23:41:21.670Z", "content": "Just use the IMU+THM+TOF model to predict all test sets, including data with null TOF values
", "rawMarkdown": "Just use the IMU+THM+TOF model to predict all test sets, including data with null TOF values", "votes": 2 } ] } ] }, { "id": 3215236, "postDate": "2025-06-01T20:47:51.573Z", "content": "1d-cnn, IMU feature only, label based stratified 1 fold local validation 0.703, lb 0.68
", "rawMarkdown": "1d-cnn, IMU feature only, label based stratified 1 fold local validation 0.703, lb 0.68", "votes": 3, "replies": [ { "id": 3215924, "postDate": "2025-06-02T21:27:29.600Z", "content": "update: IMU + TMP + TOF: 0.766 + IMU only: 0.703, LB 0.71
", "rawMarkdown": "update: IMU + TMP + TOF: 0.766 + IMU only: 0.703, LB 0.71", "votes": 2, "replies": [ { "id": 3215949, "postDate": "2025-06-02T22:47:05.570Z", "content": "Are you inferring on the ensemble of 5 fold? Dunno I get better lb lower cv though in same comparison as you for 0.68
", "rawMarkdown": "Are you inferring on the ensemble of 5 fold? Dunno I get better lb lower cv though in same comparison as you for 0.68", "votes": 1 }, { "id": 3215950, "postDate": "2025-06-02T22:55:09.073Z", "content": "For now, I’m using a simple split with a single model for a quick experiment. Given this setup, it’s expected that the gap between CV and LB would be larger compared to a 5-fold ensemble
", "rawMarkdown": "For now, I’m using a simple split with a single model for a quick experiment. Given this setup, it’s expected that the gap between CV and LB would be larger compared to a 5-fold ensemble", "votes": 3 }, { "id": 3216295, "postDate": "2025-06-03T11:14:27.963Z", "content": "Thanks ya now though for Imu only getting nice correlation 0.686 - 0.68
", "rawMarkdown": "Thanks ya now though for Imu only getting nice correlation 0.686 - 0.68", "votes": 1 }, { "id": 3216349, "postDate": "2025-06-03T12:50:16.973Z", "content": "I just applied 5 folds cv:
\nIMU only 0.705 - 0.69
\nIMU only + IMU,tof,tmp (0.705, 0.761) - 0.72
\ncorrelation seems better now
ya 5 fold cv works for me also :)
\n0.7665 ~ 0.7 LB
", "rawMarkdown": "ya 5 fold cv works for me also :) \n\n\n0.7665 ~ 0.7 LB", "votes": 1 } ] }, { "id": 3221793, "postDate": "2025-06-11T12:38:17.033Z", "content": "update:
\nusing (gender + handedness + adult_child) as the target, StratifiedGroupKFold by subject
\ncv: IMU + TMP + TOF: 0.8103 + IMU only: 0.7610, LB 0.81
so an ensemble of both ?
", "rawMarkdown": "so an ensemble of both ?" }, { "id": 3221850, "postDate": "2025-06-11T14:06:42.677Z", "content": "Use the IMU-only models if the test data does not contain TOF & TMP columns. Use the full sensor models if the test data includes TOF & TMP columns.
", "rawMarkdown": "Use the IMU-only models if the test data does not contain TOF & TMP columns. Use the full sensor models if the test data includes TOF & TMP columns.", "votes": 1 }, { "id": 3227607, "postDate": "2025-06-19T05:51:26.907Z", "content": "did u mean (gender + handedness + adult_child) + gesture as labels
", "rawMarkdown": "did u mean (gender + handedness + adult_child) + gesture as labels" }, { "id": 3227653, "postDate": "2025-06-19T06:50:58.287Z", "content": "for example:
\n \n composite_labels = []\n subject subject_groups:\n subject_demo = subject_demographics_map[subject]\n sex = subject_demo[]\n handedness = subject_demo[] \n adult_child = subject_demo[]\n\n \n \n composite_label = sex + handedness * + adult_child * \n composite_labels.append(composite_label)\nHow much improvement did the subtask bring?
\nI tried orientation prediction as a subtask, which led to a +0.007 improvement in CV (5-fold average), but the LB score was nearly the same.
It is just about using composite_labels for stratified groupk split
", "rawMarkdown": "It is just about using composite_labels for stratified groupk split" } ] } ] }, { "id": 3217253, "postDate": "2025-06-04T18:00:35.247Z", "content": "update2 : use stratifiedgroupkfold (group : \"subject\" ) is better than (group : \"sequence id\")
\n・ group : \"sequence id\" result
\nIMU only : cv 0.694 lb : 0.66
\nIMU+THM+TOF :cv 0.796, cv average 0.745, lb : 0.7
・ group : \"object\" result
\nIMU only : cv 0.665 lb : 0.65
\nIMU+THM+TOF :cv 0.751, cv average 0.705, lb : 0.7 almost same
ya similar IMU+THM+TOF :cv average
\nIs anyone finding their CV-LB less consistent on IMU + THM + TOF? My IMU only is .82 cv, .81 lb.. my full data model has cv of .88 but only giving increase of 0.02 in test submission. I’m aware test data is 50/50 and splitting models accordingly.
", "rawMarkdown": "Is anyone finding their CV-LB less consistent on IMU + THM + TOF? My IMU only is .82 cv, .81 lb.. my full data model has cv of .88 but only giving increase of 0.02 in test submission. I’m aware test data is 50/50 and splitting models accordingly. ", "votes": 1, "replies": [ { "id": 3242449, "postDate": "2025-07-05T21:42:36.957Z", "content": "I have the exact same thing you describe, except that I get .82 lb imu only, and .83 with all, despite .9 cv for imu+tof+thm.
", "rawMarkdown": "I have the exact same thing you describe, except that I get .82 lb imu only, and .83 with all, despite .9 cv for imu+tof+thm.", "replies": [ { "id": 3242457, "postDate": "2025-07-05T21:56:00.657Z", "content": "Yeah.. I think I’ll hit 0.82 IMU soon and will still be 0.83 on LB haha -_- I’m sure we’ll figure it out but a bit frustrating for now!
", "rawMarkdown": "Yeah.. I think I’ll hit 0.82 IMU soon and will still be 0.83 on LB haha -_- I’m sure we’ll figure it out but a bit frustrating for now!" }, { "id": 3242465, "postDate": "2025-07-05T23:02:24.770Z", "content": "Yeah, agreed. I decided to not tunnel on it too much and work towards a better IMU only first, I spent the last 2 weeks trying to solve this and probe but did not get to any resolution.
", "rawMarkdown": "Yeah, agreed. I decided to not tunnel on it too much and work towards a better IMU only first, I spent the last 2 weeks trying to solve this and probe but did not get to any resolution." } ] }, { "id": 3242663, "postDate": "2025-07-06T07:37:56.640Z", "content": "Same here — my CV is around 0.85 using only IMU data and goes up to 0.90 when using all sensors. Correspondingly, my public LB scores are 0.82 and 0.84. I've been stuck at this CV plateau for the past two weeks and haven’t been able to push it further.
", "rawMarkdown": "Same here — my CV is around 0.85 using only IMU data and goes up to 0.90 when using all sensors. Correspondingly, my public LB scores are 0.82 and 0.84. I've been stuck at this CV plateau for the past two weeks and haven’t been able to push it further.", "votes": 1 } ] }, { "id": 3233108, "postDate": "2025-06-26T13:37:49.573Z", "content": "| Date | \n5 Fold CV | \nPublic LB | \nCV Strategy | \n
|---|---|---|---|
| 26. June | \n0.793382 | \n0.82 | \nStratifiedGroupKFold, groups=subject | \n
| 7. July | \n0.815587 | \n0.83 | \nStratifiedGroupKFold, groups=subject | \n
Hey Mahdi, thanks for sharing! If you want, can you say what kind of model and what you stratify for? Would be interesting for me :).
\nBest,
\nJan
Hi Jan,
\nThis is based on a fork of the current best public LB notebook. I’ve added cross-validation using gesture for stratification, but the main improvement I made to the notebook has to do with feature engineering.
", "rawMarkdown": "Hi Jan,\n\nThis is based on a fork of the [current best public LB notebook](https://www.kaggle.com/code/nksusth/lb-0-78-quaternions-tf-bilstm-gru-attention ). I’ve added cross-validation using gesture for stratification, but the main improvement I made to the notebook has to do with feature engineering.", "votes": 2 }, { "id": 3233483, "postDate": "2025-06-26T20:54:20.097Z", "content": "Hey Mahdi, awesome results!
\nIf im not mistaken, that same CV strategy on that notebook yeilds a CV of 0.81 and a LB of 0.78. Yet after you added your feature engineering, the CV goes down and the LB goes up. So are you basing what feature engineering works purely on the LB?
", "rawMarkdown": "Hey Mahdi, awesome results!\n\nIf im not mistaken, that same CV strategy on that notebook yeilds a CV of 0.81 and a LB of 0.78. Yet after you added your feature engineering, the CV goes down and the LB goes up. So are you basing what feature engineering works purely on the LB?", "votes": 1 }, { "id": 3233784, "postDate": "2025-06-27T06:57:56.010Z", "content": "Hi Ravon,
\nI may be overfitting to the public LB, but that wasn't my intention. I based my feature engineering and selection on CV alone. Before I started experimenting with different features, I made a change to the notebook that gave me a better CV-LB correlation (CV went down, but LB remained the same). However, as I added more features, my CV improved only slightly, while the public LB improved by a lot.
", "rawMarkdown": "Hi Ravon,\n\nI may be overfitting to the public LB, but that wasn't my intention. I based my feature engineering and selection on CV alone. Before I started experimenting with different features, I made a change to the notebook that gave me a better CV-LB correlation (CV went down, but LB remained the same). However, as I added more features, my CV improved only slightly, while the public LB improved by a lot.", "votes": 2 }, { "id": 3234477, "postDate": "2025-06-27T22:13:22.560Z", "content": "Could you give a hint on what you did to improve CV-LB correlation? I'm really struggling to create a strong CV for that notebook. 😭
", "rawMarkdown": "Could you give a hint on what you did to improve CV-LB correlation? I'm really struggling to create a strong CV for that notebook. 😭" } ] } ] }, { "id": 3221268, "postDate": "2025-06-10T17:09:23.990Z", "content": "StratifiedGroupKfold +IMU Data only + mixup
\nCV = 0.7358
\nLB = 0.76
ahh on 5 folds that is better correlation really nice only with IMU ..
", "rawMarkdown": "ahh on 5 folds that is better correlation really nice only with IMU ..", "votes": 1, "replies": [ { "id": 3221857, "postDate": "2025-06-11T14:28:48.097Z", "content": "Yeah, it seems that figuring out what works with IMU-only data is the key to this competition. But So far, making the network deeper and wider hasn’t really helped on my side after lb 0.76.
", "rawMarkdown": "Yeah, it seems that figuring out what works with IMU-only data is the key to this competition. But So far, making the network deeper and wider hasn’t really helped on my side after lb 0.76.", "votes": 2 }, { "id": 3221889, "postDate": "2025-06-11T15:15:19.190Z", "content": "Can elaborate on your implementation of mixup? I trief it and it made everything mich worse…
", "rawMarkdown": "Can elaborate on your implementation of mixup? I trief it and it made everything mich worse…" }, { "id": 3221960, "postDate": "2025-06-11T16:21:32.630Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3221964, "postDate": "2025-06-11T16:23:45.713Z", "content": "I'm basically using the same CV setup from @alejopaullier's notebook. I modified the training code, which boosted my local CV score by around 0.005. However, my best LB score was achieved with mixup, but some of my previous submissions without it actually scored higher than other attempts that did include it.
\n ():\n model.train()\n criterion1 = nn.CrossEntropyLoss(reduction=)\n criterion2 = nn.BCEWithLogitsLoss()\n scaler = torch.amp.GradScaler(enabled=config.AMP)\n losses, losses1, losses2 = AverageMeter(), AverageMeter(), AverageMeter()\n start = end = time.time()\n global_step = \n tqdm(\n train_loader, unit=, desc=\n ) tqdm_train_loader:\n step, batch (tqdm_train_loader):\n X = batch.pop().to(device)\n y = batch.pop().to(device)\n y_hard = batch.pop().to(device)\n batch_size = y.size()\n (\n config.MIXUP_ALPHA > \n np.random.rand() < config.MIXUP_PROB\n ):\n lam = np.random.beta(config.MIXUP_ALPHA, config.MIXUP_ALPHA)\n batch_indices = torch.randperm(batch_size)\n\n \n X_mixed = lam * X + ( - lam) * X[batch_indices]\n\n \n y_a, y_b = y, y[batch_indices]\n y_hard_a, y_hard_b = y_hard, y_hard[batch_indices]\n\n torch.amp.autocast(device., enabled=config.AMP):\n y_preds, y_preds_hard = model(X_mixed)\n\n \n loss1 = lam * criterion1(y_preds, y_a) + (\n - lam\n ) * criterion1(y_preds, y_b)\n\n \n y_hard_mixed = lam * y_hard_a + ( - lam) * y_hard_b\n loss2 = criterion2(\n y_preds_hard, y_hard_mixed.unsqueeze()\n )\n\n loss = loss1 + loss2\n :\n \n torch.amp.autocast(device., enabled=config.AMP):\n y_preds, y_preds_hard = model(X)\n loss1 = criterion1(y_preds, y)\n loss2 = criterion2(y_preds_hard, y_hard.unsqueeze())\n loss = loss1 + loss2\n\n config.GRADIENT_ACCUMULATION_STEPS > :\n loss = loss / config.GRADIENT_ACCUMULATION_STEPS\n losses.update(loss.item(), batch_size)\n losses1.update(loss1.item(), batch_size)\n losses2.update(loss2.item(), batch_size)\n scaler.scale(loss).backward()\n grad_norm = torch.nn.utils.clip_grad_norm_(\n model.parameters(), config.MAX_GRAD_NORM\n )\n\n (step + ) % config.GRADIENT_ACCUMULATION_STEPS == :\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad()\n global_step += \n scheduler.step()\n end = time.time()\n\n losses.avg\nThank you for sharing! So at one point it helped but you improved to a point where it does not help anymore? Interesting find! :)
", "rawMarkdown": "Thank you for sharing! So at one point it helped but you improved to a point where it does not help anymore? Interesting find! :) " }, { "id": 3223829, "postDate": "2025-06-13T22:53:40.893Z", "content": "i think I'm still going to trust my local CV and continue using the mixup method. because I've noticed that even submissions from the exact same model can result in score variations. but that can be issues on my end though… :)
", "rawMarkdown": "i think I'm still going to trust my local CV and continue using the mixup method. because I've noticed that even submissions from the exact same model can result in score variations. but that can be issues on my end though... :)" } ] } ] }, { "id": 3219738, "postDate": "2025-06-08T09:05:03.513Z", "content": "Thank @chumajin ! Could you please explain why “for this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it”. Why don’t we use all but only the ending?
", "rawMarkdown": "Thank @chumajin ! Could you please explain why “for this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it”. Why don’t we use all but only the ending?", "votes": 1, "replies": [ { "id": 3219745, "postDate": "2025-06-08T09:09:11.610Z", "content": "@nguyenhoa Thanks for your comment.
\nIt's because it's easy to validate and the dataset is small.
\nBut you're also welcome to use the entire dataset and take the average of the predictions.
What loss function have you guys tried to use if can be shared or how have been your observations. There.. I’ll chime in just experimenting
", "rawMarkdown": "What loss function have you guys tried to use if can be shared or how have been your observations. There.. I’ll chime in just experimenting " }, { "id": 3220557, "postDate": "2025-06-09T13:56:21.920Z", "content": "\n\nWhat loss function have you guys tried to use if can be shared or how have been your observations. There.. I’ll chime in just experimenting
\n
@gauravbrills I'm using CrossEntropy like normal multi-class classification problem. My best validation losses for IMU only models are around 1.
", "rawMarkdown": "> What loss function have you guys tried to use if can be shared or how have been your observations. There.. I’ll chime in just experimenting\n\n@gauravbrills I'm using CrossEntropy like normal multi-class classification problem. My best validation losses for IMU only models are around 1.\n" } ] } ] }, { "id": 3219657, "postDate": "2025-06-08T06:20:30.917Z", "content": "StratifiedGroupKfold +IMU Data only
\nCV = 0.6742
\nLB = 0.68
@sayedathar11 Thank you. good correlation.
", "rawMarkdown": "@sayedathar11 Thank you. good correlation.", "votes": 1 } ] }, { "id": 3215579, "postDate": "2025-06-02T12:47:03.280Z", "content": "For this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it. Doesnt this reduces the number of datapoints available for training ?
", "rawMarkdown": "For this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it. Doesnt this reduces the number of datapoints available for training ? ", "votes": 1, "replies": [ { "id": 3215583, "postDate": "2025-06-02T12:51:03.960Z", "content": "@sayedathar11 Yes, you are right. Training data reduces. Just 1st step for trying.
", "rawMarkdown": "@sayedathar11 Yes, you are right. Training data reduces. Just 1st step for trying. ", "votes": 1 } ] }, { "id": 3214965, "postDate": "2025-06-01T10:49:21.937Z", "content": "Just did my first submission, nothing fancy, 5 stratified k folds.
\ncv: 0.6809, LB: 0.66
\nedit: group (on subject) stratified seems closer, :)
\ncv: 0.6727, LB: 0.67
", "rawMarkdown": "Just did my first submission, nothing fancy, 5 stratified k folds.\n\n\ncv: 0.6809, LB: 0.66\n\n\nedit: group (on subject) stratified seems closer, :)\n\ncv: 0.6727, LB: 0.67", "votes": 1 }, { "id": 3251870, "postDate": "2025-07-21T17:30:41.463Z", "content": "Since this discussion is getting a lot of attraction, I figured I would repost my question here(lmk if it's inappropriate, in which case I'll delete it).
\nDuring training I get a CV average of 0.57 but the corresponding Leader board is 0.32.
I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:
\n () -> [[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n train_idx, valid_idx sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:
() -> :\n model = model.()\n model_device = (model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = \n total_test_loss = \n torch.inference_mode():\n x, y validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // \n x_front = x[:half] \n x_back = x[half:].clone() \n\n x_back[:, :, non_imu_feats_idx] = \n x = torch.cat([x_front, x_back], dim=) \n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n {\n : total_test_loss / (validation_loader),\n : total_accuracy / (validation_loader),\n : competition_metric.score(),\n }\nThe current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, [training notebook]During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.
I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:
\n () -> [[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n train_idx, valid_idx sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:
() -> :\n model = model.()\n model_device = (model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = \n total_test_loss = \n torch.inference_mode():\n x, y validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // \n x_front = x[:half] \n x_back = x[half:].clone() \n\n x_back[:, :, non_imu_feats_idx] = \n x = torch.cat([x_front, x_back], dim=) \n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n {\n : total_test_loss / (validation_loader),\n : total_accuracy / (validation_loader),\n : competition_metric.score(),\n }\nThe current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, training notebook and inference notebook.
Do you see something I am doing wrong?
\nReading the entire pipeline would be cumbersome, so let me know if you need extra info to help.
\nEvery answer/advice is welcome.
\nThanks in advance!
CV(all): 0.867
\nCV(IMU only): 0.811
\nLB: 0.842
cv: imu only 0.8181, imu+thm+tof 0.8638
\npublic lb: 0.84
StratifiedGroupKFold, groups=subject cv Average Val Acc: 0.8194 Val Acc Std Dev: 0.0386 LB0.81
", "rawMarkdown": "StratifiedGroupKFold, groups=subject cv Average Val Acc: 0.8194 Val Acc Std Dev: 0.0386 LB0.81" }, { "id": 3244925, "postDate": "2025-07-08T17:13:45.147Z", "content": "Imu only : StratifiedGroupKFold, split by subject: cv : 0.7910 LB : 0.81
\nFull data : StratifiedKFold, split by gesture: cv : 0.83 LB : 0.8
\nEnsemble : LB : 0.82
In our split strategy, imu+tof+thm cv: 0.82~0.83, imu: 0.781, then lb is 0.82.
", "rawMarkdown": "In our split strategy, imu+tof+thm cv: 0.82~0.83, imu: 0.781, then lb is 0.82. ", "replies": [ { "id": 3242910, "postDate": "2025-07-06T14:29:31.557Z", "content": "Hey Tom, I may have missed it but can you give some details on your split strategy?
\nBest,
\nJan
CV and LB scores are correlated, but the variability between folds is concerning. We need to analyze the reasons behind the under-performing folds.
\n| fold0 | \nfold1 | \nfold2 | \nfold3 | \nfold4 | \nCV | \nLB | \nCV Strategy | \n
|---|---|---|---|---|---|---|---|
| 0.8282 | \n0.7782 | \n0.7601 | \n0.7549 | \n0.7926 | \n0.7828 | \n0.8 | \nStratifiedGroupKFold, groups=subject | \n
CV:(sgkf on subject and gesture)
\nIMU 0.825~0.828
\nAll: 0.886
LB:
\nIMU:0.839~0.844
\n(0.5IMU+0.5All):0.868
Any top teams willing to share their scores?
", "votes": 8, "replies": [ { "id": 3268029, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-12T09:22:20.007000", "content": "LB score 0.852
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
\nCV (sgkf on subject and gesture)
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
Your IMU-only score is awesome.
", "votes": 0, "replies": [] }, { "id": 3268041, "author_name": "greySnow", "author_url": "", "post_date": "2025-08-12T09:40:15.460000", "content": "TY for confirming IMU can be pushed so much! I assume it's a single model? Wondering how much it can be pushed more with ensembling~
", "votes": 0, "replies": [ { "id": 3268043, "author_name": "Ruby", "author_url": "", "post_date": "2025-08-12T09:45:44.073000", "content": "Yes, single model. I am wondering too, I don't have diversed work to ensemble currently.
", "votes": 0, "replies": [] } ] }, { "id": 3268459, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-08-13T01:43:43.033000", "content": "My best imu only CV: 0.824
\nLB: 0.834
I'm still confused about all sensors cv…
", "votes": 1, "replies": [ { "id": 3274578, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-08-25T03:48:47.503000", "content": "2025.8.25 update:
\nall sensors cv: 0.863 lb:0.865
", "votes": 0, "replies": [] } ] }, { "id": 3269417, "author_name": "yu4u", "author_url": "", "post_date": "2025-08-14T13:07:32.870000", "content": "CV(sgkf on subject and gesture)
\nIMU: 0.8390
\nAll: 0.8861
LB:
\n0.857
--overfitting?
", "votes": 1, "replies": [ { "id": 3269437, "author_name": "Ruby", "author_url": "", "post_date": "2025-08-14T13:37:31.647000", "content": "Did you test IMU only LB score?
", "votes": 0, "replies": [] }, { "id": 3269514, "author_name": "yu4u", "author_url": "", "post_date": "2025-08-14T15:22:04.420000", "content": "IMU only LB: 0.835
", "votes": 0, "replies": [] } ] }, { "id": 3269561, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-14T16:32:46.667000", "content": "Thanks for sharing, this confirms my IMU model is quite weak -
\nCV 0.827 - LB 0.822
\n(sgkf on subject and gesture)
\nMy IMU+TOF+THM on the other end looks alright :
\nCV 0.887 - LB 0.859
Hello, did you first try to sgkf first on subject before doing it on subject and gesture?
\nIf so, did you notice a diffrence in your lb score when switching from per subject sgkf to per subject and gesture sgkf?
Did not try no. It should not make any difference imo
", "votes": 0, "replies": [] }, { "id": 3271703, "author_name": "Mauro Abidal Carrer", "author_url": "", "post_date": "2025-08-19T13:25:07.627000", "content": "okok thx you for your answer
", "votes": 0, "replies": [] } ] }, { "id": 3269893, "author_name": "", "author_url": "", "post_date": "2025-08-15T09:16:54.573000", "content": "", "votes": 0, "replies": [] }, { "id": 3273198, "author_name": "liuzhangzhen", "author_url": "", "post_date": "2025-08-22T10:41:25.633000", "content": "LB:0.860
\nCV:(5folds: sgkf on subject and gesture)
\nIMU 0.807
\nAll: 0.863
\nmy cv is quiet low….
0.78
", "votes": 9, "replies": [ { "id": 3214786, "author_name": "213tubo", "author_url": "", "post_date": "2025-06-01T04:38:38.397000", "content": "Could you please share how you calculated your CV and how you split the data?
", "votes": 1, "replies": [ { "id": 3214794, "author_name": "phalanx", "author_url": "", "post_date": "2025-06-01T04:55:01.343000", "content": "I split the data with StratifiedGroupKFold, using each subject as the group label.
\nIdeally, the split should also account for metadata such as age and gender, but I haven’t implemented that yet.
Following the competition’s evaluation metric, I predict the gesture for each sequence ID and then compute the average Binary F1 and Macro F1 scores.
", "votes": 12, "replies": [] }, { "id": 3215041, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2025-06-01T14:07:41.700000", "content": "Thanks I did without strat and group and it was all over the place will try this .
", "votes": 1, "replies": [] } ] }, { "id": 3215486, "author_name": "", "author_url": "", "post_date": "2025-06-02T09:55:00.657000", "content": "", "votes": 1, "replies": [] }, { "id": 3218328, "author_name": "phalanx", "author_url": "", "post_date": "2025-06-06T04:54:18.303000", "content": "update:
\nThanks for sharing! Did you incorporate demographic data in your model?
", "votes": 1, "replies": [] }, { "id": 3218602, "author_name": "phalanx", "author_url": "", "post_date": "2025-06-06T12:20:57.917000", "content": "Yes, but I'd probably get about the same score without using it.
", "votes": 3, "replies": [] }, { "id": 3218608, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-06T12:27:50.370000", "content": "I have the same experimental results, leading me to suspect that demographic data may not provide sufficient information, as a single subject can correspond to various gesture sequences.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3251295, "author_name": "Sercan Yeşilöz", "author_url": "", "post_date": "2025-07-20T11:49:36.517000", "content": "| Model | \nCV | \nCV Strategy | \nLB | \n
|---|---|---|---|
| 1D CNN (IMU Only) | \n0.78129 | \nStratifiedGroupKFold (5 Folds) | \n0.79 | \n
| LightGBM (IMU Only) | \n0.77183 | \nStratifiedGroupKFold (5 Folds) | \n\n |
| LightGBM (All Data) | \n0.84145 | \nStratifiedGroupKFold (5 Folds) | \n\n |
| 1D CNN + LSTM (IMU Only) | \n0.77332 | \nStratifiedGroupKFold (5 Folds) | \n\n |
| 1D CNN + LSTM (All Data) | \n0.83636 | \nStratifiedGroupKFold (5 Folds) | \n\n |
Thanks for sharing! May I aks is it your ensemble used? And it is amazing lgbm! I can't push lgbm very well…
", "votes": 0, "replies": [ { "id": 3251458, "author_name": "", "author_url": "", "post_date": "2025-07-20T20:43:22.360000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 3252301, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-07-22T12:08:23.113000", "content": "@sercanyesiloz Can you share the LB scores of these models as well?
", "votes": 0, "replies": [] } ] }, { "id": 3213969, "author_name": "chumajin", "author_url": "", "post_date": "2025-05-30T17:42:32.137000", "content": "Update !
\nI use all sensors features :
\ncv: 0.73542811647 (binary f1: 0.93720089, f1 macro:0.53365533)
\nAnd for submission :
\nI switched the model by checking whether \"tof_5_v63\" is null or not.
public lb : 0.65
\nHere, I calculated the average with the cv for the only IMU sensor and all sensors.
\n(0.5798545 + 0.7354281) / 2 = 0.657639058
This seems to align perfectly for cv and public lb.
\nThis is the good competition.
thanks for posting! could you explain the rationale why using the end of the sequence is a good first step here?
", "votes": 1, "replies": [ { "id": 3214114, "author_name": "Tom", "author_url": "", "post_date": "2025-05-31T01:00:48.687000", "content": "@jonwibowo You can get the total sequence length from sequence_counter. But I think we need to build a model that takes the sequence as input, current approach uses summarized info to build a simple tabular predictor.
\nWe have five labels, it's interesting to figure out a good way handling label information.
\nOne possible new label development is:
sequence_type, ordientation, behavior, phase and gesture in the text.@tom99763 Right! Thank you for comment.
", "votes": 0, "replies": [] }, { "id": 3214194, "author_name": "chumajin", "author_url": "", "post_date": "2025-05-31T04:59:53.047000", "content": "@jonwibowo I believe the very end of a sequence tends to capture some aggregation or summary of the preceding activity. For this type of tabular data, features from the end of the sequence are often utilized.
\nHowever, in this particular case, there isn't any single feature (like those from the end) that distinctly stands out.
\nUltimately, it might be best to use all the data and apply a time-series model.
0.82 CV - 0.80 LB (don’t know if high or low 0.80), using Stratified K fold, will test with Group Stratified K fold as well. IMU only
\nL.E.: based on other competitors reaching 0.8, I believe it's high 0.8, which is in line with 0.01-0.015 gap I keep experiencing with this split strategy.
Nice just imu that is great
", "votes": 0, "replies": [] }, { "id": 3223835, "author_name": "Natsume", "author_url": "", "post_date": "2025-06-14T00:01:39.380000", "content": "When your cv exceeds 0.8, does the correlation between lb and cv weaken? In my experiment, when it exceeds 0.8, I tried to increase cv to 0.81, but lb did decrease by 0.01.
\nOr maybe i make some data leakage…
May I ask how do you split your folds?
", "votes": 0, "replies": [] }, { "id": 3223915, "author_name": "", "author_url": "", "post_date": "2025-06-14T05:33:25.437000", "content": "", "votes": 0, "replies": [] }, { "id": 3223917, "author_name": "Natsume", "author_url": "", "post_date": "2025-06-14T05:42:45.567000", "content": "Stratified K fold，split by gesture
", "votes": 0, "replies": [] }, { "id": 3224275, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-06-14T16:02:48.640000", "content": "For stratified k fold, split by gesture, I observe a constant 0.01-0.015 gap between cv and lb (cv being higher).
\nFor stratified group k fold, other way around.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3216995, "author_name": "Ye_Ai", "author_url": "", "post_date": "2025-06-04T11:42:51.907000", "content": "| Feature | \nCV | \nLB | \n
|---|---|---|
| IMU | \n0.734 | \n0.74 | \n
| IMU+THM+TOF | \n0.812 | \n0.73 | \n
| OverAll | \n0.773 | \n0.78 | \n
@ye11725 Thanks. That's amazing. So, for the IMU+THM+TOF Leaderboard (LB), you just submitted it as null, right?
", "votes": 1, "replies": [ { "id": 3217355, "author_name": "Ye_Ai", "author_url": "", "post_date": "2025-06-04T23:41:21.670000", "content": "Just use the IMU+THM+TOF model to predict all test sets, including data with null TOF values
", "votes": 2, "replies": [] } ] } ] }, { "id": 3215236, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-01T20:47:51.573000", "content": "1d-cnn, IMU feature only, label based stratified 1 fold local validation 0.703, lb 0.68
", "votes": 3, "replies": [ { "id": 3215924, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-02T21:27:29.600000", "content": "update: IMU + TMP + TOF: 0.766 + IMU only: 0.703, LB 0.71
", "votes": 2, "replies": [ { "id": 3215949, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2025-06-02T22:47:05.570000", "content": "Are you inferring on the ensemble of 5 fold? Dunno I get better lb lower cv though in same comparison as you for 0.68
", "votes": 1, "replies": [] }, { "id": 3215950, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-02T22:55:09.073000", "content": "For now, I’m using a simple split with a single model for a quick experiment. Given this setup, it’s expected that the gap between CV and LB would be larger compared to a 5-fold ensemble
", "votes": 3, "replies": [] }, { "id": 3216295, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2025-06-03T11:14:27.963000", "content": "Thanks ya now though for Imu only getting nice correlation 0.686 - 0.68
", "votes": 1, "replies": [] }, { "id": 3216349, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-03T12:50:16.973000", "content": "I just applied 5 folds cv:
\nIMU only 0.705 - 0.69
\nIMU only + IMU,tof,tmp (0.705, 0.761) - 0.72
\ncorrelation seems better now
ya 5 fold cv works for me also :)
\n0.7665 ~ 0.7 LB
", "votes": 1, "replies": [] } ] }, { "id": 3221793, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-11T12:38:17.033000", "content": "update:
\nusing (gender + handedness + adult_child) as the target, StratifiedGroupKFold by subject
\ncv: IMU + TMP + TOF: 0.8103 + IMU only: 0.7610, LB 0.81
so an ensemble of both ?
", "votes": 0, "replies": [] }, { "id": 3221850, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-11T14:06:42.677000", "content": "Use the IMU-only models if the test data does not contain TOF & TMP columns. Use the full sensor models if the test data includes TOF & TMP columns.
", "votes": 1, "replies": [] }, { "id": 3227607, "author_name": "Myo Min Htet(wnp)🇲🇲", "author_url": "", "post_date": "2025-06-19T05:51:26.907000", "content": "did u mean (gender + handedness + adult_child) + gesture as labels
", "votes": 0, "replies": [] }, { "id": 3227653, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-06-19T06:50:58.287000", "content": "for example:
\n \n composite_labels = []\n subject subject_groups:\n subject_demo = subject_demographics_map[subject]\n sex = subject_demo[]\n handedness = subject_demo[] \n adult_child = subject_demo[]\n\n \n \n composite_label = sex + handedness * + adult_child * \n composite_labels.append(composite_label)\nHow much improvement did the subtask bring?
\nI tried orientation prediction as a subtask, which led to a +0.007 improvement in CV (5-fold average), but the LB score was nearly the same.
It is just about using composite_labels for stratified groupk split
", "votes": 0, "replies": [] } ] } ] }, { "id": 3217253, "author_name": "chumajin", "author_url": "", "post_date": "2025-06-04T18:00:35.247000", "content": "update2 : use stratifiedgroupkfold (group : \"subject\" ) is better than (group : \"sequence id\")
\n・ group : \"sequence id\" result
\nIMU only : cv 0.694 lb : 0.66
\nIMU+THM+TOF :cv 0.796, cv average 0.745, lb : 0.7
・ group : \"object\" result
\nIMU only : cv 0.665 lb : 0.65
\nIMU+THM+TOF :cv 0.751, cv average 0.705, lb : 0.7 almost same
ya similar IMU+THM+TOF :cv average
\nIs anyone finding their CV-LB less consistent on IMU + THM + TOF? My IMU only is .82 cv, .81 lb.. my full data model has cv of .88 but only giving increase of 0.02 in test submission. I’m aware test data is 50/50 and splitting models accordingly.
", "votes": 1, "replies": [ { "id": 3242449, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-07-05T21:42:36.957000", "content": "I have the exact same thing you describe, except that I get .82 lb imu only, and .83 with all, despite .9 cv for imu+tof+thm.
", "votes": 0, "replies": [ { "id": 3242457, "author_name": "Jack", "author_url": "", "post_date": "2025-07-05T21:56:00.657000", "content": "Yeah.. I think I’ll hit 0.82 IMU soon and will still be 0.83 on LB haha -_- I’m sure we’ll figure it out but a bit frustrating for now!
", "votes": 0, "replies": [] }, { "id": 3242465, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-07-05T23:02:24.770000", "content": "Yeah, agreed. I decided to not tunnel on it too much and work towards a better IMU only first, I spent the last 2 weeks trying to solve this and probe but did not get to any resolution.
", "votes": 0, "replies": [] } ] }, { "id": 3242663, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-07-06T07:37:56.640000", "content": "Same here — my CV is around 0.85 using only IMU data and goes up to 0.90 when using all sensors. Correspondingly, my public LB scores are 0.82 and 0.84. I've been stuck at this CV plateau for the past two weeks and haven’t been able to push it further.
", "votes": 1, "replies": [] } ] }, { "id": 3233108, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-26T13:37:49.573000", "content": "| Date | \n5 Fold CV | \nPublic LB | \nCV Strategy | \n
|---|---|---|---|
| 26. June | \n0.793382 | \n0.82 | \nStratifiedGroupKFold, groups=subject | \n
| 7. July | \n0.815587 | \n0.83 | \nStratifiedGroupKFold, groups=subject | \n
Hey Mahdi, thanks for sharing! If you want, can you say what kind of model and what you stratify for? Would be interesting for me :).
\nBest,
\nJan
Hi Jan,
\nThis is based on a fork of the current best public LB notebook. I’ve added cross-validation using gesture for stratification, but the main improvement I made to the notebook has to do with feature engineering.
", "votes": 2, "replies": [] }, { "id": 3233483, "author_name": "Ravon", "author_url": "", "post_date": "2025-06-26T20:54:20.097000", "content": "Hey Mahdi, awesome results!
\nIf im not mistaken, that same CV strategy on that notebook yeilds a CV of 0.81 and a LB of 0.78. Yet after you added your feature engineering, the CV goes down and the LB goes up. So are you basing what feature engineering works purely on the LB?
", "votes": 1, "replies": [] }, { "id": 3233784, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-27T06:57:56.010000", "content": "Hi Ravon,
\nI may be overfitting to the public LB, but that wasn't my intention. I based my feature engineering and selection on CV alone. Before I started experimenting with different features, I made a change to the notebook that gave me a better CV-LB correlation (CV went down, but LB remained the same). However, as I added more features, my CV improved only slightly, while the public LB improved by a lot.
", "votes": 2, "replies": [] }, { "id": 3234477, "author_name": "Ravon", "author_url": "", "post_date": "2025-06-27T22:13:22.560000", "content": "Could you give a hint on what you did to improve CV-LB correlation? I'm really struggling to create a strong CV for that notebook. 😭
", "votes": 0, "replies": [] } ] } ] }, { "id": 3221268, "author_name": "Zi", "author_url": "", "post_date": "2025-06-10T17:09:23.990000", "content": "StratifiedGroupKfold +IMU Data only + mixup
\nCV = 0.7358
\nLB = 0.76
ahh on 5 folds that is better correlation really nice only with IMU ..
", "votes": 1, "replies": [ { "id": 3221857, "author_name": "Zi", "author_url": "", "post_date": "2025-06-11T14:28:48.097000", "content": "Yeah, it seems that figuring out what works with IMU-only data is the key to this competition. But So far, making the network deeper and wider hasn’t really helped on my side after lb 0.76.
", "votes": 2, "replies": [] }, { "id": 3221889, "author_name": "Jan Brederecke", "author_url": "", "post_date": "2025-06-11T15:15:19.190000", "content": "Can elaborate on your implementation of mixup? I trief it and it made everything mich worse…
", "votes": 0, "replies": [] }, { "id": 3221960, "author_name": "", "author_url": "", "post_date": "2025-06-11T16:21:32.630000", "content": "", "votes": 0, "replies": [] }, { "id": 3221964, "author_name": "Zi", "author_url": "", "post_date": "2025-06-11T16:23:45.713000", "content": "I'm basically using the same CV setup from @alejopaullier's notebook. I modified the training code, which boosted my local CV score by around 0.005. However, my best LB score was achieved with mixup, but some of my previous submissions without it actually scored higher than other attempts that did include it.
\n ():\n model.train()\n criterion1 = nn.CrossEntropyLoss(reduction=)\n criterion2 = nn.BCEWithLogitsLoss()\n scaler = torch.amp.GradScaler(enabled=config.AMP)\n losses, losses1, losses2 = AverageMeter(), AverageMeter(), AverageMeter()\n start = end = time.time()\n global_step = \n tqdm(\n train_loader, unit=, desc=\n ) tqdm_train_loader:\n step, batch (tqdm_train_loader):\n X = batch.pop().to(device)\n y = batch.pop().to(device)\n y_hard = batch.pop().to(device)\n batch_size = y.size()\n (\n config.MIXUP_ALPHA > \n np.random.rand() < config.MIXUP_PROB\n ):\n lam = np.random.beta(config.MIXUP_ALPHA, config.MIXUP_ALPHA)\n batch_indices = torch.randperm(batch_size)\n\n \n X_mixed = lam * X + ( - lam) * X[batch_indices]\n\n \n y_a, y_b = y, y[batch_indices]\n y_hard_a, y_hard_b = y_hard, y_hard[batch_indices]\n\n torch.amp.autocast(device., enabled=config.AMP):\n y_preds, y_preds_hard = model(X_mixed)\n\n \n loss1 = lam * criterion1(y_preds, y_a) + (\n - lam\n ) * criterion1(y_preds, y_b)\n\n \n y_hard_mixed = lam * y_hard_a + ( - lam) * y_hard_b\n loss2 = criterion2(\n y_preds_hard, y_hard_mixed.unsqueeze()\n )\n\n loss = loss1 + loss2\n :\n \n torch.amp.autocast(device., enabled=config.AMP):\n y_preds, y_preds_hard = model(X)\n loss1 = criterion1(y_preds, y)\n loss2 = criterion2(y_preds_hard, y_hard.unsqueeze())\n loss = loss1 + loss2\n\n config.GRADIENT_ACCUMULATION_STEPS > :\n loss = loss / config.GRADIENT_ACCUMULATION_STEPS\n losses.update(loss.item(), batch_size)\n losses1.update(loss1.item(), batch_size)\n losses2.update(loss2.item(), batch_size)\n scaler.scale(loss).backward()\n grad_norm = torch.nn.utils.clip_grad_norm_(\n model.parameters(), config.MAX_GRAD_NORM\n )\n\n (step + ) % config.GRADIENT_ACCUMULATION_STEPS == :\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad()\n global_step += \n scheduler.step()\n end = time.time()\n\n losses.avg\nThank you for sharing! So at one point it helped but you improved to a point where it does not help anymore? Interesting find! :)
", "votes": 0, "replies": [] }, { "id": 3223829, "author_name": "Zi", "author_url": "", "post_date": "2025-06-13T22:53:40.893000", "content": "i think I'm still going to trust my local CV and continue using the mixup method. because I've noticed that even submissions from the exact same model can result in score variations. but that can be issues on my end though… :)
", "votes": 0, "replies": [] } ] } ] }, { "id": 3219738, "author_name": "Pi", "author_url": "", "post_date": "2025-06-08T09:05:03.513000", "content": "Thank @chumajin ! Could you please explain why “for this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it”. Why don’t we use all but only the ending?
", "votes": 1, "replies": [ { "id": 3219745, "author_name": "chumajin", "author_url": "", "post_date": "2025-06-08T09:09:11.610000", "content": "@nguyenhoa Thanks for your comment.
\nIt's because it's easy to validate and the dataset is small.
\nBut you're also welcome to use the entire dataset and take the average of the predictions.
What loss function have you guys tried to use if can be shared or how have been your observations. There.. I’ll chime in just experimenting
", "votes": 0, "replies": [] }, { "id": 3220557, "author_name": "Pi", "author_url": "", "post_date": "2025-06-09T13:56:21.920000", "content": "\n\nWhat loss function have you guys tried to use if can be shared or how have been your observations. There.. I’ll chime in just experimenting
\n
@gauravbrills I'm using CrossEntropy like normal multi-class classification problem. My best validation losses for IMU only models are around 1.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3219657, "author_name": "Athar Sayed", "author_url": "", "post_date": "2025-06-08T06:20:30.917000", "content": "StratifiedGroupKfold +IMU Data only
\nCV = 0.6742
\nLB = 0.68
@sayedathar11 Thank you. good correlation.
", "votes": 1, "replies": [] } ] }, { "id": 3215579, "author_name": "Athar Sayed", "author_url": "", "post_date": "2025-06-02T12:47:03.280000", "content": "For this type of continuous data, the first step is to create a model using only the very end of the sequence and evaluate it. Doesnt this reduces the number of datapoints available for training ?
", "votes": 1, "replies": [ { "id": 3215583, "author_name": "chumajin", "author_url": "", "post_date": "2025-06-02T12:51:03.960000", "content": "@sayedathar11 Yes, you are right. Training data reduces. Just 1st step for trying.
", "votes": 1, "replies": [] } ] }, { "id": 3214965, "author_name": "Gerwyn", "author_url": "", "post_date": "2025-06-01T10:49:21.937000", "content": "Just did my first submission, nothing fancy, 5 stratified k folds.
\ncv: 0.6809, LB: 0.66
\nedit: group (on subject) stratified seems closer, :)
\ncv: 0.6727, LB: 0.67
", "votes": 1, "replies": [] }, { "id": 3251870, "author_name": "Mauro Abidal Carrer", "author_url": "", "post_date": "2025-07-21T17:30:41.463000", "content": "Since this discussion is getting a lot of attraction, I figured I would repost my question here(lmk if it's inappropriate, in which case I'll delete it).
\nDuring training I get a CV average of 0.57 but the corresponding Leader board is 0.32.
I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:
\n () -> [[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n train_idx, valid_idx sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:
() -> :\n model = model.()\n model_device = (model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = \n total_test_loss = \n torch.inference_mode():\n x, y validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // \n x_front = x[:half] \n x_back = x[half:].clone() \n\n x_back[:, :, non_imu_feats_idx] = \n x = torch.cat([x_front, x_back], dim=) \n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n {\n : total_test_loss / (validation_loader),\n : total_accuracy / (validation_loader),\n : competition_metric.score(),\n }\nThe current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, [training notebook]During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.
I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:
\n () -> [[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n train_idx, valid_idx sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:
() -> :\n model = model.()\n model_device = (model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = \n total_test_loss = \n torch.inference_mode():\n x, y validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // \n x_front = x[:half] \n x_back = x[half:].clone() \n\n x_back[:, :, non_imu_feats_idx] = \n x = torch.cat([x_front, x_back], dim=) \n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n {\n : total_test_loss / (validation_loader),\n : total_accuracy / (validation_loader),\n : competition_metric.score(),\n }\nThe current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, training notebook and inference notebook.
Do you see something I am doing wrong?
\nReading the entire pipeline would be cumbersome, so let me know if you need extra info to help.
\nEvery answer/advice is welcome.
\nThanks in advance!
CV(all): 0.867
\nCV(IMU only): 0.811
\nLB: 0.842
cv: imu only 0.8181, imu+thm+tof 0.8638
\npublic lb: 0.84
StratifiedGroupKFold, groups=subject cv Average Val Acc: 0.8194 Val Acc Std Dev: 0.0386 LB0.81
", "votes": 0, "replies": [] }, { "id": 3244925, "author_name": "Kurise", "author_url": "", "post_date": "2025-07-08T17:13:45.147000", "content": "Imu only : StratifiedGroupKFold, split by subject: cv : 0.7910 LB : 0.81
\nFull data : StratifiedKFold, split by gesture: cv : 0.83 LB : 0.8
\nEnsemble : LB : 0.82
In our split strategy, imu+tof+thm cv: 0.82~0.83, imu: 0.781, then lb is 0.82.
", "votes": 0, "replies": [ { "id": 3242910, "author_name": "Jan Brederecke", "author_url": "", "post_date": "2025-07-06T14:29:31.557000", "content": "Hey Tom, I may have missed it but can you give some details on your split strategy?
\nBest,
\nJan
CV and LB scores are correlated, but the variability between folds is concerning. We need to analyze the reasons behind the under-performing folds.
\n| fold0 | \nfold1 | \nfold2 | \nfold3 | \nfold4 | \nCV | \nLB | \nCV Strategy | \n
|---|---|---|---|---|---|---|---|
| 0.8282 | \n0.7782 | \n0.7601 | \n0.7549 | \n0.7926 | \n0.7828 | \n0.8 | \nStratifiedGroupKFold, groups=subject | \n