\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F190923%2Fbceeacae2b472f985ce9892ae56f538d%2FScreenshot%20from%202025-07-11%2022-23-20.png?generation=1752265423445510&alt=media\")
I had a very strong correlation until I reached 0.85 LB, after that any improvement would lower my LB.
\nMethodology:
\nHere is what I got so far:
\n
What about you ?
\nPlease share your methodology to compute CV as well otherwise comparisons are useless!
I had a very strong correlation until I reached 0.85 LB, after that any improvement would lower my LB.
\nMethodology:
\nHere is what I got so far:
\n
What about you ?
\nPlease share your methodology to compute CV as well otherwise comparisons are useless!
My cv strategy is similar to yours, whose only difference is that I just set a half of TOF validation data as zero (to simulate the situation where there are a half of imu-only data in lb).
", "rawMarkdown": "My cv strategy is similar to yours, whose only difference is that I just set a half of TOF validation data as zero (to simulate the situation where there are a half of imu-only data in lb).", "votes": 3, "replies": [ { "id": 3248460, "postDate": "2025-07-14T16:27:16.100Z", "content": "how do you pick which half to set to 0 ?
", "rawMarkdown": "how do you pick which half to set to 0 ?", "replies": [ { "id": 3248465, "postDate": "2025-07-14T16:33:13.913Z", "content": "I just set them in the dataloader iteration randomly, which is like:
\nfor y in val_dl:\n batch_size/\n x_front [:] \n x_back [:].clone() \n x_back[: : :] \n torch.cat([x_front x_back] dim)\n\n ......\nThis strategy shows great correlation for now.
", "rawMarkdown": "I just set them in the dataloader iteration randomly, which is like:\n```\nfor x, y in val_dl:\n half = batch_size/2\n x_front = x[:half] \n x_back = x[half:].clone() \n x_back[:, :, 14:] = 0.0 \n x = torch.cat([x_front, x_back], dim=0)\n\n ......\n```\nThis strategy shows great correlation for now.", "votes": 1 }, { "id": 3248479, "postDate": "2025-07-14T17:04:08.167Z", "content": "So you get some randomness in the computation of your CV? The inference is fast it's not costly to do inference twice no ?
", "rawMarkdown": "So you get some randomness in the computation of your CV? The inference is fast it's not costly to do inference twice no ?" }, { "id": 3248489, "postDate": "2025-07-14T17:19:30.463Z", "content": "I set shuffle as False so the order stays the same in each validation.
", "rawMarkdown": "I set shuffle as False so the order stays the same in each validation." }, { "id": 3248811, "postDate": "2025-07-15T08:09:33.787Z", "content": "Hello, are you guys using different models for IMU only and for ToF/THM?
", "rawMarkdown": "Hello, are you guys using different models for IMU only and for ToF/THM?" }, { "id": 3250811, "postDate": "2025-07-19T07:42:37.590Z", "content": "I tried both! No significant differences in terms of CV/LB alignement.
", "rawMarkdown": "I tried both! No significant differences in terms of CV/LB alignement." } ] }, { "id": 3252109, "postDate": "2025-07-22T04:12:31.500Z", "content": "---------------- UPDATE ----------------
\nAfter reaching 0.84，I also began to lose the relevance between cv and lb……
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16529018%2F36f9f46e96bafe040ac14edbcdea5eb5%2Favg-lb.png?generation=1753157596953029&alt=media\")
There seems an obstacle to observe correlated cv after reaching 0.84. I got same result to yours.
", "rawMarkdown": "There seems an obstacle to observe correlated cv after reaching 0.84. I got same result to yours. ", "votes": 1, "replies": [ { "id": 3250445, "postDate": "2025-07-18T12:05:38.823Z", "content": "Would you mind sharing a few CV LB results where things got tricky ?
", "rawMarkdown": "Would you mind sharing a few CV LB results where things got tricky ?", "replies": [ { "id": 3250786, "postDate": "2025-07-19T06:49:12.967Z", "content": "I can share mine:
\nCV: 0.842
\nLB: 0.83 (my best, 0.85, is from merging a few models)
\nI think that above 0.84, IMU plays a significant role. Simulating the test set is a game of luck, since some sensors are missing even in the remaining 50% of the \"full\" dataset.
cv: 0.876, lb: 0.84~0.85 (different hyperparameters). Under cv=0.84, cv-lb is very consistent.
", "rawMarkdown": "cv: 0.876, lb: 0.84~0.85 (different hyperparameters). Under cv=0.84, cv-lb is very consistent. ", "votes": 1 }, { "id": 3251706, "postDate": "2025-07-21T12:41:37.640Z", "content": "@tom99763 can you share how you compute your CV please ? Is that all features only? or all features + IMU only averaging ?
", "rawMarkdown": "@tom99763 can you share how you compute your CV please ? Is that all features only? or all features + IMU only averaging ?" }, { "id": 3252151, "postDate": "2025-07-22T06:08:42.353Z", "content": "Hi, is the result cv 0.876 using both imu and tof and mask 50% of the validation?
", "rawMarkdown": "Hi, is the result cv 0.876 using both imu and tof and mask 50% of the validation?" } ] } ] }, { "id": 3266098, "postDate": "2025-08-08T11:59:37.183Z", "content": "LB score 0.852
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
\nCV
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
Actually i dont observe much correlation above 0.80. Kind of frustrating
", "rawMarkdown": "Actually i dont observe much correlation above 0.80. Kind of frustrating ", "votes": 1 }, { "id": 3249196, "postDate": "2025-07-16T00:47:45.903Z", "content": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.
", "rawMarkdown": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.", "votes": -15 }, { "id": 3248258, "postDate": "2025-07-14T08:06:53.767Z", "content": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.
", "rawMarkdown": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.", "votes": -10 }, { "id": 3254690, "postDate": "2025-07-27T04:11:10.977Z", "content": "I think there is something that might confuse us, maybe some people used the comp metric for calculating cv while others use f1 macro for all gestures
", "rawMarkdown": "I think there is something that might confuse us, maybe some people used the comp metric for calculating cv while others use f1 macro for all gestures" }, { "id": 3254495, "postDate": "2025-07-26T16:18:06.917Z", "content": "@brunoarpa do you have some CV/LB bad correlation to share?
\nI have now seen avg CV 0.88 -> LB 0.84 …
No, nothing specific to handedness on my side… Can you share your CV ?
", "rawMarkdown": "No, nothing specific to handedness on my side... Can you share your CV ?" }, { "id": 3254546, "postDate": "2025-07-26T18:05:21.277Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3266019, "postDate": "2025-08-08T08:47:37.110Z", "content": "single model 877!
", "rawMarkdown": "single model 877!" }, { "id": 3266030, "postDate": "2025-08-08T09:15:43.327Z", "content": "Is that your CV? 87.7 CV 86.4 LB ?
", "rawMarkdown": "Is that your CV? 87.7 CV 86.4 LB ?" }, { "id": 3266037, "postDate": "2025-08-08T09:27:10.630Z", "content": "I meant the current top1 is a single model which achieved 0.877. Mine much worse. It seems to be extremely hard to improve.
", "rawMarkdown": "I meant the current top1 is a single model which achieved 0.877. Mine much worse. It seems to be extremely hard to improve.", "votes": 1 }, { "id": 3266091, "postDate": "2025-08-08T11:46:16.760Z", "content": "How do you know is a single model?
", "rawMarkdown": "How do you know is a single model?" } ] }, { "id": 3254553, "postDate": "2025-07-26T18:18:27.997Z", "content": "CV 0.88 is way too good, something is off for you
", "rawMarkdown": "CV 0.88 is way too good, something is off for you", "replies": [ { "id": 3266008, "postDate": "2025-08-08T08:25:26.077Z", "content": "@philippsinger is 0.88 CV still impossible ? :)
", "rawMarkdown": "@philippsinger is 0.88 CV still impossible ? :)", "votes": 2 } ] }, { "id": 3254554, "postDate": "2025-07-26T18:20:29.910Z", "content": "avg cv 0.88? you must be doing mad stuff 😅
", "rawMarkdown": "avg cv 0.88? you must be doing mad stuff 😅", "replies": [ { "id": 3254566, "postDate": "2025-07-26T18:42:15.307Z", "content": "@philippsinger @nlztrk I am trying to understand how I could have leaked some information across folds for the past two days but I really don't see anything suspicious. Of course since the LB is \"bad\" the only explanation is leakage…
\nI was curious to know whether 0.88 is reachable by others but you guys made it quite clear thanks!
I also feel like I lost the correlation between CV-LB. I was between 0.825x-0.832x CV and was getting 0.84x LB. Now I'm consistently hitting 0.836+ CV and consistently getting 0.83x LB. 😅
", "rawMarkdown": "I also feel like I lost the correlation between CV-LB. I was between 0.825x-0.832x CV and was getting 0.84x LB. Now I'm consistently hitting 0.836+ CV and consistently getting 0.83x LB. 😅", "votes": 1 } ] } ] }, { "id": 3247980, "postDate": "2025-07-13T18:37:22.017Z", "content": "Did you find correlation with LB better with y=sequence_type as compared to gesture?
I started with it and things were doing ok so I sticked with it. I don't think it changes much since every subject performed all the gestures, so this will only changes which subject belongs to which fold, no real difference I think. I might be wrong though!
", "rawMarkdown": "I started with it and things were doing ok so I sticked with it. I don't think it changes much since every subject performed all the gestures, so this will only changes which subject belongs to which fold, no real difference I think. I might be wrong though!", "votes": 1, "replies": [ { "id": 3248311, "postDate": "2025-07-14T10:43:11.833Z", "content": "Yea you are correct. Each subjects has 18 gesture
", "rawMarkdown": "Yea you are correct. Each subjects has 18 gesture " } ] } ] }, { "id": 3246955, "postDate": "2025-07-11T20:53:39.887Z", "content": "In the beginning, I calculated the wrong macro F1 (I did not collapse all the non-target gesture into one class), and the macro F1 is 0.05 higher than LB, after I fixed that they become consistent. I used the average of logits from 5 folds and they make the LB score more stable.
", "rawMarkdown": "In the beginning, I calculated the wrong macro F1 (I did not collapse all the non-target gesture into one class), and the macro F1 is 0.05 higher than LB, after I fixed that they become consistent. I used the average of logits from 5 folds and they make the LB score more stable." }, { "id": 3247111, "postDate": "2025-07-12T06:22:42.077Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3247020, "postDate": "2025-07-12T00:52:57.110Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 3248334, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-07-14T11:28:17.650000", "content": "My cv strategy is similar to yours, whose only difference is that I just set a half of TOF validation data as zero (to simulate the situation where there are a half of imu-only data in lb).
", "votes": 3, "replies": [ { "id": 3248460, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-14T16:27:16.100000", "content": "how do you pick which half to set to 0 ?
", "votes": 0, "replies": [ { "id": 3248465, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-07-14T16:33:13.913000", "content": "I just set them in the dataloader iteration randomly, which is like:
\nfor y in val_dl:\n batch_size/\n x_front [:] \n x_back [:].clone() \n x_back[: : :] \n torch.cat([x_front x_back] dim)\n\n ......\nThis strategy shows great correlation for now.
", "votes": 1, "replies": [] }, { "id": 3248479, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-14T17:04:08.167000", "content": "So you get some randomness in the computation of your CV? The inference is fast it's not costly to do inference twice no ?
", "votes": 0, "replies": [] }, { "id": 3248489, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-07-14T17:19:30.463000", "content": "I set shuffle as False so the order stays the same in each validation.
", "votes": 0, "replies": [] }, { "id": 3248811, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-07-15T08:09:33.787000", "content": "Hello, are you guys using different models for IMU only and for ToF/THM?
", "votes": 0, "replies": [] }, { "id": 3250811, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-19T07:42:37.590000", "content": "I tried both! No significant differences in terms of CV/LB alignement.
", "votes": 0, "replies": [] } ] }, { "id": 3252109, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-07-22T04:12:31.500000", "content": "---------------- UPDATE ----------------
\nAfter reaching 0.84，I also began to lose the relevance between cv and lb……
\n
There seems an obstacle to observe correlated cv after reaching 0.84. I got same result to yours.
", "votes": 1, "replies": [ { "id": 3250445, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-18T12:05:38.823000", "content": "Would you mind sharing a few CV LB results where things got tricky ?
", "votes": 0, "replies": [ { "id": 3250786, "author_name": "EliKal", "author_url": "", "post_date": "2025-07-19T06:49:12.967000", "content": "I can share mine:
\nCV: 0.842
\nLB: 0.83 (my best, 0.85, is from merging a few models)
\nI think that above 0.84, IMU plays a significant role. Simulating the test set is a game of luck, since some sensors are missing even in the remaining 50% of the \"full\" dataset.
cv: 0.876, lb: 0.84~0.85 (different hyperparameters). Under cv=0.84, cv-lb is very consistent.
", "votes": 1, "replies": [] }, { "id": 3251706, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-21T12:41:37.640000", "content": "@tom99763 can you share how you compute your CV please ? Is that all features only? or all features + IMU only averaging ?
", "votes": 0, "replies": [] }, { "id": 3252151, "author_name": "Wenxuan Ye", "author_url": "", "post_date": "2025-07-22T06:08:42.353000", "content": "Hi, is the result cv 0.876 using both imu and tof and mask 50% of the validation?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3266098, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-08T11:59:37.183000", "content": "LB score 0.852
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
\nCV
\nIMU only: 0.815
\nIMU + TOF + THM: 0.89
Actually i dont observe much correlation above 0.80. Kind of frustrating
", "votes": 1, "replies": [] }, { "id": 3249196, "author_name": "Mehwish", "author_url": "", "post_date": "2025-07-16T00:47:45.903000", "content": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.
", "votes": -15, "replies": [] }, { "id": 3248258, "author_name": "Sangram Patil", "author_url": "", "post_date": "2025-07-14T08:06:53.767000", "content": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.
", "votes": -10, "replies": [] }, { "id": 3254690, "author_name": "Taha_Alshatiri", "author_url": "", "post_date": "2025-07-27T04:11:10.977000", "content": "I think there is something that might confuse us, maybe some people used the comp metric for calculating cv while others use f1 macro for all gestures
", "votes": 0, "replies": [] }, { "id": 3254495, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-26T16:18:06.917000", "content": "@brunoarpa do you have some CV/LB bad correlation to share?
\nI have now seen avg CV 0.88 -> LB 0.84 …
No, nothing specific to handedness on my side… Can you share your CV ?
", "votes": 0, "replies": [] }, { "id": 3254546, "author_name": "", "author_url": "", "post_date": "2025-07-26T18:05:21.277000", "content": "", "votes": 0, "replies": [] }, { "id": 3266019, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-08-08T08:47:37.110000", "content": "single model 877!
", "votes": 0, "replies": [] }, { "id": 3266030, "author_name": "Optimo", "author_url": "", "post_date": "2025-08-08T09:15:43.327000", "content": "Is that your CV? 87.7 CV 86.4 LB ?
", "votes": 0, "replies": [] }, { "id": 3266037, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-08-08T09:27:10.630000", "content": "I meant the current top1 is a single model which achieved 0.877. Mine much worse. It seems to be extremely hard to improve.
", "votes": 1, "replies": [] }, { "id": 3266091, "author_name": "Fnoa", "author_url": "", "post_date": "2025-08-08T11:46:16.760000", "content": "How do you know is a single model?
", "votes": 0, "replies": [] } ] }, { "id": 3254553, "author_name": "Psi", "author_url": "", "post_date": "2025-07-26T18:18:27.997000", "content": "CV 0.88 is way too good, something is off for you
", "votes": 0, "replies": [ { "id": 3266008, "author_name": "Optimo", "author_url": "", "post_date": "2025-08-08T08:25:26.077000", "content": "@philippsinger is 0.88 CV still impossible ? :)
", "votes": 2, "replies": [] } ] }, { "id": 3254554, "author_name": "Anil Ozturk", "author_url": "", "post_date": "2025-07-26T18:20:29.910000", "content": "avg cv 0.88? you must be doing mad stuff 😅
", "votes": 0, "replies": [ { "id": 3254566, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-26T18:42:15.307000", "content": "@philippsinger @nlztrk I am trying to understand how I could have leaked some information across folds for the past two days but I really don't see anything suspicious. Of course since the LB is \"bad\" the only explanation is leakage…
\nI was curious to know whether 0.88 is reachable by others but you guys made it quite clear thanks!
I also feel like I lost the correlation between CV-LB. I was between 0.825x-0.832x CV and was getting 0.84x LB. Now I'm consistently hitting 0.836+ CV and consistently getting 0.83x LB. 😅
", "votes": 1, "replies": [] } ] } ] }, { "id": 3247980, "author_name": "Gerwyn", "author_url": "", "post_date": "2025-07-13T18:37:22.017000", "content": "Did you find correlation with LB better with y=sequence_type as compared to gesture?
I started with it and things were doing ok so I sticked with it. I don't think it changes much since every subject performed all the gestures, so this will only changes which subject belongs to which fold, no real difference I think. I might be wrong though!
", "votes": 1, "replies": [ { "id": 3248311, "author_name": "Gerwyn", "author_url": "", "post_date": "2025-07-14T10:43:11.833000", "content": "Yea you are correct. Each subjects has 18 gesture
", "votes": 0, "replies": [] } ] } ] }, { "id": 3246955, "author_name": "menglei", "author_url": "", "post_date": "2025-07-11T20:53:39.887000", "content": "In the beginning, I calculated the wrong macro F1 (I did not collapse all the non-target gesture into one class), and the macro F1 is 0.05 higher than LB, after I fixed that they become consistent. I used the average of logits from 5 folds and they make the LB score more stable.
", "votes": 0, "replies": [] }, { "id": 3247111, "author_name": "", "author_url": "", "post_date": "2025-07-12T06:22:42.077000", "content": "", "votes": 0, "replies": [] }, { "id": 3247020, "author_name": "", "author_url": "", "post_date": "2025-07-12T00:52:57.110000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "3246947": "I had a very strong correlation until I reached 0.85 LB, after that any improvement would lower my LB.\n\nMethodology: \n- In order to compute CV I always compute my OOF score once with all data and once with IMU data only (setting other variables to Nan beofre entering my pipeline). I consider the average of both to be my CV score. I am using 5 fold group stratified cv with group=\"subject\" and y=sequence_type.\n- In order to differentiate between same LB scores I use either the approximate ranking in LB at the time of my submission or the relative order of my ties (ranking my submissions by public score).\n\nHere is what I got so far:\n\n\nWhat about you ?\nPlease share your methodology to compute CV as well otherwise comparisons are useless!", "3248334": "My cv strategy is similar to yours, whose only difference is that I just set a half of TOF validation data as zero (to simulate the situation where there are a half of imu-only data in lb).", "3250411": "There seems an obstacle to observe correlated cv after reaching 0.84. I got same result to yours. ", "3266098": "LB score 0.852\nIMU only: 0.815\nIMU + TOF + THM: 0.89\nCV\nIMU only: 0.815\nIMU + TOF + THM: 0.89\n", "3248312": "Actually i dont observe much correlation above 0.80. Kind of frustrating ", "3249196": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.", "3248258": "Machine learning really requires patience. If I were in your place and saw that the first fold had a score lower than 0.76, I would stop training right away and change something to improve the cross-validation score.", "3254690": "I think there is something that might confuse us, maybe some people used the comp metric for calculating cv while others use f1 macro for all gestures", "3254495": " @brunoarpa do you have some CV/LB bad correlation to share?\n\nI have now seen avg CV 0.88 -> LB 0.84 ...", "3247980": "Did you find correlation with LB better with `y=sequence_type` as compared to `gesture`?", "3246955": "In the beginning, I calculated the wrong macro F1 (I did not collapse all the non-target gesture into one class), and the macro F1 is 0.05 higher than LB, after I fixed that they become consistent. I used the average of logits from 5 folds and they make the LB score more stable.", "3247111": "", "3247020": "" } }