Hey guys,
\nI've been running some experiments with neural networks and consistently noticed a 0.05–0.1 gap between my CV and LB scores. Currently, my CV score is 0.80, but the LB score is around 0.70.
Is this unusual? Am I doing something wrong, or have you all experienced this as well?
\nNote: The model is trained using all IMU, THM, and ToF features.
Hey guys,
\nI've been running some experiments with neural networks and consistently noticed a 0.05–0.1 gap between my CV and LB scores. Currently, my CV score is 0.80, but the LB score is around 0.70.
Is this unusual? Am I doing something wrong, or have you all experienced this as well?
\nNote: The model is trained using all IMU, THM, and ToF features.
I had a similar issue at the start when I created my pipeline with everything. The CV-LB scores are great with IMU only, they are almost the same (within 0.5% - 1% difference). My current IMU only score is 0.82 CV, 0.82 LB @saimanojakondi
", "rawMarkdown": "I had a similar issue at the start when I created my pipeline with everything. The CV-LB scores are great with IMU only, they are almost the same (within 0.5% - 1% difference). My current IMU only score is 0.82 CV, 0.82 LB @saimanojakondi ", "replies": [ { "id": 3251010, "postDate": "2025-07-19T17:13:43.163Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3250123, "postDate": "2025-07-17T18:11:42.940Z", "content": "are you validating on a single holdout validation set or across multiple folds? if single holdout then it's normal but for multiple folds cv you need to try various strategies to align it with lb. Also the test data is messed with half of it missing tof and thm values so there would be some divergence between cv and lb scores. Also check if you are evaluating the competition metric properly, for macro-f1 put non bfrb gestures into single category 'non-target' and then calculate
", "rawMarkdown": "are you validating on a single holdout validation set or across multiple folds? if single holdout then it's normal but for multiple folds cv you need to try various strategies to align it with lb. Also the test data is messed with half of it missing tof and thm values so there would be some divergence between cv and lb scores. Also check if you are evaluating the competition metric properly, for macro-f1 put non bfrb gestures into single category 'non-target' and then calculate" }, { "id": 3250013, "postDate": "2025-07-17T14:40:48.577Z", "content": "Two causes:
\nI also met this problem in the beginning, after changing those two the gap become much smaller.
", "rawMarkdown": "Two causes:\n1. There are 50% of data that is missing for thm and tof, you need to handle that properly\n2. You calculated the wrong metrics, for binary F1 you need to collapse all the non-gesture into one class and calculate.\n\nI also met this problem in the beginning, after changing those two the gap become much smaller." }, { "id": 3249580, "postDate": "2025-07-16T17:52:46.213Z", "content": "Please try ensembling the predictions from IMU-only data and full data. This is because 50% of the test data consists of IMU-only data.
", "rawMarkdown": "Please try ensembling the predictions from IMU-only data and full data. This is because 50% of the test data consists of IMU-only data." }, { "id": 3249567, "postDate": "2025-07-16T17:27:44.743Z", "content": "A CV score between 0.01 and -0.015 seems acceptable due to noise in the hidden test data. However, a score around 0.1 indicates a potential issue. Additionally, around 50% of the thm_ and tof_ data is missing, so please review your pipeline and recalculate your CV score according to the competition's metric.
", "rawMarkdown": "A CV score between 0.01 and -0.015 seems acceptable due to noise in the hidden test data. However, a score around 0.1 indicates a potential issue. Additionally, around 50% of the thm_ and tof_ data is missing, so please review your pipeline and recalculate your CV score according to the competition's metric." } ], "comments": [ { "id": 3250808, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-07-19T07:35:42.030000", "content": "I had a similar issue at the start when I created my pipeline with everything. The CV-LB scores are great with IMU only, they are almost the same (within 0.5% - 1% difference). My current IMU only score is 0.82 CV, 0.82 LB @saimanojakondi
", "votes": 0, "replies": [ { "id": 3251010, "author_name": "", "author_url": "", "post_date": "2025-07-19T17:13:43.163000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3250123, "author_name": "Baseer Shah", "author_url": "", "post_date": "2025-07-17T18:11:42.940000", "content": "are you validating on a single holdout validation set or across multiple folds? if single holdout then it's normal but for multiple folds cv you need to try various strategies to align it with lb. Also the test data is messed with half of it missing tof and thm values so there would be some divergence between cv and lb scores. Also check if you are evaluating the competition metric properly, for macro-f1 put non bfrb gestures into single category 'non-target' and then calculate
", "votes": 0, "replies": [] }, { "id": 3250013, "author_name": "menglei", "author_url": "", "post_date": "2025-07-17T14:40:48.577000", "content": "Two causes:
\nI also met this problem in the beginning, after changing those two the gap become much smaller.
", "votes": 0, "replies": [] }, { "id": 3249580, "author_name": "h.matsumoto", "author_url": "", "post_date": "2025-07-16T17:52:46.213000", "content": "Please try ensembling the predictions from IMU-only data and full data. This is because 50% of the test data consists of IMU-only data.
", "votes": 0, "replies": [] }, { "id": 3249567, "author_name": "Sangram Patil", "author_url": "", "post_date": "2025-07-16T17:27:44.743000", "content": "A CV score between 0.01 and -0.015 seems acceptable due to noise in the hidden test data. However, a score around 0.1 indicates a potential issue. Additionally, around 50% of the thm_ and tof_ data is missing, so please review your pipeline and recalculate your CV score according to the competition's metric.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3249564": "Hey guys,\nI've been running some experiments with neural networks and consistently noticed a 0.05–0.1 gap between my CV and LB scores. Currently, my CV score is 0.80, but the LB score is around 0.70.\n\nIs this unusual? Am I doing something wrong, or have you all experienced this as well?\nNote: The model is trained using all IMU, THM, and ToF features.", "3250808": "I had a similar issue at the start when I created my pipeline with everything. The CV-LB scores are great with IMU only, they are almost the same (within 0.5% - 1% difference). My current IMU only score is 0.82 CV, 0.82 LB @saimanojakondi ", "3250123": "are you validating on a single holdout validation set or across multiple folds? if single holdout then it's normal but for multiple folds cv you need to try various strategies to align it with lb. Also the test data is messed with half of it missing tof and thm values so there would be some divergence between cv and lb scores. Also check if you are evaluating the competition metric properly, for macro-f1 put non bfrb gestures into single category 'non-target' and then calculate", "3250013": "Two causes:\n1. There are 50% of data that is missing for thm and tof, you need to handle that properly\n2. You calculated the wrong metrics, for binary F1 you need to collapse all the non-gesture into one class and calculate.\n\nI also met this problem in the beginning, after changing those two the gap become much smaller.", "3249580": "Please try ensembling the predictions from IMU-only data and full data. This is because 50% of the test data consists of IMU-only data.", "3249567": "A CV score between 0.01 and -0.015 seems acceptable due to noise in the hidden test data. However, a score around 0.1 indicates a potential issue. Additionally, around 50% of the thm_ and tof_ data is missing, so please review your pipeline and recalculate your CV score according to the competition's metric." } }