I've been struggling with a significant CV-LB gap in this competition. I'm wondering if the primary cause is the limited size of the public test set or the overfitting caused by the high dimensionality of the features.
\nHas anyone else encountered this issue?
", "messages": [ { "id": "3000908", "postDate": "09/28/2024 08:31:26", "content": "I've been struggling with a significant CV-LB gap in this competition. I'm wondering if the primary cause is the limited size of the public test set or the overfitting caused by the high dimensionality of the features.
\nHas anyone else encountered this issue?
", "rawMarkdown": "I've been struggling with a significant CV-LB gap in this competition. I'm wondering if the primary cause is the limited size of the public test set or the overfitting caused by the high dimensionality of the features.\n\nHas anyone else encountered this issue?", "votes": null }, { "id": "3001541", "postDate": "09/29/2024 00:45:52", "content": "\n\nThe training set takes samples from the first three months of study data from nine of the participants and includes the future blood glucose value. These training samples appear in chronological order and overlap. The testing set takes samples from the remainder of the study period from fifteen of the participants (so unseen participants appear in the testing set). These testing samples do not overlap and are in a random order to avoid data leakage.
\n
IMO,Firstly, as there are new participants in the test data, I chose to use groupkfold for cross validation based on the participants. As you can see, there are significant differences in RMSE for different folds, so it is acceptable for there to be differences between the public ranking and CV. Secondly, there is overlap in the training data. For example, I predict the 7th hour using 1-6 hours and the 8th hour using 2-7 hours, while the test data does not overlap to prevent label leakage. Thirdly, the amount of data is too small. I think we should check whether CV and LB increase or decrease simultaneously.
", "rawMarkdown": ">The training set takes samples from the first three months of study data from nine of the participants and includes the future blood glucose value. These training samples appear in chronological order and overlap. The testing set takes samples from the remainder of the study period from fifteen of the participants (so unseen participants appear in the testing set). These testing samples do not overlap and are in a random order to avoid data leakage.\n\nIMO,Firstly, as there are new participants in the test data, I chose to use groupkfold for cross validation based on the participants. As you can see, there are significant differences in RMSE for different folds, so it is acceptable for there to be differences between the public ranking and CV. Secondly, there is overlap in the training data. For example, I predict the 7th hour using 1-6 hours and the 8th hour using 2-7 hours, while the test data does not overlap to prevent label leakage. Thirdly, the amount of data is too small. I think we should check whether CV and LB increase or decrease simultaneously.", "votes": null }, { "id": "3002449", "postDate": "09/30/2024 04:23:09", "content": "Just FYI: It seems like sklearn experiments with the implementation of StratifiedGroupByKfold: https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html
", "rawMarkdown": "Just FYI: It seems like sklearn experiments with the implementation of StratifiedGroupByKfold: https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html", "votes": null } ], "comments": [ { "id": 3001541, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "09/29/2024 00:45:52", "content": "\n\nThe training set takes samples from the first three months of study data from nine of the participants and includes the future blood glucose value. These training samples appear in chronological order and overlap. The testing set takes samples from the remainder of the study period from fifteen of the participants (so unseen participants appear in the testing set). These testing samples do not overlap and are in a random order to avoid data leakage.
\n
IMO,Firstly, as there are new participants in the test data, I chose to use groupkfold for cross validation based on the participants. As you can see, there are significant differences in RMSE for different folds, so it is acceptable for there to be differences between the public ranking and CV. Secondly, there is overlap in the training data. For example, I predict the 7th hour using 1-6 hours and the 8th hour using 2-7 hours, while the test data does not overlap to prevent label leakage. Thirdly, the amount of data is too small. I think we should check whether CV and LB increase or decrease simultaneously.
", "votes": null, "replies": [ { "id": 3002449, "author_name": "thomasmeiner", "author_url": "", "post_date": "09/30/2024 04:23:09", "content": "Just FYI: It seems like sklearn experiments with the implementation of StratifiedGroupByKfold: https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3000908": "I've been struggling with a significant CV-LB gap in this competition. I'm wondering if the primary cause is the limited size of the public test set or the overfitting caused by the high dimensionality of the features.\n\nHas anyone else encountered this issue?", "3001541": ">The training set takes samples from the first three months of study data from nine of the participants and includes the future blood glucose value. These training samples appear in chronological order and overlap. The testing set takes samples from the remainder of the study period from fifteen of the participants (so unseen participants appear in the testing set). These testing samples do not overlap and are in a random order to avoid data leakage.\n\nIMO,Firstly, as there are new participants in the test data, I chose to use groupkfold for cross validation based on the participants. As you can see, there are significant differences in RMSE for different folds, so it is acceptable for there to be differences between the public ranking and CV. Secondly, there is overlap in the training data. For example, I predict the 7th hour using 1-6 hours and the 8th hour using 2-7 hours, while the test data does not overlap to prevent label leakage. Thirdly, the amount of data is too small. I think we should check whether CV and LB increase or decrease simultaneously.", "3002449": "Just FYI: It seems like sklearn experiments with the implementation of StratifiedGroupByKfold: https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html" }, "source": "meta" }