To correctly calculate the CV across 5-folds, should we use the final score for each fold and average them or use the best score for each fold and average?
", "messages": [ { "id": "3232310", "postDate": "06/25/2025 15:14:08", "content": "To correctly calculate the CV across 5-folds, should we use the final score for each fold and average them or use the best score for each fold and average?
", "rawMarkdown": "To correctly calculate the CV across 5-folds, should we use the final score for each fold and average them or use the best score for each fold and average?", "votes": null }, { "id": "3232325", "postDate": "06/25/2025 15:21:27", "content": "I usually monitor both, in this competition they are very similar anyway (due to multi classification problem). Example of mine: global oof score 0.75623 ( mean score 0.7564 +-0.0181).
", "rawMarkdown": "I usually monitor both, in this competition they are very similar anyway (due to multi classification problem). Example of mine: global oof score 0.75623 ( mean score 0.7564 +-0.0181).", "votes": null }, { "id": "3232337", "postDate": "06/25/2025 15:35:12", "content": "@georgewang73 The correct way is to generate OOF (Out Of Fold) predictions and then compute your score based on this dataframe. For example, you split your data into 5 folds, so you will have 5 validation sets. Every time your model's validation score improves you save the validation predictions. You end up with 5 validation predictions (1 for each validation set). You concatenate these 5 prediction dataframes, this gives you the OOF dataframe, which must contain the y_true and y_pred columns. Finally, you compute the score using this OOF.
Thank you so much! It is a really good advice for me.
", "rawMarkdown": "Thank you so much! It is a really good advice for me.", "votes": null }, { "id": "3234222", "postDate": "06/27/2025 16:23:31", "content": "A follow up question, are you doing the stratification on subject only? I recently did that, and my CV is 0.79, yet I submitted that to the LB, but it was only 0.69, I'm wondering what might caused this issue?
", "rawMarkdown": "A follow up question, are you doing the stratification on subject only? I recently did that, and my CV is 0.79, yet I submitted that to the LB, but it was only 0.69, I'm wondering what might caused this issue?", "votes": null }, { "id": "3234245", "postDate": "06/27/2025 16:53:51", "content": "Group on subject, stratify on gesture seems a reasonnable approach. Several possibilities are discussed here.
\nYour CV LB gap seem too high, I would double check my inference pipeline on the train set and make sure that I have the exact same results as in local CV computation.
", "rawMarkdown": "Group on subject, stratify on gesture seems a reasonnable approach. Several possibilities are discussed [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/583023).\n\nYour CV LB gap seem too high, I would double check my inference pipeline on the train set and make sure that I have the exact same results as in local CV computation.", "votes": null }, { "id": "3234255", "postDate": "06/27/2025 17:03:03", "content": "Do you think it is possible that I am submitting just 1 fold's model without retraining the model with all the data?
", "rawMarkdown": "Do you think it is possible that I am submitting just 1 fold's model without retraining the model with all the data?", "votes": null }, { "id": "3234388", "postDate": "06/27/2025 18:53:45", "content": "Yes single fold gave me 0.74 for a CV of 0.756.
", "rawMarkdown": "Yes single fold gave me 0.74 for a CV of 0.756.", "votes": null }, { "id": "3234407", "postDate": "06/27/2025 19:19:28", "content": "I think it was my method's problem. In the beginning I thought the phase information would be useful since it could tell the model to distinguish between the transition and gesture phase, since the phase information is not provided in the test, I tried to train a model to detect the boundary when transition change to gesture. The performance of that boundary model was good when I am evaluating that in the validation, so I thought I could directly use that as the input in the final model. However, it seems that the boundary model might not be performing well on the test, leading to some false classification of phases. I then removed the phase as a feature, and the score between the CV and LB become almost identical.
", "rawMarkdown": "I think it was my method's problem. In the beginning I thought the phase information would be useful since it could tell the model to distinguish between the transition and gesture phase, since the phase information is not provided in the test, I tried to train a model to detect the boundary when transition change to gesture. The performance of that boundary model was good when I am evaluating that in the validation, so I thought I could directly use that as the input in the final model. However, it seems that the boundary model might not be performing well on the test, leading to some false classification of phases. I then removed the phase as a feature, and the score between the CV and LB become almost identical.", "votes": null }, { "id": "3246088", "postDate": "07/10/2025 09:27:16", "content": "We can see that the player in the bottom right corner still has a very serious expression.😐😐
", "rawMarkdown": "We can see that the player in the bottom right corner still has a very serious expression.😐😐", "votes": null }, { "id": "3246869", "postDate": "07/11/2025 17:00:39", "content": "真正想赢的人脸上是没有笑容的
", "rawMarkdown": "真正想赢的人脸上是没有笑容的", "votes": null } ], "comments": [ { "id": 3232325, "author_name": "optimo", "author_url": "", "post_date": "06/25/2025 15:21:27", "content": "I usually monitor both, in this competition they are very similar anyway (due to multi classification problem). Example of mine: global oof score 0.75623 ( mean score 0.7564 +-0.0181).
", "votes": null, "replies": [ { "id": 3234222, "author_name": "georgewang73", "author_url": "", "post_date": "06/27/2025 16:23:31", "content": "A follow up question, are you doing the stratification on subject only? I recently did that, and my CV is 0.79, yet I submitted that to the LB, but it was only 0.69, I'm wondering what might caused this issue?
", "votes": null, "replies": [ { "id": 3234245, "author_name": "optimo", "author_url": "", "post_date": "06/27/2025 16:53:51", "content": "Group on subject, stratify on gesture seems a reasonnable approach. Several possibilities are discussed here.
\nYour CV LB gap seem too high, I would double check my inference pipeline on the train set and make sure that I have the exact same results as in local CV computation.
", "votes": null, "replies": [ { "id": 3234255, "author_name": "georgewang73", "author_url": "", "post_date": "06/27/2025 17:03:03", "content": "Do you think it is possible that I am submitting just 1 fold's model without retraining the model with all the data?
", "votes": null, "replies": [ { "id": 3234388, "author_name": "optimo", "author_url": "", "post_date": "06/27/2025 18:53:45", "content": "Yes single fold gave me 0.74 for a CV of 0.756.
", "votes": null, "replies": [ { "id": 3234407, "author_name": "georgewang73", "author_url": "", "post_date": "06/27/2025 19:19:28", "content": "I think it was my method's problem. In the beginning I thought the phase information would be useful since it could tell the model to distinguish between the transition and gesture phase, since the phase information is not provided in the test, I tried to train a model to detect the boundary when transition change to gesture. The performance of that boundary model was good when I am evaluating that in the validation, so I thought I could directly use that as the input in the final model. However, it seems that the boundary model might not be performing well on the test, leading to some false classification of phases. I then removed the phase as a feature, and the score between the CV and LB become almost identical.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3232337, "author_name": "alejopaullier", "author_url": "", "post_date": "06/25/2025 15:35:12", "content": "@georgewang73 The correct way is to generate OOF (Out Of Fold) predictions and then compute your score based on this dataframe. For example, you split your data into 5 folds, so you will have 5 validation sets. Every time your model's validation score improves you save the validation predictions. You end up with 5 validation predictions (1 for each validation set). You concatenate these 5 prediction dataframes, this gives you the OOF dataframe, which must contain the y_true and y_pred columns. Finally, you compute the score using this OOF.
Thank you so much! It is a really good advice for me.
", "votes": null, "replies": [] } ] }, { "id": 3246088, "author_name": "yehongyi", "author_url": "", "post_date": "07/10/2025 09:27:16", "content": "We can see that the player in the bottom right corner still has a very serious expression.😐😐
", "votes": null, "replies": [ { "id": 3246869, "author_name": "georgewang73", "author_url": "", "post_date": "07/11/2025 17:00:39", "content": "真正想赢的人脸上是没有笑容的
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3232310": "To correctly calculate the CV across 5-folds, should we use the final score for each fold and average them or use the best score for each fold and average?", "3232325": "I usually monitor both, in this competition they are very similar anyway (due to multi classification problem). Example of mine: global oof score 0.75623 ( mean score 0.7564 +-0.0181).", "3232337": "georgewang73 The correct way is to generate OOF (Out Of Fold) predictions and then compute your score based on this dataframe. For example, you split your data into 5 folds, so you will have 5 validation sets. Every time your model's validation score improves you save the validation predictions. You end up with 5 validation predictions (1 for each validation set). You concatenate these 5 prediction dataframes, this gives you the OOF dataframe, which must contain the `y_true` and `y_pred` columns. Finally, you compute the score using this OOF.", "3232370": "Thank you so much! It is a really good advice for me.", "3234222": "A follow up question, are you doing the stratification on subject only? I recently did that, and my CV is 0.79, yet I submitted that to the LB, but it was only 0.69, I'm wondering what might caused this issue?", "3234245": "Group on subject, stratify on gesture seems a reasonnable approach. Several possibilities are discussed [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/583023).\n\nYour CV LB gap seem too high, I would double check my inference pipeline on the train set and make sure that I have the exact same results as in local CV computation.", "3234255": "Do you think it is possible that I am submitting just 1 fold's model without retraining the model with all the data?", "3234388": "Yes single fold gave me 0.74 for a CV of 0.756.", "3234407": "I think it was my method's problem. In the beginning I thought the phase information would be useful since it could tell the model to distinguish between the transition and gesture phase, since the phase information is not provided in the test, I tried to train a model to detect the boundary when transition change to gesture. The performance of that boundary model was good when I am evaluating that in the validation, so I thought I could directly use that as the input in the final model. However, it seems that the boundary model might not be performing well on the test, leading to some false classification of phases. I then removed the phase as a feature, and the score between the CV and LB become almost identical.", "3246088": "We can see that the player in the bottom right corner still has a very serious expression.😐😐", "3246869": "真正想赢的人脸上是没有笑容的" }, "source": "meta" }