I think it's still too early to have this discussion, but after looking at the content of my EDA and Discussions over the past month or so, I've come to think that the possibility of a \"shakeup\" in this competition is low. The reasons are listed below.
\nHowever, concerns remain, such as the large proportion of Private LB at 75% and the adoption of unreasonable post-processing (as past history shows).
\nIf you have any opinions, please leave a comment.
I think it's still too early to have this discussion, but after looking at the content of my EDA and Discussions over the past month or so, I've come to think that the possibility of a \"shakeup\" in this competition is low. The reasons are listed below.
\nHowever, concerns remain, such as the large proportion of Private LB at 75% and the adoption of unreasonable post-processing (as past history shows).
\nIf you have any opinions, please leave a comment.
75% of private lb is really scaring, I can't find an effective way to cope with that~
", "rawMarkdown": "75% of private lb is really scaring, I can't find an effective way to cope with that~", "votes": null }, { "id": "2483114", "postDate": "10/15/2023 13:39:07", "content": "I completely agree @liushundawang. \"Shakeup\" often occur in competitions where the proportion of Private LB exceeds 50%, and there are clear reasons for this (model used, post-processing, etc).
", "rawMarkdown": "I completely agree @liushundawang. \"Shakeup\" often occur in competitions where the proportion of Private LB exceeds 50%, and there are clear reasons for this (model used, post-processing, etc).", "votes": null }, { "id": "2483121", "postDate": "10/15/2023 13:46:39", "content": "yeah, I don't really understand why this ratio as 75%, but anyway, hope someone can give a rational declaration on that.
", "rawMarkdown": "yeah, I don't really understand why this ratio as 75%, but anyway, hope someone can give a rational declaration on that.", "votes": null }, { "id": "2483132", "postDate": "10/15/2023 13:54:54", "content": "yes. I'm sure there must be a reason for the 25-75% split, but I don't remember seeing any posts mentioning it in the past.
", "rawMarkdown": "yes. I'm sure there must be a reason for the 25-75% split, but I don't remember seeing any posts mentioning it in the past.", "votes": null }, { "id": "2483624", "postDate": "10/15/2023 20:45:00", "content": "Especially due to the topic of the competition, I would say there are going to be inconsistencies and drift for sure.
", "rawMarkdown": "Especially due to the topic of the competition, I would say there are going to be inconsistencies and drift for sure.", "votes": null }, { "id": "2483671", "postDate": "10/15/2023 22:01:42", "content": "Thank you for your comment @cody11null. In past competitions, there were warnings, signs, and omens. If I find any \"shakeup\" seeds, I'll be sure to share them.
", "rawMarkdown": "Thank you for your comment @cody11null. In past competitions, there were warnings, signs, and omens. If I find any \"shakeup\" seeds, I'll be sure to share them.", "votes": null }, { "id": "2513527", "postDate": "11/05/2023 14:10:21", "content": "With about a month left until the end of the competition, public LB is becoming more and more lively…
\nAt such times, I would like to ask the question again. As for whether or not there will be a shakeup in this competition, I'm pretty sure it won't happen.
\nPlease comment if you have any concerns.
", "rawMarkdown": "With about a month left until the end of the competition, public LB is becoming more and more lively...\n\nAt such times, I would like to ask the question again. As for whether or not there will be a shakeup in this competition, I'm pretty sure it won't happen.\n\nPlease comment if you have any concerns.", "votes": null }, { "id": "2518803", "postDate": "11/09/2023 15:02:19", "content": "@mitsuyasuhoshino usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup. But if you trust your own CV based on the strategy organizers split data for train and test - you will be safe. One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.
", "rawMarkdown": "mitsuyasuhoshino usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup. But if you trust your own CV based on the strategy organizers split data for train and test - you will be safe. One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.", "votes": null }, { "id": "2519334", "postDate": "11/09/2023 23:23:12", "content": "Usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup.
\n　Agree. I think this trend is now under control.
One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.
\n　This basic strategy can be successful or unsuccessful. I have no choice but to believe in my own experience.
Time to recover this thread….is it possible a Shakeup?
", "rawMarkdown": "Time to recover this thread....is it possible a Shakeup?", "votes": null } ], "comments": [ { "id": 2483105, "author_name": "liushundawang", "author_url": "", "post_date": "10/15/2023 13:30:30", "content": "75% of private lb is really scaring, I can't find an effective way to cope with that~
", "votes": null, "replies": [ { "id": 2483114, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "10/15/2023 13:39:07", "content": "I completely agree @liushundawang. \"Shakeup\" often occur in competitions where the proportion of Private LB exceeds 50%, and there are clear reasons for this (model used, post-processing, etc).
", "votes": null, "replies": [ { "id": 2483121, "author_name": "liushundawang", "author_url": "", "post_date": "10/15/2023 13:46:39", "content": "yeah, I don't really understand why this ratio as 75%, but anyway, hope someone can give a rational declaration on that.
", "votes": null, "replies": [ { "id": 2483132, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "10/15/2023 13:54:54", "content": "yes. I'm sure there must be a reason for the 25-75% split, but I don't remember seeing any posts mentioning it in the past.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2483624, "author_name": "cody11null", "author_url": "", "post_date": "10/15/2023 20:45:00", "content": "Especially due to the topic of the competition, I would say there are going to be inconsistencies and drift for sure.
", "votes": null, "replies": [ { "id": 2483671, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "10/15/2023 22:01:42", "content": "Thank you for your comment @cody11null. In past competitions, there were warnings, signs, and omens. If I find any \"shakeup\" seeds, I'll be sure to share them.
", "votes": null, "replies": [] } ] }, { "id": 2513527, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "11/05/2023 14:10:21", "content": "With about a month left until the end of the competition, public LB is becoming more and more lively…
\nAt such times, I would like to ask the question again. As for whether or not there will be a shakeup in this competition, I'm pretty sure it won't happen.
\nPlease comment if you have any concerns.
", "votes": null, "replies": [] }, { "id": 2518803, "author_name": "alexryzhkov", "author_url": "", "post_date": "11/09/2023 15:02:19", "content": "@mitsuyasuhoshino usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup. But if you trust your own CV based on the strategy organizers split data for train and test - you will be safe. One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.
", "votes": null, "replies": [ { "id": 2519334, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "11/09/2023 23:23:12", "content": "Usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup.
\n　Agree. I think this trend is now under control.
One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.
\n　This basic strategy can be successful or unsuccessful. I have no choice but to believe in my own experience.
Time to recover this thread….is it possible a Shakeup?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2483010": "I think it's still too early to have this discussion, but after looking at the content of my EDA and Discussions over the past month or so, **I've come to think that the possibility of a \"shakeup\" in this competition is low**. The reasons are listed below.\n\n- CV-LB correlation (size relationship and difference) is almost unchanged\n- It seems unlikely that the data distribution between train and test will change, even considering the following facts:\n * [\"The series used in train / public lb / private lb are distinct\"](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437567#2427749) as mentioned by @ryanholbrook\n * I can imagine that the way to label the correct answer is the same for train and test\n * Impact of data for \"for children and youth, especially those with mood and behavior difficulties\" as stated in the Goal of the Competition\n- (In addition) Famous and proven Kagglers are participating...\n\nHowever, concerns remain, such as the large proportion of Private LB at 75% and the adoption of unreasonable post-processing (as past history shows).\nIf you have any opinions, please leave a comment.", "2483105": "75% of private lb is really scaring, I can't find an effective way to cope with that~", "2483114": "I completely agree @liushundawang. \"Shakeup\" often occur in competitions where the proportion of Private LB exceeds 50%, and there are clear reasons for this (model used, post-processing, etc).", "2483121": "yeah, I don't really understand why this ratio as 75%, but anyway, hope someone can give a rational declaration on that.", "2483132": "yes. I'm sure there must be a reason for the 25-75% split, but I don't remember seeing any posts mentioning it in the past.", "2483624": "Especially due to the topic of the competition, I would say there are going to be inconsistencies and drift for sure.", "2483671": "Thank you for your comment @cody11null. In past competitions, there were warnings, signs, and omens. If I find any \"shakeup\" seeds, I'll be sure to share them.", "2513527": "With about a month left until the end of the competition, public LB is becoming more and more lively...\n\nAt such times, I would like to ask the question again. As for whether or not there will be a shakeup in this competition, I'm pretty sure it won't happen.\n\nPlease comment if you have any concerns.", "2518803": "mitsuyasuhoshino usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup. But if you trust your own CV based on the strategy organizers split data for train and test - you will be safe. One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.", "2519334": "**Usually based on my kaggle experience if we have the private LB significantly bigger than public one and we can't even check how they could be compared to each other (are they similar or not) - it usually means shakeup.**\n\n　Agree. I think this trend is now under control.\n\n- **One more thing is that you have 2 submissions to select for finals - if you choose 1 best public LB and 1 best CV you should also be on the safe side.**\n\n　This basic strategy can be successful or unsuccessful. I have no choice but to believe in my own experience.", "2550010": "Time to recover this thread....is it possible a Shakeup?" }, "source": "meta" }