People overfitted really bad on the ICR dataset and the result was a massive shakeup. Let's all remember to not do that same mistake again 🥲
", "messages": [ { "id": "2457738", "postDate": "09/27/2023 06:21:48", "content": "People overfitted really bad on the ICR dataset and the result was a massive shakeup. Let's all remember to not do that same mistake again 🥲
", "rawMarkdown": "People overfitted really bad on the ICR dataset and the result was a massive shakeup. Let's all remember to not do that same mistake again 🥲", "votes": null }, { "id": "2457927", "postDate": "09/27/2023 08:36:59", "content": "People overfitted really bad on the ICR dataset because the dataset was so small, while I agree that people shouldn't blindly copy public notebooks I think that it's very unlikely that the same level of shakeup could happen in this competition.
", "rawMarkdown": "People overfitted really bad on the ICR dataset because the dataset was so small, while I agree that people shouldn't blindly copy public notebooks I think that it's very unlikely that the same level of shakeup could happen in this competition.", "votes": null }, { "id": "2457963", "postDate": "09/27/2023 08:55:15", "content": "I certainly hope so. But a lot of the public notebooks are using the dataset by carl and it has very few series. Moreover, other way to do it i to sample 1 million points which also limits data variance. Not sure if this is gonna affect the final result
", "rawMarkdown": "I certainly hope so. But a lot of the public notebooks are using the dataset by carl and it has very few series. Moreover, other way to do it i to sample 1 million points which also limits data variance. Not sure if this is gonna affect the final result", "votes": null }, { "id": "2457972", "postDate": "09/27/2023 09:04:44", "content": "Yes, that's true about carl dataset but since the public leaderboard is using 50 out of 200 series to test, the score must be somewhat reliable. @abhranta
", "rawMarkdown": "Yes, that's true about carl dataset but since the public leaderboard is using 50 out of 200 series to test, the score must be somewhat reliable. @abhranta", "votes": null }, { "id": "2457975", "postDate": "09/27/2023 09:08:00", "content": "The score is good on the 50 series ID… but it may take a major fall with the other 150. I really hope not though.
", "rawMarkdown": "The score is good on the 50 series ID... but it may take a major fall with the other 150. I really hope not though.", "votes": null }, { "id": "2458006", "postDate": "09/27/2023 09:24:36", "content": "Hi, Base on the size of this train set, I feel like it is hard to perform a lot of features engineering on the entire train set, and train it. Do you have any other recommendations of approaches that doesn't limit data variance and have a lot of series? Thanks in advance.
", "rawMarkdown": "Hi, Base on the size of this train set, I feel like it is hard to perform a lot of features engineering on the entire train set, and train it. Do you have any other recommendations of approaches that doesn't limit data variance and have a lot of series? Thanks in advance.", "votes": null }, { "id": "2458105", "postDate": "09/27/2023 10:52:45", "content": "It is my understanding that the public code using limited series IDs is only a temporary measure to avoid out-of-memory errors. The train data and test data have different series IDs, so many series IDs should be used for training the final submission.
", "rawMarkdown": "It is my understanding that the public code using limited series IDs is only a temporary measure to avoid out-of-memory errors. The train data and test data have different series IDs, so many series IDs should be used for training the final submission.", "votes": null }, { "id": "2458198", "postDate": "09/27/2023 12:19:53", "content": "ICR was a unique challenge with a small dataset and a difficult-to-administer CV scheme. I think this one is a bit different where one could construct a CV scheme, build models and test across a good CV structure.
\nYour thought of not following public work blindly is good. One needs to invest time and efforts and curate his/ her own solutions to be able to score and fare well in such competitions @abhranta
No no I meant for people who are just following the public notebooks. These notebooks generally use the great but small dataset by Carl. This might lead to a tremendous fall in private LB.
", "rawMarkdown": "No no I meant for people who are just following the public notebooks. These notebooks generally use the great but small dataset by Carl. This might lead to a tremendous fall in private LB.", "votes": null }, { "id": "2458413", "postDate": "09/27/2023 15:19:19", "content": "Dear @abhranta
\n\n\n\"never blindly follow public notebooks\"
\n
Oh, that should be a banner shown in bold font at the top of the Code section of every Kaggle competition!
\nIn general the magnitude of overfitting (hence shakeup) is inversely proportional to the amount of data we have to work with, and in this competition as we are all aware there is a huge amount of data (hence the creation of the smaller dataset). However, that said we are asked to predict specific events within that data, of which there are on average 35 per series_id, so the Public LB (≈50 series) is based on matching only ≈1730 events, so I would not completely rule out some shakeup. Indeed something that has caught my eye is the the current HSNotD has a Public LB score of 0.418, but curiously the script itself indicates it has a validation score of 0.266 (?!), so as you wisely say never blindly follow public notebooks!
All the best,
\ncarl
@carlmcbrideellis HSNotD is my new favourite Kaggle lingo now. Yeah these are using the 5 min samples now. But there is a lot more that is going to be done probably to address the overfitting and increase the score.
\nLooking forward to more great work from you.
\nAll the best,
\nRegards,
\nAbhranta
\n\n\"…my new favourite Kaggle lingo now\"
\n
LOL: Here is another one: Forkmitting:
\nThe act of simply forking and directly submitting the HSNotD making no changes whatsoever and leaping to the top of the LB, and all in less time than it takes to make a nice cup of tea!
\nAll the best and good luck!
\ncarl
ICR was just a badly designed competition. No matter if you followed public notebooks, or designed your own no-postprocessing CV based model/ensemble, there was a very high chance to face a 1K positions shift up or down depending on your luckiness.
", "rawMarkdown": "ICR was just a badly designed competition. No matter if you followed public notebooks, or designed your own no-postprocessing CV based model/ensemble, there was a very high chance to face a 1K positions shift up or down depending on your luckiness.", "votes": null }, { "id": "2459900", "postDate": "09/28/2023 12:59:47", "content": "Oh yes, forkmitting. You just know a new high-scoring notebook has been made public when you drop multiple places within an hour.😁
", "rawMarkdown": "Oh yes, forkmitting. You just know a new high-scoring notebook has been made public when you drop multiple places within an hour.😁", "votes": null }, { "id": "2460137", "postDate": "09/28/2023 15:34:48", "content": "Desperate, on the edge of being surpassed by the HSNotD. Hopefully, I don't wake up tomorrow to find myself out of the medal zone because of the forkmitting. 🥲
", "rawMarkdown": "Desperate, on the edge of being surpassed by the HSNotD. Hopefully, I don't wake up tomorrow to find myself out of the medal zone because of the forkmitting. 🥲", "votes": null }, { "id": "2462258", "postDate": "09/30/2023 08:37:26", "content": "no no no no icr is the bestest competition ever
", "rawMarkdown": "no no no no icr is the bestest competition ever", "votes": null }, { "id": "2465603", "postDate": "10/03/2023 06:03:28", "content": "@celiker Congrats to your impressive 3260 jump and a solo gold!
", "rawMarkdown": "celiker Congrats to your impressive 3260 jump and a solo gold!", "votes": null } ], "comments": [ { "id": 2457927, "author_name": "annafabris", "author_url": "", "post_date": "09/27/2023 08:36:59", "content": "People overfitted really bad on the ICR dataset because the dataset was so small, while I agree that people shouldn't blindly copy public notebooks I think that it's very unlikely that the same level of shakeup could happen in this competition.
", "votes": null, "replies": [ { "id": 2457963, "author_name": "abhranta", "author_url": "", "post_date": "09/27/2023 08:55:15", "content": "I certainly hope so. But a lot of the public notebooks are using the dataset by carl and it has very few series. Moreover, other way to do it i to sample 1 million points which also limits data variance. Not sure if this is gonna affect the final result
", "votes": null, "replies": [ { "id": 2457972, "author_name": "annafabris", "author_url": "", "post_date": "09/27/2023 09:04:44", "content": "Yes, that's true about carl dataset but since the public leaderboard is using 50 out of 200 series to test, the score must be somewhat reliable. @abhranta
", "votes": null, "replies": [ { "id": 2457975, "author_name": "abhranta", "author_url": "", "post_date": "09/27/2023 09:08:00", "content": "The score is good on the 50 series ID… but it may take a major fall with the other 150. I really hope not though.
", "votes": null, "replies": [] } ] }, { "id": 2458006, "author_name": "yuchen2066", "author_url": "", "post_date": "09/27/2023 09:24:36", "content": "Hi, Base on the size of this train set, I feel like it is hard to perform a lot of features engineering on the entire train set, and train it. Do you have any other recommendations of approaches that doesn't limit data variance and have a lot of series? Thanks in advance.
", "votes": null, "replies": [] }, { "id": 2458105, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/27/2023 10:52:45", "content": "It is my understanding that the public code using limited series IDs is only a temporary measure to avoid out-of-memory errors. The train data and test data have different series IDs, so many series IDs should be used for training the final submission.
", "votes": null, "replies": [] } ] } ] }, { "id": 2458198, "author_name": "ravi20076", "author_url": "", "post_date": "09/27/2023 12:19:53", "content": "ICR was a unique challenge with a small dataset and a difficult-to-administer CV scheme. I think this one is a bit different where one could construct a CV scheme, build models and test across a good CV structure.
\nYour thought of not following public work blindly is good. One needs to invest time and efforts and curate his/ her own solutions to be able to score and fare well in such competitions @abhranta
No no I meant for people who are just following the public notebooks. These notebooks generally use the great but small dataset by Carl. This might lead to a tremendous fall in private LB.
", "votes": null, "replies": [ { "id": 2458413, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/27/2023 15:19:19", "content": "Dear @abhranta
\n\n\n\"never blindly follow public notebooks\"
\n
Oh, that should be a banner shown in bold font at the top of the Code section of every Kaggle competition!
\nIn general the magnitude of overfitting (hence shakeup) is inversely proportional to the amount of data we have to work with, and in this competition as we are all aware there is a huge amount of data (hence the creation of the smaller dataset). However, that said we are asked to predict specific events within that data, of which there are on average 35 per series_id, so the Public LB (≈50 series) is based on matching only ≈1730 events, so I would not completely rule out some shakeup. Indeed something that has caught my eye is the the current HSNotD has a Public LB score of 0.418, but curiously the script itself indicates it has a validation score of 0.266 (?!), so as you wisely say never blindly follow public notebooks!
All the best,
\ncarl
@carlmcbrideellis HSNotD is my new favourite Kaggle lingo now. Yeah these are using the 5 min samples now. But there is a lot more that is going to be done probably to address the overfitting and increase the score.
\nLooking forward to more great work from you.
\nAll the best,
\nRegards,
\nAbhranta
\n\n\"…my new favourite Kaggle lingo now\"
\n
LOL: Here is another one: Forkmitting:
\nThe act of simply forking and directly submitting the HSNotD making no changes whatsoever and leaping to the top of the LB, and all in less time than it takes to make a nice cup of tea!
\nAll the best and good luck!
\ncarl
Oh yes, forkmitting. You just know a new high-scoring notebook has been made public when you drop multiple places within an hour.😁
", "votes": null, "replies": [ { "id": 2460137, "author_name": "yuchen2066", "author_url": "", "post_date": "09/28/2023 15:34:48", "content": "Desperate, on the edge of being surpassed by the HSNotD. Hopefully, I don't wake up tomorrow to find myself out of the medal zone because of the forkmitting. 🥲
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2459414, "author_name": "kononenko", "author_url": "", "post_date": "09/28/2023 07:54:23", "content": "ICR was just a badly designed competition. No matter if you followed public notebooks, or designed your own no-postprocessing CV based model/ensemble, there was a very high chance to face a 1K positions shift up or down depending on your luckiness.
", "votes": null, "replies": [ { "id": 2462258, "author_name": "celiker", "author_url": "", "post_date": "09/30/2023 08:37:26", "content": "no no no no icr is the bestest competition ever
", "votes": null, "replies": [ { "id": 2465603, "author_name": "kononenko", "author_url": "", "post_date": "10/03/2023 06:03:28", "content": "@celiker Congrats to your impressive 3260 jump and a solo gold!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2457738": "People overfitted really bad on the ICR dataset and the result was a massive shakeup. Let's all remember to not do that same mistake again 🥲", "2457927": "People overfitted really bad on the ICR dataset because the dataset was so small, while I agree that people shouldn't blindly copy public notebooks I think that it's very unlikely that the same level of shakeup could happen in this competition.", "2457963": "I certainly hope so. But a lot of the public notebooks are using the dataset by carl and it has very few series. Moreover, other way to do it i to sample 1 million points which also limits data variance. Not sure if this is gonna affect the final result", "2457972": "Yes, that's true about carl dataset but since the public leaderboard is using 50 out of 200 series to test, the score must be somewhat reliable. @abhranta", "2457975": "The score is good on the 50 series ID... but it may take a major fall with the other 150. I really hope not though.", "2458006": "Hi, Base on the size of this train set, I feel like it is hard to perform a lot of features engineering on the entire train set, and train it. Do you have any other recommendations of approaches that doesn't limit data variance and have a lot of series? Thanks in advance.", "2458105": "It is my understanding that the public code using limited series IDs is only a temporary measure to avoid out-of-memory errors. The train data and test data have different series IDs, so many series IDs should be used for training the final submission.", "2458198": "ICR was a unique challenge with a small dataset and a difficult-to-administer CV scheme. I think this one is a bit different where one could construct a CV scheme, build models and test across a good CV structure.\nYour thought of not following public work blindly is good. One needs to invest time and efforts and curate his/ her own solutions to be able to score and fare well in such competitions @abhranta", "2458273": "No no I meant for people who are just following the public notebooks. These notebooks generally use the great but small dataset by Carl. This might lead to a tremendous fall in private LB.", "2458413": "Dear @abhranta \n\n* The latest HSNotD (\"Highest Scoring Public Notebook of the Day\" 😃) do not use the reduced dataset that I prepared, but rather they seem to take periodic samples every 5 minutes from the entire training dataset. The [reduced dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) contains the full data for 35 selected series, whilst down-sampling to 1 row in 60 has the equivalent amount of data of around 4.6 series.\n\n> \"*never blindly follow public notebooks*\"\n\nOh, that should be a banner shown in bold font at the top of the **Code** section of *every* Kaggle competition!\n\n* Overfitting:\n\nIn general the magnitude of overfitting (hence shakeup) is inversely proportional to the amount of data we have to work with, and in this competition as we are all aware there is a huge amount of data (hence the creation of the smaller dataset). However, that said we are asked to predict specific events within that data, of which there are on average 35 per `series_id`, so the Public LB (≈50 series) is based on matching only ≈1730 events, so I would not completely rule out some shakeup. Indeed something that has caught my eye is the the [current HSNotD](https://www.kaggle.com/code/siddhvr/cmi-feature-engg-with-ensemble-models) has a Public LB score of `0.418`, but curiously the script itself indicates it has a validation score of `0.266` (?!), so as you wisely say *never blindly follow public notebooks*!\n\nAll the best,\ncarl", "2458433": "carlmcbrideellis HSNotD is my new favourite Kaggle lingo now. Yeah these are using the 5 min samples now. But there is a lot more that is going to be done probably to address the overfitting and increase the score. \n\nLooking forward to more great work from you. \n\nAll the best,\nRegards,\nAbhranta", "2458479": "> \"*...my new favourite Kaggle lingo now*\"\n\nLOL: Here is another one: *Forkmitting*: \n\n The act of simply forking and directly submitting the HSNotD making no changes whatsoever and leaping to the top of the LB, and all in less time than it takes to make a nice cup of tea!\n\nAll the best and good luck!\ncarl", "2459414": "ICR was just a badly designed competition. No matter if you followed public notebooks, or designed your own no-postprocessing CV based model/ensemble, there was a very high chance to face a 1K positions shift up or down depending on your luckiness.", "2459900": "Oh yes, forkmitting. You just know a new high-scoring notebook has been made public when you drop multiple places within an hour.😁", "2460137": "Desperate, on the edge of being surpassed by the HSNotD. Hopefully, I don't wake up tomorrow to find myself out of the medal zone because of the forkmitting. 🥲", "2462258": "no no no no icr is the bestest competition ever", "2465603": "celiker Congrats to your impressive 3260 jump and a solo gold!" }, "source": "meta" }