{ "id": 342155, "title": "This really fuc#$ng LB destroy!!!", "url": "/competitions/AI4Code/discussion/342155", "author_name": "", "post_date": "2022-08-05T17:27:27.983387200Z", "votes": 9, "comment_count": 11, "views": 0, "content": "

Do you remember this guy?

\n

\"Screenshot-25\"

\n

Ah, how you can forget him, he destroyed the leaderboard…
\nWell, we can use his notebook to get some ideas or just submit and get a free medal (medals inflation!!!), but when I tried to submit it (sorry, I will never use public best scoring notebooks again), I got the error, because of I don't have some model's weights in the input directory.

\n

\"Screenshot-23\"
\n\"Screenshot-24\"

\n

So, that means that people made the dataset with weights private, and people won't be able to submit it again, hence probably half of the people in the silver and bronze medals zone are very lucky to submit it earlier (let's say \"lucky people\"). Although, now this situation is very similar to private sharing because \"lucky people\" have a submission, which was generated by private models' weights, so the \"lucky people\" should be disqualified.

\n

Well, sorry for such negativity, but that's facts!

\n

And yet another funny story, that this dude is still sharing this notebook with other account after Team Mergering deadline => private sharing => disqual!

\n

\"Screenshot-26\"

", "messages": [ { "id": "1886304", "postDate": "08/05/2022 17:27:27", "content": "

Do you remember this guy?

\n

\"Screenshot-25\"

\n

Ah, how you can forget him, he destroyed the leaderboard…
\nWell, we can use his notebook to get some ideas or just submit and get a free medal (medals inflation!!!), but when I tried to submit it (sorry, I will never use public best scoring notebooks again), I got the error, because of I don't have some model's weights in the input directory.

\n

\"Screenshot-23\"
\n\"Screenshot-24\"

\n

So, that means that people made the dataset with weights private, and people won't be able to submit it again, hence probably half of the people in the silver and bronze medals zone are very lucky to submit it earlier (let's say \"lucky people\"). Although, now this situation is very similar to private sharing because \"lucky people\" have a submission, which was generated by private models' weights, so the \"lucky people\" should be disqualified.

\n

Well, sorry for such negativity, but that's facts!

\n

And yet another funny story, that this dude is still sharing this notebook with other account after Team Mergering deadline => private sharing => disqual!

\n

\"Screenshot-26\"

", "rawMarkdown": "Do you remember this guy? \n\n\"Screenshot-25\"\n\nAh, how you can forget him, he destroyed the leaderboard...\nWell, we can use his notebook to get some ideas or just submit and get a free medal (medals inflation!!!), but when I tried to submit it (sorry, I will never use public best scoring notebooks again), I got the error, because of I don't have some model's weights in the input directory. \n\n\"Screenshot-23\"\n\"Screenshot-24\"\n\nSo, that means that people made the dataset with weights private, and people won't be able to submit it again, hence probably half of the people in the silver and bronze medals zone are very lucky to submit it earlier (let's say \"lucky people\"). Although, now this situation is very similar to private sharing because \"lucky people\" have a submission, which was generated by private models' weights, so the \"lucky people\" should be disqualified. \n\n\nWell, sorry for such negativity, but that's facts! \n\n\nAnd yet another funny story, that this dude is still sharing this notebook with other account after Team Mergering deadline => private sharing => disqual! \n\n\"Screenshot-26\"", "votes": null }, { "id": "1886312", "postDate": "08/05/2022 17:32:28", "content": "

I calculated and there are 80 teams, which are using this private shared notebook…

", "rawMarkdown": "I calculated and there are 80 teams, which are using this private shared notebook...", "votes": null }, { "id": "1886322", "postDate": "08/05/2022 17:41:26", "content": "

Dont worry. See this - https://www.kaggle.com/competitions/AI4Code/discussion/341780
\nAs this competition private LB data has not yet been created. It will happen over next 3 months, post which these models will be re-run and all forks would fail at that time (of course unless someone downloaded the models)

", "rawMarkdown": "Dont worry. See this - https://www.kaggle.com/competitions/AI4Code/discussion/341780\nAs this competition private LB data has not yet been created. It will happen over next 3 months, post which these models will be re-run and all forks would fail at that time (of course unless someone downloaded the models)", "votes": null }, { "id": "1886324", "postDate": "08/05/2022 17:43:37", "content": "

Maybe, you are right, but I would prefer the true public medal placers :)

", "rawMarkdown": "Maybe, you are right, but I would prefer the true public medal placers :)", "votes": null }, { "id": "1886723", "postDate": "08/06/2022 05:38:46", "content": "

always download the weights and create your own dataset before re-running public notebooks. point of sharing notebooks is for others to learn. if your interest is to boost your ranking in the competition then you can combine your own ideas with those in the notebook and improve your score

", "rawMarkdown": "always download the weights and create your own dataset before re-running public notebooks. point of sharing notebooks is for others to learn. if your interest is to boost your ranking in the competition then you can combine your own ideas with those in the notebook and improve your score", "votes": null }, { "id": "1887007", "postDate": "08/06/2022 11:13:25", "content": "

Kinda funny that he went full damage control mode and deleted the weights after the backlash, which somehow made the situation even worse.

", "rawMarkdown": "Kinda funny that he went full damage control mode and deleted the weights after the backlash, which somehow made the situation even worse.", "votes": null }, { "id": "1887663", "postDate": "08/06/2022 23:00:08", "content": "

good point

", "rawMarkdown": "good point", "votes": null }, { "id": "1887962", "postDate": "08/07/2022 08:16:43", "content": "

goood point

", "rawMarkdown": "goood point", "votes": null }, { "id": "1888004", "postDate": "08/07/2022 09:01:40", "content": "

Unfortunately this happened in the past already. A well known GM made his weight dataset private at the last minute in a past competition, jeopardizing lost of competitors. Why do people do this is unclear to me and not worth understanding if you ask me.

\n

The lesson is to not directly use a public notebook output for your submissions.

\n

It is more than fine to learn form it, reuse it, but not to depend on it.

\n

My preferred way is to implement public notebook ideas in my code, then test with my cross validation setup, and keep the ones that pass this test. I never directly reuse the output of a public notebook for my submissions. There is too much risk: the risk of some deletion (fortunately a rare event still), but also a risk of overfitting as public notebooks do not explain how the parameters and other settings were tuned. They often are tuned form public LB feedback, which is THE recipe for overfitting.

", "rawMarkdown": "Unfortunately this happened in the past already. A well known GM made his weight dataset private at the last minute in a past competition, jeopardizing lost of competitors. Why do people do this is unclear to me and not worth understanding if you ask me.\n\nThe lesson is to not directly use a public notebook output for your submissions.\n\nIt is more than fine to learn form it, reuse it, but not to depend on it. \n\nMy preferred way is to implement public notebook ideas in my code, then test with my cross validation setup, and keep the ones that pass this test. I never directly reuse the output of a public notebook for my submissions. There is too much risk: the risk of some deletion (fortunately a rare event still), but also a risk of overfitting as public notebooks do not explain how the parameters and other settings were tuned. They often are tuned form public LB feedback, which is THE recipe for overfitting.", "votes": null }, { "id": "1888668", "postDate": "08/07/2022 18:01:38", "content": "

Thank you, CPMP!

", "rawMarkdown": "Thank you, CPMP!", "votes": null }, { "id": "1890350", "postDate": "08/08/2022 17:24:39", "content": "

Among the other useful things GM CPMP suggested:

\n
\n

My preferred way is to implement public notebook ideas in my code

\n
\n

I learned this the very hard way in this competition by realising there was copied bug 🥲

", "rawMarkdown": "Among the other useful things GM CPMP suggested:\n\n> My preferred way is to implement public notebook ideas in my code\n\nI learned this the very hard way in this competition by realising there was copied bug 🥲", "votes": null }, { "id": "1891063", "postDate": "08/09/2022 08:14:13", "content": "

What is this bug?

", "rawMarkdown": "What is this bug?", "votes": null } ], "comments": [ { "id": 1886312, "author_name": "vad13irt", "author_url": "", "post_date": "08/05/2022 17:32:28", "content": "

I calculated and there are 80 teams, which are using this private shared notebook…

", "votes": null, "replies": [] }, { "id": 1886322, "author_name": "allohvk", "author_url": "", "post_date": "08/05/2022 17:41:26", "content": "

Dont worry. See this - https://www.kaggle.com/competitions/AI4Code/discussion/341780
\nAs this competition private LB data has not yet been created. It will happen over next 3 months, post which these models will be re-run and all forks would fail at that time (of course unless someone downloaded the models)

", "votes": null, "replies": [ { "id": 1886324, "author_name": "vad13irt", "author_url": "", "post_date": "08/05/2022 17:43:37", "content": "

Maybe, you are right, but I would prefer the true public medal placers :)

", "votes": null, "replies": [] } ] }, { "id": 1886723, "author_name": "nymfree", "author_url": "", "post_date": "08/06/2022 05:38:46", "content": "

always download the weights and create your own dataset before re-running public notebooks. point of sharing notebooks is for others to learn. if your interest is to boost your ranking in the competition then you can combine your own ideas with those in the notebook and improve your score

", "votes": null, "replies": [] }, { "id": 1887007, "author_name": "suicaokhoailang", "author_url": "", "post_date": "08/06/2022 11:13:25", "content": "

Kinda funny that he went full damage control mode and deleted the weights after the backlash, which somehow made the situation even worse.

", "votes": null, "replies": [] }, { "id": 1887663, "author_name": "grosst10", "author_url": "", "post_date": "08/06/2022 23:00:08", "content": "

good point

", "votes": null, "replies": [] }, { "id": 1887962, "author_name": "bilalsuppal", "author_url": "", "post_date": "08/07/2022 08:16:43", "content": "

goood point

", "votes": null, "replies": [] }, { "id": 1888004, "author_name": "cpmpml", "author_url": "", "post_date": "08/07/2022 09:01:40", "content": "

Unfortunately this happened in the past already. A well known GM made his weight dataset private at the last minute in a past competition, jeopardizing lost of competitors. Why do people do this is unclear to me and not worth understanding if you ask me.

\n

The lesson is to not directly use a public notebook output for your submissions.

\n

It is more than fine to learn form it, reuse it, but not to depend on it.

\n

My preferred way is to implement public notebook ideas in my code, then test with my cross validation setup, and keep the ones that pass this test. I never directly reuse the output of a public notebook for my submissions. There is too much risk: the risk of some deletion (fortunately a rare event still), but also a risk of overfitting as public notebooks do not explain how the parameters and other settings were tuned. They often are tuned form public LB feedback, which is THE recipe for overfitting.

", "votes": null, "replies": [ { "id": 1888668, "author_name": "vad13irt", "author_url": "", "post_date": "08/07/2022 18:01:38", "content": "

Thank you, CPMP!

", "votes": null, "replies": [] }, { "id": 1890350, "author_name": "init27", "author_url": "", "post_date": "08/08/2022 17:24:39", "content": "

Among the other useful things GM CPMP suggested:

\n
\n

My preferred way is to implement public notebook ideas in my code

\n
\n

I learned this the very hard way in this competition by realising there was copied bug 🥲

", "votes": null, "replies": [] }, { "id": 1891063, "author_name": "e0xextazy", "author_url": "", "post_date": "08/09/2022 08:14:13", "content": "

What is this bug?

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1886304": "Do you remember this guy? \n\n\"Screenshot-25\"\n\nAh, how you can forget him, he destroyed the leaderboard...\nWell, we can use his notebook to get some ideas or just submit and get a free medal (medals inflation!!!), but when I tried to submit it (sorry, I will never use public best scoring notebooks again), I got the error, because of I don't have some model's weights in the input directory. \n\n\"Screenshot-23\"\n\"Screenshot-24\"\n\nSo, that means that people made the dataset with weights private, and people won't be able to submit it again, hence probably half of the people in the silver and bronze medals zone are very lucky to submit it earlier (let's say \"lucky people\"). Although, now this situation is very similar to private sharing because \"lucky people\" have a submission, which was generated by private models' weights, so the \"lucky people\" should be disqualified. \n\n\nWell, sorry for such negativity, but that's facts! \n\n\nAnd yet another funny story, that this dude is still sharing this notebook with other account after Team Mergering deadline => private sharing => disqual! \n\n\"Screenshot-26\"", "1886312": "I calculated and there are 80 teams, which are using this private shared notebook...", "1886322": "Dont worry. See this - https://www.kaggle.com/competitions/AI4Code/discussion/341780\nAs this competition private LB data has not yet been created. It will happen over next 3 months, post which these models will be re-run and all forks would fail at that time (of course unless someone downloaded the models)", "1886324": "Maybe, you are right, but I would prefer the true public medal placers :)", "1886723": "always download the weights and create your own dataset before re-running public notebooks. point of sharing notebooks is for others to learn. if your interest is to boost your ranking in the competition then you can combine your own ideas with those in the notebook and improve your score", "1887007": "Kinda funny that he went full damage control mode and deleted the weights after the backlash, which somehow made the situation even worse.", "1887663": "good point", "1887962": "goood point", "1888004": "Unfortunately this happened in the past already. A well known GM made his weight dataset private at the last minute in a past competition, jeopardizing lost of competitors. Why do people do this is unclear to me and not worth understanding if you ask me.\n\nThe lesson is to not directly use a public notebook output for your submissions.\n\nIt is more than fine to learn form it, reuse it, but not to depend on it. \n\nMy preferred way is to implement public notebook ideas in my code, then test with my cross validation setup, and keep the ones that pass this test. I never directly reuse the output of a public notebook for my submissions. There is too much risk: the risk of some deletion (fortunately a rare event still), but also a risk of overfitting as public notebooks do not explain how the parameters and other settings were tuned. They often are tuned form public LB feedback, which is THE recipe for overfitting.", "1888668": "Thank you, CPMP!", "1890350": "Among the other useful things GM CPMP suggested:\n\n> My preferred way is to implement public notebook ideas in my code\n\nI learned this the very hard way in this competition by realising there was copied bug 🥲", "1891063": "What is this bug?" }, "source": "meta" }