seems everyone get 0.999 on public LB, we know it's ridiculous, but what is a reasonable best score? 0.1, 0.2, 0.3 or even higher…
", "messages": [ { "id": "1572842", "postDate": "11/06/2021 02:59:33", "content": "seems everyone get 0.999 on public LB, we know it's ridiculous, but what is a reasonable best score? 0.1, 0.2, 0.3 or even higher…
", "rawMarkdown": "seems everyone get 0.999 on public LB, we know it's ridiculous, but what is a reasonable best score? 0.1, 0.2, 0.3 or even higher...", "votes": null }, { "id": "1572989", "postDate": "11/06/2021 06:27:38", "content": "it's an interesting topic to discuss, I don't really know what is the proper range of LB score, for a well-trained model.
\nBut people how submitted early, are getting that kind of score,
and also there is a 2nd place public NB by @julian3833 . It might be possible that many people have submitted that without reading it, it says
\nBelow you can see an 18-line submission that overfits the public LB fully, using the fact that the test period is contained in the train data to get the closest point for each test row.
And also the topic Watch out!: test LB period is contained in the train csv related to this test data leak in the train is not much upvoted, maybe people have not seen that topic yet, If they have seen that then It could happen that they avoid using the test data in the train.
But there are also some GMs and Masters who are at the bottom of the leaderboard. but I think they are aware of the fact of the data leak.
can't really tell, maybe we need more data to get to a decent situation.
PS: I'm just starting with kaggle competitions, I dont have much experience. If I have said something wrong, then please pardon me.
", "rawMarkdown": "it's an interesting topic to discuss, I don't really know what is the proper range of LB score, for a well-trained model.\n\n- But people how submitted early, are getting that kind of score, \n- and also there is a 2nd place public NB by @julian3833 . It might be possible that many people have submitted that without reading it, it says\n`Below you can see an 18-line submission that overfits the public LB fully, using the fact that the test period is contained in the train data to get the closest point for each test row.` \n\n- And also the `topic` [Watch out!: test LB period is contained in the train csv ](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285505) related to this test data leak in the train is not much upvoted, maybe people have not seen that topic yet, If they have seen that then It could happen that they avoid using the test data in the train. \n\n- But there are also some GMs and Masters who are at the bottom of the leaderboard. but I think they are aware of the fact of the data leak. \n\n- can't really tell, maybe we need more data to get to a decent situation. \n\nPS: I'm just starting with kaggle competitions, I dont have much experience. If I have said something wrong, then please pardon me.", "votes": null }, { "id": "1573085", "postDate": "11/06/2021 08:56:24", "content": "\n\nBut people how submitted early, are getting that kind of score
\n
The public LB result is publicly available. You can check the data page.
\n\n", "rawMarkdown": "> But people how submitted early, are getting that kind of score\n\nThe public LB result is publicly available. You can check the data page.\n\n> THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code.", "votes": null }, { "id": "1573302", "postDate": "11/06/2021 13:46:18", "content": "THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code.
\n
Although it was written in the Data tab,
\nTHE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL
It may happen that they did not realize that at first.
", "rawMarkdown": "Although it was written in the Data tab,\n`THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL`\n\nIt may happen that they did not realize that at first.", "votes": null }, { "id": "1573385", "postDate": "11/06/2021 15:19:38", "content": "We already have some non-leaky notebooks:
\nOf course it depends on market conditions, but from these you can imagine that it will be difficult to go way above 0.05 or 0.1.
\nAnother benchmark is the numerai competition. A continuous ML competition with a similar set-up. There are some main difference with the current competition (spearman v.s. pearson correlation, the asset universe, a significant anonymisation process for numerai…). But it should give some good idea what to expect. The current #1 is at 0.037. With oscillation in the past ranging from 0 to 0.08.
", "rawMarkdown": "We already have some non-leaky notebooks:\n- My 'no-model' baseline: https://www.kaggle.com/lucasmorin/minimal-fast-baseline that score 0.015\n- Two non leaky baseline LGBM: https://www.kaggle.com/code1110/gresearch-simple-lgb-starter and https://www.kaggle.com/julian3833/s-g-research-strict-lgbm-example-lb-0-017 that score 0.022 and 0.017 respectively.\n\nOf course it depends on market conditions, but from these you can imagine that it will be difficult to go way above 0.05 or 0.1. \n\nAnother benchmark is the numerai competition. A continuous ML competition with a similar set-up. There are some main difference with the current competition (spearman v.s. pearson correlation, the asset universe, a significant anonymisation process for numerai...). But it should give some good idea what to expect. The current #1 is at 0.037. With oscillation in the past ranging from 0 to 0.08.", "votes": null } ], "comments": [ { "id": 1572989, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 06:27:38", "content": "it's an interesting topic to discuss, I don't really know what is the proper range of LB score, for a well-trained model.
\nBut people how submitted early, are getting that kind of score,
and also there is a 2nd place public NB by @julian3833 . It might be possible that many people have submitted that without reading it, it says
\nBelow you can see an 18-line submission that overfits the public LB fully, using the fact that the test period is contained in the train data to get the closest point for each test row.
And also the topic Watch out!: test LB period is contained in the train csv related to this test data leak in the train is not much upvoted, maybe people have not seen that topic yet, If they have seen that then It could happen that they avoid using the test data in the train.
But there are also some GMs and Masters who are at the bottom of the leaderboard. but I think they are aware of the fact of the data leak.
can't really tell, maybe we need more data to get to a decent situation.
PS: I'm just starting with kaggle competitions, I dont have much experience. If I have said something wrong, then please pardon me.
", "votes": null, "replies": [ { "id": 1573085, "author_name": "zarif98sjs", "author_url": "", "post_date": "11/06/2021 08:56:24", "content": "\n\nBut people how submitted early, are getting that kind of score
\n
The public LB result is publicly available. You can check the data page.
\n\n", "votes": null, "replies": [] }, { "id": 1573302, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 13:46:18", "content": "THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code.
\n
Although it was written in the Data tab,
\nTHE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL
It may happen that they did not realize that at first.
", "votes": null, "replies": [] } ] }, { "id": 1573385, "author_name": "lucasmorin", "author_url": "", "post_date": "11/06/2021 15:19:38", "content": "We already have some non-leaky notebooks:
\nOf course it depends on market conditions, but from these you can imagine that it will be difficult to go way above 0.05 or 0.1.
\nAnother benchmark is the numerai competition. A continuous ML competition with a similar set-up. There are some main difference with the current competition (spearman v.s. pearson correlation, the asset universe, a significant anonymisation process for numerai…). But it should give some good idea what to expect. The current #1 is at 0.037. With oscillation in the past ranging from 0 to 0.08.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1572842": "seems everyone get 0.999 on public LB, we know it's ridiculous, but what is a reasonable best score? 0.1, 0.2, 0.3 or even higher...", "1572989": "it's an interesting topic to discuss, I don't really know what is the proper range of LB score, for a well-trained model.\n\n- But people how submitted early, are getting that kind of score, \n- and also there is a 2nd place public NB by @julian3833 . It might be possible that many people have submitted that without reading it, it says\n`Below you can see an 18-line submission that overfits the public LB fully, using the fact that the test period is contained in the train data to get the closest point for each test row.` \n\n- And also the `topic` [Watch out!: test LB period is contained in the train csv ](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285505) related to this test data leak in the train is not much upvoted, maybe people have not seen that topic yet, If they have seen that then It could happen that they avoid using the test data in the train. \n\n- But there are also some GMs and Masters who are at the bottom of the leaderboard. but I think they are aware of the fact of the data leak. \n\n- can't really tell, maybe we need more data to get to a decent situation. \n\nPS: I'm just starting with kaggle competitions, I dont have much experience. If I have said something wrong, then please pardon me.", "1573085": "> But people how submitted early, are getting that kind of score\n\nThe public LB result is publicly available. You can check the data page.\n\n> THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code.", "1573302": "Although it was written in the Data tab,\n`THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL`\n\nIt may happen that they did not realize that at first.", "1573385": "We already have some non-leaky notebooks:\n- My 'no-model' baseline: https://www.kaggle.com/lucasmorin/minimal-fast-baseline that score 0.015\n- Two non leaky baseline LGBM: https://www.kaggle.com/code1110/gresearch-simple-lgb-starter and https://www.kaggle.com/julian3833/s-g-research-strict-lgbm-example-lb-0-017 that score 0.022 and 0.017 respectively.\n\nOf course it depends on market conditions, but from these you can imagine that it will be difficult to go way above 0.05 or 0.1. \n\nAnother benchmark is the numerai competition. A continuous ML competition with a similar set-up. There are some main difference with the current competition (spearman v.s. pearson correlation, the asset universe, a significant anonymisation process for numerai...). But it should give some good idea what to expect. The current #1 is at 0.037. With oscillation in the past ranging from 0 to 0.08." }, "source": "meta" }