{ "id": 285505, "title": "Watch out!: test LB period is contained in the train csv", "url": "/competitions/g-research-crypto-forecasting/discussion/285505", "author_name": "JuliΓ‘n Peller (dataista0)", "post_date": "2021-11-05T00:04:57.595000", "votes": 66, "comment_count": 18, "views": 0, "content": "

Hello,

\n

Probably most of you have already noticed, but I just want to make this crystal clear for everybody, since I find the original statement far from transparent.

\n

The \"Data tab\" says that:

\n
\n

Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset.

\n
\n

I found it quite obscure, but the fact is that the The test LB is very likely contained in the train set.

\n

Edit: I got a LB=.99 submission here using only train.csv data so that likelihood has skyrocketed.

\n

These are facts:

\n\n

Finally, I opened the discussion topic Meaningful submission scores / sharing the lower boundary of public test data stating that it would be great to get a public confirmation of this hypothesis (that the public LB is a continuation of the placeholder test.csv) in order to give some meaning to the submissions.

\n

EDIT

\n

[S] πŸͺ™πŸ’² Proposal for a meaningful LB

\n

πŸ‘†πŸ‘†πŸ‘† A proposal for an agreement to get comparable, non-leaky models

\n

Long story short, the idea is to drop the data after 2021-06-13 for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag [Strict] or [S].

\n
def read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df\n
", "messages": [ { "id": 1571516, "postDate": "2021-11-05T00:04:57.597Z", "content": "

Hello,

\n

Probably most of you have already noticed, but I just want to make this crystal clear for everybody, since I find the original statement far from transparent.

\n

The \"Data tab\" says that:

\n
\n

Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset.

\n
\n

I found it quite obscure, but the fact is that the The test LB is very likely contained in the train set.

\n

Edit: I got a LB=.99 submission here using only train.csv data so that likelihood has skyrocketed.

\n

These are facts:

\n\n

Finally, I opened the discussion topic Meaningful submission scores / sharing the lower boundary of public test data stating that it would be great to get a public confirmation of this hypothesis (that the public LB is a continuation of the placeholder test.csv) in order to give some meaning to the submissions.

\n

EDIT

\n

[S] πŸͺ™πŸ’² Proposal for a meaningful LB

\n

πŸ‘†πŸ‘†πŸ‘† A proposal for an agreement to get comparable, non-leaky models

\n

Long story short, the idea is to drop the data after 2021-06-13 for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag [Strict] or [S].

\n
def read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df\n
", "rawMarkdown": "Hello,\n\nProbably most of you have already noticed, but I just want to make this crystal clear for everybody, since I find the original statement far from transparent.\n\nThe \"[Data tab](https://www.kaggle.com/c/g-research-crypto-forecasting/data)\" says that:\n> Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset.\n\nI found it quite obscure, but the fact is that the __The test LB is very likely contained in the train set__.\n\n__Edit: I got a `LB=.99` submission [here](https://www.kaggle.com/julian3833/g-research-using-the-overlap-fully-lb-0-99) using only `train.csv` data so that likelihood has skyrocketed.__\n\nThese are facts: \n* The `train.csv` contains data from `2018-01-01 00:01:00` to `2021-09-21 00:00:00`\n* The placeholder `test.csv` has data from `'2021-06-13 00:00:00'` to `'2021-06-13 00:03:00'`\n* We don't know for sure the public LB date range, but I think a reasonable hypothesis is that it continues the place holder range, this is: `'2021-06-13 00:04:00'`, `'2021-06-13 00:05:00'`, ...\n* The solutions using the full train set are achieving an impossibly high LB score (In the wise words of [katsu1110](https://www.kaggle.com/code1110): \"if creating a >0.1 correlation model was so easy, everybody would be rich\" lol).\n\n\nFinally, I opened the discussion topic [Meaningful submission scores / sharing the lower boundary of public test data](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285289) stating that it would be great to get a public confirmation of this hypothesis (that the public LB is a continuation of the placeholder `test.csv`) in order to give some meaning to the submissions.\n\n\n__EDIT__\n\n# [[S] πŸͺ™πŸ’² Proposal for a meaningful LB](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n# πŸ‘†πŸ‘†πŸ‘† A proposal for an agreement to get comparable, non-leaky models\n\nLong story short, the idea is to drop the data after `2021-06-13` for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag `[Strict]` or `[S]`.\n\n```python\ndef read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n```", "votes": 66 }, { "id": 1572393, "postDate": "2021-11-05T16:26:27.843Z", "content": "

Correct, the current test set is just the tail end of the training set. I'll take a pass at clarifying the data tab. However, the public leaderboard wouldn't be meaningful even if we truncated train.csv as historic cryptocurrency prices are publicly known.

", "rawMarkdown": "Correct, the current test set is just the tail end of the training set. I'll take a pass at clarifying the data tab. However, the public leaderboard wouldn't be meaningful even if we truncated `train.csv` as historic cryptocurrency prices are publicly known.", "votes": 14, "replies": [ { "id": 1572680, "postDate": "2021-11-05T21:21:51.887Z", "content": "

Thanks for the clarification @sohier!

\n

Just to confirm: the following is true, right?

\n

Using data only from before 2021-06-13 00:00:00 is sufficient to avoid any overlapping with the public LB

", "rawMarkdown": "Thanks for the clarification @sohier!\n\nJust to confirm: the following is true, right?\n\n__Using data *only* from before `2021-06-13 00:00:00` is sufficient to avoid any overlapping with the public LB__\n", "votes": 4 }, { "id": 1573104, "postDate": "2021-11-06T09:17:57.033Z", "content": "

Hi. Probably it is a good idea to resctric usage of external data then. It will be extremely useful for the purpose of competition.

", "rawMarkdown": "Hi. Probably it is a good idea to resctric usage of external data then. It will be extremely useful for the purpose of competition.", "votes": 1 }, { "id": 1573560, "postDate": "2021-11-06T18:20:42.383Z", "content": "

So can please give an update on what is going to happen next after the leak??

", "rawMarkdown": "So can please give an update on what is going to happen next after the leak??" }, { "id": 1573765, "postDate": "2021-11-06T22:19:12.760Z", "content": "

Hey,
\nIt was not a mistake or anything wrong in the organization btw. As the organizers said, the target is publicly available (no matter if it is part of the train.csv or not).

\n

It is not a leak at all. Just a condition and we need to work under these circumstances.

\n

Here I have a proposal for sharing results: [S] πŸͺ™πŸ’² Proposal for a meaningful LB

", "rawMarkdown": "Hey,\nIt was not a mistake or anything wrong in the organization btw. As the organizers said, the target is publicly available (no matter if it is part of the train.csv or not). \n\nIt is not a leak at all. Just a condition and we need to work under these circumstances.\n\nHere I have a proposal for sharing results: [[S] πŸͺ™πŸ’² Proposal for a meaningful LB](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n" }, { "id": 1612056, "postDate": "2021-12-08T14:20:20.540Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1572661, "postDate": "2021-11-05T20:51:54.173Z", "content": "

WOW LB=0.99! I am using this for real investments!! πŸ‘Œ

", "rawMarkdown": "WOW LB=0.99! I am using this for real investments!! πŸ‘Œ", "votes": 7, "replies": [ { "id": 1572683, "postDate": "2021-11-05T21:23:21.930Z", "content": "

Lol yes Not a financial advice

", "rawMarkdown": "Lol yes _Not a financial advice_ " } ] }, { "id": 1572215, "postDate": "2021-11-05T14:16:56.767Z", "content": "

Same here. I was just playing with some parameters in the lightGBM and ended up with 96%. Didn't even try hard.

", "rawMarkdown": "Same here. I was just playing with some parameters in the lightGBM and ended up with 96%. Didn't even try hard.", "votes": 1, "replies": [ { "id": 1572688, "postDate": "2021-11-05T21:25:23.603Z", "content": "

Well, that is very high even for the leaked overfitted model. I could only reach 0.31 with a LGBM.
\nOut of curiosity: what were your hyperparameters?

", "rawMarkdown": "Well, that is very high even for the leaked overfitted model. I could only reach `0.31` with a LGBM.\nOut of curiosity: what were your hyperparameters?" }, { "id": 1572841, "postDate": "2021-11-06T02:59:21.623Z", "content": "

LGBMRegressor(n_estimators = 2500, num_leaves=700, learning_rate=0.09)

", "rawMarkdown": "LGBMRegressor(n_estimators = 2500, num_leaves=700, learning_rate=0.09)", "votes": 3 }, { "id": 1573766, "postDate": "2021-11-06T22:21:09.243Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!" } ] }, { "id": 1571836, "postDate": "2021-11-05T07:49:18.443Z", "content": "

Yesterday, I was so surprised after watching more than 0.9 scores on public LB. Thanks for sharing this. It is pretty much evident that test LB is contained in the train set. I am thinking of splitting the training dataset for meaningful validation of my initial models till there is some clarification from organizing team.

", "rawMarkdown": "Yesterday, I was so surprised after watching more than 0.9 scores on public LB. Thanks for sharing this. It is pretty much evident that test LB is contained in the train set. I am thinking of splitting the training dataset for meaningful validation of my initial models till there is some clarification from organizing team.", "votes": 2, "replies": [ { "id": 1572685, "postDate": "2021-11-05T21:24:33.870Z", "content": "

Yes! Definitely. Those were the most obvious, but almost all the models have that problem, even the low-scoring ones

", "rawMarkdown": "Yes! Definitely. Those were the most obvious, but almost all the models have that problem, even the low-scoring ones" } ] }, { "id": 1654408, "postDate": "2022-01-18T13:29:40.717Z", "content": "

a model which have corr > 0.1 will make you extremely rich

", "rawMarkdown": "a model which have corr > 0.1 will make you extremely rich" }, { "id": 1572853, "postDate": "2021-11-06T03:19:51.767Z", "rawMarkdown": "", "votes": -2, "isDeleted": true, "replies": [ { "id": 1573764, "postDate": "2021-11-06T22:18:19.237Z", "content": "

Hey,
\nIt was not a mistake or anything wrong in the organization btw. As the organizers said, the target is publicly available (no matter if it is part of the train.csv or not).

\n

Here I have a proposal for sharing results: [S] πŸͺ™πŸ’² Proposal for a meaningful LB

", "rawMarkdown": "Hey,\nIt was not a mistake or anything wrong in the organization btw. As the organizers said, the target is publicly available (no matter if it is part of the train.csv or not). \n\nHere I have a proposal for sharing results: [[S] πŸͺ™πŸ’² Proposal for a meaningful LB](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n" } ] }, { "id": 1571668, "postDate": "2021-11-05T04:12:55.107Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1572393, "author_name": "Sohier Dane", "author_url": "", "post_date": "2021-11-05T16:26:27.843000", "content": "

Correct, the current test set is just the tail end of the training set. I'll take a pass at clarifying the data tab. However, the public leaderboard wouldn't be meaningful even if we truncated train.csv as historic cryptocurrency prices are publicly known.

", "votes": 14, "replies": [ { "id": 1572680, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-05T21:21:51.887000", "content": "

Thanks for the clarification @sohier!

\n

Just to confirm: the following is true, right?

\n

Using data only from before 2021-06-13 00:00:00 is sufficient to avoid any overlapping with the public LB

", "votes": 4, "replies": [] }, { "id": 1573104, "author_name": "Artem Filippov", "author_url": "", "post_date": "2021-11-06T09:17:57.033000", "content": "

Hi. Probably it is a good idea to resctric usage of external data then. It will be extremely useful for the purpose of competition.

", "votes": 1, "replies": [] }, { "id": 1573560, "author_name": "Krish Yadav", "author_url": "", "post_date": "2021-11-06T18:20:42.383000", "content": "

So can please give an update on what is going to happen next after the leak??

", "votes": 0, "replies": [] }, { "id": 1573765, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-06T22:19:12.760000", "content": "

Hey,
\nIt was not a mistake or anything wrong in the organization btw. As the organizers said, the target is publicly available (no matter if it is part of the train.csv or not).

\n

It is not a leak at all. Just a condition and we need to work under these circumstances.

\n

Here I have a proposal for sharing results: [S] πŸͺ™πŸ’² Proposal for a meaningful LB

", "votes": 0, "replies": [] }, { "id": 1612056, "author_name": "", "author_url": "", "post_date": "2021-12-08T14:20:20.540000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1572661, "author_name": "", "author_url": "", "post_date": "2021-11-05T20:51:54.173000", "content": "

WOW LB=0.99! I am using this for real investments!! πŸ‘Œ

", "votes": 7, "replies": [ { "id": 1572683, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-05T21:23:21.930000", "content": "

Lol yes Not a financial advice

", "votes": 0, "replies": [] } ] }, { "id": 1572215, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-11-05T14:16:56.767000", "content": "

Same here. I was just playing with some parameters in the lightGBM and ended up with 96%. Didn't even try hard.

", "votes": 1, "replies": [ { "id": 1572688, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-05T21:25:23.603000", "content": "

Well, that is very high even for the leaked overfitted model. I could only reach 0.31 with a LGBM.
\nOut of curiosity: what were your hyperparameters?

", "votes": 0, "replies": [] }, { "id": 1572841, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-11-06T02:59:21.623000", "content": "

LGBMRegressor(n_estimators = 2500, num_leaves=700, learning_rate=0.09)

", "votes": 3, "replies": [] }, { "id": 1573766, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-06T22:21:09.243000", "content": "

Thanks for sharing!

", "votes": 0, "replies": [] } ] }, { "id": 1571836, "author_name": "Manny_Jain", "author_url": "", "post_date": "2021-11-05T07:49:18.443000", "content": "

Yesterday, I was so surprised after watching more than 0.9 scores on public LB. Thanks for sharing this. It is pretty much evident that test LB is contained in the train set. I am thinking of splitting the training dataset for meaningful validation of my initial models till there is some clarification from organizing team.

", "votes": 2, "replies": [ { "id": 1572685, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-05T21:24:33.870000", "content": "

Yes! Definitely. Those were the most obvious, but almost all the models have that problem, even the low-scoring ones

", "votes": 0, "replies": [] } ] }, { "id": 1654408, "author_name": "JiangShaoYin", "author_url": "", "post_date": "2022-01-18T13:29:40.717000", "content": "

a model which have corr > 0.1 will make you extremely rich

", "votes": 0, "replies": [] }, { "id": 1572853, "author_name": "", "author_url": "", "post_date": "2021-11-06T03:19:51.767000", "content": "", "votes": -2, "replies": [ { "id": 1573764, "author_name": "JuliΓ‘n Peller (dataista0)", "author_url": "", "post_date": "2021-11-06T22:18:19.237000", "content": "

Hey,
\nIt was not a mistake or anything wrong in the organization btw. As the organizers said, the target is publicly available (no matter if it is part of the train.csv or not).

\n

Here I have a proposal for sharing results: [S] πŸͺ™πŸ’² Proposal for a meaningful LB

", "votes": 0, "replies": [] } ] }, { "id": 1571668, "author_name": "", "author_url": "", "post_date": "2021-11-05T04:12:55.107000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1571516": "Hello,\n\nProbably most of you have already noticed, but I just want to make this crystal clear for everybody, since I find the original statement far from transparent.\n\nThe \"[Data tab](https://www.kaggle.com/c/g-research-crypto-forecasting/data)\" says that:\n> Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset.\n\nI found it quite obscure, but the fact is that the __The test LB is very likely contained in the train set__.\n\n__Edit: I got a `LB=.99` submission [here](https://www.kaggle.com/julian3833/g-research-using-the-overlap-fully-lb-0-99) using only `train.csv` data so that likelihood has skyrocketed.__\n\nThese are facts: \n* The `train.csv` contains data from `2018-01-01 00:01:00` to `2021-09-21 00:00:00`\n* The placeholder `test.csv` has data from `'2021-06-13 00:00:00'` to `'2021-06-13 00:03:00'`\n* We don't know for sure the public LB date range, but I think a reasonable hypothesis is that it continues the place holder range, this is: `'2021-06-13 00:04:00'`, `'2021-06-13 00:05:00'`, ...\n* The solutions using the full train set are achieving an impossibly high LB score (In the wise words of [katsu1110](https://www.kaggle.com/code1110): \"if creating a >0.1 correlation model was so easy, everybody would be rich\" lol).\n\n\nFinally, I opened the discussion topic [Meaningful submission scores / sharing the lower boundary of public test data](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285289) stating that it would be great to get a public confirmation of this hypothesis (that the public LB is a continuation of the placeholder `test.csv`) in order to give some meaning to the submissions.\n\n\n__EDIT__\n\n# [[S] πŸͺ™πŸ’² Proposal for a meaningful LB](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n# πŸ‘†πŸ‘†πŸ‘† A proposal for an agreement to get comparable, non-leaky models\n\nLong story short, the idea is to drop the data after `2021-06-13` for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag `[Strict]` or `[S]`.\n\n```python\ndef read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n```", "1572393": "Correct, the current test set is just the tail end of the training set. I'll take a pass at clarifying the data tab. However, the public leaderboard wouldn't be meaningful even if we truncated `train.csv` as historic cryptocurrency prices are publicly known.", "1572661": "WOW LB=0.99! I am using this for real investments!! πŸ‘Œ", "1572215": "Same here. I was just playing with some parameters in the lightGBM and ended up with 96%. Didn't even try hard.", "1571836": "Yesterday, I was so surprised after watching more than 0.9 scores on public LB. Thanks for sharing this. It is pretty much evident that test LB is contained in the train set. I am thinking of splitting the training dataset for meaningful validation of my initial models till there is some clarification from organizing team.", "1654408": "a model which have corr > 0.1 will make you extremely rich", "1572853": "", "1571668": "" } }