{ "id": 290265, "title": "Top 5 leader board without training on future data", "url": "/competitions/g-research-crypto-forecasting/discussion/290265", "author_name": "Ken Miller", "post_date": "2021-11-23T18:31:43.004000", "votes": 70, "comment_count": 78, "views": 0, "content": "

As a user observes: \"THE PUBLIC LEADER BOARD FOR THIS COMPETITION IS NOT MEANINGFUL \"
\nAs many have pointed out, the LB scores are useless because people are intentionally over-fitting using future data.

\n

Top Scores without training on future data:
\n@alankabisov UPD2: 0.0403
\n@yamqwe 0.0361
\n@agenlu 0.0326
\n@alexxanderlarko UPD: 0.0258
\n@saveriomazza 0.0226

\n

see the function below on how to remove the data - @alexxanderlarko was nice enough to post

\n

I am working on a notebook removing the future data and will post my score.

\n

Thanks,

", "messages": [ { "id": 1593286, "postDate": "2021-11-23T18:31:43.003Z", "content": "

As a user observes: \"THE PUBLIC LEADER BOARD FOR THIS COMPETITION IS NOT MEANINGFUL \"
\nAs many have pointed out, the LB scores are useless because people are intentionally over-fitting using future data.

\n

Top Scores without training on future data:
\n@alankabisov UPD2: 0.0403
\n@yamqwe 0.0361
\n@agenlu 0.0326
\n@alexxanderlarko UPD: 0.0258
\n@saveriomazza 0.0226

\n

see the function below on how to remove the data - @alexxanderlarko was nice enough to post

\n

I am working on a notebook removing the future data and will post my score.

\n

Thanks,

", "rawMarkdown": "As a user observes: \"THE PUBLIC LEADER BOARD FOR THIS COMPETITION IS NOT MEANINGFUL \"\nAs many have pointed out, the LB scores are useless because people are intentionally over-fitting using future data.\n\n**Top Scores without training on future data:**\n@alankabisov UPD2: 0.0403 \n@yamqwe 0.0361\n@agenlu 0.0326\n@alexxanderlarko UPD: 0.0258\n@saveriomazza 0.0226\n\nsee the function below on how to remove the data - @alexxanderlarko was nice enough to post\n\nI am working on a notebook removing the future data and will post my score.\n\nThanks,", "votes": 70 }, { "id": 1601530, "postDate": "2021-12-01T11:43:08.153Z", "content": "
\n

For those of you getting .0248 or .0361 - how are you removing the future data?

\n
\n

Like this

\n

df = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')

\n

def test_train_split(df):

\n
df[\"datetime\"] = pd.to_datetime(df[\"timestamp\"], unit=\"s\")\n\nX_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].drop(['Target','datetime'], axis=1)\n\ny_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].Target\n\nX_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].drop(['Target','datetime'], axis=1)\n\ny_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].Target\n\nreturn X_train, y_train, X_test, y_test\n
\n

X_train, y_train, X_test, y_test = test_train_split(df)

", "rawMarkdown": "> For those of you getting .0248 or .0361 - how are you removing the future data?\n\nLike this\n\ndf = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')\n\ndef test_train_split(df):\n\n df[\"datetime\"] = pd.to_datetime(df[\"timestamp\"], unit=\"s\")\n\n X_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].drop(['Target','datetime'], axis=1)\n\n y_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].Target\n\n X_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].drop(['Target','datetime'], axis=1)\n\n y_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].Target\n\n return X_train, y_train, X_test, y_test\n\nX_train, y_train, X_test, y_test = test_train_split(df)", "votes": 8, "replies": [ { "id": 1601921, "postDate": "2021-12-01T16:39:25.713Z", "content": "

Thanks Alexander

", "rawMarkdown": "Thanks Alexander" } ] }, { "id": 1597921, "postDate": "2021-11-28T04:07:00.460Z", "content": "

All of them are overfitting. (training data includes test data)
\nMy best score - 0.0248 !!!

", "rawMarkdown": "All of them are overfitting. (training data includes test data)\nMy best score - 0.0248 !!!", "votes": 8, "replies": [ { "id": 1598115, "postDate": "2021-11-28T09:07:04.717Z", "content": "

100% Agree.
\nMy best score (that I trust) is 0.0361

", "rawMarkdown": "100% Agree. \nMy best score (that I trust) is 0.0361", "votes": 5 }, { "id": 1598343, "postDate": "2021-11-28T12:13:07.953Z", "content": "

As I remember from one video, when one guy shows the accuracy for Market Prediction model as 52% all the audience was laughing, like a probability with a coin, but he explain,that even this additional 2% can helps companies make good money, and it shows that it is not a random

", "rawMarkdown": "As I remember from one video, when one guy shows the accuracy for Market Prediction model as 52% all the audience was laughing, like a probability with a coin, but he explain,that even this additional 2% can helps companies make good money, and it shows that it is not a random", "votes": 4 }, { "id": 1599928, "postDate": "2021-11-30T00:30:03.367Z", "content": "

This is absolutely correct. Imagine that you are sitting inside a Goldman Sachs trading desk and are asked to come up with a crypto forecast model. If your model explains 52% of the data, you're a rockstar. Values around 20% to 30% are much more common. I'm assuming an ARIMA model with possibly some exogenous data.

", "rawMarkdown": "This is absolutely correct. Imagine that you are sitting inside a Goldman Sachs trading desk and are asked to come up with a crypto forecast model. If your model explains 52% of the data, you're a rockstar. Values around 20% to 30% are much more common. I'm assuming an ARIMA model with possibly some exogenous data. ", "votes": 2 }, { "id": 1604950, "postDate": "2021-12-03T21:47:24.270Z", "content": "

Here is a notebook that removes the future price data and scores a .0417

\n

https://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline

", "rawMarkdown": "Here is a notebook that removes the future price data and scores a .0417\n\nhttps://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline" }, { "id": 1666024, "postDate": "2022-01-27T10:32:25.563Z", "content": "

@mlconsult are you sure it removes it? It says

\n
REMOVE_LB_TEST_OVERLAPPING_DATA = False\n
", "rawMarkdown": "@mlconsult are you sure it removes it? It says\n```\nREMOVE_LB_TEST_OVERLAPPING_DATA = False\n```" } ] }, { "id": 1605201, "postDate": "2021-12-04T04:03:41.097Z", "content": "

Hello,
\nJust a clarification:
\nThe notebook https://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline that you mention is just a copy of mine and it achieves 0.0417 with the \"leaky\" data. You can verify that the version with the score says:
\nREMOVE_LB_TEST_OVERLAPPING_DATA = False

\n

So the result of 0.0417 is leaky.

\n

The simple LGBM of that notebook, without the leakage, is presented in this other notebook:
\n🪙💲 Proposal for a meaningful LB + Strict LGBM and achieves a LB of just 0.018.

\n

I also propose a mechanism (\"Strict LB\") so we can all do the same to avoid the leakage in the same way, to obtain comparable results.

\n

The idea is the most direct one: Only keep data from before 2021-06-13 00:00:00. and the implementation is straightforward:

\n
import pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df\n
", "rawMarkdown": "Hello,\nJust a clarification:\nThe notebook https://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline that you mention is just a copy of [mine](https://www.kaggle.com/julian3833/g-research-starter-lgbm-pipeline) and it achieves `0.0417` **with** the \"leaky\" data. You can verify that the version with the score says:\n`REMOVE_LB_TEST_OVERLAPPING_DATA = False`\n\nSo the result of `0.0417` is leaky.\n\n\nThe simple LGBM of that notebook, without the leakage, is presented in this other notebook:\n[🪙💲 Proposal for a meaningful LB + Strict LGBM](https://www.kaggle.com/julian3833/proposal-for-a-meaningful-lb-strict-lgbm?scriptVersionId=80421622) and achieves a LB of just `0.018`.\n\nI also propose a mechanism (\"Strict LB\") so we can all do the same to avoid the leakage in the same way, to obtain comparable results.\n\nThe idea is the most direct one: **Only keep data from before 2021-06-13 00:00:00.** and the implementation is straightforward:\n\n```python\nimport pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n```", "votes": 5, "replies": [ { "id": 1605743, "postDate": "2021-12-04T14:04:55.517Z", "content": "

thanks @julian3833

", "rawMarkdown": "thanks @julian3833", "votes": 1 } ] }, { "id": 1645211, "postDate": "2022-01-10T18:53:42.927Z", "content": "

It would not surprise me at all if the private [edit] LB for this competition were to reveal complete carnage. I expect to see plenty of timed-out and failed submissions, and many negative scores. Maybe a genuine 0.02 prediction will be in the running for a medal?

", "rawMarkdown": "It would not surprise me at all if the private [edit] LB for this competition were to reveal complete carnage. I expect to see plenty of timed-out and failed submissions, and many negative scores. Maybe a genuine 0.02 prediction will be in the running for a medal?", "votes": 3, "replies": [ { "id": 1645392, "postDate": "2022-01-10T21:43:30.493Z", "content": "
\n

failed submissions

\n
\n

yeah, I too see that coming

", "rawMarkdown": "> failed submissions\n\nyeah, I too see that coming ", "votes": 2 } ] }, { "id": 1652681, "postDate": "2022-01-16T22:10:16.410Z", "content": "

Hello @mlconsult, thanks for the information. I usually use the LB as a reference to measure my model's improvements over time, and because now it is broken, this post will be my reference.

", "rawMarkdown": "Hello @mlconsult, thanks for the information. I usually use the LB as a reference to measure my model's improvements over time, and because now it is broken, this post will be my reference. ", "votes": 1 }, { "id": 1644805, "postDate": "2022-01-10T14:01:27.470Z", "content": "

Mine is 0.0247

", "rawMarkdown": "Mine is 0.0247 \n", "votes": 1 }, { "id": 1642942, "postDate": "2022-01-08T20:23:54.820Z", "content": "

My best so far is 0.0221

", "rawMarkdown": "My best so far is 0.0221", "votes": 1 }, { "id": 1642367, "postDate": "2022-01-08T09:48:02.630Z", "content": "

Good post, I got similar results when removing training on future data.

", "rawMarkdown": "Good post, I got similar results when removing training on future data.", "votes": 1 }, { "id": 1641300, "postDate": "2022-01-07T09:29:37.677Z", "content": "

May I ask how the scores were obtained on this leaderboard? How should I tune the hyperparameters? On the test set(which contains data on the public leaderboard) or on another held-out dataset?

", "rawMarkdown": "May I ask how the scores were obtained on this leaderboard? How should I tune the hyperparameters? On the test set(which contains data on the public leaderboard) or on another held-out dataset?", "votes": 1 }, { "id": 1633375, "postDate": "2021-12-30T17:58:58.733Z", "content": "

ah thanks for this! it's nice to have a place to really compare myself too.

", "rawMarkdown": "ah thanks for this! it's nice to have a place to really compare myself too.", "votes": 1 }, { "id": 1629028, "postDate": "2021-12-25T15:29:24.047Z", "content": "

Hi All, I have been giving some thought to both the issue relating to a fair score before going into the live valuation part of the process and agree that it makes sense to slice the data as per:

\n

import pandas as pd

\n

def read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):
\n df = pd.read_csv(file_name)
\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
\n df = df[df['datetime'] < '2021-06-13 00:00:00']
\n return df

\n

Not only will this uphold the normal lb process but ensure that the integrity of the model is inplace, however it strikes me that this may lead to a large chunk of missing data once committee in this way, especially if this is chosen as your entry. I am assuming that the dataset on inception of the live valuation will be complete to the 2nd of Feb 2022 ('/kaggle/input/g-research-crypto-forecasting/train.csv'), if not this presents problem for any lagged features being utilised. Has anyone else thought about this or have any clarity on the shape of the final dataset?

", "rawMarkdown": "Hi All, I have been giving some thought to both the issue relating to a fair score before going into the live valuation part of the process and agree that it makes sense to slice the data as per:\n\nimport pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n\nNot only will this uphold the normal lb process but ensure that the integrity of the model is inplace, however it strikes me that this may lead to a large chunk of missing data once committee in this way, especially if this is chosen as your entry. I am assuming that the dataset on inception of the live valuation will be complete to the 2nd of Feb 2022 *('/kaggle/input/g-research-crypto-forecasting/train.csv')*, if not this presents problem for any lagged features being utilised. Has anyone else thought about this or have any clarity on the shape of the final dataset?\n\n", "votes": 1 }, { "id": 1601397, "postDate": "2021-12-01T08:45:55.277Z", "content": "

Validation set will not be used if you want look high in public score.

", "rawMarkdown": "Validation set will not be used if you want look high in public score.", "votes": 1 }, { "id": 1600191, "postDate": "2021-11-30T07:40:51.787Z", "content": "

Thank you for clarifying this, I was frustrated after seeing these full scores!

", "rawMarkdown": "Thank you for clarifying this, I was frustrated after seeing these full scores!\n", "votes": 1 }, { "id": 1633020, "postDate": "2021-12-30T10:26:07.550Z", "content": "

UPD: 0.0258

", "rawMarkdown": "UPD: 0.0258", "votes": 2 }, { "id": 1626714, "postDate": "2021-12-23T06:28:04.333Z", "content": "

Let's add one more line to the top to make the discussion active. I also encourage other people to post their best scores, so we can somehow compete and have fun before the private test starts:) my current baseline has 0.0273 score.

", "rawMarkdown": "Let's add one more line to the top to make the discussion active. I also encourage other people to post their best scores, so we can somehow compete and have fun before the private test starts:) my current baseline has 0.0273 score.", "votes": 2, "replies": [ { "id": 1630596, "postDate": "2021-12-27T13:40:57.850Z", "content": "

UPD: 0.0318

", "rawMarkdown": "UPD: 0.0318" }, { "id": 1632804, "postDate": "2021-12-30T05:03:38.343Z", "content": "

UPD2: 0.0403

\n

Just want to make discussion active. Guys, why don’t you update your scores? I think it would be fun to compete here and to know approximately what is our position relative to other competitors.

", "rawMarkdown": "UPD2: 0.0403 \n\nJust want to make discussion active. Guys, why don’t you update your scores? I think it would be fun to compete here and to know approximately what is our position relative to other competitors.", "votes": 3 } ] }, { "id": 1602836, "postDate": "2021-12-02T06:55:47.857Z", "content": "

@mlconsult, my best score without overfitting, so far, 0.0326.

", "rawMarkdown": "@mlconsult, my best score without overfitting, so far, 0.0326.", "votes": 2, "replies": [ { "id": 1604548, "postDate": "2021-12-03T13:43:45.767Z", "content": "

Thanks - I added you to the list above

", "rawMarkdown": "Thanks - I added you to the list above", "votes": 1 } ] }, { "id": 1596481, "postDate": "2021-11-26T14:32:52.083Z", "content": "

MOVED TO TOP

", "rawMarkdown": "MOVED TO TOP", "replies": [ { "id": 1597070, "postDate": "2021-11-27T07:27:38.403Z", "content": "

All of them are overfitting, if not, they are loosing a great oportunity to become billionarie.

", "rawMarkdown": "All of them are overfitting, if not, they are loosing a great oportunity to become billionarie.", "votes": 14 }, { "id": 1597797, "postDate": "2021-11-27T22:57:27.333Z", "content": "

Agree, i think , the best one will be less than 0.5 possible 0.1

", "rawMarkdown": "Agree, i think , the best one will be less than 0.5 possible 0.1", "votes": 2 } ] }, { "id": 1671333, "postDate": "2022-02-01T11:28:05.603Z", "content": "

The public LB seems nonsense, what's the better metric. The only thread seems the private LB…

", "rawMarkdown": "The public LB seems nonsense, what's the better metric. The only thread seems the private LB..." }, { "id": 1668620, "postDate": "2022-01-29T20:07:13.973Z", "content": "

My submission failed best is 0.04
\nMy registered best is 0.03

", "rawMarkdown": "My submission failed best is 0.04\nMy registered best is 0.03", "replies": [ { "id": 1668961, "postDate": "2022-01-30T07:44:16.030Z", "content": "

Submission failed😂Are you having a problem of exceeding the time limit?

", "rawMarkdown": "Submission failed😂Are you having a problem of exceeding the time limit?" } ] }, { "id": 1668396, "postDate": "2022-01-29T15:23:48.480Z", "content": "

Have been anyone share a no leak notebook with 0.02+?

", "rawMarkdown": "Have been anyone share a no leak notebook with 0.02+?" }, { "id": 1663042, "postDate": "2022-01-24T18:26:22.420Z", "content": "

Any ideas and thoughts on what the score range is for a gold-prize zone?

", "rawMarkdown": "Any ideas and thoughts on what the score range is for a gold-prize zone?", "replies": [ { "id": 1663173, "postDate": "2022-01-24T20:42:36.573Z", "content": "

Probably 0.05 will be gold

", "rawMarkdown": "Probably 0.05 will be gold ", "votes": 1, "isDeleted": true }, { "id": 1664663, "postDate": "2022-01-26T06:38:48.783Z", "content": "

Ok so I'm maybe in the bronze zone with my 0.0345

", "rawMarkdown": "Ok so I'm maybe in the bronze zone with my 0.0345 " }, { "id": 1664858, "postDate": "2022-01-26T10:33:49.820Z", "content": "

Given how many people seem to be just forking and submitting the perfect-score overfitted public notebooks, I've got a feeling that a score >0 will be enough to get close to bronze 😂

", "rawMarkdown": "Given how many people seem to be just forking and submitting the perfect-score overfitted public notebooks, I've got a feeling that a score >0 will be enough to get close to bronze 😂", "votes": 1 }, { "id": 1664885, "postDate": "2022-01-26T11:13:29.900Z", "content": "

I haven't seen anyone claiming that he gets 0.05 without data leakage up till now😂 Even a score of 0.04 is rare……

", "rawMarkdown": "I haven't seen anyone claiming that he gets 0.05 without data leakage up till now😂 Even a score of 0.04 is rare......" }, { "id": 1664887, "postDate": "2022-01-26T11:17:03.080Z", "content": "

Also, I think getting 0.05 will be possible if I can construct more features. But time is another limitation here and I don't really believe that one can construct more than 100 features……

", "rawMarkdown": "Also, I think getting 0.05 will be possible if I can construct more features. But time is another limitation here and I don't really believe that one can construct more than 100 features......" }, { "id": 1664995, "postDate": "2022-01-26T13:23:55.380Z", "content": "

My score is without the leaked public/data in train time but it's so easy to overfit the charts nevertheless. Think I will use the two subs to one optimized and one more less optimized.

", "rawMarkdown": "My score is without the leaked public/data in train time but it's so easy to overfit the charts nevertheless. Think I will use the two subs to one optimized and one more less optimized." }, { "id": 1665087, "postDate": "2022-01-26T15:02:03.300Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1665132, "postDate": "2022-01-26T15:49:28.067Z", "content": "

When the contest is over, if you see gold above 0.03 they have done an insane job.. Numerai top score is 0.04, so if you think you beat that with just some TA features then you should quit the contest and run your model live, you'll make a lot more money 😁

", "rawMarkdown": "When the contest is over, if you see gold above 0.03 they have done an insane job.. Numerai top score is 0.04, so if you think you beat that with just some TA features then you should quit the contest and run your model live, you'll make a lot more money 😁 " }, { "id": 1665150, "postDate": "2022-01-26T16:05:03.770Z", "content": "

Thanks, great with some other external values to compare with, maybe ~0.01 score in the final is a good and a realistic value if the models works well, talking about my own creations.

", "rawMarkdown": "Thanks, great with some other external values to compare with, maybe ~0.01 score in the final is a good and a realistic value if the models works well, talking about my own creations." }, { "id": 1665912, "postDate": "2022-01-27T08:12:49.520Z", "content": "

I am indeed thinking about deploying my model to predict stocks. But a major problem is that a good model doesn't mean more money earned. For example, your model has a prediction accuracy of 0.8. Most of the time it's correct and you earn money, but just that 20% left will cause disasterous loss of money and your final profit will be negative if you don't have a pretty good risk controling model.

", "rawMarkdown": "I am indeed thinking about deploying my model to predict stocks. But a major problem is that a good model doesn't mean more money earned. For example, your model has a prediction accuracy of 0.8. Most of the time it's correct and you earn money, but just that 20% left will cause disasterous loss of money and your final profit will be negative if you don't have a pretty good risk controling model.", "votes": -2 }, { "id": 1665916, "postDate": "2022-01-27T08:15:35.993Z", "content": "

I also found that correlation is not a very good evaluation metric if you want to earn money, since it doesn't fit well with the logic of predicting price increase or decrease.

", "rawMarkdown": "I also found that correlation is not a very good evaluation metric if you want to earn money, since it doesn't fit well with the logic of predicting price increase or decrease." }, { "id": 1665977, "postDate": "2022-01-27T09:29:04.390Z", "content": "

How could you possibly lose money if you have 0.8 accuracy model… If you have consistent risk size its literally impossible. If you're changing your risk size each time such that a 20% loss can blow your account then ask yourself how you ended up with a 80% accurate future prediction in the first place 😂

\n

Correlation makes a lot of sense if you know how to use it, there are specific markets and strategies it would apply very well to.

\n

6 days left anyway good luck to all.

", "rawMarkdown": "How could you possibly lose money if you have 0.8 accuracy model... If you have consistent risk size its literally impossible. If you're changing your risk size each time such that a 20% loss can blow your account then ask yourself how you ended up with a 80% accurate future prediction in the first place 😂\n\nCorrelation makes a lot of sense if you know how to use it, there are specific markets and strategies it would apply very well to. \n\n6 days left anyway good luck to all.", "votes": 1 }, { "id": 1667111, "postDate": "2022-01-28T10:23:57.150Z", "content": "

I tested on the newly provided supplemental_train.csv and got a score of 0.035😂 That's a big drop from my original 0.0418…

", "rawMarkdown": "I tested on the newly provided supplemental_train.csv and got a score of 0.035😂 That's a big drop from my original 0.0418..." }, { "id": 1667266, "postDate": "2022-01-28T12:39:44.567Z", "content": "

0.035 and 0.0418, that's great, good work! Don't think many beat that scores.

", "rawMarkdown": "0.035 and 0.0418, that's great, good work! Don't think many beat that scores." }, { "id": 1670449, "postDate": "2022-01-31T14:35:56.290Z", "content": "

we've got two submissions though - I've chosen one where the score isn't that great, but it's very cheap to run, and so I'm sure it won't unexpectedly time out

", "rawMarkdown": "we've got two submissions though - I've chosen one where the score isn't that great, but it's very cheap to run, and so I'm sure it won't unexpectedly time out", "votes": 1 }, { "id": 1670453, "postDate": "2022-01-31T14:39:47.580Z", "content": "

Have done the same, selected 1 fast and 1 close to limit but better score.. Better to end up with worse score than no score 😬

", "rawMarkdown": "Have done the same, selected 1 fast and 1 close to limit but better score.. Better to end up with worse score than no score 😬", "votes": 1 } ] }, { "id": 1662408, "postDate": "2022-01-24T09:00:37.710Z", "content": "

0.0413 now.. no ensembling, can maybe squeeze out last bits in the last week but don't want to get too close to the 9hour limit..

\n

Evaluating on supplement data is closer to 0.03

", "rawMarkdown": "0.0413 now.. no ensembling, can maybe squeeze out last bits in the last week but don't want to get too close to the 9hour limit..\n\nEvaluating on supplement data is closer to 0.03", "replies": [ { "id": 1663000, "postDate": "2022-01-24T17:42:29.540Z", "content": "

What did you get using the new data in the supplemental training set?

", "rawMarkdown": "What did you get using the new data in the supplemental training set? ", "votes": 1 }, { "id": 1663380, "postDate": "2022-01-25T03:34:00.877Z", "content": "

Notice that as data change, performance would be different and ensembling might help to get a more stable score. Also I obtained some drop in the supplemental data…

", "rawMarkdown": "Notice that as data change, performance would be different and ensembling might help to get a more stable score. Also I obtained some drop in the supplemental data..." }, { "id": 1668963, "postDate": "2022-01-30T07:45:15.393Z", "content": "

Only 3 days to go and you are still improving your score😂

", "rawMarkdown": "Only 3 days to go and you are still improving your score😂" }, { "id": 1668964, "postDate": "2022-01-30T07:47:54.723Z", "content": "

To be honest my current major concern is the time limit…… Just a very simple tree model takes more than 7 hours to construct features and make predictions, let alone other complex models.

", "rawMarkdown": "To be honest my current major concern is the time limit...... Just a very simple tree model takes more than 7 hours to construct features and make predictions, let alone other complex models." }, { "id": 1670360, "postDate": "2022-01-31T13:00:56.690Z", "content": "

Yeah experimented with some different folding and added ensembling which got me the higher score on supp, though it is close to the edge on time limit too..

\n

Supp / Out of sample CV ranges from 0.0275 to 0.0328 in the ensemble so will be submitting that..

\n

3 months from here the market will be very different so not holding my breathe on these scores sticking till the end..

\n

Going to really suck if our submissions time out at some point..

", "rawMarkdown": "Yeah experimented with some different folding and added ensembling which got me the higher score on supp, though it is close to the edge on time limit too.. \n\nSupp / Out of sample CV ranges from 0.0275 to 0.0328 in the ensemble so will be submitting that..\n\n3 months from here the market will be very different so not holding my breathe on these scores sticking till the end..\n\nGoing to really suck if our submissions time out at some point.. " } ] }, { "id": 1660049, "postDate": "2022-01-22T10:54:24.387Z", "content": "

Thx for explanation. Did't know about leaked data.

", "rawMarkdown": "Thx for explanation. Did't know about leaked data." }, { "id": 1657275, "postDate": "2022-01-20T04:28:17.447Z", "content": "

Mine is 0.0292.

", "rawMarkdown": "Mine is 0.0292." }, { "id": 1656367, "postDate": "2022-01-19T09:53:33.877Z", "content": "

hello @mlconsult

\n

how to find it? \"Top Scores without training on future data:\"

\n

My score used leak data. real score is 0.01

", "rawMarkdown": "hello @mlconsult \n\nhow to find it? \"Top Scores without training on future data:\"\n\nMy score used leak data. real score is 0.01" }, { "id": 1653043, "postDate": "2022-01-17T07:33:24.920Z", "content": "

How do you measure the correlation timewise?
\nI mean, are you calculating the score based on your predictions for the 3 months as a whole, or as host pointed out: evaluate your predicted targets each-two-weeks, (so 6 independent correlation calculations) and average them?

", "rawMarkdown": "How do you measure the correlation timewise?\nI mean, are you calculating the score based on your predictions for the 3 months as a whole, or as host pointed out: evaluate your predicted targets each-two-weeks, (so 6 independent correlation calculations) and average them?" }, { "id": 1650542, "postDate": "2022-01-15T05:30:24.283Z", "content": "

Mine is 0.041 based on sample reweighting, technical indicators and ensembling.

", "rawMarkdown": "Mine is 0.041 based on sample reweighting, technical indicators and ensembling.", "replies": [ { "id": 1654305, "postDate": "2022-01-18T11:33:38.420Z", "content": "

sample reweighting? are you reweighting the assets?

", "rawMarkdown": "sample reweighting? are you reweighting the assets?" }, { "id": 1654438, "postDate": "2022-01-18T14:05:50Z", "content": "

Actually I don't train one model for all assets but train one for each asset. Both memory and performance are problems for the one-for-all model. However I currently have a submission timeout error for the 0.041 version… I can only delete 3/4 of my features and obtain 0.037 instead😂

", "rawMarkdown": "Actually I don't train one model for all assets but train one for each asset. Both memory and performance are problems for the one-for-all model. However I currently have a submission timeout error for the 0.041 version... I can only delete 3/4 of my features and obtain 0.037 instead😂", "votes": 1 }, { "id": 1654897, "postDate": "2022-01-18T23:19:40.603Z", "content": "

update:online score 0.0418

", "rawMarkdown": "update:online score 0.0418" }, { "id": 1654947, "postDate": "2022-01-19T01:26:34.893Z", "content": "

the new sup data has lots of error and not trust worthy, I got 0.05167, but I do not believe in it at all, since it only got 0.02 on old sup data.

", "rawMarkdown": "the new sup data has lots of error and not trust worthy, I got 0.05167, but I do not believe in it at all, since it only got 0.02 on old sup data." }, { "id": 1656486, "postDate": "2022-01-19T11:19:02.070Z", "content": "

You test your model on the supplemental_train.csv? This leaderboard and my result are all based on data used for public leaderboard evaluation. Maybe I should evaluate my model on the updated sup data as well.

", "rawMarkdown": "You test your model on the supplemental_train.csv? This leaderboard and my result are all based on data used for public leaderboard evaluation. Maybe I should evaluate my model on the updated sup data as well." }, { "id": 1657153, "postDate": "2022-01-20T00:37:03.813Z", "content": "

I thought the LB use the sup data, maybe I am wrong

", "rawMarkdown": "I thought the LB use the sup data, maybe I am wrong" }, { "id": 1657487, "postDate": "2022-01-20T08:17:41.833Z", "content": "

The problem is you shouldn't use all data in train.csv since test data for public leaderboard evaluation are included. But evaluating on supplemental_train.csv is a good idea, as long as all the errors mentioned in other discussion topics are corrected.

", "rawMarkdown": "The problem is you shouldn't use all data in train.csv since test data for public leaderboard evaluation are included. But evaluating on supplemental_train.csv is a good idea, as long as all the errors mentioned in other discussion topics are corrected." }, { "id": 1657835, "postDate": "2022-01-20T13:54:35.140Z", "content": "

I just finished testing on supplemental_train.csv and got 0.0403. It seems that all errors in the newly provided data have been corrected.

", "rawMarkdown": "I just finished testing on supplemental_train.csv and got 0.0403. It seems that all errors in the newly provided data have been corrected." }, { "id": 1664843, "postDate": "2022-01-26T10:11:11.113Z", "content": "

Can you show us how you tested against the supplement data? 0.04 on the supplement stuff seems pretty high considering you said you also got that on the public lb?

", "rawMarkdown": "Can you show us how you tested against the supplement data? 0.04 on the supplement stuff seems pretty high considering you said you also got that on the public lb?" }, { "id": 1664880, "postDate": "2022-01-26T11:05:53.783Z", "content": "

There is a local api: https://www.kaggle.com/jagofc/local-api-emulator.

", "rawMarkdown": "There is a local api: https://www.kaggle.com/jagofc/local-api-emulator.", "votes": 1 } ] }, { "id": 1608164, "postDate": "2021-12-06T09:12:13.993Z", "content": "

overfitting

", "rawMarkdown": "overfitting" }, { "id": 1607947, "postDate": "2021-12-06T06:40:00.310Z", "content": "

Had asked this question in a different post but see that the answer in now quite clear. Many thanks for this. St J

", "rawMarkdown": "Had asked this question in a different post but see that the answer in now quite clear. Many thanks for this. St J" }, { "id": 1607118, "postDate": "2021-12-05T15:51:42.047Z", "content": "

My best score without overfitting, so far, 0.0226.

", "rawMarkdown": "My best score without overfitting, so far, 0.0226." }, { "id": 1600672, "postDate": "2021-11-30T16:06:08.267Z", "content": "

I should not have used the term over-fititng -> what I was getting at is avoiding looking up values in the data to get a 1.00 or .9999, but point taken about everyone over-fitting. For those of you getting .0248 or .0361 - how are you removing the future data?

", "rawMarkdown": "I should not have used the term over-fititng -> what I was getting at is avoiding looking up values in the data to get a 1.00 or .9999, but point taken about everyone over-fitting. For those of you getting .0248 or .0361 - how are you removing the future data?" }, { "id": 1657706, "postDate": "2022-01-20T12:02:33.693Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1654272, "postDate": "2022-01-18T11:13:47.007Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1601530, "author_name": "Aleksandr Larko", "author_url": "", "post_date": "2021-12-01T11:43:08.153000", "content": "
\n

For those of you getting .0248 or .0361 - how are you removing the future data?

\n
\n

Like this

\n

df = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')

\n

def test_train_split(df):

\n
df[\"datetime\"] = pd.to_datetime(df[\"timestamp\"], unit=\"s\")\n\nX_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].drop(['Target','datetime'], axis=1)\n\ny_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].Target\n\nX_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].drop(['Target','datetime'], axis=1)\n\ny_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].Target\n\nreturn X_train, y_train, X_test, y_test\n
\n

X_train, y_train, X_test, y_test = test_train_split(df)

", "votes": 8, "replies": [ { "id": 1601921, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-12-01T16:39:25.713000", "content": "

Thanks Alexander

", "votes": 0, "replies": [] } ] }, { "id": 1597921, "author_name": "Aleksandr Larko", "author_url": "", "post_date": "2021-11-28T04:07:00.460000", "content": "

All of them are overfitting. (training data includes test data)
\nMy best score - 0.0248 !!!

", "votes": 8, "replies": [ { "id": 1598115, "author_name": "", "author_url": "", "post_date": "2021-11-28T09:07:04.717000", "content": "

100% Agree.
\nMy best score (that I trust) is 0.0361

", "votes": 5, "replies": [] }, { "id": 1598343, "author_name": "Vladimir Shebunyaev", "author_url": "", "post_date": "2021-11-28T12:13:07.953000", "content": "

As I remember from one video, when one guy shows the accuracy for Market Prediction model as 52% all the audience was laughing, like a probability with a coin, but he explain,that even this additional 2% can helps companies make good money, and it shows that it is not a random

", "votes": 4, "replies": [] }, { "id": 1599928, "author_name": "Ryan", "author_url": "", "post_date": "2021-11-30T00:30:03.367000", "content": "

This is absolutely correct. Imagine that you are sitting inside a Goldman Sachs trading desk and are asked to come up with a crypto forecast model. If your model explains 52% of the data, you're a rockstar. Values around 20% to 30% are much more common. I'm assuming an ARIMA model with possibly some exogenous data.

", "votes": 2, "replies": [] }, { "id": 1604950, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-12-03T21:47:24.270000", "content": "

Here is a notebook that removes the future price data and scores a .0417

\n

https://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline

", "votes": 0, "replies": [] }, { "id": 1666024, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-01-27T10:32:25.563000", "content": "

@mlconsult are you sure it removes it? It says

\n
REMOVE_LB_TEST_OVERLAPPING_DATA = False\n
", "votes": 0, "replies": [] } ] }, { "id": 1605201, "author_name": "Julián Peller (dataista0)", "author_url": "", "post_date": "2021-12-04T04:03:41.097000", "content": "

Hello,
\nJust a clarification:
\nThe notebook https://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline that you mention is just a copy of mine and it achieves 0.0417 with the \"leaky\" data. You can verify that the version with the score says:
\nREMOVE_LB_TEST_OVERLAPPING_DATA = False

\n

So the result of 0.0417 is leaky.

\n

The simple LGBM of that notebook, without the leakage, is presented in this other notebook:
\n🪙💲 Proposal for a meaningful LB + Strict LGBM and achieves a LB of just 0.018.

\n

I also propose a mechanism (\"Strict LB\") so we can all do the same to avoid the leakage in the same way, to obtain comparable results.

\n

The idea is the most direct one: Only keep data from before 2021-06-13 00:00:00. and the implementation is straightforward:

\n
import pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df\n
", "votes": 5, "replies": [ { "id": 1605743, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-12-04T14:04:55.517000", "content": "

thanks @julian3833

", "votes": 1, "replies": [] } ] }, { "id": 1645211, "author_name": "John Mitchell", "author_url": "", "post_date": "2022-01-10T18:53:42.927000", "content": "

It would not surprise me at all if the private [edit] LB for this competition were to reveal complete carnage. I expect to see plenty of timed-out and failed submissions, and many negative scores. Maybe a genuine 0.02 prediction will be in the running for a medal?

", "votes": 3, "replies": [ { "id": 1645392, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-01-10T21:43:30.493000", "content": "
\n

failed submissions

\n
\n

yeah, I too see that coming

", "votes": 2, "replies": [] } ] }, { "id": 1652681, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-01-16T22:10:16.410000", "content": "

Hello @mlconsult, thanks for the information. I usually use the LB as a reference to measure my model's improvements over time, and because now it is broken, this post will be my reference.

", "votes": 1, "replies": [] }, { "id": 1644805, "author_name": "Saumil Agrawal", "author_url": "", "post_date": "2022-01-10T14:01:27.470000", "content": "

Mine is 0.0247

", "votes": 1, "replies": [] }, { "id": 1642942, "author_name": "Michal Bogacz", "author_url": "", "post_date": "2022-01-08T20:23:54.820000", "content": "

My best so far is 0.0221

", "votes": 1, "replies": [] }, { "id": 1642367, "author_name": "Mr. Li", "author_url": "", "post_date": "2022-01-08T09:48:02.630000", "content": "

Good post, I got similar results when removing training on future data.

", "votes": 1, "replies": [] }, { "id": 1641300, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-07T09:29:37.677000", "content": "

May I ask how the scores were obtained on this leaderboard? How should I tune the hyperparameters? On the test set(which contains data on the public leaderboard) or on another held-out dataset?

", "votes": 1, "replies": [] }, { "id": 1633375, "author_name": "oliver", "author_url": "", "post_date": "2021-12-30T17:58:58.733000", "content": "

ah thanks for this! it's nice to have a place to really compare myself too.

", "votes": 1, "replies": [] }, { "id": 1629028, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2021-12-25T15:29:24.047000", "content": "

Hi All, I have been giving some thought to both the issue relating to a fair score before going into the live valuation part of the process and agree that it makes sense to slice the data as per:

\n

import pandas as pd

\n

def read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):
\n df = pd.read_csv(file_name)
\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
\n df = df[df['datetime'] < '2021-06-13 00:00:00']
\n return df

\n

Not only will this uphold the normal lb process but ensure that the integrity of the model is inplace, however it strikes me that this may lead to a large chunk of missing data once committee in this way, especially if this is chosen as your entry. I am assuming that the dataset on inception of the live valuation will be complete to the 2nd of Feb 2022 ('/kaggle/input/g-research-crypto-forecasting/train.csv'), if not this presents problem for any lagged features being utilised. Has anyone else thought about this or have any clarity on the shape of the final dataset?

", "votes": 1, "replies": [] }, { "id": 1601397, "author_name": "Ding", "author_url": "", "post_date": "2021-12-01T08:45:55.277000", "content": "

Validation set will not be used if you want look high in public score.

", "votes": 1, "replies": [] }, { "id": 1600191, "author_name": "akram001", "author_url": "", "post_date": "2021-11-30T07:40:51.787000", "content": "

Thank you for clarifying this, I was frustrated after seeing these full scores!

", "votes": 1, "replies": [] }, { "id": 1633020, "author_name": "Aleksandr Larko", "author_url": "", "post_date": "2021-12-30T10:26:07.550000", "content": "

UPD: 0.0258

", "votes": 2, "replies": [] }, { "id": 1626714, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2021-12-23T06:28:04.333000", "content": "

Let's add one more line to the top to make the discussion active. I also encourage other people to post their best scores, so we can somehow compete and have fun before the private test starts:) my current baseline has 0.0273 score.

", "votes": 2, "replies": [ { "id": 1630596, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2021-12-27T13:40:57.850000", "content": "

UPD: 0.0318

", "votes": 0, "replies": [] }, { "id": 1632804, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2021-12-30T05:03:38.343000", "content": "

UPD2: 0.0403

\n

Just want to make discussion active. Guys, why don’t you update your scores? I think it would be fun to compete here and to know approximately what is our position relative to other competitors.

", "votes": 3, "replies": [] } ] }, { "id": 1602836, "author_name": "agenlu", "author_url": "", "post_date": "2021-12-02T06:55:47.857000", "content": "

@mlconsult, my best score without overfitting, so far, 0.0326.

", "votes": 2, "replies": [ { "id": 1604548, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-12-03T13:43:45.767000", "content": "

Thanks - I added you to the list above

", "votes": 1, "replies": [] } ] }, { "id": 1596481, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-11-26T14:32:52.083000", "content": "

MOVED TO TOP

", "votes": 0, "replies": [ { "id": 1597070, "author_name": "agenlu", "author_url": "", "post_date": "2021-11-27T07:27:38.403000", "content": "

All of them are overfitting, if not, they are loosing a great oportunity to become billionarie.

", "votes": 14, "replies": [] }, { "id": 1597797, "author_name": "Vladimir Shebunyaev", "author_url": "", "post_date": "2021-11-27T22:57:27.333000", "content": "

Agree, i think , the best one will be less than 0.5 possible 0.1

", "votes": 2, "replies": [] } ] }, { "id": 1671333, "author_name": "水彩不好画", "author_url": "", "post_date": "2022-02-01T11:28:05.603000", "content": "

The public LB seems nonsense, what's the better metric. The only thread seems the private LB…

", "votes": 0, "replies": [] }, { "id": 1668620, "author_name": "Ding", "author_url": "", "post_date": "2022-01-29T20:07:13.973000", "content": "

My submission failed best is 0.04
\nMy registered best is 0.03

", "votes": 0, "replies": [ { "id": 1668961, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-30T07:44:16.030000", "content": "

Submission failed😂Are you having a problem of exceeding the time limit?

", "votes": 0, "replies": [] } ] }, { "id": 1668396, "author_name": "Xie Zejian", "author_url": "", "post_date": "2022-01-29T15:23:48.480000", "content": "

Have been anyone share a no leak notebook with 0.02+?

", "votes": 0, "replies": [] }, { "id": 1663042, "author_name": "Kirderf", "author_url": "", "post_date": "2022-01-24T18:26:22.420000", "content": "

Any ideas and thoughts on what the score range is for a gold-prize zone?

", "votes": 0, "replies": [ { "id": 1663173, "author_name": "", "author_url": "", "post_date": "2022-01-24T20:42:36.573000", "content": "

Probably 0.05 will be gold

", "votes": 1, "replies": [] }, { "id": 1664663, "author_name": "Kirderf", "author_url": "", "post_date": "2022-01-26T06:38:48.783000", "content": "

Ok so I'm maybe in the bronze zone with my 0.0345

", "votes": 0, "replies": [] }, { "id": 1664858, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-01-26T10:33:49.820000", "content": "

Given how many people seem to be just forking and submitting the perfect-score overfitted public notebooks, I've got a feeling that a score >0 will be enough to get close to bronze 😂

", "votes": 1, "replies": [] }, { "id": 1664885, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-26T11:13:29.900000", "content": "

I haven't seen anyone claiming that he gets 0.05 without data leakage up till now😂 Even a score of 0.04 is rare……

", "votes": 0, "replies": [] }, { "id": 1664887, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-26T11:17:03.080000", "content": "

Also, I think getting 0.05 will be possible if I can construct more features. But time is another limitation here and I don't really believe that one can construct more than 100 features……

", "votes": 0, "replies": [] }, { "id": 1664995, "author_name": "Kirderf", "author_url": "", "post_date": "2022-01-26T13:23:55.380000", "content": "

My score is without the leaked public/data in train time but it's so easy to overfit the charts nevertheless. Think I will use the two subs to one optimized and one more less optimized.

", "votes": 0, "replies": [] }, { "id": 1665087, "author_name": "", "author_url": "", "post_date": "2022-01-26T15:02:03.300000", "content": "", "votes": 0, "replies": [] }, { "id": 1665132, "author_name": "JM", "author_url": "", "post_date": "2022-01-26T15:49:28.067000", "content": "

When the contest is over, if you see gold above 0.03 they have done an insane job.. Numerai top score is 0.04, so if you think you beat that with just some TA features then you should quit the contest and run your model live, you'll make a lot more money 😁

", "votes": 0, "replies": [] }, { "id": 1665150, "author_name": "Kirderf", "author_url": "", "post_date": "2022-01-26T16:05:03.770000", "content": "

Thanks, great with some other external values to compare with, maybe ~0.01 score in the final is a good and a realistic value if the models works well, talking about my own creations.

", "votes": 0, "replies": [] }, { "id": 1665912, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-27T08:12:49.520000", "content": "

I am indeed thinking about deploying my model to predict stocks. But a major problem is that a good model doesn't mean more money earned. For example, your model has a prediction accuracy of 0.8. Most of the time it's correct and you earn money, but just that 20% left will cause disasterous loss of money and your final profit will be negative if you don't have a pretty good risk controling model.

", "votes": -2, "replies": [] }, { "id": 1665916, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-27T08:15:35.993000", "content": "

I also found that correlation is not a very good evaluation metric if you want to earn money, since it doesn't fit well with the logic of predicting price increase or decrease.

", "votes": 0, "replies": [] }, { "id": 1665977, "author_name": "JM", "author_url": "", "post_date": "2022-01-27T09:29:04.390000", "content": "

How could you possibly lose money if you have 0.8 accuracy model… If you have consistent risk size its literally impossible. If you're changing your risk size each time such that a 20% loss can blow your account then ask yourself how you ended up with a 80% accurate future prediction in the first place 😂

\n

Correlation makes a lot of sense if you know how to use it, there are specific markets and strategies it would apply very well to.

\n

6 days left anyway good luck to all.

", "votes": 1, "replies": [] }, { "id": 1667111, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-28T10:23:57.150000", "content": "

I tested on the newly provided supplemental_train.csv and got a score of 0.035😂 That's a big drop from my original 0.0418…

", "votes": 0, "replies": [] }, { "id": 1667266, "author_name": "Kirderf", "author_url": "", "post_date": "2022-01-28T12:39:44.567000", "content": "

0.035 and 0.0418, that's great, good work! Don't think many beat that scores.

", "votes": 0, "replies": [] }, { "id": 1670449, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-01-31T14:35:56.290000", "content": "

we've got two submissions though - I've chosen one where the score isn't that great, but it's very cheap to run, and so I'm sure it won't unexpectedly time out

", "votes": 1, "replies": [] }, { "id": 1670453, "author_name": "JM", "author_url": "", "post_date": "2022-01-31T14:39:47.580000", "content": "

Have done the same, selected 1 fast and 1 close to limit but better score.. Better to end up with worse score than no score 😬

", "votes": 1, "replies": [] } ] }, { "id": 1662408, "author_name": "JM", "author_url": "", "post_date": "2022-01-24T09:00:37.710000", "content": "

0.0413 now.. no ensembling, can maybe squeeze out last bits in the last week but don't want to get too close to the 9hour limit..

\n

Evaluating on supplement data is closer to 0.03

", "votes": 0, "replies": [ { "id": 1663000, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-01-24T17:42:29.540000", "content": "

What did you get using the new data in the supplemental training set?

", "votes": 1, "replies": [] }, { "id": 1663380, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-25T03:34:00.877000", "content": "

Notice that as data change, performance would be different and ensembling might help to get a more stable score. Also I obtained some drop in the supplemental data…

", "votes": 0, "replies": [] }, { "id": 1668963, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-30T07:45:15.393000", "content": "

Only 3 days to go and you are still improving your score😂

", "votes": 0, "replies": [] }, { "id": 1668964, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-30T07:47:54.723000", "content": "

To be honest my current major concern is the time limit…… Just a very simple tree model takes more than 7 hours to construct features and make predictions, let alone other complex models.

", "votes": 0, "replies": [] }, { "id": 1670360, "author_name": "JM", "author_url": "", "post_date": "2022-01-31T13:00:56.690000", "content": "

Yeah experimented with some different folding and added ensembling which got me the higher score on supp, though it is close to the edge on time limit too..

\n

Supp / Out of sample CV ranges from 0.0275 to 0.0328 in the ensemble so will be submitting that..

\n

3 months from here the market will be very different so not holding my breathe on these scores sticking till the end..

\n

Going to really suck if our submissions time out at some point..

", "votes": 0, "replies": [] } ] }, { "id": 1660049, "author_name": "Platon Fedorov", "author_url": "", "post_date": "2022-01-22T10:54:24.387000", "content": "

Thx for explanation. Did't know about leaked data.

", "votes": 0, "replies": [] }, { "id": 1657275, "author_name": "Willson Santoso", "author_url": "", "post_date": "2022-01-20T04:28:17.447000", "content": "

Mine is 0.0292.

", "votes": 0, "replies": [] }, { "id": 1656367, "author_name": "tensor choko", "author_url": "", "post_date": "2022-01-19T09:53:33.877000", "content": "

hello @mlconsult

\n

how to find it? \"Top Scores without training on future data:\"

\n

My score used leak data. real score is 0.01

", "votes": 0, "replies": [] }, { "id": 1653043, "author_name": "GUNER", "author_url": "", "post_date": "2022-01-17T07:33:24.920000", "content": "

How do you measure the correlation timewise?
\nI mean, are you calculating the score based on your predictions for the 3 months as a whole, or as host pointed out: evaluate your predicted targets each-two-weeks, (so 6 independent correlation calculations) and average them?

", "votes": 0, "replies": [] }, { "id": 1650542, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-15T05:30:24.283000", "content": "

Mine is 0.041 based on sample reweighting, technical indicators and ensembling.

", "votes": 0, "replies": [ { "id": 1654305, "author_name": "GUNER", "author_url": "", "post_date": "2022-01-18T11:33:38.420000", "content": "

sample reweighting? are you reweighting the assets?

", "votes": 0, "replies": [] }, { "id": 1654438, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-18T14:05:50", "content": "

Actually I don't train one model for all assets but train one for each asset. Both memory and performance are problems for the one-for-all model. However I currently have a submission timeout error for the 0.041 version… I can only delete 3/4 of my features and obtain 0.037 instead😂

", "votes": 1, "replies": [] }, { "id": 1654897, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-18T23:19:40.603000", "content": "

update:online score 0.0418

", "votes": 0, "replies": [] }, { "id": 1654947, "author_name": "liyang.chen", "author_url": "", "post_date": "2022-01-19T01:26:34.893000", "content": "

the new sup data has lots of error and not trust worthy, I got 0.05167, but I do not believe in it at all, since it only got 0.02 on old sup data.

", "votes": 0, "replies": [] }, { "id": 1656486, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-19T11:19:02.070000", "content": "

You test your model on the supplemental_train.csv? This leaderboard and my result are all based on data used for public leaderboard evaluation. Maybe I should evaluate my model on the updated sup data as well.

", "votes": 0, "replies": [] }, { "id": 1657153, "author_name": "liyang.chen", "author_url": "", "post_date": "2022-01-20T00:37:03.813000", "content": "

I thought the LB use the sup data, maybe I am wrong

", "votes": 0, "replies": [] }, { "id": 1657487, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-20T08:17:41.833000", "content": "

The problem is you shouldn't use all data in train.csv since test data for public leaderboard evaluation are included. But evaluating on supplemental_train.csv is a good idea, as long as all the errors mentioned in other discussion topics are corrected.

", "votes": 0, "replies": [] }, { "id": 1657835, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-20T13:54:35.140000", "content": "

I just finished testing on supplemental_train.csv and got 0.0403. It seems that all errors in the newly provided data have been corrected.

", "votes": 0, "replies": [] }, { "id": 1664843, "author_name": "JM", "author_url": "", "post_date": "2022-01-26T10:11:11.113000", "content": "

Can you show us how you tested against the supplement data? 0.04 on the supplement stuff seems pretty high considering you said you also got that on the public lb?

", "votes": 0, "replies": [] }, { "id": 1664880, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-01-26T11:05:53.783000", "content": "

There is a local api: https://www.kaggle.com/jagofc/local-api-emulator.

", "votes": 1, "replies": [] } ] }, { "id": 1608164, "author_name": "hongjf", "author_url": "", "post_date": "2021-12-06T09:12:13.993000", "content": "

overfitting

", "votes": 0, "replies": [] }, { "id": 1607947, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2021-12-06T06:40:00.310000", "content": "

Had asked this question in a different post but see that the answer in now quite clear. Many thanks for this. St J

", "votes": 0, "replies": [] }, { "id": 1607118, "author_name": "", "author_url": "", "post_date": "2021-12-05T15:51:42.047000", "content": "", "votes": 0, "replies": [] }, { "id": 1600672, "author_name": "", "author_url": "", "post_date": "2021-11-30T16:06:08.267000", "content": "", "votes": 0, "replies": [] }, { "id": 1657706, "author_name": "", "author_url": "", "post_date": "2022-01-20T12:02:33.693000", "content": "", "votes": 0, "replies": [] }, { "id": 1654272, "author_name": "", "author_url": "", "post_date": "2022-01-18T11:13:47.007000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1593286": "As a user observes: \"THE PUBLIC LEADER BOARD FOR THIS COMPETITION IS NOT MEANINGFUL \"\nAs many have pointed out, the LB scores are useless because people are intentionally over-fitting using future data.\n\n**Top Scores without training on future data:**\n@alankabisov UPD2: 0.0403 \n@yamqwe 0.0361\n@agenlu 0.0326\n@alexxanderlarko UPD: 0.0258\n@saveriomazza 0.0226\n\nsee the function below on how to remove the data - @alexxanderlarko was nice enough to post\n\nI am working on a notebook removing the future data and will post my score.\n\nThanks,", "1601530": "> For those of you getting .0248 or .0361 - how are you removing the future data?\n\nLike this\n\ndf = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')\n\ndef test_train_split(df):\n\n df[\"datetime\"] = pd.to_datetime(df[\"timestamp\"], unit=\"s\")\n\n X_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].drop(['Target','datetime'], axis=1)\n\n y_train = df[df['datetime'] <= \"2021-06-13 00:00:00\"].Target\n\n X_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].drop(['Target','datetime'], axis=1)\n\n y_test = df[df['datetime'] > \"2021-06-13 00:00:00\"].iloc[:-1].Target\n\n return X_train, y_train, X_test, y_test\n\nX_train, y_train, X_test, y_test = test_train_split(df)", "1597921": "All of them are overfitting. (training data includes test data)\nMy best score - 0.0248 !!!", "1605201": "Hello,\nJust a clarification:\nThe notebook https://www.kaggle.com/yliu27/g-research-starter-lgbm-pipeline that you mention is just a copy of [mine](https://www.kaggle.com/julian3833/g-research-starter-lgbm-pipeline) and it achieves `0.0417` **with** the \"leaky\" data. You can verify that the version with the score says:\n`REMOVE_LB_TEST_OVERLAPPING_DATA = False`\n\nSo the result of `0.0417` is leaky.\n\n\nThe simple LGBM of that notebook, without the leakage, is presented in this other notebook:\n[🪙💲 Proposal for a meaningful LB + Strict LGBM](https://www.kaggle.com/julian3833/proposal-for-a-meaningful-lb-strict-lgbm?scriptVersionId=80421622) and achieves a LB of just `0.018`.\n\nI also propose a mechanism (\"Strict LB\") so we can all do the same to avoid the leakage in the same way, to obtain comparable results.\n\nThe idea is the most direct one: **Only keep data from before 2021-06-13 00:00:00.** and the implementation is straightforward:\n\n```python\nimport pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n```", "1645211": "It would not surprise me at all if the private [edit] LB for this competition were to reveal complete carnage. I expect to see plenty of timed-out and failed submissions, and many negative scores. Maybe a genuine 0.02 prediction will be in the running for a medal?", "1652681": "Hello @mlconsult, thanks for the information. I usually use the LB as a reference to measure my model's improvements over time, and because now it is broken, this post will be my reference. ", "1644805": "Mine is 0.0247 \n", "1642942": "My best so far is 0.0221", "1642367": "Good post, I got similar results when removing training on future data.", "1641300": "May I ask how the scores were obtained on this leaderboard? How should I tune the hyperparameters? On the test set(which contains data on the public leaderboard) or on another held-out dataset?", "1633375": "ah thanks for this! it's nice to have a place to really compare myself too.", "1629028": "Hi All, I have been giving some thought to both the issue relating to a fair score before going into the live valuation part of the process and agree that it makes sense to slice the data as per:\n\nimport pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n\nNot only will this uphold the normal lb process but ensure that the integrity of the model is inplace, however it strikes me that this may lead to a large chunk of missing data once committee in this way, especially if this is chosen as your entry. I am assuming that the dataset on inception of the live valuation will be complete to the 2nd of Feb 2022 *('/kaggle/input/g-research-crypto-forecasting/train.csv')*, if not this presents problem for any lagged features being utilised. Has anyone else thought about this or have any clarity on the shape of the final dataset?\n\n", "1601397": "Validation set will not be used if you want look high in public score.", "1600191": "Thank you for clarifying this, I was frustrated after seeing these full scores!\n", "1633020": "UPD: 0.0258", "1626714": "Let's add one more line to the top to make the discussion active. I also encourage other people to post their best scores, so we can somehow compete and have fun before the private test starts:) my current baseline has 0.0273 score.", "1602836": "@mlconsult, my best score without overfitting, so far, 0.0326.", "1596481": "MOVED TO TOP", "1671333": "The public LB seems nonsense, what's the better metric. The only thread seems the private LB...", "1668620": "My submission failed best is 0.04\nMy registered best is 0.03", "1668396": "Have been anyone share a no leak notebook with 0.02+?", "1663042": "Any ideas and thoughts on what the score range is for a gold-prize zone?", "1662408": "0.0413 now.. no ensembling, can maybe squeeze out last bits in the last week but don't want to get too close to the 9hour limit..\n\nEvaluating on supplement data is closer to 0.03", "1660049": "Thx for explanation. Did't know about leaked data.", "1657275": "Mine is 0.0292.", "1656367": "hello @mlconsult \n\nhow to find it? \"Top Scores without training on future data:\"\n\nMy score used leak data. real score is 0.01", "1653043": "How do you measure the correlation timewise?\nI mean, are you calculating the score based on your predictions for the 3 months as a whole, or as host pointed out: evaluate your predicted targets each-two-weeks, (so 6 independent correlation calculations) and average them?", "1650542": "Mine is 0.041 based on sample reweighting, technical indicators and ensembling.", "1608164": "overfitting", "1607947": "Had asked this question in a different post but see that the answer in now quite clear. Many thanks for this. St J", "1607118": "My best score without overfitting, so far, 0.0226.", "1600672": "I should not have used the term over-fititng -> what I was getting at is avoiding looking up values in the data to get a 1.00 or .9999, but point taken about everyone over-fitting. For those of you getting .0248 or .0361 - how are you removing the future data?", "1657706": "", "1654272": "" } }