{ "id": 310718, "title": "Second score update posted", "url": "/competitions/g-research-crypto-forecasting/discussion/310718", "author_name": "Sohier Dane", "post_date": "2022-03-02T18:14:51.445000", "votes": 13, "comment_count": 48, "views": 0, "content": "

I've just published the second score update. It may take several minutes for all entries to show up on the leaderboard.

", "messages": [ { "id": 1710077, "postDate": "2022-03-02T18:14:51.447Z", "content": "

I've just published the second score update. It may take several minutes for all entries to show up on the leaderboard.

", "rawMarkdown": "I've just published the second score update. It may take several minutes for all entries to show up on the leaderboard.", "votes": 13 }, { "id": 1710091, "postDate": "2022-03-02T18:32:30.763Z", "content": "

Some teams are still only having one submission. Is there any clue on this issue? Thanks. @sohier

", "rawMarkdown": "Some teams are still only having one submission. Is there any clue on this issue? Thanks. @sohier ", "votes": 3, "replies": [ { "id": 1710106, "postDate": "2022-03-02T18:50:31.027Z", "content": "

@sohier Could you kindly double-checked the input files?

\n

Because I believe our two submissions worked fine in the initial submission. And we already added a lot of robustness logics. I will appreciate if you can double-check the input file because it is very hard for the players to figure out what is happening for a unknown dataset.

", "rawMarkdown": "@sohier Could you kindly double-checked the input files? \n\nBecause I believe our two submissions worked fine in the initial submission. And we already added a lot of robustness logics. I will appreciate if you can double-check the input file because it is very hard for the players to figure out what is happening for a unknown dataset.", "votes": 4 }, { "id": 1710247, "postDate": "2022-03-02T20:46:49.617Z", "content": "

Some teams have no submissions ;) @sohier

", "rawMarkdown": "Some teams have no submissions ;) @sohier ", "votes": 4 }, { "id": 1711187, "postDate": "2022-03-03T17:34:33.330Z", "content": "

@sohier Can you please double-check and make sure the format of the input data is not changed? Our team submitted two notebooks. The main difference is that one of the submissions uses the suplemental_train.csv. The one using the suplemental_train.csv runs well in the public leaderboard but fails in the private leaderboard.

\n

Similar issues have been reported in the following posts, but no response is given until now.

\n

https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308300
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308230

", "rawMarkdown": "@sohier Can you please double-check and make sure the format of the input data is not changed? Our team submitted two notebooks. The main difference is that one of the submissions uses the suplemental_train.csv. The one using the suplemental_train.csv runs well in the public leaderboard but fails in the private leaderboard. \n\nSimilar issues have been reported in the following posts, but no response is given until now.\n\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308300\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308230", "votes": 2 }, { "id": 1711203, "postDate": "2022-03-03T17:59:42.250Z", "content": "

The overall failure rate across all of the submissions for this iteration was roughly 1%, which is in line with what we expect for this kind of rescore. Unfortunately we aren't able to provide support or details as to what might have happened with individual submissions.

", "rawMarkdown": "The overall failure rate across all of the submissions for this iteration was roughly 1%, which is in line with what we expect for this kind of rescore. Unfortunately we aren't able to provide support or details as to what might have happened with individual submissions.", "votes": -4 }, { "id": 1711206, "postDate": "2022-03-03T18:05:41.010Z", "content": "

Can you release the supplemental data update so we can see for ourselves what went wrong? Just getting 1% of failures and saying \"yep looks right\" isn't very scientific imo..

\n

There were reports when the last supplemental data got swapped out a lot of peoples notebooks said \"asset_details.csv not found\", how can we be sure this has not happened again?

\n

These errors have caused a lot of noise, I think at least updating the supplemental train will let a large % of people see what went wrong in their notebooks and offer some closure.

", "rawMarkdown": "Can you release the supplemental data update so we can see for ourselves what went wrong? Just getting 1% of failures and saying \"yep looks right\" isn't very scientific imo..\n\nThere were reports when the last supplemental data got swapped out a lot of peoples notebooks said \"asset_details.csv not found\", how can we be sure this has not happened again?\n\nThese errors have caused a lot of noise, I think at least updating the supplemental train will let a large % of people see what went wrong in their notebooks and offer some closure.", "votes": 7 }, { "id": 1711327, "postDate": "2022-03-03T20:56:08.983Z", "content": "

@sohier
\nI think the idea from @julianmukaj is brilliant and reasonable. If the input data used for the private leaderboard is released, we'll be able to figure out what's going on on our own.

", "rawMarkdown": "@sohier \nI think the idea from @julianmukaj is brilliant and reasonable. If the input data used for the private leaderboard is released, we'll be able to figure out what's going on on our own. ", "votes": 3 }, { "id": 1711423, "postDate": "2022-03-04T01:49:37.933Z", "content": "

I am almost quite sure that you guys got something wrong in the supplemental_train.csv and the data updated afterwards. My retrained submission performed worse than the non-retrained version by a margin(0018 vs 0.019), which is due to the use of target reconstruction. This happened previously before evaluation phase started, due to the problem occurred in supplemental data. But you guys fixed it soon and everything just went right afterwards. Now similar error seems to come up again.
\nSo I would really like you to use the same code with which you got supplemental data previouly and don't optimize it any more…

", "rawMarkdown": "I am almost quite sure that you guys got something wrong in the supplemental_train.csv and the data updated afterwards. My retrained submission performed worse than the non-retrained version by a margin(0018 vs 0.019), which is due to the use of target reconstruction. This happened previously before evaluation phase started, due to the problem occurred in supplemental data. But you guys fixed it soon and everything just went right afterwards. Now similar error seems to come up again.\nSo I would really like you to use the same code with which you got supplemental data previouly and don't optimize it any more...", "votes": 3 }, { "id": 1711426, "postDate": "2022-03-04T01:52:07.920Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1711433, "postDate": "2022-03-04T02:02:14.813Z", "content": "

@sohier For verification, please check this https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota. Target reconstruction method is provided in this notebook and either he is wrong or you guys are wrong.

", "rawMarkdown": "@sohier For verification, please check this https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota. Target reconstruction method is provided in this notebook and either he is wrong or you guys are wrong.", "votes": 3 }, { "id": 1712734, "postDate": "2022-03-05T08:52:37.810Z", "content": "

Also agree. If people are having doubts about the supplemental training csv, especially given that a similar issue occurred in the past, it makes sense to share it. There's literally no reason not to.

", "rawMarkdown": "Also agree. If people are having doubts about the supplemental training csv, especially given that a similar issue occurred in the past, it makes sense to share it. There's literally no reason not to. ", "votes": 3 }, { "id": 1712830, "postDate": "2022-03-05T11:56:43.210Z", "content": "

The problem isn't here. It can be understood that they don't want to release data for free since normally we need to pay to buy these financial data. The major problem is, I have provided them with a link, a program to check whether their code is correct or not. And they just need to spend about 10 minutes on it, take their newly calculated target to compare with the target computed by that notebook. The target calculated by that notebook has a max abs error of 1e-13 on training data and thus is completely worth our trust. They don't even want to spend this little time on this competition. What else can I say??

", "rawMarkdown": "The problem isn't here. It can be understood that they don't want to release data for free since normally we need to pay to buy these financial data. The major problem is, I have provided them with a link, a program to check whether their code is correct or not. And they just need to spend about 10 minutes on it, take their newly calculated target to compare with the target computed by that notebook. The target calculated by that notebook has a max abs error of 1e-13 on training data and thus is completely worth our trust. They don't even want to spend this little time on this competition. What else can I say??", "votes": 3 }, { "id": 1712833, "postDate": "2022-03-05T12:00:45.273Z", "content": "

Another possible explanation is that, they know they made an error but think it's worth nothing. The last reason for such behavior can only be their irresponsibility.

", "rawMarkdown": "Another possible explanation is that, they know they made an error but think it's worth nothing. The last reason for such behavior can only be their irresponsibility.", "votes": 3 }, { "id": 1713092, "postDate": "2022-03-05T17:01:44.520Z", "content": "

These features are publicly available there isn't a reason for them not to release it.

", "rawMarkdown": "These features are publicly available there isn't a reason for them not to release it. " }, { "id": 1713313, "postDate": "2022-03-05T22:07:42.310Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1713691, "postDate": "2022-03-06T09:15:35.363Z", "content": "

The current problem is, the organizers don't seem to be interested in solving problems with all competitors together, in whatever ways. As you can see, they only respond to questions like \"when will be the next update\", but never respond positively to questions on data collected.

", "rawMarkdown": "The current problem is, the organizers don't seem to be interested in solving problems with all competitors together, in whatever ways. As you can see, they only respond to questions like \"when will be the next update\", but never respond positively to questions on data collected.", "votes": 3 }, { "id": 1714782, "postDate": "2022-03-07T10:24:47.457Z", "content": "

Both my notebooks used the supplemental data and both error, had a feeling with all the problems with this set we had there would be more in the evaluation phase, but alas as everyone has seen the hosts don't care because it \"looks about right\" lol

\n

\"data scientists\" eh

", "rawMarkdown": "Both my notebooks used the supplemental data and both error, had a feeling with all the problems with this set we had there would be more in the evaluation phase, but alas as everyone has seen the hosts don't care because it \"looks about right\" lol\n\n\"data scientists\" eh", "votes": 1 }, { "id": 1714820, "postDate": "2022-03-07T11:20:45.987Z", "content": "

I sent an email to @sohier and he just said they can't provide logs or whatever. I answered again and he did not reply anymore. I don't know why providing an explanation of why a notebook fails is so hard for a company such as Google.
\nI concluded that as long as \"it seems more or less OK\" they do not care. I did not work so much in this competition so it's not a big deal to not appear on the leaderboard. However, I would really like to understand why my simple submissions failed. I don't get it since there is even one that neither does any retraining nor uses supplemental data.
\nI am quite disappointed with the way the staff is handling this problem which is basically doing nothing.

", "rawMarkdown": "I sent an email to @sohier and he just said they can't provide logs or whatever. I answered again and he did not reply anymore. I don't know why providing an explanation of why a notebook fails is so hard for a company such as Google. \nI concluded that as long as \"it seems more or less OK\" they do not care. I did not work so much in this competition so it's not a big deal to not appear on the leaderboard. However, I would really like to understand why my simple submissions failed. I don't get it since there is even one that neither does any retraining nor uses supplemental data. \nI am quite disappointed with the way the staff is handling this problem which is basically doing nothing.", "votes": 1 }, { "id": 1714875, "postDate": "2022-03-07T12:45:29.957Z", "content": "

And that's why I decide not to attend any of these \"evaluate later\" competitions any more. But since my models have been successfully deployed in stock prediction and made some money, I wound't care about the medal in this competition. This will be the last time I comment on these problems. To be honest, I really don't know the goal of this competition. If the hosts are to recruit some top data scientists, their irresponsibility shown up till now makes people hard to trust them; If they are to get top solutions so that they can apply them to make money, being so careless about the target construction and the evaluation process would be so foolish since it is actual money that we are talking about, not a medal that can be won in any competition.

", "rawMarkdown": "And that's why I decide not to attend any of these \"evaluate later\" competitions any more. But since my models have been successfully deployed in stock prediction and made some money, I wound't care about the medal in this competition. This will be the last time I comment on these problems. To be honest, I really don't know the goal of this competition. If the hosts are to recruit some top data scientists, their irresponsibility shown up till now makes people hard to trust them; If they are to get top solutions so that they can apply them to make money, being so careless about the target construction and the evaluation process would be so foolish since it is actual money that we are talking about, not a medal that can be won in any competition." }, { "id": 1715201, "postDate": "2022-03-07T17:52:36.363Z", "content": "

I'm sorry that I can't give you a more satisfying answer but it's simply not possible for me to follow up on all of the requests for assistance that I receive. In this particular case, the overall submission success rate is in line with what we see in every rescore. Given that the submissions look healthy, submissions are locked, and how long it would take to fully revisit the pipeline I'm not able to prioritize your questions over the many other demands on my time. With that said, I did conduct a spot check and confirmed that the schema for the supplemental training file has not changed.

", "rawMarkdown": "I'm sorry that I can't give you a more satisfying answer but it's simply not possible for me to follow up on all of the requests for assistance that I receive. In this particular case, the overall submission success rate is in line with what we see in every rescore. Given that the submissions look healthy, submissions are locked, and how long it would take to fully revisit the pipeline I'm not able to prioritize your questions over the many other demands on my time. With that said, I did conduct a spot check and confirmed that the schema for the supplemental training file has not changed.", "votes": -6 }, { "id": 1715314, "postDate": "2022-03-07T21:33:09.203Z", "content": "

To clarify @sohier:

\n\n

After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".

\n

Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.

\n

Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.

", "rawMarkdown": "To clarify @sohier:\n\n- this is not about \"people asking for assistance for particular cases\".\n\n- this is about \"people hinting about a bug in first rerun data and asking for transparency\".\n\n- a transparency btw, that is next to effortless to achieve here. Either release data or use a simple script to confirm reconstruction error by yourself.\n\n- but now this is more becoming about \"people competing in Kaggle denied the most basic respect by Kaggle staff\".\n\n- **if a serious bug in first part of the data is not discarded (either in Kaggle pipeline or in host's test set) \nthen this competition will have been rendered meaningless. No feedback, no learning, and no true LB.** \n\nAfter about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".\n\nEven if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.\n\nAnyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended. ", "votes": 6, "replies": [ { "id": 1715484, "postDate": "2022-03-08T03:59:24.693Z", "content": "
\n

To clarify @sohier:

\n \n

After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".

\n

Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.

\n

Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.

\n

To clarify @sohier:

\n \n

After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".

\n

Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.

\n

Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.

\n
\n

Hi, I think we can use the late submission to check the submission error after the competition ends.

", "rawMarkdown": "> To clarify @sohier:\n> \n> - this is not about \"people asking for assistance for particular cases\".\n> \n> - this is about \"people hinting about a bug in first rerun data and asking for transparency\".\n> \n> - a transparency btw, that is next to effortless to achieve here. Either release data or use a simple script to confirm reconstruction error by yourself.\n> \n> - but now this is more becoming about \"people competing in Kaggle denied the most basic respect by Kaggle staff\".\n> \n> - **if a serious bug in first part of the data is not discarded (either in Kaggle pipeline or in host's test set) \n> then this competition will have been rendered meaningless. No feedback, no learning, and no true LB.** \n> \n> After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".\n> \n> Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.\n> \n> Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.\n\n> To clarify @sohier:\n> \n> - this is not about \"people asking for assistance for particular cases\".\n> \n> - this is about \"people hinting about a bug in first rerun data and asking for transparency\".\n> \n> - a transparency btw, that is next to effortless to achieve here. Either release data or use a simple script to confirm reconstruction error by yourself.\n> \n> - but now this is more becoming about \"people competing in Kaggle denied the most basic respect by Kaggle staff\".\n> \n> - **if a serious bug in first part of the data is not discarded (either in Kaggle pipeline or in host's test set) \n> then this competition will have been rendered meaningless. No feedback, no learning, and no true LB.** \n> \n> After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".\n> \n> Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.\n> \n> Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.\n\nHi, I think we can use the late submission to check the submission error after the competition ends.", "votes": 1 } ] }, { "id": 1715332, "postDate": "2022-03-07T22:11:36.313Z", "content": "
\n

In this particular case, the overall submission success rate is in line with what we see in every rescore. Given that the submissions look healthy,

\n
\n

@sohier I don't think the submission health can always be simply measured by the overall submission success rate. Instead, you need to measure the conditioned success rate to determine whether the submissions are healthy or not, e.g., the number of submissions that succeed at both training and forecasting stages divided by the number of submissions that succeed at the training stage. I would expect this ratio to be relatively low given so many people are complaining.

\n

Moreover, as far as I know, hosting a Kaggle competition is very expensive and I believe the hosts deserve the most valuable solutions. However, there is a high chance that these solutions are missed in the submission scoring or data errors that are not the responsibility of the participants. So, I humbly suggest that it is better to take more time and dig deeper into this submission failure issue for the sake of both hosts and competitors and also Kaggle's reputation. Finally, thank you very much for your consideration.

", "rawMarkdown": "> In this particular case, the overall submission success rate is in line with what we see in every rescore. Given that the submissions look healthy, \n\n\n@sohier I don't think the submission health can always be simply measured by the overall submission success rate. Instead, you need to measure the conditioned success rate to determine whether the submissions are healthy or not, e.g., the number of submissions that succeed at both training and forecasting stages divided by the number of submissions that succeed at the training stage. I would expect this ratio to be relatively low given so many people are complaining.\n\nMoreover, as far as I know, hosting a Kaggle competition is very expensive and I believe the hosts deserve the most valuable solutions. However, there is a high chance that these solutions are missed in the submission scoring or data errors that are not the responsibility of the participants. So, I humbly suggest that it is better to take more time and dig deeper into this submission failure issue for the sake of both hosts and competitors and also Kaggle's reputation. Finally, thank you very much for your consideration.", "votes": 6 }, { "id": 1715357, "postDate": "2022-03-07T23:38:54.393Z", "content": "

Hello @sohier , thank you for your reply. We are not asking for prioritizing checking our submission but rather for checking the validity of the data. I understand that our models should be as robust as possible. But nobody can deny that there could be some extreme situation in the data that cannot be easily handled on the model side. We are asking for checking the data and this is benificial for the host, all the players and the fairness of the competition. Please kindly consider this suggestion.

", "rawMarkdown": "Hello @sohier , thank you for your reply. We are not asking for prioritizing checking our submission but rather for checking the validity of the data. I understand that our models should be as robust as possible. But nobody can deny that there could be some extreme situation in the data that cannot be easily handled on the model side. We are asking for checking the data and this is benificial for the host, all the players and the fairness of the competition. Please kindly consider this suggestion.", "votes": 7 }, { "id": 1715406, "postDate": "2022-03-08T00:46:51.880Z", "content": "

Also, when the supplemental data was updated the first time during the submission part of the competition, it had a lot of outliers in it that were incorrect. This wasn't found by the Kaggle staff. It was found by someone in the competition. So, even if not that many people are getting an error in their notebooks, that doesn't mean the underlying data are correct. It might be that people are getting worse scores than what they would have gotten if the data was correct. That is why I feel like releasing the rest of the supplemental training data up to the end of the competition would be important and creating a new test set that encompasses new evaluation data will allow for full transparency.

\n

This will allow for Kagglers to use their own time to investigate the competition so none of your time gets wasted. If they can't find an error, at least it would provide assurances.

\n

Also, it seems like the scores in the second round were better than the first at least at the top of the lb. So, it might be that the supplemental training set doesn't fully extend to the last minute right before the test data set starts.

\n

However, there might be a reason why the data isn't being released. If so, that is fine, but it might be good to explain the reason to the competitors.

", "rawMarkdown": "Also, when the supplemental data was updated the first time during the submission part of the competition, it had a lot of outliers in it that were incorrect. This wasn't found by the Kaggle staff. It was found by someone in the competition. So, even if not that many people are getting an error in their notebooks, that doesn't mean the underlying data are correct. It might be that people are getting worse scores than what they would have gotten if the data was correct. That is why I feel like releasing the rest of the supplemental training data up to the end of the competition would be important and creating a new test set that encompasses new evaluation data will allow for full transparency.\n\nThis will allow for Kagglers to use their own time to investigate the competition so none of your time gets wasted. If they can't find an error, at least it would provide assurances. \n\nAlso, it seems like the scores in the second round were better than the first at least at the top of the lb. So, it might be that the supplemental training set doesn't fully extend to the last minute right before the test data set starts. \n\nHowever, there might be a reason why the data isn't being released. If so, that is fine, but it might be good to explain the reason to the competitors.", "votes": 4 }, { "id": 1715420, "postDate": "2022-03-08T01:34:11.243Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 1715828, "postDate": "2022-03-08T12:25:43.327Z", "content": "

Imagine once competition ends, we do late submissions.. we discover errors in data again, someone with a failed notebook was actually place #1… Someone in top 10 got calculated score wrong and is > 10…?

\n

Wonder how they will react then? Ignore the community again?

\n

Seems like a situation that is best to avoid just by double checking the things we brought up, or as @petersk20 nicely suggested let the community check themselves..

", "rawMarkdown": "Imagine once competition ends, we do late submissions.. we discover errors in data *again*, someone with a failed notebook was actually place #1... Someone in top 10 got calculated score wrong and is > 10...?\n\nWonder how they will react then? Ignore the community again? \n\nSeems like a situation that is best to avoid just by double checking the things we brought up, or as @petersk20 nicely suggested let the community check themselves..", "votes": 1 }, { "id": 1716697, "postDate": "2022-03-09T09:55:41.430Z", "content": "

Hi All, not too sure if it will help much but I see that Coin Gecko have a free API with all the data fields we used in this competition, might be possible to aggregate the data and compare with the first run.

", "rawMarkdown": "Hi All, not too sure if it will help much but I see that Coin Gecko have a free API with all the data fields we used in this competition, might be possible to aggregate the data and compare with the first run.", "votes": 3 }, { "id": 1716744, "postDate": "2022-03-09T11:01:44.263Z", "content": "

You can get the OHLC via https://api.coingecko.com/api/v3/coins/ethereum/ohlc?vs_currency=usd&days=14 for example

\n

But you will not get the volume or trade count unfortunately but also our data is 1 minute aggregated and coingecko is daily data

", "rawMarkdown": "You can get the OHLC via https://api.coingecko.com/api/v3/coins/ethereum/ohlc?vs_currency=usd&days=14 for example\n\nBut you will not get the volume or trade count unfortunately but also our data is 1 minute aggregated and coingecko is daily data" }, { "id": 1716928, "postDate": "2022-03-09T14:11:35.040Z", "content": "

Got you, I have access to some of the coins on a minute basis, but not all unfortunately, also dong have the volume and count values though. Guess we wait some more. :)

", "rawMarkdown": "Got you, I have access to some of the coins on a minute basis, but not all unfortunately, also dong have the volume and count values though. Guess we wait some more. :)" }, { "id": 1716982, "postDate": "2022-03-09T14:55:59.770Z", "content": "

Hi Jonny I compared the provided data with a few exchanges and aggregations between them and it did not match. I believe the provided data was pre-processed somehow. Because crypto intraday data quality is not the best, particularly cross exchanges. I think will be very hard to draw any conclusion from other data sources.

", "rawMarkdown": "Hi Jonny I compared the provided data with a few exchanges and aggregations between them and it did not match. I believe the provided data was pre-processed somehow. Because crypto intraday data quality is not the best, particularly cross exchanges. I think will be very hard to draw any conclusion from other data sources.", "votes": 1 }, { "id": 1717378, "postDate": "2022-03-09T22:03:15.723Z", "content": "

Everyone: If your solutions aren't running, I don't know what to say. But if your solution is running and you're nervous about low scores, don't be, it's illogical!

\n

In my local CV, I was getting a score standard deviation between folds of about .015, and that was for three month test folds. For a single update, I expect my model's score to be accurate to within .015 * sqrt(6), or .037.

\n

Many of us were expecting the first update to be almost all error. The second update wouldn't be much better. You can see that many grandmasters even have negative scores at this point.

\n

It gets better. Whoever is in first place on the semifinal update should not be surprised if the final update lowers them to silver medal, spot 15! Say they have a score of .04 on the semifinal update. On the final update they could easily have a score of 0 for those two weeks. So their final score would be about .04 * 5/6 + 0 * 1/6 = .033. That's 21st place by the current score distribution.

\n

(I picked .04 as the top score rather than .05 because I believe that the scores above .04 in the second update are unrealistic. The curve of the top few scores are is usually concave, not convex. I think everyone over .04 just got extra lucky this time.)

\n

1st place -> 21st place on the final update could be normal! I'm in 18th place now, and I have a lot of confidence in my solution. But I won't be surprised if I end up with a bronze medal.

\n

You should forget any nervousness that's coming from noisy low scores. It's just the nature of the competition. They could have decided to run the competition longer, but who wants to wait 6-12 months to get their score?

", "rawMarkdown": "Everyone: If your solutions aren't running, I don't know what to say. But if your solution is running and you're nervous about low scores, don't be, it's illogical!\n\nIn my local CV, I was getting a score standard deviation between folds of about .015, and that was for three month test folds. For a single update, I expect my model's score to be accurate to within .015 * sqrt(6), or .037. \n\nMany of us were expecting the first update to be almost all error. The second update wouldn't be much better. You can see that many grandmasters even have negative scores at this point.\n\nIt gets better. Whoever is in first place on the semifinal update should not be surprised if the final update lowers them to silver medal, spot 15! Say they have a score of .04 on the semifinal update. On the final update they could easily have a score of 0 for those two weeks. So their final score would be about .04 * 5/6 + 0 * 1/6 = .033. That's 21st place by the current score distribution. \n\n(I picked .04 as the top score rather than .05 because I believe that the scores above .04 in the second update are unrealistic. The curve of the top few scores are is usually concave, not convex. I think everyone over .04 just got extra lucky this time.) \n\n1st place -> 21st place on the final update could be normal! I'm in 18th place now, and I have a lot of confidence in my solution. But I won't be surprised if I end up with a bronze medal.\n\nYou should forget any nervousness that's coming from noisy low scores. It's just the nature of the competition. They could have decided to run the competition longer, but who wants to wait 6-12 months to get their score?", "votes": 1 }, { "id": 1717497, "postDate": "2022-03-10T01:31:20.160Z", "content": "

For those with submissions normally running like me, we do not question about the score out of expectation, but the way how targets are calculated and how data are collected and preprocessed. Noise should only come from the nature of financial market itself, instead of some bugs in the host's code with error. So if the host is too busy to check the full pipeline, it would be better to allow the community to find out where bugs are. That's to say data should be released.

", "rawMarkdown": "For those with submissions normally running like me, we do not question about the score out of expectation, but the way how targets are calculated and how data are collected and preprocessed. Noise should only come from the nature of financial market itself, instead of some bugs in the host's code with error. So if the host is too busy to check the full pipeline, it would be better to allow the community to find out where bugs are. That's to say data should be released.", "votes": 3 }, { "id": 1717934, "postDate": "2022-03-10T11:05:27.993Z", "content": "

As you are posting this here as well as in a separate thread, I answered there https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/311997#1717929

", "rawMarkdown": "As you are posting this here as well as in a separate thread, I answered there https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/311997#1717929" } ] }, { "id": 1710100, "postDate": "2022-03-02T18:47:48.430Z", "content": "

Some of us have raised doubts about the scoring process and posed some questions.

\n

Is this what Kaggle is now, somewhere were Kagglers are expected to devote weeks but wont deserve a 5 minute answer by the organization @sohier ?

", "rawMarkdown": "Some of us have raised doubts about the scoring process and posed some questions.\n\nIs this what Kaggle is now, somewhere were Kagglers are expected to devote weeks but wont deserve a 5 minute answer by the organization @sohier ?\n", "votes": 4, "replies": [ { "id": 1710221, "postDate": "2022-03-02T20:21:48.133Z", "content": "

I must say @miguelpm I don't think that either Kaggle nor G-Research have represented themselves well unfortunately. Their replies have ranged from non-existent to condescending which, considering how much time, effort and energy has been expended by the Kaggle community is not in the spirit of what this platform purports to represent.

", "rawMarkdown": "I must say @miguelpm I don't think that either Kaggle nor G-Research have represented themselves well unfortunately. Their replies have ranged from non-existent to condescending which, considering how much time, effort and energy has been expended by the Kaggle community is not in the spirit of what this platform purports to represent.", "votes": 14 }, { "id": 1710359, "postDate": "2022-03-03T01:43:58.653Z", "content": "

I would really like the hosts to claim that they have made a double check or not. But since I have already deployed my model in stock prediction and it performed pretty well, I won't care about this any more. We should all keep in mind that if G-Reasearch continues to do things so carelessly, top solutions may just be completely wrong. And if they deploy these models in real applications, loss of money is expected. A kaggle medal is not that important compared to actual money loss afterall.

", "rawMarkdown": "I would really like the hosts to claim that they have made a double check or not. But since I have already deployed my model in stock prediction and it performed pretty well, I won't care about this any more. We should all keep in mind that if G-Reasearch continues to do things so carelessly, top solutions may just be completely wrong. And if they deploy these models in real applications, loss of money is expected. A kaggle medal is not that important compared to actual money loss afterall.", "votes": 8 } ] }, { "id": 1710119, "postDate": "2022-03-02T19:08:11.810Z", "content": "

My two prediction models had completely opposite scores in the first update. One had a positive value close to 0 and the other had a negative value close to 0. Now their scores are close, both around 0.009.

", "rawMarkdown": "My two prediction models had completely opposite scores in the first update. One had a positive value close to 0 and the other had a negative value close to 0. Now their scores are close, both around 0.009.", "votes": 1, "replies": [ { "id": 1710883, "postDate": "2022-03-03T12:41:09.007Z", "content": "

My 2 have almost just swapped signs! It's almost like it's not easy to predict the market or something :o

", "rawMarkdown": "My 2 have almost just swapped signs! It's almost like it's not easy to predict the market or something :o ", "votes": 2 } ] }, { "id": 1712939, "postDate": "2022-03-05T13:57:06.253Z", "content": "

Why do my submission always gets no score? It always submitted competition 's default code, my first submission get 0.07，but after every update ,my submission always submit default code file,and get only 0.0001.why?

", "rawMarkdown": "Why do my submission always gets no score? It always submitted competition 's default code, my first submission get 0.07，but after every update ,my submission always submit default code file,and get only 0.0001.why?" }, { "id": 1711428, "postDate": "2022-03-04T01:55:49.280Z", "content": "

@sohier when is the next update date??

", "rawMarkdown": "@sohier when is the next update date??", "replies": [ { "id": 1711435, "postDate": "2022-03-04T02:06:20.523Z", "content": "

Hi, it is supposed to be two weeks later.

", "rawMarkdown": "Hi, it is supposed to be two weeks later.", "votes": 1 }, { "id": 1712226, "postDate": "2022-03-04T17:25:50.013Z", "content": "

@yus002 is correct; the updates will be posted every two weeks. I'll do my best to make an announcement in advance if we have to deviate from that schedule.

", "rawMarkdown": "@yus002 is correct; the updates will be posted every two weeks. I'll do my best to make an announcement in advance if we have to deviate from that schedule.", "votes": 2 }, { "id": 1712718, "postDate": "2022-03-05T08:13:46.340Z", "content": "

I just want to ask why you guys never answer my questions. I have even provided you with a notebook calculating the target. To solve all problems, just compare the target computed by your new code with the target calculated by that notebook. The provided target reconstruction method achieved a max abs error of 1e-13 on training data. If error is small enough then problems are on our side. Isn't it easy???

", "rawMarkdown": "I just want to ask why you guys never answer my questions. I have even provided you with a notebook calculating the target. To solve all problems, just compare the target computed by your new code with the target calculated by that notebook. The provided target reconstruction method achieved a max abs error of 1e-13 on training data. If error is small enough then problems are on our side. Isn't it easy???", "votes": 7 }, { "id": 1712721, "postDate": "2022-03-05T08:14:45.997Z", "content": "

Otherwise you are just too lazy to correct your code…

", "rawMarkdown": "Otherwise you are just too lazy to correct your code...", "votes": 3 } ] }, { "id": 1710084, "postDate": "2022-03-02T18:28:52.930Z", "content": "

Is this up date base on the data in last 2 weeks, or all data of 4 weeks?

", "rawMarkdown": "Is this up date base on the data in last 2 weeks, or all data of 4 weeks?", "replies": [ { "id": 1710087, "postDate": "2022-03-02T18:29:35.887Z", "content": "

Should be all 4 weeks

", "rawMarkdown": "Should be all 4 weeks", "votes": 1 }, { "id": 1711197, "postDate": "2022-03-03T17:46:52.750Z", "content": "

It's all 4 weeks.

", "rawMarkdown": "It's all 4 weeks." } ] } ], "comments": [ { "id": 1710091, "author_name": "Yirun Zhang", "author_url": "", "post_date": "2022-03-02T18:32:30.763000", "content": "

Some teams are still only having one submission. Is there any clue on this issue? Thanks. @sohier

", "votes": 3, "replies": [ { "id": 1710106, "author_name": "Hantian Zheng", "author_url": "", "post_date": "2022-03-02T18:50:31.027000", "content": "

@sohier Could you kindly double-checked the input files?

\n

Because I believe our two submissions worked fine in the initial submission. And we already added a lot of robustness logics. I will appreciate if you can double-check the input file because it is very hard for the players to figure out what is happening for a unknown dataset.

", "votes": 4, "replies": [] }, { "id": 1710247, "author_name": "Ghassen Alaya", "author_url": "", "post_date": "2022-03-02T20:46:49.617000", "content": "

Some teams have no submissions ;) @sohier

", "votes": 4, "replies": [] }, { "id": 1711187, "author_name": "jiebao1964", "author_url": "", "post_date": "2022-03-03T17:34:33.330000", "content": "

@sohier Can you please double-check and make sure the format of the input data is not changed? Our team submitted two notebooks. The main difference is that one of the submissions uses the suplemental_train.csv. The one using the suplemental_train.csv runs well in the public leaderboard but fails in the private leaderboard.

\n

Similar issues have been reported in the following posts, but no response is given until now.

\n

https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308300
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308230

", "votes": 2, "replies": [] }, { "id": 1711203, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-03-03T17:59:42.250000", "content": "

The overall failure rate across all of the submissions for this iteration was roughly 1%, which is in line with what we expect for this kind of rescore. Unfortunately we aren't able to provide support or details as to what might have happened with individual submissions.

", "votes": -4, "replies": [] }, { "id": 1711206, "author_name": "JM", "author_url": "", "post_date": "2022-03-03T18:05:41.010000", "content": "

Can you release the supplemental data update so we can see for ourselves what went wrong? Just getting 1% of failures and saying \"yep looks right\" isn't very scientific imo..

\n

There were reports when the last supplemental data got swapped out a lot of peoples notebooks said \"asset_details.csv not found\", how can we be sure this has not happened again?

\n

These errors have caused a lot of noise, I think at least updating the supplemental train will let a large % of people see what went wrong in their notebooks and offer some closure.

", "votes": 7, "replies": [] }, { "id": 1711327, "author_name": "jiebao1964", "author_url": "", "post_date": "2022-03-03T20:56:08.983000", "content": "

@sohier
\nI think the idea from @julianmukaj is brilliant and reasonable. If the input data used for the private leaderboard is released, we'll be able to figure out what's going on on our own.

", "votes": 3, "replies": [] }, { "id": 1711423, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-04T01:49:37.933000", "content": "

I am almost quite sure that you guys got something wrong in the supplemental_train.csv and the data updated afterwards. My retrained submission performed worse than the non-retrained version by a margin(0018 vs 0.019), which is due to the use of target reconstruction. This happened previously before evaluation phase started, due to the problem occurred in supplemental data. But you guys fixed it soon and everything just went right afterwards. Now similar error seems to come up again.
\nSo I would really like you to use the same code with which you got supplemental data previouly and don't optimize it any more…

", "votes": 3, "replies": [] }, { "id": 1711426, "author_name": "", "author_url": "", "post_date": "2022-03-04T01:52:07.920000", "content": "", "votes": 0, "replies": [] }, { "id": 1711433, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-04T02:02:14.813000", "content": "

@sohier For verification, please check this https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota. Target reconstruction method is provided in this notebook and either he is wrong or you guys are wrong.

", "votes": 3, "replies": [] }, { "id": 1712734, "author_name": "JacobW", "author_url": "", "post_date": "2022-03-05T08:52:37.810000", "content": "

Also agree. If people are having doubts about the supplemental training csv, especially given that a similar issue occurred in the past, it makes sense to share it. There's literally no reason not to.

", "votes": 3, "replies": [] }, { "id": 1712830, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-05T11:56:43.210000", "content": "

The problem isn't here. It can be understood that they don't want to release data for free since normally we need to pay to buy these financial data. The major problem is, I have provided them with a link, a program to check whether their code is correct or not. And they just need to spend about 10 minutes on it, take their newly calculated target to compare with the target computed by that notebook. The target calculated by that notebook has a max abs error of 1e-13 on training data and thus is completely worth our trust. They don't even want to spend this little time on this competition. What else can I say??

", "votes": 3, "replies": [] }, { "id": 1712833, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-05T12:00:45.273000", "content": "

Another possible explanation is that, they know they made an error but think it's worth nothing. The last reason for such behavior can only be their irresponsibility.

", "votes": 3, "replies": [] }, { "id": 1713092, "author_name": "JacobW", "author_url": "", "post_date": "2022-03-05T17:01:44.520000", "content": "

These features are publicly available there isn't a reason for them not to release it.

", "votes": 0, "replies": [] }, { "id": 1713313, "author_name": "", "author_url": "", "post_date": "2022-03-05T22:07:42.310000", "content": "", "votes": 1, "replies": [] }, { "id": 1713691, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-06T09:15:35.363000", "content": "

The current problem is, the organizers don't seem to be interested in solving problems with all competitors together, in whatever ways. As you can see, they only respond to questions like \"when will be the next update\", but never respond positively to questions on data collected.

", "votes": 3, "replies": [] }, { "id": 1714782, "author_name": "JM", "author_url": "", "post_date": "2022-03-07T10:24:47.457000", "content": "

Both my notebooks used the supplemental data and both error, had a feeling with all the problems with this set we had there would be more in the evaluation phase, but alas as everyone has seen the hosts don't care because it \"looks about right\" lol

\n

\"data scientists\" eh

", "votes": 1, "replies": [] }, { "id": 1714820, "author_name": "JQ", "author_url": "", "post_date": "2022-03-07T11:20:45.987000", "content": "

I sent an email to @sohier and he just said they can't provide logs or whatever. I answered again and he did not reply anymore. I don't know why providing an explanation of why a notebook fails is so hard for a company such as Google.
\nI concluded that as long as \"it seems more or less OK\" they do not care. I did not work so much in this competition so it's not a big deal to not appear on the leaderboard. However, I would really like to understand why my simple submissions failed. I don't get it since there is even one that neither does any retraining nor uses supplemental data.
\nI am quite disappointed with the way the staff is handling this problem which is basically doing nothing.

", "votes": 1, "replies": [] }, { "id": 1714875, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-07T12:45:29.957000", "content": "

And that's why I decide not to attend any of these \"evaluate later\" competitions any more. But since my models have been successfully deployed in stock prediction and made some money, I wound't care about the medal in this competition. This will be the last time I comment on these problems. To be honest, I really don't know the goal of this competition. If the hosts are to recruit some top data scientists, their irresponsibility shown up till now makes people hard to trust them; If they are to get top solutions so that they can apply them to make money, being so careless about the target construction and the evaluation process would be so foolish since it is actual money that we are talking about, not a medal that can be won in any competition.

", "votes": 0, "replies": [] }, { "id": 1715201, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-03-07T17:52:36.363000", "content": "

I'm sorry that I can't give you a more satisfying answer but it's simply not possible for me to follow up on all of the requests for assistance that I receive. In this particular case, the overall submission success rate is in line with what we see in every rescore. Given that the submissions look healthy, submissions are locked, and how long it would take to fully revisit the pipeline I'm not able to prioritize your questions over the many other demands on my time. With that said, I did conduct a spot check and confirmed that the schema for the supplemental training file has not changed.

", "votes": -6, "replies": [] }, { "id": 1715314, "author_name": "miguel perez", "author_url": "", "post_date": "2022-03-07T21:33:09.203000", "content": "

To clarify @sohier:

\n\n

After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".

\n

Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.

\n

Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.

", "votes": 6, "replies": [ { "id": 1715484, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-03-08T03:59:24.693000", "content": "
\n

To clarify @sohier:

\n \n

After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".

\n

Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.

\n

Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.

\n

To clarify @sohier:

\n \n

After about 50 comps here I have not witnessed something like this, this kind of absolute lack of respect toward Kagglers by Kaggle itself. I can not imagine, say, @inversion or many others of your coleagues answering \"Too busy to check my pipeline or run a couple of scripts, explicit errors were under 1% of subs so I just don't listen, stop bugging me\".

\n

Even if in the end there is no error this is no way to deal with this situation, ignoring legit questions for one month already.

\n

Anyway, as I imagine Kaggle will keep backups of this data the truth sooner or later will be disclosed about that first chunk of test data. Maybe long after this competition has ended.

\n
\n

Hi, I think we can use the late submission to check the submission error after the competition ends.

", "votes": 1, "replies": [] } ] }, { "id": 1715332, "author_name": "Yirun Zhang", "author_url": "", "post_date": "2022-03-07T22:11:36.313000", "content": "
\n

In this particular case, the overall submission success rate is in line with what we see in every rescore. Given that the submissions look healthy,

\n
\n

@sohier I don't think the submission health can always be simply measured by the overall submission success rate. Instead, you need to measure the conditioned success rate to determine whether the submissions are healthy or not, e.g., the number of submissions that succeed at both training and forecasting stages divided by the number of submissions that succeed at the training stage. I would expect this ratio to be relatively low given so many people are complaining.

\n

Moreover, as far as I know, hosting a Kaggle competition is very expensive and I believe the hosts deserve the most valuable solutions. However, there is a high chance that these solutions are missed in the submission scoring or data errors that are not the responsibility of the participants. So, I humbly suggest that it is better to take more time and dig deeper into this submission failure issue for the sake of both hosts and competitors and also Kaggle's reputation. Finally, thank you very much for your consideration.

", "votes": 6, "replies": [] }, { "id": 1715357, "author_name": "Hantian Zheng", "author_url": "", "post_date": "2022-03-07T23:38:54.393000", "content": "

Hello @sohier , thank you for your reply. We are not asking for prioritizing checking our submission but rather for checking the validity of the data. I understand that our models should be as robust as possible. But nobody can deny that there could be some extreme situation in the data that cannot be easily handled on the model side. We are asking for checking the data and this is benificial for the host, all the players and the fairness of the competition. Please kindly consider this suggestion.

", "votes": 7, "replies": [] }, { "id": 1715406, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-03-08T00:46:51.880000", "content": "

Also, when the supplemental data was updated the first time during the submission part of the competition, it had a lot of outliers in it that were incorrect. This wasn't found by the Kaggle staff. It was found by someone in the competition. So, even if not that many people are getting an error in their notebooks, that doesn't mean the underlying data are correct. It might be that people are getting worse scores than what they would have gotten if the data was correct. That is why I feel like releasing the rest of the supplemental training data up to the end of the competition would be important and creating a new test set that encompasses new evaluation data will allow for full transparency.

\n

This will allow for Kagglers to use their own time to investigate the competition so none of your time gets wasted. If they can't find an error, at least it would provide assurances.

\n

Also, it seems like the scores in the second round were better than the first at least at the top of the lb. So, it might be that the supplemental training set doesn't fully extend to the last minute right before the test data set starts.

\n

However, there might be a reason why the data isn't being released. If so, that is fine, but it might be good to explain the reason to the competitors.

", "votes": 4, "replies": [] }, { "id": 1715420, "author_name": "", "author_url": "", "post_date": "2022-03-08T01:34:11.243000", "content": "", "votes": 3, "replies": [] }, { "id": 1715828, "author_name": "JM", "author_url": "", "post_date": "2022-03-08T12:25:43.327000", "content": "

Imagine once competition ends, we do late submissions.. we discover errors in data again, someone with a failed notebook was actually place #1… Someone in top 10 got calculated score wrong and is > 10…?

\n

Wonder how they will react then? Ignore the community again?

\n

Seems like a situation that is best to avoid just by double checking the things we brought up, or as @petersk20 nicely suggested let the community check themselves..

", "votes": 1, "replies": [] }, { "id": 1716697, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-03-09T09:55:41.430000", "content": "

Hi All, not too sure if it will help much but I see that Coin Gecko have a free API with all the data fields we used in this competition, might be possible to aggregate the data and compare with the first run.

", "votes": 3, "replies": [] }, { "id": 1716744, "author_name": "JM", "author_url": "", "post_date": "2022-03-09T11:01:44.263000", "content": "

You can get the OHLC via https://api.coingecko.com/api/v3/coins/ethereum/ohlc?vs_currency=usd&days=14 for example

\n

But you will not get the volume or trade count unfortunately but also our data is 1 minute aggregated and coingecko is daily data

", "votes": 0, "replies": [] }, { "id": 1716928, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-03-09T14:11:35.040000", "content": "

Got you, I have access to some of the coins on a minute basis, but not all unfortunately, also dong have the volume and count values though. Guess we wait some more. :)

", "votes": 0, "replies": [] }, { "id": 1716982, "author_name": "Jose Antonio Alatorre", "author_url": "", "post_date": "2022-03-09T14:55:59.770000", "content": "

Hi Jonny I compared the provided data with a few exchanges and aggregations between them and it did not match. I believe the provided data was pre-processed somehow. Because crypto intraday data quality is not the best, particularly cross exchanges. I think will be very hard to draw any conclusion from other data sources.

", "votes": 1, "replies": [] }, { "id": 1717378, "author_name": "Nathaniel Maddux", "author_url": "", "post_date": "2022-03-09T22:03:15.723000", "content": "

Everyone: If your solutions aren't running, I don't know what to say. But if your solution is running and you're nervous about low scores, don't be, it's illogical!

\n

In my local CV, I was getting a score standard deviation between folds of about .015, and that was for three month test folds. For a single update, I expect my model's score to be accurate to within .015 * sqrt(6), or .037.

\n

Many of us were expecting the first update to be almost all error. The second update wouldn't be much better. You can see that many grandmasters even have negative scores at this point.

\n

It gets better. Whoever is in first place on the semifinal update should not be surprised if the final update lowers them to silver medal, spot 15! Say they have a score of .04 on the semifinal update. On the final update they could easily have a score of 0 for those two weeks. So their final score would be about .04 * 5/6 + 0 * 1/6 = .033. That's 21st place by the current score distribution.

\n

(I picked .04 as the top score rather than .05 because I believe that the scores above .04 in the second update are unrealistic. The curve of the top few scores are is usually concave, not convex. I think everyone over .04 just got extra lucky this time.)

\n

1st place -> 21st place on the final update could be normal! I'm in 18th place now, and I have a lot of confidence in my solution. But I won't be surprised if I end up with a bronze medal.

\n

You should forget any nervousness that's coming from noisy low scores. It's just the nature of the competition. They could have decided to run the competition longer, but who wants to wait 6-12 months to get their score?

", "votes": 1, "replies": [] }, { "id": 1717497, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-10T01:31:20.160000", "content": "

For those with submissions normally running like me, we do not question about the score out of expectation, but the way how targets are calculated and how data are collected and preprocessed. Noise should only come from the nature of financial market itself, instead of some bugs in the host's code with error. So if the host is too busy to check the full pipeline, it would be better to allow the community to find out where bugs are. That's to say data should be released.

", "votes": 3, "replies": [] }, { "id": 1717934, "author_name": "miguel perez", "author_url": "", "post_date": "2022-03-10T11:05:27.993000", "content": "

As you are posting this here as well as in a separate thread, I answered there https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/311997#1717929

", "votes": 0, "replies": [] } ] }, { "id": 1710100, "author_name": "miguel perez", "author_url": "", "post_date": "2022-03-02T18:47:48.430000", "content": "

Some of us have raised doubts about the scoring process and posed some questions.

\n

Is this what Kaggle is now, somewhere were Kagglers are expected to devote weeks but wont deserve a 5 minute answer by the organization @sohier ?

", "votes": 4, "replies": [ { "id": 1710221, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-03-02T20:21:48.133000", "content": "

I must say @miguelpm I don't think that either Kaggle nor G-Research have represented themselves well unfortunately. Their replies have ranged from non-existent to condescending which, considering how much time, effort and energy has been expended by the Kaggle community is not in the spirit of what this platform purports to represent.

", "votes": 14, "replies": [] }, { "id": 1710359, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-03T01:43:58.653000", "content": "

I would really like the hosts to claim that they have made a double check or not. But since I have already deployed my model in stock prediction and it performed pretty well, I won't care about this any more. We should all keep in mind that if G-Reasearch continues to do things so carelessly, top solutions may just be completely wrong. And if they deploy these models in real applications, loss of money is expected. A kaggle medal is not that important compared to actual money loss afterall.

", "votes": 8, "replies": [] } ] }, { "id": 1710119, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-03-02T19:08:11.810000", "content": "

My two prediction models had completely opposite scores in the first update. One had a positive value close to 0 and the other had a negative value close to 0. Now their scores are close, both around 0.009.

", "votes": 1, "replies": [ { "id": 1710883, "author_name": "JacobW", "author_url": "", "post_date": "2022-03-03T12:41:09.007000", "content": "

My 2 have almost just swapped signs! It's almost like it's not easy to predict the market or something :o

", "votes": 2, "replies": [] } ] }, { "id": 1712939, "author_name": "谭聪", "author_url": "", "post_date": "2022-03-05T13:57:06.253000", "content": "

Why do my submission always gets no score? It always submitted competition 's default code, my first submission get 0.07，but after every update ,my submission always submit default code file,and get only 0.0001.why?

", "votes": 0, "replies": [] }, { "id": 1711428, "author_name": "neoistheone", "author_url": "", "post_date": "2022-03-04T01:55:49.280000", "content": "

@sohier when is the next update date??

", "votes": 0, "replies": [ { "id": 1711435, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-03-04T02:06:20.523000", "content": "

Hi, it is supposed to be two weeks later.

", "votes": 1, "replies": [] }, { "id": 1712226, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-03-04T17:25:50.013000", "content": "

@yus002 is correct; the updates will be posted every two weeks. I'll do my best to make an announcement in advance if we have to deviate from that schedule.

", "votes": 2, "replies": [] }, { "id": 1712718, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-05T08:13:46.340000", "content": "

I just want to ask why you guys never answer my questions. I have even provided you with a notebook calculating the target. To solve all problems, just compare the target computed by your new code with the target calculated by that notebook. The provided target reconstruction method achieved a max abs error of 1e-13 on training data. If error is small enough then problems are on our side. Isn't it easy???

", "votes": 7, "replies": [] }, { "id": 1712721, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-03-05T08:14:45.997000", "content": "

Otherwise you are just too lazy to correct your code…

", "votes": 3, "replies": [] } ] }, { "id": 1710084, "author_name": "liyang.chen", "author_url": "", "post_date": "2022-03-02T18:28:52.930000", "content": "

Is this up date base on the data in last 2 weeks, or all data of 4 weeks?

", "votes": 0, "replies": [ { "id": 1710087, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-03-02T18:29:35.887000", "content": "

Should be all 4 weeks

", "votes": 1, "replies": [] }, { "id": 1711197, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-03-03T17:46:52.750000", "content": "

It's all 4 weeks.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1710077": "I've just published the second score update. It may take several minutes for all entries to show up on the leaderboard.", "1710091": "Some teams are still only having one submission. Is there any clue on this issue? Thanks. @sohier ", "1710100": "Some of us have raised doubts about the scoring process and posed some questions.\n\nIs this what Kaggle is now, somewhere were Kagglers are expected to devote weeks but wont deserve a 5 minute answer by the organization @sohier ?\n", "1710119": "My two prediction models had completely opposite scores in the first update. One had a positive value close to 0 and the other had a negative value close to 0. Now their scores are close, both around 0.009.", "1712939": "Why do my submission always gets no score? It always submitted competition 's default code, my first submission get 0.07，but after every update ,my submission always submit default code file,and get only 0.0001.why?", "1711428": "@sohier when is the next update date??", "1710084": "Is this up date base on the data in last 2 weeks, or all data of 4 weeks?" } }