![\"graph\"](\"https://i.imgur.com/bmvWwb9.png\")
Wow. This first rerun is really weird, and I would like to understand what is going on.
\nHaving very present that:
\nI don't discard bugs in my final solutions but watching the leaderboard after the first rerun has me scratching my head.
\nWhy? Well, my both models have huge negative score, worst model has negative correlation (-0.0332) score!!! . What doesn't make sense. I mean, same correlation amount positive would imply #6 in the leaderboard. That is A LOT of negative correlation. That is signal.
\nWhat leads me to two posibilities:
\n1) A dramatic bug in my code that, let's be clear, I consider perfectly possible.
\n2) A problem with the rerun.
This said, the only thing I would ask is please, @sohier have a look at the distribution of the scores in the LB and please double check they make sense, including other instances of high negative correlation, to discard some problem with the rerun dataset quality or the joining with the previous one. Just to double check that the time dedicated to this comp. is not wasted because some evaluation glitch.
\nUPDATE2 (second rerun): LB shows now huge changes and a different distribution.
\nScores in general are better and less negative. As Kaggle has given no explanation yet we can not know what is going on, if there was a glitch, if it has been fixed or simply has diminishing effect as the dataset grows. In the latter case my initial glith candidates are no longer valid and I have at this point no new ones. Still, this LB dynamics is not what I expected and still think Kaggle should review their pipeline focusing on the first rerun data.
Wow. This first rerun is really weird, and I would like to understand what is going on.
\nHaving very present that:
\nI don't discard bugs in my final solutions but watching the leaderboard after the first rerun has me scratching my head.
\nWhy? Well, my both models have huge negative score, worst model has negative correlation (-0.0332) score!!! . What doesn't make sense. I mean, same correlation amount positive would imply #6 in the leaderboard. That is A LOT of negative correlation. That is signal.
\nWhat leads me to two posibilities:
\n1) A dramatic bug in my code that, let's be clear, I consider perfectly possible.
\n2) A problem with the rerun.
This said, the only thing I would ask is please, @sohier have a look at the distribution of the scores in the LB and please double check they make sense, including other instances of high negative correlation, to discard some problem with the rerun dataset quality or the joining with the previous one. Just to double check that the time dedicated to this comp. is not wasted because some evaluation glitch.
\nUPDATE2 (second rerun): LB shows now huge changes and a different distribution.
\nScores in general are better and less negative. As Kaggle has given no explanation yet we can not know what is going on, if there was a glitch, if it has been fixed or simply has diminishing effect as the dataset grows. In the latter case my initial glith candidates are no longer valid and I have at this point no new ones. Still, this LB dynamics is not what I expected and still think Kaggle should review their pipeline focusing on the first rerun data.
@miguelpm We'll look into it with @sohier after the weekend. High negative scores are of course plausible (due to a wrong sign or volatile features), but we'll check if there's something wrong.
", "rawMarkdown": "@miguelpm We'll look into it with @sohier after the weekend. High negative scores are of course plausible (due to a wrong sign or volatile features), but we'll check if there's something wrong.", "votes": 11, "replies": [ { "id": 1697560, "postDate": "2022-02-19T17:30:27.673Z", "content": "Hooray! Can you please also double check if new supplement data or API changes has caused any unexpected notebook failures? @amedprof posted an example of his notebook that ran in 24seconds and is not complex now fails:
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308221#1694758
\nI also experienced both notebooks failing :/
", "rawMarkdown": "Hooray! Can you please also double check if new supplement data or API changes has caused any unexpected notebook failures? @amedprof posted an example of his notebook that ran in 24seconds and is not complex now fails:\n\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308221#1694758\n\nI also experienced both notebooks failing :/\n\n\n\n" }, { "id": 1697603, "postDate": "2022-02-19T17:55:37.113Z", "content": "@cstein06 thank you for answering, it's great to know you are on it.
\nI recommend beginning by discarding accidental shift of one position in target, as described here https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308564, it would explain just too well current LB.
\nagain, thank you for answering, hopefully mystery will be explained :-)
", "rawMarkdown": "@cstein06 thank you for answering, it's great to know you are on it. \n\nI recommend beginning by discarding accidental shift of one position in target, as described here https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308564, it would explain just too well current LB. \n\nagain, thank you for answering, hopefully mystery will be explained :-) \n\n\n" }, { "id": 1698043, "postDate": "2022-02-20T05:02:57.603Z", "content": "I have a feeling, the negative scores ( -.9999 < x < -.01 ) are from notebooks, who are not directly predicting but 'calculating' the target ( as in predicting t+16 log_returns and then calculating the target from there). Not sure, if that will help in hunting the bug(ghost) .
\n(Raise your hand if not the case for you.)
", "rawMarkdown": "I have a feeling, the negative scores ( -.9999 < x < -.01 ) are from notebooks, who are not directly predicting but 'calculating' the target ( as in predicting t+16 log_returns and then calculating the target from there). Not sure, if that will help in hunting the bug(ghost) . \n\n(Raise your hand if not the case for you.)\n\n" }, { "id": 1698792, "postDate": "2022-02-20T16:53:15.240Z", "content": "One of my entries had a large negative correlation but I am not calculating as above.
", "rawMarkdown": "One of my entries had a large negative correlation but I am not calculating as above." }, { "id": 1701542, "postDate": "2022-02-22T21:09:10.360Z", "content": "@cstein06 Have you guys confirmed that the data is correct or have you not got around to it? Also, is there any chance the final supplemental data and data used in the evaluation set can be released. Even if the data is correct, I feel like people that got errors would like to try to figure out why.
\nAlso, thanks for hosting a great competition. Many competitions on Kaggle that I have done had anonymized features and during this competition it was actually nice to understand the data I was analyzing.
", "rawMarkdown": "@cstein06 Have you guys confirmed that the data is correct or have you not got around to it? Also, is there any chance the final supplemental data and data used in the evaluation set can be released. Even if the data is correct, I feel like people that got errors would like to try to figure out why. \n\nAlso, thanks for hosting a great competition. Many competitions on Kaggle that I have done had anonymized features and during this competition it was actually nice to understand the data I was analyzing. ", "votes": 1 }, { "id": 1704106, "postDate": "2022-02-25T07:00:35.357Z", "content": "Doesn't look like much movement here, hopefully if there was an error in the data it will be sorted in the next run and it will all come out in the wash!!
", "rawMarkdown": "Doesn't look like much movement here, hopefully if there was an error in the data it will be sorted in the next run and it will all come out in the wash!! " }, { "id": 1704255, "postDate": "2022-02-25T11:07:42.740Z", "content": "That would not be a serious way of handling the issue, nor would contribute to trust the results even if LB turns upside down. I think competing Kagglers deserve a bit more of respect than that.
\nI am quite puzzled by the complete lack of response by Kaggle staff after more than 7 days.
", "rawMarkdown": "That would not be a serious way of handling the issue, nor would contribute to trust the results even if LB turns upside down. I think competing Kagglers deserve a bit more of respect than that.\n\nI am quite puzzled by the complete lack of response by Kaggle staff after more than 7 days.", "votes": 2 }, { "id": 1704270, "postDate": "2022-02-25T11:18:56.753Z", "content": "I agree, I see that @sohier doesn't seem to have logged in since last week. Not too sure if this is important or not?
", "rawMarkdown": "I agree, I see that @sohier doesn't seem to have logged in since last week. Not too sure if this is important or not?" }, { "id": 1704293, "postDate": "2022-02-25T11:39:12.480Z", "content": "Regarding actual time he is dedicating to this we can't know from that, he might be on it.
\nBut yes, zero comments for a week, who knows, maybe he took a week off.
@sohier will hopefully have time next week. You should be patient, as we have three months for the evaluation period.
\n@petersk20 We'll discuss the data release, as this is not common practice. Thanks, I'm glad you enjoyed it! Indeed, most difficulties came from making it possible to use non-anonymous data, in order to have, in our opinion, a much more meaningful competition.
", "rawMarkdown": "@sohier will hopefully have time next week. You should be patient, as we have three months for the evaluation period.\n\n@petersk20 We'll discuss the data release, as this is not common practice. Thanks, I'm glad you enjoyed it! Indeed, most difficulties came from making it possible to use non-anonymous data, in order to have, in our opinion, a much more meaningful competition.", "votes": 3 }, { "id": 1704441, "postDate": "2022-02-25T14:20:17.183Z", "content": "Thanks for the feedback @cstein06; we are indeed patient but many are deeply invested with the effort put in. Also grateful for the competition, it has so far been a great learning opportunity.
", "rawMarkdown": "Thanks for the feedback @cstein06; we are indeed patient but many are deeply invested with the effort put in. Also grateful for the competition, it has so far been a great learning opportunity. ", "votes": 1 }, { "id": 1704467, "postDate": "2022-02-25T14:50:21.203Z", "content": "@cstein06 again thank you for the feedback.
\nAs @jonnydossantos says, everyone here is quite patient beginning by patiently investing months in solving a problem mainly for the desire of learning (and the fun).
\nBut without proper feedback there is no learning. And I seriously doubt anyone here is fully confident of what has worked or hasn't in their solutions in this moment.
\nAbout releasing data, imo. this comp. is different in that as data is public and release is only making clear the preprocessing you are making of it, for transparency. I don't see any downside to it, it would make everyone trust the LB and learn better what has or hasn't worked both regarding solutions and code.
", "rawMarkdown": "@cstein06 again thank you for the feedback.\n\nAs @jonnydossantos says, everyone here is quite patient beginning by patiently investing months in solving a problem mainly for the desire of learning (and the fun).\n\nBut without proper feedback there is no learning. And I seriously doubt anyone here is fully confident of what has worked or hasn't in their solutions in this moment.\n\nAbout releasing data, imo. this comp. is different in that as data is public and release is only making clear the preprocessing you are making of it, for transparency. I don't see any downside to it, it would make everyone trust the LB and learn better what has or hasn't worked both regarding solutions and code.", "votes": 2 }, { "id": 1704577, "postDate": "2022-02-25T16:33:07.430Z", "content": "FYI could be a licensing issue.. I have been running my models live since competition ended but with a paid provider (since building infra myself take too long) and thus have no redistribution rights to share the data with you guys (as I'm sure many would like to also test last few weeks)
\nThe data they already provided us with is quite generous imo, its nice to have 2gb of nicely aggregated minute data for several cryptos.
", "rawMarkdown": "FYI could be a licensing issue.. I have been running my models live since competition ended but with a paid provider (since building infra myself take too long) and thus have no redistribution rights to share the data with you guys (as I'm sure many would like to also test last few weeks)\n\nThe data they already provided us with is quite generous imo, its nice to have 2gb of nicely aggregated minute data for several cryptos." }, { "id": 1704614, "postDate": "2022-02-25T17:08:00.433Z", "content": "If license is different for this last months of data than for the previous one then no officlal release is possible, but not confirmed to be the case.
\nI don't have a feed myself for crypto asets nor have time to scrape the data, I spent already enough time with this. But @julianmukaj, maybe you can tell if you are observing such wild inversion of correlations in your models predictions (vs. validation) for recent data?
The first 2 weeks after competition (what we currently have an update for) my models pretty much held up as expected, correlations matched CV more or less for the ones I ran live.. Hence why I have been pretty annoyed my models both error'd because I think I would be top 10.
\nHowever in the last week or so since then it has been more shaky for my models, we have seen a lot of whipsaw action in crypto, BTC down 9% and then up 13% within a single 24h period (yesterday) for example. Feb 19th my models done somewhat poorly too, they were quite unprepared for the steep plunges in price. I assume the quick shift in volatility (due to Russia crisis etc) played a part.
\n
You can see in this period here (marked out by red) things went pretty uncorrelated (green is long/orange is short signal/gaps I chose to not trade), so I think the next update will cause a big shake up to the leaderboard to see who handled a stress test well.
\nOfc this won't be representative of how your models perform but might offer you guys some insight.
", "rawMarkdown": "The first 2 weeks after competition (what we currently have an update for) my models pretty much held up as expected, correlations matched CV more or less for the ones I ran live.. Hence why I have been pretty annoyed my models both error'd because I think I would be top 10.\n\nHowever in the last week or so since then it has been more shaky for my models, we have seen a lot of whipsaw action in crypto, BTC down 9% and then up 13% within a single 24h period (yesterday) for example. Feb 19th my models done somewhat poorly too, they were quite unprepared for the steep plunges in price. I assume the quick shift in volatility (due to Russia crisis etc) played a part.\n\n\n\nYou can see in this period here (marked out by red) things went pretty uncorrelated (green is long/orange is short signal/gaps I chose to not trade), so I think the next update will cause a big shake up to the leaderboard to see who handled a stress test well.\n\nOfc this won't be representative of how your models perform but might offer you guys some insight.\n\n", "votes": 1 }, { "id": 1704674, "postDate": "2022-02-25T17:46:56.700Z", "content": "That's interesting @julianmukaj , thanks for sharing it. I find the fact that for the first two weeks you see no fundamental concept shift adds to the evidence pointing to a problem in the rerun.
\nAs for current week volatility (and the rest of the competition time), this should be interesting to watch, the stability of our models vs fast changing conditions. But we need a functional evaluation to enjoy that show and -I might be wrong- but that's not what I think we have at this point
", "rawMarkdown": "That's interesting @julianmukaj , thanks for sharing it. I find the fact that for the first two weeks you see no fundamental concept shift adds to the evidence pointing to a problem in the rerun. \n\nAs for current week volatility (and the rest of the competition time), this should be interesting to watch, the stability of our models vs fast changing conditions. But we need a functional evaluation to enjoy that show and -I might be wrong- but that's not what I think we have at this point", "votes": 1 }, { "id": 1704704, "postDate": "2022-02-25T18:06:11.150Z", "content": "Yeah I do find the strong negative correlations strange but it is also possible the mid way shift you see on my model has happened to other models earlier perhaps?
\nI have also seen this happen in another competition where I put correlation as the loss function for a NN, due to squaring in the loss it just tries to maximize the corr whether positive or negative, perhaps any of you guys are also using Corr as a loss function?
\nI think the organisers are right though, there is 3 months of evaluation so there is no need to rush things. Plenty of time for peoples models to jump 100 places on the leaderboard up or down 😁
", "rawMarkdown": "Yeah I do find the strong negative correlations strange but it is also possible the mid way shift you see on my model has happened to other models earlier perhaps? \n\nI have also seen this happen in another competition where I put correlation as the loss function for a NN, due to squaring in the loss it just tries to maximize the corr whether positive or negative, perhaps any of you guys are also using Corr as a loss function?\n\nI think the organisers are right though, there is 3 months of evaluation so there is no need to rush things. Plenty of time for peoples models to jump 100 places on the leaderboard up or down 😁", "votes": 1 }, { "id": 1704771, "postDate": "2022-02-25T19:37:42.513Z", "content": "What i refer to as \"concept shift\", quite more problematic than \"covariate shift\", is as a property of the data more than of the models (assuming that most properly validated models will share a good deal of the patterns used) . The assumption (I am making) is, if you validated correctly your model and for those two weeks it does as expected ---> no concept shift.
\nAbout the optimizing for absolute correlation, nope, not here at least. And I dont think hundreds of Kagglers are shifting signs or squaring accidentaly stuff at the same time, at least not likely imo :-)
\nAnd if there is indeed a problem there will be much bigger jumps in LB than 100 positions, we will have simply a brand new LB and very possibly scores will be, in general, higher.
\nAnyway, yeah, looks like there's no other option than waiting a bit more.
", "rawMarkdown": "What i refer to as \"concept shift\", quite more problematic than \"covariate shift\", is as a property of the data more than of the models (assuming that most properly validated models will share a good deal of the patterns used) . The assumption (I am making) is, if you validated correctly your model and for those two weeks it does as expected ---> no concept shift. \n\nAbout the optimizing for absolute correlation, nope, not here at least. And I dont think hundreds of Kagglers are shifting signs or squaring accidentaly stuff at the same time, at least not likely imo :-)\n\nAnd if there is indeed a problem there will be much bigger jumps in LB than 100 positions, we will have simply a brand new LB and very possibly scores will be, in general, higher.\n\nAnyway, yeah, looks like there's no other option than waiting a bit more. \n\n\n\n", "votes": 1 } ] }, { "id": 1694905, "postDate": "2022-02-17T20:17:01.953Z", "content": "My score went from (non-leak) 0.0418 on public, to 0.0416 yesterday on some strange update of scores? and now every single submission is an error, even one that finished in a couple of seconds…
\nThis has not been smooth experience at all and @sohier is not really providing frequent updates to keep our confidence high…
\nI have been running my model live myself so I know what to expect from the last 2 weeks.. If I get stuck with this \"error\" and thus no placing at all, I'm not spending this much time on a kaggle competition again, it took a lot of effort..
\nI think its very unlikely this many people (including experience data scientists and kagglers) had a \"bug\" we hadn't noticed…
", "rawMarkdown": "My score went from (non-leak) 0.0418 on public, to 0.0416 yesterday on some strange update of scores? and now every single submission is an error, even one that finished in a couple of seconds... \n\nThis has not been smooth experience at all and @sohier is not really providing frequent updates to keep our confidence high...\n\nI have been running my model live myself so I know what to expect from the last 2 weeks.. If I get stuck with this \"error\" and thus no placing at all, I'm not spending this much time on a kaggle competition again, it took a lot of effort..\n\nI think its very unlikely this many people (including experience data scientists and kagglers) had a \"bug\" we hadn't noticed...", "votes": 5, "replies": [ { "id": 1694978, "postDate": "2022-02-17T21:46:27.100Z", "content": "Yeah, there was probably something they didn't warn people about. I wish they made the public lb train, test, supplemental split as close to the actual one as possible. The only difference I could think of is the extra data in the supplemental set.
\nAlso, how did you test your model? Did you collect the data yourself. If so, it might be helpful for other people who got errors/worse scores than expected if you uploaded the extra data as a dataset.
I used a paid data provider, will have to check if I can share the data or not. If so I will try upload the last 2 weeks.
\nI am pretty sure they have done something wrong again, I can’t see how a simple LightGBM baseline (not using supplement) can result in an error, surely someone on the current ranking has submitted a LightGBM solution with a lot of features, yet every single one of mine has resulted in an error.. one of my error notebooks was simply submitting a calculation of three columns and has also error’d; I don’t have a single submission of my total 80 that doesn’t show error? I got the score update on 15th with no issues; and no word from @sohier @cstein06 or @andrewscherer on even if they are looking into further issues or not…
", "rawMarkdown": "I used a paid data provider, will have to check if I can share the data or not. If so I will try upload the last 2 weeks.\n\nI am pretty sure they have done something wrong again, I can’t see how a simple LightGBM baseline (not using supplement) can result in an error, surely someone on the current ranking has submitted a LightGBM solution with a lot of features, yet every single one of mine has resulted in an error.. one of my error notebooks was simply submitting a calculation of three columns and has also error’d; I don’t have a single submission of my total 80 that doesn’t show error? I got the score update on 15th with no issues; and no word from @sohier @cstein06 or @andrewscherer on even if they are looking into further issues or not…", "votes": 1 }, { "id": 1695905, "postDate": "2022-02-18T12:54:58.120Z", "content": "True, same case here. Most of my submissions are error
", "rawMarkdown": "True, same case here. Most of my submissions are error", "votes": 1 }, { "id": 1696071, "postDate": "2022-02-18T15:11:31.350Z", "content": "It seems a bit confusing, but I am sure that the way it works is the following: suppose you have made a total of N submissions. Out of these N submission they only run the two selected ones. After the rerun you actually get N+2 entries, the first two listed are the actual reruns with their new scores, all the old N entries are still there but now they are automatically marked with error. I hope this helps.
", "rawMarkdown": "It seems a bit confusing, but I am sure that the way it works is the following: suppose you have made a total of N submissions. Out of these N submission they only run the two selected ones. After the rerun you actually get N+2 entries, the first two listed are the actual reruns with their new scores, all the old N entries are still there but now they are automatically marked with error. I hope this helps.", "votes": 1, "isDeleted": true }, { "id": 1696073, "postDate": "2022-02-18T15:12:25.900Z", "content": "I think everyone has at most 2 submissions without error at present(the ones you choose) in their list, which matches the entries numbers in the lb now. They set the other submission with error by default, because they want to show the result only in the public lb for now, if they keep the ones you didn't choose, and if they are higher, then they may be shown on the public lb making rank meaningless. So I guess if someone has an error, there may be something wrong in their 2 submitted notebooks evaluation process, the other notebooks' error should not be concerned.
", "rawMarkdown": "I think everyone has at most 2 submissions without error at present(the ones you choose) in their list, **which matches the entries numbers in the lb now**. They set the other submission with error by default, because they want to show the result **only in the public lb for now**, if they keep the ones you didn't choose, and if they are higher, then they may be shown on the public lb making rank meaningless. So I guess if someone has an error, there may be something wrong in their 2 submitted notebooks evaluation process, the other notebooks' error should not be concerned.", "votes": 2 } ] }, { "id": 1695548, "postDate": "2022-02-18T07:55:44.423Z", "content": "I'm also currently sitting with a big negative -0.0350. Bit confused too ;) Pretty confident I didn't get my signs the wrong way around.
", "rawMarkdown": "I'm also currently sitting with a big negative **-0.0350**. Bit confused too ;) Pretty confident I didn't get my signs the wrong way around.", "votes": 3 }, { "id": 1695230, "postDate": "2022-02-18T03:05:42.913Z", "content": "our team scores is also a large negative correlation -0.0350, in 20210613-20210913 it's about 0.03, the calculation function differs with the public score function?
", "rawMarkdown": "our team scores is also a large negative correlation -0.0350, in 20210613-20210913 it's about 0.03, the calculation function differs with the public score function?", "votes": 3, "replies": [ { "id": 1696161, "postDate": "2022-02-18T16:23:42.227Z", "content": "I'm in 2nd place behind you for biggest negative score not equal to -1. :D
", "rawMarkdown": "I'm in 2nd place behind you for biggest negative score not equal to -1. :D", "votes": 2 } ] }, { "id": 1696072, "postDate": "2022-02-18T15:12:20.003Z", "content": "Same here! During submission period, no single negative score, but both of my submissions score highly negative. Interestingly the better model (previously .0224 public) scored -.027x and the second chosen model (previously .021) scored -.025x. So the better model scored more negative.
\nAll my ensemble models use online feature engineering, and using the latest data, the farthest looking to 6 days past. And this was meant to counter the momentum stuff and all that… Weird..
\nLike many others, I have a local pipeline, so awaiting the host to publish the two weeks data in question.
\nI wouldn't think they wouldn't, Why they wouldn't??
\nLet's see.
I tried that If the submissions all equal 0 then the score will be -1
", "rawMarkdown": "I tried that If the submissions all equal 0 then the score will be -1", "votes": 1 }, { "id": 1696503, "postDate": "2022-02-18T22:49:47.450Z", "content": "Totally agree, excellent thread and comments @miguelpm
\nThanks!
", "rawMarkdown": "Totally agree, excellent thread and comments @miguelpm \n\nThanks!", "votes": 1, "replies": [ { "id": 1696561, "postDate": "2022-02-18T23:50:28.893Z", "content": "Thank you for you kind words @coreacasa , yes I think it has helped to make this potential rerun error \"not a silent error\". Let's hope Kaggle will address the situation and dissipate our doubts in one way or the other!
", "rawMarkdown": "Thank you for you kind words @coreacasa , yes I think it has helped to make this potential rerun error \"not a silent error\". Let's hope Kaggle will address the situation and dissipate our doubts in one way or the other!" } ] }, { "id": 1694996, "postDate": "2022-02-17T21:58:30.393Z", "content": "@miguelpm i do share your concern despite my lb placement. It does seem possible that there could be a data provider issue similar to what happened with the first supplemental train update - there were a small number of massively incorrect crypto prices which badly affected the total correlation. However I'm doubtful/hopeful that this would be allowed to happen again.
\nI tested my models on 40 out of sample two week periods and i scored worse than my current lb score on only 3 periods.
\nSo it may also suggest it was just a difficult period for all models.
@tomforbes I've been thinking what can be compatible with negative correlations affecting to some of the subs, but not others. Right now my bet would be there is some sorting made different in the rerun pipeline that has impact depending on the notebooks. Other than that… I don't think negative high correlations should be appearing here. I mean, near -0.04 correlations. Happy to hear an explanation on how that can be happening here. (Apart of the obvious of fliping the predictions sign in the code of a great model, what is not the case here :-) )
", "rawMarkdown": "@tomforbes I've been thinking what can be compatible with negative correlations affecting to some of the subs, but not others. Right now my bet would be there is some sorting made different in the rerun pipeline that has impact depending on the notebooks. Other than that... I don't think negative high correlations should be appearing here. I mean, near -0.04 correlations. Happy to hear an explanation on how that can be happening here. (Apart of the obvious of fliping the predictions sign in the code of a great model, what is not the case here :-) )\n", "votes": 2 }, { "id": 1695943, "postDate": "2022-02-18T13:31:57.997Z", "content": "My thoughts on all of this are:
\nNon scored submissions all error, yes, but regarding momentum effect, realize target is adaptatively residualized.
\nAnd anyway, my comment acknowledges that some submissions seem to be behaving as expected, or slightly more poorly, while at the same time a very big number of other submissions seem wildly unrelated to validation scores.
\nThere can be many explanations, regime changes as you point at, a bug in the notebooks… but imo the biggest likelihood now is problems with the rerun code/data. And considering it would not be the first time such a thing happens I think it is a problem worth pointing at.
", "rawMarkdown": "Non scored submissions all error, yes, but regarding momentum effect, realize target is adaptatively residualized.\n\nAnd anyway, my comment acknowledges that some submissions seem to be behaving as expected, or slightly more poorly, while at the same time a very big number of other submissions seem wildly unrelated to validation scores. \n\nThere can be many explanations, regime changes as you point at, a bug in the notebooks... but imo the biggest likelihood now is problems with the rerun code/data. And considering it would not be the first time such a thing happens I think it is a problem worth pointing at.\n", "votes": 1 }, { "id": 1695996, "postDate": "2022-02-18T14:29:50.367Z", "content": "\n\nNon scored submissions all error, yes,
\n
Is this confirmed somewhere? On the re-run of score changes we had 2 days ago I did not get errors on my unscored notebooks..?
\nAlso yes considering we had glitches with the data and scoring already a few times, its reasonable to assume we may of had another now IMO..
", "rawMarkdown": ">Non scored submissions all error, yes,\n\nIs this confirmed somewhere? On the re-run of score changes we had 2 days ago I did not get errors on my unscored notebooks..?\n\nAlso yes considering we had glitches with the data and scoring already a few times, its reasonable to assume we may of had another now IMO.." }, { "id": 1696009, "postDate": "2022-02-18T14:37:50.363Z", "content": "I see all my non scored subs with error and consider quite possible that they just brute-forced the rerun recovering only the two chosen notebooks.
\nBut, about official confirmations, we are not having much of those here lately, no trace of Kaggle staff comments by now ;-)
Well they should be at work in 2 hours over in PST so hopefully we get some info or I'll just throw in the towel lol, wasted a month trying to break 0.04..
\nI guess will just try a late submission in 10 weeks to see what my score would have been
", "rawMarkdown": "Well they should be at work in 2 hours over in PST so hopefully we get some info or I'll just throw in the towel lol, wasted a month trying to break 0.04..\n\nI guess will just try a late submission in 10 weeks to see what my score *would* have been", "votes": 1 } ] }, { "id": 1695441, "postDate": "2022-02-18T06:39:21.090Z", "content": "@miguelpm, maybe they give us more than one minute of data per iteration? or is it just the usual variation we should expect with this kind of data?
\nI am right now understanding why grandmasters don't enter this sort of competition.
", "rawMarkdown": "@miguelpm, maybe they give us more than one minute of data per iteration? or is it just the usual variation we should expect with this kind of data?\n\nI am right now understanding why grandmasters don't enter this sort of competition.", "votes": 1, "replies": [ { "id": 1695627, "postDate": "2022-02-18T09:17:56.210Z", "content": "@agenlu might be something like that, whatever it is seems to have broken many of the submissions.
\nI think this is more serious than the usual reason not to enter high-noise time series competition. One thing is that result is less correlated with skills, another thing is that your submission is piped to a Random Number Generator before scoring. I like noisy comps, but not willing to take part if Kaggle doesn't dedicate the minimal time to make evaluation bug-free.
", "rawMarkdown": "@agenlu might be something like that, whatever it is seems to have broken many of the submissions.\n\nI think this is more serious than the usual reason not to enter high-noise time series competition. One thing is that result is less correlated with skills, another thing is that your submission is piped to a Random Number Generator before scoring. I like noisy comps, but not willing to take part if Kaggle doesn't dedicate the minimal time to make evaluation bug-free.", "votes": 3 }, { "id": 1695661, "postDate": "2022-02-18T09:44:10.867Z", "content": "@miguelpm Of course, the problem is not because is a high-noise time series is how Kaggle handles this competition. From the beginning, the host and Kaggle didn't seem very interested in it.
", "rawMarkdown": "@miguelpm Of course, the problem is not because is a high-noise time series is how Kaggle handles this competition. From the beginning, the host and Kaggle didn't seem very interested in it.", "votes": 2 } ] }, { "id": 1711977, "postDate": "2022-03-04T14:14:52.257Z", "content": "@cstein06 I'm curious as to whether the target calculation for the private LB data is consistent with that of supplemental_train.csv. Particularly, whether it has been calculated from a running start (i.e. with a warm-up of ~3770 timestamps from the contiguous end of supplemental_train.csv), or from a standing start (like at the beginning of train.csv). Please could you clarify?
", "rawMarkdown": "@cstein06 I'm curious as to whether the target calculation for the private LB data is consistent with that of supplemental_train.csv. Particularly, whether it has been calculated from a **running start** (i.e. with a warm-up of ~3770 timestamps from the contiguous end of supplemental_train.csv), or from a **standing start** (like at the beginning of train.csv). Please could you clarify? ", "replies": [ { "id": 1712026, "postDate": "2022-03-04T14:59:13.490Z", "content": "@jagofc The target is calculated from a running start.
", "rawMarkdown": "@jagofc The target is calculated from a running start.", "votes": 2 }, { "id": 1712035, "postDate": "2022-03-04T15:06:01.923Z", "content": "Good to know, thanks for confirming @cstein06
", "rawMarkdown": "Good to know, thanks for confirming @cstein06 " } ] }, { "id": 1698669, "postDate": "2022-02-20T15:09:13.783Z", "content": "couldn't find any cause too
", "rawMarkdown": "couldn't find any cause too" }, { "id": 1696844, "postDate": "2022-02-19T06:23:24.167Z", "content": "Did you use a reconstructed target ? Did you check for positive score on LB ?
\nI worked on a reconstructed target from log returns, but as I was working on log(o/c) I had a minus sign error for a while.
hi @lucasmorin, yes I used reconstructed target but I think I have found out what has probably happened here. I'm posting it in its own thread as a good glitch candidate, hopefully Kaggle staff will notice it, if I am right will make their bug hunting easier, writing it right now.
", "rawMarkdown": "hi @lucasmorin, yes I used reconstructed target but I think I have found out what has probably happened here. I'm posting it in its own thread as a good glitch candidate, hopefully Kaggle staff will notice it, if I am right will make their bug hunting easier, writing it right now.", "votes": 1 } ] }, { "id": 1696039, "postDate": "2022-02-18T14:55:46.777Z", "content": "I think negative correlations could make perfect sense. At least in my case. I used very simple models that work with the \"reverting to the mean\" idea. It uses info from the last 30 minutes. In most months it works OK and scores around 0.03 or 0.04 but in a few other months (probably when the market was strongly momentum driven) it scores -0.01 or -0.02. So how well this \"model\" works depends on the type of market. Perhaps the last two weeks was more momentum driven than usual, so that would explain my -0.0162 and -.0169 scores. Too bad, but at least it makes sense.
", "rawMarkdown": "I think negative correlations could make perfect sense. At least in my case. I used very simple models that work with the \"reverting to the mean\" idea. It uses info from the last 30 minutes. In most months it works OK and scores around 0.03 or 0.04 but in a few other months (probably when the market was strongly momentum driven) it scores -0.01 or -0.02. So how well this \"model\" works depends on the type of market. Perhaps the last two weeks was more momentum driven than usual, so that would explain my -0.0162 and -.0169 scores. Too bad, but at least it makes sense.", "votes": 3, "isDeleted": true, "replies": [ { "id": 1696137, "postDate": "2022-02-18T16:05:32.500Z", "content": "Well, it's possible. But I think that point misses something important:
\nAnyway, we'll see, I hope.
", "rawMarkdown": "Well, it's possible. But I think that point misses something important:\n- target is not market prices here, but residualized prices that are, by design mean reverting no matter market conditions, and show a pretty stable autocorrelation structure in all train dataset.\n- can this be different at test time? sure, maybe this two weeks exhibit opposite pattern of autocorrelations to the one in train, a nice example of \"concept shift\". But it would be a bit weird, hence this thread. Also with such a concept shift hardly any model would show positive scores, unless trained disregarding train autocorrelation structure (not likely).\n\nAnyway, we'll see, I hope.", "votes": 1 }, { "id": 1696193, "postDate": "2022-02-18T16:41:23.763Z", "content": "Agreed, will be interesting to see if indeed there is an error in to test data that has slipped by and if we see a return to positive correlation!
", "rawMarkdown": "Agreed, will be interesting to see if indeed there is an error in to test data that has slipped by and if we see a return to positive correlation!", "votes": 1 } ] }, { "id": 1695758, "postDate": "2022-02-18T10:49:05.933Z", "rawMarkdown": "", "votes": 4, "isDeleted": true } ], "comments": [ { "id": 1697550, "author_name": "Carlos Stein N Brito", "author_url": "", "post_date": "2022-02-19T17:24:28.107000", "content": "@miguelpm We'll look into it with @sohier after the weekend. High negative scores are of course plausible (due to a wrong sign or volatile features), but we'll check if there's something wrong.
", "votes": 11, "replies": [ { "id": 1697560, "author_name": "JM", "author_url": "", "post_date": "2022-02-19T17:30:27.673000", "content": "Hooray! Can you please also double check if new supplement data or API changes has caused any unexpected notebook failures? @amedprof posted an example of his notebook that ran in 24seconds and is not complex now fails:
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308221#1694758
\nI also experienced both notebooks failing :/
", "votes": 0, "replies": [] }, { "id": 1697603, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-19T17:55:37.113000", "content": "@cstein06 thank you for answering, it's great to know you are on it.
\nI recommend beginning by discarding accidental shift of one position in target, as described here https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308564, it would explain just too well current LB.
\nagain, thank you for answering, hopefully mystery will be explained :-)
", "votes": 0, "replies": [] }, { "id": 1698043, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-20T05:02:57.603000", "content": "I have a feeling, the negative scores ( -.9999 < x < -.01 ) are from notebooks, who are not directly predicting but 'calculating' the target ( as in predicting t+16 log_returns and then calculating the target from there). Not sure, if that will help in hunting the bug(ghost) .
\n(Raise your hand if not the case for you.)
", "votes": 0, "replies": [] }, { "id": 1698792, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-20T16:53:15.240000", "content": "One of my entries had a large negative correlation but I am not calculating as above.
", "votes": 0, "replies": [] }, { "id": 1701542, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-22T21:09:10.360000", "content": "@cstein06 Have you guys confirmed that the data is correct or have you not got around to it? Also, is there any chance the final supplemental data and data used in the evaluation set can be released. Even if the data is correct, I feel like people that got errors would like to try to figure out why.
\nAlso, thanks for hosting a great competition. Many competitions on Kaggle that I have done had anonymized features and during this competition it was actually nice to understand the data I was analyzing.
", "votes": 1, "replies": [] }, { "id": 1704106, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-25T07:00:35.357000", "content": "Doesn't look like much movement here, hopefully if there was an error in the data it will be sorted in the next run and it will all come out in the wash!!
", "votes": 0, "replies": [] }, { "id": 1704255, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-25T11:07:42.740000", "content": "That would not be a serious way of handling the issue, nor would contribute to trust the results even if LB turns upside down. I think competing Kagglers deserve a bit more of respect than that.
\nI am quite puzzled by the complete lack of response by Kaggle staff after more than 7 days.
", "votes": 2, "replies": [] }, { "id": 1704270, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-25T11:18:56.753000", "content": "I agree, I see that @sohier doesn't seem to have logged in since last week. Not too sure if this is important or not?
", "votes": 0, "replies": [] }, { "id": 1704293, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-25T11:39:12.480000", "content": "Regarding actual time he is dedicating to this we can't know from that, he might be on it.
\nBut yes, zero comments for a week, who knows, maybe he took a week off.
@sohier will hopefully have time next week. You should be patient, as we have three months for the evaluation period.
\n@petersk20 We'll discuss the data release, as this is not common practice. Thanks, I'm glad you enjoyed it! Indeed, most difficulties came from making it possible to use non-anonymous data, in order to have, in our opinion, a much more meaningful competition.
", "votes": 3, "replies": [] }, { "id": 1704441, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-25T14:20:17.183000", "content": "Thanks for the feedback @cstein06; we are indeed patient but many are deeply invested with the effort put in. Also grateful for the competition, it has so far been a great learning opportunity.
", "votes": 1, "replies": [] }, { "id": 1704467, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-25T14:50:21.203000", "content": "@cstein06 again thank you for the feedback.
\nAs @jonnydossantos says, everyone here is quite patient beginning by patiently investing months in solving a problem mainly for the desire of learning (and the fun).
\nBut without proper feedback there is no learning. And I seriously doubt anyone here is fully confident of what has worked or hasn't in their solutions in this moment.
\nAbout releasing data, imo. this comp. is different in that as data is public and release is only making clear the preprocessing you are making of it, for transparency. I don't see any downside to it, it would make everyone trust the LB and learn better what has or hasn't worked both regarding solutions and code.
", "votes": 2, "replies": [] }, { "id": 1704577, "author_name": "JM", "author_url": "", "post_date": "2022-02-25T16:33:07.430000", "content": "FYI could be a licensing issue.. I have been running my models live since competition ended but with a paid provider (since building infra myself take too long) and thus have no redistribution rights to share the data with you guys (as I'm sure many would like to also test last few weeks)
\nThe data they already provided us with is quite generous imo, its nice to have 2gb of nicely aggregated minute data for several cryptos.
", "votes": 0, "replies": [] }, { "id": 1704614, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-25T17:08:00.433000", "content": "If license is different for this last months of data than for the previous one then no officlal release is possible, but not confirmed to be the case.
\nI don't have a feed myself for crypto asets nor have time to scrape the data, I spent already enough time with this. But @julianmukaj, maybe you can tell if you are observing such wild inversion of correlations in your models predictions (vs. validation) for recent data?
The first 2 weeks after competition (what we currently have an update for) my models pretty much held up as expected, correlations matched CV more or less for the ones I ran live.. Hence why I have been pretty annoyed my models both error'd because I think I would be top 10.
\nHowever in the last week or so since then it has been more shaky for my models, we have seen a lot of whipsaw action in crypto, BTC down 9% and then up 13% within a single 24h period (yesterday) for example. Feb 19th my models done somewhat poorly too, they were quite unprepared for the steep plunges in price. I assume the quick shift in volatility (due to Russia crisis etc) played a part.
\n
You can see in this period here (marked out by red) things went pretty uncorrelated (green is long/orange is short signal/gaps I chose to not trade), so I think the next update will cause a big shake up to the leaderboard to see who handled a stress test well.
\nOfc this won't be representative of how your models perform but might offer you guys some insight.
", "votes": 1, "replies": [] }, { "id": 1704674, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-25T17:46:56.700000", "content": "That's interesting @julianmukaj , thanks for sharing it. I find the fact that for the first two weeks you see no fundamental concept shift adds to the evidence pointing to a problem in the rerun.
\nAs for current week volatility (and the rest of the competition time), this should be interesting to watch, the stability of our models vs fast changing conditions. But we need a functional evaluation to enjoy that show and -I might be wrong- but that's not what I think we have at this point
", "votes": 1, "replies": [] }, { "id": 1704704, "author_name": "JM", "author_url": "", "post_date": "2022-02-25T18:06:11.150000", "content": "Yeah I do find the strong negative correlations strange but it is also possible the mid way shift you see on my model has happened to other models earlier perhaps?
\nI have also seen this happen in another competition where I put correlation as the loss function for a NN, due to squaring in the loss it just tries to maximize the corr whether positive or negative, perhaps any of you guys are also using Corr as a loss function?
\nI think the organisers are right though, there is 3 months of evaluation so there is no need to rush things. Plenty of time for peoples models to jump 100 places on the leaderboard up or down 😁
", "votes": 1, "replies": [] }, { "id": 1704771, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-25T19:37:42.513000", "content": "What i refer to as \"concept shift\", quite more problematic than \"covariate shift\", is as a property of the data more than of the models (assuming that most properly validated models will share a good deal of the patterns used) . The assumption (I am making) is, if you validated correctly your model and for those two weeks it does as expected ---> no concept shift.
\nAbout the optimizing for absolute correlation, nope, not here at least. And I dont think hundreds of Kagglers are shifting signs or squaring accidentaly stuff at the same time, at least not likely imo :-)
\nAnd if there is indeed a problem there will be much bigger jumps in LB than 100 positions, we will have simply a brand new LB and very possibly scores will be, in general, higher.
\nAnyway, yeah, looks like there's no other option than waiting a bit more.
", "votes": 1, "replies": [] } ] }, { "id": 1694905, "author_name": "JM", "author_url": "", "post_date": "2022-02-17T20:17:01.953000", "content": "My score went from (non-leak) 0.0418 on public, to 0.0416 yesterday on some strange update of scores? and now every single submission is an error, even one that finished in a couple of seconds…
\nThis has not been smooth experience at all and @sohier is not really providing frequent updates to keep our confidence high…
\nI have been running my model live myself so I know what to expect from the last 2 weeks.. If I get stuck with this \"error\" and thus no placing at all, I'm not spending this much time on a kaggle competition again, it took a lot of effort..
\nI think its very unlikely this many people (including experience data scientists and kagglers) had a \"bug\" we hadn't noticed…
", "votes": 5, "replies": [ { "id": 1694978, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-17T21:46:27.100000", "content": "Yeah, there was probably something they didn't warn people about. I wish they made the public lb train, test, supplemental split as close to the actual one as possible. The only difference I could think of is the extra data in the supplemental set.
\nAlso, how did you test your model? Did you collect the data yourself. If so, it might be helpful for other people who got errors/worse scores than expected if you uploaded the extra data as a dataset.
I used a paid data provider, will have to check if I can share the data or not. If so I will try upload the last 2 weeks.
\nI am pretty sure they have done something wrong again, I can’t see how a simple LightGBM baseline (not using supplement) can result in an error, surely someone on the current ranking has submitted a LightGBM solution with a lot of features, yet every single one of mine has resulted in an error.. one of my error notebooks was simply submitting a calculation of three columns and has also error’d; I don’t have a single submission of my total 80 that doesn’t show error? I got the score update on 15th with no issues; and no word from @sohier @cstein06 or @andrewscherer on even if they are looking into further issues or not…
", "votes": 1, "replies": [] }, { "id": 1695905, "author_name": "neoistheone", "author_url": "", "post_date": "2022-02-18T12:54:58.120000", "content": "True, same case here. Most of my submissions are error
", "votes": 1, "replies": [] }, { "id": 1696071, "author_name": "", "author_url": "", "post_date": "2022-02-18T15:11:31.350000", "content": "It seems a bit confusing, but I am sure that the way it works is the following: suppose you have made a total of N submissions. Out of these N submission they only run the two selected ones. After the rerun you actually get N+2 entries, the first two listed are the actual reruns with their new scores, all the old N entries are still there but now they are automatically marked with error. I hope this helps.
", "votes": 1, "replies": [] }, { "id": 1696073, "author_name": "Ryexxl", "author_url": "", "post_date": "2022-02-18T15:12:25.900000", "content": "I think everyone has at most 2 submissions without error at present(the ones you choose) in their list, which matches the entries numbers in the lb now. They set the other submission with error by default, because they want to show the result only in the public lb for now, if they keep the ones you didn't choose, and if they are higher, then they may be shown on the public lb making rank meaningless. So I guess if someone has an error, there may be something wrong in their 2 submitted notebooks evaluation process, the other notebooks' error should not be concerned.
", "votes": 2, "replies": [] } ] }, { "id": 1695548, "author_name": "jagofc", "author_url": "", "post_date": "2022-02-18T07:55:44.423000", "content": "I'm also currently sitting with a big negative -0.0350. Bit confused too ;) Pretty confident I didn't get my signs the wrong way around.
", "votes": 3, "replies": [] }, { "id": 1695230, "author_name": "Hayden234", "author_url": "", "post_date": "2022-02-18T03:05:42.913000", "content": "our team scores is also a large negative correlation -0.0350, in 20210613-20210913 it's about 0.03, the calculation function differs with the public score function?
", "votes": 3, "replies": [ { "id": 1696161, "author_name": "jagofc", "author_url": "", "post_date": "2022-02-18T16:23:42.227000", "content": "I'm in 2nd place behind you for biggest negative score not equal to -1. :D
", "votes": 2, "replies": [] } ] }, { "id": 1696072, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-18T15:12:20.003000", "content": "Same here! During submission period, no single negative score, but both of my submissions score highly negative. Interestingly the better model (previously .0224 public) scored -.027x and the second chosen model (previously .021) scored -.025x. So the better model scored more negative.
\nAll my ensemble models use online feature engineering, and using the latest data, the farthest looking to 6 days past. And this was meant to counter the momentum stuff and all that… Weird..
\nLike many others, I have a local pipeline, so awaiting the host to publish the two weeks data in question.
\nI wouldn't think they wouldn't, Why they wouldn't??
\nLet's see.
I tried that If the submissions all equal 0 then the score will be -1
", "votes": 1, "replies": [] }, { "id": 1696503, "author_name": "Manuel Campos", "author_url": "", "post_date": "2022-02-18T22:49:47.450000", "content": "Totally agree, excellent thread and comments @miguelpm
\nThanks!
", "votes": 1, "replies": [ { "id": 1696561, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-18T23:50:28.893000", "content": "Thank you for you kind words @coreacasa , yes I think it has helped to make this potential rerun error \"not a silent error\". Let's hope Kaggle will address the situation and dissipate our doubts in one way or the other!
", "votes": 0, "replies": [] } ] }, { "id": 1694996, "author_name": "Tom Forbes", "author_url": "", "post_date": "2022-02-17T21:58:30.393000", "content": "@miguelpm i do share your concern despite my lb placement. It does seem possible that there could be a data provider issue similar to what happened with the first supplemental train update - there were a small number of massively incorrect crypto prices which badly affected the total correlation. However I'm doubtful/hopeful that this would be allowed to happen again.
\nI tested my models on 40 out of sample two week periods and i scored worse than my current lb score on only 3 periods.
\nSo it may also suggest it was just a difficult period for all models.
@tomforbes I've been thinking what can be compatible with negative correlations affecting to some of the subs, but not others. Right now my bet would be there is some sorting made different in the rerun pipeline that has impact depending on the notebooks. Other than that… I don't think negative high correlations should be appearing here. I mean, near -0.04 correlations. Happy to hear an explanation on how that can be happening here. (Apart of the obvious of fliping the predictions sign in the code of a great model, what is not the case here :-) )
", "votes": 2, "replies": [] }, { "id": 1695943, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-18T13:31:57.997000", "content": "My thoughts on all of this are:
\nNon scored submissions all error, yes, but regarding momentum effect, realize target is adaptatively residualized.
\nAnd anyway, my comment acknowledges that some submissions seem to be behaving as expected, or slightly more poorly, while at the same time a very big number of other submissions seem wildly unrelated to validation scores.
\nThere can be many explanations, regime changes as you point at, a bug in the notebooks… but imo the biggest likelihood now is problems with the rerun code/data. And considering it would not be the first time such a thing happens I think it is a problem worth pointing at.
", "votes": 1, "replies": [] }, { "id": 1695996, "author_name": "JM", "author_url": "", "post_date": "2022-02-18T14:29:50.367000", "content": "\n\nNon scored submissions all error, yes,
\n
Is this confirmed somewhere? On the re-run of score changes we had 2 days ago I did not get errors on my unscored notebooks..?
\nAlso yes considering we had glitches with the data and scoring already a few times, its reasonable to assume we may of had another now IMO..
", "votes": 0, "replies": [] }, { "id": 1696009, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-18T14:37:50.363000", "content": "I see all my non scored subs with error and consider quite possible that they just brute-forced the rerun recovering only the two chosen notebooks.
\nBut, about official confirmations, we are not having much of those here lately, no trace of Kaggle staff comments by now ;-)
Well they should be at work in 2 hours over in PST so hopefully we get some info or I'll just throw in the towel lol, wasted a month trying to break 0.04..
\nI guess will just try a late submission in 10 weeks to see what my score would have been
", "votes": 1, "replies": [] } ] }, { "id": 1695441, "author_name": "agenlu", "author_url": "", "post_date": "2022-02-18T06:39:21.090000", "content": "@miguelpm, maybe they give us more than one minute of data per iteration? or is it just the usual variation we should expect with this kind of data?
\nI am right now understanding why grandmasters don't enter this sort of competition.
", "votes": 1, "replies": [ { "id": 1695627, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-18T09:17:56.210000", "content": "@agenlu might be something like that, whatever it is seems to have broken many of the submissions.
\nI think this is more serious than the usual reason not to enter high-noise time series competition. One thing is that result is less correlated with skills, another thing is that your submission is piped to a Random Number Generator before scoring. I like noisy comps, but not willing to take part if Kaggle doesn't dedicate the minimal time to make evaluation bug-free.
", "votes": 3, "replies": [] }, { "id": 1695661, "author_name": "agenlu", "author_url": "", "post_date": "2022-02-18T09:44:10.867000", "content": "@miguelpm Of course, the problem is not because is a high-noise time series is how Kaggle handles this competition. From the beginning, the host and Kaggle didn't seem very interested in it.
", "votes": 2, "replies": [] } ] }, { "id": 1711977, "author_name": "jagofc", "author_url": "", "post_date": "2022-03-04T14:14:52.257000", "content": "@cstein06 I'm curious as to whether the target calculation for the private LB data is consistent with that of supplemental_train.csv. Particularly, whether it has been calculated from a running start (i.e. with a warm-up of ~3770 timestamps from the contiguous end of supplemental_train.csv), or from a standing start (like at the beginning of train.csv). Please could you clarify?
", "votes": 0, "replies": [ { "id": 1712026, "author_name": "Carlos Stein N Brito", "author_url": "", "post_date": "2022-03-04T14:59:13.490000", "content": "@jagofc The target is calculated from a running start.
", "votes": 2, "replies": [] }, { "id": 1712035, "author_name": "jagofc", "author_url": "", "post_date": "2022-03-04T15:06:01.923000", "content": "Good to know, thanks for confirming @cstein06
", "votes": 0, "replies": [] } ] }, { "id": 1698669, "author_name": "kiwikiwi2021", "author_url": "", "post_date": "2022-02-20T15:09:13.783000", "content": "couldn't find any cause too
", "votes": 0, "replies": [] }, { "id": 1696844, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-02-19T06:23:24.167000", "content": "Did you use a reconstructed target ? Did you check for positive score on LB ?
\nI worked on a reconstructed target from log returns, but as I was working on log(o/c) I had a minus sign error for a while.
hi @lucasmorin, yes I used reconstructed target but I think I have found out what has probably happened here. I'm posting it in its own thread as a good glitch candidate, hopefully Kaggle staff will notice it, if I am right will make their bug hunting easier, writing it right now.
", "votes": 1, "replies": [] } ] }, { "id": 1696039, "author_name": "", "author_url": "", "post_date": "2022-02-18T14:55:46.777000", "content": "I think negative correlations could make perfect sense. At least in my case. I used very simple models that work with the \"reverting to the mean\" idea. It uses info from the last 30 minutes. In most months it works OK and scores around 0.03 or 0.04 but in a few other months (probably when the market was strongly momentum driven) it scores -0.01 or -0.02. So how well this \"model\" works depends on the type of market. Perhaps the last two weeks was more momentum driven than usual, so that would explain my -0.0162 and -.0169 scores. Too bad, but at least it makes sense.
", "votes": 3, "replies": [ { "id": 1696137, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-18T16:05:32.500000", "content": "Well, it's possible. But I think that point misses something important:
\nAnyway, we'll see, I hope.
", "votes": 1, "replies": [] }, { "id": 1696193, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-18T16:41:23.763000", "content": "Agreed, will be interesting to see if indeed there is an error in to test data that has slipped by and if we see a return to positive correlation!
", "votes": 1, "replies": [] } ] }, { "id": 1695758, "author_name": "", "author_url": "", "post_date": "2022-02-18T10:49:05.933000", "content": "", "votes": 4, "replies": [] } ], "raw_markdown_by_id": { "1694894": "Wow. This first rerun is really weird, and I would like to understand what is going on.\n\nHaving very present that:\n\n- this is a low signal, high noise comp.\n- the target had a weird design with weird implications, some of them being data quality dependence, residualization weights used also for evaluation, unavailable future period to include blindly...\n- and, of course, Kaggle time series API, not very debugging friendly to put it mildly.\n\nI don't discard bugs in my final solutions but watching the leaderboard after the first rerun has me scratching my head.\n\nWhy? Well, my both models have huge negative score, worst model has **negative correlation (-0.0332) score!!!** . What doesn't make sense. I mean, same correlation amount positive would imply #6 in the leaderboard. That is A LOT of negative correlation. That is signal.\n\nWhat leads me to two posibilities:\n1) A dramatic bug in my code that, let's be clear, I consider perfectly possible.\n2) A problem with the rerun.\n\nThis said, the only thing I would ask is please, @sohier have a look at the distribution of the scores in the LB and please double check they make sense, including other instances of high negative correlation, to discard some problem with the rerun dataset quality or the joining with the previous one. Just to double check that the time dedicated to this comp. is not wasted because some evaluation glitch.\n\n**UPDATE2 (second rerun): LB shows now huge changes and a different distribution**. \nScores in general are better and less negative. As Kaggle has given no explanation yet we can not know what is going on, if there was a glitch, if it has been fixed or simply has diminishing effect as the dataset grows. In the latter case my initial glith candidates are no longer valid and I have at this point no new ones. Still, this LB dynamics is not what I expected and still think Kaggle should review their pipeline focusing on the first rerun data.\n", "1697550": "@miguelpm We'll look into it with @sohier after the weekend. High negative scores are of course plausible (due to a wrong sign or volatile features), but we'll check if there's something wrong.", "1694905": "My score went from (non-leak) 0.0418 on public, to 0.0416 yesterday on some strange update of scores? and now every single submission is an error, even one that finished in a couple of seconds... \n\nThis has not been smooth experience at all and @sohier is not really providing frequent updates to keep our confidence high...\n\nI have been running my model live myself so I know what to expect from the last 2 weeks.. If I get stuck with this \"error\" and thus no placing at all, I'm not spending this much time on a kaggle competition again, it took a lot of effort..\n\nI think its very unlikely this many people (including experience data scientists and kagglers) had a \"bug\" we hadn't noticed...", "1695548": "I'm also currently sitting with a big negative **-0.0350**. Bit confused too ;) Pretty confident I didn't get my signs the wrong way around.", "1695230": "our team scores is also a large negative correlation -0.0350, in 20210613-20210913 it's about 0.03, the calculation function differs with the public score function?", "1696072": "Same here! During submission period, no single negative score, but both of my submissions score highly negative. Interestingly the better model (previously .0224 public) scored -.027x and the second chosen model (previously .021) scored -.025x. So the better model scored more negative.\n\nAll my ensemble models use online feature engineering, and using the latest data, the farthest looking to 6 days past. And this was meant to counter the momentum stuff and all that... Weird..\n\nLike many others, I have a local pipeline, so awaiting the host to publish the two weeks data in question. \n\nI wouldn't think they wouldn't, Why they wouldn't??\nLet's see.", "1719058": "I tried that If the submissions all equal 0 then the score will be -1", "1696503": "Totally agree, excellent thread and comments @miguelpm \n\nThanks!", "1694996": "@miguelpm i do share your concern despite my lb placement. It does seem possible that there could be a data provider issue similar to what happened with the first supplemental train update - there were a small number of massively incorrect crypto prices which badly affected the total correlation. However I'm doubtful/hopeful that this would be allowed to happen again.\n\nI tested my models on 40 out of sample two week periods and i scored worse than my current lb score on only 3 periods.\nSo it may also suggest it was just a difficult period for all models.", "1695441": "@miguelpm, maybe they give us more than one minute of data per iteration? or is it just the usual variation we should expect with this kind of data?\n\nI am right now understanding why grandmasters don't enter this sort of competition.", "1711977": "@cstein06 I'm curious as to whether the target calculation for the private LB data is consistent with that of supplemental_train.csv. Particularly, whether it has been calculated from a **running start** (i.e. with a warm-up of ~3770 timestamps from the contiguous end of supplemental_train.csv), or from a **standing start** (like at the beginning of train.csv). Please could you clarify? ", "1698669": "couldn't find any cause too", "1696844": "Did you use a reconstructed target ? Did you check for positive score on LB ? \nI worked on a reconstructed target from log returns, but as I was working on log(o/c) I had a minus sign error for a while. ", "1696039": "I think negative correlations could make perfect sense. At least in my case. I used very simple models that work with the \"reverting to the mean\" idea. It uses info from the last 30 minutes. In most months it works OK and scores around 0.03 or 0.04 but in a few other months (probably when the market was strongly momentum driven) it scores -0.01 or -0.02. So how well this \"model\" works depends on the type of market. Perhaps the last two weeks was more momentum driven than usual, so that would explain my -0.0162 and -.0169 scores. Too bad, but at least it makes sense.", "1695758": "" } }