UPDATE (second rerun) : the anomaly of very negative correlations seems to have been moderated,with \"naive submission\" reported to have become neutral (from -0.035 !). \nThis discards the \"promising candidate\" as the cause of the glitch, which seems now more of a single impact issue in the beginning of the dataset (old/new data join before target calculation?). Given LB evolution there's still high likelihood of a problem with the scoring process
\nI have posted before about the weirdness of the leaderboard scores after first rerun. What bugged me ;-) was I couldn't think of a simple explanation of what were seeing; poor near-random performance and very negative correlations
\nI initially though of some sorting issue but that was not a good candidate, as reversing order doesn't imply necesarily inverting correlations.The question I was making to myself was, what is the simplest mistake at rerun time that would explain this weirdnes?
\nAnd I think I got it. The big hint is the autocorrelation structure of target.
\nThere you have it. If time series API mistakenly feeds target ahead 15 instead of 16 then:
\nMakes sense? If this is finally the issue @sohier 's bug hunting has suddenly become much easier.
", "messages": [ { "id": "1696930", "postDate": "02/19/2022 08:17:17", "content": "UPDATE (second rerun) : the anomaly of very negative correlations seems to have been moderated,with \"naive submission\" reported to have become neutral (from -0.035 !). \nThis discards the \"promising candidate\" as the cause of the glitch, which seems now more of a single impact issue in the beginning of the dataset (old/new data join before target calculation?). Given LB evolution there's still high likelihood of a problem with the scoring process
\nI have posted before about the weirdness of the leaderboard scores after first rerun. What bugged me ;-) was I couldn't think of a simple explanation of what were seeing; poor near-random performance and very negative correlations
\nI initially though of some sorting issue but that was not a good candidate, as reversing order doesn't imply necesarily inverting correlations.The question I was making to myself was, what is the simplest mistake at rerun time that would explain this weirdnes?
\nAnd I think I got it. The big hint is the autocorrelation structure of target.
\nThere you have it. If time series API mistakenly feeds target ahead 15 instead of 16 then:
\nMakes sense? If this is finally the issue @sohier 's bug hunting has suddenly become much easier.
", "rawMarkdown": "**UPDATE (second rerun) :** *the anomaly of very negative correlations seems to have been moderated,with \"naive submission\" reported to have become neutral (from -0.035 !). \nThis discards the \"promising candidate\" as the cause of the glitch, which seems now more of a single impact issue in the beginning of the dataset (old/new data join before target calculation?). Given LB evolution there's still high likelihood of a problem with the scoring process*\n\n\nI have posted before about the weirdness of the leaderboard scores after first rerun. What bugged me ;-) was I couldn't think of a simple explanation of what were seeing; poor near-random performance and very negative correlations\n\nI initially though of some sorting issue but that was not a good candidate, as reversing order doesn't imply necesarily inverting correlations.The question I was making to myself was, what is the simplest mistake at rerun time that would explain this weirdnes?\n\nAnd I think I got it. The big hint is the autocorrelation structure of target. \n\n- Remember our target has an overlap by design that assures decreasing **positive correlation up to lag 15**? \n- Remember that **the moment you reach lag 16 that correlation consistently turns negative**, so negative that naive submissions using just that obtained score > 0.02?\n\nThere you have it. **If time series API mistakenly feeds target ahead 15 instead of 16** then:\n\n- everyone does worse than expected\n- the ones with models relying on lag 16 heavily, including \"robust\" naive prediction get consistent negative correlations\n- the ones with a time machine can do even better as this positive correlation is indeed a leak.\n\nMakes sense? If this is finally the issue @sohier 's bug hunting has suddenly become much easier.", "votes": null }, { "id": "1697055", "postDate": "02/19/2022 10:37:57", "content": "Nice thread. Just in case it is a helpful data point, my best 'non -1' negative correlation submission is pure naive running target calculation. It's currently on the LB with -0.0350. I did a decent amount of testing on it, so I'm sure that the calculation is consistent with the calculation methodology for train.csv and supplemental_train.csv. (And I'm also sure that I put a negative sign in front of it before prediction!)
\nMy other candidate for a bug (if there is a bug ;)) is that the test set used target calculation from a standing start (as used at beginning of train.csv) rather than from a running start (as used for the updated supplemental_train.csv). This would mean that my online target calculation would only become correct after ~3750 timestamps, with havoc ensuing in-between.
", "rawMarkdown": "Nice thread. Just in case it is a helpful data point, my best 'non -1' negative correlation submission is pure naive running target calculation. It's currently on the LB with -0.0350. I did a decent amount of testing on it, so I'm sure that the calculation is consistent with the calculation methodology for train.csv and supplemental_train.csv. (And I'm also sure that I put a negative sign in front of it before prediction!)\n\nMy other candidate for a bug (if there is a bug ;)) is that the test set used target calculation from a **standing start** (as used at beginning of train.csv) rather than from a **running start** (as used for the updated supplemental_train.csv). This would mean that my online target calculation would only become correct after ~3750 timestamps, with havoc ensuing in-between.", "votes": null }, { "id": "1697082", "postDate": "02/19/2022 11:06:11", "content": "So good to get that info about the behaviour of pure naive submission.
\nThat naive score is fully consistent with my hypothesis, pretty relevant info. Thanks for sharing it @jagofc !
", "rawMarkdown": "So good to get that info about the behaviour of pure naive submission.\n\nThat naive score is fully consistent with my hypothesis, pretty relevant info. Thanks for sharing it @jagofc !", "votes": null }, { "id": "1697395", "postDate": "02/19/2022 15:46:19", "content": "I would say there are two things that can be done easily to see what is going on:
\nRun the 'concerned submissions' with old API (yes I want my old slow API back!)
Share the first two weeks data, (why not?)
And for those two things or anything beyond, I am not sure if @sohier or @cstein06 is the right person to talk.
\n(if there is any intention to attend this issue at all)
@jagofc second update is here, would you mind sharing the score of that pure naive sub. of yours?
", "rawMarkdown": "jagofc second update is here, would you mind sharing the score of that pure naive sub. of yours?", "votes": null }, { "id": "1710160", "postDate": "03/02/2022 19:41:37", "content": "Hi, yes, it’s -0.0053. This time around the distribution of scores is very different. I don't want to comment too commitally, but my best guess is that target calculation was not done from a running start, thereby throwing everyone's predictions for the first 3750 timestamps, and most models are only now catching up. If my score increases to ~0.03 next time then I think this is compelling evidence.
", "rawMarkdown": "Hi, yes, it’s -0.0053. This time around the distribution of scores is very different. I don't want to comment too commitally, but my best guess is that target calculation was not done from a running start, thereby throwing everyone's predictions for the first 3750 timestamps, and most models are only now catching up. If my score increases to ~0.03 next time then I think this is compelling evidence.", "votes": null }, { "id": "1710204", "postDate": "03/02/2022 19:59:35", "content": "That's super useful info. Yes, LB looks different, and the possibility you comment is plausible.
\nAs we lack any kind of explanation or comment by Kaggle we can not know if something has been fixed/changed in the scoring process or it is, as you suggest, something with decreasing effect as dataset grows. (Or, of course, data was just that weird).
\nNot everyday you see multiple subs jumping 1500 positions in the LB, as has been my case and some others, I'm still inclined to think something is not being done right here.
\nBut if naive pred doesn't score as negative as before something has definitely changed and target shift is no longer a good candidate for a glitch. Much more possible something of the kind you suggest, that would still be a glitch btw, only one less impactful as time advances.
\nBut thanks a lot @jagofc , for me understanding what goes on and assessing the efectiveness of my approach is much more important than LB, and this is very useful to know.
", "rawMarkdown": "That's super useful info. Yes, LB looks different, and the possibility you comment is plausible.\n\nAs we lack any kind of explanation or comment by Kaggle we can not know if something has been fixed/changed in the scoring process or it is, as you suggest, something with decreasing effect as dataset grows. (Or, of course, data was just that weird).\n\nNot everyday you see multiple subs jumping 1500 positions in the LB, as has been my case and some others, I'm still inclined to think something is not being done right here.\n\nBut if naive pred doesn't score as negative as before something has definitely changed and target shift is no longer a good candidate for a glitch. Much more possible something of the kind you suggest, that would still be a glitch btw, only one less impactful as time advances.\n\nBut thanks a lot @jagofc , for me understanding what goes on and assessing the efectiveness of my approach is much more important than LB, and this is very useful to know.", "votes": null }, { "id": "1710219", "postDate": "03/02/2022 20:12:22", "content": "This is especially true considering the war in Ukraine and sanctions waged against Russia should have made many models perform worse during this period than last.
", "rawMarkdown": "This is especially true considering the war in Ukraine and sanctions waged against Russia should have made many models perform worse during this period than last.", "votes": null }, { "id": "1710227", "postDate": "03/02/2022 20:27:06", "content": "No problem @miguelpm. Yes, it would amount to something like a randomised starting position - but possibly with quite a large spread. Obviously can't be sure without transparency on the data, but if the evidence mounts that that's how the target has been calculated (standing start), then I think I will try and raise it, since the supplemental train was calculated from a running start. I'd be happy with some assurance that I'm competing in a fair game. :)
", "rawMarkdown": "No problem @miguelpm. Yes, it would amount to something like a randomised starting position - but possibly with quite a large spread. Obviously can't be sure without transparency on the data, but if the evidence mounts that that's how the target has been calculated (standing start), then I think I will try and raise it, since the supplemental train was calculated from a running start. I'd be happy with some assurance that I'm competing in a fair game. :)", "votes": null }, { "id": "1711981", "postDate": "03/04/2022 14:19:31", "content": "Decided to post a question about running vs standing start as a comment here, because otherwise it'll be deemed less significant as more updates accumulate. Let's see whether there's a response!
", "rawMarkdown": "Decided to post a question about running vs standing start as a comment [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308239#1711977), because otherwise it'll be deemed less significant as more updates accumulate. Let's see whether there's a response!", "votes": null }, { "id": "1721318", "postDate": "03/13/2022 15:21:25", "content": "@miguelpm Hey, I've seen your several posts, thanks for your sharing of analysis.
\nActually, I would like to ask is it possible that the more anomaly of second return is due to the general adjustment of score, in brief, to make scores on the leaderboard look better. So base on luck, some more compatible simple notebooks are more likely to profit from the adjustment.
\nBesides, if there were adjustments regarding the environment and the way dealing with raw data, it might influence the notebooks using pre-trained models. Take my personal case as an example, most of data for my pre-trained model is based on \"live data\" (data from 'supplementary_train.csv'). During the middle phase of the competition period, I sensed wiredness of data from 'supplementary_train.csv', after the organiser announcing the update of 'supplementary_train.csv', I chose keep using great amount of data from 'supplementary_train.csv' to train my models.
\nLast but not least, the submission finished running just several hours before deadline in return of -1 score, similar to the problem mentioned in https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/304630 remains mystery.
\nSo personally, I also remain some doubts towards the competition.
", "rawMarkdown": "miguelpm Hey, I've seen your several posts, thanks for your sharing of analysis.\n\nActually, I would like to ask is it possible that the more anomaly of second return is due to the general adjustment of score, in brief, to make scores on the leaderboard look better. So base on luck, some more compatible simple notebooks are more likely to profit from the adjustment.\n\nBesides, if there were adjustments regarding the environment and the way dealing with raw data, it might influence the notebooks using pre-trained models. Take my personal case as an example, most of data for my pre-trained model is based on \"live data\" (data from 'supplementary_train.csv'). During the middle phase of the competition period, I sensed wiredness of data from 'supplementary_train.csv', after the organiser announcing the update of 'supplementary_train.csv', I chose keep using great amount of data from 'supplementary_train.csv' to train my models. \n\nLast but not least, the submission finished running just several hours before deadline in return of -1 score, similar to the problem mentioned in https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/304630 remains mystery.\n\nSo personally, I also remain some doubts towards the competition.", "votes": null }, { "id": "1721346", "postDate": "03/13/2022 15:51:10", "content": "\nYou are suggesting interesting possibilities, truth is I don't know for sure what is really going on. There seems to be indeed a very distinguishable group of submissions affected by problems in the first rerun and others relatively unaffected.
\nWe will have some more bits of information as competition advances. The next hint I am waiting for is the evolution of scores going forward, specially of naive sumission. We have already seen huge improvements in second rerun. If instead of oscilating they steadily converge to expected score then it will be aditional evidence that something went really wrong with first chunk of data.
\nUnfortunately, without cooperation by organization we will not be able to prove/disprove anything.
\nTo me this is no longer about this particular comp. but more about the consideration Kaggle has to their competitors and how much you can trust them going forward. I want to think this is not all Kaggle staff, nor today's Kaggle way of doing things.
", "rawMarkdown": "Hi @zhaoyuanyuanzyy \n\nYou are suggesting interesting possibilities, truth is I don't know for sure what is really going on. There seems to be indeed a very distinguishable group of submissions affected by problems in the first rerun and others relatively unaffected.\n\nWe will have some more bits of information as competition advances. The next hint I am waiting for is the evolution of scores going forward, specially of naive sumission. We have already seen huge improvements in second rerun. If instead of oscilating they steadily converge to expected score then it will be aditional evidence that something went really wrong with first chunk of data.\n\n Unfortunately, without cooperation by organization we will not be able to prove/disprove anything. \n\nTo me this is no longer about this particular comp. but more about the consideration Kaggle has to their competitors and how much you can trust them going forward. I want to think this is not all Kaggle staff, nor today's Kaggle way of doing things.", "votes": null }, { "id": "1721640", "postDate": "03/13/2022 22:03:40", "content": "Hey @miguelpm
\nI suppose the cause behind some wired results in second return might be complicated. And I remain in doubt that a good submission can reach what kind of score, that's why I'm quite interested in your future sharing of the convergence of expected score.
\nSeeing there were certain amount of participants come from finance domain in this competition, for them truly job related. So they might have a different understanding of scores published on leaderboard and more sensible towards transparency of data during critical period.
\nBesides, with limited budgets, a lot of changes all of a sudden appeared in global environment might forced some adjustments of codes on the organiser’s side. So they might pay priority to certain group of notebooks.
\nI'm from IT domain, I couldn't tell more, just from the tutorial and api description provided by the organiser, I think their initial purpose remained fine. And crypto currency system isn't stock market nor free market, I suppose, people in that domain maybe more interested in the analytical perspectives with the help of codes.
", "rawMarkdown": "Hey @miguelpm \n\nI suppose the cause behind some wired results in second return might be complicated. And I remain in doubt that a good submission can reach what kind of score, that's why I'm quite interested in your future sharing of the convergence of expected score. \n\nSeeing there were certain amount of participants come from finance domain in this competition, for them truly job related. So they might have a different understanding of scores published on leaderboard and more sensible towards transparency of data during critical period.\n\nBesides, with limited budgets, a lot of changes all of a sudden appeared in global environment might forced some adjustments of codes on the organiser’s side. So they might pay priority to certain group of notebooks. \n\nI'm from IT domain, I couldn't tell more, just from the tutorial and api description provided by the organiser, I think their initial purpose remained fine. And crypto currency system isn't stock market nor free market, I suppose, people in that domain maybe more interested in the analytical perspectives with the help of codes.", "votes": null } ], "comments": [ { "id": 1697055, "author_name": "jagofc", "author_url": "", "post_date": "02/19/2022 10:37:57", "content": "Nice thread. Just in case it is a helpful data point, my best 'non -1' negative correlation submission is pure naive running target calculation. It's currently on the LB with -0.0350. I did a decent amount of testing on it, so I'm sure that the calculation is consistent with the calculation methodology for train.csv and supplemental_train.csv. (And I'm also sure that I put a negative sign in front of it before prediction!)
\nMy other candidate for a bug (if there is a bug ;)) is that the test set used target calculation from a standing start (as used at beginning of train.csv) rather than from a running start (as used for the updated supplemental_train.csv). This would mean that my online target calculation would only become correct after ~3750 timestamps, with havoc ensuing in-between.
", "votes": null, "replies": [ { "id": 1697082, "author_name": "miguelpm", "author_url": "", "post_date": "02/19/2022 11:06:11", "content": "So good to get that info about the behaviour of pure naive submission.
\nThat naive score is fully consistent with my hypothesis, pretty relevant info. Thanks for sharing it @jagofc !
", "votes": null, "replies": [] }, { "id": 1710108, "author_name": "miguelpm", "author_url": "", "post_date": "03/02/2022 18:51:54", "content": "@jagofc second update is here, would you mind sharing the score of that pure naive sub. of yours?
", "votes": null, "replies": [] }, { "id": 1710160, "author_name": "jagofc", "author_url": "", "post_date": "03/02/2022 19:41:37", "content": "Hi, yes, it’s -0.0053. This time around the distribution of scores is very different. I don't want to comment too commitally, but my best guess is that target calculation was not done from a running start, thereby throwing everyone's predictions for the first 3750 timestamps, and most models are only now catching up. If my score increases to ~0.03 next time then I think this is compelling evidence.
", "votes": null, "replies": [] }, { "id": 1710204, "author_name": "miguelpm", "author_url": "", "post_date": "03/02/2022 19:59:35", "content": "That's super useful info. Yes, LB looks different, and the possibility you comment is plausible.
\nAs we lack any kind of explanation or comment by Kaggle we can not know if something has been fixed/changed in the scoring process or it is, as you suggest, something with decreasing effect as dataset grows. (Or, of course, data was just that weird).
\nNot everyday you see multiple subs jumping 1500 positions in the LB, as has been my case and some others, I'm still inclined to think something is not being done right here.
\nBut if naive pred doesn't score as negative as before something has definitely changed and target shift is no longer a good candidate for a glitch. Much more possible something of the kind you suggest, that would still be a glitch btw, only one less impactful as time advances.
\nBut thanks a lot @jagofc , for me understanding what goes on and assessing the efectiveness of my approach is much more important than LB, and this is very useful to know.
", "votes": null, "replies": [] }, { "id": 1710219, "author_name": "petersk20", "author_url": "", "post_date": "03/02/2022 20:12:22", "content": "This is especially true considering the war in Ukraine and sanctions waged against Russia should have made many models perform worse during this period than last.
", "votes": null, "replies": [] }, { "id": 1710227, "author_name": "jagofc", "author_url": "", "post_date": "03/02/2022 20:27:06", "content": "No problem @miguelpm. Yes, it would amount to something like a randomised starting position - but possibly with quite a large spread. Obviously can't be sure without transparency on the data, but if the evidence mounts that that's how the target has been calculated (standing start), then I think I will try and raise it, since the supplemental train was calculated from a running start. I'd be happy with some assurance that I'm competing in a fair game. :)
", "votes": null, "replies": [] }, { "id": 1711981, "author_name": "jagofc", "author_url": "", "post_date": "03/04/2022 14:19:31", "content": "Decided to post a question about running vs standing start as a comment here, because otherwise it'll be deemed less significant as more updates accumulate. Let's see whether there's a response!
", "votes": null, "replies": [] } ] }, { "id": 1697395, "author_name": "abdulkadirguner", "author_url": "", "post_date": "02/19/2022 15:46:19", "content": "I would say there are two things that can be done easily to see what is going on:
\nRun the 'concerned submissions' with old API (yes I want my old slow API back!)
Share the first two weeks data, (why not?)
And for those two things or anything beyond, I am not sure if @sohier or @cstein06 is the right person to talk.
\n(if there is any intention to attend this issue at all)
@miguelpm Hey, I've seen your several posts, thanks for your sharing of analysis.
\nActually, I would like to ask is it possible that the more anomaly of second return is due to the general adjustment of score, in brief, to make scores on the leaderboard look better. So base on luck, some more compatible simple notebooks are more likely to profit from the adjustment.
\nBesides, if there were adjustments regarding the environment and the way dealing with raw data, it might influence the notebooks using pre-trained models. Take my personal case as an example, most of data for my pre-trained model is based on \"live data\" (data from 'supplementary_train.csv'). During the middle phase of the competition period, I sensed wiredness of data from 'supplementary_train.csv', after the organiser announcing the update of 'supplementary_train.csv', I chose keep using great amount of data from 'supplementary_train.csv' to train my models.
\nLast but not least, the submission finished running just several hours before deadline in return of -1 score, similar to the problem mentioned in https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/304630 remains mystery.
\nSo personally, I also remain some doubts towards the competition.
", "votes": null, "replies": [ { "id": 1721346, "author_name": "miguelpm", "author_url": "", "post_date": "03/13/2022 15:51:10", "content": "\nYou are suggesting interesting possibilities, truth is I don't know for sure what is really going on. There seems to be indeed a very distinguishable group of submissions affected by problems in the first rerun and others relatively unaffected.
\nWe will have some more bits of information as competition advances. The next hint I am waiting for is the evolution of scores going forward, specially of naive sumission. We have already seen huge improvements in second rerun. If instead of oscilating they steadily converge to expected score then it will be aditional evidence that something went really wrong with first chunk of data.
\nUnfortunately, without cooperation by organization we will not be able to prove/disprove anything.
\nTo me this is no longer about this particular comp. but more about the consideration Kaggle has to their competitors and how much you can trust them going forward. I want to think this is not all Kaggle staff, nor today's Kaggle way of doing things.
", "votes": null, "replies": [] }, { "id": 1721640, "author_name": "zhaoyuanyuanzyy", "author_url": "", "post_date": "03/13/2022 22:03:40", "content": "Hey @miguelpm
\nI suppose the cause behind some wired results in second return might be complicated. And I remain in doubt that a good submission can reach what kind of score, that's why I'm quite interested in your future sharing of the convergence of expected score.
\nSeeing there were certain amount of participants come from finance domain in this competition, for them truly job related. So they might have a different understanding of scores published on leaderboard and more sensible towards transparency of data during critical period.
\nBesides, with limited budgets, a lot of changes all of a sudden appeared in global environment might forced some adjustments of codes on the organiser’s side. So they might pay priority to certain group of notebooks.
\nI'm from IT domain, I couldn't tell more, just from the tutorial and api description provided by the organiser, I think their initial purpose remained fine. And crypto currency system isn't stock market nor free market, I suppose, people in that domain maybe more interested in the analytical perspectives with the help of codes.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1696930": "**UPDATE (second rerun) :** *the anomaly of very negative correlations seems to have been moderated,with \"naive submission\" reported to have become neutral (from -0.035 !). \nThis discards the \"promising candidate\" as the cause of the glitch, which seems now more of a single impact issue in the beginning of the dataset (old/new data join before target calculation?). Given LB evolution there's still high likelihood of a problem with the scoring process*\n\n\nI have posted before about the weirdness of the leaderboard scores after first rerun. What bugged me ;-) was I couldn't think of a simple explanation of what were seeing; poor near-random performance and very negative correlations\n\nI initially though of some sorting issue but that was not a good candidate, as reversing order doesn't imply necesarily inverting correlations.The question I was making to myself was, what is the simplest mistake at rerun time that would explain this weirdnes?\n\nAnd I think I got it. The big hint is the autocorrelation structure of target. \n\n- Remember our target has an overlap by design that assures decreasing **positive correlation up to lag 15**? \n- Remember that **the moment you reach lag 16 that correlation consistently turns negative**, so negative that naive submissions using just that obtained score > 0.02?\n\nThere you have it. **If time series API mistakenly feeds target ahead 15 instead of 16** then:\n\n- everyone does worse than expected\n- the ones with models relying on lag 16 heavily, including \"robust\" naive prediction get consistent negative correlations\n- the ones with a time machine can do even better as this positive correlation is indeed a leak.\n\nMakes sense? If this is finally the issue @sohier 's bug hunting has suddenly become much easier.", "1697055": "Nice thread. Just in case it is a helpful data point, my best 'non -1' negative correlation submission is pure naive running target calculation. It's currently on the LB with -0.0350. I did a decent amount of testing on it, so I'm sure that the calculation is consistent with the calculation methodology for train.csv and supplemental_train.csv. (And I'm also sure that I put a negative sign in front of it before prediction!)\n\nMy other candidate for a bug (if there is a bug ;)) is that the test set used target calculation from a **standing start** (as used at beginning of train.csv) rather than from a **running start** (as used for the updated supplemental_train.csv). This would mean that my online target calculation would only become correct after ~3750 timestamps, with havoc ensuing in-between.", "1697082": "So good to get that info about the behaviour of pure naive submission.\n\nThat naive score is fully consistent with my hypothesis, pretty relevant info. Thanks for sharing it @jagofc !", "1697395": "I would say there are two things that can be done easily to see what is going on:\n\n1. Run the 'concerned submissions' with old API (yes I want my old slow API back!)\n\n2. Share the first two weeks data, (why not?)\n\nAnd for those two things or anything beyond, I am not sure if @sohier or @cstein06 is the right person to talk.\n(if there is any intention to attend this issue at all)", "1710108": "jagofc second update is here, would you mind sharing the score of that pure naive sub. of yours?", "1710160": "Hi, yes, it’s -0.0053. This time around the distribution of scores is very different. I don't want to comment too commitally, but my best guess is that target calculation was not done from a running start, thereby throwing everyone's predictions for the first 3750 timestamps, and most models are only now catching up. If my score increases to ~0.03 next time then I think this is compelling evidence.", "1710204": "That's super useful info. Yes, LB looks different, and the possibility you comment is plausible.\n\nAs we lack any kind of explanation or comment by Kaggle we can not know if something has been fixed/changed in the scoring process or it is, as you suggest, something with decreasing effect as dataset grows. (Or, of course, data was just that weird).\n\nNot everyday you see multiple subs jumping 1500 positions in the LB, as has been my case and some others, I'm still inclined to think something is not being done right here.\n\nBut if naive pred doesn't score as negative as before something has definitely changed and target shift is no longer a good candidate for a glitch. Much more possible something of the kind you suggest, that would still be a glitch btw, only one less impactful as time advances.\n\nBut thanks a lot @jagofc , for me understanding what goes on and assessing the efectiveness of my approach is much more important than LB, and this is very useful to know.", "1710219": "This is especially true considering the war in Ukraine and sanctions waged against Russia should have made many models perform worse during this period than last.", "1710227": "No problem @miguelpm. Yes, it would amount to something like a randomised starting position - but possibly with quite a large spread. Obviously can't be sure without transparency on the data, but if the evidence mounts that that's how the target has been calculated (standing start), then I think I will try and raise it, since the supplemental train was calculated from a running start. I'd be happy with some assurance that I'm competing in a fair game. :)", "1711981": "Decided to post a question about running vs standing start as a comment [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308239#1711977), because otherwise it'll be deemed less significant as more updates accumulate. Let's see whether there's a response!", "1721318": "miguelpm Hey, I've seen your several posts, thanks for your sharing of analysis.\n\nActually, I would like to ask is it possible that the more anomaly of second return is due to the general adjustment of score, in brief, to make scores on the leaderboard look better. So base on luck, some more compatible simple notebooks are more likely to profit from the adjustment.\n\nBesides, if there were adjustments regarding the environment and the way dealing with raw data, it might influence the notebooks using pre-trained models. Take my personal case as an example, most of data for my pre-trained model is based on \"live data\" (data from 'supplementary_train.csv'). During the middle phase of the competition period, I sensed wiredness of data from 'supplementary_train.csv', after the organiser announcing the update of 'supplementary_train.csv', I chose keep using great amount of data from 'supplementary_train.csv' to train my models. \n\nLast but not least, the submission finished running just several hours before deadline in return of -1 score, similar to the problem mentioned in https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/304630 remains mystery.\n\nSo personally, I also remain some doubts towards the competition.", "1721346": "Hi @zhaoyuanyuanzyy \n\nYou are suggesting interesting possibilities, truth is I don't know for sure what is really going on. There seems to be indeed a very distinguishable group of submissions affected by problems in the first rerun and others relatively unaffected.\n\nWe will have some more bits of information as competition advances. The next hint I am waiting for is the evolution of scores going forward, specially of naive sumission. We have already seen huge improvements in second rerun. If instead of oscilating they steadily converge to expected score then it will be aditional evidence that something went really wrong with first chunk of data.\n\n Unfortunately, without cooperation by organization we will not be able to prove/disprove anything. \n\nTo me this is no longer about this particular comp. but more about the consideration Kaggle has to their competitors and how much you can trust them going forward. I want to think this is not all Kaggle staff, nor today's Kaggle way of doing things.", "1721640": "Hey @miguelpm \n\nI suppose the cause behind some wired results in second return might be complicated. And I remain in doubt that a good submission can reach what kind of score, that's why I'm quite interested in your future sharing of the convergence of expected score. \n\nSeeing there were certain amount of participants come from finance domain in this competition, for them truly job related. So they might have a different understanding of scores published on leaderboard and more sensible towards transparency of data during critical period.\n\nBesides, with limited budgets, a lot of changes all of a sudden appeared in global environment might forced some adjustments of codes on the organiser’s side. So they might pay priority to certain group of notebooks. \n\nI'm from IT domain, I couldn't tell more, just from the tutorial and api description provided by the organiser, I think their initial purpose remained fine. And crypto currency system isn't stock market nor free market, I suppose, people in that domain maybe more interested in the analytical perspectives with the help of codes." }, "source": "meta" }