After the first rerun there has been quite a bit of discussion going on about high negative scores on the LB. In that regard I wondered how much a few predictions can mess up (or improve) a models score. I conducted a small experiment to see what correlation can be achieved by setting a random block of 500 values within the prediction to 1, and leaving the rest of the values to 0. I back tested this strategy on the first 2 weeks of the supplemental_train.csv.
\nHere is a small notebook with the results:
\nhttps://www.kaggle.com/fritzcremer/consecutive-ones-prediction/notebook
The number of targets is 14 * 60 * 24 * 14 = 282240 (minus a few nan values which I ignored) and for simplicity I disregarded the fact that different coins have different weights, but I think that shouldn't matter too much.
\nI tested 5000-6000 different block positions and it seems that just by setting random segments containing 500 values to 1, you can reach a correlation of 0.053770. This may seem surprising as this score would be rank 1 on the current LB, but it just shows are much the pearson correlation is affected by a few single predictions.
Now, I hope the kaggle team and the competition host can clear up any possible mishaps that lead to the large negative scores and few notebook failings.
\nBut I think there are also plausible theories how such high negative score could be achieved, even with models that might perform well on similar test sets. If we look at crypto markets, we can observe very high negative or positive returns over a short period of time. Sometimes the market bounces back from these events, and sometimes it does not. Either way, during these moments the market often experiences extreme volatility. Therefore any predictions get either rewarded a lot, or punished a lot. In fact, so much so, that a few single predictions may overshadow the rest of the models performance. I repeated the experiment above while setting only 15 values to 1. In that case, the largest observed correlation was 0.031047, which is still impressive considering the amount targets. Now why would a model make such bad predictions during these volatile moments? Well, the model gets rewarded a lot for getting these few predictions right during training phase, therefore it also very easy to overfit on these scenarios. If a model makes a strong decision during these kind of scenarios, it is similar to making an all or nothing bet.
\nAfter the first experiment I conducted another, even simpler experiment, where I simply set the prediction for the maximum absolute target to 1 (only one single value set!). This gives us an even higher correlation of 0.071587. This is a short code example for that:
\ncount = 14*60*24*14 # 2 Weeks\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/supplemental_train.csv\", nrows=count)\n\ntargets = train['Target'].to_numpy()\ntargets = targets[~np.isnan(targets)] # Ignore the few NaN values\n\npreds = np.zeros(len(targets))\npreds[np.argmax(np.abs(targets))] = 1 # Get the highest targets index and set the prediction to 1\n\ncorr = np.corrcoef(targets, preds)[0, 1] # Calculate the correlation\n\nprint(\"Correlation: {:.6f}\".format(corr))\nRemember that this holds only for 2 weeks. Over 3 month, these single predictions have much less impact (though still quite a lot!). That is also the reason we might not know how the final top 10 will look like, until the very last rerun.
\nNo matter what the reason for the high negative LB scores may be, I think everyone should be aware of how much impact a few predictions can have on the score, for the better or for the worse.
", "messages": [ { "id": 1700428, "postDate": "2022-02-22T00:49:37.040Z", "content": "After the first rerun there has been quite a bit of discussion going on about high negative scores on the LB. In that regard I wondered how much a few predictions can mess up (or improve) a models score. I conducted a small experiment to see what correlation can be achieved by setting a random block of 500 values within the prediction to 1, and leaving the rest of the values to 0. I back tested this strategy on the first 2 weeks of the supplemental_train.csv.
\nHere is a small notebook with the results:
\nhttps://www.kaggle.com/fritzcremer/consecutive-ones-prediction/notebook
The number of targets is 14 * 60 * 24 * 14 = 282240 (minus a few nan values which I ignored) and for simplicity I disregarded the fact that different coins have different weights, but I think that shouldn't matter too much.
\nI tested 5000-6000 different block positions and it seems that just by setting random segments containing 500 values to 1, you can reach a correlation of 0.053770. This may seem surprising as this score would be rank 1 on the current LB, but it just shows are much the pearson correlation is affected by a few single predictions.
Now, I hope the kaggle team and the competition host can clear up any possible mishaps that lead to the large negative scores and few notebook failings.
\nBut I think there are also plausible theories how such high negative score could be achieved, even with models that might perform well on similar test sets. If we look at crypto markets, we can observe very high negative or positive returns over a short period of time. Sometimes the market bounces back from these events, and sometimes it does not. Either way, during these moments the market often experiences extreme volatility. Therefore any predictions get either rewarded a lot, or punished a lot. In fact, so much so, that a few single predictions may overshadow the rest of the models performance. I repeated the experiment above while setting only 15 values to 1. In that case, the largest observed correlation was 0.031047, which is still impressive considering the amount targets. Now why would a model make such bad predictions during these volatile moments? Well, the model gets rewarded a lot for getting these few predictions right during training phase, therefore it also very easy to overfit on these scenarios. If a model makes a strong decision during these kind of scenarios, it is similar to making an all or nothing bet.
\nAfter the first experiment I conducted another, even simpler experiment, where I simply set the prediction for the maximum absolute target to 1 (only one single value set!). This gives us an even higher correlation of 0.071587. This is a short code example for that:
\ncount = 14*60*24*14 # 2 Weeks\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/supplemental_train.csv\", nrows=count)\n\ntargets = train['Target'].to_numpy()\ntargets = targets[~np.isnan(targets)] # Ignore the few NaN values\n\npreds = np.zeros(len(targets))\npreds[np.argmax(np.abs(targets))] = 1 # Get the highest targets index and set the prediction to 1\n\ncorr = np.corrcoef(targets, preds)[0, 1] # Calculate the correlation\n\nprint(\"Correlation: {:.6f}\".format(corr))\nRemember that this holds only for 2 weeks. Over 3 month, these single predictions have much less impact (though still quite a lot!). That is also the reason we might not know how the final top 10 will look like, until the very last rerun.
\nNo matter what the reason for the high negative LB scores may be, I think everyone should be aware of how much impact a few predictions can have on the score, for the better or for the worse.
", "rawMarkdown": "After the first rerun there has been quite a bit of discussion going on about high negative scores on the LB. In that regard I wondered how much a few predictions can mess up (or improve) a models score. I conducted a small experiment to see what correlation can be achieved by setting a random block of 500 values within the prediction to 1, and leaving the rest of the values to 0. I back tested this strategy on the first 2 weeks of the supplemental_train.csv.\nHere is a small notebook with the results: \nhttps://www.kaggle.com/fritzcremer/consecutive-ones-prediction/notebook\n\nThe number of targets is 14 * 60 * 24 * 14 = 282240 (minus a few nan values which I ignored) and for simplicity I disregarded the fact that different coins have different weights, but I think that shouldn't matter too much.\nI tested 5000-6000 different block positions and it seems that just by setting random segments containing 500 values to 1, you can reach a correlation of 0.053770. This may seem surprising as this score would be rank 1 on the current LB, but it just shows are much the pearson correlation is affected by a few single predictions.\n\nNow, I hope the kaggle team and the competition host can clear up any possible mishaps that lead to the large negative scores and few notebook failings.\n\nBut I think there are also plausible theories how such high negative score could be achieved, even with models that might perform well on similar test sets. If we look at crypto markets, we can observe very high negative or positive returns over a short period of time. Sometimes the market bounces back from these events, and sometimes it does not. Either way, during these moments the market often experiences extreme volatility. Therefore any predictions get either rewarded a lot, or punished a lot. In fact, so much so, that a few single predictions may overshadow the rest of the models performance. I repeated the experiment above while setting only 15 values to 1. In that case, the largest observed correlation was 0.031047, which is still impressive considering the amount targets. Now why would a model make such bad predictions during these volatile moments? Well, the model gets rewarded a lot for getting these few predictions right during training phase, therefore it also very easy to overfit on these scenarios. If a model makes a strong decision during these kind of scenarios, it is similar to making an all or nothing bet.\n\nAfter the first experiment I conducted another, even simpler experiment, where I simply set the prediction for the maximum absolute target to 1 (only one single value set!). This gives us an even higher correlation of 0.071587. This is a short code example for that:\n\n```\ncount = 14*60*24*14 # 2 Weeks\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/supplemental_train.csv\", nrows=count)\n\ntargets = train['Target'].to_numpy()\ntargets = targets[~np.isnan(targets)] # Ignore the few NaN values\n\npreds = np.zeros(len(targets))\npreds[np.argmax(np.abs(targets))] = 1 # Get the highest targets index and set the prediction to 1\n\ncorr = np.corrcoef(targets, preds)[0, 1] # Calculate the correlation\n \nprint(\"Correlation: {:.6f}\".format(corr))\n```\nRemember that this holds only for 2 weeks. Over 3 month, these single predictions have much less impact (though still quite a lot!). That is also the reason we might not know how the final top 10 will look like, until the very last rerun.\n\nNo matter what the reason for the high negative LB scores may be, I think everyone should be aware of how much impact a few predictions can have on the score, for the better or for the worse.", "votes": 8 } ], "comments": [], "raw_markdown_by_id": { "1700428": "After the first rerun there has been quite a bit of discussion going on about high negative scores on the LB. In that regard I wondered how much a few predictions can mess up (or improve) a models score. I conducted a small experiment to see what correlation can be achieved by setting a random block of 500 values within the prediction to 1, and leaving the rest of the values to 0. I back tested this strategy on the first 2 weeks of the supplemental_train.csv.\nHere is a small notebook with the results: \nhttps://www.kaggle.com/fritzcremer/consecutive-ones-prediction/notebook\n\nThe number of targets is 14 * 60 * 24 * 14 = 282240 (minus a few nan values which I ignored) and for simplicity I disregarded the fact that different coins have different weights, but I think that shouldn't matter too much.\nI tested 5000-6000 different block positions and it seems that just by setting random segments containing 500 values to 1, you can reach a correlation of 0.053770. This may seem surprising as this score would be rank 1 on the current LB, but it just shows are much the pearson correlation is affected by a few single predictions.\n\nNow, I hope the kaggle team and the competition host can clear up any possible mishaps that lead to the large negative scores and few notebook failings.\n\nBut I think there are also plausible theories how such high negative score could be achieved, even with models that might perform well on similar test sets. If we look at crypto markets, we can observe very high negative or positive returns over a short period of time. Sometimes the market bounces back from these events, and sometimes it does not. Either way, during these moments the market often experiences extreme volatility. Therefore any predictions get either rewarded a lot, or punished a lot. In fact, so much so, that a few single predictions may overshadow the rest of the models performance. I repeated the experiment above while setting only 15 values to 1. In that case, the largest observed correlation was 0.031047, which is still impressive considering the amount targets. Now why would a model make such bad predictions during these volatile moments? Well, the model gets rewarded a lot for getting these few predictions right during training phase, therefore it also very easy to overfit on these scenarios. If a model makes a strong decision during these kind of scenarios, it is similar to making an all or nothing bet.\n\nAfter the first experiment I conducted another, even simpler experiment, where I simply set the prediction for the maximum absolute target to 1 (only one single value set!). This gives us an even higher correlation of 0.071587. This is a short code example for that:\n\n```\ncount = 14*60*24*14 # 2 Weeks\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/supplemental_train.csv\", nrows=count)\n\ntargets = train['Target'].to_numpy()\ntargets = targets[~np.isnan(targets)] # Ignore the few NaN values\n\npreds = np.zeros(len(targets))\npreds[np.argmax(np.abs(targets))] = 1 # Get the highest targets index and set the prediction to 1\n\ncorr = np.corrcoef(targets, preds)[0, 1] # Calculate the correlation\n \nprint(\"Correlation: {:.6f}\".format(corr))\n```\nRemember that this holds only for 2 weeks. Over 3 month, these single predictions have much less impact (though still quite a lot!). That is also the reason we might not know how the final top 10 will look like, until the very last rerun.\n\nNo matter what the reason for the high negative LB scores may be, I think everyone should be aware of how much impact a few predictions can have on the score, for the better or for the worse." } }