The market is almost always unpredictable, even very good model can not guarantee stable performance, since the pattern shown in the history may change in the future. I tested my models using 2*3months and got about roughly 0.05 on each(no leakage), but I got -0.0025 on the first update(the submitted model used all data). I feel so sad as many of you, but I think it is normal and I really enjoyed the competition.
\nI think the useful things for us would be sharing our thoughts this task, and learning from each other. Let me share my idea first.
\nWe are asked to predict r15-beta*m15, where beta is the slope of linear fitting of passed return, so if you think linear relation exist, the expected target should be the fitting intercept(very close to zero). So we are actually asked to predict the unexpected extra return of each asset based on passed linear fitting.
\nBeta would be a feature that directly affect target. The beta is set to 'zero' if there is missing value in 3750 window, and we are actually asked to predict r15. So I have two types of models, one for zero beta(predic r15) and the other for non-zero beta(predict r15-betam15).
Other features I used are [asset return, portfolio return, asset return-portfolio return, asset return for moving average] in passed [1,2,3,5,10,15,20,30,40,50,60] minute.
I used 12*5 LGBM models ensemble for this competition, every time I added Gaussian noise to all features and target.
The market is almost always unpredictable, even very good model can not guarantee stable performance, since the pattern shown in the history may change in the future. I tested my models using 2*3months and got about roughly 0.05 on each(no leakage), but I got -0.0025 on the first update(the submitted model used all data). I feel so sad as many of you, but I think it is normal and I really enjoyed the competition.
\nI think the useful things for us would be sharing our thoughts this task, and learning from each other. Let me share my idea first.
\nWe are asked to predict r15-beta*m15, where beta is the slope of linear fitting of passed return, so if you think linear relation exist, the expected target should be the fitting intercept(very close to zero). So we are actually asked to predict the unexpected extra return of each asset based on passed linear fitting.
\nBeta would be a feature that directly affect target. The beta is set to 'zero' if there is missing value in 3750 window, and we are actually asked to predict r15. So I have two types of models, one for zero beta(predic r15) and the other for non-zero beta(predict r15-betam15).
Other features I used are [asset return, portfolio return, asset return-portfolio return, asset return for moving average] in passed [1,2,3,5,10,15,20,30,40,50,60] minute.
I used 12*5 LGBM models ensemble for this competition, every time I added Gaussian noise to all features and target.
Some of us spent a lot of work on this so it is a bit upsetting to see in my case suddenly every single submission has changed to an error despite them being a mix between simple pred for testing to the more complex solutions, I think we have a right to be a little less calm…
\nThe reason you got a negative score is because you were overfitting to the public LB. You do not get a 0.05 just by using your simplistic features.
\nYou also added noise to your features and target what did you think would happen in a high noise low signal challenge?
", "rawMarkdown": "Some of us spent a lot of work on this so it is a bit upsetting to see in my case suddenly every single submission has changed to an error despite them being a mix between simple pred for testing to the more complex solutions, I think we have a right to be a little less calm…\n\nThe reason you got a negative score is because you were overfitting to the public LB. You do not get a 0.05 just by using your simplistic features. \n\nYou also added noise to your features and target what did you think would happen in a high noise low signal challenge?", "votes": -1, "replies": [ { "id": 1695568, "postDate": "2022-02-18T08:19:55.567Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 1695640, "postDate": "2022-02-18T09:28:02.213Z", "content": "There is no place because as of yesterday, 100% of my submissions have turned into an \"error\", hence why a lot of us are upset because something seems to have gone wrong. My score before was 0.0418. There are other strange things happening to others submissions such as their expected scores coming through as negatives and others had tests to prove something was not right too.. OP has completely missed the point with the non \"calm\"-ness
", "rawMarkdown": "There is no place because as of yesterday, 100% of my submissions have turned into an \"error\", hence why a lot of us are upset because something seems to have gone wrong. My score before was 0.0418. There are other strange things happening to others submissions such as their expected scores coming through as negatives and others had tests to prove something was not right too.. OP has completely missed the point with the non \"calm\"-ness ", "votes": -2 }, { "id": 1696216, "postDate": "2022-02-18T16:58:59.867Z", "content": "@julianmukaj I agree that I definitely have some level of overfitting, but I dont think wit would such huge. In first test, I used LB data as test, use data before that as training, the CV is about 0.066 and test is 0.05; in second test, I used SUP data as test and the data before that as training, the CV is still about 0.06 and test around 0.05.
\nFor the Gaussian noise, it is used to make more smooth prediction and deal with over fitting. Since tree based models can give discrete prediction. It works well when I did test on the 2*3 months test mentioned above.
\nIf you still dont agree with the noise I added, think about this situation. Imagine the function y=sin(x)+N, where N is some large noise signal. We know the truth is y=sin(x), but the over fitting LGBM would give wiggles at every place. Basically speaking, the predicted value at some bin [x1,x2] would be mean(yi|xi in [x1,x2]) = mean(sin(xi)| xi in [x1,x2])+mean(N(xi)| xi in [x1,x2]). If bin is large, mean(N(xi)| xi in [x1,x2]) would be close to zero but model will under fit; if bin is small, mean(N(xi)| xi in [x1,x2]) will have large variation and cause wiggles, and it is overfitting. Add nosie to {xi} will make the data in one bin goes into nearby bins, and each model will have different mean(N(xi)| xi in [x1,x2]), the expected ensemble contribution of noise will decay as number of model increase.
\nI think there would be better way to add noise to deal with overfitting, but I think the idea itself is right, especially for noisy data. What do you think about this?
\nFor the features part, what do you think would be better features? I am just curious about this.
", "rawMarkdown": "@julianmukaj I agree that I definitely have some level of overfitting, but I dont think wit would such huge. In first test, I used LB data as test, use data before that as training, the CV is about 0.066 and test is 0.05; in second test, I used SUP data as test and the data before that as training, the CV is still about 0.06 and test around 0.05.\n\nFor the Gaussian noise, it is used to make more smooth prediction and deal with over fitting. Since tree based models can give discrete prediction. It works well when I did test on the 2*3 months test mentioned above.\n\nIf you still dont agree with the noise I added, think about this situation. Imagine the function y=sin(x)+N, where N is some large noise signal. We know the truth is y=sin(x), but the over fitting LGBM would give wiggles at every place. Basically speaking, the predicted value at some bin [x1,x2] would be mean(yi|xi in [x1,x2]) = mean(sin(xi)| xi in [x1,x2])+mean(N(xi)| xi in [x1,x2]). If bin is large, mean(N(xi)| xi in [x1,x2]) would be close to zero but model will under fit; if bin is small, mean(N(xi)| xi in [x1,x2]) will have large variation and cause wiggles, and it is overfitting. Add nosie to {xi} will make the data in one bin goes into nearby bins, and each model will have different mean(N(xi)| xi in [x1,x2]), the expected ensemble contribution of noise will decay as number of model increase.\n\n I think there would be better way to add noise to deal with overfitting, but I think the idea itself is right, especially for noisy data. What do you think about this?\n\nFor the features part, what do you think would be better features? I am just curious about this.\n\n", "votes": -2 }, { "id": 1696230, "postDate": "2022-02-18T17:14:28.040Z", "content": "I understand the principle around adding noise to \"smooth\" the prediction but I think you will achieve better results just by L1, L2 regularization your tree model, especially in a contest like this because I just think its counter intuitive to add noise to something that is already very hard to separate noise from, especially if adding it to features where you are trying to extract as much signal as possible. Did you get 0.05 on the public LB or CV?
\nSome things I put down regarding features here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308100 most of my feature engineering is driven by my domain knowledge of trading and crypto
", "rawMarkdown": "I understand the principle around adding noise to \"smooth\" the prediction but I think you will achieve better results just by L1, L2 regularization your tree model, especially in a contest like this because I just think its counter intuitive to add noise to something that is already very hard to separate noise from, especially if adding it to features where you are trying to extract as much signal as possible. Did you get 0.05 on the public LB or CV?\n\nSome things I put down regarding features here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308100 most of my feature engineering is driven by my domain knowledge of trading and crypto", "votes": -2 }, { "id": 1696262, "postDate": "2022-02-18T17:45:55.493Z", "content": "I added gaussian noise at each step in my neural network. It seemed to help prevent overfitting. For tree based methods though I think it would not improve the score since you can't change the noise at each iteration.
", "rawMarkdown": "I added gaussian noise at each step in my neural network. It seemed to help prevent overfitting. For tree based methods though I think it would not improve the score since you can't change the noise at each iteration. ", "votes": -1 }, { "id": 1696376, "postDate": "2022-02-18T19:41:43.027Z", "content": "@petersk20 you are right, that why I have totally 12*5 models, each models have data input with different Gaussian noise(both add N(0,sigma) and multiply N(1,0.03)). The score keeps increasing for first 40 models, and seems saturate after that.
\nOne thing I did not pay attention is that the noise I add is independent, however there should be some correlation between the features, so ideally you should adjust your input noise based on that. I was busy in some internship interview and do not have time do the fine tuning.
@julianmukaj I did used lots of regularization including [L1, L2, max_depth, min_data_leaf, feature_fraction, bagging_fraction, num_leaves]. The 0.05 is LB and on updated SUP data, CV is above 0.06, training is 0.1-0.13.
\nFor the adding noise part, I can not guarantee it is good for this task, but I know it works in some task with noisy input, just like the example I mentioned, and I think mathematical fomula make sense.
", "rawMarkdown": "@julianmukaj I did used lots of regularization including [L1, L2, max_depth, min_data_leaf, feature_fraction, bagging_fraction, num_leaves]. The 0.05 is LB and on updated SUP data, CV is above 0.06, training is 0.1-0.13.\n\nFor the adding noise part, I can not guarantee it is good for this task, but I know it works in some task with noisy input, just like the example I mentioned, and I think mathematical fomula make sense.", "votes": -2 }, { "id": 1696401, "postDate": "2022-02-18T20:02:14.837Z", "content": "I think it helps to augment the training set which will be good wether there is noise already in the data or not. Did you include a model in your ensemble with no noise? Because that might have also helped.
", "rawMarkdown": "I think it helps to augment the training set which will be good wether there is noise already in the data or not. Did you include a model in your ensemble with no noise? Because that might have also helped. ", "votes": -1 }, { "id": 1696443, "postDate": "2022-02-18T20:49:06.357Z", "content": "Ahh, I think you are right that at least I need to add the model without extra noise. All my models were trained on data with extra noise. But actually the noise intensity I add is small, all the models perform similar as the model without nosie.
", "rawMarkdown": "Ahh, I think you are right that at least I need to add the model without extra noise. All my models were trained on data with extra noise. But actually the noise intensity I add is small, all the models perform similar as the model without nosie.", "votes": -2 } ] } ], "comments": [ { "id": 1695539, "author_name": "JM", "author_url": "", "post_date": "2022-02-18T07:50:00.240000", "content": "Some of us spent a lot of work on this so it is a bit upsetting to see in my case suddenly every single submission has changed to an error despite them being a mix between simple pred for testing to the more complex solutions, I think we have a right to be a little less calm…
\nThe reason you got a negative score is because you were overfitting to the public LB. You do not get a 0.05 just by using your simplistic features.
\nYou also added noise to your features and target what did you think would happen in a high noise low signal challenge?
", "votes": -1, "replies": [ { "id": 1695568, "author_name": "", "author_url": "", "post_date": "2022-02-18T08:19:55.567000", "content": "", "votes": -2, "replies": [] }, { "id": 1695640, "author_name": "JM", "author_url": "", "post_date": "2022-02-18T09:28:02.213000", "content": "There is no place because as of yesterday, 100% of my submissions have turned into an \"error\", hence why a lot of us are upset because something seems to have gone wrong. My score before was 0.0418. There are other strange things happening to others submissions such as their expected scores coming through as negatives and others had tests to prove something was not right too.. OP has completely missed the point with the non \"calm\"-ness
", "votes": -2, "replies": [] }, { "id": 1696216, "author_name": "liyang.chen", "author_url": "", "post_date": "2022-02-18T16:58:59.867000", "content": "@julianmukaj I agree that I definitely have some level of overfitting, but I dont think wit would such huge. In first test, I used LB data as test, use data before that as training, the CV is about 0.066 and test is 0.05; in second test, I used SUP data as test and the data before that as training, the CV is still about 0.06 and test around 0.05.
\nFor the Gaussian noise, it is used to make more smooth prediction and deal with over fitting. Since tree based models can give discrete prediction. It works well when I did test on the 2*3 months test mentioned above.
\nIf you still dont agree with the noise I added, think about this situation. Imagine the function y=sin(x)+N, where N is some large noise signal. We know the truth is y=sin(x), but the over fitting LGBM would give wiggles at every place. Basically speaking, the predicted value at some bin [x1,x2] would be mean(yi|xi in [x1,x2]) = mean(sin(xi)| xi in [x1,x2])+mean(N(xi)| xi in [x1,x2]). If bin is large, mean(N(xi)| xi in [x1,x2]) would be close to zero but model will under fit; if bin is small, mean(N(xi)| xi in [x1,x2]) will have large variation and cause wiggles, and it is overfitting. Add nosie to {xi} will make the data in one bin goes into nearby bins, and each model will have different mean(N(xi)| xi in [x1,x2]), the expected ensemble contribution of noise will decay as number of model increase.
\nI think there would be better way to add noise to deal with overfitting, but I think the idea itself is right, especially for noisy data. What do you think about this?
\nFor the features part, what do you think would be better features? I am just curious about this.
", "votes": -2, "replies": [] }, { "id": 1696230, "author_name": "JM", "author_url": "", "post_date": "2022-02-18T17:14:28.040000", "content": "I understand the principle around adding noise to \"smooth\" the prediction but I think you will achieve better results just by L1, L2 regularization your tree model, especially in a contest like this because I just think its counter intuitive to add noise to something that is already very hard to separate noise from, especially if adding it to features where you are trying to extract as much signal as possible. Did you get 0.05 on the public LB or CV?
\nSome things I put down regarding features here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/308100 most of my feature engineering is driven by my domain knowledge of trading and crypto
", "votes": -2, "replies": [] }, { "id": 1696262, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-18T17:45:55.493000", "content": "I added gaussian noise at each step in my neural network. It seemed to help prevent overfitting. For tree based methods though I think it would not improve the score since you can't change the noise at each iteration.
", "votes": -1, "replies": [] }, { "id": 1696376, "author_name": "liyang.chen", "author_url": "", "post_date": "2022-02-18T19:41:43.027000", "content": "@petersk20 you are right, that why I have totally 12*5 models, each models have data input with different Gaussian noise(both add N(0,sigma) and multiply N(1,0.03)). The score keeps increasing for first 40 models, and seems saturate after that.
\nOne thing I did not pay attention is that the noise I add is independent, however there should be some correlation between the features, so ideally you should adjust your input noise based on that. I was busy in some internship interview and do not have time do the fine tuning.
@julianmukaj I did used lots of regularization including [L1, L2, max_depth, min_data_leaf, feature_fraction, bagging_fraction, num_leaves]. The 0.05 is LB and on updated SUP data, CV is above 0.06, training is 0.1-0.13.
\nFor the adding noise part, I can not guarantee it is good for this task, but I know it works in some task with noisy input, just like the example I mentioned, and I think mathematical fomula make sense.
", "votes": -2, "replies": [] }, { "id": 1696401, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-18T20:02:14.837000", "content": "I think it helps to augment the training set which will be good wether there is noise already in the data or not. Did you include a model in your ensemble with no noise? Because that might have also helped.
", "votes": -1, "replies": [] }, { "id": 1696443, "author_name": "liyang.chen", "author_url": "", "post_date": "2022-02-18T20:49:06.357000", "content": "Ahh, I think you are right that at least I need to add the model without extra noise. All my models were trained on data with extra noise. But actually the noise intensity I add is small, all the models perform similar as the model without nosie.
", "votes": -2, "replies": [] } ] } ], "raw_markdown_by_id": { "1695455": "The market is almost always unpredictable, even very good model can not guarantee stable performance, since the pattern shown in the history may change in the future. I tested my models using 2*3months and got about roughly 0.05 on each(no leakage), but I got -0.0025 on the first update(the submitted model used all data). I feel so sad as many of you, but I think it is normal and I really enjoyed the competition.\n\nI think the useful things for us would be sharing our thoughts this task, and learning from each other. Let me share my idea first. \n\n1. We are asked to predict r15-beta*m15, where beta is the slope of linear fitting of passed return, so if you think linear relation exist, the expected target should be the fitting intercept(very close to zero). So we are actually asked to predict the unexpected extra return of each asset based on passed linear fitting. \nBeta would be a feature that directly affect target. The beta is set to 'zero' if there is missing value in 3750 window, and we are actually asked to predict r15. So I have two types of models, one for zero beta(predic r15) and the other for non-zero beta(predict r15-betam15).\n\n2. Other features I used are [asset return, portfolio return, asset return-portfolio return, asset return for moving average] in passed [1,2,3,5,10,15,20,30,40,50,60] minute.\n\n3. I used 12*5 LGBM models ensemble for this competition, every time I added Gaussian noise to all features and target.\n\n", "1695539": "Some of us spent a lot of work on this so it is a bit upsetting to see in my case suddenly every single submission has changed to an error despite them being a mix between simple pred for testing to the more complex solutions, I think we have a right to be a little less calm…\n\nThe reason you got a negative score is because you were overfitting to the public LB. You do not get a 0.05 just by using your simplistic features. \n\nYou also added noise to your features and target what did you think would happen in a high noise low signal challenge?" }, "complete": true, "source": "existing_discussion", "competition_id": 30894, "competition_slug": "g-research-crypto-forecasting", "team_id": 7900757, "solution_url": "/competitions/g-research-crypto-forecasting/writeups/liyang-chen-calm-down-even-if-you-got-negative-sco" }