I've seen a lot of people in this competition use gradient boosted decision trees, but I think they were the wrong model to use for this problem. Let me know your thought.
\nFor many Kaggle competition, gradient boosted decision trees like lightgbm and xgboost perform very well and usually end up at the top of the leaderboard. However, I think this competition is a bit different.
\nIn this competition, we are predicting the residual returns of cryptocurrencies and not the actual return over a 15 minute period. The residual return is equal to R(t)−βM(t) where R(T) is the actual return of the coin, β is equal to beta and M is equal to the weighted return of the market.
\nMost people created features for individual coins that could be used in a GBDT to predict the return of that associated coin. However, the weighted return of the market is also used in the calculation of the target, meaning that features for the other 13 coins have to be used in order to encapsulate all of the information needed to predict the target.
\nI created a new notebook to illustrate that: kaggle.com/petersk20/tree-based-methods/notebook. I trained two lightgbm models to predict the target for Bitcoin. Model one included the beta value and the return of Bitcoin used in the target calculation. This one had a correlation coefficient of 0.382. The second model utilized the beta value, the return of bitcoin, and the return of the 13 other coins. This one had a correlation coefficient of 0.941. This shows that if you don't include information about other coins in your model your maximum score drops significantly.
\nTo solve this problem, one could include all of the features created for each coin in their GBDT problem. However, this would multiply the number of features by a factor of 14 which would most likely cause the GBDT to overfit and not find good patterns.
\nThis problem is perfect for neural networks because it is easy in Pytorch or Tensorflow to optimize multiple things at the same time. Secondary losses can be utilized to create rich feature representation for each coin and these representations can be combined in a final output MLP. For example, a neural network could be created with multiple input branches for each coin. At the end of each branch, a loss function for the individual return of the coin in the next 15 minutes could be predicted. Feature representations from previous layers of each branch can then be combined to predict the actual target for the competition.
", "messages": [ { "id": "1673677", "postDate": "02/02/2022 21:58:44", "content": "I've seen a lot of people in this competition use gradient boosted decision trees, but I think they were the wrong model to use for this problem. Let me know your thought.
\nFor many Kaggle competition, gradient boosted decision trees like lightgbm and xgboost perform very well and usually end up at the top of the leaderboard. However, I think this competition is a bit different.
\nIn this competition, we are predicting the residual returns of cryptocurrencies and not the actual return over a 15 minute period. The residual return is equal to R(t)−βM(t) where R(T) is the actual return of the coin, β is equal to beta and M is equal to the weighted return of the market.
\nMost people created features for individual coins that could be used in a GBDT to predict the return of that associated coin. However, the weighted return of the market is also used in the calculation of the target, meaning that features for the other 13 coins have to be used in order to encapsulate all of the information needed to predict the target.
\nI created a new notebook to illustrate that: kaggle.com/petersk20/tree-based-methods/notebook. I trained two lightgbm models to predict the target for Bitcoin. Model one included the beta value and the return of Bitcoin used in the target calculation. This one had a correlation coefficient of 0.382. The second model utilized the beta value, the return of bitcoin, and the return of the 13 other coins. This one had a correlation coefficient of 0.941. This shows that if you don't include information about other coins in your model your maximum score drops significantly.
\nTo solve this problem, one could include all of the features created for each coin in their GBDT problem. However, this would multiply the number of features by a factor of 14 which would most likely cause the GBDT to overfit and not find good patterns.
\nThis problem is perfect for neural networks because it is easy in Pytorch or Tensorflow to optimize multiple things at the same time. Secondary losses can be utilized to create rich feature representation for each coin and these representations can be combined in a final output MLP. For example, a neural network could be created with multiple input branches for each coin. At the end of each branch, a loss function for the individual return of the coin in the next 15 minutes could be predicted. Feature representations from previous layers of each branch can then be combined to predict the actual target for the competition.
", "rawMarkdown": "I've seen a lot of people in this competition use gradient boosted decision trees, but I think they were the wrong model to use for this problem. Let me know your thought.\n\nFor many Kaggle competition, gradient boosted decision trees like lightgbm and xgboost perform very well and usually end up at the top of the leaderboard. However, I think this competition is a bit different. \n\nIn this competition, we are predicting the residual returns of cryptocurrencies and not the actual return over a 15 minute period. The residual return is equal to R(t)−βM(t) where R(T) is the actual return of the coin, β is equal to beta and M is equal to the weighted return of the market.\n\nMost people created features for individual coins that could be used in a GBDT to predict the return of that associated coin. However, the weighted return of the market is also used in the calculation of the target, meaning that features for the other 13 coins have to be used in order to encapsulate all of the information needed to predict the target.\n\nI created a new notebook to illustrate that: kaggle.com/petersk20/tree-based-methods/notebook. I trained two lightgbm models to predict the target for Bitcoin. Model one included the beta value and the return of Bitcoin used in the target calculation. This one had a correlation coefficient of 0.382. The second model utilized the beta value, the return of bitcoin, and the return of the 13 other coins. This one had a correlation coefficient of 0.941. This shows that if you don't include information about other coins in your model your maximum score drops significantly.\n\nTo solve this problem, one could include all of the features created for each coin in their GBDT problem. However, this would multiply the number of features by a factor of 14 which would most likely cause the GBDT to overfit and not find good patterns.\n\nThis problem is perfect for neural networks because it is easy in Pytorch or Tensorflow to optimize multiple things at the same time. Secondary losses can be utilized to create rich feature representation for each coin and these representations can be combined in a final output MLP. For example, a neural network could be created with multiple input branches for each coin. At the end of each branch, a loss function for the individual return of the coin in the next 15 minutes could be predicted. Feature representations from previous layers of each branch can then be combined to predict the actual target for the competition.", "votes": null }, { "id": "1673690", "postDate": "02/02/2022 22:08:29", "content": "That is very interesting. It makes sense that nueral networks would be better at predicting the competition target directly. I personally chose to predict normal returns using lgbm, predict beta by using the last 3734 minute data, and construct m with the return predictions. For features, I did as you suggested and used returns for all coins. This does take up space though, giving 14 new features for each return feature. I personally think a mix of gbm and nn will do best as is often the case.
", "rawMarkdown": "That is very interesting. It makes sense that nueral networks would be better at predicting the competition target directly. I personally chose to predict normal returns using lgbm, predict beta by using the last 3734 minute data, and construct m with the return predictions. For features, I did as you suggested and used returns for all coins. This does take up space though, giving 14 new features for each return feature. I personally think a mix of gbm and nn will do best as is often the case.", "votes": null }, { "id": "1673713", "postDate": "02/02/2022 22:51:21", "content": "I also utilized a beta found in the same way.
\nBut, the problem with lgbm is that you can't optimize everything together. If I understand you correctly, you had one model per coin that predicted the normal returns. You then took this result and fed it into another lgbm. It would work better with a neural network because you can optimize the two losses that the lgbm is trying optimize together in the same model. Because once you make predictions with the first lgbm you are leaking the true return value into the training features. You then use those leaked return features with another lgbm.
Sorry I was misleading. I used price(0) / price(-15) for each asset as for a total of 14 features. So for each row in the training dataset, I had single target, say asset_0, but I had features p(0) / p(-15) for all 14 assets. I trained one lgbm model with all the assets at once, predicting the normal returns. Then I'd use those predictions and my predicted beta to construct the competition target. You're right, a neural net could predict all 14 at once, which could be more efficient and much better training. I joined late and went with lgbm because I'm more comfortable with it, but the more I think about it, the nueral net method seems really strong.
", "rawMarkdown": "Sorry I was misleading. I used price(0) / price(-15) for each asset as for a total of 14 features. So for each row in the training dataset, I had single target, say asset_0, but I had features p(0) / p(-15) for all 14 assets. I trained one lgbm model with all the assets at once, predicting the normal returns. Then I'd use those predictions and my predicted beta to construct the competition target. You're right, a neural net could predict all 14 at once, which could be more efficient and much better training. I joined late and went with lgbm because I'm more comfortable with it, but the more I think about it, the nueral net method seems really strong.", "votes": null }, { "id": "1674054", "postDate": "02/03/2022 07:36:14", "content": "What do you mean by predicting beta? Isn't \"beta\" a function of only the logreturns ( and it's derivatives), so once you have logreturns there need not be another prediction layer, but just matrix multiplications to the 🎯?
", "rawMarkdown": "What do you mean by predicting beta? Isn't \"beta\" a function of only the logreturns ( and it's derivatives), so once you have logreturns there need not be another prediction layer, but just matrix multiplications to the 🎯?", "votes": null }, { "id": "1674067", "postDate": "02/03/2022 07:46:16", "content": "I think the problem with NN approaches is more in overfitting. You get too many interim features (hidden state outputs) where all the features are endlessly combined. Yes those actual endless combination is the power of NN's. But for this task: 'market prediction', where we talk about .05 being maybe the winner, those combinations easily pass the boundary of generalization.
\nThis is probably not unmanagable, but my belief: it would take lots of time and effort.
\nThe same overfitting effect is also present in gbm s, once you start using more and more \"derivative features\" in a single model. But it is probably much easier to manage.
", "rawMarkdown": "I think the problem with NN approaches is more in overfitting. You get too many interim features (hidden state outputs) where all the features are endlessly combined. Yes those actual endless combination is the power of NN's. But for this task: 'market prediction', where we talk about .05 being maybe the winner, those combinations easily pass the boundary of generalization. \n\nThis is probably not unmanagable, but my belief: it would take lots of time and effort. \n\nThe same overfitting effect is also present in gbm s, once you start using more and more \"derivative features\" in a single model. But it is probably much easier to manage.", "votes": null }, { "id": "1674079", "postDate": "02/03/2022 08:05:11", "content": "0.382 and 0.941 are both very overfitted. They're models with leakage right? I think you helped your model memorize the leaderboard period, by providing it with features that it used to uniquely identify and locate each leaderboard sample. Working purely with CV and no leakage, typical scores are more in the .03 to .05 range. I don't think you can rely on any conclusions made from setups that give scores in the high range you gave.
", "rawMarkdown": "0.382 and 0.941 are both very overfitted. They're models with leakage right? I think you helped your model memorize the leaderboard period, by providing it with features that it used to uniquely identify and locate each leaderboard sample. Working purely with CV and no leakage, typical scores are more in the .03 to .05 range. I don't think you can rely on any conclusions made from setups that give scores in the high range you gave.", "votes": null }, { "id": "1674243", "postDate": "02/03/2022 10:45:34", "content": "I have found TF Keras NN better, mostly because of the possibility to customize it. for exemle it is easy to build a custom loss for NN with autodiff, while standard lgbm loss implementaiton need gradient and hessian. I tried to use tf autodiff in a custom lgbm loss without success. TF Trees are not well develloped enough but it could solve this problem if you can input a custom tf loss in a tf tree model.
\nRegarding overfitting I think both can overfit. IMO avoiding overfitting is mainly a matter of what feature you use over what cv scheme. Not really a matter of boosting versus NN. After that both framework offer regularisation approaches. I think the most important form of regularisation is size of the model, which is customisable easily in both framework. After that you have l1, l2 penalisation which is a standard parameter for both framework. The main difference being in the ability to add noise / drop out in NN.
", "rawMarkdown": "I have found TF Keras NN better, mostly because of the possibility to customize it. for exemle it is easy to build a custom loss for NN with autodiff, while standard lgbm loss implementaiton need gradient and hessian. I tried to use tf autodiff in a custom lgbm loss without success. TF Trees are not well develloped enough but it could solve this problem if you can input a custom tf loss in a tf tree model. \n\nRegarding overfitting I think both can overfit. IMO avoiding overfitting is mainly a matter of what feature you use over what cv scheme. Not really a matter of boosting versus NN. After that both framework offer regularisation approaches. I think the most important form of regularisation is size of the model, which is customisable easily in both framework. After that you have l1, l2 penalisation which is a standard parameter for both framework. The main difference being in the ability to add noise / drop out in NN.", "votes": null }, { "id": "1674617", "postDate": "02/03/2022 16:36:50", "content": "That wasn't my point. I am using the actual values used in the creation of the target so of course they are overfit. They are just used to show that if you use all information present for a single coin .382 is the maximum score you could get theoretically. Whereas if you use the returns of all coins, .941 is the maximum score. This shows you need to include information about other coins to win this competition which neural networks should be able to do better
", "rawMarkdown": "That wasn't my point. I am using the actual values used in the creation of the target so of course they are overfit. They are just used to show that if you use all information present for a single coin .382 is the maximum score you could get theoretically. Whereas if you use the returns of all coins, .941 is the maximum score. This shows you need to include information about other coins to win this competition which neural networks should be able to do better", "votes": null }, { "id": "1674627", "postDate": "02/03/2022 16:40:12", "content": "We mean estimate beta by using the beta calculation on the past 3734 data.
", "rawMarkdown": "We mean estimate beta by using the beta calculation on the past 3734 data.", "votes": null }, { "id": "1685647", "postDate": "02/11/2022 12:52:45", "content": "I don't think you can make the claim that just because the score of an overfitted model is better, the model architecture also performs better on unseen data. Though I agree that it is probably necessary to utilize the data to the fullest extend by taking other coins into account. But you could make a model that e.g. takes features from bitcoin (as bitcoin probably has the biggest influence on the market) and the coin you want to make a prediction for as input for a tree based model.
", "rawMarkdown": "I don't think you can make the claim that just because the score of an overfitted model is better, the model architecture also performs better on unseen data. Though I agree that it is probably necessary to utilize the data to the fullest extend by taking other coins into account. But you could make a model that e.g. takes features from bitcoin (as bitcoin probably has the biggest influence on the market) and the coin you want to make a prediction for as input for a tree based model.", "votes": null }, { "id": "1686096", "postDate": "02/11/2022 18:51:59", "content": "@fritzcremer I never said that because the overfitted model is better, the model architecture is better. The model architecture between the .382 and .941 model are completely the same. The only difference is the data. I feel like a lot of people aren't understanding my argument so I'll make it again to see if it changes your mind. The target is composed of three things: the return of the coin, beta, and the market return. Because of this, I created two lightgbm models with exactly the same architecture. I then sliced the dataframe to only include bitcoin. In the first one, I put in the return of bitcoin over the next 15 days and the beta value. Both of these are unrealistic because they include future data. However, this only scored a correlation of .382 to the target. This means if we used the actual return of bitcoin and beta to predict the target it still doesn't even get close to the maximum 1.0 correlation. I then took the returns of the other coins and put them into the model and got .941 correlation. This means that a model for bitcoin must use features of all the other coins to encompass all of the information needed to predict the target. I then made an argument for why neural networks allowing multiple branches and multiple losses would perform better because they can be used to process the large number of features from each coin independently and then combine the condensed rich feature representations for each coin in a final MLP model.
", "rawMarkdown": "fritzcremer I never said that because the overfitted model is better, the model architecture is better. The model architecture between the .382 and .941 model are completely the same. The only difference is the data. I feel like a lot of people aren't understanding my argument so I'll make it again to see if it changes your mind. The target is composed of three things: the return of the coin, beta, and the market return. Because of this, I created two lightgbm models with exactly the same architecture. I then sliced the dataframe to only include bitcoin. In the first one, I put in the return of bitcoin over the next 15 days and the beta value. Both of these are unrealistic because they include future data. However, this only scored a correlation of .382 to the target. This means if we used the actual return of bitcoin and beta to predict the target it still doesn't even get close to the maximum 1.0 correlation. I then took the returns of the other coins and put them into the model and got .941 correlation. This means that a model for bitcoin must use features of all the other coins to encompass all of the information needed to predict the target. I then made an argument for why neural networks allowing multiple branches and multiple losses would perform better because they can be used to process the large number of features from each coin independently and then combine the condensed rich feature representations for each coin in a final MLP model.", "votes": null }, { "id": "1686113", "postDate": "02/11/2022 19:04:06", "content": "I also think that neural nets have that advantage, but I am still unsure why the numbers .382 and .941 tell us a lot, when we expect the actual top notebooks to be way below 0.1. I agree with the hypothesis you make but I just don't think such a comparison is meaningful.
", "rawMarkdown": "I also think that neural nets have that advantage, but I am still unsure why the numbers .382 and .941 tell us a lot, when we expect the actual top notebooks to be way below 0.1. I agree with the hypothesis you make but I just don't think such a comparison is meaningful.", "votes": null }, { "id": "1686184", "postDate": "02/11/2022 20:42:36", "content": "The .384 number tells us that even if we include the actual return of bitcoin and the real beta value we will only get a value of .384 for the correlation. This tells us that we need to include the information from other coins to encapsulate all of the information needed to predict the target.
", "rawMarkdown": "The .384 number tells us that even if we include the actual return of bitcoin and the real beta value we will only get a value of .384 for the correlation. This tells us that we need to include the information from other coins to encapsulate all of the information needed to predict the target.", "votes": null }, { "id": "1688577", "postDate": "02/13/2022 17:53:54", "content": "Yes, like @petersk20 said, estimating beta from the known 3734 measurements that will be included.
", "rawMarkdown": "Yes, like @petersk20 said, estimating beta from the known 3734 measurements that will be included.", "votes": null } ], "comments": [ { "id": 1673690, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "02/02/2022 22:08:29", "content": "That is very interesting. It makes sense that nueral networks would be better at predicting the competition target directly. I personally chose to predict normal returns using lgbm, predict beta by using the last 3734 minute data, and construct m with the return predictions. For features, I did as you suggested and used returns for all coins. This does take up space though, giving 14 new features for each return feature. I personally think a mix of gbm and nn will do best as is often the case.
", "votes": null, "replies": [ { "id": 1673713, "author_name": "petersk20", "author_url": "", "post_date": "02/02/2022 22:51:21", "content": "I also utilized a beta found in the same way.
\nBut, the problem with lgbm is that you can't optimize everything together. If I understand you correctly, you had one model per coin that predicted the normal returns. You then took this result and fed it into another lgbm. It would work better with a neural network because you can optimize the two losses that the lgbm is trying optimize together in the same model. Because once you make predictions with the first lgbm you are leaking the true return value into the training features. You then use those leaked return features with another lgbm.
Sorry I was misleading. I used price(0) / price(-15) for each asset as for a total of 14 features. So for each row in the training dataset, I had single target, say asset_0, but I had features p(0) / p(-15) for all 14 assets. I trained one lgbm model with all the assets at once, predicting the normal returns. Then I'd use those predictions and my predicted beta to construct the competition target. You're right, a neural net could predict all 14 at once, which could be more efficient and much better training. I joined late and went with lgbm because I'm more comfortable with it, but the more I think about it, the nueral net method seems really strong.
", "votes": null, "replies": [] }, { "id": 1674054, "author_name": "abdulkadirguner", "author_url": "", "post_date": "02/03/2022 07:36:14", "content": "What do you mean by predicting beta? Isn't \"beta\" a function of only the logreturns ( and it's derivatives), so once you have logreturns there need not be another prediction layer, but just matrix multiplications to the 🎯?
", "votes": null, "replies": [] }, { "id": 1674627, "author_name": "petersk20", "author_url": "", "post_date": "02/03/2022 16:40:12", "content": "We mean estimate beta by using the beta calculation on the past 3734 data.
", "votes": null, "replies": [] }, { "id": 1688577, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "02/13/2022 17:53:54", "content": "Yes, like @petersk20 said, estimating beta from the known 3734 measurements that will be included.
", "votes": null, "replies": [] } ] }, { "id": 1674067, "author_name": "abdulkadirguner", "author_url": "", "post_date": "02/03/2022 07:46:16", "content": "I think the problem with NN approaches is more in overfitting. You get too many interim features (hidden state outputs) where all the features are endlessly combined. Yes those actual endless combination is the power of NN's. But for this task: 'market prediction', where we talk about .05 being maybe the winner, those combinations easily pass the boundary of generalization.
\nThis is probably not unmanagable, but my belief: it would take lots of time and effort.
\nThe same overfitting effect is also present in gbm s, once you start using more and more \"derivative features\" in a single model. But it is probably much easier to manage.
", "votes": null, "replies": [] }, { "id": 1674079, "author_name": "nathanrm", "author_url": "", "post_date": "02/03/2022 08:05:11", "content": "0.382 and 0.941 are both very overfitted. They're models with leakage right? I think you helped your model memorize the leaderboard period, by providing it with features that it used to uniquely identify and locate each leaderboard sample. Working purely with CV and no leakage, typical scores are more in the .03 to .05 range. I don't think you can rely on any conclusions made from setups that give scores in the high range you gave.
", "votes": null, "replies": [ { "id": 1674617, "author_name": "petersk20", "author_url": "", "post_date": "02/03/2022 16:36:50", "content": "That wasn't my point. I am using the actual values used in the creation of the target so of course they are overfit. They are just used to show that if you use all information present for a single coin .382 is the maximum score you could get theoretically. Whereas if you use the returns of all coins, .941 is the maximum score. This shows you need to include information about other coins to win this competition which neural networks should be able to do better
", "votes": null, "replies": [] }, { "id": 1685647, "author_name": "fritzcremer", "author_url": "", "post_date": "02/11/2022 12:52:45", "content": "I don't think you can make the claim that just because the score of an overfitted model is better, the model architecture also performs better on unseen data. Though I agree that it is probably necessary to utilize the data to the fullest extend by taking other coins into account. But you could make a model that e.g. takes features from bitcoin (as bitcoin probably has the biggest influence on the market) and the coin you want to make a prediction for as input for a tree based model.
", "votes": null, "replies": [] }, { "id": 1686096, "author_name": "petersk20", "author_url": "", "post_date": "02/11/2022 18:51:59", "content": "@fritzcremer I never said that because the overfitted model is better, the model architecture is better. The model architecture between the .382 and .941 model are completely the same. The only difference is the data. I feel like a lot of people aren't understanding my argument so I'll make it again to see if it changes your mind. The target is composed of three things: the return of the coin, beta, and the market return. Because of this, I created two lightgbm models with exactly the same architecture. I then sliced the dataframe to only include bitcoin. In the first one, I put in the return of bitcoin over the next 15 days and the beta value. Both of these are unrealistic because they include future data. However, this only scored a correlation of .382 to the target. This means if we used the actual return of bitcoin and beta to predict the target it still doesn't even get close to the maximum 1.0 correlation. I then took the returns of the other coins and put them into the model and got .941 correlation. This means that a model for bitcoin must use features of all the other coins to encompass all of the information needed to predict the target. I then made an argument for why neural networks allowing multiple branches and multiple losses would perform better because they can be used to process the large number of features from each coin independently and then combine the condensed rich feature representations for each coin in a final MLP model.
", "votes": null, "replies": [] }, { "id": 1686113, "author_name": "fritzcremer", "author_url": "", "post_date": "02/11/2022 19:04:06", "content": "I also think that neural nets have that advantage, but I am still unsure why the numbers .382 and .941 tell us a lot, when we expect the actual top notebooks to be way below 0.1. I agree with the hypothesis you make but I just don't think such a comparison is meaningful.
", "votes": null, "replies": [] }, { "id": 1686184, "author_name": "petersk20", "author_url": "", "post_date": "02/11/2022 20:42:36", "content": "The .384 number tells us that even if we include the actual return of bitcoin and the real beta value we will only get a value of .384 for the correlation. This tells us that we need to include the information from other coins to encapsulate all of the information needed to predict the target.
", "votes": null, "replies": [] } ] }, { "id": 1674243, "author_name": "lucasmorin", "author_url": "", "post_date": "02/03/2022 10:45:34", "content": "I have found TF Keras NN better, mostly because of the possibility to customize it. for exemle it is easy to build a custom loss for NN with autodiff, while standard lgbm loss implementaiton need gradient and hessian. I tried to use tf autodiff in a custom lgbm loss without success. TF Trees are not well develloped enough but it could solve this problem if you can input a custom tf loss in a tf tree model.
\nRegarding overfitting I think both can overfit. IMO avoiding overfitting is mainly a matter of what feature you use over what cv scheme. Not really a matter of boosting versus NN. After that both framework offer regularisation approaches. I think the most important form of regularisation is size of the model, which is customisable easily in both framework. After that you have l1, l2 penalisation which is a standard parameter for both framework. The main difference being in the ability to add noise / drop out in NN.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1673677": "I've seen a lot of people in this competition use gradient boosted decision trees, but I think they were the wrong model to use for this problem. Let me know your thought.\n\nFor many Kaggle competition, gradient boosted decision trees like lightgbm and xgboost perform very well and usually end up at the top of the leaderboard. However, I think this competition is a bit different. \n\nIn this competition, we are predicting the residual returns of cryptocurrencies and not the actual return over a 15 minute period. The residual return is equal to R(t)−βM(t) where R(T) is the actual return of the coin, β is equal to beta and M is equal to the weighted return of the market.\n\nMost people created features for individual coins that could be used in a GBDT to predict the return of that associated coin. However, the weighted return of the market is also used in the calculation of the target, meaning that features for the other 13 coins have to be used in order to encapsulate all of the information needed to predict the target.\n\nI created a new notebook to illustrate that: kaggle.com/petersk20/tree-based-methods/notebook. I trained two lightgbm models to predict the target for Bitcoin. Model one included the beta value and the return of Bitcoin used in the target calculation. This one had a correlation coefficient of 0.382. The second model utilized the beta value, the return of bitcoin, and the return of the 13 other coins. This one had a correlation coefficient of 0.941. This shows that if you don't include information about other coins in your model your maximum score drops significantly.\n\nTo solve this problem, one could include all of the features created for each coin in their GBDT problem. However, this would multiply the number of features by a factor of 14 which would most likely cause the GBDT to overfit and not find good patterns.\n\nThis problem is perfect for neural networks because it is easy in Pytorch or Tensorflow to optimize multiple things at the same time. Secondary losses can be utilized to create rich feature representation for each coin and these representations can be combined in a final output MLP. For example, a neural network could be created with multiple input branches for each coin. At the end of each branch, a loss function for the individual return of the coin in the next 15 minutes could be predicted. Feature representations from previous layers of each branch can then be combined to predict the actual target for the competition.", "1673690": "That is very interesting. It makes sense that nueral networks would be better at predicting the competition target directly. I personally chose to predict normal returns using lgbm, predict beta by using the last 3734 minute data, and construct m with the return predictions. For features, I did as you suggested and used returns for all coins. This does take up space though, giving 14 new features for each return feature. I personally think a mix of gbm and nn will do best as is often the case.", "1673713": "I also utilized a beta found in the same way. \nBut, the problem with lgbm is that you can't optimize everything together. If I understand you correctly, you had one model per coin that predicted the normal returns. You then took this result and fed it into another lgbm. It would work better with a neural network because you can optimize the two losses that the lgbm is trying optimize together in the same model. Because once you make predictions with the first lgbm you are leaking the true return value into the training features. You then use those leaked return features with another lgbm.", "1673870": "Sorry I was misleading. I used price(0) / price(-15) for each asset as for a total of 14 features. So for each row in the training dataset, I had single target, say asset_0, but I had features p(0) / p(-15) for all 14 assets. I trained one lgbm model with all the assets at once, predicting the normal returns. Then I'd use those predictions and my predicted beta to construct the competition target. You're right, a neural net could predict all 14 at once, which could be more efficient and much better training. I joined late and went with lgbm because I'm more comfortable with it, but the more I think about it, the nueral net method seems really strong.", "1674054": "What do you mean by predicting beta? Isn't \"beta\" a function of only the logreturns ( and it's derivatives), so once you have logreturns there need not be another prediction layer, but just matrix multiplications to the 🎯?", "1674067": "I think the problem with NN approaches is more in overfitting. You get too many interim features (hidden state outputs) where all the features are endlessly combined. Yes those actual endless combination is the power of NN's. But for this task: 'market prediction', where we talk about .05 being maybe the winner, those combinations easily pass the boundary of generalization. \n\nThis is probably not unmanagable, but my belief: it would take lots of time and effort. \n\nThe same overfitting effect is also present in gbm s, once you start using more and more \"derivative features\" in a single model. But it is probably much easier to manage.", "1674079": "0.382 and 0.941 are both very overfitted. They're models with leakage right? I think you helped your model memorize the leaderboard period, by providing it with features that it used to uniquely identify and locate each leaderboard sample. Working purely with CV and no leakage, typical scores are more in the .03 to .05 range. I don't think you can rely on any conclusions made from setups that give scores in the high range you gave.", "1674243": "I have found TF Keras NN better, mostly because of the possibility to customize it. for exemle it is easy to build a custom loss for NN with autodiff, while standard lgbm loss implementaiton need gradient and hessian. I tried to use tf autodiff in a custom lgbm loss without success. TF Trees are not well develloped enough but it could solve this problem if you can input a custom tf loss in a tf tree model. \n\nRegarding overfitting I think both can overfit. IMO avoiding overfitting is mainly a matter of what feature you use over what cv scheme. Not really a matter of boosting versus NN. After that both framework offer regularisation approaches. I think the most important form of regularisation is size of the model, which is customisable easily in both framework. After that you have l1, l2 penalisation which is a standard parameter for both framework. The main difference being in the ability to add noise / drop out in NN.", "1674617": "That wasn't my point. I am using the actual values used in the creation of the target so of course they are overfit. They are just used to show that if you use all information present for a single coin .382 is the maximum score you could get theoretically. Whereas if you use the returns of all coins, .941 is the maximum score. This shows you need to include information about other coins to win this competition which neural networks should be able to do better", "1674627": "We mean estimate beta by using the beta calculation on the past 3734 data.", "1685647": "I don't think you can make the claim that just because the score of an overfitted model is better, the model architecture also performs better on unseen data. Though I agree that it is probably necessary to utilize the data to the fullest extend by taking other coins into account. But you could make a model that e.g. takes features from bitcoin (as bitcoin probably has the biggest influence on the market) and the coin you want to make a prediction for as input for a tree based model.", "1686096": "fritzcremer I never said that because the overfitted model is better, the model architecture is better. The model architecture between the .382 and .941 model are completely the same. The only difference is the data. I feel like a lot of people aren't understanding my argument so I'll make it again to see if it changes your mind. The target is composed of three things: the return of the coin, beta, and the market return. Because of this, I created two lightgbm models with exactly the same architecture. I then sliced the dataframe to only include bitcoin. In the first one, I put in the return of bitcoin over the next 15 days and the beta value. Both of these are unrealistic because they include future data. However, this only scored a correlation of .382 to the target. This means if we used the actual return of bitcoin and beta to predict the target it still doesn't even get close to the maximum 1.0 correlation. I then took the returns of the other coins and put them into the model and got .941 correlation. This means that a model for bitcoin must use features of all the other coins to encompass all of the information needed to predict the target. I then made an argument for why neural networks allowing multiple branches and multiple losses would perform better because they can be used to process the large number of features from each coin independently and then combine the condensed rich feature representations for each coin in a final MLP model.", "1686113": "I also think that neural nets have that advantage, but I am still unsure why the numbers .382 and .941 tell us a lot, when we expect the actual top notebooks to be way below 0.1. I agree with the hypothesis you make but I just don't think such a comparison is meaningful.", "1686184": "The .384 number tells us that even if we include the actual return of bitcoin and the real beta value we will only get a value of .384 for the correlation. This tells us that we need to include the information from other coins to encapsulate all of the information needed to predict the target.", "1688577": "Yes, like @petersk20 said, estimating beta from the known 3734 measurements that will be included." }, "source": "meta" }