![\"download.jpg\"](\"https://i.postimg.cc/mZ5JbV2B/download.jpg\")
View the final submission notebook here.
\nFinal Update - Congratulations to all the winners! While this comp was a bit of a rollercoaster, I'm very pleased to get my first competition Gold. Disappointing to just miss out on the top 10 after being there so long, but thats always possible with this type of challenge.
\nOriginal post below.
\nFirst of all thankyou to G-Research and Cambridge Spark for hosting this competition and to the Kaggle staff for making everything run relatively smoothly. I found it so interesting and more open ended than other comps I've competed in, in my short Kaggler career.
\nIt goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts. Hopefully you find this solution useful regardless of the final outcome.
\nElephant in the room - there has been a lot of concern about the validity of the final LB results and requests to release the final test data after the competition ends so that competitors can validate their final scores. I think this would be a good idea and should put any concerns to bed. If it's possible & not too much effort for Kaggle to accommodate the request please consider it. However, as far as im concerned my current score seems reasonable based on all the scores I achieved in testing. I have no reason to believe any of the LB scores are incorrect until there is evidence to the contrary.
\nI would encourage those who feel their score is unexpected share their solutions as the Kaggle community might be able to debug any issues/misunderstandings.
\nIt is obviously difficult to accurately predict crypto prices, but it's possible to find some signal - hopefully this notebook will shed some light on how I got mine.
\nDisclaimers
\nTest Scores (no lookahead bias):
\nI downloaded free public data from multiple exchange apis (Binance, FTX, Coinbase, Kucoin… etc.) to see if this extra data would improve my models. Some of my final models were trained on extra data. Specifically, using some currencies not included in the competition (e.g. XRP, ZEC…) from Binance seemed to provide a consistent small improvement over multiple timeframes.
\nI settled on a group of 17 features for all models. There are 8 lagged features - a simple mixture of EMA's, historical returns and historical volatility over various lookback periods - and these features were averaged across timestamps to produce 8 more. Asset_ID was also included. It was tricky to find a group that performed the best consistently across different time periods, adding and removing features from this set seemed to reduce performance. Im sure this isn't the optimal feature set, but this group seemed to work well enough.
\nIt was also important to perform some kind on binning on the features, especially for training the LGBM model. The commonly used reduce_mem_usage function and some rounding functions seemed to provide a suitable amount of bins. I found binning to 500-1000 unique values worked well for any given continuous feature.
\nI think a crucial part of this competition was manipulating the target. Thanks to the work of @alexfir and other Kagglers we found out how the target was being calculated.
\nI thought it would be useful to split the target into two components:
As @gengdaiziwang illustrates in his helpful notebook, we can see that in the case where an observation is missing for a given asset, the beta component is automatically set to 0.
\n
But this makes it a completely different target! A model trained on the target given as standard will have to deal with training on a target that randomly switches between a 15 minute forward return for one asset vs what is essentially a 15 minute forward return relative to the other assets - very different things.
\nThe approach I took was to create recreate two targets to represent these two different cases. Then I could train separate models on these recreated targets which would learn the two cases much more effectively.
\nSee an implementation of this target engineering.
Now, having two models trained on different targets, I can alter my predictions to match the target produced by the api. The api provides all the information i need to figure out whether the beta component will be == 0 because if an asset is missing from an iteration i know that for the next 3750 iterations the beta component == 0 for this asset. Therefore my next 3750 predictions will come from a model that was training on a target constructued without a beta component.
\nThis target engineering and dual model method added roughly 0.01 to my score on the original test period, although this did vary quite significantly for other testing periods.
\nTo summarise as simply as possible:
\n![\"gres.png\"](\"https://i.postimg.cc/ZRVkFSSd/gres.png\")
I found LightGBM worked well and was easy to experiment with, i used the weighted correlation evaluation metric and fairly out of the box hyperparameters, tuning didnt seem to add much and wasn't consistent across time periods.
\nI also used a Keras NN heavily influenced by @lucasmorin's
\nexcellent notebook. This just worked really well for me out of the box and i couldnt find many ways to improve the architecture or hyperparameters.
Ensembling these two models worked well and added an extra 0.005 to my score (roughly) although the improvement varies in different market regimes. I also experimented with ensembles using catboost and XGboost but the predictions from most gradient boosting models were too highly correlated to provide much ensembling benefit. I found the diversity in predictions between NN and GB were a good match.
\nI used a fairly simple CV method for training and testing. My folds were based on timestamp values similar to the below:
\nSince this public lb was useless in this competition, it was essential to construct a robust CV framework to have a reliable benchmark for making improvements to models. It was also important to have test folds in several different market regimes to avoid overfitting to one period.
\nMy best single model was a NN trained on fold 0 data only (surprisingly).
\nI was fairly selective on picking models to include in the final submission as some folds just trained much better than others and had more consistent outperformance on unseen data.
\nThis is almost the exact notebook I submitted for my final predictions. I used lists to store historical data for feature calculations and did most calculations using numpy or lists, avoiding pandas at all costs.
\nThe submission completes in roughly 7 hours.
\nView the final submission notebook here.
\nFinal Update - Congratulations to all the winners! While this comp was a bit of a rollercoaster, I'm very pleased to get my first competition Gold. Disappointing to just miss out on the top 10 after being there so long, but thats always possible with this type of challenge.
\nOriginal post below.
\nFirst of all thankyou to G-Research and Cambridge Spark for hosting this competition and to the Kaggle staff for making everything run relatively smoothly. I found it so interesting and more open ended than other comps I've competed in, in my short Kaggler career.
\nIt goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts. Hopefully you find this solution useful regardless of the final outcome.
\nElephant in the room - there has been a lot of concern about the validity of the final LB results and requests to release the final test data after the competition ends so that competitors can validate their final scores. I think this would be a good idea and should put any concerns to bed. If it's possible & not too much effort for Kaggle to accommodate the request please consider it. However, as far as im concerned my current score seems reasonable based on all the scores I achieved in testing. I have no reason to believe any of the LB scores are incorrect until there is evidence to the contrary.
\nI would encourage those who feel their score is unexpected share their solutions as the Kaggle community might be able to debug any issues/misunderstandings.
\nIt is obviously difficult to accurately predict crypto prices, but it's possible to find some signal - hopefully this notebook will shed some light on how I got mine.
\nDisclaimers
\nTest Scores (no lookahead bias):
\nI downloaded free public data from multiple exchange apis (Binance, FTX, Coinbase, Kucoin… etc.) to see if this extra data would improve my models. Some of my final models were trained on extra data. Specifically, using some currencies not included in the competition (e.g. XRP, ZEC…) from Binance seemed to provide a consistent small improvement over multiple timeframes.
\nI settled on a group of 17 features for all models. There are 8 lagged features - a simple mixture of EMA's, historical returns and historical volatility over various lookback periods - and these features were averaged across timestamps to produce 8 more. Asset_ID was also included. It was tricky to find a group that performed the best consistently across different time periods, adding and removing features from this set seemed to reduce performance. Im sure this isn't the optimal feature set, but this group seemed to work well enough.
\nIt was also important to perform some kind on binning on the features, especially for training the LGBM model. The commonly used reduce_mem_usage function and some rounding functions seemed to provide a suitable amount of bins. I found binning to 500-1000 unique values worked well for any given continuous feature.
\nI think a crucial part of this competition was manipulating the target. Thanks to the work of @alexfir and other Kagglers we found out how the target was being calculated.
\nI thought it would be useful to split the target into two components:
As @gengdaiziwang illustrates in his helpful notebook, we can see that in the case where an observation is missing for a given asset, the beta component is automatically set to 0.
\n
But this makes it a completely different target! A model trained on the target given as standard will have to deal with training on a target that randomly switches between a 15 minute forward return for one asset vs what is essentially a 15 minute forward return relative to the other assets - very different things.
\nThe approach I took was to create recreate two targets to represent these two different cases. Then I could train separate models on these recreated targets which would learn the two cases much more effectively.
\nSee an implementation of this target engineering.
Now, having two models trained on different targets, I can alter my predictions to match the target produced by the api. The api provides all the information i need to figure out whether the beta component will be == 0 because if an asset is missing from an iteration i know that for the next 3750 iterations the beta component == 0 for this asset. Therefore my next 3750 predictions will come from a model that was training on a target constructued without a beta component.
\nThis target engineering and dual model method added roughly 0.01 to my score on the original test period, although this did vary quite significantly for other testing periods.
\nTo summarise as simply as possible:
\n
I found LightGBM worked well and was easy to experiment with, i used the weighted correlation evaluation metric and fairly out of the box hyperparameters, tuning didnt seem to add much and wasn't consistent across time periods.
\nI also used a Keras NN heavily influenced by @lucasmorin's
\nexcellent notebook. This just worked really well for me out of the box and i couldnt find many ways to improve the architecture or hyperparameters.
Ensembling these two models worked well and added an extra 0.005 to my score (roughly) although the improvement varies in different market regimes. I also experimented with ensembles using catboost and XGboost but the predictions from most gradient boosting models were too highly correlated to provide much ensembling benefit. I found the diversity in predictions between NN and GB were a good match.
\nI used a fairly simple CV method for training and testing. My folds were based on timestamp values similar to the below:
\nSince this public lb was useless in this competition, it was essential to construct a robust CV framework to have a reliable benchmark for making improvements to models. It was also important to have test folds in several different market regimes to avoid overfitting to one period.
\nMy best single model was a NN trained on fold 0 data only (surprisingly).
\nI was fairly selective on picking models to include in the final submission as some folds just trained much better than others and had more consistent outperformance on unseen data.
\nThis is almost the exact notebook I submitted for my final predictions. I used lists to store historical data for feature calculations and did most calculations using numpy or lists, avoiding pandas at all costs.
\nThe submission completes in roughly 7 hours.
\nGreat write up. Thank you for sharing. I am surprised you were able to fit all the feature engineering and ensembles inside the predict loop… I kept running into time limits when trying similar. As for me, after three runs…56 -> 84 -> 164 … 😬
", "rawMarkdown": "Great write up. Thank you for sharing. I am surprised you were able to fit all the feature engineering and ensembles inside the predict loop... I kept running into time limits when trying similar. As for me, after three runs...56 -> 84 -> 164 ... 😬", "votes": null }, { "id": "1725138", "postDate": "03/16/2022 20:55:40", "content": "Can you post a link to where you got data from those exchanges in the same format as competition?
\nI think you are jumping the gun a bit on LB placings, but nevertheless I think you have an interesting solution, I did not consider the target engineering for mine I think that was a good idea for sure.
\nWell done on the 6 week performance and good luck for the rest! 🙂
", "rawMarkdown": "Can you post a link to where you got data from those exchanges in the same format as competition? \n\nI think you are jumping the gun a bit on LB placings, but nevertheless I think you have an interesting solution, I did not consider the target engineering for mine I think that was a good idea for sure.\n\nWell done on the 6 week performance and good luck for the rest! 🙂", "votes": null }, { "id": "1725140", "postDate": "03/16/2022 20:56:08", "content": "Great job Tom. I just has a couple questions about the time period of the training set you used? What time period did you use to train your final models on? Did it include the supplemental training set. Did it vary across the models in your ensemble?
", "rawMarkdown": "Great job Tom. I just has a couple questions about the time period of the training set you used? What time period did you use to train your final models on? Did it include the supplemental training set. Did it vary across the models in your ensemble?", "votes": null }, { "id": "1725143", "postDate": "03/16/2022 21:03:54", "content": "Also, maybe the reason there are discrepancies in other people's models and not your models is because of a heightened amount of NA's in the data. This would cause more betas to be zero which would throw off other people's models but not yours due to the feature extraction. I noticed an uptick in NA's from the public lb to supplemental dataset but only a slight one.
", "rawMarkdown": "Also, maybe the reason there are discrepancies in other people's models and not your models is because of a heightened amount of NA's in the data. This would cause more betas to be zero which would throw off other people's models but not yours due to the feature extraction. I noticed an uptick in NA's from the public lb to supplemental dataset but only a slight one.", "votes": null }, { "id": "1725151", "postDate": "03/16/2022 21:10:15", "content": "Good work! Same here, majority of the time feature engineering, very little modeling in contrast to other competitions. Will be interesting weeks ahead following the progress :)
", "rawMarkdown": "Good work! Same here, majority of the time feature engineering, very little modeling in contrast to other competitions. Will be interesting weeks ahead following the progress :)", "votes": null }, { "id": "1725165", "postDate": "03/16/2022 21:36:00", "content": "Looks great @tomforbes , thanks for sharing that. I find very interesting that you have managed to make your approach robust to data quality which was a huge condition for label generation.
\nI have given already clearly enough my opinion that the statistical nature of the labels at test time is very different to the previous one, and not because of market forces. And I still consider that huge label quality shift at test time isn't legit for a Kaggle comp, but that doesn't take away the merit that your solution is cleverly prepared to deal with it, so congratulations for that.
\nAs for myself, I have spent way too much energy in this competition, especially addressing the \"elephant in the room\" issue but I am goint to spend no more time on it. Every competition teaches you something and this one has not been an exception, hopefully I will choose a better way to invest my time next time.
\nAgain, thanks for sharing and good luck!
", "rawMarkdown": "Looks great @tomforbes , thanks for sharing that. I find very interesting that you have managed to make your approach robust to data quality which was a huge condition for label generation. \n\nI have given already clearly enough my opinion that the statistical nature of the labels at test time is very different to the previous one, and not because of market forces. And I still consider that huge label quality shift at test time isn't legit for a Kaggle comp, but that doesn't take away the merit that your solution is cleverly prepared to deal with it, so congratulations for that. \n\nAs for myself, I have spent way too much energy in this competition, especially addressing the \"elephant in the room\" issue but I am goint to spend no more time on it. Every competition teaches you something and this one has not been an exception, hopefully I will choose a better way to invest my time next time. \n\nAgain, thanks for sharing and good luck!", "votes": null }, { "id": "1725170", "postDate": "03/16/2022 21:46:49", "content": "Thanks Kyle! I have outlined some of the training and validation folds i used in this post. For example my best model was a NN trained on data from 1514764860 - 1570000000 (timestamp). I choose whichever folds performed best overall on my testing folds. I didn't use any supplemental_train data for training, only testing. I didn't train any models that included data after ~1610000000
", "rawMarkdown": "Thanks Kyle! I have outlined some of the training and validation folds i used in this post. For example my best model was a NN trained on data from 1514764860 - 1570000000 (timestamp). I choose whichever folds performed best overall on my testing folds. I didn't use any supplemental_train data for training, only testing. I didn't train any models that included data after ~1610000000", "votes": null }, { "id": "1725172", "postDate": "03/16/2022 21:50:07", "content": "Maybe, but then I imagine most people trained their models on the target as originally constructed which had plenty of beta == zero, so whatever results they got in testing should probably still apply unless there is an unusual amount of missing data. It seemed to be the trend that the most recent data had the least missing data
", "rawMarkdown": "Maybe, but then I imagine most people trained their models on the target as originally constructed which had plenty of beta == zero, so whatever results they got in testing should probably still apply unless there is an unusual amount of missing data. It seemed to be the trend that the most recent data had the least missing data", "votes": null }, { "id": "1725174", "postDate": "03/16/2022 21:54:43", "content": "I was only interested in OHLC data so didn't need to get the more hard to source features like VWAP. This is pretty easy to find if you google most exchanges have public api docs. E.g. binance
\nYeah as i said its 1st place 6 weeks in - I could slip a few of places by the end but thought i'd share what i'd written up
", "rawMarkdown": "I was only interested in OHLC data so didn't need to get the more hard to source features like VWAP. This is pretty easy to find if you google most exchanges have public api docs. E.g. [binance](https://www.binance.com/en/landing/data)\n\nYeah as i said its 1st place 6 weeks in - I could slip a few of places by the end but thought i'd share what i'd written up", "votes": null }, { "id": "1725178", "postDate": "03/16/2022 21:57:00", "content": "That makes sense. I saw that but didn't know if that was just used for cv or if it was also used in the final model.
", "rawMarkdown": "That makes sense. I saw that but didn't know if that was just used for cv or if it was also used in the final model.", "votes": null }, { "id": "1725180", "postDate": "03/16/2022 22:02:20", "content": "Nice solution :-) Congrats on pulling the hard data sourcing / target engineering work. So in the end you only use very few features ? Did you keep the noise from my nb ? (btw thanks for citing my work - hope to get gold)
", "rawMarkdown": "Nice solution :-) Congrats on pulling the hard data sourcing / target engineering work. So in the end you only use very few features ? Did you keep the noise from my nb ? (btw thanks for citing my work - hope to get gold)", "votes": null }, { "id": "1725535", "postDate": "03/17/2022 07:21:06", "content": "Great work Tom Forbes!! Looks like you might just be deserving of first place after all :).
", "rawMarkdown": "Great work Tom Forbes!! Looks like you might just be deserving of first place after all :).", "votes": null }, { "id": "1725751", "postDate": "03/17/2022 12:06:44", "content": "To take into account the beta=0 cases I just added the approximated beta calculated with the existing history as a feature to the model and hoped the model will learn from this feature. In this way I did not have to split the models for the two cases. Would have been interesting actually to check if separate models improve.
", "rawMarkdown": "To take into account the beta=0 cases I just added the approximated beta calculated with the existing history as a feature to the model and hoped the model will learn from this feature. In this way I did not have to split the models for the two cases. Would have been interesting actually to check if separate models improve.", "votes": null }, { "id": "1725977", "postDate": "03/17/2022 14:57:34", "content": "Respect. The target switching approach is very neat!
", "rawMarkdown": "Respect. The target switching approach is very neat!", "votes": null }, { "id": "1726151", "postDate": "03/17/2022 17:29:49", "content": "Thanks! Yeah only 17 features for all models. Yes for some models i kept the noise and some seemed to train better without. It certainly deserves a gold!
", "rawMarkdown": "Thanks! Yeah only 17 features for all models. Yes for some models i kept the noise and some seemed to train better without. It certainly deserves a gold!", "votes": null }, { "id": "1726153", "postDate": "03/17/2022 17:31:46", "content": "Yeah i tried using beta as a feature but i didn't get a lot out of it. Think the benefit of splitting out the models is the target is consistent and allows for better training
", "rawMarkdown": "Yeah i tried using beta as a feature but i didn't get a lot out of it. Think the benefit of splitting out the models is the target is consistent and allows for better training", "votes": null }, { "id": "1726205", "postDate": "03/17/2022 18:45:27", "content": "Great writeup!
", "rawMarkdown": "Great writeup!", "votes": null }, { "id": "1728336", "postDate": "03/18/2022 19:40:44", "content": "This is great. Thanks for sharing
", "rawMarkdown": "This is great. Thanks for sharing", "votes": null }, { "id": "1728396", "postDate": "03/18/2022 21:09:49", "content": "Great Job @tomforbes, I hope you have good luck in the rest of the competition. I have a question, what was the longest period of time that your models worked on the validation? …10000000 from timestamps?
", "rawMarkdown": "Great Job @tomforbes, I hope you have good luck in the rest of the competition. I have a question, what was the longest period of time that your models worked on the validation? ...10000000 from timestamps?", "votes": null }, { "id": "1729046", "postDate": "03/19/2022 15:16:40", "content": "Great work. Congrats.
", "rawMarkdown": "Great work. Congrats.", "votes": null }, { "id": "1729052", "postDate": "03/19/2022 15:20:24", "content": "Thanks Manuel! Yeah i used roughly 10000000 timestamp periods. I found using larger validation periods didn't make too much difference to training but i preferred to keep a smaller validation set - so I could keep my test sets large.
", "rawMarkdown": "Thanks Manuel! Yeah i used roughly 10000000 timestamp periods. I found using larger validation periods didn't make too much difference to training but i preferred to keep a smaller validation set - so I could keep my test sets large.", "votes": null }, { "id": "1740790", "postDate": "03/31/2022 09:23:41", "content": "\n\nIt goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts
\n
This aged well - congrats!
", "rawMarkdown": "> It goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts\n\nThis aged well - congrats!", "votes": null }, { "id": "1813866", "postDate": "06/07/2022 09:56:01", "content": "Nice work. Congrats.
", "rawMarkdown": "Nice work. Congrats.", "votes": null }, { "id": "1855991", "postDate": "07/15/2022 04:04:52", "content": "Thanks for your contribution
", "rawMarkdown": "Thanks for your contribution", "votes": null } ], "comments": [ { "id": 1725134, "author_name": "marketneutral", "author_url": "", "post_date": "03/16/2022 20:51:03", "content": "Great write up. Thank you for sharing. I am surprised you were able to fit all the feature engineering and ensembles inside the predict loop… I kept running into time limits when trying similar. As for me, after three runs…56 -> 84 -> 164 … 😬
", "votes": null, "replies": [] }, { "id": 1725138, "author_name": "julianmukaj", "author_url": "", "post_date": "03/16/2022 20:55:40", "content": "Can you post a link to where you got data from those exchanges in the same format as competition?
\nI think you are jumping the gun a bit on LB placings, but nevertheless I think you have an interesting solution, I did not consider the target engineering for mine I think that was a good idea for sure.
\nWell done on the 6 week performance and good luck for the rest! 🙂
", "votes": null, "replies": [ { "id": 1725174, "author_name": "tomforbes", "author_url": "", "post_date": "03/16/2022 21:54:43", "content": "I was only interested in OHLC data so didn't need to get the more hard to source features like VWAP. This is pretty easy to find if you google most exchanges have public api docs. E.g. binance
\nYeah as i said its 1st place 6 weeks in - I could slip a few of places by the end but thought i'd share what i'd written up
", "votes": null, "replies": [] } ] }, { "id": 1725140, "author_name": "petersk20", "author_url": "", "post_date": "03/16/2022 20:56:08", "content": "Great job Tom. I just has a couple questions about the time period of the training set you used? What time period did you use to train your final models on? Did it include the supplemental training set. Did it vary across the models in your ensemble?
", "votes": null, "replies": [ { "id": 1725170, "author_name": "tomforbes", "author_url": "", "post_date": "03/16/2022 21:46:49", "content": "Thanks Kyle! I have outlined some of the training and validation folds i used in this post. For example my best model was a NN trained on data from 1514764860 - 1570000000 (timestamp). I choose whichever folds performed best overall on my testing folds. I didn't use any supplemental_train data for training, only testing. I didn't train any models that included data after ~1610000000
", "votes": null, "replies": [] }, { "id": 1725178, "author_name": "petersk20", "author_url": "", "post_date": "03/16/2022 21:57:00", "content": "That makes sense. I saw that but didn't know if that was just used for cv or if it was also used in the final model.
", "votes": null, "replies": [] } ] }, { "id": 1725143, "author_name": "petersk20", "author_url": "", "post_date": "03/16/2022 21:03:54", "content": "Also, maybe the reason there are discrepancies in other people's models and not your models is because of a heightened amount of NA's in the data. This would cause more betas to be zero which would throw off other people's models but not yours due to the feature extraction. I noticed an uptick in NA's from the public lb to supplemental dataset but only a slight one.
", "votes": null, "replies": [ { "id": 1725172, "author_name": "tomforbes", "author_url": "", "post_date": "03/16/2022 21:50:07", "content": "Maybe, but then I imagine most people trained their models on the target as originally constructed which had plenty of beta == zero, so whatever results they got in testing should probably still apply unless there is an unusual amount of missing data. It seemed to be the trend that the most recent data had the least missing data
", "votes": null, "replies": [] } ] }, { "id": 1725151, "author_name": "kirderf", "author_url": "", "post_date": "03/16/2022 21:10:15", "content": "Good work! Same here, majority of the time feature engineering, very little modeling in contrast to other competitions. Will be interesting weeks ahead following the progress :)
", "votes": null, "replies": [] }, { "id": 1725165, "author_name": "miguelpm", "author_url": "", "post_date": "03/16/2022 21:36:00", "content": "Looks great @tomforbes , thanks for sharing that. I find very interesting that you have managed to make your approach robust to data quality which was a huge condition for label generation.
\nI have given already clearly enough my opinion that the statistical nature of the labels at test time is very different to the previous one, and not because of market forces. And I still consider that huge label quality shift at test time isn't legit for a Kaggle comp, but that doesn't take away the merit that your solution is cleverly prepared to deal with it, so congratulations for that.
\nAs for myself, I have spent way too much energy in this competition, especially addressing the \"elephant in the room\" issue but I am goint to spend no more time on it. Every competition teaches you something and this one has not been an exception, hopefully I will choose a better way to invest my time next time.
\nAgain, thanks for sharing and good luck!
", "votes": null, "replies": [] }, { "id": 1725180, "author_name": "lucasmorin", "author_url": "", "post_date": "03/16/2022 22:02:20", "content": "Nice solution :-) Congrats on pulling the hard data sourcing / target engineering work. So in the end you only use very few features ? Did you keep the noise from my nb ? (btw thanks for citing my work - hope to get gold)
", "votes": null, "replies": [ { "id": 1726151, "author_name": "tomforbes", "author_url": "", "post_date": "03/17/2022 17:29:49", "content": "Thanks! Yeah only 17 features for all models. Yes for some models i kept the noise and some seemed to train better without. It certainly deserves a gold!
", "votes": null, "replies": [] } ] }, { "id": 1725535, "author_name": "jonnydossantos", "author_url": "", "post_date": "03/17/2022 07:21:06", "content": "Great work Tom Forbes!! Looks like you might just be deserving of first place after all :).
", "votes": null, "replies": [] }, { "id": 1725751, "author_name": "casras", "author_url": "", "post_date": "03/17/2022 12:06:44", "content": "To take into account the beta=0 cases I just added the approximated beta calculated with the existing history as a feature to the model and hoped the model will learn from this feature. In this way I did not have to split the models for the two cases. Would have been interesting actually to check if separate models improve.
", "votes": null, "replies": [ { "id": 1726153, "author_name": "tomforbes", "author_url": "", "post_date": "03/17/2022 17:31:46", "content": "Yeah i tried using beta as a feature but i didn't get a lot out of it. Think the benefit of splitting out the models is the target is consistent and allows for better training
", "votes": null, "replies": [] } ] }, { "id": 1725977, "author_name": "jagofc", "author_url": "", "post_date": "03/17/2022 14:57:34", "content": "Respect. The target switching approach is very neat!
", "votes": null, "replies": [] }, { "id": 1726205, "author_name": "justinkrieger001", "author_url": "", "post_date": "03/17/2022 18:45:27", "content": "Great writeup!
", "votes": null, "replies": [] }, { "id": 1728336, "author_name": "ishanmehta115", "author_url": "", "post_date": "03/18/2022 19:40:44", "content": "This is great. Thanks for sharing
", "votes": null, "replies": [] }, { "id": 1728396, "author_name": "coreacasa", "author_url": "", "post_date": "03/18/2022 21:09:49", "content": "Great Job @tomforbes, I hope you have good luck in the rest of the competition. I have a question, what was the longest period of time that your models worked on the validation? …10000000 from timestamps?
", "votes": null, "replies": [ { "id": 1729052, "author_name": "tomforbes", "author_url": "", "post_date": "03/19/2022 15:20:24", "content": "Thanks Manuel! Yeah i used roughly 10000000 timestamp periods. I found using larger validation periods didn't make too much difference to training but i preferred to keep a smaller validation set - so I could keep my test sets large.
", "votes": null, "replies": [] } ] }, { "id": 1729046, "author_name": "indureghu", "author_url": "", "post_date": "03/19/2022 15:16:40", "content": "Great work. Congrats.
", "votes": null, "replies": [] }, { "id": 1740790, "author_name": "marcogorelli", "author_url": "", "post_date": "03/31/2022 09:23:41", "content": "\n\nIt goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts
\n
This aged well - congrats!
", "votes": null, "replies": [] }, { "id": 1813866, "author_name": "liletta", "author_url": "", "post_date": "06/07/2022 09:56:01", "content": "Nice work. Congrats.
", "votes": null, "replies": [] }, { "id": 1855991, "author_name": "gectrade", "author_url": "", "post_date": "07/15/2022 04:04:52", "content": "Thanks for your contribution
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1725106": "[View the final submission notebook here.](https://www.kaggle.com/tomforbes/1st-place-6-weeks-in-final-submission)\n\nFinal Update - Congratulations to all the winners! While this comp was a bit of a rollercoaster, I'm very pleased to get my first competition Gold. Disappointing to just miss out on the top 10 after being there so long, but thats always possible with this type of challenge.\n\nOriginal post below.\n\n### Initial Comments\n\nFirst of all thankyou to G-Research and Cambridge Spark for hosting this competition and to the Kaggle staff for making everything run relatively smoothly. I found it so interesting and more open ended than other comps I've competed in, in my short Kaggler career.\n\nIt goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts. Hopefully you find this solution useful regardless of the final outcome.\n\nElephant in the room - there has been a lot of concern about the validity of the final LB results and requests to release the final test data after the competition ends so that competitors can validate their final scores. I think this would be a good idea and should put any concerns to bed. If it's possible & not too much effort for Kaggle to accommodate the request please consider it. However, as far as im concerned my current score seems reasonable based on all the scores I achieved in testing. I have no reason to believe any of the LB scores are incorrect until there is evidence to the contrary.\n\n**I would encourage those who feel their score is unexpected share their solutions as the Kaggle community might be able to debug any issues/misunderstandings.**\n\nIt is obviously difficult to accurately predict crypto prices, but it's possible to find *some* signal - hopefully this notebook will shed some light on how I got mine.\n\n*Disclaimers*\n- *this was not my exact final submission but very similar*\n- *I have removed the dataset containing my pretrained models and scalers - so some commands will error*\n\n### Summary\n\n- 17 features with lagged and timestamp averages for all models.\n- Ensembles of LGBM and Keras NN models.\n- Target Engineering and prediction switching.\n- Some training on extra cryptos outside competition scope.\n\nTest Scores (no lookahead bias):\n- 0.0565 best score on original testing period (Jun 13 2021 - Sep 21 2021)\n- 0.0475 best score on supplemental_train.csv update. (Sep 21 2021 - Jan 24 2022)\n- 0.0465 best single model score (Jun 13 2021 - Sep 21 2021)\n\n### External Data\nI downloaded free public data from multiple exchange apis (Binance, FTX, Coinbase, Kucoin... etc.) to see if this extra data would improve my models. Some of my final models were trained on extra data. Specifically, using some currencies not included in the competition (e.g. XRP, ZEC...) from Binance seemed to provide a consistent small improvement over multiple timeframes.\n\n### Feature Engineering\nI settled on a group of 17 features for all models. There are 8 lagged features - **a simple mixture of EMA's, historical returns and historical volatility over various lookback periods** - and these features were averaged across timestamps to produce 8 more. Asset_ID was also included. It was tricky to find a group that performed the best consistently across different time periods, adding and removing features from this set seemed to reduce performance. Im sure this isn't the optimal feature set, but this group seemed to work well enough.\n\nIt was also important to perform some kind on binning on the features, especially for training the LGBM model. The commonly used reduce_mem_usage function and some rounding functions seemed to provide a suitable amount of bins. I found binning to 500-1000 unique values worked well for any given continuous feature.\n\n### Target Engineering\nI think a crucial part of this competition was manipulating the target. Thanks to the work of **@alexfir** and other Kagglers [we found out how the target was being calculated](https://www.kaggle.com/alexfir/recreating-target).\nI thought it would be useful to split the target into two components:\n- The forward 15 minute return of an asset\n- The beta component, where we calculate the mean of forward 15 minute returns for all assets and incorporate past 15 minute returns for the previous 3750 timestamps\n\nAs **@gengdaiziwang** illustrates [in his helpful notebook](https://www.kaggle.com/gengdaiziwang/are-we-training-models-on-an-inaccurate-target), we can see that in the case where an observation is missing for a given asset, the beta component is automatically set to 0.\n\n[](https://postimg.cc/Dm1B6rXM)\n\nBut this makes it a completely different target! A model trained on the target given as standard will have to deal with training on a target that randomly switches between a 15 minute forward return for one asset vs what is essentially a 15 minute forward return *relative to the other assets* - very different things.\n\nThe approach I took was to create recreate two targets to represent these two different cases. Then I could train separate models on these recreated targets which would learn the two cases much more effectively.\n[See an implementation of this target engineering](https://www.kaggle.com/tomforbes/target-engineering).\n\nNow, having two models trained on different targets, I can alter my predictions to match the target produced by the api. The api provides all the information i need to figure out whether the beta component will be == 0 because if an asset is missing from an iteration i know that for the next 3750 iterations the beta component == 0 for this asset. **Therefore my next 3750 predictions will come from a model that was training on a target constructued without a beta component.**\n\nThis target engineering and dual model method added roughly 0.01 to my score on the original test period, although this did vary quite significantly for other testing periods.\n\nTo summarise as simply as possible:\n\n- TargetZero = A target based on 15 minute Forward Return only\n- TargetBeta = TargetZero + Beta Component\n- ModelZero = Model(s) trained on TargetZero\n- ModelBeta = Model(s) trained on TargetBeta\n\n[](https://postimg.cc/xJJFnBqf)\n\n### Models\n\nI found LightGBM worked well and was easy to experiment with, i used the weighted correlation evaluation metric and fairly out of the box hyperparameters, tuning didnt seem to add much and wasn't consistent across time periods.\n\nI also used a Keras NN heavily influenced by **@lucasmorin's**\n[excellent notebook](http://https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap). This just worked really well for me out of the box and i couldnt find many ways to improve the architecture or hyperparameters.\n\nEnsembling these two models worked well and added an extra 0.005 to my score (roughly) although the improvement varies in different market regimes. I also experimented with ensembles using catboost and XGboost but the predictions from most gradient boosting models were too highly correlated to provide much ensembling benefit. I found the diversity in predictions between NN and GB were a good match.\n\nI used a fairly simple CV method for training and testing. My folds were based on timestamp values similar to the below:\n- Training Fold 0: 1514764860 - 1570000000 \n- Training Fold 1: 1514764860 - 1580000000 \n- Training Fold 2: 1514764860 - 1590000000 \n- Training Fold 3: 1514764860 - 1600000000 \n- Training Fold 4: 1514764860 - 1610000000\n- Validation Fold 0: 1570000000 - 1580000000\n- Validation Fold 1: 1580000000 - 1590000000 \n- Validation Fold 2: 1590000000 - 1600000000 \n- Validation Fold 3: 1600000000 - 1610000000 \n- Validation Fold 4: 1610000000 - 1620000000 \n- Test Fold A: 1615000000 - 1623542400\n- Test Fold B (Supplemental Train Update): 1623542400 - 1643000000\n\nSince this public lb was useless in this competition, it was essential to construct a robust CV framework to have a reliable benchmark for making improvements to models. It was also important to have test folds in several different market regimes to avoid overfitting to one period.\n\n**My best single model was a NN trained on fold 0 data only (surprisingly).**\n\nI was fairly selective on picking models to include in the final submission as some folds just trained much better than others and had more consistent outperformance on unseen data.\n\n### Submission\n\nThis is almost the exact notebook I submitted for my final predictions. I used lists to store historical data for feature calculations and did most calculations using numpy or lists, avoiding pandas at all costs.\n\nThe submission completes in roughly 7 hours.\n\n\n### What didnt work\n- Layering extra data from different exchanges for the same asset. E.g training on BTC data from Binance + Coinbase + FTX + G-Research version. Didnt harm but didnt improve models significantly either.\n- Framing as a classification problem - poor results\n- Hyperparameter tuning LGBM\n- PCA features\n- Using all 14 assets features per row / predicting on 14 targets.\n- Meta modelling GB and NN predictions\n\n### With more time id try\n- Extra Feature engineering\n- Pytorch implementation of keras NN for ensembling\n- LSTM\n- Other model architectures", "1725134": "Great write up. Thank you for sharing. I am surprised you were able to fit all the feature engineering and ensembles inside the predict loop... I kept running into time limits when trying similar. As for me, after three runs...56 -> 84 -> 164 ... 😬", "1725138": "Can you post a link to where you got data from those exchanges in the same format as competition? \n\nI think you are jumping the gun a bit on LB placings, but nevertheless I think you have an interesting solution, I did not consider the target engineering for mine I think that was a good idea for sure.\n\nWell done on the 6 week performance and good luck for the rest! 🙂", "1725140": "Great job Tom. I just has a couple questions about the time period of the training set you used? What time period did you use to train your final models on? Did it include the supplemental training set. Did it vary across the models in your ensemble?", "1725143": "Also, maybe the reason there are discrepancies in other people's models and not your models is because of a heightened amount of NA's in the data. This would cause more betas to be zero which would throw off other people's models but not yours due to the feature extraction. I noticed an uptick in NA's from the public lb to supplemental dataset but only a slight one.", "1725151": "Good work! Same here, majority of the time feature engineering, very little modeling in contrast to other competitions. Will be interesting weeks ahead following the progress :)", "1725165": "Looks great @tomforbes , thanks for sharing that. I find very interesting that you have managed to make your approach robust to data quality which was a huge condition for label generation. \n\nI have given already clearly enough my opinion that the statistical nature of the labels at test time is very different to the previous one, and not because of market forces. And I still consider that huge label quality shift at test time isn't legit for a Kaggle comp, but that doesn't take away the merit that your solution is cleverly prepared to deal with it, so congratulations for that. \n\nAs for myself, I have spent way too much energy in this competition, especially addressing the \"elephant in the room\" issue but I am goint to spend no more time on it. Every competition teaches you something and this one has not been an exception, hopefully I will choose a better way to invest my time next time. \n\nAgain, thanks for sharing and good luck!", "1725170": "Thanks Kyle! I have outlined some of the training and validation folds i used in this post. For example my best model was a NN trained on data from 1514764860 - 1570000000 (timestamp). I choose whichever folds performed best overall on my testing folds. I didn't use any supplemental_train data for training, only testing. I didn't train any models that included data after ~1610000000", "1725172": "Maybe, but then I imagine most people trained their models on the target as originally constructed which had plenty of beta == zero, so whatever results they got in testing should probably still apply unless there is an unusual amount of missing data. It seemed to be the trend that the most recent data had the least missing data", "1725174": "I was only interested in OHLC data so didn't need to get the more hard to source features like VWAP. This is pretty easy to find if you google most exchanges have public api docs. E.g. [binance](https://www.binance.com/en/landing/data)\n\nYeah as i said its 1st place 6 weeks in - I could slip a few of places by the end but thought i'd share what i'd written up", "1725178": "That makes sense. I saw that but didn't know if that was just used for cv or if it was also used in the final model.", "1725180": "Nice solution :-) Congrats on pulling the hard data sourcing / target engineering work. So in the end you only use very few features ? Did you keep the noise from my nb ? (btw thanks for citing my work - hope to get gold)", "1725535": "Great work Tom Forbes!! Looks like you might just be deserving of first place after all :).", "1725751": "To take into account the beta=0 cases I just added the approximated beta calculated with the existing history as a feature to the model and hoped the model will learn from this feature. In this way I did not have to split the models for the two cases. Would have been interesting actually to check if separate models improve.", "1725977": "Respect. The target switching approach is very neat!", "1726151": "Thanks! Yeah only 17 features for all models. Yes for some models i kept the noise and some seemed to train better without. It certainly deserves a gold!", "1726153": "Yeah i tried using beta as a feature but i didn't get a lot out of it. Think the benefit of splitting out the models is the target is consistent and allows for better training", "1726205": "Great writeup!", "1728336": "This is great. Thanks for sharing", "1728396": "Great Job @tomforbes, I hope you have good luck in the rest of the competition. I have a question, what was the longest period of time that your models worked on the validation? ...10000000 from timestamps?", "1729046": "Great work. Congrats.", "1729052": "Thanks Manuel! Yeah i used roughly 10000000 timestamp periods. I found using larger validation periods didn't make too much difference to training but i preferred to keep a smaller validation set - so I could keep my test sets large.", "1740790": "> It goes without saying that while I'm in 1st Place now there is no guarantee it will stay that way in the next 6 weeks, but I'm confident I'll be there or thereabouts\n\nThis aged well - congrats!", "1813866": "Nice work. Congrats.", "1855991": "Thanks for your contribution" }, "source": "meta" }