![\"Gresearch](\"https://i.imgur.com/nBmonr4.png\")
Hello everyone. As this competition is rather tough I tought I might share a bit of work that could be helpful to others.
\nFeature Engineering, Feature Importance, CV design and objective functions :
\n
Embargo are designed to avoid leakage and for building 1 month rolling features without problems. Another approach to reduce datasets size would be undersampling. I have not set my mind if and how to do it. Inputs more than welcome on the topics of CV design and undersampling. The usefullness of those features are validated by the next notebook.
\nResults from the lgbm eval function:
\n![\"lgbm](\"https://i.imgur.com/E2tadRf.png\")
Interestingly there seems to be different regimes.
\nTop 20 results from the lgbm importance:
\n![\"lgbm](\"https://i.imgur.com/5v2iXDY.png\")
Asset_ids, log returns, VWAP, volatilities, we can better see the main risk factors.
\nHope you'll appreciate those two notebooks.
", "messages": [ { "id": "1586237", "postDate": "11/17/2021 23:11:56", "content": "Hello everyone. As this competition is rather tough I tought I might share a bit of work that could be helpful to others.
\nFeature Engineering, Feature Importance, CV design and objective functions :
\n
Embargo are designed to avoid leakage and for building 1 month rolling features without problems. Another approach to reduce datasets size would be undersampling. I have not set my mind if and how to do it. Inputs more than welcome on the topics of CV design and undersampling. The usefullness of those features are validated by the next notebook.
\nResults from the lgbm eval function:
\n
Interestingly there seems to be different regimes.
\nTop 20 results from the lgbm importance:
\n
Asset_ids, log returns, VWAP, volatilities, we can better see the main risk factors.
\nHope you'll appreciate those two notebooks.
", "rawMarkdown": "Hello everyone. As this competition is rather tough I tought I might share a bit of work that could be helpful to others. \n\n**Feature Engineering, Feature Importance, CV design and objective functions** :\n\n- [**Feature Engineering, Cv Design**](https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors): Before going too much on complex feature engineering I tought we could use some basic feature exploration. This notebook does that by building around 200 features. I aimed to get common risk factors (see for exemple: [common risk factors in cryptocurrency](https://www.nber.org/system/files/working_papers/w25882/w25882.pdf)) and some basic ML features (aggregated over market, beta, lagged features, time encoding ...). Some parts of the notebook are inspired from other competitions (notably @slawekbiel work in numpy Feature Enginerring in Optiver). The notebook is a bit convoluted to avoid out of memory errors (saving intermediate work as parquet files then aggregating them). I pushed the notebook up to CV engineering so as to manipulate smaller datasets. I've retained the following CV design: \n\n\n\nEmbargo are designed to avoid leakage and for building 1 month rolling features without problems. Another approach to reduce datasets size would be undersampling. I have not set my mind if and how to do it. Inputs more than welcome on the topics of CV design and undersampling. The usefullness of those features are validated by the next notebook.\n\n- [**Feature Importance, lgbm evaluation function**](https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance): a basic lgbm notebooks using the previous Feature Engineering notebook to help priroritize feature engineering work. It rely on the initial intro lgbm notebook from @code1110. It mainly introduce three things that I haven't seen available here: (1) Significant Feature Engineering Work (see notebook above) (2) an feval function that is compatitble with the lgbm framework. This is not an objective function that can be optimised but it allows to follow our target metric over rounds and can be used for early stopping. And (3) a beautifull boxplot to compare feature importance accross folds. It come from currently winning Optiver solution (https://www.kaggle.com/nyanpn/public-2nd-place-solution from @nyanpn). It should be helpfull to weed out unusefull features before getting into heavier feature engineering frameworks.\n\n**Results from the lgbm eval function**:\n\n\n\nInterestingly there seems to be different regimes.\n\n**Top 20 results from the lgbm importance**:\n\n\n\nAsset_ids, log returns, VWAP, volatilities, we can better see the main risk factors. \n\nHope you'll appreciate those two notebooks.", "votes": null }, { "id": "1586872", "postDate": "11/18/2021 09:58:17", "content": "Thank you so much for these notebooks! I'm currently using PurgedGroupTimeSeries CV from YAM PELEG's notebook here: https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm. Are there reasons to prefer your CV method?
", "rawMarkdown": "Thank you so much for these notebooks! I'm currently using PurgedGroupTimeSeries CV from YAM PELEG's notebook here: https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm. Are there reasons to prefer your CV method?", "votes": null }, { "id": "1586910", "postDate": "11/18/2021 10:46:03", "content": "I wanted to avoid overlaps, get gaps, mostly to understand what is happening (if things change over time… etc) and a simple implementation. I might change the cv design depending on what I find and the implementation if I have to put it in a tuner. Regarding yam work, I don't know, I have a hard time following everything he does, what is usefull, how it works (and what is his).
", "rawMarkdown": "I wanted to avoid overlaps, get gaps, mostly to understand what is happening (if things change over time... etc) and a simple implementation. I might change the cv design depending on what I find and the implementation if I have to put it in a tuner. Regarding yam work, I don't know, I have a hard time following everything he does, what is usefull, how it works (and what is his).", "votes": null }, { "id": "1589412", "postDate": "11/20/2021 08:46:07", "content": "Noted about the code complexity, I will try and reduce some of the overhead in the notebooks and in general remove as much \"fat\" as possible. The main reason there is so much excessive code is that I try to make many cells as \"self-sufficient\" as possible so people can just copy-paste them if they want.
About the subtle criticism about \"what is mine\": I try to give credit to anyone I use code from on every notebook I share [to an extent that some of the notebooks are just filled \"credit\" that they become annoying to read in my opinion..]. however, since I shared a lot in this competition, some might have slipped through the cracks [I had once 10 tabs open at the same time with notebooks for this competition..]. If you find such case please do call me out, I will fix it. this is absolutely not my intent.
And nice lgbm evaluation function BTW!
Hey @lucasmorin, Here's a model-free approach to pruning non-informative and redundant features that you might find useful: https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii.
", "rawMarkdown": "Hey @lucasmorin, Here's a model-free approach to pruning non-informative and redundant features that you might find useful: https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii.", "votes": null }, { "id": "1593426", "postDate": "11/23/2021 22:30:12", "content": "Somehow I missed your answer. This is not exactly the sort of conversation I hoped to get here but I will answer you. Being halfway trough notebook GM I can appreciate the energy and work you put in this. You have managed to build a very impressive network of all sort of stuff. My criticism was a bit about credit (burying a very short acknowledgment of someone else work below flashy stuff doesn't seems entirely fair when the original works may have taken weeks and you just straight up copypaste it and make no work / argument as to why it will work here).
\nIt's also about quantity versus quality. It is fairly notorious that fancy architectures (and a big part of technical analysis) aren't really usefull in tackling financial problems. Going all in Fancy models like you do seems a bit counter productive. In my opinion it's exactly how one end up asking himself 'what is going on? and how come so many are overfitted?' .
\nAlso I am a bit concerned by the side effects. I enjoy kaggle, reading the diversity of opinions, the challenge to build better stuff, the contribution of domain experts. Now I can't help but wonder how many newcommers will fall in the fancy ML pitfall you set up and how many contributors aren't participating because of the space you manage to occupy on the code tab.
", "rawMarkdown": "Somehow I missed your answer. This is not exactly the sort of conversation I hoped to get here but I will answer you. Being halfway trough notebook GM I can appreciate the energy and work you put in this. You have managed to build a very impressive network of all sort of stuff. My criticism was a bit about credit (burying a very short acknowledgment of someone else work below flashy stuff doesn't seems entirely fair when the original works may have taken weeks and you just straight up copypaste it and make no work / argument as to why it will work here). \n\nIt's also about quantity versus quality. It is fairly notorious that fancy architectures (and a big part of technical analysis) aren't really usefull in tackling financial problems. Going all in Fancy models like you do seems a bit counter productive. In my opinion it's exactly how one end up asking himself[ 'what is going on? and how come so many are overfitted?' ](https://www.kaggle.com/c/optiver-realized-volatility-prediction/discussion/286220). \n\nAlso I am a bit concerned by the side effects. I enjoy kaggle, reading the diversity of opinions, the challenge to build better stuff, the contribution of domain experts. Now I can't help but wonder how many newcommers will fall in the fancy ML pitfall you set up and how many contributors aren't participating because of the space you manage to occupy on the code tab.", "votes": null }, { "id": "1593427", "postDate": "11/23/2021 22:30:58", "content": "Thanks for the link. Are you the autor of the package ?
", "rawMarkdown": "Thanks for the link. Are you the autor of the package ?", "votes": null }, { "id": "1593441", "postDate": "11/23/2021 23:01:42", "content": "Yes, I am! Also, the author of the linked references. Feel free to shoot me an email here if you have any question: ml@kxy.ai.
", "rawMarkdown": "Yes, I am! Also, the author of the linked references. Feel free to shoot me an email here if you have any question: ml@kxy.ai.", "votes": null } ], "comments": [ { "id": 1586872, "author_name": "mingyangyi", "author_url": "", "post_date": "11/18/2021 09:58:17", "content": "Thank you so much for these notebooks! I'm currently using PurgedGroupTimeSeries CV from YAM PELEG's notebook here: https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm. Are there reasons to prefer your CV method?
", "votes": null, "replies": [ { "id": 1586910, "author_name": "lucasmorin", "author_url": "", "post_date": "11/18/2021 10:46:03", "content": "I wanted to avoid overlaps, get gaps, mostly to understand what is happening (if things change over time… etc) and a simple implementation. I might change the cv design depending on what I find and the implementation if I have to put it in a tuner. Regarding yam work, I don't know, I have a hard time following everything he does, what is usefull, how it works (and what is his).
", "votes": null, "replies": [] }, { "id": 1589412, "author_name": "", "author_url": "", "post_date": "11/20/2021 08:46:07", "content": "Noted about the code complexity, I will try and reduce some of the overhead in the notebooks and in general remove as much \"fat\" as possible. The main reason there is so much excessive code is that I try to make many cells as \"self-sufficient\" as possible so people can just copy-paste them if they want.
About the subtle criticism about \"what is mine\": I try to give credit to anyone I use code from on every notebook I share [to an extent that some of the notebooks are just filled \"credit\" that they become annoying to read in my opinion..]. however, since I shared a lot in this competition, some might have slipped through the cracks [I had once 10 tabs open at the same time with notebooks for this competition..]. If you find such case please do call me out, I will fix it. this is absolutely not my intent.
And nice lgbm evaluation function BTW!
Somehow I missed your answer. This is not exactly the sort of conversation I hoped to get here but I will answer you. Being halfway trough notebook GM I can appreciate the energy and work you put in this. You have managed to build a very impressive network of all sort of stuff. My criticism was a bit about credit (burying a very short acknowledgment of someone else work below flashy stuff doesn't seems entirely fair when the original works may have taken weeks and you just straight up copypaste it and make no work / argument as to why it will work here).
\nIt's also about quantity versus quality. It is fairly notorious that fancy architectures (and a big part of technical analysis) aren't really usefull in tackling financial problems. Going all in Fancy models like you do seems a bit counter productive. In my opinion it's exactly how one end up asking himself 'what is going on? and how come so many are overfitted?' .
\nAlso I am a bit concerned by the side effects. I enjoy kaggle, reading the diversity of opinions, the challenge to build better stuff, the contribution of domain experts. Now I can't help but wonder how many newcommers will fall in the fancy ML pitfall you set up and how many contributors aren't participating because of the space you manage to occupy on the code tab.
", "votes": null, "replies": [] } ] }, { "id": 1590177, "author_name": "leanboosting", "author_url": "", "post_date": "11/21/2021 01:38:28", "content": "Hey @lucasmorin, Here's a model-free approach to pruning non-informative and redundant features that you might find useful: https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii.
", "votes": null, "replies": [ { "id": 1593427, "author_name": "lucasmorin", "author_url": "", "post_date": "11/23/2021 22:30:58", "content": "Thanks for the link. Are you the autor of the package ?
", "votes": null, "replies": [] }, { "id": 1593441, "author_name": "leanboosting", "author_url": "", "post_date": "11/23/2021 23:01:42", "content": "Yes, I am! Also, the author of the linked references. Feel free to shoot me an email here if you have any question: ml@kxy.ai.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1586237": "Hello everyone. As this competition is rather tough I tought I might share a bit of work that could be helpful to others. \n\n**Feature Engineering, Feature Importance, CV design and objective functions** :\n\n- [**Feature Engineering, Cv Design**](https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors): Before going too much on complex feature engineering I tought we could use some basic feature exploration. This notebook does that by building around 200 features. I aimed to get common risk factors (see for exemple: [common risk factors in cryptocurrency](https://www.nber.org/system/files/working_papers/w25882/w25882.pdf)) and some basic ML features (aggregated over market, beta, lagged features, time encoding ...). Some parts of the notebook are inspired from other competitions (notably @slawekbiel work in numpy Feature Enginerring in Optiver). The notebook is a bit convoluted to avoid out of memory errors (saving intermediate work as parquet files then aggregating them). I pushed the notebook up to CV engineering so as to manipulate smaller datasets. I've retained the following CV design: \n\n\n\nEmbargo are designed to avoid leakage and for building 1 month rolling features without problems. Another approach to reduce datasets size would be undersampling. I have not set my mind if and how to do it. Inputs more than welcome on the topics of CV design and undersampling. The usefullness of those features are validated by the next notebook.\n\n- [**Feature Importance, lgbm evaluation function**](https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance): a basic lgbm notebooks using the previous Feature Engineering notebook to help priroritize feature engineering work. It rely on the initial intro lgbm notebook from @code1110. It mainly introduce three things that I haven't seen available here: (1) Significant Feature Engineering Work (see notebook above) (2) an feval function that is compatitble with the lgbm framework. This is not an objective function that can be optimised but it allows to follow our target metric over rounds and can be used for early stopping. And (3) a beautifull boxplot to compare feature importance accross folds. It come from currently winning Optiver solution (https://www.kaggle.com/nyanpn/public-2nd-place-solution from @nyanpn). It should be helpfull to weed out unusefull features before getting into heavier feature engineering frameworks.\n\n**Results from the lgbm eval function**:\n\n\n\nInterestingly there seems to be different regimes.\n\n**Top 20 results from the lgbm importance**:\n\n\n\nAsset_ids, log returns, VWAP, volatilities, we can better see the main risk factors. \n\nHope you'll appreciate those two notebooks.", "1586872": "Thank you so much for these notebooks! I'm currently using PurgedGroupTimeSeries CV from YAM PELEG's notebook here: https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm. Are there reasons to prefer your CV method?", "1586910": "I wanted to avoid overlaps, get gaps, mostly to understand what is happening (if things change over time... etc) and a simple implementation. I might change the cv design depending on what I find and the implementation if I have to put it in a tuner. Regarding yam work, I don't know, I have a hard time following everything he does, what is usefull, how it works (and what is his).", "1589412": "Noted about the code complexity, I will try and reduce some of the overhead in the notebooks and in general remove as much \"fat\" as possible. The main reason there is so much excessive code is that I try to make many cells as \"self-sufficient\" as possible so people can just copy-paste them if they want.\n- About the subtle criticism about \"what is mine\": I try to give credit to anyone I use code from on every notebook I share [to an extent that some of the notebooks are just filled \"credit\" that they become annoying to read in my opinion..]. however, since I shared a lot in this competition, some might have slipped through the cracks [I had once 10 tabs open at the same time with notebooks for this competition..]. If you find such case please do call me out, I will fix it. this is absolutely not my intent. \n\n- And nice lgbm evaluation function BTW!", "1590177": "Hey @lucasmorin, Here's a model-free approach to pruning non-informative and redundant features that you might find useful: https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii.", "1593426": "Somehow I missed your answer. This is not exactly the sort of conversation I hoped to get here but I will answer you. Being halfway trough notebook GM I can appreciate the energy and work you put in this. You have managed to build a very impressive network of all sort of stuff. My criticism was a bit about credit (burying a very short acknowledgment of someone else work below flashy stuff doesn't seems entirely fair when the original works may have taken weeks and you just straight up copypaste it and make no work / argument as to why it will work here). \n\nIt's also about quantity versus quality. It is fairly notorious that fancy architectures (and a big part of technical analysis) aren't really usefull in tackling financial problems. Going all in Fancy models like you do seems a bit counter productive. In my opinion it's exactly how one end up asking himself[ 'what is going on? and how come so many are overfitted?' ](https://www.kaggle.com/c/optiver-realized-volatility-prediction/discussion/286220). \n\nAlso I am a bit concerned by the side effects. I enjoy kaggle, reading the diversity of opinions, the challenge to build better stuff, the contribution of domain experts. Now I can't help but wonder how many newcommers will fall in the fancy ML pitfall you set up and how many contributors aren't participating because of the space you manage to occupy on the code tab.", "1593427": "Thanks for the link. Are you the autor of the package ?", "1593441": "Yes, I am! Also, the author of the linked references. Feel free to shoot me an email here if you have any question: ml@kxy.ai." }, "source": "meta" }