{ "id": 293994, "title": "On-line Feature Engineering, Ensemble Calibration and Submission", "url": "/competitions/g-research-crypto-forecasting/discussion/293994", "author_name": "", "post_date": "2021-12-07T23:02:25.493752200Z", "votes": 39, "comment_count": 6, "views": 0, "content": "

Online Feature Engineering

\n

Using some of my previous work in the JaneStreet competition (running algos FE for fast inference) I managed to build an online version of my common factors - mostly pandas - study notebook. The idea are (1) use mostly numpy (2) use some reordering to work on a constant size array and (3) a class to keep lags in memory. The main trick is to use a 'running algorithm'. To perform a online sum (or a mean) over n instances you don't need to sum all of them. Just keep track of the cumulated sum and add the most recent value / remove the oldest one. I estimates it will only takes a bit more than 10 minutes of submission time to build 350 features, so that's pretty neat.

\n

On-line Feature Engineering Notebook

\n

There is room for improvement (better handling nans, cleaning the code / optimising lags / testing the coherence with the pandas notebooks / making the algo for the second moment work). But this should give some of you an idea of what to do for feature engineering.

\n

Ensemble Calibration and Submission

\n

In the notebook above I also kept the CV design and included a 0.1 sampling. It allowed me to build folds and calibrate some lgbm models in a notebook. Then ensemble them for submission with on-line features in another notebook. I use a previous lgbm notebook framework. More specifficaly: Online Feature Engineering + custom feval function + importance tools. The results vary widely across folds and seeds. So the bad news is it still seems difficult to calibrate models. The good news is that we get chunk of 14 predictions at a time, so we actually get way more time to predict than in the Jane Street competition. I demonstrate that in an ensemble submission notebook where I submit an ensemble of 25 lgbm models (5 folds x 5 seeds) on 350 features thanks to On-line Feature Engineering. Note that I save and reuse the running means from the first notebook so that I have the correct long-term means at the begining of the prediction phase.

\n

Ensemble Calibration Notebook
\nEnsemble Submission Notebook

\n

These two notebooks are also under construction (model calibration of course, but also adding supplemental train data to the submission notebook).

\n

It took me quite a lot of personnal time and work to build those notebooks, so If you find them usefull please take the time to upvote them.

", "messages": [ { "id": "1611371", "postDate": "12/07/2021 23:02:25", "content": "

Online Feature Engineering

\n

Using some of my previous work in the JaneStreet competition (running algos FE for fast inference) I managed to build an online version of my common factors - mostly pandas - study notebook. The idea are (1) use mostly numpy (2) use some reordering to work on a constant size array and (3) a class to keep lags in memory. The main trick is to use a 'running algorithm'. To perform a online sum (or a mean) over n instances you don't need to sum all of them. Just keep track of the cumulated sum and add the most recent value / remove the oldest one. I estimates it will only takes a bit more than 10 minutes of submission time to build 350 features, so that's pretty neat.

\n

On-line Feature Engineering Notebook

\n

There is room for improvement (better handling nans, cleaning the code / optimising lags / testing the coherence with the pandas notebooks / making the algo for the second moment work). But this should give some of you an idea of what to do for feature engineering.

\n

Ensemble Calibration and Submission

\n

In the notebook above I also kept the CV design and included a 0.1 sampling. It allowed me to build folds and calibrate some lgbm models in a notebook. Then ensemble them for submission with on-line features in another notebook. I use a previous lgbm notebook framework. More specifficaly: Online Feature Engineering + custom feval function + importance tools. The results vary widely across folds and seeds. So the bad news is it still seems difficult to calibrate models. The good news is that we get chunk of 14 predictions at a time, so we actually get way more time to predict than in the Jane Street competition. I demonstrate that in an ensemble submission notebook where I submit an ensemble of 25 lgbm models (5 folds x 5 seeds) on 350 features thanks to On-line Feature Engineering. Note that I save and reuse the running means from the first notebook so that I have the correct long-term means at the begining of the prediction phase.

\n

Ensemble Calibration Notebook
\nEnsemble Submission Notebook

\n

These two notebooks are also under construction (model calibration of course, but also adding supplemental train data to the submission notebook).

\n

It took me quite a lot of personnal time and work to build those notebooks, so If you find them usefull please take the time to upvote them.

", "rawMarkdown": "# Online Feature Engineering\n\nUsing some of my previous work in the JaneStreet competition ([running algos FE for fast inference](https://www.kaggle.com/lucasmorin/running-algos-fe-for-fast-inference)) I managed to build an online version of my [common factors](https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors) - mostly pandas - study notebook. The idea are (1) use mostly numpy (2) use some reordering to work on a constant size array and (3) a class to keep lags in memory. The main trick is to use a 'running algorithm'. To perform a online sum (or a mean) over n instances you don't need to sum all of them. Just keep track of the cumulated sum and add the most recent value / remove the oldest one. I estimates it will only takes a bit more than 10 minutes of submission time to build 350 features, so that's pretty neat.\n\n[**On-line Feature Engineering Notebook**](https://www.kaggle.com/lucasmorin/on-line-feature-engineering)\n\nThere is room for improvement (better handling nans, cleaning the code / optimising lags / testing the coherence with the pandas notebooks / making the algo for the second moment work). But this should give some of you an idea of what to do for feature engineering. \n\n# Ensemble Calibration and Submission\n\nIn the notebook above I also kept the CV design and included a 0.1 sampling. It allowed me to build folds and calibrate some lgbm models in a notebook. Then ensemble them for submission with on-line features in another notebook. I use a previous [lgbm notebook framework](https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance). More specifficaly: Online Feature Engineering + custom feval function + importance tools. The results vary widely across folds and seeds. So the bad news is it still seems difficult to calibrate models. The good news is that we get chunk of 14 predictions at a time, so we actually get way more time to predict than in the Jane Street competition. I demonstrate that in an ensemble submission notebook where I submit an ensemble of 25 lgbm models (5 folds x 5 seeds) on 350 features thanks to On-line Feature Engineering. Note that I save and reuse the running means from the first notebook so that I have the correct long-term means at the begining of the prediction phase. \n\n[**Ensemble Calibration Notebook**](https://www.kaggle.com/lucasmorin/online-fe-lgbm-feval-importances)\n[**Ensemble Submission Notebook**](https://www.kaggle.com/lucasmorin/ensemble-submission)\n\nThese two notebooks are also under construction (model calibration of course, but also adding supplemental train data to the submission notebook).\n\n**It took me quite a lot of personnal time and work to build those notebooks, so If you find them usefull please take the time to upvote them.**", "votes": null }, { "id": "1612061", "postDate": "12/08/2021 14:21:48", "content": "

Great job. I will refer to it as I was also having trouble with submission error.

", "rawMarkdown": "Great job. I will refer to it as I was also having trouble with submission error.", "votes": null }, { "id": "1613850", "postDate": "12/10/2021 10:27:55", "content": "

Good job, thanks!

", "rawMarkdown": "Good job, thanks!", "votes": null }, { "id": "1615522", "postDate": "12/12/2021 10:52:30", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "1615861", "postDate": "12/12/2021 19:57:20", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing", "votes": null }, { "id": "1616539", "postDate": "12/13/2021 13:38:29", "content": "

I really like it that you brought your Running Features notebook! It is a great one! to this day I am not sure how come you was the only one that published lag features on Jane Street..

", "rawMarkdown": "I really like it that you brought your Running Features notebook! It is a great one! to this day I am not sure how come you was the only one that published lag features on Jane Street..", "votes": null }, { "id": "1616804", "postDate": "12/13/2021 16:20:48", "content": "

There was no identifier in the JS competition, so no real way to track which stock was each sample. We managed to notice some features were only taking a few constant values across the day so we inferred that each constant value was a different stock. It allowed to match stock at the intraday level, but not over longer horizons. There was no way to build long term features. That explains it mostly. That and the idea that below anonymised features we probably got some relevant lagged features.

", "rawMarkdown": "There was no identifier in the JS competition, so no real way to track which stock was each sample. We managed to notice some features were only taking a few constant values across the day so we inferred that each constant value was a different stock. It allowed to match stock at the intraday level, but not over longer horizons. There was no way to build long term features. That explains it mostly. That and the idea that below anonymised features we probably got some relevant lagged features.", "votes": null } ], "comments": [ { "id": 1612061, "author_name": "matrixneo", "author_url": "", "post_date": "12/08/2021 14:21:48", "content": "

Great job. I will refer to it as I was also having trouble with submission error.

", "votes": null, "replies": [] }, { "id": 1613850, "author_name": "kibkim14", "author_url": "", "post_date": "12/10/2021 10:27:55", "content": "

Good job, thanks!

", "votes": null, "replies": [] }, { "id": 1615522, "author_name": "carol916", "author_url": "", "post_date": "12/12/2021 10:52:30", "content": "

Thanks for sharing!

", "votes": null, "replies": [] }, { "id": 1615861, "author_name": "flaviocavalcante", "author_url": "", "post_date": "12/12/2021 19:57:20", "content": "

Thanks for sharing

", "votes": null, "replies": [] }, { "id": 1616539, "author_name": "", "author_url": "", "post_date": "12/13/2021 13:38:29", "content": "

I really like it that you brought your Running Features notebook! It is a great one! to this day I am not sure how come you was the only one that published lag features on Jane Street..

", "votes": null, "replies": [ { "id": 1616804, "author_name": "lucasmorin", "author_url": "", "post_date": "12/13/2021 16:20:48", "content": "

There was no identifier in the JS competition, so no real way to track which stock was each sample. We managed to notice some features were only taking a few constant values across the day so we inferred that each constant value was a different stock. It allowed to match stock at the intraday level, but not over longer horizons. There was no way to build long term features. That explains it mostly. That and the idea that below anonymised features we probably got some relevant lagged features.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1611371": "# Online Feature Engineering\n\nUsing some of my previous work in the JaneStreet competition ([running algos FE for fast inference](https://www.kaggle.com/lucasmorin/running-algos-fe-for-fast-inference)) I managed to build an online version of my [common factors](https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors) - mostly pandas - study notebook. The idea are (1) use mostly numpy (2) use some reordering to work on a constant size array and (3) a class to keep lags in memory. The main trick is to use a 'running algorithm'. To perform a online sum (or a mean) over n instances you don't need to sum all of them. Just keep track of the cumulated sum and add the most recent value / remove the oldest one. I estimates it will only takes a bit more than 10 minutes of submission time to build 350 features, so that's pretty neat.\n\n[**On-line Feature Engineering Notebook**](https://www.kaggle.com/lucasmorin/on-line-feature-engineering)\n\nThere is room for improvement (better handling nans, cleaning the code / optimising lags / testing the coherence with the pandas notebooks / making the algo for the second moment work). But this should give some of you an idea of what to do for feature engineering. \n\n# Ensemble Calibration and Submission\n\nIn the notebook above I also kept the CV design and included a 0.1 sampling. It allowed me to build folds and calibrate some lgbm models in a notebook. Then ensemble them for submission with on-line features in another notebook. I use a previous [lgbm notebook framework](https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance). More specifficaly: Online Feature Engineering + custom feval function + importance tools. The results vary widely across folds and seeds. So the bad news is it still seems difficult to calibrate models. The good news is that we get chunk of 14 predictions at a time, so we actually get way more time to predict than in the Jane Street competition. I demonstrate that in an ensemble submission notebook where I submit an ensemble of 25 lgbm models (5 folds x 5 seeds) on 350 features thanks to On-line Feature Engineering. Note that I save and reuse the running means from the first notebook so that I have the correct long-term means at the begining of the prediction phase. \n\n[**Ensemble Calibration Notebook**](https://www.kaggle.com/lucasmorin/online-fe-lgbm-feval-importances)\n[**Ensemble Submission Notebook**](https://www.kaggle.com/lucasmorin/ensemble-submission)\n\nThese two notebooks are also under construction (model calibration of course, but also adding supplemental train data to the submission notebook).\n\n**It took me quite a lot of personnal time and work to build those notebooks, so If you find them usefull please take the time to upvote them.**", "1612061": "Great job. I will refer to it as I was also having trouble with submission error.", "1613850": "Good job, thanks!", "1615522": "Thanks for sharing!", "1615861": "Thanks for sharing", "1616539": "I really like it that you brought your Running Features notebook! It is a great one! to this day I am not sure how come you was the only one that published lag features on Jane Street..", "1616804": "There was no identifier in the JS competition, so no real way to track which stock was each sample. We managed to notice some features were only taking a few constant values across the day so we inferred that each constant value was a different stock. It allowed to match stock at the intraday level, but not over longer horizons. There was no way to build long term features. That explains it mostly. That and the idea that below anonymised features we probably got some relevant lagged features." }, "source": "meta" }