\n\nTL;DR: Code & Datasets | also see baselines below 👇
\n
Alright, Guys!
\nThe following tutorial details some of my initial guesses for some of the concepts that probably will surface throughout the competition. Probably some will even be incorporated into some of the final solutions.
\nHierarchical Time-series (HTS)
\nThis is a time-series competition.
\nmoreover - this can be modeled with hierarchical time series (HTS) model.
This basically means that you have a \"model per asset\" and \"one model for all\" maybe even \"model per weekday\"… and so on. by the end you ensemble and merge all the models.
\nThis method had proven to work over and over and over again in previous competitions.
\nMost notably the last M5 prediction. the winning solution was an extremely simple LGB running in an HTS fashion.. At the end of the day, It destroyed much more complex solutions with ease. by the way, there were nearly no fancy deep-learning models whatsoever at the top.
Note: It is important to note however that in both recent financial competitions there was no HTS solution at the top.
\nFuture Issues that future notebooks are going to address
\nMemory limits
\n1.0 It might look like the data is small and it won't be a problem. trust me, when doing HTS. it will become a problem. You will have to address this head-on.
\n1.1 You have to find a way to have more and more meaningful features (splits by \"type\"/date limits)
\n1.2 lags, aggs, nan cutoffs, etc.. all need to be taken into consideration.
Models
\n2.1. There is no \"winner\" here but if there was, I think that the winner of this competition will be an ensemble of NNs [Important will be FC, Tabnet, AutoEncoders..]. They are performing pretty well on the extrapolation of time series data.
\n2.2 The simple models will include LGB, XGB, Catboost, HistGBM. And strong solutions will have both.
\n2.3 Some novel ideas might include ARIMA, SARIMAX with some unique tricks for ensembling.
\n3.4 Ensembling and stacking, it works.
\n3.5 About deep learning: I don't think there will be an end2end deep learning solution (like pure LSTM or WaveNet) at the beginning. These are absolutely extremely hard to get right. however, from experience: when you do manage to get them to work (mostly wavenet), they are REALLY GOOD. However, we might see pretty early on some NBEATS or some AutoML NBEATS solution (like gluonTS). these are not as hard to train and they are pretty stable. (the 2nd solution on M5 used gluonTS with some cool tricks).
\n3.6 Neural networks that we will see for sure: simple FC networks trained on simple tabular feature engineering (lags, aggs.. etc..). They will also work pretty well on the LB.
Advanced Ideas we will probably see sometime
\n4.0 Fancy Loss function (the most important when training neural networks) If you want to do this, You have to create your own one that takes into account the specific issues of this dataset. It is hard and takes time.
\n4.1 Sample weights that \"stimulate\" the network to learn better
\n4.2 Some fancy incorporation of prior knowledge from the dataset into the network architecture.
About Features
\n5.1 If you want a stable solution that also might work well in real life and that will push your skills toward things that are common and you are going to see again and again: try to work on a simple single model (like LGB) and push its score above and above.
\nYou want to try playing around mostly with with lags (df[numeticsl_column] - df.groupby(time_column)[numetical_column].shift(-1)) and agg features (df.groupby(time_column)[numetical_column].agg(std)). these are baseline time-series features. they will work here for sure.
An overview of some M solutions - the top time-series competitions out there to date
\nInfo about M4
\nhttps://www.sciencedirect.com/science/article/pii/S0169207019301128
M4 winning solution was a hybrid of both exponential smoothing and a recurrent neural network. The model used was LSTM.
\nLink: https://www.sciencedirect.com/science/article/pii/S0169207019301153
Another interesting article and probably relevant for Kaggle. The article describes ensembles of statistical models: https://www.sciencedirect.com/science/article/pii/S0169207019301190#b5
Overall, the entire journal for the M4 competition can be found here:
\nhttps://www.sciencedirect.com/journal/international-journal-of-forecasting/vol/36/issue/1
Some other useful Resources
\nCoursera has a course about Sequences, Time series prediction. It is a basic course that provides a preliminary overview of the concept And building models to Univariate Time series. The course uses Keras and TensorFlow, but it provides a good starting point for this topic
\nLink: https://www.coursera.org/learn/tensorflow-sequences-time-series-and-prediction/home/welcome
LSTM networks - Great post/Blog that explains the concept beyond LSTM networks
\nLink: https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Post about building Time Series deep learning - The post covers the Keras/TensorFlow framework, but it also gives a great overview of the main concepts and how to prepare the data.
\n**Link: **https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/
Pytorch Basic example - This is a simple example of how to build a Pytorch LSTM network for simple Univariate Time Series.
\nLink: https://github.com/pytorch/examples/tree/master/time_sequence_prediction
Some misc links for time-series things you might want to take a look at
\n[credit: https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/133463]
\nThere have been quite a few time-series competitions hosted on Kaggle.
Here are a few of them:
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting
\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting
\nhttps://www.kaggle.com/c/rossmann-store-sales
Here are a few of the winning solutions from each of the competitions:
\nWeb Traffic Time Series
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/43795
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39395
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39370
Recruit Restaurant Visitor Forecasting
\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/49129
\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/49166
Favorita Grocery Sales Forecasting
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47582
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47556
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47667
Rossman Store Sales
\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/18024
\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/17974
\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/17896
I am currently in the process of implementing the ideas in this post.
\nThis is a work in progress. The following table updates with the current status of the project.
| CV + Model | \nHyperparam Optimization | \nTime Series Models | \nFeature Engineering | \n
|---|---|---|---|
| Neural Network Starter | \nMLP + AE | \nLSTM | \nTechnical Analysis #1 | \n
| LightGBM Starter | \nLightGBM | \nWavenet | \nTechnical Analysis #2 | \n
| Catboost Starter | \nCatboost | \nMultivariate-Transformer [written from scratch] | \nTime Series Agg | \n
| XGBoost Starter | \nXGboost | \nN-BEATS | \nNeutralization | \n
| Supervised AE [Janestreet 1st] | \nSupervised AE [Janestreet 1st] | \nDeepAR | \nQuant's Volatility Features | \n
| Transformer) | \nTransformer | \n\n | ⏳Target Engineering | \n
| TabNet Starter | \n\n | \n | ⏳Fourier Analysis | \n
| Reinforcement Learning (PPO) Starter | \n\n | \n | ⏳Wavelets | \n
About the validation: GroupTimeSeriesSplit
\n(⏳ - in the making..)
\nHappy Kaggling!
\nCheers ^^
\n\nTL;DR: Code & Datasets | also see baselines below 👇
\n
Alright, Guys!
\nThe following tutorial details some of my initial guesses for some of the concepts that probably will surface throughout the competition. Probably some will even be incorporated into some of the final solutions.
\nHierarchical Time-series (HTS)
\nThis is a time-series competition.
\nmoreover - this can be modeled with hierarchical time series (HTS) model.
This basically means that you have a \"model per asset\" and \"one model for all\" maybe even \"model per weekday\"… and so on. by the end you ensemble and merge all the models.
\nThis method had proven to work over and over and over again in previous competitions.
\nMost notably the last M5 prediction. the winning solution was an extremely simple LGB running in an HTS fashion.. At the end of the day, It destroyed much more complex solutions with ease. by the way, there were nearly no fancy deep-learning models whatsoever at the top.
Note: It is important to note however that in both recent financial competitions there was no HTS solution at the top.
\nFuture Issues that future notebooks are going to address
\nMemory limits
\n1.0 It might look like the data is small and it won't be a problem. trust me, when doing HTS. it will become a problem. You will have to address this head-on.
\n1.1 You have to find a way to have more and more meaningful features (splits by \"type\"/date limits)
\n1.2 lags, aggs, nan cutoffs, etc.. all need to be taken into consideration.
Models
\n2.1. There is no \"winner\" here but if there was, I think that the winner of this competition will be an ensemble of NNs [Important will be FC, Tabnet, AutoEncoders..]. They are performing pretty well on the extrapolation of time series data.
\n2.2 The simple models will include LGB, XGB, Catboost, HistGBM. And strong solutions will have both.
\n2.3 Some novel ideas might include ARIMA, SARIMAX with some unique tricks for ensembling.
\n3.4 Ensembling and stacking, it works.
\n3.5 About deep learning: I don't think there will be an end2end deep learning solution (like pure LSTM or WaveNet) at the beginning. These are absolutely extremely hard to get right. however, from experience: when you do manage to get them to work (mostly wavenet), they are REALLY GOOD. However, we might see pretty early on some NBEATS or some AutoML NBEATS solution (like gluonTS). these are not as hard to train and they are pretty stable. (the 2nd solution on M5 used gluonTS with some cool tricks).
\n3.6 Neural networks that we will see for sure: simple FC networks trained on simple tabular feature engineering (lags, aggs.. etc..). They will also work pretty well on the LB.
Advanced Ideas we will probably see sometime
\n4.0 Fancy Loss function (the most important when training neural networks) If you want to do this, You have to create your own one that takes into account the specific issues of this dataset. It is hard and takes time.
\n4.1 Sample weights that \"stimulate\" the network to learn better
\n4.2 Some fancy incorporation of prior knowledge from the dataset into the network architecture.
About Features
\n5.1 If you want a stable solution that also might work well in real life and that will push your skills toward things that are common and you are going to see again and again: try to work on a simple single model (like LGB) and push its score above and above.
\nYou want to try playing around mostly with with lags (df[numeticsl_column] - df.groupby(time_column)[numetical_column].shift(-1)) and agg features (df.groupby(time_column)[numetical_column].agg(std)). these are baseline time-series features. they will work here for sure.
An overview of some M solutions - the top time-series competitions out there to date
\nInfo about M4
\nhttps://www.sciencedirect.com/science/article/pii/S0169207019301128
M4 winning solution was a hybrid of both exponential smoothing and a recurrent neural network. The model used was LSTM.
\nLink: https://www.sciencedirect.com/science/article/pii/S0169207019301153
Another interesting article and probably relevant for Kaggle. The article describes ensembles of statistical models: https://www.sciencedirect.com/science/article/pii/S0169207019301190#b5
Overall, the entire journal for the M4 competition can be found here:
\nhttps://www.sciencedirect.com/journal/international-journal-of-forecasting/vol/36/issue/1
Some other useful Resources
\nCoursera has a course about Sequences, Time series prediction. It is a basic course that provides a preliminary overview of the concept And building models to Univariate Time series. The course uses Keras and TensorFlow, but it provides a good starting point for this topic
\nLink: https://www.coursera.org/learn/tensorflow-sequences-time-series-and-prediction/home/welcome
LSTM networks - Great post/Blog that explains the concept beyond LSTM networks
\nLink: https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Post about building Time Series deep learning - The post covers the Keras/TensorFlow framework, but it also gives a great overview of the main concepts and how to prepare the data.
\n**Link: **https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/
Pytorch Basic example - This is a simple example of how to build a Pytorch LSTM network for simple Univariate Time Series.
\nLink: https://github.com/pytorch/examples/tree/master/time_sequence_prediction
Some misc links for time-series things you might want to take a look at
\n[credit: https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/133463]
\nThere have been quite a few time-series competitions hosted on Kaggle.
Here are a few of them:
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting
\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting
\nhttps://www.kaggle.com/c/rossmann-store-sales
Here are a few of the winning solutions from each of the competitions:
\nWeb Traffic Time Series
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/43795
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39395
\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39370
Recruit Restaurant Visitor Forecasting
\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/49129
\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/49166
Favorita Grocery Sales Forecasting
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47582
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47556
\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47667
Rossman Store Sales
\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/18024
\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/17974
\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/17896
I am currently in the process of implementing the ideas in this post.
\nThis is a work in progress. The following table updates with the current status of the project.
| CV + Model | \nHyperparam Optimization | \nTime Series Models | \nFeature Engineering | \n
|---|---|---|---|
| Neural Network Starter | \nMLP + AE | \nLSTM | \nTechnical Analysis #1 | \n
| LightGBM Starter | \nLightGBM | \nWavenet | \nTechnical Analysis #2 | \n
| Catboost Starter | \nCatboost | \nMultivariate-Transformer [written from scratch] | \nTime Series Agg | \n
| XGBoost Starter | \nXGboost | \nN-BEATS | \nNeutralization | \n
| Supervised AE [Janestreet 1st] | \nSupervised AE [Janestreet 1st] | \nDeepAR | \nQuant's Volatility Features | \n
| Transformer) | \nTransformer | \n\n | ⏳Target Engineering | \n
| TabNet Starter | \n\n | \n | ⏳Fourier Analysis | \n
| Reinforcement Learning (PPO) Starter | \n\n | \n | ⏳Wavelets | \n
About the validation: GroupTimeSeriesSplit
\n(⏳ - in the making..)
\nHappy Kaggling!
\nCheers ^^
Upvoted
\nM5 forecasting competition has a very complex metric for different level predictions(item-level, department-level, store-level, etc). I think this is why HTC model is applied there. Not sure if we can do this here.
", "rawMarkdown": "Upvoted\n\nM5 forecasting competition has a very complex metric for different level predictions(item-level, department-level, store-level, etc). I think this is why HTC model is applied there. Not sure if we can do this here.", "votes": null }, { "id": "1568767", "postDate": "11/03/2021 03:07:53", "content": "wow thank you so much - as someone just delving into time series analysis I wanted to read other people's kernels and see what people were doing and this post is incredibly helpful.
", "rawMarkdown": "wow thank you so much - as someone just delving into time series analysis I wanted to read other people's kernels and see what people were doing and this post is incredibly helpful.", "votes": null }, { "id": "1568854", "postDate": "11/03/2021 04:48:11", "content": "I think lag features will not be useful because each time you only predict one row, so you have no more information to take.
", "rawMarkdown": "I think lag features will not be useful because each time you only predict one row, so you have no more information to take.", "votes": null }, { "id": "1568867", "postDate": "11/03/2021 05:02:45", "content": "The competitions recently have become more timeseries oriented. Optiver, Google Brain Ventilator and now the crypto currencies predictions. What could be the reason?
", "rawMarkdown": "The competitions recently have become more timeseries oriented. Optiver, Google Brain Ventilator and now the crypto currencies predictions. What could be the reason?", "votes": null }, { "id": "1568922", "postDate": "11/03/2021 06:24:19", "content": "I don't think HTS models will work as good as global models either. There are lots of cross-asset information, bitcoin correlation, seasonality and etc. to use. I'm not sure HTS models can use all of those information.
", "rawMarkdown": "I don't think HTS models will work as good as global models either. There are lots of cross-asset information, bitcoin correlation, seasonality and etc. to use. I'm not sure HTS models can use all of those information.", "votes": null }, { "id": "1569931", "postDate": "11/03/2021 20:57:08", "content": "its fun! :)
", "rawMarkdown": "its fun! :)", "votes": null }, { "id": "1569961", "postDate": "11/03/2021 21:21:04", "content": "M6 launching in february ?
", "rawMarkdown": "M6 launching in february ?", "votes": null }, { "id": "1571483", "postDate": "11/04/2021 22:56:17", "content": "I have the same question here. If the test sets, especially the evaluation after submission deadline, do not include long enough time series information, how could the models predict?
", "rawMarkdown": "I have the same question here. If the test sets, especially the evaluation after submission deadline, do not include long enough time series information, how could the models predict?", "votes": null }, { "id": "1571497", "postDate": "11/04/2021 23:21:04", "content": "The model can predict. You treat the data as cross sectional data instead of time series data. Your main work is to find out the right models and the right features and to use them to predict.
", "rawMarkdown": "The model can predict. You treat the data as cross sectional data instead of time series data. Your main work is to find out the right models and the right features and to use them to predict.", "votes": null }, { "id": "1572178", "postDate": "11/05/2021 13:36:38", "content": "Thanks, sounds like it might lose time-related information from training but it might be the approach they want us to go with.
\nOr it might depend on how external data can be used after the submission deadline.
Just noticed this from the data descriptions:
\nsupplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.
then I think the models are expected to get trained with updated data after the submission deadline.
", "rawMarkdown": "Just noticed this from the data descriptions:\nsupplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\nthen I think the models are expected to get trained with updated data after the submission deadline.", "votes": null }, { "id": "1572591", "postDate": "11/05/2021 19:07:21", "content": "It seems like Time-series Forecasting is one of your territories U_U, Thanks for the topic.
I would like to know your opinion on using TabNet with attention. Is it a good idea to implement? Adding Attention to other Deep Learning Models might also yield some good results.
", "rawMarkdown": "It seems like Time-series Forecasting is one of your territories `U_U`, Thanks for the topic.\n\n**I would like to know your opinion on using TabNet with attention. Is it a good idea to implement? Adding Attention to other Deep Learning Models might also yield some good results.**", "votes": null }, { "id": "1572617", "postDate": "11/05/2021 19:53:22", "content": "I think it will be much more simpler than you think - as is the case on real quant trading desks
\nMy Prediction
\nSimple/Weighted Ensemble of >10Fold NN+Boosted Decison Tree Model will perform the best here with around 25-50 features in the dataset
Agree. There doesn't seem to be a lot of hierarchy between the different assets.
", "rawMarkdown": "Agree. There doesn't seem to be a lot of hierarchy between the different assets.", "votes": null }, { "id": "1572956", "postDate": "11/06/2021 05:49:09", "content": "This is fantastic summary, guide and comments. Thank you. I feel like I could delve into the links and notebooks posted for hours and hours.
\nMuch appreciated
Hi,
\n\n\nNote: It is important to note however that in both recent financial competitions there was no HTS solution at the top.
\n
Which financial competitions are you referring? Which were the top solutions?
", "rawMarkdown": "Hi,\n\n> Note: It is important to note however that in both recent financial competitions there was no HTS solution at the top.\n\nWhich financial competitions are you referring? Which were the top solutions?", "votes": null }, { "id": "1574067", "postDate": "11/07/2021 07:46:26", "content": "Hi all,
\nCan any explain me how submission.csv works
", "rawMarkdown": "Hi all, \n\nCan any explain me how submission.csv works", "votes": null }, { "id": "1574079", "postDate": "11/07/2021 08:04:08", "content": "you are a phenomenon :-) join our team :-)
", "rawMarkdown": "you are a phenomenon :-) join our team :-)", "votes": null }, { "id": "1576539", "postDate": "11/09/2021 09:38:30", "content": "I really appreciate you taking the time to post this, especially since you linked examples. Thank you.
", "rawMarkdown": "I really appreciate you taking the time to post this, especially since you linked examples. Thank you.", "votes": null }, { "id": "1577519", "postDate": "11/10/2021 06:50:22", "content": "Status update 10-11-2021:
\nI have 10 years of investment experience, so I want to learn how to analyze data. My experience will be helpful for learning.
", "rawMarkdown": "I have 10 years of investment experience, so I want to learn how to analyze data. My experience will be helpful for learning.", "votes": null }, { "id": "1582496", "postDate": "11/15/2021 00:24:43", "content": "Hello, @yamqwe thanks for all the information provided; it will help me with my goals for this competition.
", "rawMarkdown": "Hello, @yamqwe thanks for all the information provided; it will help me with my goals for this competition.", "votes": null }, { "id": "1583075", "postDate": "11/15/2021 14:14:05", "content": "Thanks for the write-up very insightful and useful for learning purposes.
\nWhat do you think of the Optiver Realized Volatility Prediction competition? Pretty close goals isnt it?
", "rawMarkdown": "Thanks for the write-up very insightful and useful for learning purposes.\n\nWhat do you think of the [Optiver Realized Volatility Prediction](https://www.kaggle.com/c/optiver-realized-volatility-prediction) competition? Pretty close goals isnt it?", "votes": null }, { "id": "1585980", "postDate": "11/17/2021 17:31:05", "content": "Status update 17-11-2021:
\nTarget Computation: Targets creation had been updated to the code released by the hosts. And was validated to make sure it is similar [~0.00012526]
Bug fix: one of the updates to the collection code introduced a bug where all the data frames were of one asset. This was pointed out in the discussions of the dataset and fixed within minutes. The datasets are fixed now and usable again.
Feature engineering baselines: Three new feature engineering baselines had been added.
Baselines genera; update: All the baselines had been updated to support submitting to the LB [compute time issues..] also many small bugs were fixed. I advise you to recheck them if you use code from them.
Distance to the competition data: Some pointed out that in some eras the data is slightly different than the competition data. I am checking this out now. [I mean, right now. the comparison runs in the background while I type this 😄]
Uptime: Data collection had been going smoothly for 9/10 days since the project started!
Thank you all for your support! 😍
Do you even HODL bruh?
", "rawMarkdown": "Do you even HODL bruh?", "votes": null }, { "id": "1586242", "postDate": "11/17/2021 23:22:56", "content": "Amazing, tons and tons of information in all of your tutorials!
", "rawMarkdown": "Amazing, tons and tons of information in all of your tutorials!", "votes": null }, { "id": "1592128", "postDate": "11/22/2021 23:08:17", "content": "Thank you @yamqwe for sharing all this information, this is really useful 👍
", "rawMarkdown": "Thank you @yamqwe for sharing all this information, this is really useful 👍", "votes": null }, { "id": "1612105", "postDate": "12/08/2021 14:55:24", "content": "Amazing! Thanks for the detailed analysis! :)
", "rawMarkdown": "Amazing! Thanks for the detailed analysis! :)", "votes": null }, { "id": "1625929", "postDate": "12/22/2021 09:52:08", "content": "404 - https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn
", "rawMarkdown": "404 - https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn", "votes": null }, { "id": "1635663", "postDate": "01/02/2022 04:03:18", "content": "hey thanks for your hardwork , you really did nice work
", "rawMarkdown": "hey thanks for your hardwork , you really did nice work", "votes": null }, { "id": "1636445", "postDate": "01/02/2022 22:29:51", "content": "thank you so much.
", "rawMarkdown": "thank you so much.", "votes": null }, { "id": "1639298", "postDate": "01/05/2022 14:36:31", "content": "Super informative. Thanks!
", "rawMarkdown": "Super informative. Thanks!", "votes": null }, { "id": "1642318", "postDate": "01/08/2022 09:01:42", "content": "Very useful, thanks!
", "rawMarkdown": "Very useful, thanks!", "votes": null }, { "id": "1685517", "postDate": "02/11/2022 11:07:02", "content": "Thank you for your insights :)
", "rawMarkdown": "Thank you for your insights :)", "votes": null } ], "comments": [ { "id": 1568688, "author_name": "hantianzheng", "author_url": "", "post_date": "11/03/2021 01:28:46", "content": "Upvoted
\nM5 forecasting competition has a very complex metric for different level predictions(item-level, department-level, store-level, etc). I think this is why HTC model is applied there. Not sure if we can do this here.
", "votes": null, "replies": [ { "id": 1568922, "author_name": "gunesevitan", "author_url": "", "post_date": "11/03/2021 06:24:19", "content": "I don't think HTS models will work as good as global models either. There are lots of cross-asset information, bitcoin correlation, seasonality and etc. to use. I'm not sure HTS models can use all of those information.
", "votes": null, "replies": [] }, { "id": 1572808, "author_name": "gerbooz", "author_url": "", "post_date": "11/06/2021 01:33:11", "content": "Agree. There doesn't seem to be a lot of hierarchy between the different assets.
", "votes": null, "replies": [] } ] }, { "id": 1568767, "author_name": "everwitt7", "author_url": "", "post_date": "11/03/2021 03:07:53", "content": "wow thank you so much - as someone just delving into time series analysis I wanted to read other people's kernels and see what people were doing and this post is incredibly helpful.
", "votes": null, "replies": [] }, { "id": 1568854, "author_name": "yus002", "author_url": "", "post_date": "11/03/2021 04:48:11", "content": "I think lag features will not be useful because each time you only predict one row, so you have no more information to take.
", "votes": null, "replies": [ { "id": 1571483, "author_name": "tinycup", "author_url": "", "post_date": "11/04/2021 22:56:17", "content": "I have the same question here. If the test sets, especially the evaluation after submission deadline, do not include long enough time series information, how could the models predict?
", "votes": null, "replies": [] }, { "id": 1571497, "author_name": "yus002", "author_url": "", "post_date": "11/04/2021 23:21:04", "content": "The model can predict. You treat the data as cross sectional data instead of time series data. Your main work is to find out the right models and the right features and to use them to predict.
", "votes": null, "replies": [] }, { "id": 1572178, "author_name": "tinycup", "author_url": "", "post_date": "11/05/2021 13:36:38", "content": "Thanks, sounds like it might lose time-related information from training but it might be the approach they want us to go with.
\nOr it might depend on how external data can be used after the submission deadline.
Just noticed this from the data descriptions:
\nsupplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.
then I think the models are expected to get trained with updated data after the submission deadline.
", "votes": null, "replies": [] } ] }, { "id": 1568867, "author_name": "kamaljp", "author_url": "", "post_date": "11/03/2021 05:02:45", "content": "The competitions recently have become more timeseries oriented. Optiver, Google Brain Ventilator and now the crypto currencies predictions. What could be the reason?
", "votes": null, "replies": [ { "id": 1569931, "author_name": "endremoen", "author_url": "", "post_date": "11/03/2021 20:57:08", "content": "its fun! :)
", "votes": null, "replies": [] }, { "id": 1569961, "author_name": "lucasmorin", "author_url": "", "post_date": "11/03/2021 21:21:04", "content": "M6 launching in february ?
", "votes": null, "replies": [] } ] }, { "id": 1572591, "author_name": "soumya9977", "author_url": "", "post_date": "11/05/2021 19:07:21", "content": "It seems like Time-series Forecasting is one of your territories U_U, Thanks for the topic.
I would like to know your opinion on using TabNet with attention. Is it a good idea to implement? Adding Attention to other Deep Learning Models might also yield some good results.
", "votes": null, "replies": [] }, { "id": 1572617, "author_name": "munumbutt", "author_url": "", "post_date": "11/05/2021 19:53:22", "content": "I think it will be much more simpler than you think - as is the case on real quant trading desks
\nMy Prediction
\nSimple/Weighted Ensemble of >10Fold NN+Boosted Decison Tree Model will perform the best here with around 25-50 features in the dataset
This is fantastic summary, guide and comments. Thank you. I feel like I could delve into the links and notebooks posted for hours and hours.
\nMuch appreciated
Hi,
\n\n\nNote: It is important to note however that in both recent financial competitions there was no HTS solution at the top.
\n
Which financial competitions are you referring? Which were the top solutions?
", "votes": null, "replies": [] }, { "id": 1574067, "author_name": "pardhu555", "author_url": "", "post_date": "11/07/2021 07:46:26", "content": "Hi all,
\nCan any explain me how submission.csv works
", "votes": null, "replies": [] }, { "id": 1574079, "author_name": "hananshteingart", "author_url": "", "post_date": "11/07/2021 08:04:08", "content": "you are a phenomenon :-) join our team :-)
", "votes": null, "replies": [] }, { "id": 1576539, "author_name": "matteyritch", "author_url": "", "post_date": "11/09/2021 09:38:30", "content": "I really appreciate you taking the time to post this, especially since you linked examples. Thank you.
", "votes": null, "replies": [] }, { "id": 1577519, "author_name": "", "author_url": "", "post_date": "11/10/2021 06:50:22", "content": "Status update 10-11-2021:
\nI have 10 years of investment experience, so I want to learn how to analyze data. My experience will be helpful for learning.
", "votes": null, "replies": [] }, { "id": 1582496, "author_name": "cv13j0", "author_url": "", "post_date": "11/15/2021 00:24:43", "content": "Hello, @yamqwe thanks for all the information provided; it will help me with my goals for this competition.
", "votes": null, "replies": [] }, { "id": 1583075, "author_name": "neomatrix369", "author_url": "", "post_date": "11/15/2021 14:14:05", "content": "Thanks for the write-up very insightful and useful for learning purposes.
\nWhat do you think of the Optiver Realized Volatility Prediction competition? Pretty close goals isnt it?
", "votes": null, "replies": [] }, { "id": 1585980, "author_name": "", "author_url": "", "post_date": "11/17/2021 17:31:05", "content": "Status update 17-11-2021:
\nTarget Computation: Targets creation had been updated to the code released by the hosts. And was validated to make sure it is similar [~0.00012526]
Bug fix: one of the updates to the collection code introduced a bug where all the data frames were of one asset. This was pointed out in the discussions of the dataset and fixed within minutes. The datasets are fixed now and usable again.
Feature engineering baselines: Three new feature engineering baselines had been added.
Baselines genera; update: All the baselines had been updated to support submitting to the LB [compute time issues..] also many small bugs were fixed. I advise you to recheck them if you use code from them.
Distance to the competition data: Some pointed out that in some eras the data is slightly different than the competition data. I am checking this out now. [I mean, right now. the comparison runs in the background while I type this 😄]
Uptime: Data collection had been going smoothly for 9/10 days since the project started!
Thank you all for your support! 😍
Do you even HODL bruh?
", "votes": null, "replies": [] }, { "id": 1586242, "author_name": "asnunes", "author_url": "", "post_date": "11/17/2021 23:22:56", "content": "Amazing, tons and tons of information in all of your tutorials!
", "votes": null, "replies": [] }, { "id": 1592128, "author_name": "alincijov", "author_url": "", "post_date": "11/22/2021 23:08:17", "content": "Thank you @yamqwe for sharing all this information, this is really useful 👍
", "votes": null, "replies": [] }, { "id": 1612105, "author_name": "lukekim0418", "author_url": "", "post_date": "12/08/2021 14:55:24", "content": "Amazing! Thanks for the detailed analysis! :)
", "votes": null, "replies": [] }, { "id": 1625929, "author_name": "vladimircape", "author_url": "", "post_date": "12/22/2021 09:52:08", "content": "404 - https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn
", "votes": null, "replies": [] }, { "id": 1635663, "author_name": "hiteshkumars", "author_url": "", "post_date": "01/02/2022 04:03:18", "content": "hey thanks for your hardwork , you really did nice work
", "votes": null, "replies": [] }, { "id": 1636445, "author_name": "haiyunhu", "author_url": "", "post_date": "01/02/2022 22:29:51", "content": "thank you so much.
", "votes": null, "replies": [] }, { "id": 1639298, "author_name": "markzhaoty", "author_url": "", "post_date": "01/05/2022 14:36:31", "content": "Super informative. Thanks!
", "votes": null, "replies": [] }, { "id": 1642318, "author_name": "ezvhaun", "author_url": "", "post_date": "01/08/2022 09:01:42", "content": "Very useful, thanks!
", "votes": null, "replies": [] }, { "id": 1685517, "author_name": "matami8", "author_url": "", "post_date": "02/11/2022 11:07:02", "content": "Thank you for your insights :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1568505": "> **TL;DR:** [Code & Datasets](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726) | also see baselines below 👇\n\n\nAlright, Guys! \n\nThe following tutorial details some of my initial guesses for some of the concepts that probably will surface throughout the competition. Probably some will even be incorporated into some of the final solutions. \n\n____\n\n**Hierarchical Time-series (HTS)**\nThis is a time-series competition. \nmoreover - this can be modeled with hierarchical time series (HTS) model. \n\nThis basically means that you have a \"model per asset\" and \"one model for all\" maybe even \"model per weekday\"... and so on. by the end you ensemble and merge all the models.\n\n**This method had proven to work over and over and over again in previous competitions.** \nMost notably the[ last M5 prediction](https://www.kaggle.com/c/m5-forecasting-accuracy).[ the winning solution was an extremely simple LGB running in an HTS fashion.](https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/163684). At the end of the day, It destroyed much more complex solutions with ease. by the way, there were nearly no fancy deep-learning models whatsoever at the top. \n\nNote: It is important to note however that in both recent financial competitions there was no HTS solution at the top. \n\n____\n\n**Future Issues that future notebooks are going to address**\n\n**Memory limits**\n1.0 It might look like the data is small and it won't be a problem. trust me, when doing HTS. it will become a problem. You will have to address this head-on. \n1.1 You have to find a way to have more and more meaningful features (splits by \"type\"/date limits)\n1.2 lags, aggs, nan cutoffs, etc.. all need to be taken into consideration.\n\n**Models**\n2.1. There is no \"winner\" here but if there was, I think that the winner of this competition will be an ensemble of NNs [Important will be FC, Tabnet, AutoEncoders..]. They are performing pretty well on the extrapolation of time series data. \n2.2 The simple models will include LGB, XGB, Catboost, HistGBM. And strong solutions will have both.\n2.3 Some novel ideas might include ARIMA, SARIMAX with some unique tricks for ensembling.\n3.4 Ensembling and stacking, it works.\n3.5 About deep learning: I don't think there will be an end2end deep learning solution (like pure LSTM or WaveNet) at the beginning. These are absolutely extremely hard to get right. however, from experience: when you do manage to get them to work (mostly wavenet), they are REALLY GOOD. However, we might see pretty early on some NBEATS or some AutoML NBEATS solution (like gluonTS). these are not as hard to train and they are pretty stable. ([the 2nd solution on M5 used gluonTS with some cool tricks](https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/164599)). \n3.6 Neural networks that we will see for sure: simple FC networks trained on simple tabular feature engineering (lags, aggs.. etc..). They will also work pretty well on the LB. \n\n____\n\n**Advanced Ideas we will probably see sometime**\n\n4.0 Fancy Loss function (the most important when training neural networks) If you want to do this, You have to create your own one that takes into account the specific issues of this dataset. It is hard and takes time. \n4.1 Sample weights that \"stimulate\" the network to learn better \n4.2 Some fancy incorporation of prior knowledge from the dataset into the network architecture. \n\n____\n\n**About Features**\n5.1 If you want a stable solution that also might work well in real life and that will push your skills toward things that are common and you are going to see again and again: try to work on a simple single model (like LGB) and push its score above and above. \nYou want to try playing around mostly with with lags (df[numeticsl_column] - df.groupby(time_column)[numetical_column].shift(-1)) and agg features (df.groupby(time_column)[numetical_column].agg(std)). these are baseline time-series features. they will work here for sure. \n\n____\n\n**An overview of some M solutions - the top time-series competitions out there to date**\n\n- Info about M4 \nhttps://www.sciencedirect.com/science/article/pii/S0169207019301128\n\n- M4 winning solution was a hybrid of both exponential smoothing and a recurrent neural network. The model used was LSTM.\nLink: https://www.sciencedirect.com/science/article/pii/S0169207019301153\n\n- Another interesting article and probably relevant for Kaggle. The article describes ensembles of statistical models: https://www.sciencedirect.com/science/article/pii/S0169207019301190#b5\n\n- Overall, the entire journal for the M4 competition can be found here:\nhttps://www.sciencedirect.com/journal/international-journal-of-forecasting/vol/36/issue/1\n\n____\n\n**Some other useful Resources**\nCoursera has a course about Sequences, Time series prediction. It is a basic course that provides a preliminary overview of the concept And building models to Univariate Time series. The course uses Keras and TensorFlow, but it provides a good starting point for this topic\nLink: https://www.coursera.org/learn/tensorflow-sequences-time-series-and-prediction/home/welcome\n\n- LSTM networks - Great post/Blog that explains the concept beyond LSTM networks\n**Link:** https://colah.github.io/posts/2015-08-Understanding-LSTMs/\n\n- Post about building Time Series deep learning - The post covers the Keras/TensorFlow framework, but it also gives a great overview of the main concepts and how to prepare the data.\n**Link: **https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/\n\n- Pytorch Basic example - This is a simple example of how to build a Pytorch LSTM network for simple Univariate Time Series.\n**Link:** https://github.com/pytorch/examples/tree/master/time_sequence_prediction\n\n**Some misc links for time-series things you might want to take a look at**\n[credit: https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/133463]\nThere have been quite a few time-series competitions hosted on Kaggle. \n\nHere are a few of them:\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting\nhttps://www.kaggle.com/c/rossmann-store-sales\n\n**Here are a few of the winning solutions from each of the competitions:**\n\n____\n\n**Web Traffic Time Series**\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/43795\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39395\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39370\n\n____\n\n**Recruit Restaurant Visitor Forecasting**\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/49129\nhttps://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/49166\n\n____\n\n**Favorita Grocery Sales Forecasting**\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47582\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47556\nhttps://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47667\n\n____\n\n**Rossman Store Sales**\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/18024\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/17974\nhttps://www.kaggle.com/c/rossmann-store-sales/discussion/17896\n\n____\n\n### Implementations:\n\nI am currently in the [process of implementing](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726) the ideas in this post. \nThis is a work in progress. The following table updates with the current status of the project. \n\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | [Quant's Volatility Features](https://www.kaggle.com/yamqwe/crypto-prediction-volatility-features) |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | | ⏳Target Engineering |\n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |⏳Fourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | ⏳Wavelets | \n\n\n\n[About the validation: GroupTimeSeriesSplit](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit)\n\n(⏳ - in the making..)\n\n\n\n\nHappy Kaggling! \nCheers ^^", "1568688": "Upvoted\n\nM5 forecasting competition has a very complex metric for different level predictions(item-level, department-level, store-level, etc). I think this is why HTC model is applied there. Not sure if we can do this here.", "1568767": "wow thank you so much - as someone just delving into time series analysis I wanted to read other people's kernels and see what people were doing and this post is incredibly helpful.", "1568854": "I think lag features will not be useful because each time you only predict one row, so you have no more information to take.", "1568867": "The competitions recently have become more timeseries oriented. Optiver, Google Brain Ventilator and now the crypto currencies predictions. What could be the reason?", "1568922": "I don't think HTS models will work as good as global models either. There are lots of cross-asset information, bitcoin correlation, seasonality and etc. to use. I'm not sure HTS models can use all of those information.", "1569931": "its fun! :)", "1569961": "M6 launching in february ?", "1571483": "I have the same question here. If the test sets, especially the evaluation after submission deadline, do not include long enough time series information, how could the models predict?", "1571497": "The model can predict. You treat the data as cross sectional data instead of time series data. Your main work is to find out the right models and the right features and to use them to predict.", "1572178": "Thanks, sounds like it might lose time-related information from training but it might be the approach they want us to go with.\nOr it might depend on how external data can be used after the submission deadline.", "1572184": "Just noticed this from the data descriptions:\nsupplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\nthen I think the models are expected to get trained with updated data after the submission deadline.", "1572591": "It seems like Time-series Forecasting is one of your territories `U_U`, Thanks for the topic.\n\n**I would like to know your opinion on using TabNet with attention. Is it a good idea to implement? Adding Attention to other Deep Learning Models might also yield some good results.**", "1572617": "I think it will be much more simpler than you think - as is the case on real quant trading desks\n\nMy Prediction\nSimple/Weighted Ensemble of >10Fold NN+Boosted Decison Tree Model will perform the best here with around 25-50 features in the dataset", "1572808": "Agree. There doesn't seem to be a lot of hierarchy between the different assets.", "1572956": "This is fantastic summary, guide and comments. Thank you. I feel like I could delve into the links and notebooks posted for hours and hours.\nMuch appreciated", "1573180": "Hi,\n\n> Note: It is important to note however that in both recent financial competitions there was no HTS solution at the top.\n\nWhich financial competitions are you referring? Which were the top solutions?", "1574067": "Hi all, \n\nCan any explain me how submission.csv works", "1574079": "you are a phenomenon :-) join our team :-)", "1576539": "I really appreciate you taking the time to post this, especially since you linked examples. Thank you.", "1577519": "**Status update 10-11-2021:**\n- Data collection had been going smoothly for the past 3 days. \n- Working on VWAP & Targets creation. Minimizing the mean distance to the originals. \n- Just updated the post with the full baselines table. \n- You guys rock! Many many forks of the baseline notebooks in just a couple of days! 🤩", "1578463": "I have 10 years of investment experience, so I want to learn how to analyze data. My experience will be helpful for learning.", "1582496": "Hello, @yamqwe thanks for all the information provided; it will help me with my goals for this competition.", "1583075": "Thanks for the write-up very insightful and useful for learning purposes.\n\nWhat do you think of the [Optiver Realized Volatility Prediction](https://www.kaggle.com/c/optiver-realized-volatility-prediction) competition? Pretty close goals isnt it?", "1585980": "**Status update 17-11-2021:**\n- **Target Computation:** Targets creation had been updated to the [code](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778) released by the hosts. And was validated to make sure it is similar [~0.00012526]\n- **Bug fix:** one of the updates to the collection code introduced a bug where all the data frames were of one asset. This was pointed out in the discussions of the dataset and fixed within minutes. The datasets are fixed now and usable again. \n- **Feature engineering baselines:** Three new feature engineering baselines had been added.\n- **Baselines genera; update:** All the baselines had been updated to support submitting to the LB [compute time issues..] also many small bugs were fixed. I advise you to recheck them if you use code from them. \n- **Distance to the competition data:** Some pointed out that in some eras the data is slightly different than the competition data. I am checking this out now. [I mean, right now. the comparison runs in the background while I type this 😄]\n- **Uptime**: Data collection had been going smoothly for 9/10 days since the project started! \n\n- Thank you all for your support! 😍", "1586021": "Do you even HODL bruh?", "1586242": "Amazing, tons and tons of information in all of your tutorials!", "1592128": "Thank you @yamqwe for sharing all this information, this is really useful 👍", "1612105": "Amazing! Thanks for the detailed analysis! :)", "1625929": "404 - https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn", "1635663": "hey thanks for your hardwork , you really did nice work", "1636445": "thank you so much.", "1639298": "Super informative. Thanks!", "1642318": "Very useful, thanks!", "1685517": "Thank you for your insights :)" }, "source": "meta" }