{ "id": 286677, "title": "Transformer-based Multivariate Time Series Representation Learning", "url": "/competitions/g-research-crypto-forecasting/discussion/286677", "author_name": "", "post_date": "2021-11-10T08:12:40.631500Z", "votes": 17, "comment_count": 1, "views": 0, "content": "

\n
\n

TL;DR: Here is the code

\n
\n

Hi Guys,

\n

I just finished converting my implementation of the paper \"Transformer-based Multivariate Time Series Representation Learning\" from the google ventilators competition earlier this year.

\n

The notebook is right here.

\n

It supports training as a supervised model end2end. based on the same architecture as the original paper. You can pay around with the code and get it to be training self-supervised pretty easily [the generator for the masking noise of the paper is there and tested] but as of this moment, it is at end2end mode.

\n

I was also not 100% sure if I should have included the masking while supervised training since it can be treated as an augmentation, anyway it is still there at the moment and plugged in.

\n

Basically, it is a notebook demonstrating the use case of extracting unsupervised representation vectors or transfer learning from a pretrained transformer for time-series data.

\n

Hope it helps someone, probably some fellow Kagglers.

\n

\"\"

\n

Multivariate Time Series Transformer Framework

\n

Source: https://arxiv.org/abs/2010.02803

\n

Reference:
\nGeorge Zerveas et al. (2021). \"A transformer-based Framework for Multivariate Time Series Representation Learning,\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21).

\n

This notebook follows the ideas presented in my Initial Thoughts.

\n

Happy Kaggling!
\nCheers ^^

", "messages": [ { "id": "1577604", "postDate": "11/10/2021 08:12:40", "content": "

\n
\n

TL;DR: Here is the code

\n
\n

Hi Guys,

\n

I just finished converting my implementation of the paper \"Transformer-based Multivariate Time Series Representation Learning\" from the google ventilators competition earlier this year.

\n

The notebook is right here.

\n

It supports training as a supervised model end2end. based on the same architecture as the original paper. You can pay around with the code and get it to be training self-supervised pretty easily [the generator for the masking noise of the paper is there and tested] but as of this moment, it is at end2end mode.

\n

I was also not 100% sure if I should have included the masking while supervised training since it can be treated as an augmentation, anyway it is still there at the moment and plugged in.

\n

Basically, it is a notebook demonstrating the use case of extracting unsupervised representation vectors or transfer learning from a pretrained transformer for time-series data.

\n

Hope it helps someone, probably some fellow Kagglers.

\n

\"\"

\n

Multivariate Time Series Transformer Framework

\n

Source: https://arxiv.org/abs/2010.02803

\n

Reference:
\nGeorge Zerveas et al. (2021). \"A transformer-based Framework for Multivariate Time Series Representation Learning,\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21).

\n

This notebook follows the ideas presented in my Initial Thoughts.

\n

Happy Kaggling!
\nCheers ^^

", "rawMarkdown": "#### Transformer-based Multivariate Time Series Representation Learning\n\n> **TL;DR:** [Here is the code](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer)\n\nHi Guys, \n\nI just finished converting [my implementation](https://www.kaggle.com/yamqwe/keras-time-series-transformer) of the paper \"Transformer-based Multivariate Time Series Representation Learning\" from the google ventilators competition earlier this year. \n\nThe notebook is [right here](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer).\n\n\nIt supports training as a supervised model end2end. based on the same architecture as the original paper. You can pay around with the code and get it to be training self-supervised pretty easily [the generator for the masking noise of the paper is there and tested] but as of this moment, it is at end2end mode. \n\nI was also not 100% sure if I should have included the masking while supervised training since it can be treated as an augmentation, anyway it is still there at the moment and plugged in. \n\nBasically, it is a notebook demonstrating the use case of extracting unsupervised representation vectors or transfer learning from a pretrained transformer for time-series data.\n\nHope it helps someone, probably some fellow Kagglers.\n\n![](https://storage.googleapis.com/kaggle-markpeng/GoogleBrainVentilator/mvts_transformer_architecture.png)\n\n**Multivariate Time Series Transformer Framework**\n\nSource: https://arxiv.org/abs/2010.02803\n\n**Reference:**\nGeorge Zerveas et al. (2021). \"A transformer-based Framework for Multivariate Time Series Representation Learning,\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21).\n\n\nThis notebook follows the ideas presented in my [Initial Thoughts](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903).\n\nHappy Kaggling! \nCheers ^^", "votes": null }, { "id": "2485176", "postDate": "10/17/2023 02:15:32", "content": "

The links you provided for the notebook or implementation, are not working.
\nSharing the implementation would be appritiated.

", "rawMarkdown": "The links you provided for the notebook or implementation, are not working.\nSharing the implementation would be appritiated.", "votes": null } ], "comments": [ { "id": 2485176, "author_name": "itzzsayeed", "author_url": "", "post_date": "10/17/2023 02:15:32", "content": "

The links you provided for the notebook or implementation, are not working.
\nSharing the implementation would be appritiated.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1577604": "#### Transformer-based Multivariate Time Series Representation Learning\n\n> **TL;DR:** [Here is the code](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer)\n\nHi Guys, \n\nI just finished converting [my implementation](https://www.kaggle.com/yamqwe/keras-time-series-transformer) of the paper \"Transformer-based Multivariate Time Series Representation Learning\" from the google ventilators competition earlier this year. \n\nThe notebook is [right here](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer).\n\n\nIt supports training as a supervised model end2end. based on the same architecture as the original paper. You can pay around with the code and get it to be training self-supervised pretty easily [the generator for the masking noise of the paper is there and tested] but as of this moment, it is at end2end mode. \n\nI was also not 100% sure if I should have included the masking while supervised training since it can be treated as an augmentation, anyway it is still there at the moment and plugged in. \n\nBasically, it is a notebook demonstrating the use case of extracting unsupervised representation vectors or transfer learning from a pretrained transformer for time-series data.\n\nHope it helps someone, probably some fellow Kagglers.\n\n![](https://storage.googleapis.com/kaggle-markpeng/GoogleBrainVentilator/mvts_transformer_architecture.png)\n\n**Multivariate Time Series Transformer Framework**\n\nSource: https://arxiv.org/abs/2010.02803\n\n**Reference:**\nGeorge Zerveas et al. (2021). \"A transformer-based Framework for Multivariate Time Series Representation Learning,\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21).\n\n\nThis notebook follows the ideas presented in my [Initial Thoughts](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903).\n\nHappy Kaggling! \nCheers ^^", "2485176": "The links you provided for the notebook or implementation, are not working.\nSharing the implementation would be appritiated." }, "source": "meta" }