","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.043234,"end_time":"2021-12-22T10:54:00.518073","exception":false,"start_time":"2021-12-22T10:54:00.474839","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.043888,"end_time":"2021-12-22T10:54:00.606291","exception":false,"start_time":"2021-12-22T10:54:00.562403","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n>### Transformer-based Multivariate Time Series Representation Learning \n>This is a simple starter notebook for Kaggle's Crypto Comp showing the usage of Time Series Transformer model from the paper \"Transformer-based Multivariate Time Series Representation Learning\" with Keras. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n\n
\n","metadata":{"papermill":{"duration":0.043852,"end_time":"2021-12-22T10:54:00.867551","exception":false,"start_time":"2021-12-22T10:54:00.823699","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content π\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.043331,"end_time":"2021-12-22T10:54:00.954557","exception":false,"start_time":"2021-12-22T10:54:00.911226","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [π Introduction](#introduction)\n\n2. [π Table Of Content](#outline) \n\n3. [π€Ώ Diving into the Data](#diving) \n\n4. [π Imports](#imports) \n\n5. [ποΈ Configurations](#config) \n\n6. [ποΈ Data Loading](#loading) \n\n7. [π¬ Feature Engineering](#features) \n\n8. [βοΈ Configure The Model](#modelconf)\n\n9. [ποΈ Training](#training)\n\n10. [π° Submit to Kaggle](#submit)","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.043724,"end_time":"2021-12-22T10:54:01.044507","exception":false,"start_time":"2021-12-22T10:54:01.000783","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data π€Ώ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.043703,"end_time":"2021-12-22T10:54:01.132575","exception":false,"start_time":"2021-12-22T10:54:01.088872","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - π There are 14 coins in the dataset\n>\n> - π There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.0439,"end_time":"2021-12-22T10:54:01.221177","exception":false,"start_time":"2021-12-22T10:54:01.177277","status":"completed"},"tags":[]}},{"cell_type":"code","source":"css_file = '''\ndiv #notebook {\nbackground-color: white;\nfont-family: 'Open Sans', Helvetica, sans-serif;\nline-height: 20px;\n}\n\n#notebook-container {\nmargin-top: 2em;\npadding-top: 2em;\nborder-top: 4px solid %s; /* light orange */\n-webkit-box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n}\n\ndiv .input {\nmargin-bottom: 1em;\n}\n\n.rendered_html h1, .rendered_html h2, .rendered_html h3, .rendered_html h4, .rendered_html h5, .rendered_html h6 {\ncolor: %s; /* light orange */\nfont-weight: 600;\n}\n\n.rendered_html code {\n background-color: #efefef; /* light gray */\n}\n\n.CodeMirror {\ncolor: #8c8c8c; /* dark gray */\npadding: 0.7em;\n}\n\ndiv.input_area {\nborder: none;\n background-color: %s; /* rgba(229, 143, 101, 0.1); light orange [exactly #E58F65] */\n border-top: 2px solid %s; /* light orange */\n}\n\ndiv.input_prompt {\ncolor: %s; /* light blue */\n}\n\ndiv.output_prompt {\ncolor: %s; /* strong orange */\n}\n\ndiv.cell.selected:before, div.cell.selected.jupyter-soft-selected:before {\nbackground: %s; /* light orange */\n}\n\ndiv.cell.selected, div.cell.selected.jupyter-soft-selected {\n border-color: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected:before {\nbackground: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected {\nborder-color: %s; /* light orange */\n\n}\n'''\ndef to_rgb(h): return tuple(int(h[i:i+2], 16) for i in (0, 2, 4))\nmain_color_rgba = 'rgba(%s, %s, %s, 0.1)' % (to_rgb(main_color[1:])[0], to_rgb(main_color[1:])[1], to_rgb(main_color[1:])[2])\nopen('notebook.css', 'w').write(css_file % (main_color, main_color, main_color_rgba, main_color, prompt, strong_main_color, main_color, main_color, main_color, main_color))\nfrom IPython.core.display import display, HTML, Javascript\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75405d52-594f-4791-a45a-16354bc24cfa","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"775496bb-f633-46f4-a642-10e6cd29c1bb","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.063682,"end_time":"2021-12-22T10:54:01.330369","exception":false,"start_time":"2021-12-22T10:54:01.266687","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-28T12:58:43.916375Z","iopub.execute_input":"2021-12-28T12:58:43.916828Z","iopub.status.idle":"2021-12-28T12:58:43.947863Z","shell.execute_reply.started":"2021-12-28T12:58:43.916773Z","shell.execute_reply":"2021-12-28T12:58:43.946137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\n# def nb(): return HTML(\"\")\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75d3e80e-0c31-48f5-88dd-5b6a3c79cf38","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"968efaca-ede8-4817-b3aa-baedaba3205e","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.119196,"end_time":"2021-12-22T10:54:01.496668","exception":false,"start_time":"2021-12-22T10:54:01.377472","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-28T12:58:43.976722Z","iopub.execute_input":"2021-12-28T12:58:43.97705Z","iopub.status.idle":"2021-12-28T12:58:43.986729Z","shell.execute_reply.started":"2021-12-28T12:58:43.977018Z","shell.execute_reply":"2021-12-28T12:58:43.985376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n Transformer-based Multivariate Time Series Representation Learning \n
https://arxiv.org/abs/2010.02803
\n \n
\n\n\nThis is a simple starter notebook for Kaggle's Crypto Comp showing the usage of Time Series Transformer model from the paper \"Transformer-based Multivariate Time Series Representation Learning\" with Keras. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n\n\nThe main difference is that this notebook supports training end2end and, well.. that it uses tf-keras rather then pytroch-lightning of the original implementation.\n\nThis notebook implements Transformer Encoder model for learning the representation of time series dataset in G-Research Crypto prediction competition. A **masking probability of 15%** is applied to sampled sequences features independently at random in geometric distribution. It learns to attend both to preceding and succeeding segments in individual features, as well as the inter-dependencies between features.\n\nThe model is trained with keras for about 1,000 epochs using combined training and test sets. The loss function is **masked MSE loss** trained with masking noise end2end to the targets. \nYou could play around the hyperparameters for a larger model or using more hand-crafted features as the input.\n\n\n\n\n\n\n
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n ![](\"https://i.ibb.co/2M2MsgH/upvote7.png\")
![\"Multivariate](\"https://storage.googleapis.com/kaggle-markpeng/GoogleBrainVentilator/mvts_transformer_architecture.png\")
\n\n