{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## GResearch Crypto Forecasting\n### Introduction\nIn this competition you will forecast the prices of several cyrptoassets. Once you make that prediction, you can move on to the next batch and will get additional data.\n\nThis competition is different from most Kaggle Competitions in that:\n* You can only submit from Kaggle Notebooks\n* You must use our custom `gresearch_crypto` Python module.  The purpose of this module is to control the flow of information to ensure that you are not using future data to make predictions.  If you do not use this module properly, your code may fail.\n\n### This starter notebook demonstrates how to use the `gresearch_crypto` module to get the test features and make predictions.\n### TL;DR: End-to-End Usage Example\n```\nimport gresearch_crypto\nenv = gresearch_crypto.make_env()\n\n# Training data is in the competition dataset as usual\ntrain_df = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv', low_memory=False)\ntgt_1_model.fit(train_df)\ntgt_2_model.fit(train_df)\niter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = tgt_1_model.predict(test_df)\n    env.predict(sample_prediction_df)\n```\nNote that `tgt_1_model.fit` and `tgt_2_model.fit` are examples of the functions you need to write for the above example to work.","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.010991,"end_time":"2020-10-16T21:06:02.406829","exception":false,"start_time":"2020-10-16T21:06:02.395838","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Introduction\nFirst let's import the module and create an environment. Adding the directory holding the module to the pythonpath with `sys.path.append` isn't strictly necessary within Kaggle notebooks, which handles that behind the scenes, but will be necessary if you are testing your code offline.","metadata":{"papermill":{"duration":0.009526,"end_time":"2020-10-16T21:06:02.426349","exception":false,"start_time":"2020-10-16T21:06:02.416823","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import gresearch_crypto\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2021-10-06T16:46:41.417495Z","iopub.execute_input":"2021-10-06T16:46:41.418023Z","iopub.status.idle":"2021-10-06T16:46:41.47296Z","shell.execute_reply.started":"2021-10-06T16:46:41.417921Z","shell.execute_reply":"2021-10-06T16:46:41.472181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can only call make_env() **once**, so don't lose it!","metadata":{}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()","metadata":{"papermill":{"duration":0.02778,"end_time":"2020-10-16T21:06:02.463894","exception":false,"start_time":"2020-10-16T21:06:02.436114","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:46:41.47456Z","iopub.execute_input":"2021-10-06T16:46:41.475121Z","iopub.status.idle":"2021-10-06T16:46:41.479105Z","shell.execute_reply.started":"2021-10-06T16:46:41.475086Z","shell.execute_reply":"2021-10-06T16:46:41.478507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"papermill":{"duration":0.020032,"end_time":"2020-10-16T21:06:02.494341","exception":false,"start_time":"2020-10-16T21:06:02.474309","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:46:41.480341Z","iopub.execute_input":"2021-10-06T16:46:41.480791Z","iopub.status.idle":"2021-10-06T16:46:41.499158Z","shell.execute_reply.started":"2021-10-06T16:46:41.480736Z","shell.execute_reply":"2021-10-06T16:46:41.498468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training data is in the competition dataset as usual","metadata":{"papermill":{"duration":0.011622,"end_time":"2020-10-16T21:06:02.516712","exception":false,"start_time":"2020-10-16T21:06:02.50509","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv', low_memory=False, \n                       dtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32', \n                              'Open': 'float64', 'High': 'float64', 'Low': 'float64', 'Close': 'float64', \n                              'Volume': 'float64', 'VWAP': 'float64'\n                             }\n                      )\ntrain_df.head(3)","metadata":{"papermill":{"duration":0.273123,"end_time":"2020-10-16T21:06:02.800188","exception":false,"start_time":"2020-10-16T21:06:02.527065","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:46:41.500342Z","iopub.execute_input":"2021-10-06T16:46:41.501173Z","iopub.status.idle":"2021-10-06T16:47:54.523448Z","shell.execute_reply.started":"2021-10-06T16:46:41.501125Z","shell.execute_reply":"2021-10-06T16:47:54.522537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `iter_test` function\n\nThis is a generator which loops through each timestamp in the test set. You have direct access to the example test rows for your convenience, but your code will only be able to get rows from the real test set via the API. Once you call `predict` you can continue on to the next batch.\n\nYields:\n* While there are more batch(es) and `predict` was called successfully since the last yield, yields a tuple of:\n    * `test_df`: DataFrame with the test features for the next batch, and user responses for the previous batch.\n    * `sample_prediction_df`: DataFrame with an example prediction.  Intended to be filled in and passed back to the `predict` function.\n* If `predict` has not been called successfully since the last yield, prints an error and yields `None`.","metadata":{"papermill":{"duration":0.010818,"end_time":"2020-10-16T21:06:02.82238","exception":false,"start_time":"2020-10-16T21:06:02.811562","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()","metadata":{"papermill":{"duration":0.018224,"end_time":"2020-10-16T21:06:02.851885","exception":false,"start_time":"2020-10-16T21:06:02.833661","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:47:54.525697Z","iopub.execute_input":"2021-10-06T16:47:54.525957Z","iopub.status.idle":"2021-10-06T16:47:54.529921Z","shell.execute_reply.started":"2021-10-06T16:47:54.525929Z","shell.execute_reply":"2021-10-06T16:47:54.529102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's get the data for the first test batch and check it out.","metadata":{"papermill":{"duration":0.011037,"end_time":"2020-10-16T21:06:02.874345","exception":false,"start_time":"2020-10-16T21:06:02.863308","status":"completed"},"tags":[]}},{"cell_type":"code","source":"(test_df, sample_prediction_df) = next(iter_test)\ntest_df.head(3)","metadata":{"papermill":{"duration":0.051867,"end_time":"2020-10-16T21:06:02.937505","exception":false,"start_time":"2020-10-16T21:06:02.885638","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:47:54.53104Z","iopub.execute_input":"2021-10-06T16:47:54.531242Z","iopub.status.idle":"2021-10-06T16:47:54.591641Z","shell.execute_reply.started":"2021-10-06T16:47:54.531219Z","shell.execute_reply":"2021-10-06T16:47:54.590715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note the warning about the lack of optimization! The version of the API that will deliver the hidden test set is both more efficient and going to deliver substantially more data. It's highly recommended that you read to [the data page](https://www.kaggle.com/c/g-research-crypto-forecasting/data) timeseries section for a discussion of how to plan for the impact of the API on your notebook's runtime and memory use.","metadata":{}},{"cell_type":"code","source":"sample_prediction_df.head(3)","metadata":{"papermill":{"duration":0.026029,"end_time":"2020-10-16T21:06:02.975857","exception":false,"start_time":"2020-10-16T21:06:02.949828","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:47:54.592695Z","iopub.execute_input":"2021-10-06T16:47:54.592923Z","iopub.status.idle":"2021-10-06T16:47:54.602435Z","shell.execute_reply.started":"2021-10-06T16:47:54.592899Z","shell.execute_reply":"2021-10-06T16:47:54.601856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll get an error if we try to continue on to the next batch without making our predictions for the current batch.","metadata":{"papermill":{"duration":0.012732,"end_time":"2020-10-16T21:06:03.001392","exception":false,"start_time":"2020-10-16T21:06:02.98866","status":"completed"},"tags":[]}},{"cell_type":"code","source":"next(iter_test)","metadata":{"papermill":{"duration":0.020584,"end_time":"2020-10-16T21:06:03.035182","exception":false,"start_time":"2020-10-16T21:06:03.014598","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:47:54.603465Z","iopub.execute_input":"2021-10-06T16:47:54.603876Z","iopub.status.idle":"2021-10-06T16:47:54.614603Z","shell.execute_reply.started":"2021-10-06T16:47:54.603845Z","shell.execute_reply":"2021-10-06T16:47:54.613746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **`predict`** function\nStores your predictions for the current batch.  Expects the same format as `sample_prediction_df`.\n\nArgs:\n* `predictions_df`: DataFrame which must have the same format as `sample_prediction_df`.\n\nThis function will raise an Exception if not called after a successful iteration of the `iter_test` generator.","metadata":{"papermill":{"duration":0.012685,"end_time":"2020-10-16T21:06:03.06105","exception":false,"start_time":"2020-10-16T21:06:03.048365","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Let's make a dummy prediction using the sample provided by `iter_test`.","metadata":{"papermill":{"duration":0.012697,"end_time":"2020-10-16T21:06:03.086804","exception":false,"start_time":"2020-10-16T21:06:03.074107","status":"completed"},"tags":[]}},{"cell_type":"code","source":"env.predict(sample_prediction_df)","metadata":{"papermill":{"duration":0.020002,"end_time":"2020-10-16T21:06:03.119765","exception":false,"start_time":"2020-10-16T21:06:03.099763","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:47:54.616349Z","iopub.execute_input":"2021-10-06T16:47:54.616828Z","iopub.status.idle":"2021-10-06T16:47:54.626873Z","shell.execute_reply.started":"2021-10-06T16:47:54.616787Z","shell.execute_reply":"2021-10-06T16:47:54.625955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Main Loop\nLet's loop through all the remaining batches in the test set generator and make the default prediction for each.  The `iter_test` generator will simply stop returning values once you've reached the end.\n\nWhen writing your own notebooks, be sure to write robust code that makes as few assumptions about the `iter_test`/`predict` loop as possible.  For example there may be large gaps between timestamps for one or more cryptoassets. In the unlikely event that a cryptoasset were dropped from enough exchanges it might go missing from the dataset entirely.\n\nYou may assume that the structure of `sample_prediction_df` will not change in this competition.","metadata":{"papermill":{"duration":0.012972,"end_time":"2020-10-16T21:06:03.14595","exception":false,"start_time":"2020-10-16T21:06:03.132978","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = 0\n    env.predict(sample_prediction_df)","metadata":{"papermill":{"duration":0.363455,"end_time":"2020-10-16T21:06:03.522581","exception":false,"start_time":"2020-10-16T21:06:03.159126","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-06T16:47:54.627962Z","iopub.execute_input":"2021-10-06T16:47:54.628386Z","iopub.status.idle":"2021-10-06T16:47:54.647981Z","shell.execute_reply.started":"2021-10-06T16:47:54.628356Z","shell.execute_reply":"2021-10-06T16:47:54.647177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Restart the Notebook to run your code again\nIn order to combat cheating you are only allowed to call `make_env` or iterate through `iter_test` once per Notebook run.  However, while you're iterating on your model it's reasonable to try something out, change the model a bit, and try it again.  Unfortunately, if you try to simply re-run the code, or even refresh the browser page, you'll still be running on the same Notebook execution session you had been running before, and the `gresearch_crypto` module will still throw errors.  To get around this you need to explicitly restart your Notebook execution session.","metadata":{"papermill":{"duration":0.013013,"end_time":"2020-10-16T21:06:03.549216","exception":false,"start_time":"2020-10-16T21:06:03.536203","status":"completed"},"tags":[]}}]}