{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

G-Research Crypto Forecasting

\n

Exploratory Data Analysis

\n
CREATED BY: DARIEN SCHETTLER
\n\n
\n\n---\n\n
\n\n
\n 📖   I USED THE FOLLOWING NOTEBOOKS... PLEASE UPVOTE THEM AS WELL!

\n - G-Research: Plot financial data using mplfinanc
\n
\n\n
\n\n
VERY WIP
\n","metadata":{}},{"cell_type":"markdown","source":"

\n\n

\n\n

TABLE OF CONTENTS

\n\n---\n\n

0    IMPORTS

\n\n---\n\n

1    BACKGROUND INFORMATION

\n\n---\n\n

2    SETUP

\n\n---\n\n

3    HELPER FUNCTIONS

\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

0  IMPORTS    ⤒

","metadata":{}},{"cell_type":"code","source":"print(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n# Machine Learning and Data Science Imports\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_addons as tfa; print(f\"\\t\\t– TENSORFLOW ADDONS VERSION: {tfa.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None;\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\nfrom sklearn.preprocessing import RobustScaler, PolynomialFeatures\nfrom sklearn.model_selection import GroupKFold;\n\n# Competition Specific Imports\nimport gresearch_crypto\nimport cudf, cuml, cupy\n\n!pip -q install mplfinance\n!pip -q install chart_studio\nimport mplfinance as mpf\nimport cufflinks as cf; cf.set_config_file(offline=True)\n\n# Built In Imports\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom glob import glob\nimport warnings\nimport requests\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports\nfrom matplotlib.colors import ListedColormap\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport seaborn as sns\nfrom PIL import Image\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nimport plotly\nimport PIL\nimport cv2\n\n\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\n \nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")\n \nprint(\"\\n... SEEDING FOR DETERMINISTIC BEHAVIOUR ...\\n\")\nseed_it_all()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:41:10.893647Z","iopub.execute_input":"2021-11-03T12:41:10.894282Z","iopub.status.idle":"2021-11-03T12:41:39.253024Z","shell.execute_reply.started":"2021-11-03T12:41:10.894171Z","shell.execute_reply":"2021-11-03T12:41:39.251708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

1  BACKGROUND INFORMATION    ⤒

\n\n
PRIMARY TASK DESCRIPTION\n\n\n**In this competition you will forecast the prices of several cyrptoassets. Once you make that prediction, you can move on to the next batch and will get additional data.**\n\nThis competition is different from most Kaggle Competitions in that:\n* You can only submit from Kaggle Notebooks\n* You must use our custom **`gresearch_crypto`** Python module. \n * The purpose of this module is to control the flow of information to ensure that you are not using future data to make predictions. \n * If you do not use this module properly, your code may fail.\n\n\n---\n\n
\n\n
INTERACTING WITH THE TEST API\n\nYou must leverage an environment from a python package and iterate through using the **`iter_test`** function:\n\niter_test\n* This is a generator which loops through each timestamp in the test set. \n* You have direct access to the example test rows for your convenience, but your code will only be able to get rows from the real test set via the API. \n* Once you call **`predict`** you can continue on to the next batch.\n\n**What Does It Yield**\n* While there are more batch(es) and `predict` was called successfully since the last yield, yields a tuple of:\n * **`test_df`**: DataFrame with the test features for the next batch, and user responses for the previous batch.\n * **`sample_prediction_df`**: DataFrame with an example prediction. \n * Intended to be filled in and passed back to the `predict` function.\n* If **`predict`** has not been called successfully since the last yield, prints an error and yields `None`.\n\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n\n\n

2  SETUP    ⤒\n

","metadata":{}},{"cell_type":"markdown","source":"

2.1 ACCELERATOR DETECTION

\n\n---\n\nIn order to use **`TPU`**, we use **`TPUClusterResolver`** for the initialization which is necessary to connect to the remote cluster and initialize cloud TPUs. Let's go over two important points\n\n1. When using TPU on Kaggle, you don't need to specify arguments for **`TPUClusterResolver`**\n2. However, on **G**oogle **C**ompute **E**ngine (**GCE**), you will need to do the following:\n\n
\n\n```python\n# The name you gave to the TPU to use\nTPU_WORKER = 'my-tpu-name'\n\n# or you can also specify the grpc path directly\n# TPU_WORKER = 'grpc://xxx.xxx.xxx.xxx:8470'\n\n# The zone you chose when you created the TPU to use on GCP.\nZONE = 'us-east1-b'\n\n# The name of the GCP project where you created the TPU to use on GCP.\nPROJECT = 'my-tpu-project'\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=TPU_WORKER, zone=ZONE, project=PROJECT)\n```\n\n
\n 🛑   WARNING:

- Although the Tensorflow documentation says it is the project name that should be provided for the argument `project`, it is actually the Project ID, that you should provide. This can be found on the GCP project dashboard page.
\n
\n\n
\n 📖   REFERENCES:

\n - Guide - Use TPUs
\n - Doc - TPUClusterResolver
\n\n
","metadata":{}},{"cell_type":"code","source":"print(f\"\\n... ACCELERATOR SETUP STARTING ...\\n\")\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n TPU = tf.distribute.cluster_resolver.TPUClusterResolver() \nexcept ValueError:\n TPU = None\n\nif TPU:\n print(f\"\\n... RUNNING ON TPU - {TPU.master()}...\")\n tf.config.experimental_connect_to_cluster(TPU)\n tf.tpu.experimental.initialize_tpu_system(TPU)\n strategy = tf.distribute.experimental.TPUStrategy(TPU)\nelse:\n print(f\"\\n... RUNNING ON CPU/GPU ...\")\n # Yield the default distribution strategy in Tensorflow\n # --> Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy() \n\n# What Is a Replica?\n# --> A single Cloud TPU device consists of FOUR chips, each of which has TWO TPU cores. \n# --> Therefore, for efficient utilization of Cloud TPU, a program should make use of each of the EIGHT (4x2) cores. \n# --> Each replica is essentially a copy of the training graph that is run on each core and \n# trains a mini-batch containing 1/8th of the overall batch size\nN_REPLICAS = strategy.num_replicas_in_sync\n \nprint(f\"... # OF REPLICAS: {N_REPLICAS} ...\\n\")\n\nprint(f\"\\n... ACCELERATOR SETUP COMPLTED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:41:39.255167Z","iopub.execute_input":"2021-11-03T12:41:39.256081Z","iopub.status.idle":"2021-11-03T12:41:39.272301Z","shell.execute_reply.started":"2021-11-03T12:41:39.256034Z","shell.execute_reply":"2021-11-03T12:41:39.27126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

2.2 COMPETITION DATA ACCESS

\n\n---\n\nTPUs read data must be read directly from **G**oogle **C**loud **S**torage **(GCS)**. Kaggle provides a utility library – **`KaggleDatasets`** – which has a utility function **`.get_gcs_path`** that will allow us to access the location of our input datasets within **GCS**.

\n\n
\n 📌   TIPS:

- If you have multiple datasets attached to the notebook, you should pass the name of a specific dataset to the `get_gcs_path()` function. In our case, the name of the dataset is the name of the directory the dataset is mounted within.

\n
","metadata":{}},{"cell_type":"code","source":"print(\"\\n... DATA ACCESS SETUP STARTED ...\\n\")\n\nif TPU:\n # Google Cloud Dataset path to training and validation images\n DATA_DIR = KaggleDatasets().get_gcs_path('g-research-crypto-forecasting')\n save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\nelse:\n # Local path to training and validation images\n DATA_DIR = \"/kaggle/input/g-research-crypto-forecasting\"\n save_locally = None\n \nprint(f\"\\n... DATA DIRECTORY PATH IS:\\n\\t--> {DATA_DIR}\")\n\nprint(f\"\\n... IMMEDIATE CONTENTS OF DATA DIRECTORY IS:\")\nfor file in tf.io.gfile.glob(os.path.join(DATA_DIR, \"*\")): print(f\"\\t--> {file}\")\n\n \nprint(\"\\n\\n... DATA ACCESS SETUP COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:41:39.277679Z","iopub.execute_input":"2021-11-03T12:41:39.27797Z","iopub.status.idle":"2021-11-03T12:41:39.29703Z","shell.execute_reply.started":"2021-11-03T12:41:39.277938Z","shell.execute_reply":"2021-11-03T12:41:39.295405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

2.3 BASIC DATA DEFINITIONS & INITIALIZATIONS

\n\n---\n\nLet's import the module and create an environment. \n* Adding the directory holding the module to the pythonpath with **`sys.path.append`** isn't strictly necessary within Kaggle notebooks, which handles that behind the scenes, but will be necessary if you are testing your code offline.\n* You can only call make_env() **once**, so don't lose it!","metadata":{}},{"cell_type":"code","source":"print(\"\\n... BASIC DATA SETUP STARTING ...\\n\\n\")\n\nprint(\"\\n... ASSET DETAILS DATAFRAME ..\\n\")\nASSET_CSV = os.path.join(DATA_DIR, \"asset_details.csv\")\nasset_df = cudf.read_csv(ASSET_CSV)\nasset_mapping = dict(asset_df[['Asset_ID', 'Asset_Name']].to_pandas().values)\nasset_reverse_mapping = {v:k for k,v in asset_mapping.items()}\ndisplay(asset_df.head())\n\nprint(\"\\n... TRAIN DATAFRAME ..\\n\")\nTRAIN_CSV = os.path.join(DATA_DIR, \"train.csv\")\ntrain_df = cudf.read_csv(TRAIN_CSV)\ntrain_df[\"Count\"] = train_df[\"Count\"].astype(\"int32\")\ntrain_df[\"asset_name\"] = train_df[\"Asset_ID\"].map(asset_mapping)\ntrain_df['timestamp'] = cudf.to_datetime(train_df['timestamp'], unit='s')\ntrain_df.columns = [_c.lower() for _c in train_df.columns]\ndisplay(train_df.head())\n\nprint(\"\\n... SUPPLEMENTAL TRAIN DATAFRAME ..\\n\")\nSUPP_TRAIN_CSV = os.path.join(DATA_DIR, \"supplemental_train.csv\")\nsupp_train_df = cudf.read_csv(SUPP_TRAIN_CSV)\nsupp_train_df[\"Count\"] = supp_train_df[\"Count\"].astype(\"int32\")\nsupp_train_df[\"asset_name\"] = supp_train_df[\"Asset_ID\"].map(asset_mapping)\nsupp_train_df['timestamp'] = cudf.to_datetime(supp_train_df['timestamp'], unit='s')\nsupp_train_df.columns = [_c.lower() for _c in supp_train_df.columns]\ndisplay(supp_train_df.head())\n\nprint(\"\\n... EXAMPLE TEST DATAFRAME ..\\n\")\nEX_TEST_CSV = os.path.join(DATA_DIR, \"example_test.csv\")\nex_test_df = cudf.read_csv(EX_TEST_CSV)\nex_test_df[\"Count\"] = ex_test_df[\"Count\"].astype(\"int32\")\nex_test_df[\"asset_name\"] = ex_test_df[\"Asset_ID\"].map(asset_mapping)\nex_test_df['timestamp'] = cudf.to_datetime(ex_test_df['timestamp'], unit='s')\nex_test_df.columns = [_c.lower() for _c in ex_test_df.columns]\ndisplay(ex_test_df.head())\n\nprint(\"\\n... EXAMPLE SAMPLE SUBMISSION DATAFRAME ..\\n\")\nEX_SS_CSV = os.path.join(DATA_DIR, \"example_sample_submission.csv\")\nex_ss_df = cudf.read_csv(EX_SS_CSV)\ndisplay(ex_ss_df.head())\n\n# Set Other Variables\nprint(\"\\n... SETTING OTHER VARIABLES ..\\n\")\n\n# Create a new contest\nenv = gresearch_crypto.make_env()\n\n# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()\nprint(\"\\n\\n... BASIC DATA SETUP FINISHING ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:41:39.298371Z","iopub.execute_input":"2021-11-03T12:41:39.298648Z","iopub.status.idle":"2021-11-03T12:42:17.290491Z","shell.execute_reply.started":"2021-11-03T12:41:39.298615Z","shell.execute_reply":"2021-11-03T12:42:17.289657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

2.4 LEVERAGING XLA OPTIMIZATIONS

\n\n---\n\n\n**XLA** (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes. **The results are improvements in speed and memory usage**.\n\n
\n\nWhen a TensorFlow program is run, all of the operations are executed individually by the TensorFlow executor. Each TensorFlow operation has a precompiled GPU/TPU kernel implementation that the executor dispatches to.\n\nXLA provides us with an alternative mode of running models: it compiles the TensorFlow graph into a sequence of computation kernels generated specifically for the given model. Because these kernels are unique to the model, they can exploit model-specific information for optimization.

\n\n
\n 🛑   WARNING:

- XLA can not currently compile functions where dimensions are not inferrable: that is, if it's not possible to infer the dimensions of all tensors without running the entire computation
\n
\n\n
\n 📌   NOTE:

- XLA compilation is only applied to code that is compiled into a graph (in TF2 that's only a code inside tf.function).
- The jit_compile API has must-compile semantics, i.e. either the entire function is compiled with XLA, or an errors.InvalidArgumentError exception is thrown)\n
\n\n
\n 📖   REFERENCE:

- XLA: Optimizing Compiler for Machine Learning
\n
","metadata":{}},{"cell_type":"code","source":"print(f\"\\n... XLA OPTIMIZATIONS STARTING ...\\n\")\n\nprint(f\"\\n... CONFIGURE JIT (JUST IN TIME) COMPILATION ...\\n\")\n# enable XLA optmizations (10% speedup when using @tf.function calls)\ntf.config.optimizer.set_jit(True)\n\nprint(f\"\\n... XLA OPTIMIZATIONS COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:42:17.291726Z","iopub.execute_input":"2021-11-03T12:42:17.292054Z","iopub.status.idle":"2021-11-03T12:42:17.299595Z","shell.execute_reply.started":"2021-11-03T12:42:17.292016Z","shell.execute_reply":"2021-11-03T12:42:17.298915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n

\n 3  HELPER FUNCTION & CLASSES    ⤒\n

","metadata":{}},{"cell_type":"markdown","source":"

3.1 GENERAL HELPER FUNCTIONS

\n\n---","metadata":{}},{"cell_type":"code","source":"def flatten_l_o_l(nested_list):\n \"\"\" Flatten a list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\ndef reduce_crypto_df(df, asset_id=None, asset_name=None, dt_start='2021-04-14 05:45:00', dt_end='2021-04-14 07:00:00', dt_interval=1):\n \"\"\" Useful function to return the subset of a dataframe with desired information \n \n Args:\n df (CuDF.DataFrame)\n asset_id (int, optional): Integer mapping for respective Cryptocurrency\n - 0 through 13 (14 total currencies as shown below)\n asset_name (str, optional): Name of the asset (cryptocurrency)\n - [Binance Coin|Bitcoin|Bitcoin Cash|Cardano]\n - [Dogecoin|EOS.IO|Ethereum|Ethereum Classic]\n - [IOTA|Litecoin|Maker|Monero|Stellar|TRON]\n dt_start (str, optional): The datetime to start the plot from (inclusive of this)\n dt_end (str, optional): The datetime to plot up until (non-inclusive of this) \n dt_interval (int, optional): The number of minutes per displayed candlestick\n \n Returns:\n a CuDF.DataFrame containing the relevant stock information for the desired time period\n \"\"\"\n \n if asset_id is None:\n if asset_name is None:\n raise ValueError(\"asset_id or asset_name must be provided\")\n else:\n asset_id = asset_reverse_mapping[asset_name]\n else:\n asset_name = asset_mapping[asset_id]\n \n return df[df.asset_id==asset_id].set_index('timestamp').loc[dt_start:dt_end:dt_interval]\n","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:48:44.092872Z","iopub.execute_input":"2021-11-03T12:48:44.093144Z","iopub.status.idle":"2021-11-03T12:48:44.100862Z","shell.execute_reply.started":"2021-11-03T12:48:44.093116Z","shell.execute_reply":"2021-11-03T12:48:44.099809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's look at the latest data (first half of September 2021)\nDT_START = \"2021-09-01 00:00:00\"\nDT_END = \"2021-09-21 00:00:00\"\nDT_INTERVAL = 60 # every 2 hours\n\nfor _asset_id in train_df.asset_id.unique().to_pandas():\n sub_crypto_df = reduce_crypto_df(train_df, asset_id=_asset_id, dt_start=DT_START, dt_end=DT_END, dt_interval=DT_INTERVAL)\n qf = cf.QuantFig(sub_crypto_df.to_pandas(), columns={k.title():k for k in sub_crypto_df.columns}, name=asset_mapping[_asset_id])\n qf.iplot()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:49:19.342768Z","iopub.execute_input":"2021-11-03T12:49:19.343342Z","iopub.status.idle":"2021-11-03T12:49:21.735278Z","shell.execute_reply.started":"2021-11-03T12:49:19.343296Z","shell.execute_reply":"2021-11-03T12:49:21.734631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}