{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nThis notebook is an attempt to gauge, at a first glance, all the tasks we will have to undertake to participate in this competition. So, here I will be looking at the data files provided to us. Specifically, *the asset_details.csv* and *train.csv* files and noting my thoughts as I go through them.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime\nimport os\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport seaborn as sns\ncmap = sns.color_palette()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:47:59.800504Z","iopub.execute_input":"2021-11-05T16:47:59.801473Z","iopub.status.idle":"2021-11-05T16:48:02.116212Z","shell.execute_reply.started":"2021-11-05T16:47:59.80135Z","shell.execute_reply":"2021-11-05T16:48:02.115326Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = '../input/g-research-crypto-forecasting'","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:48:11.654992Z","iopub.execute_input":"2021-11-05T16:48:11.655296Z","iopub.status.idle":"2021-11-05T16:48:11.658882Z","shell.execute_reply.started":"2021-11-05T16:48:11.655263Z","shell.execute_reply":"2021-11-05T16:48:11.658202Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls $data_path","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:48:16.505266Z","iopub.execute_input":"2021-11-05T16:48:16.50607Z","iopub.status.idle":"2021-11-05T16:48:17.281797Z","shell.execute_reply.started":"2021-11-05T16:48:16.50602Z","shell.execute_reply":"2021-11-05T16:48:17.280622Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data","metadata":{}},{"cell_type":"code","source":"assets = pd.read_csv(os.path.join(data_path, 'asset_details.csv'))\nassets.sort_values('Weight', ascending = False)","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:48:22.460871Z","iopub.execute_input":"2021-11-05T16:48:22.461579Z","iopub.status.idle":"2021-11-05T16:48:22.512317Z","shell.execute_reply.started":"2021-11-05T16:48:22.461533Z","shell.execute_reply":"2021-11-05T16:48:22.511404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So this file contains the names of all 14 coins along with their id to which they must be mapped. Also, contains a weighting value presumably based on how important they will be in our predicitons.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(data_path, 'train.csv'))\ntrain_df['asset_name'] = train_df.Asset_ID.map(assets.set_index('Asset_ID').Asset_Name)\nprint(f'There are {len(train_df)} rows in the dataset')\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:48:28.920844Z","iopub.execute_input":"2021-11-05T16:48:28.921936Z","iopub.status.idle":"2021-11-05T16:49:55.564521Z","shell.execute_reply.started":"2021-11-05T16:48:28.921892Z","shell.execute_reply":"2021-11-05T16:49:55.563615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is what we'll mainly be working with. There are almost 25 million rows of data. Each row represents a minute's worth of data for a particular coin. Here's what the hosts say about the variables:\n* timestamp: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n* Asset_ID: The asset ID corresponding to one of the crytocurrencies (e.g. Asset_ID = 1 for Bitcoin). The mapping from Asset_ID to crypto asset is contained in asset_details.csv.\n* Count: Total number of trades in the time interval (last minute).\n* Open: Opening price of the time interval (in USD).\n* High: Highest price reached during time interval (in USD).\n* Low: Lowest price reached during time interval (in USD).\n* Close: Closing price of the time interval (in USD).\n* Volume: Quantity of asset bought or sold, displayed in base currency USD.\n* VWAP: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n* Target: Residual log-returns for the asset over a 15 minute horizon.","metadata":{}},{"cell_type":"markdown","source":"Let's see if all the coins are equally represented in our dataset.","metadata":{}},{"cell_type":"code","source":"asset_counts = []\nfor asset in assets.Asset_ID:\n    x = assets[assets.Asset_ID == asset].Asset_Name.values[0]\n    y = train_df[train_df.Asset_ID == asset].Asset_ID.value_counts().values[0]\n    asset_counts.append([x,y])\nasset_count_df = pd.DataFrame(asset_counts)\nasset_count_df.columns = ['coin', 'count']\nasset_count_df","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:50:03.435662Z","iopub.execute_input":"2021-11-05T16:50:03.435996Z","iopub.status.idle":"2021-11-05T16:50:09.624955Z","shell.execute_reply.started":"2021-11-05T16:50:03.435962Z","shell.execute_reply":"2021-11-05T16:50:09.623882Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig_1 = px.pie(asset_count_df, values = 'count', names = 'coin',\n              title = 'Distribution of data in the dataset')\nfig_1.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:50:15.224959Z","iopub.execute_input":"2021-11-05T16:50:15.225291Z","iopub.status.idle":"2021-11-05T16:50:16.387016Z","shell.execute_reply.started":"2021-11-05T16:50:15.225255Z","shell.execute_reply":"2021-11-05T16:50:16.386123Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, not all coins are equally represented. In some cases there is quite a big disparity. Like in the case of **Maker** where there is almost a disparity of 1.3 million samples when compared to **Bitcoin**\n\nThis could be caused by data not being available for the same time period for all the different coins. We verify this by finding the start and end dates for all the different coins.","metadata":{}},{"cell_type":"code","source":"dates = []\nfor ind,coin in enumerate(assets.Asset_Name.values):\n    crypto = train_df[train_df.asset_name == coin].set_index('timestamp')\n    start_time = crypto.index[0].astype('datetime64[s]')\n    end_time = crypto.index[-1].astype('datetime64[s]')\n    dates.append([coin, start_time, end_time])\ndates_df = pd.DataFrame(dates)\ndates_df.columns = ['coin','start', 'end']\ndates_df","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:50:23.796403Z","iopub.execute_input":"2021-11-05T16:50:23.797395Z","iopub.status.idle":"2021-11-05T16:51:19.373802Z","shell.execute_reply.started":"2021-11-05T16:50:23.797351Z","shell.execute_reply":"2021-11-05T16:51:19.372879Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From this we can see that almost all coins have the same start and end date. All of them, with the exception of Dogecoin, have a start date in 2018 and all of them end at the same period. While this does account for some disparity, it doesn not explain the magnitude of it we see in our data distribution. \n\nWe have been informed by the host that:\n\n> Missing asset data, for a given minute, is not represented by NaN's, but instead by the absence of those rows.\n\nThis could explain the missing rows. We can find these by looking for instances where the timegap between consecutive rows is greater than 60 seconds. ","metadata":{}},{"cell_type":"code","source":"missing_rows = []\nfor ind,coin in enumerate(assets.Asset_Name.values):\n    crypto = train_df[train_df.asset_name == coin].set_index('timestamp')\n    v, c = np.unique(np.diff(crypto.index), return_counts = True)\n    vc = zip(v,c)\n    vc_df = pd.DataFrame(vc)\n    vc_df.columns = ['interval', 'counts']\n    missing_data = vc_df[1:]\n    missing_data['interval(mins)'] = missing_data.interval/60\n    missing_data['rows'] = missing_data['interval(mins)']*missing_data.counts\n    gaps = missing_data.counts.sum()\n    rows = missing_data.rows.sum()\n    missing_rows.append([coin, gaps, rows])\nmissing_rows_df = pd.DataFrame(missing_rows)\nmissing_rows_df.columns = ['coin', 'gaps','rows_missing']\nmissing_rows_df","metadata":{"execution":{"iopub.status.busy":"2021-11-05T17:03:43.503709Z","iopub.execute_input":"2021-11-05T17:03:43.504034Z","iopub.status.idle":"2021-11-05T17:04:38.918215Z","shell.execute_reply.started":"2021-11-05T17:03:43.503975Z","shell.execute_reply":"2021-11-05T17:04:38.917164Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This goes someway to explaining the disparity in our dataset. Some of these coins have hundreds of thousands of rows of missing data. We're going to have to find a way to fill these gaps, or at least account for them, when we build our model. ","metadata":{}},{"cell_type":"markdown","source":"Let's take a deeper look at one of the coins to see if we can expect more missing data from this dataset, such as NaN's.","metadata":{}},{"cell_type":"code","source":"bit = train_df[train_df.Asset_ID == 1].set_index('timestamp')\nbit.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T16:54:56.426431Z","iopub.execute_input":"2021-11-05T16:54:56.42726Z","iopub.status.idle":"2021-11-05T16:54:57.198525Z","shell.execute_reply.started":"2021-11-05T16:54:56.427192Z","shell.execute_reply":"2021-11-05T16:54:57.197427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bit.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T12:45:39.022725Z","iopub.execute_input":"2021-11-05T12:45:39.022981Z","iopub.status.idle":"2021-11-05T12:45:39.064607Z","shell.execute_reply.started":"2021-11-05T12:45:39.022955Z","shell.execute_reply":"2021-11-05T12:45:39.063745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There's another thing we'll have to figure then. There are some targets missing.","metadata":{}},{"cell_type":"code","source":"import time\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plots","metadata":{}},{"cell_type":"markdown","source":"Let's see how the closing price has behaved for bitcoin over the last 3 years.","metadata":{}},{"cell_type":"code","source":"f = plt.figure(figsize = (10,6))\n\nax = f.add_subplot(111)\nax.set_facecolor('azure')\nplt.plot(bit['Close'], c = 'darkviolet')\nplt.axvline(x = 1546300800, label = 'Start of 2019', c = 'orange')\nplt.axvline(x = 1577836800, label = 'Start of 2020', c = 'forestgreen')\nplt.axvline(x = 1609459200, label = 'Start of 2021', c = 'crimson')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin Close')\nplt.grid()\nplt.title('Close price of bitcoin')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T13:35:14.205828Z","iopub.execute_input":"2021-11-05T13:35:14.206195Z","iopub.status.idle":"2021-11-05T13:35:16.301268Z","shell.execute_reply.started":"2021-11-05T13:35:14.206155Z","shell.execute_reply":"2021-11-05T13:35:16.300232Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Something has clearly changed in 2021 where we see extremely volatile behaviour. The period before looks relatively stable compared to what has happened since the end of 2019. Let's see if this behaviour is replicated by the other coins.","metadata":{}},{"cell_type":"code","source":"f = plt.figure(figsize = (15,30))\n\nfor ind,coin in enumerate(assets.Asset_Name.values):\n    crypto = train_df[train_df.asset_name == coin].set_index('timestamp')\n    ax = f.add_subplot(7,2,ind+1)\n    ax.set_facecolor('azure')\n    plt.plot(crypto['Close'], c = cmap[ind%10])\n    plt.axvline(x = 1546300800, label = 'Start of 2019', c = 'orange')\n    plt.axvline(x = 1577836800, label = 'Start of 2020', c = 'forestgreen')\n    plt.axvline(x = 1609459200, label = 'Start of 2021', c = 'crimson')\n    plt.legend()\n    plt.xlabel('Time')\n    plt.ylabel('Close price')\n    plt.grid()\n    plt.title(coin)\n    \nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T14:22:45.70718Z","iopub.execute_input":"2021-11-05T14:22:45.708263Z","iopub.status.idle":"2021-11-05T14:24:04.997869Z","shell.execute_reply.started":"2021-11-05T14:22:45.7082Z","shell.execute_reply":"2021-11-05T14:24:04.996947Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A lot of coins share a lot of similarities with each other. And most coins exhibit the same behaviour in 2021. There's probably an exogenous reason for it. Like, perhaps stemming from increased speculation because of cryptos going mainstream and people having a lot of time on their hands because of being home during the pandemic? It's just a hypothesis, a spurious one at that. We'll explore this later.","metadata":{}},{"cell_type":"markdown","source":"# Upcoming\nI intend to continue working on this notebook and pick up from where I left off. Do return if you have found this helpful.\n\n**References**\n* [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n* [Simple Exploration Notebook - Crypto Forecasting](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-crypto-forecasting)\nI have learned a great deal from these notebooks and I hope you will as well.","metadata":{}}]}