{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# Exploration of Missing Data in G-Research Crypto Forecasting\n", "\n", "This notebook presents results from my exploration of missing data in the G-Research Crypto Forecasting dataset. \n", "\n", "I explore the missing data from 2 aspects:\n", "\n", "1. Missing timepoints, further reffered to as gaps.\n", "2. Missing values in `Target`, which typically stem from the time gaps (point 1.)\n", "\n", "A big portion of the code is abstracted in the `gresearch_crypto_utils` file, which is available if you are interested in the computational details."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": ["import matplotlib.pyplot as plt\n", "import gresearch_crypto_utils as utils"]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": ["MAKE_LARGE_PLOTS = True\n", "dataset = utils.GResearchDataset(\n", "    base_path = \"/kaggle/input/g-research-crypto-forecasting\"\n", ")"]}, {"cell_type": "markdown", "metadata": {}, "source": ["For some cryptoassets, the data is available since 2019, so that we will subsample all the training set such as to have the data for all cryptoassets. More precisely, we will use the data in the 2019-04-12 14:34:00 - 2021-09-21 00:00:00 time period in this analysis.\n", "\n", "The preprocessing of the raw training data is done inside the `dataset.get_processed_dfs_dict_by_asset`. It includes addition of rows (with NaNs) for each minute in the targeted time period."]}, {"cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": ["assets_train = dataset.get_processed_dfs_dict_by_asset()\n", "assets_train[0].head(3)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## NAs in prices\n", "\n", "In the training data, there are no cases for which all the prices (HLCO) are missing. Also, if there is data for one type of price, there is data for all the other prices."]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": ["na_counts = dataset.train[['Close', 'Open', 'Low', 'High']].isna().sum(axis=\"columns\")\n", "(na_counts == 4).sum(), ((1 <= na_counts) & (na_counts <= 3)).sum()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Gaps in time-series for each asset\n", "\n", "We compute a dataframe where each row corresponds to a time gap in a particular asset. \n", "\n", "For example, row number 1 below is a time gap of length 1 min in asset with id==2, starting at 2019-06-01 00:00:00."]}, {"cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": ["gap_stats = utils.compute_gap_stats_all_assets(dataset.assets_train, dataset.asset_details)\n", "gap_stats.head()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Given this computed table, let's evaluate, for each asset:\n", "- distribution of gap durations\n", "- distribution of intergap intervals\n", "- gap length in time"]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": ["if MAKE_LARGE_PLOTS:\n", "    for i, row in dataset.get_assets_iter():\n", "        fig, ax = plt.subplots(figsize=(12, 50 // 14))\n", "        (gap_stats\n", "            .loc[(gap_stats['Asset_ID'] == row['Asset_ID'])]\n", "            ['gap_duration_mins']\n", "            .plot.hist(bins=50, title=f\"Gap duration (mins): {row['Asset_Name']}\", ax=ax)\n", "        )"]}, {"cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": ["if MAKE_LARGE_PLOTS:\n", "    for i, row in dataset.get_assets_iter():\n", "        fig, ax = plt.subplots(figsize=(12,50 // 14))\n", "        (gap_stats\n", "            .loc[(gap_stats['Asset_ID'] == row['Asset_ID'])]\n", "            ['mins_from_last_gap']\n", "            .plot.hist(bins=50, title=f\"Intergap duration (mins): {row['Asset_Name']}\", ax=ax)\n", "        )"]}, {"cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": ["if MAKE_LARGE_PLOTS:\n", "    for i, row in dataset.get_assets_iter():\n", "        fig, ax = plt.subplots(figsize=(15,80 // 14))\n", "        (gap_stats\n", "            .loc[(gap_stats['Asset_ID'] == row['Asset_ID'])] \n", "            .plot.scatter(\n", "                x=\"gap_start\", y=\"gap_duration_mins\", \n", "                title=f\"Gap lengths in time: {row['Asset_Name']}\", ax=ax)\n", "        )"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Missing data in `Target`\n", "\n", "We'll explore:\n", "- Evolution of `Target` in time (not strictly related to missing data exploration)\n", "- Distribution of `Target` in time (not strictly related to missing data exploration)\n", "- Missing data in `Target` evolution in time:\n", "  - We'll compare missing data cumulative plots of `Target` with those computed from time gaps. The similarity of these 2 graphs, for each cryptoasset, indicates that they are associated."]}, {"cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": ["if MAKE_LARGE_PLOTS:\n", "    for i, row in dataset.get_assets_iter():\n", "        fig, ax = plt.subplots(figsize=(15,100 // 14))\n", "        asset_id = row['Asset_ID']\n", "        df_asset = assets_train[asset_id]\n", "        df_asset[\"Target\"].plot(\n", "            ax=ax, title=f\"Target: {row['Asset_Name']}\")"]}, {"cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": ["if MAKE_LARGE_PLOTS:\n", "    fig, axs = plt.subplots(nrows=dataset.asset_details.shape[0], figsize=(10,70), sharex=True)\n", "\n", "    for i, row in dataset.get_assets_iter():\n", "        asset_id = row['Asset_ID']\n", "        df_asset = assets_train[asset_id]\n", "        df_asset[\"Target\"].plot.hist(\n", "            bins=50,\n", "            ax=axs[i], \n", "            title=f\"Target histogram: {row['Asset_Name']}\")"]}, {"cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": ["if MAKE_LARGE_PLOTS:\n", "    for i, row in dataset.get_assets_iter():\n", "        fig, axs = plt.subplots(ncols=2, figsize=(20,80 // 14))\n", "        df_asset = assets_train[row['Asset_ID']]\n", "        df_asset['Asset_ID'].isna().cumsum().plot(\n", "            ax=axs[0], \n", "            title=f\"No time data cumsum: {row['Asset_Name']}\"\n", "        )\n", "        df_asset['Target_original_na'].fillna(False).cumsum().plot(\n", "            ax=axs[1], \n", "            title=f\"No target data (original train dataset): {row['Asset_Name']}\"\n", "        )"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Explaining the Target NaNs\n", "\n", "From the plots above (cumulative), it looks like, generally speaking, `Target` NaNs stem from the time gaps.\n", "\n", "We'll evaluate how many `Target` NaNs are there when the prices at times t+1 and t+16 are available. The code exploits the verified fact that if one price is available from HLCO, every other one is for that timepoint."]}, {"cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": ["for _, row in dataset.get_assets_iter():\n", "    df_asset = assets_train[row['Asset_ID']]\n", "    \n", "    df_asset['Close_t+1'] = df_asset['Close'].shift(-1)\n", "    df_asset['Close_t+16'] = df_asset['Close'].shift(-16)\n", "    \n", "    mask_na_with_available_prices = (\n", "        (df_asset['Target_original_na'] == True)\n", "        & (\n", "            (~df_asset['Close_t+1'].isna()) \n", "            & (~df_asset['Close_t+16'].isna())\n", "        )\n", "    )\n", "\n", "    mask_na_no_price = (\n", "        (df_asset['Target_original_na'] == True)\n", "        & (\n", "            (df_asset['Close_t+1'].isna()) \n", "            | (df_asset['Close_t+16'].isna())\n", "        )\n", "    )\n", "\n", "    # mask_na_no_t1 = (\n", "    #     (df_asset['Target_original_na'] == True)\n", "    #     & (df_asset['Close_t+1'].isna())\n", "    # )\n", "\n", "    # mask_na_no_t16 = (\n", "    #     (df_asset['Target_original_na'] == True)\n", "    #     & (df_asset['Close_t+16'].isna())\n", "    # )\n", "\n", "    mask_target_na = df_asset['Target_original_na'] == True\n", "    \n", "    print(\n", "        f\"Asset: {row['Asset_Name']:<16s} | \",\n", "        f\"P(t+1) && P(t+16) exist: {mask_na_with_available_prices.sum():<2d} | \",\n", "        f\"not P(t+1) || not P(t+16): {mask_na_no_price.sum():<6d} | \"\n", "        # f\"not P(t+1): {mask_na_no_t1.sum():<7d}\",\n", "        # f\" | not P(t+16): {mask_na_no_t16.sum():<7d}\"\n", "        f\"Target NaNs: {mask_target_na.sum():<7d}\"\n", "    )"]}, {"cell_type": "markdown", "metadata": {}, "source": ["It looks like NaNs in `Target` come from the time gaps, therefore making it impossible to compute log returns due to inavailability of prices at t+1 and t+16. There are negligible exceptions (<20 per cryptoasset) which might come from boundary cases.\n", "\n", "It also means that the hosts likely interpolated the values at the missing time points and used them to compute the rolling averages that are required for weighted average market returns $M(t)$ and the $\\beta^a$."]}], "metadata": {"interpreter": {"hash": "05981d0fefd8f6aa50eb4dbfe52cc7e5c628fc68afd5a1c09ce41a74ff948e5f"}, "kernelspec": {"display_name": "Python 3.9.7 64-bit ('ds': conda)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7"}, "orig_nbformat": 4}, "nbformat": 4, "nbformat_minor": 2}