{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":30894,"databundleVersionId":3147400,"sourceType":"competition"}],"dockerImageVersionId":30587,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto Forecasting\n\n## 1.Problem Definiton\n> We need to use a machine learning model to forecast short term returns in 14 popular cryptocurrencies?\n  Our dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum and our challenge is to predict their future returns.\n    \n## 2.Data\nThe data is dowloaded from the Kaggle's G-Research Crypto Forecasting competition.\n\nThere are 3 main datasets:\n\n* train.csv - The training set, which contains data through the\n* asset_details.csv - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n* example_test.csv - An example of the data that will be delivered by the time series API.\n* supplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the entire submission period.\n\n## 3.Evaluation\n\n* Submissions are evaluated on a weighted version of the Pearson correlation coefficient.\n\nFor more one evaluation of this project check:\n`www.kaggle.com/competitions/g-research-crypto-forecasting/overview/evaluation`\n\n**Note:** The goal for most regression evaluation metrics is to minimize the error.\n\n## 4.Features\n\nKaggle provides a data dicitionary detailing all of the  features of the dataset.\nEach row of the data setin the train.csv has the trading data for an asset, at a given minute timestamp, described in detail below.\n\n* timestamp: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n* Asset_ID: The asset ID corresponding to one of the crytocurrencies (e.g. Asset_ID = 1 for Bitcoin). The mapping from Asset_ID to crypto asset is contained in asset_details.csv.\n* Count: Total number of trades in the time interval (last minute).\n* Open: Opening price of the time interval (in USD).\n* High: Highest price reached during time interval (in USD).\n* Low: Lowest price reached during time interval (in USD).\n* Close: Closing price of the time interval (in USD).\n* Volume: Quantity of asset bought or sold, displayed in base currency USD.\n* VWAP: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n* Target: Residual log-returns for the asset over a 15 minute horizon.\n\nThe first two columns define the time and asset indexes for this data row. The 6 middle columns are feature columns with the trading data for this asset and minute in time. The last column is the prediction target, which we will get to later in more detail.\n\nWe also view the asset information, including the list of all assets, the Asset_ID to asset mapping, and the weight of each asset used to weigh their relative importance in the evaluation metric.\n","metadata":{"execution":{"iopub.status.busy":"2023-11-25T13:55:07.794010Z","iopub.execute_input":"2023-11-25T13:55:07.794472Z","iopub.status.idle":"2023-11-25T13:55:07.868366Z","shell.execute_reply.started":"2023-11-25T13:55:07.794433Z","shell.execute_reply":"2023-11-25T13:55:07.864637Z"}}},{"cell_type":"code","source":"import pandas as pd \nimport numpy as np \nimport time \nimport gc \nfrom sklearn.preprocessing import StandardScaler\nfrom lightgbm import LGBMRegressor\nfrom sklearn.model_selection import TimeSeriesSplit\nfrom scipy.stats import pearsonr\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom datetime import datetime\n\n\nimport warnings\nwarnings.simplefilter(\"ignore\") \n\npd.set_option(\"display.max_columns\", None)\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:58:25.961681Z","iopub.execute_input":"2023-11-29T17:58:25.962118Z","iopub.status.idle":"2023-11-29T17:58:29.217070Z","shell.execute_reply.started":"2023-11-29T17:58:25.962084Z","shell.execute_reply":"2023-11-29T17:58:29.215485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == \"int\":\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        # else:\n            # df[col] = df[col].astype('category')\n           \n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:58:29.220125Z","iopub.execute_input":"2023-11-29T17:58:29.220655Z","iopub.status.idle":"2023-11-29T17:58:29.234478Z","shell.execute_reply.started":"2023-11-29T17:58:29.220613Z","shell.execute_reply":"2023-11-29T17:58:29.233498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading our Training Data","metadata":{}},{"cell_type":"code","source":"data_folder = \"../input/g-research-crypto-forecasting/\"\n!ls $data_folder","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:58:29.235915Z","iopub.execute_input":"2023-11-29T17:58:29.236511Z","iopub.status.idle":"2023-11-29T17:58:29.576973Z","shell.execute_reply.started":"2023-11-29T17:58:29.236476Z","shell.execute_reply":"2023-11-29T17:58:29.576038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details = pd.read_csv(data_folder + \"/asset_details.csv\").sort_values(\"Asset_ID\")\ndf_asset_details","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:58:29.578018Z","iopub.execute_input":"2023-11-29T17:58:29.578258Z","iopub.status.idle":"2023-11-29T17:58:29.664488Z","shell.execute_reply.started":"2023-11-29T17:58:29.578234Z","shell.execute_reply":"2023-11-29T17:58:29.663460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_csv_strict(file_name= data_folder + \"/train.csv\"):\n    df = pd.read_csv(file_name).pipe(reduce_mem_usage)\n    df[\"datetime\"] = pd.to_datetime(df[\"timestamp\"], unit=\"s\")\n    df = df[df[\"datetime\"] >= \"2020-05-13 00:00:00\"]\n    return df","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:58:29.667100Z","iopub.execute_input":"2023-11-29T17:58:29.667648Z","iopub.status.idle":"2023-11-29T17:58:29.672632Z","shell.execute_reply.started":"2023-11-29T17:58:29.667618Z","shell.execute_reply":"2023-11-29T17:58:29.671775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = read_csv_strict()\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:58:29.674134Z","iopub.execute_input":"2023-11-29T17:58:29.674708Z","iopub.status.idle":"2023-11-29T17:59:37.997888Z","shell.execute_reply.started":"2023-11-29T17:58:29.674674Z","shell.execute_reply":"2023-11-29T17:59:37.996666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_supp_train = read_csv_strict(file_name=data_folder + \"/supplemental_train.csv\")\ndf_supp_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:38.000144Z","iopub.execute_input":"2023-11-29T17:59:38.000907Z","iopub.status.idle":"2023-11-29T17:59:45.268243Z","shell.execute_reply.started":"2023-11-29T17:59:38.000872Z","shell.execute_reply":"2023-11-29T17:59:45.266987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = read_csv_strict(file_name=data_folder + \"/example_test.csv\")\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.269595Z","iopub.execute_input":"2023-11-29T17:59:45.269900Z","iopub.status.idle":"2023-11-29T17:59:45.308321Z","shell.execute_reply.started":"2023-11-29T17:59:45.269874Z","shell.execute_reply":"2023-11-29T17:59:45.307071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_train[\"timestamp\"].min(), df_train[\"timestamp\"].max())","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.310212Z","iopub.execute_input":"2023-11-29T17:59:45.311250Z","iopub.status.idle":"2023-11-29T17:59:45.322118Z","shell.execute_reply.started":"2023-11-29T17:59:45.311213Z","shell.execute_reply":"2023-11-29T17:59:45.321137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_supp_train[\"timestamp\"].min(), df_supp_train[\"timestamp\"].max())","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.323156Z","iopub.execute_input":"2023-11-29T17:59:45.323430Z","iopub.status.idle":"2023-11-29T17:59:45.330278Z","shell.execute_reply.started":"2023-11-29T17:59:45.323387Z","shell.execute_reply":"2023-11-29T17:59:45.329394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.concat([df_train, df_supp_train, df_test])\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.331750Z","iopub.execute_input":"2023-11-29T17:59:45.332136Z","iopub.status.idle":"2023-11-29T17:59:45.579390Z","shell.execute_reply.started":"2023-11-29T17:59:45.332067Z","shell.execute_reply":"2023-11-29T17:59:45.578304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.shape, df_supp_train.shape, df_test.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.581126Z","iopub.execute_input":"2023-11-29T17:59:45.582060Z","iopub.status.idle":"2023-11-29T17:59:45.589378Z","shell.execute_reply.started":"2023-11-29T17:59:45.582019Z","shell.execute_reply":"2023-11-29T17:59:45.588178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_supp_train, df_test\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.590893Z","iopub.execute_input":"2023-11-29T17:59:45.591275Z","iopub.status.idle":"2023-11-29T17:59:45.694434Z","shell.execute_reply.started":"2023-11-29T17:59:45.591205Z","shell.execute_reply":"2023-11-29T17:59:45.693170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.memory_usage().sum() / 1024**3","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.699689Z","iopub.execute_input":"2023-11-29T17:59:45.699970Z","iopub.status.idle":"2023-11-29T17:59:45.711941Z","shell.execute_reply.started":"2023-11-29T17:59:45.699949Z","shell.execute_reply":"2023-11-29T17:59:45.710177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Negative volume implies lack of liquidity in the market\ndf_train[df_train.Volume < 0]","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.713077Z","iopub.execute_input":"2023-11-29T17:59:45.713440Z","iopub.status.idle":"2023-11-29T17:59:45.743716Z","shell.execute_reply.started":"2023-11-29T17:59:45.713378Z","shell.execute_reply":"2023-11-29T17:59:45.742591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[df_train.Target.isna()].groupby(\"Asset_ID\")[\"timestamp\"].count()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.745165Z","iopub.execute_input":"2023-11-29T17:59:45.745619Z","iopub.status.idle":"2023-11-29T17:59:45.822492Z","shell.execute_reply.started":"2023-11-29T17:59:45.745580Z","shell.execute_reply":"2023-11-29T17:59:45.821353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[(df_train.Target.isna()) & (df_train.Asset_ID==8)].describe(percentiles=[0.25, 0.50, 0.75, 0.90, 0.95, 0.99])","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.823755Z","iopub.execute_input":"2023-11-29T17:59:45.824012Z","iopub.status.idle":"2023-11-29T17:59:45.937416Z","shell.execute_reply.started":"2023-11-29T17:59:45.823989Z","shell.execute_reply":"2023-11-29T17:59:45.936041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = df_train.sort_values(by=[\"Asset_ID\", \"timestamp\"], ascending=True)\n\ndf_train.head(20)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:45.938707Z","iopub.execute_input":"2023-11-29T17:59:45.938975Z","iopub.status.idle":"2023-11-29T17:59:50.786549Z","shell.execute_reply.started":"2023-11-29T17:59:45.938953Z","shell.execute_reply":"2023-11-29T17:59:50.785237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Candlestick charts \nThe trading data format is an aggregated form of market data including for Open,\nHigh, Low and Close. We can visualize this data through the commonly used \ncandlestick bar chart, which allows traders to perform technical analysis on intraday values. \nThe bar's body length represents the price range between the open and close of that day's trading.\nWhen the bar is red, it means the close was lower than the open, and green otherwise. These are also referred to as bullish and bearish candlesticks. \nThe wicks above and below the bars show the high and low prices of that interval's trading.","metadata":{}},{"cell_type":"code","source":"btc = df_train[df_train[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 is for Bitcoin\nbtc_mini = btc.iloc[-200:] # Select recent data rows","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:50.788004Z","iopub.execute_input":"2023-11-29T17:59:50.788390Z","iopub.status.idle":"2023-11-29T17:59:50.888738Z","shell.execute_reply.started":"2023-11-29T17:59:50.788365Z","shell.execute_reply":"2023-11-29T17:59:50.887648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\n\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:50.890195Z","iopub.execute_input":"2023-11-29T17:59:50.890520Z","iopub.status.idle":"2023-11-29T17:59:51.251782Z","shell.execute_reply.started":"2023-11-29T17:59:50.890493Z","shell.execute_reply":"2023-11-29T17:59:51.250707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# inspecting ethereum which is yet another important asset.\neth = df_train[df_train[\"Asset_ID\"]==6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\neth.info(show_counts =True)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:51.253322Z","iopub.execute_input":"2023-11-29T17:59:51.254608Z","iopub.status.idle":"2023-11-29T17:59:51.373691Z","shell.execute_reply.started":"2023-11-29T17:59:51.254565Z","shell.execute_reply":"2023-11-29T17:59:51.372690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Visualization\nWe will start by visualising the Close prices for the two most popular assets i.e Bitcoin and Ethereum","metadata":{}},{"cell_type":"code","source":"# plotting vwap time series for both chosen assets\nf = plt.figure(figsize=(15,4))\n\nax = f.add_subplot(121)\nplt.plot(btc['Close'], label='BTC')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin')\n\nax2 = f.add_subplot(122)\nax2.plot(eth['Close'], color='red', label='ETH')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:51.375608Z","iopub.execute_input":"2023-11-29T17:59:51.376274Z","iopub.status.idle":"2023-11-29T17:59:53.392005Z","shell.execute_reply.started":"2023-11-29T17:59:51.376233Z","shell.execute_reply":"2023-11-29T17:59:53.391191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like these assets have quite different history, no direct corelation is visible in graphs, but we can check if they correlate in recent times(We will check for the last months upto which data is available)","metadata":{}},{"cell_type":"code","source":"# helper function for conversion from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n\n# creating intervals\nbtc_mini_2021 = btc.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]\neth_mini_2021 = eth.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:53.393096Z","iopub.execute_input":"2023-11-29T17:59:53.394105Z","iopub.status.idle":"2023-11-29T17:59:53.445101Z","shell.execute_reply.started":"2023-11-29T17:59:53.394067Z","shell.execute_reply":"2023-11-29T17:59:53.444047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plotting time series for both chosen assets\nf = plt.figure(figsize=(7,8))\n\nax = f.add_subplot(211)\nplt.plot(btc_mini_2021['Close'], label='btc')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin Close')\n\nax2 = f.add_subplot(212)\nax2.plot(eth_mini_2021['Close'], color='red', label='eth')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum Close')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:53.446943Z","iopub.execute_input":"2023-11-29T17:59:53.447717Z","iopub.status.idle":"2023-11-29T17:59:53.926070Z","shell.execute_reply.started":"2023-11-29T17:59:53.447662Z","shell.execute_reply":"2023-11-29T17:59:53.924783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On shorter intervals we can visually see some potential correlation between both assets, with some simultaneous ups and downs. A better format for analyzing such movements is by calculating asset returns.","metadata":{}},{"cell_type":"markdown","source":"### Log returns\nIn order to analyze price changes for an asset we can deal with the price difference. However, different assets exhibit different price scales, so that the their returns are not readily comparable. We can solve this problem by computing the percentage change in price instead, also known as the return. This return coincides with the percentage change in our invested capital.\n\nReturns are widely used in finance, however log returns are preferred for mathematical modelling of time series, as they are additive across time. Also, while regular returns cannot go below -100%, log returns are not bounded.\n\nTo compute the log return, we can simply take the logarithm of the ratio between two consecutive prices. The first row will have an empty return as the previous value is unknown, therefore the empty return data point will be dropped.\n\n**Source**: Tutorial to the G-Research Crypto Competition","metadata":{}},{"cell_type":"code","source":"# we are making a function which will  compute log returns\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:53.927557Z","iopub.execute_input":"2023-11-29T17:59:53.927829Z","iopub.status.idle":"2023-11-29T17:59:53.933519Z","shell.execute_reply.started":"2023-11-29T17:59:53.927808Z","shell.execute_reply":"2023-11-29T17:59:53.932007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can know visualize the log return for our two assets. ","metadata":{}},{"cell_type":"code","source":"import scipy.stats as stats\n\nlret_btc = log_return(btc_mini_2021.Close)[1:]\nlret_eth = log_return(eth_mini_2021.Close)[1:]\nlret_btc.rename('lret_btc', inplace=True)\nlret_eth.rename('lret_eth', inplace=True)\n\nplt.figure(figsize=(8,4))\nplt.plot(lret_btc);\nplt.plot(lret_eth);\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:53.935289Z","iopub.execute_input":"2023-11-29T17:59:53.935563Z","iopub.status.idle":"2023-11-29T17:59:54.208749Z","shell.execute_reply.started":"2023-11-29T17:59:53.935541Z","shell.execute_reply":"2023-11-29T17:59:54.207320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can see how the signal now looks more like white noise, with less drift than the time series for prices.","metadata":{}},{"cell_type":"markdown","source":"### Correlation between assets¶\nWe hypothesized before that crypto asset returns may exhibit some correlation. we will try to look this in more detail now.\n\nWe will now check the correlation between all assets visualizing the correlation matrix. Note how some assets have much higher pairwise correlation than others.","metadata":{}},{"cell_type":"code","source":"# creating dataframe with returns for all assets\nall_assets_2021 = pd.DataFrame([])\nfor asset_id, asset_name in zip(df_asset_details.Asset_ID, df_asset_details.Asset_Name):\n  asset = df_train[df_train[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n  asset = asset.loc[totimestamp('01/01/2021'):totimestamp('01/05/2021')]\n  asset = asset.reindex(range(asset.index[0],asset.index[-1]+60,60),method='pad')\n  lret = log_return(asset.Close.fillna(0))[1:]\n  all_assets_2021 = all_assets_2021.join(lret, rsuffix=asset_name, how=\"outer\")","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:54.210030Z","iopub.execute_input":"2023-11-29T17:59:54.210270Z","iopub.status.idle":"2023-11-29T17:59:55.927648Z","shell.execute_reply.started":"2023-11-29T17:59:54.210249Z","shell.execute_reply":"2023-11-29T17:59:55.926727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_asset_details.Asset_ID, df_asset_details.Asset_Name)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:55.928848Z","iopub.execute_input":"2023-11-29T17:59:55.929185Z","iopub.status.idle":"2023-11-29T17:59:55.937168Z","shell.execute_reply.started":"2023-11-29T17:59:55.929157Z","shell.execute_reply":"2023-11-29T17:59:55.936088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_assets_2021.corr()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:55.938629Z","iopub.execute_input":"2023-11-29T17:59:55.939621Z","iopub.status.idle":"2023-11-29T17:59:56.034932Z","shell.execute_reply.started":"2023-11-29T17:59:55.939559Z","shell.execute_reply":"2023-11-29T17:59:56.033554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.heatmap(all_assets_2021.corr(),\n            xticklabels = df_asset_details.Asset_Name.values,\n            yticklabels = df_asset_details.Asset_Name.values)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:56.036751Z","iopub.execute_input":"2023-11-29T17:59:56.037096Z","iopub.status.idle":"2023-11-29T17:59:56.480251Z","shell.execute_reply.started":"2023-11-29T17:59:56.037070Z","shell.execute_reply":"2023-11-29T17:59:56.479364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dealing with missing data","metadata":{}},{"cell_type":"code","source":"df_train.info()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:56.482009Z","iopub.execute_input":"2023-11-29T17:59:56.482953Z","iopub.status.idle":"2023-11-29T17:59:56.498306Z","shell.execute_reply.started":"2023-11-29T17:59:56.482883Z","shell.execute_reply":"2023-11-29T17:59:56.495901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"total missing: \")\nprint(df_train.isna().sum())\nprint(\"----------------------------\")\nprint(\"percent missing: \")\nprint(df_train.isna().sum() * 100. / df_train.shape[0])","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:56.500576Z","iopub.execute_input":"2023-11-29T17:59:56.501260Z","iopub.status.idle":"2023-11-29T17:59:56.792329Z","shell.execute_reply.started":"2023-11-29T17:59:56.501204Z","shell.execute_reply":"2023-11-29T17:59:56.790943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:56.793865Z","iopub.execute_input":"2023-11-29T17:59:56.794176Z","iopub.status.idle":"2023-11-29T17:59:56.814225Z","shell.execute_reply.started":"2023-11-29T17:59:56.794146Z","shell.execute_reply":"2023-11-29T17:59:56.812630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ffill for each group for Target missing value","metadata":{}},{"cell_type":"code","source":"df_train[\"Target\"] = df_train.sort_values([\"Asset_ID\", \"timestamp\"]).groupby(\"Asset_ID\")[\"Target\"].ffill()\ndf_train[\"VWAP\"] = df_train.sort_values([\"Asset_ID\", \"timestamp\"]).groupby(\"Asset_ID\")[\"VWAP\"].ffill()\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T17:59:56.815779Z","iopub.execute_input":"2023-11-29T17:59:56.816136Z","iopub.status.idle":"2023-11-29T18:00:02.797131Z","shell.execute_reply.started":"2023-11-29T17:59:56.816102Z","shell.execute_reply":"2023-11-29T18:00:02.795890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:02.798877Z","iopub.execute_input":"2023-11-29T18:00:02.799690Z","iopub.status.idle":"2023-11-29T18:00:02.941612Z","shell.execute_reply.started":"2023-11-29T18:00:02.799651Z","shell.execute_reply":"2023-11-29T18:00:02.940476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[ (df_train[\"Target\"].isna()) & (df_train[\"Asset_ID\"] == 10)]","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:02.943376Z","iopub.execute_input":"2023-11-29T18:00:02.943952Z","iopub.status.idle":"2023-11-29T18:00:03.032886Z","shell.execute_reply.started":"2023-11-29T18:00:02.943914Z","shell.execute_reply":"2023-11-29T18:00:03.031525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[\"Target\"] = df_train[\"Target\"].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:03.034543Z","iopub.execute_input":"2023-11-29T18:00:03.034849Z","iopub.status.idle":"2023-11-29T18:00:03.098316Z","shell.execute_reply.started":"2023-11-29T18:00:03.034823Z","shell.execute_reply":"2023-11-29T18:00:03.096510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[ (df_train[\"Target\"].isna()) & (df_train[\"Asset_ID\"] == 10)]","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:03.100085Z","iopub.execute_input":"2023-11-29T18:00:03.100443Z","iopub.status.idle":"2023-11-29T18:00:03.152582Z","shell.execute_reply.started":"2023-11-29T18:00:03.100418Z","shell.execute_reply":"2023-11-29T18:00:03.151293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[\"Target\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:03.156017Z","iopub.execute_input":"2023-11-29T18:00:03.156447Z","iopub.status.idle":"2023-11-29T18:00:03.193807Z","shell.execute_reply.started":"2023-11-29T18:00:03.156396Z","shell.execute_reply":"2023-11-29T18:00:03.192832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utility Functions","metadata":{}},{"cell_type":"code","source":"# Two new features from the competition tutorial\ndef upper_shadow(df):\n    return df[\"High\"] - np.maximum(df[\"Close\"], df[\"Open\"])\n\ndef lower_shadow(df):\n    return np.minimum(df[\"Close\"], df[\"Open\"]) - df[\"Low\"]\n\n## notebook: crypto-prediction-technical-analysis-features\ndef SM_A_M(df, colname, n):\n    mean = df[colname].rolling(window=n).mean()\n    median = df[colname].rolling(window=n).median()\n    \n    return mean, median\n\ndef EMA1(df, colname, n):\n    ## EMA\n    \"\"\"\n    https://qiita.com/MuAuan/items/b08616a841be25d29817\n    \"\"\"\n    a= 2/(n+1)\n    return df[colname].ewm(alpha=a).mean()\n\ndef MACD(df, colname, span1=12, span2=26, span3=9):\n    ## MACD\n    \"\"\"\n    Compute MACD\n    # https://www.learnpythonwithrune.org/pandas-calculate-the-moving-average-convergence-divergence-macd-for-a-stock/\n    \"\"\"\n    \n    exp1 = EMA1(df, colname, span1)\n    exp2 = EMA1(df, colname, span2)\n    macd = 100 * (exp1 - exp2) / exp2\n    signal = macd.ewm(alpha=2./(span3+1)).mean() ##EMA1(df, colname, macd, span3)\n\n    return macd, signal\n\ndef BollingerBand(df, colname, window, no_of_std):\n    mean = df[colname].rolling(window=window).mean()\n    std = df[colname].rolling(window=window).std() \n    bb_high = mean + no_of_std * std\n    bb_low = mean - no_of_std * std\n    \n    return bb_high, bb_low, std\n## RSI\ndef rsiFunc(df, colname, n=14):\n    prices = df[colname].values\n    \n    deltas = np.diff(prices)\n    seed = deltas[:n+1]\n    up = seed[seed>=0].sum()/n\n    down = -seed[seed<0].sum()/n\n    rs = up/down\n    rsi = np.zeros_like(prices)\n    rsi[:n] = 100. - 100./(1.+rs)\n\n    for i in range(n, len(prices)):\n        delta = deltas[i-1] # cause the diff is 1 shorter\n\n        if delta>0:\n            upval = delta\n            downval = 0.\n        else:\n            upval = 0.\n            downval = -delta\n\n        up = (up*(n-1) + upval)/n\n        down = (down*(n-1) + downval)/n\n\n        rs = up/down\n        rsi[i] = 100. - 100./(1.+rs)\n\n    return rsi\n\n## https://stackoverflow.com/questions/42138357/pandas-rolling-slope-calculation\ndef calc_slope(x):\n    slope = np.polyfit(range(len(x)), x, 1)[0]\n    return slope","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:03.195705Z","iopub.execute_input":"2023-11-29T18:00:03.196119Z","iopub.status.idle":"2023-11-29T18:00:03.211666Z","shell.execute_reply.started":"2023-11-29T18:00:03.196085Z","shell.execute_reply":"2023-11-29T18:00:03.209859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"code","source":"def get_feat(df):\n    df[\"High-Low\"] = df[\"High\"] - df[\"Low\"]\n    df[\"Close-Open\"] = df[\"Close\"] - df[\"Open\"]\n\n    df[\"dayofweek\"] = df[\"datetime\"].dt.dayofweek\n    df[\"weekofyear\"] = df[\"datetime\"].dt.isocalendar().week\n ## df[\"weekofyear\"] = df[\"datetime\"].dt.day_name()\n##df['weekofyear'] = df['datetime'].dt.isocalendar().week\n\n    df[\"dayofweek\"] = df[\"dayofweek\"].astype(np.int32)\n    df[\"weekofyear\"] = df[\"weekofyear\"].astype(np.int32)\n  \n\n    df = df.drop([\"datetime\", \"VWAP\", \"Open\", \"High\", \"Low\"], axis=1)\n    gc.collect()\n   \n\n    macd, signal = MACD(df=df, colname=\"Close\")\n    df['MACD_' + \"Close\" + '_macd'] = macd\n    df['MACD_' + \"Close\" + '_signal'] = signal\n\n    df['MACD_' + \"Close\" + '_macd'] = df['MACD_' + \"Close\" + '_macd'].astype(np.float32)\n    df['MACD_' + \"Close\" + '_signal'] = df['MACD_' + \"Close\" + '_signal'].astype(np.float32)\n\n    arr_n = [5, 10, 15]\n  \n    arr_cols = [\"Close\", \"Volume\"]\n    no_of_std = 2.5\n\n    for colname in arr_cols: \n        print(\"colname = \" + colname)\n        for n in arr_n:\n            print(\"n = \"+ str(n))\n            df['Target_lag_' + str(n)] = df['Target'].shift(n)\n            df['Target_lag_' + str(n)].fillna(value=df[\"Target\"], inplace=True)\n            \n\n\n            print(\"SMA\")\n            mean, median = SM_A_M(df=df, colname=colname, n=n)\n            df['SM_' + colname + '_mean_' + str(n)] = mean\n            df['SM_' + colname + '_median_' + str(n)] = median\n\n            df['SM_' + colname + '_mean_' + str(n)] = df['SM_' + colname + '_mean_' + str(n)].astype(np.float32)\n            df['SM_' + colname + '_median_' + str(n)] = df['SM_' + colname + '_median_' + str(n)].astype(np.float32)\n\n            df['SM_' + colname + '_mean_' + str(n)].fillna(value=df[colname], inplace=True)\n            df['SM_' + colname + '_median_' + str(n)].fillna(value=df[colname], inplace=True)\n            \n\n            print(\"EMA1\")\n            ewmean = EMA1(df=df_train, colname=colname, n=n)\n            df['EWM_' + colname + '_mean_' + str(n)] = ewmean\n            df['EWM_' + colname + '_mean_' + str(n)] = df['EWM_' + colname + '_mean_' + str(n)].astype(np.float32)\n            df['EWM_' + colname + '_mean_' + str(n)].fillna(value=1, inplace=True)\n            \n            if colname in ['VWAP', 'Volume']:\n                continue\n\n            if n==5:\n                continue\n                \n            print(\"RSI\")\n            rsi = rsiFunc(df=df,  colname=colname, n=n)\n            df['RSI_' + str(n)] = rsi\n            df['RSI_' + str(n)] = df['RSI_' + str(n)].astype(np.float32)\n            df['RSI_' + str(n)].fillna(value=1, inplace=True)\n            \n\n            print(\"BollingerBand\")\n            bb_high, bb_low, std = BollingerBand(df=df, colname=colname, window=n, no_of_std=no_of_std)\n            df['SM_' + colname + '_std_' + str(n)] = std\n            df['SM_' + colname + '_BB_High_' + str(n)] = bb_high\n            df['SM_' + colname + '_BB_Low_' + str(n)] = bb_low\n\n            df['SM_' + colname + '_std_' + str(n)] = df['SM_' + colname + '_std_' + str(n)].astype(np.float32)\n            df['SM_' + colname + '_BB_High_' + str(n)] = df['SM_' + colname + '_BB_High_' + str(n)].astype(np.float32)\n            df['SM_' + colname + '_BB_Low_' + str(n)] = df['SM_' + colname + '_BB_Low_' + str(n)].astype(np.float32)\n            df['SM_' + colname + '_std_' + str(n)].fillna(value=0, inplace=True)\n            df['SM_' + colname + '_BB_High_' + str(n)].fillna(value=df[colname], inplace=True) \n            df['SM_' + colname + '_BB_Low_' + str(n)].fillna(value=df[colname], inplace=True) \n            \n            \n            gc.collect()\n\n    df = reduce_mem_usage(df)\n\n    df.head()\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:03.220729Z","iopub.execute_input":"2023-11-29T18:00:03.221095Z","iopub.status.idle":"2023-11-29T18:00:03.241420Z","shell.execute_reply.started":"2023-11-29T18:00:03.221067Z","shell.execute_reply":"2023-11-29T18:00:03.240046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.describe()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:03.243046Z","iopub.execute_input":"2023-11-29T18:00:03.243716Z","iopub.status.idle":"2023-11-29T18:00:08.174473Z","shell.execute_reply.started":"2023-11-29T18:00:03.243682Z","shell.execute_reply":"2023-11-29T18:00:08.172894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train=get_feat(df=df_train)\ndf_train.memory_usage(deep=True)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:00:08.175929Z","iopub.execute_input":"2023-11-29T18:00:08.176493Z","iopub.status.idle":"2023-11-29T18:01:45.632338Z","shell.execute_reply.started":"2023-11-29T18:00:08.176465Z","shell.execute_reply":"2023-11-29T18:01:45.630811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.info(verbose=False,memory_usage='deep')","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:45.634069Z","iopub.execute_input":"2023-11-29T18:01:45.634505Z","iopub.status.idle":"2023-11-29T18:01:45.646006Z","shell.execute_reply.started":"2023-11-29T18:01:45.634470Z","shell.execute_reply":"2023-11-29T18:01:45.643925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:45.647951Z","iopub.execute_input":"2023-11-29T18:01:45.648379Z","iopub.status.idle":"2023-11-29T18:01:45.836014Z","shell.execute_reply.started":"2023-11-29T18:01:45.648342Z","shell.execute_reply":"2023-11-29T18:01:45.834684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import memory_profiler\nm1 = memory_profiler.memory_usage()\nm1","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:45.839211Z","iopub.execute_input":"2023-11-29T18:01:45.839719Z","iopub.status.idle":"2023-11-29T18:01:45.956895Z","shell.execute_reply.started":"2023-11-29T18:01:45.839683Z","shell.execute_reply":"2023-11-29T18:01:45.955551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:45.961775Z","iopub.execute_input":"2023-11-29T18:01:45.962276Z","iopub.status.idle":"2023-11-29T18:01:46.011298Z","shell.execute_reply.started":"2023-11-29T18:01:45.962237Z","shell.execute_reply":"2023-11-29T18:01:46.009757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:46.013641Z","iopub.execute_input":"2023-11-29T18:01:46.014088Z","iopub.status.idle":"2023-11-29T18:01:46.137029Z","shell.execute_reply.started":"2023-11-29T18:01:46.014048Z","shell.execute_reply":"2023-11-29T18:01:46.135888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = df_train[~((df_train[\"group_num\"].isna()) & (df_train[\"row_id\"].isna()))]\nprint(df_test.shape)\n\ngc.collect()\n\ndf_train.drop(  df_train[~((df_train[\"group_num\"].isna()) & (df_train[\"row_id\"].isna()))].index, inplace=True)\ny_train = df_train['Target']\nprint(df_train.shape)\ny_test = df_test['Target']\ndf_test.drop([\"Target\"], axis=1, inplace=True)\ndf_train.drop([\"group_num\", \"row_id\"], axis=1, inplace=True)\n\ndf_train.shape, df_test.shape\n","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:46.138372Z","iopub.execute_input":"2023-11-29T18:01:46.138725Z","iopub.status.idle":"2023-11-29T18:01:53.993719Z","shell.execute_reply.started":"2023-11-29T18:01:46.138693Z","shell.execute_reply":"2023-11-29T18:01:53.992621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:53.995178Z","iopub.execute_input":"2023-11-29T18:01:53.995674Z","iopub.status.idle":"2023-11-29T18:01:54.041848Z","shell.execute_reply.started":"2023-11-29T18:01:53.995646Z","shell.execute_reply":"2023-11-29T18:01:54.039696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.043604Z","iopub.execute_input":"2023-11-29T18:01:54.043977Z","iopub.status.idle":"2023-11-29T18:01:54.092056Z","shell.execute_reply.started":"2023-11-29T18:01:54.043949Z","shell.execute_reply":"2023-11-29T18:01:54.090703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.093634Z","iopub.execute_input":"2023-11-29T18:01:54.094064Z","iopub.status.idle":"2023-11-29T18:01:54.635334Z","shell.execute_reply.started":"2023-11-29T18:01:54.094009Z","shell.execute_reply":"2023-11-29T18:01:54.633641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Standardization\nWe will now standarize the input data. \nStandarization is the process of putting different variables on the same scale. \nIn regression analysis, it is often crucial to standarize our independent variables or \nwe may risk obtaining misleading results.","metadata":{}},{"cell_type":"code","source":"# X_train = X_train.drop('timestamp',axis = 1)\n# X_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.636717Z","iopub.execute_input":"2023-11-29T18:01:54.637268Z","iopub.status.idle":"2023-11-29T18:01:54.981621Z","shell.execute_reply.started":"2023-11-29T18:01:54.637242Z","shell.execute_reply":"2023-11-29T18:01:54.979945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.982673Z","iopub.status.idle":"2023-11-29T18:01:54.983030Z","shell.execute_reply.started":"2023-11-29T18:01:54.982851Z","shell.execute_reply":"2023-11-29T18:01:54.982868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# from sklearn.preprocessing import StandardScaler\n# X_test = df_test.drop(['group_num','row_id','timestamp'], axis=1)\n# # simple preprocessing of the data \n# scaler = StandardScaler() #using StandardScalar() as it is Robust scalar\n# X_train_scaled = scaler.fit_transform(X_train)\n# X_test_scaled = scaler.transform(X_test)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:02:18.620254Z","iopub.execute_input":"2023-11-29T18:02:18.620748Z","iopub.status.idle":"2023-11-29T18:02:18.656637Z","shell.execute_reply.started":"2023-11-29T18:02:18.620710Z","shell.execute_reply":"2023-11-29T18:02:18.654496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_train_scaled.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.985748Z","iopub.status.idle":"2023-11-29T18:01:54.986063Z","shell.execute_reply.started":"2023-11-29T18:01:54.985903Z","shell.execute_reply":"2023-11-29T18:01:54.985917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_test_scaled.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.986936Z","iopub.status.idle":"2023-11-29T18:01:54.987236Z","shell.execute_reply.started":"2023-11-29T18:01:54.987092Z","shell.execute_reply":"2023-11-29T18:01:54.987106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Models\n# Linear Regression","metadata":{}},{"cell_type":"code","source":"# from sklearn.linear_model import LinearRegression\n\n# # implement basic ML baseline \n# lr = LinearRegression()\n# lr.fit(X_train_scaled,y_train)\n# y_pred_lr = lr.predict(X_test_scaled)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.988697Z","iopub.status.idle":"2023-11-29T18:01:54.988986Z","shell.execute_reply.started":"2023-11-29T18:01:54.988848Z","shell.execute_reply":"2023-11-29T18:01:54.988863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Evaluating our model with different metrics","metadata":{}},{"cell_type":"code","source":"\n\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Training With LGBM","metadata":{}},{"cell_type":"code","source":"# 10-fold time series cross validation\ndef timecv_model(model, X, y):\n    tfold = TimeSeriesSplit(n_splits = 5)\n    pcc_list = []\n    for _, (train_index, test_index) in tqdm(enumerate(tfold.split(X), start=1)):\n        X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n        y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n        clf = model.fit(X_train, y_train)\n        pred = clf.predict(X_test)\n        pcc = pearsonr(pred, y_test) \n        pcc_list.append(pcc[0])\n        gc.collect()\n    \n    return pcc_list\n\ndef cv_result(model, X, y):\n    model_name = model.__class__.__name__\n    pcc_ = timecv_model(model, X, y)\n    for i, pcc in enumerate(pcc_):\n        print(f'{i}th fold: {model_name} PCC: {pcc:.4f}')\n    print(f'\\n{model_name} average PCC: {np.mean(pcc_):.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:02:56.840664Z","iopub.execute_input":"2023-11-29T18:02:56.842167Z","iopub.status.idle":"2023-11-29T18:02:56.853330Z","shell.execute_reply.started":"2023-11-29T18:02:56.842115Z","shell.execute_reply":"2023-11-29T18:02:56.852067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_lgb(model, X, y):\n    # print(\"model fitting ...\")\n    # clf = model.fit(X, y)\n    # print(\"predicting ...\")\n    # pred = clf.predict(X)\n    # pcc = pearsonr(pred, y)[0]\n    \n    # print(\"pcc = \", str(pcc))\n    # gc.collect()\n    cv_result(model,X,y)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:03:00.907871Z","iopub.execute_input":"2023-11-29T18:03:00.908319Z","iopub.status.idle":"2023-11-29T18:03:00.915154Z","shell.execute_reply.started":"2023-11-29T18:03:00.908294Z","shell.execute_reply":"2023-11-29T18:03:00.913740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_model = LGBMRegressor(n_estimators = 576,\n                          max_depth = 17,\n                          num_leaves = 35,\n                          learning_rate = 0.06891972238739223, \n                          min_child_samples = 47, \n                          colsample_bytree=0.4000480946836777,\n                          subsample = 0.6058272745943716,\n                          seed = 42)\n\n\ny = df_train[\"Target\"]\nX = df_train.drop(['timestamp', \"Target\"], axis=1)\n\ngc.collect()\n\nmodel = train_lgb(lgb_model, X, y)\nmodel","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:03:23.239958Z","iopub.execute_input":"2023-11-29T18:03:23.240473Z","iopub.status.idle":"2023-11-29T18:17:49.648445Z","shell.execute_reply.started":"2023-11-29T18:03:23.240434Z","shell.execute_reply":"2023-11-29T18:17:49.647545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:03:12.641998Z","iopub.status.idle":"2023-11-29T18:03:12.642445Z","shell.execute_reply.started":"2023-11-29T18:03:12.642221Z","shell.execute_reply":"2023-11-29T18:03:12.642240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# y.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:03:12.644045Z","iopub.status.idle":"2023-11-29T18:03:12.644438Z","shell.execute_reply.started":"2023-11-29T18:03:12.644242Z","shell.execute_reply":"2023-11-29T18:03:12.644257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_prediction(row):\n    y_pred = df_test.loc[((df_test[\"timestamp\"] == row[\"timestamp\"]) & (df_test[\"Asset_ID\"] == row[\"Asset_ID\"])), \"Target\"]\n    print(y_pred.values[0])\n    return y_pred.values[0]","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.996459Z","iopub.status.idle":"2023-11-29T18:01:54.996749Z","shell.execute_reply.started":"2023-11-29T18:01:54.996608Z","shell.execute_reply":"2023-11-29T18:01:54.996623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict and Submit","metadata":{}},{"cell_type":"code","source":"df_pred = pd.read_csv(data_folder + \"/example_test.csv\")\ndf_pred.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.997668Z","iopub.status.idle":"2023-11-29T18:01:54.997954Z","shell.execute_reply.started":"2023-11-29T18:01:54.997817Z","shell.execute_reply":"2023-11-29T18:01:54.997831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_sub = pd.read_csv(data_folder+\"/example_test.csv\")\ndf_sample_sub.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:54.999050Z","iopub.status.idle":"2023-11-29T18:01:54.999358Z","shell.execute_reply.started":"2023-11-29T18:01:54.999205Z","shell.execute_reply":"2023-11-29T18:01:54.999221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.columns","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.000195Z","iopub.status.idle":"2023-11-29T18:01:55.000502Z","shell.execute_reply.started":"2023-11-29T18:01:55.000334Z","shell.execute_reply":"2023-11-29T18:01:55.000348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = df_test.drop([\"group_num\", \"row_id\", \"timestamp\"], axis=1)\ndf_test[\"Target\"] = model.predict(X_test)\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.001594Z","iopub.status.idle":"2023-11-29T18:01:55.001882Z","shell.execute_reply.started":"2023-11-29T18:01:55.001745Z","shell.execute_reply":"2023-11-29T18:01:55.001760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test[[\"group_num\", \"row_id\", \"Target\"]].to_csv('submission01.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.003267Z","iopub.status.idle":"2023-11-29T18:01:55.003890Z","shell.execute_reply.started":"2023-11-29T18:01:55.003623Z","shell.execute_reply":"2023-11-29T18:01:55.003647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result=pd.read_csv('submission01.csv')\nplt.scatter(result['row_id'],result['Target'],cmap='hsv')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.005191Z","iopub.status.idle":"2023-11-29T18:01:55.005563Z","shell.execute_reply.started":"2023-11-29T18:01:55.005376Z","shell.execute_reply":"2023-11-29T18:01:55.005391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.006428Z","iopub.status.idle":"2023-11-29T18:01:55.006737Z","shell.execute_reply.started":"2023-11-29T18:01:55.006593Z","shell.execute_reply":"2023-11-29T18:01:55.006608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_pred[\"Target\"] = 0.0\n\nfor j, row in df_test.iterrows():\n    print(row[\"Target\"])\n    df_pred.loc[((df_pred[\"group_num\"] == int(row[\"group_num\"])) & (df_pred[\"row_id\"] == int(row[\"row_id\"]))), \"Target\"] = row[\"Target\"]\n    print(\"pred \" + str(row[\"row_id\"])  + \" \" + str(df_pred.loc[df_pred[\"row_id\"] == row[\"row_id\"], \"Target\"]))","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.007496Z","iopub.status.idle":"2023-11-29T18:01:55.007784Z","shell.execute_reply.started":"2023-11-29T18:01:55.007645Z","shell.execute_reply":"2023-11-29T18:01:55.007659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_pred","metadata":{"execution":{"iopub.status.busy":"2023-11-29T18:01:55.009056Z","iopub.status.idle":"2023-11-29T18:01:55.009362Z","shell.execute_reply.started":"2023-11-29T18:01:55.009209Z","shell.execute_reply":"2023-11-29T18:01:55.009226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}