{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 目標和進展\n在這個筆記本中，我們嘗試重新創建 [G-Research 加密競賽教程](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition/notebook#) 中描述的目標計算建立你的預測模型）。\n\n此處提供的代碼側重於正確的目標計算，並未進行優化。當前的筆記本版本提供了接近的結果，但並不理想。平均而言，差異很小，但對於某些行給出的值超出了正常目標範圍。我希望我們能在主持人、kaggle 團隊和社區的幫助下解決這個問題。\n\n**從 2021 年 11 月 9 日起更新**\n\n感謝 Ernesto Budia 的調查（參見他的 [notebook](https://www.kaggle.com/ebudia/recreating-target-min-periods-3750/)），我們知道如何獲得更接近的匹配 - 在計算 $\\ beta^a$ 滾動平均 $\\langle .\\rangle$ 應該完成整整 3750 分鐘。\n\n由於這種變化\n平均絕對誤差提高 0.00099 => 0.00006\n最大絕對誤差提高 2.44 => 0.045\n\n**從 2021 年 11 月 12 日起更新**\n\n事實證明，我們一開始使用的樸素方法為 Recreated Target 提供了很多 NA 值，因此 Target 和 Recreated target 之間的比較僅針對部分數據進行（計算平均/最大誤差並不反映真實情況）。另外託管已發布的部分代碼，因此我們嘗試使用此代碼創建更快的當前計算版本。\n\n平均絕對誤差 0.000190\n最大絕對誤差 0.289100\n標準差 0.001135\n\n**2021 年 11 月 17 日更新**\n\n* 在計算回報 R 時使用 x-1 而不是 log(x) - 請參閱 Branden Murray 的 [comment](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778#1582764)\n* 使用唯一的時間戳（並非所有可能的分鐘都像以前的版本） - 檢查行“all_timestamps = np.sort(data['timestamp'].unique())”\n\n最後一個更改意味著所有班次/滾動操作都不是以分鐘為單位，而是以可用記錄為單位。當所有資產都缺少時間戳時，就會出現差異。\n\n平均絕對誤差 0.000190 => 0.000000\n\n最大絕對誤差 0.289100 => 0.003798\n\n標準偏差 0.001135 = 0.000001","metadata":{}},{"cell_type":"markdown","source":"# 幼稚的方法\n在本節中，我們將根據 Target 計算描述進行逐步計算。 它很慢，並且在 Recreated Target 中給出了很多 NA 值。 因此，如果您需要更快/更正確的版本並且不了解它的計算方式，請跳轉到更優化部分。\n\n讓我們從閱讀資產詳細信息、訓練數據和添加時間列開始。","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport gc\n\ndirectory = '../input/g-research-crypto-forecasting'\nfile_path = os.path.join(directory, 'train.csv')\ndtypes = {\n    'timestamp': np.int64,\n    'Asset_ID': np.int8,\n#     'Count': np.int32,\n#     'Open': np.float64,\n#     'High': np.float64,\n#     'Low': np.float64,\n    'Close': np.float64,\n#     'Volume': np.float64,\n#     'VWAP': np.float64,\n    'Target': np.float64,\n}\ndata = pd.read_csv(file_path, dtype=dtypes, usecols=list(dtypes.keys()))\ndata['Time'] = pd.to_datetime(data['timestamp'], unit='s')\n\nfile_path = os.path.join(directory, 'asset_details.csv')\ndetails = pd.read_csv(file_path)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:21:58.175695Z","iopub.execute_input":"2021-11-17T19:21:58.175989Z","iopub.status.idle":"2021-11-17T19:22:41.169916Z","shell.execute_reply.started":"2021-11-17T19:21:58.175959Z","shell.execute_reply":"2021-11-17T19:22:41.169017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"然後按照下面的公式計算回報。\n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\n這是分別為每個資產完成的。 我們不知道應該使用哪個價格。 有五種不同的價格：開盤價、最高價、最低價、收盤價和 VWAP。 可能有一個組合，比如時間 + 1 分鐘的開盤價和時間 + 16 分鐘的收盤價。 我們在下面的計算中使用**收盤**價格。","metadata":{}},{"cell_type":"code","source":"price_column = 'Close'\nids = list(details.Asset_ID)\nchunks = []\nfor id in ids:    \n    asset = data[data.Asset_ID == id].copy()\n    asset.sort_values(by='Time', inplace=True)\n    asset.set_index(keys='Time', inplace=True)\n    asset['p1'] = asset[price_column].shift(freq='-1T')\n    asset['p16'] = asset[price_column].shift(freq='-16T')\n    asset['r'] = np.log(asset.p16/asset.p1)\n    asset.drop(['p1', 'p16'], axis=1, inplace=True)\n    asset.reset_index(inplace=True)\n    chunks.append(asset)\n\ndata = pd.concat(chunks)\ndata.sort_values(by='Time', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:22:41.171696Z","iopub.execute_input":"2021-11-17T19:22:41.172028Z","iopub.status.idle":"2021-11-17T19:22:57.257006Z","shell.execute_reply.started":"2021-11-17T19:22:41.171995Z","shell.execute_reply":"2021-11-17T19:22:57.256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"接下來，為每一行分配權重。 併計算 M(t)。 請注意，所有資產的 M(t) 都是相同的，並且僅取決於時間。\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a}$$\n\n我們不知道是應該為所有資產計算 ${\\sum_a w^a}$ 還是只為在時間 t 有數據的資產計算 ${\\sum_a w^a}$。\n","metadata":{}},{"cell_type":"code","source":"data['w'] = data['Asset_ID'].map(details.set_index(keys='Asset_ID')['Weight'])\nweight_sum = details.Weight.sum()\n\ndata['weighted_asset_r'] = data.w * data.r\ntime_group = data.groupby('Time')\n\nm = time_group['weighted_asset_r'].sum() / time_group['w'].sum()\n#m = time_group['weighted_asset_r'].sum() / weight_sum\n\ndata.set_index(keys=['Time'], inplace=True)\ndata['m'] = m\ndata.reset_index(inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:22:57.258271Z","iopub.execute_input":"2021-11-17T19:22:57.258541Z","iopub.status.idle":"2021-11-17T19:23:00.306049Z","shell.execute_reply.started":"2021-11-17T19:22:57.258509Z","shell.execute_reply":"2021-11-17T19:23:00.305129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"之後，計算 Beta。 括號 $\\langle .\\rangle$ 表示隨時間推移的滾動平均值（3750 分鐘窗口）。 如果沒有完整的 3750 分鐘窗口，$\\beta$ 變為零。\n\n$$\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle}$$","metadata":{}},{"cell_type":"code","source":"data['m2'] = data.m ** 2\ndata['mr'] = data.r * data.m\n\nchunks = []\nfor id in ids:\n    # type: pd.DataFrame\n    asset = data[data.Asset_ID == id].copy()\n    asset.sort_values(by='Time', inplace=True)\n    asset.set_index(keys='Time', inplace=True)\n    asset['mr_rolling'] = asset['mr'].rolling(window='3750T', min_periods=3750).mean()\n    asset['m2_rolling'] = asset['m2'].rolling(window='3750T', min_periods=3750).mean()\n    asset.reset_index(inplace=True)\n    chunks.append(asset)\n    debug = 1\n\ndata = pd.concat(chunks)\ndata.sort_values(by='Time', inplace=True)\ndata['beta'] = data['mr_rolling'] / data['m2_rolling']","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:00.307453Z","iopub.execute_input":"2021-11-17T19:23:00.307778Z","iopub.status.idle":"2021-11-17T19:23:14.635862Z","shell.execute_reply.started":"2021-11-17T19:23:00.307728Z","shell.execute_reply":"2021-11-17T19:23:14.635071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"最後計算目標。\n$$\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$","metadata":{}},{"cell_type":"code","source":"data['Target_recreated'] = data['r'] - data['beta'] * data['m']","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:14.638503Z","iopub.execute_input":"2021-11-17T19:23:14.638826Z","iopub.status.idle":"2021-11-17T19:23:14.790872Z","shell.execute_reply.started":"2021-11-17T19:23:14.638783Z","shell.execute_reply":"2021-11-17T19:23:14.790106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"現在我們比較給定和重新創建的目標。","metadata":{}},{"cell_type":"code","source":"data['Target_diff'] = np.abs(data['Target'] - data['Target_recreated'])\n\nprint(f'Average absolute error {data.Target_diff.mean():8.6f}')\nprint(f'Max absolute error {data.Target_diff.max():8.6f}')\nprint(f'Standard deviation {data.Target_diff.std():8.6f}')","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:14.792534Z","iopub.execute_input":"2021-11-17T19:23:14.792853Z","iopub.status.idle":"2021-11-17T19:23:15.327275Z","shell.execute_reply.started":"2021-11-17T19:23:14.792814Z","shell.execute_reply":"2021-11-17T19:23:15.326338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"正常目標在 [-0.5, 0.96] 範圍內更改，因此最大絕對誤差 2.4 意味著重新創建的目標對於某些記錄是完全錯誤的。","metadata":{}},{"cell_type":"code","source":"data['Target'].agg(['min', 'max'])","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.328367Z","iopub.execute_input":"2021-11-17T19:23:15.328605Z","iopub.status.idle":"2021-11-17T19:23:15.407083Z","shell.execute_reply.started":"2021-11-17T19:23:15.328575Z","shell.execute_reply":"2021-11-17T19:23:15.406208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"讓我們檢查一下我們有多少重新創建的 Target 錯誤（值超出範圍）的記錄。","metadata":{}},{"cell_type":"code","source":"(data.Target_recreated < -0.509351).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.408211Z","iopub.execute_input":"2021-11-17T19:23:15.408416Z","iopub.status.idle":"2021-11-17T19:23:15.454978Z","shell.execute_reply.started":"2021-11-17T19:23:15.408384Z","shell.execute_reply":"2021-11-17T19:23:15.454285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(data.Target_recreated > 0.96417).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.456027Z","iopub.execute_input":"2021-11-17T19:23:15.456574Z","iopub.status.idle":"2021-11-17T19:23:15.503673Z","shell.execute_reply.started":"2021-11-17T19:23:15.456535Z","shell.execute_reply":"2021-11-17T19:23:15.502796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"我嘗試了不同的價格，更改了公式中的時間間隔，將滾動平均值從 3750 分鐘替換為 3750 條最後記錄等，但結果相同或更糟。 有可能最小化最大誤差，但平均值變得更高。\n\n如果您在計算中發現錯誤或對原始 Target 的實際計算方式有所了解，請告訴我。","metadata":{}},{"cell_type":"markdown","source":"事實證明，這段代碼在 Recreated Target 中給出了很多 NA 值，因此最小化 Abs/Max 誤差可能會產生誤導。","metadata":{}},{"cell_type":"code","source":"pd.isna(data.Target).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.50486Z","iopub.execute_input":"2021-11-17T19:23:15.505193Z","iopub.status.idle":"2021-11-17T19:23:15.558513Z","shell.execute_reply.started":"2021-11-17T19:23:15.505149Z","shell.execute_reply":"2021-11-17T19:23:15.557761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.isna(data.Target_recreated).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.559584Z","iopub.execute_input":"2021-11-17T19:23:15.559997Z","iopub.status.idle":"2021-11-17T19:23:15.614092Z","shell.execute_reply.started":"2021-11-17T19:23:15.559956Z","shell.execute_reply":"2021-11-17T19:23:15.613334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Resulting Recreated Target 具有超過 10 倍的 NA 值。 讓我們檢查一下這些 NA 是什麼時候引入的。","metadata":{}},{"cell_type":"code","source":"pd.isna(data.r).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.61527Z","iopub.execute_input":"2021-11-17T19:23:15.615555Z","iopub.status.idle":"2021-11-17T19:23:15.668672Z","shell.execute_reply.started":"2021-11-17T19:23:15.615523Z","shell.execute_reply":"2021-11-17T19:23:15.667963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"在計算 R 時，NA 的數量幾乎等於目標 NA 的數量。差值是 1086。可以用每個資產每個月的 [missing first minite](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286095) 來解釋。\n\n有趣的是，目標存在於火車數據集中，日期為 YYYY-MM-01 23:44:00 (t+16) 和 YYYY-MM-01 23:59:00 (t+1)。對於這些時間，計算算法需要 YYYY-MM-01 00:00:00 時間的價格，但測試數據中沒有這個時間。它可以被認為是Target是在其他數據上計算的證明，然後用已經計算的Target構建訓練數據集。\n\n這個缺少一個月的第一分鐘的錯誤給出了大約 44 個月 x 2（每一行影響輪班 1 和 16 的兩個目標）x 14（資產數量）=> 1232。一些資產沒有所有月份的數據。\n\n唯一可以產生NA的地方是下面的代碼\n> 滾動（窗口='3750T'，min_periods=3750）\n\n3750 分鐘間隔內的缺失值會導致 NA 值。\n\n現在讓我們看一下host提供的[code](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778)：\n> num = df.multiply(mkt.values, axis=0).rolling(window).mean().values\n> denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values\n\n滾動接受兩種可能的窗口類型 - [可以是](https://pandas.pydata.org/pandas-docs/version/1.3.0/reference/api/pandas.DataFrame.rolling.html) shift/int（要移動的數字或行數）或時間偏移量（要移動的分鐘數），但沒有 **min_periods** 參數。這意味著對於 int shift 它根本不會產生 NA，對於時間偏移它可能僅在最後 3750 分鐘沒有數據時才產生 NA。\n\n代碼的下一行計算 beta 並用零刪除所有生成的非數字值：\n> beta = np.nan_to_num(num.T / denom, nan=0., posinf=0., neginf=0.)\n\n這裡的結論是，添加 min_periods=3750 看似改進了我們的指標，實際上添加了許多 NA 值，有效地最小化了這些指標計算的數據。主機代碼顯示滾動平均值不會產生額外的 NA 值。","metadata":{}},{"cell_type":"markdown","source":"# 更優化的版本\n\n在這裡，我們嘗試做兩件事：解決 NA 的問題並使這段代碼運行得更快，以及重用來自主機的代碼。\n來自主機的代碼表明，數據應該以稍微不同的方式重新呈現，以避免昂貴的基於分鐘的輪班/滾動操作，並立即對所有資產執行計算。","metadata":{}},{"cell_type":"markdown","source":"我們首先創建所有時間戳（分鐘）都存在的新數據框。 在火車上，某些資產可能會缺少幾分鐘。We start with creating new data frame where all timestamps (minutes) present. In the train some minutes may be missing for some assets.","metadata":{}},{"cell_type":"code","source":"ids = list(details.Asset_ID)\nasset_names = list(details.Asset_Name)\n\n# times = data['timestamp'].agg(['min', 'max']).to_dict()\n# all_timestamps = np.arange(times['min'], times['max'] + 60, 60)\nall_timestamps = np.sort(data['timestamp'].unique())\ntargets = pd.DataFrame(index=all_timestamps)\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.669765Z","iopub.execute_input":"2021-11-17T19:23:15.670191Z","iopub.status.idle":"2021-11-17T19:23:16.221689Z","shell.execute_reply.started":"2021-11-17T19:23:15.67014Z","shell.execute_reply":"2021-11-17T19:23:16.22088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"接下來，我們計算每個資產的 R 並將其值作為列添加到新的目標數據框。 請注意，某些行將包含 NA，因為可能缺少帶班次的所需價格。","metadata":{}},{"cell_type":"code","source":"for i, id in enumerate(ids):\n    asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n    price = pd.Series(index=all_timestamps, data=asset[price_column])\n#     targets[asset_names[i]] = np.log(\n#         price.shift(periods=-16) /\n#         price.shift(periods=-1)\n#     )\n    targets[asset_names[i]] = (\n        price.shift(periods=-16) /\n        price.shift(periods=-1)\n    ) - 1\n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:16.225319Z","iopub.execute_input":"2021-11-17T19:23:16.22595Z","iopub.status.idle":"2021-11-17T19:23:31.028861Z","shell.execute_reply.started":"2021-11-17T19:23:16.225906Z","shell.execute_reply":"2021-11-17T19:23:31.028036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"接下來計算 M 作為每行的列的加權平均值（當前為每個資產保存 R 值）。 此實現將不可用的 R 值計數為零。 請注意，在計算加權平均值時，這不是處理 NA 值的唯一方法。","metadata":{}},{"cell_type":"code","source":"weights = np.array(list(details.Weight))\ntargets['m'] = np.average(targets.fillna(0), axis=1, weights=weights)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:31.029939Z","iopub.execute_input":"2021-11-17T19:23:31.030159Z","iopub.status.idle":"2021-11-17T19:23:31.416563Z","shell.execute_reply.started":"2021-11-17T19:23:31.030132Z","shell.execute_reply":"2021-11-17T19:23:31.415705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"最後是時候應用主機提供的代碼來計算 beta 和 Target。","metadata":{}},{"cell_type":"code","source":"m = targets['m']\n\nnum = targets.multiply(m.values, axis=0).rolling(3750).mean().values\ndenom = m.multiply(m.values, axis=0).rolling(3750).mean().values\nbeta = np.nan_to_num(num.T / denom, nan=0., posinf=0., neginf=0.)\n\ntargets = targets - (beta * m.values).T","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:31.417648Z","iopub.execute_input":"2021-11-17T19:23:31.417862Z","iopub.status.idle":"2021-11-17T19:23:33.094494Z","shell.execute_reply.started":"2021-11-17T19:23:31.417836Z","shell.execute_reply":"2021-11-17T19:23:33.093663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"讓我們檢查一下它與給定的 Target 有何不同以及我們有多少 NA 值。","metadata":{}},{"cell_type":"code","source":"diffs = []\n\nfor i, id in enumerate(ids):\n    print(asset_names[i])\n    # type: pd.DataFrame\n    asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n    print(f'asset size {asset.shape[0]}')\n    recreated = pd.Series(index=asset.index, data=targets[asset_names[i]])\n    diff = np.abs(asset['Target'] - recreated)\n    diffs.append(diff[~pd.isna(diff)].values)\n    print(f'Average absolute error {diff.mean():8.6f}')\n    print(f'Max absolute error {diff.max():8.6f}')\n    print(f'Standard deviation {diff.std():8.6f}')\n    print(f'Target na {pd.isna(asset.Target).sum()}')\n    print(f'Target_calculated na {pd.isna(recreated).sum()}')\n    print()\n\ndiffs = np.concatenate(diffs, axis=0)\nprint('For all assets')\nprint(f'Average absolute error {diffs.mean():8.6f}')\nprint(f'Max absolute error {diffs.max():8.6f}')\nprint(f'Standard deviation {diffs.std():8.6f}')\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:33.095922Z","iopub.execute_input":"2021-11-17T19:23:33.096359Z","iopub.status.idle":"2021-11-17T19:23:41.767416Z","shell.execute_reply.started":"2021-11-17T19:23:33.096324Z","shell.execute_reply":"2021-11-17T19:23:41.766446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"它仍然不理想，但至少我們的指標是根據幾乎所有可用數據計算得出的，NA 的數量與 Target 中的幾乎相同。\n","metadata":{}},{"cell_type":"markdown","source":"有一個關於 Beta 等於零的討論。 它將目標計算僅轉換為 R 計算。 讓我們看看有多少非數字值以及有多少行 beta=0。","metadata":{}},{"cell_type":"code","source":"beta_ = num.T / denom\n\nfor i, id in enumerate(ids):\n    print(asset_names[i])\n    print(f'Infiinte beta rows {np.isinf(beta_[i]).sum()}')\n    nan_sum = np.isnan(beta_[i]).sum()\n    print(f'NAN beta rows {nan_sum} ({100 * nan_sum / beta_.shape[1]:5.2f}%)')\n\n    eps = 1e-6\n    zero_sum = ((beta[i] > -eps) & (beta[i] < eps)).sum()\n    print(f'Zero beta rows {zero_sum} ({100 * zero_sum / beta.shape[1]:5.2f}%)')\n    print()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:41.768517Z","iopub.execute_input":"2021-11-17T19:23:41.768726Z","iopub.status.idle":"2021-11-17T19:23:42.058484Z","shell.execute_reply.started":"2021-11-17T19:23:41.768695Z","shell.execute_reply":"2021-11-17T19:23:42.057544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below is the same code but in the form of a function you can copy and paste. Note that it does not require Time column, it uses timestamp instead.","metadata":{}},{"cell_type":"code","source":"def calculate_target(data: pd.DataFrame, details: pd.DataFrame, price_column: str):\n    ids = list(details.Asset_ID)\n    asset_names = list(details.Asset_Name)\n    weights = np.array(list(details.Weight))\n\n    all_timestamps = np.sort(data['timestamp'].unique())\n    targets = pd.DataFrame(index=all_timestamps)\n\n    for i, id in enumerate(ids):\n        asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n        price = pd.Series(index=all_timestamps, data=asset[price_column])\n        targets[asset_names[i]] = (\n            price.shift(periods=-16) /\n            price.shift(periods=-1)\n        ) - 1\n    \n    targets['m'] = np.average(targets.fillna(0), axis=1, weights=weights)\n    \n    m = targets['m']\n\n    num = targets.multiply(m.values, axis=0).rolling(3750).mean().values\n    denom = m.multiply(m.values, axis=0).rolling(3750).mean().values\n    beta = np.nan_to_num(num.T / denom, nan=0., posinf=0., neginf=0.)\n\n    targets = targets - (beta * m.values).T\n    targets.drop('m', axis=1, inplace=True)\n    \n    return targets","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:42.060096Z","iopub.execute_input":"2021-11-17T19:23:42.060429Z","iopub.status.idle":"2021-11-17T19:23:42.072548Z","shell.execute_reply.started":"2021-11-17T19:23:42.060388Z","shell.execute_reply":"2021-11-17T19:23:42.071504Z"},"trusted":true},"execution_count":null,"outputs":[]}]}