{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Goals and progress\nIn this notebook we try to recreate Target calculation described in [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition/notebook#Building-your-prediction-model).\n\nThe code presented here is focused on proper Target calculation and is not optimized. The current notebook version provides close result, but not ideal. In average the difference is quite small, but for some rows gives values outside of normal Target range. I hope we could fix this with help of host, kaggle team and community.\n\n**Update from 9-Nov-2021**\n\nThanks to investigation from Ernesto Budia (see his [notebook](https://www.kaggle.com/ebudia/recreating-target-min-periods-3750/)) we know how to get much closer match - when calculating $\\beta^a$ rolling average $\\langle .\\rangle$ should be done for full 3750 minutes.\n\nAs a result of this change  \nAverage absolute error improved 0.00099 => 0.00006  \nMax absolute error improved 2.44 => 0.045\n\n**Update from 12-Nov-2021**\n\nIt turns out that naive approach we used at the beginning gives a lot of NA values for Recreated Target, so comparison between Target and Recreated target was done only for part of data (calculating Average/Max error does not reflect real picture). Additionally host published part of code, so we try to create faster version of current calculations using this code.\n\nAverage absolute error 0.000190  \nMax absolute error 0.289100  \nStandard deviation 0.001135  \n\n**Update from 17 Nov-2021**\n\n* use x-1 instead of log(x) when calculating return R - see [comment](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778#1582764) by Branden Murray\n* use unique timestamps (not all possible minutes as in previous version) - check line \"all_timestamps = np.sort(data['timestamp'].unique())\"\n\nThe last change means that all shift/rolling operations are done not in terms of minutes, but in terms of available records. The difference appears when a timestamp is missing for all assets.\n\nAverage absolute error 0.000190 => 0.000000  \nMax absolute error 0.289100 => 0.003798  \nStandard deviation 0.001135 = 0.000001  ","metadata":{}},{"cell_type":"markdown","source":"# Naive approach\nIn this section we are implementing step by step calculation based on Target calculation description. It is slow and gives a lot of NA values in Recreated Target. So if you need faster/more correct version and not understanding of how it is calculated, jump to More optimized section.\n\nLet's start with reading assets details, train data and adding Time column.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport gc\n\ndirectory = '../input/g-research-crypto-forecasting'\nfile_path = os.path.join(directory, 'train.csv')\ndtypes = {\n    'timestamp': np.int64,\n    'Asset_ID': np.int8,\n#     'Count': np.int32,\n#     'Open': np.float64,\n#     'High': np.float64,\n#     'Low': np.float64,\n    'Close': np.float64,\n#     'Volume': np.float64,\n#     'VWAP': np.float64,\n    'Target': np.float64,\n}\ndata = pd.read_csv(file_path, dtype=dtypes, usecols=list(dtypes.keys()))\ndata['Time'] = pd.to_datetime(data['timestamp'], unit='s')\n\nfile_path = os.path.join(directory, 'asset_details.csv')\ndetails = pd.read_csv(file_path)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:21:58.175695Z","iopub.execute_input":"2021-11-17T19:21:58.175989Z","iopub.status.idle":"2021-11-17T19:22:41.169916Z","shell.execute_reply.started":"2021-11-17T19:21:58.175959Z","shell.execute_reply":"2021-11-17T19:22:41.169017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then calculate return as per formula below. \n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\nThis is done for each asset separately. We do not know which price should be used. There are five different prices: Open, High, Low, Close, and VWAP. Probably there is a mix like Open price for time + 1 min, and Close for time + 16 min. We use **Close** price in calculation below.","metadata":{}},{"cell_type":"code","source":"price_column = 'Close'\nids = list(details.Asset_ID)\nchunks = []\nfor id in ids:    \n    asset = data[data.Asset_ID == id].copy()\n    asset.sort_values(by='Time', inplace=True)\n    asset.set_index(keys='Time', inplace=True)\n    asset['p1'] = asset[price_column].shift(freq='-1T')\n    asset['p16'] = asset[price_column].shift(freq='-16T')\n    asset['r'] = np.log(asset.p16/asset.p1)\n    asset.drop(['p1', 'p16'], axis=1, inplace=True)\n    asset.reset_index(inplace=True)\n    chunks.append(asset)\n\ndata = pd.concat(chunks)\ndata.sort_values(by='Time', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:22:41.171696Z","iopub.execute_input":"2021-11-17T19:22:41.172028Z","iopub.status.idle":"2021-11-17T19:22:57.257006Z","shell.execute_reply.started":"2021-11-17T19:22:41.171995Z","shell.execute_reply":"2021-11-17T19:22:57.256000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, assign weight for each row. And calculate M(t). Note that M(t) is the same for all assets and depend only on time.\n\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a}$$\n\nWe do not know if ${\\sum_a w^a}$ should be calculated for all assets or only for assets having data at time t.\n","metadata":{}},{"cell_type":"code","source":"data['w'] = data['Asset_ID'].map(details.set_index(keys='Asset_ID')['Weight'])\nweight_sum = details.Weight.sum()\n\ndata['weighted_asset_r'] = data.w * data.r\ntime_group = data.groupby('Time')\n\nm = time_group['weighted_asset_r'].sum() / time_group['w'].sum()\n#m = time_group['weighted_asset_r'].sum() / weight_sum\n\ndata.set_index(keys=['Time'], inplace=True)\ndata['m'] = m\ndata.reset_index(inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:22:57.258271Z","iopub.execute_input":"2021-11-17T19:22:57.258541Z","iopub.status.idle":"2021-11-17T19:23:00.306049Z","shell.execute_reply.started":"2021-11-17T19:22:57.258509Z","shell.execute_reply":"2021-11-17T19:23:00.305129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After that, Beta is calculated. Bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows). If there is no full 3750 minute window, $\\beta$ becomes zero.\n\n$$\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle}$$","metadata":{}},{"cell_type":"code","source":"data['m2'] = data.m ** 2\ndata['mr'] = data.r * data.m\n\nchunks = []\nfor id in ids:\n    # type: pd.DataFrame\n    asset = data[data.Asset_ID == id].copy()\n    asset.sort_values(by='Time', inplace=True)\n    asset.set_index(keys='Time', inplace=True)\n    asset['mr_rolling'] = asset['mr'].rolling(window='3750T', min_periods=3750).mean()\n    asset['m2_rolling'] = asset['m2'].rolling(window='3750T', min_periods=3750).mean()\n    asset.reset_index(inplace=True)\n    chunks.append(asset)\n    debug = 1\n\ndata = pd.concat(chunks)\ndata.sort_values(by='Time', inplace=True)\ndata['beta'] = data['mr_rolling'] / data['m2_rolling']","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:00.307453Z","iopub.execute_input":"2021-11-17T19:23:00.307778Z","iopub.status.idle":"2021-11-17T19:23:14.635862Z","shell.execute_reply.started":"2021-11-17T19:23:00.307728Z","shell.execute_reply":"2021-11-17T19:23:14.635071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And finallly Target is calculated.\n$$\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$","metadata":{}},{"cell_type":"code","source":"data['Target_recreated'] = data['r'] - data['beta'] * data['m']","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:14.638503Z","iopub.execute_input":"2021-11-17T19:23:14.638826Z","iopub.status.idle":"2021-11-17T19:23:14.790872Z","shell.execute_reply.started":"2021-11-17T19:23:14.638783Z","shell.execute_reply":"2021-11-17T19:23:14.790106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we compare given and recreated Target.","metadata":{}},{"cell_type":"code","source":"data['Target_diff'] = np.abs(data['Target'] - data['Target_recreated'])\n\nprint(f'Average absolute error {data.Target_diff.mean():8.6f}')\nprint(f'Max absolute error {data.Target_diff.max():8.6f}')\nprint(f'Standard deviation {data.Target_diff.std():8.6f}')","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:14.792534Z","iopub.execute_input":"2021-11-17T19:23:14.792853Z","iopub.status.idle":"2021-11-17T19:23:15.327275Z","shell.execute_reply.started":"2021-11-17T19:23:14.792814Z","shell.execute_reply":"2021-11-17T19:23:15.326338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Normal Target is changed in range [-0.5, 0.96], so Max absolute error 2.4 means recreated Target is completely wrong for some records.","metadata":{}},{"cell_type":"code","source":"data['Target'].agg(['min', 'max'])","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.328367Z","iopub.execute_input":"2021-11-17T19:23:15.328605Z","iopub.status.idle":"2021-11-17T19:23:15.407083Z","shell.execute_reply.started":"2021-11-17T19:23:15.328575Z","shell.execute_reply":"2021-11-17T19:23:15.406208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check how many records with wrong (values outside of range) recreated Target do we have.","metadata":{}},{"cell_type":"code","source":"(data.Target_recreated < -0.509351).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.408211Z","iopub.execute_input":"2021-11-17T19:23:15.408416Z","iopub.status.idle":"2021-11-17T19:23:15.454978Z","shell.execute_reply.started":"2021-11-17T19:23:15.408384Z","shell.execute_reply":"2021-11-17T19:23:15.454285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(data.Target_recreated > 0.96417).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.456027Z","iopub.execute_input":"2021-11-17T19:23:15.456574Z","iopub.status.idle":"2021-11-17T19:23:15.503673Z","shell.execute_reply.started":"2021-11-17T19:23:15.456535Z","shell.execute_reply":"2021-11-17T19:23:15.502796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I have tried different prices, changed time intervals in formula, replaced rolling average from 3750 minutes to 3750 last records, etc. but result is the same or worse. It is possible minimize Max error, but avreage became higher.\n\nPlease let me know if you find an error in calculations or have some insight of how original Target is really calculated.","metadata":{}},{"cell_type":"markdown","source":"It turns out that this code gives a lot of NA values in Recreated Target, so minimizing Abs/Max error was perhaps misleading.","metadata":{}},{"cell_type":"code","source":"pd.isna(data.Target).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.504860Z","iopub.execute_input":"2021-11-17T19:23:15.505193Z","iopub.status.idle":"2021-11-17T19:23:15.558513Z","shell.execute_reply.started":"2021-11-17T19:23:15.505149Z","shell.execute_reply":"2021-11-17T19:23:15.557761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.isna(data.Target_recreated).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.559584Z","iopub.execute_input":"2021-11-17T19:23:15.559997Z","iopub.status.idle":"2021-11-17T19:23:15.614092Z","shell.execute_reply.started":"2021-11-17T19:23:15.559956Z","shell.execute_reply":"2021-11-17T19:23:15.613334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Resulting Recreated Target has more than 10 times more NA values. Let's check when these NAs were intorduced.","metadata":{}},{"cell_type":"code","source":"pd.isna(data.r).sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.615270Z","iopub.execute_input":"2021-11-17T19:23:15.615555Z","iopub.status.idle":"2021-11-17T19:23:15.668672Z","shell.execute_reply.started":"2021-11-17T19:23:15.615523Z","shell.execute_reply":"2021-11-17T19:23:15.667963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When calculating R number of NAs is almost equal to number of Target NA. The difference is 1086. It can be explained by [missing first minite](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286095) of each month for each asset. \n\nIt is interesting that the Target is present in train dataset for dates like YYYY-MM-01 23:44:00 (t+16) and YYYY-MM-01 23:59:00 (t+1). For these times calculation algorithm requires price at time YYYY-MM-01 00:00:00, but this time is absent in test data. It can be considered as a proof that Target was calculated on other data, and then train dataset was built with already calculated Target.\n\nThis missing first minute of a month bug gives approximately 44 month x 2 (every row affect two targets for shift 1 and 16) x 14 (number of assets) => 1232. Some assets do not have data for all months.\n\nThe only other place that can produce NA is the following code\n> rolling(window='3750T', min_periods=3750)\n\nmissing values inside 3750 minutes interval result in NA values. \n\nNow let's look at the [code](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778) provided by host:\n> num = df.multiply(mkt.values, axis=0).rolling(window).mean().values  \n> denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values  \n\nThere are two possible types of window that rolling accepts - [it can be](https://pandas.pydata.org/pandas-docs/version/1.3.0/reference/api/pandas.DataFrame.rolling.html) either shift/int (number or rows to shift) or time offset (number of minutes to shift), but there is no **min_periods** parameter. This means that for int shift it does not produce NA at all, for time offset it may produce NA only if there is no data for last 3750 minutes.\n\nNext line of the code calculates beta and removes all resulting non-number values with zero:\n> beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.)  \n\nThe conclusion here is that adding min_periods=3750 which seemingly improved our metrics in fact added lots of NA values effectively minimizing data where these metrics were calculating. Host code shows that rolling average does not produce additional NA values.","metadata":{}},{"cell_type":"markdown","source":"# More optimized version\n\nHere we try to do two things: solve problem with NA and make this code working faster, and re-use code from host.\nThe code from the host shows that data should be re-presented in a little bit different way to avoid expensive minutes-based shift/rolling operations and perform calculation for all assets at once.","metadata":{}},{"cell_type":"markdown","source":"We start with creating new data frame where all timestamps (minutes) present. In the train some minutes may be missing for some assets.","metadata":{}},{"cell_type":"code","source":"ids = list(details.Asset_ID)\nasset_names = list(details.Asset_Name)\n\n# times = data['timestamp'].agg(['min', 'max']).to_dict()\n# all_timestamps = np.arange(times['min'], times['max'] + 60, 60)\nall_timestamps = np.sort(data['timestamp'].unique())\ntargets = pd.DataFrame(index=all_timestamps)\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:15.669765Z","iopub.execute_input":"2021-11-17T19:23:15.670191Z","iopub.status.idle":"2021-11-17T19:23:16.221689Z","shell.execute_reply.started":"2021-11-17T19:23:15.670140Z","shell.execute_reply":"2021-11-17T19:23:16.220880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we calculate R for each asset and add its values as a column to new targets data frame. Note that some rows will contain NA as required prices with shift may be absent.","metadata":{}},{"cell_type":"code","source":"for i, id in enumerate(ids):\n    asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n    price = pd.Series(index=all_timestamps, data=asset[price_column])\n#     targets[asset_names[i]] = np.log(\n#         price.shift(periods=-16) /\n#         price.shift(periods=-1)\n#     )\n    targets[asset_names[i]] = (\n        price.shift(periods=-16) /\n        price.shift(periods=-1)\n    ) - 1\n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:16.225319Z","iopub.execute_input":"2021-11-17T19:23:16.225950Z","iopub.status.idle":"2021-11-17T19:23:31.028861Z","shell.execute_reply.started":"2021-11-17T19:23:16.225906Z","shell.execute_reply":"2021-11-17T19:23:31.028036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next calculate M as weighted mean of columns (currently holding R value for each asset) for each row. This implementation count unavailable R values as zero. Please note it is not the only way to deal with NA values when calculating weighted average.","metadata":{}},{"cell_type":"code","source":"weights = np.array(list(details.Weight))\ntargets['m'] = np.average(targets.fillna(0), axis=1, weights=weights)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:31.029939Z","iopub.execute_input":"2021-11-17T19:23:31.030159Z","iopub.status.idle":"2021-11-17T19:23:31.416563Z","shell.execute_reply.started":"2021-11-17T19:23:31.030132Z","shell.execute_reply":"2021-11-17T19:23:31.415705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally it is time to apply code provided by the host to calculate beta and Target. ","metadata":{}},{"cell_type":"code","source":"m = targets['m']\n\nnum = targets.multiply(m.values, axis=0).rolling(3750).mean().values\ndenom = m.multiply(m.values, axis=0).rolling(3750).mean().values\nbeta = np.nan_to_num(num.T / denom, nan=0., posinf=0., neginf=0.)\n\ntargets = targets - (beta * m.values).T","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:31.417648Z","iopub.execute_input":"2021-11-17T19:23:31.417862Z","iopub.status.idle":"2021-11-17T19:23:33.094494Z","shell.execute_reply.started":"2021-11-17T19:23:31.417836Z","shell.execute_reply":"2021-11-17T19:23:33.093663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check how it is different from given Target and how many NA values we have.","metadata":{}},{"cell_type":"code","source":"diffs = []\n\nfor i, id in enumerate(ids):\n    print(asset_names[i])\n    # type: pd.DataFrame\n    asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n    print(f'asset size {asset.shape[0]}')\n    recreated = pd.Series(index=asset.index, data=targets[asset_names[i]])\n    diff = np.abs(asset['Target'] - recreated)\n    diffs.append(diff[~pd.isna(diff)].values)\n    print(f'Average absolute error {diff.mean():8.6f}')\n    print(f'Max absolute error {diff.max():8.6f}')\n    print(f'Standard deviation {diff.std():8.6f}')\n    print(f'Target na {pd.isna(asset.Target).sum()}')\n    print(f'Target_calculated na {pd.isna(recreated).sum()}')\n    print()\n\ndiffs = np.concatenate(diffs, axis=0)\nprint('For all assets')\nprint(f'Average absolute error {diffs.mean():8.6f}')\nprint(f'Max absolute error {diffs.max():8.6f}')\nprint(f'Standard deviation {diffs.std():8.6f}')\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:33.095922Z","iopub.execute_input":"2021-11-17T19:23:33.096359Z","iopub.status.idle":"2021-11-17T19:23:41.767416Z","shell.execute_reply.started":"2021-11-17T19:23:33.096324Z","shell.execute_reply":"2021-11-17T19:23:41.766446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is still not ideal, but at least our metrics are calculated on almost all data available and number of NA almost the same as in Target.\n","metadata":{}},{"cell_type":"markdown","source":"There is a discussion about Beta equal to zero. It turns Target calculation just to R calculation. Let's see how many non-number values we have and how many rows have beta=0.","metadata":{}},{"cell_type":"code","source":"beta_ = num.T / denom\n\nfor i, id in enumerate(ids):\n    print(asset_names[i])\n    print(f'Infiinte beta rows {np.isinf(beta_[i]).sum()}')\n    nan_sum = np.isnan(beta_[i]).sum()\n    print(f'NAN beta rows {nan_sum} ({100 * nan_sum / beta_.shape[1]:5.2f}%)')\n\n    eps = 1e-6\n    zero_sum = ((beta[i] > -eps) & (beta[i] < eps)).sum()\n    print(f'Zero beta rows {zero_sum} ({100 * zero_sum / beta.shape[1]:5.2f}%)')\n    print()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:41.768517Z","iopub.execute_input":"2021-11-17T19:23:41.768726Z","iopub.status.idle":"2021-11-17T19:23:42.058484Z","shell.execute_reply.started":"2021-11-17T19:23:41.768695Z","shell.execute_reply":"2021-11-17T19:23:42.057544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below is the same code but in the form of a function you can copy and paste. Note that it does not require Time column, it uses timestamp instead.","metadata":{}},{"cell_type":"code","source":"def calculate_target(data: pd.DataFrame, details: pd.DataFrame, price_column: str):\n    ids = list(details.Asset_ID)\n    asset_names = list(details.Asset_Name)\n    weights = np.array(list(details.Weight))\n\n    all_timestamps = np.sort(data['timestamp'].unique())\n    targets = pd.DataFrame(index=all_timestamps)\n\n    for i, id in enumerate(ids):\n        asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n        price = pd.Series(index=all_timestamps, data=asset[price_column])\n        targets[asset_names[i]] = (\n            price.shift(periods=-16) /\n            price.shift(periods=-1)\n        ) - 1\n    \n    targets['m'] = np.average(targets.fillna(0), axis=1, weights=weights)\n    \n    m = targets['m']\n\n    num = targets.multiply(m.values, axis=0).rolling(3750).mean().values\n    denom = m.multiply(m.values, axis=0).rolling(3750).mean().values\n    beta = np.nan_to_num(num.T / denom, nan=0., posinf=0., neginf=0.)\n\n    targets = targets - (beta * m.values).T\n    targets.drop('m', axis=1, inplace=True)\n    \n    return targets","metadata":{"execution":{"iopub.status.busy":"2021-11-17T19:23:42.060096Z","iopub.execute_input":"2021-11-17T19:23:42.060429Z","iopub.status.idle":"2021-11-17T19:23:42.072548Z","shell.execute_reply.started":"2021-11-17T19:23:42.060388Z","shell.execute_reply":"2021-11-17T19:23:42.071504Z"},"trusted":true},"execution_count":null,"outputs":[]}]}