{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#
TABLE OF CONTENTS
\n\n* [INTRODUCTION](#0)\n* [DATA IMPORT](#1)\n* [ML MODELS](#2)\n * [MODEL INITIATION](#2.1)\n * [I-O](#2.2)\n * [TRAINING](#2.3)\n* [INFERENCING](#3)\n* [NEXT STEPS AND OUTRO](#4)","metadata":{}},{"cell_type":"markdown","source":"\n#
INTRODUCTION
","metadata":{}},{"cell_type":"markdown","source":"\n##
PACKAGE IMPORTS
","metadata":{}},{"cell_type":"code","source":"%%time \n\n# General library imports:-\nfrom IPython.display import display_html, clear_output, Markdown;\nfrom gc import collect;\nfrom itertools import groupby as grp_by;\n\nfrom copy import deepcopy;\nimport pandas as pd;\nimport numpy as np;\nimport polars as pl;\nimport re;\n\nfrom pprint import pprint;\nfrom functools import partial;\nfrom collections import Counter;\nfrom itertools import product;\nfrom colorama import Fore, Style, init;\nfrom warnings import filterwarnings;\nfilterwarnings('ignore');\n\nimport ctypes\nlibc = ctypes.CDLL(\"libc.so.6\")\n\nfrom tqdm.notebook import tqdm;\n\nclear_output();\nprint();\nlibc.malloc_trim(0);","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:36.924199Z","iopub.execute_input":"2023-09-18T16:09:36.924626Z","iopub.status.idle":"2023-09-18T16:09:37.762149Z","shell.execute_reply.started":"2023-09-18T16:09:36.924588Z","shell.execute_reply":"2023-09-18T16:09:37.760965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\n# Model development:-\nfrom xgboost import XGBClassifier;\nfrom lightgbm import LGBMClassifier, log_evaluation, early_stopping;\nfrom catboost import CatBoostClassifier;\nfrom sklearn.ensemble import (RandomForestClassifier as RFC,\n ExtraTreesClassifier as ETC,\n GradientBoostingClassifier as GBC,\n HistGradientBoostingClassifier as HGBC,\n );\n\nclear_output();\nprint();\ncollect();\nlibc.malloc_trim(0);","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:37.764182Z","iopub.execute_input":"2023-09-18T16:09:37.764678Z","iopub.status.idle":"2023-09-18T16:09:41.948961Z","shell.execute_reply.started":"2023-09-18T16:09:37.764645Z","shell.execute_reply":"2023-09-18T16:09:41.947431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\n# Color printing \ndef PrintColor(text:str, color = Fore.BLUE, style = Style.BRIGHT):\n \"Prints color outputs using colorama using a text F-string\";\n print(style + color + text + Style.RESET_ALL); \n\n# Making sklearn pipeline outputs as dataframe:-\nfrom sklearn import set_config; \nset_config(transform_output = \"pandas\");\npd.set_option('display.max_columns', 50);\npd.set_option('display.max_rows', 50);\n\n# Setting global configurations for polars:-\npl.Config.activate_decimals(True).set_tbl_hide_column_data_types(True);\npl.Config(**dict(tbl_formatting = 'ASCII_FULL_CONDENSED',\n tbl_hide_column_data_types = True,\n tbl_hide_dataframe_shape = True,\n fmt_float = \"mixed\",\n tbl_cell_alignment = 'CENTER',\n tbl_hide_dtype_separator = True,\n tbl_cols = 100,\n tbl_rows = 100,\n fmt_str_lengths = 100,\n )\n );\n\nclear_output();\nprint();\ncollect();\nlibc.malloc_trim(0);","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:41.950628Z","iopub.execute_input":"2023-09-18T16:09:41.951132Z","iopub.status.idle":"2023-09-18T16:09:42.082877Z","shell.execute_reply.started":"2023-09-18T16:09:41.951086Z","shell.execute_reply":"2023-09-18T16:09:42.081233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##
CONFIGURATION
","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Defining a Configuration class:-\nclass CFG:\n \"This is a Configuration class for model training\";\n \n # Data preparation:- \n version_nb = 1;\n \n # Used during code proto-typing and syntax testing:- \n test_req = \"N\";\n n_samples = 0.02;\n # Used during testing with train-sample series id for inference:- \n test_tr_smpl = \"N\";\n \n gpu_switch = \"OFF\"; \n state = 42;\n target = \"awake\";\n path = f\"/kaggle/input/sleepstate-datacuration/\";\n alt_path = f\"/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet\";\n \n # Data transforms and scaling:- \n roll_nper = 20;\n \n # Model Training:- \n methods = ['CBC', 'LGBMC', 'XGBC', 'RFC', 'HGBC'];\n method = \"RFC\";\n \n # Inference:- \n sm_len = 2 * 230;\n \n # Global variables for plotting:-\n grid_specs = {'visible': True, 'which': 'both', 'linestyle': '--', \n 'color': 'lightgrey', 'linewidth': 0.75};\n title_specs = {'fontsize': 9, 'fontweight': 'bold', 'color': 'tab:blue'};\n\nprint();\nPrintColor(f\"--> Configuration done!\\n\");\ncollect();\nlibc.malloc_trim(0);","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:42.086560Z","iopub.execute_input":"2023-09-18T16:09:42.087117Z","iopub.status.idle":"2023-09-18T16:09:42.215595Z","shell.execute_reply.started":"2023-09-18T16:09:42.087073Z","shell.execute_reply":"2023-09-18T16:09:42.214304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"| Version
Number | Version Details | Preparation date|LB score|Submission error
Y/N|User|\n| :-: | --- | :-: | :-: |:-: |:-: \n| **V1** |* Used Ravi's preprocessed dataset - V1 as input
* Random Forest baseline
* Inference based on successive loading from the test data series id|18Sep2023|||**Ravi**|","metadata":{}},{"cell_type":"markdown","source":"\n##
NOTEBOOK OBJECTIVES
","metadata":{}},{"cell_type":"markdown","source":"
\nIn this assignment, we entail the below-
\n1. This is a time series classifier for sleep state detection
\n2. We are provided acceleromater data for the train set and are supposed to make predictions for the sleep state in the test set
\n3. The target is converted to a binary state in the alternative dataset
\n4. Evaluation metric is derived from the adjutant notebook and is average precision over timestamp error tolerance thresholds, averaged over event classes.
\n
In this notebook, we entail the below-
\n1. To initialize and train a single baseline ML model on the preprocessed data imported from the previous part
\n2. To infer and submit the test set results

This notebook continues from my data-curation kernel
Please peruse that one to complete the process appropriately
\n
\n\n
**Note**
\n**Data curation kernel input link- (https://www.kaggle.com/code/ravi20076/sleepstate-datacuration)**","metadata":{}},{"cell_type":"markdown","source":"\n#
DATA IMPORT
","metadata":{}},{"cell_type":"code","source":"%%time\n\nXtrain = \\\npd.read_parquet(CFG.path + f\"Xtrain_V{CFG.version_nb}.parquet\").\\\ndrop(columns = ['series_id', 'step'], errors = \"ignore\");\nytrain = pd.read_parquet(CFG.path + f\"ytrain_V{CFG.version_nb}.parquet\").squeeze();\n\nPrintColor(f\"\\nViewing the Xtrain dataframe\\n\");\ndisplay(Xtrain.head(5).style.format(precision = 2));\n\nif CFG.test_req == \"Y\":\n if isinstance(CFG.n_samples, int) == True :\n PrintColor(f\"\\nTesting the code execution with {CFG.n_samples} per target class\\n\", \n Fore.RED);\n ytrain = ytrain.groupby(ytrain).sample(n = CFG.n_samples, random_state = CFG.state).sort_index();\n Xtrain = Xtrain.iloc[ytrain.index];\n else:\n PrintColor(f\"\\nTesting the code execution with {CFG.n_samples * 100:.3f} percent per target class\\n\", \n Fore.RED);\n ytrain = ytrain.groupby(ytrain).sample(frac = CFG.n_samples, random_state = CFG.state).sort_index();\n Xtrain = Xtrain.iloc[ytrain.index]; \n \n PrintColor(f\"---> Data shape after testing sampling = {Xtrain.shape} {ytrain.shape}\",);\n pprint(Counter(ytrain), depth = 1, indent = 10);\n \nelse:\n PrintColor(f\"\\nWe are not testing the code execution\\n\", Fore.RED); \n \nsel_cols = Xtrain.iloc[0:2].columns;\nPrintColor(f\"\\n\\nTraining set columns\"); \npprint(sel_cols);\n\nprint();\ncollect();\nlibc.malloc_trim(0);","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:42.217214Z","iopub.execute_input":"2023-09-18T16:09:42.217556Z","iopub.status.idle":"2023-09-18T16:09:48.351908Z","shell.execute_reply.started":"2023-09-18T16:09:42.217527Z","shell.execute_reply":"2023-09-18T16:09:48.350777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#
ML MODEL TRAINING
","metadata":{}},{"cell_type":"markdown","source":"
\nNote-
\n1. We aim to make a successful baseline submission without a CV score here to test the efficacy of the features ensued.
\n2. We aim to incorporate the competition metric, include CV results, build better models and use complex techniques appropriate to the assignment in future versions of the notebook
\n3. To use this kernel, select one option from the configuration methods and initialize it as method = your option. This will execute and infer herewith
\n
","metadata":{}},{"cell_type":"code","source":"%%time \n\n# Initializing model I-O:-\n\nMdl_Master = \\\n{'CBC': CatBoostClassifier(**{'task_type' : \"GPU\" if CFG.gpu_switch == \"ON\" else \"CPU\",\n 'random_state' : CFG.state,\n 'classes_count' : 3,\n 'bagging_temperature' : 0.425,\n 'colsample_bylevel' : 0.667,\n 'iterations' : 1000,\n 'learning_rate' : 0.05,\n 'od_wait' : 32,\n 'max_depth' : 6,\n 'l2_leaf_reg' : 0.45,\n 'min_data_in_leaf' : 300,\n 'random_strength' : 0.15, \n 'max_bin' : 200,\n 'verbose' : 0, \n }\n ), \n \n 'LGBMC': LGBMClassifier(**{'device' : \"gpu\" if CFG.gpu_switch == \"ON\" else \"cpu\",\n 'objective' : 'multiclass',\n 'metric' : 'none',\n 'boosting_type' : 'gbdt',\n 'random_state' : CFG.state,\n 'colsample_bytree' : 0.675,\n 'subsample' : 0.925,\n 'learning_rate' : 0.045,\n 'max_depth' : 5,\n 'n_estimators' : 100,\n 'num_leaves' : 90,\n 'min_child_samples' : 350,\n 'reg_alpha' : 0.01,\n 'reg_lambda' : 1.5,\n 'verbose' : -1,\n }\n ),\n\n 'XGBC': XGBClassifier(**{'tree_method' : \"gpu_hist\" if CFG.gpu_switch == \"ON\" else \"hist\",\n 'objective' : 'multi:logistic',\n 'random_state' : CFG.state,\n 'colsample_bytree' : 0.45,\n 'learning_rate' : 0.05,\n 'max_depth' : 6,\n 'n_estimators' : 1500, \n 'reg_alpha' : 0.01,\n 'reg_lambda' : 1.25,\n 'min_child_samples' : 350,\n }\n ),\n \n 'HGBC': HGBC(learning_rate = 0.055,\n max_iter = 2000,\n max_depth = 7,\n min_samples_leaf = 300,\n l2_regularization= 1.25,\n max_bins = 200,\n n_iter_no_change = 100,\n random_state = CFG.state,\n ),\n \n 'RFC' : RFC(n_estimators = 75,\n criterion = 'gini',\n max_depth = 6,\n min_samples_leaf = 350,\n max_features = \"log2\",\n n_jobs = -1,\n random_state = CFG.state,\n verbose = 0,\n ),\n \n \"GBC\" : GBC(n_estimators = 50,\n max_depth = 8,\n min_samples_leaf = 350,\n random_state = CFG.state,\n )\n};\n\nprint();\ncollect();\nlibc.malloc_trim(0);","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:48.355816Z","iopub.execute_input":"2023-09-18T16:09:48.356386Z","iopub.status.idle":"2023-09-18T16:09:48.501245Z","shell.execute_reply.started":"2023-09-18T16:09:48.356333Z","shell.execute_reply":"2023-09-18T16:09:48.500053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\ntry:\n PrintColor(f\"--> Initializing {CFG.method} model training\");\n model = Mdl_Master[CFG.method];\nexcept:\n PrintColor(f\"--> Defaulting to random forest to prevent submission error\", Fore.RED);\n model = Mdl_Master[\"RFC\"]; \n \nmodel.fit(Xtrain, ytrain);\n\nif CFG.test_req == \"Y\": \n pass\nelse:\n del Xtrain, ytrain, Mdl_Master;\n \ncollect();\nlibc.malloc_trim(0);\nprint(); ","metadata":{"execution":{"iopub.status.busy":"2023-09-18T16:09:48.502862Z","iopub.execute_input":"2023-09-18T16:09:48.504235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#
INFERENCING
","metadata":{}},{"cell_type":"markdown","source":"
\nSection strategy-
\n1. We incorporate the test set and aggregate the series id numbers first
\n2. For each series, we load the test set, curate features and make model predictions
\n3. We dump the results into the working folder to conserve memory
\n4. We input the results (predictions) from the working folder at the end and combine the results to curate the final test set predictions \n
\n\n
**Source**
\nhttps://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set
\nI picked up the strategy from the source and coded it myself using a class with appropriate initialization options
","metadata":{}},{"cell_type":"code","source":"%%time \n\nclass InferenceMaker():\n \"\"\"\n This class infers the model onto the test set with chunks, each chunk is a series id\n \"\"\";\n \n def __init__(self, sel_cols: np.array, model): \n self.alt_path = CFG.alt_path;\n self.path = CFG.path;\n self.sel_cols = sel_cols.to_list();\n self.nper = CFG.roll_nper;\n self.model = model;\n self.target = CFG.target;\n self.sm_len = CFG.sm_len;\n self.test_req = CFG.test_tr_smpl;\n \n if self.test_req == \"Y\":\n self.new_path = re.sub(r\"test_series\", \"train_series\", self.alt_path);\n PrintColor(f\"\\n---> Testing the code with a couple of sample series in the training data\", \n color = Fore.RED);\n print(self.new_path);\n else: \n self.new_path = \"\";\n \n def _CollateSID(self):\n \"This method collates the series id in the test data/ sample train data for testing\";\n \n if self.test_req == \"Y\":\n PrintColor(f\"\\n---> Testing the code with a couple of sample series in the training data\", \n color = Fore.RED); \n self.test_sid = \\\n pl.scan_parquet(self.new_path).select([\"series_id\"]).\\\n unique().\\\n collect().\\\n to_numpy().flatten()[0:5]; \n \n else:\n self.test_sid = \\\n pl.scan_parquet(self.alt_path).select([\"series_id\"]).\\\n unique().\\\n collect().\\\n to_numpy().flatten();\n return self;\n \n def _PreprocessDF(self, series_id: str = ''):\n \"\"\"\n We import the test set from the path in configuration file as a lazy frame.\n We use the date time column and rolling statistics from public work.\n We create secondary features on the competition test set using the inputs\n If we test the code, then a random input series is taken for inference, else the provided test series is taken for inference.\n\n Returns:- secondary feature built test dataframe ready for model implementation\n \"\"\";\n\n # Importing the dataframe:-\n if self.test_req == \"Y\": \n self.Xtest = \\\n pl.scan_parquet(self.new_path).\\\n with_columns(pl.col('timestamp').str.to_datetime().dt.offset_by(\"-4h\")).\\\n filter(pl.col('series_id') == series_id);\n \n else:\n self.Xtest = \\\n pl.scan_parquet(self.alt_path).\\\n with_columns(pl.col('timestamp').str.to_datetime().dt.offset_by(\"-4h\")).\\\n filter(pl.col('series_id') == series_id);\n\n # Curating date features:- \n self.Xtest = \\\n self.Xtest.\\\n with_columns([pl.col(\"timestamp\").dt.hour().cast(pl.Int8).alias(\"hour_nb\"),\n pl.col(\"timestamp\").dt.day().cast(pl.Int8).alias(\"day_nb\"),\n pl.col(\"timestamp\").dt.weekday().cast(pl.Int8).alias(\"weekday_nb\"),\n pl.col(\"timestamp\").dt.week().cast(pl.Int8).alias(\"week_nb\"),\n pl.col(\"timestamp\").dt.month().cast(pl.Int8).alias(\"month_nb\"),\n pl.col(\"timestamp\").dt.year().cast(pl.Int16).alias(\"year_nb\"),\n (pl.col(\"anglez\") * pl.col(\"enmo\")).alias(\"anglez_enmo\")\n ]\n ).\\\n with_columns([pl.when(pl.col(\"hour_nb\") <= 12).then(pl.lit(1)).otherwise(pl.lit(0)).cast(pl.Int8).alias(\"is_morning\"),\n pl.when(pl.col(\"weekday_nb\") >= 6).then(pl.lit(1)).otherwise(pl.lit(0)).cast(pl.Int8).alias(\"is_weekend\") \n ]\n );\n\n # Curating rolling features and window results:-\n self.Xtest = \\\n self.Xtest.\\\n with_columns([pl.col(\"anglez\").diff(self.nper).over(\"series_id\").\\\n fill_null(strategy = 'backward').alias(\"d_anglez\"),\n pl.col(\"enmo\").diff(self.nper).over(\"series_id\").\\\n fill_null(strategy = 'backward').alias(\"d_enmo\"),\n pl.col(\"anglez\").rolling_mean(self.nper).over(\"series_id\").\\\n fill_null(strategy = 'backward').fill_null(strategy = 'forward').alias('ma_anglez'),\n pl.col(\"enmo\").rolling_mean(self.nper).over(\"series_id\").\\\n fill_null(strategy = 'backward').fill_null(strategy = 'forward').alias('ma_enmo')\n ]\n ).\\\n with_columns([pl.col('d_anglez').rolling_mean(self.nper).over(\"series_id\").\\\n fill_null(strategy = 'backward').fill_null(strategy = 'forward').alias('ma_d_anglez'),\n pl.col('d_enmo').rolling_mean(self.nper).over(\"series_id\").\\\n fill_null(strategy = 'backward').fill_null(strategy = 'forward').alias('ma_d_enmo')\n ]\n );\n \n # Collating relevant columns for model implementation:- \n self.Xtest = \\\n self.Xtest.select(self.sel_cols + ['series_id', 'step']).\\\n collect().\\\n to_pandas();\n \n collect();\n return self;\n \n @staticmethod\n def GetEvent(df):\n lstCV = zip(df.series_id, df.smooth);\n lstPOI = [];\n \n for (c, v), g in grp_by(lstCV, lambda cv: (cv[0], cv[1] != 0 and not pd.isnull(cv[1]))):\n llg = sum(1 for item in g);\n if v is False:\n lstPOI.extend([0] * llg);\n else:\n lstPOI.extend(['onset'] + (llg - 2) * [0] + ['wakeup'] if llg > 1 else [0]);\n return lstPOI;\n \n def _CuratePreds(self, series_id: str):\n \"This method curates model predictions for the test data series id\";\n \n self.Xtest['score'] =\\\n self.model.\\\n predict_proba(self.Xtest[self.sel_cols])[:,1];\n self.Xtest[f\"not_{self.target}\"] = 1 - self.Xtest[\"score\"];\n \n self.Xtest[\"smooth\"] = \\\n self.Xtest[f\"not_{self.target}\"].\\\n rolling(self.sm_len, center=True).\\\n mean().\\\n fillna(method=\"bfill\").fillna(method=\"ffill\").round();\n \n # Obtaining the event information:- \n self.Xtest[\"event\"] = self.GetEvent(self.Xtest);\n \n # Curating the submission file component:-\n self.Xtest = \\\n self.Xtest.loc[self.Xtest[\"event\"] != 0][[\"series_id\",\"step\",\"event\",\"score\"]].\\\n reset_index(drop= True);\n self.Xtest.to_csv(f'{series_id}.csv', index = False);\n \n collect();\n return self;\n \n def _CombinePreds(self):\n \"\"\"\n In this method, we combine the final predictions to curate a single submission file for the test data\n \"\"\";\n\n self.sub_fl = pd.read_csv(f'{self.test_sid[0]}.csv');\n for s in self.test_sid[1:]:\n temp = pd.read_csv(f'{s}.csv')\n self.sub_fl = pd.concat([self.sub_fl, temp], ignore_index=True);\n del temp;\n collect();\n\n self.sub_fl = self.sub_fl.copy().reset_index(drop=True).reset_index(names=\"row_id\");\n return self;\n \n def CurateTestPreds(self):\n \"This method combines all other methods and curates the test set predictions\";\n \n self._CollateSID();\n collect();\n for i, series_id in tqdm(enumerate(self.test_sid)):\n num_space = 3 if i <= 9 else 2;\n PrintColor(f\"---> {i+1}.{' ' * num_space} Current series id = {series_id}\");\n del num_space;\n \n self._PreprocessDF(series_id = series_id);\n self._CuratePreds(series_id = series_id);\n \n collect();\n self._CombinePreds();\n collect();\n \n return self.sub_fl;\n \nprint();\ncollect();\nlibc.malloc_trim(0);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\ninf_prf = InferenceMaker(sel_cols = sel_cols, model = model);\n\ntry:\n sub_fl = inf_prf.CurateTestPreds();\n sub_fl.to_csv(\"submission.csv\", index = None);\n PrintColor(f\"\\n---> Submission file\\n\");\n display(sub_fl.head(10).style.format(precision = 3));\nexcept:\n PrintColor(f\"\\n---> Check the submission file\\n\", Fore.RED);\n\ncollect();\nprint();\nlibc.malloc_trim(0);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#
OUTRO
","metadata":{}},{"cell_type":"markdown","source":"
\nNext steps-
\n1. Incorporating CV strategy
\n2. Better features
\n3. Better algorithms
\n4. Incorporation of the competition metric into the submission process and inferencing
\n5. Hardware management
\n
","metadata":{}},{"cell_type":"markdown","source":"
\nMy learnings from this process insofar-
\n1. libc.malloc_trim(0) - thanks to my fellow contributors!
\n2. Data curation and inferencing in separate kernels
\n3. I have used polars before but not in a competition environment. Thanks to this challenge, I used polars in a competition setting!
\n
","metadata":{}},{"cell_type":"markdown","source":"Our collective contribution to the assignment thus far-
\n1. Baseline notebook- https://www.kaggle.com/code/ravi20076/sleepstate-eda-baseline
\n2. Preprocessed dataset - https://www.kaggle.com/datasets/ravi20076/sleepstatepreprocesseddata
\n3. Data curation component kernel- https://www.kaggle.com/code/ravi20076/sleepstate-datacuration
\n4. Model training and inferencing baseline - this one.\n

**Please feel free to use these artefacts and improve upon them. You may reach out in case of any misgivings!**
\nBest regards!!","metadata":{}}]}