{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Local API Emulator\n\n**NEW**: the updated version of Local API Emulator is neater and packaged as a module with a demo available here:\nhttps://www.kaggle.com/jagofc/local-api-emulator-is-now-a-module\n\n**OLD**: Wouldn't it be great if we could test the API locally on our own slices of data? Well, now you can - here's my crack at a local API emulator with LB scoring. \n\n#### It has the properties you've come to expect and love from the real API:\n\n+ Delivers data with the same formatting as that from the real API, grouped by `timestamp`.\n+ Has a `predict()` method.\n+ Berates you if you try to:\n    + get data for time `t_2` before calling `predict()` for time `t_1`.\n    + predict for time `t_1` before getting the data for time `t_1`.\n\n#### It also has the following functionality:\n\n+ Collects your predictions in a list of dataframe slices which is accessible as an attribute `predictions`.\n+ Has a length method which counts the number of unique timestamps remaining to be served.\n+ Gives familiar error messages if you don't follow protocol.\n+ Has a `score` method which:\n    + computes the weighted correlation between your predictions and the true targets.\n    + correctly handles constant predictions (giving a score of -1).\n\n#### This is not the *real* API.\n\nThis is just an emulator. It is easy to cheat it in ways that I hope won't be possible for real API.\\\ne.g. I think it's highly unlikely that - when running code for the private LB - the hosts store a list containing all of your predictions that can be modified post-hoc by participants... (However if you've played around with API in your interactive notebook you'll see that **is** possible to modify the predictions list in the local `gresearch_crypto` env :O)\n\n#### You might find this code useful for:\n\n+ realistically testing your models on a different time period to that used for the public LB,\n+ avoiding the opaque submission procedure,\n+ racking up more submissions attempts than the ordained 5 per day.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-12-18T16:24:01.340788Z","iopub.execute_input":"2021-12-18T16:24:01.341213Z","iopub.status.idle":"2021-12-18T16:24:01.361741Z","shell.execute_reply.started":"2021-12-18T16:24:01.341111Z","shell.execute_reply":"2021-12-18T16:24:01.360507Z"}}},{"cell_type":"markdown","source":"## Code & step-by-step Demo","metadata":{}},{"cell_type":"markdown","source":"Preliminaries.","metadata":{}},{"cell_type":"code","source":"import time\nimport numpy as np\nimport pandas as pd\nimport gresearch_crypto\nfrom numpy import dtype\n\nasset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nid_2_weight = dict(zip(asset_details.Asset_ID, asset_details.Weight))\n\ndtypes = {'timestamp': np.int64, 'Asset_ID': np.int8,\n          'Count': np.int32,     'Open': np.float64,\n          'High': np.float64,    'Low': np.float64,\n          'Close': np.float64,   'Volume': np.float64,\n          'VWAP': np.float64,    'Target': np.float64}\n\ndef datestring_to_timestamp(ts):\n    return int(pd.Timestamp(ts).timestamp())\n\ndef read_csv_slice(file_path='../input/g-research-crypto-forecasting/train.csv', dtypes=dtypes, use_window=None):\n    df = pd.read_csv(file_path, dtype=dtypes)\n    if use_window is not None: \n        df = df[(df.timestamp >= use_window[0]) & (df.timestamp < use_window[1])]\n    return df\n\ndef weighted_correlation(a, b, weights):\n    w = np.ravel(weights)\n    a = np.ravel(a)\n    b = np.ravel(b)\n    sum_w = np.sum(w)\n    mean_a = np.sum(a * w) / sum_w\n    mean_b = np.sum(b * w) / sum_w\n    var_a = np.sum(w * np.square(a - mean_a)) / sum_w\n    var_b = np.sum(w * np.square(b - mean_b)) / sum_w\n    cov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\n    corr = cov / np.sqrt(var_a * var_b)\n    return corr","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-19T22:03:28.702413Z","iopub.execute_input":"2021-12-19T22:03:28.70334Z","iopub.status.idle":"2021-12-19T22:03:28.722647Z","shell.execute_reply.started":"2021-12-19T22:03:28.703296Z","shell.execute_reply":"2021-12-19T22:03:28.721717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Select and load the subset that you're interested in filling the API with. *The current setting is for the slice used by the public LB.*","metadata":{}},{"cell_type":"code","source":"start = datestring_to_timestamp('2021-06-13T00:00')\nend = datestring_to_timestamp('2021-09-22T01:00')\ntrain_df = read_csv_slice(use_window=[start, end])","metadata":{"execution":{"iopub.status.busy":"2021-12-19T22:03:30.882741Z","iopub.execute_input":"2021-12-19T22:03:30.883191Z","iopub.status.idle":"2021-12-19T22:04:29.328043Z","shell.execute_reply.started":"2021-12-19T22:03:30.883138Z","shell.execute_reply":"2021-12-19T22:04:29.326821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here's code for the local API emulator.","metadata":{}},{"cell_type":"code","source":"class API:\n    def __init__(self, df):\n        df = df.astype(dtypes)\n        df['row_id'] = df.index\n        dfg = df.groupby('timestamp')\n        \n        self.data_iter = dfg.__iter__()\n        self.init_num_times = len(dfg)\n        self.next_calls = 0\n        self.pred_calls = 0\n        self.predictions = []\n        self.targets = []\n        \n        print(\"This version of the API is not optimized and should not be used to estimate the runtime of your code on the hidden test set. ;)\")\n\n    def __iter__(self):\n        return self\n    \n    def __len__(self):\n        return self.init_num_times - self.next_calls\n        \n    def __next__(self):\n        assert self.pred_calls == self.next_calls, \"You must call `predict()` successfully before you can get the next batch of data.\"\n        timestamp, df = next(self.data_iter)\n        self.next_calls += 1\n        data_df = df.drop(columns=['Target'])\n        true_df = df.drop(columns=['timestamp','Count','Open','High','Low','Close','Volume','VWAP'])\n        true_df = true_df[['row_id', 'Target', 'Asset_ID']]\n        self.targets.append(true_df)\n        pred_df = true_df.drop(columns=['Asset_ID'])\n        pred_df['Target'] = 0.\n        return data_df, pred_df\n    \n    def predict(self, pred_df):\n        assert self.pred_calls == self.next_calls - 1, \"You must get the next batch of data from the API before making a new prediction.\"\n        assert pred_df.columns.to_list() == ['row_id', 'Target'], \"Prediction dataframe should have columns `row_id` and `Target`.\"\n        pred_df = pred_df.astype({'row_id': dtype('int64'), 'Target': dtype('float64')})\n        self.predictions.append(pred_df)\n        self.pred_calls += 1\n        \n    def score(self, id_2_weight=id_2_weight):\n        pred_df = pd.concat(self.predictions).rename(columns={'Target':'Prediction'})\n        true_df = pd.concat(self.targets)\n        scoring_df = pd.merge(true_df, pred_df, on='row_id', how='left')\n        scoring_df['Weight'] = scoring_df.Asset_ID.map(id_2_weight)\n        scoring_df = scoring_df[scoring_df.Target.isna()==False]\n        if scoring_df.Prediction.var(ddof=0) < 1e-10:\n            score = -1\n        else:\n            score = weighted_correlation(scoring_df.Prediction, scoring_df.Target, scoring_df.Weight)\n        return scoring_df, score","metadata":{"execution":{"iopub.status.busy":"2021-12-19T22:04:29.331171Z","iopub.execute_input":"2021-12-19T22:04:29.331451Z","iopub.status.idle":"2021-12-19T22:04:29.345466Z","shell.execute_reply.started":"2021-12-19T22:04:29.331418Z","shell.execute_reply":"2021-12-19T22:04:29.344479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create an API instance.","metadata":{"execution":{"iopub.status.busy":"2021-12-19T20:17:41.332783Z","iopub.execute_input":"2021-12-19T20:17:41.333156Z","iopub.status.idle":"2021-12-19T20:17:41.362971Z","shell.execute_reply.started":"2021-12-19T20:17:41.333029Z","shell.execute_reply":"2021-12-19T20:17:41.361171Z"}}},{"cell_type":"code","source":"api = API(train_df)","metadata":{"execution":{"iopub.status.busy":"2021-12-19T20:57:50.105046Z","iopub.execute_input":"2021-12-19T20:57:50.105339Z","iopub.status.idle":"2021-12-19T20:57:51.208316Z","shell.execute_reply.started":"2021-12-19T20:57:50.105306Z","shell.execute_reply":"2021-12-19T20:57:51.207276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Get the first batch of data.","metadata":{}},{"cell_type":"code","source":"(data_df, pred_df) = next(api)\ndata_df.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:50:18.009044Z","iopub.execute_input":"2021-12-18T18:50:18.009383Z","iopub.status.idle":"2021-12-18T18:50:18.284654Z","shell.execute_reply.started":"2021-12-18T18:50:18.009348Z","shell.execute_reply":"2021-12-18T18:50:18.283655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll get an error if we try to continue on to the next batch without making our predictions for the current batch. *Commented out so that the notebook doesn't Fail.*","metadata":{}},{"cell_type":"code","source":"# next(api)","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:50:21.069998Z","iopub.execute_input":"2021-12-18T18:50:21.070356Z","iopub.status.idle":"2021-12-18T18:50:21.075421Z","shell.execute_reply.started":"2021-12-18T18:50:21.070313Z","shell.execute_reply":"2021-12-18T18:50:21.07455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's make a dummy prediction using `pred_df`.","metadata":{}},{"cell_type":"code","source":"api.predict(pred_df)","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:50:22.89661Z","iopub.execute_input":"2021-12-18T18:50:22.896931Z","iopub.status.idle":"2021-12-18T18:50:22.904125Z","shell.execute_reply.started":"2021-12-18T18:50:22.896899Z","shell.execute_reply":"2021-12-18T18:50:22.903019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now you can continue to iterate. Lets get another slice of data and make another dummy prediction:","metadata":{}},{"cell_type":"code","source":"(data_df, pred_df) = next(api)\napi.predict(pred_df)","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:50:25.730363Z","iopub.execute_input":"2021-12-18T18:50:25.730675Z","iopub.status.idle":"2021-12-18T18:50:25.739954Z","shell.execute_reply.started":"2021-12-18T18:50:25.730645Z","shell.execute_reply":"2021-12-18T18:50:25.739081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Your predictions are stored by the API. Let's just look at the first two prediction batches we made:","metadata":{}},{"cell_type":"code","source":"api.predictions","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:50:29.328332Z","iopub.execute_input":"2021-12-18T18:50:29.329538Z","iopub.status.idle":"2021-12-18T18:50:29.341165Z","shell.execute_reply.started":"2021-12-18T18:50:29.329484Z","shell.execute_reply":"2021-12-18T18:50:29.340544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The API also has a length method, which tracks the number of timestamps still to be served:","metadata":{}},{"cell_type":"code","source":"len(api)","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:50:46.119155Z","iopub.execute_input":"2021-12-18T18:50:46.120085Z","iopub.status.idle":"2021-12-18T18:50:46.126025Z","shell.execute_reply.started":"2021-12-18T18:50:46.120042Z","shell.execute_reply":"2021-12-18T18:50:46.125183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that you don't need to to restart the notebook kernel in order to make a new emulator (or to refresh the current one), in contrast with the `gresearch_crypto` env.","metadata":{}},{"cell_type":"code","source":"api2 = API(train_df)","metadata":{"execution":{"iopub.status.busy":"2021-12-18T18:52:40.389461Z","iopub.execute_input":"2021-12-18T18:52:40.389845Z","iopub.status.idle":"2021-12-18T18:52:42.41605Z","shell.execute_reply.started":"2021-12-18T18:52:40.389806Z","shell.execute_reply":"2021-12-18T18:52:42.414923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Main loop\n\nAn example loop (making dummy predictions of Target=0) with a timing estimate for 100 days worth of data.","metadata":{}},{"cell_type":"code","source":"start_time = time.time()\n\nfor (data_df, pred_df) in api:\n    pred_df['Target'] = 0.\n    api.predict(pred_df)\n    \nfinish_time = time.time()\n\ntotal_time = finish_time - start_time\niter_speed = api.init_num_times/total_time\n\nprint(f\"Iterations/s = {round(iter_speed, 2)}.\")\ntest_iters = 60 * 24 * 100\nprint(f\"Expected number of iterations in test set is approx. {test_iters}\",\n      f\"which will take {round(test_iters / iter_speed, 2)}s\",\n      \"using this API emulator while making dummy predictions.\")","metadata":{"execution":{"iopub.status.busy":"2021-12-19T20:57:53.722758Z","iopub.execute_input":"2021-12-19T20:57:53.72352Z","iopub.status.idle":"2021-12-19T20:57:54.8406Z","shell.execute_reply.started":"2021-12-19T20:57:53.723471Z","shell.execute_reply":"2021-12-19T20:57:54.83899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Calculate your LB score!\n\nThe API now has a `score` method. This returns:\n+ a dataframe containing your predictions, the targets, and weights,\n+ the LB score: weighted correlation between predictions and targets.","metadata":{}},{"cell_type":"code","source":"df, score = api.score()\nprint(f\"Your LB score is {round(score, 4)}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-19T20:50:53.421619Z","iopub.execute_input":"2021-12-19T20:50:53.42192Z","iopub.status.idle":"2021-12-19T20:51:29.987883Z","shell.execute_reply.started":"2021-12-19T20:50:53.42189Z","shell.execute_reply":"2021-12-19T20:51:29.98687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A TL;DR example with random predictions","metadata":{}},{"cell_type":"code","source":"api = API(train_df)\n\nfor (data_df, pred_df) in api:\n    pred_df['Target'] = np.random.randn(len(pred_df), 1)\n    api.predict(pred_df)\n    \ndf, score = api.score()\n\nprint(f\"Your LB score is {round(score, 4)}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-19T22:04:29.34672Z","iopub.execute_input":"2021-12-19T22:04:29.347071Z","iopub.status.idle":"2021-12-19T22:12:36.1312Z","shell.execute_reply.started":"2021-12-19T22:04:29.347038Z","shell.execute_reply":"2021-12-19T22:12:36.130282Z"},"trusted":true},"execution_count":null,"outputs":[]}]}