{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🪙💲 G-Research - Full overlap-exploiting (\"leaky\") solution\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/30894/logos/header.png)\n\n\n \n \n \n \n\n---\n# 👇👇👇\n# [🪙💲 Proposal for a meaningful LB + LGBM [S]](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n# 👆👆👆 \n# A proposal for a method to get comparable, non-leaky models \n# + one non-leaky model scoring `LB=0.018`\n\n---\n\n \n \n \n \n\n\n### After the early dissapointment with [my original model](https://www.kaggle.com/julian3833/g-research-starter-lgbm-pipeline-lb)'s LB contamination and while I make some time to work on a more reasonable starter model, I thought: The LB is _already_ contaminated and there is no going back, so why not? It is just a matter or time, and, actually, the LB, in the current situation, is meaningless.\n\n### Below you can see a 18-line submission that overfits the public LB fully, using the fact that the test period is contained in the train data to get the closest point for each test row.\n\nIt is not _actually_ leaky because the fact that the LB is part of the public data was publicly disclosed by the organizers. In my opinion, though, the statement was not clear enough regarding the fact that the public LB test period was contained in the `train.csv`, but I guess it doesn't matter.\n\nI don't know the public test time range, but here I'm assuming that is starts from the placeholder and goes on.\nThe placeholder `test.csv` starts at `'2021-06-13 00:00:00'`. I tried with the full `train.csv` but it takes a long time and it's probably going to fail the submission.\n\nThe LB doesn't seem to be meaningful at all as long as we don't know which training data points are leaking into the public test set and which data points are not. See [this discussion](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285289)\n\nReferences: [Stackoverflow: Pandas DataFrame How to query the closest datetime index?](https://stackoverflow.com/questions/42264848/pandas-dataframe-how-to-query-the-closest-datetime-index).\n\n\n\n----\n\n# See also:\n* __[Watch out!: test LB period is contained in the train csv](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285505) (topic)__\n\n\nBe aware of this situation if you forked one of the early versions of [🪙💲 G-Research - Starter LGBM Pipeline](https://www.kaggle.com/julian3833/g-research-starter-lgbm-pipeline-lb) as well.\n\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport gresearch_crypto\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\ndf = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv', \n                 usecols=['Target', 'Asset_ID','timestamp'], dtype={'Asset_ID': 'int8'})\ndf['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\ndf = df.set_index('datetime').drop('timestamp', axis=1)\ndf = df[(df.index.year == 2021) & (df.index.month > 5)]\ndfs = {asset_id: df[df['Asset_ID'] == asset_id].resample('1min').interpolate().copy() for asset_id in df['Asset_ID'].unique()}\ndel df\nfor df_test, df_pred in iter_test:\n    df_test['datetime'] = pd.to_datetime(df_test['timestamp'], unit='s')\n    for _, row in df_test.iterrows():\n        try:\n            df = dfs[row['Asset_ID']]\n            closest_train_sample = df.iloc[df.index.get_loc(row['datetime'], method='nearest')]\n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = closest_train_sample['Target']\n        except:\n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n    df_pred['Target'] = df_pred['Target'].fillna(0)\n    env.predict(df_pred)","metadata":{"execution":{"iopub.status.busy":"2022-01-24T19:23:44.304968Z","iopub.execute_input":"2022-01-24T19:23:44.305519Z","iopub.status.idle":"2022-01-24T19:24:39.881149Z","shell.execute_reply.started":"2022-01-24T19:23:44.305432Z","shell.execute_reply":"2022-01-24T19:24:39.880543Z"},"trusted":true},"execution_count":null,"outputs":[]}]}