{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🪙💲 Proposal for a meaningful LB + LGBM [S]\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/30894/logos/header.png)\n\n# Ok, so LB is meaningless. Now what?\n\nIt was confirmed that `2021-06-13 00:00:00` is the beginning of the test data of the public LB [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285505#1572680).\n\nPlease note that the very nature of the Target is \"leakable\". There is no fault at all at any level here. In any case, the organizers could have been more clear about the fact that the train csv contained the LB data, but they clarified that already. The target is publicly accesible with almost no effort and therefore there is no possible \"realistic leaderboard\".\n\nOn a final note before continuing, the LB was already meaningless even when the solutions were in the `0.01` range. Any solution using the full `train.csv` overfits. Period. I shared the `0.999` to rip off the band aid and move one.\n\n\nThere are __two problems at hand now__:\n\n## 1)  The public LB data is contained in the `train.csv`. \n\nThat is one of the problems, the one that breaks the LB.\n\nThe corolaries are various: \n* The leaderboard is meaningless, rendering the \"LB validation\" useless. \n* It turns impossible to assess whether a publicly shared model is useful or not.\n\nI think we can address it quite easily and find a common ground for sharing solutions (and actually using the LB score of them as an accurate measure of something). The leaderboard is broken and useless, but we can use the submission scores still to communicate, compare results and share good models.\n\nThe second problem is:\n\n## 2) The distance between the public and the private/final LB.\n\nI will ignore this problem for now, and focus on the first one: how to get some common ground for comparing and assessing LB scores.\n\n\n---\n\n\n\n# Not sure what I am talking about? Check these links:\n* __[Watch out!: test LB period is contained in the train csv](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285505) (topic)__\n* __[🪙💲 G-Research- Using the overlap fully [LB=0.99]](https://www.kaggle.com/julian3833/g-research-using-the-overlap-fully-lb-0-99) (notebook)__\n* __[Meaningful submission scores / sharing the lower boundary of public test data](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285289) (topic)__\n\n\n\n\n---","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# A proposal for a meaningful LB - \"Strict submissions\" [Strict] / [S]\n\nThe idea is quite obvious and probably you have thought of it. I just want to propose it as a \"convention\" so we have a common ground of understanding:\n\n<h2 style=\"text-align:center; background-color:#00FF00;padding:40px;border-radius: 30px;\"> Only keep data from before 2021-06-13 00:00:00.</h2>\n\nThis can be done by replacing the `pd.read_csv()` with the following function `read_csv_strict()` or however you want.\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-11-21T04:09:29.907112Z","iopub.execute_input":"2021-11-21T04:09:29.907462Z","iopub.status.idle":"2021-11-21T04:09:29.913288Z","shell.execute_reply.started":"2021-11-21T04:09:29.907427Z","shell.execute_reply":"2021-11-21T04:09:29.912623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = read_csv_strict()","metadata":{"execution":{"iopub.status.busy":"2021-11-21T04:09:32.262919Z","iopub.execute_input":"2021-11-21T04:09:32.263421Z","iopub.status.idle":"2021-11-21T04:10:17.722477Z","shell.execute_reply.started":"2021-11-21T04:09:32.263386Z","shell.execute_reply":"2021-11-21T04:10:17.72176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are other files around, if you use them make sure you do the same filtering. Of course, you will have a validation scheme inside the notebook: anything inside the notebook should be before `2021-06-13`, even the validation.\nOnly the submission iteration will see that time horizon.\n\n\n## If you do this, your public LB score can be labelled as \"Strict\". And it will be comparable to any other fellow kaggler reporting a \"Strict\" score.\n\nFor public shared kernels, we can use the tag `[Strict]` or `[S]` for labelling them. The visible scores of notebooks that always have run with this strict mechanism will be reallistic representations of the performance of the model.\n\n\n## So that is the proposal: __drop all the data  after `2021-06-13` and flag your work as `[Strict]` or `[S]`.__\n\n# I wish I see comparable results soon! \n# Can you beat my current `[S] 0.017`? It seems pretty weak, I bet you can! 😁😁\n\n\n\n---\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;","metadata":{}},{"cell_type":"markdown","source":"# Strict model: 🪙 G-Research Crypto - Starter LGBM Pipeline\n\n### This is just a copy of the original [🪙💲 G-Research- Starter LGBM Pipeline](https://www.kaggle.com/julian3833/g-research-starter-lgbm-pipeline), but without the LB score contamination with the \"leaky\" data. Therefore, this is a valid \"`[S]`\" (or \"`[Strict]`\") notebook, as it follows the convention.\n\n# Import and load dfs\n\nReferences: [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom lightgbm import LGBMRegressor\nfrom sklearn.linear_model import LinearRegression\nimport gresearch_crypto\n\n\nTRAIN_CSV = '../input/g-research-crypto-forecasting/train.csv'\nASSET_DETAILS_CSV = '../input/g-research-crypto-forecasting/asset_details.csv'\n\ndef read_csv_strict(file_name='../input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-11-21T04:10:17.724322Z","iopub.execute_input":"2021-11-21T04:10:17.725716Z","iopub.status.idle":"2021-11-21T04:10:17.731422Z","shell.execute_reply.started":"2021-11-21T04:10:17.725665Z","shell.execute_reply":"2021-11-21T04:10:17.730837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = read_csv_strict()","metadata":{"execution":{"iopub.status.busy":"2021-11-21T04:10:17.732834Z","iopub.execute_input":"2021-11-21T04:10:17.73307Z","iopub.status.idle":"2021-11-21T04:11:08.02049Z","shell.execute_reply.started":"2021-11-21T04:10:17.733045Z","shell.execute_reply":"2021-11-21T04:11:08.016572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details = pd.read_csv(ASSET_DETAILS_CSV).sort_values(\"Asset_ID\")\ndf_asset_details","metadata":{"execution":{"iopub.status.busy":"2021-11-21T04:11:08.027428Z","iopub.execute_input":"2021-11-21T04:11:08.02813Z","iopub.status.idle":"2021-11-21T04:11:08.080498Z","shell.execute_reply.started":"2021-11-21T04:11:08.028076Z","shell.execute_reply":"2021-11-21T04:11:08.079648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training\n\n## Utility functions to train a model for one asset\n\n### Features from [G-Research - Starter [0.361 LB]](https://www.kaggle.com/danofer/g-research-starter-0-361-lb)\n### And [[GResearch] Simple LGB Starter](https://www.kaggle.com/code1110/gresearch-simple-lgb-starter#Feature-Engineering)","metadata":{}},{"cell_type":"code","source":"# Two new features from the competition tutorial\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\n\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# A utility function to build features from the original df\n# It works for rows to, so we can reutilize it.\ndef get_features(df):\n    df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n    return df_feat\n\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n    df = df_train[df_train[\"Asset_ID\"] == asset_id]\n    \n    # TODO: Try different features here!\n    df_proc = get_features(df)\n    df_proc['y'] = df['Target']\n    df_proc = df_proc.replace([np.inf, -np.inf], np.nan).dropna(how=\"any\")\n    \n    X = df_proc.drop(\"y\", axis=1)\n    y = df_proc[\"y\"]\n\n    # TODO: Try different models here!\n    model = LGBMRegressor(n_estimators=10)\n    model.fit(X, y)\n    return X, y, model","metadata":{"execution":{"iopub.status.busy":"2021-11-21T03:51:54.879745Z","iopub.execute_input":"2021-11-21T03:51:54.880348Z","iopub.status.idle":"2021-11-21T03:51:54.8931Z","shell.execute_reply.started":"2021-11-21T03:51:54.880302Z","shell.execute_reply":"2021-11-21T03:51:54.892133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loop over all assets","metadata":{}},{"cell_type":"code","source":"Xs = {}\nys = {}\nmodels = {}\n\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    X, y, model = get_Xy_and_model_for_asset(df_train, asset_id)    \n    Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model","metadata":{"execution":{"iopub.status.busy":"2021-11-21T03:52:02.2924Z","iopub.execute_input":"2021-11-21T03:52:02.292811Z","iopub.status.idle":"2021-11-21T03:52:53.032089Z","shell.execute_reply.started":"2021-11-21T03:52:02.292779Z","shell.execute_reply":"2021-11-21T03:52:53.031102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the model interface\nx = get_features(df_train.iloc[1])\ny_pred = models[0].predict([x])\ny_pred[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-21T03:52:53.035184Z","iopub.execute_input":"2021-11-21T03:52:53.035748Z","iopub.status.idle":"2021-11-21T03:52:53.0796Z","shell.execute_reply.started":"2021-11-21T03:52:53.035706Z","shell.execute_reply":"2021-11-21T03:52:53.078602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict & submit\n\nReferences: [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n\nSomething that helped me understand this iterator was adding a pdb checkpoint inside of the for loop:\n\n```python\nimport pdb; pdb.set_trace()\n```\n\nSee [Python Debugging With Pdb](https://realpython.com/python-debugging-pdb/) if you want to use it and you don't know how to.\n","metadata":{}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():\n        \n        model = models[row['Asset_ID']]\n        x_test = get_features(row)\n        y_pred = model.predict([x_test])[0]\n        \n        df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n        \n        \n        # Print just one sample row to get a feeling of what it looks like\n        if i == 0 and j == 0:\n            display(x_test)\n\n    # Display the first prediction dataframe\n    if i == 0:\n        display(df_pred)\n\n    # Send submissions\n    env.predict(df_pred)","metadata":{"execution":{"iopub.status.busy":"2021-11-21T03:52:53.081226Z","iopub.execute_input":"2021-11-21T03:52:53.081693Z","iopub.status.idle":"2021-11-21T03:52:53.672597Z","shell.execute_reply.started":"2021-11-21T03:52:53.081646Z","shell.execute_reply":"2021-11-21T03:52:53.671587Z"},"trusted":true},"execution_count":null,"outputs":[]}]}