{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🪙 G-Research Crypto - Starter LGBM Pipeline\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/30894/logos/header.png)\n\n\n### Just a simple pipeline going from zero to a valid submission\n\nWe train one `LGBMRegressor` for each asset over a very very naive set of features (the input dataframe `['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']`), we get the predictions correctly using the iterator and we submit. No validation for now, no cross validation... nothing at all lol: just the bare pipeline!\n\n\n## Please _DO_ upvote if you find this useful!\n\n## References:\n* [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n* [Basic Submission Template](https://www.kaggle.com/sohier/basic-submission-template)\n* [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n\n\n\n__Changelog__\n\n__V5__: Added two non-timely features from the tutorial: `Upper_Shadow` and `Lower_Shadow`","metadata":{}},{"cell_type":"markdown","source":"This is a fork of the original notebook. My intention here to show is what can be done with this method to improve your score.","metadata":{}},{"cell_type":"markdown","source":"# Import and load dfs\n\nReferences: [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom lightgbm import LGBMRegressor\nimport gresearch_crypto\n\n\nTRAIN_CSV = '/kaggle/input/g-research-crypto-forecasting/train.csv'\nASSET_DETAILS_CSV = '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:06:25.810498Z","iopub.execute_input":"2021-11-30T17:06:25.811018Z","iopub.status.idle":"2021-11-30T17:06:28.154344Z","shell.execute_reply.started":"2021-11-30T17:06:25.810922Z","shell.execute_reply":"2021-11-30T17:06:28.153350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_CSV)\nasset_df = pd.read_csv(ASSET_DETAILS_CSV)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:10:20.458725Z","iopub.execute_input":"2021-11-30T17:10:20.459414Z","iopub.status.idle":"2021-11-30T17:11:26.652686Z","shell.execute_reply.started":"2021-11-30T17:10:20.459362Z","shell.execute_reply":"2021-11-30T17:11:26.651663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_df = asset_df.sort_values('Asset_ID')\nasset_df","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:12:06.820697Z","iopub.execute_input":"2021-11-30T17:12:06.821142Z","iopub.status.idle":"2021-11-30T17:12:06.845827Z","shell.execute_reply.started":"2021-11-30T17:12:06.821075Z","shell.execute_reply":"2021-11-30T17:12:06.844541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"markdown","source":"## Utility functions to train a model for one asset","metadata":{}},{"cell_type":"code","source":"def upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df[\"Open\"])\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# A utility function that builds the features from the original dataset\ndef get_features(df):\n    df_feature = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feature['Lower_Shadow'] = lower_shadow(df_feature)\n    df_feature['Upper_Shadow'] = upper_shadow(df_feature)\n    return df_feature\n\ndef get_model(df_train, asset_id):\n    df = df_train[df_train['Asset_ID'] == asset_id]\n    \n    model_df = get_features(df)\n    model_df['y'] = df['Target']\n    model_df = model_df.dropna(how='any')\n    \n    X = model_df.drop('y', axis=1)\n    y = model_df['y']\n    \n    model = LGBMRegressor(n_estimators=1000)\n    model.fit(X, y)\n    return X, y, model","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:52:39.645169Z","iopub.execute_input":"2021-11-30T17:52:39.645563Z","iopub.status.idle":"2021-11-30T17:52:39.657514Z","shell.execute_reply.started":"2021-11-30T17:52:39.645525Z","shell.execute_reply":"2021-11-30T17:52:39.656657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loop over all assets","metadata":{}},{"cell_type":"code","source":"X_all = {}\ny_all = {}\nmodels = {}\n\nfor asset_id, asset_name in zip(asset_df['Asset_ID'], asset_df['Asset_Name']):\n    print(f\"Training model for {asset_name:<16}  (ID={asset_id:<2})\")\n    X, y, model = get_model(train_df, asset_id)\n    X_all[asset_id], y_all[asset_id], models[asset_id] = X, y, model","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:54:36.265645Z","iopub.execute_input":"2021-11-30T17:54:36.265973Z","iopub.status.idle":"2021-11-30T18:00:48.202709Z","shell.execute_reply.started":"2021-11-30T17:54:36.265938Z","shell.execute_reply":"2021-11-30T18:00:48.201909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#  CHECKING MODEL INTERFACE\nx = get_features(train_df.iloc[1])\ny_preds = models[0].predict([x])\ny_preds[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-30T18:01:13.617732Z","iopub.execute_input":"2021-11-30T18:01:13.618296Z","iopub.status.idle":"2021-11-30T18:01:13.632748Z","shell.execute_reply.started":"2021-11-30T18:01:13.618258Z","shell.execute_reply":"2021-11-30T18:01:13.631905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict & submit\n\nReferences: [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n\nSomething that helped me understand this iterator was adding a pdb checkpoint inside of the for loop:\n\n```python\nimport pdb; pdb.set_trace()\n```\n\nSee [Python Debugging With Pdb](https://realpython.com/python-debugging-pdb/) if you want to use it and you don't know how to.\n","metadata":{"execution":{"iopub.status.busy":"2021-11-02T20:57:49.349459Z","iopub.status.idle":"2021-11-02T20:57:49.349757Z","shell.execute_reply.started":"2021-11-02T20:57:49.349596Z","shell.execute_reply":"2021-11-02T20:57:49.349613Z"}}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():\n        \n        model = models[row['Asset_ID']]\n        x_test = get_features(row)\n        y_pred = model.predict([x_test])[0]\n        \n        df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n        \n        \n        # Print just one sample row to get a feeling of what it looks like\n        if i == 0 and j == 0:\n            display(x_test)\n            \n    # Display the first prediction dataframe\n    if i == 0:\n        display(df_pred)\n\n    # Send submissions\n    env.predict(df_pred)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Run Faster Damn It!\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}