{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook we will be using Weights and Biases (W&B) integration for XGBoost for experiment tracking and use W&B Sweep for hyperparameter sweep. \n\nNote that I am using my own [fork](https://github.com/ayulockin/client) of the `wandb/client` repo where I have improved the existing integration for XGBoost. You can find the pending PR [here](https://github.com/wandb/client/pull/2929). It is expected to be merged soon. I hope this notebook shows you the benefits of using this callback. ","metadata":{}},{"cell_type":"markdown","source":"# Imports and Setup","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/ayulockin/client\n%cd client\n!pip -qq install .\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2021-12-01T16:21:20.579679Z","iopub.execute_input":"2021-12-01T16:21:20.580025Z","iopub.status.idle":"2021-12-01T16:22:00.219880Z","shell.execute_reply.started":"2021-12-01T16:21:20.579937Z","shell.execute_reply":"2021-12-01T16:22:00.218814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport json\nimport time\nimport numpy as np\nimport pandas as pd\nfrom datetime import datetime\n\nimport xgboost as xgb\nfrom xgboost.callback import EarlyStopping\nfrom sklearn.metrics import mean_squared_error","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-12-01T07:29:06.037726Z","iopub.execute_input":"2021-12-01T07:29:06.038343Z","iopub.status.idle":"2021-12-01T07:29:06.882557Z","shell.execute_reply.started":"2021-12-01T07:29:06.0383Z","shell.execute_reply":"2021-12-01T07:29:06.881834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The existing integration of XGBoost (`wandb_callback`) uses an old style callback that will be deprecated in favor of `WandbCallback`. ","metadata":{}},{"cell_type":"code","source":"import wandb\nfrom wandb.xgboost import WandbCallback\n\n# Login\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:37:22.768538Z","iopub.execute_input":"2021-12-01T07:37:22.769131Z","iopub.status.idle":"2021-12-01T07:37:36.290033Z","shell.execute_reply.started":"2021-12-01T07:37:22.769093Z","shell.execute_reply":"2021-12-01T07:37:36.289338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Dataset\n\nIf you haven't already check out the [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition).","metadata":{}},{"cell_type":"code","source":"crypto_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\nassets = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv').sort_values(\"Asset_ID\").reset_index(drop=True)\ncrypto_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:37:49.456802Z","iopub.execute_input":"2021-12-01T07:37:49.457072Z","iopub.status.idle":"2021-12-01T07:38:42.677998Z","shell.execute_reply.started":"2021-12-01T07:37:49.457044Z","shell.execute_reply":"2021-12-01T07:38:42.677037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare Split","metadata":{}},{"cell_type":"code","source":"crypto_df['datetime'] = pd.to_datetime(crypto_df['timestamp'], unit='s')\ntrain_df = crypto_df[crypto_df['datetime'] < '2021-06-13 00:00:00']\nvalid_df = crypto_df[crypto_df['datetime'] >= '2021-06-13 00:00:00']\n\nprint(\"Number of samples in train_df: \", len(train_df))\nprint(\"Number of samples in valid_df: \", len(valid_df))","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:43:00.088703Z","iopub.execute_input":"2021-12-01T07:43:00.089392Z","iopub.status.idle":"2021-12-01T07:43:02.676001Z","shell.execute_reply.started":"2021-12-01T07:43:00.089352Z","shell.execute_reply":"2021-12-01T07:43:02.674493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Features","metadata":{}},{"cell_type":"code","source":"# Features\nfeatues_col = [\"Count\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\", \"VWAP\"]\n\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\n\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)\n\ndef fill_nan_inf(df):\n    # Fill NaN values\n    df = df.fillna(0)\n    # Fill Inf values\n    df = df.replace([np.inf, -np.inf], 0)\n    \n    return df\n\ndef create_features(df, label=False):\n    \"\"\"\n    Create time series features\n    \"\"\"\n    # Build features\n    up_shadow = upper_shadow(df)\n    low_shadow = lower_shadow(df)    \n    five_min_log_return = log_return(df.VWAP, periods=5)\n    abs_one_min_log_return = log_return(df.VWAP,periods=1).abs()    \n    features = df[featues_col]\n\n    # Concat all the features into one dataframe\n    X = pd.concat([features, up_shadow, low_shadow, \n                   five_min_log_return, abs_one_min_log_return], \n                  axis=1)\n    \n    # Rename feature columns\n    X.columns = featues_col+[\"up_shadow\", \"low_shadow\", \"five_min_log_return\", \"abs_one_min_log_return\"]\n    \n    # Fill NaN and Inf\n    X = fill_nan_inf(X)\n    \n    if label:\n        y = df.Target\n        # Fill NaN and Inf\n        y = fill_nan_inf(y)\n        \n        return X, y\n    \n    return X","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:43:17.474522Z","iopub.execute_input":"2021-12-01T07:43:17.475208Z","iopub.status.idle":"2021-12-01T07:43:17.484664Z","shell.execute_reply.started":"2021-12-01T07:43:17.475171Z","shell.execute_reply":"2021-12-01T07:43:17.484004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will take just one crypto asset and find the best combination of hyperparameters to forecast the target for the validation set. \n\nThere are two reasons to do so:\n\n* `MultiOutputRegressor` wrapper for `XGBRegressor` is limited. We can't perform multi-output prediction/evaluation using this wrapper. Check out the GitHub issue [here](https://github.com/scikit-learn/scikit-learn/issues/15953). \n\n* We will use Bitcoin which is responsible to move the target because of having the hightest weightage (6.779922). ","metadata":{}},{"cell_type":"code","source":"# Get single crypto trading data\nbtc_train = train_df[train_df.Asset_ID==1]\nbtc_valid = valid_df[valid_df.Asset_ID==1]\n\n# Fill missing value\nbtc_train = btc_train.reindex(range(btc_train.index[0],btc_train.index[-1]+60,60),method='pad')\nbtc_valid = btc_valid.reindex(range(btc_valid.index[0],btc_valid.index[-1]+60,60),method='pad')\n\n# Create features\nX_train, y_train = create_features(btc_train, label=True)\nX_valid, y_valid = create_features(btc_valid, label=True)","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:43:24.236844Z","iopub.execute_input":"2021-12-01T07:43:24.237496Z","iopub.status.idle":"2021-12-01T07:43:24.839773Z","shell.execute_reply.started":"2021-12-01T07:43:24.237439Z","shell.execute_reply":"2021-12-01T07:43:24.83901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hyperparameter Tuning","metadata":{}},{"cell_type":"code","source":"def train():\n    with wandb.init() as run:\n        bst_params = {\n            'objective': 'reg:squarederror', \n            'n_estimators': 60,\n            'booster': run.config.booster,\n            'learning_rate': run.config.learning_rate,     \n            'gamma': run.config.gamma,\n            'max_depth': run.config.max_depth,\n            'min_child_weight': run.config.min_child_weight,  \n            'eval_metric': ['rmse'],\n            'tree_method': 'gpu_hist',\n        }\n\n        # Initialize the XGBoostClassifier\n        xgbmodel = xgb.XGBRegressor(**bst_params)\n\n        # Train the model, using the wandb_callback for logging\n        xgbmodel.fit(X_train, y_train, \n                     eval_set=[(X_valid, y_valid)],\n                     callbacks=[\n                         WandbCallback(log_model=True,\n                                       log_feature_importance=False,\n                                       define_metric=True)\n                     ],\n                     verbose=False)\n        \n        preds = xgbmodel.predict(X_valid)\n        rmse = np.sqrt(mean_squared_error(y_valid, preds))\n        print(\"RMSE: %f\" % (rmse))\n        wandb.log({\"Valid_RMSE\": rmse})","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:54:06.826066Z","iopub.execute_input":"2021-12-01T07:54:06.826363Z","iopub.status.idle":"2021-12-01T07:54:06.834369Z","shell.execute_reply.started":"2021-12-01T07:54:06.826331Z","shell.execute_reply":"2021-12-01T07:54:06.833403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sweep_config = {\n  \"name\" : \"btc_hyperparam_search\",\n  \"method\" : \"random\",\n  \"parameters\" : {\n    \"booster\": {\n        \"values\": [\"gbtree\", \"gblinear\"]\n    },\n    \"learning_rate\": {\n      \"min\": 0.001,\n      \"max\": 1.0\n    },\n    \"gamma\": {\n      \"min\": 0.001,\n      \"max\": 1.0\n    },\n    \"max_depth\": {\n        \"values\": [3, 5, 7]\n    },\n    \"min_child_weight\": {\n      \"min\": 1,\n      \"max\": 150\n    },\n    \"early_stopping_rounds\": {\n      \"values\" : [10, 20, 40, 40,]\n    },\n  }\n}\n\nsweep_id = wandb.sweep(sweep_config, project='btc_hyperparam_search')","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:54:15.983007Z","iopub.execute_input":"2021-12-01T07:54:15.983842Z","iopub.status.idle":"2021-12-01T07:54:17.217034Z","shell.execute_reply.started":"2021-12-01T07:54:15.9838Z","shell.execute_reply":"2021-12-01T07:54:17.215166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.agent(sweep_id, project='btc_hyperparam_search', function=train)","metadata":{"execution":{"iopub.status.busy":"2021-12-01T07:54:23.593572Z","iopub.execute_input":"2021-12-01T07:54:23.594551Z","iopub.status.idle":"2021-12-01T08:09:08.064686Z","shell.execute_reply.started":"2021-12-01T07:54:23.5945Z","shell.execute_reply":"2021-12-01T08:09:08.063735Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With just 2-3 lines of extra code you could monitor so much more and make sense of the most important hyperparameters for your XGBoost model. \n\n## [Check out the Sweeps page here $\\rightarrow$](https://wandb.ai/ayut/btc_hyperparam_search/sweeps/c1pgsztw?workspace=user-ayut)\n\n<!-- ![img](https://media.giphy.com/media/2ji2l1fsec75l6yoFD/giphy.gif) -->\n![sweepdemo_4.gif](https://s10.gifyu.com/images/sweepdemo_4.gif)","metadata":{}}]}