","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.021861,"end_time":"2021-11-28T07:36:02.237722","exception":false,"start_time":"2021-11-28T07:36:02.215861","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.021823,"end_time":"2021-11-28T07:36:02.281636","exception":false,"start_time":"2021-11-28T07:36:02.259813","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":">### ๐ฅ๐ฅ Crypto Prediction: Xgboost Regressor ๐ฅ๐ฅ\n>\n> **Just a simple pipeline going from zero to a valid submission**\n>\n> We train one `XGBRegressor` for each asset over a very very naive set of features (the input dataframe `['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']`), we get the predictions correctly using the iterator and we submit. No validation for now, no cross validation... nothing at all lol: just the bare pipeline!\n>\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n\n
","metadata":{"_cell_guid":"8af163ff-915a-4fae-aefe-6447e64952e5","_uuid":"b328cc9e-a536-4347-beed-d033e9f5ac6a","papermill":{"duration":0.021411,"end_time":"2021-11-28T07:36:02.32514","exception":false,"start_time":"2021-11-28T07:36:02.303729","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content ๐\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.021432,"end_time":"2021-11-28T07:36:02.411849","exception":false,"start_time":"2021-11-28T07:36:02.390417","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [๐ Introduction](#introduction)\n\n2. [๐ Table Of Content](#outline) \n\n3. [๐คฟ Diving into the Data](#diving) \n\n4. [๐ Imports](#imports) \n\n5. [๐๏ธ Training](#training)\n\n6. [๐ฐ Submit to Kaggle](#submit)","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.02375,"end_time":"2021-11-28T07:36:02.461951","exception":false,"start_time":"2021-11-28T07:36:02.438201","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data ๐คฟ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.026492,"end_time":"2021-11-28T07:36:02.519222","exception":false,"start_time":"2021-11-28T07:36:02.49273","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - ๐ There are 14 coins in the dataset\n>\n> - ๐ There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.021857,"end_time":"2021-11-28T07:36:02.564694","exception":false,"start_time":"2021-11-28T07:36:02.542837","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n
XGBoost = ๐ณ + ๐ง๐ป + ๐ช๐ป
","metadata":{"papermill":{"duration":0.022352,"end_time":"2021-11-28T07:36:02.723629","exception":false,"start_time":"2021-11-28T07:36:02.701277","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Libraries ๐\n","metadata":{"_cell_guid":"347324aa-490a-4556-bcc4-36379bdbec7c","_uuid":"03824ce3-fc08-4d19-926d-27edbb773466","papermill":{"duration":0.022277,"end_time":"2021-11-28T07:36:02.814188","exception":false,"start_time":"2021-11-28T07:36:02.791911","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"XGBoost is a favorite choice on kaggle and it doesn't look like it is going anywhere! \nIt is basiclly the a version of gradient boosting machines framework that made the approach so popular.\n\n**It is usually included in winning ensembles on Kaggle when solving a tabular problem**\n\nXGBoost algorithm provides large range of hyperparameters. In order to get the best performance out of it, we need to know to tune them.\n\n>
TL;DR: What makes XGBoost great:
\n>\n>1. XGBoost was the first wide-spread GBM framework so it has \"more mileage\" then all other frameworks.\n>2. Easy to use \n>3. When using GPU it is usually faster than nearly all other gradient boosting algorithms that use GPU.\n>4. A very powerful gradient boosting. \n\n
Leaf growth in XGBoost
\n\nXGboost splits up to the specified max_depth hyperparameter and then starts pruning the tree backwards and removes splits beyond which there is no positive gain. It uses this approach since sometimes a split of no loss reduction may be followed by a split with loss reduction. XGBoost can also perform leaf-wise tree growth (as LightGBM).\n\nNormally it is impossible to enumerate all the possible tree structures q. A greedy algorithm that starts from a single leaf and iteratively adds branches to the tree is used instead. Assume that I_L and I_R are the instance sets of left and right nodes after the split. Then the loss reduction after the split is given by,\n\n\n\n
XGBoost vs LightGBM
\n\nLightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value while XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split. Here instances mean observations/samples.\n\nLet's see how pre-sorting splitting works-\n\n- For each node, enumerate over all features\n\n- For each feature, sort the instances by feature value\n\n- Use a linear scan to decide the best split along that feature basis information gain\n\n- Take the best split solution along all the features\n\nIn simple terms, Histogram-based algorithm splits all the data points for a feature into discrete bins and uses these bins to find the split value of histogram. While, it is efficient than pre-sorted algorithm in training speed which enumerates all possible split points on the pre-sorted feature values, it is still behind GOSS in terms of speed.\n\n
XGBoost Model Parameters
\n\n>For an exhaustive overview of all parameters [see here](https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/)\n\n**objective [default=reg:linear]**\n\nThis defines the loss function to be minimized. Mostly used values are:\n\n- binary:logistic โlogistic regression for binary classification, returns predicted probability (not class)\n\n- multi:softmax โmulticlass classification using the softmax objective, returns predicted class (not probabilities)\nyou also need to set an additional num_class (number of classes) parameter defining the number of unique classes\n\n- multi:softprob โsame as softmax, but returns predicted probability of each data point belonging to each class.\n\n**eval_metric [ default according to objective ]**\n\nThe metric to be used for validation data. The default values are rmse for regression and error for classification.\nTypical values are:\n\n- rmse โ root mean square error\n\n- mae โ mean absolute error\n\n- logloss โ negative log-likelihood\n\n- error โ Binary classification error rate (0.5 threshold)\n\n- merror โ Multiclass classification error rate\n\n- mlogloss โ Multiclass logloss\n\n- auc: Area under the curve\n\n\n**eta [default=0.3]**\n\n- Analogous to learning rate in GBM.\n- Makes the model more robust by shrinking the weights on each step.\n- Typical final values to be used: 0.01-0.2\n\n**colsample_bytree**: We can create a random sample of the features (or columns) to use prior to creating each decision tree in the boosted model. That is, tuning Column Sub-sampling in XGBoost By Tree. This is controlled by the colsample_bytree parameter. The default value is 1.0 meaning that all columns are used in each decision tree. A fraction (e.g. 0.6) means a fraction of columns to be subsampled. We can evaluate values for colsample_bytree between 0.1 and 1.0 incrementing by 0.1.\n\n
Regularization in XGBoost
\n\nXGBoost adds built-in regularization to achieve accuracy gains beyond gradient boosting. Regularization is the process of adding information to reduce variance and prevent overfitting.\n\nAlthough data may be regularized through hyperparameter fine-tuning, regularized algorithms may also be attempted. For example, Ridge and Lasso are regularized machine learning alternatives to LinearRegression.\n\nXGBoost includes regularization as part of the learning objective, as contrasted with gradient boosting and random forests. The regularized parameters penalize complexity and smooth out the final weights to prevent overfitting. XGBoost is a regularized version of gradient boosting.\n\nMathematically, XGBoost's learning objective may be defined as follows:\n\n$$obj(ฮธ) = l(ฮธ) + ฮฉ (ฮธ)$$\n\nHere, **l(ฮธ)** is the loss function, which is the Mean Squared Error (MSE) for regression, or the log loss for classification, and **ฮฉ (ฮธ)** is the regularization function, a penalty term to prevent over-fitting. Including a regularization term as part of the objective function distinguishes XGBoost from most tree ensembles.\n\nThe learning objective for the th boosted tree can now be rewritten as follows:\n\n\n\n**reg_alpha and reg_lambda** : First note the loss function is defined as\n\n\n\n>So the above is how the regularized objective function looks like if you want to allow for the inclusion of a L1 and a L2 parameter in the same model\n\n`reg_alpha` and `reg_lambda` control the L1 and L2 regularization terms, which in this case limit how extreme the weights at the leaves can become. Higher values of alpha mean more L1 regularization. See the documentation [here](http://xgboost.readthedocs.io/en/latest///parameter.html#parameters-for-tree-booster).\n\nSince L1 regularization in GBDTs is applied to leaf scores rather than directly to features as in logistic regression, it actually serves to reduce the depth of trees. This in turn will tend to reduce the impact of less-predictive features. We might think of L1 regularization as more aggressive against less-predictive features than L2 regularization.\n\nThese two regularization terms have different effects on the weights; L2 regularization (controlled by the lambda term) encourages the weights to be small, whereas L1 regularization (controlled by the alpha term) encourages sparsity โ so it encourages weights to go to 0. This is helpful in models such as logistic regression, where you want some feature selection, but in decision trees weโve already selected our features, so zeroing their weights isnโt super helpful. For this reason, I found setting a high lambda value and a low (or 0) alpha value to be the most effective when regularizing.\n\n>[From this Paper](https://arxiv.org/pdf/1603.02754.pdf)\n\n\n
\n\n____\n\nBefore diving into the actual parameters of XGBoost, Let's define three types of parameters: General Parameters, Booster Parameters and Task Parameters.\n\n1. **General parameters** Relate to chosing which booster algorithm we will be using, usually tree or linear model.\n\n2. **Booster parameters** Are the actual parameters of the booster you have chosen.\n\n3. **Task parameters** Tells the framework what problem are we trying to solve. For example, regression tasks may use different parameters with ranking tasks.\n\n____\n\n\n
How to tune XGBoost like a boss?
\n\nHyperparameters tuning guide:\n\n
General Parameters
\n\n1. **booster** [default= gbtree ] \n - Which booster to use. Can be gbtree, gblinear or dart; \n - gbtree and dart use tree based models while gblinear uses linear functions.\n\n2. **verbosity** [default=1]\n - Verbosity of printing messages. Valid values are 0 (silent), 1 (warning), 2 (info), 3 (debug). \n - Sometimes XGBoost tries to change configurations based on heuristics, which is displayed as warning message. \n - If thereโs unexpected behaviour, please try to increase value of verbosity.\n\n3. **nthread** [default to maximum number of threads available if not set]\n - Number of parallel threads used to run XGBoost. When choosing it, please keep thread contention and hyperthreading in mind.\n\n____\n\n\n
Tree Booster Parameters
\n\n1. **eta [default=0.3, ]**\n - alias: learning_rate\n - Step size shrinkage used in update to prevents overfitting.\n - After each boosting step, we can directly get the weights of new features\n - It makes the model more robust by shrinking the weights on each step.\n - range: [0,1]\n\n2. **gamma [default=0]**\n - Minimum loss reduction required to make a further partition on a leaf node of the tree. \n - The larger gamma is, the more conservative the algorithm will be.\n - range: [0,โ]\n\n3. **max_depth [default=6]**\n\n - Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. \n - 0 is only accepted in lossguided growing policy when tree_method is set as hist or gpu_hist and it indicates no limit on depth. \n - Beware that XGBoost aggressively consumes memory when training a deep tree.\n - range: [0,โ] )\n\n4. **min_child_weight [default=1]**\n\n - its Minimum sum of instance weight (hessian) needed in a child. I\n - if the tree partition step results in a leaf node with the sum of instance weight less than min_child_weight, then the building process will give up further partitioning. \n - In linear regression task, this simply corresponds to minimum number of instances needed to be in each node. \n - The larger min_child_weight is, the more conservative the algorithm will be.\n - range: [0,โ]\n\n5. **max_delta_step [default=0]**\n\n - Maximum delta step we allow each leaf output to be. \n - If the value is set to 0, it means there is no constraint. \n - If it is set to a positive value, it can help making the update step more conservative. Usually this parameter is not needed, but it might help in logistic regression when class is extremely imbalanced. Set it to value of 1-10 might help control the update.\n - range: [0,โ]\n\n6. **subsample [default=1]**\n\n - It denotes the fraction of observations to be randomly samples for each tree.\n - Subsample ratio of the training instances.\n - Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. - This will prevent overfitting.\n - Subsampling will occur once in every boosting iteration.\n - Lower values make the algorithm more conservative and prevents overfitting but too small alues might lead to under-fitting.\n - typical values: 0.5-1\n - range: (0,1]\n\n7. **sampling_method [default= uniform]**\n\n - The method to use to sample the training instances.\n - **uniform:** each training instance has an equal probability of being selected. Typically set subsample >= 0.5 for good results.\n - **gradient_based:** the selection probability for each training instance is proportional to the regularized absolute value of gradients \n\n8. **colsample_bytree, colsample_bylevel, colsample_bynode [default=1]**\n\n>
This is a family of parameters for subsampling of columns.
\n>\n>**All colsample_by** parameters have a range of (0, 1], the default value of 1, and specify the fraction of columns to be subsampled.\n>\n>**lsample_bytree**s the subsample ratio of columns when constructing each tree. Subsampling occurs once for every tree constructed.\n>\n>**colsample_bylevel** is the subsample ratio of columns for each level. Subsampling occurs once for every new depth level reached in a tree. Columns are subsampled from the set of columns chosen for the current tree.\n>\n>**colsample_bynode** is the subsample ratio of columns for each node (split). Subsampling occurs once every time a new split is evaluated. Columns are subsampled from the set of columns chosen for the current level.\n>\n>**colsample_by** parameters work cumulatively. For instance, the combination **{'colsample_bytree':0.5, 'colsample_bylevel':0.5, 'colsample_bynode':0.5}** with 64 features will leave 8 features to choose from at each split.\n>\n\n9. **lambda [default=1]**\n - alias: reg_lambda\n - L2 regularization term on weights. \n - Increasing this value will make model more conservative.\n\n10. **alpha [default=0]**\n - alias: reg_alpha\n - L1 regularization term on weights.\n - Increasing this value will make model more conservative.\n\n11. **grow_policy [default= depthwise]**\n - Controls a way new nodes are added to the tree.\n - Currently supported only if tree_method is set to hist or gpu_hist.\n - **Choices:* - depthwise, lossguide\n - **depthwise:* - split at nodes closest to the root.\n - **lossguide:* - split at nodes with highest loss change.\n\n12. **max_leaves [default=0]**\n - Maximum number of nodes to be added. \n - Only relevant when grow_policy=lossguide is set.\n\n[Read More](https://xgboost.readthedocs.io/en/latest/parameter.html)\n\n____\n\n\n\n
Task Parameters
\n\n1. **objective [default=reg:squarederror]**\n\nIt defines the loss function to be minimized. Most commonly used values are given below -\n\n - reg:squarederror : regression with squared loss.\n\n - reg:squaredlogerror: regression with squared log loss 1/2[log(pred+1)โlog(label+1)]2. - All input labels are required to be greater than -1.\n\n - reg:logistic : logistic regression\n\n - binary:logistic : logistic regression for binary classification, output probability\n\n - binary:logitraw: logistic regression for binary classification, output score before logistic transformation\n\n - binary:hinge : hinge loss for binary classification. This makes predictions of 0 or 1, rather than producing probabilities.\n\n - multi:softmax : set XGBoost to do multiclass classification using the softmax objective, you also need to set num_class(number of classes)\n\n - multi:softprob : same as softmax, but output a vector of ndata nclass, which can be further reshaped to ndata nclass matrix. The result contains predicted probability of each data point belonging to each class.\n\n2. **eval_metric [default according to objective]**\n - The metric to be used for validation data.\n - The default values are rmse for regression, error for classification and mean average precision for ranking.\n - We can add multiple evaluation metrics.\n - Python users must pass the metrices as list of parameters pairs instead of map.\n - The most common values are given below -\n\n - rmse : root mean square error\n - mae : mean absolute error\n - logloss : negative log-likelihood\n - error : Binary classification error rate (0.5 threshold). \n - merror : Multiclass classification error rate.\n - mlogloss : Multiclass logloss\n - auc: Area under the curve\n - aucpr : Area under the PR curve\n \n \n\n
","metadata":{"papermill":{"duration":0.022787,"end_time":"2021-11-28T07:36:02.769106","exception":false,"start_time":"2021-11-28T07:36:02.746319","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import traceback\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\nimport gresearch_crypto\nfrom lightgbm import LGBMRegressor\n\nTRAIN_JAY = '../input/cryptocurrency-extra-data-binance-coin/orig_train.jay'\nASSET_DETAILS_JAY = '../input/cryptocurrency-extra-data-binance-coin/orig_asset_details.jay'","metadata":{"papermill":{"duration":1.913774,"end_time":"2021-11-28T07:36:04.795829","exception":false,"start_time":"2021-11-28T07:36:02.882055","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-01T08:21:38.197953Z","iopub.execute_input":"2021-12-01T08:21:38.198520Z","iopub.status.idle":"2021-12-01T08:21:41.072870Z","shell.execute_reply.started":"2021-12-01T08:21:38.198480Z","shell.execute_reply":"2021-12-01T08:21:41.071959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = dt.fread('../input/cryptocurrency-extra-data-binance-coin/orig_train.jay').to_pandas()\nprint(df_train.info())\ndf_train.head()","metadata":{"papermill":{"duration":20.468653,"end_time":"2021-11-28T07:36:25.288829","exception":false,"start_time":"2021-11-28T07:36:04.820176","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-01T08:21:41.078120Z","iopub.execute_input":"2021-12-01T08:21:41.079290Z","iopub.status.idle":"2021-12-01T08:22:01.649711Z","shell.execute_reply.started":"2021-12-01T08:21:41.079248Z","shell.execute_reply":"2021-12-01T08:22:01.648978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details = dt.fread('../input/cryptocurrency-extra-data-binance-coin/orig_asset_details.jay').to_pandas().sort_values(\"Asset_ID\")\nprint(df_asset_details.info())\ndf_asset_details.head()","metadata":{"papermill":{"duration":0.050158,"end_time":"2021-11-28T07:36:25.36318","exception":false,"start_time":"2021-11-28T07:36:25.313022","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-01T08:22:01.651277Z","iopub.execute_input":"2021-12-01T08:22:01.651792Z","iopub.status.idle":"2021-12-01T08:22:01.676855Z","shell.execute_reply.started":"2021-12-01T08:22:01.651750Z","shell.execute_reply":"2021-12-01T08:22:01.676102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{"papermill":{"duration":0.024225,"end_time":"2021-11-28T07:36:25.412631","exception":false,"start_time":"2021-11-28T07:36:25.388406","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Feature Extraction**","metadata":{"papermill":{"duration":0.02426,"end_time":"2021-11-28T07:36:25.510651","exception":false,"start_time":"2021-11-28T07:36:25.486391","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import xgboost as xgb\n\n# Two new features from the competition tutorial\ndef upper_shadow(df):\n return df['High'] - np.maximum(df['Close'], df['Open'])\n\ndef lower_shadow(df):\n return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# A utility function to build features from the original df\n# It works for rows to, so we can reutilize it.\ndef get_features(df):\n df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n return df_feat","metadata":{"papermill":{"duration":0.130913,"end_time":"2021-11-28T07:36:25.666076","exception":false,"start_time":"2021-11-28T07:36:25.535163","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-01T08:22:01.679177Z","iopub.execute_input":"2021-12-01T08:22:01.679467Z","iopub.status.idle":"2021-12-01T08:22:01.766728Z","shell.execute_reply.started":"2021-12-01T08:22:01.679430Z","shell.execute_reply":"2021-12-01T08:22:01.766063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Main Training Function**","metadata":{"papermill":{"duration":0.024945,"end_time":"2021-11-28T07:36:25.716157","exception":false,"start_time":"2021-11-28T07:36:25.691212","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import xgboost as xgb\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n df = df_train[df_train[\"Asset_ID\"] == asset_id]\n \n # TODO: Try different features here!\n df_proc = get_features(df)\n df_proc['y'] = df['Target']\n df_proc = df_proc.dropna(how=\"any\")\n \n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n \n model = xgb.XGBRegressor(\n n_estimators=500,\n max_depth=11,\n learning_rate=0.05,\n subsample=0.9,\n colsample_bytree=0.7,\n missing=-999,\n random_state=2020,\n tree_method='gpu_hist' # THE MAGICAL PARAMETER\n )\n model.fit(X, y)\n\n return X, y, model","metadata":{"papermill":{"duration":0.035218,"end_time":"2021-11-28T07:36:25.776185","exception":false,"start_time":"2021-11-28T07:36:25.740967","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-01T08:22:01.767999Z","iopub.execute_input":"2021-12-01T08:22:01.768254Z","iopub.status.idle":"2021-12-01T08:22:01.774997Z","shell.execute_reply.started":"2021-12-01T08:22:01.768218Z","shell.execute_reply":"2021-12-01T08:22:01.774039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loop over all assets","metadata":{"papermill":{"duration":0.024205,"end_time":"2021-11-28T07:36:25.825204","exception":false,"start_time":"2021-11-28T07:36:25.800999","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from time import time ","metadata":{"execution":{"iopub.status.busy":"2021-12-01T08:22:01.776516Z","iopub.execute_input":"2021-12-01T08:22:01.776811Z","iopub.status.idle":"2021-12-01T08:22:01.783617Z","shell.execute_reply.started":"2021-12-01T08:22:01.776775Z","shell.execute_reply":"2021-12-01T08:22:01.782828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xs = {}\nys = {}\nmodels = {}\n\nt1 = time()\n\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\", end=\" \")\n try:\n X, y, model = get_Xy_and_model_for_asset(df_train, asset_id) \n Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model\n except: \n Xs[asset_id], ys[asset_id], models[asset_id] = None, None, None \n print(f\" time spent {time()-t1} \")\n t1= time()","metadata":{"papermill":{"duration":11703.741092,"end_time":"2021-11-28T10:51:29.590958","exception":false,"start_time":"2021-11-28T07:36:25.849866","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-01T08:22:01.785008Z","iopub.execute_input":"2021-12-01T08:22:01.785295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the model interface\nx = get_features(df_train.iloc[1])\ny_pred = models[0].predict(pd.DataFrame([x]))\ny_pred[0]","metadata":{"papermill":{"duration":0.101664,"end_time":"2021-11-28T10:51:29.722678","exception":false,"start_time":"2021-11-28T10:51:29.621014","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit To Kaggle ๐ฐ\n","metadata":{"execution":{"iopub.status.busy":"2021-11-02T20:57:49.349459Z","iopub.status.idle":"2021-11-02T20:57:49.349757Z","shell.execute_reply":"2021-11-02T20:57:49.349613Z","shell.execute_reply.started":"2021-11-02T20:57:49.349596Z"},"papermill":{"duration":0.028827,"end_time":"2021-11-28T10:51:29.781263","exception":false,"start_time":"2021-11-28T10:51:29.752436","status":"completed"},"tags":[]}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n if models[row['Asset_ID']] is not None:\n try:\n model = models[row['Asset_ID']]\n x_test = get_features(row)\n y_pred = model.predict(pd.DataFrame([x_test]))[0]\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n except:\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n traceback.print_exc()\n else: \n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n \n env.predict(df_pred)","metadata":{"papermill":{"duration":0.716563,"end_time":"2021-11-28T10:51:30.527927","exception":false,"start_time":"2021-11-28T10:51:29.811364","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n ![](\"https://i.ibb.co/8bvJY8B/xgboost-logo.png\"){kind=logo}
![](\"https://i.ibb.co/2M2MsgH/upvote7.png\")