","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.021861,"end_time":"2021-11-28T07:36:02.237722","exception":false,"start_time":"2021-11-28T07:36:02.215861","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.021823,"end_time":"2021-11-28T07:36:02.281636","exception":false,"start_time":"2021-11-28T07:36:02.259813","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":">### π₯π₯ Crypto Prediction: Xgboost Regressor π₯π₯\n>\n> **Just a simple pipeline going from zero to a valid submission**\n>\n> We train one `XGBRegressor` for each asset over a very very naive set of features (the input dataframe `['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']`), we get the predictions correctly using the iterator and we submit. No validation for now, no cross validation... nothing at all lol: just the bare pipeline!\n>\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n\n
\n\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | [Quant's Volatility Features](https://www.kaggle.com/yamqwe/crypto-prediction-volatility-features) |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | | β³Target Engineering |\n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |β³Fourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | β³Wavelets | \n\n____","metadata":{"papermill":{"duration":0.021578,"end_time":"2021-11-28T07:36:02.368518","exception":false,"start_time":"2021-11-28T07:36:02.34694","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content π\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.021432,"end_time":"2021-11-28T07:36:02.411849","exception":false,"start_time":"2021-11-28T07:36:02.390417","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [π Introduction](#introduction)\n\n2. [π Table Of Content](#outline) \n\n3. [π€Ώ Diving into the Data](#diving) \n\n4. [π Imports](#imports) \n\n5. [ποΈ Training](#training)\n\n6. [π° Submit to Kaggle](#submit)","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.02375,"end_time":"2021-11-28T07:36:02.461951","exception":false,"start_time":"2021-11-28T07:36:02.438201","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data π€Ώ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.026492,"end_time":"2021-11-28T07:36:02.519222","exception":false,"start_time":"2021-11-28T07:36:02.49273","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - π There are 14 coins in the dataset\n>\n> - π There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.021857,"end_time":"2021-11-28T07:36:02.564694","exception":false,"start_time":"2021-11-28T07:36:02.542837","status":"completed"},"tags":[]}},{"cell_type":"code","source":"css_file = '''\ndiv #notebook {\nbackground-color: white;\nfont-family: 'Open Sans', Helvetica, sans-serif;\nline-height: 20px;\n}\n\n#notebook-container {\nmargin-top: 2em;\npadding-top: 2em;\nborder-top: 4px solid %s; /* light orange */\n-webkit-box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n}\n\ndiv .input {\nmargin-bottom: 1em;\n}\n\n.rendered_html h1, .rendered_html h2, .rendered_html h3, .rendered_html h4, .rendered_html h5, .rendered_html h6 {\ncolor: %s; /* light orange */\nfont-weight: 600;\n}\n\n.rendered_html code {\n background-color: #efefef; /* light gray */\n}\n\n.CodeMirror {\ncolor: #8c8c8c; /* dark gray */\npadding: 0.7em;\n}\n\ndiv.input_area {\nborder: none;\n background-color: %s; /* rgba(229, 143, 101, 0.1); light orange [exactly #E58F65] */\n border-top: 2px solid %s; /* light orange */\n}\n\ndiv.input_prompt {\ncolor: %s; /* light blue */\n}\n\ndiv.output_prompt {\ncolor: %s; /* strong orange */\n}\n\ndiv.cell.selected:before, div.cell.selected.jupyter-soft-selected:before {\nbackground: %s; /* light orange */\n}\n\ndiv.cell.selected, div.cell.selected.jupyter-soft-selected {\n border-color: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected:before {\nbackground: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected {\nborder-color: %s; /* light orange */\n\n}\n'''\ndef to_rgb(h): return tuple(int(h[i:i+2], 16) for i in (0, 2, 4))\nmain_color_rgba = 'rgba(%s, %s, %s, 0.1)' % (to_rgb(main_color[1:])[0], to_rgb(main_color[1:])[1], to_rgb(main_color[1:])[2])\nopen('notebook.css', 'w').write(css_file % (main_color, main_color, main_color_rgba, main_color, prompt, strong_main_color, main_color, main_color, main_color, main_color))\nfrom IPython.core.display import display, HTML, Javascript\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75405d52-594f-4791-a45a-16354bc24cfa","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"775496bb-f633-46f4-a642-10e6cd29c1bb","jupyter":{"outputs_hidden":false},"papermill":{"duration":0.037642,"end_time":"2021-11-28T07:36:02.624398","exception":false,"start_time":"2021-11-28T07:36:02.586756","status":"completed"},"tags":[],"collapsed":false,"execution":{"iopub.status.busy":"2021-12-20T07:49:30.281395Z","iopub.execute_input":"2021-12-20T07:49:30.281874Z","iopub.status.idle":"2021-12-20T07:49:30.293719Z","shell.execute_reply.started":"2021-12-20T07:49:30.281838Z","shell.execute_reply":"2021-12-20T07:49:30.293035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\n# def nb(): return HTML(\"\")\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75d3e80e-0c31-48f5-88dd-5b6a3c79cf38","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"968efaca-ede8-4817-b3aa-baedaba3205e","jupyter":{"outputs_hidden":false},"papermill":{"duration":0.031566,"end_time":"2021-11-28T07:36:02.678512","exception":false,"start_time":"2021-11-28T07:36:02.646946","status":"completed"},"tags":[],"collapsed":false,"execution":{"iopub.status.busy":"2021-12-20T07:49:30.362335Z","iopub.execute_input":"2021-12-20T07:49:30.362551Z","iopub.status.idle":"2021-12-20T07:49:30.369225Z","shell.execute_reply.started":"2021-12-20T07:49:30.362525Z","shell.execute_reply":"2021-12-20T07:49:30.368384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
LightGBM = π³ + π + β’οΈ
","metadata":{"papermill":{"duration":0.022352,"end_time":"2021-11-28T07:36:02.723629","exception":false,"start_time":"2021-11-28T07:36:02.701277","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\nLightGBM is the current \"Meta\" on kaggle and it doesn't look like it is going to get Nerfed anytime soon! \nIt is basiclly a \"light\" version of gradient boosting machines framework that aims to increases efficiency and reduces memory usage.\n\n**It is usually THE Algorithm everyone on Kaggle try when facing a tabular dataset**\n\n>
TL;DR: What makes LightGBM so great:
\n>\n>1. LGBM was developed and maintained by Microsoft themselves so it gets constant maintenance and support.\n>2. Easy to use \n>3. Faster than nearly all other gradient boosting algorithms.\n>4. Usually the most powerful gradient boosting. \n\n\nIt is a **gradient boosting** model that makes use of tree based learning algorithms. It is considered to be a fast processing algorithm.\n\nWhile other algorithms trees grow horizontally, LightGBM algorithm grows vertically, meaning it grows leaf-wise and other algorithms grow level-wise. LightGBM chooses the leaf with large loss to grow. It can lower down more loss than a level wise algorithm when growing the same leaf.\n\n\n\n[Source of Image](https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/)\n\nLight GBM is prefixed as Light because of its high speed. Light GBM can handle the large size of data and takes lower memory to run.\n\nAnother reason why Light GBM is so popular is because it focuses on accuracy of results. LGBM also supports GPU learning and thus data scientists are widely using LGBM for data science application development.\n\n
Leaf growth technique in LightGBM
\n\nLightGBM uses leaf-wise (best-first) tree growth. It chooses to grow the leaf that minimizes the loss, allowing a growth of an imbalanced tree. Because it doesnβt grow level-wise, but leaf-wise, over-fitting can happen when data is small. In these cases, it is important to control the tree depth.\n\n
LightGBM vs XGBoost
\n\nbase learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting but rather through histogram approximations (XGBoost now has this functionality as well but it's still not as fast as LightGBM). \n\nThis results in slight decrease of predictive performance buy much larger increase of speed. This means more opportunity for feature engineering/experimentation/model tuning which inevitably yields larger increases in predictive performance. (Feature engineering are the key to winning most Kaggle competitions)\n\n\n
LightGBM vs Catboost
\n\nCatBoost is not used as much, mostly because it tends to be much slower than LightGBM and XGBoost. That being said, CatBoost is very different when it comes to the implementation of gradient boosting. This can give slightly more accurate predictions, in particular if you have large amounts of categorical features. Because rapid experimentation is vital in Kaggle competitions, LightGBM tends to be the go-to algorithm when first creating strong base learners.\n\nIn general, it is important to note that a large amount of approaches involves combining all three boosting algorithms in an ensemble. LightGBM, CatBoost, and XGBoost might be thrown together in a mix to create a strong ensemble. This is done to really squeeze spots on the leaderboard and it usually works.\n\n
\n\n____\n \nParameter Tuning is an important part that is usually done by data scientists to achieve a good accuracy, fast result and to deal with overfitting. Let us see quickly some of the parameter tuning you can do for better results.\nWhile, LightGBM has more than 100 parameters that are given in the [documentation of LightGBM](https://github.com/microsoft/LightGBM), we are going to check the most important ones.\n\n**num_leaves**: This parameter is responsible for the complexity of the model. I normally start by trying values in the range [10,100]. But if you have a solid heuristic to choose tree depth you can always use it and set num_leaves to 2^tree_depth - 1\n\n[LightGBM Documentation](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html) says in respect -\nThis is the main parameter to control the complexity of the tree model. Theoretically, we can set num_leaves = 2^(max_depth) to obtain the same number of leaves as depth-wise tree. However, this simple conversion is not good in practice. The reason is that a leaf-wise tree is typically much deeper than a depth-wise tree for a fixed number of leaves. Unconstrained depth can induce over-fitting. Thus, when trying to tune the num_leaves, we should let it be smaller than 2^(max_depth). For example, when the max_depth=7 the depth-wise tree can get good accuracy, but setting num_leaves to 127 may cause over-fitting, and setting it to 70 or 80 may get better accuracy than depth-wise.\n\n**Min_data_in_leaf**: Assigning bigger value to this parameter can result in underfitting of the model. Giving it a value of 100 or 1000 is sufficient for a large dataset.\n\n**Max_depth**: Controls the depth of the individual trees. Typical values range from a depth of 3β8 but it is not uncommon to see a tree depth of 1. Smaller depth trees are computationally efficient (but require more trees); however, higher depth trees allow the algorithm to capture unique interactions but also increase the risk of over-fitting. Larger training data sets are more tolerable to deeper trees.\n\n**num_iterations**: Num_iterations specifies the number of boosting iterations (trees to build). The more trees you build the more accurate your model can be at the cost of:\n - Longer training time\n - Higher chance of over-fitting\nSo typically start with a lower number of trees to build a baseline and increase it later when you want to squeeze the last % out of your model.\n\nIt is recommended to use smaller `learning_rate` with larger `num_iterations`. Also, we should use `early_stopping_rounds` if we go for higher `num_iterations` to stop your training when it is not learning anything useful.\n\n**early_stopping_rounds** - \"early stopping\" refers to stopping the training process if the model's performance on a given validation set does not improve for several consecutive iterations. This parameter will stop training if the validation metric is not improving after the last early stopping round. It should be defined in pair with a number of iterations. If we set it too large we increase the chance of over-fitting. **The rule of thumb is to have it at 10% of your `num_iterations`**.\n\n____\n \n
Other Parameters Overview
\n\n____\n \n**Parameters that control the trees of LightGBM**\n\n- num_leaves: controls the number of decision leaves in a single tree. there will be multiple trees in pool.\n- min_data_in_leaf: the minimum number of data/sample/count per leaf (default is 20; lower min_data_in_leaf means less conservative/control, potentially overfitting).\n- max_depth: this the height of a decision tree. if its more possibility of overfitting but too low may underfit.\n>**NOTE:** max_depth directly impacts:\n>1. The best value for the num_leaves parameter\n>2. Model Performance\n>3. Training Time\n\n____\n\n**Parameters For Better Accuracy**\n\n- Use large max_bin (may be slower)\n\n Use small learning_rate with large num_iterations\n\n- Use large num_leaves (may cause over-fitting)\n\n- Use bigger training data\n\n- Try dart\n\n____\n\n**Parameters for Dealing with Over-fitting**\n\n- Use small max_bin\n\n- Use small num_leaves\n\n- Use min_data_in_leaf and min_sum_hessian_in_leaf\n\n- Use bagging by set bagging_fraction and bagging_freq\n\n- Use feature sub-sampling by set feature_fraction\n\n- Use bigger training data\n\n- Try lambda_l1, lambda_l2 and min_gain_to_split for regularization\n\n- Try max_depth to avoid growing deep tree\n\n- Try extra_trees\n\n- Try increasing path_smooth\n\n____\n\n\n
How to tune LightGBM like a boss?
\n\nHyperparameters tuning guide:\n\n**objective**\n * When you change it affects other parameters\tSpecify the type of ML model\n * default- value regression\n * aliases- Objective_type\n\n**boosting**\n * If you set it RF, that would be a bagging approach\n * default- gbdt\n * Range- [gbdt, rf, dart, goss]\n * aliases- boosting_type\n\n**lambda_l1**\n * regularization parameter\n * default- 0.0\n * Range- [0, β]\n * aliases- reg_alpha\n * constraints- lambda_l1 >= 0.0\n\n**bagging_fraction**\n * randomly select part of data without resampling\n * default-1.0\n * range- [0, 1]\n * aliases- Subsample\n * constarints- 0.0 < bagging_fraction <= 1.0\n\n**bagging_freq**\n * default- 0.0\n * range- [0, β]\n * aliases- subsample_freq\n * bagging_fraction should be set to value smaller than 1.0 as well 0 means disable bagging\n\n**num_leaves**\n * max number of leaves in one tree\n * default- 31\n * Range- [1, β]\n * Note- 1 < num_leaves <= 131072\n\n**feature_fraction**\n * if you set it to 0.8, LightGBM will select 80% of features\n * default- 1.0\n * Range- [0, 1]\n * aliases- sub_feature\n * constarint- 0.0 < feature_fraction <= 1.0\n\n**max_depth**\n * default- [-1]\n * range- [-1, β]m\n * Larger is usually better, but overfitting speed increases.\n * limit the max depth Forr tree model\n\n**max_bin**\n * deal with over-fitting\n * default- 255\n * range- [2, β]\n * aliases- Histogram Binning\n * max_bin > 1\n\n**num_iterations**\n * number of boosting iterations\n * default- 100\n * range- [1, β]\n * AKA- Num_boost_round, n_iter\n * constarints- num_iterations >= 0\n\n**learning_rate**\n * default- 0.1\n * range- [0 1]\n * aliases- eta\n * general values- learning_rate > 0.0Typical: 0.05.\n\n**early_stopping_round**\n * will stop training if validation doesnβt improve in last early_stopping_round\n * Model Performance, Number of Iterations, Training Time\n * default- 0\n * Range- [0, β]\n\n**categorical_feature** \n * to sepecify or Handle categorical features\n * i.e LGBM automatically handels categorical variable we dont need to one hot encode them.\n\n**bagging_freq**\n * default-0.0\n * Range-[0, β]\n * aliases- subsample_freq\n * note- 0 means disable bagging; k means perform bagging at every k iteration\n * enable bagging, bagging_fraction should be set to value smaller than 1.0 as well\n\n**verbosity**\n * default- 0\n * range- [-β, β]\n * aliases- verbose\n * constraints- {< 0: Fatal, = 0: Error (Warning), = 1: Info, > 1}\n\n**min_data_in_leaf**\n * Can be used to deal with over-fitting:\n * default- 20\n * constarint-min_data_in_leaf >= 0 \n\n\n
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n ![](\"https://lightgbm.readthedocs.io/en/latest/_images/LightGBM_logo_black_text.svg\"){kind=logo}
![](\"https://i.ibb.co/2M2MsgH/upvote7.png\")