{
  "id": 123654,
  "title": "20 Private LB rank solution",
  "url": "/competitions/ashrae-energy-prediction/discussion/123654",
  "author_name": "Volodymyr",
  "post_date": "2019-12-29T10:52:49.816000",
  "votes": 18,
  "comment_count": 1,
  "views": 0,
  "content": "<p>It was a long run and finally it is completed and we can share some tricks and ideas that we have used. According to this official <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123462\">discussion</a> and this <a href=\"https://www.kaggle.com/robikscube/ashrae-leaderboard-and-shake\">Kernel</a> [ods.ai]PowerRangers has taken 20 Private LB rank. So let's start :)</p>\n\n<p>Sorry for all misspelling and not really good code style in Kernels and here too ( LGBT-&gt; LGBM, Bland-&gt; Blend, ...). We did not have enough time to write really good code base :( </p>\n\n<h1>Our team had 2 main obstacles:</h1>\n\n<ul>\n<li>Time (The exams were coming)</li>\n<li>Computing resources \nAs for computing resources - we carried out all our experiments and development in Kaggle Kernels. So we had to deal with RAM and execution time limitations.\n<h1>Our main scheme:</h1></li>\n</ul>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F17ea6a8a8d308c1f47886461830cc5e1%2Fwin_scheme.dio(1\" alt=\"\">.png?generation=1577617664090195&amp;alt=media)</p>\n\n<h1>First of all, we created some Baseline models:</h1>\n\n<ol>\n<li>We started from statistics - simple mean or median over each <code>meter</code>. Then same statistics over several categorical features ( <code>meter</code>, <code>day_of_week</code>, <code>building_id</code>, <code>month</code> ). You can find it in this <a href=\"https://www.kaggle.com/vladimirsydor/naivemeanpredictor\">here</a> <code>NaiveMeanModel</code> . Of course, they were performing really poor (1.39-1.4 on public LB). But we used these feature for more sophisticated models.</li>\n<li>Then we tried <a href=\"https://www.kaggle.com/vladimirsydor/randomforestbaseline\">RandomForestRegressor</a>. But it was pretty slow and perform not really good.</li>\n<li>Finally we realized that <img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F8a17e2376bd2e09f3d756e37077c1aae%2F2x3fpr.jpg?generation=1577608619775546&amp;alt=media\" alt=\"\"></li>\n</ol>\n\n<h2>So we made our first experiments with LGBM and passed into silver zone</h2>\n\n<h1>And now it was time for PREPROCESSING!</h1>\n\n<p><a href=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2Fe958aa1905d44ec6a5dd083a2b4e95e5%2FData%20Science%20Tom%20and%20Jerry.jpg?generation=1577608845429010&amp;alt=media\"></a></p>\n\n<h2>Preprocessing:</h2>\n\n<ul>\n<li>First off all - detecting outliers in nearly the same way, as in this <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122471\">discussion</a>, but less sophisticated.</li>\n<li>Weather data preprocessing - interpolation on NaNs and creating features <code>is_nan</code> for these columns </li>\n<li>Adding features: <code>day</code>, <code>day_of_week</code>, <code>month</code>,  rolling features ( <code>{feature}_mean_lag{window}</code> ), max/min features by categorical features ( <code>air_temperature_max</code>)</li>\n<li>Finally we tried to add Leak data from 2017-2018 but it was not a good idea for us. But adding leak data from 2016 gave us some more data for training</li>\n</ul>\n\n<h2>You can have more detailed look at our Data Preprocessing and Feature engineering:</h2>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/baseline-preprocessing\">Preprocessing</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks-train-fe\">Preprocessing + Feature Engineering + Leaks from 2016</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks\">Preprocessing + Leaks from 2016-2018 )</a> . Here we created several datasets, taking leaks from different sites in order to train uncorrelated models.</li>\n</ul>\n\n<p>Also most ideas ( and code :) ) were taken from these kernels:\n- <a href=\"https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks\">first</a>\n- The second one was deleted</p>\n\n<h1>Now was time for modelling!</h1>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F0a78a2dcebaf5a9d74f185f741a6db17%2F1x7P7gqjo8k2_bj2rTQWAfg.jpeg?generation=1577610244398959&amp;alt=media\" alt=\"\"></p>\n\n<h1>Models:</h1>\n\n<h2>1. Of course, LGBM:</h2>\n\n<ul>\n<li>We are nor really professionals in LGBM hyperparams optimization + it was training really long, so we did not spend a lot of time for hyperparams optimization. But what we find out is that it was not overfitting much and increasing the number of leaves mostly helped the model (we tried 145 and 82)</li>\n<li>Even with high LR our Boost has not converged even for 7k iterations. But Leak scores and Public LB scores did not really improve for Boost trained for 7k, comparing with boost on 5k. Also it was a great difference for Boost trained for 3k. We could not try more iterations, cause Kernel has time limitations :(</li>\n<li>Also we had 5 kernels for one Booost in order to train it in CV style and then blend results.</li>\n<li>One more interesting fact is that - all DataFrames that passed to LGBM model are converted to <code>float64</code> and if you have real BIG DATA, you will RUN OUT OF MEMORY, so you need to convert it into <code>np.array</code>. Small tip but it helped us a lot :)\nYou can take a look at our Boost models here:</li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/lgbt-on-pp-leaks-train-fe-fold-1\">with leaked data from train(2016)</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/lgbt-on-leaks-fold-1\">with leaked data from train(2016) and test(2017-2018)</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/lgbt-fold-1\">without leaked data</a></li>\n</ul>\n\n<h2>2. Neural Net ( going out from forest!!! )</h2>\n\n<p><a href=\"https://www.kaggle.com/vladimirsydor/nn-on-pp-leaks-train-fe-fold-1\">Kernel</a>\nInspired by <a href=\"https://www.kaggle.com/abazdyrev\">abazdyrev</a> and his <a href=\"https://www.kaggle.com/abazdyrev/energy-consumption-keras-approach\">Kernel</a>\n-  First of all, usual preprocessing for NN: scaling and label encoding for Embeddings. You can find it in  <code>PreprocessingUnit</code>\n-  So we have chosen Embeddings for categorical features, because we have really a lot of data and they could train ( I hope )\n- Also we tried several optimizers (Nadam, Adam, Adamax) and as for final activations ( no-activation and softplus). Best results were for softplus and Adamax. We did not try to submit NN predictions alone but on Leak Validation it even outperformed LGBM ( WOW!!! ).</p>\n\n<h1>And finallly we are ready for the most interesting part - BLEEEEEEEEEND</h1>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F93bdaffad2efe430a77c96dec638e842%2Findex.jpeg?generation=1577612207103871&amp;alt=media\" alt=\"\"></p>\n\n<p>We were inspired by this <a href=\"https://www.kaggle.com/khoongweihao/ashrae-leak-validation-bruteforce-heuristic-search\">kernel</a> , but it was too simple for us :)</p>\n\n<p><a href=\"https://www.kaggle.com/vladimirsydor/bland-by-leak\">Kernel</a>\n1. Firstly we gathered a lot of submissions  ( our and some public ones ), finally we had 32 .csv files\n2. Some public kernels and some our submissions were created with models trained on Leak data , so we can not use them for Leak Validation and they were excluded. Also we exclude some bad submissions. You can find out them in <code>EXCLUDE_LIST</code>\n3. Then we have to choose several submissions for BLENDING:\n- Firstly, we tried Hyperopt on indexes for median blending. We have taken 10 submissions. Also we penalized Hyperopt for taking same files for blending \n- Secondly, we have created some kind of Genetic Algorithm for the same purpose. Mostly it was taken from our practical work from the university.\n- Thirdly, we tried to stack submissions with one layer Perceptron in CV style but it was hardly overfitting, so we did not try it more.</p>\n\n<h1>Finally all that Leaked data was added to our final submission.</h1>\n\n<p><a href=\"https://www.kaggle.com/vladimirsydor/add-leak\">Kernel</a> </p>\n\n<h1>Also add some less valuable Kernels to build the complete scheme:</h1>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-nn-on-pp-leaks-train-fe\">Blending NN trained on different Folds</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-lgbt-on-pp-leaks-train-fe\">Blend LGBM (PP + Leaks Train + FE) on different folds </a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-lgbt-folds\">Blend LGBM on different folds </a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-lgbt-on-leaks\">Blend LGBM (trained on train + leaked data) on different folds </a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/leakaggregator\">Leak Aggregator </a></li>\n</ul>\n\n<h1>Great thanks to my team members:</h1>\n\n<p><a href=\"https://www.kaggle.com/zekamrozek\">Evgeniy</a> and <a href=\"https://www.kaggle.com/vladyelisieiev\">Vladislav</a>. They made a real good job!!!</p>\n\n<h2>Good Kaggling and happy Holidays !!!</h2>",
  "messages": [
    {
      "id": 705718,
      "postDate": "2019-12-29T10:52:49.817Z",
      "content": "<p>It was a long run and finally it is completed and we can share some tricks and ideas that we have used. According to this official <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123462\">discussion</a> and this <a href=\"https://www.kaggle.com/robikscube/ashrae-leaderboard-and-shake\">Kernel</a> [ods.ai]PowerRangers has taken 20 Private LB rank. So let's start :)</p>\n\n<p>Sorry for all misspelling and not really good code style in Kernels and here too ( LGBT-&gt; LGBM, Bland-&gt; Blend, ...). We did not have enough time to write really good code base :( </p>\n\n<h1>Our team had 2 main obstacles:</h1>\n\n<ul>\n<li>Time (The exams were coming)</li>\n<li>Computing resources \nAs for computing resources - we carried out all our experiments and development in Kaggle Kernels. So we had to deal with RAM and execution time limitations.\n<h1>Our main scheme:</h1></li>\n</ul>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F17ea6a8a8d308c1f47886461830cc5e1%2Fwin_scheme.dio(1\" alt=\"\">.png?generation=1577617664090195&amp;alt=media)</p>\n\n<h1>First of all, we created some Baseline models:</h1>\n\n<ol>\n<li>We started from statistics - simple mean or median over each <code>meter</code>. Then same statistics over several categorical features ( <code>meter</code>, <code>day_of_week</code>, <code>building_id</code>, <code>month</code> ). You can find it in this <a href=\"https://www.kaggle.com/vladimirsydor/naivemeanpredictor\">here</a> <code>NaiveMeanModel</code> . Of course, they were performing really poor (1.39-1.4 on public LB). But we used these feature for more sophisticated models.</li>\n<li>Then we tried <a href=\"https://www.kaggle.com/vladimirsydor/randomforestbaseline\">RandomForestRegressor</a>. But it was pretty slow and perform not really good.</li>\n<li>Finally we realized that <img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F8a17e2376bd2e09f3d756e37077c1aae%2F2x3fpr.jpg?generation=1577608619775546&amp;alt=media\" alt=\"\"></li>\n</ol>\n\n<h2>So we made our first experiments with LGBM and passed into silver zone</h2>\n\n<h1>And now it was time for PREPROCESSING!</h1>\n\n<p><a href=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2Fe958aa1905d44ec6a5dd083a2b4e95e5%2FData%20Science%20Tom%20and%20Jerry.jpg?generation=1577608845429010&amp;alt=media\"></a></p>\n\n<h2>Preprocessing:</h2>\n\n<ul>\n<li>First off all - detecting outliers in nearly the same way, as in this <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122471\">discussion</a>, but less sophisticated.</li>\n<li>Weather data preprocessing - interpolation on NaNs and creating features <code>is_nan</code> for these columns </li>\n<li>Adding features: <code>day</code>, <code>day_of_week</code>, <code>month</code>,  rolling features ( <code>{feature}_mean_lag{window}</code> ), max/min features by categorical features ( <code>air_temperature_max</code>)</li>\n<li>Finally we tried to add Leak data from 2017-2018 but it was not a good idea for us. But adding leak data from 2016 gave us some more data for training</li>\n</ul>\n\n<h2>You can have more detailed look at our Data Preprocessing and Feature engineering:</h2>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/baseline-preprocessing\">Preprocessing</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks-train-fe\">Preprocessing + Feature Engineering + Leaks from 2016</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks\">Preprocessing + Leaks from 2016-2018 )</a> . Here we created several datasets, taking leaks from different sites in order to train uncorrelated models.</li>\n</ul>\n\n<p>Also most ideas ( and code :) ) were taken from these kernels:\n- <a href=\"https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks\">first</a>\n- The second one was deleted</p>\n\n<h1>Now was time for modelling!</h1>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F0a78a2dcebaf5a9d74f185f741a6db17%2F1x7P7gqjo8k2_bj2rTQWAfg.jpeg?generation=1577610244398959&amp;alt=media\" alt=\"\"></p>\n\n<h1>Models:</h1>\n\n<h2>1. Of course, LGBM:</h2>\n\n<ul>\n<li>We are nor really professionals in LGBM hyperparams optimization + it was training really long, so we did not spend a lot of time for hyperparams optimization. But what we find out is that it was not overfitting much and increasing the number of leaves mostly helped the model (we tried 145 and 82)</li>\n<li>Even with high LR our Boost has not converged even for 7k iterations. But Leak scores and Public LB scores did not really improve for Boost trained for 7k, comparing with boost on 5k. Also it was a great difference for Boost trained for 3k. We could not try more iterations, cause Kernel has time limitations :(</li>\n<li>Also we had 5 kernels for one Booost in order to train it in CV style and then blend results.</li>\n<li>One more interesting fact is that - all DataFrames that passed to LGBM model are converted to <code>float64</code> and if you have real BIG DATA, you will RUN OUT OF MEMORY, so you need to convert it into <code>np.array</code>. Small tip but it helped us a lot :)\nYou can take a look at our Boost models here:</li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/lgbt-on-pp-leaks-train-fe-fold-1\">with leaked data from train(2016)</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/lgbt-on-leaks-fold-1\">with leaked data from train(2016) and test(2017-2018)</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/lgbt-fold-1\">without leaked data</a></li>\n</ul>\n\n<h2>2. Neural Net ( going out from forest!!! )</h2>\n\n<p><a href=\"https://www.kaggle.com/vladimirsydor/nn-on-pp-leaks-train-fe-fold-1\">Kernel</a>\nInspired by <a href=\"https://www.kaggle.com/abazdyrev\">abazdyrev</a> and his <a href=\"https://www.kaggle.com/abazdyrev/energy-consumption-keras-approach\">Kernel</a>\n-  First of all, usual preprocessing for NN: scaling and label encoding for Embeddings. You can find it in  <code>PreprocessingUnit</code>\n-  So we have chosen Embeddings for categorical features, because we have really a lot of data and they could train ( I hope )\n- Also we tried several optimizers (Nadam, Adam, Adamax) and as for final activations ( no-activation and softplus). Best results were for softplus and Adamax. We did not try to submit NN predictions alone but on Leak Validation it even outperformed LGBM ( WOW!!! ).</p>\n\n<h1>And finallly we are ready for the most interesting part - BLEEEEEEEEEND</h1>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F93bdaffad2efe430a77c96dec638e842%2Findex.jpeg?generation=1577612207103871&amp;alt=media\" alt=\"\"></p>\n\n<p>We were inspired by this <a href=\"https://www.kaggle.com/khoongweihao/ashrae-leak-validation-bruteforce-heuristic-search\">kernel</a> , but it was too simple for us :)</p>\n\n<p><a href=\"https://www.kaggle.com/vladimirsydor/bland-by-leak\">Kernel</a>\n1. Firstly we gathered a lot of submissions  ( our and some public ones ), finally we had 32 .csv files\n2. Some public kernels and some our submissions were created with models trained on Leak data , so we can not use them for Leak Validation and they were excluded. Also we exclude some bad submissions. You can find out them in <code>EXCLUDE_LIST</code>\n3. Then we have to choose several submissions for BLENDING:\n- Firstly, we tried Hyperopt on indexes for median blending. We have taken 10 submissions. Also we penalized Hyperopt for taking same files for blending \n- Secondly, we have created some kind of Genetic Algorithm for the same purpose. Mostly it was taken from our practical work from the university.\n- Thirdly, we tried to stack submissions with one layer Perceptron in CV style but it was hardly overfitting, so we did not try it more.</p>\n\n<h1>Finally all that Leaked data was added to our final submission.</h1>\n\n<p><a href=\"https://www.kaggle.com/vladimirsydor/add-leak\">Kernel</a> </p>\n\n<h1>Also add some less valuable Kernels to build the complete scheme:</h1>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-nn-on-pp-leaks-train-fe\">Blending NN trained on different Folds</a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-lgbt-on-pp-leaks-train-fe\">Blend LGBM (PP + Leaks Train + FE) on different folds </a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-lgbt-folds\">Blend LGBM on different folds </a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/bland-lgbt-on-leaks\">Blend LGBM (trained on train + leaked data) on different folds </a></li>\n<li><a href=\"https://www.kaggle.com/vladimirsydor/leakaggregator\">Leak Aggregator </a></li>\n</ul>\n\n<h1>Great thanks to my team members:</h1>\n\n<p><a href=\"https://www.kaggle.com/zekamrozek\">Evgeniy</a> and <a href=\"https://www.kaggle.com/vladyelisieiev\">Vladislav</a>. They made a real good job!!!</p>\n\n<h2>Good Kaggling and happy Holidays !!!</h2>",
      "rawMarkdown": "It was a long run and finally it is completed and we can share some tricks and ideas that we have used. According to this official [discussion](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123462) and this [Kernel](https://www.kaggle.com/robikscube/ashrae-leaderboard-and-shake) [ods.ai]PowerRangers has taken 20 Private LB rank. So let's start :)\n\nSorry for all misspelling and not really good code style in Kernels and here too ( LGBT-&gt; LGBM, Bland-&gt; Blend, ...). We did not have enough time to write really good code base :( \n\n# Our team had 2 main obstacles:\n- Time (The exams were coming)\n- Computing resources \nAs for computing resources - we carried out all our experiments and development in Kaggle Kernels. So we had to deal with RAM and execution time limitations.\n# Our main scheme:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F17ea6a8a8d308c1f47886461830cc5e1%2Fwin_scheme.dio(1).png?generation=1577617664090195&amp;alt=media)\n\n\n# First of all, we created some Baseline models:\n1. We started from statistics - simple mean or median over each `meter`. Then same statistics over several categorical features ( `meter`, `day_of_week`, `building_id`, `month` ). You can find it in this [here](https://www.kaggle.com/vladimirsydor/naivemeanpredictor) `NaiveMeanModel` . Of course, they were performing really poor (1.39-1.4 on public LB). But we used these feature for more sophisticated models.\n2. Then we tried [RandomForestRegressor](https://www.kaggle.com/vladimirsydor/randomforestbaseline). But it was pretty slow and perform not really good.\n3. Finally we realized that ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F8a17e2376bd2e09f3d756e37077c1aae%2F2x3fpr.jpg?generation=1577608619775546&amp;alt=media)\n\n## So we made our first experiments with LGBM and passed into silver zone\n\n# And now it was time for PREPROCESSING!\n[](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2Fe958aa1905d44ec6a5dd083a2b4e95e5%2FData%20Science%20Tom%20and%20Jerry.jpg?generation=1577608845429010&amp;alt=media)\n\n## Preprocessing:\n- First off all - detecting outliers in nearly the same way, as in this [discussion](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122471), but less sophisticated.\n- Weather data preprocessing - interpolation on NaNs and creating features `is_nan` for these columns \n- Adding features: `day`, `day_of_week`, `month`,  rolling features ( `{feature}_mean_lag{window}` ), max/min features by categorical features ( `air_temperature_max`)\n- Finally we tried to add Leak data from 2017-2018 but it was not a good idea for us. But adding leak data from 2016 gave us some more data for training\n\n## You can have more detailed look at our Data Preprocessing and Feature engineering:\n- [Preprocessing](https://www.kaggle.com/vladimirsydor/baseline-preprocessing)\n- [Preprocessing + Feature Engineering + Leaks from 2016](https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks-train-fe)\n- [Preprocessing + Leaks from 2016-2018 )](https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks) . Here we created several datasets, taking leaks from different sites in order to train uncorrelated models.\n\nAlso most ideas ( and code :) ) were taken from these kernels:\n- [first](https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks)\n- The second one was deleted\n\n# Now was time for modelling!\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F0a78a2dcebaf5a9d74f185f741a6db17%2F1x7P7gqjo8k2_bj2rTQWAfg.jpeg?generation=1577610244398959&amp;alt=media)\n\n# Models:\n## 1. Of course, LGBM:\n-  We are nor really professionals in LGBM hyperparams optimization + it was training really long, so we did not spend a lot of time for hyperparams optimization. But what we find out is that it was not overfitting much and increasing the number of leaves mostly helped the model (we tried 145 and 82)\n- Even with high LR our Boost has not converged even for 7k iterations. But Leak scores and Public LB scores did not really improve for Boost trained for 7k, comparing with boost on 5k. Also it was a great difference for Boost trained for 3k. We could not try more iterations, cause Kernel has time limitations :(\n- Also we had 5 kernels for one Booost in order to train it in CV style and then blend results.\n- One more interesting fact is that - all DataFrames that passed to LGBM model are converted to `float64` and if you have real BIG DATA, you will RUN OUT OF MEMORY, so you need to convert it into `np.array`. Small tip but it helped us a lot :)\nYou can take a look at our Boost models here:\n- [with leaked data from train(2016)](https://www.kaggle.com/vladimirsydor/lgbt-on-pp-leaks-train-fe-fold-1)\n- [with leaked data from train(2016) and test(2017-2018)](https://www.kaggle.com/vladimirsydor/lgbt-on-leaks-fold-1)\n- [without leaked data](https://www.kaggle.com/vladimirsydor/lgbt-fold-1)\n\n## 2. Neural Net ( going out from forest!!! )\n[Kernel](https://www.kaggle.com/vladimirsydor/nn-on-pp-leaks-train-fe-fold-1)\nInspired by [abazdyrev](https://www.kaggle.com/abazdyrev) and his [Kernel](https://www.kaggle.com/abazdyrev/energy-consumption-keras-approach)\n-  First of all, usual preprocessing for NN: scaling and label encoding for Embeddings. You can find it in  `PreprocessingUnit`\n-  So we have chosen Embeddings for categorical features, because we have really a lot of data and they could train ( I hope )\n- Also we tried several optimizers (Nadam, Adam, Adamax) and as for final activations ( no-activation and softplus). Best results were for softplus and Adamax. We did not try to submit NN predictions alone but on Leak Validation it even outperformed LGBM ( WOW!!! ).\n\n# And finallly we are ready for the most interesting part - BLEEEEEEEEEND\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F93bdaffad2efe430a77c96dec638e842%2Findex.jpeg?generation=1577612207103871&amp;alt=media)\n\nWe were inspired by this [kernel](https://www.kaggle.com/khoongweihao/ashrae-leak-validation-bruteforce-heuristic-search) , but it was too simple for us :)\n\n[Kernel](https://www.kaggle.com/vladimirsydor/bland-by-leak)\n1. Firstly we gathered a lot of submissions  ( our and some public ones ), finally we had 32 .csv files\n2. Some public kernels and some our submissions were created with models trained on Leak data , so we can not use them for Leak Validation and they were excluded. Also we exclude some bad submissions. You can find out them in `EXCLUDE_LIST`\n3. Then we have to choose several submissions for BLENDING:\n- Firstly, we tried Hyperopt on indexes for median blending. We have taken 10 submissions. Also we penalized Hyperopt for taking same files for blending \n- Secondly, we have created some kind of Genetic Algorithm for the same purpose. Mostly it was taken from our practical work from the university.\n- Thirdly, we tried to stack submissions with one layer Perceptron in CV style but it was hardly overfitting, so we did not try it more.\n\n\n# Finally all that Leaked data was added to our final submission.  \n[Kernel](https://www.kaggle.com/vladimirsydor/add-leak) \n\n# Also add some less valuable Kernels to build the complete scheme:\n- [Blending NN trained on different Folds](https://www.kaggle.com/vladimirsydor/bland-nn-on-pp-leaks-train-fe)\n- [Blend LGBM (PP + Leaks Train + FE) on different folds ](https://www.kaggle.com/vladimirsydor/bland-lgbt-on-pp-leaks-train-fe)\n- [Blend LGBM on different folds ](https://www.kaggle.com/vladimirsydor/bland-lgbt-folds)\n- [Blend LGBM (trained on train + leaked data) on different folds ](https://www.kaggle.com/vladimirsydor/bland-lgbt-on-leaks)\n- [Leak Aggregator ](https://www.kaggle.com/vladimirsydor/leakaggregator)\n\n#Great thanks to my team members:\n [Evgeniy](https://www.kaggle.com/zekamrozek) and [Vladislav](https://www.kaggle.com/vladyelisieiev). They made a real good job!!!\n\n## Good Kaggling and happy Holidays !!!",
      "votes": 18
    },
    {
      "id": 705846,
      "postDate": "2019-12-29T15:14:43.740Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 705846,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-12-29T15:14:43.740000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "705718": "It was a long run and finally it is completed and we can share some tricks and ideas that we have used. According to this official [discussion](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123462) and this [Kernel](https://www.kaggle.com/robikscube/ashrae-leaderboard-and-shake) [ods.ai]PowerRangers has taken 20 Private LB rank. So let's start :)\n\nSorry for all misspelling and not really good code style in Kernels and here too ( LGBT-&gt; LGBM, Bland-&gt; Blend, ...). We did not have enough time to write really good code base :( \n\n# Our team had 2 main obstacles:\n- Time (The exams were coming)\n- Computing resources \nAs for computing resources - we carried out all our experiments and development in Kaggle Kernels. So we had to deal with RAM and execution time limitations.\n# Our main scheme:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F17ea6a8a8d308c1f47886461830cc5e1%2Fwin_scheme.dio(1).png?generation=1577617664090195&amp;alt=media)\n\n\n# First of all, we created some Baseline models:\n1. We started from statistics - simple mean or median over each `meter`. Then same statistics over several categorical features ( `meter`, `day_of_week`, `building_id`, `month` ). You can find it in this [here](https://www.kaggle.com/vladimirsydor/naivemeanpredictor) `NaiveMeanModel` . Of course, they were performing really poor (1.39-1.4 on public LB). But we used these feature for more sophisticated models.\n2. Then we tried [RandomForestRegressor](https://www.kaggle.com/vladimirsydor/randomforestbaseline). But it was pretty slow and perform not really good.\n3. Finally we realized that ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F8a17e2376bd2e09f3d756e37077c1aae%2F2x3fpr.jpg?generation=1577608619775546&amp;alt=media)\n\n## So we made our first experiments with LGBM and passed into silver zone\n\n# And now it was time for PREPROCESSING!\n[](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2Fe958aa1905d44ec6a5dd083a2b4e95e5%2FData%20Science%20Tom%20and%20Jerry.jpg?generation=1577608845429010&amp;alt=media)\n\n## Preprocessing:\n- First off all - detecting outliers in nearly the same way, as in this [discussion](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122471), but less sophisticated.\n- Weather data preprocessing - interpolation on NaNs and creating features `is_nan` for these columns \n- Adding features: `day`, `day_of_week`, `month`,  rolling features ( `{feature}_mean_lag{window}` ), max/min features by categorical features ( `air_temperature_max`)\n- Finally we tried to add Leak data from 2017-2018 but it was not a good idea for us. But adding leak data from 2016 gave us some more data for training\n\n## You can have more detailed look at our Data Preprocessing and Feature engineering:\n- [Preprocessing](https://www.kaggle.com/vladimirsydor/baseline-preprocessing)\n- [Preprocessing + Feature Engineering + Leaks from 2016](https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks-train-fe)\n- [Preprocessing + Leaks from 2016-2018 )](https://www.kaggle.com/vladimirsydor/baseline-preprocessing-leaks) . Here we created several datasets, taking leaks from different sites in order to train uncorrelated models.\n\nAlso most ideas ( and code :) ) were taken from these kernels:\n- [first](https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks)\n- The second one was deleted\n\n# Now was time for modelling!\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F0a78a2dcebaf5a9d74f185f741a6db17%2F1x7P7gqjo8k2_bj2rTQWAfg.jpeg?generation=1577610244398959&amp;alt=media)\n\n# Models:\n## 1. Of course, LGBM:\n-  We are nor really professionals in LGBM hyperparams optimization + it was training really long, so we did not spend a lot of time for hyperparams optimization. But what we find out is that it was not overfitting much and increasing the number of leaves mostly helped the model (we tried 145 and 82)\n- Even with high LR our Boost has not converged even for 7k iterations. But Leak scores and Public LB scores did not really improve for Boost trained for 7k, comparing with boost on 5k. Also it was a great difference for Boost trained for 3k. We could not try more iterations, cause Kernel has time limitations :(\n- Also we had 5 kernels for one Booost in order to train it in CV style and then blend results.\n- One more interesting fact is that - all DataFrames that passed to LGBM model are converted to `float64` and if you have real BIG DATA, you will RUN OUT OF MEMORY, so you need to convert it into `np.array`. Small tip but it helped us a lot :)\nYou can take a look at our Boost models here:\n- [with leaked data from train(2016)](https://www.kaggle.com/vladimirsydor/lgbt-on-pp-leaks-train-fe-fold-1)\n- [with leaked data from train(2016) and test(2017-2018)](https://www.kaggle.com/vladimirsydor/lgbt-on-leaks-fold-1)\n- [without leaked data](https://www.kaggle.com/vladimirsydor/lgbt-fold-1)\n\n## 2. Neural Net ( going out from forest!!! )\n[Kernel](https://www.kaggle.com/vladimirsydor/nn-on-pp-leaks-train-fe-fold-1)\nInspired by [abazdyrev](https://www.kaggle.com/abazdyrev) and his [Kernel](https://www.kaggle.com/abazdyrev/energy-consumption-keras-approach)\n-  First of all, usual preprocessing for NN: scaling and label encoding for Embeddings. You can find it in  `PreprocessingUnit`\n-  So we have chosen Embeddings for categorical features, because we have really a lot of data and they could train ( I hope )\n- Also we tried several optimizers (Nadam, Adam, Adamax) and as for final activations ( no-activation and softplus). Best results were for softplus and Adamax. We did not try to submit NN predictions alone but on Leak Validation it even outperformed LGBM ( WOW!!! ).\n\n# And finallly we are ready for the most interesting part - BLEEEEEEEEEND\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F93bdaffad2efe430a77c96dec638e842%2Findex.jpeg?generation=1577612207103871&amp;alt=media)\n\nWe were inspired by this [kernel](https://www.kaggle.com/khoongweihao/ashrae-leak-validation-bruteforce-heuristic-search) , but it was too simple for us :)\n\n[Kernel](https://www.kaggle.com/vladimirsydor/bland-by-leak)\n1. Firstly we gathered a lot of submissions  ( our and some public ones ), finally we had 32 .csv files\n2. Some public kernels and some our submissions were created with models trained on Leak data , so we can not use them for Leak Validation and they were excluded. Also we exclude some bad submissions. You can find out them in `EXCLUDE_LIST`\n3. Then we have to choose several submissions for BLENDING:\n- Firstly, we tried Hyperopt on indexes for median blending. We have taken 10 submissions. Also we penalized Hyperopt for taking same files for blending \n- Secondly, we have created some kind of Genetic Algorithm for the same purpose. Mostly it was taken from our practical work from the university.\n- Thirdly, we tried to stack submissions with one layer Perceptron in CV style but it was hardly overfitting, so we did not try it more.\n\n\n# Finally all that Leaked data was added to our final submission.  \n[Kernel](https://www.kaggle.com/vladimirsydor/add-leak) \n\n# Also add some less valuable Kernels to build the complete scheme:\n- [Blending NN trained on different Folds](https://www.kaggle.com/vladimirsydor/bland-nn-on-pp-leaks-train-fe)\n- [Blend LGBM (PP + Leaks Train + FE) on different folds ](https://www.kaggle.com/vladimirsydor/bland-lgbt-on-pp-leaks-train-fe)\n- [Blend LGBM on different folds ](https://www.kaggle.com/vladimirsydor/bland-lgbt-folds)\n- [Blend LGBM (trained on train + leaked data) on different folds ](https://www.kaggle.com/vladimirsydor/bland-lgbt-on-leaks)\n- [Leak Aggregator ](https://www.kaggle.com/vladimirsydor/leakaggregator)\n\n#Great thanks to my team members:\n [Evgeniy](https://www.kaggle.com/zekamrozek) and [Vladislav](https://www.kaggle.com/vladyelisieiev). They made a real good job!!!\n\n## Good Kaggling and happy Holidays !!!",
    "705846": ""
  }
}