{ "id": 123496, "title": "13th place gold solution", "url": "/competitions/ashrae-energy-prediction/discussion/123496", "author_name": "Tim Yee", "post_date": "2019-12-28T06:20:54.025000", "votes": 25, "comment_count": 15, "views": 0, "content": "

Thanks to ASHRAE for providing this unclean dataset! Thanks to Kaggle for continuing and keeping the competition alive. Congrats to everyone who managed to survive the LB shakeup!

\n\n

Thanks to @rohanrao, @kailex, @nz0722, @aitude, @purist1024 for your excellent notebooks which had direct impact on helping me achieve this outcome.

\n\n

Summary

\n\n

Final 2 submissions: equal weighted blend of the following plus some regularization.\n1. https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks\n2. https://www.kaggle.com/rohanrao/ashrae-half-and-half (credit for original work https://www.kaggle.com/kailex/ac-dc)\n3. https://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\n4. https://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type

\n\n

public LB: 0.935 (1.032 w/o leak estimated) with aggressive regularization 0.80 and 0.91\npublic LB: 0.944 (1.039 w/o leak estimated) with conservative regularization 0.91 for all\npublic LB: 0.950 (1.045 w/o leak) no tricks - I did not select for submission

\n\n

What worked

\n\n
    \n
  1. Data Cleaning - garbage in, garbage out. This is probably the single most important aspect of the the competition. I did this manually by plotting heatmaps and also going into each building's meter to inspect the target meter readings if they looked reasonable. I also reverse engineered the heat map to show only zeros by added the following line:
    train_df = train_df.query('not (meter_reading != 0)')
    2. \n
  2. Regularization - (this is what I'm calling it, maybe some will call it postprocessing, coefficients, tricks, etc.) - multiplying by some value < 1.0. For aggressive regularization, I used two different values 0.80 for responsive meters and 0.91 for less responsive meters. I probed each site's meter individually by multiplying by 0.95 to start then as I went through all the meters, I noted which meter or meters dropped the public LB by 0.001, which I noted to be responsive to regularization). I got the idea from LANL Earthquake prediction. Fortunately, I was confident that the LB would not shake me down if I tried to overfit the LB this way. I ran some tests on the leaked sites 0,1,2,4,15 to test each site's meters effect to various values. 0.90-0.95 seemed fairly safe values to start out with when I began probing. I gained about 0.015 on LB through a series of 29 submissions, which took about 2 weeks. As my remaining submissions diminished, I ramped up the overall aggressiveness of regularization and began probing the responsive site's meters as a collective submission because there wasn't enough submissions to try all possible values. This gave between 0.004 - 0.005 reduction on Private LB score. Without any regularization, I would have ended up placing 35 on private LB, so this trick definitely gave my scores the extra kick to finish in gold.
    3. \n
  3. Feature Engineering - In the half-and-half model, I added a feature that grouped building_id, meter, weekday, hour and mean target encoded it using full train (after data cleaning). I got the idea from this unassuming kernel. In Kfold-lightgbm-without-leak-1-08 model, I added a feature that combined site and meter as a categorical with no mean encoding. I noticed that for some sites, the mean encoded bm_week_hour feature performed worse while others performed better, but overall, it seemed favorable.
    4. \n
  4. Validating using sites 0,1,2,4,15 - using actual test ground truths for various sites individually and together helped to monitor whether my experiments improved out of sample test data.
    5. \n
  5. Addition by Subtraction - removing certain features for certain models helped improve ground truth (GT) test validation as well as local cross validation (CV). For half-and-half model variant, I dropped site_id, sea_level_pressure, and precip_depth_1_hr. For kfold-lightgbm-without-leak-1-08 I dropped site_id, sea_level_pressure, wind_direction, wind_speed, year_built, floor_count. I removed holiday features for all existing models because they made CV and GT validation worse. For aligned-timestamp-lgbm-by-meter-type, I dropped all the lag3 features.
    6. \n
\n\n

Notes: very little hyper-parameter optimization was performed. Just very limited basic tuning.

\n\n

What didn't work

\n\n
    \n
  1. Training using leaked test GT labels - training with GT from 2017-2018 did not improve out of sample site_id. For ex: training with site_id 0 did not improve validation scores for sites 1,2,4,15 dramatically. I only performed that one test and realized that adding a site_id to testing doesn't improve LB scores for out of site validation because each site_id is it's own microcosm and behaves different from other sites.
    2. \n
  2. A lot of feature engineering did not work including weather features.
    3. \n
  3. CatBoost/XGBoost/NN Embeddings/Linear Regression/Kernel Ridge Regression(KRR)/KNN all performed worse. I tried building 1449 (1 for each building) and 2380 (1 for each building's meter) Linear Regression, KRR, and LGBM models. I tried blending with Catboost and NN Embeddings models separately, but GT validation didn't seem to improve.
    4. \n
  4. Training with full year of training data and validating using sites 0,1,2,4,15 didn't seem to help much.
    5. \n
  5. Correcting site 0 meter 0 didn't help - discussion
    6. \n
\n\n

Some experiments I tried

\n\n
    \n
  1. Using half-and-half methodology to split train into 2 separate halves - middle months (4-9)/ending months (1-3, 10-12) → performed worse. First half of the day 0-11, later half of the day 12-23. This split by hour in the day performed surprisingly well and GT validation had shown this, however I didn't select it for final submission blend. notebook
    2. \n
  2. L2 model - I stacked a model on top of individual model predictions (for the same folds), which only worked for half-and-half model. I left this out of final blend. I also tried ensembling through stacking final models using LGBM but there didn't seem to be any improvement in GT validation.
    3. \n
\n\n

What I didn't try

\n\n
    \n
  1. Divide and Conquer notebook - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.
    2. \n
  2. Different blending methods, different weights, etc. - I just kept it simple (introduced no additional complexity or bias)
    3. \n
  3. Making submission with certain models in final blend due to limited submissions. Namely I wanted to submit my blend with NN Embeddings, but based on GT validation, I filtered out a lot of potentially better performing blends. It is possible that sites 0,1,2,4,15 did not represent other sites well enough, but that was the trade off I had to make.
    4. \n
\n\n

notes: I mention these, because it may have been the difference for finishing closer in-the-money.

\n\n

What I learned

\n\n
    \n
  1. I took my mean target encoding game to a new level with this competition. Before this, I was stuck at basic single categorical feature mean target encoding. Below is the code I used to do multi categorical feature mean encoding:\n> bm_cols = ['building_id', 'meter', 'weekday', 'hour',]\ndf_train['hour'] = df_train['timestamp'].dt.hour\ndf_train['weekday'] = df_train['timestamp'].dt.weekday\nbm = df_train.groupby(bm_cols)['meter_reading'].mean().rename('bm_week_hour').to_frame()\ndf_train = df_train.merge(bm, right_index=True, left_on=bm_cols, how='left')
    2. \n
\n\n

Final submissions used for blending

\n\n

half-and-half variant\nkfold-lightgbm-without-leak-1-08 variant\naligned-timestamp-lgbm-by-meter-type variant

\n\n

Final thoughts

\n\n

It took me a year to finally reach a gold model. I was an absolute beginner when I started and I've learned a lot since joining Kaggle. I hope to continue competing and learning from all the bright and inspiring people on here. Although the kernels I blended with are all public, there were a lot of details that led to obtaining a gold model. Most notably, stable validation and logging each experiment meticulously so that after a few weeks or near the end, I wouldn't forget what led to improvements and what did not.

", "messages": [ { "id": 704911, "postDate": "2019-12-28T06:20:54.027Z", "content": "

Thanks to ASHRAE for providing this unclean dataset! Thanks to Kaggle for continuing and keeping the competition alive. Congrats to everyone who managed to survive the LB shakeup!

\n\n

Thanks to @rohanrao, @kailex, @nz0722, @aitude, @purist1024 for your excellent notebooks which had direct impact on helping me achieve this outcome.

\n\n

Summary

\n\n

Final 2 submissions: equal weighted blend of the following plus some regularization.\n1. https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks\n2. https://www.kaggle.com/rohanrao/ashrae-half-and-half (credit for original work https://www.kaggle.com/kailex/ac-dc)\n3. https://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\n4. https://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type

\n\n

public LB: 0.935 (1.032 w/o leak estimated) with aggressive regularization 0.80 and 0.91\npublic LB: 0.944 (1.039 w/o leak estimated) with conservative regularization 0.91 for all\npublic LB: 0.950 (1.045 w/o leak) no tricks - I did not select for submission

\n\n

What worked

\n\n
    \n
  1. Data Cleaning - garbage in, garbage out. This is probably the single most important aspect of the the competition. I did this manually by plotting heatmaps and also going into each building's meter to inspect the target meter readings if they looked reasonable. I also reverse engineered the heat map to show only zeros by added the following line:
    train_df = train_df.query('not (meter_reading != 0)')
    2. \n
  2. Regularization - (this is what I'm calling it, maybe some will call it postprocessing, coefficients, tricks, etc.) - multiplying by some value < 1.0. For aggressive regularization, I used two different values 0.80 for responsive meters and 0.91 for less responsive meters. I probed each site's meter individually by multiplying by 0.95 to start then as I went through all the meters, I noted which meter or meters dropped the public LB by 0.001, which I noted to be responsive to regularization). I got the idea from LANL Earthquake prediction. Fortunately, I was confident that the LB would not shake me down if I tried to overfit the LB this way. I ran some tests on the leaked sites 0,1,2,4,15 to test each site's meters effect to various values. 0.90-0.95 seemed fairly safe values to start out with when I began probing. I gained about 0.015 on LB through a series of 29 submissions, which took about 2 weeks. As my remaining submissions diminished, I ramped up the overall aggressiveness of regularization and began probing the responsive site's meters as a collective submission because there wasn't enough submissions to try all possible values. This gave between 0.004 - 0.005 reduction on Private LB score. Without any regularization, I would have ended up placing 35 on private LB, so this trick definitely gave my scores the extra kick to finish in gold.
    3. \n
  3. Feature Engineering - In the half-and-half model, I added a feature that grouped building_id, meter, weekday, hour and mean target encoded it using full train (after data cleaning). I got the idea from this unassuming kernel. In Kfold-lightgbm-without-leak-1-08 model, I added a feature that combined site and meter as a categorical with no mean encoding. I noticed that for some sites, the mean encoded bm_week_hour feature performed worse while others performed better, but overall, it seemed favorable.
    4. \n
  4. Validating using sites 0,1,2,4,15 - using actual test ground truths for various sites individually and together helped to monitor whether my experiments improved out of sample test data.
    5. \n
  5. Addition by Subtraction - removing certain features for certain models helped improve ground truth (GT) test validation as well as local cross validation (CV). For half-and-half model variant, I dropped site_id, sea_level_pressure, and precip_depth_1_hr. For kfold-lightgbm-without-leak-1-08 I dropped site_id, sea_level_pressure, wind_direction, wind_speed, year_built, floor_count. I removed holiday features for all existing models because they made CV and GT validation worse. For aligned-timestamp-lgbm-by-meter-type, I dropped all the lag3 features.
    6. \n
\n\n

Notes: very little hyper-parameter optimization was performed. Just very limited basic tuning.

\n\n

What didn't work

\n\n
    \n
  1. Training using leaked test GT labels - training with GT from 2017-2018 did not improve out of sample site_id. For ex: training with site_id 0 did not improve validation scores for sites 1,2,4,15 dramatically. I only performed that one test and realized that adding a site_id to testing doesn't improve LB scores for out of site validation because each site_id is it's own microcosm and behaves different from other sites.
    2. \n
  2. A lot of feature engineering did not work including weather features.
    3. \n
  3. CatBoost/XGBoost/NN Embeddings/Linear Regression/Kernel Ridge Regression(KRR)/KNN all performed worse. I tried building 1449 (1 for each building) and 2380 (1 for each building's meter) Linear Regression, KRR, and LGBM models. I tried blending with Catboost and NN Embeddings models separately, but GT validation didn't seem to improve.
    4. \n
  4. Training with full year of training data and validating using sites 0,1,2,4,15 didn't seem to help much.
    5. \n
  5. Correcting site 0 meter 0 didn't help - discussion
    6. \n
\n\n

Some experiments I tried

\n\n
    \n
  1. Using half-and-half methodology to split train into 2 separate halves - middle months (4-9)/ending months (1-3, 10-12) → performed worse. First half of the day 0-11, later half of the day 12-23. This split by hour in the day performed surprisingly well and GT validation had shown this, however I didn't select it for final submission blend. notebook
    2. \n
  2. L2 model - I stacked a model on top of individual model predictions (for the same folds), which only worked for half-and-half model. I left this out of final blend. I also tried ensembling through stacking final models using LGBM but there didn't seem to be any improvement in GT validation.
    3. \n
\n\n

What I didn't try

\n\n
    \n
  1. Divide and Conquer notebook - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.
    2. \n
  2. Different blending methods, different weights, etc. - I just kept it simple (introduced no additional complexity or bias)
    3. \n
  3. Making submission with certain models in final blend due to limited submissions. Namely I wanted to submit my blend with NN Embeddings, but based on GT validation, I filtered out a lot of potentially better performing blends. It is possible that sites 0,1,2,4,15 did not represent other sites well enough, but that was the trade off I had to make.
    4. \n
\n\n

notes: I mention these, because it may have been the difference for finishing closer in-the-money.

\n\n

What I learned

\n\n
    \n
  1. I took my mean target encoding game to a new level with this competition. Before this, I was stuck at basic single categorical feature mean target encoding. Below is the code I used to do multi categorical feature mean encoding:\n> bm_cols = ['building_id', 'meter', 'weekday', 'hour',]\ndf_train['hour'] = df_train['timestamp'].dt.hour\ndf_train['weekday'] = df_train['timestamp'].dt.weekday\nbm = df_train.groupby(bm_cols)['meter_reading'].mean().rename('bm_week_hour').to_frame()\ndf_train = df_train.merge(bm, right_index=True, left_on=bm_cols, how='left')
    2. \n
\n\n

Final submissions used for blending

\n\n

half-and-half variant\nkfold-lightgbm-without-leak-1-08 variant\naligned-timestamp-lgbm-by-meter-type variant

\n\n

Final thoughts

\n\n

It took me a year to finally reach a gold model. I was an absolute beginner when I started and I've learned a lot since joining Kaggle. I hope to continue competing and learning from all the bright and inspiring people on here. Although the kernels I blended with are all public, there were a lot of details that led to obtaining a gold model. Most notably, stable validation and logging each experiment meticulously so that after a few weeks or near the end, I wouldn't forget what led to improvements and what did not.

", "rawMarkdown": "Thanks to ASHRAE for providing this unclean dataset! Thanks to Kaggle for continuing and keeping the competition alive. Congrats to everyone who managed to survive the LB shakeup!\n\nThanks to @rohanrao, @kailex, @nz0722, @aitude, @purist1024 for your excellent notebooks which had direct impact on helping me achieve this outcome.\n\n## Summary\n\nFinal 2 submissions: equal weighted blend of the following plus some regularization.\n1. https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks\n2. https://www.kaggle.com/rohanrao/ashrae-half-and-half (credit for original work https://www.kaggle.com/kailex/ac-dc)\n3. https://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\n4. https://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\n\npublic LB: 0.935 (1.032 w/o leak estimated) with aggressive regularization 0.80 and 0.91\npublic LB: 0.944 (1.039 w/o leak estimated) with conservative regularization 0.91 for all\npublic LB: 0.950 (1.045 w/o leak) no tricks - I did not select for submission\n\n### What worked\n1. Data Cleaning - garbage in, garbage out. This is probably the single most important aspect of the the competition. I did this manually by plotting [heatmaps](https://www.kaggle.com/ganfear/missing-data-and-zeros-visualized) and also going into each building's meter to inspect the target meter readings if they looked reasonable. I also reverse engineered the heat map to show only zeros by added the following line:
`train_df = train_df.query('not (meter_reading != 0)')`\n2. Regularization - (this is what I'm calling it, maybe some will call it postprocessing, coefficients, tricks, etc.) - multiplying by some value < 1.0. For aggressive regularization, I used two different values 0.80 for responsive meters and 0.91 for less responsive meters. I probed each site's meter individually by multiplying by 0.95 to start then as I went through all the meters, I noted which meter or meters dropped the public LB by 0.001, which I noted to be responsive to regularization). I got the idea from [LANL Earthquake prediction](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/94324). Fortunately, I was confident that the LB would not shake me down if I tried to overfit the LB this way. I ran some tests on the leaked sites 0,1,2,4,15 to test each site's meters effect to various values. 0.90-0.95 seemed fairly safe values to start out with when I began probing. I gained about 0.015 on LB through a series of 29 submissions, which took about 2 weeks. As my remaining submissions diminished, I ramped up the overall aggressiveness of regularization and began probing the responsive site's meters as a collective submission because there wasn't enough submissions to try all possible values. This gave between 0.004 - 0.005 reduction on Private LB score. Without any regularization, I would have ended up placing 35 on private LB, so this trick definitely gave my scores the extra kick to finish in gold.\n3. Feature Engineering - In the half-and-half model, I added a feature that grouped `building_id`, `meter`, `weekday`, `hour` and mean target encoded it using full train (after data cleaning). I got the idea from this [unassuming kernel](https://www.kaggle.com/mlisovyi/no-ml-benchmark). In Kfold-lightgbm-without-leak-1-08 model, I added a feature that combined site and meter as a categorical with no mean encoding. I noticed that for some sites, the mean encoded `bm_week_hour` feature performed worse while others performed better, but overall, it seemed favorable.\n4. Validating using sites 0,1,2,4,15 - using actual test ground truths for various sites individually and together helped to monitor whether my experiments improved out of sample test data.\n5. Addition by Subtraction - removing certain features for certain models helped improve ground truth (GT) test validation as well as local cross validation (CV). For half-and-half model variant, I dropped `site_id`, `sea_level_pressure`, and `precip_depth_1_hr`. For kfold-lightgbm-without-leak-1-08 I dropped `site_id`, `sea_level_pressure`, `wind_direction`, `wind_speed`, `year_built`, `floor_count`. I removed holiday features for all existing models because they made CV and GT validation worse. For aligned-timestamp-lgbm-by-meter-type, I dropped all the lag3 features.\n\n\nNotes: very little hyper-parameter optimization was performed. Just very limited basic tuning.\n\n### What didn't work\n1. Training using leaked test GT labels - training with GT from 2017-2018 did not improve out of sample `site_id`. For ex: training with `site_id` 0 did not improve validation scores for sites 1,2,4,15 dramatically. I only performed that one test and realized that adding a `site_id` to testing doesn't improve LB scores for out of site validation because each `site_id` is it's own microcosm and behaves different from other sites.\n2. A lot of feature engineering did not work including weather features.\n3. CatBoost/XGBoost/NN Embeddings/Linear Regression/Kernel Ridge Regression(KRR)/KNN all performed worse. I tried building 1449 (1 for each building) and 2380 (1 for each building's meter) Linear Regression, KRR, and LGBM models. I tried blending with Catboost and NN Embeddings models separately, but GT validation didn't seem to improve.\n4. Training with full year of training data and validating using sites 0,1,2,4,15 didn't seem to help much.\n5. Correcting site 0 meter 0 didn't help - [discussion](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/119261)\n\n### Some experiments I tried\n1. Using half-and-half methodology to split train into 2 separate halves - middle months (4-9)/ending months (1-3, 10-12) → performed worse. First half of the day 0-11, later half of the day 12-23. This split by hour in the day performed surprisingly well and GT validation had shown this, however I didn't select it for final submission blend. [notebook](https://www.kaggle.com/teeyee314/best-single-lgbm-lb-1-08-morning-evening)\n2. L2 model - I stacked a model on top of individual model predictions (for the same folds), which only worked for half-and-half model. I left this out of final blend. I also tried ensembling through stacking final models using LGBM but there didn't seem to be any improvement in GT validation.\n\n### What I didn't try\n1. [Divide and Conquer notebook](https://www.kaggle.com/rohanrao/ashrae-divide-and-conquer) - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.\n2. Different blending methods, different weights, etc. - I just kept it simple (introduced no additional complexity or bias)\n3. Making submission with certain models in final blend due to limited submissions. Namely I wanted to submit my blend with NN Embeddings, but based on GT validation, I filtered out a lot of potentially better performing blends. It is possible that sites 0,1,2,4,15 did not represent other sites well enough, but that was the trade off I had to make. \n\nnotes: I mention these, because it may have been the difference for finishing closer in-the-money. \n\n### What I learned\n1. I took my mean target encoding game to a new level with this competition. Before this, I was stuck at basic single categorical feature mean target encoding. Below is the code I used to do multi categorical feature mean encoding:\n> bm\\_cols = ['building\\_id', 'meter', 'weekday', 'hour',]\ndf\\_train['hour'] = df\\_train['timestamp'].dt.hour\ndf\\_train['weekday'] = df\\_train['timestamp'].dt.weekday\nbm = df\\_train.groupby(bm\\_cols)['meter\\_reading'].mean().rename('bm\\_week\\_hour').to\\_frame()\ndf\\_train = df\\_train.merge(bm, right\\_index=True, left\\_on=bm\\_cols, how='left')\n\n### Final submissions used for blending\n[half-and-half variant](https://www.kaggle.com/teeyee314/best-single-lgbm-lb-1-08)\n[kfold-lightgbm-without-leak-1-08 variant](https://www.kaggle.com/teeyee314/kfold-lightgbm)\n[aligned-timestamp-lgbm-by-meter-type variant](https://www.kaggle.com/teeyee314/aligned-timestamp-lgbm-by-meter-type-1-09)\n\n### Final thoughts\nIt took me a year to finally reach a gold model. I was an absolute beginner when I started and I've learned a lot since joining Kaggle. I hope to continue competing and learning from all the bright and inspiring people on here. Although the kernels I blended with are all public, there were a lot of details that led to obtaining a gold model. Most notably, stable validation and logging each experiment meticulously so that after a few weeks or near the end, I wouldn't forget what led to improvements and what did not.", "votes": 25 }, { "id": 717399, "postDate": "2020-01-13T04:35:22.650Z", "content": "

Thank you very much @teeyee314 for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!

\n\n

Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.

", "rawMarkdown": "Thank you very much @teeyee314 for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "votes": 1, "replies": [ { "id": 717928, "postDate": "2020-01-13T19:41:04.167Z", "content": "

Hi @claytonmiller My solutions have been made public under the Final submissions used for blending section. There's also a notebook under Some experiments I tried section that you may want to checkout at as well. Feel free to use my solutions and reference them any way you wish. I will make my blending/post processing notebook public when I have more time - I need to clean it up first.

", "rawMarkdown": "Hi @claytonmiller My solutions have been made public under the `Final submissions used for blending` section. There's also a notebook under `Some experiments I tried` section that you may want to checkout at as well. Feel free to use my solutions and reference them any way you wish. I will make my blending/post processing notebook public when I have more time - I need to clean it up first.", "votes": 1 }, { "id": 718220, "postDate": "2020-01-14T07:29:06.577Z", "content": "

Hi Tim - thanks so much! Just get the others added when you get a chance

", "rawMarkdown": "Hi Tim - thanks so much! Just get the others added when you get a chance" } ] }, { "id": 706253, "postDate": "2019-12-30T06:20:39.160Z", "content": "

Congrats @teeyee314 for your first gold!

\n\n
\n

Divide and Conquer notebook - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.

\n
\n\n

Fun fact: Our best single model is a tuned version of the Divide and Conquer concept :-)

", "rawMarkdown": "Congrats @teeyee314 for your first gold!\n\n>Divide and Conquer notebook - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.\n\nFun fact: Our best single model is a tuned version of the Divide and Conquer concept :-)", "votes": 2, "replies": [ { "id": 706259, "postDate": "2019-12-30T06:35:33.403Z", "content": "

@rohanrao Thanks! It was an oversight I regret :/. I came within striking distance of finishing in the money. Had I not made the oversight, I may well have squeezed a little more out of my scores.

", "rawMarkdown": "@rohanrao Thanks! It was an oversight I regret :/. I came within striking distance of finishing in the money. Had I not made the oversight, I may well have squeezed a little more out of my scores." } ] }, { "id": 712189, "postDate": "2020-01-06T23:26:23.380Z", "content": "

Congrats!

", "rawMarkdown": "Congrats!" }, { "id": 706852, "postDate": "2019-12-30T23:20:17.177Z", "content": "

Thanks for the shout-out, and congratulations. I'm glad my kernel was able to be some help.

", "rawMarkdown": "Thanks for the shout-out, and congratulations. I'm glad my kernel was able to be some help.\n" }, { "id": 706376, "postDate": "2019-12-30T10:02:53.707Z", "content": "

Congrats @teeyee314

", "rawMarkdown": "Congrats @teeyee314 " }, { "id": 705867, "postDate": "2019-12-29T15:46:45.753Z", "content": "

Congrats @teeyee314 and thanks for sharing.

", "rawMarkdown": "Congrats @teeyee314 and thanks for sharing." }, { "id": 705193, "postDate": "2019-12-28T16:25:41.827Z", "content": "

@teeyee314 hi, I want to know your target encoding is as a new feature ? Or replace some features ？

", "rawMarkdown": "@teeyee314 hi, I want to know your target encoding is as a new feature ? Or replace some features ？", "replies": [ { "id": 706249, "postDate": "2019-12-30T06:14:19.173Z", "content": "

New feature, however, always good idea to try all possibilities (sometimes its better to replace, sometimes its better to as new feature). You can play around with the feature in my notebook and watch the validation.

", "rawMarkdown": "New feature, however, always good idea to try all possibilities (sometimes its better to replace, sometimes its better to as new feature). You can play around with the feature in my [notebook](https://www.kaggle.com/teeyee314/best-single-lgbm-lb-1-08) and watch the validation." }, { "id": 706251, "postDate": "2019-12-30T06:16:59.933Z", "content": "

ok,thx

", "rawMarkdown": "ok,thx" } ] }, { "id": 704918, "postDate": "2019-12-28T06:35:25.960Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!" }, { "id": 706311, "postDate": "2019-12-30T08:30:38.827Z", "rawMarkdown": "", "isDeleted": true }, { "id": 705122, "postDate": "2019-12-28T13:19:54.760Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 717399, "author_name": "Clayton Miller", "author_url": "", "post_date": "2020-01-13T04:35:22.650000", "content": "

Thank you very much @teeyee314 for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!

\n\n

Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.

", "votes": 1, "replies": [ { "id": 717928, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-01-13T19:41:04.167000", "content": "

Hi @claytonmiller My solutions have been made public under the Final submissions used for blending section. There's also a notebook under Some experiments I tried section that you may want to checkout at as well. Feel free to use my solutions and reference them any way you wish. I will make my blending/post processing notebook public when I have more time - I need to clean it up first.

", "votes": 1, "replies": [] }, { "id": 718220, "author_name": "Clayton Miller", "author_url": "", "post_date": "2020-01-14T07:29:06.577000", "content": "

Hi Tim - thanks so much! Just get the others added when you get a chance

", "votes": 0, "replies": [] } ] }, { "id": 706253, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-30T06:20:39.160000", "content": "

Congrats @teeyee314 for your first gold!

\n\n
\n

Divide and Conquer notebook - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.

\n
\n\n

Fun fact: Our best single model is a tuned version of the Divide and Conquer concept :-)

", "votes": 2, "replies": [ { "id": 706259, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-12-30T06:35:33.403000", "content": "

@rohanrao Thanks! It was an oversight I regret :/. I came within striking distance of finishing in the money. Had I not made the oversight, I may well have squeezed a little more out of my scores.

", "votes": 0, "replies": [] } ] }, { "id": 712189, "author_name": "corochann", "author_url": "", "post_date": "2020-01-06T23:26:23.380000", "content": "

Congrats!

", "votes": 0, "replies": [] }, { "id": 706852, "author_name": "Robert Stockton", "author_url": "", "post_date": "2019-12-30T23:20:17.177000", "content": "

Thanks for the shout-out, and congratulations. I'm glad my kernel was able to be some help.

", "votes": 0, "replies": [] }, { "id": 706376, "author_name": "Sandeep Kumar", "author_url": "", "post_date": "2019-12-30T10:02:53.707000", "content": "

Congrats @teeyee314

", "votes": 0, "replies": [] }, { "id": 705867, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-12-29T15:46:45.753000", "content": "

Congrats @teeyee314 and thanks for sharing.

", "votes": 0, "replies": [] }, { "id": 705193, "author_name": "LJH", "author_url": "", "post_date": "2019-12-28T16:25:41.827000", "content": "

@teeyee314 hi, I want to know your target encoding is as a new feature ? Or replace some features ？

", "votes": 0, "replies": [ { "id": 706249, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-12-30T06:14:19.173000", "content": "

New feature, however, always good idea to try all possibilities (sometimes its better to replace, sometimes its better to as new feature). You can play around with the feature in my notebook and watch the validation.

", "votes": 0, "replies": [] }, { "id": 706251, "author_name": "LJH", "author_url": "", "post_date": "2019-12-30T06:16:59.933000", "content": "

ok,thx

", "votes": 0, "replies": [] } ] }, { "id": 704918, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2019-12-28T06:35:25.960000", "content": "

Congratulations!

", "votes": 0, "replies": [] }, { "id": 706311, "author_name": "", "author_url": "", "post_date": "2019-12-30T08:30:38.827000", "content": "", "votes": 0, "replies": [] }, { "id": 705122, "author_name": "", "author_url": "", "post_date": "2019-12-28T13:19:54.760000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "704911": "Thanks to ASHRAE for providing this unclean dataset! Thanks to Kaggle for continuing and keeping the competition alive. Congrats to everyone who managed to survive the LB shakeup!\n\nThanks to @rohanrao, @kailex, @nz0722, @aitude, @purist1024 for your excellent notebooks which had direct impact on helping me achieve this outcome.\n\n## Summary\n\nFinal 2 submissions: equal weighted blend of the following plus some regularization.\n1. https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks\n2. https://www.kaggle.com/rohanrao/ashrae-half-and-half (credit for original work https://www.kaggle.com/kailex/ac-dc)\n3. https://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\n4. https://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\n\npublic LB: 0.935 (1.032 w/o leak estimated) with aggressive regularization 0.80 and 0.91\npublic LB: 0.944 (1.039 w/o leak estimated) with conservative regularization 0.91 for all\npublic LB: 0.950 (1.045 w/o leak) no tricks - I did not select for submission\n\n### What worked\n1. Data Cleaning - garbage in, garbage out. This is probably the single most important aspect of the the competition. I did this manually by plotting [heatmaps](https://www.kaggle.com/ganfear/missing-data-and-zeros-visualized) and also going into each building's meter to inspect the target meter readings if they looked reasonable. I also reverse engineered the heat map to show only zeros by added the following line:
`train_df = train_df.query('not (meter_reading != 0)')`\n2. Regularization - (this is what I'm calling it, maybe some will call it postprocessing, coefficients, tricks, etc.) - multiplying by some value < 1.0. For aggressive regularization, I used two different values 0.80 for responsive meters and 0.91 for less responsive meters. I probed each site's meter individually by multiplying by 0.95 to start then as I went through all the meters, I noted which meter or meters dropped the public LB by 0.001, which I noted to be responsive to regularization). I got the idea from [LANL Earthquake prediction](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/94324). Fortunately, I was confident that the LB would not shake me down if I tried to overfit the LB this way. I ran some tests on the leaked sites 0,1,2,4,15 to test each site's meters effect to various values. 0.90-0.95 seemed fairly safe values to start out with when I began probing. I gained about 0.015 on LB through a series of 29 submissions, which took about 2 weeks. As my remaining submissions diminished, I ramped up the overall aggressiveness of regularization and began probing the responsive site's meters as a collective submission because there wasn't enough submissions to try all possible values. This gave between 0.004 - 0.005 reduction on Private LB score. Without any regularization, I would have ended up placing 35 on private LB, so this trick definitely gave my scores the extra kick to finish in gold.\n3. Feature Engineering - In the half-and-half model, I added a feature that grouped `building_id`, `meter`, `weekday`, `hour` and mean target encoded it using full train (after data cleaning). I got the idea from this [unassuming kernel](https://www.kaggle.com/mlisovyi/no-ml-benchmark). In Kfold-lightgbm-without-leak-1-08 model, I added a feature that combined site and meter as a categorical with no mean encoding. I noticed that for some sites, the mean encoded `bm_week_hour` feature performed worse while others performed better, but overall, it seemed favorable.\n4. Validating using sites 0,1,2,4,15 - using actual test ground truths for various sites individually and together helped to monitor whether my experiments improved out of sample test data.\n5. Addition by Subtraction - removing certain features for certain models helped improve ground truth (GT) test validation as well as local cross validation (CV). For half-and-half model variant, I dropped `site_id`, `sea_level_pressure`, and `precip_depth_1_hr`. For kfold-lightgbm-without-leak-1-08 I dropped `site_id`, `sea_level_pressure`, `wind_direction`, `wind_speed`, `year_built`, `floor_count`. I removed holiday features for all existing models because they made CV and GT validation worse. For aligned-timestamp-lgbm-by-meter-type, I dropped all the lag3 features.\n\n\nNotes: very little hyper-parameter optimization was performed. Just very limited basic tuning.\n\n### What didn't work\n1. Training using leaked test GT labels - training with GT from 2017-2018 did not improve out of sample `site_id`. For ex: training with `site_id` 0 did not improve validation scores for sites 1,2,4,15 dramatically. I only performed that one test and realized that adding a `site_id` to testing doesn't improve LB scores for out of site validation because each `site_id` is it's own microcosm and behaves different from other sites.\n2. A lot of feature engineering did not work including weather features.\n3. CatBoost/XGBoost/NN Embeddings/Linear Regression/Kernel Ridge Regression(KRR)/KNN all performed worse. I tried building 1449 (1 for each building) and 2380 (1 for each building's meter) Linear Regression, KRR, and LGBM models. I tried blending with Catboost and NN Embeddings models separately, but GT validation didn't seem to improve.\n4. Training with full year of training data and validating using sites 0,1,2,4,15 didn't seem to help much.\n5. Correcting site 0 meter 0 didn't help - [discussion](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/119261)\n\n### Some experiments I tried\n1. Using half-and-half methodology to split train into 2 separate halves - middle months (4-9)/ending months (1-3, 10-12) → performed worse. First half of the day 0-11, later half of the day 12-23. This split by hour in the day performed surprisingly well and GT validation had shown this, however I didn't select it for final submission blend. [notebook](https://www.kaggle.com/teeyee314/best-single-lgbm-lb-1-08-morning-evening)\n2. L2 model - I stacked a model on top of individual model predictions (for the same folds), which only worked for half-and-half model. I left this out of final blend. I also tried ensembling through stacking final models using LGBM but there didn't seem to be any improvement in GT validation.\n\n### What I didn't try\n1. [Divide and Conquer notebook](https://www.kaggle.com/rohanrao/ashrae-divide-and-conquer) - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.\n2. Different blending methods, different weights, etc. - I just kept it simple (introduced no additional complexity or bias)\n3. Making submission with certain models in final blend due to limited submissions. Namely I wanted to submit my blend with NN Embeddings, but based on GT validation, I filtered out a lot of potentially better performing blends. It is possible that sites 0,1,2,4,15 did not represent other sites well enough, but that was the trade off I had to make. \n\nnotes: I mention these, because it may have been the difference for finishing closer in-the-money. \n\n### What I learned\n1. I took my mean target encoding game to a new level with this competition. Before this, I was stuck at basic single categorical feature mean target encoding. Below is the code I used to do multi categorical feature mean encoding:\n> bm\\_cols = ['building\\_id', 'meter', 'weekday', 'hour',]\ndf\\_train['hour'] = df\\_train['timestamp'].dt.hour\ndf\\_train['weekday'] = df\\_train['timestamp'].dt.weekday\nbm = df\\_train.groupby(bm\\_cols)['meter\\_reading'].mean().rename('bm\\_week\\_hour').to\\_frame()\ndf\\_train = df\\_train.merge(bm, right\\_index=True, left\\_on=bm\\_cols, how='left')\n\n### Final submissions used for blending\n[half-and-half variant](https://www.kaggle.com/teeyee314/best-single-lgbm-lb-1-08)\n[kfold-lightgbm-without-leak-1-08 variant](https://www.kaggle.com/teeyee314/kfold-lightgbm)\n[aligned-timestamp-lgbm-by-meter-type variant](https://www.kaggle.com/teeyee314/aligned-timestamp-lgbm-by-meter-type-1-09)\n\n### Final thoughts\nIt took me a year to finally reach a gold model. I was an absolute beginner when I started and I've learned a lot since joining Kaggle. I hope to continue competing and learning from all the bright and inspiring people on here. Although the kernels I blended with are all public, there were a lot of details that led to obtaining a gold model. Most notably, stable validation and logging each experiment meticulously so that after a few weeks or near the end, I wouldn't forget what led to improvements and what did not.", "717399": "Thank you very much @teeyee314 for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "706253": "Congrats @teeyee314 for your first gold!\n\n>Divide and Conquer notebook - I didn't bother with this notebook at all so I can't tell whether it was any good. Looking back now, I should have at least tried playing around with it. It is possible it could have helped reduce variance like @rohanrao mentioned.\n\nFun fact: Our best single model is a tuned version of the Divide and Conquer concept :-)", "712189": "Congrats!", "706852": "Thanks for the shout-out, and congratulations. I'm glad my kernel was able to be some help.\n", "706376": "Congrats @teeyee314 ", "705867": "Congrats @teeyee314 and thanks for sharing.", "705193": "@teeyee314 hi, I want to know your target encoding is as a new feature ? Or replace some features ？", "704918": "Congratulations!", "706311": "", "705122": "" } }