![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F331718%2Fc9acb654b6bdae701c9a9aa54c1bdd33%2FScreenshot%20from%202020-01-07%2013-15-21.png?generation=1578431784407091&alt=media\")
Thank you to Kaggle and ASHRAE for hosting this competition. The decision to use only non-leak data for the private test set helped to make this competition fair. Thank you to all those who contributed to the kernels and discussions and especially those who made the leaks public. Last but not least, I'd like to thank and congratulate my teammate Isamu Yamashita for being a great teammate and becoming a Competitions Master.
\n\nDuring the competition, we shared ideas and discussed progress within our team, but we tested and trained separate models. This helped us maintain diversity in our final ensemble. This is a combined summary our team's solution.
\n\n
As others have noted, cleaning the data was very important in this competition. The assumption is that there are unpredictable and hence unlearnable anomalies in the data that, if trained on, degrade the quality of the predictions. We identified and filtered out three types of anomalies:\n1. Long streaks of constant values\n1. Large positive/negative spikes\n1. Additional anomalies determined by visual inspection
\n\nWe noticed that some of these anomalies were consistent across multiple buildings at a site. We validated potential anomalies using all buildings in a site--if an anomaly showed up at the same time at multiple buildings, we could be reasonably certain that this was indeed a true anomaly. This allowed us to remove anomalies that were not necessarily part of a long streak of constant values or a large spike.
\n\nThere were a lot of missing values in temperature metadata. We found that imputing the missing data using linear interpolation helped our models.
\n\nAs noted in the competition forum, the timezone in the train/test data was different from the timezone in the weather metadata. We used the information in this discussion post to correct the timezones.
\n\nLike most competitors, we started by predicting log1p(meter_reading). We also corrected the units for site 0 as per this discussion post.
Near the end of the competition, we tried standardizing meter_reading by dividing by square_feet; i.e., we predicted log1p(meter_reading/square_feet). Isamu came up with the idea after reading this discussion post by Artyom Vorobyov. The models trained with the standardized target added diversity to our final ensemble and improved our score by about 0.002. If we had more time we would have liked to explore this idea further; for example, we could have tried to predict log1p(meter_reading)/square_feet or created features using the standardized targets.
We took different approaches to feature engineering and feature selection in this competition. Isamu took a conservative approach and carefully selected features; on the other hand, Matt took a brute force approach and used most of them. Here are the features that helped:
\n\nbuilding_id and meter![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F331718%2Fd5ec815eedab54ae3262680957de2fbd%2Fsmoothed_temperature.png?generation=1578280466482319&alt=media\")
hour gets mapped to hour_x = cos(2*pi*hour/24) and hour_y = sin(2*pi*hour/24)We trained CatBoost, LightGBM, and MLP models on different subsets of the data:\n* 1 model per meter\n* 1 model per site_id\n* 1 model per (building_id, meter)
\n\nOur team tried different approaches to validation in this competition. Like other competitors, we tried K-Fold CV using consecutive months as the validation set. The following code shows one approach to getting validation months: \n\n>>> import numpy as np\n>>> def get_validation_months(n):\n... return [np.arange(i, i+n) % 12 + 1 for i in range(12)]\n...\n>>> get_validation_months(6)\n[array([1, 2, 3, 4, 5, 6]),\n array([2, 3, 4, 5, 6, 7]),\n array([3, 4, 5, 6, 7, 8]),\n array([4, 5, 6, 7, 8, 9]),\n array([ 5, 6, 7, 8, 9, 10]),\n array([ 6, 7, 8, 9, 10, 11]),\n array([ 7, 8, 9, 10, 11, 12]),\n array([ 8, 9, 10, 11, 12, 1]),\n array([ 9, 10, 11, 12, 1, 2]),\n array([10, 11, 12, 1, 2, 3]),\n array([11, 12, 1, 2, 3, 4]),\n array([12, 1, 2, 3, 4, 5])]\n\nTrying different validation schemes allowed us to train models that added diversity to our final ensemble.
To reduce the risk of overfitting to the public LB and improve robustness, we ensembled predictions from many different models. Here are some of the things we did:\n* Used cleaned leak data as a holdout set to tune our second stage model\n* Averaged log values; i.e., expm1(mean(log1p(x))) rather than averaged the raw values\n* Used the generalized weighted mean and tuned the parameters using Optuna\n* Hedged our bets by including leak-free public kernels on cleaned data:\n * ASHRAE: Half and Half by Vopani\n * ASHRAE- KFold LightGBM - without leak (1.08) by Sandeep Kumar\n * Ashrae: simple data cleanup (LB 1.08 no leaks) by Robert Stockton
This kernel shows how we ensembled our predictions. Our final ensemble was a plain average of our top 4 submissions with respect to the public LB score.
\n\nAgain, following this discussion post by Artyom Vorobyov, we thought of ensembling the predictions for buildings with the same meter_readings; e.g., buildings 1225 and 1226 with meter 0. However, we were not able to improve our public LB result with this approach. It is possible that we did something wrong here, but we didn't have enough time to go back and explore this idea further.
Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth.
Thank you to Kaggle and ASHRAE for hosting this competition. The decision to use only non-leak data for the private test set helped to make this competition fair. Thank you to all those who contributed to the kernels and discussions and especially those who made the leaks public. Last but not least, I'd like to thank and congratulate my teammate Isamu Yamashita for being a great teammate and becoming a Competitions Master.
\n\nDuring the competition, we shared ideas and discussed progress within our team, but we tested and trained separate models. This helped us maintain diversity in our final ensemble. This is a combined summary our team's solution.
\n\n
As others have noted, cleaning the data was very important in this competition. The assumption is that there are unpredictable and hence unlearnable anomalies in the data that, if trained on, degrade the quality of the predictions. We identified and filtered out three types of anomalies:\n1. Long streaks of constant values\n1. Large positive/negative spikes\n1. Additional anomalies determined by visual inspection
\n\nWe noticed that some of these anomalies were consistent across multiple buildings at a site. We validated potential anomalies using all buildings in a site--if an anomaly showed up at the same time at multiple buildings, we could be reasonably certain that this was indeed a true anomaly. This allowed us to remove anomalies that were not necessarily part of a long streak of constant values or a large spike.
\n\nThere were a lot of missing values in temperature metadata. We found that imputing the missing data using linear interpolation helped our models.
\n\nAs noted in the competition forum, the timezone in the train/test data was different from the timezone in the weather metadata. We used the information in this discussion post to correct the timezones.
\n\nLike most competitors, we started by predicting log1p(meter_reading). We also corrected the units for site 0 as per this discussion post.
Near the end of the competition, we tried standardizing meter_reading by dividing by square_feet; i.e., we predicted log1p(meter_reading/square_feet). Isamu came up with the idea after reading this discussion post by Artyom Vorobyov. The models trained with the standardized target added diversity to our final ensemble and improved our score by about 0.002. If we had more time we would have liked to explore this idea further; for example, we could have tried to predict log1p(meter_reading)/square_feet or created features using the standardized targets.
We took different approaches to feature engineering and feature selection in this competition. Isamu took a conservative approach and carefully selected features; on the other hand, Matt took a brute force approach and used most of them. Here are the features that helped:
\n\nbuilding_id and meter
hour gets mapped to hour_x = cos(2*pi*hour/24) and hour_y = sin(2*pi*hour/24)We trained CatBoost, LightGBM, and MLP models on different subsets of the data:\n* 1 model per meter\n* 1 model per site_id\n* 1 model per (building_id, meter)
\n\nOur team tried different approaches to validation in this competition. Like other competitors, we tried K-Fold CV using consecutive months as the validation set. The following code shows one approach to getting validation months: \n\n>>> import numpy as np\n>>> def get_validation_months(n):\n... return [np.arange(i, i+n) % 12 + 1 for i in range(12)]\n...\n>>> get_validation_months(6)\n[array([1, 2, 3, 4, 5, 6]),\n array([2, 3, 4, 5, 6, 7]),\n array([3, 4, 5, 6, 7, 8]),\n array([4, 5, 6, 7, 8, 9]),\n array([ 5, 6, 7, 8, 9, 10]),\n array([ 6, 7, 8, 9, 10, 11]),\n array([ 7, 8, 9, 10, 11, 12]),\n array([ 8, 9, 10, 11, 12, 1]),\n array([ 9, 10, 11, 12, 1, 2]),\n array([10, 11, 12, 1, 2, 3]),\n array([11, 12, 1, 2, 3, 4]),\n array([12, 1, 2, 3, 4, 5])]\n\nTrying different validation schemes allowed us to train models that added diversity to our final ensemble.
To reduce the risk of overfitting to the public LB and improve robustness, we ensembled predictions from many different models. Here are some of the things we did:\n* Used cleaned leak data as a holdout set to tune our second stage model\n* Averaged log values; i.e., expm1(mean(log1p(x))) rather than averaged the raw values\n* Used the generalized weighted mean and tuned the parameters using Optuna\n* Hedged our bets by including leak-free public kernels on cleaned data:\n * ASHRAE: Half and Half by Vopani\n * ASHRAE- KFold LightGBM - without leak (1.08) by Sandeep Kumar\n * Ashrae: simple data cleanup (LB 1.08 no leaks) by Robert Stockton
This kernel shows how we ensembled our predictions. Our final ensemble was a plain average of our top 4 submissions with respect to the public LB score.
\n\nAgain, following this discussion post by Artyom Vorobyov, we thought of ensembling the predictions for buildings with the same meter_readings; e.g., buildings 1225 and 1226 with meter 0. However, we were not able to improve our public LB result with this approach. It is possible that we did something wrong here, but we didn't have enough time to go back and explore this idea further.
Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth.
Congratulations to our first-place team!!!!!! 🙌 💪 👍 👊 Their solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "rawMarkdown": "Congratulations to our first-place team!!!!!! 🙌 💪 👍 👊 Their solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned! \n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "votes": 3 }, { "id": 717738, "postDate": "2020-01-13T14:49:21.277Z", "content": "Hi and congrats
\n\nWhen you say in the feature engineering section, \n> - Raw features from train/test, weather metadata, and building metadata\n- Categorical interactions such as the concatenation of building_id and meter\n- Count (frequency) features\n- ...
\n\nLets say X1, X2 two categorical features and X1_X2 the interaction between X1 and X2
\n\nDoes this mean that in your final data set you had the following features at the same time:\nX1, X1_count, X2, X2_count, X1_X2, X1_X2_count? or only X1_count, X2_count, X1_X2_count ?
Congratulations, awesome work!
\n\n\n\n\nCount (frequency) features
\n
What kind of features are these?
", "rawMarkdown": "Congratulations, awesome work! \n\n> Count (frequency) features\n\nWhat kind of features are these?\n", "votes": 1, "replies": [ { "id": 716753, "postDate": "2020-01-12T07:56:18.840Z", "content": "Thank mavillan! The count features are the result of mapping categorical features to their counts (frequencies); e.g., we could create an hourly_count feature like this \n\ndf[\"hourly_count\"] = df.groupby(\"hour\")[\"hour\"].transform(\"count\")\n
Congrats! Nice write up!
\n\n> Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth.
\n\nby \"smoothing\", you mean postprocessing right?
", "rawMarkdown": "Congrats! Nice write up!\n\n> Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth.\n\nby \"smoothing\", you mean [postprocessing](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123528) right?\n", "votes": 1, "replies": [ { "id": 711911, "postDate": "2020-01-06T16:35:28.560Z", "content": "Hi Tim, if I understand the 2nd place team's postprocessing correctly, it is not the same as what we tried. By smoothing I mean something like a moving average of the predictions.
", "rawMarkdown": "Hi Tim, if I understand the 2nd place team's postprocessing correctly, it is not the same as what we tried. By smoothing I mean something like a moving average of the predictions. ", "votes": 1 } ] }, { "id": 711859, "postDate": "2020-01-06T15:31:43.740Z", "content": "Great job. Your feature Engineering and Blending techniques are awesome!\nWhat was your MLP architecture?
", "rawMarkdown": "Great job. Your feature Engineering and Blending techniques are awesome!\nWhat was your MLP architecture?", "votes": 2, "replies": [ { "id": 712749, "postDate": "2020-01-07T14:55:40.527Z", "content": "Thanks! The MLP architecture was pretty standard: entity emeddings + numerical features, followed by a few dense layers. I used Keras and the code was similar to these kernels\n* Keras NN with embeddings for cat. features (1.15)\n* same old entity embeddings
", "rawMarkdown": "Thanks! The MLP architecture was pretty standard: entity emeddings + numerical features, followed by a few dense layers. I used Keras and the code was similar to these kernels\n* [Keras NN with embeddings for cat. features (1.15)](https://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15)\n* [same old entity embeddings](https://www.kaggle.com/abhishek/same-old-entity-embeddings)", "votes": 1 } ] }, { "id": 711414, "postDate": "2020-01-06T03:54:04.723Z", "content": "Congratulations
\n\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @mmotoki
", "rawMarkdown": "Congratulations\n\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @mmotoki " }, { "id": 1685910, "postDate": "2022-02-11T16:01:17.200Z", "content": "Very interesting!
", "rawMarkdown": "Very interesting!" }, { "id": 802036, "postDate": "2020-04-09T04:39:20.180Z", "content": "Congratulations! \nBy the way, How much did \"1st, 2nd-order differentiation temperature features using Savitzky-Golay filter\" make accuracy better?
", "rawMarkdown": "Congratulations! \nBy the way, How much did \"1st, 2nd-order differentiation temperature features using Savitzky-Golay filter\" make accuracy better?" }, { "id": 749861, "postDate": "2020-02-19T00:12:48.997Z", "content": "Congratulations and thank you for sharing !\nI will try your approach and use Outlier Detection from another group of gold medalist.
", "rawMarkdown": "Congratulations and thank you for sharing !\nI will try your approach and use Outlier Detection from another group of gold medalist." }, { "id": 735255, "postDate": "2020-02-02T19:26:34.833Z", "content": "Congrats
", "rawMarkdown": "Congrats" }, { "id": 729838, "postDate": "2020-01-26T17:46:43.140Z", "content": "Thank you for sharing this post with the community 👍
", "rawMarkdown": "Thank you for sharing this post with the community 👍 " }, { "id": 726732, "postDate": "2020-01-23T07:46:01.237Z", "content": "congratulation!
", "rawMarkdown": "congratulation!" }, { "id": 723314, "postDate": "2020-01-19T20:45:36.253Z", "content": "Congrats, nice work!
", "rawMarkdown": "Congrats, nice work!" }, { "id": 721156, "postDate": "2020-01-17T05:31:14.497Z", "content": "Congratulations @mmotoki and @yamsam and thanks for sharing your approach.
", "rawMarkdown": "Congratulations @mmotoki and @yamsam and thanks for sharing your approach." }, { "id": 717598, "postDate": "2020-01-13T10:57:24Z", "content": "Congratulations! Thanks for the nice write up!
", "rawMarkdown": "Congratulations! Thanks for the nice write up!" }, { "id": 717185, "postDate": "2020-01-12T20:59:46.310Z", "content": "Nice! Congrats
", "rawMarkdown": "Nice! Congrats" }, { "id": 717066, "postDate": "2020-01-12T17:21:56.680Z", "content": "congratulations and thanks for sharing.💯
", "rawMarkdown": "congratulations and thanks for sharing.💯 " }, { "id": 716110, "postDate": "2020-01-11T09:10:26.297Z", "content": "Congrats. Awesome work!! Thank you for sharing.
", "rawMarkdown": "Congrats. Awesome work!! Thank you for sharing. " }, { "id": 715591, "postDate": "2020-01-10T16:40:32.663Z", "content": "Thanks for sharing solutions,Great work really deserving @mmotoki !!!!!!!!!!!!
", "rawMarkdown": "Thanks for sharing solutions,Great work really deserving @mmotoki !!!!!!!!!!!!" }, { "id": 715449, "postDate": "2020-01-10T14:49:26.063Z", "content": "Thank you very much. Not clear what are final timezones shifts (?) for sites, and how they helped to improve the score (I thought weather data had been already given with local time).\nAlso not clear what to do with some missed values for pressure, precip_depth, wind_direction - need to take smoothed average too?\nAlso day-of-time categorization/cyclic model is worse that raw 24-hour model (separate model for every hour, because of non-smoothed behaviour for some buildings). Not sure this is applicable for your ensemble.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4152487%2F7ebe6137198d85ce687530d683bd84f8%2F2020-01-10.energy0.hourly.png?generation=1578667751585545&alt=media\")
Additionally normalization all features to 0-1 (or 0-1000) will help in precision. But not investigate, how exactly.
", "rawMarkdown": "Thank you very much. Not clear what are final timezones shifts (?) for sites, and how they helped to improve the score (I thought weather data had been already given with local time).\nAlso not clear what to do with some missed values for pressure, precip_depth, wind_direction - need to take smoothed average too?\nAlso day-of-time categorization/cyclic model is worse that raw 24-hour model (separate model for every hour, because of non-smoothed behaviour for some buildings). Not sure this is applicable for your ensemble.\n\n\nAdditionally normalization all features to 0-1 (or 0-1000) will help in precision. But not investigate, how exactly.", "replies": [ { "id": 716749, "postDate": "2020-01-12T07:45:34.470Z", "content": "Thanks Nikolay, the timezone shifts are originally from this discussion post and kernel. To see how to fill in the missing values, have a look at my teammate Isamu's starter kernel. About your last point, you might be right--we didn't try building one model for each hour.
", "rawMarkdown": "Thanks Nikolay, the timezone shifts are originally from this [discussion post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114483#latest-659257) and [kernel](https://www.kaggle.com/frednavruzov/aligning-temperature-timestamp). To see how to fill in the missing values, have a look at my teammate Isamu's [starter kernel](https://www.kaggle.com/yamsam/new-ucf-starter-kernel#Fill-Nan-value-in-weather-dataframe-by-interpolation). About your last point, you might be right--we didn't try building one model for each hour." } ] }, { "id": 715423, "postDate": "2020-01-10T14:14:07.157Z", "content": "Congratulations! And thanks for sharing.
\n\nBy the way, this cyclic hour encoding of yours, this is some usual trick in ML prediction? I think, this the first time I see it and I love it!
", "rawMarkdown": "Congratulations! And thanks for sharing.\n\nBy the way, this cyclic hour encoding of yours, this is some usual trick in ML prediction? I think, this the first time I see it and I love it!", "replies": [ { "id": 716680, "postDate": "2020-01-12T05:05:47.570Z", "content": "Thanks Tomas! I think cyclic encoding is a relatively well known technique (blogs are easy to find if you google \"cyclic feature encoding\"), but I don't see it being used on Kaggle or in practice too frequently. From my experience, these features lead to only small improvements (if any) and so maybe that's why they aren't more widely used.
", "rawMarkdown": "Thanks Tomas! I think cyclic encoding is a relatively well known technique (blogs are easy to find if you google \"cyclic feature encoding\"), but I don't see it being used on Kaggle or in practice too frequently. From my experience, these features lead to only small improvements (if any) and so maybe that's why they aren't more widely used.", "votes": 2 }, { "id": 716958, "postDate": "2020-01-12T14:10:41.897Z", "content": "Thank you, Matt! I will look at it.
", "rawMarkdown": "Thank you, Matt! I will look at it. " } ] }, { "id": 715078, "postDate": "2020-01-10T05:27:08.637Z", "content": "Thank you sharing the great work you did, this will help to understand lots of things.
", "rawMarkdown": "Thank you sharing the great work you did, this will help to understand lots of things." }, { "id": 714865, "postDate": "2020-01-09T21:31:09.687Z", "content": "Congratulations, great job!!
", "rawMarkdown": "Congratulations, great job!!" }, { "id": 714363, "postDate": "2020-01-09T10:51:10.430Z", "content": "So great!!! Congratulations, hope one day I could do half of this work!!
", "rawMarkdown": "So great!!! Congratulations, hope one day I could do half of this work!!" }, { "id": 713849, "postDate": "2020-01-08T18:00:22.197Z", "content": "Congrats @mmotoki
", "rawMarkdown": "Congrats @mmotoki " }, { "id": 713156, "postDate": "2020-01-08T01:12:12.073Z", "content": "Congrats @mmotoki and @teeyee314. Thanks for sharing.
", "rawMarkdown": "Congrats @mmotoki and @teeyee314. Thanks for sharing." }, { "id": 712169, "postDate": "2020-01-06T22:36:54.653Z", "content": "Thank you for sharing detailed write up. Congratulations!
", "rawMarkdown": "Thank you for sharing detailed write up. Congratulations!" }, { "id": 711691, "postDate": "2020-01-06T12:19:27.553Z", "content": "Hi Matt! Thanks for sharing this writeup! Congrats again!
", "rawMarkdown": "Hi Matt! Thanks for sharing this writeup! Congrats again!" }, { "id": 711682, "postDate": "2020-01-06T11:50:12.387Z", "content": "Great job, as usual.
", "rawMarkdown": "Great job, as usual." }, { "id": 711480, "postDate": "2020-01-06T07:04:19.443Z", "content": "thank you for the solution, congratulations!
", "rawMarkdown": "thank you for the solution, congratulations!" }, { "id": 712004, "postDate": "2020-01-06T18:38:57.420Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 712618, "postDate": "2020-01-07T12:59:15.063Z", "content": "Thanks for sharing, great work!
", "rawMarkdown": "Thanks for sharing, great work!", "votes": 1 }, { "id": 1051981, "postDate": "2020-10-17T06:25:39.770Z", "content": "Thank you for sharing, great work!
", "rawMarkdown": "Thank you for sharing, great work!" }, { "id": 720733, "postDate": "2020-01-16T17:22:21.497Z", "content": "Thank you for sharing
", "rawMarkdown": "Thank you for sharing" }, { "id": 717120, "postDate": "2020-01-12T19:01:36.150Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 716915, "postDate": "2020-01-12T13:07:31.283Z", "content": "Congratulations! Thanks for sharing.
", "rawMarkdown": "Congratulations! Thanks for sharing." }, { "id": 713976, "postDate": "2020-01-08T21:42:37.420Z", "content": "Congrats! thanks for the writeup.
", "rawMarkdown": "Congrats! thanks for the writeup." }, { "id": 713153, "postDate": "2020-01-08T01:07:20.967Z", "content": "Thanks
", "rawMarkdown": "Thanks" }, { "id": 712580, "postDate": "2020-01-07T11:58:57.830Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 712505, "postDate": "2020-01-07T10:06:32.913Z", "content": "Great job, thank's for sharing ! 👍
", "rawMarkdown": "Great job, thank's for sharing ! 👍" }, { "id": 711410, "postDate": "2020-01-06T03:38:14.087Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 717351, "author_name": "Clayton Miller", "author_url": "", "post_date": "2020-01-13T03:45:51.343000", "content": "Congratulations to our first-place team!!!!!! 🙌 💪 👍 👊 Their solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "votes": 3, "replies": [] }, { "id": 717738, "author_name": "armbodj", "author_url": "", "post_date": "2020-01-13T14:49:21.277000", "content": "Hi and congrats
\n\nWhen you say in the feature engineering section, \n> - Raw features from train/test, weather metadata, and building metadata\n- Categorical interactions such as the concatenation of building_id and meter\n- Count (frequency) features\n- ...
\n\nLets say X1, X2 two categorical features and X1_X2 the interaction between X1 and X2
\n\nDoes this mean that in your final data set you had the following features at the same time:\nX1, X1_count, X2, X2_count, X1_X2, X1_X2_count? or only X1_count, X2_count, X1_X2_count ?
Congratulations, awesome work!
\n\n\n\n\nCount (frequency) features
\n
What kind of features are these?
", "votes": 1, "replies": [ { "id": 716753, "author_name": "Matt Motoki", "author_url": "", "post_date": "2020-01-12T07:56:18.840000", "content": "Thank mavillan! The count features are the result of mapping categorical features to their counts (frequencies); e.g., we could create an hourly_count feature like this \n\ndf[\"hourly_count\"] = df.groupby(\"hour\")[\"hour\"].transform(\"count\")\n
Congrats! Nice write up!
\n\n> Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth.
\n\nby \"smoothing\", you mean postprocessing right?
", "votes": 1, "replies": [ { "id": 711911, "author_name": "Matt Motoki", "author_url": "", "post_date": "2020-01-06T16:35:28.560000", "content": "Hi Tim, if I understand the 2nd place team's postprocessing correctly, it is not the same as what we tried. By smoothing I mean something like a moving average of the predictions.
", "votes": 1, "replies": [] } ] }, { "id": 711859, "author_name": "Volodymyr", "author_url": "", "post_date": "2020-01-06T15:31:43.740000", "content": "Great job. Your feature Engineering and Blending techniques are awesome!\nWhat was your MLP architecture?
", "votes": 2, "replies": [ { "id": 712749, "author_name": "Matt Motoki", "author_url": "", "post_date": "2020-01-07T14:55:40.527000", "content": "Thanks! The MLP architecture was pretty standard: entity emeddings + numerical features, followed by a few dense layers. I used Keras and the code was similar to these kernels\n* Keras NN with embeddings for cat. features (1.15)\n* same old entity embeddings
", "votes": 1, "replies": [] } ] }, { "id": 711414, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-06T03:54:04.723000", "content": "Congratulations
\n\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @mmotoki
", "votes": 0, "replies": [] }, { "id": 1685910, "author_name": "Dasha Tereshkina", "author_url": "", "post_date": "2022-02-11T16:01:17.200000", "content": "Very interesting!
", "votes": 0, "replies": [] }, { "id": 802036, "author_name": "Takumi Ban", "author_url": "", "post_date": "2020-04-09T04:39:20.180000", "content": "Congratulations! \nBy the way, How much did \"1st, 2nd-order differentiation temperature features using Savitzky-Golay filter\" make accuracy better?
", "votes": 0, "replies": [] }, { "id": 749861, "author_name": "Xuanlong Yu", "author_url": "", "post_date": "2020-02-19T00:12:48.997000", "content": "Congratulations and thank you for sharing !\nI will try your approach and use Outlier Detection from another group of gold medalist.
", "votes": 0, "replies": [] }, { "id": 735255, "author_name": "John Nesbit", "author_url": "", "post_date": "2020-02-02T19:26:34.833000", "content": "Congrats
", "votes": 0, "replies": [] }, { "id": 729838, "author_name": "Ivan Vigorito", "author_url": "", "post_date": "2020-01-26T17:46:43.140000", "content": "Thank you for sharing this post with the community 👍
", "votes": 0, "replies": [] }, { "id": 726732, "author_name": "Pradeep", "author_url": "", "post_date": "2020-01-23T07:46:01.237000", "content": "congratulation!
", "votes": 0, "replies": [] }, { "id": 723314, "author_name": "Andy Crowe", "author_url": "", "post_date": "2020-01-19T20:45:36.253000", "content": "Congrats, nice work!
", "votes": 0, "replies": [] }, { "id": 721156, "author_name": "Deep Chatterjee", "author_url": "", "post_date": "2020-01-17T05:31:14.497000", "content": "Congratulations @mmotoki and @yamsam and thanks for sharing your approach.
", "votes": 0, "replies": [] }, { "id": 717598, "author_name": "Mohamad Hallak", "author_url": "", "post_date": "2020-01-13T10:57:24", "content": "Congratulations! Thanks for the nice write up!
", "votes": 0, "replies": [] }, { "id": 717185, "author_name": "Eren Ali Aslangiray", "author_url": "", "post_date": "2020-01-12T20:59:46.310000", "content": "Nice! Congrats
", "votes": 0, "replies": [] }, { "id": 717066, "author_name": "Shahules", "author_url": "", "post_date": "2020-01-12T17:21:56.680000", "content": "congratulations and thanks for sharing.💯
", "votes": 0, "replies": [] }, { "id": 716110, "author_name": "", "author_url": "", "post_date": "2020-01-11T09:10:26.297000", "content": "Congrats. Awesome work!! Thank you for sharing.
", "votes": 0, "replies": [] }, { "id": 715591, "author_name": "ask9", "author_url": "", "post_date": "2020-01-10T16:40:32.663000", "content": "Thanks for sharing solutions,Great work really deserving @mmotoki !!!!!!!!!!!!
", "votes": 0, "replies": [] }, { "id": 715449, "author_name": "Nikolay Matsievsky", "author_url": "", "post_date": "2020-01-10T14:49:26.063000", "content": "Thank you very much. Not clear what are final timezones shifts (?) for sites, and how they helped to improve the score (I thought weather data had been already given with local time).\nAlso not clear what to do with some missed values for pressure, precip_depth, wind_direction - need to take smoothed average too?\nAlso day-of-time categorization/cyclic model is worse that raw 24-hour model (separate model for every hour, because of non-smoothed behaviour for some buildings). Not sure this is applicable for your ensemble.\n
Additionally normalization all features to 0-1 (or 0-1000) will help in precision. But not investigate, how exactly.
", "votes": 0, "replies": [ { "id": 716749, "author_name": "Matt Motoki", "author_url": "", "post_date": "2020-01-12T07:45:34.470000", "content": "Thanks Nikolay, the timezone shifts are originally from this discussion post and kernel. To see how to fill in the missing values, have a look at my teammate Isamu's starter kernel. About your last point, you might be right--we didn't try building one model for each hour.
", "votes": 0, "replies": [] } ] }, { "id": 715423, "author_name": "Tomas Pechacek", "author_url": "", "post_date": "2020-01-10T14:14:07.157000", "content": "Congratulations! And thanks for sharing.
\n\nBy the way, this cyclic hour encoding of yours, this is some usual trick in ML prediction? I think, this the first time I see it and I love it!
", "votes": 0, "replies": [ { "id": 716680, "author_name": "Matt Motoki", "author_url": "", "post_date": "2020-01-12T05:05:47.570000", "content": "Thanks Tomas! I think cyclic encoding is a relatively well known technique (blogs are easy to find if you google \"cyclic feature encoding\"), but I don't see it being used on Kaggle or in practice too frequently. From my experience, these features lead to only small improvements (if any) and so maybe that's why they aren't more widely used.
", "votes": 2, "replies": [] }, { "id": 716958, "author_name": "Tomas Pechacek", "author_url": "", "post_date": "2020-01-12T14:10:41.897000", "content": "Thank you, Matt! I will look at it.
", "votes": 0, "replies": [] } ] }, { "id": 715078, "author_name": "laksh", "author_url": "", "post_date": "2020-01-10T05:27:08.637000", "content": "Thank you sharing the great work you did, this will help to understand lots of things.
", "votes": 0, "replies": [] }, { "id": 714865, "author_name": "MuriloRazoli", "author_url": "", "post_date": "2020-01-09T21:31:09.687000", "content": "Congratulations, great job!!
", "votes": 0, "replies": [] }, { "id": 714363, "author_name": "Ghawkuser", "author_url": "", "post_date": "2020-01-09T10:51:10.430000", "content": "So great!!! Congratulations, hope one day I could do half of this work!!
", "votes": 0, "replies": [] }, { "id": 713849, "author_name": "Paul Tan", "author_url": "", "post_date": "2020-01-08T18:00:22.197000", "content": "Congrats @mmotoki
", "votes": 0, "replies": [] }, { "id": 713156, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-01-08T01:12:12.073000", "content": "Congrats @mmotoki and @teeyee314. Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 712169, "author_name": "corochann", "author_url": "", "post_date": "2020-01-06T22:36:54.653000", "content": "Thank you for sharing detailed write up. Congratulations!
", "votes": 0, "replies": [] }, { "id": 711691, "author_name": "Sri Yogesh", "author_url": "", "post_date": "2020-01-06T12:19:27.553000", "content": "Hi Matt! Thanks for sharing this writeup! Congrats again!
", "votes": 0, "replies": [] }, { "id": 711682, "author_name": "Fredrik Jonsson", "author_url": "", "post_date": "2020-01-06T11:50:12.387000", "content": "Great job, as usual.
", "votes": 0, "replies": [] }, { "id": 711480, "author_name": "", "author_url": "", "post_date": "2020-01-06T07:04:19.443000", "content": "", "votes": 0, "replies": [] }, { "id": 712004, "author_name": "", "author_url": "", "post_date": "2020-01-06T18:38:57.420000", "content": "", "votes": 1, "replies": [] }, { "id": 712618, "author_name": "", "author_url": "", "post_date": "2020-01-07T12:59:15.063000", "content": "", "votes": 1, "replies": [] }, { "id": 1051981, "author_name": "", "author_url": "", "post_date": "2020-10-17T06:25:39.770000", "content": "", "votes": 0, "replies": [] }, { "id": 720733, "author_name": "", "author_url": "", "post_date": "2020-01-16T17:22:21.497000", "content": "", "votes": 0, "replies": [] }, { "id": 717120, "author_name": "", "author_url": "", "post_date": "2020-01-12T19:01:36.150000", "content": "", "votes": 0, "replies": [] }, { "id": 716915, "author_name": "", "author_url": "", "post_date": "2020-01-12T13:07:31.283000", "content": "", "votes": 0, "replies": [] }, { "id": 713976, "author_name": "", "author_url": "", "post_date": "2020-01-08T21:42:37.420000", "content": "", "votes": 0, "replies": [] }, { "id": 713153, "author_name": "", "author_url": "", "post_date": "2020-01-08T01:07:20.967000", "content": "", "votes": 0, "replies": [] }, { "id": 712580, "author_name": "", "author_url": "", "post_date": "2020-01-07T11:58:57.830000", "content": "", "votes": 0, "replies": [] }, { "id": 712505, "author_name": "", "author_url": "", "post_date": "2020-01-07T10:06:32.913000", "content": "", "votes": 0, "replies": [] }, { "id": 711410, "author_name": "", "author_url": "", "post_date": "2020-01-06T03:38:14.087000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "711401": "Thank you to Kaggle and ASHRAE for hosting this competition. The decision to use only non-leak data for the private test set helped to make this competition fair. Thank you to all those who contributed to the kernels and discussions and especially those who made the leaks public. Last but not least, I'd like to thank and congratulate my teammate [Isamu Yamashita](https://www.kaggle.com/yamsam) for being a great teammate and becoming a Competitions Master.\n\nDuring the competition, we shared ideas and discussed progress within our team, but we tested and trained separate models. This helped us maintain diversity in our final ensemble. This is a combined summary our team's solution. \n\n\n\n\n## Preprocessing\n\n### Remove anomalies\nAs others have noted, cleaning the data was very important in this competition. The assumption is that there are unpredictable and hence unlearnable anomalies in the data that, if trained on, degrade the quality of the predictions. We identified and filtered out three types of anomalies:\n1. Long streaks of constant values\n1. Large positive/negative spikes\n1. Additional anomalies determined by visual inspection\n\nWe noticed that some of these anomalies were consistent across multiple buildings at a site. We validated potential anomalies using all buildings in a site--if an anomaly showed up at the same time at multiple buildings, we could be reasonably certain that this was indeed a true anomaly. This allowed us to remove anomalies that were not necessarily part of a long streak of constant values or a large spike.\n\n### Impute Missing Temperature Values\nThere were a lot of missing values in temperature metadata. We found that imputing the missing data using linear interpolation helped our models.\n\n### Local Time Zone Correlation\nAs noted in the competition forum, the timezone in the train/test data was different from the timezone in the weather metadata. We used the information in this [discussion post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112841) to correct the timezones.\n\n### Target Transformations\nLike most competitors, we started by predicting `log1p(meter_reading)`. We also corrected the units for site 0 as per this [discussion post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/119261). \n\nNear the end of the competition, we tried standardizing `meter_reading` by dividing by `square_feet`; i.e., we predicted `log1p(meter_reading/square_feet)`. Isamu came up with the idea after reading this [discussion post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122263) by [Artyom Vorobyov](https://www.kaggle.com/artoby). The models trained with the standardized target added diversity to our final ensemble and improved our score by about 0.002. If we had more time we would have liked to explore this idea further; for example, we could have tried to predict `log1p(meter_reading)/square_feet` or created features using the standardized targets.\n\n## Feature Engineering and Feature Selection\nWe took different approaches to feature engineering and feature selection in this competition. Isamu took a conservative approach and carefully selected features; on the other hand, Matt took a brute force approach and used most of them. Here are the features that helped:\n\n* Raw features from train/test, weather metadata, and building metadata\n* Categorical interactions such as the concatenation of `building_id` and `meter`\n* Time series features including holiday flags and time of day features\n* Count (frequency) features\n* Lag temperature features similar to those found in the public kernels\n* Smoothed and 1st, 2nd-order differentiation temperature features using Savitzky-Golay filter (see the figure below)\n\n\n\n* Cyclic encoding of periodic features; e.g., `hour` gets mapped to `hour_x = cos(2*pi*hour/24)` and `hour_y = sin(2*pi*hour/24)`\n* Bayesian target encoding (see this [kernel](https://www.kaggle.com/mmotoki/hierarchical-bayesian-target-encoding))\n\n## Models\nWe trained CatBoost, LightGBM, and MLP models on different subsets of the data:\n* 1 model per meter\n* 1 model per site_id\n* 1 model per (building_id, meter)\n\nOur team tried different approaches to validation in this competition. Like other competitors, we tried K-Fold CV using consecutive months as the validation set. The following code shows one approach to getting validation months: \n```\n>>> import numpy as np\n>>> def get_validation_months(n):\n... return [np.arange(i, i+n) % 12 + 1 for i in range(12)]\n...\n>>> get_validation_months(6)\n[array([1, 2, 3, 4, 5, 6]),\n array([2, 3, 4, 5, 6, 7]),\n array([3, 4, 5, 6, 7, 8]),\n array([4, 5, 6, 7, 8, 9]),\n array([ 5, 6, 7, 8, 9, 10]),\n array([ 6, 7, 8, 9, 10, 11]),\n array([ 7, 8, 9, 10, 11, 12]),\n array([ 8, 9, 10, 11, 12, 1]),\n array([ 9, 10, 11, 12, 1, 2]),\n array([10, 11, 12, 1, 2, 3]),\n array([11, 12, 1, 2, 3, 4]),\n array([12, 1, 2, 3, 4, 5])]\n```\nTrying different validation schemes allowed us to train models that added diversity to our final ensemble.\n\n## Ensembling\nTo reduce the risk of overfitting to the public LB and improve robustness, we ensembled predictions from many different models. Here are some of the things we did:\n* Used cleaned leak data as a holdout set to tune our second stage model\n* Averaged log values; i.e., `expm1(mean(log1p(x)))` rather than averaged the raw values\n* Used the generalized weighted mean and tuned the parameters using Optuna\n* Hedged our bets by including leak-free public kernels on cleaned data:\n * [ASHRAE: Half and Half](https://www.kaggle.com/rohanrao/ashrae-half-and-half) by [Vopani](https://www.kaggle.com/rohanrao)\n * [ASHRAE- KFold LightGBM - without leak (1.08)](https://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08) by [Sandeep Kumar](https://www.kaggle.com/aitude)\n * [Ashrae: simple data cleanup (LB 1.08 no leaks)](https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks) by [Robert Stockton](https://www.kaggle.com/purist1024)\n\nThis [kernel](https://www.kaggle.com/mmotoki/generalized-weighted-mean) shows how we ensembled our predictions. Our final ensemble was a plain average of our top 4 submissions with respect to the public LB score. \n\n## What Didn't Work\n\n* Again, following this [discussion post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122263) by [Artyom Vorobyov](https://www.kaggle.com/artoby), we thought of ensembling the predictions for buildings with the same meter_readings; e.g., buildings 1225 and 1226 with meter 0. However, we were not able to improve our public LB result with this approach. It is possible that we did something wrong here, but we didn't have enough time to go back and explore this idea further. \n\n* Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth. \n\n", "717351": "Congratulations to our first-place team!!!!!! 🙌 💪 👍 👊 Their solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned! \n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "717738": "Hi and congrats \n\nWhen you say in the feature engineering section, \n> - Raw features from train/test, weather metadata, and building metadata\n- Categorical interactions such as the concatenation of building_id and meter\n- Count (frequency) features\n- ...\n\nLets say X1, X2 two categorical features and X1_X2 the interaction between X1 and X2\n\nDoes this mean that in your final data set you had the following features at the same time:\n`X1, X1_count, X2, X2_count, X1_X2, X1_X2_count `? or only `X1_count, X2_count, X1_X2_count` ?\n", "713971": "Congratulations, awesome work! \n\n> Count (frequency) features\n\nWhat kind of features are these?\n", "711611": "Congrats! Nice write up!\n\n> Smoothing the predictions of our models. We had initial success with smoothing the final predictions of our models, but after a certain point, we started to find that smoothing hurt our public LB score. Our guess is that smoothing helps spiky low quality predictions, but our ensemble predictions were already sufficiently smooth.\n\nby \"smoothing\", you mean [postprocessing](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123528) right?\n", "711859": "Great job. Your feature Engineering and Blending techniques are awesome!\nWhat was your MLP architecture?", "711414": "Congratulations\n\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @mmotoki ", "1685910": "Very interesting!", "802036": "Congratulations! \nBy the way, How much did \"1st, 2nd-order differentiation temperature features using Savitzky-Golay filter\" make accuracy better?", "749861": "Congratulations and thank you for sharing !\nI will try your approach and use Outlier Detection from another group of gold medalist.", "735255": "Congrats", "729838": "Thank you for sharing this post with the community 👍 ", "726732": "congratulation!", "723314": "Congrats, nice work!", "721156": "Congratulations @mmotoki and @yamsam and thanks for sharing your approach.", "717598": "Congratulations! Thanks for the nice write up!", "717185": "Nice! Congrats", "717066": "congratulations and thanks for sharing.💯 ", "716110": "Congrats. Awesome work!! Thank you for sharing. ", "715591": "Thanks for sharing solutions,Great work really deserving @mmotoki !!!!!!!!!!!!", "715449": "Thank you very much. Not clear what are final timezones shifts (?) for sites, and how they helped to improve the score (I thought weather data had been already given with local time).\nAlso not clear what to do with some missed values for pressure, precip_depth, wind_direction - need to take smoothed average too?\nAlso day-of-time categorization/cyclic model is worse that raw 24-hour model (separate model for every hour, because of non-smoothed behaviour for some buildings). Not sure this is applicable for your ensemble.\n\n\nAdditionally normalization all features to 0-1 (or 0-1000) will help in precision. But not investigate, how exactly.", "715423": "Congratulations! And thanks for sharing.\n\nBy the way, this cyclic hour encoding of yours, this is some usual trick in ML prediction? I think, this the first time I see it and I love it!", "715078": "Thank you sharing the great work you did, this will help to understand lots of things.", "714865": "Congratulations, great job!!", "714363": "So great!!! Congratulations, hope one day I could do half of this work!!", "713849": "Congrats @mmotoki ", "713156": "Congrats @mmotoki and @teeyee314. Thanks for sharing.", "712169": "Thank you for sharing detailed write up. Congratulations!", "711691": "Hi Matt! Thanks for sharing this writeup! Congrats again!", "711682": "Great job, as usual.", "711480": "thank you for the solution, congratulations!", "712004": "", "712618": "Thanks for sharing, great work!", "1051981": "Thank you for sharing, great work!", "720733": "Thank you for sharing", "717120": "Thank you for sharing!", "716915": "Congratulations! Thanks for sharing.", "713976": "Congrats! thanks for the writeup.", "713153": "Thanks", "712580": "Thanks for sharing!", "712505": "Great job, thank's for sharing ! 👍", "711410": "Thanks for sharing!" } }