As we are spending these festivities glancing and glancing again the LB waiting for the definitive results, we decided to already share our approach in this competition in order to let some time pass sharing some knowledge and ideas and also because we may be too much shocked later (it could be either in positive or negative ¯_(ツ)_/¯ ) for writing it down later.
\n\nBefore starting, a big mention to all the team that made this possible and that fought until the very end, in spite of a lot of disappointing events (cheating and private sharing) about which you already know: Pietro Marinelli, Stefano Morelli, Serigne Cisse, Amine Marzouki :-)
\n\nHere we start:
\n\n[Feature Engineering and Feature Selection]
\n\nWe first processed the weather data, by integrating the missing timestamps by site, then we estimated the missing values for \"air_temperature\", \"cloud_coverage\", \"dew_temperature\", \"precip_depth_1_hr\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\", using interpolation.
\n\nWe then extracted temporal data, both based on UTC time and on the local time we presumed for each site. Apart from month, week, day, hour, notably we extracted also the day of the week, and the day of the year, correcting for the fact that 2016 was a bisestile year (and the following two were not). Then we fused all training, weather and building metadata together.
\n\nAs for as feature creation:\n• We created the delta min-max air temperature by hour at site\n• We computed the sin and cos transformation of time data, such as hour, day of the week, day of the year (it actually didn’t prove so much useful)\n• We created group stats (min, max, mean, median, variance) for weather measures by grouping by site, month, day and hour.\n• We found useful to compute the delta wind in respect of 1, 2 and 3 hours before.\n• We computed some holidays by site and modelled them using the radial basis function in order to catch their effect not simply on the holiday day.\n• We used a 7 day rolling mean for air temperature that we used in order to compare with the actual temperature\n• We computed the expected average floor size of the building
\n\n[Validation Strategy]\nWe adopted two learning strategies: the first requiring downsampling the meter 0 and meter 1 cases (by 50% and 35% respectively) in order for a model to learn all the meter together, the second building a single model for each meter. In the end we also tried varying the strategy, such as building a model for working days and another one for weekend and festivities or such as a model for each meter by site combination, but the first two strategies proved the most efficient. Stefano also tried some pseudo-labelling.\nIn all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified CV, but alter we found out that a 3 fold strategy brought better LB results. We also noticed that by increasing the number of folds we obtained better oof scores, which didn’t translate in better LB performances, anyway.
\n\n[Outlier Removal]\nWe observed for outlier removal, both in train and in prediction phases. In the training phase we decided to remove the building 1099 and all the readings from building 104 for meter 0 before 2016-05-20. In the prediction phase we monitored for outlying predictions, but we noticed that by ensembling anomalous values were naturally flatten out.
\n\n[Leak]\nAfter the leak was exposed for sites 0,1,2,15 and it become public to everyone, we started using the leaked data for ensembling, that is for finding the best combination of our models. In our submissions to the LB, after finding the best combination, we also replaced our estimations for sites 0,1,2,15 with the leaked labels since we expected everyone did the same O_o and because it helped us to understand if we were improving also on the other sites, which we were sure they would have been used for the final test. We didn't use the leaked data in any other way.
\n\n[Single Models]\nWe built a large number of models: lightgbm (also dart), catboost, xgboost, dnn, random forest. In the end we had over 30 models, though we decided to use only about half of them, because the most effective accordingly to our ensembling strategy.
\n\n[Ensembling]\nWe ensembled using the leaked data as a test data. In fact, we expected our model to predict well on all sites if it was performing fine both on the leaked data and on the public leaderboard. In order to obtain a fine ensembling for this purpose, we developed a Bayesian optimization that used a linear combination of the different meter predictions. Since we noticed that certain models worked best on specific meters, we decided to give a separated weight for each meter in a prediction. \nWe also put a constrain on the optimization, thus all weights were positive or at most zero. Through successive iterations we found out that there was a sweet point, that is optimizing more for the leaked data at a certain point the LB score decreased. This information helped us to focus on certain models to be blended and exclude others.
\n\n[No one expects the Spanish Inquisition, aka the “shake up”]\nWell, but we did. Since long it has been clear for all of the team that optimizing for the leaked sites could have proved a good strategy for scoring high on the public LB but it may also have turned into a huge overfitting penalizing predictions for other sites (in spite of our hopes for generalization). \nConsequently, we decided to prepare a B-plan, what we called with the codename “the Big Mean”. In this alternative plan, we simply took all our best LB submissions and their oof and we averaged them, slightly overweighting the submissions whose oof scored higher on sites that were not part of the leak. Based on our experience of many other forecasting challenges we took, especially when you do have to forecast far in the future (as in the case of our competition), sometimes simple stuff like a plain mean does more the kind of magic you need.
\n\nWill all our work lead to a good final position? We don’t know, but in the end we found that we would have liked to explore more about holidays at the different sites and doing more data cleaning of possible outliers in training data, instead of relying on a blend that could smooth out wrong estimations. Anyway, as a team we tried really a lot,with everyone bringing ideas and testing solution. In any case we all learned a lot and enjoyed exploring solutions for this competition. Let’s see what happens with the new year :-) Meanwhile have great festivities!
", "messages": [ { "id": "702650", "postDate": "12/25/2019 01:01:32", "content": "As we are spending these festivities glancing and glancing again the LB waiting for the definitive results, we decided to already share our approach in this competition in order to let some time pass sharing some knowledge and ideas and also because we may be too much shocked later (it could be either in positive or negative ¯_(ツ)_/¯ ) for writing it down later.
\n\nBefore starting, a big mention to all the team that made this possible and that fought until the very end, in spite of a lot of disappointing events (cheating and private sharing) about which you already know: Pietro Marinelli, Stefano Morelli, Serigne Cisse, Amine Marzouki :-)
\n\nHere we start:
\n\n[Feature Engineering and Feature Selection]
\n\nWe first processed the weather data, by integrating the missing timestamps by site, then we estimated the missing values for \"air_temperature\", \"cloud_coverage\", \"dew_temperature\", \"precip_depth_1_hr\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\", using interpolation.
\n\nWe then extracted temporal data, both based on UTC time and on the local time we presumed for each site. Apart from month, week, day, hour, notably we extracted also the day of the week, and the day of the year, correcting for the fact that 2016 was a bisestile year (and the following two were not). Then we fused all training, weather and building metadata together.
\n\nAs for as feature creation:\n• We created the delta min-max air temperature by hour at site\n• We computed the sin and cos transformation of time data, such as hour, day of the week, day of the year (it actually didn’t prove so much useful)\n• We created group stats (min, max, mean, median, variance) for weather measures by grouping by site, month, day and hour.\n• We found useful to compute the delta wind in respect of 1, 2 and 3 hours before.\n• We computed some holidays by site and modelled them using the radial basis function in order to catch their effect not simply on the holiday day.\n• We used a 7 day rolling mean for air temperature that we used in order to compare with the actual temperature\n• We computed the expected average floor size of the building
\n\n[Validation Strategy]\nWe adopted two learning strategies: the first requiring downsampling the meter 0 and meter 1 cases (by 50% and 35% respectively) in order for a model to learn all the meter together, the second building a single model for each meter. In the end we also tried varying the strategy, such as building a model for working days and another one for weekend and festivities or such as a model for each meter by site combination, but the first two strategies proved the most efficient. Stefano also tried some pseudo-labelling.\nIn all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified CV, but alter we found out that a 3 fold strategy brought better LB results. We also noticed that by increasing the number of folds we obtained better oof scores, which didn’t translate in better LB performances, anyway.
\n\n[Outlier Removal]\nWe observed for outlier removal, both in train and in prediction phases. In the training phase we decided to remove the building 1099 and all the readings from building 104 for meter 0 before 2016-05-20. In the prediction phase we monitored for outlying predictions, but we noticed that by ensembling anomalous values were naturally flatten out.
\n\n[Leak]\nAfter the leak was exposed for sites 0,1,2,15 and it become public to everyone, we started using the leaked data for ensembling, that is for finding the best combination of our models. In our submissions to the LB, after finding the best combination, we also replaced our estimations for sites 0,1,2,15 with the leaked labels since we expected everyone did the same O_o and because it helped us to understand if we were improving also on the other sites, which we were sure they would have been used for the final test. We didn't use the leaked data in any other way.
\n\n[Single Models]\nWe built a large number of models: lightgbm (also dart), catboost, xgboost, dnn, random forest. In the end we had over 30 models, though we decided to use only about half of them, because the most effective accordingly to our ensembling strategy.
\n\n[Ensembling]\nWe ensembled using the leaked data as a test data. In fact, we expected our model to predict well on all sites if it was performing fine both on the leaked data and on the public leaderboard. In order to obtain a fine ensembling for this purpose, we developed a Bayesian optimization that used a linear combination of the different meter predictions. Since we noticed that certain models worked best on specific meters, we decided to give a separated weight for each meter in a prediction. \nWe also put a constrain on the optimization, thus all weights were positive or at most zero. Through successive iterations we found out that there was a sweet point, that is optimizing more for the leaked data at a certain point the LB score decreased. This information helped us to focus on certain models to be blended and exclude others.
\n\n[No one expects the Spanish Inquisition, aka the “shake up”]\nWell, but we did. Since long it has been clear for all of the team that optimizing for the leaked sites could have proved a good strategy for scoring high on the public LB but it may also have turned into a huge overfitting penalizing predictions for other sites (in spite of our hopes for generalization). \nConsequently, we decided to prepare a B-plan, what we called with the codename “the Big Mean”. In this alternative plan, we simply took all our best LB submissions and their oof and we averaged them, slightly overweighting the submissions whose oof scored higher on sites that were not part of the leak. Based on our experience of many other forecasting challenges we took, especially when you do have to forecast far in the future (as in the case of our competition), sometimes simple stuff like a plain mean does more the kind of magic you need.
\n\nWill all our work lead to a good final position? We don’t know, but in the end we found that we would have liked to explore more about holidays at the different sites and doing more data cleaning of possible outliers in training data, instead of relying on a blend that could smooth out wrong estimations. Anyway, as a team we tried really a lot,with everyone bringing ideas and testing solution. In any case we all learned a lot and enjoyed exploring solutions for this competition. Let’s see what happens with the new year :-) Meanwhile have great festivities!
", "rawMarkdown": "As we are spending these festivities glancing and glancing again the LB waiting for the definitive results, we decided to already share our approach in this competition in order to let some time pass sharing some knowledge and ideas and also because we may be too much shocked later (it could be either in positive or negative ¯\\_(ツ)_/¯ ) for writing it down later. \n\nBefore starting, a big mention to all the team that made this possible and that fought until the very end, in spite of a lot of disappointing events (cheating and private sharing) about which you already know: Pietro Marinelli, Stefano Morelli, Serigne Cisse, Amine Marzouki :-)\n\nHere we start:\n\n[Feature Engineering and Feature Selection]\n\nWe first processed the weather data, by integrating the missing timestamps by site, then we estimated the missing values for \"air_temperature\", \"cloud_coverage\", \"dew_temperature\", \"precip_depth_1_hr\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\", using interpolation. \n\nWe then extracted temporal data, both based on UTC time and on the local time we presumed for each site. Apart from month, week, day, hour, notably we extracted also the day of the week, and the day of the year, correcting for the fact that 2016 was a bisestile year (and the following two were not). Then we fused all training, weather and building metadata together.\n\nAs for as feature creation:\n•\tWe created the delta min-max air temperature by hour at site\n•\tWe computed the sin and cos transformation of time data, such as hour, day of the week, day of the year (it actually didn’t prove so much useful)\n•\tWe created group stats (min, max, mean, median, variance) for weather measures by grouping by site, month, day and hour.\n•\tWe found useful to compute the delta wind in respect of 1, 2 and 3 hours before.\n•\tWe computed some holidays by site and modelled them using the radial basis function in order to catch their effect not simply on the holiday day.\n•\tWe used a 7 day rolling mean for air temperature that we used in order to compare with the actual temperature\n•\tWe computed the expected average floor size of the building\n\n[Validation Strategy]\nWe adopted two learning strategies: the first requiring downsampling the meter 0 and meter 1 cases (by 50% and 35% respectively) in order for a model to learn all the meter together, the second building a single model for each meter. In the end we also tried varying the strategy, such as building a model for working days and another one for weekend and festivities or such as a model for each meter by site combination, but the first two strategies proved the most efficient. Stefano also tried some pseudo-labelling.\nIn all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified CV, but alter we found out that a 3 fold strategy brought better LB results. We also noticed that by increasing the number of folds we obtained better oof scores, which didn’t translate in better LB performances, anyway. \n\n[Outlier Removal]\nWe observed for outlier removal, both in train and in prediction phases. In the training phase we decided to remove the building 1099 and all the readings from building 104 for meter 0 before 2016-05-20. In the prediction phase we monitored for outlying predictions, but we noticed that by ensembling anomalous values were naturally flatten out.\n\n[Leak]\nAfter the leak was exposed for sites 0,1,2,15 and it become public to everyone, we started using the leaked data for ensembling, that is for finding the best combination of our models. In our submissions to the LB, after finding the best combination, we also replaced our estimations for sites 0,1,2,15 with the leaked labels since we expected everyone did the same O_o and because it helped us to understand if we were improving also on the other sites, which we were sure they would have been used for the final test. We didn't use the leaked data in any other way.\n\n[Single Models]\nWe built a large number of models: lightgbm (also dart), catboost, xgboost, dnn, random forest. In the end we had over 30 models, though we decided to use only about half of them, because the most effective accordingly to our ensembling strategy.\n\n[Ensembling]\nWe ensembled using the leaked data as a test data. In fact, we expected our model to predict well on all sites if it was performing fine both on the leaked data and on the public leaderboard. In order to obtain a fine ensembling for this purpose, we developed a Bayesian optimization that used a linear combination of the different meter predictions. Since we noticed that certain models worked best on specific meters, we decided to give a separated weight for each meter in a prediction. \nWe also put a constrain on the optimization, thus all weights were positive or at most zero. Through successive iterations we found out that there was a sweet point, that is optimizing more for the leaked data at a certain point the LB score decreased. This information helped us to focus on certain models to be blended and exclude others.\n\n[No one expects the Spanish Inquisition, aka the “shake up”]\nWell, but we did. Since long it has been clear for all of the team that optimizing for the leaked sites could have proved a good strategy for scoring high on the public LB but it may also have turned into a huge overfitting penalizing predictions for other sites (in spite of our hopes for generalization). \nConsequently, we decided to prepare a B-plan, what we called with the codename “the Big Mean”. In this alternative plan, we simply took all our best LB submissions and their oof and we averaged them, slightly overweighting the submissions whose oof scored higher on sites that were not part of the leak. Based on our experience of many other forecasting challenges we took, especially when you do have to forecast far in the future (as in the case of our competition), sometimes simple stuff like a plain mean does more the kind of magic you need.\n\nWill all our work lead to a good final position? We don’t know, but in the end we found that we would have liked to explore more about holidays at the different sites and doing more data cleaning of possible outliers in training data, instead of relying on a blend that could smooth out wrong estimations. Anyway, as a team we tried really a lot,with everyone bringing ideas and testing solution. In any case we all learned a lot and enjoyed exploring solutions for this competition. Let’s see what happens with the new year :-) Meanwhile have great festivities!", "votes": null }, { "id": "702812", "postDate": "12/25/2019 07:49:39", "content": "Thank you for sharing
", "rawMarkdown": "Thank you for sharing", "votes": null }, { "id": "702925", "postDate": "12/25/2019 11:12:26", "content": "Nice work! Thanks for sharing!👍
", "rawMarkdown": "Nice work! Thanks for sharing!👍", "votes": null }, { "id": "703739", "postDate": "12/26/2019 15:34:32", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": null }, { "id": "703775", "postDate": "12/26/2019 15:58:47", "content": "Great work! Congrats team Electrium Z!
", "rawMarkdown": "Great work! Congrats team Electrium Z!", "votes": null }, { "id": "703908", "postDate": "12/26/2019 19:26:49", "content": "Very informative explanation! Thank you for sharing your knowledge. I learned so much from \"Machine Learning for Dummies\" when I was first starting out!
", "rawMarkdown": "Very informative explanation! Thank you for sharing your knowledge. I learned so much from \"Machine Learning for Dummies\" when I was first starting out!", "votes": null }, { "id": "704130", "postDate": "12/27/2019 04:56:48", "content": "Thanks for sharing the 2 different models idea in 'Validation Strategy', will try something like this in my future competitions.
\n\nGood Luck in Private LB!
", "rawMarkdown": "Thanks for sharing the 2 different models idea in 'Validation Strategy', will try something like this in my future competitions.\n\nGood Luck in Private LB!", "votes": null }, { "id": "704198", "postDate": "12/27/2019 06:46:38", "content": "Many thanks for sharing your team solutions! :-)
", "rawMarkdown": "Many thanks for sharing your team solutions! :-)", "votes": null }, { "id": "704332", "postDate": "12/27/2019 10:31:17", "content": "Amazing work👍
", "rawMarkdown": "Amazing work👍", "votes": null }, { "id": "704471", "postDate": "12/27/2019 14:22:39", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "706774", "postDate": "12/30/2019 20:03:24", "content": "thank you for sharing!\nI have one question.
\n\n>In all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified >CV, but alter we found out that a 3 fold strategy brought better LB results.
\n\n'a 3 fold strategy' means 'StratifiedKFold(n=3)' ? or 'KFold(n=3)'?
", "rawMarkdown": "thank you for sharing!\nI have one question.\n\n>In all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified >CV, but alter we found out that a 3 fold strategy brought better LB results. \n\n'a 3 fold strategy' means 'StratifiedKFold(n=3)' ? or 'KFold(n=3)'?", "votes": null }, { "id": "706784", "postDate": "12/30/2019 20:18:53", "content": "Always stratified.
", "rawMarkdown": "Always stratified.", "votes": null }, { "id": "706790", "postDate": "12/30/2019 20:28:55", "content": "thank you.
", "rawMarkdown": "thank you.", "votes": null }, { "id": "717405", "postDate": "01/13/2020 04:42:27", "content": "Hey @lucamassaron -- thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well.\nIt would also be helpful if you could indicate whether you are an ASHRAE member.
", "rawMarkdown": "Hey @lucamassaron -- thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well.**\n*It would also be helpful if you could indicate whether you are an ASHRAE member.*", "votes": null }, { "id": "717459", "postDate": "01/13/2020 06:22:24", "content": "@claytonmiller Thank you very much, you can acknoledge me using the details that you can find on https://www.linkedin.com/in/lmassaron/. Please do no forget to acknoledge also the other members of the team who contributed to this solution: @pietromarinelli @steubk @serigne @aminepy : none of us is member of ASHRAE at the present time.
", "rawMarkdown": "claytonmiller Thank you very much, you can acknoledge me using the details that you can find on [https://www.linkedin.com/in/lmassaron/](https://www.linkedin.com/in/lmassaron/). Please do no forget to acknoledge also the other members of the team who contributed to this solution: @pietromarinelli @steubk @serigne @aminepy : none of us is member of ASHRAE at the present time.", "votes": null }, { "id": "718214", "postDate": "01/14/2020 07:26:10", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null } ], "comments": [ { "id": 702812, "author_name": "skinyx", "author_url": "", "post_date": "12/25/2019 07:49:39", "content": "Thank you for sharing
", "votes": null, "replies": [] }, { "id": 702925, "author_name": "mimoudata", "author_url": "", "post_date": "12/25/2019 11:12:26", "content": "Nice work! Thanks for sharing!👍
", "votes": null, "replies": [] }, { "id": 703739, "author_name": "agastya124", "author_url": "", "post_date": "12/26/2019 15:34:32", "content": "Great work!
", "votes": null, "replies": [] }, { "id": 703775, "author_name": "caesarlupum", "author_url": "", "post_date": "12/26/2019 15:58:47", "content": "Great work! Congrats team Electrium Z!
", "votes": null, "replies": [] }, { "id": 703908, "author_name": "carlolepelaars", "author_url": "", "post_date": "12/26/2019 19:26:49", "content": "Very informative explanation! Thank you for sharing your knowledge. I learned so much from \"Machine Learning for Dummies\" when I was first starting out!
", "votes": null, "replies": [] }, { "id": 704130, "author_name": "mengnanyi", "author_url": "", "post_date": "12/27/2019 04:56:48", "content": "Thanks for sharing the 2 different models idea in 'Validation Strategy', will try something like this in my future competitions.
\n\nGood Luck in Private LB!
", "votes": null, "replies": [] }, { "id": 704198, "author_name": "projdev", "author_url": "", "post_date": "12/27/2019 06:46:38", "content": "Many thanks for sharing your team solutions! :-)
", "votes": null, "replies": [] }, { "id": 704332, "author_name": "purumalgi", "author_url": "", "post_date": "12/27/2019 10:31:17", "content": "Amazing work👍
", "votes": null, "replies": [] }, { "id": 704471, "author_name": "agastya124", "author_url": "", "post_date": "12/27/2019 14:22:39", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 706774, "author_name": "yu19900724", "author_url": "", "post_date": "12/30/2019 20:03:24", "content": "thank you for sharing!\nI have one question.
\n\n>In all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified >CV, but alter we found out that a 3 fold strategy brought better LB results.
\n\n'a 3 fold strategy' means 'StratifiedKFold(n=3)' ? or 'KFold(n=3)'?
", "votes": null, "replies": [ { "id": 706784, "author_name": "lucamassaron", "author_url": "", "post_date": "12/30/2019 20:18:53", "content": "Always stratified.
", "votes": null, "replies": [] }, { "id": 706790, "author_name": "yu19900724", "author_url": "", "post_date": "12/30/2019 20:28:55", "content": "thank you.
", "votes": null, "replies": [] } ] }, { "id": 717405, "author_name": "claytonmiller", "author_url": "", "post_date": "01/13/2020 04:42:27", "content": "Hey @lucamassaron -- thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well.\nIt would also be helpful if you could indicate whether you are an ASHRAE member.
", "votes": null, "replies": [ { "id": 717459, "author_name": "lucamassaron", "author_url": "", "post_date": "01/13/2020 06:22:24", "content": "@claytonmiller Thank you very much, you can acknoledge me using the details that you can find on https://www.linkedin.com/in/lmassaron/. Please do no forget to acknoledge also the other members of the team who contributed to this solution: @pietromarinelli @steubk @serigne @aminepy : none of us is member of ASHRAE at the present time.
", "votes": null, "replies": [] }, { "id": 718214, "author_name": "claytonmiller", "author_url": "", "post_date": "01/14/2020 07:26:10", "content": "Thank you!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "702650": "As we are spending these festivities glancing and glancing again the LB waiting for the definitive results, we decided to already share our approach in this competition in order to let some time pass sharing some knowledge and ideas and also because we may be too much shocked later (it could be either in positive or negative ¯\\_(ツ)_/¯ ) for writing it down later. \n\nBefore starting, a big mention to all the team that made this possible and that fought until the very end, in spite of a lot of disappointing events (cheating and private sharing) about which you already know: Pietro Marinelli, Stefano Morelli, Serigne Cisse, Amine Marzouki :-)\n\nHere we start:\n\n[Feature Engineering and Feature Selection]\n\nWe first processed the weather data, by integrating the missing timestamps by site, then we estimated the missing values for \"air_temperature\", \"cloud_coverage\", \"dew_temperature\", \"precip_depth_1_hr\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\", using interpolation. \n\nWe then extracted temporal data, both based on UTC time and on the local time we presumed for each site. Apart from month, week, day, hour, notably we extracted also the day of the week, and the day of the year, correcting for the fact that 2016 was a bisestile year (and the following two were not). Then we fused all training, weather and building metadata together.\n\nAs for as feature creation:\n•\tWe created the delta min-max air temperature by hour at site\n•\tWe computed the sin and cos transformation of time data, such as hour, day of the week, day of the year (it actually didn’t prove so much useful)\n•\tWe created group stats (min, max, mean, median, variance) for weather measures by grouping by site, month, day and hour.\n•\tWe found useful to compute the delta wind in respect of 1, 2 and 3 hours before.\n•\tWe computed some holidays by site and modelled them using the radial basis function in order to catch their effect not simply on the holiday day.\n•\tWe used a 7 day rolling mean for air temperature that we used in order to compare with the actual temperature\n•\tWe computed the expected average floor size of the building\n\n[Validation Strategy]\nWe adopted two learning strategies: the first requiring downsampling the meter 0 and meter 1 cases (by 50% and 35% respectively) in order for a model to learn all the meter together, the second building a single model for each meter. In the end we also tried varying the strategy, such as building a model for working days and another one for weekend and festivities or such as a model for each meter by site combination, but the first two strategies proved the most efficient. Stefano also tried some pseudo-labelling.\nIn all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified CV, but alter we found out that a 3 fold strategy brought better LB results. We also noticed that by increasing the number of folds we obtained better oof scores, which didn’t translate in better LB performances, anyway. \n\n[Outlier Removal]\nWe observed for outlier removal, both in train and in prediction phases. In the training phase we decided to remove the building 1099 and all the readings from building 104 for meter 0 before 2016-05-20. In the prediction phase we monitored for outlying predictions, but we noticed that by ensembling anomalous values were naturally flatten out.\n\n[Leak]\nAfter the leak was exposed for sites 0,1,2,15 and it become public to everyone, we started using the leaked data for ensembling, that is for finding the best combination of our models. In our submissions to the LB, after finding the best combination, we also replaced our estimations for sites 0,1,2,15 with the leaked labels since we expected everyone did the same O_o and because it helped us to understand if we were improving also on the other sites, which we were sure they would have been used for the final test. We didn't use the leaked data in any other way.\n\n[Single Models]\nWe built a large number of models: lightgbm (also dart), catboost, xgboost, dnn, random forest. In the end we had over 30 models, though we decided to use only about half of them, because the most effective accordingly to our ensembling strategy.\n\n[Ensembling]\nWe ensembled using the leaked data as a test data. In fact, we expected our model to predict well on all sites if it was performing fine both on the leaked data and on the public leaderboard. In order to obtain a fine ensembling for this purpose, we developed a Bayesian optimization that used a linear combination of the different meter predictions. Since we noticed that certain models worked best on specific meters, we decided to give a separated weight for each meter in a prediction. \nWe also put a constrain on the optimization, thus all weights were positive or at most zero. Through successive iterations we found out that there was a sweet point, that is optimizing more for the leaked data at a certain point the LB score decreased. This information helped us to focus on certain models to be blended and exclude others.\n\n[No one expects the Spanish Inquisition, aka the “shake up”]\nWell, but we did. Since long it has been clear for all of the team that optimizing for the leaked sites could have proved a good strategy for scoring high on the public LB but it may also have turned into a huge overfitting penalizing predictions for other sites (in spite of our hopes for generalization). \nConsequently, we decided to prepare a B-plan, what we called with the codename “the Big Mean”. In this alternative plan, we simply took all our best LB submissions and their oof and we averaged them, slightly overweighting the submissions whose oof scored higher on sites that were not part of the leak. Based on our experience of many other forecasting challenges we took, especially when you do have to forecast far in the future (as in the case of our competition), sometimes simple stuff like a plain mean does more the kind of magic you need.\n\nWill all our work lead to a good final position? We don’t know, but in the end we found that we would have liked to explore more about holidays at the different sites and doing more data cleaning of possible outliers in training data, instead of relying on a blend that could smooth out wrong estimations. Anyway, as a team we tried really a lot,with everyone bringing ideas and testing solution. In any case we all learned a lot and enjoyed exploring solutions for this competition. Let’s see what happens with the new year :-) Meanwhile have great festivities!", "702812": "Thank you for sharing", "702925": "Nice work! Thanks for sharing!👍", "703739": "Great work!", "703775": "Great work! Congrats team Electrium Z!", "703908": "Very informative explanation! Thank you for sharing your knowledge. I learned so much from \"Machine Learning for Dummies\" when I was first starting out!", "704130": "Thanks for sharing the 2 different models idea in 'Validation Strategy', will try something like this in my future competitions.\n\nGood Luck in Private LB!", "704198": "Many thanks for sharing your team solutions! :-)", "704332": "Amazing work👍", "704471": "Thanks for sharing!", "706774": "thank you for sharing!\nI have one question.\n\n>In all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified >CV, but alter we found out that a 3 fold strategy brought better LB results. \n\n'a 3 fold strategy' means 'StratifiedKFold(n=3)' ? or 'KFold(n=3)'?", "706784": "Always stratified.", "706790": "thank you.", "717405": "Hey @lucamassaron -- thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well.**\n*It would also be helpful if you could indicate whether you are an ASHRAE member.*", "717459": "claytonmiller Thank you very much, you can acknoledge me using the details that you can find on [https://www.linkedin.com/in/lmassaron/](https://www.linkedin.com/in/lmassaron/). Please do no forget to acknoledge also the other members of the team who contributed to this solution: @pietromarinelli @steubk @serigne @aminepy : none of us is member of ASHRAE at the present time.", "718214": "Thank you!" }, "source": "existing_discussion", "complete": true, "competition_id": 9994, "competition_slug": "ashrae-energy-prediction", "team_id": 3809803, "solution_url": "/competitions/ashrae-energy-prediction/writeups/electrium-z-public-lb-24th-solution-by-team-electr" }