meter_reading value by meter type through the year of data we are given you can see a clear curve:\nHaving spent a good portion of my career working in the power industry, I've had a lot of experience with electrical power consumption data (however most of my experience is at the substation and not building level).
\n\nLast year I uploaded a public dataset with energy consumption (at the power utility level) and created a few kernels about predicting time series data. This data may be most similar to the 'electricity' meter types in the ASHRAE dataset. Hopefully some find it helpful:\n- https://www.kaggle.com/robikscube/hourly-energy-consumption\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-prophet\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-xgboost
\n\nOne thing that will become very apparent as you look at the data is how energy consumption is strongly correlated with the time of year, day of week, and hour within the day. At least for electricty this is because much of the power feeds air conditioning units - which are used more in the summer months, and can be shut off during weekends and at nights.
\n\nIf we look at the median meter_reading value by meter type through the year of data we are given you can see a clear curve:\n.png?generation=1571187811086197&alt=media)
It's also interesting to see that we are provided with only 1 year of data, and expected to predict 2 years. This means we must select models and validation schemes that can account for these annual trends, without having more than one year to work with.
\n\n.png?generation=1571187915239061&alt=media)
One thing I'm not a fan of is that the weather data is provided. It's clear that weather is strongly correlated with energy consumption, but we can't predict exactly what the future weather will be- so why would we incorporate it into our models if we are trying to predict the future? I've had a few arguments with consultants about this exact fact- weather data is only valuable if you believe the weather forecast is strong enough to improve your model. This means you need to train your models on the forecast not the actualized weather data. At least in real life that's the way it works đ .
\n\nI'm excited about this competition though, it looks like it will be a lot of fun. I'll be updating my EDA kernel with more insights as I continue to dig through the data.
\n\nHappy Kaggling!
", "messages": [ { "id": "649975", "postDate": "10/16/2019 01:16:17", "content": "Having spent a good portion of my career working in the power industry, I've had a lot of experience with electrical power consumption data (however most of my experience is at the substation and not building level).
\n\nLast year I uploaded a public dataset with energy consumption (at the power utility level) and created a few kernels about predicting time series data. This data may be most similar to the 'electricity' meter types in the ASHRAE dataset. Hopefully some find it helpful:\n- https://www.kaggle.com/robikscube/hourly-energy-consumption\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-prophet\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-xgboost
\n\nOne thing that will become very apparent as you look at the data is how energy consumption is strongly correlated with the time of year, day of week, and hour within the day. At least for electricty this is because much of the power feeds air conditioning units - which are used more in the summer months, and can be shut off during weekends and at nights.
\n\nIf we look at the median meter_reading value by meter type through the year of data we are given you can see a clear curve:\n.png?generation=1571187811086197&alt=media)
It's also interesting to see that we are provided with only 1 year of data, and expected to predict 2 years. This means we must select models and validation schemes that can account for these annual trends, without having more than one year to work with.
\n\n.png?generation=1571187915239061&alt=media)
One thing I'm not a fan of is that the weather data is provided. It's clear that weather is strongly correlated with energy consumption, but we can't predict exactly what the future weather will be- so why would we incorporate it into our models if we are trying to predict the future? I've had a few arguments with consultants about this exact fact- weather data is only valuable if you believe the weather forecast is strong enough to improve your model. This means you need to train your models on the forecast not the actualized weather data. At least in real life that's the way it works đ .
\n\nI'm excited about this competition though, it looks like it will be a lot of fun. I'll be updating my EDA kernel with more insights as I continue to dig through the data.
\n\nHappy Kaggling!
", "rawMarkdown": "Having spent a good portion of my career working in the power industry, I've had a lot of experience with electrical power consumption data (however most of my experience is at the substation and not building level).\n\nLast year I uploaded a public dataset with energy consumption (at the power utility level) and created a few kernels about predicting time series data. This data may be most similar to the 'electricity' meter types in the ASHRAE dataset. Hopefully some find it helpful:\n- https://www.kaggle.com/robikscube/hourly-energy-consumption\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-prophet\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-xgboost\n\nOne thing that will become very apparent as you look at the data is how energy consumption is strongly correlated with the time of year, day of week, and hour within the day. At least for **electricty** this is because much of the power feeds air conditioning units - which are used more in the summer months, and can be shut off during weekends and at nights.\n\nIf we look at the median `meter_reading` value by meter type through the year of data we are given you can see a clear curve:\n.png?generation=1571187811086197&alt=media)\n\nIt's also interesting to see that we are provided with only 1 year of data, and expected to predict 2 years. This means we must select models and validation schemes that can account for these annual trends, without having more than one year to work with.\n\n.png?generation=1571187915239061&alt=media)\n\n\n\nOne thing I'm not a fan of is that the weather data is provided. It's clear that weather is strongly correlated with energy consumption, **but we can't predict exactly what the future weather will be**- so why would we incorporate it into our models if we are trying to predict the future? I've had a few arguments with consultants about this exact fact- weather data is only valuable if you believe the weather *forecast* is strong enough to improve your model. This means you need to train your models on the *forecast* not the actualized weather data. At least in real life that's the way it works đ .\n\nI'm excited about this competition though, it looks like it will be a lot of fun. I'll be updating my [EDA kernel ](https://www.kaggle.com/robikscube/predicting-energy-consumption-eda-and-starter)with more insights as I continue to dig through the data.\n\nHappy Kaggling!", "votes": null }, { "id": "650006", "postDate": "10/16/2019 02:17:26", "content": "Indeed! It will to be a lot of fun!
", "rawMarkdown": "Indeed! It will to be a lot of fun!", "votes": null }, { "id": "650098", "postDate": "10/16/2019 05:16:41", "content": "Amazing Summary..\nThanks for Sharing Thoughts... @robikscube
", "rawMarkdown": "Amazing Summary..\nThanks for Sharing Thoughts... @robikscube", "votes": null }, { "id": "650191", "postDate": "10/16/2019 07:10:44", "content": "I totally agree with you (especially on the aspect on the part about using weather data) @robikscube.
\n\nMicrosoft recently released their building energy saving solution which uses multiple machine learning models (1 of which is a weather forecasting model).
\n\n", "rawMarkdown": "I totally agree with you (especially on the aspect on the part about using weather data) @robikscube.\n\nMicrosoft recently released their building energy saving solution which uses multiple machine learning models (1 of which is a weather forecasting model).\n\nhttps://www.microsoft.com/en-us/itshowcase/microsoft-uses-machine-learning-to-develop-smart-energy-solutions", "votes": null }, { "id": "650277", "postDate": "10/16/2019 08:38:44", "content": "IMHO the use case should have been the following:
\n\nBased on today's data[ X(day=today) ], predict the energy consumption N days in future [Y(day=today+ N Days)].
\n\nBeing a competition organized by Kaggle, I am finding the current goal to be little fishy. :-)
", "rawMarkdown": "IMHO the use case should have been the following: \n\nBased on today's data[ X(day=today) ], predict the energy consumption N days in future [Y(day=today+ N Days)]. \n\nBeing a competition organized by Kaggle, I am finding the current goal to be little fishy. :-)", "votes": null }, { "id": "650308", "postDate": "10/16/2019 08:59:16", "content": "Am I missing something?\n I thought the point of this challenge was to build an estimation-model to show what the energy consumption of a building would have been without energy saving changes that has been made to it. Thus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model).... Therefore actual historical weather data for the period we are interested in would be available.... was my thought.
", "rawMarkdown": "Am I missing something?\n I thought the point of this challenge was to build an estimation-model to show what the energy consumption of a building would have been without energy saving changes that has been made to it. Thus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model).... Therefore actual historical weather data for the period we are interested in would be available.... was my thought.", "votes": null }, { "id": "650345", "postDate": "10/16/2019 09:39:35", "content": "I agree with your points on weather and the short time-frame of the data. I do, however, appreciate that the competition is looking at this from a long-term perspective and energy efficiency. EE occurs behind the meter so itâs not easy to really know itâs impact. Usually, there are profiles on hypothetical demand for appliances, etc. But these are generalized shapes... not something that really sells to individual customers on the fence with spending money on EE.
\n\nThe benefit of this competition as opposed to real-world is that the historical load demand and weather have been provided. If we had to also forecast weather, most of our models would do much poorer. Most utilities rely on \"simple\" approaches such as 20 year averages. This approach actually didn't pan out well for the past few years where heat waves has driven temperatures higher than usual.. you can image how that affected our forecastsl However, the sub-seasonal RODEO forecast competition is promising... maybe one day. Anyway, the point is that weather forecasting in itself is time consuming (see RODEO competition) and would eat up a lot of time in this competition...
\n\n... which in the end a hypothetical approach to estimating EE. Utilities moves at a glacial pace. Throwing in something like, âwe estimate your savings to be _ using our ML/AI modelsâ would make almost anyoneâs head explode when talking about utilities and energy savings.
", "rawMarkdown": "I agree with your points on weather and the short time-frame of the data. I do, however, appreciate that the competition is looking at this from a long-term perspective and energy efficiency. EE occurs behind the meter so itâs not easy to really know itâs impact. Usually, there are profiles on hypothetical demand for appliances, etc. But these are generalized shapes... not something that really sells to individual customers on the fence with spending money on EE.\n\nThe benefit of this competition as opposed to real-world is that the historical load demand and weather have been provided. If we had to also forecast weather, most of our models would do much poorer. Most utilities rely on \"simple\" approaches such as 20 year averages. This approach actually didn't pan out well for the past few years where heat waves has driven temperatures higher than usual.. you can image how that affected our forecastsl However, the sub-seasonal RODEO forecast competition is promising... maybe one day. Anyway, the point is that weather forecasting in itself is time consuming (see RODEO competition) and would eat up a lot of time in this competition...\n\n... which in the end a hypothetical approach to estimating EE. Utilities moves at a glacial pace. Throwing in something like, âwe estimate your savings to be ___ using our ML/AI modelsâ would make almost anyoneâs head explode when talking about utilities and energy savings.", "votes": null }, { "id": "650377", "postDate": "10/16/2019 10:00:02", "content": "@magnusab I think you are correct. From the description of the competition:
\n\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.
@magnusab - you maybe right, I'm not quite clear on exactly what the sponsors are trying to get out of this competition. The description says:
\n\n```\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.
\n\nIn this competition, youâll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.
\n\nWith better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n```\nI'm not sure what they mean here, or they hope to achieve it by having us predict meter readings:\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.
\n\nI still think my point stands. Clearly they have the actual meter data from 2017-2018 which they will be using to score our submission. What is valuable to them is being able to have models that predict the energy consumption for 2020 and beyond for buildings with and without the improvements. If that's what the goal then creating models that use actualized weather data will overestimate their predictive power.
\n\nPlease correct me if I'm missing something...
", "rawMarkdown": "magnusab - you maybe right, I'm not quite clear on exactly what the sponsors are trying to get out of this competition. The description says:\n\n```\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.\n\nIn this competition, youâll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.\n\nWith better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n```\nI'm not sure what they mean here, or they hope to achieve it by having us predict meter readings:\n**Developing energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.**\n\nI still think my point stands. Clearly they have the actual meter data from 2017-2018 which they will be using to score our submission. What is valuable to them is being able to have models that predict the energy consumption for 2020 and beyond for buildings with and without the improvements. If that's what the goal then creating models that use **actualized** weather data will overestimate their predictive power.\n\nPlease correct me if I'm missing something...", "votes": null }, { "id": "650513", "postDate": "10/16/2019 13:10:16", "content": "Re reading your post again and when you say this:
\n\n\n\n\nThus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model)
\n
If this were the case - how would they score our submissions? If the changes were made then they have no \"true\" measurement if the changes were not made.
", "rawMarkdown": "Re reading your post again and when you say this:\n> Thus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model)\n\nIf this were the case - how would they score our submissions? If the changes were made then they have no \"true\" measurement if the changes were not made.", "votes": null }, { "id": "650528", "postDate": "10/16/2019 13:30:14", "content": "They have 100% real data before change.
\n\nThey made changes let's say for 50% of buildings
\n\n50% without changes went to test set to evaluate model.\nIf model is good they will predict 50% with change and see consumption difference to prove energy saving and eco transformation strategy.
", "rawMarkdown": "They have 100% real data before change.\n\nThey made changes let's say for 50% of buildings\n\n50% without changes went to test set to evaluate model.\nIf model is good they will predict 50% with change and see consumption difference to prove energy saving and eco transformation strategy.", "votes": null }, { "id": "650539", "postDate": "10/16/2019 13:40:25", "content": "@kyakovlev I see what you're saying, but I don't follow the logic. If that were the case it would be a classic A/B test. Why would they need our models to predict anything? They already have the meter data for the 50% without changes (control group) and 50% who made changes (treatment group) for the years we are predicting.
\n\nI've asked this question to the sponsor in their discussion thread here. Hopefully they can clear things up.
\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112985#latest-650506
", "rawMarkdown": "kyakovlev I see what you're saying, but I don't follow the logic. If that were the case it would be a classic A/B test. Why would they need our models to predict anything? They already have the meter data for the 50% without changes (control group) and 50% who made changes (treatment group) for the years we are predicting.\n\nI've asked this question to the sponsor in their discussion thread here. Hopefully they can clear things up.\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112985#latest-650506", "votes": null }, { "id": "650545", "postDate": "10/16/2019 13:47:45", "content": "Ohh, it's easy.\nMoney)))
\n\nThey go to NY municipality\n* Let's do eco/energy transformation\n* But we don't have money for it\n* Give us consumption historical data\n* Here it is\n* Our model predicts SUCH HUGE savings for future two yeas\n* WOW!! No more words, take our money, do the job!
\n\nSo now we are predicting consumption for next two years with changes our without we don't know.\nProbably in dataset are similar buildings with and without change, probably only with change, or only without. ASHRAE people want to know how good we can predict consumption for such long periods.
\n\nOf course it's not about money itself - Money is a key to make decisions by people how not care about making world better. Less brutal earth consumption will be - more \"Grand\" prefix for grandgrandgrandson will be real.
", "rawMarkdown": "Ohh, it's easy.\nMoney))) \n\nThey go to NY municipality\n* Let's do eco/energy transformation\n* But we don't have money for it\n* Give us consumption historical data\n* Here it is\n* Our model predicts SUCH HUGE savings for future two yeas\n* WOW!! No more words, take our money, do the job!\n\nSo now we are predicting consumption for next two years with changes our without we don't know.\nProbably in dataset are similar buildings with and without change, probably only with change, or only without. ASHRAE people want to know how good we can predict consumption for such long periods. \n\nOf course it's not about money itself - Money is a key to make decisions by people how not care about making world better. Less brutal earth consumption will be - more \"Grand\" prefix for grandgrandgrandson will be real.", "votes": null }, { "id": "652292", "postDate": "10/18/2019 16:29:06", "content": "@robikscube \nWhen it comes to scoring us. Dont they just score our model on how well it estimates for that point in time on actual data..... Then to utilize it in practice you train it on data up until you did the change on the building, then you feed it with data for the period after making the change... you compare the \"estimated energy expenditure without change\" with actual expenditure and you put a dollar sign on how much eneergy the change saved you?
\n\nIm in a hurry, so didnt think this properly through, but please point out where I went wrong anyway :) Just so I get it clearified :)
", "rawMarkdown": "robikscube \nWhen it comes to scoring us. Dont they just score our model on how well it estimates for that point in time on actual data..... Then to utilize it in practice you train it on data up until you did the change on the building, then you feed it with data for the period after making the change... you compare the \"estimated energy expenditure without change\" with actual expenditure and you put a dollar sign on how much eneergy the change saved you?\n\nIm in a hurry, so didnt think this properly through, but please point out where I went wrong anyway :) Just so I get it clearified :)", "votes": null }, { "id": "659687", "postDate": "10/28/2019 04:48:55", "content": "Hi Rob, \nI try to explain your confusion:\n1. You motivation is: predict the future energy usage.\n2. ASHRAE's motivation is: after install new building equipments, what is the potential energy reducing if without the installations? This is different from A/B testing,** since they never saved 50% data without changes.** They have renewed all equipments! What they want to is to find the 'counterfactual' model to calculate the energy reducing.
", "rawMarkdown": "Hi Rob, \nI try to explain your confusion:\n1. You motivation is: predict the future energy usage.\n2. ASHRAE's motivation is: after install new building equipments, what is the potential energy reducing if without the installations? This is different from A/B testing,** since they never saved 50% data without changes.** They have renewed all equipments! What they want to is to find the 'counterfactual' model to calculate the energy reducing.", "votes": null }, { "id": "698797", "postDate": "12/19/2019 17:43:11", "content": "An interesting comment and thread. I'm new to Kaggle and enjoyed the challenge, but I also had questions about the utility of any models developed in this process. I've have experience reviewing building energy performance, including simulation models. A few thoughts below. Would be interested in other perspectives. \n1. The challenge goal was to accurately predict energy use that under a hypothetical scenario would have occurred if not for the installation of energy savings measures. The difference in observed post-installation energy use and the predicted energy use would be deemed the realized energy savings. Energy Service Contacts, were a building manager only pays for realized energy savings, carefully consider how energy savings will be determined and documented. More accurate ways to establish \"what if\" energy use assuming no energy savings measures were installed would be useful. \n2. Accurately predicting \"annual\" energy use would be useful as it is strongly correlated with annual energy costs. There may be errors in predicting each month, day or hour but as long as the negatives offset the positives then the sum over a year can be accurate. I had models predicting annual energy use for 2017 & 2018 that were very close to the actual energy use. But, these models only ran in the middle of pack using the RMSLE. I wonder if using the RMSLE results in higher error when predicting annual energy use as opposed hourly energy use. The RMSLE would bias toward lower energy consumption hours, while a simple mean error would bias toward the higher energy use hours and I would think be more consistent with a simple annual sum for energy. \n3. I did not check on how well my models did for peak periods, i.e. the highest energy use during each month, or the annual peak. I also suspect a RMSLE fitted model might be more likely to under-predict the energy use in these periods as opposed to say using a simple mean error. These peaks which can set distribution , transmission or capacity costs would be important to get right.\n4. In real life the first thing a modeler would do I think would be to clean up the data: establish units (kBTU, therms, kWh..) and remove any data that doesn't pass the laugh test. In my case I found data cleaning of limited value given (1) the Private data set also had data issues, so cleaning up the Train data set could actually result in a worse score, and (2) I chose not to view or use the leaked data as doing so would seem to violate the spirit of the challenge. Seems like the goal should be to take sloppy data and make clean predictions with it -- which would suggest the Private data set should be cleaned up and made as close as possible to actual energy use. Taking \"garbage\" data and intentionally making \"garbage\" predictions doesn't seem like it would be very useful. A model that could take garbage data and make \"clean\" predictions, on the other hand, could be very useful.
\n\nGood luck with your Private scoring
", "rawMarkdown": "An interesting comment and thread. I'm new to Kaggle and enjoyed the challenge, but I also had questions about the utility of any models developed in this process. I've have experience reviewing building energy performance, including simulation models. A few thoughts below. Would be interested in other perspectives. \n1. The challenge goal was to accurately predict energy use that under a hypothetical scenario would have occurred if not for the installation of energy savings measures. The difference in observed post-installation energy use and the predicted energy use would be deemed the realized energy savings. Energy Service Contacts, were a building manager only pays for realized energy savings, carefully consider how energy savings will be determined and documented. More accurate ways to establish \"what if\" energy use assuming no energy savings measures were installed would be useful. \n2. Accurately predicting \"annual\" energy use would be useful as it is strongly correlated with annual energy costs. There may be errors in predicting each month, day or hour but as long as the negatives offset the positives then the sum over a year can be accurate. I had models predicting annual energy use for 2017 & 2018 that were very close to the actual energy use. But, these models only ran in the middle of pack using the RMSLE. I wonder if using the RMSLE results in higher error when predicting annual energy use as opposed hourly energy use. The RMSLE would bias toward lower energy consumption hours, while a simple mean error would bias toward the higher energy use hours and I would think be more consistent with a simple annual sum for energy. \n3. I did not check on how well my models did for peak periods, i.e. the highest energy use during each month, or the annual peak. I also suspect a RMSLE fitted model might be more likely to under-predict the energy use in these periods as opposed to say using a simple mean error. These peaks which can set distribution , transmission or capacity costs would be important to get right.\n4. In real life the first thing a modeler would do I think would be to clean up the data: establish units (kBTU, therms, kWh..) and remove any data that doesn't pass the laugh test. In my case I found data cleaning of limited value given (1) the Private data set also had data issues, so cleaning up the Train data set could actually result in a worse score, and (2) I chose not to view or use the leaked data as doing so would seem to violate the spirit of the challenge. Seems like the goal should be to take sloppy data and make clean predictions with it -- which would suggest the Private data set should be cleaned up and made as close as possible to actual energy use. Taking \"garbage\" data and intentionally making \"garbage\" predictions doesn't seem like it would be very useful. A model that could take garbage data and make \"clean\" predictions, on the other hand, could be very useful. \n\nGood luck with your Private scoring", "votes": null } ], "comments": [ { "id": 650006, "author_name": "tayorm", "author_url": "", "post_date": "10/16/2019 02:17:26", "content": "Indeed! It will to be a lot of fun!
", "votes": null, "replies": [] }, { "id": 650098, "author_name": "veeralakrishna", "author_url": "", "post_date": "10/16/2019 05:16:41", "content": "Amazing Summary..\nThanks for Sharing Thoughts... @robikscube
", "votes": null, "replies": [] }, { "id": 650191, "author_name": "tiannnn", "author_url": "", "post_date": "10/16/2019 07:10:44", "content": "I totally agree with you (especially on the aspect on the part about using weather data) @robikscube.
\n\nMicrosoft recently released their building energy saving solution which uses multiple machine learning models (1 of which is a weather forecasting model).
\n\n", "votes": null, "replies": [] }, { "id": 650277, "author_name": "arnabbiswas1", "author_url": "", "post_date": "10/16/2019 08:38:44", "content": "IMHO the use case should have been the following:
\n\nBased on today's data[ X(day=today) ], predict the energy consumption N days in future [Y(day=today+ N Days)].
\n\nBeing a competition organized by Kaggle, I am finding the current goal to be little fishy. :-)
", "votes": null, "replies": [] }, { "id": 650308, "author_name": "magnusab", "author_url": "", "post_date": "10/16/2019 08:59:16", "content": "Am I missing something?\n I thought the point of this challenge was to build an estimation-model to show what the energy consumption of a building would have been without energy saving changes that has been made to it. Thus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model).... Therefore actual historical weather data for the period we are interested in would be available.... was my thought.
", "votes": null, "replies": [ { "id": 650377, "author_name": "arnabbiswas1", "author_url": "", "post_date": "10/16/2019 10:00:02", "content": "@magnusab I think you are correct. From the description of the competition:
\n\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.
@magnusab - you maybe right, I'm not quite clear on exactly what the sponsors are trying to get out of this competition. The description says:
\n\n```\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.
\n\nIn this competition, youâll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.
\n\nWith better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n```\nI'm not sure what they mean here, or they hope to achieve it by having us predict meter readings:\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.
\n\nI still think my point stands. Clearly they have the actual meter data from 2017-2018 which they will be using to score our submission. What is valuable to them is being able to have models that predict the energy consumption for 2020 and beyond for buildings with and without the improvements. If that's what the goal then creating models that use actualized weather data will overestimate their predictive power.
\n\nPlease correct me if I'm missing something...
", "votes": null, "replies": [] }, { "id": 650513, "author_name": "robikscube", "author_url": "", "post_date": "10/16/2019 13:10:16", "content": "Re reading your post again and when you say this:
\n\n\n\n\nThus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model)
\n
If this were the case - how would they score our submissions? If the changes were made then they have no \"true\" measurement if the changes were not made.
", "votes": null, "replies": [] }, { "id": 650528, "author_name": "kyakovlev", "author_url": "", "post_date": "10/16/2019 13:30:14", "content": "They have 100% real data before change.
\n\nThey made changes let's say for 50% of buildings
\n\n50% without changes went to test set to evaluate model.\nIf model is good they will predict 50% with change and see consumption difference to prove energy saving and eco transformation strategy.
", "votes": null, "replies": [] }, { "id": 650539, "author_name": "robikscube", "author_url": "", "post_date": "10/16/2019 13:40:25", "content": "@kyakovlev I see what you're saying, but I don't follow the logic. If that were the case it would be a classic A/B test. Why would they need our models to predict anything? They already have the meter data for the 50% without changes (control group) and 50% who made changes (treatment group) for the years we are predicting.
\n\nI've asked this question to the sponsor in their discussion thread here. Hopefully they can clear things up.
\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112985#latest-650506
", "votes": null, "replies": [] }, { "id": 650545, "author_name": "kyakovlev", "author_url": "", "post_date": "10/16/2019 13:47:45", "content": "Ohh, it's easy.\nMoney)))
\n\nThey go to NY municipality\n* Let's do eco/energy transformation\n* But we don't have money for it\n* Give us consumption historical data\n* Here it is\n* Our model predicts SUCH HUGE savings for future two yeas\n* WOW!! No more words, take our money, do the job!
\n\nSo now we are predicting consumption for next two years with changes our without we don't know.\nProbably in dataset are similar buildings with and without change, probably only with change, or only without. ASHRAE people want to know how good we can predict consumption for such long periods.
\n\nOf course it's not about money itself - Money is a key to make decisions by people how not care about making world better. Less brutal earth consumption will be - more \"Grand\" prefix for grandgrandgrandson will be real.
", "votes": null, "replies": [] }, { "id": 659687, "author_name": "alphado", "author_url": "", "post_date": "10/28/2019 04:48:55", "content": "Hi Rob, \nI try to explain your confusion:\n1. You motivation is: predict the future energy usage.\n2. ASHRAE's motivation is: after install new building equipments, what is the potential energy reducing if without the installations? This is different from A/B testing,** since they never saved 50% data without changes.** They have renewed all equipments! What they want to is to find the 'counterfactual' model to calculate the energy reducing.
", "votes": null, "replies": [] } ] }, { "id": 650345, "author_name": "learnmower", "author_url": "", "post_date": "10/16/2019 09:39:35", "content": "I agree with your points on weather and the short time-frame of the data. I do, however, appreciate that the competition is looking at this from a long-term perspective and energy efficiency. EE occurs behind the meter so itâs not easy to really know itâs impact. Usually, there are profiles on hypothetical demand for appliances, etc. But these are generalized shapes... not something that really sells to individual customers on the fence with spending money on EE.
\n\nThe benefit of this competition as opposed to real-world is that the historical load demand and weather have been provided. If we had to also forecast weather, most of our models would do much poorer. Most utilities rely on \"simple\" approaches such as 20 year averages. This approach actually didn't pan out well for the past few years where heat waves has driven temperatures higher than usual.. you can image how that affected our forecastsl However, the sub-seasonal RODEO forecast competition is promising... maybe one day. Anyway, the point is that weather forecasting in itself is time consuming (see RODEO competition) and would eat up a lot of time in this competition...
\n\n... which in the end a hypothetical approach to estimating EE. Utilities moves at a glacial pace. Throwing in something like, âwe estimate your savings to be _ using our ML/AI modelsâ would make almost anyoneâs head explode when talking about utilities and energy savings.
", "votes": null, "replies": [] }, { "id": 652292, "author_name": "magnusab", "author_url": "", "post_date": "10/18/2019 16:29:06", "content": "@robikscube \nWhen it comes to scoring us. Dont they just score our model on how well it estimates for that point in time on actual data..... Then to utilize it in practice you train it on data up until you did the change on the building, then you feed it with data for the period after making the change... you compare the \"estimated energy expenditure without change\" with actual expenditure and you put a dollar sign on how much eneergy the change saved you?
\n\nIm in a hurry, so didnt think this properly through, but please point out where I went wrong anyway :) Just so I get it clearified :)
", "votes": null, "replies": [] }, { "id": 698797, "author_name": "dzbeavers", "author_url": "", "post_date": "12/19/2019 17:43:11", "content": "An interesting comment and thread. I'm new to Kaggle and enjoyed the challenge, but I also had questions about the utility of any models developed in this process. I've have experience reviewing building energy performance, including simulation models. A few thoughts below. Would be interested in other perspectives. \n1. The challenge goal was to accurately predict energy use that under a hypothetical scenario would have occurred if not for the installation of energy savings measures. The difference in observed post-installation energy use and the predicted energy use would be deemed the realized energy savings. Energy Service Contacts, were a building manager only pays for realized energy savings, carefully consider how energy savings will be determined and documented. More accurate ways to establish \"what if\" energy use assuming no energy savings measures were installed would be useful. \n2. Accurately predicting \"annual\" energy use would be useful as it is strongly correlated with annual energy costs. There may be errors in predicting each month, day or hour but as long as the negatives offset the positives then the sum over a year can be accurate. I had models predicting annual energy use for 2017 & 2018 that were very close to the actual energy use. But, these models only ran in the middle of pack using the RMSLE. I wonder if using the RMSLE results in higher error when predicting annual energy use as opposed hourly energy use. The RMSLE would bias toward lower energy consumption hours, while a simple mean error would bias toward the higher energy use hours and I would think be more consistent with a simple annual sum for energy. \n3. I did not check on how well my models did for peak periods, i.e. the highest energy use during each month, or the annual peak. I also suspect a RMSLE fitted model might be more likely to under-predict the energy use in these periods as opposed to say using a simple mean error. These peaks which can set distribution , transmission or capacity costs would be important to get right.\n4. In real life the first thing a modeler would do I think would be to clean up the data: establish units (kBTU, therms, kWh..) and remove any data that doesn't pass the laugh test. In my case I found data cleaning of limited value given (1) the Private data set also had data issues, so cleaning up the Train data set could actually result in a worse score, and (2) I chose not to view or use the leaked data as doing so would seem to violate the spirit of the challenge. Seems like the goal should be to take sloppy data and make clean predictions with it -- which would suggest the Private data set should be cleaned up and made as close as possible to actual energy use. Taking \"garbage\" data and intentionally making \"garbage\" predictions doesn't seem like it would be very useful. A model that could take garbage data and make \"clean\" predictions, on the other hand, could be very useful.
\n\nGood luck with your Private scoring
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "649975": "Having spent a good portion of my career working in the power industry, I've had a lot of experience with electrical power consumption data (however most of my experience is at the substation and not building level).\n\nLast year I uploaded a public dataset with energy consumption (at the power utility level) and created a few kernels about predicting time series data. This data may be most similar to the 'electricity' meter types in the ASHRAE dataset. Hopefully some find it helpful:\n- https://www.kaggle.com/robikscube/hourly-energy-consumption\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-prophet\n- https://www.kaggle.com/robikscube/tutorial-time-series-forecasting-with-xgboost\n\nOne thing that will become very apparent as you look at the data is how energy consumption is strongly correlated with the time of year, day of week, and hour within the day. At least for **electricty** this is because much of the power feeds air conditioning units - which are used more in the summer months, and can be shut off during weekends and at nights.\n\nIf we look at the median `meter_reading` value by meter type through the year of data we are given you can see a clear curve:\n.png?generation=1571187811086197&alt=media)\n\nIt's also interesting to see that we are provided with only 1 year of data, and expected to predict 2 years. This means we must select models and validation schemes that can account for these annual trends, without having more than one year to work with.\n\n.png?generation=1571187915239061&alt=media)\n\n\n\nOne thing I'm not a fan of is that the weather data is provided. It's clear that weather is strongly correlated with energy consumption, **but we can't predict exactly what the future weather will be**- so why would we incorporate it into our models if we are trying to predict the future? I've had a few arguments with consultants about this exact fact- weather data is only valuable if you believe the weather *forecast* is strong enough to improve your model. This means you need to train your models on the *forecast* not the actualized weather data. At least in real life that's the way it works đ .\n\nI'm excited about this competition though, it looks like it will be a lot of fun. I'll be updating my [EDA kernel ](https://www.kaggle.com/robikscube/predicting-energy-consumption-eda-and-starter)with more insights as I continue to dig through the data.\n\nHappy Kaggling!", "650006": "Indeed! It will to be a lot of fun!", "650098": "Amazing Summary..\nThanks for Sharing Thoughts... @robikscube", "650191": "I totally agree with you (especially on the aspect on the part about using weather data) @robikscube.\n\nMicrosoft recently released their building energy saving solution which uses multiple machine learning models (1 of which is a weather forecasting model).\n\nhttps://www.microsoft.com/en-us/itshowcase/microsoft-uses-machine-learning-to-develop-smart-energy-solutions", "650277": "IMHO the use case should have been the following: \n\nBased on today's data[ X(day=today) ], predict the energy consumption N days in future [Y(day=today+ N Days)]. \n\nBeing a competition organized by Kaggle, I am finding the current goal to be little fishy. :-)", "650308": "Am I missing something?\n I thought the point of this challenge was to build an estimation-model to show what the energy consumption of a building would have been without energy saving changes that has been made to it. Thus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model).... Therefore actual historical weather data for the period we are interested in would be available.... was my thought.", "650345": "I agree with your points on weather and the short time-frame of the data. I do, however, appreciate that the competition is looking at this from a long-term perspective and energy efficiency. EE occurs behind the meter so itâs not easy to really know itâs impact. Usually, there are profiles on hypothetical demand for appliances, etc. But these are generalized shapes... not something that really sells to individual customers on the fence with spending money on EE.\n\nThe benefit of this competition as opposed to real-world is that the historical load demand and weather have been provided. If we had to also forecast weather, most of our models would do much poorer. Most utilities rely on \"simple\" approaches such as 20 year averages. This approach actually didn't pan out well for the past few years where heat waves has driven temperatures higher than usual.. you can image how that affected our forecastsl However, the sub-seasonal RODEO forecast competition is promising... maybe one day. Anyway, the point is that weather forecasting in itself is time consuming (see RODEO competition) and would eat up a lot of time in this competition...\n\n... which in the end a hypothetical approach to estimating EE. Utilities moves at a glacial pace. Throwing in something like, âwe estimate your savings to be ___ using our ML/AI modelsâ would make almost anyoneâs head explode when talking about utilities and energy savings.", "650377": "magnusab I think you are correct. From the description of the competition:\n\n`Developing energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.`", "650493": "magnusab - you maybe right, I'm not quite clear on exactly what the sponsors are trying to get out of this competition. The description says:\n\n```\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.\n\nIn this competition, youâll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.\n\nWith better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n```\nI'm not sure what they mean here, or they hope to achieve it by having us predict meter readings:\n**Developing energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall.**\n\nI still think my point stands. Clearly they have the actual meter data from 2017-2018 which they will be using to score our submission. What is valuable to them is being able to have models that predict the energy consumption for 2020 and beyond for buildings with and without the improvements. If that's what the goal then creating models that use **actualized** weather data will overestimate their predictive power.\n\nPlease correct me if I'm missing something...", "650513": "Re reading your post again and when you say this:\n> Thus it is used restrospectively to compare the actual energy consumption, and the estimated one if changes had not been made (our model)\n\nIf this were the case - how would they score our submissions? If the changes were made then they have no \"true\" measurement if the changes were not made.", "650528": "They have 100% real data before change.\n\nThey made changes let's say for 50% of buildings\n\n50% without changes went to test set to evaluate model.\nIf model is good they will predict 50% with change and see consumption difference to prove energy saving and eco transformation strategy.", "650539": "kyakovlev I see what you're saying, but I don't follow the logic. If that were the case it would be a classic A/B test. Why would they need our models to predict anything? They already have the meter data for the 50% without changes (control group) and 50% who made changes (treatment group) for the years we are predicting.\n\nI've asked this question to the sponsor in their discussion thread here. Hopefully they can clear things up.\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112985#latest-650506", "650545": "Ohh, it's easy.\nMoney))) \n\nThey go to NY municipality\n* Let's do eco/energy transformation\n* But we don't have money for it\n* Give us consumption historical data\n* Here it is\n* Our model predicts SUCH HUGE savings for future two yeas\n* WOW!! No more words, take our money, do the job!\n\nSo now we are predicting consumption for next two years with changes our without we don't know.\nProbably in dataset are similar buildings with and without change, probably only with change, or only without. ASHRAE people want to know how good we can predict consumption for such long periods. \n\nOf course it's not about money itself - Money is a key to make decisions by people how not care about making world better. Less brutal earth consumption will be - more \"Grand\" prefix for grandgrandgrandson will be real.", "652292": "robikscube \nWhen it comes to scoring us. Dont they just score our model on how well it estimates for that point in time on actual data..... Then to utilize it in practice you train it on data up until you did the change on the building, then you feed it with data for the period after making the change... you compare the \"estimated energy expenditure without change\" with actual expenditure and you put a dollar sign on how much eneergy the change saved you?\n\nIm in a hurry, so didnt think this properly through, but please point out where I went wrong anyway :) Just so I get it clearified :)", "659687": "Hi Rob, \nI try to explain your confusion:\n1. You motivation is: predict the future energy usage.\n2. ASHRAE's motivation is: after install new building equipments, what is the potential energy reducing if without the installations? This is different from A/B testing,** since they never saved 50% data without changes.** They have renewed all equipments! What they want to is to find the 'counterfactual' model to calculate the energy reducing.", "698797": "An interesting comment and thread. I'm new to Kaggle and enjoyed the challenge, but I also had questions about the utility of any models developed in this process. I've have experience reviewing building energy performance, including simulation models. A few thoughts below. Would be interested in other perspectives. \n1. The challenge goal was to accurately predict energy use that under a hypothetical scenario would have occurred if not for the installation of energy savings measures. The difference in observed post-installation energy use and the predicted energy use would be deemed the realized energy savings. Energy Service Contacts, were a building manager only pays for realized energy savings, carefully consider how energy savings will be determined and documented. More accurate ways to establish \"what if\" energy use assuming no energy savings measures were installed would be useful. \n2. Accurately predicting \"annual\" energy use would be useful as it is strongly correlated with annual energy costs. There may be errors in predicting each month, day or hour but as long as the negatives offset the positives then the sum over a year can be accurate. I had models predicting annual energy use for 2017 & 2018 that were very close to the actual energy use. But, these models only ran in the middle of pack using the RMSLE. I wonder if using the RMSLE results in higher error when predicting annual energy use as opposed hourly energy use. The RMSLE would bias toward lower energy consumption hours, while a simple mean error would bias toward the higher energy use hours and I would think be more consistent with a simple annual sum for energy. \n3. I did not check on how well my models did for peak periods, i.e. the highest energy use during each month, or the annual peak. I also suspect a RMSLE fitted model might be more likely to under-predict the energy use in these periods as opposed to say using a simple mean error. These peaks which can set distribution , transmission or capacity costs would be important to get right.\n4. In real life the first thing a modeler would do I think would be to clean up the data: establish units (kBTU, therms, kWh..) and remove any data that doesn't pass the laugh test. In my case I found data cleaning of limited value given (1) the Private data set also had data issues, so cleaning up the Train data set could actually result in a worse score, and (2) I chose not to view or use the leaked data as doing so would seem to violate the spirit of the challenge. Seems like the goal should be to take sloppy data and make clean predictions with it -- which would suggest the Private data set should be cleaned up and made as close as possible to actual energy use. Taking \"garbage\" data and intentionally making \"garbage\" predictions doesn't seem like it would be very useful. A model that could take garbage data and make \"clean\" predictions, on the other hand, could be very useful. \n\nGood luck with your Private scoring" }, "source": "meta" }