![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1557202%2Feed32e74ea7d94ccd3163c6e514a37e6%2Fsite0buildingElectricity.png?generation=1571920422092472&alt=media\")
according my analysis\n```\ntotal number of readings for meter type electricity 12060910\ntotal number of non-zero readings for meter type electricity 11530741\ntotal number of readings for meter type chilledwater 4182440\ntotal number of non-zero readings for meter type chilledwater 3525936\ntotal number of readings for meter type steam 2708713\ntotal number of non-zero readings for meter type steam 2361753\ntotal number of readings for meter type hotwater 1264037\ntotal number of non-zero readings for meter type hotwater 923694
\n\n```\nso what does it means?\nis is zero energy consumption or missing data?\nasking because using 0.0 kwh electricity should be very rare.
", "messages": [ { "id": 650811, "postDate": "2019-10-16T17:52:55.360Z", "content": "according my analysis\n```\ntotal number of readings for meter type electricity 12060910\ntotal number of non-zero readings for meter type electricity 11530741\ntotal number of readings for meter type chilledwater 4182440\ntotal number of non-zero readings for meter type chilledwater 3525936\ntotal number of readings for meter type steam 2708713\ntotal number of non-zero readings for meter type steam 2361753\ntotal number of readings for meter type hotwater 1264037\ntotal number of non-zero readings for meter type hotwater 923694
\n\n```\nso what does it means?\nis is zero energy consumption or missing data?\nasking because using 0.0 kwh electricity should be very rare.
", "rawMarkdown": "according my analysis\n```\ntotal number of readings for meter type electricity 12060910\ntotal number of non-zero readings for meter type electricity 11530741\ntotal number of readings for meter type chilledwater 4182440\ntotal number of non-zero readings for meter type chilledwater 3525936\ntotal number of readings for meter type steam 2708713\ntotal number of non-zero readings for meter type steam 2361753\ntotal number of readings for meter type hotwater 1264037\ntotal number of non-zero readings for meter type hotwater 923694\n\n```\nso what does it means?\nis is zero energy consumption or missing data?\nasking because using 0.0 kwh electricity should be very rare.\n", "votes": 58 }, { "id": 655306, "postDate": "2019-10-22T22:34:27.377Z", "content": "Let me explain something about building meters - especially for chilled water, steam, and hot water. Many of these are 'pulse' meters, meaning they don't record how much they consume at a given timestep interval. Rather, they record every time they surpass a threshold. For example, a chilled water meter may send a timestamp every time it hits 100 ton hrs. This is tricky, and the thresholds are not necessarily well set up for a building. A building may use 100 ton hrs of cooling in 45 minutes, or once every 6 months. In this case, '0.0' values aren't null or zero consumption - they are a time when the pulse meter didn't record.
\n\nPer Dmitriy's comments, this is going to very difficult to pull out. Not only do you have to estimate energy consumption at a given hour, but you have to estimate when the next pulse will occur to record the total for the test meter submission.
", "rawMarkdown": "Let me explain something about building meters - especially for chilled water, steam, and hot water. Many of these are 'pulse' meters, meaning they don't record how much they consume at a given timestep interval. Rather, they record every time they surpass a threshold. For example, a chilled water meter may send a timestamp every time it hits 100 ton hrs. This is tricky, and the thresholds are not necessarily well set up for a building. A building may use 100 ton hrs of cooling in 45 minutes, or once every 6 months. In this case, '0.0' values aren't null or zero consumption - they are a time when the pulse meter didn't record.\n\nPer Dmitriy's comments, this is going to very difficult to pull out. Not only do you have to estimate energy consumption at a given hour, but you have to estimate when the next pulse will occur to record the total for the test meter submission.", "votes": 35, "replies": [ { "id": 661223, "postDate": "2019-10-30T03:16:08.380Z", "content": "@mdahlhausen Thanks for the domain knowledge, super helpful.
\n\nJust to confirm I'm understanding this right: a meter reading of, say 3.0 means that the meter has exceeded its threshold 3 times since the last interval (i.e. in this case 3 times in the last hour)?
The value is: Energy consumption in kWh (or equivalent).
\n\nThe software normally convert the pulses to the energy unit.
", "rawMarkdown": "The value is: Energy consumption in kWh (or equivalent).\n\nThe software normally convert the pulses to the energy unit." } ] }, { "id": 656588, "postDate": "2019-10-24T12:43:34.107Z", "content": "As others have seen, most buildings in site 0 has either zero or small readings for the first 5 months. Because of the similar pattern and the same geographical site, perhaps the data is collected by the same party in a similar way.
\n\nSite 0, meter 0 (electricity), all buildings meter_reading:\n
Thank you for pointing out.
\n\nI removed all those values (electricity of site_id 0 until May 20) from training data, and score improved a lot!\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type
", "rawMarkdown": "Thank you for pointing out.\n\nI removed all those values (electricity of site_id 0 until May 20) from training data, and score improved a lot!\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type", "votes": 11 }, { "id": 659618, "postDate": "2019-10-28T01:38:25.383Z", "content": "I found the same insight in my notebook.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645169%2F91917080a0509b1d823bcb2e43998dec%2FCaptura%20de%20pantalla%20de%202019-10-28%2002-33-34.png?generation=1572226486608876&alt=media\")
You can see this cluster of buildings down left.
\n\nThere are not the only buildings that have the 0 phenomenon at the very same timestamps. There are at least big four other clusters of buildings that share the same 0 patterns.
", "rawMarkdown": "I found the same insight in my [notebook](https://www.kaggle.com/juanmah/ashrae-zeros).\n\n\n\nYou can see this cluster of buildings down left.\n\nThere are not the only buildings that have the 0 phenomenon at the very same timestamps. There are at least big four other clusters of buildings that share the same 0 patterns.", "votes": 17 }, { "id": 659882, "postDate": "2019-10-28T11:35:13.357Z", "content": "@corochann My score did too. It does take some guess work to decide what is pattern to fit and what is bad data to discard :)
", "rawMarkdown": "@corochann My score did too. It does take some guess work to decide what is pattern to fit and what is bad data to discard :)\n", "votes": 3 }, { "id": 659884, "postDate": "2019-10-28T11:44:42.707Z", "content": "@juanmah Yours is a very illustrative plot! Much more thorough than my eye ball findings. Will go to investigate those yellow blocks. Thanks!
", "rawMarkdown": "@juanmah Yours is a very illustrative plot! Much more thorough than my eye ball findings. Will go to investigate those yellow blocks. Thanks!", "votes": 2 }, { "id": 659911, "postDate": "2019-10-28T12:33:42.660Z", "content": "@juanmah That visualization is really nice to find out weird patterns!!
", "rawMarkdown": "@juanmah That visualization is really nice to find out weird patterns!!", "votes": 1 }, { "id": 661740, "postDate": "2019-10-30T16:50:07.273Z", "content": "@barnwellguy Terrific visualization! Thanks!
", "rawMarkdown": "@barnwellguy Terrific visualization! Thanks! ", "votes": 1 } ] }, { "id": 651729, "postDate": "2019-10-17T21:23:44.760Z", "content": "If 0.0 means that a reading wasn't taken, and then the next positive reading \"catches up\" on the incremental changes since the last positive reading, then it's actually a problem with the RMSLE metric.
\n\nWith a mean-preserving metric, this bunching would just create noise, but on average you would still get the right incremental value; if you were supposed to have readings of \"100\" and \"200\" for the two hours, but you get \"0\" and \"300\" instead, either way you get 150 on average. However, (log(101) + log(201)) / 2 amounts to a different number than (log(1) + log(301)) / 2; it's 4.96 vs 2.85, in this example. The implication is that your prediction of the building's energy use should take into account the likelihood of the meter not updating every hour, implicitly or explicitly.
\n\nIs the prediction for the building low because it's energy efficient, or because its meters get stuck more often? The organizers won't know, so there will be a big question mark over any conclusion that they reach from the models trained to optimize the RMSLE metric. This is the kind of nonsense that happens when you use a kludge metric, and don't think through what side effects you're introducing by using it.
", "rawMarkdown": "If 0.0 means that a reading wasn't taken, and then the next positive reading \"catches up\" on the incremental changes since the last positive reading, then it's actually a problem with the RMSLE metric. \n\nWith a mean-preserving metric, this bunching would just create noise, but on average you would still get the right incremental value; if you were supposed to have readings of \"100\" and \"200\" for the two hours, but you get \"0\" and \"300\" instead, either way you get 150 on average. However, (log(101) + log(201)) / 2 amounts to a different number than (log(1) + log(301)) / 2; it's 4.96 vs 2.85, in this example. The implication is that your prediction of the building's energy use should take into account the likelihood of the meter not updating every hour, implicitly or explicitly. \n\nIs the prediction for the building low because it's energy efficient, or because its meters get stuck more often? The organizers won't know, so there will be a big question mark over any conclusion that they reach from the models trained to optimize the RMSLE metric. This is the kind of nonsense that happens when you use a kludge metric, and don't think through what side effects you're introducing by using it.", "votes": 20, "replies": [ { "id": 651904, "postDate": "2019-10-18T04:11:32.633Z", "content": "If they ignore 0-values when computing the scores, it'd take care of the problem.
", "rawMarkdown": "If they ignore 0-values when computing the scores, it'd take care of the problem." }, { "id": 652135, "postDate": "2019-10-18T12:01:34.687Z", "content": "Ignoring 0-values won't take care of the problem, it will make the problem different. Instead of under-estimating the energy use for a building with a stuttering meter, it will over-estimate it. Another problem is that we don't actually know whether zeroes are ignored or not, so how are we supposed to build the model intelligently?
\n\nIf I were to guess, I would say that the big CV-LB differences are due to either different behavior of zeroes in the test set, or different treatment of them. This is why I'm staying out of this competition: the bad choice of metric is going to turn it into nonsense. You're going to get a much bigger bang for the buck figuring out how to deal with zeroes than you will figuring out how to build an intelligent model.
", "rawMarkdown": "Ignoring 0-values won't take care of the problem, it will make the problem different. Instead of under-estimating the energy use for a building with a stuttering meter, it will over-estimate it. Another problem is that we don't actually know whether zeroes are ignored or not, so how are we supposed to build the model intelligently? \n\nIf I were to guess, I would say that the big CV-LB differences are due to either different behavior of zeroes in the test set, or different treatment of them. This is why I'm staying out of this competition: the bad choice of metric is going to turn it into nonsense. You're going to get a much bigger bang for the buck figuring out how to deal with zeroes than you will figuring out how to build an intelligent model.", "votes": 16 }, { "id": 652154, "postDate": "2019-10-18T12:31:21.890Z", "content": "wise words @dmitriyguller thank you for share your knowledge :)
", "rawMarkdown": "wise words @dmitriyguller thank you for share your knowledge :)", "votes": -1 }, { "id": 652233, "postDate": "2019-10-18T14:48:47.987Z", "content": "The data description page does say
\n\n\n\n\nThere are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.
\n
but we don't know if zeros are treated as gaps.
", "rawMarkdown": "The data description page does say\n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\n\nbut we don't know if zeros are treated as gaps.", "votes": 2 }, { "id": 652367, "postDate": "2019-10-18T18:41:36.697Z", "content": "My guess is that zeros are not treated as gaps, because in the training dataset, we do have gaps in observations, while the test dataset has an entry for every timestamp (to avoid leakage, I presume). The observations that were added to complete the grid in the test set are probably going to be the only ones excluded from scoring.
", "rawMarkdown": "My guess is that zeros are not treated as gaps, because in the training dataset, we do have gaps in observations, while the test dataset has an entry for every timestamp (to avoid leakage, I presume). The observations that were added to complete the grid in the test set are probably going to be the only ones excluded from scoring. ", "votes": 5 }, { "id": 657712, "postDate": "2019-10-25T11:36:32.250Z", "content": "I think removing zeros (i.e. \"gaps\") solves the issue as they are not scored in the test set, one should not score in the train/validation either. \nConcerning the large CV-LB gap, I guess it depends on your CV schema ;)
", "rawMarkdown": "I think removing zeros (i.e. \"gaps\") solves the issue as they are not scored in the test set, one should not score in the train/validation either. \nConcerning the large CV-LB gap, I guess it depends on your CV schema ;)", "votes": 1 }, { "id": 657879, "postDate": "2019-10-25T13:48:54.907Z", "content": "\n\n\nI think removing zeros (i.e. \"gaps\") solves the issue as they are not scored in the test set, one should not score in the train/validation either.
\n
Do you have evidence of this? My assumption is the same as Dmitriy's -- that 0s are handled differently than explicit gaps and would be scored in the test data. The competition details / sponsors haven't said anything to indicate that the test data is processed or represented differently than the train data, with the only exception being the artificial inclusion of timestamps that are missing to mask the true gaps. It seems like part of the problem is that meaningful 0s can't always be cleanly disambiguated from measurement gaps, so it doesn't seem possible to me that just excluding them would be the way to go.
", "rawMarkdown": "> I think removing zeros (i.e. \"gaps\") solves the issue as they are not scored in the test set, one should not score in the train/validation either.\n\nDo you have evidence of this? My assumption is the same as Dmitriy's -- that 0s are handled differently than explicit gaps and would be scored in the test data. The competition details / sponsors haven't said anything to indicate that the test data is processed or represented differently than the train data, with the only exception being the artificial inclusion of timestamps that are missing to mask the true gaps. It seems like part of the problem is that meaningful 0s can't always be cleanly disambiguated from measurement gaps, so it doesn't seem possible to me that just excluding them would be the way to go. ", "votes": 2 }, { "id": 657976, "postDate": "2019-10-25T14:59:53.960Z", "content": "\"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"\nI really think they could not have been more clear than this :)
\n\nGaps are not simply zeros. Gaps are sequences of zeros, like the one that someone nicely plotted above. \nFind a logic that identifies these gaps and remove them from your training data. \nGood luck and happy kaggling!
", "rawMarkdown": "\"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"\nI really think they could not have been more clear than this :)\n\nGaps are not simply zeros. Gaps are sequences of zeros, like the one that someone nicely plotted above. \nFind a logic that identifies these gaps and remove them from your training data. \nGood luck and happy kaggling!", "votes": 2 }, { "id": 658009, "postDate": "2019-10-25T15:29:11.057Z", "content": "I agree with you that sequences of zeros may be reasonably interpreted as gaps, but there are plenty of explicit temporal gaps in the timestamps. For just one example, look at building id 1012, which has an explicit gap in August. It's possible that explicit gaps and sequences of zeroes are actually the same or similar things, but they are definitely represented differently in the training data. If we assume that the test data is processed/represented the same as train, then we could expect there to be sequences of zeros that aren't explicit gaps and therefore will be scored if we take that quote literally.
\n\nI think that quote actually isn't very clear but rather leaves ambiguity because we're not told how to interpret zeroes. If those zero sequences are actual gaps, why wouldn't the sponsors just remove them? Or maybe to reframe the question, if those zero sequences weren't processed out of train, why should we expect them to be processed out of test for scoring purposes? My occam's razor view is that train and test were processed exactly the same way and then missing timestamps were added to test, and not that the sponsors have special processing code that will clean zero sequences from test that they didn't bother to apply to train as well if they know those data points are problematic.
", "rawMarkdown": "I agree with you that sequences of zeros may be reasonably interpreted as gaps, but there are plenty of explicit temporal gaps in the timestamps. For just one example, look at building id 1012, which has an explicit gap in August. It's possible that explicit gaps and sequences of zeroes are actually the same or similar things, but they are definitely represented differently in the training data. If we assume that the test data is processed/represented the same as train, then we could expect there to be sequences of zeros that aren't explicit gaps and therefore will be scored if we take that quote literally. \n\nI think that quote actually isn't very clear but rather leaves ambiguity because we're not told how to interpret zeroes. If those zero sequences are actual gaps, why wouldn't the sponsors just remove them? Or maybe to reframe the question, if those zero sequences weren't processed out of train, why should we expect them to be processed out of test for scoring purposes? My occam's razor view is that train and test were processed exactly the same way and then missing timestamps were added to test, and not that the sponsors have special processing code that will clean zero sequences from test that they didn't bother to apply to train as well if they know those data points are problematic.", "votes": 1 }, { "id": 658019, "postDate": "2019-10-25T15:40:10.843Z", "content": "If the gaps are really missing records and not zeros, why bother specifying that they are not scored in the test set? If there is no data it's pretty clear that it will not be scored...\nAlso, from a logical perspective, it makes a lot of sense to exclude the zero-induced gaps from scoring, because they are clearly not predictable by any means.
\n\nHowever, I appreciate your point about the ambiguity and I agree that they could have cleaned the train set too, if my hypothesis was true.
", "rawMarkdown": "If the gaps are really missing records and not zeros, why bother specifying that they are not scored in the test set? If there is no data it's pretty clear that it will not be scored...\nAlso, from a logical perspective, it makes a lot of sense to exclude the zero-induced gaps from scoring, because they are clearly not predictable by any means.\n\nHowever, I appreciate your point about the ambiguity and I agree that they could have cleaned the train set too, if my hypothesis was true." }, { "id": 658111, "postDate": "2019-10-25T17:09:38.290Z", "content": "\n\n\nIf the gaps are really missing records and not zeros, why bother specifying that they are not scored in the test set? If there is no data it's pretty clear that it will not be scored…
\n
I think they specify just because it's non-obvious once they fill in the missing timestamps and include weather data to obscure the gaps. They didn't want to just show no data for the missing timestamps in test because that would probably run the risk of being a major data leak, but they wanted to make it clear that they weren't going to impute those target values or something like that.
\n\n\n\n\nAlso, from a logical perspective, it makes a lot of sense to exclude the zero-induced gaps from scoring, because they are clearly not predictable by any means.
\n
I agree, scoring our models against obvious measurement errors that don't reflect real energy usage doesn't seem to make much sense, but unfortunately that doesn't guarantee that it won't happen. I actually do wonder if there's any predictability in the zero gaps -- could it be triggered by extreme weather conditions or be a recurring pattern for certain buildings? Could be an interesting thing to look into, but even if errors are systematic in some way I highly doubt we have enough data (or the right features) for a model to properly anticipate them.
", "rawMarkdown": "> If the gaps are really missing records and not zeros, why bother specifying that they are not scored in the test set? If there is no data it's pretty clear that it will not be scored…\n\nI think they specify just because it's non-obvious once they fill in the missing timestamps and include weather data to obscure the gaps. They didn't want to just show no data for the missing timestamps in test because that would probably run the risk of being a major data leak, but they wanted to make it clear that they weren't going to impute those target values or something like that.\n\n> Also, from a logical perspective, it makes a lot of sense to exclude the zero-induced gaps from scoring, because they are clearly not predictable by any means.\n\nI agree, scoring our models against obvious measurement errors that don't reflect real energy usage doesn't seem to make much sense, but unfortunately that doesn't guarantee that it won't happen. I actually do wonder if there's any predictability in the zero gaps -- could it be triggered by extreme weather conditions or be a recurring pattern for certain buildings? Could be an interesting thing to look into, but even if errors are systematic in some way I highly doubt we have enough data (or the right features) for a model to properly anticipate them.\n\n" }, { "id": 658198, "postDate": "2019-10-25T19:09:46.250Z", "content": "Maybe a good point would be removing only those zero values that are at the beginig of the ts, and leavo those ones between two blocks of meter_readings > 0
", "rawMarkdown": "Maybe a good point would be removing only those zero values that are at the beginig of the ts, and leavo those ones between two blocks of meter_readings > 0\n" }, { "id": 660266, "postDate": "2019-10-29T00:22:04.030Z", "rawMarkdown": "", "isDeleted": true }, { "id": 660268, "postDate": "2019-10-29T00:22:51.590Z", "rawMarkdown": "", "isDeleted": true }, { "id": 663612, "postDate": "2019-11-02T12:22:25.040Z", "content": "I get a .4 difference in score with(out) a zero discriminator, so I second your notion, but I'll \"compete\"
", "rawMarkdown": "I get a .4 difference in score with(out) a zero discriminator, so I second your notion, but I'll \"compete\"" } ] }, { "id": 650912, "postDate": "2019-10-16T20:35:21.763Z", "content": "This data set is more tricky than it appears. There are green kernels that don't do any eda at the moment, but that will change soon. Many meters are missing entire months worth of data (forget 0.0 or np.nan values—I'm talking about no timestamp or sample recorded at all). Figuring out how to deal with these will be a game changer.
", "rawMarkdown": "This data set is more tricky than it appears. There are green kernels that don't do any eda at the moment, but that will change soon. Many meters are missing entire months worth of data (forget 0.0 or np.nan values—I'm talking about no timestamp or sample recorded at all). Figuring out how to deal with these will be a game changer.", "votes": 9 }, { "id": 651352, "postDate": "2019-10-17T11:45:02.727Z", "content": "@authman @tiannnn @dhananjay3 @gunesevitan \nat my kernel Starter ⚡ Great Energy Predictor \nI explain this.
\n\nmeter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.
Why is meter_reading 0?
\n\nIf you check the builing 0 (building_id = 0) you will see:\n- there is no information between January - May 2016 (months 1 to 5). meter_reading==0.\n- From June 2016 (month=6) we have information!
reason? \nIn my opinion they started to measure the consume of that specific building in June 2016. \nIn order to complete the database from 2016, they filled the other months with 0.\nNote that they can do it because the meteorological variables aren't a problem, you can access to historical data from external sources, and other variables about the building are constant like the year_built, how big is the building, floors etc... And that's why they could create the database and include new buildings since 2016 :)
\n\nSo... would be amazing to know when they started to measure each building... I'll post more insights and cool information.\nMore information at my kernel, I'll keep updating everyday ;)
", "rawMarkdown": "@authman @tiannnn @dhananjay3 @gunesevitan \nat my kernel [Starter ⚡ Great Energy Predictor ](https://www.kaggle.com/jesucristo/starter-great-energy-predictor)\nI explain this.\n\n```meter_reading``` - The target variable. **Energy consumption in kWh (or equivalent).** Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n\nWhy is meter_reading 0?\n\n- Those buildings don't consume energy (weird).\n- The stuff they use for measuring was broken.\n- Missing data! See the next example with the building 0. Spoiler: They started to measure Building 0 at June 2016! (month=6)\n\nIf you check the builing ```0``` (building_id = 0) you will see:\n- there is no information between January - May 2016 (months 1 to 5). meter_reading==0.\n- From June 2016 (month=6) we have information! \n\n**reason?** \nIn my opinion they started to measure the consume of that specific building in June 2016. \nIn order to complete the database from 2016, they filled the other months with 0.\nNote that they can do it because the *meteorological variables* aren't a problem, you can access to historical data from external sources, and other *variables about the building* are constant like the year_built, how big is the building, floors etc... And that's why they could create the database and include new buildings since 2016 :)\n\nSo... would be amazing to know when they started to measure each building... I'll post more insights and cool information.\nMore information at my kernel, I'll keep updating everyday ;)", "votes": 6, "replies": [ { "id": 651382, "postDate": "2019-10-17T12:42:57.667Z", "content": "Interesting. In your kernel we can see that some(e.g. 1099, 778) building have zero reading in between or after non-zero readings than this explanation is not valid in these cases. anyway we just want to know if to treat these values as missing values or not.
", "rawMarkdown": "Interesting. In your kernel we can see that some(e.g. 1099, 778) building have zero reading in between or after non-zero readings than this explanation is not valid in these cases. anyway we just want to know if to treat these values as missing values or not.", "votes": 2 }, { "id": 651387, "postDate": "2019-10-17T12:47:00.183Z", "content": "good point I didn't realize about that, maybe is a mix of both: missing values and errors :)
", "rawMarkdown": "good point I didn't realize about that, maybe is a mix of both: missing values and errors :)", "votes": 2 }, { "id": 651579, "postDate": "2019-10-17T16:19:18.763Z", "content": "Yeah, what I was saying is that there are instances where we only have e.g. <500 samples for a building/meter. In the worst case scenario, there's a particular building-meter pair where we literally only have 2016 december transactions. The rest of the year is completely missing, as in no 0-fill, no np.nan, just no data period. There's more to this story than meets the eye ;-).
", "rawMarkdown": "Yeah, what I was saying is that there are instances where we only have e.g. <500 samples for a building/meter. In the worst case scenario, there's a particular building-meter pair where we literally only have 2016 december transactions. The rest of the year is completely missing, as in no 0-fill, no np.nan, just no data period. There's more to this story than meets the eye ;-).", "votes": 1 } ] }, { "id": 652352, "postDate": "2019-10-18T18:07:57.553Z", "content": "0 gaps can be meaningful as well in my opinion. e.g. vacation in school building.
", "rawMarkdown": "0 gaps can be meaningful as well in my opinion. e.g. vacation in school building. ", "votes": 4 }, { "id": 651544, "postDate": "2019-10-17T15:29:42.560Z", "content": "IMHO my hypothesis is:\n- Zero readings mean zero energy consumption, but meter installed and working\n- Missing readings could mean a number of things: no meter installed, meter broke down, meter installed and running yet data unavailable...
\n\nQuick visualization to support my point:\nhttps://www.kaggle.com/ganfear/missing-data-and-zeros-visualized?scriptVersionId=22110712
", "rawMarkdown": "IMHO my hypothesis is:\n- Zero readings mean zero energy consumption, but meter installed and working\n- Missing readings could mean a number of things: no meter installed, meter broke down, meter installed and running yet data unavailable...\n\nQuick visualization to support my point:\nhttps://www.kaggle.com/ganfear/missing-data-and-zeros-visualized?scriptVersionId=22110712", "votes": 1, "replies": [ { "id": 651705, "postDate": "2019-10-17T20:29:15.233Z", "content": "as @ganfar mentionned, if you look at your house meters, the reading is on cumulative value not instantaneous value and a meter is almost never reset. For this competition, they most likely took substracted the previous value to get the consumption between two timestamp.
", "rawMarkdown": "as @ganfar mentionned, if you look at your house meters, the reading is on cumulative value not instantaneous value and a meter is almost never reset. For this competition, they most likely took substracted the previous value to get the consumption between two timestamp.", "votes": 1 }, { "id": 655236, "postDate": "2019-10-22T20:59:20.143Z", "content": "In this case it's necessary handle missing values and 0 gaps.✔️\nBut Which are the appropriate approaches to overcoming this problem?
", "rawMarkdown": "In this case it's necessary handle missing values and 0 gaps.✔️\nBut Which are the appropriate approaches to overcoming this problem?" } ] }, { "id": 651480, "postDate": "2019-10-17T14:28:05.623Z", "content": "Are the zeros the gap?
\n\nFrom the data page:\n'There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.'
", "rawMarkdown": "Are the zeros the gap?\n\nFrom the data page:\n'There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.'", "votes": 1, "replies": [ { "id": 651898, "postDate": "2019-10-18T04:03:32.640Z", "content": "gaps most likely means absence of some timestamps.
\nbecause we don't have all timestamps for each unique meter. can be seen in my EDA.
I am not sure if this is taken from an actual dataset. But in practise, sensors can drift or malfunction. When that happens, it is possible for the reading to be different from the ground truth.
", "rawMarkdown": "I am not sure if this is taken from an actual dataset. But in practise, sensors can drift or malfunction. When that happens, it is possible for the reading to be different from the ground truth.", "votes": 1 }, { "id": 675811, "postDate": "2019-11-18T15:37:27.417Z", "content": "Hello there,
\n\nWhen a (pulse) meter is mounted and configured for a certain measurement point, his sensitivity is chosen accordingly with the site consumption. That is, it should properly record each settlement interval (one hour in this case; the power is auctioned on an hourly base, or even quarterly in some countries). For example, if you want to weigh o piece of bread is not the best practice to use a meter that weighs trains.\nThere are very rare cases when a meter behaves like the ones with long 0 intervals. The site may be disconnected from the grid or the measurements are wrongly collected.
", "rawMarkdown": "Hello there,\n\nWhen a (pulse) meter is mounted and configured for a certain measurement point, his **sensitivity** is chosen accordingly with the site consumption. That is, it should properly record each settlement interval (one hour in this case; the power is auctioned on an hourly base, or even quarterly in some countries). For example, if you want to weigh o piece of bread is not the best practice to use a meter that weighs trains.\nThere are very rare cases when a meter behaves like the ones with long 0 intervals. The site may be disconnected from the grid or the measurements are wrongly collected.", "votes": 2 }, { "id": 650828, "postDate": "2019-10-16T18:15:09.437Z", "content": "I think 0.0 probably means that building doesn't have that meter type. If the building has it, there has to be some energy consumption since it is real world data.
", "rawMarkdown": "I think 0.0 probably means that building doesn't have that meter type. If the building has it, there has to be some energy consumption since it is real world data.", "replies": [ { "id": 650839, "postDate": "2019-10-16T18:30:44.560Z", "content": "sometimes zero value occurs in between non zero values so ..\nplus if a building does not have some meter type then that combination is not in dataset.
", "rawMarkdown": "sometimes zero value occurs in between non zero values so ..\nplus if a building does not have some meter type then that combination is not in dataset.", "votes": 2 }, { "id": 655103, "postDate": "2019-10-22T17:33:12.593Z", "content": "I think that it could be that the zeros in the beginning of the measure are meters not yet installed.
\n\nAnd the zeros in between non zeros are broken meters that they replaced, so that after few months they installed new ones and it came back to meter.
\n\nIs it possible?
", "rawMarkdown": "I think that it could be that the zeros in the beginning of the measure are meters not yet installed.\n\nAnd the zeros in between non zeros are broken meters that they replaced, so that after few months they installed new ones and it came back to meter.\n\nIs it possible?" } ] }, { "id": 652496, "postDate": "2019-10-18T23:41:04.973Z", "content": "USA DOE likely has guidance on how to fill out survey's that they have conducted. Since it looked like from my brief exam they wanted a count of number of PC is use, they likely had some specifics on what to do about meter readings.
\n\nIf we can find that guidance that may give us a better understanding of the data collection process.
", "rawMarkdown": "USA DOE likely has guidance on how to fill out survey's that they have conducted. Since it looked like from my brief exam they wanted a count of number of PC is use, they likely had some specifics on what to do about meter readings.\n\nIf we can find that guidance that may give us a better understanding of the data collection process." }, { "id": 656587, "postDate": "2019-10-24T12:43:31.593Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 656593, "postDate": "2019-10-24T12:51:56.977Z", "content": "Perhaps it is a site 0 characteristic. I zeroed out Jan-May site-0 meter_reading submissions to mimic the 2016 training set behavior, the score is worse. The zeros perhaps does not repeat each year.
", "rawMarkdown": "Perhaps it is a site 0 characteristic. I zeroed out Jan-May site-0 meter_reading submissions to mimic the 2016 training set behavior, the score is worse. The zeros perhaps does not repeat each year.", "votes": 1 } ] } ], "comments": [ { "id": 655306, "author_name": "Matthew Dahlhausen", "author_url": "", "post_date": "2019-10-22T22:34:27.377000", "content": "Let me explain something about building meters - especially for chilled water, steam, and hot water. Many of these are 'pulse' meters, meaning they don't record how much they consume at a given timestep interval. Rather, they record every time they surpass a threshold. For example, a chilled water meter may send a timestamp every time it hits 100 ton hrs. This is tricky, and the thresholds are not necessarily well set up for a building. A building may use 100 ton hrs of cooling in 45 minutes, or once every 6 months. In this case, '0.0' values aren't null or zero consumption - they are a time when the pulse meter didn't record.
\n\nPer Dmitriy's comments, this is going to very difficult to pull out. Not only do you have to estimate energy consumption at a given hour, but you have to estimate when the next pulse will occur to record the total for the test meter submission.
", "votes": 35, "replies": [ { "id": 661223, "author_name": "Louka Ewington-Pitsos", "author_url": "", "post_date": "2019-10-30T03:16:08.380000", "content": "@mdahlhausen Thanks for the domain knowledge, super helpful.
\n\nJust to confirm I'm understanding this right: a meter reading of, say 3.0 means that the meter has exceeded its threshold 3 times since the last interval (i.e. in this case 3 times in the last hour)?
The value is: Energy consumption in kWh (or equivalent).
\n\nThe software normally convert the pulses to the energy unit.
", "votes": 0, "replies": [] } ] }, { "id": 656588, "author_name": "Buffalo Spdwy", "author_url": "", "post_date": "2019-10-24T12:43:34.107000", "content": "As others have seen, most buildings in site 0 has either zero or small readings for the first 5 months. Because of the similar pattern and the same geographical site, perhaps the data is collected by the same party in a similar way.
\n\nSite 0, meter 0 (electricity), all buildings meter_reading:\n
Thank you for pointing out.
\n\nI removed all those values (electricity of site_id 0 until May 20) from training data, and score improved a lot!\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type
", "votes": 11, "replies": [] }, { "id": 659618, "author_name": "Juanma Hernández", "author_url": "", "post_date": "2019-10-28T01:38:25.383000", "content": "I found the same insight in my notebook.
\n\n
You can see this cluster of buildings down left.
\n\nThere are not the only buildings that have the 0 phenomenon at the very same timestamps. There are at least big four other clusters of buildings that share the same 0 patterns.
", "votes": 17, "replies": [] }, { "id": 659882, "author_name": "Buffalo Spdwy", "author_url": "", "post_date": "2019-10-28T11:35:13.357000", "content": "@corochann My score did too. It does take some guess work to decide what is pattern to fit and what is bad data to discard :)
", "votes": 3, "replies": [] }, { "id": 659884, "author_name": "Buffalo Spdwy", "author_url": "", "post_date": "2019-10-28T11:44:42.707000", "content": "@juanmah Yours is a very illustrative plot! Much more thorough than my eye ball findings. Will go to investigate those yellow blocks. Thanks!
", "votes": 2, "replies": [] }, { "id": 659911, "author_name": "corochann", "author_url": "", "post_date": "2019-10-28T12:33:42.660000", "content": "@juanmah That visualization is really nice to find out weird patterns!!
", "votes": 1, "replies": [] }, { "id": 661740, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-30T16:50:07.273000", "content": "@barnwellguy Terrific visualization! Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 651729, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2019-10-17T21:23:44.760000", "content": "If 0.0 means that a reading wasn't taken, and then the next positive reading \"catches up\" on the incremental changes since the last positive reading, then it's actually a problem with the RMSLE metric.
\n\nWith a mean-preserving metric, this bunching would just create noise, but on average you would still get the right incremental value; if you were supposed to have readings of \"100\" and \"200\" for the two hours, but you get \"0\" and \"300\" instead, either way you get 150 on average. However, (log(101) + log(201)) / 2 amounts to a different number than (log(1) + log(301)) / 2; it's 4.96 vs 2.85, in this example. The implication is that your prediction of the building's energy use should take into account the likelihood of the meter not updating every hour, implicitly or explicitly.
\n\nIs the prediction for the building low because it's energy efficient, or because its meters get stuck more often? The organizers won't know, so there will be a big question mark over any conclusion that they reach from the models trained to optimize the RMSLE metric. This is the kind of nonsense that happens when you use a kludge metric, and don't think through what side effects you're introducing by using it.
", "votes": 20, "replies": [ { "id": 651904, "author_name": "عثمان", "author_url": "", "post_date": "2019-10-18T04:11:32.633000", "content": "If they ignore 0-values when computing the scores, it'd take care of the problem.
", "votes": 0, "replies": [] }, { "id": 652135, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2019-10-18T12:01:34.687000", "content": "Ignoring 0-values won't take care of the problem, it will make the problem different. Instead of under-estimating the energy use for a building with a stuttering meter, it will over-estimate it. Another problem is that we don't actually know whether zeroes are ignored or not, so how are we supposed to build the model intelligently?
\n\nIf I were to guess, I would say that the big CV-LB differences are due to either different behavior of zeroes in the test set, or different treatment of them. This is why I'm staying out of this competition: the bad choice of metric is going to turn it into nonsense. You're going to get a much bigger bang for the buck figuring out how to deal with zeroes than you will figuring out how to build an intelligent model.
", "votes": 16, "replies": [] }, { "id": 652154, "author_name": "Nanashi", "author_url": "", "post_date": "2019-10-18T12:31:21.890000", "content": "wise words @dmitriyguller thank you for share your knowledge :)
", "votes": -1, "replies": [] }, { "id": 652233, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2019-10-18T14:48:47.987000", "content": "The data description page does say
\n\n\n\n\nThere are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.
\n
but we don't know if zeros are treated as gaps.
", "votes": 2, "replies": [] }, { "id": 652367, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2019-10-18T18:41:36.697000", "content": "My guess is that zeros are not treated as gaps, because in the training dataset, we do have gaps in observations, while the test dataset has an entry for every timestamp (to avoid leakage, I presume). The observations that were added to complete the grid in the test set are probably going to be the only ones excluded from scoring.
", "votes": 5, "replies": [] }, { "id": 657712, "author_name": "bluetrain", "author_url": "", "post_date": "2019-10-25T11:36:32.250000", "content": "I think removing zeros (i.e. \"gaps\") solves the issue as they are not scored in the test set, one should not score in the train/validation either. \nConcerning the large CV-LB gap, I guess it depends on your CV schema ;)
", "votes": 1, "replies": [] }, { "id": 657879, "author_name": "Joe Eddy", "author_url": "", "post_date": "2019-10-25T13:48:54.907000", "content": "\n\n\nI think removing zeros (i.e. \"gaps\") solves the issue as they are not scored in the test set, one should not score in the train/validation either.
\n
Do you have evidence of this? My assumption is the same as Dmitriy's -- that 0s are handled differently than explicit gaps and would be scored in the test data. The competition details / sponsors haven't said anything to indicate that the test data is processed or represented differently than the train data, with the only exception being the artificial inclusion of timestamps that are missing to mask the true gaps. It seems like part of the problem is that meaningful 0s can't always be cleanly disambiguated from measurement gaps, so it doesn't seem possible to me that just excluding them would be the way to go.
", "votes": 2, "replies": [] }, { "id": 657976, "author_name": "bluetrain", "author_url": "", "post_date": "2019-10-25T14:59:53.960000", "content": "\"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"\nI really think they could not have been more clear than this :)
\n\nGaps are not simply zeros. Gaps are sequences of zeros, like the one that someone nicely plotted above. \nFind a logic that identifies these gaps and remove them from your training data. \nGood luck and happy kaggling!
", "votes": 2, "replies": [] }, { "id": 658009, "author_name": "Joe Eddy", "author_url": "", "post_date": "2019-10-25T15:29:11.057000", "content": "I agree with you that sequences of zeros may be reasonably interpreted as gaps, but there are plenty of explicit temporal gaps in the timestamps. For just one example, look at building id 1012, which has an explicit gap in August. It's possible that explicit gaps and sequences of zeroes are actually the same or similar things, but they are definitely represented differently in the training data. If we assume that the test data is processed/represented the same as train, then we could expect there to be sequences of zeros that aren't explicit gaps and therefore will be scored if we take that quote literally.
\n\nI think that quote actually isn't very clear but rather leaves ambiguity because we're not told how to interpret zeroes. If those zero sequences are actual gaps, why wouldn't the sponsors just remove them? Or maybe to reframe the question, if those zero sequences weren't processed out of train, why should we expect them to be processed out of test for scoring purposes? My occam's razor view is that train and test were processed exactly the same way and then missing timestamps were added to test, and not that the sponsors have special processing code that will clean zero sequences from test that they didn't bother to apply to train as well if they know those data points are problematic.
", "votes": 1, "replies": [] }, { "id": 658019, "author_name": "bluetrain", "author_url": "", "post_date": "2019-10-25T15:40:10.843000", "content": "If the gaps are really missing records and not zeros, why bother specifying that they are not scored in the test set? If there is no data it's pretty clear that it will not be scored...\nAlso, from a logical perspective, it makes a lot of sense to exclude the zero-induced gaps from scoring, because they are clearly not predictable by any means.
\n\nHowever, I appreciate your point about the ambiguity and I agree that they could have cleaned the train set too, if my hypothesis was true.
", "votes": 0, "replies": [] }, { "id": 658111, "author_name": "Joe Eddy", "author_url": "", "post_date": "2019-10-25T17:09:38.290000", "content": "\n\n\nIf the gaps are really missing records and not zeros, why bother specifying that they are not scored in the test set? If there is no data it's pretty clear that it will not be scored…
\n
I think they specify just because it's non-obvious once they fill in the missing timestamps and include weather data to obscure the gaps. They didn't want to just show no data for the missing timestamps in test because that would probably run the risk of being a major data leak, but they wanted to make it clear that they weren't going to impute those target values or something like that.
\n\n\n\n\nAlso, from a logical perspective, it makes a lot of sense to exclude the zero-induced gaps from scoring, because they are clearly not predictable by any means.
\n
I agree, scoring our models against obvious measurement errors that don't reflect real energy usage doesn't seem to make much sense, but unfortunately that doesn't guarantee that it won't happen. I actually do wonder if there's any predictability in the zero gaps -- could it be triggered by extreme weather conditions or be a recurring pattern for certain buildings? Could be an interesting thing to look into, but even if errors are systematic in some way I highly doubt we have enough data (or the right features) for a model to properly anticipate them.
", "votes": 0, "replies": [] }, { "id": 658198, "author_name": "EnricRovira", "author_url": "", "post_date": "2019-10-25T19:09:46.250000", "content": "Maybe a good point would be removing only those zero values that are at the beginig of the ts, and leavo those ones between two blocks of meter_readings > 0
", "votes": 0, "replies": [] }, { "id": 660266, "author_name": "", "author_url": "", "post_date": "2019-10-29T00:22:04.030000", "content": "", "votes": 0, "replies": [] }, { "id": 660268, "author_name": "", "author_url": "", "post_date": "2019-10-29T00:22:51.590000", "content": "", "votes": 0, "replies": [] }, { "id": 663612, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2019-11-02T12:22:25.040000", "content": "I get a .4 difference in score with(out) a zero discriminator, so I second your notion, but I'll \"compete\"
", "votes": 0, "replies": [] } ] }, { "id": 650912, "author_name": "عثمان", "author_url": "", "post_date": "2019-10-16T20:35:21.763000", "content": "This data set is more tricky than it appears. There are green kernels that don't do any eda at the moment, but that will change soon. Many meters are missing entire months worth of data (forget 0.0 or np.nan values—I'm talking about no timestamp or sample recorded at all). Figuring out how to deal with these will be a game changer.
", "votes": 9, "replies": [] }, { "id": 651352, "author_name": "Nanashi", "author_url": "", "post_date": "2019-10-17T11:45:02.727000", "content": "@authman @tiannnn @dhananjay3 @gunesevitan \nat my kernel Starter ⚡ Great Energy Predictor \nI explain this.
\n\nmeter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.
Why is meter_reading 0?
\n\nIf you check the builing 0 (building_id = 0) you will see:\n- there is no information between January - May 2016 (months 1 to 5). meter_reading==0.\n- From June 2016 (month=6) we have information!
reason? \nIn my opinion they started to measure the consume of that specific building in June 2016. \nIn order to complete the database from 2016, they filled the other months with 0.\nNote that they can do it because the meteorological variables aren't a problem, you can access to historical data from external sources, and other variables about the building are constant like the year_built, how big is the building, floors etc... And that's why they could create the database and include new buildings since 2016 :)
\n\nSo... would be amazing to know when they started to measure each building... I'll post more insights and cool information.\nMore information at my kernel, I'll keep updating everyday ;)
", "votes": 6, "replies": [ { "id": 651382, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2019-10-17T12:42:57.667000", "content": "Interesting. In your kernel we can see that some(e.g. 1099, 778) building have zero reading in between or after non-zero readings than this explanation is not valid in these cases. anyway we just want to know if to treat these values as missing values or not.
", "votes": 2, "replies": [] }, { "id": 651387, "author_name": "Nanashi", "author_url": "", "post_date": "2019-10-17T12:47:00.183000", "content": "good point I didn't realize about that, maybe is a mix of both: missing values and errors :)
", "votes": 2, "replies": [] }, { "id": 651579, "author_name": "عثمان", "author_url": "", "post_date": "2019-10-17T16:19:18.763000", "content": "Yeah, what I was saying is that there are instances where we only have e.g. <500 samples for a building/meter. In the worst case scenario, there's a particular building-meter pair where we literally only have 2016 december transactions. The rest of the year is completely missing, as in no 0-fill, no np.nan, just no data period. There's more to this story than meets the eye ;-).
", "votes": 1, "replies": [] } ] }, { "id": 652352, "author_name": "Bhavesh Ghodasara", "author_url": "", "post_date": "2019-10-18T18:07:57.553000", "content": "0 gaps can be meaningful as well in my opinion. e.g. vacation in school building.
", "votes": 4, "replies": [] }, { "id": 651544, "author_name": "Ganfear", "author_url": "", "post_date": "2019-10-17T15:29:42.560000", "content": "IMHO my hypothesis is:\n- Zero readings mean zero energy consumption, but meter installed and working\n- Missing readings could mean a number of things: no meter installed, meter broke down, meter installed and running yet data unavailable...
\n\nQuick visualization to support my point:\nhttps://www.kaggle.com/ganfear/missing-data-and-zeros-visualized?scriptVersionId=22110712
", "votes": 1, "replies": [ { "id": 651705, "author_name": "eagle4", "author_url": "", "post_date": "2019-10-17T20:29:15.233000", "content": "as @ganfar mentionned, if you look at your house meters, the reading is on cumulative value not instantaneous value and a meter is almost never reset. For this competition, they most likely took substracted the previous value to get the consumption between two timestamp.
", "votes": 1, "replies": [] }, { "id": 655236, "author_name": "Caesar Lupum", "author_url": "", "post_date": "2019-10-22T20:59:20.143000", "content": "In this case it's necessary handle missing values and 0 gaps.✔️\nBut Which are the appropriate approaches to overcoming this problem?
", "votes": 0, "replies": [] } ] }, { "id": 651480, "author_name": "FChmiel", "author_url": "", "post_date": "2019-10-17T14:28:05.623000", "content": "Are the zeros the gap?
\n\nFrom the data page:\n'There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.'
", "votes": 1, "replies": [ { "id": 651898, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2019-10-18T04:03:32.640000", "content": "gaps most likely means absence of some timestamps.
\nbecause we don't have all timestamps for each unique meter. can be seen in my EDA.
I am not sure if this is taken from an actual dataset. But in practise, sensors can drift or malfunction. When that happens, it is possible for the reading to be different from the ground truth.
", "votes": 1, "replies": [] }, { "id": 675811, "author_name": "Sorin", "author_url": "", "post_date": "2019-11-18T15:37:27.417000", "content": "Hello there,
\n\nWhen a (pulse) meter is mounted and configured for a certain measurement point, his sensitivity is chosen accordingly with the site consumption. That is, it should properly record each settlement interval (one hour in this case; the power is auctioned on an hourly base, or even quarterly in some countries). For example, if you want to weigh o piece of bread is not the best practice to use a meter that weighs trains.\nThere are very rare cases when a meter behaves like the ones with long 0 intervals. The site may be disconnected from the grid or the measurements are wrongly collected.
", "votes": 2, "replies": [] }, { "id": 650828, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2019-10-16T18:15:09.437000", "content": "I think 0.0 probably means that building doesn't have that meter type. If the building has it, there has to be some energy consumption since it is real world data.
", "votes": 0, "replies": [ { "id": 650839, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2019-10-16T18:30:44.560000", "content": "sometimes zero value occurs in between non zero values so ..\nplus if a building does not have some meter type then that combination is not in dataset.
", "votes": 2, "replies": [] }, { "id": 655103, "author_name": "luis cascelli", "author_url": "", "post_date": "2019-10-22T17:33:12.593000", "content": "I think that it could be that the zeros in the beginning of the measure are meters not yet installed.
\n\nAnd the zeros in between non zeros are broken meters that they replaced, so that after few months they installed new ones and it came back to meter.
\n\nIs it possible?
", "votes": 0, "replies": [] } ] }, { "id": 652496, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-10-18T23:41:04.973000", "content": "USA DOE likely has guidance on how to fill out survey's that they have conducted. Since it looked like from my brief exam they wanted a count of number of PC is use, they likely had some specifics on what to do about meter readings.
\n\nIf we can find that guidance that may give us a better understanding of the data collection process.
", "votes": 0, "replies": [] }, { "id": 656587, "author_name": "", "author_url": "", "post_date": "2019-10-24T12:43:31.593000", "content": "", "votes": 0, "replies": [ { "id": 656593, "author_name": "Buffalo Spdwy", "author_url": "", "post_date": "2019-10-24T12:51:56.977000", "content": "Perhaps it is a site 0 characteristic. I zeroed out Jan-May site-0 meter_reading submissions to mimic the 2016 training set behavior, the score is worse. The zeros perhaps does not repeat each year.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "650811": "according my analysis\n```\ntotal number of readings for meter type electricity 12060910\ntotal number of non-zero readings for meter type electricity 11530741\ntotal number of readings for meter type chilledwater 4182440\ntotal number of non-zero readings for meter type chilledwater 3525936\ntotal number of readings for meter type steam 2708713\ntotal number of non-zero readings for meter type steam 2361753\ntotal number of readings for meter type hotwater 1264037\ntotal number of non-zero readings for meter type hotwater 923694\n\n```\nso what does it means?\nis is zero energy consumption or missing data?\nasking because using 0.0 kwh electricity should be very rare.\n", "655306": "Let me explain something about building meters - especially for chilled water, steam, and hot water. Many of these are 'pulse' meters, meaning they don't record how much they consume at a given timestep interval. Rather, they record every time they surpass a threshold. For example, a chilled water meter may send a timestamp every time it hits 100 ton hrs. This is tricky, and the thresholds are not necessarily well set up for a building. A building may use 100 ton hrs of cooling in 45 minutes, or once every 6 months. In this case, '0.0' values aren't null or zero consumption - they are a time when the pulse meter didn't record.\n\nPer Dmitriy's comments, this is going to very difficult to pull out. Not only do you have to estimate energy consumption at a given hour, but you have to estimate when the next pulse will occur to record the total for the test meter submission.", "656588": "As others have seen, most buildings in site 0 has either zero or small readings for the first 5 months. Because of the similar pattern and the same geographical site, perhaps the data is collected by the same party in a similar way.\n\n- I think it is likely that at about May 2016, they either start to take measure or start to use a new way of taking measure of electricity usage, which will persist into the future. So, the zero/small readings will **not** repeat next year.\n- If the small/readings are caused by any yearly periodic systematic regulation, I think it is not likely that the small period is the beginning 5 months of the year, since season/temperature does not typically starts at the beginning of the year.\n\nSite 0, meter 0 (electricity), all buildings meter_reading:\n\n", "651729": "If 0.0 means that a reading wasn't taken, and then the next positive reading \"catches up\" on the incremental changes since the last positive reading, then it's actually a problem with the RMSLE metric. \n\nWith a mean-preserving metric, this bunching would just create noise, but on average you would still get the right incremental value; if you were supposed to have readings of \"100\" and \"200\" for the two hours, but you get \"0\" and \"300\" instead, either way you get 150 on average. However, (log(101) + log(201)) / 2 amounts to a different number than (log(1) + log(301)) / 2; it's 4.96 vs 2.85, in this example. The implication is that your prediction of the building's energy use should take into account the likelihood of the meter not updating every hour, implicitly or explicitly. \n\nIs the prediction for the building low because it's energy efficient, or because its meters get stuck more often? The organizers won't know, so there will be a big question mark over any conclusion that they reach from the models trained to optimize the RMSLE metric. This is the kind of nonsense that happens when you use a kludge metric, and don't think through what side effects you're introducing by using it.", "650912": "This data set is more tricky than it appears. There are green kernels that don't do any eda at the moment, but that will change soon. Many meters are missing entire months worth of data (forget 0.0 or np.nan values—I'm talking about no timestamp or sample recorded at all). Figuring out how to deal with these will be a game changer.", "651352": "@authman @tiannnn @dhananjay3 @gunesevitan \nat my kernel [Starter ⚡ Great Energy Predictor ](https://www.kaggle.com/jesucristo/starter-great-energy-predictor)\nI explain this.\n\n```meter_reading``` - The target variable. **Energy consumption in kWh (or equivalent).** Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n\nWhy is meter_reading 0?\n\n- Those buildings don't consume energy (weird).\n- The stuff they use for measuring was broken.\n- Missing data! See the next example with the building 0. Spoiler: They started to measure Building 0 at June 2016! (month=6)\n\nIf you check the builing ```0``` (building_id = 0) you will see:\n- there is no information between January - May 2016 (months 1 to 5). meter_reading==0.\n- From June 2016 (month=6) we have information! \n\n**reason?** \nIn my opinion they started to measure the consume of that specific building in June 2016. \nIn order to complete the database from 2016, they filled the other months with 0.\nNote that they can do it because the *meteorological variables* aren't a problem, you can access to historical data from external sources, and other *variables about the building* are constant like the year_built, how big is the building, floors etc... And that's why they could create the database and include new buildings since 2016 :)\n\nSo... would be amazing to know when they started to measure each building... I'll post more insights and cool information.\nMore information at my kernel, I'll keep updating everyday ;)", "652352": "0 gaps can be meaningful as well in my opinion. e.g. vacation in school building. ", "651544": "IMHO my hypothesis is:\n- Zero readings mean zero energy consumption, but meter installed and working\n- Missing readings could mean a number of things: no meter installed, meter broke down, meter installed and running yet data unavailable...\n\nQuick visualization to support my point:\nhttps://www.kaggle.com/ganfear/missing-data-and-zeros-visualized?scriptVersionId=22110712", "651480": "Are the zeros the gap?\n\nFrom the data page:\n'There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.'", "651232": "I am not sure if this is taken from an actual dataset. But in practise, sensors can drift or malfunction. When that happens, it is possible for the reading to be different from the ground truth.", "675811": "Hello there,\n\nWhen a (pulse) meter is mounted and configured for a certain measurement point, his **sensitivity** is chosen accordingly with the site consumption. That is, it should properly record each settlement interval (one hour in this case; the power is auctioned on an hourly base, or even quarterly in some countries). For example, if you want to weigh o piece of bread is not the best practice to use a meter that weighs trains.\nThere are very rare cases when a meter behaves like the ones with long 0 intervals. The site may be disconnected from the grid or the measurements are wrongly collected.", "650828": "I think 0.0 probably means that building doesn't have that meter type. If the building has it, there has to be some energy consumption since it is real world data.", "652496": "USA DOE likely has guidance on how to fill out survey's that they have conducted. Since it looked like from my brief exam they wanted a count of number of PC is use, they likely had some specifics on what to do about meter readings.\n\nIf we can find that guidance that may give us a better understanding of the data collection process.", "656587": "" } }