![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3406117%2F0c8bf43850e9f55e6a953dd4f0f38525%2Fplot-building_id-meter_reading.png?generation=1572358953678236&alt=media\")
Hello,
\n\nPlease look at these two plots.
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3406117%2F0e912b16e691788357e0b308e4baf209%2Fregression-building_id-meter_reading.png?generation=1572358958761349&alt=media\")
It seems that meter reading for building id 1099 is not correct if we compare to other buildings that exist with the same site id. Below is plot -
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3406117%2F9064e44cfb2d62c132b0e3530e8ec253%2Fbuildings-same-site-id.png?generation=1572359672024333&alt=media\")
So my question is do you recommend to remove these observations from the training data?
\n\nThanks in advance for your response!
", "messages": [ { "id": "660751", "postDate": "10/29/2019 14:45:13", "content": "Hello,
\n\nPlease look at these two plots.
\n\n
It seems that meter reading for building id 1099 is not correct if we compare to other buildings that exist with the same site id. Below is plot -
\n\n
So my question is do you recommend to remove these observations from the training data?
\n\nThanks in advance for your response!
", "rawMarkdown": "Hello, \n\nPlease look at these two plots.\n\n### Building vs Meter Reading Plot\n\n\n\n### Regression Plot between Building and Meter Reading\n\n\n\n### Building Details\n\n- building_id\t- 1099\n- site_id\t- 13\n- primary_use - Education\t\n- square_feet - 332884\t\n- year_built - NaN\t\n- floor_count - NaN\n\nIt seems that meter reading for building id 1099 is not correct if we compare to other buildings that exist with the same site id. Below is plot -\n\n\n\nSo my question is do you recommend to remove these observations from the training data? \n\n\nThanks in advance for your response!", "votes": null }, { "id": "660793", "postDate": "10/29/2019 16:07:27", "content": "It is an 'outlier' within buildings. Here is some EDA
", "rawMarkdown": "It is an 'outlier' within buildings. Here is some [EDA](https://www.kaggle.com/nroman/eda-for-ashrae)", "votes": null }, { "id": "660810", "postDate": "10/29/2019 16:31:23", "content": "remove them and make a submission to the leaderboard, let us know if your score increases
", "rawMarkdown": "remove them and make a submission to the leaderboard, let us know if your score increases", "votes": null }, { "id": "660877", "postDate": "10/29/2019 18:19:13", "content": "What is the exact value of meter_reading for this data point?
", "rawMarkdown": "What is the exact value of meter_reading for this data point?", "votes": null }, { "id": "661320", "postDate": "10/30/2019 06:02:57", "content": "Mean of meter_reading for building id 1099 is 13372.445088 kWh
", "rawMarkdown": "Mean of meter_reading for building id 1099 is 13372.445088 kWh", "votes": null }, { "id": "661438", "postDate": "10/30/2019 09:47:45", "content": "I removed them, the lb score doubled to 2.37 from 1.13😂 Idk if i did everything right...
", "rawMarkdown": "I removed them, the lb score doubled to 2.37 from 1.13😂 Idk if i did everything right...", "votes": null }, { "id": "661458", "postDate": "10/30/2019 10:29:14", "content": "Same experience here, after removing some consistent meter reading period (0 or something) my score increased from 1.10 to 1.18, I had thought that it would help.\nI think that there's also some strange meter readings like them in the test set for us to predict.
", "rawMarkdown": "Same experience here, after removing some consistent meter reading period (0 or something) my score increased from 1.10 to 1.18, I had thought that it would help.\nI think that there's also some strange meter readings like them in the test set for us to predict.", "votes": null }, { "id": "661464", "postDate": "10/30/2019 10:39:12", "content": "I believe that this training set has more outliers because when I plot to scatter between hourly temperature and hourly meter reading, I don't see any correlation.
", "rawMarkdown": "I believe that this training set has more outliers because when I plot to scatter between hourly temperature and hourly meter reading, I don't see any correlation.", "votes": null }, { "id": "661487", "postDate": "10/30/2019 11:19:06", "content": "since we only have one year data, it is hard to know if they are real outliers or 'systematic outliers'. I can't see how to deal with thess expect keeping LB probing🤒 ..
", "rawMarkdown": "since we only have one year data, it is hard to know if they are real outliers or 'systematic outliers'. I can't see how to deal with thess expect keeping LB probing🤒 ..", "votes": null }, { "id": "661501", "postDate": "10/30/2019 11:56:18", "content": "You can try and remove the data and see if it improves score. See the discussion on what zero meter reading means. The meter is likely a pulse meter.
", "rawMarkdown": "You can try and remove the data and see if it improves score. See the discussion on [what zero meter reading means](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113054#655306). The meter is likely a pulse meter.", "votes": null }, { "id": "661579", "postDate": "10/30/2019 13:30:30", "content": "@aitude That could be because the timestamps aren't aligned!
", "rawMarkdown": "aitude That could be because the timestamps aren't aligned!", "votes": null }, { "id": "661590", "postDate": "10/30/2019 13:45:31", "content": "@sdoria But if you separate training data by meter type then it makes sense. Below is a plot between chilledwater's energy consumption and air temperature.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3406117%2Fbb359e64896cfe4fba8967596a9828eb%2Fchilled-water.png?generation=1572443088830563&alt=media\")
True, I have also seen patterns after filtering the data.
", "rawMarkdown": "True, I have also seen patterns after filtering the data.", "votes": null }, { "id": "661611", "postDate": "10/30/2019 14:04:04", "content": "what about this building?
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1343910%2F1501412360d41f3bcc2270b3f0581f33%2Fbuild_1018_chilledwater.png?generation=1572443822686133&alt=media\")
Building Details
\n\nThis building has only 1 meter reading different from 0, which is 0.0334, it seems quite suspicious cause it looks like a closed build or maybe it has no record for meter chilledwater
\n\nBut looking at meter electricity (which is the most common):
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1343910%2F638516706ca24b2e4d88bf770a7ef3fa%2Fbuild_1018_electricity.png?generation=1572443883861672&alt=media\")
It looks like an active building.
\n\nWhat do you think about it? \nHow can we predict chilledwater consumpion without any (only 1) useful observation?
", "rawMarkdown": "what about this building?\n\n\n\nBuilding Details\n\n- building_id - 1018\n- site_id - 10\n- meter - chilledwater\n- primary_use - Education\n- square_feet - 186840\n- year_built - NaN\n- floor_count - 2.0\n\nThis building has only 1 meter reading different from 0, which is 0.0334, it seems quite suspicious cause it looks like a closed build or maybe it has no record for meter chilledwater\n\nBut looking at meter electricity (which is the most common):\n\n\n\n\nIt looks like an active building.\n\nWhat do you think about it? \nHow can we predict chilledwater consumpion without any (only 1) useful observation?", "votes": null }, { "id": "662989", "postDate": "11/01/2019 09:42:18", "content": "Thanks for sharing. Nice findings
", "rawMarkdown": "Thanks for sharing. Nice findings", "votes": null }, { "id": "665178", "postDate": "11/04/2019 17:38:50", "content": "I identified the site 13 as Twin Cities (see https://www.kaggle.com/c/ashrae-energy-prediction/discussion/115698). Therefore, the 1099 building can be identified with certainty as Coffman Memorial Union at U.of Minnesota. See https://en.wikipedia.org/wiki/Coffman_Memorial_Union\nIt should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset ...
", "rawMarkdown": "I identified the site 13 as Twin Cities (see https://www.kaggle.com/c/ashrae-energy-prediction/discussion/115698). Therefore, the 1099 building can be identified with certainty as Coffman Memorial Union at U.of Minnesota. See https://en.wikipedia.org/wiki/Coffman_Memorial_Union\nIt should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset ...", "votes": null }, { "id": "666076", "postDate": "11/05/2019 17:47:20", "content": "Thank you for sharing, EDA is really interesting
", "rawMarkdown": "Thank you for sharing, EDA is really interesting", "votes": null }, { "id": "674546", "postDate": "11/16/2019 17:30:50", "content": "> It should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset …
\n\nProbably this is an error, but there is public info that confirms these numbers somehow, I described it in UPD section here: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/116773#latest-674046
", "rawMarkdown": "> It should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset …\n\n\nProbably this is an error, but there is public info that confirms these numbers somehow, I described it in UPD section here: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/116773#latest-674046", "votes": null }, { "id": "680086", "postDate": "11/24/2019 01:31:56", "content": "If you follow the crumb trail left by Dmitry Labazkin to the Coffman Memorial Union - you will find that the dashboard shows steam on a meter that red lines at 20,000 lbs/hour.
\n\nPounds/hour is common measurement system for steam - it appears to me that a conversion error was made - if you convert 20,000 lbs/hour to BTU you get values that look similar to the data reported. These conversions need more info, like the psi level of the steam, etc. so they are not simple conversions, but my operating assumption is that kilo btu's per hour is what we are seeing.
\n\nI am going to try a pre-process conversion but it looks like I need a post process to convert back as folks have noted the test also seems to contain high values.
", "rawMarkdown": "If you follow the crumb trail left by Dmitry Labazkin to the Coffman Memorial Union - you will find that the dashboard shows steam on a meter that red lines at 20,000 lbs/hour.\n\nPounds/hour is common measurement system for steam - it appears to me that a conversion error was made - if you convert 20,000 lbs/hour to BTU you get values that look similar to the data reported. These conversions need more info, like the psi level of the steam, etc. so they are not simple conversions, but my operating assumption is that kilo btu's per hour is what we are seeing.\n\nI am going to try a pre-process conversion but it looks like I need a post process to convert back as folks have noted the test also seems to contain high values.", "votes": null } ], "comments": [ { "id": 660793, "author_name": "nroman", "author_url": "", "post_date": "10/29/2019 16:07:27", "content": "It is an 'outlier' within buildings. Here is some EDA
", "votes": null, "replies": [ { "id": 666076, "author_name": "negi009", "author_url": "", "post_date": "11/05/2019 17:47:20", "content": "Thank you for sharing, EDA is really interesting
", "votes": null, "replies": [] } ] }, { "id": 660810, "author_name": "abednadir", "author_url": "", "post_date": "10/29/2019 16:31:23", "content": "remove them and make a submission to the leaderboard, let us know if your score increases
", "votes": null, "replies": [] }, { "id": 660877, "author_name": "kaushal2896", "author_url": "", "post_date": "10/29/2019 18:19:13", "content": "What is the exact value of meter_reading for this data point?
", "votes": null, "replies": [ { "id": 661320, "author_name": "aitude", "author_url": "", "post_date": "10/30/2019 06:02:57", "content": "Mean of meter_reading for building id 1099 is 13372.445088 kWh
", "votes": null, "replies": [] } ] }, { "id": 661438, "author_name": "zgzjnbzl", "author_url": "", "post_date": "10/30/2019 09:47:45", "content": "I removed them, the lb score doubled to 2.37 from 1.13😂 Idk if i did everything right...
", "votes": null, "replies": [ { "id": 661458, "author_name": "jielu0728", "author_url": "", "post_date": "10/30/2019 10:29:14", "content": "Same experience here, after removing some consistent meter reading period (0 or something) my score increased from 1.10 to 1.18, I had thought that it would help.\nI think that there's also some strange meter readings like them in the test set for us to predict.
", "votes": null, "replies": [] }, { "id": 661464, "author_name": "aitude", "author_url": "", "post_date": "10/30/2019 10:39:12", "content": "I believe that this training set has more outliers because when I plot to scatter between hourly temperature and hourly meter reading, I don't see any correlation.
", "votes": null, "replies": [] }, { "id": 661487, "author_name": "zgzjnbzl", "author_url": "", "post_date": "10/30/2019 11:19:06", "content": "since we only have one year data, it is hard to know if they are real outliers or 'systematic outliers'. I can't see how to deal with thess expect keeping LB probing🤒 ..
", "votes": null, "replies": [] }, { "id": 661579, "author_name": "sdoria", "author_url": "", "post_date": "10/30/2019 13:30:30", "content": "@aitude That could be because the timestamps aren't aligned!
", "votes": null, "replies": [] }, { "id": 661590, "author_name": "aitude", "author_url": "", "post_date": "10/30/2019 13:45:31", "content": "@sdoria But if you separate training data by meter type then it makes sense. Below is a plot between chilledwater's energy consumption and air temperature.
\n\n
True, I have also seen patterns after filtering the data.
", "votes": null, "replies": [] } ] }, { "id": 661501, "author_name": "teeyee314", "author_url": "", "post_date": "10/30/2019 11:56:18", "content": "You can try and remove the data and see if it improves score. See the discussion on what zero meter reading means. The meter is likely a pulse meter.
", "votes": null, "replies": [] }, { "id": 661611, "author_name": "stenford23", "author_url": "", "post_date": "10/30/2019 14:04:04", "content": "what about this building?
\n\n
Building Details
\n\nThis building has only 1 meter reading different from 0, which is 0.0334, it seems quite suspicious cause it looks like a closed build or maybe it has no record for meter chilledwater
\n\nBut looking at meter electricity (which is the most common):
\n\n
It looks like an active building.
\n\nWhat do you think about it? \nHow can we predict chilledwater consumpion without any (only 1) useful observation?
", "votes": null, "replies": [] }, { "id": 662989, "author_name": "longyin2", "author_url": "", "post_date": "11/01/2019 09:42:18", "content": "Thanks for sharing. Nice findings
", "votes": null, "replies": [] }, { "id": 665178, "author_name": "poedator", "author_url": "", "post_date": "11/04/2019 17:38:50", "content": "I identified the site 13 as Twin Cities (see https://www.kaggle.com/c/ashrae-energy-prediction/discussion/115698). Therefore, the 1099 building can be identified with certainty as Coffman Memorial Union at U.of Minnesota. See https://en.wikipedia.org/wiki/Coffman_Memorial_Union\nIt should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset ...
", "votes": null, "replies": [ { "id": 674546, "author_name": "labdmitriy", "author_url": "", "post_date": "11/16/2019 17:30:50", "content": "> It should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset …
\n\nProbably this is an error, but there is public info that confirms these numbers somehow, I described it in UPD section here: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/116773#latest-674046
", "votes": null, "replies": [] } ] }, { "id": 680086, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/24/2019 01:31:56", "content": "If you follow the crumb trail left by Dmitry Labazkin to the Coffman Memorial Union - you will find that the dashboard shows steam on a meter that red lines at 20,000 lbs/hour.
\n\nPounds/hour is common measurement system for steam - it appears to me that a conversion error was made - if you convert 20,000 lbs/hour to BTU you get values that look similar to the data reported. These conversions need more info, like the psi level of the steam, etc. so they are not simple conversions, but my operating assumption is that kilo btu's per hour is what we are seeing.
\n\nI am going to try a pre-process conversion but it looks like I need a post process to convert back as folks have noted the test also seems to contain high values.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "660751": "Hello, \n\nPlease look at these two plots.\n\n### Building vs Meter Reading Plot\n\n\n\n### Regression Plot between Building and Meter Reading\n\n\n\n### Building Details\n\n- building_id\t- 1099\n- site_id\t- 13\n- primary_use - Education\t\n- square_feet - 332884\t\n- year_built - NaN\t\n- floor_count - NaN\n\nIt seems that meter reading for building id 1099 is not correct if we compare to other buildings that exist with the same site id. Below is plot -\n\n\n\nSo my question is do you recommend to remove these observations from the training data? \n\n\nThanks in advance for your response!", "660793": "It is an 'outlier' within buildings. Here is some [EDA](https://www.kaggle.com/nroman/eda-for-ashrae)", "660810": "remove them and make a submission to the leaderboard, let us know if your score increases", "660877": "What is the exact value of meter_reading for this data point?", "661320": "Mean of meter_reading for building id 1099 is 13372.445088 kWh", "661438": "I removed them, the lb score doubled to 2.37 from 1.13😂 Idk if i did everything right...", "661458": "Same experience here, after removing some consistent meter reading period (0 or something) my score increased from 1.10 to 1.18, I had thought that it would help.\nI think that there's also some strange meter readings like them in the test set for us to predict.", "661464": "I believe that this training set has more outliers because when I plot to scatter between hourly temperature and hourly meter reading, I don't see any correlation.", "661487": "since we only have one year data, it is hard to know if they are real outliers or 'systematic outliers'. I can't see how to deal with thess expect keeping LB probing🤒 ..", "661501": "You can try and remove the data and see if it improves score. See the discussion on [what zero meter reading means](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113054#655306). The meter is likely a pulse meter.", "661579": "aitude That could be because the timestamps aren't aligned!", "661590": "sdoria But if you separate training data by meter type then it makes sense. Below is a plot between chilledwater's energy consumption and air temperature. \n\n", "661606": "True, I have also seen patterns after filtering the data.", "661611": "what about this building?\n\n\n\nBuilding Details\n\n- building_id - 1018\n- site_id - 10\n- meter - chilledwater\n- primary_use - Education\n- square_feet - 186840\n- year_built - NaN\n- floor_count - 2.0\n\nThis building has only 1 meter reading different from 0, which is 0.0334, it seems quite suspicious cause it looks like a closed build or maybe it has no record for meter chilledwater\n\nBut looking at meter electricity (which is the most common):\n\n\n\n\nIt looks like an active building.\n\nWhat do you think about it? \nHow can we predict chilledwater consumpion without any (only 1) useful observation?", "662989": "Thanks for sharing. Nice findings", "665178": "I identified the site 13 as Twin Cities (see https://www.kaggle.com/c/ashrae-energy-prediction/discussion/115698). Therefore, the 1099 building can be identified with certainty as Coffman Memorial Union at U.of Minnesota. See https://en.wikipedia.org/wiki/Coffman_Memorial_Union\nIt should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset ...", "666076": "Thank you for sharing, EDA is really interesting", "674546": "> It should not consume that much steam energy. Most likely we deal with the reading or conversion error. The good question is whether there is the same error in the test dataset …\n\n\nProbably this is an error, but there is public info that confirms these numbers somehow, I described it in UPD section here: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/116773#latest-674046", "680086": "If you follow the crumb trail left by Dmitry Labazkin to the Coffman Memorial Union - you will find that the dashboard shows steam on a meter that red lines at 20,000 lbs/hour.\n\nPounds/hour is common measurement system for steam - it appears to me that a conversion error was made - if you convert 20,000 lbs/hour to BTU you get values that look similar to the data reported. These conversions need more info, like the psi level of the steam, etc. so they are not simple conversions, but my operating assumption is that kilo btu's per hour is what we are seeing.\n\nI am going to try a pre-process conversion but it looks like I need a post process to convert back as folks have noted the test also seems to contain high values." }, "source": "meta" }