\nCorrelations\n- Correlations of all features (buildings and weather included) differ substantially between sites\n- Only weak correlations to meter_reading\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F941950%2F5a0cdfecb02b3ea8b9df6193c4f91b0f%2Fcorrelation%20to%20meter_reading.png?generation=1571682417386003&alt=media\")
While examining and plotting the data I've made these observations so far:
\n\n
\nTraining data\n- 20 Mio. samples, no missing values, no duplicates\n- One year worth of data, all timestamps from 2016\n- Timewise the measurements are quite evenly distributed.\n- Target variable highly skewed\n- Most readings measure electricity.\n- Generating steam consumes most energy.\n- Energy consumption peaks significantly in Spring, steam energy extremely so (but only due to two outliers, see below).\n- Chilled water most consumed in late summer.\n- Education buildings consume by far most of energy, then come office buildings. \n- Buildings between 300-400k square feet by far with highest consumption (again due to the two outliers, see below)\n- Buildings have up to four types of meters, most have one (or three) types.\n- Distinct ranges of meter readings per building observable.\n- 3750 outliers for meter_reading. They stem from just 2 buildings (1099, 778) and come from steam and chilledwater. \n- Building 1099 consumes lots of steam energy.\n- Buildings 1099 and 778 are massive outliers. Examining the data with and without these two buildings yields very different results. \n- Possible meter failures eg building 218, 192. See this thread..\n- Buildings on site_id == 0 have either zero or small readings in the first 5 months of 2016. Thread.
\nTest data\n- Private dataset seems to be the last 22% of the data, time ordered (test set and row_id column are not strictly time ordered). Thread here.
\nBuilding data\n- 1449 unique buildings on 16 sites built between 1900 and 2017 \n- 7 buildings (building_id = 28, 45, 46, 99, 103, 409, 567) built in timespan of training data 2016. \n- Building 363 built in 2017, but consumes energy in 2016. More here.\n- Wide variety of size and height: 283 to 875k sqft, 1 to 26 floors\n- 16 types of usage\n- 14 sites with mixed types of buildings, 2 with just one type\n- site_id likely a city or area rather than a specific compound of buildings\n- Distribution of square_feet highly skewed\n- Parking facilities much bigger on average than all other types\n- Food, parking, retail among the most recent built buildings\n- Technology and science related buildings on average the oldest\n- Lodging/residential on average the highest buildings\n- 29 outliers for square_feet and 5 for floor_count
\nWeather data\n- 417016 samples, several duplicates, lots of missing values\n- All buildings from northern hemisphere, since seasonal temperature in general follows the same pattern\n- cloud_coverage for site_id 12 is much higher than on all other sites.\n- sea_level_pressure for site_id 5 is entirely missing (NaN).\n- Sites 1, 5, 12 likely in Europe due to timezone and use of Celsius, sites 7 and 11 in Canada due to the timezone, use of C and July 1st, others potentially in US due to the timezone and use of F. Thread here.\n- Weather data likely not in local time. Thread.\n- Wind direction has implicit duplicates with values 0 and 360. Can easily be deduplicated. Wind direction as modelling features as such likely negligible though.
\nCorrelations\n- Correlations of all features (buildings and weather included) differ substantially between sites\n- Only weak correlations to meter_reading\n
\nKernel: \nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip
While examining and plotting the data I've made these observations so far:
\n\n
\nTraining data\n- 20 Mio. samples, no missing values, no duplicates\n- One year worth of data, all timestamps from 2016\n- Timewise the measurements are quite evenly distributed.\n- Target variable highly skewed\n- Most readings measure electricity.\n- Generating steam consumes most energy.\n- Energy consumption peaks significantly in Spring, steam energy extremely so (but only due to two outliers, see below).\n- Chilled water most consumed in late summer.\n- Education buildings consume by far most of energy, then come office buildings. \n- Buildings between 300-400k square feet by far with highest consumption (again due to the two outliers, see below)\n- Buildings have up to four types of meters, most have one (or three) types.\n- Distinct ranges of meter readings per building observable.\n- 3750 outliers for meter_reading. They stem from just 2 buildings (1099, 778) and come from steam and chilledwater. \n- Building 1099 consumes lots of steam energy.\n- Buildings 1099 and 778 are massive outliers. Examining the data with and without these two buildings yields very different results. \n- Possible meter failures eg building 218, 192. See this thread..\n- Buildings on site_id == 0 have either zero or small readings in the first 5 months of 2016. Thread.
\nTest data\n- Private dataset seems to be the last 22% of the data, time ordered (test set and row_id column are not strictly time ordered). Thread here.
\nBuilding data\n- 1449 unique buildings on 16 sites built between 1900 and 2017 \n- 7 buildings (building_id = 28, 45, 46, 99, 103, 409, 567) built in timespan of training data 2016. \n- Building 363 built in 2017, but consumes energy in 2016. More here.\n- Wide variety of size and height: 283 to 875k sqft, 1 to 26 floors\n- 16 types of usage\n- 14 sites with mixed types of buildings, 2 with just one type\n- site_id likely a city or area rather than a specific compound of buildings\n- Distribution of square_feet highly skewed\n- Parking facilities much bigger on average than all other types\n- Food, parking, retail among the most recent built buildings\n- Technology and science related buildings on average the oldest\n- Lodging/residential on average the highest buildings\n- 29 outliers for square_feet and 5 for floor_count
\nWeather data\n- 417016 samples, several duplicates, lots of missing values\n- All buildings from northern hemisphere, since seasonal temperature in general follows the same pattern\n- cloud_coverage for site_id 12 is much higher than on all other sites.\n- sea_level_pressure for site_id 5 is entirely missing (NaN).\n- Sites 1, 5, 12 likely in Europe due to timezone and use of Celsius, sites 7 and 11 in Canada due to the timezone, use of C and July 1st, others potentially in US due to the timezone and use of F. Thread here.\n- Weather data likely not in local time. Thread.\n- Wind direction has implicit duplicates with values 0 and 360. Can easily be deduplicated. Wind direction as modelling features as such likely negligible though.
\nCorrelations\n- Correlations of all features (buildings and weather included) differ substantially between sites\n- Only weak correlations to meter_reading\n
\nKernel: \nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip
Quite helpful EDA
", "rawMarkdown": "Quite helpful EDA", "votes": 5 }, { "id": 663306, "postDate": "2019-11-01T19:26:58.140Z", "content": "How do you identify \"outliers\"?
", "rawMarkdown": "How do you identify \"outliers\"?", "votes": 1, "replies": [ { "id": 663886, "postDate": "2019-11-02T20:01:02.707Z", "content": "Code and explanation is in my notebook:
\n\nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip
\n\nOr this answer to a similar question:
\n\nhttps://www.kaggle.com/general/113313#652302
\n\nCode and procedure is based on this article:
\n\nhttps://machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/
", "rawMarkdown": "Code and explanation is in my notebook:\n\nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip\n\nOr this answer to a similar question:\n\nhttps://www.kaggle.com/general/113313#652302\n\nCode and procedure is based on this article:\n\nhttps://machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/" } ] }, { "id": 662058, "postDate": "2019-10-31T03:31:49.280Z", "content": "Thanks for briefing! Helpful.
", "rawMarkdown": "Thanks for briefing! Helpful.", "votes": 1, "replies": [ { "id": 673672, "postDate": "2019-11-15T10:03:25.453Z", "content": "杰少，daidaiwo啊
", "rawMarkdown": "杰少，daidaiwo啊", "votes": -1 } ] }, { "id": 654112, "postDate": "2019-10-21T13:19:03.127Z", "content": "Lots of stuff in a nutshell. Very useful. Can you also add the kernel links where you found these info ?
", "rawMarkdown": "Lots of stuff in a nutshell. Very useful. Can you also add the kernel links where you found these info ? ", "votes": 2, "replies": [ { "id": 654135, "postDate": "2019-10-21T13:54:15.270Z", "content": "Sure. Many things I found out by myself – see this kernel:
\n\nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip
\n\nAt the end I reference the kernels of the fellow Kagglers that have inspired the EDA., e.g.:
\n\nhttps://www.kaggle.com/blue07/eda-insights-on-weather-buildings\nhttps://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-ashrae
", "rawMarkdown": "Sure. Many things I found out by myself – see this kernel:\n\nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip\n\nAt the end I reference the kernels of the fellow Kagglers that have inspired the EDA., e.g.:\n\nhttps://www.kaggle.com/blue07/eda-insights-on-weather-buildings\nhttps://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-ashrae", "votes": 5 }, { "id": 660898, "postDate": "2019-10-29T18:52:08.523Z", "content": "This is really great. All at one place. Thanks
", "rawMarkdown": "This is really great. All at one place. Thanks", "votes": 1 }, { "id": 660928, "postDate": "2019-10-29T19:27:01.627Z", "content": "Everyone please stop with the contentless 'Thanks' or 'Very helpful' posts (borderline spam). Just simply click the upvote button, that's what it's there for. (Whereas if you post a contentless response, it triggers an individual email to everyone subscribed to each thread.)
", "rawMarkdown": "Everyone please stop with the contentless 'Thanks' or 'Very helpful' posts (borderline spam). Just simply click the upvote button, that's what it's there for. (Whereas if you post a contentless response, it triggers an individual email to everyone subscribed to each thread.)", "votes": 2 }, { "id": 662070, "postDate": "2019-10-31T04:05:04.393Z", "content": "I think that is because it is an easy way to get bronze medals in discussion. Kagglers are kind enough to appreciate those kind words.
", "rawMarkdown": "I think that is because it is an easy way to get bronze medals in discussion. Kagglers are kind enough to appreciate those kind words. " } ] }, { "id": 2513809, "postDate": "2023-11-05T18:11:49.587Z", "content": "Very Informative EDA Insights
", "rawMarkdown": "Very Informative EDA Insights" }, { "id": 673748, "postDate": "2019-11-15T12:35:21.803Z", "content": "Guys, give me advice what to do with features yea built, floor count if there is no data for almost 50% of buildings in set?
", "rawMarkdown": "Guys, give me advice what to do with features yea built, floor count if there is no data for almost 50% of buildings in set?" }, { "id": 664334, "postDate": "2019-11-03T14:19:29.927Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 660098, "postDate": "2019-10-28T17:14:39.173Z", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": 1 }, { "id": 654617, "postDate": "2019-10-22T05:18:11.500Z", "content": "thanks!
", "rawMarkdown": "thanks!", "votes": 1 }, { "id": 663240, "postDate": "2019-11-01T16:58:43.183Z", "content": "Thanks for helpful insight of data.
", "rawMarkdown": "Thanks for helpful insight of data.", "votes": 2 }, { "id": 654084, "postDate": "2019-10-21T12:25:52.903Z", "content": "Thanks for briefing!
", "rawMarkdown": "Thanks for briefing! " } ], "comments": [ { "id": 656368, "author_name": "itsshavar", "author_url": "", "post_date": "2019-10-24T07:12:27.370000", "content": "Quite helpful EDA
", "votes": 5, "replies": [] }, { "id": 663306, "author_name": "bluetrain", "author_url": "", "post_date": "2019-11-01T19:26:58.140000", "content": "How do you identify \"outliers\"?
", "votes": 1, "replies": [ { "id": 663886, "author_name": "chmaxx", "author_url": "", "post_date": "2019-11-02T20:01:02.707000", "content": "Code and explanation is in my notebook:
\n\nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip
\n\nOr this answer to a similar question:
\n\nhttps://www.kaggle.com/general/113313#652302
\n\nCode and procedure is based on this article:
\n\nhttps://machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/
", "votes": 0, "replies": [] } ] }, { "id": 662058, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2019-10-31T03:31:49.280000", "content": "Thanks for briefing! Helpful.
", "votes": 1, "replies": [ { "id": 673672, "author_name": "Cobalt", "author_url": "", "post_date": "2019-11-15T10:03:25.453000", "content": "杰少，daidaiwo啊
", "votes": -1, "replies": [] } ] }, { "id": 654112, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-21T13:19:03.127000", "content": "Lots of stuff in a nutshell. Very useful. Can you also add the kernel links where you found these info ?
", "votes": 2, "replies": [ { "id": 654135, "author_name": "chmaxx", "author_url": "", "post_date": "2019-10-21T13:54:15.270000", "content": "Sure. Many things I found out by myself – see this kernel:
\n\nhttps://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip
\n\nAt the end I reference the kernels of the fellow Kagglers that have inspired the EDA., e.g.:
\n\nhttps://www.kaggle.com/blue07/eda-insights-on-weather-buildings\nhttps://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-ashrae
", "votes": 5, "replies": [] }, { "id": 660898, "author_name": "itsshavar", "author_url": "", "post_date": "2019-10-29T18:52:08.523000", "content": "This is really great. All at one place. Thanks
", "votes": 1, "replies": [] }, { "id": 660928, "author_name": "Stephen McInerney", "author_url": "", "post_date": "2019-10-29T19:27:01.627000", "content": "Everyone please stop with the contentless 'Thanks' or 'Very helpful' posts (borderline spam). Just simply click the upvote button, that's what it's there for. (Whereas if you post a contentless response, it triggers an individual email to everyone subscribed to each thread.)
", "votes": 2, "replies": [] }, { "id": 662070, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-31T04:05:04.393000", "content": "I think that is because it is an easy way to get bronze medals in discussion. Kagglers are kind enough to appreciate those kind words.
", "votes": 0, "replies": [] } ] }, { "id": 2513809, "author_name": "Nithiyashree Venkatachalam Kannan", "author_url": "", "post_date": "2023-11-05T18:11:49.587000", "content": "Very Informative EDA Insights
", "votes": 0, "replies": [] }, { "id": 673748, "author_name": "AvtandylRudenko", "author_url": "", "post_date": "2019-11-15T12:35:21.803000", "content": "Guys, give me advice what to do with features yea built, floor count if there is no data for almost 50% of buildings in set?
", "votes": 0, "replies": [] }, { "id": 664334, "author_name": "", "author_url": "", "post_date": "2019-11-03T14:19:29.927000", "content": "", "votes": 1, "replies": [] }, { "id": 660098, "author_name": "AJ Aberdein", "author_url": "", "post_date": "2019-10-28T17:14:39.173000", "content": "Thanks!
", "votes": 1, "replies": [] }, { "id": 654617, "author_name": "Krishna Katyal", "author_url": "", "post_date": "2019-10-22T05:18:11.500000", "content": "thanks!
", "votes": 1, "replies": [] }, { "id": 663240, "author_name": "Neelam", "author_url": "", "post_date": "2019-11-01T16:58:43.183000", "content": "Thanks for helpful insight of data.
", "votes": 2, "replies": [] }, { "id": 654084, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-21T12:25:52.903000", "content": "Thanks for briefing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "654079": "While [examining and plotting the data](https://www.kaggle.com/chmaxx/ashrae-eda-and-visualization-wip) I've made these observations so far:\n\n