![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2F60cb73d0b99415922e2e8521563552b6%2Fmissing_data_train.png?generation=1572806448865078&alt=media\")
Hi all,
\n\nJust joining. A quick count of non-null observations shows more than half data is missing in training (especially steam/hotwater/chilledwater). Only Electricity meter is acceptable.
\n\n
For this challenge should we predict for a building when meter_reading is 100% missing?\nFor me it leads to something like:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2Fac5f22c7021217d4a07168c8cc4b45b8%2Fmissing_data_prediction.png?generation=1572806583031161&alt=media\")
We could try to predict by ignoring building_id. Is that what organizer want from us?
\n\nThanks for any clarification.
", "messages": [ { "id": "664498", "postDate": "11/03/2019 18:49:30", "content": "Hi all,
\n\nJust joining. A quick count of non-null observations shows more than half data is missing in training (especially steam/hotwater/chilledwater). Only Electricity meter is acceptable.
\n\n
For this challenge should we predict for a building when meter_reading is 100% missing?\nFor me it leads to something like:\n
We could try to predict by ignoring building_id. Is that what organizer want from us?
\n\nThanks for any clarification.
", "rawMarkdown": "Hi all, \n\nJust joining. A quick count of non-null observations shows more than half data is missing in training (especially steam/hotwater/chilledwater). Only Electricity meter is acceptable. \n\n\n\nFor this challenge should we predict for a building when meter_reading is 100% missing?\nFor me it leads to something like:\n\n\nWe could try to predict by ignoring building_id. Is that what organizer want from us?\n\nThanks for any clarification.", "votes": null }, { "id": "664523", "postDate": "11/03/2019 19:42:42", "content": "Nice visualization @mpware. Would love to see your code.
", "rawMarkdown": "Nice visualization @mpware. Would love to see your code.", "votes": null }, { "id": "664527", "postDate": "11/03/2019 19:54:14", "content": "\"Not every building has all meter types.\"\nYou don't need to predict the missing meters
", "rawMarkdown": "\"Not every building has all meter types.\"\nYou don't need to predict the missing meters", "votes": null }, { "id": "664776", "postDate": "11/04/2019 08:00:26", "content": "\n\n\nWe could try to predict by ignoring building_id. Is that what organizer want from us?
\n
This is most likely what organizers are trying to get from us, but you know Kaggle doesn't work that way. The most overfitting model to the test set wins the price. You should use everything they gave you.
", "rawMarkdown": "> We could try to predict by ignoring building_id. Is that what organizer want from us?\n\nThis is most likely what organizers are trying to get from us, but you know Kaggle doesn't work that way. The most overfitting model to the test set wins the price. You should use everything they gave you.", "votes": null }, { "id": "664777", "postDate": "11/04/2019 08:02:54", "content": "Thanks, I realize that the answer is already in test dataset, we have the same missing data for the same buildings. Only the following buildings have all meter available: 1232, 1241, 1249, 1258, 1259, 1293, 1294, 1295, 1296, 1297, 1298, 1301, 1331. For others we can have partial data (so we can predict) or no data at all (we don't predict). For instance:
\n\ntest_pd.query(\"building_id == 504 & meter == 1\")
return zero lines.
", "rawMarkdown": "Thanks, I realize that the answer is already in test dataset, we have the same missing data for the same buildings. Only the following buildings have all meter available: 1232, 1241, 1249, 1258, 1259, 1293, 1294, 1295, 1296, 1297, 1298, 1301, 1331. For others we can have partial data (so we can predict) or no data at all (we don't predict). For instance:\n\n`test_pd.query(\"building_id == 504 & meter == 1\")`\n\nreturn zero lines.", "votes": null }, { "id": "664780", "postDate": "11/04/2019 08:07:00", "content": "Here is how we can see missing data (in test dataset):
\n\n```
\n\nMETER_DICT = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\ntest_pd[\"meter_reading\"] = 1.0 # Dummy prediction\ntest_meter_pd = test_pd.groupby([\"building_id\", \"meter\"]).agg({\"meter_reading\": 'count'}).reset_index()\ntest_meter_pd[\"meter\"] = test_meter_pd[\"meter\"].map(METER_DICT)\ntest_meter_pd = test_meter_pd.pivot(index='building_id', columns='meter', values='meter_reading').fillna(0)\nfig, ax = plt.subplots(figsize=(26, 4))\nd = sns.heatmap(data=test_meter_pd.T, ax=ax)\nd = plt.title(\"Count of non-null meter_reading observations\")\nd = plt.yticks(rotation=0)\n```
", "rawMarkdown": "Here is how we can see missing data (in test dataset):\n\n```\n# Missing data per building_id in test data\nMETER_DICT = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\ntest_pd[\"meter_reading\"] = 1.0 # Dummy prediction\ntest_meter_pd = test_pd.groupby([\"building_id\", \"meter\"]).agg({\"meter_reading\": 'count'}).reset_index()\ntest_meter_pd[\"meter\"] = test_meter_pd[\"meter\"].map(METER_DICT)\ntest_meter_pd = test_meter_pd.pivot(index='building_id', columns='meter', values='meter_reading').fillna(0)\nfig, ax = plt.subplots(figsize=(26, 4))\nd = sns.heatmap(data=test_meter_pd.T, ax=ax)\nd = plt.title(\"Count of non-null meter_reading observations\")\nd = plt.yticks(rotation=0)\n```", "votes": null }, { "id": "664790", "postDate": "11/04/2019 08:23:33", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1187996%2F36f6697819ffe8745a38ab34974a04d6%2Fbuilding-meter.png?generation=1572855794991175&alt=media\")
Out of the 1449 buildings only 1413 have electricity meters installed. 498 buildings have chilled water connections, 324 have stem connection and 145 have hot water connections.
\n\nAgain 875 buildings have only one type of meter, 230 have two types, 331 have three types and only 13 buildings have all the types.
\n\nIf the image is not clear check this this kernel. for details.
", "rawMarkdown": "\n\n\nOut of the 1449 buildings only 1413 have electricity meters installed. 498 buildings have chilled water connections, 324 have stem connection and 145 have hot water connections.\n\nAgain 875 buildings have only one type of meter, 230 have two types, 331 have three types and only 13 buildings have all the types.\n\nIf the image is not clear check this [this kernel.](https://www.kaggle.com/rajeshcv/exploring-buildings-weather-and-time-series-data) for details.", "votes": null }, { "id": "664815", "postDate": "11/04/2019 09:12:08", "content": "For your information I've already tried to predict without building_id. That requires much longer time to converge (early stop). And yet no better result than my model with building_id.
", "rawMarkdown": "For your information I've already tried to predict without building\\_id. That requires much longer time to converge (early stop). And yet no better result than my model with building\\_id.", "votes": null }, { "id": "664829", "postDate": "11/04/2019 09:29:59", "content": "Looks like a good feature candidate 🤔
", "rawMarkdown": "Looks like a good feature candidate 🤔", "votes": null } ], "comments": [ { "id": 664523, "author_name": "themonologue", "author_url": "", "post_date": "11/03/2019 19:42:42", "content": "Nice visualization @mpware. Would love to see your code.
", "votes": null, "replies": [ { "id": 664780, "author_name": "mpware", "author_url": "", "post_date": "11/04/2019 08:07:00", "content": "Here is how we can see missing data (in test dataset):
\n\n```
\n\nMETER_DICT = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\ntest_pd[\"meter_reading\"] = 1.0 # Dummy prediction\ntest_meter_pd = test_pd.groupby([\"building_id\", \"meter\"]).agg({\"meter_reading\": 'count'}).reset_index()\ntest_meter_pd[\"meter\"] = test_meter_pd[\"meter\"].map(METER_DICT)\ntest_meter_pd = test_meter_pd.pivot(index='building_id', columns='meter', values='meter_reading').fillna(0)\nfig, ax = plt.subplots(figsize=(26, 4))\nd = sns.heatmap(data=test_meter_pd.T, ax=ax)\nd = plt.title(\"Count of non-null meter_reading observations\")\nd = plt.yticks(rotation=0)\n```
", "votes": null, "replies": [] } ] }, { "id": 664527, "author_name": "stecasasso", "author_url": "", "post_date": "11/03/2019 19:54:14", "content": "\"Not every building has all meter types.\"\nYou don't need to predict the missing meters
", "votes": null, "replies": [ { "id": 664777, "author_name": "mpware", "author_url": "", "post_date": "11/04/2019 08:02:54", "content": "Thanks, I realize that the answer is already in test dataset, we have the same missing data for the same buildings. Only the following buildings have all meter available: 1232, 1241, 1249, 1258, 1259, 1293, 1294, 1295, 1296, 1297, 1298, 1301, 1331. For others we can have partial data (so we can predict) or no data at all (we don't predict). For instance:
\n\ntest_pd.query(\"building_id == 504 & meter == 1\")
return zero lines.
", "votes": null, "replies": [] } ] }, { "id": 664776, "author_name": "gunesevitan", "author_url": "", "post_date": "11/04/2019 08:00:26", "content": "\n\n\nWe could try to predict by ignoring building_id. Is that what organizer want from us?
\n
This is most likely what organizers are trying to get from us, but you know Kaggle doesn't work that way. The most overfitting model to the test set wins the price. You should use everything they gave you.
", "votes": null, "replies": [] }, { "id": 664790, "author_name": "rajeshcv", "author_url": "", "post_date": "11/04/2019 08:23:33", "content": "
Out of the 1449 buildings only 1413 have electricity meters installed. 498 buildings have chilled water connections, 324 have stem connection and 145 have hot water connections.
\n\nAgain 875 buildings have only one type of meter, 230 have two types, 331 have three types and only 13 buildings have all the types.
\n\nIf the image is not clear check this this kernel. for details.
", "votes": null, "replies": [ { "id": 664829, "author_name": "jielu0728", "author_url": "", "post_date": "11/04/2019 09:29:59", "content": "Looks like a good feature candidate 🤔
", "votes": null, "replies": [] } ] }, { "id": 664815, "author_name": "jielu0728", "author_url": "", "post_date": "11/04/2019 09:12:08", "content": "For your information I've already tried to predict without building_id. That requires much longer time to converge (early stop). And yet no better result than my model with building_id.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "664498": "Hi all, \n\nJust joining. A quick count of non-null observations shows more than half data is missing in training (especially steam/hotwater/chilledwater). Only Electricity meter is acceptable. \n\n\n\nFor this challenge should we predict for a building when meter_reading is 100% missing?\nFor me it leads to something like:\n\n\nWe could try to predict by ignoring building_id. Is that what organizer want from us?\n\nThanks for any clarification.", "664523": "Nice visualization @mpware. Would love to see your code.", "664527": "\"Not every building has all meter types.\"\nYou don't need to predict the missing meters", "664776": "> We could try to predict by ignoring building_id. Is that what organizer want from us?\n\nThis is most likely what organizers are trying to get from us, but you know Kaggle doesn't work that way. The most overfitting model to the test set wins the price. You should use everything they gave you.", "664777": "Thanks, I realize that the answer is already in test dataset, we have the same missing data for the same buildings. Only the following buildings have all meter available: 1232, 1241, 1249, 1258, 1259, 1293, 1294, 1295, 1296, 1297, 1298, 1301, 1331. For others we can have partial data (so we can predict) or no data at all (we don't predict). For instance:\n\n`test_pd.query(\"building_id == 504 & meter == 1\")`\n\nreturn zero lines.", "664780": "Here is how we can see missing data (in test dataset):\n\n```\n# Missing data per building_id in test data\nMETER_DICT = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\ntest_pd[\"meter_reading\"] = 1.0 # Dummy prediction\ntest_meter_pd = test_pd.groupby([\"building_id\", \"meter\"]).agg({\"meter_reading\": 'count'}).reset_index()\ntest_meter_pd[\"meter\"] = test_meter_pd[\"meter\"].map(METER_DICT)\ntest_meter_pd = test_meter_pd.pivot(index='building_id', columns='meter', values='meter_reading').fillna(0)\nfig, ax = plt.subplots(figsize=(26, 4))\nd = sns.heatmap(data=test_meter_pd.T, ax=ax)\nd = plt.title(\"Count of non-null meter_reading observations\")\nd = plt.yticks(rotation=0)\n```", "664790": "\n\n\nOut of the 1449 buildings only 1413 have electricity meters installed. 498 buildings have chilled water connections, 324 have stem connection and 145 have hot water connections.\n\nAgain 875 buildings have only one type of meter, 230 have two types, 331 have three types and only 13 buildings have all the types.\n\nIf the image is not clear check this [this kernel.](https://www.kaggle.com/rajeshcv/exploring-buildings-weather-and-time-series-data) for details.", "664815": "For your information I've already tried to predict without building\\_id. That requires much longer time to converge (early stop). And yet no better result than my model with building\\_id.", "664829": "Looks like a good feature candidate 🤔" }, "source": "meta" }