Due to a miscommunication, the electric meter readings for site 0 were not properly converted to units of kWh and are in kBTU. We apologize for the error and the resulting inconvenience.
\n\nThe units of the electric meter readings for site 0 will not be updated (i.e., they will remain as is), since this could potentially disrupt the workflows and models of many teams. For models that are sensitive to units/absolute values, the issue can be avoided with the right conversion factors. Multiply by 0.2931 to get to model inputs into kWh like the other sites, and 3.4118 to get back to kBTU for scoring.
\n\nThanks to @teeyee314 for catching the issue.
", "messages": [ { "id": "682635", "postDate": "11/27/2019 17:47:28", "content": "Due to a miscommunication, the electric meter readings for site 0 were not properly converted to units of kWh and are in kBTU. We apologize for the error and the resulting inconvenience.
\n\nThe units of the electric meter readings for site 0 will not be updated (i.e., they will remain as is), since this could potentially disrupt the workflows and models of many teams. For models that are sensitive to units/absolute values, the issue can be avoided with the right conversion factors. Multiply by 0.2931 to get to model inputs into kWh like the other sites, and 3.4118 to get back to kBTU for scoring.
\n\nThanks to @teeyee314 for catching the issue.
", "rawMarkdown": "Due to a miscommunication, the electric meter readings for site 0 were not properly converted to units of kWh and are in kBTU. We apologize for the error and the resulting inconvenience. \n\nThe units of the electric meter readings for site 0 will not be updated (i.e., they will remain as is), since this could potentially disrupt the workflows and models of many teams. For models that are sensitive to units/absolute values, the issue can be avoided with the right conversion factors. Multiply by 0.2931 to get to model inputs into kWh like the other sites, and 3.4118 to get back to kBTU for scoring.\n\nThanks to @teeyee314 for [catching the issue](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/118753).", "votes": null }, { "id": "683474", "postDate": "11/28/2019 13:17:52", "content": "Am I right that if you do modelling building-wise or site-wise this changes essentially nothing?
", "rawMarkdown": "Am I right that if you do modelling building-wise or site-wise this changes essentially nothing?", "votes": null }, { "id": "683809", "postDate": "11/28/2019 20:54:40", "content": "Could you please explain why it's necessary to multiply by 3.14118 for scoring? Am I right that the evaluation is in kWh?
", "rawMarkdown": "Could you please explain why it's necessary to multiply by 3.14118 for scoring? Am I right that the evaluation is in kWh?", "votes": null }, { "id": "684101", "postDate": "11/29/2019 08:30:12", "content": "I assume it is because the same error is in the test data as well, i.e. for site 0 are values are in kBTU unit in both train and test data.\nHence, if we convert train data to kWh, we need to convert back in our predictions.
", "rawMarkdown": "I assume it is because the same error is in the test data as well, i.e. for site 0 are values are in kBTU unit in both train and test data.\nHence, if we convert train data to kWh, we need to convert back in our predictions.", "votes": null }, { "id": "684102", "postDate": "11/29/2019 08:31:14", "content": "Think so, yes. I you have modeling elements that do cross site comparisons, you might have a problem
", "rawMarkdown": "Think so, yes. I you have modeling elements that do cross site comparisons, you might have a problem", "votes": null }, { "id": "685189", "postDate": "12/01/2019 05:38:02", "content": "When I read your message I understand that something wrong on site_0 - but a tiny bit confused.
\n\nOption 1 - readings for meter types 0 and 1 are all wrong.\nor\nOption 2 - only readings for meter type 0 are wrong.
\n\nI think your saying Option 2 - is that correct?
\n\nIf it's option 1 - they were both wrong - than what conversion is needed.
\n\nWhile the data description indicated that all 4 meters were in kwh - the information from the leak sites does not seem to support that - certainly looks like the meters are in different units - lbs, tons, whatever.
\n\nRather than simply pointing to the data description (which has now proved to be wrong) it would be nice of you did a deep dive on your data and supply a more convincing statement that all sites have been checked and all meter readings (expect site 0) are in kWh. Because I think I see many that make no sense if truly kWh.
", "rawMarkdown": "When I read your message I understand that something wrong on site_0 - but a tiny bit confused.\n\nOption 1 - readings for meter types 0 and 1 are all wrong.\nor\nOption 2 - only readings for meter type 0 are wrong.\n\nI think your saying Option 2 - is that correct?\n\n\nIf it's option 1 - they were both wrong - than what conversion is needed. \n\nWhile the data description indicated that all 4 meters were in kwh - the information from the leak sites does not seem to support that - certainly looks like the meters are in different units - lbs, tons, whatever. \n\nRather than simply pointing to the data description (which has now proved to be wrong) it would be nice of you did a deep dive on your data and supply a more convincing statement that all sites have been checked and all meter readings (expect site 0) are in kWh. Because I think I see many that make no sense if truly kWh.", "votes": null }, { "id": "685633", "postDate": "12/02/2019 03:02:42", "content": "It seems like readings for meter type 1 are wrong, isn't it?
", "rawMarkdown": "It seems like readings for meter type 1 are wrong, isn't it?", "votes": null }, { "id": "686170", "postDate": "12/02/2019 21:08:05", "content": "yes - it seems to me that type 1 is wrong and type 0 is ok - but he used the word \"electric\" which I think is type 0.
", "rawMarkdown": "yes - it seems to me that type 1 is wrong and type 0 is ok - but he used the word \"electric\" which I think is type 0.", "votes": null }, { "id": "686631", "postDate": "12/03/2019 11:07:01", "content": "I suggest adding this to the \"data\" page's description for visibility. This is a big change.
", "rawMarkdown": "I suggest adding this to the \"data\" page's description for visibility. This is a big change.", "votes": null }, { "id": "687297", "postDate": "12/04/2019 07:38:01", "content": "Martin Is following what you want to say if I want to do a code correction?
\n\ndef correct_error_meter0_model_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*0.2931) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading
\n\ndef correct_error_meter0_scoring_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*3.4118) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading
\n\nSo, to summaries, we will use correct_error_meter0_model_use for our train and valudation dataset and will use and will use function **correct_error_meter0_scoring_use ** function for our outcome in validation
", "rawMarkdown": "Martin Is following what you want to say if I want to do a code correction?\n\ndef correct_error_meter0_model_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*0.2931) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading\n\ndef correct_error_meter0_scoring_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*3.4118) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading\n\nSo, to summaries, we will use **correct_error_meter0_model_use** for our train and valudation dataset and will use and will use function **correct_error_meter0_scoring_use ** function for our outcome in validation", "votes": null }, { "id": "688714", "postDate": "12/06/2019 01:04:29", "content": "I multiply 0.2931 the electric meter readings for site 0 and then log1p before tranning, exem1 and then multiply 3.4118 before submit, but get very worse result, what's wrong with my understanding?
", "rawMarkdown": "I multiply 0.2931 the electric meter readings for site 0 and then log1p before tranning, exem1 and then multiply 3.4118 before submit, but get very worse result, what's wrong with my understanding?", "votes": null }, { "id": "688836", "postDate": "12/06/2019 06:39:43", "content": "@djongo did you consider the meter type? He refers to type 0 as of my understanding.
", "rawMarkdown": "djongo did you consider the meter type? He refers to type 0 as of my understanding.", "votes": null }, { "id": "688979", "postDate": "12/06/2019 10:01:53", "content": "Yes， I considered the electric meter.
", "rawMarkdown": "Yes， I considered the electric meter.", "votes": null }, { "id": "691797", "postDate": "12/10/2019 14:02:28", "content": "yes, that's it.\nI did it like so:\ntrain.loc[(train['site_id'] == 0) & (train['meter'] == 0), 'meter_reading'] = train[(train['site_id'] == 0) & (train['meter'] == 0)]['meter_reading'] * 0.2931
\n\nSame same
", "rawMarkdown": "yes, that's it.\nI did it like so:\ntrain.loc[(train['site_id'] == 0) & (train['meter'] == 0), 'meter_reading'] = train[(train['site_id'] == 0) & (train['meter'] == 0)]['meter_reading'] * 0.2931\n\nSame same", "votes": null }, { "id": "691799", "postDate": "12/10/2019 14:05:31", "content": "Actually, buildingwise no, since it only applies to electric meters, absolute values within a building between meters will be affected. So, meter wise model requires no handling, building-wise does, IF you use relation between meters within building =)
", "rawMarkdown": "Actually, buildingwise no, since it only applies to electric meters, absolute values within a building between meters will be affected. So, meter wise model requires no handling, building-wise does, IF you use relation between meters within building =)", "votes": null }, { "id": "692825", "postDate": "12/11/2019 19:12:24", "content": "\n\n\ntype 0 is ok
\n
Nope. Based on the table of the referenced post, type 0 (electric) is in kBTU while the other sites are in kWh. Hence, type 0 should be updated to kWh. Regarding type 1 (Chilled), the unit for site 0 is kBTU while for sites 2 and 15 is Tons (1 ton (refrigeration) = 12000 Btu (IT)/hour).
", "rawMarkdown": "> type 0 is ok\n\nNope. Based on the table of the [referenced post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/118753), type 0 (electric) is in kBTU while the other sites are in kWh. Hence, type 0 should be updated to kWh. Regarding type 1 (Chilled), the unit for site 0 is kBTU while for sites 2 and 15 is Tons (1 ton (refrigeration) = 12000 Btu (IT)/hour).", "votes": null }, { "id": "692909", "postDate": "12/11/2019 20:56:27", "content": "So, this only affect 'train.csv'? Is that correct? Please confirm the meter reading units from original and desired. What then is the 'test.csv' supposed to predict in 'kWh' or 'bBTU'??
", "rawMarkdown": "So, this only affect 'train.csv'? Is that correct? Please confirm the meter reading units from original and desired. What then is the 'test.csv' supposed to predict in 'kWh' or 'bBTU'??", "votes": null }, { "id": "693904", "postDate": "12/12/2019 23:13:59", "content": "Interesting, mine did not go worse or better, pretty much the same. From the CV I see they're different on the 5th digit
", "rawMarkdown": "Interesting, mine did not go worse or better, pretty much the same. From the CV I see they're different on the 5th digit", "votes": null }, { "id": "696488", "postDate": "12/16/2019 17:00:01", "content": "come on!
", "rawMarkdown": "come on!", "votes": null }, { "id": "697547", "postDate": "12/18/2019 04:18:13", "content": "Evaluation Exception: Submission must have 41697600 row
", "rawMarkdown": "Evaluation Exception: Submission must have 41697600 row", "votes": null } ], "comments": [ { "id": 683474, "author_name": "larryd87", "author_url": "", "post_date": "11/28/2019 13:17:52", "content": "Am I right that if you do modelling building-wise or site-wise this changes essentially nothing?
", "votes": null, "replies": [ { "id": 684102, "author_name": "diginfo", "author_url": "", "post_date": "11/29/2019 08:31:14", "content": "Think so, yes. I you have modeling elements that do cross site comparisons, you might have a problem
", "votes": null, "replies": [] }, { "id": 691799, "author_name": "diginfo", "author_url": "", "post_date": "12/10/2019 14:05:31", "content": "Actually, buildingwise no, since it only applies to electric meters, absolute values within a building between meters will be affected. So, meter wise model requires no handling, building-wise does, IF you use relation between meters within building =)
", "votes": null, "replies": [] } ] }, { "id": 683809, "author_name": "eastashk", "author_url": "", "post_date": "11/28/2019 20:54:40", "content": "Could you please explain why it's necessary to multiply by 3.14118 for scoring? Am I right that the evaluation is in kWh?
", "votes": null, "replies": [ { "id": 684101, "author_name": "diginfo", "author_url": "", "post_date": "11/29/2019 08:30:12", "content": "I assume it is because the same error is in the test data as well, i.e. for site 0 are values are in kBTU unit in both train and test data.\nHence, if we convert train data to kWh, we need to convert back in our predictions.
", "votes": null, "replies": [] }, { "id": 687297, "author_name": "ydhiman", "author_url": "", "post_date": "12/04/2019 07:38:01", "content": "Martin Is following what you want to say if I want to do a code correction?
\n\ndef correct_error_meter0_model_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*0.2931) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading
\n\ndef correct_error_meter0_scoring_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*3.4118) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading
\n\nSo, to summaries, we will use correct_error_meter0_model_use for our train and valudation dataset and will use and will use function **correct_error_meter0_scoring_use ** function for our outcome in validation
", "votes": null, "replies": [] }, { "id": 691797, "author_name": "diginfo", "author_url": "", "post_date": "12/10/2019 14:02:28", "content": "yes, that's it.\nI did it like so:\ntrain.loc[(train['site_id'] == 0) & (train['meter'] == 0), 'meter_reading'] = train[(train['site_id'] == 0) & (train['meter'] == 0)]['meter_reading'] * 0.2931
\n\nSame same
", "votes": null, "replies": [] } ] }, { "id": 685189, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/01/2019 05:38:02", "content": "When I read your message I understand that something wrong on site_0 - but a tiny bit confused.
\n\nOption 1 - readings for meter types 0 and 1 are all wrong.\nor\nOption 2 - only readings for meter type 0 are wrong.
\n\nI think your saying Option 2 - is that correct?
\n\nIf it's option 1 - they were both wrong - than what conversion is needed.
\n\nWhile the data description indicated that all 4 meters were in kwh - the information from the leak sites does not seem to support that - certainly looks like the meters are in different units - lbs, tons, whatever.
\n\nRather than simply pointing to the data description (which has now proved to be wrong) it would be nice of you did a deep dive on your data and supply a more convincing statement that all sites have been checked and all meter readings (expect site 0) are in kWh. Because I think I see many that make no sense if truly kWh.
", "votes": null, "replies": [ { "id": 685633, "author_name": "mipang", "author_url": "", "post_date": "12/02/2019 03:02:42", "content": "It seems like readings for meter type 1 are wrong, isn't it?
", "votes": null, "replies": [] }, { "id": 686170, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/02/2019 21:08:05", "content": "yes - it seems to me that type 1 is wrong and type 0 is ok - but he used the word \"electric\" which I think is type 0.
", "votes": null, "replies": [] }, { "id": 692825, "author_name": "wittmannf", "author_url": "", "post_date": "12/11/2019 19:12:24", "content": "\n\n\ntype 0 is ok
\n
Nope. Based on the table of the referenced post, type 0 (electric) is in kBTU while the other sites are in kWh. Hence, type 0 should be updated to kWh. Regarding type 1 (Chilled), the unit for site 0 is kBTU while for sites 2 and 15 is Tons (1 ton (refrigeration) = 12000 Btu (IT)/hour).
", "votes": null, "replies": [] } ] }, { "id": 686631, "author_name": "danofer", "author_url": "", "post_date": "12/03/2019 11:07:01", "content": "I suggest adding this to the \"data\" page's description for visibility. This is a big change.
", "votes": null, "replies": [] }, { "id": 688714, "author_name": "djongo", "author_url": "", "post_date": "12/06/2019 01:04:29", "content": "I multiply 0.2931 the electric meter readings for site 0 and then log1p before tranning, exem1 and then multiply 3.4118 before submit, but get very worse result, what's wrong with my understanding?
", "votes": null, "replies": [ { "id": 688836, "author_name": "shahules", "author_url": "", "post_date": "12/06/2019 06:39:43", "content": "@djongo did you consider the meter type? He refers to type 0 as of my understanding.
", "votes": null, "replies": [] }, { "id": 688979, "author_name": "djongo", "author_url": "", "post_date": "12/06/2019 10:01:53", "content": "Yes， I considered the electric meter.
", "votes": null, "replies": [] }, { "id": 693904, "author_name": "lftuwujie", "author_url": "", "post_date": "12/12/2019 23:13:59", "content": "Interesting, mine did not go worse or better, pretty much the same. From the CV I see they're different on the 5th digit
", "votes": null, "replies": [] } ] }, { "id": 692909, "author_name": "sumpatds", "author_url": "", "post_date": "12/11/2019 20:56:27", "content": "So, this only affect 'train.csv'? Is that correct? Please confirm the meter reading units from original and desired. What then is the 'test.csv' supposed to predict in 'kWh' or 'bBTU'??
", "votes": null, "replies": [] }, { "id": 696488, "author_name": "tzozeze", "author_url": "", "post_date": "12/16/2019 17:00:01", "content": "come on!
", "votes": null, "replies": [] }, { "id": 697547, "author_name": "amitsorde", "author_url": "", "post_date": "12/18/2019 04:18:13", "content": "Evaluation Exception: Submission must have 41697600 row
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "682635": "Due to a miscommunication, the electric meter readings for site 0 were not properly converted to units of kWh and are in kBTU. We apologize for the error and the resulting inconvenience. \n\nThe units of the electric meter readings for site 0 will not be updated (i.e., they will remain as is), since this could potentially disrupt the workflows and models of many teams. For models that are sensitive to units/absolute values, the issue can be avoided with the right conversion factors. Multiply by 0.2931 to get to model inputs into kWh like the other sites, and 3.4118 to get back to kBTU for scoring.\n\nThanks to @teeyee314 for [catching the issue](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/118753).", "683474": "Am I right that if you do modelling building-wise or site-wise this changes essentially nothing?", "683809": "Could you please explain why it's necessary to multiply by 3.14118 for scoring? Am I right that the evaluation is in kWh?", "684101": "I assume it is because the same error is in the test data as well, i.e. for site 0 are values are in kBTU unit in both train and test data.\nHence, if we convert train data to kWh, we need to convert back in our predictions.", "684102": "Think so, yes. I you have modeling elements that do cross site comparisons, you might have a problem", "685189": "When I read your message I understand that something wrong on site_0 - but a tiny bit confused.\n\nOption 1 - readings for meter types 0 and 1 are all wrong.\nor\nOption 2 - only readings for meter type 0 are wrong.\n\nI think your saying Option 2 - is that correct?\n\n\nIf it's option 1 - they were both wrong - than what conversion is needed. \n\nWhile the data description indicated that all 4 meters were in kwh - the information from the leak sites does not seem to support that - certainly looks like the meters are in different units - lbs, tons, whatever. \n\nRather than simply pointing to the data description (which has now proved to be wrong) it would be nice of you did a deep dive on your data and supply a more convincing statement that all sites have been checked and all meter readings (expect site 0) are in kWh. Because I think I see many that make no sense if truly kWh.", "685633": "It seems like readings for meter type 1 are wrong, isn't it?", "686170": "yes - it seems to me that type 1 is wrong and type 0 is ok - but he used the word \"electric\" which I think is type 0.", "686631": "I suggest adding this to the \"data\" page's description for visibility. This is a big change.", "687297": "Martin Is following what you want to say if I want to do a code correction?\n\ndef correct_error_meter0_model_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*0.2931) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading\n\ndef correct_error_meter0_scoring_use(meter_reading,meter_type):\n meter_reading = (meter_reading*(meter_type == 0).astype(int)*3.4118) + (meter_reading+(meter_type != 0).astype(int))\n return meter_reading\n\nSo, to summaries, we will use **correct_error_meter0_model_use** for our train and valudation dataset and will use and will use function **correct_error_meter0_scoring_use ** function for our outcome in validation", "688714": "I multiply 0.2931 the electric meter readings for site 0 and then log1p before tranning, exem1 and then multiply 3.4118 before submit, but get very worse result, what's wrong with my understanding?", "688836": "djongo did you consider the meter type? He refers to type 0 as of my understanding.", "688979": "Yes， I considered the electric meter.", "691797": "yes, that's it.\nI did it like so:\ntrain.loc[(train['site_id'] == 0) & (train['meter'] == 0), 'meter_reading'] = train[(train['site_id'] == 0) & (train['meter'] == 0)]['meter_reading'] * 0.2931\n\nSame same", "691799": "Actually, buildingwise no, since it only applies to electric meters, absolute values within a building between meters will be affected. So, meter wise model requires no handling, building-wise does, IF you use relation between meters within building =)", "692825": "> type 0 is ok\n\nNope. Based on the table of the [referenced post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/118753), type 0 (electric) is in kBTU while the other sites are in kWh. Hence, type 0 should be updated to kWh. Regarding type 1 (Chilled), the unit for site 0 is kBTU while for sites 2 and 15 is Tons (1 ton (refrigeration) = 12000 Btu (IT)/hour).", "692909": "So, this only affect 'train.csv'? Is that correct? Please confirm the meter reading units from original and desired. What then is the 'test.csv' supposed to predict in 'kWh' or 'bBTU'??", "693904": "Interesting, mine did not go worse or better, pretty much the same. From the CV I see they're different on the 5th digit", "696488": "come on!", "697547": "Evaluation Exception: Submission must have 41697600 row" }, "source": "meta" }