{ "id": 115256, "title": "Site specific holidays", "url": "/competitions/ashrae-energy-prediction/discussion/115256", "author_name": "Gunes Evitan", "post_date": "2019-11-01T11:03:28.131000", "votes": 71, "comment_count": 11, "views": 0, "content": "

https://github.com/dr-prodigy/python-holidays

\n\n

I found this package which is very convenient for finding holidays with timestamps. Sadly, there is no cv score improvement from the feature.

\n\n

```\nimport holidays

\n\n

en_holidays = holidays.England()\nir_holidays = holidays.Ireland()\nca_holidays = holidays.Canada()\nus_holidays = holidays.UnitedStates()

\n\n

en_idx = df_weather.query('site_id == 1 or site_id == 5').index\nir_idx = df_weather.query('site_id == 12').index\nca_idx = df_weather.query('site_id == 7 or site_id == 11').index\nus_idx = df_weather.query('site_id == 0 or site_id == 2 or site_id == 3 or site_id == 4 or site_id == 6 or site_id == 8 or site_id == 9 or site_id == 10 or site_id == 13 or site_id == 14 or site_id == 15').index

\n\n

df_weather['IsHoliday'] = 0\ndf_weather.loc[en_idx, 'IsHoliday'] = df_weather.loc[en_idx, 'timestamp'].apply(lambda x: en_holidays.get(x, default=0))\ndf_weather.loc[ir_idx, 'IsHoliday'] = df_weather.loc[ir_idx, 'timestamp'].apply(lambda x: ir_holidays.get(x, default=0))\ndf_weather.loc[ca_idx, 'IsHoliday'] = df_weather.loc[ca_idx, 'timestamp'].apply(lambda x: ca_holidays.get(x, default=0))\ndf_weather.loc[us_idx, 'IsHoliday'] = df_weather.loc[us_idx, 'timestamp'].apply(lambda x: us_holidays.get(x, default=0))

\n\n

holiday_idx = df_weather['IsHoliday'] != 0\ndf_weather.loc[holiday_idx, 'IsHoliday'] = 1\ndf_weather['IsHoliday'] = df_weather['IsHoliday'].astype(np.uint8)\n```

", "messages": [ { "id": 663029, "postDate": "2019-11-01T11:03:28.130Z", "content": "

https://github.com/dr-prodigy/python-holidays

\n\n

I found this package which is very convenient for finding holidays with timestamps. Sadly, there is no cv score improvement from the feature.

\n\n

```\nimport holidays

\n\n

en_holidays = holidays.England()\nir_holidays = holidays.Ireland()\nca_holidays = holidays.Canada()\nus_holidays = holidays.UnitedStates()

\n\n

en_idx = df_weather.query('site_id == 1 or site_id == 5').index\nir_idx = df_weather.query('site_id == 12').index\nca_idx = df_weather.query('site_id == 7 or site_id == 11').index\nus_idx = df_weather.query('site_id == 0 or site_id == 2 or site_id == 3 or site_id == 4 or site_id == 6 or site_id == 8 or site_id == 9 or site_id == 10 or site_id == 13 or site_id == 14 or site_id == 15').index

\n\n

df_weather['IsHoliday'] = 0\ndf_weather.loc[en_idx, 'IsHoliday'] = df_weather.loc[en_idx, 'timestamp'].apply(lambda x: en_holidays.get(x, default=0))\ndf_weather.loc[ir_idx, 'IsHoliday'] = df_weather.loc[ir_idx, 'timestamp'].apply(lambda x: ir_holidays.get(x, default=0))\ndf_weather.loc[ca_idx, 'IsHoliday'] = df_weather.loc[ca_idx, 'timestamp'].apply(lambda x: ca_holidays.get(x, default=0))\ndf_weather.loc[us_idx, 'IsHoliday'] = df_weather.loc[us_idx, 'timestamp'].apply(lambda x: us_holidays.get(x, default=0))

\n\n

holiday_idx = df_weather['IsHoliday'] != 0\ndf_weather.loc[holiday_idx, 'IsHoliday'] = 1\ndf_weather['IsHoliday'] = df_weather['IsHoliday'].astype(np.uint8)\n```

", "rawMarkdown": "https://github.com/dr-prodigy/python-holidays\n\nI found this package which is very convenient for finding holidays with timestamps. Sadly, there is no cv score improvement from the feature.\n\n```\nimport holidays\n\nen_holidays = holidays.England()\nir_holidays = holidays.Ireland()\nca_holidays = holidays.Canada()\nus_holidays = holidays.UnitedStates()\n\nen_idx = df_weather.query('site_id == 1 or site_id == 5').index\nir_idx = df_weather.query('site_id == 12').index\nca_idx = df_weather.query('site_id == 7 or site_id == 11').index\nus_idx = df_weather.query('site_id == 0 or site_id == 2 or site_id == 3 or site_id == 4 or site_id == 6 or site_id == 8 or site_id == 9 or site_id == 10 or site_id == 13 or site_id == 14 or site_id == 15').index\n\ndf_weather['IsHoliday'] = 0\ndf_weather.loc[en_idx, 'IsHoliday'] = df_weather.loc[en_idx, 'timestamp'].apply(lambda x: en_holidays.get(x, default=0))\ndf_weather.loc[ir_idx, 'IsHoliday'] = df_weather.loc[ir_idx, 'timestamp'].apply(lambda x: ir_holidays.get(x, default=0))\ndf_weather.loc[ca_idx, 'IsHoliday'] = df_weather.loc[ca_idx, 'timestamp'].apply(lambda x: ca_holidays.get(x, default=0))\ndf_weather.loc[us_idx, 'IsHoliday'] = df_weather.loc[us_idx, 'timestamp'].apply(lambda x: us_holidays.get(x, default=0))\n\nholiday_idx = df_weather['IsHoliday'] != 0\ndf_weather.loc[holiday_idx, 'IsHoliday'] = 1\ndf_weather['IsHoliday'] = df_weather['IsHoliday'].astype(np.uint8)\n```", "votes": 71 }, { "id": 663560, "postDate": "2019-11-02T09:53:31.927Z", "content": "

The impact of any magic with holidays, besides maybe Christmas/NYE but for this one needs no calendars, is likely to be marginal. Firstly there are buildings which seem to be fully immune to any holidays, secondly we are talking at best like 12 observations per holiday and then we have like 10-15 holidays per year outside of regular weekend/weekday pattern.\nMy point I would not waste time on playing with holidays at this stage of the comp, even though it is easier to dig in this direction then to figure out how to actually do CV in this comp :)

", "rawMarkdown": "The impact of any magic with holidays, besides maybe Christmas/NYE but for this one needs no calendars, is likely to be marginal. Firstly there are buildings which seem to be fully immune to any holidays, secondly we are talking at best like 12 observations per holiday and then we have like 10-15 holidays per year outside of regular weekend/weekday pattern.\nMy point I would not waste time on playing with holidays at this stage of the comp, even though it is easier to dig in this direction then to figure out how to actually do CV in this comp :)", "votes": 8, "replies": [ { "id": 663561, "postDate": "2019-11-02T09:54:39.523Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 663682, "postDate": "2019-11-02T14:19:40.560Z", "content": "

I've seen a small improvement encoding the holiday string as a category. Additionally I've seen a bigger improvement removing training rows that could potentially be a holiday altogether.

", "rawMarkdown": "I've seen a small improvement encoding the holiday string as a category. Additionally I've seen a bigger improvement removing training rows that could potentially be a holiday altogether.", "votes": 5, "replies": [ { "id": 663943, "postDate": "2019-11-02T23:04:18.073Z", "content": "

@rcpeters, so you removed all the rows where \"IsHoliday\"=1 and got significant improvement? By how much if you don't mind my asking? That is interesting.

", "rawMarkdown": "@rcpeters, so you removed all the rows where \"IsHoliday\"=1 and got significant improvement? By how much if you don't mind my asking? That is interesting." }, { "id": 664636, "postDate": "2019-11-04T02:10:37.050Z", "content": "

At the time my score went from 1.25 to 1.24 but with only two decimal places who knows.

", "rawMarkdown": "At the time my score went from 1.25 to 1.24 but with only two decimal places who knows. ", "votes": 1 } ] }, { "id": 674087, "postDate": "2019-11-15T21:55:03.780Z", "content": "

You should probably share this on the external data disclosure thread, even if you don't find improvement.

", "rawMarkdown": "You should probably share this on the external data disclosure thread, even if you don't find improvement.", "votes": 1 }, { "id": 666536, "postDate": "2019-11-06T07:36:50.313Z", "content": "

Gunes, thank you for this post with holidays in all 4 countries.\nIn a similar post Konstantin claims 0.02 improvement. I'd expect even better results with Ireland, UK and Canada added. Why is this not the case?

", "rawMarkdown": "Gunes, thank you for this post with holidays in all 4 countries.\nIn a [similar post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113286) Konstantin claims 0.02 improvement. I'd expect even better results with Ireland, UK and Canada added. Why is this not the case?", "votes": 1, "replies": [ { "id": 667323, "postDate": "2019-11-07T05:52:20.990Z", "content": "

I believe Canada, UK and Ireland sites have less buildings than US sites. When we remove those timestamps, the data becomes even less and score doesn't improve.

", "rawMarkdown": "I believe Canada, UK and Ireland sites have less buildings than US sites. When we remove those timestamps, the data becomes even less and score doesn't improve." } ] }, { "id": 669944, "postDate": "2019-11-10T18:20:15.880Z", "content": "

As said and shown by different kagglers, there are mainly 4 countries US, Canada, England and Ireland in the dataset. And I have seen there are dicussions about F and C in air temperature but when I checked them empirically I found that Canada and US both have Celsius air temperature. Can anyone enlighten me? Thanks a lot.

", "rawMarkdown": "As said and shown by different kagglers, there are mainly 4 countries US, Canada, England and Ireland in the dataset. And I have seen there are dicussions about F and C in air temperature but when I checked them empirically I found that Canada and US both have Celsius air temperature. Can anyone enlighten me? Thanks a lot." }, { "id": 671174, "postDate": "2019-11-12T10:56:06.610Z", "rawMarkdown": "", "isDeleted": true }, { "id": 663050, "postDate": "2019-11-01T11:54:28.900Z", "content": "

Thanks for sharing @gunesevitan.

", "rawMarkdown": "Thanks for sharing @gunesevitan." } ], "comments": [ { "id": 663560, "author_name": "snovik", "author_url": "", "post_date": "2019-11-02T09:53:31.927000", "content": "

The impact of any magic with holidays, besides maybe Christmas/NYE but for this one needs no calendars, is likely to be marginal. Firstly there are buildings which seem to be fully immune to any holidays, secondly we are talking at best like 12 observations per holiday and then we have like 10-15 holidays per year outside of regular weekend/weekday pattern.\nMy point I would not waste time on playing with holidays at this stage of the comp, even though it is easier to dig in this direction then to figure out how to actually do CV in this comp :)

", "votes": 8, "replies": [ { "id": 663561, "author_name": "", "author_url": "", "post_date": "2019-11-02T09:54:39.523000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 663682, "author_name": "rcpeters", "author_url": "", "post_date": "2019-11-02T14:19:40.560000", "content": "

I've seen a small improvement encoding the holiday string as a category. Additionally I've seen a bigger improvement removing training rows that could potentially be a holiday altogether.

", "votes": 5, "replies": [ { "id": 663943, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-02T23:04:18.073000", "content": "

@rcpeters, so you removed all the rows where \"IsHoliday\"=1 and got significant improvement? By how much if you don't mind my asking? That is interesting.

", "votes": 0, "replies": [] }, { "id": 664636, "author_name": "rcpeters", "author_url": "", "post_date": "2019-11-04T02:10:37.050000", "content": "

At the time my score went from 1.25 to 1.24 but with only two decimal places who knows.

", "votes": 1, "replies": [] } ] }, { "id": 674087, "author_name": "Rob Rose", "author_url": "", "post_date": "2019-11-15T21:55:03.780000", "content": "

You should probably share this on the external data disclosure thread, even if you don't find improvement.

", "votes": 1, "replies": [] }, { "id": 666536, "author_name": "Poe Dator", "author_url": "", "post_date": "2019-11-06T07:36:50.313000", "content": "

Gunes, thank you for this post with holidays in all 4 countries.\nIn a similar post Konstantin claims 0.02 improvement. I'd expect even better results with Ireland, UK and Canada added. Why is this not the case?

", "votes": 1, "replies": [ { "id": 667323, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2019-11-07T05:52:20.990000", "content": "

I believe Canada, UK and Ireland sites have less buildings than US sites. When we remove those timestamps, the data becomes even less and score doesn't improve.

", "votes": 0, "replies": [] } ] }, { "id": 669944, "author_name": "Mukul Sharma", "author_url": "", "post_date": "2019-11-10T18:20:15.880000", "content": "

As said and shown by different kagglers, there are mainly 4 countries US, Canada, England and Ireland in the dataset. And I have seen there are dicussions about F and C in air temperature but when I checked them empirically I found that Canada and US both have Celsius air temperature. Can anyone enlighten me? Thanks a lot.

", "votes": 0, "replies": [] }, { "id": 671174, "author_name": "", "author_url": "", "post_date": "2019-11-12T10:56:06.610000", "content": "", "votes": 0, "replies": [] }, { "id": 663050, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-01T11:54:28.900000", "content": "

Thanks for sharing @gunesevitan.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "663029": "https://github.com/dr-prodigy/python-holidays\n\nI found this package which is very convenient for finding holidays with timestamps. Sadly, there is no cv score improvement from the feature.\n\n```\nimport holidays\n\nen_holidays = holidays.England()\nir_holidays = holidays.Ireland()\nca_holidays = holidays.Canada()\nus_holidays = holidays.UnitedStates()\n\nen_idx = df_weather.query('site_id == 1 or site_id == 5').index\nir_idx = df_weather.query('site_id == 12').index\nca_idx = df_weather.query('site_id == 7 or site_id == 11').index\nus_idx = df_weather.query('site_id == 0 or site_id == 2 or site_id == 3 or site_id == 4 or site_id == 6 or site_id == 8 or site_id == 9 or site_id == 10 or site_id == 13 or site_id == 14 or site_id == 15').index\n\ndf_weather['IsHoliday'] = 0\ndf_weather.loc[en_idx, 'IsHoliday'] = df_weather.loc[en_idx, 'timestamp'].apply(lambda x: en_holidays.get(x, default=0))\ndf_weather.loc[ir_idx, 'IsHoliday'] = df_weather.loc[ir_idx, 'timestamp'].apply(lambda x: ir_holidays.get(x, default=0))\ndf_weather.loc[ca_idx, 'IsHoliday'] = df_weather.loc[ca_idx, 'timestamp'].apply(lambda x: ca_holidays.get(x, default=0))\ndf_weather.loc[us_idx, 'IsHoliday'] = df_weather.loc[us_idx, 'timestamp'].apply(lambda x: us_holidays.get(x, default=0))\n\nholiday_idx = df_weather['IsHoliday'] != 0\ndf_weather.loc[holiday_idx, 'IsHoliday'] = 1\ndf_weather['IsHoliday'] = df_weather['IsHoliday'].astype(np.uint8)\n```", "663560": "The impact of any magic with holidays, besides maybe Christmas/NYE but for this one needs no calendars, is likely to be marginal. Firstly there are buildings which seem to be fully immune to any holidays, secondly we are talking at best like 12 observations per holiday and then we have like 10-15 holidays per year outside of regular weekend/weekday pattern.\nMy point I would not waste time on playing with holidays at this stage of the comp, even though it is easier to dig in this direction then to figure out how to actually do CV in this comp :)", "663682": "I've seen a small improvement encoding the holiday string as a category. Additionally I've seen a bigger improvement removing training rows that could potentially be a holiday altogether.", "674087": "You should probably share this on the external data disclosure thread, even if you don't find improvement.", "666536": "Gunes, thank you for this post with holidays in all 4 countries.\nIn a [similar post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113286) Konstantin claims 0.02 improvement. I'd expect even better results with Ireland, UK and Canada added. Why is this not the case?", "669944": "As said and shown by different kagglers, there are mainly 4 countries US, Canada, England and Ireland in the dataset. And I have seen there are dicussions about F and C in air temperature but when I checked them empirically I found that Canada and US both have Celsius air temperature. Can anyone enlighten me? Thanks a lot.", "671174": "", "663050": "Thanks for sharing @gunesevitan." } }