Finally nice competition! Not easy due to really noisy data but new lessons learnt again. Some insights of my solution that reached place #9 and gold.
\n\nOne of my objectives was to survive to shake-up. As soon as leaked data were discovered I decided to use it mainly for hold-out validation. I started training without any leak data, playing with features engineering, different models, different time CV folds and drove my work only with correlated CV + hold-out + LB results. It was a bit frustrating to have very low ranking with this strategy but I knew it should help at the end. On the 2 last week I included 2017 leak data in training and kept 2018 leak data for hold-out.
\n\nMy solution is ensemble (ridge regression) of several models:\n- LightGBM (x7)\n- CatBoost (x4)\n- Neural Network with categories embedding and features standard normalization (x4)\n- LiteMORT (x1)
\n\nFeatures are quite simple, no model above 15 features, 12 as average:\n- building_id, meter, site_id, primary_use, week_day, is_holiday\n- square_feet, cloud coverage, precip_depth_1_h\n- feels like temperature, building age, square_feet * floor_count\n- air_temperature roll mean 24h, sea_level_pressure trend roll mean 24h\n- air_temperature cooling degree per day, meter_reading median per building per meter per year
Notice that some of my models are not using building_id to try to generalize better.\nA few models are per meter, others not.\nFor each model CV I applied different time split, x3, x4 and x6 .
\n\nCleansing and imputation was important too. For weather data, I tried to find different external sources to fill gaps but it did not give any boost mainly because we're not 100% sure of location of site_id and some data such as cloud coverage was not consistent with the ones in training data. Finally, I trained an additional simple LightGBM model to impute missing data based on provided data (similar to this kernel). For meter reading, cleansing was not obvious, removing zero patterns (electricity, hot water in summer ...) looked a good idea but one can notice that such pattern also appear in 2017/2018 leaked data. Some buildings could also have some renovations slots that would explain zero patterns. So each of my model had a zero-pattern drop strategy different (site_id = 0 before May 2016 only, full drop, partial drop based on duration and/or on season).
Post-processing: None and it was a mistake when I see top solutions.
\n\nFor final submission I selected the ones with best CV/Hold-Out/LB correlation and it was a good choice as they're the best score in Private LB!
\n\nWhat did not work:\n- Too many features lead to overfit especially with target encoding.\n- Tree-based second level model for ensembling (overfit again)\n- External data for weather (not in inline with provided data)\n- Non time-split CV (auto-correlation)
\n\nThanks to organizers, Kaggle and competitors for this challenge!
", "messages": [ { "id": "705094", "postDate": "12/28/2019 11:59:43", "content": "Finally nice competition! Not easy due to really noisy data but new lessons learnt again. Some insights of my solution that reached place #9 and gold.
\n\nOne of my objectives was to survive to shake-up. As soon as leaked data were discovered I decided to use it mainly for hold-out validation. I started training without any leak data, playing with features engineering, different models, different time CV folds and drove my work only with correlated CV + hold-out + LB results. It was a bit frustrating to have very low ranking with this strategy but I knew it should help at the end. On the 2 last week I included 2017 leak data in training and kept 2018 leak data for hold-out.
\n\nMy solution is ensemble (ridge regression) of several models:\n- LightGBM (x7)\n- CatBoost (x4)\n- Neural Network with categories embedding and features standard normalization (x4)\n- LiteMORT (x1)
\n\nFeatures are quite simple, no model above 15 features, 12 as average:\n- building_id, meter, site_id, primary_use, week_day, is_holiday\n- square_feet, cloud coverage, precip_depth_1_h\n- feels like temperature, building age, square_feet * floor_count\n- air_temperature roll mean 24h, sea_level_pressure trend roll mean 24h\n- air_temperature cooling degree per day, meter_reading median per building per meter per year
Notice that some of my models are not using building_id to try to generalize better.\nA few models are per meter, others not.\nFor each model CV I applied different time split, x3, x4 and x6 .
\n\nCleansing and imputation was important too. For weather data, I tried to find different external sources to fill gaps but it did not give any boost mainly because we're not 100% sure of location of site_id and some data such as cloud coverage was not consistent with the ones in training data. Finally, I trained an additional simple LightGBM model to impute missing data based on provided data (similar to this kernel). For meter reading, cleansing was not obvious, removing zero patterns (electricity, hot water in summer ...) looked a good idea but one can notice that such pattern also appear in 2017/2018 leaked data. Some buildings could also have some renovations slots that would explain zero patterns. So each of my model had a zero-pattern drop strategy different (site_id = 0 before May 2016 only, full drop, partial drop based on duration and/or on season).
Post-processing: None and it was a mistake when I see top solutions.
\n\nFor final submission I selected the ones with best CV/Hold-Out/LB correlation and it was a good choice as they're the best score in Private LB!
\n\nWhat did not work:\n- Too many features lead to overfit especially with target encoding.\n- Tree-based second level model for ensembling (overfit again)\n- External data for weather (not in inline with provided data)\n- Non time-split CV (auto-correlation)
\n\nThanks to organizers, Kaggle and competitors for this challenge!
", "rawMarkdown": "Finally nice competition! Not easy due to really noisy data but new lessons learnt again. Some insights of my solution that reached place #9 and gold.\n\nOne of my objectives was to survive to shake-up. As soon as leaked data were discovered I decided to use it mainly for hold-out validation. I started training without any leak data, playing with features engineering, different models, different time CV folds and drove my work only with **correlated CV + hold-out + LB results**. It was a bit frustrating to have very low ranking with this strategy but I knew it should help at the end. On the 2 last week I included 2017 leak data in training and kept 2018 leak data for hold-out.\n\nMy solution is **ensemble (ridge regression) of several models**:\n- LightGBM (x7)\n- CatBoost (x4)\n- Neural Network with [categories embedding](https://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15#685556) and features standard normalization (x4)\n- LiteMORT (x1)\n\n**Features are quite simple**, no model above 15 features, 12 as average:\n- `building_id`, `meter`, `site_id`, `primary_use`, `week_day`, `is_holiday`\n- `square_feet`, `cloud coverage`, `precip_depth_1_h`\n- [feels like](https://github.com/malexer/meteocalc) temperature, building age, `square_feet` * `floor_count`\n- `air_temperature` roll mean 24h, `sea_level_pressure` trend roll mean 24h\n- `air_temperature` [cooling degree](https://www.investopedia.com/terms/c/colddegreeday.asp) per day, `meter_reading` median per building per meter per year\n\nNotice that some of my models are not using building_id to try to generalize better.\nA few models are per meter, others not.\nFor each model CV I applied different time split, x3, x4 and x6 .\n\n**Cleansing and imputation was important too**. For weather data, I tried to find different external sources to fill gaps but it did not give any boost mainly because we're not 100% sure of location of `site_id` and some data such as cloud coverage was not consistent with the ones in training data. Finally, I trained an additional simple LightGBM model to impute missing data based on provided data (similar to this [kernel](https://www.kaggle.com/frednavruzov/nan-restoration-techniques-for-weather-data)). For meter reading, cleansing was not obvious, removing zero patterns (electricity, hot water in summer ...) looked a good idea but one can notice that such pattern also appear in 2017/2018 leaked data. Some buildings could also have some renovations slots that would explain zero patterns. So each of my model had a zero-pattern drop strategy different (`site_id` = 0 before May 2016 only, full drop, partial drop based on duration and/or on season).\n\n**Post-processing**: None and it was a mistake when I see top solutions.\n\nFor final submission I selected the ones with best CV/Hold-Out/LB correlation and it was a good choice as they're the best score in Private LB!\n\n**What did not work:**\n- Too many features lead to overfit especially with target encoding.\n- Tree-based second level model for ensembling (overfit again)\n- External data for weather (not in inline with provided data)\n- Non time-split CV (auto-correlation)\n\nThanks to organizers, Kaggle and competitors for this challenge!", "votes": null }, { "id": "705098", "postDate": "12/28/2019 12:13:25", "content": "Nice work! Forgot all about trying model based imputation.
", "rawMarkdown": "Nice work! Forgot all about trying model based imputation.", "votes": null }, { "id": "705120", "postDate": "12/28/2019 13:16:13", "content": "Congratulations\nThanks for sharing your Approach & Insights!! @mpware
", "rawMarkdown": "Congratulations\nThanks for sharing your Approach & Insights!! @mpware", "votes": null }, { "id": "712191", "postDate": "01/06/2020 23:29:05", "content": "Congrats, thank you for sharing.
", "rawMarkdown": "Congrats, thank you for sharing.", "votes": null }, { "id": "712978", "postDate": "01/07/2020 19:34:54", "content": "Thanks! \nAnd the private LB is official now!
", "rawMarkdown": "Thanks! \nAnd the private LB is official now!", "votes": null }, { "id": "713158", "postDate": "01/08/2020 01:15:43", "content": "Congrats @mpware and thanks for sharing.
", "rawMarkdown": "Congrats @mpware and thanks for sharing.", "votes": null }, { "id": "717357", "postDate": "01/13/2020 03:51:22", "content": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "rawMarkdown": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "votes": null }, { "id": "717907", "postDate": "01/13/2020 19:04:34", "content": "@claytonmiller Sure, you can add it and you can use info on my profile.\nI'm quite busy with another competition (deepfakes) so I won't add notebooks immediately.
", "rawMarkdown": "claytonmiller Sure, you can add it and you can use info on my profile.\nI'm quite busy with another competition (deepfakes) so I won't add notebooks immediately.", "votes": null }, { "id": "717916", "postDate": "01/13/2020 19:21:42", "content": "Felicitations!
", "rawMarkdown": "Felicitations!", "votes": null }, { "id": "717929", "postDate": "01/13/2020 19:51:11", "content": "Merci, t'as fait mieux que moi 😃
", "rawMarkdown": "Merci, t'as fait mieux que moi 😃", "votes": null }, { "id": "718218", "postDate": "01/14/2020 07:28:24", "content": "No worries -- add the kernels when you have a chance. Good luck in the Deep Fakes competition!
", "rawMarkdown": "No worries -- add the kernels when you have a chance. Good luck in the Deep Fakes competition!", "votes": null } ], "comments": [ { "id": 705098, "author_name": "teeyee314", "author_url": "", "post_date": "12/28/2019 12:13:25", "content": "Nice work! Forgot all about trying model based imputation.
", "votes": null, "replies": [] }, { "id": 705120, "author_name": "veeralakrishna", "author_url": "", "post_date": "12/28/2019 13:16:13", "content": "Congratulations\nThanks for sharing your Approach & Insights!! @mpware
", "votes": null, "replies": [] }, { "id": 712191, "author_name": "corochann", "author_url": "", "post_date": "01/06/2020 23:29:05", "content": "Congrats, thank you for sharing.
", "votes": null, "replies": [ { "id": 712978, "author_name": "mpware", "author_url": "", "post_date": "01/07/2020 19:34:54", "content": "Thanks! \nAnd the private LB is official now!
", "votes": null, "replies": [] } ] }, { "id": 713158, "author_name": "sheriytm", "author_url": "", "post_date": "01/08/2020 01:15:43", "content": "Congrats @mpware and thanks for sharing.
", "votes": null, "replies": [] }, { "id": 717357, "author_name": "claytonmiller", "author_url": "", "post_date": "01/13/2020 03:51:22", "content": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "votes": null, "replies": [ { "id": 717907, "author_name": "mpware", "author_url": "", "post_date": "01/13/2020 19:04:34", "content": "@claytonmiller Sure, you can add it and you can use info on my profile.\nI'm quite busy with another competition (deepfakes) so I won't add notebooks immediately.
", "votes": null, "replies": [] }, { "id": 718218, "author_name": "claytonmiller", "author_url": "", "post_date": "01/14/2020 07:28:24", "content": "No worries -- add the kernels when you have a chance. Good luck in the Deep Fakes competition!
", "votes": null, "replies": [] } ] }, { "id": 717916, "author_name": "chabir", "author_url": "", "post_date": "01/13/2020 19:21:42", "content": "Felicitations!
", "votes": null, "replies": [ { "id": 717929, "author_name": "mpware", "author_url": "", "post_date": "01/13/2020 19:51:11", "content": "Merci, t'as fait mieux que moi 😃
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "705094": "Finally nice competition! Not easy due to really noisy data but new lessons learnt again. Some insights of my solution that reached place #9 and gold.\n\nOne of my objectives was to survive to shake-up. As soon as leaked data were discovered I decided to use it mainly for hold-out validation. I started training without any leak data, playing with features engineering, different models, different time CV folds and drove my work only with **correlated CV + hold-out + LB results**. It was a bit frustrating to have very low ranking with this strategy but I knew it should help at the end. On the 2 last week I included 2017 leak data in training and kept 2018 leak data for hold-out.\n\nMy solution is **ensemble (ridge regression) of several models**:\n- LightGBM (x7)\n- CatBoost (x4)\n- Neural Network with [categories embedding](https://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15#685556) and features standard normalization (x4)\n- LiteMORT (x1)\n\n**Features are quite simple**, no model above 15 features, 12 as average:\n- `building_id`, `meter`, `site_id`, `primary_use`, `week_day`, `is_holiday`\n- `square_feet`, `cloud coverage`, `precip_depth_1_h`\n- [feels like](https://github.com/malexer/meteocalc) temperature, building age, `square_feet` * `floor_count`\n- `air_temperature` roll mean 24h, `sea_level_pressure` trend roll mean 24h\n- `air_temperature` [cooling degree](https://www.investopedia.com/terms/c/colddegreeday.asp) per day, `meter_reading` median per building per meter per year\n\nNotice that some of my models are not using building_id to try to generalize better.\nA few models are per meter, others not.\nFor each model CV I applied different time split, x3, x4 and x6 .\n\n**Cleansing and imputation was important too**. For weather data, I tried to find different external sources to fill gaps but it did not give any boost mainly because we're not 100% sure of location of `site_id` and some data such as cloud coverage was not consistent with the ones in training data. Finally, I trained an additional simple LightGBM model to impute missing data based on provided data (similar to this [kernel](https://www.kaggle.com/frednavruzov/nan-restoration-techniques-for-weather-data)). For meter reading, cleansing was not obvious, removing zero patterns (electricity, hot water in summer ...) looked a good idea but one can notice that such pattern also appear in 2017/2018 leaked data. Some buildings could also have some renovations slots that would explain zero patterns. So each of my model had a zero-pattern drop strategy different (`site_id` = 0 before May 2016 only, full drop, partial drop based on duration and/or on season).\n\n**Post-processing**: None and it was a mistake when I see top solutions.\n\nFor final submission I selected the ones with best CV/Hold-Out/LB correlation and it was a good choice as they're the best score in Private LB!\n\n**What did not work:**\n- Too many features lead to overfit especially with target encoding.\n- Tree-based second level model for ensembling (overfit again)\n- External data for weather (not in inline with provided data)\n- Non time-split CV (auto-correlation)\n\nThanks to organizers, Kaggle and competitors for this challenge!", "705098": "Nice work! Forgot all about trying model based imputation.", "705120": "Congratulations\nThanks for sharing your Approach & Insights!! @mpware", "712191": "Congrats, thank you for sharing.", "712978": "Thanks! \nAnd the private LB is official now!", "713158": "Congrats @mpware and thanks for sharing.", "717357": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "717907": "claytonmiller Sure, you can add it and you can use info on my profile.\nI'm quite busy with another competition (deepfakes) so I won't add notebooks immediately.", "717916": "Felicitations!", "717929": "Merci, t'as fait mieux que moi 😃", "718218": "No worries -- add the kernels when you have a chance. Good luck in the Deep Fakes competition!" }, "source": "existing_discussion", "complete": true, "competition_id": 9994, "competition_slug": "ashrae-energy-prediction", "team_id": 3907238, "solution_url": "/competitions/ashrae-energy-prediction/writeups/mpware-9th-place-solution" }