![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2749718%2Fa58b8c52f3a372b5331cd9270d7ce3c3%2Foriginal.PNG?generation=1577536664878419&alt=media\")
It looks like data cleaning was the key to getting a good score. We mainly looked through most of the buildings individually and removed the outlier. The effect was mostly seen when you corrected the predictions with a value less than 1, since we mainly deleted values with 0 and raised the mean value of the prediction.
\n\nI think this strategy enabled us to take 2nd place in private (14th. Public, without leak site-4)
\n\nFrom the beginning, I have tried to make sure that this strategy does not backfire and did an experiment.
\n\nI took a building (building_id == 1000, meter == 0) that had no outlier. \n
If I train the model with this data and predict the same building, a cv: 0.15 and lb: 0.11 will result. With postprocessing> 1 or <1 lb gets worse. (Just to illustrate, that postprocess doesn't work for data without outlier)
\n\nHowever, if I have data with outliers (which I simply generated by setting month 2 to 0 in the train and testing month 11 to 0), a CV: 1.16 and LB: 1.255 come out
\n\nTRAIN:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2749718%2Fc571666e8d8d106ea7354ebb0242fdc7%2Ftrain_original.PNG?generation=1577537183326125&alt=media\")
TEST:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2749718%2F2b4b9d9d979db0dfda68790b460fd7a7%2Ftest_outlier.PNG?generation=1577537198520972&alt=media\")
With postprocess (* 0.94) slightly better LB: 1.23
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2749718%2F8b7978e06e3377c3e7ca053e1e979bb8%2Ftrain_clean.PNG?generation=1577537794017163&alt=media\")
At the end, if you remove the outlier from train, you get a better cv: 0.17 and LB: 1.19. With postprocess (* 0.92, which regulates the mean of prediction to train) even LB: 1.14
\n\nShout out to my team-mates @rohanrao, @berserker408 and @isanton.
", "messages": [ { "id": 705115, "postDate": "2019-12-28T13:06:11.923Z", "content": "It looks like data cleaning was the key to getting a good score. We mainly looked through most of the buildings individually and removed the outlier. The effect was mostly seen when you corrected the predictions with a value less than 1, since we mainly deleted values with 0 and raised the mean value of the prediction.
\n\nI think this strategy enabled us to take 2nd place in private (14th. Public, without leak site-4)
\n\nFrom the beginning, I have tried to make sure that this strategy does not backfire and did an experiment.
\n\nI took a building (building_id == 1000, meter == 0) that had no outlier. \n
If I train the model with this data and predict the same building, a cv: 0.15 and lb: 0.11 will result. With postprocessing> 1 or <1 lb gets worse. (Just to illustrate, that postprocess doesn't work for data without outlier)
\n\nHowever, if I have data with outliers (which I simply generated by setting month 2 to 0 in the train and testing month 11 to 0), a CV: 1.16 and LB: 1.255 come out
\n\nTRAIN:\n
TEST:\n
With postprocess (* 0.94) slightly better LB: 1.23
\n\n
At the end, if you remove the outlier from train, you get a better cv: 0.17 and LB: 1.19. With postprocess (* 0.92, which regulates the mean of prediction to train) even LB: 1.14
\n\nShout out to my team-mates @rohanrao, @berserker408 and @isanton.
", "rawMarkdown": "It looks like data cleaning was the key to getting a good score. We mainly looked through most of the buildings individually and removed the outlier. The effect was mostly seen when you corrected the predictions with a value less than 1, since we mainly deleted values with 0 and raised the mean value of the prediction.\n\nI think this strategy enabled us to take 2nd place in private (14th. Public, without leak site-4)\n\nFrom the beginning, I have tried to make sure that this strategy does not backfire and did an experiment. \n\n_____________________________________________________________________________________________________\nI took a building (building_id == 1000, meter == 0) that had no outlier. \n\n\n\nIf I train the model with this data and predict the same building, a cv: 0.15 and lb: 0.11 will result. With postprocessing> 1 or <1 lb gets worse. (Just to illustrate, that postprocess doesn't work for data without outlier)\n\n_____________________________________________________________________________________________________\n\nHowever, if I have data with outliers (which I simply generated by setting month 2 to 0 in the train and testing month 11 to 0), a CV: 1.16 and **LB: 1.255** come out\n\nTRAIN:\n\n\nTEST:\n\n\nWith postprocess (* 0.94) slightly better **LB: 1.23**\n_____________________________________________________________________________________________________\n\n\n\nAt the end, if you remove the outlier from train, you get a better cv: 0.17 and LB: 1.19. With postprocess (* 0.92, which regulates the mean of prediction to train) even **LB: 1.14**\n\nShout out to my team-mates @rohanrao, @berserker408 and @isanton.", "votes": 40 }, { "id": 717360, "postDate": "2020-01-13T03:54:15.167Z", "content": "That's an interesting find -- thanks for sharing!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "rawMarkdown": "That's an interesting find -- thanks for sharing! \n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "votes": 1 }, { "id": 710531, "postDate": "2020-01-04T21:27:46.623Z", "content": "Very interesting. By chance, did you look at modeling the energy consumption with a generalized additive model, such as Prophet? This could allow you to take the input model it and replace values with the GAM that were above a certain error threshold.
", "rawMarkdown": "Very interesting. By chance, did you look at modeling the energy consumption with a generalized additive model, such as Prophet? This could allow you to take the input model it and replace values with the GAM that were above a certain error threshold.", "votes": 1, "replies": [ { "id": 710689, "postDate": "2020-01-05T04:45:42.307Z", "content": "We did look at some time series models and I even published a skeletal kernel Prophet's Prophecy. The performance was not even close to boosting methods and didn't add any value whatsoever. So we dropped the approach completely.
\n\nThis dataset is very suitable for boosting models.
", "rawMarkdown": "We did look at some time series models and I even published a skeletal kernel [Prophet's Prophecy](https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy). The performance was not even close to boosting methods and didn't add any value whatsoever. So we dropped the approach completely.\n\nThis dataset is very suitable for boosting models.", "votes": 1 } ] }, { "id": 708510, "postDate": "2020-01-02T11:14:12.660Z", "content": "congratulations
", "rawMarkdown": "congratulations ", "votes": 1 }, { "id": 705386, "postDate": "2019-12-28T21:16:42.390Z", "content": "@oleg90777 Thanks for sharing. How did you compute the lb score? Did you use a hold-out set?
", "rawMarkdown": "@oleg90777 Thanks for sharing. How did you compute the lb score? Did you use a hold-out set?", "votes": 1, "replies": [ { "id": 705711, "postDate": "2019-12-29T10:28:18.930Z", "content": "We just multiplied the predictions with 0.80. That worked best for public. We selected two final submissions. One with large postprocess (*0.80) and one without. Actually the private score was almost the same for both, because postprocessing with around 0.9 was the best. So we didn't select the best submission and landed only on second place :D
", "rawMarkdown": "We just multiplied the predictions with 0.80. That worked best for public. We selected two final submissions. One with large postprocess (*0.80) and one without. Actually the private score was almost the same for both, because postprocessing with around 0.9 was the best. So we didn't select the best submission and landed only on second place :D", "votes": 3 }, { "id": 706532, "postDate": "2019-12-30T14:26:42.027Z", "content": "Congratulations @oleg90777 , great work!\nSame thing happened to me as well. To avoid this, is it better to not select any submissions at all and let Kaggle figure out the best submissions according to the scoring on private test set? \nThis is my first competition in Kaggle, I was not sure what to do so I selected two submissions to be 'safe'.
", "rawMarkdown": "Congratulations @oleg90777 , great work!\nSame thing happened to me as well. To avoid this, is it better to not select any submissions at all and let Kaggle figure out the best submissions according to the scoring on private test set? \nThis is my first competition in Kaggle, I was not sure what to do so I selected two submissions to be 'safe'.\n" }, { "id": 706567, "postDate": "2019-12-30T15:12:02.707Z", "content": "If you don't select two submissions, kaggle take two best submissions according to public score.
", "rawMarkdown": "If you don't select two submissions, kaggle take two best submissions according to public score. ", "votes": 1 } ] }, { "id": 705238, "postDate": "2019-12-28T17:19:20.767Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 705163, "postDate": "2019-12-28T15:07:58.933Z", "content": "Thank you for sharing. Congratulations :)
", "rawMarkdown": "Thank you for sharing. Congratulations :)", "votes": 1 }, { "id": 706466, "postDate": "2019-12-30T12:39:53.387Z", "content": "Very insightful! Thanks and congratulations with 2nd place.
", "rawMarkdown": "Very insightful! Thanks and congratulations with 2nd place.", "votes": 2 }, { "id": 705860, "postDate": "2019-12-29T15:35:24.357Z", "content": "@oleg90777 , congrats and thanks for sharing. My post processing value was 0.96 chosen after experimenting with a range of 0.96-1.0 now I wish I have tested a bigger window :(
", "rawMarkdown": "@oleg90777 , congrats and thanks for sharing. My post processing value was 0.96 chosen after experimenting with a range of 0.96-1.0 now I wish I have tested a bigger window :(", "votes": 2 }, { "id": 705119, "postDate": "2019-12-28T13:15:30.063Z", "content": "Congratulations\nThanks for sharing your Insights!! @oleg90777
", "rawMarkdown": "Congratulations\nThanks for sharing your Insights!! @oleg90777 " }, { "id": 716977, "postDate": "2020-01-12T14:42:29.020Z", "content": "Also @oleg90777, Since we are talking about Time Series Models, I am really curious to know that if at all there is no trend or patterns whatsoever present in the data then which model should be considered? At times I have faced this issue a lot on real world data that the data is so random without any tren, seasonal or cyclic component that most of the available models seems to fail. Kindly let me know your thoughts.
\n\nLooking forward to hear from you! 😊
", "rawMarkdown": "Also @oleg90777, Since we are talking about Time Series Models, I am really curious to know that if at all there is no trend or patterns whatsoever present in the data then which model should be considered? At times I have faced this issue a lot on real world data that the data is so random without any tren, seasonal or cyclic component that most of the available models seems to fail. Kindly let me know your thoughts. \n\nLooking forward to hear from you! 😊 " }, { "id": 716976, "postDate": "2020-01-12T14:41:50.477Z", "content": "Thanks @oleg90777 for Sharing this ! This is very insightful. Congratulations for holding the 2nd place :)
", "rawMarkdown": "Thanks @oleg90777 for Sharing this ! This is very insightful. Congratulations for holding the 2nd place :)" }, { "id": 705368, "postDate": "2019-12-28T20:40:26.427Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 712183, "postDate": "2020-01-06T23:11:35.810Z", "content": "Conglats, Thanks for sharing \"magic\".
", "rawMarkdown": "Conglats, Thanks for sharing \"magic\".", "votes": 1 } ], "comments": [ { "id": 717360, "author_name": "Clayton Miller", "author_url": "", "post_date": "2020-01-13T03:54:15.167000", "content": "That's an interesting find -- thanks for sharing!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "votes": 1, "replies": [] }, { "id": 710531, "author_name": "Adam Peretti", "author_url": "", "post_date": "2020-01-04T21:27:46.623000", "content": "Very interesting. By chance, did you look at modeling the energy consumption with a generalized additive model, such as Prophet? This could allow you to take the input model it and replace values with the GAM that were above a certain error threshold.
", "votes": 1, "replies": [ { "id": 710689, "author_name": "Vopani", "author_url": "", "post_date": "2020-01-05T04:45:42.307000", "content": "We did look at some time series models and I even published a skeletal kernel Prophet's Prophecy. The performance was not even close to boosting methods and didn't add any value whatsoever. So we dropped the approach completely.
\n\nThis dataset is very suitable for boosting models.
", "votes": 1, "replies": [] } ] }, { "id": 708510, "author_name": "Firat Gonen", "author_url": "", "post_date": "2020-01-02T11:14:12.660000", "content": "congratulations
", "votes": 1, "replies": [] }, { "id": 705386, "author_name": "Waylon Wu", "author_url": "", "post_date": "2019-12-28T21:16:42.390000", "content": "@oleg90777 Thanks for sharing. How did you compute the lb score? Did you use a hold-out set?
", "votes": 1, "replies": [ { "id": 705711, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-12-29T10:28:18.930000", "content": "We just multiplied the predictions with 0.80. That worked best for public. We selected two final submissions. One with large postprocess (*0.80) and one without. Actually the private score was almost the same for both, because postprocessing with around 0.9 was the best. So we didn't select the best submission and landed only on second place :D
", "votes": 3, "replies": [] }, { "id": 706532, "author_name": "Nono", "author_url": "", "post_date": "2019-12-30T14:26:42.027000", "content": "Congratulations @oleg90777 , great work!\nSame thing happened to me as well. To avoid this, is it better to not select any submissions at all and let Kaggle figure out the best submissions according to the scoring on private test set? \nThis is my first competition in Kaggle, I was not sure what to do so I selected two submissions to be 'safe'.
", "votes": 0, "replies": [] }, { "id": 706567, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-12-30T15:12:02.707000", "content": "If you don't select two submissions, kaggle take two best submissions according to public score.
", "votes": 1, "replies": [] } ] }, { "id": 705238, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2019-12-28T17:19:20.767000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 705163, "author_name": "Vishal Sharma", "author_url": "", "post_date": "2019-12-28T15:07:58.933000", "content": "Thank you for sharing. Congratulations :)
", "votes": 1, "replies": [] }, { "id": 706466, "author_name": "Carlo", "author_url": "", "post_date": "2019-12-30T12:39:53.387000", "content": "Very insightful! Thanks and congratulations with 2nd place.
", "votes": 2, "replies": [] }, { "id": 705860, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-12-29T15:35:24.357000", "content": "@oleg90777 , congrats and thanks for sharing. My post processing value was 0.96 chosen after experimenting with a range of 0.96-1.0 now I wish I have tested a bigger window :(
", "votes": 2, "replies": [] }, { "id": 705119, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-12-28T13:15:30.063000", "content": "Congratulations\nThanks for sharing your Insights!! @oleg90777
", "votes": 0, "replies": [] }, { "id": 716977, "author_name": "Risab Biswas", "author_url": "", "post_date": "2020-01-12T14:42:29.020000", "content": "Also @oleg90777, Since we are talking about Time Series Models, I am really curious to know that if at all there is no trend or patterns whatsoever present in the data then which model should be considered? At times I have faced this issue a lot on real world data that the data is so random without any tren, seasonal or cyclic component that most of the available models seems to fail. Kindly let me know your thoughts.
\n\nLooking forward to hear from you! 😊
", "votes": 0, "replies": [] }, { "id": 716976, "author_name": "Risab Biswas", "author_url": "", "post_date": "2020-01-12T14:41:50.477000", "content": "Thanks @oleg90777 for Sharing this ! This is very insightful. Congratulations for holding the 2nd place :)
", "votes": 0, "replies": [] }, { "id": 705368, "author_name": "", "author_url": "", "post_date": "2019-12-28T20:40:26.427000", "content": "", "votes": 1, "replies": [] }, { "id": 712183, "author_name": "corochann", "author_url": "", "post_date": "2020-01-06T23:11:35.810000", "content": "Conglats, Thanks for sharing \"magic\".
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "705115": "It looks like data cleaning was the key to getting a good score. We mainly looked through most of the buildings individually and removed the outlier. The effect was mostly seen when you corrected the predictions with a value less than 1, since we mainly deleted values with 0 and raised the mean value of the prediction.\n\nI think this strategy enabled us to take 2nd place in private (14th. Public, without leak site-4)\n\nFrom the beginning, I have tried to make sure that this strategy does not backfire and did an experiment. \n\n_____________________________________________________________________________________________________\nI took a building (building_id == 1000, meter == 0) that had no outlier. \n\n\n\nIf I train the model with this data and predict the same building, a cv: 0.15 and lb: 0.11 will result. With postprocessing> 1 or <1 lb gets worse. (Just to illustrate, that postprocess doesn't work for data without outlier)\n\n_____________________________________________________________________________________________________\n\nHowever, if I have data with outliers (which I simply generated by setting month 2 to 0 in the train and testing month 11 to 0), a CV: 1.16 and **LB: 1.255** come out\n\nTRAIN:\n\n\nTEST:\n\n\nWith postprocess (* 0.94) slightly better **LB: 1.23**\n_____________________________________________________________________________________________________\n\n\n\nAt the end, if you remove the outlier from train, you get a better cv: 0.17 and LB: 1.19. With postprocess (* 0.92, which regulates the mean of prediction to train) even **LB: 1.14**\n\nShout out to my team-mates @rohanrao, @berserker408 and @isanton.", "717360": "That's an interesting find -- thanks for sharing! \n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "710531": "Very interesting. By chance, did you look at modeling the energy consumption with a generalized additive model, such as Prophet? This could allow you to take the input model it and replace values with the GAM that were above a certain error threshold.", "708510": "congratulations ", "705386": "@oleg90777 Thanks for sharing. How did you compute the lb score? Did you use a hold-out set?", "705238": "Congratulations!", "705163": "Thank you for sharing. Congratulations :)", "706466": "Very insightful! Thanks and congratulations with 2nd place.", "705860": "@oleg90777 , congrats and thanks for sharing. My post processing value was 0.96 chosen after experimenting with a range of 0.96-1.0 now I wish I have tested a bigger window :(", "705119": "Congratulations\nThanks for sharing your Insights!! @oleg90777 ", "716977": "Also @oleg90777, Since we are talking about Time Series Models, I am really curious to know that if at all there is no trend or patterns whatsoever present in the data then which model should be considered? At times I have faced this issue a lot on real world data that the data is so random without any tren, seasonal or cyclic component that most of the available models seems to fail. Kindly let me know your thoughts. \n\nLooking forward to hear from you! 😊 ", "716976": "Thanks @oleg90777 for Sharing this ! This is very insightful. Congratulations for holding the 2nd place :)", "705368": "", "712183": "Conglats, Thanks for sharing \"magic\"." } }