Hey there!\nSince the amount of data leakage is constantly growing I wonder how to deal with it...\nThe LB seems to be quite useless by now, however just replacing values in your submission won't give you any boost in the private LB, as the authors of this competition announced to exclude the leaked data from the final score.
\n\nHas anybody tried to replace missing data in the training set (year 2016) or use the leaked 2017 data in training to boost the performance of other sides?
\n\nAny other idea is also welcome :)
", "messages": [ { "id": "677992", "postDate": "11/20/2019 21:20:13", "content": "Hey there!\nSince the amount of data leakage is constantly growing I wonder how to deal with it...\nThe LB seems to be quite useless by now, however just replacing values in your submission won't give you any boost in the private LB, as the authors of this competition announced to exclude the leaked data from the final score.
\n\nHas anybody tried to replace missing data in the training set (year 2016) or use the leaked 2017 data in training to boost the performance of other sides?
\n\nAny other idea is also welcome :)
", "rawMarkdown": "Hey there!\nSince the amount of data leakage is constantly growing I wonder how to deal with it...\nThe LB seems to be quite useless by now, however just replacing values in your submission won't give you any boost in the private LB, as the authors of this competition announced to exclude the leaked data from the final score.\n\nHas anybody tried to replace missing data in the training set (year 2016) or use the leaked 2017 data in training to boost the performance of other sides? \n\nAny other idea is also welcome :)", "votes": null }, { "id": "677997", "postDate": "11/20/2019 21:33:14", "content": "Fill with disclosed useless predictions will show you clearer improvement of the prediction of your model when you evaluate the model with still valid predictions by checking LB.
", "rawMarkdown": "Fill with disclosed useless predictions will show you clearer improvement of the prediction of your model when you evaluate the model with still valid predictions by checking LB.", "votes": null }, { "id": "678030", "postDate": "11/20/2019 23:00:19", "content": "Thanks! You're right using the ground truth labels will show us clearer improvement on the data that is not leaked... \nI'm still interested if we can use the data in a different way to improve our models on other sides :)
", "rawMarkdown": "Thanks! You're right using the ground truth labels will show us clearer improvement on the data that is not leaked... \nI'm still interested if we can use the data in a different way to improve our models on other sides :)", "votes": null }, { "id": "678220", "postDate": "11/21/2019 05:29:38", "content": "Exactly, I am also wondering how can we use the leaked data without overfitting the model? Suggestions are welcome.
", "rawMarkdown": "Exactly, I am also wondering how can we use the leaked data without overfitting the model? Suggestions are welcome.", "votes": null }, { "id": "678298", "postDate": "11/21/2019 07:47:56", "content": "If organizers had provided us with 15 sites - all of them having 2016 data and 5 of them have 2016 to 2018 data what would you do?
", "rawMarkdown": "If organizers had provided us with 15 sites - all of them having 2016 data and 5 of them have 2016 to 2018 data what would you do?", "votes": null }, { "id": "678403", "postDate": "11/21/2019 10:31:12", "content": "That is pretty much exactly my question!
", "rawMarkdown": "That is pretty much exactly my question!", "votes": null }, { "id": "678634", "postDate": "11/21/2019 17:11:04", "content": "Lets assume it turns out we do have 5 sites of added data in the next weeks. Lets also pretend that the 5 sites with all 3 years of data had been part of the initial data offered.
\n\nOK - using leaked data to simply fill in the submission for the appearance of a better score would NOT be something you would have done if the data had been offered right up front. So those kernels that show you how to back fill your predictions with the scraped data are useless and would never have occurred if the added 5 had been offered up front. Always try to remember that in the real world you don't get a leader board to see how your doing.
\n\nOne of my major issues with the original data set was the need to predict two years given only a single year. Having data for all three years for some sites lets me see a number of features in action. Biggest one that comes to mind is holidays. Assuming a USA site there are I think around a dozen legal holidays. Some of them are major, like Christmas where almost everyone takes that day off, schools are closed, additional days off around the holiday, etc. Some are minor and should only have a small blip on the power usage. Even adding in days off around the major holidays the impact is probably hard to see. Now we can look at those kind of features over the full three year span.
\n\nI started a feature set of football games at the USA sites - the population of many colleges doubles on home football games, homecoming weekend, etc. But less than 6 home games for most teams - now I can look at 18 home games. SO - features that were infrequent in the original data set now occur more often with greater potential that your model can see them. It's looking like from discussion posts that we are also closing in on knowing almost all of the cities - so lots more potential for events as features.
\n\nMost of the models I have created have very nice matches (ok you could call them over-trained) to the 2016 data. Some of the very best matches are the worst performers on leader board. Cannot think of too many folks who would have looked at the 5 added sites with all three years and said - hmm - better not put this data right into my train data set for my model. So pretty much my exact answer would be - put the added data into your training set - that's would you would have done if it was offered as part of the original data package. Now you do have some additional worries as you now have an unbalanced data set so you get the chance to learn how to handle that (rather than the current learning of how to google and scrape).
\n\nAs almost a universal truth - adding more data does NOT lead to over-fitting - it's one of the cures for over fitting.
", "rawMarkdown": "Lets assume it turns out we do have 5 sites of added data in the next weeks. Lets also pretend that the 5 sites with all 3 years of data had been part of the initial data offered.\n\nOK - using leaked data to simply fill in the submission for the appearance of a better score would NOT be something you would have done if the data had been offered right up front. So those kernels that show you how to back fill your predictions with the scraped data are useless and would never have occurred if the added 5 had been offered up front. Always try to remember that in the real world you don't get a leader board to see how your doing.\n\nOne of my major issues with the original data set was the need to predict two years given only a single year. Having data for all three years for some sites lets me see a number of features in action. Biggest one that comes to mind is holidays. Assuming a USA site there are I think around a dozen legal holidays. Some of them are major, like Christmas where almost everyone takes that day off, schools are closed, additional days off around the holiday, etc. Some are minor and should only have a small blip on the power usage. Even adding in days off around the major holidays the impact is probably hard to see. Now we can look at those kind of features over the full three year span.\n\nI started a feature set of football games at the USA sites - the population of many colleges doubles on home football games, homecoming weekend, etc. But less than 6 home games for most teams - now I can look at 18 home games. SO - features that were infrequent in the original data set now occur more often with greater potential that your model can see them. It's looking like from discussion posts that we are also closing in on knowing almost all of the cities - so lots more potential for events as features.\n\nMost of the models I have created have very nice matches (ok you could call them over-trained) to the 2016 data. Some of the very best matches are the worst performers on leader board. Cannot think of too many folks who would have looked at the 5 added sites with all three years and said - hmm - better not put this data right into my train data set for my model. So pretty much my exact answer would be - put the added data into your training set - that's would you would have done if it was offered as part of the original data package. Now you do have some additional worries as you now have an unbalanced data set so you get the chance to learn how to handle that (rather than the current learning of how to google and scrape).\n\nAs almost a universal truth - adding more data does NOT lead to over-fitting - it's one of the cures for over fitting.", "votes": null }, { "id": "679015", "postDate": "11/22/2019 07:02:29", "content": "I just started with this competition. I know there is a leak so it is affecting the leaderboard. So should I use leaked data or not into my modeling?
", "rawMarkdown": "I just started with this competition. I know there is a leak so it is affecting the leaderboard. So should I use leaked data or not into my modeling?", "votes": null }, { "id": "679065", "postDate": "11/22/2019 08:26:22", "content": "you should try training without and with additional data and see what happens. you can also use external data for validation setup if you wish. You can check out my notebook for ideas.
", "rawMarkdown": "you should try training without and with additional data and see what happens. you can also use external data for validation setup if you wish. You can check out my [notebook](https://www.kaggle.com/teeyee314/ucf-counterfactual-prediction-analysis) for ideas.", "votes": null } ], "comments": [ { "id": 677997, "author_name": "hanjoonchoe", "author_url": "", "post_date": "11/20/2019 21:33:14", "content": "Fill with disclosed useless predictions will show you clearer improvement of the prediction of your model when you evaluate the model with still valid predictions by checking LB.
", "votes": null, "replies": [ { "id": 678030, "author_name": "elysii", "author_url": "", "post_date": "11/20/2019 23:00:19", "content": "Thanks! You're right using the ground truth labels will show us clearer improvement on the data that is not leaked... \nI'm still interested if we can use the data in a different way to improve our models on other sides :)
", "votes": null, "replies": [] } ] }, { "id": 678220, "author_name": "arnabbiswas1", "author_url": "", "post_date": "11/21/2019 05:29:38", "content": "Exactly, I am also wondering how can we use the leaked data without overfitting the model? Suggestions are welcome.
", "votes": null, "replies": [ { "id": 678298, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/21/2019 07:47:56", "content": "If organizers had provided us with 15 sites - all of them having 2016 data and 5 of them have 2016 to 2018 data what would you do?
", "votes": null, "replies": [] }, { "id": 678403, "author_name": "elysii", "author_url": "", "post_date": "11/21/2019 10:31:12", "content": "That is pretty much exactly my question!
", "votes": null, "replies": [] }, { "id": 678634, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/21/2019 17:11:04", "content": "Lets assume it turns out we do have 5 sites of added data in the next weeks. Lets also pretend that the 5 sites with all 3 years of data had been part of the initial data offered.
\n\nOK - using leaked data to simply fill in the submission for the appearance of a better score would NOT be something you would have done if the data had been offered right up front. So those kernels that show you how to back fill your predictions with the scraped data are useless and would never have occurred if the added 5 had been offered up front. Always try to remember that in the real world you don't get a leader board to see how your doing.
\n\nOne of my major issues with the original data set was the need to predict two years given only a single year. Having data for all three years for some sites lets me see a number of features in action. Biggest one that comes to mind is holidays. Assuming a USA site there are I think around a dozen legal holidays. Some of them are major, like Christmas where almost everyone takes that day off, schools are closed, additional days off around the holiday, etc. Some are minor and should only have a small blip on the power usage. Even adding in days off around the major holidays the impact is probably hard to see. Now we can look at those kind of features over the full three year span.
\n\nI started a feature set of football games at the USA sites - the population of many colleges doubles on home football games, homecoming weekend, etc. But less than 6 home games for most teams - now I can look at 18 home games. SO - features that were infrequent in the original data set now occur more often with greater potential that your model can see them. It's looking like from discussion posts that we are also closing in on knowing almost all of the cities - so lots more potential for events as features.
\n\nMost of the models I have created have very nice matches (ok you could call them over-trained) to the 2016 data. Some of the very best matches are the worst performers on leader board. Cannot think of too many folks who would have looked at the 5 added sites with all three years and said - hmm - better not put this data right into my train data set for my model. So pretty much my exact answer would be - put the added data into your training set - that's would you would have done if it was offered as part of the original data package. Now you do have some additional worries as you now have an unbalanced data set so you get the chance to learn how to handle that (rather than the current learning of how to google and scrape).
\n\nAs almost a universal truth - adding more data does NOT lead to over-fitting - it's one of the cures for over fitting.
", "votes": null, "replies": [] } ] }, { "id": 679015, "author_name": "adish333", "author_url": "", "post_date": "11/22/2019 07:02:29", "content": "I just started with this competition. I know there is a leak so it is affecting the leaderboard. So should I use leaked data or not into my modeling?
", "votes": null, "replies": [ { "id": 679065, "author_name": "teeyee314", "author_url": "", "post_date": "11/22/2019 08:26:22", "content": "you should try training without and with additional data and see what happens. you can also use external data for validation setup if you wish. You can check out my notebook for ideas.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "677992": "Hey there!\nSince the amount of data leakage is constantly growing I wonder how to deal with it...\nThe LB seems to be quite useless by now, however just replacing values in your submission won't give you any boost in the private LB, as the authors of this competition announced to exclude the leaked data from the final score.\n\nHas anybody tried to replace missing data in the training set (year 2016) or use the leaked 2017 data in training to boost the performance of other sides? \n\nAny other idea is also welcome :)", "677997": "Fill with disclosed useless predictions will show you clearer improvement of the prediction of your model when you evaluate the model with still valid predictions by checking LB.", "678030": "Thanks! You're right using the ground truth labels will show us clearer improvement on the data that is not leaked... \nI'm still interested if we can use the data in a different way to improve our models on other sides :)", "678220": "Exactly, I am also wondering how can we use the leaked data without overfitting the model? Suggestions are welcome.", "678298": "If organizers had provided us with 15 sites - all of them having 2016 data and 5 of them have 2016 to 2018 data what would you do?", "678403": "That is pretty much exactly my question!", "678634": "Lets assume it turns out we do have 5 sites of added data in the next weeks. Lets also pretend that the 5 sites with all 3 years of data had been part of the initial data offered.\n\nOK - using leaked data to simply fill in the submission for the appearance of a better score would NOT be something you would have done if the data had been offered right up front. So those kernels that show you how to back fill your predictions with the scraped data are useless and would never have occurred if the added 5 had been offered up front. Always try to remember that in the real world you don't get a leader board to see how your doing.\n\nOne of my major issues with the original data set was the need to predict two years given only a single year. Having data for all three years for some sites lets me see a number of features in action. Biggest one that comes to mind is holidays. Assuming a USA site there are I think around a dozen legal holidays. Some of them are major, like Christmas where almost everyone takes that day off, schools are closed, additional days off around the holiday, etc. Some are minor and should only have a small blip on the power usage. Even adding in days off around the major holidays the impact is probably hard to see. Now we can look at those kind of features over the full three year span.\n\nI started a feature set of football games at the USA sites - the population of many colleges doubles on home football games, homecoming weekend, etc. But less than 6 home games for most teams - now I can look at 18 home games. SO - features that were infrequent in the original data set now occur more often with greater potential that your model can see them. It's looking like from discussion posts that we are also closing in on knowing almost all of the cities - so lots more potential for events as features.\n\nMost of the models I have created have very nice matches (ok you could call them over-trained) to the 2016 data. Some of the very best matches are the worst performers on leader board. Cannot think of too many folks who would have looked at the 5 added sites with all three years and said - hmm - better not put this data right into my train data set for my model. So pretty much my exact answer would be - put the added data into your training set - that's would you would have done if it was offered as part of the original data package. Now you do have some additional worries as you now have an unbalanced data set so you get the chance to learn how to handle that (rather than the current learning of how to google and scrape).\n\nAs almost a universal truth - adding more data does NOT lead to over-fitting - it's one of the cures for over fitting.", "679015": "I just started with this competition. I know there is a leak so it is affecting the leaderboard. So should I use leaked data or not into my modeling?", "679065": "you should try training without and with additional data and see what happens. you can also use external data for validation setup if you wish. You can check out my [notebook](https://www.kaggle.com/teeyee314/ucf-counterfactual-prediction-analysis) for ideas." }, "source": "meta" }