Hi,\nReading the overview is useless now and the purpose of the competition has been defeated.\nThe purpose was to make models that could be used in the future to predict how much energy a building would use without improving efficiency. Now, if so many notebooks are just using leaked data, are we really solving the problem? I am wondering how we could discourage this, atleast for future competitions?
", "messages": [ { "id": "679927", "postDate": "11/23/2019 17:10:39", "content": "Hi,\nReading the overview is useless now and the purpose of the competition has been defeated.\nThe purpose was to make models that could be used in the future to predict how much energy a building would use without improving efficiency. Now, if so many notebooks are just using leaked data, are we really solving the problem? I am wondering how we could discourage this, atleast for future competitions?
", "rawMarkdown": "Hi,\nReading the overview is useless now and the purpose of the competition has been defeated.\nThe purpose was to make models that could be used in the future to predict how much energy a building would use without improving efficiency. Now, if so many notebooks are just using leaked data, are we really solving the problem? I am wondering how we could discourage this, atleast for future competitions?", "votes": null }, { "id": "679934", "postDate": "11/23/2019 17:31:54", "content": "In normal situations, leaked information is typically only used accidentally. But in competitions, participants often find and intentionally exploit leakage where it is present. It a common problem!
\n\n\n\n\nWhen leakage is found in a competition, there are many ways that we can address it. These may include:
\n \n\n
\n- \n
Let the competition continue as is (especially if the leakage only has a small impact)
- \n
Remove the leakage from the set and relaunch the competition
- \n
Generate a new test set that does not have the leakage present
You can find out more info in the competition documentation here:\nhttps://www.kaggle.com/docs/competitions#what-is-leakage
", "rawMarkdown": "In normal situations, leaked information is typically only used accidentally. But in competitions, participants often find and intentionally exploit leakage where it is present. It a common problem!\n\n> When leakage is found in a competition, there are many ways that we can address it. These may include:\n\n> - Let the competition continue as is (especially if the leakage only has a small impact)\n\n> - Remove the leakage from the set and relaunch the competition\n\n> - Generate a new test set that does not have the leakage present\n\nYou can find out more info in the competition documentation here:\nhttps://www.kaggle.com/docs/competitions#what-is-leakage", "votes": null }, { "id": "679988", "postDate": "11/23/2019 19:26:22", "content": "If you focus on \"THE PLACE\", it might be useless, but If you focus on improvement of LB score of yourself, it still useful. Specifically focusing on valid predictions, You just replace all meter readings leaked with your raw prediction and see how your score behaves, then it is still useful.
\n\nLeaks are nothing or could be useful in predictions, don't know yet because there is no one to say if they could get benefits for the still valids other than replacing from leaks. I use it just to erase useless predictions
", "rawMarkdown": "If you focus on \"THE PLACE\", it might be useless, but If you focus on improvement of LB score of yourself, it still useful. Specifically focusing on valid predictions, You just replace all meter readings leaked with your raw prediction and see how your score behaves, then it is still useful.\n\nLeaks are nothing or could be useful in predictions, don't know yet because there is no one to say if they could get benefits for the still valids other than replacing from leaks. I use it just to erase useless predictions", "votes": null }, { "id": "680026", "postDate": "11/23/2019 21:23:27", "content": "In this case, you're better off using the leaked data. But there are multiple ways to use leaked data, such as training on leak, or validating on leak, but not both. Just because a few sites' ground truth labels can be found online through webscraping, doesn't mean the competition should be over. In fact, it just means we have more visibility on how our model behaves on the test set particularly through validating. Part of the test set is still hidden and thus still meaningful. Currently there is about 20-33% of the test that can be found online, that just means there the testing set is smaller that it originally was, which is no big deal considering test is 2x training data.
", "rawMarkdown": "In this case, you're better off using the leaked data. But there are multiple ways to use leaked data, such as training on leak, or validating on leak, but not both. Just because a few sites' ground truth labels can be found online through webscraping, doesn't mean the competition should be over. In fact, it just means we have more visibility on how our model behaves on the test set particularly through validating. Part of the test set is still hidden and thus still meaningful. Currently there is about 20-33% of the test that can be found online, that just means there the testing set is smaller that it originally was, which is no big deal considering test is 2x training data.", "votes": null } ], "comments": [ { "id": 679934, "author_name": "andreagarritano", "author_url": "", "post_date": "11/23/2019 17:31:54", "content": "In normal situations, leaked information is typically only used accidentally. But in competitions, participants often find and intentionally exploit leakage where it is present. It a common problem!
\n\n\n\n\nWhen leakage is found in a competition, there are many ways that we can address it. These may include:
\n \n\n
\n- \n
Let the competition continue as is (especially if the leakage only has a small impact)
- \n
Remove the leakage from the set and relaunch the competition
- \n
Generate a new test set that does not have the leakage present
You can find out more info in the competition documentation here:\nhttps://www.kaggle.com/docs/competitions#what-is-leakage
", "votes": null, "replies": [] }, { "id": 679988, "author_name": "hanjoonchoe", "author_url": "", "post_date": "11/23/2019 19:26:22", "content": "If you focus on \"THE PLACE\", it might be useless, but If you focus on improvement of LB score of yourself, it still useful. Specifically focusing on valid predictions, You just replace all meter readings leaked with your raw prediction and see how your score behaves, then it is still useful.
\n\nLeaks are nothing or could be useful in predictions, don't know yet because there is no one to say if they could get benefits for the still valids other than replacing from leaks. I use it just to erase useless predictions
", "votes": null, "replies": [] }, { "id": 680026, "author_name": "teeyee314", "author_url": "", "post_date": "11/23/2019 21:23:27", "content": "In this case, you're better off using the leaked data. But there are multiple ways to use leaked data, such as training on leak, or validating on leak, but not both. Just because a few sites' ground truth labels can be found online through webscraping, doesn't mean the competition should be over. In fact, it just means we have more visibility on how our model behaves on the test set particularly through validating. Part of the test set is still hidden and thus still meaningful. Currently there is about 20-33% of the test that can be found online, that just means there the testing set is smaller that it originally was, which is no big deal considering test is 2x training data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "679927": "Hi,\nReading the overview is useless now and the purpose of the competition has been defeated.\nThe purpose was to make models that could be used in the future to predict how much energy a building would use without improving efficiency. Now, if so many notebooks are just using leaked data, are we really solving the problem? I am wondering how we could discourage this, atleast for future competitions?", "679934": "In normal situations, leaked information is typically only used accidentally. But in competitions, participants often find and intentionally exploit leakage where it is present. It a common problem!\n\n> When leakage is found in a competition, there are many ways that we can address it. These may include:\n\n> - Let the competition continue as is (especially if the leakage only has a small impact)\n\n> - Remove the leakage from the set and relaunch the competition\n\n> - Generate a new test set that does not have the leakage present\n\nYou can find out more info in the competition documentation here:\nhttps://www.kaggle.com/docs/competitions#what-is-leakage", "679988": "If you focus on \"THE PLACE\", it might be useless, but If you focus on improvement of LB score of yourself, it still useful. Specifically focusing on valid predictions, You just replace all meter readings leaked with your raw prediction and see how your score behaves, then it is still useful.\n\nLeaks are nothing or could be useful in predictions, don't know yet because there is no one to say if they could get benefits for the still valids other than replacing from leaks. I use it just to erase useless predictions", "680026": "In this case, you're better off using the leaked data. But there are multiple ways to use leaked data, such as training on leak, or validating on leak, but not both. Just because a few sites' ground truth labels can be found online through webscraping, doesn't mean the competition should be over. In fact, it just means we have more visibility on how our model behaves on the test set particularly through validating. Part of the test set is still hidden and thus still meaningful. Currently there is about 20-33% of the test that can be found online, that just means there the testing set is smaller that it originally was, which is no big deal considering test is 2x training data." }, "source": "meta" }