Others have mentioned that the oil prices data lets us leak information about future oil prices into our forecasts, which is not helpful for the sponsors at all. People can easily widen this leak by using other future data such as weather, national disasters, economic fluctuations, etc. This both hurts the value of the winning solutions for the sponsor and makes the competition unappealing for those who are looking for a fair forecasting competition. Please do something, thanks.
", "messages": [ { "id": "240520", "postDate": "11/06/2017 20:01:15", "content": "Others have mentioned that the oil prices data lets us leak information about future oil prices into our forecasts, which is not helpful for the sponsors at all. People can easily widen this leak by using other future data such as weather, national disasters, economic fluctuations, etc. This both hurts the value of the winning solutions for the sponsor and makes the competition unappealing for those who are looking for a fair forecasting competition. Please do something, thanks.
", "rawMarkdown": "Others have mentioned that the oil prices data lets us leak information about future oil prices into our forecasts, which is not helpful for the sponsors at all. People can easily widen this leak by using other future data such as weather, national disasters, economic fluctuations, etc. This both hurts the value of the winning solutions for the sponsor and makes the competition unappealing for those who are looking for a fair forecasting competition. Please do something, thanks.", "votes": null }, { "id": "240586", "postDate": "11/06/2017 23:11:59", "content": "Your concerns have merit. There are pros and cons for allowing external data that can introduce a small amount of leakage into the test timeframe. We did discuss these at length prior to the competition, and believe the current policy strikes the correct balance.
", "rawMarkdown": "Your concerns have merit. There are pros and cons for allowing external data that can introduce a small amount of leakage into the test timeframe. We did discuss these at length prior to the competition, and believe the current policy strikes the correct balance.", "votes": null }, { "id": "240632", "postDate": "11/07/2017 03:01:57", "content": "I have been among the others discussing data leakage in this competition. From one of the subsequent comments, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability.
", "rawMarkdown": "I have been among the others discussing data leakage in this competition. From one of the subsequent comments, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability.", "votes": null }, { "id": "240888", "postDate": "11/07/2017 15:36:46", "content": "Honestly, using the oil data in a leaky way overfits the dataset extremely quickly, because it lets your model learn specific day patterns, and as @Murray Hays mentions there is a time lag between the effect. Plus, I have found that most of the correlation between oil prices and product sales is overshadowed by the fact that sales are a random walk. My solutions are better without it.
\n\nThat said, widening the leak could have potential. I almost feel like we are obligated to if we want to do well...
", "rawMarkdown": "Honestly, using the oil data in a leaky way overfits the dataset extremely quickly, because it lets your model learn specific day patterns, and as @Murray Hays mentions there is a time lag between the effect. Plus, I have found that most of the correlation between oil prices and product sales is overshadowed by the fact that sales are a random walk. My solutions are better without it.\n\nThat said, widening the leak could have potential. I almost feel like we are obligated to if we want to do well...", "votes": null } ], "comments": [ { "id": 240586, "author_name": "inversion", "author_url": "", "post_date": "11/06/2017 23:11:59", "content": "Your concerns have merit. There are pros and cons for allowing external data that can introduce a small amount of leakage into the test timeframe. We did discuss these at length prior to the competition, and believe the current policy strikes the correct balance.
", "votes": null, "replies": [] }, { "id": 240632, "author_name": "gmurrayhays", "author_url": "", "post_date": "11/07/2017 03:01:57", "content": "I have been among the others discussing data leakage in this competition. From one of the subsequent comments, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability.
", "votes": null, "replies": [] }, { "id": 240888, "author_name": "rhgrossm", "author_url": "", "post_date": "11/07/2017 15:36:46", "content": "Honestly, using the oil data in a leaky way overfits the dataset extremely quickly, because it lets your model learn specific day patterns, and as @Murray Hays mentions there is a time lag between the effect. Plus, I have found that most of the correlation between oil prices and product sales is overshadowed by the fact that sales are a random walk. My solutions are better without it.
\n\nThat said, widening the leak could have potential. I almost feel like we are obligated to if we want to do well...
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "240520": "Others have mentioned that the oil prices data lets us leak information about future oil prices into our forecasts, which is not helpful for the sponsors at all. People can easily widen this leak by using other future data such as weather, national disasters, economic fluctuations, etc. This both hurts the value of the winning solutions for the sponsor and makes the competition unappealing for those who are looking for a fair forecasting competition. Please do something, thanks.", "240586": "Your concerns have merit. There are pros and cons for allowing external data that can introduce a small amount of leakage into the test timeframe. We did discuss these at length prior to the competition, and believe the current policy strikes the correct balance.", "240632": "I have been among the others discussing data leakage in this competition. From one of the subsequent comments, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability.", "240888": "Honestly, using the oil data in a leaky way overfits the dataset extremely quickly, because it lets your model learn specific day patterns, and as @Murray Hays mentions there is a time lag between the effect. Plus, I have found that most of the correlation between oil prices and product sales is overshadowed by the fact that sales are a random walk. My solutions are better without it.\n\nThat said, widening the leak could have potential. I almost feel like we are obligated to if we want to do well..." }, "source": "meta" }