![\"\"](\"https://github.com/stkbailey/stkbailey.github.io/blob/master/assets/kaggle_favorita/oil_prices.png?raw=True\" "\\"\\"")
I am confused. A historical oil price data set has been provided by Favorita for our use in predicting unit_sales. I have no problem understanding that oil prices could be an influence on unit_sales.
\n\nHowever, the goal of Favorita is to predict FUTURE unit_sales, for which oil prices will not be known at the time of prediction. I can see how a Kaggle competitor could rank higher through their use of oil prices, but I don't understand why the allowance of oil prices as an input is not counter-productive to Favorita's goal? (From my understanding, this would be an instance of data leakage - https://www.kaggle.com/wiki/Leakage).)
", "messages": [ { "id": "239042", "postDate": "11/02/2017 16:01:38", "content": "I am confused. A historical oil price data set has been provided by Favorita for our use in predicting unit_sales. I have no problem understanding that oil prices could be an influence on unit_sales.
\n\nHowever, the goal of Favorita is to predict FUTURE unit_sales, for which oil prices will not be known at the time of prediction. I can see how a Kaggle competitor could rank higher through their use of oil prices, but I don't understand why the allowance of oil prices as an input is not counter-productive to Favorita's goal? (From my understanding, this would be an instance of data leakage - https://www.kaggle.com/wiki/Leakage).)
", "rawMarkdown": "I am confused. A historical oil price data set has been provided by Favorita for our use in predicting unit_sales. I have no problem understanding that oil prices could be an influence on unit_sales. \n\nHowever, the goal of Favorita is to predict FUTURE unit_sales, for which oil prices will not be known at the time of prediction. I can see how a Kaggle competitor could rank higher through their use of oil prices, but I don't understand why the allowance of oil prices as an input is not counter-productive to Favorita's goal? (From my understanding, this would be an instance of data leakage - https://www.kaggle.com/wiki/Leakage)[][1].)\n\n\n [1]: https://www.kaggle.com/wiki/Leakage", "votes": null }, { "id": "239287", "postDate": "11/03/2017 05:41:50", "content": "Hello,
\n\nI have just started on the competition, my thoughts on above question is, if the oil price has a pattern and cycle on time, then it might be a time series feature which can be used in the forecasting. But if we don't have any pattern or cycle then i feel you are right.
", "rawMarkdown": "Hello,\n\n I have just started on the competition, my thoughts on above question is, if the oil price has a pattern and cycle on time, then it might be a time series feature which can be used in the forecasting. But if we don't have any pattern or cycle then i feel you are right.", "votes": null }, { "id": "239541", "postDate": "11/03/2017 18:25:53", "content": "It might be correlated with consumer sentiment. Perhaps while increasing, people will worry and spend less, while decreasing people will be more optimistic and start spending more. Since groceries are a necessity it might affect people buying more expensive vs cheaper items.
", "rawMarkdown": "It might be correlated with consumer sentiment. Perhaps while increasing, people will worry and spend less, while decreasing people will be more optimistic and start spending more. Since groceries are a necessity it might affect people buying more expensive vs cheaper items.", "votes": null }, { "id": "239712", "postDate": "11/04/2017 09:05:36", "content": "Presumably, any influence oil prices have on sales will involve a time delay, so by knowing the length of that delay and today's prices it will be possible to more precisely predict (near) future sales.
", "rawMarkdown": "Presumably, any influence oil prices have on sales will involve a time delay, so by knowing the length of that delay and today's prices it will be possible to more precisely predict (near) future sales.", "votes": null }, { "id": "240105", "postDate": "11/05/2017 16:07:20", "content": "Hi,\nJust a note, as Ecuador is an oil exporting country which relies heavy on the oil export, your comment makes sense when inverting the \"increasing\" and \"decreasing\".\nI'd also assume that oil prices would indeed have a delayed effect on people buying cheaper items due to getting fired from oil companies and govt. jobs.
", "rawMarkdown": "Hi,\nJust a note, as Ecuador is an oil exporting country which relies heavy on the oil export, your comment makes sense when inverting the \"increasing\" and \"decreasing\".\nI'd also assume that oil prices would indeed have a delayed effect on people buying cheaper items due to getting fired from oil companies and govt. jobs.", "votes": null }, { "id": "240873", "postDate": "11/07/2017 14:35:57", "content": "There was a major drop in oil prices in the middle of the training data, so at the very least, I think we'll want to make sure that the pre-recession and post-recession data are not treated exactly the same.
\n\n
Good point, Stephen. From one of the comments in this thread, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability. To ensure no data leakage, I make sure that the lag between a specific oil price and the affected unit_sales is always greater than the amount of time covered by the test data (which is 16 days).
", "rawMarkdown": "Good point, Stephen. From one of the comments in this thread, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability. To ensure no data leakage, I make sure that the lag between a specific oil price and the affected unit_sales is always greater than the amount of time covered by the test data (which is 16 days).", "votes": null }, { "id": "240933", "postDate": "11/07/2017 17:02:20", "content": "Agreeing with this. One of the EDAs shows a trend where there is an obvious lag. The problem is that the time between the change in oil price and the corrollary change in the target variable seems to be pretty variable.
", "rawMarkdown": "Agreeing with this. One of the EDAs shows a trend where there is an obvious lag. The problem is that the time between the change in oil price and the corrollary change in the target variable seems to be pretty variable.", "votes": null }, { "id": "241427", "postDate": "11/08/2017 19:58:32", "content": "I haven't investigated this yet, but I would imagine it oil price percentage change may not be a \"lagged\" variable, but a \"lead\" variable. If a consumer feels prices will go up/down in the near future, then they may shop differently in the present.
\n\nSo there might be some lag/lead correspondence --although I'm not sure to what extent we'll be able to detec the signal.
", "rawMarkdown": "I haven't investigated this yet, but I would imagine it oil price percentage change may not be a \"lagged\" variable, but a \"lead\" variable. If a consumer feels prices will go up/down in the near future, then they may shop differently in the present.\n\nSo there might be some lag/lead correspondence --although I'm not sure to what extent we'll be able to detec the signal.", "votes": null }, { "id": "241471", "postDate": "11/08/2017 22:34:44", "content": "You're right, lead not lag. Good catch!
", "rawMarkdown": "You're right, _lead_ not lag. Good catch!", "votes": null }, { "id": "266078", "postDate": "01/07/2018 18:33:42", "content": "I concur with OP's concerns
", "rawMarkdown": "I concur with OP's concerns", "votes": null } ], "comments": [ { "id": 239287, "author_name": "yatishbn", "author_url": "", "post_date": "11/03/2017 05:41:50", "content": "Hello,
\n\nI have just started on the competition, my thoughts on above question is, if the oil price has a pattern and cycle on time, then it might be a time series feature which can be used in the forecasting. But if we don't have any pattern or cycle then i feel you are right.
", "votes": null, "replies": [] }, { "id": 239541, "author_name": "smist08", "author_url": "", "post_date": "11/03/2017 18:25:53", "content": "It might be correlated with consumer sentiment. Perhaps while increasing, people will worry and spend less, while decreasing people will be more optimistic and start spending more. Since groceries are a necessity it might affect people buying more expensive vs cheaper items.
", "votes": null, "replies": [ { "id": 240105, "author_name": "raistorx", "author_url": "", "post_date": "11/05/2017 16:07:20", "content": "Hi,\nJust a note, as Ecuador is an oil exporting country which relies heavy on the oil export, your comment makes sense when inverting the \"increasing\" and \"decreasing\".\nI'd also assume that oil prices would indeed have a delayed effect on people buying cheaper items due to getting fired from oil companies and govt. jobs.
", "votes": null, "replies": [] } ] }, { "id": 239712, "author_name": "ascotth", "author_url": "", "post_date": "11/04/2017 09:05:36", "content": "Presumably, any influence oil prices have on sales will involve a time delay, so by knowing the length of that delay and today's prices it will be possible to more precisely predict (near) future sales.
", "votes": null, "replies": [ { "id": 240933, "author_name": "jmbull", "author_url": "", "post_date": "11/07/2017 17:02:20", "content": "Agreeing with this. One of the EDAs shows a trend where there is an obvious lag. The problem is that the time between the change in oil price and the corrollary change in the target variable seems to be pretty variable.
", "votes": null, "replies": [] }, { "id": 241427, "author_name": "puremath86", "author_url": "", "post_date": "11/08/2017 19:58:32", "content": "I haven't investigated this yet, but I would imagine it oil price percentage change may not be a \"lagged\" variable, but a \"lead\" variable. If a consumer feels prices will go up/down in the near future, then they may shop differently in the present.
\n\nSo there might be some lag/lead correspondence --although I'm not sure to what extent we'll be able to detec the signal.
", "votes": null, "replies": [] }, { "id": 241471, "author_name": "jmbull", "author_url": "", "post_date": "11/08/2017 22:34:44", "content": "You're right, lead not lag. Good catch!
", "votes": null, "replies": [] } ] }, { "id": 240873, "author_name": "stkbailey", "author_url": "", "post_date": "11/07/2017 14:35:57", "content": "There was a major drop in oil prices in the middle of the training data, so at the very least, I think we'll want to make sure that the pre-recession and post-recession data are not treated exactly the same.
\n\n
Good point, Stephen. From one of the comments in this thread, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability. To ensure no data leakage, I make sure that the lag between a specific oil price and the affected unit_sales is always greater than the amount of time covered by the test data (which is 16 days).
", "votes": null, "replies": [] } ] }, { "id": 266078, "author_name": "dbcerigo", "author_url": "", "post_date": "01/07/2018 18:33:42", "content": "I concur with OP's concerns
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "239042": "I am confused. A historical oil price data set has been provided by Favorita for our use in predicting unit_sales. I have no problem understanding that oil prices could be an influence on unit_sales. \n\nHowever, the goal of Favorita is to predict FUTURE unit_sales, for which oil prices will not be known at the time of prediction. I can see how a Kaggle competitor could rank higher through their use of oil prices, but I don't understand why the allowance of oil prices as an input is not counter-productive to Favorita's goal? (From my understanding, this would be an instance of data leakage - https://www.kaggle.com/wiki/Leakage)[][1].)\n\n\n [1]: https://www.kaggle.com/wiki/Leakage", "239287": "Hello,\n\n I have just started on the competition, my thoughts on above question is, if the oil price has a pattern and cycle on time, then it might be a time series feature which can be used in the forecasting. But if we don't have any pattern or cycle then i feel you are right.", "239541": "It might be correlated with consumer sentiment. Perhaps while increasing, people will worry and spend less, while decreasing people will be more optimistic and start spending more. Since groceries are a necessity it might affect people buying more expensive vs cheaper items.", "239712": "Presumably, any influence oil prices have on sales will involve a time delay, so by knowing the length of that delay and today's prices it will be possible to more precisely predict (near) future sales.", "240105": "Hi,\nJust a note, as Ecuador is an oil exporting country which relies heavy on the oil export, your comment makes sense when inverting the \"increasing\" and \"decreasing\".\nI'd also assume that oil prices would indeed have a delayed effect on people buying cheaper items due to getting fired from oil companies and govt. jobs.", "240873": "There was a major drop in oil prices in the middle of the training data, so at the very least, I think we'll want to make sure that the pre-recession and post-recession data are not treated exactly the same. \n\n", "240876": "Good point, Stephen. From one of the comments in this thread, I got the idea of how to use the oil data without introducing leakage. The comment talked about how unit_sales might be affected some time after specific oil price behavior. So, I added a parameter to my analysis that allows me to 'pretend' that older oil prices are associated with newer test data. This gave a small boost to my prediction capability. To ensure no data leakage, I make sure that the lag between a specific oil price and the affected unit_sales is always greater than the amount of time covered by the test data (which is 16 days).", "240933": "Agreeing with this. One of the EDAs shows a trend where there is an obvious lag. The problem is that the time between the change in oil price and the corrollary change in the target variable seems to be pretty variable.", "241427": "I haven't investigated this yet, but I would imagine it oil price percentage change may not be a \"lagged\" variable, but a \"lead\" variable. If a consumer feels prices will go up/down in the near future, then they may shop differently in the present.\n\nSo there might be some lag/lead correspondence --although I'm not sure to what extent we'll be able to detec the signal.", "241471": "You're right, _lead_ not lag. Good catch!", "266078": "I concur with OP's concerns" }, "source": "meta" }