Scoring this competition on the basis of validation performance may be problematic.
\n\nThe problem is, in this competition, validation performance won't penalize biased estimators because the validation data is generated under the same pricing/stocking regime as the training data:
\n\n\n\n\nThe train and test dataset are split based on time, as well as the public and private leaderboard dataset split.
\n
Bias is bad because it can contaminate policy forecasts. In other words, a biased model may lose predictive power if and when Bimbo modifies their pricing/stocking strategy.
\n\nOne potential source of bias in this competition comes from product stockouts (see, for example, Mortimer and Conlon, 2013).
\n\nCan you think of others? What are your thoughts?
", "messages": [ { "id": "123003", "postDate": "06/09/2016 05:21:40", "content": "Scoring this competition on the basis of validation performance may be problematic.
\n\nThe problem is, in this competition, validation performance won't penalize biased estimators because the validation data is generated under the same pricing/stocking regime as the training data:
\n\n\n\n\nThe train and test dataset are split based on time, as well as the public and private leaderboard dataset split.
\n
Bias is bad because it can contaminate policy forecasts. In other words, a biased model may lose predictive power if and when Bimbo modifies their pricing/stocking strategy.
\n\nOne potential source of bias in this competition comes from product stockouts (see, for example, Mortimer and Conlon, 2013).
\n\nCan you think of others? What are your thoughts?
", "rawMarkdown": "Scoring this competition on the basis of validation performance may be problematic. \r\n\r\nThe problem is, in this competition, validation performance won't penalize biased estimators because the validation data is generated under the same pricing/stocking regime as the training data:\r\n\r\n> The train and test dataset are split based on time, as well as the public and private leaderboard dataset split.\r\n \r\nBias is bad because it can contaminate policy forecasts. In other words, a biased model may lose predictive power if and when Bimbo modifies their pricing/stocking strategy. \r\n\r\nOne potential source of bias in this competition comes from product stockouts (see, for example, [Mortimer and Conlon, 2013][1]).\r\n\r\nCan you think of others? What are your thoughts?\r\n\r\n[1]: https://www.aeaweb.org/articles?id=10.1257/mic.5.4.1", "votes": null }, { "id": "124105", "postDate": "06/15/2016 14:54:04", "content": "I agree that pricing/stocking strategies can affect the demand.
\n\nHowever, for you can still get some sense of the pricing/stocking strategy by looking at price per product on different weeks and across multiple retailers (for pricing), and at the Sales depot / Route for the stocking strategy. It is not the best data to do it but you can get a sense of what's going on.
\n\nFor the stockouts, it is a different beast. We don't have any data around stockouts on the supply side, and partal data in the retailer side. When the total returns is zero it might be that the store run out of product 1 minute, 1 day, 3days, etc before the restocking; It might also mean that they will do more returns next week... I have also noticed that big retailers have a small % of returns, which is expected since they have the best data about the demand and will never run out of stock; smaller retailers sometimes get no returns (they probably run out of stock) or have big returns (they weren't good at estimating demand). In both cases I am still not sure on how to handle during the analysis, as the goal is not to estimate the real (final consumer) demand, but the retailers' demand, so even if they are doing it the wrong way our goal is to predict that they will do so...
", "rawMarkdown": "I agree that pricing/stocking strategies can affect the demand.\r\n\r\nHowever, for you can still get some sense of the pricing/stocking strategy by looking at price per product on different weeks and across multiple retailers (for pricing), and at the Sales depot / Route for the stocking strategy. It is not the best data to do it but you can get a sense of what's going on.\r\n\r\nFor the stockouts, it is a different beast. We don't have any data around stockouts on the supply side, and partal data in the retailer side. When the total returns is zero it might be that the store run out of product 1 minute, 1 day, 3days, etc before the restocking; It might also mean that they will do more returns next week... I have also noticed that big retailers have a small % of returns, which is expected since they have the best data about the demand and will never run out of stock; smaller retailers sometimes get no returns (they probably run out of stock) or have big returns (they weren't good at estimating demand). In both cases I am still not sure on how to handle during the analysis, as the goal is not to estimate the real (final consumer) demand, but the retailers' demand, so even if they are doing it the wrong way our goal is to predict that they will do so...", "votes": null }, { "id": "124139", "postDate": "06/15/2016 19:20:17", "content": "Are the public and private leaderboard dataset really split based on time? Do you mean that the public leaderboard test set is equivalent to Week=10 and the private leaderboard to Week=11? Aren't they random? The only information that I found was that 51% of the test data corresponds to the public leaderboard and 49% to the private leaderboard.
", "rawMarkdown": "Are the public and private leaderboard dataset really split based on time? Do you mean that the public leaderboard test set is equivalent to Week=10 and the private leaderboard to Week=11? Aren't they random? The only information that I found was that 51% of the test data corresponds to the public leaderboard and 49% to the private leaderboard.", "votes": null }, { "id": "124143", "postDate": "06/15/2016 19:39:14", "content": "That's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.
", "rawMarkdown": "That's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.", "votes": null }, { "id": "125395", "postDate": "06/29/2016 04:42:11", "content": "Thought it might be helpful to provide a data point on how frequent stockouts are in practice. Here is a link to the SAS user's guide to inventory planning (SAS is the leading ERP solution):
\n\nhttps://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_91/inventory_ug_7307.pdf
\n\nThe SAS inventory replenishment planning tool calculates an optimal inventory policy using both a demand forecast and information on costs. One of the costs that SAS's tool considers is the cost of stockouts. Because the cost of a stockout can be hard to measure, SAS's tool offers the option to target one of several "service measures", including:
\n\nMany of the examples use fill rate, which also happens to be SAS's default service measure. All examples that use fill rate target a level on the order of 95%.
", "rawMarkdown": "Thought it might be helpful to provide a data point on how frequent stockouts are in practice. Here is a link to the SAS user's guide to inventory planning (SAS is the [leading ERP solution][1]):\r\n\r\nhttps://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_91/inventory_ug_7307.pdf\r\n\r\nThe SAS inventory replenishment planning tool calculates an optimal inventory policy using both a demand forecast and information on costs. One of the costs that SAS's tool considers is the cost of stockouts. Because the cost of a stockout can be hard to measure, SAS's tool offers the option to target one of several \"service measures\", including:\r\n\r\n - *Ready rate*: the likelihood of stockouts during a given planning period\r\n - *Fill rate*: the fraction of demand satisfied directly from on-hand inventory\r\n - *Backorder ratio*: average backorders divided by average demand\r\n\r\nMany of the examples use fill rate, which also happens to be SAS's default service measure. All examples that use fill rate target a level on the order of 95%.\r\n\r\n [1]: https://www.google.com/#q=top+ERP+solutions", "votes": null }, { "id": "125414", "postDate": "06/29/2016 09:09:33", "content": "[quote=Dmitriy Guller;124143]
\n\nThat's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.
\n\n[/quote]
\n\nSplitting by time is the usual setup for forecasting model.
", "rawMarkdown": "[quote=Dmitriy Guller;124143]\r\n\r\nThat's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.\r\n\r\n[/quote]\r\n\r\nSplitting by time is the usual setup for forecasting model.", "votes": null }, { "id": "125419", "postDate": "06/29/2016 10:21:33", "content": "Would you also say that splitting by time is the usual setup for training/validation also? I.e. using the last week for validation and the rest for training?
\n\n/m
", "rawMarkdown": "Would you also say that splitting by time is the usual setup for training/validation also? I.e. using the last week for validation and the rest for training?\r\n\r\n/m", "votes": null }, { "id": "125421", "postDate": "06/29/2016 10:33:42", "content": "yep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.
", "rawMarkdown": "yep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.", "votes": null }, { "id": "125423", "postDate": "06/29/2016 10:39:57", "content": "Yeah, that's a stellar result. Good work! My features are overfitting; I do worse in validation than training and even worse in testing. I'm trying to figure out which ones are worst, so I can get rid of them...
", "rawMarkdown": "Yeah, that's a stellar result. Good work! My features are overfitting; I do worse in validation than training and even worse in testing. I'm trying to figure out which ones are worst, so I can get rid of them...", "votes": null }, { "id": "125437", "postDate": "06/29/2016 13:00:59", "content": "[quote=Michael Jahrer;125421]
\n\nyep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.
\n\n[/quote]
\n\nYeah, this validates my observations. I wasn't doing so earlier and got my model overfit validation errors too much. I will catch you 1st place runner!
", "rawMarkdown": "[quote=Michael Jahrer;125421]\r\n\r\nyep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.\r\n\r\n\r\n[/quote]\r\n\r\nYeah, this validates my observations. I wasn't doing so earlier and got my model overfit validation errors too much. I will catch you 1st place runner!", "votes": null } ], "comments": [ { "id": 124105, "author_name": "juanrdz", "author_url": "", "post_date": "06/15/2016 14:54:04", "content": "I agree that pricing/stocking strategies can affect the demand.
\n\nHowever, for you can still get some sense of the pricing/stocking strategy by looking at price per product on different weeks and across multiple retailers (for pricing), and at the Sales depot / Route for the stocking strategy. It is not the best data to do it but you can get a sense of what's going on.
\n\nFor the stockouts, it is a different beast. We don't have any data around stockouts on the supply side, and partal data in the retailer side. When the total returns is zero it might be that the store run out of product 1 minute, 1 day, 3days, etc before the restocking; It might also mean that they will do more returns next week... I have also noticed that big retailers have a small % of returns, which is expected since they have the best data about the demand and will never run out of stock; smaller retailers sometimes get no returns (they probably run out of stock) or have big returns (they weren't good at estimating demand). In both cases I am still not sure on how to handle during the analysis, as the goal is not to estimate the real (final consumer) demand, but the retailers' demand, so even if they are doing it the wrong way our goal is to predict that they will do so...
", "votes": null, "replies": [] }, { "id": 124139, "author_name": "regisely", "author_url": "", "post_date": "06/15/2016 19:20:17", "content": "Are the public and private leaderboard dataset really split based on time? Do you mean that the public leaderboard test set is equivalent to Week=10 and the private leaderboard to Week=11? Aren't they random? The only information that I found was that 51% of the test data corresponds to the public leaderboard and 49% to the private leaderboard.
", "votes": null, "replies": [] }, { "id": 124143, "author_name": "dmitriyguller", "author_url": "", "post_date": "06/15/2016 19:39:14", "content": "That's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.
", "votes": null, "replies": [] }, { "id": 125395, "author_name": "dietcoke", "author_url": "", "post_date": "06/29/2016 04:42:11", "content": "Thought it might be helpful to provide a data point on how frequent stockouts are in practice. Here is a link to the SAS user's guide to inventory planning (SAS is the leading ERP solution):
\n\nhttps://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_91/inventory_ug_7307.pdf
\n\nThe SAS inventory replenishment planning tool calculates an optimal inventory policy using both a demand forecast and information on costs. One of the costs that SAS's tool considers is the cost of stockouts. Because the cost of a stockout can be hard to measure, SAS's tool offers the option to target one of several "service measures", including:
\n\nMany of the examples use fill rate, which also happens to be SAS's default service measure. All examples that use fill rate target a level on the order of 95%.
", "votes": null, "replies": [] }, { "id": 125414, "author_name": "amazingbob", "author_url": "", "post_date": "06/29/2016 09:09:33", "content": "[quote=Dmitriy Guller;124143]
\n\nThat's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.
\n\n[/quote]
\n\nSplitting by time is the usual setup for forecasting model.
", "votes": null, "replies": [] }, { "id": 125419, "author_name": "mfagerlund", "author_url": "", "post_date": "06/29/2016 10:21:33", "content": "Would you also say that splitting by time is the usual setup for training/validation also? I.e. using the last week for validation and the rest for training?
\n\n/m
", "votes": null, "replies": [] }, { "id": 125421, "author_name": "mjahrer", "author_url": "", "post_date": "06/29/2016 10:33:42", "content": "yep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.
", "votes": null, "replies": [] }, { "id": 125423, "author_name": "mfagerlund", "author_url": "", "post_date": "06/29/2016 10:39:57", "content": "Yeah, that's a stellar result. Good work! My features are overfitting; I do worse in validation than training and even worse in testing. I'm trying to figure out which ones are worst, so I can get rid of them...
", "votes": null, "replies": [] }, { "id": 125437, "author_name": "amazingbob", "author_url": "", "post_date": "06/29/2016 13:00:59", "content": "[quote=Michael Jahrer;125421]
\n\nyep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.
\n\n[/quote]
\n\nYeah, this validates my observations. I wasn't doing so earlier and got my model overfit validation errors too much. I will catch you 1st place runner!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "123003": "Scoring this competition on the basis of validation performance may be problematic. \r\n\r\nThe problem is, in this competition, validation performance won't penalize biased estimators because the validation data is generated under the same pricing/stocking regime as the training data:\r\n\r\n> The train and test dataset are split based on time, as well as the public and private leaderboard dataset split.\r\n \r\nBias is bad because it can contaminate policy forecasts. In other words, a biased model may lose predictive power if and when Bimbo modifies their pricing/stocking strategy. \r\n\r\nOne potential source of bias in this competition comes from product stockouts (see, for example, [Mortimer and Conlon, 2013][1]).\r\n\r\nCan you think of others? What are your thoughts?\r\n\r\n[1]: https://www.aeaweb.org/articles?id=10.1257/mic.5.4.1", "124105": "I agree that pricing/stocking strategies can affect the demand.\r\n\r\nHowever, for you can still get some sense of the pricing/stocking strategy by looking at price per product on different weeks and across multiple retailers (for pricing), and at the Sales depot / Route for the stocking strategy. It is not the best data to do it but you can get a sense of what's going on.\r\n\r\nFor the stockouts, it is a different beast. We don't have any data around stockouts on the supply side, and partal data in the retailer side. When the total returns is zero it might be that the store run out of product 1 minute, 1 day, 3days, etc before the restocking; It might also mean that they will do more returns next week... I have also noticed that big retailers have a small % of returns, which is expected since they have the best data about the demand and will never run out of stock; smaller retailers sometimes get no returns (they probably run out of stock) or have big returns (they weren't good at estimating demand). In both cases I am still not sure on how to handle during the analysis, as the goal is not to estimate the real (final consumer) demand, but the retailers' demand, so even if they are doing it the wrong way our goal is to predict that they will do so...", "124139": "Are the public and private leaderboard dataset really split based on time? Do you mean that the public leaderboard test set is equivalent to Week=10 and the private leaderboard to Week=11? Aren't they random? The only information that I found was that 51% of the test data corresponds to the public leaderboard and 49% to the private leaderboard.", "124143": "That's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.", "125395": "Thought it might be helpful to provide a data point on how frequent stockouts are in practice. Here is a link to the SAS user's guide to inventory planning (SAS is the [leading ERP solution][1]):\r\n\r\nhttps://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_91/inventory_ug_7307.pdf\r\n\r\nThe SAS inventory replenishment planning tool calculates an optimal inventory policy using both a demand forecast and information on costs. One of the costs that SAS's tool considers is the cost of stockouts. Because the cost of a stockout can be hard to measure, SAS's tool offers the option to target one of several \"service measures\", including:\r\n\r\n - *Ready rate*: the likelihood of stockouts during a given planning period\r\n - *Fill rate*: the fraction of demand satisfied directly from on-hand inventory\r\n - *Backorder ratio*: average backorders divided by average demand\r\n\r\nMany of the examples use fill rate, which also happens to be SAS's default service measure. All examples that use fill rate target a level on the order of 95%.\r\n\r\n [1]: https://www.google.com/#q=top+ERP+solutions", "125414": "[quote=Dmitriy Guller;124143]\r\n\r\nThat's what I took it to mean. Usually even forecasting competitions split the test set randomly between private and public, but not in this case. That actually makes sense to me, I think the random test split is cheating.\r\n\r\n[/quote]\r\n\r\nSplitting by time is the usual setup for forecasting model.", "125419": "Would you also say that splitting by time is the usual setup for training/validation also? I.e. using the last week for validation and the rest for training?\r\n\r\n/m", "125421": "yep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.", "125423": "Yeah, that's a stellar result. Good work! My features are overfitting; I do worse in validation than training and even worse in testing. I'm trying to figure out which ones are worst, so I can get rid of them...", "125437": "[quote=Michael Jahrer;125421]\r\n\r\nyep if u have time sorted data. I use the last week as validation set and error on valid / leaderboard are very consistent. eg. 0.443686 on valid and 0.43722 on leaderboard.\r\n\r\n\r\n[/quote]\r\n\r\nYeah, this validates my observations. I wasn't doing so earlier and got my model overfit validation errors too much. I will catch you 1st place runner!" }, "source": "meta" }