As i understand the unit demand is actually min(demand, supply) in both the test data and the train data. This problem would be a much more interesting to think if test data somehow didn't have this limitation, i.e. the competition is to predict the 'True' demand of items given only conditional demand.
\n\nOne possible solution according to me can be to assume a latent variable which tells me whether demand >= supply or not. And then trying to iteratively find out the probability of this latent variable being 1 and True demand (may be using some adaptation of EM algorithm)
\n\nLet me know your comments
", "messages": [ { "id": "248278", "postDate": "11/25/2017 14:09:40", "content": "As i understand the unit demand is actually min(demand, supply) in both the test data and the train data. This problem would be a much more interesting to think if test data somehow didn't have this limitation, i.e. the competition is to predict the 'True' demand of items given only conditional demand.
\n\nOne possible solution according to me can be to assume a latent variable which tells me whether demand >= supply or not. And then trying to iteratively find out the probability of this latent variable being 1 and True demand (may be using some adaptation of EM algorithm)
\n\nLet me know your comments
", "rawMarkdown": "As i understand the unit demand is actually min(demand, supply) in both the test data and the train data. This problem would be a much more interesting to think if test data somehow didn't have this limitation, i.e. the competition is to predict the 'True' demand of items given only conditional demand.\n\nOne possible solution according to me can be to assume a latent variable which tells me whether demand >= supply or not. And then trying to iteratively find out the probability of this latent variable being 1 and True demand (may be using some adaptation of EM algorithm)\n\nLet me know your comments", "votes": null }, { "id": "249846", "postDate": "11/29/2017 10:13:01", "content": "what did you mean by the unit demand is actually min(demand, supply) in both the test data and the train data ? How did you get the demand or the supply? Can you explain to me in detail ：）
", "rawMarkdown": "what did you mean by the unit demand is actually min(demand, supply) in both the test data and the train data ? How did you get the demand or the supply? Can you explain to me in detail ：）", "votes": null }, { "id": "249949", "postDate": "11/29/2017 14:44:56", "content": "I think the unit_sales (observed) = min(demand , supply) (hypothetical demand and supply variable, not observed) in both train and test data. Because the sales will be limited by the units of that item present in that store on that day.
\n\nI was hoping that atleast demand is not limited by supply in the test sample , but i think it is, as the scores do not drop a lot from train and test.
\n\nInteresting scenario would be to predict the True demand (which is unobserved in the train sample) instead of the limited demand (unit_sales). I was thinking if this can be handled by assuming a latent variable and then using EM.
", "rawMarkdown": "I think the unit_sales (observed) = min(demand , supply) (hypothetical demand and supply variable, not observed) in both train and test data. Because the sales will be limited by the units of that item present in that store on that day.\n\nI was hoping that atleast demand is not limited by supply in the test sample , but i think it is, as the scores do not drop a lot from train and test. \n\nInteresting scenario would be to predict the True demand (which is unobserved in the train sample) instead of the limited demand (unit_sales). I was thinking if this can be handled by assuming a latent variable and then using EM.", "votes": null }, { "id": "249984", "postDate": "11/29/2017 15:40:50", "content": "Brilliant！
", "rawMarkdown": "Brilliant！", "votes": null }, { "id": "250506", "postDate": "11/30/2017 04:16:23", "content": "how could we know demand and supply？
", "rawMarkdown": "how could we know demand and supply？", "votes": null }, { "id": "252025", "postDate": "12/02/2017 04:56:31", "content": "Interesting idea. In my understanding the test dataset is collected the same way as the train dataset, so the target is still the actual sales, not theoretical demand. But I wonder if we could find a way to predict an item in a certain area is running out of supply so the model can accommodate that constraint.
", "rawMarkdown": "Interesting idea. In my understanding the test dataset is collected the same way as the train dataset, so the target is still the actual sales, not theoretical demand. But I wonder if we could find a way to predict an item in a certain area is running out of supply so the model can accommodate that constraint.", "votes": null }, { "id": "253195", "postDate": "12/04/2017 15:44:01", "content": "I actually don't get the goal of that competition. What's the point of predicting the unit_sales? IMO the demand shall be predicted.
\n\nCorrect me if I am wrong but training a model using the unit_sales in the loss indicator will result in a prediction that reproduces some of the \"incorrect supply previsions\" made by the previous methods.
\n\nIf unit_sales = min(demand, supply) = 0, it means demand=0 and suply>0 OR supply=0 and demand>0. The two cases result in a loss of money for the company, but the actions to be undertaken are totally different (first case: reducing supply, second case: increase supply). What am I suppose to do as the owner of the shop if the predicted unit_sales=0 ? Choose the action in function of the current supply? How much should I buy if current supply=0?
", "rawMarkdown": "I actually don't get the goal of that competition. What's the point of predicting the unit_sales? IMO the demand shall be predicted.\n\nCorrect me if I am wrong but training a model using the unit_sales in the loss indicator will result in a prediction that reproduces some of the \"incorrect supply previsions\" made by the previous methods.\n\nIf unit_sales = min(demand, supply) = 0, it means demand=0 and suply>0 OR supply=0 and demand>0. The two cases result in a loss of money for the company, but the actions to be undertaken are totally different (first case: reducing supply, second case: increase supply). What am I suppose to do as the owner of the shop if the predicted unit_sales=0 ? Choose the action in function of the current supply? How much should I buy if current supply=0?", "votes": null }, { "id": "253518", "postDate": "12/05/2017 05:12:37", "content": "@tanguy73: i agree predicting unit_sales will definitely has lot of old logic bias.
\n\nBut there is some logic in predicting it as well (its not interesting to predict it but it is not completely baseless). Customers stickiness explains it somewhat, i.e. the products which were sold in recent past in a store are the items which are more likely to be searched for by the customers (call it stickiness or customers expectation that he/she will find that particular item in that shop and hence goes to the shop searching for it) and the items which were not in stock in recent past will most likely not be searched.
\n\nSo, the old supply will set the future selling patterns on a store.
\n\nAlthough, i also agree that predicting the correct demand would be much more helpful in the long run, but the company should not use the correct demand from the start of this new strategy, they should gradually increase the demand to the correct demand.
\n\nLet me know your comments!
", "rawMarkdown": "tanguy73: i agree predicting unit_sales will definitely has lot of old logic bias.\n\nBut there is some logic in predicting it as well (its not interesting to predict it but it is not completely baseless). Customers stickiness explains it somewhat, i.e. the products which were sold in recent past in a store are the items which are more likely to be searched for by the customers (call it stickiness or customers expectation that he/she will find that particular item in that shop and hence goes to the shop searching for it) and the items which were not in stock in recent past will most likely not be searched.\n\nSo, the old supply will set the future selling patterns on a store.\n\nAlthough, i also agree that predicting the correct demand would be much more helpful in the long run, but the company should not use the correct demand from the start of this new strategy, they should gradually increase the demand to the correct demand.\n\nLet me know your comments!", "votes": null }, { "id": "253519", "postDate": "12/05/2017 05:14:35", "content": "Hey ceshine, yes the test data is similar to train data. As i though EM iterations might work to estimate the latent demand. But wont be useful for this competition as the test and train are both constraint by the supply
", "rawMarkdown": "Hey ceshine, yes the test data is similar to train data. As i though EM iterations might work to estimate the latent demand. But wont be useful for this competition as the test and train are both constraint by the supply", "votes": null }, { "id": "254242", "postDate": "12/06/2017 14:43:14", "content": "Not sure how you deduce the customer stickiness from the unit_sales, how to quantify it, and which actions to undertake knowing this. But I am a total noob in retailing.
\n\nStill I am struggling with the fact that a model which will predict the actual demand will be hardly penalized by the old logic bias (=old supply mismatches) made in the past. I was thrilled to join this competition but thinking that a good model will perform badly according to the competition ranking system kinda put an end to my motivation.
\n\nBut maybe I missed something cause people don't seem to mind this.
", "rawMarkdown": "Not sure how you deduce the customer stickiness from the unit_sales, how to quantify it, and which actions to undertake knowing this. But I am a total noob in retailing.\n\nStill I am struggling with the fact that a model which will predict the actual *demand* will be hardly penalized by the old logic bias (=old supply mismatches) made in the past. I was thrilled to join this competition but thinking that a *good* model will perform badly according to the competition ranking system kinda put an end to my motivation.\n\nBut maybe I missed something cause people don't seem to mind this.", "votes": null }, { "id": "255311", "postDate": "12/08/2017 20:29:40", "content": "\n\n\nLazyGUY wrote
\n \n\n\nyes the test data is similar to train data
\n
I'm not agree with that.
\n\nAs I highlighted in this kernel (https://www.kaggle.com/ericdfr/new-items-in-stores/), the count of new store/item combo in the test dataset is unusual.
\n\nAs FengLi spotted in this topic (https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/42964) :
\n\n\n\n\nThey just create the test data for convenience by including all item/store combo although like store A never sells item 1
\n
I've asked Corporación Favorita about these 40k+ new item/store combo but they haven't yet answered. :-)
\n\nI'm pretty sure that demand is limited by supply in the test set. I give an example here : https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/43083
", "rawMarkdown": "> **LazyGUY wrote**\n> \n> > yes the test data is similar to train data\n\nI'm not agree with that. \n\nAs I highlighted in this kernel ([https://www.kaggle.com/ericdfr/new-items-in-stores/][1]), the count of new store/item combo in the test dataset is unusual. \n\nAs FengLi spotted in this topic ([https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/42964][2]) :\n> They just create the test data for convenience by including all item/store combo although like store A never sells item 1 \n\nI've asked Corporación Favorita about these 40k+ new item/store combo but they haven't yet answered. :-)\n\nI'm pretty sure that demand is limited by supply in the test set. I give an example here : [https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/43083][3]\n\n\n [1]: https://www.kaggle.com/ericdfr/new-items-in-stores/\n [2]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/42964\n [3]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/43083", "votes": null }, { "id": "255554", "postDate": "12/09/2017 11:24:35", "content": "Hey i meant the test data and train are similar in the manner that they both have demand limited by supply. Also, i think most of the combinations which you think do not exist in the train data and exist in test data will have zero sales even in the test period, because in my opinion the data for test and train are not created differently. I don't think they did any kind of test and control in the test period or removed observations for certain combinations from train data.
\n\nI believe this because the performance although drops a little but not a lot from val to test data.
", "rawMarkdown": "Hey i meant the test data and train are similar in the manner that they both have demand limited by supply. Also, i think most of the combinations which you think do not exist in the train data and exist in test data will have zero sales even in the test period, because in my opinion the data for test and train are not created differently. I don't think they did any kind of test and control in the test period or removed observations for certain combinations from train data. \n\nI believe this because the performance although drops a little but not a lot from val to test data.", "votes": null }, { "id": "255557", "postDate": "12/09/2017 11:37:26", "content": "By old logic i mean the logic used to determine the supply of various items to various stores. And unit_sales is demand limited by supply hence using predicted unit_sales to set future demand forecast is biasing the forecast by the old logic and should not work logically.
\n\nBut on the other hand i think its not completely useless to predict it as well (considering it will be difficult to predict the actual demand because of data limitation).
\n\nI explain that by the following (please mind this argument as this is mostly an intuitive argument), the items which are absent from a store for a while, if introduced in huge quantity suddenly as might be suggested by the true demand (found out from EM or some other method) might not sell for two potential reasons\n1. The customers have shifted to competitive products and might not revert back or look for old product.\n2. They just don't look for that specific product in that store anymore as they didn't find that product in that store in recent past.
", "rawMarkdown": "By old logic i mean the logic used to determine the supply of various items to various stores. And unit_sales is demand limited by supply hence using predicted unit_sales to set future demand forecast is biasing the forecast by the old logic and should not work logically.\n\nBut on the other hand i think its not completely useless to predict it as well (considering it will be difficult to predict the actual demand because of data limitation). \n\nI explain that by the following (please mind this argument as this is mostly an intuitive argument), the items which are absent from a store for a while, if introduced in huge quantity suddenly as might be suggested by the true demand (found out from EM or some other method) might not sell for two potential reasons\n1. The customers have shifted to competitive products and might not revert back or look for old product.\n2. They just don't look for that specific product in that store anymore as they didn't find that product in that store in recent past.", "votes": null }, { "id": "255559", "postDate": "12/09/2017 11:50:03", "content": "I'm afraid you are discussing something of no practical interest. How could you ever know what you call the 'true demand'? Think of it, and let us know if you find a way, because this would revolutionize retail industry worldwide.\nThis point was made already by @july520 but none reacted to it.
\n\nThe only thing that can be collected in practice, in addition to sales volumes, is whether an item is out of stock. This does not necessarily mean that supply was limited, as we cannot know if someone wanted to buy something and found an empty shelf instead. Anyway, this is a theoretical discussion as we don't have the out of stock data.
", "rawMarkdown": "I'm afraid you are discussing something of no practical interest. How could you ever know what you call the 'true demand'? Think of it, and let us know if you find a way, because this would revolutionize retail industry worldwide.\nThis point was made already by @july520 but none reacted to it. \n\nThe only thing that can be collected in practice, in addition to sales volumes, is whether an item is out of stock. This does not necessarily mean that supply was limited, as we cannot know if someone wanted to buy something and found an empty shelf instead. Anyway, this is a theoretical discussion as we don't have the out of stock data.", "votes": null }, { "id": "255568", "postDate": "12/09/2017 12:09:17", "content": "Hi CPMP, i agree the true demand can not be available in the data directly but that doesn't mean we cannot get some idea about it at all. I think if you assume 2 latent variables demand and supply with there respective distribution assumptions then by using Expectation maximisation you will be able to get the distribution parameters and hence can get some idea about the true demand and true supply. \nNote: Although, i have not tried it in this competition as the test data has the same limitation so doesn't make much of a difference but in theory you should be able to get more than just the limited demand from the given data.
", "rawMarkdown": "Hi CPMP, i agree the true demand can not be available in the data directly but that doesn't mean we cannot get some idea about it at all. I think if you assume 2 latent variables demand and supply with there respective distribution assumptions then by using Expectation maximisation you will be able to get the distribution parameters and hence can get some idea about the true demand and true supply. \nNote: Although, i have not tried it in this competition as the test data has the same limitation so doesn't make much of a difference but in theory you should be able to get more than just the limited demand from the given data.", "votes": null }, { "id": "255569", "postDate": "12/09/2017 12:11:15", "content": "How would you estimate the distribution of the demand latent variable? This is chicken and egg.
", "rawMarkdown": "How would you estimate the distribution of the demand latent variable? This is chicken and egg.", "votes": null }, { "id": "255573", "postDate": "12/09/2017 12:32:30", "content": "With all respect, I think this discussion can be shortened if you can just go through the EM algorithm and few of its applications like Gaussian Mixture model for clustering, HMM (where we try to get the distribution of hidden unobserved variable given the observed variable) and PLSA (getting topic distribution and word topic distribution, given only corpus of text).\nIn all these model you make some distribution assumptions about the unobserved latent variable and also about the structure of the graphical model and most of the time we get meaningful results.
\n\nIn this case for example the following probabilistic model might work
\n\nAssume (1+latent demand) and (1+latent supply) to be log normally distributed with mean and variance of these distribution given as a function of X, Pd,Ps (parameters).
You are given a random variable unit_sales = min(latent demand, latent supply)
So instead of finding argmax of P(unit_sales/X,P) with the assumption that (1+unit_sales) is log normally distributed with parameters P. We can find argmax of P(unit_sales/X,Pd,Ps) with the assumption mentioned in step 1.
This can be achieved by Expectation Maximisation algorithm (which is an iterative algorithm) to find Pd\nand Ps and hence the latent demand and latent supply distribution.
Thanks
", "rawMarkdown": "With all respect, I think this discussion can be shortened if you can just go through the EM algorithm and few of its applications like Gaussian Mixture model for clustering, HMM (where we try to get the distribution of hidden unobserved variable given the observed variable) and PLSA (getting topic distribution and word topic distribution, given only corpus of text).\nIn all these model you make some distribution assumptions about the unobserved latent variable and also about the structure of the graphical model and most of the time we get meaningful results.\n\nIn this case for example the following probabilistic model might work \n\n1. Assume (1+latent demand) and (1+latent supply) to be log normally distributed with mean and variance of these distribution given as a function of X, Pd,Ps (parameters). \n\n2. You are given a random variable unit_sales = min(latent demand, latent supply)\n\n3. So instead of finding argmax of P(unit_sales/X,P) with the assumption that (1+unit_sales) is log normally distributed with parameters P. We can find argmax of P(unit_sales/X,Pd,Ps) with the assumption mentioned in step 1. \n\n4. This can be achieved by Expectation Maximisation algorithm (which is an iterative algorithm) to find Pd\n and Ps and hence the latent demand and latent supply distribution. \n\nThanks", "votes": null }, { "id": "255832", "postDate": "12/10/2017 09:17:55", "content": "I'm afraid you do not answer my question: how would you estimate the true demand distribution. What you do is to use priors that do not look realistic at all, especially for supply. In a store, supply increases instantaneously when a delivery truck brings inventory in, then supply decreases steadily when sales happen. This is by no way near a log normal distributed random variable.
", "rawMarkdown": "I'm afraid you do not answer my question: how would you estimate the true demand distribution. What you do is to use priors that do not look realistic at all, especially for supply. In a store, supply increases instantaneously when a delivery truck brings inventory in, then supply decreases steadily when sales happen. This is by no way near a log normal distributed random variable.", "votes": null }, { "id": "255836", "postDate": "12/10/2017 09:31:32", "content": "I think i did answer your question CPMP, i suggest please read the above answer again (latent demand distribution will give me something of a True Demand) . And i think the lognormal distribution assumption with say parameter (mu,sigma) as a function of X,P (depending on the way you create) is perfectly right at a particular point in time (considering you can use yesterdays sales figures in X).
", "rawMarkdown": "I think i did answer your question CPMP, i suggest please read the above answer again (latent demand distribution will give me something of a True Demand) . And i think the lognormal distribution assumption with say parameter (mu,sigma) as a function of X,P (depending on the way you create) is perfectly right at a particular point in time (considering you can use yesterdays sales figures in X).", "votes": null }, { "id": "255852", "postDate": "12/10/2017 10:42:44", "content": "It's been a long time I have been patronized like this. Quite a refresher.
\n\nNot very useful still. Let me give you and advice as well: always check your assumptions. Seems you assume I don't know abut Bayesian approaches, seems you assume I don't know EM, HMM, Gaussian mixture models, etc. Seems you assume I never worked on inventory optimization for retailers. Please correct me if I'm wrong about your assumptions. See, I eat my own food: I check if my assumptions about you are right or wrong. Tell me.
\n\nAnd we'll see at competition end which model is useful, and which is not.
", "rawMarkdown": "It's been a long time I have been patronized like this. Quite a refresher. \n\nNot very useful still. Let me give you and advice as well: always check your assumptions. Seems you assume I don't know abut Bayesian approaches, seems you assume I don't know EM, HMM, Gaussian mixture models, etc. Seems you assume I never worked on inventory optimization for retailers. Please correct me if I'm wrong about your assumptions. See, I eat my own food: I check if my assumptions about you are right or wrong. Tell me.\n\nAnd we'll see at competition end which model is useful, and which is not.", "votes": null }, { "id": "255914", "postDate": "12/10/2017 15:11:45", "content": "Hey was not my intention to patronise you at all!
\n\nBut quite frankly i either did not understand your question or what seems wrong to you in the above EM method?
Also, I genuinely think we can use lognormal distribution for (supply given History, r.v. ) considering that this distribution can be assumed to be a stationary distribution if we are able to create good historical variables.
Thanks!
", "rawMarkdown": "Hey was not my intention to patronise you at all!\n\n1. But quite frankly i either did not understand your question or what seems wrong to you in the above EM method?\n\n2. Also, I genuinely think we can use lognormal distribution for (**supply given History**, r.v. ) considering that this distribution can be assumed to be a stationary distribution if we are able to create good historical variables. \n\nThanks!", "votes": null }, { "id": "255972", "postDate": "12/10/2017 18:47:54", "content": "Fair enough.
\n\nI tried to explain that supply follows a distribution that is not log normal. And given we don't know when delivery trucks come into each store, nor when items are out of stock, I think it is very difficult to estimate supply distribution. If you find a way then you have a bright future in retail industry.
", "rawMarkdown": "Fair enough.\n\nI tried to explain that supply follows a distribution that is not log normal. And given we don't know when delivery trucks come into each store, nor when items are out of stock, I think it is very difficult to estimate supply distribution. If you find a way then you have a bright future in retail industry.", "votes": null }, { "id": "255977", "postDate": "12/10/2017 19:33:00", "content": "I would think that you would enforce the modeling onto the Demand Side, and use a boolean soup for the supply function.
", "rawMarkdown": "I would think that you would enforce the modeling onto the Demand Side, and use a boolean soup for the supply function.", "votes": null }, { "id": "256344", "postDate": "12/11/2017 19:10:20", "content": "Hey Michael, please elaborate. Did not understand what you said.
", "rawMarkdown": "Hey Michael, please elaborate. Did not understand what you said.", "votes": null }, { "id": "256500", "postDate": "12/12/2017 04:27:25", "content": "Basically, supply and demand are working on completely different kinds of fundamentals. Personally, I think modeling Demand might be easier than CPMP suggests but I do agree that Supply is just not going to happen in the kind of way you're describing.
", "rawMarkdown": "Basically, supply and demand are working on completely different kinds of fundamentals. Personally, I think modeling Demand might be easier than CPMP suggests but I do agree that Supply is just not going to happen in the kind of way you're describing.", "votes": null } ], "comments": [ { "id": 249846, "author_name": "wangziwei02", "author_url": "", "post_date": "11/29/2017 10:13:01", "content": "what did you mean by the unit demand is actually min(demand, supply) in both the test data and the train data ? How did you get the demand or the supply? Can you explain to me in detail ：）
", "votes": null, "replies": [] }, { "id": 249949, "author_name": "nit458", "author_url": "", "post_date": "11/29/2017 14:44:56", "content": "I think the unit_sales (observed) = min(demand , supply) (hypothetical demand and supply variable, not observed) in both train and test data. Because the sales will be limited by the units of that item present in that store on that day.
\n\nI was hoping that atleast demand is not limited by supply in the test sample , but i think it is, as the scores do not drop a lot from train and test.
\n\nInteresting scenario would be to predict the True demand (which is unobserved in the train sample) instead of the limited demand (unit_sales). I was thinking if this can be handled by assuming a latent variable and then using EM.
", "votes": null, "replies": [ { "id": 249984, "author_name": "wangziwei02", "author_url": "", "post_date": "11/29/2017 15:40:50", "content": "Brilliant！
", "votes": null, "replies": [] }, { "id": 252025, "author_name": "ceshine", "author_url": "", "post_date": "12/02/2017 04:56:31", "content": "Interesting idea. In my understanding the test dataset is collected the same way as the train dataset, so the target is still the actual sales, not theoretical demand. But I wonder if we could find a way to predict an item in a certain area is running out of supply so the model can accommodate that constraint.
", "votes": null, "replies": [] }, { "id": 253519, "author_name": "nit458", "author_url": "", "post_date": "12/05/2017 05:14:35", "content": "Hey ceshine, yes the test data is similar to train data. As i though EM iterations might work to estimate the latent demand. But wont be useful for this competition as the test and train are both constraint by the supply
", "votes": null, "replies": [] }, { "id": 255311, "author_name": "ericdfr", "author_url": "", "post_date": "12/08/2017 20:29:40", "content": "\n\n\nLazyGUY wrote
\n \n\n\nyes the test data is similar to train data
\n
I'm not agree with that.
\n\nAs I highlighted in this kernel (https://www.kaggle.com/ericdfr/new-items-in-stores/), the count of new store/item combo in the test dataset is unusual.
\n\nAs FengLi spotted in this topic (https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/42964) :
\n\n\n\n\nThey just create the test data for convenience by including all item/store combo although like store A never sells item 1
\n
I've asked Corporación Favorita about these 40k+ new item/store combo but they haven't yet answered. :-)
\n\nI'm pretty sure that demand is limited by supply in the test set. I give an example here : https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/43083
", "votes": null, "replies": [] }, { "id": 255554, "author_name": "nit458", "author_url": "", "post_date": "12/09/2017 11:24:35", "content": "Hey i meant the test data and train are similar in the manner that they both have demand limited by supply. Also, i think most of the combinations which you think do not exist in the train data and exist in test data will have zero sales even in the test period, because in my opinion the data for test and train are not created differently. I don't think they did any kind of test and control in the test period or removed observations for certain combinations from train data.
\n\nI believe this because the performance although drops a little but not a lot from val to test data.
", "votes": null, "replies": [] } ] }, { "id": 250506, "author_name": "july520", "author_url": "", "post_date": "11/30/2017 04:16:23", "content": "how could we know demand and supply？
", "votes": null, "replies": [] }, { "id": 253195, "author_name": "tanguy73", "author_url": "", "post_date": "12/04/2017 15:44:01", "content": "I actually don't get the goal of that competition. What's the point of predicting the unit_sales? IMO the demand shall be predicted.
\n\nCorrect me if I am wrong but training a model using the unit_sales in the loss indicator will result in a prediction that reproduces some of the \"incorrect supply previsions\" made by the previous methods.
\n\nIf unit_sales = min(demand, supply) = 0, it means demand=0 and suply>0 OR supply=0 and demand>0. The two cases result in a loss of money for the company, but the actions to be undertaken are totally different (first case: reducing supply, second case: increase supply). What am I suppose to do as the owner of the shop if the predicted unit_sales=0 ? Choose the action in function of the current supply? How much should I buy if current supply=0?
", "votes": null, "replies": [] }, { "id": 253518, "author_name": "nit458", "author_url": "", "post_date": "12/05/2017 05:12:37", "content": "@tanguy73: i agree predicting unit_sales will definitely has lot of old logic bias.
\n\nBut there is some logic in predicting it as well (its not interesting to predict it but it is not completely baseless). Customers stickiness explains it somewhat, i.e. the products which were sold in recent past in a store are the items which are more likely to be searched for by the customers (call it stickiness or customers expectation that he/she will find that particular item in that shop and hence goes to the shop searching for it) and the items which were not in stock in recent past will most likely not be searched.
\n\nSo, the old supply will set the future selling patterns on a store.
\n\nAlthough, i also agree that predicting the correct demand would be much more helpful in the long run, but the company should not use the correct demand from the start of this new strategy, they should gradually increase the demand to the correct demand.
\n\nLet me know your comments!
", "votes": null, "replies": [ { "id": 254242, "author_name": "tanguy73", "author_url": "", "post_date": "12/06/2017 14:43:14", "content": "Not sure how you deduce the customer stickiness from the unit_sales, how to quantify it, and which actions to undertake knowing this. But I am a total noob in retailing.
\n\nStill I am struggling with the fact that a model which will predict the actual demand will be hardly penalized by the old logic bias (=old supply mismatches) made in the past. I was thrilled to join this competition but thinking that a good model will perform badly according to the competition ranking system kinda put an end to my motivation.
\n\nBut maybe I missed something cause people don't seem to mind this.
", "votes": null, "replies": [] }, { "id": 255557, "author_name": "nit458", "author_url": "", "post_date": "12/09/2017 11:37:26", "content": "By old logic i mean the logic used to determine the supply of various items to various stores. And unit_sales is demand limited by supply hence using predicted unit_sales to set future demand forecast is biasing the forecast by the old logic and should not work logically.
\n\nBut on the other hand i think its not completely useless to predict it as well (considering it will be difficult to predict the actual demand because of data limitation).
\n\nI explain that by the following (please mind this argument as this is mostly an intuitive argument), the items which are absent from a store for a while, if introduced in huge quantity suddenly as might be suggested by the true demand (found out from EM or some other method) might not sell for two potential reasons\n1. The customers have shifted to competitive products and might not revert back or look for old product.\n2. They just don't look for that specific product in that store anymore as they didn't find that product in that store in recent past.
", "votes": null, "replies": [] } ] }, { "id": 255559, "author_name": "cpmpml", "author_url": "", "post_date": "12/09/2017 11:50:03", "content": "I'm afraid you are discussing something of no practical interest. How could you ever know what you call the 'true demand'? Think of it, and let us know if you find a way, because this would revolutionize retail industry worldwide.\nThis point was made already by @july520 but none reacted to it.
\n\nThe only thing that can be collected in practice, in addition to sales volumes, is whether an item is out of stock. This does not necessarily mean that supply was limited, as we cannot know if someone wanted to buy something and found an empty shelf instead. Anyway, this is a theoretical discussion as we don't have the out of stock data.
", "votes": null, "replies": [ { "id": 255568, "author_name": "nit458", "author_url": "", "post_date": "12/09/2017 12:09:17", "content": "Hi CPMP, i agree the true demand can not be available in the data directly but that doesn't mean we cannot get some idea about it at all. I think if you assume 2 latent variables demand and supply with there respective distribution assumptions then by using Expectation maximisation you will be able to get the distribution parameters and hence can get some idea about the true demand and true supply. \nNote: Although, i have not tried it in this competition as the test data has the same limitation so doesn't make much of a difference but in theory you should be able to get more than just the limited demand from the given data.
", "votes": null, "replies": [] }, { "id": 255569, "author_name": "cpmpml", "author_url": "", "post_date": "12/09/2017 12:11:15", "content": "How would you estimate the distribution of the demand latent variable? This is chicken and egg.
", "votes": null, "replies": [] }, { "id": 255573, "author_name": "nit458", "author_url": "", "post_date": "12/09/2017 12:32:30", "content": "With all respect, I think this discussion can be shortened if you can just go through the EM algorithm and few of its applications like Gaussian Mixture model for clustering, HMM (where we try to get the distribution of hidden unobserved variable given the observed variable) and PLSA (getting topic distribution and word topic distribution, given only corpus of text).\nIn all these model you make some distribution assumptions about the unobserved latent variable and also about the structure of the graphical model and most of the time we get meaningful results.
\n\nIn this case for example the following probabilistic model might work
\n\nAssume (1+latent demand) and (1+latent supply) to be log normally distributed with mean and variance of these distribution given as a function of X, Pd,Ps (parameters).
You are given a random variable unit_sales = min(latent demand, latent supply)
So instead of finding argmax of P(unit_sales/X,P) with the assumption that (1+unit_sales) is log normally distributed with parameters P. We can find argmax of P(unit_sales/X,Pd,Ps) with the assumption mentioned in step 1.
This can be achieved by Expectation Maximisation algorithm (which is an iterative algorithm) to find Pd\nand Ps and hence the latent demand and latent supply distribution.
Thanks
", "votes": null, "replies": [] }, { "id": 255832, "author_name": "cpmpml", "author_url": "", "post_date": "12/10/2017 09:17:55", "content": "I'm afraid you do not answer my question: how would you estimate the true demand distribution. What you do is to use priors that do not look realistic at all, especially for supply. In a store, supply increases instantaneously when a delivery truck brings inventory in, then supply decreases steadily when sales happen. This is by no way near a log normal distributed random variable.
", "votes": null, "replies": [] }, { "id": 255836, "author_name": "nit458", "author_url": "", "post_date": "12/10/2017 09:31:32", "content": "I think i did answer your question CPMP, i suggest please read the above answer again (latent demand distribution will give me something of a True Demand) . And i think the lognormal distribution assumption with say parameter (mu,sigma) as a function of X,P (depending on the way you create) is perfectly right at a particular point in time (considering you can use yesterdays sales figures in X).
", "votes": null, "replies": [] }, { "id": 255852, "author_name": "cpmpml", "author_url": "", "post_date": "12/10/2017 10:42:44", "content": "It's been a long time I have been patronized like this. Quite a refresher.
\n\nNot very useful still. Let me give you and advice as well: always check your assumptions. Seems you assume I don't know abut Bayesian approaches, seems you assume I don't know EM, HMM, Gaussian mixture models, etc. Seems you assume I never worked on inventory optimization for retailers. Please correct me if I'm wrong about your assumptions. See, I eat my own food: I check if my assumptions about you are right or wrong. Tell me.
\n\nAnd we'll see at competition end which model is useful, and which is not.
", "votes": null, "replies": [] }, { "id": 255914, "author_name": "nit458", "author_url": "", "post_date": "12/10/2017 15:11:45", "content": "Hey was not my intention to patronise you at all!
\n\nBut quite frankly i either did not understand your question or what seems wrong to you in the above EM method?
Also, I genuinely think we can use lognormal distribution for (supply given History, r.v. ) considering that this distribution can be assumed to be a stationary distribution if we are able to create good historical variables.
Thanks!
", "votes": null, "replies": [] }, { "id": 255972, "author_name": "cpmpml", "author_url": "", "post_date": "12/10/2017 18:47:54", "content": "Fair enough.
\n\nI tried to explain that supply follows a distribution that is not log normal. And given we don't know when delivery trucks come into each store, nor when items are out of stock, I think it is very difficult to estimate supply distribution. If you find a way then you have a bright future in retail industry.
", "votes": null, "replies": [] }, { "id": 255977, "author_name": "vannak", "author_url": "", "post_date": "12/10/2017 19:33:00", "content": "I would think that you would enforce the modeling onto the Demand Side, and use a boolean soup for the supply function.
", "votes": null, "replies": [] }, { "id": 256344, "author_name": "nit458", "author_url": "", "post_date": "12/11/2017 19:10:20", "content": "Hey Michael, please elaborate. Did not understand what you said.
", "votes": null, "replies": [] }, { "id": 256500, "author_name": "vannak", "author_url": "", "post_date": "12/12/2017 04:27:25", "content": "Basically, supply and demand are working on completely different kinds of fundamentals. Personally, I think modeling Demand might be easier than CPMP suggests but I do agree that Supply is just not going to happen in the kind of way you're describing.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "248278": "As i understand the unit demand is actually min(demand, supply) in both the test data and the train data. This problem would be a much more interesting to think if test data somehow didn't have this limitation, i.e. the competition is to predict the 'True' demand of items given only conditional demand.\n\nOne possible solution according to me can be to assume a latent variable which tells me whether demand >= supply or not. And then trying to iteratively find out the probability of this latent variable being 1 and True demand (may be using some adaptation of EM algorithm)\n\nLet me know your comments", "249846": "what did you mean by the unit demand is actually min(demand, supply) in both the test data and the train data ? How did you get the demand or the supply? Can you explain to me in detail ：）", "249949": "I think the unit_sales (observed) = min(demand , supply) (hypothetical demand and supply variable, not observed) in both train and test data. Because the sales will be limited by the units of that item present in that store on that day.\n\nI was hoping that atleast demand is not limited by supply in the test sample , but i think it is, as the scores do not drop a lot from train and test. \n\nInteresting scenario would be to predict the True demand (which is unobserved in the train sample) instead of the limited demand (unit_sales). I was thinking if this can be handled by assuming a latent variable and then using EM.", "249984": "Brilliant！", "250506": "how could we know demand and supply？", "252025": "Interesting idea. In my understanding the test dataset is collected the same way as the train dataset, so the target is still the actual sales, not theoretical demand. But I wonder if we could find a way to predict an item in a certain area is running out of supply so the model can accommodate that constraint.", "253195": "I actually don't get the goal of that competition. What's the point of predicting the unit_sales? IMO the demand shall be predicted.\n\nCorrect me if I am wrong but training a model using the unit_sales in the loss indicator will result in a prediction that reproduces some of the \"incorrect supply previsions\" made by the previous methods.\n\nIf unit_sales = min(demand, supply) = 0, it means demand=0 and suply>0 OR supply=0 and demand>0. The two cases result in a loss of money for the company, but the actions to be undertaken are totally different (first case: reducing supply, second case: increase supply). What am I suppose to do as the owner of the shop if the predicted unit_sales=0 ? Choose the action in function of the current supply? How much should I buy if current supply=0?", "253518": "tanguy73: i agree predicting unit_sales will definitely has lot of old logic bias.\n\nBut there is some logic in predicting it as well (its not interesting to predict it but it is not completely baseless). Customers stickiness explains it somewhat, i.e. the products which were sold in recent past in a store are the items which are more likely to be searched for by the customers (call it stickiness or customers expectation that he/she will find that particular item in that shop and hence goes to the shop searching for it) and the items which were not in stock in recent past will most likely not be searched.\n\nSo, the old supply will set the future selling patterns on a store.\n\nAlthough, i also agree that predicting the correct demand would be much more helpful in the long run, but the company should not use the correct demand from the start of this new strategy, they should gradually increase the demand to the correct demand.\n\nLet me know your comments!", "253519": "Hey ceshine, yes the test data is similar to train data. As i though EM iterations might work to estimate the latent demand. But wont be useful for this competition as the test and train are both constraint by the supply", "254242": "Not sure how you deduce the customer stickiness from the unit_sales, how to quantify it, and which actions to undertake knowing this. But I am a total noob in retailing.\n\nStill I am struggling with the fact that a model which will predict the actual *demand* will be hardly penalized by the old logic bias (=old supply mismatches) made in the past. I was thrilled to join this competition but thinking that a *good* model will perform badly according to the competition ranking system kinda put an end to my motivation.\n\nBut maybe I missed something cause people don't seem to mind this.", "255311": "> **LazyGUY wrote**\n> \n> > yes the test data is similar to train data\n\nI'm not agree with that. \n\nAs I highlighted in this kernel ([https://www.kaggle.com/ericdfr/new-items-in-stores/][1]), the count of new store/item combo in the test dataset is unusual. \n\nAs FengLi spotted in this topic ([https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/42964][2]) :\n> They just create the test data for convenience by including all item/store combo although like store A never sells item 1 \n\nI've asked Corporación Favorita about these 40k+ new item/store combo but they haven't yet answered. :-)\n\nI'm pretty sure that demand is limited by supply in the test set. I give an example here : [https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/43083][3]\n\n\n [1]: https://www.kaggle.com/ericdfr/new-items-in-stores/\n [2]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/42964\n [3]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/43083", "255554": "Hey i meant the test data and train are similar in the manner that they both have demand limited by supply. Also, i think most of the combinations which you think do not exist in the train data and exist in test data will have zero sales even in the test period, because in my opinion the data for test and train are not created differently. I don't think they did any kind of test and control in the test period or removed observations for certain combinations from train data. \n\nI believe this because the performance although drops a little but not a lot from val to test data.", "255557": "By old logic i mean the logic used to determine the supply of various items to various stores. And unit_sales is demand limited by supply hence using predicted unit_sales to set future demand forecast is biasing the forecast by the old logic and should not work logically.\n\nBut on the other hand i think its not completely useless to predict it as well (considering it will be difficult to predict the actual demand because of data limitation). \n\nI explain that by the following (please mind this argument as this is mostly an intuitive argument), the items which are absent from a store for a while, if introduced in huge quantity suddenly as might be suggested by the true demand (found out from EM or some other method) might not sell for two potential reasons\n1. The customers have shifted to competitive products and might not revert back or look for old product.\n2. They just don't look for that specific product in that store anymore as they didn't find that product in that store in recent past.", "255559": "I'm afraid you are discussing something of no practical interest. How could you ever know what you call the 'true demand'? Think of it, and let us know if you find a way, because this would revolutionize retail industry worldwide.\nThis point was made already by @july520 but none reacted to it. \n\nThe only thing that can be collected in practice, in addition to sales volumes, is whether an item is out of stock. This does not necessarily mean that supply was limited, as we cannot know if someone wanted to buy something and found an empty shelf instead. Anyway, this is a theoretical discussion as we don't have the out of stock data.", "255568": "Hi CPMP, i agree the true demand can not be available in the data directly but that doesn't mean we cannot get some idea about it at all. I think if you assume 2 latent variables demand and supply with there respective distribution assumptions then by using Expectation maximisation you will be able to get the distribution parameters and hence can get some idea about the true demand and true supply. \nNote: Although, i have not tried it in this competition as the test data has the same limitation so doesn't make much of a difference but in theory you should be able to get more than just the limited demand from the given data.", "255569": "How would you estimate the distribution of the demand latent variable? This is chicken and egg.", "255573": "With all respect, I think this discussion can be shortened if you can just go through the EM algorithm and few of its applications like Gaussian Mixture model for clustering, HMM (where we try to get the distribution of hidden unobserved variable given the observed variable) and PLSA (getting topic distribution and word topic distribution, given only corpus of text).\nIn all these model you make some distribution assumptions about the unobserved latent variable and also about the structure of the graphical model and most of the time we get meaningful results.\n\nIn this case for example the following probabilistic model might work \n\n1. Assume (1+latent demand) and (1+latent supply) to be log normally distributed with mean and variance of these distribution given as a function of X, Pd,Ps (parameters). \n\n2. You are given a random variable unit_sales = min(latent demand, latent supply)\n\n3. So instead of finding argmax of P(unit_sales/X,P) with the assumption that (1+unit_sales) is log normally distributed with parameters P. We can find argmax of P(unit_sales/X,Pd,Ps) with the assumption mentioned in step 1. \n\n4. This can be achieved by Expectation Maximisation algorithm (which is an iterative algorithm) to find Pd\n and Ps and hence the latent demand and latent supply distribution. \n\nThanks", "255832": "I'm afraid you do not answer my question: how would you estimate the true demand distribution. What you do is to use priors that do not look realistic at all, especially for supply. In a store, supply increases instantaneously when a delivery truck brings inventory in, then supply decreases steadily when sales happen. This is by no way near a log normal distributed random variable.", "255836": "I think i did answer your question CPMP, i suggest please read the above answer again (latent demand distribution will give me something of a True Demand) . And i think the lognormal distribution assumption with say parameter (mu,sigma) as a function of X,P (depending on the way you create) is perfectly right at a particular point in time (considering you can use yesterdays sales figures in X).", "255852": "It's been a long time I have been patronized like this. Quite a refresher. \n\nNot very useful still. Let me give you and advice as well: always check your assumptions. Seems you assume I don't know abut Bayesian approaches, seems you assume I don't know EM, HMM, Gaussian mixture models, etc. Seems you assume I never worked on inventory optimization for retailers. Please correct me if I'm wrong about your assumptions. See, I eat my own food: I check if my assumptions about you are right or wrong. Tell me.\n\nAnd we'll see at competition end which model is useful, and which is not.", "255914": "Hey was not my intention to patronise you at all!\n\n1. But quite frankly i either did not understand your question or what seems wrong to you in the above EM method?\n\n2. Also, I genuinely think we can use lognormal distribution for (**supply given History**, r.v. ) considering that this distribution can be assumed to be a stationary distribution if we are able to create good historical variables. \n\nThanks!", "255972": "Fair enough.\n\nI tried to explain that supply follows a distribution that is not log normal. And given we don't know when delivery trucks come into each store, nor when items are out of stock, I think it is very difficult to estimate supply distribution. If you find a way then you have a bright future in retail industry.", "255977": "I would think that you would enforce the modeling onto the Demand Side, and use a boolean soup for the supply function.", "256344": "Hey Michael, please elaborate. Did not understand what you said.", "256500": "Basically, supply and demand are working on completely different kinds of fundamentals. Personally, I think modeling Demand might be easier than CPMP suggests but I do agree that Supply is just not going to happen in the kind of way you're describing." }, "source": "meta" }