I've been testing out a few different strategies:
\n\nI haven't had much luck with 2 or 3. Although I haven't tried much in regards to NNs yet. So take that with a grain of salt.
\n\nI have, however, had some really good results using FB's Prophet. How is everyone else fairing? Similar results?
", "messages": [ { "id": "243461", "postDate": "11/14/2017 04:32:55", "content": "I've been testing out a few different strategies:
\n\nI haven't had much luck with 2 or 3. Although I haven't tried much in regards to NNs yet. So take that with a grain of salt.
\n\nI have, however, had some really good results using FB's Prophet. How is everyone else fairing? Similar results?
", "rawMarkdown": "I've been testing out a few different strategies:\n\n 1. Using Prophet,\n 2. Using LSTMs,\n 3. Using MLPs.\n\nI haven't had much luck with 2 or 3. Although I haven't tried much in regards to NNs yet. So take that with a grain of salt.\n\nI have, however, had some really good results using FB's Prophet. How is everyone else fairing? Similar results?", "votes": null }, { "id": "243469", "postDate": "11/14/2017 05:19:48", "content": "I am not familiar with Prophet, would you like to share some detail?
", "rawMarkdown": "I am not familiar with Prophet, would you like to share some detail?", "votes": null }, { "id": "243516", "postDate": "11/14/2017 08:15:31", "content": "https://research.fb.com/prophet-forecasting-at-scale/
", "rawMarkdown": "https://research.fb.com/prophet-forecasting-at-scale/", "votes": null }, { "id": "243520", "postDate": "11/14/2017 08:18:41", "content": "did u add more new features?
", "rawMarkdown": "did u add more new features?", "votes": null }, { "id": "243574", "postDate": "11/14/2017 12:41:33", "content": "Haven't tried NN yet, I have decent results with prophet - but still inferior to the median-based approach. My best guess is that several of the sales series are almost random walks => not much structure to extract using prophet.
", "rawMarkdown": "Haven't tried NN yet, I have decent results with prophet - but still inferior to the median-based approach. My best guess is that several of the sales series are almost random walks => not much structure to extract using prophet.", "votes": null }, { "id": "243635", "postDate": "11/14/2017 14:59:31", "content": "\n\n\nMy best guess is that several of the sales series are almost random walks ...
\n
I noticed this as well. I was thinking to handle series with a RMSLE above a certain threshold (or too many missing values) differently. Some of this data looks contrived to me --but it could simply be incomplete.
", "rawMarkdown": "> My best guess is that several of the sales series are almost random walks ...\n\nI noticed this as well. I was thinking to handle series with a RMSLE above a certain threshold (or too many missing values) differently. Some of this data looks contrived to me --but it could simply be incomplete.", "votes": null }, { "id": "243636", "postDate": "11/14/2017 15:01:13", "content": "You get two variables (one feature and the target) with Prophet (it handles basic 1-dim'l time series):
\n\nProphet looks interesting and good to know about it, as I am new to Kaggle, I am trying a few regressors. As prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition, and as have been highlighted by facebook, the date is the major input to be used (I have not further than the front page at https://research.fb.com/prophet-forecasting-at-scale/), am I missing something here?
", "rawMarkdown": "Prophet looks interesting and good to know about it, as I am new to Kaggle, I am trying a few regressors. As prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition, and as have been highlighted by facebook, the date is the major input to be used (I have not further than the front page at https://research.fb.com/prophet-forecasting-at-scale/), am I missing something here?", "votes": null }, { "id": "246911", "postDate": "11/22/2017 00:11:21", "content": "I recently managed to get an lb 0.533 out of an lstm model. What have the other methods scored for you guys?
", "rawMarkdown": "I recently managed to get an lb 0.533 out of an lstm model. What have the other methods scored for you guys?", "votes": null }, { "id": "248544", "postDate": "11/26/2017 12:33:15", "content": "Hi Michael, i am new to machine learning. I finally managed to have LSTM working but for now i have it only per product and store (not generic), so if i want to predict test i don't see any other option than iterate thru stores and products, generate different models and then make the prediction. How did you managed to predict your data? Thanks in advance
", "rawMarkdown": "Hi Michael, i am new to machine learning. I finally managed to have LSTM working but for now i have it only per product and store (not generic), so if i want to predict test i don't see any other option than iterate thru stores and products, generate different models and then make the prediction. How did you managed to predict your data? Thanks in advance", "votes": null }, { "id": "248554", "postDate": "11/26/2017 13:30:42", "content": "You should try to assemble a 2D array of sales. Each row belongs to a unique store-product pair, and each column is a day of the history given. \nIn my model, I iterate over every row and randomly select a region to feed into the LSTM and have it predict the next 16 values after.
\n\nBasically, however you got the single store-item pair's history, do the same for all of them. Train the LSTM on the aggregate model. To get this to work successfully, the model will probably need to know the different stores and products so feed those in as Embeddings.
", "rawMarkdown": "You should try to assemble a 2D array of sales. Each row belongs to a unique store-product pair, and each column is a day of the history given. \nIn my model, I iterate over every row and randomly select a region to feed into the LSTM and have it predict the next 16 values after. \n\nBasically, however you got the single store-item pair's history, do the same for all of them. Train the LSTM on the aggregate model. To get this to work successfully, the model will probably need to know the different stores and products so feed those in as Embeddings.", "votes": null }, { "id": "248631", "postDate": "11/26/2017 16:30:28", "content": "Nice approach! Would you mind clarifying?
\n\nAs I understand it each row represents a store-product pair and each column is a day. You choose a random region and have the LSTM predict the next 16 values after. Fine.
\n\nAfter this you explain \"train the LSTM on the aggregate model\". Aggregate of what? Perhaps you mean you input to the LSTM many of these random regions for just one store-item pair? And then you do this for all pairs?
", "rawMarkdown": "Nice approach! Would you mind clarifying?\n\nAs I understand it each row represents a store-product pair and each column is a day. You choose a random region and have the LSTM predict the next 16 values after. Fine.\n\nAfter this you explain \"train the LSTM on the aggregate model\". Aggregate of what? Perhaps you mean you input to the LSTM many of these random regions for just one store-item pair? And then you do this for all pairs?", "votes": null }, { "id": "248650", "postDate": "11/26/2017 17:29:18", "content": "Hi, I will try using Prophet thanks for pointing to it. How is the standard xgboost performing, in Rosmann competition it gave good scores to a similar problem for store level sales data. I am not getting good result with xgboost here.
", "rawMarkdown": "Hi, I will try using Prophet thanks for pointing to it. How is the standard xgboost performing, in Rosmann competition it gave good scores to a similar problem for store level sales data. I am not getting good result with xgboost here.", "votes": null }, { "id": "248663", "postDate": "11/26/2017 18:02:51", "content": "thanks for the clarification. BTW, one thing that i notice in the LSTM prediction, if i want to predict n item in advance, all of them seems to have a high correlation to the last \"known\" unit sold , and there is no much variation over the predictions, is this the same case for you?
", "rawMarkdown": "thanks for the clarification. BTW, one thing that i notice in the LSTM prediction, if i want to predict n item in advance, all of them seems to have a high correlation to the last \"known\" unit sold , and there is no much variation over the predictions, is this the same case for you?", "votes": null }, { "id": "248710", "postDate": "11/26/2017 20:11:13", "content": "I meant the model trained on the aggregate data of all store-item pairs. On each eopch I take one window from each store-item pair, though I'm sure a full history or multiple samples might make sense as well.
", "rawMarkdown": "I meant the model trained on the aggregate data of all store-item pairs. On each eopch I take one window from each store-item pair, though I'm sure a full history or multiple samples might make sense as well.", "votes": null }, { "id": "248714", "postDate": "11/26/2017 20:36:12", "content": "@Dani that's a pretty common problem with Time Series Prediction. Adding in more relevant data is the best solution.
", "rawMarkdown": "Dani that's a pretty common problem with Time Series Prediction. Adding in more relevant data is the best solution.", "votes": null }, { "id": "249040", "postDate": "11/27/2017 15:33:27", "content": "I tried one \"sanity check\" implementation of xgboost. It didn't perform well and I haven't come back to fix it yet. I'm getting the best results out of a combination of LSTMs, Prophet, and Linear Models (according to CV scores w/o normalization).
", "rawMarkdown": "I tried one \"sanity check\" implementation of xgboost. It didn't perform well and I haven't come back to fix it yet. I'm getting the best results out of a combination of LSTMs, Prophet, and Linear Models (according to CV scores w/o normalization).", "votes": null }, { "id": "249044", "postDate": "11/27/2017 15:43:34", "content": "I would use Prophet to create new features (and, in general, extend the dataset). I wouldn't use it directly for the prediction.
\n\n\n\n\nAs prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition...
\n
While that is true... the most important feature in a time-series is what occurred in the recent past. That is the whole idea underlying the structure of sARIMA models (and all other time-series models). So I expect that the time-series itself will be the most important data. Hence, LSTMs, sARIMA models, Prophet, GARCH, etc. should do well here.
", "rawMarkdown": "I would use Prophet to create new features (and, in general, extend the dataset). I wouldn't use it directly for the prediction.\n\n> As prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition...\n\nWhile that is true... the most important feature in a time-series is what occurred in the recent past. That is the whole idea underlying the structure of sARIMA models (and all other time-series models). So I expect that the time-series itself will be the most important data. Hence, LSTMs, sARIMA models, Prophet, GARCH, etc. should do well here.", "votes": null }, { "id": "249046", "postDate": "11/27/2017 15:49:21", "content": "May I ask how fast is your implementation of Prophet? I am trying it now with part of the Favorita data, not sure how would it scale up! My guess is that it will take hours to finish.
", "rawMarkdown": "May I ask how fast is your implementation of Prophet? I am trying it now with part of the Favorita data, not sure how would it scale up! My guess is that it will take hours to finish.", "votes": null }, { "id": "249080", "postDate": "11/27/2017 16:57:34", "content": "I tried using prophet as well. It takes 4 seconds by item x store combination for me with an i7 + 16Go RAM machine.\nSo when you extrapolate that to 200.000 items x store combinations, it takes around 10 days to complete...
\n\nOne way to make Prophet slightly faster is to set the parameter \"uncertainty_samples = 0\" (without this, it takes 6.5 seconds by item x store...)
\n\nI was putting great hope into prophet, the results look quite good when I check some items : the promotions and holidays are well accounted for, the weekly and yearly seasonality as well.\nBut when I submitted the results, I was quite disappointed because it scored lower than using just a simple xgboost model.
", "rawMarkdown": "I tried using prophet as well. It takes 4 seconds by item x store combination for me with an i7 + 16Go RAM machine.\nSo when you extrapolate that to 200.000 items x store combinations, it takes around 10 days to complete...\n\nOne way to make Prophet slightly faster is to set the parameter \"uncertainty_samples = 0\" (without this, it takes 6.5 seconds by item x store...)\n\nI was putting great hope into prophet, the results look quite good when I check some items : the promotions and holidays are well accounted for, the weekly and yearly seasonality as well.\nBut when I submitted the results, I was quite disappointed because it scored lower than using just a simple xgboost model.", "votes": null }, { "id": "249084", "postDate": "11/27/2017 17:03:44", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "249100", "postDate": "11/27/2017 17:59:18", "content": "@ChristopheS \n10 days' time is frustrating! \nOne reason for this slowness is due to stan's stdout (Initial log joint probability = -139.766\nIteration 1. Log joint probability = 15.5265. Improved by 155.292.\nIteration 2. Log joint probability = 25.1928. Improved by 9.66629\n....)\nI could not toggle this off!
\n\nMy guess, however, is that the good Prophet results are due to overfitting, hence the unseen/test results scored lower than expected.\nPart of the problem could be due to the negative sales (refunds/returns), which I am currently struggling to model (I think there is no way to correctly predict these -ve's in the test set)
\n\nI wonder if you have considered all the dates in the training of FB Prophet?!!
", "rawMarkdown": "ChristopheS \n10 days' time is frustrating! \nOne reason for this slowness is due to stan's stdout (Initial log joint probability = -139.766\nIteration 1. Log joint probability = 15.5265. Improved by 155.292.\nIteration 2. Log joint probability = 25.1928. Improved by 9.66629\n....)\nI could not toggle this off!\n\nMy guess, however, is that the good Prophet results are due to overfitting, hence the unseen/test results scored lower than expected.\nPart of the problem could be due to the negative sales (refunds/returns), which I am currently struggling to model (I think there is no way to correctly predict these -ve's in the test set)\n\nI wonder if you have considered all the dates in the training of FB Prophet?!!", "votes": null }, { "id": "249440", "postDate": "11/28/2017 13:23:31", "content": "No, I just used data for the last 2 years and a half.\nI doubt expanding it to 4 years and a half would improve the results, but who knows...
", "rawMarkdown": "No, I just used data for the last 2 years and a half.\nI doubt expanding it to 4 years and a half would improve the results, but who knows...", "votes": null }, { "id": "249512", "postDate": "11/28/2017 16:45:00", "content": "@ChristopheS: have you considered running it parallel? In R, you essentially you can adapt the public time series kernel:
\n\nhttps://www.kaggle.com/dongxu027/time-series-ets-starter-lb-0-590
", "rawMarkdown": "ChristopheS: have you considered running it parallel? In R, you essentially you can adapt the public time series kernel:\n\nhttps://www.kaggle.com/dongxu027/time-series-ets-starter-lb-0-590", "votes": null }, { "id": "249520", "postDate": "11/28/2017 16:56:59", "content": "No, I searched a little bit how I could run it parallel but I didn't find anything to do it. I'm using Python, so unfortunately the post you refer doesn't solve my problem.
\n\nI tried using a free EC2 instance, but the free tier only gives you a machine with 1Go RAM and you need a minimum of 4Go of RAM to install Prophet (and it would be very long anyway).
", "rawMarkdown": "No, I searched a little bit how I could run it parallel but I didn't find anything to do it. I'm using Python, so unfortunately the post you refer doesn't solve my problem.\n\nI tried using a free EC2 instance, but the free tier only gives you a machine with 1Go RAM and you need a minimum of 4Go of RAM to install Prophet (and it would be very long anyway).", "votes": null }, { "id": "255316", "postDate": "12/08/2017 20:43:16", "content": "I've forecasted the 200k+ item/store combo on a c5.18.xlarge (72 last gen cpus). :-)\nI tooks 5 hours. Note that there is an iowait limitation and only 50+ cpus are really used.
\nBut it costs around 9$ on a spot instance. :-/
Hi Michael, my prediction using LSTM for one store and one item does not work very well. Is it because the data is two small (2 year, 712 observations)?
\n\nTo expand to all the item and stores, I understand your idea that each row is a store-item pair. For the columns, when I build an LSTM for one store and one item, I use a look-back of 12 and have 637 rows and 12 columns as the train dataset (split train and val as 650 and 62). But as to all the stores and items, how can I achieve this? If still use 2 year data, does it mean I should create a dataset with each store-item pair has like 650 rows, 12 columns? If there are 1M pair, That will be a huge matrix....Do I think something wrong? thx.
", "rawMarkdown": "Hi Michael, my prediction using LSTM for one store and one item does not work very well. Is it because the data is two small (2 year, 712 observations)?\n\nTo expand to all the item and stores, I understand your idea that each row is a store-item pair. For the columns, when I build an LSTM for one store and one item, I use a look-back of 12 and have 637 rows and 12 columns as the train dataset (split train and val as 650 and 62). But as to all the stores and items, how can I achieve this? If still use 2 year data, does it mean I should create a dataset with each store-item pair has like 650 rows, 12 columns? If there are 1M pair, That will be a huge matrix....Do I think something wrong? thx.", "votes": null }, { "id": "259441", "postDate": "12/18/2017 11:24:46", "content": "My main chunk of Sales data is size (219094,1246), which represents 219,094 actual pairs mentioned in either the test or train csv. The 16 test days + 1230 the number of days present since \"onpromo\" kicked in (Not Including Christmas days since they don't appear in train set).
\n\nFor the simplest sampling method, you can shuffle the rows, select random t_0's in batches, and take in an inputs and pred windows from those t_0.
", "rawMarkdown": "My main chunk of Sales data is size (219094,1246), which represents 219,094 actual pairs mentioned in either the test or train csv. The 16 test days + 1230 the number of days present since \"onpromo\" kicked in (Not Including Christmas days since they don't appear in train set). \n\nFor the simplest sampling method, you can shuffle the rows, select random t_0's in batches, and take in an inputs and pred windows from those t_0.", "votes": null }, { "id": "259570", "postDate": "12/18/2017 16:38:17", "content": "Thanks for explanation. For the columns for one pair, what I did is transferred the data into shifted windows: t-1, t-2, ..., t-12 (12 columns with look-back=12) for X, t for y, so that is why I have 12 columns for one pair. It seems you do it in a different way?
", "rawMarkdown": "Thanks for explanation. For the columns for one pair, what I did is transferred the data into shifted windows: t-1, t-2, ..., t-12 (12 columns with look-back=12) for X, t for y, so that is why I have 12 columns for one pair. It seems you do it in a different way?", "votes": null }, { "id": "259636", "postDate": "12/18/2017 19:03:19", "content": "Yeah so the matrix I described is what I use to build the batches for the model to train on. My input isn't the same shape as that matrix. If I were to use a look back of 12, my input shape would be (batch, 12, 1), with target size (batch, 16, 1)
", "rawMarkdown": "Yeah so the matrix I described is what I use to build the batches for the model to train on. My input isn't the same shape as that matrix. If I were to use a look back of 12, my input shape would be (batch, 12, 1), with target size (batch, 16, 1)", "votes": null }, { "id": "259715", "postDate": "12/18/2017 22:28:42", "content": "thx! so if use all the column (1230), means the look_back is 1230?\nif want to use look_back=12, we need to create another new data with 12 columns?\nsorry for so many questions, I just begin to learn LSTM and get quite confused.
", "rawMarkdown": "thx! so if use all the column (1230), means the look_back is 1230?\nif want to use look_back=12, we need to create another new data with 12 columns?\nsorry for so many questions, I just begin to learn LSTM and get quite confused.", "votes": null }, { "id": "259932", "postDate": "12/19/2017 08:04:29", "content": "No. I think you might be doing something like xdata = data[:-12], ydata = data[12:]. I don't do continuous prediction. I think you take in 12 steps and predict 1, right?
\n\nI split like xdata = data[123:123+24], ydata = data[123+24:123+24+16] | where 24 is the input length, 16 is the test prediction length, and 123 is a random number. So my single sample predicts 16 values and inputs 24.
", "rawMarkdown": "No. I think you might be doing something like xdata = data[:-12], ydata = data[12:]. I don't do continuous prediction. I think you take in 12 steps and predict 1, right? \n\nI split like xdata = data[123:123+24], ydata = data[123+24:123+24+16] | where 24 is the input length, 16 is the test prediction length, and 123 is a random number. So my single sample predicts 16 values and inputs 24.", "votes": null }, { "id": "260185", "postDate": "12/19/2017 20:29:53", "content": "yeah I did model.add(Dense(16)), so I can predict 16 outputs once with all the previous data.
\n\nSince your data columns are the dates, do you mean that you randomly choose a date point (123), then 24 days as input, next 16 days as prediction? 24 is kind like look_back, because you look back 24 days?\nHow can you differentiate certain store-item pairs, do we need an embedding layer before the model.add(LSTM). Thx!
", "rawMarkdown": "yeah I did model.add(Dense(16)), so I can predict 16 outputs once with all the previous data.\n\nSince your data columns are the dates, do you mean that you randomly choose a date point (123), then 24 days as input, next 16 days as prediction? 24 is kind like look_back, because you look back 24 days?\nHow can you differentiate certain store-item pairs, do we need an embedding layer before the model.add(LSTM). Thx!", "votes": null }, { "id": "260192", "postDate": "12/19/2017 20:53:14", "content": "Since the 24 input, predict 16 is only one sample, then we can write a loop to iterate say 100 times with shifting time perieds, so we have 100 samples, and train the model for every store-item pair? so, it means every store-item pair has 100 samples, every sample inputs 24, outputs 16 values. Do I understand it correctly? Will that be very slow with so many different store-item pairs?
\n\nThx!
", "rawMarkdown": "Since the 24 input, predict 16 is only one sample, then we can write a loop to iterate say 100 times with shifting time perieds, so we have 100 samples, and train the model for every store-item pair? so, it means every store-item pair has 100 samples, every sample inputs 24, outputs 16 values. Do I understand it correctly? Will that be very slow with so many different store-item pairs?\n\nThx!", "votes": null }, { "id": "260201", "postDate": "12/19/2017 21:32:40", "content": "Yeah. Be careful about how you sample a random time step. You probably want to avoid x and y being all zero sales.
\n\nAlso yeah, I use store and product embeddings mixed in with the lstm.
\n\nIt's also easier to mix embeddings with sequences with the functional keras api, over the sequential.
", "rawMarkdown": "Yeah. Be careful about how you sample a random time step. You probably want to avoid x and y being all zero sales.\n\nAlso yeah, I use store and product embeddings mixed in with the lstm.\n\nIt's also easier to mix embeddings with sequences with the functional keras api, over the sequential.", "votes": null }, { "id": "260231", "postDate": "12/19/2017 22:12:54", "content": "I do one random time step per sample, so each epoch is around 200k samples. I typically run ~1e2 epochs.
", "rawMarkdown": "I do one random time step per sample, so each epoch is around 200k samples. I typically run ~1e2 epochs.", "votes": null }, { "id": "260457", "postDate": "12/20/2017 08:23:07", "content": "thx Michael. It helps a lot!
\n\nI tried to build the model step by step, now I add two embedding layers for stores and items, like:\nmodel_store.add(Embedding(54, 10, input_length=1))\nmodel_item.add(Embedding(4036, 10, input_length=1)) \n(I randomly choose 10, I am not sure how to choose this.)
\n\nFor the final model. fit(train_X, train_y), how should I construct the dimensions for train_X and train_y? Since it is multiple inputs, so I think it should be a list of Numpy arrays. I tried to search the solution, but cannot find any. Could you give me some hints?
", "rawMarkdown": "thx Michael. It helps a lot! \n\nI tried to build the model step by step, now I add two embedding layers for stores and items, like:\nmodel_store.add(Embedding(54, 10, input_length=1))\nmodel_item.add(Embedding(4036, 10, input_length=1)) \n(I randomly choose 10, I am not sure how to choose this.)\n\nFor the final model. fit(train_X, train_y), how should I construct the dimensions for train_X and train_y? Since it is multiple inputs, so I think it should be a list of Numpy arrays. I tried to search the solution, but cannot find any. Could you give me some hints?", "votes": null }, { "id": "260610", "postDate": "12/20/2017 14:48:37", "content": "Yeah, you probably want to supply a list containing the sequential data in shape (batch_size, time_steps, features), and feeding in each embedding as separate (batch_size,1) arrays.
\n\nIf you want to do more complex models, you'll need to get used to the Functional API. Here's a link on both that and multi-input:\nFunctional API Guide
", "rawMarkdown": "Yeah, you probably want to supply a list containing the sequential data in shape (batch_size, time_steps, features), and feeding in each embedding as separate (batch_size,1) arrays. \n\nIf you want to do more complex models, you'll need to get used to the Functional API. Here's a link on both that and multi-input:\n[Functional API Guide][1]\n\n\n [1]: https://keras.io/getting-started/functional-api-guide/", "votes": null }, { "id": "261728", "postDate": "12/23/2017 17:45:58", "content": "Thanks for your help! I figured out the shape of the embedding and LSTM layers, but I am using the whole time series (for me, it is 576 days) now for each epoch for 20 epochs in total. I think it makes sense to train the model, it's just slower and uses quite a lot of memory, but I am not sure. What do you think about that?
\n\nIn addition, when I do model.predict() for validation data, I input a list of arrays of features (past 576 days sales, past 567 days promotion history, store-item pairs). For the test data prediction, I am thinking to first create the same structure past 567 days data with sales, promotion and store-item pairs(same with test dataset), and supply them as a list of features for model.predict(). Does that seem right?
", "rawMarkdown": "Thanks for your help! I figured out the shape of the embedding and LSTM layers, but I am using the whole time series (for me, it is 576 days) now for each epoch for 20 epochs in total. I think it makes sense to train the model, it's just slower and uses quite a lot of memory, but I am not sure. What do you think about that?\n\nIn addition, when I do model.predict() for validation data, I input a list of arrays of features (past 576 days sales, past 567 days promotion history, store-item pairs). For the test data prediction, I am thinking to first create the same structure past 567 days data with sales, promotion and store-item pairs(same with test dataset), and supply them as a list of features for model.predict(). Does that seem right?", "votes": null }, { "id": "263250", "postDate": "12/29/2017 13:27:23", "content": "Given this approach, how do you handle the fact that there are some store-item combinations on test set that don't appear on train set ? There's no 24 previous days to feed the LSTM in order to predict the next 16 days.
\n\nYour comments have been very valuable for me. Thanks.
", "rawMarkdown": "Given this approach, how do you handle the fact that there are some store-item combinations on test set that don't appear on train set ? There's no 24 previous days to feed the LSTM in order to predict the next 16 days.\n\nYour comments have been very valuable for me. Thanks.", "votes": null }, { "id": "263258", "postDate": "12/29/2017 14:30:17", "content": "Firstly, a simple heuristic like \"zero input = zero output\" is probably going to be fine- they aren't a large percentage of items.
\n\nSecondly, the answer in a pure Time-Series/LSTM approach implies that two different products with identical histories should have identical forecasts. This means a completely zero history will always have the same output. To get around this limitation, I use Product and Store embeddings which can introduce differences between item 9000 at store 1 and at store 45, even if they both have the same zero history.
\n\nTo actually do this training I construct a solid numpy array of size (store-item count, total days), which means I have a zero in every (store-item,date) pair that is missing from Training.csv. For an item introduced in 1-1-2015, there are approximately 700 zeros before the first non zero value. Instead of making the first index be ~700, I'll use ~700 minus the input steps, and maybe minus a little more. Does that make sense?
\n\nI believe that there are less than 10 items which have completely zero history in any stores, so keep that in mind.
", "rawMarkdown": "Firstly, a simple heuristic like \"zero input = zero output\" is probably going to be fine- they aren't a large percentage of items. \n\nSecondly, the answer in a pure Time-Series/LSTM approach implies that two different products with identical histories should have identical forecasts. This means a completely zero history will always have the same output. To get around this limitation, I use Product and Store embeddings which can introduce differences between item 9000 at store 1 and at store 45, even if they both have the same zero history. \n\nTo actually do this training I construct a solid numpy array of size (store-item count, total days), which means I have a zero in every (store-item,date) pair that is missing from Training.csv. For an item introduced in 1-1-2015, there are approximately 700 zeros before the first non zero value. Instead of making the first index be ~700, I'll use ~700 minus the input steps, and maybe minus a little more. Does that make sense? \n\nI believe that there are less than 10 items which have completely zero history in any stores, so keep that in mind.", "votes": null }, { "id": "263350", "postDate": "12/29/2017 20:38:49", "content": "Just to keep you updated: I am using an lgbm model similar to my xgboost model from before. It does a decent job with some data cleansing and feature engineering.
", "rawMarkdown": "Just to keep you updated: I am using an lgbm model similar to my xgboost model from before. It does a decent job with some data cleansing and feature engineering.", "votes": null }, { "id": "264011", "postDate": "01/02/2018 01:39:33", "content": "Since each epoch is around 200k samples ( all the store-item pairs that appears), to construct the validation dataset, will it also be 200k samples with different time periods?
", "rawMarkdown": "Since each epoch is around 200k samples ( all the store-item pairs that appears), to construct the validation dataset, will it also be 200k samples with different time periods?", "votes": null }, { "id": "264024", "postDate": "01/02/2018 02:52:27", "content": "Yep. I use variables alen and blen, and tlen=alen+blen. So test starts at seqlen-tlen, Val at seqlen-blen-tlen
", "rawMarkdown": "Yep. I use variables alen and blen, and tlen=alen+blen. So test starts at seqlen-tlen, Val at seqlen-blen-tlen", "votes": null }, { "id": "264046", "postDate": "01/02/2018 04:29:11", "content": "thx! May I ask what are alen, blen, seqlen?
", "rawMarkdown": "thx! May I ask what are alen, blen, seqlen?", "votes": null }, { "id": "264048", "postDate": "01/02/2018 04:33:10", "content": "Alen = input length, blen=16, tlen = total length, around 1k depending on how you deal with on promo.
", "rawMarkdown": "Alen = input length, blen=16, tlen = total length, around 1k depending on how you deal with on promo.", "votes": null }, { "id": "267299", "postDate": "01/11/2018 01:00:42", "content": "Do you mean for each epoch, you select the same random input and output window for all the samples? or the windows vary for each sample?
", "rawMarkdown": "Do you mean for each epoch, you select the same random input and output window for all the samples? or the windows vary for each sample?", "votes": null } ], "comments": [ { "id": 243469, "author_name": "lynndai", "author_url": "", "post_date": "11/14/2017 05:19:48", "content": "I am not familiar with Prophet, would you like to share some detail?
", "votes": null, "replies": [ { "id": 243516, "author_name": "puremath86", "author_url": "", "post_date": "11/14/2017 08:15:31", "content": "https://research.fb.com/prophet-forecasting-at-scale/
", "votes": null, "replies": [] }, { "id": 243520, "author_name": "lynndai", "author_url": "", "post_date": "11/14/2017 08:18:41", "content": "did u add more new features?
", "votes": null, "replies": [] }, { "id": 243636, "author_name": "puremath86", "author_url": "", "post_date": "11/14/2017 15:01:13", "content": "You get two variables (one feature and the target) with Prophet (it handles basic 1-dim'l time series):
\n\nHaven't tried NN yet, I have decent results with prophet - but still inferior to the median-based approach. My best guess is that several of the sales series are almost random walks => not much structure to extract using prophet.
", "votes": null, "replies": [ { "id": 243635, "author_name": "puremath86", "author_url": "", "post_date": "11/14/2017 14:59:31", "content": "\n\n\nMy best guess is that several of the sales series are almost random walks ...
\n
I noticed this as well. I was thinking to handle series with a RMSLE above a certain threshold (or too many missing values) differently. Some of this data looks contrived to me --but it could simply be incomplete.
", "votes": null, "replies": [] } ] }, { "id": 246900, "author_name": "rawi707", "author_url": "", "post_date": "11/21/2017 23:16:39", "content": "Prophet looks interesting and good to know about it, as I am new to Kaggle, I am trying a few regressors. As prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition, and as have been highlighted by facebook, the date is the major input to be used (I have not further than the front page at https://research.fb.com/prophet-forecasting-at-scale/), am I missing something here?
", "votes": null, "replies": [ { "id": 249044, "author_name": "puremath86", "author_url": "", "post_date": "11/27/2017 15:43:34", "content": "I would use Prophet to create new features (and, in general, extend the dataset). I wouldn't use it directly for the prediction.
\n\n\n\n\nAs prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition...
\n
While that is true... the most important feature in a time-series is what occurred in the recent past. That is the whole idea underlying the structure of sARIMA models (and all other time-series models). So I expect that the time-series itself will be the most important data. Hence, LSTMs, sARIMA models, Prophet, GARCH, etc. should do well here.
", "votes": null, "replies": [] }, { "id": 249046, "author_name": "rawi707", "author_url": "", "post_date": "11/27/2017 15:49:21", "content": "May I ask how fast is your implementation of Prophet? I am trying it now with part of the Favorita data, not sure how would it scale up! My guess is that it will take hours to finish.
", "votes": null, "replies": [] }, { "id": 249080, "author_name": "christophes", "author_url": "", "post_date": "11/27/2017 16:57:34", "content": "I tried using prophet as well. It takes 4 seconds by item x store combination for me with an i7 + 16Go RAM machine.\nSo when you extrapolate that to 200.000 items x store combinations, it takes around 10 days to complete...
\n\nOne way to make Prophet slightly faster is to set the parameter \"uncertainty_samples = 0\" (without this, it takes 6.5 seconds by item x store...)
\n\nI was putting great hope into prophet, the results look quite good when I check some items : the promotions and holidays are well accounted for, the weekly and yearly seasonality as well.\nBut when I submitted the results, I was quite disappointed because it scored lower than using just a simple xgboost model.
", "votes": null, "replies": [] }, { "id": 249100, "author_name": "rawi707", "author_url": "", "post_date": "11/27/2017 17:59:18", "content": "@ChristopheS \n10 days' time is frustrating! \nOne reason for this slowness is due to stan's stdout (Initial log joint probability = -139.766\nIteration 1. Log joint probability = 15.5265. Improved by 155.292.\nIteration 2. Log joint probability = 25.1928. Improved by 9.66629\n....)\nI could not toggle this off!
\n\nMy guess, however, is that the good Prophet results are due to overfitting, hence the unseen/test results scored lower than expected.\nPart of the problem could be due to the negative sales (refunds/returns), which I am currently struggling to model (I think there is no way to correctly predict these -ve's in the test set)
\n\nI wonder if you have considered all the dates in the training of FB Prophet?!!
", "votes": null, "replies": [] }, { "id": 249440, "author_name": "christophes", "author_url": "", "post_date": "11/28/2017 13:23:31", "content": "No, I just used data for the last 2 years and a half.\nI doubt expanding it to 4 years and a half would improve the results, but who knows...
", "votes": null, "replies": [] }, { "id": 249512, "author_name": "konradb", "author_url": "", "post_date": "11/28/2017 16:45:00", "content": "@ChristopheS: have you considered running it parallel? In R, you essentially you can adapt the public time series kernel:
\n\nhttps://www.kaggle.com/dongxu027/time-series-ets-starter-lb-0-590
", "votes": null, "replies": [] }, { "id": 249520, "author_name": "christophes", "author_url": "", "post_date": "11/28/2017 16:56:59", "content": "No, I searched a little bit how I could run it parallel but I didn't find anything to do it. I'm using Python, so unfortunately the post you refer doesn't solve my problem.
\n\nI tried using a free EC2 instance, but the free tier only gives you a machine with 1Go RAM and you need a minimum of 4Go of RAM to install Prophet (and it would be very long anyway).
", "votes": null, "replies": [] }, { "id": 255316, "author_name": "ericdfr", "author_url": "", "post_date": "12/08/2017 20:43:16", "content": "I've forecasted the 200k+ item/store combo on a c5.18.xlarge (72 last gen cpus). :-)\nI tooks 5 hours. Note that there is an iowait limitation and only 50+ cpus are really used.
\nBut it costs around 9$ on a spot instance. :-/
I recently managed to get an lb 0.533 out of an lstm model. What have the other methods scored for you guys?
", "votes": null, "replies": [ { "id": 248544, "author_name": "danirodao", "author_url": "", "post_date": "11/26/2017 12:33:15", "content": "Hi Michael, i am new to machine learning. I finally managed to have LSTM working but for now i have it only per product and store (not generic), so if i want to predict test i don't see any other option than iterate thru stores and products, generate different models and then make the prediction. How did you managed to predict your data? Thanks in advance
", "votes": null, "replies": [] }, { "id": 248554, "author_name": "vannak", "author_url": "", "post_date": "11/26/2017 13:30:42", "content": "You should try to assemble a 2D array of sales. Each row belongs to a unique store-product pair, and each column is a day of the history given. \nIn my model, I iterate over every row and randomly select a region to feed into the LSTM and have it predict the next 16 values after.
\n\nBasically, however you got the single store-item pair's history, do the same for all of them. Train the LSTM on the aggregate model. To get this to work successfully, the model will probably need to know the different stores and products so feed those in as Embeddings.
", "votes": null, "replies": [] }, { "id": 248631, "author_name": "jeveromy", "author_url": "", "post_date": "11/26/2017 16:30:28", "content": "Nice approach! Would you mind clarifying?
\n\nAs I understand it each row represents a store-product pair and each column is a day. You choose a random region and have the LSTM predict the next 16 values after. Fine.
\n\nAfter this you explain \"train the LSTM on the aggregate model\". Aggregate of what? Perhaps you mean you input to the LSTM many of these random regions for just one store-item pair? And then you do this for all pairs?
", "votes": null, "replies": [] }, { "id": 248663, "author_name": "danirodao", "author_url": "", "post_date": "11/26/2017 18:02:51", "content": "thanks for the clarification. BTW, one thing that i notice in the LSTM prediction, if i want to predict n item in advance, all of them seems to have a high correlation to the last \"known\" unit sold , and there is no much variation over the predictions, is this the same case for you?
", "votes": null, "replies": [] }, { "id": 248710, "author_name": "vannak", "author_url": "", "post_date": "11/26/2017 20:11:13", "content": "I meant the model trained on the aggregate data of all store-item pairs. On each eopch I take one window from each store-item pair, though I'm sure a full history or multiple samples might make sense as well.
", "votes": null, "replies": [] }, { "id": 248714, "author_name": "vannak", "author_url": "", "post_date": "11/26/2017 20:36:12", "content": "@Dani that's a pretty common problem with Time Series Prediction. Adding in more relevant data is the best solution.
", "votes": null, "replies": [] }, { "id": 259390, "author_name": "floralqy", "author_url": "", "post_date": "12/18/2017 09:19:36", "content": "Hi Michael, my prediction using LSTM for one store and one item does not work very well. Is it because the data is two small (2 year, 712 observations)?
\n\nTo expand to all the item and stores, I understand your idea that each row is a store-item pair. For the columns, when I build an LSTM for one store and one item, I use a look-back of 12 and have 637 rows and 12 columns as the train dataset (split train and val as 650 and 62). But as to all the stores and items, how can I achieve this? If still use 2 year data, does it mean I should create a dataset with each store-item pair has like 650 rows, 12 columns? If there are 1M pair, That will be a huge matrix....Do I think something wrong? thx.
", "votes": null, "replies": [] }, { "id": 259441, "author_name": "vannak", "author_url": "", "post_date": "12/18/2017 11:24:46", "content": "My main chunk of Sales data is size (219094,1246), which represents 219,094 actual pairs mentioned in either the test or train csv. The 16 test days + 1230 the number of days present since \"onpromo\" kicked in (Not Including Christmas days since they don't appear in train set).
\n\nFor the simplest sampling method, you can shuffle the rows, select random t_0's in batches, and take in an inputs and pred windows from those t_0.
", "votes": null, "replies": [] }, { "id": 259570, "author_name": "floralqy", "author_url": "", "post_date": "12/18/2017 16:38:17", "content": "Thanks for explanation. For the columns for one pair, what I did is transferred the data into shifted windows: t-1, t-2, ..., t-12 (12 columns with look-back=12) for X, t for y, so that is why I have 12 columns for one pair. It seems you do it in a different way?
", "votes": null, "replies": [] }, { "id": 259636, "author_name": "vannak", "author_url": "", "post_date": "12/18/2017 19:03:19", "content": "Yeah so the matrix I described is what I use to build the batches for the model to train on. My input isn't the same shape as that matrix. If I were to use a look back of 12, my input shape would be (batch, 12, 1), with target size (batch, 16, 1)
", "votes": null, "replies": [] }, { "id": 259715, "author_name": "floralqy", "author_url": "", "post_date": "12/18/2017 22:28:42", "content": "thx! so if use all the column (1230), means the look_back is 1230?\nif want to use look_back=12, we need to create another new data with 12 columns?\nsorry for so many questions, I just begin to learn LSTM and get quite confused.
", "votes": null, "replies": [] }, { "id": 259932, "author_name": "vannak", "author_url": "", "post_date": "12/19/2017 08:04:29", "content": "No. I think you might be doing something like xdata = data[:-12], ydata = data[12:]. I don't do continuous prediction. I think you take in 12 steps and predict 1, right?
\n\nI split like xdata = data[123:123+24], ydata = data[123+24:123+24+16] | where 24 is the input length, 16 is the test prediction length, and 123 is a random number. So my single sample predicts 16 values and inputs 24.
", "votes": null, "replies": [] }, { "id": 260185, "author_name": "floralqy", "author_url": "", "post_date": "12/19/2017 20:29:53", "content": "yeah I did model.add(Dense(16)), so I can predict 16 outputs once with all the previous data.
\n\nSince your data columns are the dates, do you mean that you randomly choose a date point (123), then 24 days as input, next 16 days as prediction? 24 is kind like look_back, because you look back 24 days?\nHow can you differentiate certain store-item pairs, do we need an embedding layer before the model.add(LSTM). Thx!
", "votes": null, "replies": [] }, { "id": 260192, "author_name": "floralqy", "author_url": "", "post_date": "12/19/2017 20:53:14", "content": "Since the 24 input, predict 16 is only one sample, then we can write a loop to iterate say 100 times with shifting time perieds, so we have 100 samples, and train the model for every store-item pair? so, it means every store-item pair has 100 samples, every sample inputs 24, outputs 16 values. Do I understand it correctly? Will that be very slow with so many different store-item pairs?
\n\nThx!
", "votes": null, "replies": [] }, { "id": 260201, "author_name": "vannak", "author_url": "", "post_date": "12/19/2017 21:32:40", "content": "Yeah. Be careful about how you sample a random time step. You probably want to avoid x and y being all zero sales.
\n\nAlso yeah, I use store and product embeddings mixed in with the lstm.
\n\nIt's also easier to mix embeddings with sequences with the functional keras api, over the sequential.
", "votes": null, "replies": [] }, { "id": 260231, "author_name": "vannak", "author_url": "", "post_date": "12/19/2017 22:12:54", "content": "I do one random time step per sample, so each epoch is around 200k samples. I typically run ~1e2 epochs.
", "votes": null, "replies": [] }, { "id": 260457, "author_name": "floralqy", "author_url": "", "post_date": "12/20/2017 08:23:07", "content": "thx Michael. It helps a lot!
\n\nI tried to build the model step by step, now I add two embedding layers for stores and items, like:\nmodel_store.add(Embedding(54, 10, input_length=1))\nmodel_item.add(Embedding(4036, 10, input_length=1)) \n(I randomly choose 10, I am not sure how to choose this.)
\n\nFor the final model. fit(train_X, train_y), how should I construct the dimensions for train_X and train_y? Since it is multiple inputs, so I think it should be a list of Numpy arrays. I tried to search the solution, but cannot find any. Could you give me some hints?
", "votes": null, "replies": [] }, { "id": 260610, "author_name": "vannak", "author_url": "", "post_date": "12/20/2017 14:48:37", "content": "Yeah, you probably want to supply a list containing the sequential data in shape (batch_size, time_steps, features), and feeding in each embedding as separate (batch_size,1) arrays.
\n\nIf you want to do more complex models, you'll need to get used to the Functional API. Here's a link on both that and multi-input:\nFunctional API Guide
", "votes": null, "replies": [] }, { "id": 261728, "author_name": "floralqy", "author_url": "", "post_date": "12/23/2017 17:45:58", "content": "Thanks for your help! I figured out the shape of the embedding and LSTM layers, but I am using the whole time series (for me, it is 576 days) now for each epoch for 20 epochs in total. I think it makes sense to train the model, it's just slower and uses quite a lot of memory, but I am not sure. What do you think about that?
\n\nIn addition, when I do model.predict() for validation data, I input a list of arrays of features (past 576 days sales, past 567 days promotion history, store-item pairs). For the test data prediction, I am thinking to first create the same structure past 567 days data with sales, promotion and store-item pairs(same with test dataset), and supply them as a list of features for model.predict(). Does that seem right?
", "votes": null, "replies": [] }, { "id": 263250, "author_name": "martinpella", "author_url": "", "post_date": "12/29/2017 13:27:23", "content": "Given this approach, how do you handle the fact that there are some store-item combinations on test set that don't appear on train set ? There's no 24 previous days to feed the LSTM in order to predict the next 16 days.
\n\nYour comments have been very valuable for me. Thanks.
", "votes": null, "replies": [] }, { "id": 263258, "author_name": "vannak", "author_url": "", "post_date": "12/29/2017 14:30:17", "content": "Firstly, a simple heuristic like \"zero input = zero output\" is probably going to be fine- they aren't a large percentage of items.
\n\nSecondly, the answer in a pure Time-Series/LSTM approach implies that two different products with identical histories should have identical forecasts. This means a completely zero history will always have the same output. To get around this limitation, I use Product and Store embeddings which can introduce differences between item 9000 at store 1 and at store 45, even if they both have the same zero history.
\n\nTo actually do this training I construct a solid numpy array of size (store-item count, total days), which means I have a zero in every (store-item,date) pair that is missing from Training.csv. For an item introduced in 1-1-2015, there are approximately 700 zeros before the first non zero value. Instead of making the first index be ~700, I'll use ~700 minus the input steps, and maybe minus a little more. Does that make sense?
\n\nI believe that there are less than 10 items which have completely zero history in any stores, so keep that in mind.
", "votes": null, "replies": [] }, { "id": 264011, "author_name": "floralqy", "author_url": "", "post_date": "01/02/2018 01:39:33", "content": "Since each epoch is around 200k samples ( all the store-item pairs that appears), to construct the validation dataset, will it also be 200k samples with different time periods?
", "votes": null, "replies": [] }, { "id": 264024, "author_name": "vannak", "author_url": "", "post_date": "01/02/2018 02:52:27", "content": "Yep. I use variables alen and blen, and tlen=alen+blen. So test starts at seqlen-tlen, Val at seqlen-blen-tlen
", "votes": null, "replies": [] }, { "id": 264046, "author_name": "floralqy", "author_url": "", "post_date": "01/02/2018 04:29:11", "content": "thx! May I ask what are alen, blen, seqlen?
", "votes": null, "replies": [] }, { "id": 264048, "author_name": "vannak", "author_url": "", "post_date": "01/02/2018 04:33:10", "content": "Alen = input length, blen=16, tlen = total length, around 1k depending on how you deal with on promo.
", "votes": null, "replies": [] }, { "id": 267299, "author_name": "floralqy", "author_url": "", "post_date": "01/11/2018 01:00:42", "content": "Do you mean for each epoch, you select the same random input and output window for all the samples? or the windows vary for each sample?
", "votes": null, "replies": [] } ] }, { "id": 248650, "author_name": "chinta", "author_url": "", "post_date": "11/26/2017 17:29:18", "content": "Hi, I will try using Prophet thanks for pointing to it. How is the standard xgboost performing, in Rosmann competition it gave good scores to a similar problem for store level sales data. I am not getting good result with xgboost here.
", "votes": null, "replies": [ { "id": 249040, "author_name": "puremath86", "author_url": "", "post_date": "11/27/2017 15:33:27", "content": "I tried one \"sanity check\" implementation of xgboost. It didn't perform well and I haven't come back to fix it yet. I'm getting the best results out of a combination of LSTMs, Prophet, and Linear Models (according to CV scores w/o normalization).
", "votes": null, "replies": [] }, { "id": 249084, "author_name": "chinta", "author_url": "", "post_date": "11/27/2017 17:03:44", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 263350, "author_name": "puremath86", "author_url": "", "post_date": "12/29/2017 20:38:49", "content": "Just to keep you updated: I am using an lgbm model similar to my xgboost model from before. It does a decent job with some data cleansing and feature engineering.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "243461": "I've been testing out a few different strategies:\n\n 1. Using Prophet,\n 2. Using LSTMs,\n 3. Using MLPs.\n\nI haven't had much luck with 2 or 3. Although I haven't tried much in regards to NNs yet. So take that with a grain of salt.\n\nI have, however, had some really good results using FB's Prophet. How is everyone else fairing? Similar results?", "243469": "I am not familiar with Prophet, would you like to share some detail?", "243516": "https://research.fb.com/prophet-forecasting-at-scale/", "243520": "did u add more new features?", "243574": "Haven't tried NN yet, I have decent results with prophet - but still inferior to the median-based approach. My best guess is that several of the sales series are almost random walks => not much structure to extract using prophet.", "243635": "> My best guess is that several of the sales series are almost random walks ...\n\nI noticed this as well. I was thinking to handle series with a RMSLE above a certain threshold (or too many missing values) differently. Some of this data looks contrived to me --but it could simply be incomplete.", "243636": "You get two variables (one feature and the target) with Prophet (it handles basic 1-dim'l time series): \n\n* $ds$ and \n* $y$.", "246900": "Prophet looks interesting and good to know about it, as I am new to Kaggle, I am trying a few regressors. As prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition, and as have been highlighted by facebook, the date is the major input to be used (I have not further than the front page at https://research.fb.com/prophet-forecasting-at-scale/), am I missing something here?", "246911": "I recently managed to get an lb 0.533 out of an lstm model. What have the other methods scored for you guys?", "248544": "Hi Michael, i am new to machine learning. I finally managed to have LSTM working but for now i have it only per product and store (not generic), so if i want to predict test i don't see any other option than iterate thru stores and products, generate different models and then make the prediction. How did you managed to predict your data? Thanks in advance", "248554": "You should try to assemble a 2D array of sales. Each row belongs to a unique store-product pair, and each column is a day of the history given. \nIn my model, I iterate over every row and randomly select a region to feed into the LSTM and have it predict the next 16 values after. \n\nBasically, however you got the single store-item pair's history, do the same for all of them. Train the LSTM on the aggregate model. To get this to work successfully, the model will probably need to know the different stores and products so feed those in as Embeddings.", "248631": "Nice approach! Would you mind clarifying?\n\nAs I understand it each row represents a store-product pair and each column is a day. You choose a random region and have the LSTM predict the next 16 values after. Fine.\n\nAfter this you explain \"train the LSTM on the aggregate model\". Aggregate of what? Perhaps you mean you input to the LSTM many of these random regions for just one store-item pair? And then you do this for all pairs?", "248650": "Hi, I will try using Prophet thanks for pointing to it. How is the standard xgboost performing, in Rosmann competition it gave good scores to a similar problem for store level sales data. I am not getting good result with xgboost here.", "248663": "thanks for the clarification. BTW, one thing that i notice in the LSTM prediction, if i want to predict n item in advance, all of them seems to have a high correlation to the last \"known\" unit sold , and there is no much variation over the predictions, is this the same case for you?", "248710": "I meant the model trained on the aggregate data of all store-item pairs. On each eopch I take one window from each store-item pair, though I'm sure a full history or multiple samples might make sense as well.", "248714": "Dani that's a pretty common problem with Time Series Prediction. Adding in more relevant data is the best solution.", "249040": "I tried one \"sanity check\" implementation of xgboost. It didn't perform well and I haven't come back to fix it yet. I'm getting the best results out of a combination of LSTMs, Prophet, and Linear Models (according to CV scores w/o normalization).", "249044": "I would use Prophet to create new features (and, in general, extend the dataset). I wouldn't use it directly for the prediction.\n\n> As prophet is a one-to-one mapper, it is not clear how someone could make use of all the available inputs by the competition...\n\nWhile that is true... the most important feature in a time-series is what occurred in the recent past. That is the whole idea underlying the structure of sARIMA models (and all other time-series models). So I expect that the time-series itself will be the most important data. Hence, LSTMs, sARIMA models, Prophet, GARCH, etc. should do well here.", "249046": "May I ask how fast is your implementation of Prophet? I am trying it now with part of the Favorita data, not sure how would it scale up! My guess is that it will take hours to finish.", "249080": "I tried using prophet as well. It takes 4 seconds by item x store combination for me with an i7 + 16Go RAM machine.\nSo when you extrapolate that to 200.000 items x store combinations, it takes around 10 days to complete...\n\nOne way to make Prophet slightly faster is to set the parameter \"uncertainty_samples = 0\" (without this, it takes 6.5 seconds by item x store...)\n\nI was putting great hope into prophet, the results look quite good when I check some items : the promotions and holidays are well accounted for, the weekly and yearly seasonality as well.\nBut when I submitted the results, I was quite disappointed because it scored lower than using just a simple xgboost model.", "249084": "Thanks", "249100": "ChristopheS \n10 days' time is frustrating! \nOne reason for this slowness is due to stan's stdout (Initial log joint probability = -139.766\nIteration 1. Log joint probability = 15.5265. Improved by 155.292.\nIteration 2. Log joint probability = 25.1928. Improved by 9.66629\n....)\nI could not toggle this off!\n\nMy guess, however, is that the good Prophet results are due to overfitting, hence the unseen/test results scored lower than expected.\nPart of the problem could be due to the negative sales (refunds/returns), which I am currently struggling to model (I think there is no way to correctly predict these -ve's in the test set)\n\nI wonder if you have considered all the dates in the training of FB Prophet?!!", "249440": "No, I just used data for the last 2 years and a half.\nI doubt expanding it to 4 years and a half would improve the results, but who knows...", "249512": "ChristopheS: have you considered running it parallel? In R, you essentially you can adapt the public time series kernel:\n\nhttps://www.kaggle.com/dongxu027/time-series-ets-starter-lb-0-590", "249520": "No, I searched a little bit how I could run it parallel but I didn't find anything to do it. I'm using Python, so unfortunately the post you refer doesn't solve my problem.\n\nI tried using a free EC2 instance, but the free tier only gives you a machine with 1Go RAM and you need a minimum of 4Go of RAM to install Prophet (and it would be very long anyway).", "255316": "I've forecasted the 200k+ item/store combo on a c5.18.xlarge (72 last gen cpus). :-)\nI tooks 5 hours. Note that there is an iowait limitation and only 50+ cpus are really used. \nBut it costs around 9$ on a spot instance. :-/", "259390": "Hi Michael, my prediction using LSTM for one store and one item does not work very well. Is it because the data is two small (2 year, 712 observations)?\n\nTo expand to all the item and stores, I understand your idea that each row is a store-item pair. For the columns, when I build an LSTM for one store and one item, I use a look-back of 12 and have 637 rows and 12 columns as the train dataset (split train and val as 650 and 62). But as to all the stores and items, how can I achieve this? If still use 2 year data, does it mean I should create a dataset with each store-item pair has like 650 rows, 12 columns? If there are 1M pair, That will be a huge matrix....Do I think something wrong? thx.", "259441": "My main chunk of Sales data is size (219094,1246), which represents 219,094 actual pairs mentioned in either the test or train csv. The 16 test days + 1230 the number of days present since \"onpromo\" kicked in (Not Including Christmas days since they don't appear in train set). \n\nFor the simplest sampling method, you can shuffle the rows, select random t_0's in batches, and take in an inputs and pred windows from those t_0.", "259570": "Thanks for explanation. For the columns for one pair, what I did is transferred the data into shifted windows: t-1, t-2, ..., t-12 (12 columns with look-back=12) for X, t for y, so that is why I have 12 columns for one pair. It seems you do it in a different way?", "259636": "Yeah so the matrix I described is what I use to build the batches for the model to train on. My input isn't the same shape as that matrix. If I were to use a look back of 12, my input shape would be (batch, 12, 1), with target size (batch, 16, 1)", "259715": "thx! so if use all the column (1230), means the look_back is 1230?\nif want to use look_back=12, we need to create another new data with 12 columns?\nsorry for so many questions, I just begin to learn LSTM and get quite confused.", "259932": "No. I think you might be doing something like xdata = data[:-12], ydata = data[12:]. I don't do continuous prediction. I think you take in 12 steps and predict 1, right? \n\nI split like xdata = data[123:123+24], ydata = data[123+24:123+24+16] | where 24 is the input length, 16 is the test prediction length, and 123 is a random number. So my single sample predicts 16 values and inputs 24.", "260185": "yeah I did model.add(Dense(16)), so I can predict 16 outputs once with all the previous data.\n\nSince your data columns are the dates, do you mean that you randomly choose a date point (123), then 24 days as input, next 16 days as prediction? 24 is kind like look_back, because you look back 24 days?\nHow can you differentiate certain store-item pairs, do we need an embedding layer before the model.add(LSTM). Thx!", "260192": "Since the 24 input, predict 16 is only one sample, then we can write a loop to iterate say 100 times with shifting time perieds, so we have 100 samples, and train the model for every store-item pair? so, it means every store-item pair has 100 samples, every sample inputs 24, outputs 16 values. Do I understand it correctly? Will that be very slow with so many different store-item pairs?\n\nThx!", "260201": "Yeah. Be careful about how you sample a random time step. You probably want to avoid x and y being all zero sales.\n\nAlso yeah, I use store and product embeddings mixed in with the lstm.\n\nIt's also easier to mix embeddings with sequences with the functional keras api, over the sequential.", "260231": "I do one random time step per sample, so each epoch is around 200k samples. I typically run ~1e2 epochs.", "260457": "thx Michael. It helps a lot! \n\nI tried to build the model step by step, now I add two embedding layers for stores and items, like:\nmodel_store.add(Embedding(54, 10, input_length=1))\nmodel_item.add(Embedding(4036, 10, input_length=1)) \n(I randomly choose 10, I am not sure how to choose this.)\n\nFor the final model. fit(train_X, train_y), how should I construct the dimensions for train_X and train_y? Since it is multiple inputs, so I think it should be a list of Numpy arrays. I tried to search the solution, but cannot find any. Could you give me some hints?", "260610": "Yeah, you probably want to supply a list containing the sequential data in shape (batch_size, time_steps, features), and feeding in each embedding as separate (batch_size,1) arrays. \n\nIf you want to do more complex models, you'll need to get used to the Functional API. Here's a link on both that and multi-input:\n[Functional API Guide][1]\n\n\n [1]: https://keras.io/getting-started/functional-api-guide/", "261728": "Thanks for your help! I figured out the shape of the embedding and LSTM layers, but I am using the whole time series (for me, it is 576 days) now for each epoch for 20 epochs in total. I think it makes sense to train the model, it's just slower and uses quite a lot of memory, but I am not sure. What do you think about that?\n\nIn addition, when I do model.predict() for validation data, I input a list of arrays of features (past 576 days sales, past 567 days promotion history, store-item pairs). For the test data prediction, I am thinking to first create the same structure past 567 days data with sales, promotion and store-item pairs(same with test dataset), and supply them as a list of features for model.predict(). Does that seem right?", "263250": "Given this approach, how do you handle the fact that there are some store-item combinations on test set that don't appear on train set ? There's no 24 previous days to feed the LSTM in order to predict the next 16 days.\n\nYour comments have been very valuable for me. Thanks.", "263258": "Firstly, a simple heuristic like \"zero input = zero output\" is probably going to be fine- they aren't a large percentage of items. \n\nSecondly, the answer in a pure Time-Series/LSTM approach implies that two different products with identical histories should have identical forecasts. This means a completely zero history will always have the same output. To get around this limitation, I use Product and Store embeddings which can introduce differences between item 9000 at store 1 and at store 45, even if they both have the same zero history. \n\nTo actually do this training I construct a solid numpy array of size (store-item count, total days), which means I have a zero in every (store-item,date) pair that is missing from Training.csv. For an item introduced in 1-1-2015, there are approximately 700 zeros before the first non zero value. Instead of making the first index be ~700, I'll use ~700 minus the input steps, and maybe minus a little more. Does that make sense? \n\nI believe that there are less than 10 items which have completely zero history in any stores, so keep that in mind.", "263350": "Just to keep you updated: I am using an lgbm model similar to my xgboost model from before. It does a decent job with some data cleansing and feature engineering.", "264011": "Since each epoch is around 200k samples ( all the store-item pairs that appears), to construct the validation dataset, will it also be 200k samples with different time periods?", "264024": "Yep. I use variables alen and blen, and tlen=alen+blen. So test starts at seqlen-tlen, Val at seqlen-blen-tlen", "264046": "thx! May I ask what are alen, blen, seqlen?", "264048": "Alen = input length, blen=16, tlen = total length, around 1k depending on how you deal with on promo.", "267299": "Do you mean for each epoch, you select the same random input and output window for all the samples? or the windows vary for each sample?" }, "source": "meta" }