![\"not](\"https://storage.googleapis.com/kaggle-forum-message-attachments/269183/8271/2018-01-16%2020-09-34%E5%B1%8F%E5%B9%95%E6%88%AA%E5%9B%BE.png\")
First of all, I'd like to thank Giba, my team mate, without whom I would not have entered this competition. I also want to thank Kaggle and the host for providing this challenging dataset. Congrats to all gold winning teams and individuals, you did an awesome job here. And there is lots of solution sharing already, which is great. Last but not least, final standing is a great, positive, surprise to us, esp considering we both had the flu.
\n\nOur solution is mostly a single NN model that scores 0.515 private and 0.508 public, with a secondary lgb model. Believe it or not, we missed the deadline by few seconds and could not select our second sub ourselves. It scored 0.514 on private LB, maybe we would have advanced a couple of place, but we are very happy with the result anyway. Details below.
\n\nMissing values
\n\nonpromotion is fully known after 2014-03-31. We ignored the missing values issue by only using data posterior to that. We actually did not use data prior December 1st 2015. We also assumed that any item/store that appears somewhere in train or test data has 0 sales for the dates it does not appear. This seems to be what the data description says. It means that store/item that do not appear in train have zero sales during the train period.
\n\nSeasonality
\n\nIn time series problems there are two key ingredients: seasonality, and cv setting. Here weekly seasonality is extremely strong, yearly seasonality is weak, and there is a monthly seasonality due to pay day. We dealt with weekly seasonality by using time periods that align on weekdays for everything. We did not find a good way to use the monthly seasonality. And we captured the yearly seasonality by using sales figures from 364 days ago. We used 364 and not 365 to align with weekdays.
\n\nCV setting
\n\nWe selected 2 validation periods plus the test period: (2017-07-16, 2017-07-31), (2017-08-01, 2017-08-15), and (2017-08-16, 2017-08-31). For each period we construct a series of training datasets aligned on weekday, for instance for the first validation period:
\n\ndataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12\ndataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19\ndataset up to 2017-03-25 predict period of length 16 starting on 2017-03-26\ndataset up to 2017-04-01 predict period of length 16 starting on 2017-04-02\ndataset up to 2017-04-08 predict period of length 16 starting on 2017-04-09\ndataset up to 2017-04-15 predict period of length 16 starting on 2017-04-16\ndataset up to 2017-04-22 predict period of length 16 starting on 2017-04-23\ndataset up to 2017-04-29 predict period of length 16 starting on 2017-04-30\ndataset up to 2017-05-06 predict period of length 16 starting on 2017-05-07\ndataset up to 2017-05-13 predict period of length 16 starting on 2017-05-14\ndataset up to 2017-05-20 predict period of length 16 starting on 2017-05-21\ndataset up to 2017-05-27 predict period of length 16 starting on 2017-05-28\ndataset up to 2017-06-03 predict period of length 16 starting on 2017-06-04\ndataset up to 2017-06-10 predict period of length 16 starting on 2017-06-11\ndataset up to 2017-06-17 predict period of length 16 starting on 2017-06-18\ndataset up to 2017-06-24 predict period of length 16 starting on 2017-06-25\ndataset up to 2017-07-15 predict period of length 16 starting on 2017-07-16\nWe use all datasets but the last to train, and we use the last one for early stopping. We use 10 fold CV on the training periods. The only caveat is to make sure all store/item pairs for a given time series are in the same fold. I used a very similar CV setting in WTF competition.
\n\nFeature engineering
\n\nAll unit sales are clipped to be non negative, then log1p transformed. For each of the above datasets, we use:
\n\nlags over periods similar to @Ceshine Lee starter LGB kernel. We use mean, max, and proportion of zero entries, as well as proportion of promotion days.
Sales for each of the last 7 days
Average sales per weekday over the last 8 weeks
For each of the test/validation days: sales from 364 days earlier, and promotion status.
Class of the item
Store
We did not use oil price, item number, and other info in our primary model.
\n\nNN Model
\n\nA feedforward model with 3 dense layers, relu activation. Class and store are embedded in 4 dimension vectors, and appended at the second level. Let x be the output of the dense layers. Its length is equal to the length of the test or validation period. Last level is a 1D convolution with x, the on promotion input vector, and the product of the onpromotion vector with x:
\n\ny = Multiply()([x, promo_test_input])\ny = Reshape((-1,1))(y)\nx = Reshape((-1,1))(x)\nz = Reshape((-1,1))(promo_test_input)\nx = Concatenate(axis=-1)([x, y, z])\nx = Conv1D(1,1, activation='linear')(x)\nThis captures the influence of onpromotion day per day quite well.\nThe target for the model is the sequence of train/validation sales, i.e. 15 or 16 days depending on the period.
\n\nOther models
\n\nWe trained a lgb model using the same features and cv setting, the only difference being the target: we train one model per day, like in @Ceshine Lee starter LGB kernel. Best lgb scores 0.510 on pblic LB and 0.517 on private LB. I don't know how others got better results with LGB than with NN.
\n\nGiba created a classifier model to predict when sales are zero or not. Alone this model is useless but it adds when ensembling. I'll let him describe it if need be.
\n\nEnsembling
\n\nTurns out that what would have been our best sub is an unweighted average of 3 different runs of the NN model using a different sets of training periods, and one lgb model. We tried some limited stacking with a NN with one hidden layer. Giba also looked to how to post process submissions. By default we set to 0 all store/item pairs that do not appear in train, but he found that we should not do it if that pair has a promotion in test. He got a sub that scores 0.514 private that way, but, due to few second glitch we did not not select it... And we did not have time to combine his postprocessing with our best blend either.
\n\nConclusion
\n\nMain takeaway from me is once again to ignore public LB feedback. Indeed, that feedback was on the first 5 days of the test period, and it gives zero useful information on what happens for days further in the future. Having some reasonable CV setting was key to estimate how well a model predicts further in the future. I could not say I was expecting to get a result as good as what we got in the end, but I knew we were not overfitting to the first few days of the test period ;)
\n\nLet me end with a special notice to Ceshine Lee without whom many would not have fared as well as they did. He shared a great kernel, and he shared very useful tips in the early days of the competition. There should be a sharing gold medal in Kaggle competitions, and he would have won that one here!
\n\nEdit: \nThe discussion below makes me add this. We added 0 sales row with onpromotion=False for all store/item pairs that appear in our training datasets. We did not add all those store/items pairs that appear in the test set. We did compare with and without these: the results were similar, but training time were higher with the test items. We validated adding rows with onpromotion=False by using the original data in the validation periods.
", "messages": [ { "id": "269174", "postDate": "01/16/2018 11:42:09", "content": "First of all, I'd like to thank Giba, my team mate, without whom I would not have entered this competition. I also want to thank Kaggle and the host for providing this challenging dataset. Congrats to all gold winning teams and individuals, you did an awesome job here. And there is lots of solution sharing already, which is great. Last but not least, final standing is a great, positive, surprise to us, esp considering we both had the flu.
\n\nOur solution is mostly a single NN model that scores 0.515 private and 0.508 public, with a secondary lgb model. Believe it or not, we missed the deadline by few seconds and could not select our second sub ourselves. It scored 0.514 on private LB, maybe we would have advanced a couple of place, but we are very happy with the result anyway. Details below.
\n\nMissing values
\n\nonpromotion is fully known after 2014-03-31. We ignored the missing values issue by only using data posterior to that. We actually did not use data prior December 1st 2015. We also assumed that any item/store that appears somewhere in train or test data has 0 sales for the dates it does not appear. This seems to be what the data description says. It means that store/item that do not appear in train have zero sales during the train period.
\n\nSeasonality
\n\nIn time series problems there are two key ingredients: seasonality, and cv setting. Here weekly seasonality is extremely strong, yearly seasonality is weak, and there is a monthly seasonality due to pay day. We dealt with weekly seasonality by using time periods that align on weekdays for everything. We did not find a good way to use the monthly seasonality. And we captured the yearly seasonality by using sales figures from 364 days ago. We used 364 and not 365 to align with weekdays.
\n\nCV setting
\n\nWe selected 2 validation periods plus the test period: (2017-07-16, 2017-07-31), (2017-08-01, 2017-08-15), and (2017-08-16, 2017-08-31). For each period we construct a series of training datasets aligned on weekday, for instance for the first validation period:
\n\ndataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12\ndataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19\ndataset up to 2017-03-25 predict period of length 16 starting on 2017-03-26\ndataset up to 2017-04-01 predict period of length 16 starting on 2017-04-02\ndataset up to 2017-04-08 predict period of length 16 starting on 2017-04-09\ndataset up to 2017-04-15 predict period of length 16 starting on 2017-04-16\ndataset up to 2017-04-22 predict period of length 16 starting on 2017-04-23\ndataset up to 2017-04-29 predict period of length 16 starting on 2017-04-30\ndataset up to 2017-05-06 predict period of length 16 starting on 2017-05-07\ndataset up to 2017-05-13 predict period of length 16 starting on 2017-05-14\ndataset up to 2017-05-20 predict period of length 16 starting on 2017-05-21\ndataset up to 2017-05-27 predict period of length 16 starting on 2017-05-28\ndataset up to 2017-06-03 predict period of length 16 starting on 2017-06-04\ndataset up to 2017-06-10 predict period of length 16 starting on 2017-06-11\ndataset up to 2017-06-17 predict period of length 16 starting on 2017-06-18\ndataset up to 2017-06-24 predict period of length 16 starting on 2017-06-25\ndataset up to 2017-07-15 predict period of length 16 starting on 2017-07-16\nWe use all datasets but the last to train, and we use the last one for early stopping. We use 10 fold CV on the training periods. The only caveat is to make sure all store/item pairs for a given time series are in the same fold. I used a very similar CV setting in WTF competition.
\n\nFeature engineering
\n\nAll unit sales are clipped to be non negative, then log1p transformed. For each of the above datasets, we use:
\n\nlags over periods similar to @Ceshine Lee starter LGB kernel. We use mean, max, and proportion of zero entries, as well as proportion of promotion days.
Sales for each of the last 7 days
Average sales per weekday over the last 8 weeks
For each of the test/validation days: sales from 364 days earlier, and promotion status.
Class of the item
Store
We did not use oil price, item number, and other info in our primary model.
\n\nNN Model
\n\nA feedforward model with 3 dense layers, relu activation. Class and store are embedded in 4 dimension vectors, and appended at the second level. Let x be the output of the dense layers. Its length is equal to the length of the test or validation period. Last level is a 1D convolution with x, the on promotion input vector, and the product of the onpromotion vector with x:
\n\ny = Multiply()([x, promo_test_input])\ny = Reshape((-1,1))(y)\nx = Reshape((-1,1))(x)\nz = Reshape((-1,1))(promo_test_input)\nx = Concatenate(axis=-1)([x, y, z])\nx = Conv1D(1,1, activation='linear')(x)\nThis captures the influence of onpromotion day per day quite well.\nThe target for the model is the sequence of train/validation sales, i.e. 15 or 16 days depending on the period.
\n\nOther models
\n\nWe trained a lgb model using the same features and cv setting, the only difference being the target: we train one model per day, like in @Ceshine Lee starter LGB kernel. Best lgb scores 0.510 on pblic LB and 0.517 on private LB. I don't know how others got better results with LGB than with NN.
\n\nGiba created a classifier model to predict when sales are zero or not. Alone this model is useless but it adds when ensembling. I'll let him describe it if need be.
\n\nEnsembling
\n\nTurns out that what would have been our best sub is an unweighted average of 3 different runs of the NN model using a different sets of training periods, and one lgb model. We tried some limited stacking with a NN with one hidden layer. Giba also looked to how to post process submissions. By default we set to 0 all store/item pairs that do not appear in train, but he found that we should not do it if that pair has a promotion in test. He got a sub that scores 0.514 private that way, but, due to few second glitch we did not not select it... And we did not have time to combine his postprocessing with our best blend either.
\n\nConclusion
\n\nMain takeaway from me is once again to ignore public LB feedback. Indeed, that feedback was on the first 5 days of the test period, and it gives zero useful information on what happens for days further in the future. Having some reasonable CV setting was key to estimate how well a model predicts further in the future. I could not say I was expecting to get a result as good as what we got in the end, but I knew we were not overfitting to the first few days of the test period ;)
\n\nLet me end with a special notice to Ceshine Lee without whom many would not have fared as well as they did. He shared a great kernel, and he shared very useful tips in the early days of the competition. There should be a sharing gold medal in Kaggle competitions, and he would have won that one here!
\n\nEdit: \nThe discussion below makes me add this. We added 0 sales row with onpromotion=False for all store/item pairs that appear in our training datasets. We did not add all those store/items pairs that appear in the test set. We did compare with and without these: the results were similar, but training time were higher with the test items. We validated adding rows with onpromotion=False by using the original data in the validation periods.
", "rawMarkdown": "First of all, I'd like to thank Giba, my team mate, without whom I would not have entered this competition. I also want to thank Kaggle and the host for providing this challenging dataset. Congrats to all gold winning teams and individuals, you did an awesome job here. And there is lots of solution sharing already, which is great. Last but not least, final standing is a great, positive, surprise to us, esp considering we both had the flu.\n\nOur solution is mostly a single NN model that scores 0.515 private and 0.508 public, with a secondary lgb model. Believe it or not, we missed the deadline by few seconds and could not select our second sub ourselves. It scored 0.514 on private LB, maybe we would have advanced a couple of place, but we are very happy with the result anyway. Details below.\n\n**Missing values**\n\nonpromotion is fully known after 2014-03-31. We ignored the missing values issue by only using data posterior to that. We actually did not use data prior December 1st 2015. We also assumed that any item/store that appears somewhere in train or test data has 0 sales for the dates it does not appear. This seems to be what the data description says. It means that store/item that do not appear in train have zero sales during the train period.\n\n**Seasonality**\n\nIn time series problems there are two key ingredients: seasonality, and cv setting. Here weekly seasonality is extremely strong, yearly seasonality is weak, and there is a monthly seasonality due to pay day. We dealt with weekly seasonality by using time periods that align on weekdays for everything. We did not find a good way to use the monthly seasonality. And we captured the yearly seasonality by using sales figures from 364 days ago. We used 364 and not 365 to align with weekdays.\n\n**CV setting**\n\nWe selected 2 validation periods plus the test period: (2017-07-16, 2017-07-31), (2017-08-01, 2017-08-15), and (2017-08-16, 2017-08-31). For each period we construct a series of training datasets aligned on weekday, for instance for the first validation period:\n\n dataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12\n dataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19\n dataset up to 2017-03-25 predict period of length 16 starting on 2017-03-26\n dataset up to 2017-04-01 predict period of length 16 starting on 2017-04-02\n dataset up to 2017-04-08 predict period of length 16 starting on 2017-04-09\n dataset up to 2017-04-15 predict period of length 16 starting on 2017-04-16\n dataset up to 2017-04-22 predict period of length 16 starting on 2017-04-23\n dataset up to 2017-04-29 predict period of length 16 starting on 2017-04-30\n dataset up to 2017-05-06 predict period of length 16 starting on 2017-05-07\n dataset up to 2017-05-13 predict period of length 16 starting on 2017-05-14\n dataset up to 2017-05-20 predict period of length 16 starting on 2017-05-21\n dataset up to 2017-05-27 predict period of length 16 starting on 2017-05-28\n dataset up to 2017-06-03 predict period of length 16 starting on 2017-06-04\n dataset up to 2017-06-10 predict period of length 16 starting on 2017-06-11\n dataset up to 2017-06-17 predict period of length 16 starting on 2017-06-18\n dataset up to 2017-06-24 predict period of length 16 starting on 2017-06-25\n dataset up to 2017-07-15 predict period of length 16 starting on 2017-07-16\n\nWe use all datasets but the last to train, and we use the last one for early stopping. We use 10 fold CV on the training periods. The only caveat is to make sure all store/item pairs for a given time series are in the same fold. I used a very similar CV setting in WTF competition.\n\n**Feature engineering**\n\nAll unit sales are clipped to be non negative, then log1p transformed. For each of the above datasets, we use:\n\n - lags over periods similar to @Ceshine Lee starter LGB kernel. We use mean, max, and proportion of zero entries, as well as proportion of promotion days.\n\n - Sales for each of the last 7 days\n\n - Average sales per weekday over the last 8 weeks\n\n - For each of the test/validation days: sales from 364 days earlier, and promotion status.\n\n - Class of the item\n\n - Store\n\nWe did not use oil price, item number, and other info in our primary model.\n\n**NN Model**\n\nA feedforward model with 3 dense layers, relu activation. Class and store are embedded in 4 dimension vectors, and appended at the second level. Let x be the output of the dense layers. Its length is equal to the length of the test or validation period. Last level is a 1D convolution with x, the on promotion input vector, and the product of the onpromotion vector with x:\n\n y = Multiply()([x, promo_test_input])\n y = Reshape((-1,1))(y)\n x = Reshape((-1,1))(x)\n z = Reshape((-1,1))(promo_test_input)\n x = Concatenate(axis=-1)([x, y, z])\n x = Conv1D(1,1, activation='linear')(x)\n\nThis captures the influence of onpromotion day per day quite well.\nThe target for the model is the sequence of train/validation sales, i.e. 15 or 16 days depending on the period.\n\n**Other models**\n\nWe trained a lgb model using the same features and cv setting, the only difference being the target: we train one model per day, like in @Ceshine Lee starter LGB kernel. Best lgb scores 0.510 on pblic LB and 0.517 on private LB. I don't know how others got better results with LGB than with NN.\n\nGiba created a classifier model to predict when sales are zero or not. Alone this model is useless but it adds when ensembling. I'll let him describe it if need be.\n\n**Ensembling**\n\nTurns out that what would have been our best sub is an unweighted average of 3 different runs of the NN model using a different sets of training periods, and one lgb model. We tried some limited stacking with a NN with one hidden layer. Giba also looked to how to post process submissions. By default we set to 0 all store/item pairs that do not appear in train, but he found that we should not do it if that pair has a promotion in test. He got a sub that scores 0.514 private that way, but, due to few second glitch we did not not select it... And we did not have time to combine his postprocessing with our best blend either.\n\n**Conclusion**\n\nMain takeaway from me is once again to ignore public LB feedback. Indeed, that feedback was on the first 5 days of the test period, and it gives zero useful information on what happens for days further in the future. Having some reasonable CV setting was key to estimate how well a model predicts further in the future. I could not say I was expecting to get a result as good as what we got in the end, but I knew we were not overfitting to the first few days of the test period ;) \n\nLet me end with a special notice to Ceshine Lee without whom many would not have fared as well as they did. He shared a great kernel, and he shared very useful tips in the early days of the competition. There should be a sharing gold medal in Kaggle competitions, and he would have won that one here!\n\n**Edit:** \nThe discussion below makes me add this. We added 0 sales row with onpromotion=False for all store/item pairs that appear in our training datasets. We did not add all those store/items pairs that appear in the test set. We did compare with and without these: the results were similar, but training time were higher with the test items. We validated adding rows with onpromotion=False by using the original data in the validation periods.", "votes": null }, { "id": "269183", "postDate": "01/16/2018 12:11:20", "content": "onpromotion is not fully known after 2014-03-31 !
\n\njust look this figure , it obviously the item on promotion from 2017.8.1~2017.8.15
\n\nbut because someday there is no sales for this item，we lost some information about promotion.
\n\nIf you use these bias data to train, you will get a result that all items onpromotion have a sale>1.
\n\nthat is not true in test data.
\n\n
Not sure why you say that. Running this code:
\n\nimport pandas as pd\n\ntrain = pd.read_csv('../input/train.csv', usecols=['date', 'onpromotion'])\n\ntrain.onpromotion.fillna(-1, inplace=True)\n\ntrain.groupby('onpromotion').date.max()\nyields
\n\nonpromotion\n-1 2014-03-31\nFalse 2017-08-15\nTrue 2017-08-15\nName: date, dtype: object\nWhat am I missing?
", "rawMarkdown": "Not sure why you say that. Running this code:\n\n import pandas as pd\n \n train = pd.read_csv('../input/train.csv', usecols=['date', 'onpromotion'])\n \n train.onpromotion.fillna(-1, inplace=True)\n \n train.groupby('onpromotion').date.max()\n\nyields\n\n onpromotion\n -1 2014-03-31\n False 2017-08-15\n True 2017-08-15\n Name: date, dtype: object\n\nWhat am I missing?", "votes": null }, { "id": "269186", "postDate": "01/16/2018 12:23:01", "content": "Records with zero unit sales
", "rawMarkdown": "Records with zero unit sales", "votes": null }, { "id": "269188", "postDate": "01/16/2018 12:27:47", "content": "We don't know the onpromotion information in the records with zero unit sales.
\n\nIf we fillna with zero unit sales records to False.
\n\nOur model will get a result that if a item on promotion , its sales will >1
", "rawMarkdown": "We don't know the onpromotion information in the records with zero unit sales.\n\nIf we fillna with zero unit sales records to False.\n\nOur model will get a result that if a item on promotion , its sales will >1", "votes": null }, { "id": "269189", "postDate": "01/16/2018 12:29:06", "content": "> it obviously the item on promotion from 2017.8.1~2017.8.15
\n\n> that is not true in test data.
\n\nHow do you know both?
\n\nAnyway, we did not use that assumption. The way we use onpromotion (in the NN) is to modify on average the sales level. I guess that way was good enough to yield reasonable predictions.
\n\nI did play with how to fillna for onpromotion when adding 0 sales rows. Best CV result was obtained with fillna(0). Sure, it is a bias, but it is the best bias I found.
\n\nHow did you validate your way of filling missing onpromotion values?
", "rawMarkdown": "> it obviously the item on promotion from 2017.8.1~2017.8.15\n\n> that is not true in test data.\n\nHow do you know both? \n\nAnyway, we did not use that assumption. The way we use onpromotion (in the NN) is to modify on average the sales level. I guess that way was good enough to yield reasonable predictions.\n\nI did play with how to fillna for onpromotion when adding 0 sales rows. Best CV result was obtained with fillna(0). Sure, it is a bias, but it is the best bias I found. \n\nHow did you validate your way of filling missing onpromotion values?", "votes": null }, { "id": "269190", "postDate": "01/16/2018 12:32:17", "content": "So how do you deal with this item promotion information in 2017.8.12
", "rawMarkdown": "So how do you deal with this item promotion information in 2017.8.12", "votes": null }, { "id": "269193", "postDate": "01/16/2018 12:37:38", "content": "Just check your model's result(not merged with test data) ,all items on promotion's sales >1
\n\nI had probed leaderboard , that's not true in test data
\n\nA lot items'sales on promotion is 0.
", "rawMarkdown": "Just check your model's result(not merged with test data) ,all items on promotion's sales >1\n\nI had probed leaderboard , that's not true in test data\n\nA lot items'sales on promotion is 0.", "votes": null }, { "id": "269194", "postDate": "01/16/2018 12:43:32", "content": "As I said, when adding zero sales rows we put onpromotion to False. We tried other ways, and they led to worse CV values. How did you validate your way of filling NA? You say medals are not due to good use of CV, but I'm afraid this precisely shows the opposite: we used CV to find out how to best deal with that issue, and it worked reasonably well.
\n\nWe also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).
", "rawMarkdown": "As I said, when adding zero sales rows we put onpromotion to False. We tried other ways, and they led to worse CV values. How did you validate your way of filling NA? You say medals are not due to good use of CV, but I'm afraid this precisely shows the opposite: we used CV to find out how to best deal with that issue, and it worked reasonably well.\n\nWe also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).", "votes": null }, { "id": "269195", "postDate": "01/16/2018 12:44:08", "content": "If you fillna(0), that caused bias both train and valiation data,
\n\nthe items'sales on promotion>1
\n\nso you will get a good cv.
\n\nbut not in test data
", "rawMarkdown": "If you fillna(0), that caused bias both train and valiation data,\n\nthe items'sales on promotion>1\n\nso you will get a good cv.\n\nbut not in test data", "votes": null }, { "id": "269196", "postDate": "01/16/2018 12:46:04", "content": "We are circling: \"How did you validate your way of filling NA? \"
\n\nre test data:
\n\n\n", "rawMarkdown": "We are circling: \"How did you validate your way of filling NA? \"\n\nre test data: \n> We also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).", "votes": null }, { "id": "269197", "postDate": "01/16/2018 12:50:29", "content": "We also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).
\n
I also probed LB, i found make some rules to deal with bias ( \"last 21 days no sales → onpromotopn to False\") will get 0.003 booster. The bias can't be handled by local cv. So I used leadboard to validate.\nUnfortunely , the bias exist between 5days and 11days too.
\n\nThat's the reason why a lot team drop a lot
\n\nyou can see sjv‘s post . he used leadboard to validate too.
", "rawMarkdown": "I also probed LB, i found make some rules to deal with bias ( \"last 21 days no sales → onpromotopn to False\") will get 0.003 booster. The bias can't be handled by local cv. So I used leadboard to validate.\nUnfortunely , the bias exist between 5days and 11days too.\n\nThat's the reason why a lot team drop a lot\n\nyou can see sjv‘s post . he used leadboard to validate too.", "votes": null }, { "id": "269204", "postDate": "01/16/2018 13:00:19", "content": "\n\n\nyou can see sjv‘s post . he used leadboard to validate too.
\n
And he drops in performance too.
\n\nBack to this:
\n\n\n\n\nIf you fillna(0), that caused bias both train and valiation data, the items'sales on promotion>1 so you will get a good cv.
\n
Now that you insist, I remember I ran lgb at point with the original data as evaluation, i.e. without any additional 0 sales rows. I wanted to make sure that adding these rows was fine. That's how I validated that filling missing onpromotion with 0 was the best I could get. And our limited LB probing confirmed later than it was a reasonable bet.
", "rawMarkdown": "> you can see sjv‘s post . he used leadboard to validate too.\n\nAnd he drops in performance too.\n\nBack to this:\n\n> If you fillna(0), that caused bias both train and valiation data, the items'sales on promotion>1 so you will get a good cv.\n\nNow that you insist, I remember I ran lgb at point with the original data as evaluation, i.e. without any additional 0 sales rows. I wanted to make sure that adding these rows was fine. That's how I validated that filling missing onpromotion with 0 was the best I could get. And our limited LB probing confirmed later than it was a reasonable bet.", "votes": null }, { "id": "269243", "postDate": "01/16/2018 14:16:50", "content": "Congrats CPMP and Giba for gold medal again. I thought that you would be using your WTF model here, your solution there was great and here insight into data was great. After Caesar I do not have time to participate fully in this competition but giba and you have achieved quite a lot in short duration..
", "rawMarkdown": "Congrats CPMP and Giba for gold medal again. I thought that you would be using your WTF model here, your solution there was great and here insight into data was great. After Caesar I do not have time to participate fully in this competition but giba and you have achieved quite a lot in short duration..", "votes": null }, { "id": "269248", "postDate": "01/16/2018 14:24:23", "content": "Thanks! I did try to reuse my WTF model but made a mistake in it, and lost a week trying to get it running correctly. Then we restarted from scratch, but ended up with a model similar to that of WTF, except for the last convolution layer.
\n\nDon't mention Caesars to me ;) I see people complaining about data preparation here, but there it was an order of magnitude worse, at least... Not to mention the reset when I was fighting for top spot... Very bad memories to me. Almost made me quit Kaggle.
\n\nI feel better now of course with a gold medal :)
", "rawMarkdown": "Thanks! I did try to reuse my WTF model but made a mistake in it, and lost a week trying to get it running correctly. Then we restarted from scratch, but ended up with a model similar to that of WTF, except for the last convolution layer. \n\nDon't mention Caesars to me ;) I see people complaining about data preparation here, but there it was an order of magnitude worse, at least... Not to mention the reset when I was fighting for top spot... Very bad memories to me. Almost made me quit Kaggle. \n\nI feel better now of course with a gold medal :)", "votes": null }, { "id": "269293", "postDate": "01/16/2018 15:48:18", "content": "@CPMP Based on your and Giba's comments, your team hardly began to understand the issue of bias between the train and test sets. Crediting your good performance to your CV setup demonstrates further ignorance of the issue, and it's honestly a bit insulting.
\n\n\n\n\nAnd he drops in performance too.
\n
If you use the fact that my position dropped and yours raised as a retroactive justification for your CV setup, then you're a victim of a very basic statistical fallacy. I think you may learn something from this:\nhttps://www.youtube.com/watch?v=q-JgG0ECp2U
", "rawMarkdown": "CPMP Based on your and Giba's comments, your team hardly began to understand the issue of bias between the train and test sets. Crediting your good performance to your CV setup demonstrates further ignorance of the issue, and it's honestly a bit insulting. \n\n> And he drops in performance too.\n\nIf you use the fact that my position dropped and yours raised as a retroactive justification for your CV setup, then you're a victim of a very basic statistical fallacy. I think you may learn something from this:\nhttps://www.youtube.com/watch?v=q-JgG0ECp2U", "votes": null }, { "id": "269329", "postDate": "01/16/2018 16:33:17", "content": "@sjv, Sure, you can dismiss what we did as ignorance, hope it makes you feel better. Giba and I did discuss what to do when adding 0 sales rows, and we quickly settled on no promotion as being simple, reliable, and not dangerous. Seems we were right. Sorry if you find this insulting.
\n\nI still value what you did, and I'm one that pointed many participants to your WTF code.
\n\nEdit. My CV setting is the same as in WTF. The fact that it worked twice reasonably well may be a statistical fallacy, but I doubt it. I don't like videos, do you have the same in writing?
", "rawMarkdown": "sjv, Sure, you can dismiss what we did as ignorance, hope it makes you feel better. Giba and I did discuss what to do when adding 0 sales rows, and we quickly settled on no promotion as being simple, reliable, and not dangerous. Seems we were right. Sorry if you find this insulting.\n\n I still value what you did, and I'm one that pointed many participants to your WTF code.\n\nEdit. My CV setting is the same as in WTF. The fact that it worked twice reasonably well may be a statistical fallacy, but I doubt it. I don't like videos, do you have the same in writing?", "votes": null }, { "id": "269331", "postDate": "01/16/2018 16:34:23", "content": "@sjv, it would be great to know more about how your teammates calculated 'p'. Did this method of imputing help on the private LB also?
", "rawMarkdown": "sjv, it would be great to know more about how your teammates calculated 'p'. Did this method of imputing help on the private LB also?", "votes": null }, { "id": "269350", "postDate": "01/16/2018 17:07:31", "content": "@sjv, I have watched that video. Is that the best you can do when confronted with reality? Not only am I ignorant, but I got 2 gold medals by sheer luck using a flawed CV setting? And worse, in WTF, my flawed CV setting resulted in a better submission than yours? BTW, you did not tell us what your model performance was in private LB here. Don't tell me it isn't better than mine. That would hurt.
\n\nMore seriously, let me forget about this conversation, and let me remember instead your great instacart and your great wtf code. As I wrote elsewhere, i hope you'll cool down and come compete with us again.
", "rawMarkdown": "sjv, I have watched that video. Is that the best you can do when confronted with reality? Not only am I ignorant, but I got 2 gold medals by sheer luck using a flawed CV setting? And worse, in WTF, my flawed CV setting resulted in a better submission than yours? BTW, you did not tell us what your model performance was in private LB here. Don't tell me it isn't better than mine. That would hurt.\n\nMore seriously, let me forget about this conversation, and let me remember instead your great instacart and your great wtf code. As I wrote elsewhere, i hope you'll cool down and come compete with us again.", "votes": null }, { "id": "269358", "postDate": "01/16/2018 17:24:52", "content": "@sjv Sorry for your frustration about dropping some places in Private LB, but it happens. It happened dozen of times with me in the past and sometimes are due our fault and other times are just bad luck. Our \"ignorance\" in understand the problem placed us where we are and not the drop of onpromotion information on zero sales dates in train. But if you had took a look at some properties of Public/Private split you won't had completely trusted your solution in Public LB. I always say and continue saying, don't trust public LB. Specially in a time series challenge where public LB is calculated using 5 days ahead and private is from 6 to 16 days ahead, they are completely different models. \nBut maybe you're right, our CV set and feature engineering strategy is a fallacy that \"luckily\" worked well on private LB. \nBest of Luck in your research from now, take some time to rest and I hope you can come back to Kaggle in the future.
", "rawMarkdown": "sjv Sorry for your frustration about dropping some places in Private LB, but it happens. It happened dozen of times with me in the past and sometimes are due our fault and other times are just bad luck. Our \"ignorance\" in understand the problem placed us where we are and not the drop of onpromotion information on zero sales dates in train. But if you had took a look at some properties of Public/Private split you won't had completely trusted your solution in Public LB. I always say and continue saying, don't trust public LB. Specially in a time series challenge where public LB is calculated using 5 days ahead and private is from 6 to 16 days ahead, they are completely different models. \nBut maybe you're right, our CV set and feature engineering strategy is a fallacy that \"luckily\" worked well on private LB. \nBest of Luck in your research from now, take some time to rest and I hope you can come back to Kaggle in the future.", "votes": null }, { "id": "270571", "postDate": "01/18/2018 14:37:04", "content": "Congratulations @Giba and @CPMP for great result, and thank you for sharing .
\n\nProper cross validation in time series is quite a discussed subject so to know different approaches is of high interest imo. And your approach looks a very promising one :-)
\n\nThat said, even with your detailed explanation, not sure I got it right about your CV setup; My initial understanding:
\n\nBut then, you mention k-fold CV (10 fold) cross validation. Not sure how does k-fold fit in here so I am just getting it wrong about your setup, maybe you could kindly give me some aditional hint on how that piece fits in the puzzle.
\n\n(btw, am I the only one intrigued by \"random\" k-fold working so well on time series? It is not the first time I hear of it, as long as some precautions are taken with adjacent folds but, well... I think its something to notice)
", "rawMarkdown": "**Congratulations @Giba and @CPMP for great result, and thank you for sharing** . \n\nProper cross validation in time series is quite a discussed subject so to know different approaches is of high interest imo. And your approach looks a very promising one :-)\n\nThat said, even with your detailed explanation, not sure I got it right about your CV setup; My initial understanding:\n\n- you feed a NN with (all) sequence input and sequence outputs built from 16 bi-weekly chunks, using 17th to get n_epochs, and then retrain all for prediction of test, and you do this twice (last two be-weekly periods) \n\nBut then, you mention k-fold CV (10 fold) cross validation. Not sure **how does k-fold fit in here** so I am just getting it wrong about your setup, maybe you could kindly give me some aditional hint on how that piece fits in the puzzle. \n\n(btw, am I the only one intrigued by \"random\" k-fold working so well on time series? It is not the first time I hear of it, as long as some precautions are taken with adjacent folds but, well... I think its something to notice)", "votes": null }, { "id": "270579", "postDate": "01/18/2018 14:53:41", "content": "Thanks. Let me try to explain again, the write up may be confusing. For one validation period, say period of length 16 starting on 2017-07-16, we create training datasets as described above. We concatenate these into one dataset, and use kfold on it. We then train one model per fold, as usual in kfold cv, and keep the epoch with the best average cv value on the folds to compute the prediction on the validation data using the average of the fold models predictions. We then use that prediction score as our measure of the trained model. When we are done with model tuning, we use the exact same kfold on the training dataset for the test period, use the best epoch on the folds to compute the final test prediction. With our NN model, this yields a 0.003 improvement over Ceshine Lee's approach. I reported this on the forum, moving from 0.51 to 0.508 on public LB. I'm not sure why, but probably because we average the predictions of several models.
", "rawMarkdown": "Thanks. Let me try to explain again, the write up may be confusing. For one validation period, say period of length 16 starting on 2017-07-16, we create training datasets as described above. We concatenate these into one dataset, and use kfold on it. We then train one model per fold, as usual in kfold cv, and keep the epoch with the best average cv value on the folds to compute the prediction on the validation data using the average of the fold models predictions. We then use that prediction score as our measure of the trained model. When we are done with model tuning, we use the exact same kfold on the training dataset for the test period, use the best epoch on the folds to compute the final test prediction. With our NN model, this yields a 0.003 improvement over Ceshine Lee's approach. I reported this on the forum, moving from 0.51 to 0.508 on public LB. I'm not sure why, but probably because we average the predictions of several models.", "votes": null }, { "id": "270596", "postDate": "01/18/2018 15:07:41", "content": "So kfold for parameter setting and then validation periods used for model performance estimation. Thaks a lot @CPMP, its clear now :-)
", "rawMarkdown": "So kfold for parameter setting and then validation periods used for model performance estimation. Thaks a lot @CPMP, its clear now :-)", "votes": null }, { "id": "271660", "postDate": "01/21/2018 03:18:40", "content": "Thanks for sharing!\nI have a little question here.I remember one of the stores just start operation at 2017.4.20 while your team (actually a lot of teams)add the samples before 4.20 without special treatment.Could you tell me why it doesn't need special treatment?
", "rawMarkdown": "Thanks for sharing!\nI have a little question here.I remember one of the stores just start operation at 2017.4.20 while your team (actually a lot of teams)add the samples before 4.20 without special treatment.Could you tell me why it doesn't need special treatment?", "votes": null }, { "id": "271765", "postDate": "01/21/2018 11:15:42", "content": "The model must have learned that for this store (store_nbr is one of our features) the early data isn't useful. The good question would be: what special treatment should we have used? Maybe dropping older data from that store?
", "rawMarkdown": "The model must have learned that for this store (store_nbr is one of our features) the early data isn't useful. The good question would be: what special treatment should we have used? Maybe dropping older data from that store?", "votes": null }, { "id": "274780", "postDate": "01/27/2018 09:43:21", "content": "", "rawMarkdown": "", "votes": null }, { "id": "274810", "postDate": "01/27/2018 11:58:37", "content": "@CPMP hi, Congrats first! I still have some question about the training data construction.
\n\nIf you create the training datasets as described above, then the label (y) will duplicate, is that true ?
\n\nFor example:
\n\n1) dataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12
\n\n2) dataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19 ....
\n\nThen the label of dataset 2) will have the same label of dataset 1) (Seven days will duplicates, but with different features.).
\n\nAnd every dataset will use the same features to predict the future 16 days, isn't? (Row numbers = num items * 16, the datasets will be very large!)
\n\nThanks for you time.
", "rawMarkdown": "CPMP hi, Congrats first! I still have some question about the training data construction.\n\nIf you create the training datasets as described above, then the label (y) will duplicate, is that true ?\n\nFor example:\n\n1) dataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12\n\n2) dataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19 ....\n\nThen the label of dataset 2) will have the same label of dataset 1) (Seven days will duplicates, but with different features.).\n\nAnd every dataset will use the same features to predict the future 16 days, isn't? (Row numbers = num items * 16, the datasets will be very large!)\n\nThanks for you time.", "votes": null }, { "id": "275486", "postDate": "01/29/2018 09:31:25", "content": "Yes, same labels with different features for overlapping days. And yes, for one dataset all features are the same for the 16 days.
", "rawMarkdown": "Yes, same labels with different features for overlapping days. And yes, for one dataset all features are the same for the 16 days.", "votes": null }, { "id": "275513", "postDate": "01/29/2018 11:19:46", "content": "@CPMP thanks for you help!!!
", "rawMarkdown": "CPMP thanks for you help!!!", "votes": null }, { "id": "275568", "postDate": "01/29/2018 13:59:03", "content": "hi @CPMP , in you approach, it seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?). In this cross validation settings, is there any good method to account for these?
\n\nMaybe the count features will be useful. But how to generate the features in you cross validation settings? Or the count features should be used in other cv settings? Look forward to you give some insights.
\n\nThanks again.
", "rawMarkdown": "hi @CPMP , in you approach, it seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?). In this cross validation settings, is there any good method to account for these? \n\nMaybe the [count features][1] will be useful. But how to generate the features in you cross validation settings? Or the count features should be used in other cv settings? Look forward to you give some insights.\n\nThanks again.\n\n [1]: http://blog.kaggle.com/2016/02/03/rossmann-store-sales-winners-interview-2nd-place-nima-shahbazi/", "votes": null } ], "comments": [ { "id": 269183, "author_name": "h4211819", "author_url": "", "post_date": "01/16/2018 12:11:20", "content": "onpromotion is not fully known after 2014-03-31 !
\n\njust look this figure , it obviously the item on promotion from 2017.8.1~2017.8.15
\n\nbut because someday there is no sales for this item，we lost some information about promotion.
\n\nIf you use these bias data to train, you will get a result that all items onpromotion have a sale>1.
\n\nthat is not true in test data.
\n\n
Not sure why you say that. Running this code:
\n\nimport pandas as pd\n\ntrain = pd.read_csv('../input/train.csv', usecols=['date', 'onpromotion'])\n\ntrain.onpromotion.fillna(-1, inplace=True)\n\ntrain.groupby('onpromotion').date.max()\nyields
\n\nonpromotion\n-1 2014-03-31\nFalse 2017-08-15\nTrue 2017-08-15\nName: date, dtype: object\nWhat am I missing?
", "votes": null, "replies": [] }, { "id": 269186, "author_name": "mykper", "author_url": "", "post_date": "01/16/2018 12:23:01", "content": "Records with zero unit sales
", "votes": null, "replies": [] }, { "id": 269188, "author_name": "h4211819", "author_url": "", "post_date": "01/16/2018 12:27:47", "content": "We don't know the onpromotion information in the records with zero unit sales.
\n\nIf we fillna with zero unit sales records to False.
\n\nOur model will get a result that if a item on promotion , its sales will >1
", "votes": null, "replies": [] }, { "id": 269189, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 12:29:06", "content": "> it obviously the item on promotion from 2017.8.1~2017.8.15
\n\n> that is not true in test data.
\n\nHow do you know both?
\n\nAnyway, we did not use that assumption. The way we use onpromotion (in the NN) is to modify on average the sales level. I guess that way was good enough to yield reasonable predictions.
\n\nI did play with how to fillna for onpromotion when adding 0 sales rows. Best CV result was obtained with fillna(0). Sure, it is a bias, but it is the best bias I found.
\n\nHow did you validate your way of filling missing onpromotion values?
", "votes": null, "replies": [] }, { "id": 269190, "author_name": "h4211819", "author_url": "", "post_date": "01/16/2018 12:32:17", "content": "So how do you deal with this item promotion information in 2017.8.12
", "votes": null, "replies": [] }, { "id": 269193, "author_name": "h4211819", "author_url": "", "post_date": "01/16/2018 12:37:38", "content": "Just check your model's result(not merged with test data) ,all items on promotion's sales >1
\n\nI had probed leaderboard , that's not true in test data
\n\nA lot items'sales on promotion is 0.
", "votes": null, "replies": [] }, { "id": 269194, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 12:43:32", "content": "As I said, when adding zero sales rows we put onpromotion to False. We tried other ways, and they led to worse CV values. How did you validate your way of filling NA? You say medals are not due to good use of CV, but I'm afraid this precisely shows the opposite: we used CV to find out how to best deal with that issue, and it worked reasonably well.
\n\nWe also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).
", "votes": null, "replies": [] }, { "id": 269195, "author_name": "h4211819", "author_url": "", "post_date": "01/16/2018 12:44:08", "content": "If you fillna(0), that caused bias both train and valiation data,
\n\nthe items'sales on promotion>1
\n\nso you will get a good cv.
\n\nbut not in test data
", "votes": null, "replies": [] }, { "id": 269196, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 12:46:04", "content": "We are circling: \"How did you validate your way of filling NA? \"
\n\nre test data:
\n\n\n", "votes": null, "replies": [] }, { "id": 269197, "author_name": "h4211819", "author_url": "", "post_date": "01/16/2018 12:50:29", "content": "We also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).
\n
I also probed LB, i found make some rules to deal with bias ( \"last 21 days no sales → onpromotopn to False\") will get 0.003 booster. The bias can't be handled by local cv. So I used leadboard to validate.\nUnfortunely , the bias exist between 5days and 11days too.
\n\nThat's the reason why a lot team drop a lot
\n\nyou can see sjv‘s post . he used leadboard to validate too.
", "votes": null, "replies": [] }, { "id": 269204, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 13:00:19", "content": "\n\n\nyou can see sjv‘s post . he used leadboard to validate too.
\n
And he drops in performance too.
\n\nBack to this:
\n\n\n\n\nIf you fillna(0), that caused bias both train and valiation data, the items'sales on promotion>1 so you will get a good cv.
\n
Now that you insist, I remember I ran lgb at point with the original data as evaluation, i.e. without any additional 0 sales rows. I wanted to make sure that adding these rows was fine. That's how I validated that filling missing onpromotion with 0 was the best I could get. And our limited LB probing confirmed later than it was a reasonable bet.
", "votes": null, "replies": [] }, { "id": 269293, "author_name": "seanjv", "author_url": "", "post_date": "01/16/2018 15:48:18", "content": "@CPMP Based on your and Giba's comments, your team hardly began to understand the issue of bias between the train and test sets. Crediting your good performance to your CV setup demonstrates further ignorance of the issue, and it's honestly a bit insulting.
\n\n\n\n\nAnd he drops in performance too.
\n
If you use the fact that my position dropped and yours raised as a retroactive justification for your CV setup, then you're a victim of a very basic statistical fallacy. I think you may learn something from this:\nhttps://www.youtube.com/watch?v=q-JgG0ECp2U
", "votes": null, "replies": [] }, { "id": 269329, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 16:33:17", "content": "@sjv, Sure, you can dismiss what we did as ignorance, hope it makes you feel better. Giba and I did discuss what to do when adding 0 sales rows, and we quickly settled on no promotion as being simple, reliable, and not dangerous. Seems we were right. Sorry if you find this insulting.
\n\nI still value what you did, and I'm one that pointed many participants to your WTF code.
\n\nEdit. My CV setting is the same as in WTF. The fact that it worked twice reasonably well may be a statistical fallacy, but I doubt it. I don't like videos, do you have the same in writing?
", "votes": null, "replies": [] }, { "id": 269331, "author_name": "npa02012", "author_url": "", "post_date": "01/16/2018 16:34:23", "content": "@sjv, it would be great to know more about how your teammates calculated 'p'. Did this method of imputing help on the private LB also?
", "votes": null, "replies": [] }, { "id": 269350, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 17:07:31", "content": "@sjv, I have watched that video. Is that the best you can do when confronted with reality? Not only am I ignorant, but I got 2 gold medals by sheer luck using a flawed CV setting? And worse, in WTF, my flawed CV setting resulted in a better submission than yours? BTW, you did not tell us what your model performance was in private LB here. Don't tell me it isn't better than mine. That would hurt.
\n\nMore seriously, let me forget about this conversation, and let me remember instead your great instacart and your great wtf code. As I wrote elsewhere, i hope you'll cool down and come compete with us again.
", "votes": null, "replies": [] }, { "id": 269358, "author_name": "titericz", "author_url": "", "post_date": "01/16/2018 17:24:52", "content": "@sjv Sorry for your frustration about dropping some places in Private LB, but it happens. It happened dozen of times with me in the past and sometimes are due our fault and other times are just bad luck. Our \"ignorance\" in understand the problem placed us where we are and not the drop of onpromotion information on zero sales dates in train. But if you had took a look at some properties of Public/Private split you won't had completely trusted your solution in Public LB. I always say and continue saying, don't trust public LB. Specially in a time series challenge where public LB is calculated using 5 days ahead and private is from 6 to 16 days ahead, they are completely different models. \nBut maybe you're right, our CV set and feature engineering strategy is a fallacy that \"luckily\" worked well on private LB. \nBest of Luck in your research from now, take some time to rest and I hope you can come back to Kaggle in the future.
", "votes": null, "replies": [] } ] }, { "id": 269243, "author_name": "adityakumarsinha", "author_url": "", "post_date": "01/16/2018 14:16:50", "content": "Congrats CPMP and Giba for gold medal again. I thought that you would be using your WTF model here, your solution there was great and here insight into data was great. After Caesar I do not have time to participate fully in this competition but giba and you have achieved quite a lot in short duration..
", "votes": null, "replies": [ { "id": 269248, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 14:24:23", "content": "Thanks! I did try to reuse my WTF model but made a mistake in it, and lost a week trying to get it running correctly. Then we restarted from scratch, but ended up with a model similar to that of WTF, except for the last convolution layer.
\n\nDon't mention Caesars to me ;) I see people complaining about data preparation here, but there it was an order of magnitude worse, at least... Not to mention the reset when I was fighting for top spot... Very bad memories to me. Almost made me quit Kaggle.
\n\nI feel better now of course with a gold medal :)
", "votes": null, "replies": [] } ] }, { "id": 270571, "author_name": "miguelpm", "author_url": "", "post_date": "01/18/2018 14:37:04", "content": "Congratulations @Giba and @CPMP for great result, and thank you for sharing .
\n\nProper cross validation in time series is quite a discussed subject so to know different approaches is of high interest imo. And your approach looks a very promising one :-)
\n\nThat said, even with your detailed explanation, not sure I got it right about your CV setup; My initial understanding:
\n\nBut then, you mention k-fold CV (10 fold) cross validation. Not sure how does k-fold fit in here so I am just getting it wrong about your setup, maybe you could kindly give me some aditional hint on how that piece fits in the puzzle.
\n\n(btw, am I the only one intrigued by \"random\" k-fold working so well on time series? It is not the first time I hear of it, as long as some precautions are taken with adjacent folds but, well... I think its something to notice)
", "votes": null, "replies": [ { "id": 270579, "author_name": "cpmpml", "author_url": "", "post_date": "01/18/2018 14:53:41", "content": "Thanks. Let me try to explain again, the write up may be confusing. For one validation period, say period of length 16 starting on 2017-07-16, we create training datasets as described above. We concatenate these into one dataset, and use kfold on it. We then train one model per fold, as usual in kfold cv, and keep the epoch with the best average cv value on the folds to compute the prediction on the validation data using the average of the fold models predictions. We then use that prediction score as our measure of the trained model. When we are done with model tuning, we use the exact same kfold on the training dataset for the test period, use the best epoch on the folds to compute the final test prediction. With our NN model, this yields a 0.003 improvement over Ceshine Lee's approach. I reported this on the forum, moving from 0.51 to 0.508 on public LB. I'm not sure why, but probably because we average the predictions of several models.
", "votes": null, "replies": [] }, { "id": 270596, "author_name": "miguelpm", "author_url": "", "post_date": "01/18/2018 15:07:41", "content": "So kfold for parameter setting and then validation periods used for model performance estimation. Thaks a lot @CPMP, its clear now :-)
", "votes": null, "replies": [] }, { "id": 274780, "author_name": "hubifeng", "author_url": "", "post_date": "01/27/2018 09:43:21", "content": "", "votes": null, "replies": [] } ] }, { "id": 271660, "author_name": "lyuxinwei", "author_url": "", "post_date": "01/21/2018 03:18:40", "content": "Thanks for sharing!\nI have a little question here.I remember one of the stores just start operation at 2017.4.20 while your team (actually a lot of teams)add the samples before 4.20 without special treatment.Could you tell me why it doesn't need special treatment?
", "votes": null, "replies": [ { "id": 271765, "author_name": "cpmpml", "author_url": "", "post_date": "01/21/2018 11:15:42", "content": "The model must have learned that for this store (store_nbr is one of our features) the early data isn't useful. The good question would be: what special treatment should we have used? Maybe dropping older data from that store?
", "votes": null, "replies": [] } ] }, { "id": 274810, "author_name": "hubifeng", "author_url": "", "post_date": "01/27/2018 11:58:37", "content": "@CPMP hi, Congrats first! I still have some question about the training data construction.
\n\nIf you create the training datasets as described above, then the label (y) will duplicate, is that true ?
\n\nFor example:
\n\n1) dataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12
\n\n2) dataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19 ....
\n\nThen the label of dataset 2) will have the same label of dataset 1) (Seven days will duplicates, but with different features.).
\n\nAnd every dataset will use the same features to predict the future 16 days, isn't? (Row numbers = num items * 16, the datasets will be very large!)
\n\nThanks for you time.
", "votes": null, "replies": [ { "id": 275486, "author_name": "cpmpml", "author_url": "", "post_date": "01/29/2018 09:31:25", "content": "Yes, same labels with different features for overlapping days. And yes, for one dataset all features are the same for the 16 days.
", "votes": null, "replies": [] }, { "id": 275513, "author_name": "hubifeng", "author_url": "", "post_date": "01/29/2018 11:19:46", "content": "@CPMP thanks for you help!!!
", "votes": null, "replies": [] } ] }, { "id": 275568, "author_name": "hubifeng", "author_url": "", "post_date": "01/29/2018 13:59:03", "content": "hi @CPMP , in you approach, it seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?). In this cross validation settings, is there any good method to account for these?
\n\nMaybe the count features will be useful. But how to generate the features in you cross validation settings? Or the count features should be used in other cv settings? Look forward to you give some insights.
\n\nThanks again.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "269174": "First of all, I'd like to thank Giba, my team mate, without whom I would not have entered this competition. I also want to thank Kaggle and the host for providing this challenging dataset. Congrats to all gold winning teams and individuals, you did an awesome job here. And there is lots of solution sharing already, which is great. Last but not least, final standing is a great, positive, surprise to us, esp considering we both had the flu.\n\nOur solution is mostly a single NN model that scores 0.515 private and 0.508 public, with a secondary lgb model. Believe it or not, we missed the deadline by few seconds and could not select our second sub ourselves. It scored 0.514 on private LB, maybe we would have advanced a couple of place, but we are very happy with the result anyway. Details below.\n\n**Missing values**\n\nonpromotion is fully known after 2014-03-31. We ignored the missing values issue by only using data posterior to that. We actually did not use data prior December 1st 2015. We also assumed that any item/store that appears somewhere in train or test data has 0 sales for the dates it does not appear. This seems to be what the data description says. It means that store/item that do not appear in train have zero sales during the train period.\n\n**Seasonality**\n\nIn time series problems there are two key ingredients: seasonality, and cv setting. Here weekly seasonality is extremely strong, yearly seasonality is weak, and there is a monthly seasonality due to pay day. We dealt with weekly seasonality by using time periods that align on weekdays for everything. We did not find a good way to use the monthly seasonality. And we captured the yearly seasonality by using sales figures from 364 days ago. We used 364 and not 365 to align with weekdays.\n\n**CV setting**\n\nWe selected 2 validation periods plus the test period: (2017-07-16, 2017-07-31), (2017-08-01, 2017-08-15), and (2017-08-16, 2017-08-31). For each period we construct a series of training datasets aligned on weekday, for instance for the first validation period:\n\n dataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12\n dataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19\n dataset up to 2017-03-25 predict period of length 16 starting on 2017-03-26\n dataset up to 2017-04-01 predict period of length 16 starting on 2017-04-02\n dataset up to 2017-04-08 predict period of length 16 starting on 2017-04-09\n dataset up to 2017-04-15 predict period of length 16 starting on 2017-04-16\n dataset up to 2017-04-22 predict period of length 16 starting on 2017-04-23\n dataset up to 2017-04-29 predict period of length 16 starting on 2017-04-30\n dataset up to 2017-05-06 predict period of length 16 starting on 2017-05-07\n dataset up to 2017-05-13 predict period of length 16 starting on 2017-05-14\n dataset up to 2017-05-20 predict period of length 16 starting on 2017-05-21\n dataset up to 2017-05-27 predict period of length 16 starting on 2017-05-28\n dataset up to 2017-06-03 predict period of length 16 starting on 2017-06-04\n dataset up to 2017-06-10 predict period of length 16 starting on 2017-06-11\n dataset up to 2017-06-17 predict period of length 16 starting on 2017-06-18\n dataset up to 2017-06-24 predict period of length 16 starting on 2017-06-25\n dataset up to 2017-07-15 predict period of length 16 starting on 2017-07-16\n\nWe use all datasets but the last to train, and we use the last one for early stopping. We use 10 fold CV on the training periods. The only caveat is to make sure all store/item pairs for a given time series are in the same fold. I used a very similar CV setting in WTF competition.\n\n**Feature engineering**\n\nAll unit sales are clipped to be non negative, then log1p transformed. For each of the above datasets, we use:\n\n - lags over periods similar to @Ceshine Lee starter LGB kernel. We use mean, max, and proportion of zero entries, as well as proportion of promotion days.\n\n - Sales for each of the last 7 days\n\n - Average sales per weekday over the last 8 weeks\n\n - For each of the test/validation days: sales from 364 days earlier, and promotion status.\n\n - Class of the item\n\n - Store\n\nWe did not use oil price, item number, and other info in our primary model.\n\n**NN Model**\n\nA feedforward model with 3 dense layers, relu activation. Class and store are embedded in 4 dimension vectors, and appended at the second level. Let x be the output of the dense layers. Its length is equal to the length of the test or validation period. Last level is a 1D convolution with x, the on promotion input vector, and the product of the onpromotion vector with x:\n\n y = Multiply()([x, promo_test_input])\n y = Reshape((-1,1))(y)\n x = Reshape((-1,1))(x)\n z = Reshape((-1,1))(promo_test_input)\n x = Concatenate(axis=-1)([x, y, z])\n x = Conv1D(1,1, activation='linear')(x)\n\nThis captures the influence of onpromotion day per day quite well.\nThe target for the model is the sequence of train/validation sales, i.e. 15 or 16 days depending on the period.\n\n**Other models**\n\nWe trained a lgb model using the same features and cv setting, the only difference being the target: we train one model per day, like in @Ceshine Lee starter LGB kernel. Best lgb scores 0.510 on pblic LB and 0.517 on private LB. I don't know how others got better results with LGB than with NN.\n\nGiba created a classifier model to predict when sales are zero or not. Alone this model is useless but it adds when ensembling. I'll let him describe it if need be.\n\n**Ensembling**\n\nTurns out that what would have been our best sub is an unweighted average of 3 different runs of the NN model using a different sets of training periods, and one lgb model. We tried some limited stacking with a NN with one hidden layer. Giba also looked to how to post process submissions. By default we set to 0 all store/item pairs that do not appear in train, but he found that we should not do it if that pair has a promotion in test. He got a sub that scores 0.514 private that way, but, due to few second glitch we did not not select it... And we did not have time to combine his postprocessing with our best blend either.\n\n**Conclusion**\n\nMain takeaway from me is once again to ignore public LB feedback. Indeed, that feedback was on the first 5 days of the test period, and it gives zero useful information on what happens for days further in the future. Having some reasonable CV setting was key to estimate how well a model predicts further in the future. I could not say I was expecting to get a result as good as what we got in the end, but I knew we were not overfitting to the first few days of the test period ;) \n\nLet me end with a special notice to Ceshine Lee without whom many would not have fared as well as they did. He shared a great kernel, and he shared very useful tips in the early days of the competition. There should be a sharing gold medal in Kaggle competitions, and he would have won that one here!\n\n**Edit:** \nThe discussion below makes me add this. We added 0 sales row with onpromotion=False for all store/item pairs that appear in our training datasets. We did not add all those store/items pairs that appear in the test set. We did compare with and without these: the results were similar, but training time were higher with the test items. We validated adding rows with onpromotion=False by using the original data in the validation periods.", "269183": "onpromotion is not fully known after 2014-03-31 !\n\njust look this figure , it obviously the item on promotion from 2017.8.1~2017.8.15\n\nbut because someday there is no sales for this item，we lost some information about promotion.\n\nIf you use these bias data to train, you will get a result that all items onpromotion have a sale>1.\n\nthat is not true in test data.\n\n\n![not known][1]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/269183/8271/2018-01-16%2020-09-34%E5%B1%8F%E5%B9%95%E6%88%AA%E5%9B%BE.png", "269184": "Not sure why you say that. Running this code:\n\n import pandas as pd\n \n train = pd.read_csv('../input/train.csv', usecols=['date', 'onpromotion'])\n \n train.onpromotion.fillna(-1, inplace=True)\n \n train.groupby('onpromotion').date.max()\n\nyields\n\n onpromotion\n -1 2014-03-31\n False 2017-08-15\n True 2017-08-15\n Name: date, dtype: object\n\nWhat am I missing?", "269186": "Records with zero unit sales", "269188": "We don't know the onpromotion information in the records with zero unit sales.\n\nIf we fillna with zero unit sales records to False.\n\nOur model will get a result that if a item on promotion , its sales will >1", "269189": "> it obviously the item on promotion from 2017.8.1~2017.8.15\n\n> that is not true in test data.\n\nHow do you know both? \n\nAnyway, we did not use that assumption. The way we use onpromotion (in the NN) is to modify on average the sales level. I guess that way was good enough to yield reasonable predictions.\n\nI did play with how to fillna for onpromotion when adding 0 sales rows. Best CV result was obtained with fillna(0). Sure, it is a bias, but it is the best bias I found. \n\nHow did you validate your way of filling missing onpromotion values?", "269190": "So how do you deal with this item promotion information in 2017.8.12", "269193": "Just check your model's result(not merged with test data) ,all items on promotion's sales >1\n\nI had probed leaderboard , that's not true in test data\n\nA lot items'sales on promotion is 0.", "269194": "As I said, when adding zero sales rows we put onpromotion to False. We tried other ways, and they led to worse CV values. How did you validate your way of filling NA? You say medals are not due to good use of CV, but I'm afraid this precisely shows the opposite: we used CV to find out how to best deal with that issue, and it worked reasonably well.\n\nWe also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).", "269195": "If you fillna(0), that caused bias both train and valiation data,\n\nthe items'sales on promotion>1\n\nso you will get a good cv.\n\nbut not in test data", "269196": "We are circling: \"How did you validate your way of filling NA? \"\n\nre test data: \n> We also probed LB a bit, and found that setting sales > 0 for items that have at least on promotion day was improving LB (both public, and private actually).", "269197": "I also probed LB, i found make some rules to deal with bias ( \"last 21 days no sales → onpromotopn to False\") will get 0.003 booster. The bias can't be handled by local cv. So I used leadboard to validate.\nUnfortunely , the bias exist between 5days and 11days too.\n\nThat's the reason why a lot team drop a lot\n\nyou can see sjv‘s post . he used leadboard to validate too.", "269204": "> you can see sjv‘s post . he used leadboard to validate too.\n\nAnd he drops in performance too.\n\nBack to this:\n\n> If you fillna(0), that caused bias both train and valiation data, the items'sales on promotion>1 so you will get a good cv.\n\nNow that you insist, I remember I ran lgb at point with the original data as evaluation, i.e. without any additional 0 sales rows. I wanted to make sure that adding these rows was fine. That's how I validated that filling missing onpromotion with 0 was the best I could get. And our limited LB probing confirmed later than it was a reasonable bet.", "269243": "Congrats CPMP and Giba for gold medal again. I thought that you would be using your WTF model here, your solution there was great and here insight into data was great. After Caesar I do not have time to participate fully in this competition but giba and you have achieved quite a lot in short duration..", "269248": "Thanks! I did try to reuse my WTF model but made a mistake in it, and lost a week trying to get it running correctly. Then we restarted from scratch, but ended up with a model similar to that of WTF, except for the last convolution layer. \n\nDon't mention Caesars to me ;) I see people complaining about data preparation here, but there it was an order of magnitude worse, at least... Not to mention the reset when I was fighting for top spot... Very bad memories to me. Almost made me quit Kaggle. \n\nI feel better now of course with a gold medal :)", "269293": "CPMP Based on your and Giba's comments, your team hardly began to understand the issue of bias between the train and test sets. Crediting your good performance to your CV setup demonstrates further ignorance of the issue, and it's honestly a bit insulting. \n\n> And he drops in performance too.\n\nIf you use the fact that my position dropped and yours raised as a retroactive justification for your CV setup, then you're a victim of a very basic statistical fallacy. I think you may learn something from this:\nhttps://www.youtube.com/watch?v=q-JgG0ECp2U", "269329": "sjv, Sure, you can dismiss what we did as ignorance, hope it makes you feel better. Giba and I did discuss what to do when adding 0 sales rows, and we quickly settled on no promotion as being simple, reliable, and not dangerous. Seems we were right. Sorry if you find this insulting.\n\n I still value what you did, and I'm one that pointed many participants to your WTF code.\n\nEdit. My CV setting is the same as in WTF. The fact that it worked twice reasonably well may be a statistical fallacy, but I doubt it. I don't like videos, do you have the same in writing?", "269331": "sjv, it would be great to know more about how your teammates calculated 'p'. Did this method of imputing help on the private LB also?", "269350": "sjv, I have watched that video. Is that the best you can do when confronted with reality? Not only am I ignorant, but I got 2 gold medals by sheer luck using a flawed CV setting? And worse, in WTF, my flawed CV setting resulted in a better submission than yours? BTW, you did not tell us what your model performance was in private LB here. Don't tell me it isn't better than mine. That would hurt.\n\nMore seriously, let me forget about this conversation, and let me remember instead your great instacart and your great wtf code. As I wrote elsewhere, i hope you'll cool down and come compete with us again.", "269358": "sjv Sorry for your frustration about dropping some places in Private LB, but it happens. It happened dozen of times with me in the past and sometimes are due our fault and other times are just bad luck. Our \"ignorance\" in understand the problem placed us where we are and not the drop of onpromotion information on zero sales dates in train. But if you had took a look at some properties of Public/Private split you won't had completely trusted your solution in Public LB. I always say and continue saying, don't trust public LB. Specially in a time series challenge where public LB is calculated using 5 days ahead and private is from 6 to 16 days ahead, they are completely different models. \nBut maybe you're right, our CV set and feature engineering strategy is a fallacy that \"luckily\" worked well on private LB. \nBest of Luck in your research from now, take some time to rest and I hope you can come back to Kaggle in the future.", "270571": "**Congratulations @Giba and @CPMP for great result, and thank you for sharing** . \n\nProper cross validation in time series is quite a discussed subject so to know different approaches is of high interest imo. And your approach looks a very promising one :-)\n\nThat said, even with your detailed explanation, not sure I got it right about your CV setup; My initial understanding:\n\n- you feed a NN with (all) sequence input and sequence outputs built from 16 bi-weekly chunks, using 17th to get n_epochs, and then retrain all for prediction of test, and you do this twice (last two be-weekly periods) \n\nBut then, you mention k-fold CV (10 fold) cross validation. Not sure **how does k-fold fit in here** so I am just getting it wrong about your setup, maybe you could kindly give me some aditional hint on how that piece fits in the puzzle. \n\n(btw, am I the only one intrigued by \"random\" k-fold working so well on time series? It is not the first time I hear of it, as long as some precautions are taken with adjacent folds but, well... I think its something to notice)", "270579": "Thanks. Let me try to explain again, the write up may be confusing. For one validation period, say period of length 16 starting on 2017-07-16, we create training datasets as described above. We concatenate these into one dataset, and use kfold on it. We then train one model per fold, as usual in kfold cv, and keep the epoch with the best average cv value on the folds to compute the prediction on the validation data using the average of the fold models predictions. We then use that prediction score as our measure of the trained model. When we are done with model tuning, we use the exact same kfold on the training dataset for the test period, use the best epoch on the folds to compute the final test prediction. With our NN model, this yields a 0.003 improvement over Ceshine Lee's approach. I reported this on the forum, moving from 0.51 to 0.508 on public LB. I'm not sure why, but probably because we average the predictions of several models.", "270596": "So kfold for parameter setting and then validation periods used for model performance estimation. Thaks a lot @CPMP, its clear now :-)", "271660": "Thanks for sharing!\nI have a little question here.I remember one of the stores just start operation at 2017.4.20 while your team (actually a lot of teams)add the samples before 4.20 without special treatment.Could you tell me why it doesn't need special treatment?", "271765": "The model must have learned that for this store (store_nbr is one of our features) the early data isn't useful. The good question would be: what special treatment should we have used? Maybe dropping older data from that store?", "274780": "", "274810": "CPMP hi, Congrats first! I still have some question about the training data construction.\n\nIf you create the training datasets as described above, then the label (y) will duplicate, is that true ?\n\nFor example:\n\n1) dataset up to 2017-03-11 predict period of length 16 starting on 2017-03-12\n\n2) dataset up to 2017-03-18 predict period of length 16 starting on 2017-03-19 ....\n\nThen the label of dataset 2) will have the same label of dataset 1) (Seven days will duplicates, but with different features.).\n\nAnd every dataset will use the same features to predict the future 16 days, isn't? (Row numbers = num items * 16, the datasets will be very large!)\n\nThanks for you time.", "275486": "Yes, same labels with different features for overlapping days. And yes, for one dataset all features are the same for the 16 days.", "275513": "CPMP thanks for you help!!!", "275568": "hi @CPMP , in you approach, it seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?). In this cross validation settings, is there any good method to account for these? \n\nMaybe the [count features][1] will be useful. But how to generate the features in you cross validation settings? Or the count features should be used in other cv settings? Look forward to you give some insights.\n\nThanks again.\n\n [1]: http://blog.kaggle.com/2016/02/03/rossmann-store-sales-winners-interview-2nd-place-nima-shahbazi/" }, "source": "meta" }