I use several validation set (different time period) to validate my feature. All of them have quite robust improvement. But the public score drops after including those feature. I'm quite sure I don't have leakage when I train and validate the model. Is there anyone having the same problem?
", "messages": [ { "id": "252329", "postDate": "12/02/2017 19:01:16", "content": "I use several validation set (different time period) to validate my feature. All of them have quite robust improvement. But the public score drops after including those feature. I'm quite sure I don't have leakage when I train and validate the model. Is there anyone having the same problem?
", "rawMarkdown": "I use several validation set (different time period) to validate my feature. All of them have quite robust improvement. But the public score drops after including those feature. I'm quite sure I don't have leakage when I train and validate the model. Is there anyone having the same problem?", "votes": null }, { "id": "252427", "postDate": "12/02/2017 22:24:35", "content": "I have the same issue, and it's baffling.
", "rawMarkdown": "I have the same issue, and it's baffling.", "votes": null }, { "id": "252503", "postDate": "12/03/2017 03:52:54", "content": "lol
", "rawMarkdown": "lol", "votes": null }, { "id": "252867", "postDate": "12/04/2017 03:16:43", "content": "The more work I do, the more bias I feel the train data has. On average it has more than 40k id (store , item) are missing every day. However, we have no information about these missing value. And in test data, this information is included.
", "rawMarkdown": "The more work I do, the more bias I feel the train data has. On average it has more than 40k id (store , item) are missing every day. However, we have no information about these missing value. And in test data, this information is included.", "votes": null }, { "id": "252908", "postDate": "12/04/2017 05:51:52", "content": "I have similar experiences. Selecting features and models based on validation scores can be somewhat biased for this dataset (I guess this is natural for a time-based prediction task). Usually I'd lose 0.001 ~ 0.002 expected gain when it happens.
", "rawMarkdown": "I have similar experiences. Selecting features and models based on validation scores can be somewhat biased for this dataset (I guess this is natural for a time-based prediction task). Usually I'd lose 0.001 ~ 0.002 expected gain when it happens.", "votes": null }, { "id": "253423", "postDate": "12/04/2017 23:08:02", "content": "I've had similar problems when trying to improve the median/mean predictor models.
\nMy XGBoost model does similar on my validation and the LB test set -- but i've been having trouble getting past .56 with it!
Be careful about validation - models that are better on a 2-week period are not necessarily better on the 5-day period that the public leaderboard is based on.
", "rawMarkdown": "Be careful about validation - models that are better on a 2-week period are not necessarily better on the 5-day period that the public leaderboard is based on.", "votes": null }, { "id": "254031", "postDate": "12/06/2017 04:32:46", "content": "Seems like the best strategy might be to have two validations, one designed to match the public LB and the other designed to match the private, but otherwise using the same approach. Then you can use the public LB as a check on the validation but still validate for the private LB.
", "rawMarkdown": "Seems like the best strategy might be to have two validations, one designed to match the public LB and the other designed to match the private, but otherwise using the same approach. Then you can use the public LB as a check on the validation but still validate for the private LB.", "votes": null }, { "id": "254064", "postDate": "12/06/2017 05:46:35", "content": "Yes, I'm been doing exactly that. But the validation score for the first 5 day and the public score can still not match. Does anyone find differently?
", "rawMarkdown": "Yes, I'm been doing exactly that. But the validation score for the first 5 day and the public score can still not match. Does anyone find differently?", "votes": null }, { "id": "254070", "postDate": "12/06/2017 05:57:03", "content": "I think the problem is because we have a bias train data. The bias is from the missing value and its corresponding promotion information.
", "rawMarkdown": "I think the problem is because we have a bias train data. The bias is from the missing value and its corresponding promotion information.", "votes": null }, { "id": "254197", "postDate": "12/06/2017 12:38:27", "content": "I just started on this competition, and when I run the ma script (https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-529) with another train/test split I have a huge difference with the public LB score. Is it just me (which means I screwed Paulo Pinto's code) or did others found the same?
", "rawMarkdown": "I just started on this competition, and when I run the ma script (https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-529) with another train/test split I have a huge difference with the public LB score. Is it just me (which means I screwed Paulo Pinto's code) or did others found the same?", "votes": null }, { "id": "254260", "postDate": "12/06/2017 15:43:14", "content": "Are you talking about items on promotion but have no unit sold? I agree that missing that information certainly can make the model be too optimistic on items. But since we have no such information, how much mismatch can be attribute to it is unknown.
", "rawMarkdown": "Are you talking about items on promotion but have no unit sold? I agree that missing that information certainly can make the model be too optimistic on items. But since we have no such information, how much mismatch can be attribute to it is unknown.", "votes": null }, { "id": "254307", "postDate": "12/06/2017 17:45:14", "content": "Exactly
", "rawMarkdown": "Exactly", "votes": null }, { "id": "254313", "postDate": "12/06/2017 17:56:15", "content": "I also noticed this asymmetry between the train and test sets. If you use onpromotion as a feature, you have to accept that its distribution differs in the train and test sets. If you ignore the feature altogether you remove the asymmetry but also lose an important source of information. For what it's worth, I get about the same scores with and without using onpromotion. Curious if you guys are seeing the same thing?
", "rawMarkdown": "I also noticed this asymmetry between the train and test sets. If you use onpromotion as a feature, you have to accept that its distribution differs in the train and test sets. If you ignore the feature altogether you remove the asymmetry but also lose an important source of information. For what it's worth, I get about the same scores with and without using onpromotion. Curious if you guys are seeing the same thing?", "votes": null }, { "id": "254322", "postDate": "12/06/2017 18:07:52", "content": "Yes. Amazing you can get same score with and without using onpromotion.
", "rawMarkdown": "Yes. Amazing you can get same score with and without using onpromotion.", "votes": null }, { "id": "254441", "postDate": "12/07/2017 00:29:21", "content": "Interesting. Haven't actually tried removing onpromotion for a while. Will try it soon and see how it goes. Thanks for the tip.
", "rawMarkdown": "Interesting. Haven't actually tried removing onpromotion for a while. Will try it soon and see how it goes. Thanks for the tip.", "votes": null }, { "id": "254721", "postDate": "12/07/2017 13:34:11", "content": "It seems removing onpromotion still makes my model converge extremely slowly and I gave up midway. Evidently my models are far less robust as @sjv's.
", "rawMarkdown": "It seems removing onpromotion still makes my model converge extremely slowly and I gave up midway. Evidently my models are far less robust as @sjv's.", "votes": null }, { "id": "255860", "postDate": "12/10/2017 11:19:50", "content": "You don't know how long it takes to train @sjv model(s)... I bet he is training on the full train set based on past competitions, and based on the comment about NaN in onpromotion.
", "rawMarkdown": "You don't know how long it takes to train @sjv model(s)... I bet he is training on the full train set based on past competitions, and based on the comment about NaN in onpromotion.", "votes": null }, { "id": "255868", "postDate": "12/10/2017 12:12:01", "content": "I'd love to hear from anyone that observes the same thing to get some clues of where that robustness came from.
", "rawMarkdown": "I'd love to hear from anyone that observes the same thing to get some clues of where that robustness came from.", "votes": null }, { "id": "256757", "postDate": "12/12/2017 17:02:34", "content": "", "rawMarkdown": "", "votes": null }, { "id": "257501", "postDate": "12/14/2017 11:18:13", "content": "Guys, I am seeing similar issue. Basically I am using XGBoost model for this challenge and use rmse as a validation mechanism. Over a period of time, I think the rmse error has dropped all the way to 0.5-0.6 ish however my public score is showing in opposite direction. Much appreciate if someone can help!!
", "rawMarkdown": "Guys, I am seeing similar issue. Basically I am using XGBoost model for this challenge and use rmse as a validation mechanism. Over a period of time, I think the rmse error has dropped all the way to 0.5-0.6 ish however my public score is showing in opposite direction. Much appreciate if someone can help!!", "votes": null }, { "id": "257621", "postDate": "12/14/2017 16:38:36", "content": "Is anyone willing to share numbers?
\nOn my XGBoost model, I have LB: .527 --- CV: .557 (7/31 --> 8/15) and .548 (7/31 --> 8/4)
I was originally getting a CV-error of .635 -- I reduced the difference by adding all item/store/date combinations to the test set and assuming 0 unit_sales.
", "rawMarkdown": "Is anyone willing to share numbers? \nOn my XGBoost model, I have LB: .527 --- CV: .557 (7/31 --> 8/15) and .548 (7/31 --> 8/4)\n \nI was originally getting a CV-error of .635 -- I reduced the difference by adding all item/store/date combinations to the test set and assuming 0 unit_sales.", "votes": null }, { "id": "257958", "postDate": "12/15/2017 06:42:55", "content": "Even I am using XGBoost mode and I am happy to share the details:
\n\nLatest version of my code was showing:\ntrain-rmse:0.4802 eval-rmse:0.4802, whereas public leaderboard score is 1.297.
\n\nThis is something I don;t understand. How come my public leaderboard score is increasing everytime I make submission. This is my first competition and I think I'm missing something here. Could someone please help in understanding how public leaderboard score is validating.
\n\n@Nicolas I didn;t get about 7/31 --> 8/15 could you please clarify.
\n\nManjy Thanks\nManju
", "rawMarkdown": "Even I am using XGBoost mode and I am happy to share the details:\n\nLatest version of my code was showing:\n**train-rmse:0.4802\teval-rmse:0.4802**, whereas public leaderboard score is **1.297**.\n\nThis is something I don;t understand. How come my public leaderboard score is increasing everytime I make submission. This is my first competition and I think I'm missing something here. Could someone please help in understanding how public leaderboard score is validating.\n\n@Nicolas I didn;t get about 7/31 --> 8/15 could you please clarify.\n\nManjy Thanks\nManju", "votes": null }, { "id": "257996", "postDate": "12/15/2017 08:14:16", "content": "Hello there! I've come to similar results at some point. The thing is how you compose your eval set. From the numbers you provide I guess you take it from the same set as train set, i.e. with maybe some outbursts excluded and so on, but the main thing is that your eval set already contains information on what items were sold on what dates. So that 0.4802 is a misleading number. Try to construct your eval set from all the item nbrs that are there, as well as all dates. And you'll see eval score drop significantly.
", "rawMarkdown": "Hello there! I've come to similar results at some point. The thing is how you compose your eval set. From the numbers you provide I guess you take it from the same set as train set, i.e. with maybe some outbursts excluded and so on, but the main thing is that your eval set already contains information on what items were sold on what dates. So that 0.4802 is a misleading number. Try to construct your eval set from all the item nbrs that are there, as well as all dates. And you'll see eval score drop significantly.", "votes": null }, { "id": "258004", "postDate": "12/15/2017 08:26:08", "content": "And therefore I think there's an overfitting - we do better and better on a training set and losing more and more on test. But I tried quite extensively many scenarios of limitation of item nbrs to actually build predictions for, and it doesn't get better in any significant way. So I'm still puzzled how to come up with decent results with xbg, there's surely something I miss here. Maybe it's still just feature engineering to be done better... with means calculated in various ways as the best features so far.
", "rawMarkdown": "And therefore I think there's an overfitting - we do better and better on a training set and losing more and more on test. But I tried quite extensively many scenarios of limitation of item nbrs to actually build predictions for, and it doesn't get better in any significant way. So I'm still puzzled how to come up with decent results with xbg, there's surely something I miss here. Maybe it's still just feature engineering to be done better... with means calculated in various ways as the best features so far.", "votes": null }, { "id": "258047", "postDate": "12/15/2017 10:36:45", "content": "Thank you Alexander for the response.\nI am using train_test_split method to split the training data.
\n\n*X_train, X_valid = train_test_split(train_items, test_size=0.30, random_state=20)\ndtrain = xgb.DMatrix(X_train[features], y_train)\ndvalid = xgb.DMatrix(X_valid[features], y_valid)*\nThe items_sales is dropped from the X_train and X_valid dataframe.
\n\nAnd in the XGB train, I set evals=watchlist, where watchlist is [(dtrain, 'train'), (dvalid, 'eval')]
\n\nI like your suggestion about constructing eval set with just item_nbrs, date and try to predict and see what happens to rmse score. I had a look at other kernels with LB 0.5ish and they didn't carry out feature engineering, which is interesting.\nRegarding LB score, my understanding is that, once we submit our prediction results, which will compared against the actual number/item_sales in this case. The comparison method used is RMSLE?? The result of RMSLE will decide where I stand in the competition. One important point, I found is that every I submit the prediction, kaggle compares against the 50% of the test set and this set would be randomly selected from the test set.
\n\nAs this is my first competition so bit of struggle, but I really appreciate people are very helpful.
", "rawMarkdown": "Thank you Alexander for the response.\nI am using train_test_split method to split the training data. \n\n*X_train, X_valid = train_test_split(train_items, test_size=0.30, random_state=20)\ndtrain = xgb.DMatrix(X_train[features], y_train)\ndvalid = xgb.DMatrix(X_valid[features], y_valid)*\nThe items_sales is dropped from the X_train and X_valid dataframe.\n\nAnd in the XGB train, I set evals=watchlist, where watchlist is [(dtrain, 'train'), (dvalid, 'eval')]\n\n\nI like your suggestion about constructing eval set with just item_nbrs, date and try to predict and see what happens to rmse score. I had a look at other kernels with LB 0.5ish and they didn't carry out feature engineering, which is interesting.\nRegarding LB score, my understanding is that, once we submit our prediction results, which will compared against the actual number/item_sales in this case. The comparison method used is RMSLE?? The result of RMSLE will decide where I stand in the competition. One important point, I found is that every I submit the prediction, kaggle compares against the 50% of the test set and this set would be randomly selected from the test set.\n\nAs this is my first competition so bit of struggle, but I really appreciate people are very helpful.", "votes": null }, { "id": "258110", "postDate": "12/15/2017 13:58:09", "content": "@gmanjunath 7/31 --> 8/15 refers to the dates that I use in my CV test set.
\nI included the 7/31 --> 8/4 as a CV error check on the first 4 days, as we know that the public leader board is only calculated on 8/16 --> 8/21.
\nFrom your code snippet, it is not completely clear how you are generating your test set - it seems that you may be taking random data-points from train. In my opinion, it is best to create it based on time.
@Alexander check out one of these kernels to see how to build a strong model. They use LGBM, but I was able to easily convert to XGBoost and have similar results.
\nLGBM Starter in R
\nLGBM Starter in Python
@Nicolas Thank you very much for the suggestion and also for sharing the reference code.\nRegarding the test data, yes you are correct I am taking random data-points from the training set.\nI will have a look at your code and modify my code accordingly.
", "rawMarkdown": "Nicolas Thank you very much for the suggestion and also for sharing the reference code.\nRegarding the test data, yes you are correct I am taking random data-points from the training set.\nI will have a look at your code and modify my code accordingly.", "votes": null }, { "id": "261817", "postDate": "12/24/2017 04:09:45", "content": "Same problem, difference between scores from local validations (say 2017.8.1~2017.8.15) and PB can be around 0.3. It's rather baffling and confusing. It's my first time join a Kaggle competition so I don't know if it's normal situations in other Kaggle competitions.
", "rawMarkdown": "Same problem, difference between scores from local validations (say 2017.8.1~2017.8.15) and PB can be around 0.3. It's rather baffling and confusing. It's my first time join a Kaggle competition so I don't know if it's normal situations in other Kaggle competitions.", "votes": null }, { "id": "262157", "postDate": "12/25/2017 15:53:25", "content": "So Is there a good validation strategy up to now~ I think I possibly overfit public LB~
", "rawMarkdown": "So Is there a good validation strategy up to now~ I think I possibly overfit public LB~", "votes": null }, { "id": "262166", "postDate": "12/25/2017 16:36:18", "content": "As long as the LB-CV=constant, I suppose the validation can be trusted.
", "rawMarkdown": "As long as the LB-CV=constant, I suppose the validation can be trusted.", "votes": null }, { "id": "262392", "postDate": "12/26/2017 12:23:10", "content": "@Nicolas I did modify the code to train the model from 06/01 --> 07/31 and test using 8/01 --> 8/15. As mentioned the validation score was not good so I had to fine tune some of the XGBoost parameters. I am seeing good rmse score using the test dataset, however I am still getting 1.2ish LB score. \nHere is the link to my code\nhttps://www.kaggle.com/gmanjunath/xgboost-predictions\nCould you please cast your eyes to the code and any suggestions would be much appreciated.
\n\nI can see many people are having this issue, it would be good to know the reason for inconsistency in the LB score and the validation score.
", "rawMarkdown": "Nicolas I did modify the code to train the model from 06/01 --> 07/31 and test using 8/01 --> 8/15. As mentioned the validation score was not good so I had to fine tune some of the XGBoost parameters. I am seeing good rmse score using the test dataset, however I am still getting 1.2ish LB score. \nHere is the link to my code\nhttps://www.kaggle.com/gmanjunath/xgboost-predictions\nCould you please cast your eyes to the code and any suggestions would be much appreciated.\n\nI can see many people are having this issue, it would be good to know the reason for inconsistency in the LB score and the validation score.", "votes": null }, { "id": "262524", "postDate": "12/27/2017 01:14:23", "content": "Complete missing values for existing tuples (stores_nbr, items_nbr) with 0 sales. Therefore your algorithm predicts higher value.
\n\nYou probably missed this in R/Python LGBM scripts.
", "rawMarkdown": "Complete missing values for existing tuples (stores_nbr, items_nbr) with 0 sales. Therefore your algorithm predicts higher value.\n\nYou probably missed this in R/Python LGBM scripts.", "votes": null }, { "id": "262769", "postDate": "12/27/2017 22:27:25", "content": "@Jacques, Thank you very much for the suggestion. Much appreciated.\nI had a look at the Python version of LGBM, I can see towards the end of the code, there is code submission = df_test[[\"id\"]].join(df_preds, how=\"left\").fillna(0) which fills 0, this is something I was not doing in my code. So I modified my code by doing this:\nsubmit.loc[:, \"unit_sales\"].fillna(0, inplace=True) just before converting to csv file. I did try with this change, unfortunately no luck.Any pointers please.
", "rawMarkdown": "Jacques, Thank you very much for the suggestion. Much appreciated.\nI had a look at the Python version of LGBM, I can see towards the end of the code, there is code **submission = df_test[[\"id\"]].join(df_preds, how=\"left\").fillna(0)** which fills 0, this is something I was not doing in my code. So I modified my code by doing this:\n**submit.loc[:, \"unit_sales\"].fillna(0, inplace=True)** just before converting to csv file. I did try with this change, unfortunately no luck.Any pointers please.", "votes": null }, { "id": "262948", "postDate": "12/28/2017 12:03:52", "content": "I appreciate the fact you are trying to learn and not blindly copy/pasting public script and tuning hyperparameters like a lot of people are doing. However you'll study Ceshine's script more accurately. Use it, execute it line by line and see what are the input and output. It will help you understand what's going on.
\n\nYes you need to fill by 0 tuples (stores_nbr, items_nbr) not present in train set. However in your training sets you only have tuples (stores_nbr, items_nbr, dates) where sales are positives. You need to \"complete/expand\" missing values of date for tuples (stores_nbr, items_nbr) with 0 sales and onpromotion.
\n\nCeshine does it by using \"unstack\" function and using (stores_nbr, items_nbr) as index.
\n\nHave a look at this kernel to better understand what's going on. \nhttps://www.kaggle.com/aharless/dissecting-ceshine-lee-s-lgbm-kernel
", "rawMarkdown": "I appreciate the fact you are trying to learn and not blindly copy/pasting public script and tuning hyperparameters like a lot of people are doing. However you'll study Ceshine's script more accurately. Use it, execute it line by line and see what are the input and output. It will help you understand what's going on.\n\nYes you need to fill by 0 tuples (stores_nbr, items_nbr) not present in train set. However in your training sets you only have tuples (stores_nbr, items_nbr, dates) where sales are positives. You need to \"complete/expand\" missing values of date for tuples (stores_nbr, items_nbr) with 0 sales and onpromotion. \n\nCeshine does it by using \"unstack\" function and using (stores_nbr, items_nbr) as index. \n\nHave a look at this kernel to better understand what's going on. \nhttps://www.kaggle.com/aharless/dissecting-ceshine-lee-s-lgbm-kernel", "votes": null }, { "id": "263010", "postDate": "12/28/2017 17:03:45", "content": "@gmanjunath - I took a look at your code. I think your biggest problem is your lack of features. Remember that XGBoost does NOT take categorical variables. So XGBoost will not decipher item_nbr, store_nbr, nor family as you might hope.
\n\nThe day/month/year, perishable and onpromotion are your other features and are good, but not nearly enough to determine unit_sales.
\n\nTry adding 'median_sales' to each item/store combination as a predictor variable. This should give your model a lot more predictive power (probably ~ LB .65).
\n\nI would say do this first to gain some momentum. Afterwards, as Jacques Peeters mentioned, decipher and run Ceshine's code until you understand it both line-by-line and conceptually. The structure is more complex then what you have done, but also more powerful/faster.
\n\nGood luck!
", "rawMarkdown": "gmanjunath - I took a look at your code. I think your biggest problem is your lack of features. Remember that XGBoost does NOT take categorical variables. So XGBoost will not decipher item_nbr, store_nbr, nor family as you might hope. \n \nThe day/month/year, perishable and onpromotion are your other features and are good, but not nearly enough to determine unit_sales. \n \nTry adding 'median_sales' to each item/store combination as a predictor variable. This should give your model a lot more predictive power (probably ~ LB .65). \n \nI would say do this first to gain some momentum. Afterwards, as Jacques Peeters mentioned, decipher and run Ceshine's code until you understand it both line-by-line and conceptually. The structure is more complex then what you have done, but also more powerful/faster. \n \nGood luck!", "votes": null }, { "id": "263073", "postDate": "12/28/2017 21:26:48", "content": "@Jacques and @Nicolas thank you very much for taking time to review my code. I really appreciate your suggestions. \nI will modify the code as per your suggestions and give it a try.
", "rawMarkdown": "Jacques and @Nicolas thank you very much for taking time to review my code. I really appreciate your suggestions. \nI will modify the code as per your suggestions and give it a try.", "votes": null }, { "id": "266824", "postDate": "01/09/2018 22:07:12", "content": "@Jacques and @Nicolas, I managed to crack the prediction results (0.53). As you guys pointed out, I was not filling up missing values for (stores/item nbr), which I am doing now using the pivot table. I must admit that my code is based on Ceshine's code, however this was done to gain the momentum. Next I am planning to play around the training data by picking up the data for the month of Aug from previous year and see how much it will make difference.\nOnce again thank you so much guys for the support.
", "rawMarkdown": "Jacques and @Nicolas, I managed to crack the prediction results (0.53). As you guys pointed out, I was not filling up missing values for (stores/item nbr), which I am doing now using the pivot table. I must admit that my code is based on Ceshine's code, however this was done to gain the momentum. Next I am planning to play around the training data by picking up the data for the month of Aug from previous year and see how much it will make difference.\nOnce again thank you so much guys for the support.", "votes": null }, { "id": "278539", "postDate": "02/06/2018 12:27:51", "content": "Agree!
", "rawMarkdown": "Agree!", "votes": null } ], "comments": [ { "id": 252427, "author_name": "kubilai", "author_url": "", "post_date": "12/02/2017 22:24:35", "content": "I have the same issue, and it's baffling.
", "votes": null, "replies": [] }, { "id": 252503, "author_name": "liuboli", "author_url": "", "post_date": "12/03/2017 03:52:54", "content": "lol
", "votes": null, "replies": [] }, { "id": 252867, "author_name": "beedata", "author_url": "", "post_date": "12/04/2017 03:16:43", "content": "The more work I do, the more bias I feel the train data has. On average it has more than 40k id (store , item) are missing every day. However, we have no information about these missing value. And in test data, this information is included.
", "votes": null, "replies": [] }, { "id": 252908, "author_name": "ceshine", "author_url": "", "post_date": "12/04/2017 05:51:52", "content": "I have similar experiences. Selecting features and models based on validation scores can be somewhat biased for this dataset (I guess this is natural for a time-based prediction task). Usually I'd lose 0.001 ~ 0.002 expected gain when it happens.
", "votes": null, "replies": [] }, { "id": 253423, "author_name": "npa02012", "author_url": "", "post_date": "12/04/2017 23:08:02", "content": "I've had similar problems when trying to improve the median/mean predictor models.
\nMy XGBoost model does similar on my validation and the LB test set -- but i've been having trouble getting past .56 with it!
Be careful about validation - models that are better on a 2-week period are not necessarily better on the 5-day period that the public leaderboard is based on.
", "votes": null, "replies": [ { "id": 254031, "author_name": "aharless", "author_url": "", "post_date": "12/06/2017 04:32:46", "content": "Seems like the best strategy might be to have two validations, one designed to match the public LB and the other designed to match the private, but otherwise using the same approach. Then you can use the public LB as a check on the validation but still validate for the private LB.
", "votes": null, "replies": [] }, { "id": 254064, "author_name": "ceshine", "author_url": "", "post_date": "12/06/2017 05:46:35", "content": "Yes, I'm been doing exactly that. But the validation score for the first 5 day and the public score can still not match. Does anyone find differently?
", "votes": null, "replies": [] }, { "id": 254070, "author_name": "beedata", "author_url": "", "post_date": "12/06/2017 05:57:03", "content": "I think the problem is because we have a bias train data. The bias is from the missing value and its corresponding promotion information.
", "votes": null, "replies": [] }, { "id": 254260, "author_name": "ceshine", "author_url": "", "post_date": "12/06/2017 15:43:14", "content": "Are you talking about items on promotion but have no unit sold? I agree that missing that information certainly can make the model be too optimistic on items. But since we have no such information, how much mismatch can be attribute to it is unknown.
", "votes": null, "replies": [] }, { "id": 254307, "author_name": "beedata", "author_url": "", "post_date": "12/06/2017 17:45:14", "content": "Exactly
", "votes": null, "replies": [] }, { "id": 254313, "author_name": "seanjv", "author_url": "", "post_date": "12/06/2017 17:56:15", "content": "I also noticed this asymmetry between the train and test sets. If you use onpromotion as a feature, you have to accept that its distribution differs in the train and test sets. If you ignore the feature altogether you remove the asymmetry but also lose an important source of information. For what it's worth, I get about the same scores with and without using onpromotion. Curious if you guys are seeing the same thing?
", "votes": null, "replies": [] }, { "id": 254322, "author_name": "beedata", "author_url": "", "post_date": "12/06/2017 18:07:52", "content": "Yes. Amazing you can get same score with and without using onpromotion.
", "votes": null, "replies": [] }, { "id": 254441, "author_name": "ceshine", "author_url": "", "post_date": "12/07/2017 00:29:21", "content": "Interesting. Haven't actually tried removing onpromotion for a while. Will try it soon and see how it goes. Thanks for the tip.
", "votes": null, "replies": [] }, { "id": 254721, "author_name": "ceshine", "author_url": "", "post_date": "12/07/2017 13:34:11", "content": "It seems removing onpromotion still makes my model converge extremely slowly and I gave up midway. Evidently my models are far less robust as @sjv's.
", "votes": null, "replies": [] }, { "id": 255860, "author_name": "cpmpml", "author_url": "", "post_date": "12/10/2017 11:19:50", "content": "You don't know how long it takes to train @sjv model(s)... I bet he is training on the full train set based on past competitions, and based on the comment about NaN in onpromotion.
", "votes": null, "replies": [] }, { "id": 255868, "author_name": "ceshine", "author_url": "", "post_date": "12/10/2017 12:12:01", "content": "I'd love to hear from anyone that observes the same thing to get some clues of where that robustness came from.
", "votes": null, "replies": [] } ] }, { "id": 254197, "author_name": "cpmpml", "author_url": "", "post_date": "12/06/2017 12:38:27", "content": "I just started on this competition, and when I run the ma script (https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-529) with another train/test split I have a huge difference with the public LB score. Is it just me (which means I screwed Paulo Pinto's code) or did others found the same?
", "votes": null, "replies": [ { "id": 256757, "author_name": "askorn", "author_url": "", "post_date": "12/12/2017 17:02:34", "content": "", "votes": null, "replies": [] }, { "id": 261817, "author_name": "houxudan63", "author_url": "", "post_date": "12/24/2017 04:09:45", "content": "Same problem, difference between scores from local validations (say 2017.8.1~2017.8.15) and PB can be around 0.3. It's rather baffling and confusing. It's my first time join a Kaggle competition so I don't know if it's normal situations in other Kaggle competitions.
", "votes": null, "replies": [] } ] }, { "id": 257501, "author_name": "gmanjunath", "author_url": "", "post_date": "12/14/2017 11:18:13", "content": "Guys, I am seeing similar issue. Basically I am using XGBoost model for this challenge and use rmse as a validation mechanism. Over a period of time, I think the rmse error has dropped all the way to 0.5-0.6 ish however my public score is showing in opposite direction. Much appreciate if someone can help!!
", "votes": null, "replies": [] }, { "id": 257621, "author_name": "npa02012", "author_url": "", "post_date": "12/14/2017 16:38:36", "content": "Is anyone willing to share numbers?
\nOn my XGBoost model, I have LB: .527 --- CV: .557 (7/31 --> 8/15) and .548 (7/31 --> 8/4)
I was originally getting a CV-error of .635 -- I reduced the difference by adding all item/store/date combinations to the test set and assuming 0 unit_sales.
", "votes": null, "replies": [ { "id": 257958, "author_name": "gmanjunath", "author_url": "", "post_date": "12/15/2017 06:42:55", "content": "Even I am using XGBoost mode and I am happy to share the details:
\n\nLatest version of my code was showing:\ntrain-rmse:0.4802 eval-rmse:0.4802, whereas public leaderboard score is 1.297.
\n\nThis is something I don;t understand. How come my public leaderboard score is increasing everytime I make submission. This is my first competition and I think I'm missing something here. Could someone please help in understanding how public leaderboard score is validating.
\n\n@Nicolas I didn;t get about 7/31 --> 8/15 could you please clarify.
\n\nManjy Thanks\nManju
", "votes": null, "replies": [] }, { "id": 257996, "author_name": "askorn", "author_url": "", "post_date": "12/15/2017 08:14:16", "content": "Hello there! I've come to similar results at some point. The thing is how you compose your eval set. From the numbers you provide I guess you take it from the same set as train set, i.e. with maybe some outbursts excluded and so on, but the main thing is that your eval set already contains information on what items were sold on what dates. So that 0.4802 is a misleading number. Try to construct your eval set from all the item nbrs that are there, as well as all dates. And you'll see eval score drop significantly.
", "votes": null, "replies": [] }, { "id": 258004, "author_name": "askorn", "author_url": "", "post_date": "12/15/2017 08:26:08", "content": "And therefore I think there's an overfitting - we do better and better on a training set and losing more and more on test. But I tried quite extensively many scenarios of limitation of item nbrs to actually build predictions for, and it doesn't get better in any significant way. So I'm still puzzled how to come up with decent results with xbg, there's surely something I miss here. Maybe it's still just feature engineering to be done better... with means calculated in various ways as the best features so far.
", "votes": null, "replies": [] }, { "id": 258047, "author_name": "gmanjunath", "author_url": "", "post_date": "12/15/2017 10:36:45", "content": "Thank you Alexander for the response.\nI am using train_test_split method to split the training data.
\n\n*X_train, X_valid = train_test_split(train_items, test_size=0.30, random_state=20)\ndtrain = xgb.DMatrix(X_train[features], y_train)\ndvalid = xgb.DMatrix(X_valid[features], y_valid)*\nThe items_sales is dropped from the X_train and X_valid dataframe.
\n\nAnd in the XGB train, I set evals=watchlist, where watchlist is [(dtrain, 'train'), (dvalid, 'eval')]
\n\nI like your suggestion about constructing eval set with just item_nbrs, date and try to predict and see what happens to rmse score. I had a look at other kernels with LB 0.5ish and they didn't carry out feature engineering, which is interesting.\nRegarding LB score, my understanding is that, once we submit our prediction results, which will compared against the actual number/item_sales in this case. The comparison method used is RMSLE?? The result of RMSLE will decide where I stand in the competition. One important point, I found is that every I submit the prediction, kaggle compares against the 50% of the test set and this set would be randomly selected from the test set.
\n\nAs this is my first competition so bit of struggle, but I really appreciate people are very helpful.
", "votes": null, "replies": [] }, { "id": 258110, "author_name": "npa02012", "author_url": "", "post_date": "12/15/2017 13:58:09", "content": "@gmanjunath 7/31 --> 8/15 refers to the dates that I use in my CV test set.
\nI included the 7/31 --> 8/4 as a CV error check on the first 4 days, as we know that the public leader board is only calculated on 8/16 --> 8/21.
\nFrom your code snippet, it is not completely clear how you are generating your test set - it seems that you may be taking random data-points from train. In my opinion, it is best to create it based on time.
@Alexander check out one of these kernels to see how to build a strong model. They use LGBM, but I was able to easily convert to XGBoost and have similar results.
\nLGBM Starter in R
\nLGBM Starter in Python
@Nicolas Thank you very much for the suggestion and also for sharing the reference code.\nRegarding the test data, yes you are correct I am taking random data-points from the training set.\nI will have a look at your code and modify my code accordingly.
", "votes": null, "replies": [] }, { "id": 262392, "author_name": "gmanjunath", "author_url": "", "post_date": "12/26/2017 12:23:10", "content": "@Nicolas I did modify the code to train the model from 06/01 --> 07/31 and test using 8/01 --> 8/15. As mentioned the validation score was not good so I had to fine tune some of the XGBoost parameters. I am seeing good rmse score using the test dataset, however I am still getting 1.2ish LB score. \nHere is the link to my code\nhttps://www.kaggle.com/gmanjunath/xgboost-predictions\nCould you please cast your eyes to the code and any suggestions would be much appreciated.
\n\nI can see many people are having this issue, it would be good to know the reason for inconsistency in the LB score and the validation score.
", "votes": null, "replies": [] }, { "id": 262524, "author_name": "jacquespeeters", "author_url": "", "post_date": "12/27/2017 01:14:23", "content": "Complete missing values for existing tuples (stores_nbr, items_nbr) with 0 sales. Therefore your algorithm predicts higher value.
\n\nYou probably missed this in R/Python LGBM scripts.
", "votes": null, "replies": [] }, { "id": 262769, "author_name": "gmanjunath", "author_url": "", "post_date": "12/27/2017 22:27:25", "content": "@Jacques, Thank you very much for the suggestion. Much appreciated.\nI had a look at the Python version of LGBM, I can see towards the end of the code, there is code submission = df_test[[\"id\"]].join(df_preds, how=\"left\").fillna(0) which fills 0, this is something I was not doing in my code. So I modified my code by doing this:\nsubmit.loc[:, \"unit_sales\"].fillna(0, inplace=True) just before converting to csv file. I did try with this change, unfortunately no luck.Any pointers please.
", "votes": null, "replies": [] }, { "id": 262948, "author_name": "jacquespeeters", "author_url": "", "post_date": "12/28/2017 12:03:52", "content": "I appreciate the fact you are trying to learn and not blindly copy/pasting public script and tuning hyperparameters like a lot of people are doing. However you'll study Ceshine's script more accurately. Use it, execute it line by line and see what are the input and output. It will help you understand what's going on.
\n\nYes you need to fill by 0 tuples (stores_nbr, items_nbr) not present in train set. However in your training sets you only have tuples (stores_nbr, items_nbr, dates) where sales are positives. You need to \"complete/expand\" missing values of date for tuples (stores_nbr, items_nbr) with 0 sales and onpromotion.
\n\nCeshine does it by using \"unstack\" function and using (stores_nbr, items_nbr) as index.
\n\nHave a look at this kernel to better understand what's going on. \nhttps://www.kaggle.com/aharless/dissecting-ceshine-lee-s-lgbm-kernel
", "votes": null, "replies": [] }, { "id": 263010, "author_name": "npa02012", "author_url": "", "post_date": "12/28/2017 17:03:45", "content": "@gmanjunath - I took a look at your code. I think your biggest problem is your lack of features. Remember that XGBoost does NOT take categorical variables. So XGBoost will not decipher item_nbr, store_nbr, nor family as you might hope.
\n\nThe day/month/year, perishable and onpromotion are your other features and are good, but not nearly enough to determine unit_sales.
\n\nTry adding 'median_sales' to each item/store combination as a predictor variable. This should give your model a lot more predictive power (probably ~ LB .65).
\n\nI would say do this first to gain some momentum. Afterwards, as Jacques Peeters mentioned, decipher and run Ceshine's code until you understand it both line-by-line and conceptually. The structure is more complex then what you have done, but also more powerful/faster.
\n\nGood luck!
", "votes": null, "replies": [] }, { "id": 263073, "author_name": "gmanjunath", "author_url": "", "post_date": "12/28/2017 21:26:48", "content": "@Jacques and @Nicolas thank you very much for taking time to review my code. I really appreciate your suggestions. \nI will modify the code as per your suggestions and give it a try.
", "votes": null, "replies": [] }, { "id": 266824, "author_name": "gmanjunath", "author_url": "", "post_date": "01/09/2018 22:07:12", "content": "@Jacques and @Nicolas, I managed to crack the prediction results (0.53). As you guys pointed out, I was not filling up missing values for (stores/item nbr), which I am doing now using the pivot table. I must admit that my code is based on Ceshine's code, however this was done to gain the momentum. Next I am planning to play around the training data by picking up the data for the month of Aug from previous year and see how much it will make difference.\nOnce again thank you so much guys for the support.
", "votes": null, "replies": [] } ] }, { "id": 262157, "author_name": "wuzuping", "author_url": "", "post_date": "12/25/2017 15:53:25", "content": "So Is there a good validation strategy up to now~ I think I possibly overfit public LB~
", "votes": null, "replies": [] }, { "id": 262166, "author_name": "kelexu", "author_url": "", "post_date": "12/25/2017 16:36:18", "content": "As long as the LB-CV=constant, I suppose the validation can be trusted.
", "votes": null, "replies": [ { "id": 278539, "author_name": "longyin2", "author_url": "", "post_date": "02/06/2018 12:27:51", "content": "Agree!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "252329": "I use several validation set (different time period) to validate my feature. All of them have quite robust improvement. But the public score drops after including those feature. I'm quite sure I don't have leakage when I train and validate the model. Is there anyone having the same problem?", "252427": "I have the same issue, and it's baffling.", "252503": "lol", "252867": "The more work I do, the more bias I feel the train data has. On average it has more than 40k id (store , item) are missing every day. However, we have no information about these missing value. And in test data, this information is included.", "252908": "I have similar experiences. Selecting features and models based on validation scores can be somewhat biased for this dataset (I guess this is natural for a time-based prediction task). Usually I'd lose 0.001 ~ 0.002 expected gain when it happens.", "253423": "I've had similar problems when trying to improve the median/mean predictor models. \nMy XGBoost model does similar on my validation and the LB test set -- but i've been having trouble getting past .56 with it!", "253764": "Be careful about validation - models that are better on a 2-week period are not necessarily better on the 5-day period that the public leaderboard is based on.", "254031": "Seems like the best strategy might be to have two validations, one designed to match the public LB and the other designed to match the private, but otherwise using the same approach. Then you can use the public LB as a check on the validation but still validate for the private LB.", "254064": "Yes, I'm been doing exactly that. But the validation score for the first 5 day and the public score can still not match. Does anyone find differently?", "254070": "I think the problem is because we have a bias train data. The bias is from the missing value and its corresponding promotion information.", "254197": "I just started on this competition, and when I run the ma script (https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-529) with another train/test split I have a huge difference with the public LB score. Is it just me (which means I screwed Paulo Pinto's code) or did others found the same?", "254260": "Are you talking about items on promotion but have no unit sold? I agree that missing that information certainly can make the model be too optimistic on items. But since we have no such information, how much mismatch can be attribute to it is unknown.", "254307": "Exactly", "254313": "I also noticed this asymmetry between the train and test sets. If you use onpromotion as a feature, you have to accept that its distribution differs in the train and test sets. If you ignore the feature altogether you remove the asymmetry but also lose an important source of information. For what it's worth, I get about the same scores with and without using onpromotion. Curious if you guys are seeing the same thing?", "254322": "Yes. Amazing you can get same score with and without using onpromotion.", "254441": "Interesting. Haven't actually tried removing onpromotion for a while. Will try it soon and see how it goes. Thanks for the tip.", "254721": "It seems removing onpromotion still makes my model converge extremely slowly and I gave up midway. Evidently my models are far less robust as @sjv's.", "255860": "You don't know how long it takes to train @sjv model(s)... I bet he is training on the full train set based on past competitions, and based on the comment about NaN in onpromotion.", "255868": "I'd love to hear from anyone that observes the same thing to get some clues of where that robustness came from.", "256757": "", "257501": "Guys, I am seeing similar issue. Basically I am using XGBoost model for this challenge and use rmse as a validation mechanism. Over a period of time, I think the rmse error has dropped all the way to 0.5-0.6 ish however my public score is showing in opposite direction. Much appreciate if someone can help!!", "257621": "Is anyone willing to share numbers? \nOn my XGBoost model, I have LB: .527 --- CV: .557 (7/31 --> 8/15) and .548 (7/31 --> 8/4)\n \nI was originally getting a CV-error of .635 -- I reduced the difference by adding all item/store/date combinations to the test set and assuming 0 unit_sales.", "257958": "Even I am using XGBoost mode and I am happy to share the details:\n\nLatest version of my code was showing:\n**train-rmse:0.4802\teval-rmse:0.4802**, whereas public leaderboard score is **1.297**.\n\nThis is something I don;t understand. How come my public leaderboard score is increasing everytime I make submission. This is my first competition and I think I'm missing something here. Could someone please help in understanding how public leaderboard score is validating.\n\n@Nicolas I didn;t get about 7/31 --> 8/15 could you please clarify.\n\nManjy Thanks\nManju", "257996": "Hello there! I've come to similar results at some point. The thing is how you compose your eval set. From the numbers you provide I guess you take it from the same set as train set, i.e. with maybe some outbursts excluded and so on, but the main thing is that your eval set already contains information on what items were sold on what dates. So that 0.4802 is a misleading number. Try to construct your eval set from all the item nbrs that are there, as well as all dates. And you'll see eval score drop significantly.", "258004": "And therefore I think there's an overfitting - we do better and better on a training set and losing more and more on test. But I tried quite extensively many scenarios of limitation of item nbrs to actually build predictions for, and it doesn't get better in any significant way. So I'm still puzzled how to come up with decent results with xbg, there's surely something I miss here. Maybe it's still just feature engineering to be done better... with means calculated in various ways as the best features so far.", "258047": "Thank you Alexander for the response.\nI am using train_test_split method to split the training data. \n\n*X_train, X_valid = train_test_split(train_items, test_size=0.30, random_state=20)\ndtrain = xgb.DMatrix(X_train[features], y_train)\ndvalid = xgb.DMatrix(X_valid[features], y_valid)*\nThe items_sales is dropped from the X_train and X_valid dataframe.\n\nAnd in the XGB train, I set evals=watchlist, where watchlist is [(dtrain, 'train'), (dvalid, 'eval')]\n\n\nI like your suggestion about constructing eval set with just item_nbrs, date and try to predict and see what happens to rmse score. I had a look at other kernels with LB 0.5ish and they didn't carry out feature engineering, which is interesting.\nRegarding LB score, my understanding is that, once we submit our prediction results, which will compared against the actual number/item_sales in this case. The comparison method used is RMSLE?? The result of RMSLE will decide where I stand in the competition. One important point, I found is that every I submit the prediction, kaggle compares against the 50% of the test set and this set would be randomly selected from the test set.\n\nAs this is my first competition so bit of struggle, but I really appreciate people are very helpful.", "258110": "gmanjunath 7/31 --> 8/15 refers to the dates that I use in my CV test set. \nI included the 7/31 --> 8/4 as a CV error check on the first 4 days, as we know that the public leader board is only calculated on 8/16 --> 8/21. \nFrom your code snippet, it is not completely clear how you are generating your test set - it seems that you may be taking random data-points from train. In my opinion, it is best to create it based on time. \n \n@Alexander check out one of these kernels to see how to build a strong model. They use LGBM, but I was able to easily convert to XGBoost and have similar results. \n[LGBM Starter in R][1] \n[LGBM Starter in Python][2]\n\n\n [1]: https://www.kaggle.com/npa02012/ceshine-s-lgbm-starter-in-r-lb-0-529\n [2]: https://www.kaggle.com/ceshine/lgbm-starter", "258229": "Nicolas Thank you very much for the suggestion and also for sharing the reference code.\nRegarding the test data, yes you are correct I am taking random data-points from the training set.\nI will have a look at your code and modify my code accordingly.", "261817": "Same problem, difference between scores from local validations (say 2017.8.1~2017.8.15) and PB can be around 0.3. It's rather baffling and confusing. It's my first time join a Kaggle competition so I don't know if it's normal situations in other Kaggle competitions.", "262157": "So Is there a good validation strategy up to now~ I think I possibly overfit public LB~", "262166": "As long as the LB-CV=constant, I suppose the validation can be trusted.", "262392": "Nicolas I did modify the code to train the model from 06/01 --> 07/31 and test using 8/01 --> 8/15. As mentioned the validation score was not good so I had to fine tune some of the XGBoost parameters. I am seeing good rmse score using the test dataset, however I am still getting 1.2ish LB score. \nHere is the link to my code\nhttps://www.kaggle.com/gmanjunath/xgboost-predictions\nCould you please cast your eyes to the code and any suggestions would be much appreciated.\n\nI can see many people are having this issue, it would be good to know the reason for inconsistency in the LB score and the validation score.", "262524": "Complete missing values for existing tuples (stores_nbr, items_nbr) with 0 sales. Therefore your algorithm predicts higher value.\n\nYou probably missed this in R/Python LGBM scripts.", "262769": "Jacques, Thank you very much for the suggestion. Much appreciated.\nI had a look at the Python version of LGBM, I can see towards the end of the code, there is code **submission = df_test[[\"id\"]].join(df_preds, how=\"left\").fillna(0)** which fills 0, this is something I was not doing in my code. So I modified my code by doing this:\n**submit.loc[:, \"unit_sales\"].fillna(0, inplace=True)** just before converting to csv file. I did try with this change, unfortunately no luck.Any pointers please.", "262948": "I appreciate the fact you are trying to learn and not blindly copy/pasting public script and tuning hyperparameters like a lot of people are doing. However you'll study Ceshine's script more accurately. Use it, execute it line by line and see what are the input and output. It will help you understand what's going on.\n\nYes you need to fill by 0 tuples (stores_nbr, items_nbr) not present in train set. However in your training sets you only have tuples (stores_nbr, items_nbr, dates) where sales are positives. You need to \"complete/expand\" missing values of date for tuples (stores_nbr, items_nbr) with 0 sales and onpromotion. \n\nCeshine does it by using \"unstack\" function and using (stores_nbr, items_nbr) as index. \n\nHave a look at this kernel to better understand what's going on. \nhttps://www.kaggle.com/aharless/dissecting-ceshine-lee-s-lgbm-kernel", "263010": "gmanjunath - I took a look at your code. I think your biggest problem is your lack of features. Remember that XGBoost does NOT take categorical variables. So XGBoost will not decipher item_nbr, store_nbr, nor family as you might hope. \n \nThe day/month/year, perishable and onpromotion are your other features and are good, but not nearly enough to determine unit_sales. \n \nTry adding 'median_sales' to each item/store combination as a predictor variable. This should give your model a lot more predictive power (probably ~ LB .65). \n \nI would say do this first to gain some momentum. Afterwards, as Jacques Peeters mentioned, decipher and run Ceshine's code until you understand it both line-by-line and conceptually. The structure is more complex then what you have done, but also more powerful/faster. \n \nGood luck!", "263073": "Jacques and @Nicolas thank you very much for taking time to review my code. I really appreciate your suggestions. \nI will modify the code as per your suggestions and give it a try.", "266824": "Jacques and @Nicolas, I managed to crack the prediction results (0.53). As you guys pointed out, I was not filling up missing values for (stores/item nbr), which I am doing now using the pivot table. I must admit that my code is based on Ceshine's code, however this was done to gain the momentum. Next I am planning to play around the training data by picking up the data for the month of Aug from previous year and see how much it will make difference.\nOnce again thank you so much guys for the support.", "278539": "Agree!" }, "source": "meta" }