{ "id": 22758, "title": "Expecting a Huge Shake-up", "url": "/competitions/grupo-bimbo-inventory-demand/discussion/22758", "author_name": "", "post_date": "2016-08-08T15:12:56.467Z", "votes": 1, "comment_count": 66, "views": 8205, "content": "

I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!

", "messages": [ { "id": "130565", "postDate": "08/08/2016 15:12:56", "content": "

I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!

", "rawMarkdown": "I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!", "votes": null }, { "id": "130569", "postDate": "08/08/2016 16:40:41", "content": "

[quote=NimaShahbazi;130565]

\n\n

I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!

\n\n

[/quote]\nwow, we didn't use any public scripts. Are we safe?

", "rawMarkdown": "[quote=NimaShahbazi;130565]\r\n\r\nI am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!\r\n\r\n[/quote]\r\nwow, we didn't use any public scripts. Are we safe?", "votes": null }, { "id": "130570", "postDate": "08/08/2016 16:46:36", "content": "

@rcarson: Well, I don't know, it depends on the features that you have used! Hope not! you are good at not overfitting buddy :)

", "rawMarkdown": "rcarson: Well, I don't know, it depends on the features that you have used! Hope not! you are good at not overfitting buddy :)", "votes": null }, { "id": "130571", "postDate": "08/08/2016 16:54:55", "content": "

With the public log-means scripts (mine included) the two week gap has fewer of the easiest types to predict, so they will probably jump up by ~.01. My coefficients were calculated using a gap week, so they might hold up a bit better - but they will all be slaughtered by an xgboost solution that avoids a should-be-obvious-but-isn't pitfall...

\n\n

edit: I expect some/all of the top 10'ers to figure out how to avoid the pitfall I'm thinking of. But much of the top 10% is going to take a hit. The drop range is likely going to be between .007 and .02, depending on what you're doing...

", "rawMarkdown": "With the public log-means scripts (mine included) the two week gap has fewer of the easiest types to predict, so they will probably jump up by ~.01. My coefficients were calculated using a gap week, so they might hold up a bit better - but they will all be slaughtered by an xgboost solution that avoids a should-be-obvious-but-isn't pitfall...\r\n\r\nedit: I expect some/all of the top 10'ers to figure out how to avoid the pitfall I'm thinking of. But much of the top 10% is going to take a hit. The drop range is likely going to be between .007 and .02, depending on what you're doing...", "votes": null }, { "id": "130573", "postDate": "08/08/2016 16:55:41", "content": "

Thank you for letting us know. I guess the risky feature might be the target rate encoding but we will review our validation process.

", "rawMarkdown": "Thank you for letting us know. I guess the risky feature might be the target rate encoding but we will review our validation process.", "votes": null }, { "id": "131250", "postDate": "08/15/2016 00:38:04", "content": "

I did many cross validation. However due to small RAM I could only use 2 weeks of data per fitting.

\n\n

My XGBoost model uses lagged demand features, I guess this is the main factor.

\n\n

What I found: the closer evaluation's week to train's weeks, the better the CV RMSLE.

\n\n

For example,

\n\n\n\n

Week 5 and 8 has better score, they're closer to 6,7. \nWhile week 9 has worse score. \nThe same pattern in all other week pairs, 5+6, 5+8, .., 6+8, 6+9, .., 8+9.

\n\n

Test data:

\n\n

Week 10 = public LB, closer to training data by week. Should have better score.

\n\n

Week 11 = private LB, has far 2+ week distance from training data. Worse score ;)

\n\n

So, I'm expecting worse score by around 0.005+ in the private LB, maybe a rank drop by ~50 ;))

", "rawMarkdown": "I did many cross validation. However due to small RAM I could only use 2 weeks of data per fitting. \r\n\r\nMy XGBoost model uses lagged demand features, I guess this is the main factor. \r\n\r\nWhat I found: the closer evaluation's week to train's weeks, the better the CV RMSLE. \r\n\r\nFor example, \r\n\r\n - train week 6,7 ; eval week 5 ; rmsle 0.444189828967\r\n \r\n - train week 6,7 ; eval week 8 ; rmsle 0.444531862635 \r\n \r\n - train week 6,7 ; eval week 9 ; rmsle 0.450169899702\r\n\r\nWeek 5 and 8 has better score, they're closer to 6,7. \r\nWhile week 9 has worse score. \r\nThe same pattern in all other week pairs, 5+6, 5+8, .., 6+8, 6+9, .., 8+9. \r\n\r\nTest data: \r\n\r\nWeek 10 = public LB, closer to training data by week. Should have better score. \r\n \r\nWeek 11 = private LB, has far 2+ week distance from training data. Worse score ;) \r\n \r\nSo, I'm expecting worse score by around 0.005+ in the private LB, maybe a rank drop by ~50 ;))", "votes": null }, { "id": "131253", "postDate": "08/15/2016 02:09:34", "content": "

aldente - .005 sounds reasonable and should hold up on the private LB, if you're setting up week 9's processing correctly.

", "rawMarkdown": "aldente - .005 sounds reasonable and should hold up on the private LB, if you're setting up week 9's processing correctly.", "votes": null }, { "id": "131257", "postDate": "08/15/2016 04:22:01", "content": "

@happycube, hopefully ;)

\n\n

You're correct, looks like most important is weeks closer to test data.

", "rawMarkdown": "happycube, hopefully ;) \r\n \r\nYou're correct, looks like most important is weeks closer to test data.", "votes": null }, { "id": "131294", "postDate": "08/15/2016 15:03:55", "content": "

@happycube, may I ask what you mean by "processing week 9 correctly"? Thank you!

", "rawMarkdown": "happycube, may I ask what you mean by \"processing week 9 correctly\"? Thank you!", "votes": null }, { "id": "131300", "postDate": "08/15/2016 16:29:29", "content": "

@happycube\nI also wonder.\nMay I ask what you mean by "processing week 9 correctly"? \nThank you!\n:)))

", "rawMarkdown": "happycube\r\nI also wonder.\r\nMay I ask what you mean by \"processing week 9 correctly\"? \r\nThank you!\r\n:)))", "votes": null }, { "id": "131301", "postDate": "08/15/2016 16:31:32", "content": "

Not using any information from week 8...

", "rawMarkdown": "Not using any information from week 8...", "votes": null }, { "id": "131346", "postDate": "08/16/2016 07:47:42", "content": "

Thank you!

", "rawMarkdown": "Thank you!", "votes": null }, { "id": "131382", "postDate": "08/16/2016 16:38:31", "content": "

@rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P

", "rawMarkdown": "rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P", "votes": null }, { "id": "131520", "postDate": "08/17/2016 21:29:18", "content": "

What is the point of this post? We all know that PB is using week 10 when private LB is using week 11. Either you disclose what you think is making a difference, or you use it at your advantage.

", "rawMarkdown": "What is the point of this post? We all know that PB is using week 10 when private LB is using week 11. Either you disclose what you think is making a difference, or you use it at your advantage.", "votes": null }, { "id": "131540", "postDate": "08/18/2016 01:10:09", "content": "

@CPMP, if you read my post carefully, I discussed the "reason" why I think I'll have worse private LB score (rank drop), because my XGB model uses lagged demand features which I suspect to hurt week 10/11 split. Based on my cross validation results.

\n\n

People using the same lagged demand features can reproduce it easily, to prove that the behavior of biweekly cross validation results I shared is the truth.

\n\n

(... deleted for misunderstanding, my apologies)

", "rawMarkdown": "CPMP, if you read my post carefully, I discussed the \"reason\" why I think I'll have worse private LB score (rank drop), because my XGB model uses lagged demand features which I suspect to hurt week 10/11 split. Based on my cross validation results.\r\n\r\nPeople using the same lagged demand features can reproduce it easily, to prove that the behavior of biweekly cross validation results I shared is the truth. \r\n\r\n(... deleted for misunderstanding, my apologies)", "votes": null }, { "id": "131553", "postDate": "08/18/2016 03:25:56", "content": "

[quote=Davut Polat;131382]

\n\n

@rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P

\n\n

[/quote]

\n\n

Hi, Davut Polat! Glad to see you.

\n\n

We'll wait :)))

", "rawMarkdown": "[quote=Davut Polat;131382]\r\n\r\n@rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P \r\n\r\n\r\n[/quote]\r\n\r\nHi, Davut Polat! Glad to see you.\r\n\r\nWe'll wait :)))", "votes": null }, { "id": "131565", "postDate": "08/18/2016 07:58:24", "content": "

@aldente I think CPMP was talking about this post of NimaShahbazi as it doesn't make any sense. He didn't tell why there would be a huge shake-up, but just posted that there would be a shake-up which I think most of us know.

", "rawMarkdown": "aldente I think CPMP was talking about this post of NimaShahbazi as it doesn't make any sense. He didn't tell why there would be a huge shake-up, but just posted that there would be a shake-up which I think most of us know.", "votes": null }, { "id": "131566", "postDate": "08/18/2016 08:07:44", "content": "

All, yes, I am asking about the post that started the thread. I may have overreacted because I am a bit nervous (I have a dental abscess ...) Apologies to whomever finds my question incorrect.

\n\n

@aldente, your post is informative and useful for those who did not realize that we have a 2 week ahead forecast to do, not a one week ahead forecast. Same comment applies to @happycube answer on processing week 9 correctly.

", "rawMarkdown": "All, yes, I am asking about the post that started the thread. I may have overreacted because I am a bit nervous (I have a dental abscess ...) Apologies to whomever finds my question incorrect.\r\n\r\n@aldente, your post is informative and useful for those who did not realize that we have a 2 week ahead forecast to do, not a one week ahead forecast. Same comment applies to @happycube answer on processing week 9 correctly.", "votes": null }, { "id": "131596", "postDate": "08/18/2016 13:47:39", "content": "

I think the "huge" shake-up happens in the middle of the competition who uses mean-value approach and ignores the time indicator.

\n\n

But Davut gave us a hope to jump to the top thanks to the shake-up ;-)

", "rawMarkdown": "I think the \"huge\" shake-up happens in the middle of the competition who uses mean-value approach and ignores the time indicator. \r\n\r\nBut Davut gave us a hope to jump to the top thanks to the shake-up ;-)", "votes": null }, { "id": "131598", "postDate": "08/18/2016 13:51:58", "content": "

@Taro, there is always a hope to jump to the top :) I think forum posts are enough for a good score\nListen this song carefully :)\nhttps://www.youtube.com/watch?v=DQWI1kvmwRg

", "rawMarkdown": "Taro, there is always a hope to jump to the top :) I think forum posts are enough for a good score\r\nListen this song carefully :)\r\nhttps://www.youtube.com/watch?v=DQWI1kvmwRg", "votes": null }, { "id": "131616", "postDate": "08/18/2016 16:19:45", "content": "

@Davut That video is great!

", "rawMarkdown": "Davut That video is great!", "votes": null }, { "id": "131640", "postDate": "08/18/2016 20:13:21", "content": "

@Abhilash, @CPMP, my apologies for misunderstanding, I edited my post :) .. I supposed to ask first who are intended, my fault ;)) \n+1

", "rawMarkdown": "Abhilash, @CPMP, my apologies for misunderstanding, I edited my post :) .. I supposed to ask first who are intended, my fault ;)) \r\n+1", "votes": null }, { "id": "131644", "postDate": "08/18/2016 20:47:08", "content": "

@CPMP @AbhilashAwasthi, \n1- As you can notice some good discussions for the shakeup is done "after" I posted this thread!\n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules! \n3- Ofcourse I have my own reasons to post this thread. I was looking for the ideas, if someone likes to share! (My own strategy to see what others think, and maybe contribute if I like to).\n4- a "Single Thread" for shakeup makes everyone aware! Okay you (CPMP) and your friend (Ablihash) know about it, good for you, take advantage.\n5- if you don't find this informative, I am sorry that I waste your time with this post!\n6- I ranked 2nd in Rossman competition (there was a huge shakeup there too, and it was a pretty same competition), so "maybe" when I pointing out a huge shakeup it has more strength.\n7- thanks to @aldente @happycube for the info and @Davut for the video!

", "rawMarkdown": "CPMP @AbhilashAwasthi, \r\n1- As you can notice some good discussions for the shakeup is done \"after\" I posted this thread!\r\n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules! \r\n3- Ofcourse I have my own reasons to post this thread. I was looking for the ideas, if someone likes to share! (My own strategy to see what others think, and maybe contribute if I like to).\r\n4- a \"Single Thread\" for shakeup makes everyone aware! Okay you (CPMP) and your friend (Ablihash) know about it, good for you, take advantage.\r\n5- if you don't find this informative, I am sorry that I waste your time with this post!\r\n6- I ranked 2nd in Rossman competition (there was a huge shakeup there too, and it was a pretty same competition), so \"maybe\" when I pointing out a huge shakeup it has more strength.\r\n7- thanks to @aldente @happycube for the info and @Davut for the video!", "votes": null }, { "id": "131646", "postDate": "08/18/2016 21:04:15", "content": "

btw, kaggle a friendly community we all make good friends here, for example when @davut says I know what makes you .42x wait for me and the other guy says we are waiting! (is it informative?). we are all friends here and share what we think and what we "like" to! and there is no informative filter on posting threads (although I have my own reasons for this post).

", "rawMarkdown": "btw, kaggle a friendly community we all make good friends here, for example when @davut says I know what makes you .42x wait for me and the other guy says we are waiting! (is it informative?). we are all friends here and share what we think and what we \"like\" to! and there is no informative filter on posting threads (although I have my own reasons for this post).", "votes": null }, { "id": "131662", "postDate": "08/18/2016 22:28:27", "content": "

I am new and this is my first competition, currently 70 on LB. I have one question for the big guns here: Do u ensemble different models to get better results (on private LB) or its just one model ? If you don't want to answer, don't :]

", "rawMarkdown": "I am new and this is my first competition, currently 70 on LB. I have one question for the big guns here: Do u ensemble different models to get better results (on private LB) or its just one model ? If you don't want to answer, don't :]", "votes": null }, { "id": "131666", "postDate": "08/18/2016 22:46:31", "content": "

@salenator I got 0.437xx public LB without ensembling, and I think I'll get better soon. There is hope therefore ;)

", "rawMarkdown": "salenator I got 0.437xx public LB without ensembling, and I think I'll get better soon. There is hope therefore ;)", "votes": null }, { "id": "131667", "postDate": "08/18/2016 22:53:16", "content": "

@NimaShahbazi, I apologized, see above.

\n\n

You wrote: \n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules!

\n\n

This applies to me as well: I can express my thoughts same as you.

", "rawMarkdown": "NimaShahbazi, I apologized, see above.\r\n\r\nYou wrote: \r\n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules!\r\n\r\nThis applies to me as well: I can express my thoughts same as you.", "votes": null }, { "id": "131912", "postDate": "08/20/2016 14:48:39", "content": "

Dear Kagglers, I am rookie here and I want to ask a tiny humble question regarding this thrilling and scaring overfitting.

\n\n

I have worse metric on CV than on LB. Can I feel myself safe? My CV strategy is to use first 5 known weeks as train and two last known weeks as test.

\n\n

Many thanks for sharing your experience!

\n\n

SC

\n\n

PS Song is great and funny, but I am scared even more now.

", "rawMarkdown": "Dear Kagglers, I am rookie here and I want to ask a tiny humble question regarding this thrilling and scaring overfitting. \r\n\r\nI have worse metric on CV than on LB. Can I feel myself safe? My CV strategy is to use first 5 known weeks as train and two last known weeks as test.\r\n\r\nMany thanks for sharing your experience!\r\n\r\nSC\r\n\r\nPS Song is great and funny, but I am scared even more now.", "votes": null }, { "id": "131917", "postDate": "08/20/2016 15:38:16", "content": "

Sergey, you should not be scared but instead try to adjust your solution.\nYour validation strategy seems fine to me, although you might want to notice that leaderboard is split by time, which means that the actual target of the competition is to predict horison 2. \nA better validation strategy can be training on first 6 weeks validate on the eighth week (public leader board), then add the additional seventh week info to generate the prediction (by same model) for the nineth week (private leader board)

\n\n

Good luck

\n\n

Nati

", "rawMarkdown": "Sergey, you should not be scared but instead try to adjust your solution.\r\nYour validation strategy seems fine to me, although you might want to notice that leaderboard is split by time, which means that the actual target of the competition is to predict horison 2. \r\nA better validation strategy can be training on first 6 weeks validate on the eighth week (public leader board), then add the additional seventh week info to generate the prediction (by same model) for the nineth week (private leader board)\r\n\r\nGood luck\r\n\r\nNati", "votes": null }, { "id": "131922", "postDate": "08/20/2016 16:28:20", "content": "

@Nathaniel, thank you for advice! I will try for sure as soon as my machine will finish parameters training. But what do you think, usually, according to you experience and knowledge - does better metric on CV leads to overfitting and worse to under fitting? Or this is independent?

\n\n

SC

", "rawMarkdown": "Nathaniel, thank you for advice! I will try for sure as soon as my machine will finish parameters training. But what do you think, usually, according to you experience and knowledge - does better metric on CV leads to overfitting and worse to under fitting? Or this is independent?\r\n\r\nSC", "votes": null }, { "id": "131923", "postDate": "08/20/2016 16:39:47", "content": "

I made a habit of selecting my two submissions one by best lb score and the other by best CV score.\nIn general if CV method is robust you're better select your submission by CV.\nThe CV method i described above combines the two, as you can use the public leader board score as additional information.

", "rawMarkdown": "I made a habit of selecting my two submissions one by best lb score and the other by best CV score.\r\nIn general if CV method is robust you're better select your submission by CV.\r\nThe CV method i described above combines the two, as you can use the public leader board score as additional information.", "votes": null }, { "id": "132611", "postDate": "08/27/2016 13:33:00", "content": "

Hi everybody,

\n\n

Right now the competition is close to the end so let's see if your guess on the leaderboard shakeup is right or not ;-) You can anonymously vote at http://goo.gl/Jjkxjh and see the results at http://goo.gl/ip0bSV. Personally I have enjoyed the journey so far and this poll might be fun as well.

\n\n

Here is the screenshot of the current leaderboard \"current.

\n\n

PS: I have just corrected the links, please use the links here not the ones in your email notification! Many thanks!

", "rawMarkdown": "Hi everybody,\r\n\r\nRight now the competition is close to the end so let's see if your guess on the leaderboard shakeup is right or not ;-) You can anonymously vote at [http://goo.gl/Jjkxjh][1] and see the results at [http://goo.gl/ip0bSV][2]. Personally I have enjoyed the journey so far and this poll might be fun as well.\r\n\r\nHere is the screenshot of the current leaderboard ![current leaderboard][3].\r\n\r\nPS: I have just corrected the links, please use the links here not the ones in your email notification! Many thanks!\r\n\r\n\r\n [1]: http://goo.gl/Jjkxjh\r\n [2]: http://goo.gl/ip0bSV\r\n [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/132611/4740/Screen%20Shot%202016-08-27%20at%2015.26.51.png?sv=2012-02-12&se=2016-08-30T13%3A34%3A41Z&sr=b&sp=r&sig=DB6kbZcVWkIp8BTWV7ymSVZSZ%2BiWZfU5GtfI%2BjIJeqs%3D", "votes": null }, { "id": "132619", "postDate": "08/27/2016 14:16:49", "content": "

@Tien-Dung Le, I voted, but I feel that this poll doesn't represent the overall board shake up, since people at the top are usually experts, so it's less likely that they are overfitting badly to the public LB. On the other hand, I do expect major shakeup around the XGB public script score, which probably encompasses most of the field.

", "rawMarkdown": "Tien-Dung Le, I voted, but I feel that this poll doesn't represent the overall board shake up, since people at the top are usually experts, so it's less likely that they are overfitting badly to the public LB. On the other hand, I do expect major shakeup around the XGB public script score, which probably encompasses most of the field.", "votes": null }, { "id": "132637", "postDate": "08/27/2016 20:14:52", "content": "

@Caio, thanks for your vote. I do agree with you about "this poll doesn't represent the overall board shake up". The reason I use "top 10" because it is much easy to measure - by just looking and it is more interesting.

\n\n

"people at the top are usually experts" => So you expect the leaderboard is stable, probably like this one https://www.kaggle.com/c/avito-duplicate-ads-detection/leaderboard/private. However, don't forget that there exists some huge shake-ups like this https://www.kaggle.com/c/restaurant-revenue-prediction/leaderboard/private

\n\n

Let's see what other people vote!

", "rawMarkdown": "Caio, thanks for your vote. I do agree with you about \"this poll doesn't represent the overall board shake up\". The reason I use \"top 10\" because it is much easy to measure - by just looking and it is more interesting.\r\n\r\n\"people at the top are usually experts\" => So you expect the leaderboard is stable, probably like this one [https://www.kaggle.com/c/avito-duplicate-ads-detection/leaderboard/private][1]. However, don't forget that there exists some huge shake-ups like this [https://www.kaggle.com/c/restaurant-revenue-prediction/leaderboard/private][2]\r\n\r\nLet's see what other people vote! \r\n\r\n\r\n [1]: https://www.kaggle.com/c/avito-duplicate-ads-detection/leaderboard/private\r\n [2]: https://www.kaggle.com/c/restaurant-revenue-prediction/leaderboard/private", "votes": null }, { "id": "132684", "postDate": "08/28/2016 12:21:40", "content": "

@Sergey Cherkasov

\n\n

What matters really is if your LB value increases when your local CV value increases. if this is the case, then you can focus on your local CV value.

\n\n

But, as was said, issue is that the private LB is a 2 week horizon forecast, hence your local CV setting must reflect that.

", "rawMarkdown": "Sergey Cherkasov\r\n\r\nWhat matters really is if your LB value increases when your local CV value increases. if this is the case, then you can focus on your local CV value.\r\n\r\nBut, as was said, issue is that the private LB is a 2 week horizon forecast, hence your local CV setting must reflect that.", "votes": null }, { "id": "132744", "postDate": "08/28/2016 23:37:52", "content": "

For myself, I'm doing what I can to hold up on the private LB - my public LB score isn't great, but it loses less than .005 by removing any week 9 data at least. So hopefully it'll hold up OK for week 11...

\n\n

... and if not, I'll grumble and figure out WTF happened... ;)

", "rawMarkdown": "For myself, I'm doing what I can to hold up on the private LB - my public LB score isn't great, but it loses less than .005 by removing any week 9 data at least. So hopefully it'll hold up OK for week 11...\r\n\r\n... and if not, I'll grumble and figure out WTF happened... ;)", "votes": null }, { "id": "132759", "postDate": "08/29/2016 04:14:20", "content": "

@happycube, same here.

", "rawMarkdown": "happycube, same here.", "votes": null }, { "id": "132987", "postDate": "08/30/2016 18:07:43", "content": "

Another data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!

\n\n

So what does it say that a team capable of being #1 isn't even in the top 3% right now...?

", "rawMarkdown": "Another data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!\r\n\r\nSo what does it say that a team capable of being #1 isn't even in the top 3% right now...?", "votes": null }, { "id": "132997", "postDate": "08/30/2016 18:53:23", "content": "

[quote=happycube;132987]

\n\n

Another data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!

\n\n

So what does it say that a team capable of being #1 isn't even in the top 3% right now...?

\n\n

[/quote]

\n\n

We haven't update our LB score for more than 45 days until today. But we are making some progress along the way. Sticking to the highest possible LB score isn't that useful.

\n\n

Jianbo

", "rawMarkdown": "[quote=happycube;132987]\r\n\r\nAnother data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!\r\n\r\nSo what does it say that a team capable of being #1 isn't even in the top 3% right now...?\r\n\r\n[/quote]\r\n\r\nWe haven't update our LB score for more than 45 days until today. But we are making some progress along the way. Sticking to the highest possible LB score isn't that useful.\r\n\r\nJianbo", "votes": null }, { "id": "133002", "postDate": "08/30/2016 19:05:38", "content": "

Agreed - but they've submitted in the last day, before I made my post. They're definitely not angling for a public LB score, and I personally expect them to wind up in the top ten tonight.

\n\n

In this compo the public LB is at best a type of non-local validation, just another week to compare against occasionally and make sure the submission isn't garbage for some reason ;)

", "rawMarkdown": "Agreed - but they've submitted in the last day, before I made my post. They're definitely not angling for a public LB score, and I personally expect them to wind up in the top ten tonight.\r\n\r\nIn this compo the public LB is at best a type of non-local validation, just another week to compare against occasionally and make sure the submission isn't garbage for some reason ;)", "votes": null }, { "id": "133011", "postDate": "08/30/2016 21:25:13", "content": "

@happycube, I didn't notice Kazanova and Gert are also in this competition. Hmmm, this makes it interesting.

\n\n

I am always surprise and sad about how worse the score I got in these days. Sometime I think maybe a lot of high LB score team overfitting and I should be confident on my validation scheme. Sometime I think maybe I am not on the correct direction and I am too conservative.

\n\n

We will know the result in 2.5 hours. This is an interesting competition.

", "rawMarkdown": "happycube, I didn't notice Kazanova and Gert are also in this competition. Hmmm, this makes it interesting. \r\n\r\nI am always surprise and sad about how worse the score I got in these days. Sometime I think maybe a lot of high LB score team overfitting and I should be confident on my validation scheme. Sometime I think maybe I am not on the correct direction and I am too conservative.\r\n\r\nWe will know the result in 2.5 hours. This is an interesting competition.", "votes": null }, { "id": "133025", "postDate": "08/30/2016 23:41:17", "content": "

Me in 20 minutes.

\n\n

\"enter

", "rawMarkdown": "Me in 20 minutes.\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://media.giphy.com/media/ENt72LgUE2gV2/giphy.gif", "votes": null }, { "id": "133026", "postDate": "08/30/2016 23:54:15", "content": "

In this case, isn't the public leaderboard exactly the week 10 samples? So, you'd want to fit your predictions for week 10 to the leaderboard as best you can, meanwhile trusting your local validation for week 11 ...Right?

", "rawMarkdown": "In this case, isn't the public leaderboard exactly the week 10 samples? So, you'd want to fit your predictions **for week 10** to the leaderboard as best you can, meanwhile trusting your local validation for week 11 ...Right?", "votes": null }, { "id": "133027", "postDate": "08/31/2016 00:02:52", "content": "

Not as big as I figured, but Nima made it to #10 (edit: 7!)... and I had a pretty nice gain, but not a great final score.

\n\n

Interestingly, my multi-linear model (inspired by the public scripts) got .458/.47308, which would still've been top 10%. Go figure.

\n\n

post-finalization edit: Ironically my best private LB score (still would've in the 80's IIRC) was the test where I didn't use week 9 data for the submission. Huh.

", "rawMarkdown": "Not as big as I figured, but Nima made it to #10 (edit: 7!)... and I had a pretty nice gain, but not a great final score.\r\n\r\nInterestingly, my multi-linear model (inspired by the public scripts) got .458/.47308, which would still've been top 10%. Go figure. \r\n\r\npost-finalization edit: Ironically my best private LB score (still would've in the 80's IIRC) was the test where I didn't use week 9 data for the submission. Huh.", "votes": null }, { "id": "133033", "postDate": "08/31/2016 00:52:20", "content": "

My approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.

\n\n

My basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.

\n\n

For my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.

\n\n

My highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...

\n\n

Anyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!

", "rawMarkdown": "My approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.\r\n\r\nMy basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.\r\n\r\nFor my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.\r\n\r\nMy highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...\r\n\r\nAnyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!", "votes": null }, { "id": "133034", "postDate": "08/31/2016 00:52:33", "content": "

bingo, 1996

", "rawMarkdown": "bingo, 1996", "votes": null }, { "id": "133035", "postDate": "08/31/2016 01:09:07", "content": "

[quote=Tod Wright;133033]

\n\n

My approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.

\n\n

My basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.

\n\n

For my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.

\n\n

My highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...

\n\n

Anyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!

\n\n

[/quote]

\n\n

Yes. we are doing similarly. But I was surprised that the week 11 score (0.449xxx) is worse than our week 10 score 0.447xxx (by withholding week 9). Do you observe this difference on your case?

", "rawMarkdown": "[quote=Tod Wright;133033]\r\n\r\nMy approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.\r\n\r\nMy basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.\r\n\r\nFor my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.\r\n\r\nMy highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...\r\n\r\nAnyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!\r\n\r\n[/quote]\r\n\r\nYes. we are doing similarly. But I was surprised that the week 11 score (0.449xxx) is worse than our week 10 score 0.447xxx (by withholding week 9). Do you observe this difference on your case?", "votes": null }, { "id": "133036", "postDate": "08/31/2016 01:12:53", "content": "

@Jianbo Ye

\n\n

Yes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).

", "rawMarkdown": "Jianbo Ye\r\n\r\nYes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).", "votes": null }, { "id": "133038", "postDate": "08/31/2016 01:23:24", "content": "

[quote=Tod Wright;133036]

\n\n

@Jianbo Ye

\n\n

Yes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).

\n\n

[/quote]

\n\n

I am feeling that maybe @NimaShahbazi is doing smarter than us. Because his pub score was worse than ours, but then did better on private score.

\n\n

Jianbo

", "rawMarkdown": "[quote=Tod Wright;133036]\r\n\r\n@Jianbo Ye\r\n\r\nYes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).\r\n\r\n[/quote]\r\n\r\nI am feeling that maybe @NimaShahbazi is doing smarter than us. Because his pub score was worse than ours, but then did better on private score. \r\n\r\nJianbo", "votes": null }, { "id": "133043", "postDate": "08/31/2016 02:06:53", "content": "

Jianbo Ye, Tod Wright - I really appreciate the explanation of your respective CV routines. I thought I had a decent CV routine (I did different variations of holding out one of weeks 6,7,8,9 and training on the others). Anyhow, I feel like I lucked into my top 15% finish =p. I really struggled with the appropriate CV approach for weeks during this competition. I am trying to learn (I am a student), and am going to use this project for my Machine Learning Engineer Nanodegree Capstone on Udacity.

\n\n

Anyhow, I'd love some insights into how you decided upon this particular CV routine - I am really looking forward to hearing what the people in the top 10 did as well!

", "rawMarkdown": "Jianbo Ye, Tod Wright - I really appreciate the explanation of your respective CV routines. I thought I had a decent CV routine (I did different variations of holding out one of weeks 6,7,8,9 and training on the others). Anyhow, I feel like I lucked into my top 15% finish =p. I really struggled with the appropriate CV approach for weeks during this competition. I am trying to learn (I am a student), and am going to use this project for my Machine Learning Engineer Nanodegree Capstone on Udacity. \r\n\r\nAnyhow, I'd love some insights into how you decided upon this particular CV routine - I am really looking forward to hearing what the people in the top 10 did as well!", "votes": null }, { "id": "133044", "postDate": "08/31/2016 02:23:00", "content": "

I am not totally sure if the thing is related to the performance of the models, but I did notice something quite strange and would like to share. As I am not comfortable with SQL and had trouble with limited memory of my laptop to run python, I ended up creating lag features using C++ hashmaps. \nWhat I discovered is that when I use a client_id and product_id combined (vector of bool ) as a hash of the entries and iterating through the entries using default order of C++ unordered_map, I noticed that a lot (literally half) of the consectuive entries in that order share the same depot_id, route_id, which is hard to imagine as the client_id and product_id pair are not the same for these entries. Artificially created data injected in the data set maybe ?

", "rawMarkdown": "I am not totally sure if the thing is related to the performance of the models, but I did notice something quite strange and would like to share. As I am not comfortable with SQL and had trouble with limited memory of my laptop to run python, I ended up creating lag features using C++ hashmaps. \r\nWhat I discovered is that when I use a client_id and product_id combined (vector of bool ) as a hash of the entries and iterating through the entries using default order of C++ unordered_map, I noticed that a lot (literally half) of the consectuive entries **in that order** share the same depot_id, route_id, which is hard to imagine as the client_id and product_id pair are not the same for these entries. Artificially created data injected in the data set maybe ?", "votes": null }, { "id": "133046", "postDate": "08/31/2016 02:44:34", "content": "

@TenaciousTerrier

\n\n

I'm not sure there are any deep insights behind this choice. For a problem with a time-series aspect you generally want to create a validation split analogous to the overall train/test split (rather than cross-validating). Given that the public/private split corresponds to weeks 10/11, it is clear that the predictions for the private leaderboard (which are the only ones that actually matter) will not have the benefit of having been trained on the preceding week. So the local validation setup should similarly omit the week before the hold out, to reflect the conditions of the final prediction for week 11. It then makes sense to do the same for the week 10 predictions, so that you have a local validation and an external test that both mimic the conditions found for the private leaderboard.

\n\n

The issue of course is that this is somewhat restricting, in that, e.g., you cannot form a 5-week lag for local validation, even though you could form one for the public test and the private test. Obviously you are then leaving some money on the table by forcing your public and private tests to suffer the same constraints as your local validation. You could start building different models for the public and private tests than you used for local validation (e.g., include the 5-week lag for these), but then you are trading off against transparency of the validation setup. Similarly for imputing 1-week lags in some way: no doubt there are gains to be had from doing something such as this, but it is then not so straightforward to interpret the validation and test scores.

\n\n

So, I chose to have a more transparent and (hopefully) more trustworthy validation setup and poorer scores rather than potentially better scores subject to more uncertainty and complexity.

", "rawMarkdown": "TenaciousTerrier\r\n\r\nI'm not sure there are any deep insights behind this choice. For a problem with a time-series aspect you generally want to create a validation split analogous to the overall train/test split (rather than cross-validating). Given that the public/private split corresponds to weeks 10/11, it is clear that the predictions for the private leaderboard (which are the only ones that actually matter) will not have the benefit of having been trained on the preceding week. So the local validation setup should similarly omit the week before the hold out, to reflect the conditions of the final prediction for week 11. It then makes sense to do the same for the week 10 predictions, so that you have a local validation and an external test that both mimic the conditions found for the private leaderboard.\r\n\r\nThe issue of course is that this is somewhat restricting, in that, e.g., you cannot form a 5-week lag for local validation, even though you could form one for the public test and the private test. Obviously you are then leaving some money on the table by forcing your public and private tests to suffer the same constraints as your local validation. You could start building different models for the public and private tests than you used for local validation (e.g., include the 5-week lag for these), but then you are trading off against transparency of the validation setup. Similarly for imputing 1-week lags in some way: no doubt there are gains to be had from doing something such as this, but it is then not so straightforward to interpret the validation and test scores. \r\n\r\nSo, I chose to have a more transparent and (hopefully) more trustworthy validation setup and poorer scores rather than potentially better scores subject to more uncertainty and complexity.", "votes": null }, { "id": "133048", "postDate": "08/31/2016 02:55:42", "content": "

Tod Wright - Thanks for the perspective! I definitely will take this feedback into consideration for my report, and my next time series Kaggle competition!

", "rawMarkdown": "Tod Wright - Thanks for the perspective! I definitely will take this feedback into consideration for my report, and my next time series Kaggle competition!", "votes": null }, { "id": "133052", "postDate": "08/31/2016 03:39:15", "content": "

@To Wright, I used a very similar cv setting except that I used one week more, for instance using weeks 5-7 to predict week 9.

", "rawMarkdown": "To Wright, I used a very similar cv setting except that I used one week more, for instance using weeks 5-7 to predict week 9.", "votes": null }, { "id": "133055", "postDate": "08/31/2016 04:04:28", "content": "

@CPMP

\n\n

I am not sure I follow what you mean exactly, but I realize now I should probably clarify that, in the case of xgboost models, by "[using] week 9 as my local validation set, training on weeks 3-7" I actually mean "using weeks 3-6 (3-7) to build features for week 7 (9), then training on week 7 and validating on week 9".

\n\n

(Note that, e.g., week 3 doesn't contribute a lag for week 9, but I had as a feature the mean demand over all previous weeks — except of course the week that was omitted from the train set.)

", "rawMarkdown": "CPMP\r\n\r\nI am not sure I follow what you mean exactly, but I realize now I should probably clarify that, in the case of xgboost models, by \"[using] week 9 as my local validation set, training on weeks 3-7\" I actually mean \"using weeks 3-6 (3-7) to build features for week 7 (9), then training on week 7 and validating on week 9\". \r\n\r\n(Note that, e.g., week 3 doesn't contribute a lag for week 9, but I had as a feature the mean demand over *all* previous weeks — except of course the week that was omitted from the train set.)", "votes": null }, { "id": "133061", "postDate": "08/31/2016 05:12:33", "content": "

Tod,

\n\n

I used a very similar approach, except that I used more weeks than you, see the post I just wrote.

", "rawMarkdown": "Tod,\r\n\r\nI used a very similar approach, except that I used more weeks than you, see the post I just wrote.", "votes": null }, { "id": "133248", "postDate": "09/01/2016 03:45:59", "content": "

[quote=NimaShahbazi;130565]

\n\n

I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!

\n\n

[/quote]

\n\n

It turns out you are pretty much correct. Will you share your validation strategy? How can you achieve this rank boost! @NimaShahbazi

\n\n

Jianbo

", "rawMarkdown": "[quote=NimaShahbazi;130565]\r\n\r\nI am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!\r\n\r\n[/quote]\r\n\r\nIt turns out you are pretty much correct. Will you share your validation strategy? How can you achieve this rank boost! @NimaShahbazi\r\n\r\nJianbo", "votes": null }, { "id": "133276", "postDate": "09/01/2016 07:28:28", "content": "

Well there was a huge shakeup - but mainly due to the culling of pixies!

", "rawMarkdown": "Well ***there was*** a huge shakeup - but mainly due to the culling of pixies!", "votes": null }, { "id": "133280", "postDate": "09/01/2016 07:34:42", "content": "

I wish I could upvote you more than once for the phrase "culling of pixies" alone :-)

", "rawMarkdown": "I wish I could upvote you more than once for the phrase \"culling of pixies\" alone :-)", "votes": null }, { "id": "133292", "postDate": "09/01/2016 11:10:20", "content": "

We got ~60th with quite a different strategy I suppose. Just slam client and product into libFM with as many latent factors as you can handle. That's it. No CV; didn't touch the time variables; no validation strategy; just train it as long as you can and submit it. (The MCMC fitting algorithm doesn't have any hyper-parameters to tune...)

\n\n

I'm lamenting we didn't spend more time on this. Sounds like a boosted decision tree model on some lagged features would have been a nice blend. I just assumed the people ahead of us were at least using more than 2 features in a libFM.

", "rawMarkdown": "We got ~60th with quite a different strategy I suppose. Just slam client and product into libFM with as many latent factors as you can handle. That's it. No CV; didn't touch the time variables; no validation strategy; just train it as long as you can and submit it. (The MCMC fitting algorithm doesn't have any hyper-parameters to tune...)\r\n\r\nI'm lamenting we didn't spend more time on this. Sounds like a boosted decision tree model on some lagged features would have been a nice blend. I just assumed the people ahead of us were at least using more than 2 features in a libFM.", "votes": null }, { "id": "133297", "postDate": "09/01/2016 11:43:48", "content": "

I think @NimaShahbazi refered to the fact that the final score is on week 11 only, and that many of the top 1% and the public scripts seemed to optimize the forecast for week 10 as well (with 1-lag data). Which of course is not needed and only distracts from the final goal.

", "rawMarkdown": "I think @NimaShahbazi refered to the fact that the final score is on week 11 only, and that many of the top 1% and the public scripts seemed to optimize the forecast for week 10 as well (with 1-lag data). Which of course is not needed and only distracts from the final goal.", "votes": null }, { "id": "134408", "postDate": "09/06/2016 14:44:21", "content": "

@Jianbo Ye, sorry for my late response, I was on a vacation and no internet access. I was expecting a huge one actually, but the shakeup is not the one I expected! the reason was using the prediction for week 10 as an actual lag1 for week 11, the one that many users suffers from, like MrOojer I guess! even those that move 2-3 position up/down may suffer from this error propagation problem. There was a popular public script and forum post based on this feature. The validation strategy that I used was training on week (3,8)+(4,8)+(5,8)+(6,8)+(7,8) and validation on week 9. same as for the test set, (3,9)+(4,9)+(5,9)+(6,9)+(7,9)+(8,9) ==> for week 10 and 11. and All my moving averages was >=lag2. I have find this combination from all 2^7 combinations for the best result. By the way, all my moving averages was on not only on #Demanda_uni_equil but on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima as well. Please let me know if you want more details on the feature or the model itself.\nps: the moving average that I used for the "unit sales" was mean(log(x)) moving average.

", "rawMarkdown": "Jianbo Ye, sorry for my late response, I was on a vacation and no internet access. I was expecting a huge one actually, but the shakeup is not the one I expected! the reason was using the prediction for week 10 as an actual lag1 for week 11, the one that many users suffers from, like MrOojer I guess! even those that move 2-3 position up/down may suffer from this error propagation problem. There was a popular public script and forum post based on this feature. The validation strategy that I used was training on week (3,8)+(4,8)+(5,8)+(6,8)+(7,8) and validation on week 9. same as for the test set, (3,9)+(4,9)+(5,9)+(6,9)+(7,9)+(8,9) ==> for week 10 and 11. and All my moving averages was >=lag2. I have find this combination from all 2^7 combinations for the best result. By the way, all my moving averages was on not only on #Demanda_uni_equil but on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima as well. Please let me know if you want more details on the feature or the model itself.\r\nps: the moving average that I used for the \"unit sales\" was mean(log(x)) moving average.", "votes": null }, { "id": "134423", "postDate": "09/06/2016 15:11:07", "content": "

@NimaShahbazi

\n\n

what do you mean by train on (3,8)+(4,8)+(5,8)+(6,8)+(7,8) ?

\n\n

For example, (3,8) : are you creating features for week 8 using week 3 (only)?

\n\n

It seems we have different validation strategies. Although I am not suffering the kind of effects you mentioned and many other teams have suffered, your validation seems still better than mine.

\n\n

Jianbo

", "rawMarkdown": "NimaShahbazi \r\n\r\nwhat do you mean by train on (3,8)+(4,8)+(5,8)+(6,8)+(7,8) ?\r\n\r\nFor example, (3,8) : are you creating features for week 8 using week 3 (only)?\r\n\r\nIt seems we have different validation strategies. Although I am not suffering the kind of effects you mentioned and many other teams have suffered, your validation seems still better than mine. \r\n\r\nJianbo", "votes": null }, { "id": "134430", "postDate": "09/06/2016 15:36:55", "content": "

@NimaShahbazi, Tx for sharing.

\n\n

I used an approach very close to yours apparently, but I decomposed by product because I did not have enough ram. What is your HW?

", "rawMarkdown": "NimaShahbazi, Tx for sharing.\r\n\r\nI used an approach very close to yours apparently, but I decomposed by product because I did not have enough ram. What is your HW?", "votes": null }, { "id": "134604", "postDate": "09/07/2016 13:50:49", "content": "

@JianboYe, thanks, I mean I train on individual set of (x,8) where x=(3,4,5,6,7), and evaluate the bag result on week 9. same for the leader board (x,9) x=(3,4,5,6,7,8). \n@CPMP, welcome. Yes in that case you will lose some similar-type products in order to generalize your model better. by the way my RAM footprint was around 27GB. I was using 16 core CPU with 32GB of RAM. MovingAverages on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima was really helpful though!

", "rawMarkdown": "JianboYe, thanks, I mean I train on individual set of (x,8) where x=(3,4,5,6,7), and evaluate the bag result on week 9. same for the leader board (x,9) x=(3,4,5,6,7,8). \r\n@CPMP, welcome. Yes in that case you will lose some similar-type products in order to generalize your model better. by the way my RAM footprint was around 27GB. I was using 16 core CPU with 32GB of RAM. MovingAverages on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima was really helpful though!", "votes": null }, { "id": "135125", "postDate": "09/11/2016 09:54:41", "content": "

@Tod Wright

\n\n

What features did you use? besides the lags of course.

", "rawMarkdown": "Tod Wright\r\n\r\nWhat features did you use? besides the lags of course.", "votes": null }, { "id": "135337", "postDate": "09/13/2016 07:41:45", "content": "

@PabloMarin

\n\n

The other features were all simple ones that were mentioned on the forums here at some point: the extracted product weights and piece counts, factor frequencies for Agencia_ID, Ruta_SAK, Cliente_ID, and Producto_ID, the client count per town (i.e., the number of clients in the client's town), and the mean (under log(1+p)) of Demanda_uni_equil over all previous weeks (except the 'skipped' week).

", "rawMarkdown": "PabloMarin\r\n\r\nThe other features were all simple ones that were mentioned on the forums here at some point: the extracted product weights and piece counts, factor frequencies for Agencia_ID, Ruta_SAK, Cliente_ID, and Producto_ID, the client count per town (i.e., the number of clients in the client's town), and the mean (under log(1+p)) of Demanda_uni_equil over all previous weeks (except the 'skipped' week).", "votes": null } ], "comments": [ { "id": 130569, "author_name": "jiweiliu", "author_url": "", "post_date": "08/08/2016 16:40:41", "content": "

[quote=NimaShahbazi;130565]

\n\n

I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!

\n\n

[/quote]\nwow, we didn't use any public scripts. Are we safe?

", "votes": null, "replies": [] }, { "id": 130570, "author_name": "nimashahbazi", "author_url": "", "post_date": "08/08/2016 16:46:36", "content": "

@rcarson: Well, I don't know, it depends on the features that you have used! Hope not! you are good at not overfitting buddy :)

", "votes": null, "replies": [] }, { "id": 130571, "author_name": "happycube", "author_url": "", "post_date": "08/08/2016 16:54:55", "content": "

With the public log-means scripts (mine included) the two week gap has fewer of the easiest types to predict, so they will probably jump up by ~.01. My coefficients were calculated using a gap week, so they might hold up a bit better - but they will all be slaughtered by an xgboost solution that avoids a should-be-obvious-but-isn't pitfall...

\n\n

edit: I expect some/all of the top 10'ers to figure out how to avoid the pitfall I'm thinking of. But much of the top 10% is going to take a hit. The drop range is likely going to be between .007 and .02, depending on what you're doing...

", "votes": null, "replies": [] }, { "id": 130573, "author_name": "jiweiliu", "author_url": "", "post_date": "08/08/2016 16:55:41", "content": "

Thank you for letting us know. I guess the risky feature might be the target rate encoding but we will review our validation process.

", "votes": null, "replies": [] }, { "id": 131250, "author_name": "aldente", "author_url": "", "post_date": "08/15/2016 00:38:04", "content": "

I did many cross validation. However due to small RAM I could only use 2 weeks of data per fitting.

\n\n

My XGBoost model uses lagged demand features, I guess this is the main factor.

\n\n

What I found: the closer evaluation's week to train's weeks, the better the CV RMSLE.

\n\n

For example,

\n\n\n\n

Week 5 and 8 has better score, they're closer to 6,7. \nWhile week 9 has worse score. \nThe same pattern in all other week pairs, 5+6, 5+8, .., 6+8, 6+9, .., 8+9.

\n\n

Test data:

\n\n

Week 10 = public LB, closer to training data by week. Should have better score.

\n\n

Week 11 = private LB, has far 2+ week distance from training data. Worse score ;)

\n\n

So, I'm expecting worse score by around 0.005+ in the private LB, maybe a rank drop by ~50 ;))

", "votes": null, "replies": [] }, { "id": 131253, "author_name": "happycube", "author_url": "", "post_date": "08/15/2016 02:09:34", "content": "

aldente - .005 sounds reasonable and should hold up on the private LB, if you're setting up week 9's processing correctly.

", "votes": null, "replies": [] }, { "id": 131257, "author_name": "aldente", "author_url": "", "post_date": "08/15/2016 04:22:01", "content": "

@happycube, hopefully ;)

\n\n

You're correct, looks like most important is weeks closer to test data.

", "votes": null, "replies": [] }, { "id": 131294, "author_name": "jiweiliu", "author_url": "", "post_date": "08/15/2016 15:03:55", "content": "

@happycube, may I ask what you mean by "processing week 9 correctly"? Thank you!

", "votes": null, "replies": [] }, { "id": 131300, "author_name": "alexxanderlarko", "author_url": "", "post_date": "08/15/2016 16:29:29", "content": "

@happycube\nI also wonder.\nMay I ask what you mean by "processing week 9 correctly"? \nThank you!\n:)))

", "votes": null, "replies": [] }, { "id": 131301, "author_name": "happycube", "author_url": "", "post_date": "08/15/2016 16:31:32", "content": "

Not using any information from week 8...

", "votes": null, "replies": [] }, { "id": 131346, "author_name": "alexxanderlarko", "author_url": "", "post_date": "08/16/2016 07:47:42", "content": "

Thank you!

", "votes": null, "replies": [] }, { "id": 131382, "author_name": "davutpolat", "author_url": "", "post_date": "08/16/2016 16:38:31", "content": "

@rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P

", "votes": null, "replies": [] }, { "id": 131520, "author_name": "cpmpml", "author_url": "", "post_date": "08/17/2016 21:29:18", "content": "

What is the point of this post? We all know that PB is using week 10 when private LB is using week 11. Either you disclose what you think is making a difference, or you use it at your advantage.

", "votes": null, "replies": [] }, { "id": 131540, "author_name": "aldente", "author_url": "", "post_date": "08/18/2016 01:10:09", "content": "

@CPMP, if you read my post carefully, I discussed the "reason" why I think I'll have worse private LB score (rank drop), because my XGB model uses lagged demand features which I suspect to hurt week 10/11 split. Based on my cross validation results.

\n\n

People using the same lagged demand features can reproduce it easily, to prove that the behavior of biweekly cross validation results I shared is the truth.

\n\n

(... deleted for misunderstanding, my apologies)

", "votes": null, "replies": [] }, { "id": 131553, "author_name": "alexxanderlarko", "author_url": "", "post_date": "08/18/2016 03:25:56", "content": "

[quote=Davut Polat;131382]

\n\n

@rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P

\n\n

[/quote]

\n\n

Hi, Davut Polat! Glad to see you.

\n\n

We'll wait :)))

", "votes": null, "replies": [] }, { "id": 131565, "author_name": "abhilashawasthi", "author_url": "", "post_date": "08/18/2016 07:58:24", "content": "

@aldente I think CPMP was talking about this post of NimaShahbazi as it doesn't make any sense. He didn't tell why there would be a huge shake-up, but just posted that there would be a shake-up which I think most of us know.

", "votes": null, "replies": [] }, { "id": 131566, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2016 08:07:44", "content": "

All, yes, I am asking about the post that started the thread. I may have overreacted because I am a bit nervous (I have a dental abscess ...) Apologies to whomever finds my question incorrect.

\n\n

@aldente, your post is informative and useful for those who did not realize that we have a 2 week ahead forecast to do, not a one week ahead forecast. Same comment applies to @happycube answer on processing week 9 correctly.

", "votes": null, "replies": [] }, { "id": 131596, "author_name": "tarobxl", "author_url": "", "post_date": "08/18/2016 13:47:39", "content": "

I think the "huge" shake-up happens in the middle of the competition who uses mean-value approach and ignores the time indicator.

\n\n

But Davut gave us a hope to jump to the top thanks to the shake-up ;-)

", "votes": null, "replies": [] }, { "id": 131598, "author_name": "davutpolat", "author_url": "", "post_date": "08/18/2016 13:51:58", "content": "

@Taro, there is always a hope to jump to the top :) I think forum posts are enough for a good score\nListen this song carefully :)\nhttps://www.youtube.com/watch?v=DQWI1kvmwRg

", "votes": null, "replies": [] }, { "id": 131616, "author_name": "manuelsh", "author_url": "", "post_date": "08/18/2016 16:19:45", "content": "

@Davut That video is great!

", "votes": null, "replies": [] }, { "id": 131640, "author_name": "aldente", "author_url": "", "post_date": "08/18/2016 20:13:21", "content": "

@Abhilash, @CPMP, my apologies for misunderstanding, I edited my post :) .. I supposed to ask first who are intended, my fault ;)) \n+1

", "votes": null, "replies": [] }, { "id": 131644, "author_name": "nimashahbazi", "author_url": "", "post_date": "08/18/2016 20:47:08", "content": "

@CPMP @AbhilashAwasthi, \n1- As you can notice some good discussions for the shakeup is done "after" I posted this thread!\n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules! \n3- Ofcourse I have my own reasons to post this thread. I was looking for the ideas, if someone likes to share! (My own strategy to see what others think, and maybe contribute if I like to).\n4- a "Single Thread" for shakeup makes everyone aware! Okay you (CPMP) and your friend (Ablihash) know about it, good for you, take advantage.\n5- if you don't find this informative, I am sorry that I waste your time with this post!\n6- I ranked 2nd in Rossman competition (there was a huge shakeup there too, and it was a pretty same competition), so "maybe" when I pointing out a huge shakeup it has more strength.\n7- thanks to @aldente @happycube for the info and @Davut for the video!

", "votes": null, "replies": [] }, { "id": 131646, "author_name": "nimashahbazi", "author_url": "", "post_date": "08/18/2016 21:04:15", "content": "

btw, kaggle a friendly community we all make good friends here, for example when @davut says I know what makes you .42x wait for me and the other guy says we are waiting! (is it informative?). we are all friends here and share what we think and what we "like" to! and there is no informative filter on posting threads (although I have my own reasons for this post).

", "votes": null, "replies": [] }, { "id": 131662, "author_name": "", "author_url": "", "post_date": "08/18/2016 22:28:27", "content": "

I am new and this is my first competition, currently 70 on LB. I have one question for the big guns here: Do u ensemble different models to get better results (on private LB) or its just one model ? If you don't want to answer, don't :]

", "votes": null, "replies": [] }, { "id": 131666, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2016 22:46:31", "content": "

@salenator I got 0.437xx public LB without ensembling, and I think I'll get better soon. There is hope therefore ;)

", "votes": null, "replies": [] }, { "id": 131667, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2016 22:53:16", "content": "

@NimaShahbazi, I apologized, see above.

\n\n

You wrote: \n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules!

\n\n

This applies to me as well: I can express my thoughts same as you.

", "votes": null, "replies": [] }, { "id": 131912, "author_name": "sergeycherkasov", "author_url": "", "post_date": "08/20/2016 14:48:39", "content": "

Dear Kagglers, I am rookie here and I want to ask a tiny humble question regarding this thrilling and scaring overfitting.

\n\n

I have worse metric on CV than on LB. Can I feel myself safe? My CV strategy is to use first 5 known weeks as train and two last known weeks as test.

\n\n

Many thanks for sharing your experience!

\n\n

SC

\n\n

PS Song is great and funny, but I am scared even more now.

", "votes": null, "replies": [] }, { "id": 131917, "author_name": "interesting", "author_url": "", "post_date": "08/20/2016 15:38:16", "content": "

Sergey, you should not be scared but instead try to adjust your solution.\nYour validation strategy seems fine to me, although you might want to notice that leaderboard is split by time, which means that the actual target of the competition is to predict horison 2. \nA better validation strategy can be training on first 6 weeks validate on the eighth week (public leader board), then add the additional seventh week info to generate the prediction (by same model) for the nineth week (private leader board)

\n\n

Good luck

\n\n

Nati

", "votes": null, "replies": [] }, { "id": 131922, "author_name": "sergeycherkasov", "author_url": "", "post_date": "08/20/2016 16:28:20", "content": "

@Nathaniel, thank you for advice! I will try for sure as soon as my machine will finish parameters training. But what do you think, usually, according to you experience and knowledge - does better metric on CV leads to overfitting and worse to under fitting? Or this is independent?

\n\n

SC

", "votes": null, "replies": [] }, { "id": 131923, "author_name": "interesting", "author_url": "", "post_date": "08/20/2016 16:39:47", "content": "

I made a habit of selecting my two submissions one by best lb score and the other by best CV score.\nIn general if CV method is robust you're better select your submission by CV.\nThe CV method i described above combines the two, as you can use the public leader board score as additional information.

", "votes": null, "replies": [] }, { "id": 132611, "author_name": "tarobxl", "author_url": "", "post_date": "08/27/2016 13:33:00", "content": "

Hi everybody,

\n\n

Right now the competition is close to the end so let's see if your guess on the leaderboard shakeup is right or not ;-) You can anonymously vote at http://goo.gl/Jjkxjh and see the results at http://goo.gl/ip0bSV. Personally I have enjoyed the journey so far and this poll might be fun as well.

\n\n

Here is the screenshot of the current leaderboard \"current.

\n\n

PS: I have just corrected the links, please use the links here not the ones in your email notification! Many thanks!

", "votes": null, "replies": [] }, { "id": 132619, "author_name": "caiotaniguchi", "author_url": "", "post_date": "08/27/2016 14:16:49", "content": "

@Tien-Dung Le, I voted, but I feel that this poll doesn't represent the overall board shake up, since people at the top are usually experts, so it's less likely that they are overfitting badly to the public LB. On the other hand, I do expect major shakeup around the XGB public script score, which probably encompasses most of the field.

", "votes": null, "replies": [] }, { "id": 132637, "author_name": "tarobxl", "author_url": "", "post_date": "08/27/2016 20:14:52", "content": "

@Caio, thanks for your vote. I do agree with you about "this poll doesn't represent the overall board shake up". The reason I use "top 10" because it is much easy to measure - by just looking and it is more interesting.

\n\n

"people at the top are usually experts" => So you expect the leaderboard is stable, probably like this one https://www.kaggle.com/c/avito-duplicate-ads-detection/leaderboard/private. However, don't forget that there exists some huge shake-ups like this https://www.kaggle.com/c/restaurant-revenue-prediction/leaderboard/private

\n\n

Let's see what other people vote!

", "votes": null, "replies": [] }, { "id": 132684, "author_name": "cpmpml", "author_url": "", "post_date": "08/28/2016 12:21:40", "content": "

@Sergey Cherkasov

\n\n

What matters really is if your LB value increases when your local CV value increases. if this is the case, then you can focus on your local CV value.

\n\n

But, as was said, issue is that the private LB is a 2 week horizon forecast, hence your local CV setting must reflect that.

", "votes": null, "replies": [] }, { "id": 132744, "author_name": "happycube", "author_url": "", "post_date": "08/28/2016 23:37:52", "content": "

For myself, I'm doing what I can to hold up on the private LB - my public LB score isn't great, but it loses less than .005 by removing any week 9 data at least. So hopefully it'll hold up OK for week 11...

\n\n

... and if not, I'll grumble and figure out WTF happened... ;)

", "votes": null, "replies": [] }, { "id": 132759, "author_name": "cpmpml", "author_url": "", "post_date": "08/29/2016 04:14:20", "content": "

@happycube, same here.

", "votes": null, "replies": [] }, { "id": 132987, "author_name": "happycube", "author_url": "", "post_date": "08/30/2016 18:07:43", "content": "

Another data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!

\n\n

So what does it say that a team capable of being #1 isn't even in the top 3% right now...?

", "votes": null, "replies": [] }, { "id": 132997, "author_name": "amazingbob", "author_url": "", "post_date": "08/30/2016 18:53:23", "content": "

[quote=happycube;132987]

\n\n

Another data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!

\n\n

So what does it say that a team capable of being #1 isn't even in the top 3% right now...?

\n\n

[/quote]

\n\n

We haven't update our LB score for more than 45 days until today. But we are making some progress along the way. Sticking to the highest possible LB score isn't that useful.

\n\n

Jianbo

", "votes": null, "replies": [] }, { "id": 133002, "author_name": "happycube", "author_url": "", "post_date": "08/30/2016 19:05:38", "content": "

Agreed - but they've submitted in the last day, before I made my post. They're definitely not angling for a public LB score, and I personally expect them to wind up in the top ten tonight.

\n\n

In this compo the public LB is at best a type of non-local validation, just another week to compare against occasionally and make sure the submission isn't garbage for some reason ;)

", "votes": null, "replies": [] }, { "id": 133011, "author_name": "aikinogard", "author_url": "", "post_date": "08/30/2016 21:25:13", "content": "

@happycube, I didn't notice Kazanova and Gert are also in this competition. Hmmm, this makes it interesting.

\n\n

I am always surprise and sad about how worse the score I got in these days. Sometime I think maybe a lot of high LB score team overfitting and I should be confident on my validation scheme. Sometime I think maybe I am not on the correct direction and I am too conservative.

\n\n

We will know the result in 2.5 hours. This is an interesting competition.

", "votes": null, "replies": [] }, { "id": 133025, "author_name": "crimeprophet", "author_url": "", "post_date": "08/30/2016 23:41:17", "content": "

Me in 20 minutes.

\n\n

\"enter

", "votes": null, "replies": [] }, { "id": 133026, "author_name": "ben519", "author_url": "", "post_date": "08/30/2016 23:54:15", "content": "

In this case, isn't the public leaderboard exactly the week 10 samples? So, you'd want to fit your predictions for week 10 to the leaderboard as best you can, meanwhile trusting your local validation for week 11 ...Right?

", "votes": null, "replies": [] }, { "id": 133027, "author_name": "happycube", "author_url": "", "post_date": "08/31/2016 00:02:52", "content": "

Not as big as I figured, but Nima made it to #10 (edit: 7!)... and I had a pretty nice gain, but not a great final score.

\n\n

Interestingly, my multi-linear model (inspired by the public scripts) got .458/.47308, which would still've been top 10%. Go figure.

\n\n

post-finalization edit: Ironically my best private LB score (still would've in the 80's IIRC) was the test where I didn't use week 9 data for the submission. Huh.

", "votes": null, "replies": [] }, { "id": 133033, "author_name": "tmwright", "author_url": "", "post_date": "08/31/2016 00:52:20", "content": "

My approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.

\n\n

My basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.

\n\n

For my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.

\n\n

My highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...

\n\n

Anyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!

", "votes": null, "replies": [] }, { "id": 133034, "author_name": "stevendu", "author_url": "", "post_date": "08/31/2016 00:52:33", "content": "

bingo, 1996

", "votes": null, "replies": [] }, { "id": 133035, "author_name": "amazingbob", "author_url": "", "post_date": "08/31/2016 01:09:07", "content": "

[quote=Tod Wright;133033]

\n\n

My approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.

\n\n

My basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.

\n\n

For my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.

\n\n

My highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...

\n\n

Anyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!

\n\n

[/quote]

\n\n

Yes. we are doing similarly. But I was surprised that the week 11 score (0.449xxx) is worse than our week 10 score 0.447xxx (by withholding week 9). Do you observe this difference on your case?

", "votes": null, "replies": [] }, { "id": 133036, "author_name": "tmwright", "author_url": "", "post_date": "08/31/2016 01:12:53", "content": "

@Jianbo Ye

\n\n

Yes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).

", "votes": null, "replies": [] }, { "id": 133038, "author_name": "amazingbob", "author_url": "", "post_date": "08/31/2016 01:23:24", "content": "

[quote=Tod Wright;133036]

\n\n

@Jianbo Ye

\n\n

Yes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).

\n\n

[/quote]

\n\n

I am feeling that maybe @NimaShahbazi is doing smarter than us. Because his pub score was worse than ours, but then did better on private score.

\n\n

Jianbo

", "votes": null, "replies": [] }, { "id": 133043, "author_name": "tenaciousterrier", "author_url": "", "post_date": "08/31/2016 02:06:53", "content": "

Jianbo Ye, Tod Wright - I really appreciate the explanation of your respective CV routines. I thought I had a decent CV routine (I did different variations of holding out one of weeks 6,7,8,9 and training on the others). Anyhow, I feel like I lucked into my top 15% finish =p. I really struggled with the appropriate CV approach for weeks during this competition. I am trying to learn (I am a student), and am going to use this project for my Machine Learning Engineer Nanodegree Capstone on Udacity.

\n\n

Anyhow, I'd love some insights into how you decided upon this particular CV routine - I am really looking forward to hearing what the people in the top 10 did as well!

", "votes": null, "replies": [] }, { "id": 133044, "author_name": "tonyzhangrt", "author_url": "", "post_date": "08/31/2016 02:23:00", "content": "

I am not totally sure if the thing is related to the performance of the models, but I did notice something quite strange and would like to share. As I am not comfortable with SQL and had trouble with limited memory of my laptop to run python, I ended up creating lag features using C++ hashmaps. \nWhat I discovered is that when I use a client_id and product_id combined (vector of bool ) as a hash of the entries and iterating through the entries using default order of C++ unordered_map, I noticed that a lot (literally half) of the consectuive entries in that order share the same depot_id, route_id, which is hard to imagine as the client_id and product_id pair are not the same for these entries. Artificially created data injected in the data set maybe ?

", "votes": null, "replies": [] }, { "id": 133046, "author_name": "tmwright", "author_url": "", "post_date": "08/31/2016 02:44:34", "content": "

@TenaciousTerrier

\n\n

I'm not sure there are any deep insights behind this choice. For a problem with a time-series aspect you generally want to create a validation split analogous to the overall train/test split (rather than cross-validating). Given that the public/private split corresponds to weeks 10/11, it is clear that the predictions for the private leaderboard (which are the only ones that actually matter) will not have the benefit of having been trained on the preceding week. So the local validation setup should similarly omit the week before the hold out, to reflect the conditions of the final prediction for week 11. It then makes sense to do the same for the week 10 predictions, so that you have a local validation and an external test that both mimic the conditions found for the private leaderboard.

\n\n

The issue of course is that this is somewhat restricting, in that, e.g., you cannot form a 5-week lag for local validation, even though you could form one for the public test and the private test. Obviously you are then leaving some money on the table by forcing your public and private tests to suffer the same constraints as your local validation. You could start building different models for the public and private tests than you used for local validation (e.g., include the 5-week lag for these), but then you are trading off against transparency of the validation setup. Similarly for imputing 1-week lags in some way: no doubt there are gains to be had from doing something such as this, but it is then not so straightforward to interpret the validation and test scores.

\n\n

So, I chose to have a more transparent and (hopefully) more trustworthy validation setup and poorer scores rather than potentially better scores subject to more uncertainty and complexity.

", "votes": null, "replies": [] }, { "id": 133048, "author_name": "tenaciousterrier", "author_url": "", "post_date": "08/31/2016 02:55:42", "content": "

Tod Wright - Thanks for the perspective! I definitely will take this feedback into consideration for my report, and my next time series Kaggle competition!

", "votes": null, "replies": [] }, { "id": 133052, "author_name": "cpmpml", "author_url": "", "post_date": "08/31/2016 03:39:15", "content": "

@To Wright, I used a very similar cv setting except that I used one week more, for instance using weeks 5-7 to predict week 9.

", "votes": null, "replies": [] }, { "id": 133055, "author_name": "tmwright", "author_url": "", "post_date": "08/31/2016 04:04:28", "content": "

@CPMP

\n\n

I am not sure I follow what you mean exactly, but I realize now I should probably clarify that, in the case of xgboost models, by "[using] week 9 as my local validation set, training on weeks 3-7" I actually mean "using weeks 3-6 (3-7) to build features for week 7 (9), then training on week 7 and validating on week 9".

\n\n

(Note that, e.g., week 3 doesn't contribute a lag for week 9, but I had as a feature the mean demand over all previous weeks — except of course the week that was omitted from the train set.)

", "votes": null, "replies": [] }, { "id": 133061, "author_name": "cpmpml", "author_url": "", "post_date": "08/31/2016 05:12:33", "content": "

Tod,

\n\n

I used a very similar approach, except that I used more weeks than you, see the post I just wrote.

", "votes": null, "replies": [] }, { "id": 133248, "author_name": "amazingbob", "author_url": "", "post_date": "09/01/2016 03:45:59", "content": "

[quote=NimaShahbazi;130565]

\n\n

I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!

\n\n

[/quote]

\n\n

It turns out you are pretty much correct. Will you share your validation strategy? How can you achieve this rank boost! @NimaShahbazi

\n\n

Jianbo

", "votes": null, "replies": [] }, { "id": 133276, "author_name": "scirpus", "author_url": "", "post_date": "09/01/2016 07:28:28", "content": "

Well there was a huge shakeup - but mainly due to the culling of pixies!

", "votes": null, "replies": [] }, { "id": 133280, "author_name": "konradb", "author_url": "", "post_date": "09/01/2016 07:34:42", "content": "

I wish I could upvote you more than once for the phrase "culling of pixies" alone :-)

", "votes": null, "replies": [] }, { "id": 133292, "author_name": "biasvariance", "author_url": "", "post_date": "09/01/2016 11:10:20", "content": "

We got ~60th with quite a different strategy I suppose. Just slam client and product into libFM with as many latent factors as you can handle. That's it. No CV; didn't touch the time variables; no validation strategy; just train it as long as you can and submit it. (The MCMC fitting algorithm doesn't have any hyper-parameters to tune...)

\n\n

I'm lamenting we didn't spend more time on this. Sounds like a boosted decision tree model on some lagged features would have been a nice blend. I just assumed the people ahead of us were at least using more than 2 features in a libFM.

", "votes": null, "replies": [] }, { "id": 133297, "author_name": "mrooijer", "author_url": "", "post_date": "09/01/2016 11:43:48", "content": "

I think @NimaShahbazi refered to the fact that the final score is on week 11 only, and that many of the top 1% and the public scripts seemed to optimize the forecast for week 10 as well (with 1-lag data). Which of course is not needed and only distracts from the final goal.

", "votes": null, "replies": [] }, { "id": 134408, "author_name": "nimashahbazi", "author_url": "", "post_date": "09/06/2016 14:44:21", "content": "

@Jianbo Ye, sorry for my late response, I was on a vacation and no internet access. I was expecting a huge one actually, but the shakeup is not the one I expected! the reason was using the prediction for week 10 as an actual lag1 for week 11, the one that many users suffers from, like MrOojer I guess! even those that move 2-3 position up/down may suffer from this error propagation problem. There was a popular public script and forum post based on this feature. The validation strategy that I used was training on week (3,8)+(4,8)+(5,8)+(6,8)+(7,8) and validation on week 9. same as for the test set, (3,9)+(4,9)+(5,9)+(6,9)+(7,9)+(8,9) ==> for week 10 and 11. and All my moving averages was >=lag2. I have find this combination from all 2^7 combinations for the best result. By the way, all my moving averages was on not only on #Demanda_uni_equil but on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima as well. Please let me know if you want more details on the feature or the model itself.\nps: the moving average that I used for the "unit sales" was mean(log(x)) moving average.

", "votes": null, "replies": [] }, { "id": 134423, "author_name": "amazingbob", "author_url": "", "post_date": "09/06/2016 15:11:07", "content": "

@NimaShahbazi

\n\n

what do you mean by train on (3,8)+(4,8)+(5,8)+(6,8)+(7,8) ?

\n\n

For example, (3,8) : are you creating features for week 8 using week 3 (only)?

\n\n

It seems we have different validation strategies. Although I am not suffering the kind of effects you mentioned and many other teams have suffered, your validation seems still better than mine.

\n\n

Jianbo

", "votes": null, "replies": [] }, { "id": 134430, "author_name": "cpmpml", "author_url": "", "post_date": "09/06/2016 15:36:55", "content": "

@NimaShahbazi, Tx for sharing.

\n\n

I used an approach very close to yours apparently, but I decomposed by product because I did not have enough ram. What is your HW?

", "votes": null, "replies": [] }, { "id": 134604, "author_name": "nimashahbazi", "author_url": "", "post_date": "09/07/2016 13:50:49", "content": "

@JianboYe, thanks, I mean I train on individual set of (x,8) where x=(3,4,5,6,7), and evaluate the bag result on week 9. same for the leader board (x,9) x=(3,4,5,6,7,8). \n@CPMP, welcome. Yes in that case you will lose some similar-type products in order to generalize your model better. by the way my RAM footprint was around 27GB. I was using 16 core CPU with 32GB of RAM. MovingAverages on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima was really helpful though!

", "votes": null, "replies": [] }, { "id": 135125, "author_name": "pablomarin", "author_url": "", "post_date": "09/11/2016 09:54:41", "content": "

@Tod Wright

\n\n

What features did you use? besides the lags of course.

", "votes": null, "replies": [] }, { "id": 135337, "author_name": "tmwright", "author_url": "", "post_date": "09/13/2016 07:41:45", "content": "

@PabloMarin

\n\n

The other features were all simple ones that were mentioned on the forums here at some point: the extracted product weights and piece counts, factor frequencies for Agencia_ID, Ruta_SAK, Cliente_ID, and Producto_ID, the client count per town (i.e., the number of clients in the client's town), and the mean (under log(1+p)) of Demanda_uni_equil over all previous weeks (except the 'skipped' week).

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "130565": "I am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!", "130569": "[quote=NimaShahbazi;130565]\r\n\r\nI am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!\r\n\r\n[/quote]\r\nwow, we didn't use any public scripts. Are we safe?", "130570": "rcarson: Well, I don't know, it depends on the features that you have used! Hope not! you are good at not overfitting buddy :)", "130571": "With the public log-means scripts (mine included) the two week gap has fewer of the easiest types to predict, so they will probably jump up by ~.01. My coefficients were calculated using a gap week, so they might hold up a bit better - but they will all be slaughtered by an xgboost solution that avoids a should-be-obvious-but-isn't pitfall...\r\n\r\nedit: I expect some/all of the top 10'ers to figure out how to avoid the pitfall I'm thinking of. But much of the top 10% is going to take a hit. The drop range is likely going to be between .007 and .02, depending on what you're doing...", "130573": "Thank you for letting us know. I guess the risky feature might be the target rate encoding but we will review our validation process.", "131250": "I did many cross validation. However due to small RAM I could only use 2 weeks of data per fitting. \r\n\r\nMy XGBoost model uses lagged demand features, I guess this is the main factor. \r\n\r\nWhat I found: the closer evaluation's week to train's weeks, the better the CV RMSLE. \r\n\r\nFor example, \r\n\r\n - train week 6,7 ; eval week 5 ; rmsle 0.444189828967\r\n \r\n - train week 6,7 ; eval week 8 ; rmsle 0.444531862635 \r\n \r\n - train week 6,7 ; eval week 9 ; rmsle 0.450169899702\r\n\r\nWeek 5 and 8 has better score, they're closer to 6,7. \r\nWhile week 9 has worse score. \r\nThe same pattern in all other week pairs, 5+6, 5+8, .., 6+8, 6+9, .., 8+9. \r\n\r\nTest data: \r\n\r\nWeek 10 = public LB, closer to training data by week. Should have better score. \r\n \r\nWeek 11 = private LB, has far 2+ week distance from training data. Worse score ;) \r\n \r\nSo, I'm expecting worse score by around 0.005+ in the private LB, maybe a rank drop by ~50 ;))", "131253": "aldente - .005 sounds reasonable and should hold up on the private LB, if you're setting up week 9's processing correctly.", "131257": "happycube, hopefully ;) \r\n \r\nYou're correct, looks like most important is weeks closer to test data.", "131294": "happycube, may I ask what you mean by \"processing week 9 correctly\"? Thank you!", "131300": "happycube\r\nI also wonder.\r\nMay I ask what you mean by \"processing week 9 correctly\"? \r\nThank you!\r\n:)))", "131301": "Not using any information from week 8...", "131346": "Thank you!", "131382": "rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P", "131520": "What is the point of this post? We all know that PB is using week 10 when private LB is using week 11. Either you disclose what you think is making a difference, or you use it at your advantage.", "131540": "CPMP, if you read my post carefully, I discussed the \"reason\" why I think I'll have worse private LB score (rank drop), because my XGB model uses lagged demand features which I suspect to hurt week 10/11 split. Based on my cross validation results.\r\n\r\nPeople using the same lagged demand features can reproduce it easily, to prove that the behavior of biweekly cross validation results I shared is the truth. \r\n\r\n(... deleted for misunderstanding, my apologies)", "131553": "[quote=Davut Polat;131382]\r\n\r\n@rcarson and @Alexander Larko, i know what you have for .42xx score, wait for me :P i will be back soon :P \r\n\r\n\r\n[/quote]\r\n\r\nHi, Davut Polat! Glad to see you.\r\n\r\nWe'll wait :)))", "131565": "aldente I think CPMP was talking about this post of NimaShahbazi as it doesn't make any sense. He didn't tell why there would be a huge shake-up, but just posted that there would be a shake-up which I think most of us know.", "131566": "All, yes, I am asking about the post that started the thread. I may have overreacted because I am a bit nervous (I have a dental abscess ...) Apologies to whomever finds my question incorrect.\r\n\r\n@aldente, your post is informative and useful for those who did not realize that we have a 2 week ahead forecast to do, not a one week ahead forecast. Same comment applies to @happycube answer on processing week 9 correctly.", "131596": "I think the \"huge\" shake-up happens in the middle of the competition who uses mean-value approach and ignores the time indicator. \r\n\r\nBut Davut gave us a hope to jump to the top thanks to the shake-up ;-)", "131598": "Taro, there is always a hope to jump to the top :) I think forum posts are enough for a good score\r\nListen this song carefully :)\r\nhttps://www.youtube.com/watch?v=DQWI1kvmwRg", "131616": "Davut That video is great!", "131640": "Abhilash, @CPMP, my apologies for misunderstanding, I edited my post :) .. I supposed to ask first who are intended, my fault ;)) \r\n+1", "131644": "CPMP @AbhilashAwasthi, \r\n1- As you can notice some good discussions for the shakeup is done \"after\" I posted this thread!\r\n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules! \r\n3- Ofcourse I have my own reasons to post this thread. I was looking for the ideas, if someone likes to share! (My own strategy to see what others think, and maybe contribute if I like to).\r\n4- a \"Single Thread\" for shakeup makes everyone aware! Okay you (CPMP) and your friend (Ablihash) know about it, good for you, take advantage.\r\n5- if you don't find this informative, I am sorry that I waste your time with this post!\r\n6- I ranked 2nd in Rossman competition (there was a huge shakeup there too, and it was a pretty same competition), so \"maybe\" when I pointing out a huge shakeup it has more strength.\r\n7- thanks to @aldente @happycube for the info and @Davut for the video!", "131646": "btw, kaggle a friendly community we all make good friends here, for example when @davut says I know what makes you .42x wait for me and the other guy says we are waiting! (is it informative?). we are all friends here and share what we think and what we \"like\" to! and there is no informative filter on posting threads (although I have my own reasons for this post).", "131662": "I am new and this is my first competition, currently 70 on LB. I have one question for the big guns here: Do u ensemble different models to get better results (on private LB) or its just one model ? If you don't want to answer, don't :]", "131666": "salenator I got 0.437xx public LB without ensembling, and I think I'll get better soon. There is hope therefore ;)", "131667": "NimaShahbazi, I apologized, see above.\r\n\r\nYou wrote: \r\n2- You have no right to tell someone what to write or what not to! everyone can express their thoughts as long as it is not against the rules!\r\n\r\nThis applies to me as well: I can express my thoughts same as you.", "131912": "Dear Kagglers, I am rookie here and I want to ask a tiny humble question regarding this thrilling and scaring overfitting. \r\n\r\nI have worse metric on CV than on LB. Can I feel myself safe? My CV strategy is to use first 5 known weeks as train and two last known weeks as test.\r\n\r\nMany thanks for sharing your experience!\r\n\r\nSC\r\n\r\nPS Song is great and funny, but I am scared even more now.", "131917": "Sergey, you should not be scared but instead try to adjust your solution.\r\nYour validation strategy seems fine to me, although you might want to notice that leaderboard is split by time, which means that the actual target of the competition is to predict horison 2. \r\nA better validation strategy can be training on first 6 weeks validate on the eighth week (public leader board), then add the additional seventh week info to generate the prediction (by same model) for the nineth week (private leader board)\r\n\r\nGood luck\r\n\r\nNati", "131922": "Nathaniel, thank you for advice! I will try for sure as soon as my machine will finish parameters training. But what do you think, usually, according to you experience and knowledge - does better metric on CV leads to overfitting and worse to under fitting? Or this is independent?\r\n\r\nSC", "131923": "I made a habit of selecting my two submissions one by best lb score and the other by best CV score.\r\nIn general if CV method is robust you're better select your submission by CV.\r\nThe CV method i described above combines the two, as you can use the public leader board score as additional information.", "132611": "Hi everybody,\r\n\r\nRight now the competition is close to the end so let's see if your guess on the leaderboard shakeup is right or not ;-) You can anonymously vote at [http://goo.gl/Jjkxjh][1] and see the results at [http://goo.gl/ip0bSV][2]. Personally I have enjoyed the journey so far and this poll might be fun as well.\r\n\r\nHere is the screenshot of the current leaderboard ![current leaderboard][3].\r\n\r\nPS: I have just corrected the links, please use the links here not the ones in your email notification! Many thanks!\r\n\r\n\r\n [1]: http://goo.gl/Jjkxjh\r\n [2]: http://goo.gl/ip0bSV\r\n [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/132611/4740/Screen%20Shot%202016-08-27%20at%2015.26.51.png?sv=2012-02-12&se=2016-08-30T13%3A34%3A41Z&sr=b&sp=r&sig=DB6kbZcVWkIp8BTWV7ymSVZSZ%2BiWZfU5GtfI%2BjIJeqs%3D", "132619": "Tien-Dung Le, I voted, but I feel that this poll doesn't represent the overall board shake up, since people at the top are usually experts, so it's less likely that they are overfitting badly to the public LB. On the other hand, I do expect major shakeup around the XGB public script score, which probably encompasses most of the field.", "132637": "Caio, thanks for your vote. I do agree with you about \"this poll doesn't represent the overall board shake up\". The reason I use \"top 10\" because it is much easy to measure - by just looking and it is more interesting.\r\n\r\n\"people at the top are usually experts\" => So you expect the leaderboard is stable, probably like this one [https://www.kaggle.com/c/avito-duplicate-ads-detection/leaderboard/private][1]. However, don't forget that there exists some huge shake-ups like this [https://www.kaggle.com/c/restaurant-revenue-prediction/leaderboard/private][2]\r\n\r\nLet's see what other people vote! \r\n\r\n\r\n [1]: https://www.kaggle.com/c/avito-duplicate-ads-detection/leaderboard/private\r\n [2]: https://www.kaggle.com/c/restaurant-revenue-prediction/leaderboard/private", "132684": "Sergey Cherkasov\r\n\r\nWhat matters really is if your LB value increases when your local CV value increases. if this is the case, then you can focus on your local CV value.\r\n\r\nBut, as was said, issue is that the private LB is a 2 week horizon forecast, hence your local CV setting must reflect that.", "132744": "For myself, I'm doing what I can to hold up on the private LB - my public LB score isn't great, but it loses less than .005 by removing any week 9 data at least. So hopefully it'll hold up OK for week 11...\r\n\r\n... and if not, I'll grumble and figure out WTF happened... ;)", "132759": "happycube, same here.", "132987": "Another data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!\r\n\r\nSo what does it say that a team capable of being #1 isn't even in the top 3% right now...?", "132997": "[quote=happycube;132987]\r\n\r\nAnother data point - and a huge one IMO - is that Kazanova and Gert are at #69 (.44173) and haven't improved their public LB score in almost 4 weeks... and either one by themselves are more than capable of finishing in the top 10!\r\n\r\nSo what does it say that a team capable of being #1 isn't even in the top 3% right now...?\r\n\r\n[/quote]\r\n\r\nWe haven't update our LB score for more than 45 days until today. But we are making some progress along the way. Sticking to the highest possible LB score isn't that useful.\r\n\r\nJianbo", "133002": "Agreed - but they've submitted in the last day, before I made my post. They're definitely not angling for a public LB score, and I personally expect them to wind up in the top ten tonight.\r\n\r\nIn this compo the public LB is at best a type of non-local validation, just another week to compare against occasionally and make sure the submission isn't garbage for some reason ;)", "133011": "happycube, I didn't notice Kazanova and Gert are also in this competition. Hmmm, this makes it interesting. \r\n\r\nI am always surprise and sad about how worse the score I got in these days. Sometime I think maybe a lot of high LB score team overfitting and I should be confident on my validation scheme. Sometime I think maybe I am not on the correct direction and I am too conservative.\r\n\r\nWe will know the result in 2.5 hours. This is an interesting competition.", "133025": "Me in 20 minutes.\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://media.giphy.com/media/ENt72LgUE2gV2/giphy.gif", "133026": "In this case, isn't the public leaderboard exactly the week 10 samples? So, you'd want to fit your predictions **for week 10** to the leaderboard as best you can, meanwhile trusting your local validation for week 11 ...Right?", "133027": "Not as big as I figured, but Nima made it to #10 (edit: 7!)... and I had a pretty nice gain, but not a great final score.\r\n\r\nInterestingly, my multi-linear model (inspired by the public scripts) got .458/.47308, which would still've been top 10%. Go figure. \r\n\r\npost-finalization edit: Ironically my best private LB score (still would've in the 80's IIRC) was the test where I didn't use week 9 data for the submission. Huh.", "133033": "My approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.\r\n\r\nMy basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.\r\n\r\nFor my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.\r\n\r\nMy highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...\r\n\r\nAnyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!", "133034": "bingo, 1996", "133035": "[quote=Tod Wright;133033]\r\n\r\nMy approach, for whatever it's worth (38th place, apparently) was to use week 9 as my local validation set, training on weeks 3-7 (i.e., skipping week 8). I then trained on weeks 3-8 (skipping week 9) to test against the public leader board (week 10). For the final submission I then trained on weeks 3-9 to predict week 11. I imagine this scheme was probably pretty common.\r\n\r\nMy basic philosophy was to not try and do anything too cute. I decided to not even bother trying to use model predictions for week N-1 to form a noisy 1-week lag for week N. So I used (for xgboost models) lags starting from 2 weeks. For the local validation training set omitting week 8 I couldn't form a 5-week lag, so I only considered 2-, 3-, and 4-week lags.\r\n\r\nFor my final model I took an unweighted average (under the log(1+p)) of three xgboost models, trained on three slightly different feature sets and then averaged this in roughly 60:40 proportion (chosen by my local validation scores) with a single FTRL-Proximal model (thanks @Scirpus) that was also trained on the one-week-short training sets discussed above.\r\n\r\nMy highest public score was a model trained earlier using week 9, and did worse by 0.02 on the private leaderboard. It was kind of spooky submitting models that omitted week 9 and therefore did much worse on the public leaderboard, knowing that no doubt many other people were now doing similarly. I was really not sure what was going to happen when the private leaderboard was revealed...\r\n\r\nAnyway, my congratulations to the prize winners and those achieving new personal bests, and any (?) newly anointed (grand) masters!\r\n\r\n[/quote]\r\n\r\nYes. we are doing similarly. But I was surprised that the week 11 score (0.449xxx) is worse than our week 10 score 0.447xxx (by withholding week 9). Do you observe this difference on your case?", "133036": "Jianbo Ye\r\n\r\nYes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).", "133038": "[quote=Tod Wright;133036]\r\n\r\n@Jianbo Ye\r\n\r\nYes, I observed the same and similarly did a double-take when I first saw it. My score tuple (VAL, PUB, PRV) = (0.46035, 0.45138, 0.45580).\r\n\r\n[/quote]\r\n\r\nI am feeling that maybe @NimaShahbazi is doing smarter than us. Because his pub score was worse than ours, but then did better on private score. \r\n\r\nJianbo", "133043": "Jianbo Ye, Tod Wright - I really appreciate the explanation of your respective CV routines. I thought I had a decent CV routine (I did different variations of holding out one of weeks 6,7,8,9 and training on the others). Anyhow, I feel like I lucked into my top 15% finish =p. I really struggled with the appropriate CV approach for weeks during this competition. I am trying to learn (I am a student), and am going to use this project for my Machine Learning Engineer Nanodegree Capstone on Udacity. \r\n\r\nAnyhow, I'd love some insights into how you decided upon this particular CV routine - I am really looking forward to hearing what the people in the top 10 did as well!", "133044": "I am not totally sure if the thing is related to the performance of the models, but I did notice something quite strange and would like to share. As I am not comfortable with SQL and had trouble with limited memory of my laptop to run python, I ended up creating lag features using C++ hashmaps. \r\nWhat I discovered is that when I use a client_id and product_id combined (vector of bool ) as a hash of the entries and iterating through the entries using default order of C++ unordered_map, I noticed that a lot (literally half) of the consectuive entries **in that order** share the same depot_id, route_id, which is hard to imagine as the client_id and product_id pair are not the same for these entries. Artificially created data injected in the data set maybe ?", "133046": "TenaciousTerrier\r\n\r\nI'm not sure there are any deep insights behind this choice. For a problem with a time-series aspect you generally want to create a validation split analogous to the overall train/test split (rather than cross-validating). Given that the public/private split corresponds to weeks 10/11, it is clear that the predictions for the private leaderboard (which are the only ones that actually matter) will not have the benefit of having been trained on the preceding week. So the local validation setup should similarly omit the week before the hold out, to reflect the conditions of the final prediction for week 11. It then makes sense to do the same for the week 10 predictions, so that you have a local validation and an external test that both mimic the conditions found for the private leaderboard.\r\n\r\nThe issue of course is that this is somewhat restricting, in that, e.g., you cannot form a 5-week lag for local validation, even though you could form one for the public test and the private test. Obviously you are then leaving some money on the table by forcing your public and private tests to suffer the same constraints as your local validation. You could start building different models for the public and private tests than you used for local validation (e.g., include the 5-week lag for these), but then you are trading off against transparency of the validation setup. Similarly for imputing 1-week lags in some way: no doubt there are gains to be had from doing something such as this, but it is then not so straightforward to interpret the validation and test scores. \r\n\r\nSo, I chose to have a more transparent and (hopefully) more trustworthy validation setup and poorer scores rather than potentially better scores subject to more uncertainty and complexity.", "133048": "Tod Wright - Thanks for the perspective! I definitely will take this feedback into consideration for my report, and my next time series Kaggle competition!", "133052": "To Wright, I used a very similar cv setting except that I used one week more, for instance using weeks 5-7 to predict week 9.", "133055": "CPMP\r\n\r\nI am not sure I follow what you mean exactly, but I realize now I should probably clarify that, in the case of xgboost models, by \"[using] week 9 as my local validation set, training on weeks 3-7\" I actually mean \"using weeks 3-6 (3-7) to build features for week 7 (9), then training on week 7 and validating on week 9\". \r\n\r\n(Note that, e.g., week 3 doesn't contribute a lag for week 9, but I had as a feature the mean demand over *all* previous weeks — except of course the week that was omitted from the train set.)", "133061": "Tod,\r\n\r\nI used a very similar approach, except that I used more weeks than you, see the post I just wrote.", "133248": "[quote=NimaShahbazi;130565]\r\n\r\nI am expecting a really huge shake up, specially in top 1% in this competition due to some reason (that I have noticed in the forum and the public scripts). If I was correct I will point out after the competition ends!\r\n\r\n[/quote]\r\n\r\nIt turns out you are pretty much correct. Will you share your validation strategy? How can you achieve this rank boost! @NimaShahbazi\r\n\r\nJianbo", "133276": "Well ***there was*** a huge shakeup - but mainly due to the culling of pixies!", "133280": "I wish I could upvote you more than once for the phrase \"culling of pixies\" alone :-)", "133292": "We got ~60th with quite a different strategy I suppose. Just slam client and product into libFM with as many latent factors as you can handle. That's it. No CV; didn't touch the time variables; no validation strategy; just train it as long as you can and submit it. (The MCMC fitting algorithm doesn't have any hyper-parameters to tune...)\r\n\r\nI'm lamenting we didn't spend more time on this. Sounds like a boosted decision tree model on some lagged features would have been a nice blend. I just assumed the people ahead of us were at least using more than 2 features in a libFM.", "133297": "I think @NimaShahbazi refered to the fact that the final score is on week 11 only, and that many of the top 1% and the public scripts seemed to optimize the forecast for week 10 as well (with 1-lag data). Which of course is not needed and only distracts from the final goal.", "134408": "Jianbo Ye, sorry for my late response, I was on a vacation and no internet access. I was expecting a huge one actually, but the shakeup is not the one I expected! the reason was using the prediction for week 10 as an actual lag1 for week 11, the one that many users suffers from, like MrOojer I guess! even those that move 2-3 position up/down may suffer from this error propagation problem. There was a popular public script and forum post based on this feature. The validation strategy that I used was training on week (3,8)+(4,8)+(5,8)+(6,8)+(7,8) and validation on week 9. same as for the test set, (3,9)+(4,9)+(5,9)+(6,9)+(7,9)+(8,9) ==> for week 10 and 11. and All my moving averages was >=lag2. I have find this combination from all 2^7 combinations for the best result. By the way, all my moving averages was on not only on #Demanda_uni_equil but on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima as well. Please let me know if you want more details on the feature or the model itself.\r\nps: the moving average that I used for the \"unit sales\" was mean(log(x)) moving average.", "134423": "NimaShahbazi \r\n\r\nwhat do you mean by train on (3,8)+(4,8)+(5,8)+(6,8)+(7,8) ?\r\n\r\nFor example, (3,8) : are you creating features for week 8 using week 3 (only)?\r\n\r\nIt seems we have different validation strategies. Although I am not suffering the kind of effects you mentioned and many other teams have suffered, your validation seems still better than mine. \r\n\r\nJianbo", "134430": "NimaShahbazi, Tx for sharing.\r\n\r\nI used an approach very close to yours apparently, but I decomposed by product because I did not have enough ram. What is your HW?", "134604": "JianboYe, thanks, I mean I train on individual set of (x,8) where x=(3,4,5,6,7), and evaluate the bag result on week 9. same for the leader board (x,9) x=(3,4,5,6,7,8). \r\n@CPMP, welcome. Yes in that case you will lose some similar-type products in order to generalize your model better. by the way my RAM footprint was around 27GB. I was using 16 core CPU with 32GB of RAM. MovingAverages on #Venta_uni_hoy #Venta_hoy #Dev_uni_proxima was really helpful though!", "135125": "Tod Wright\r\n\r\nWhat features did you use? besides the lags of course.", "135337": "PabloMarin\r\n\r\nThe other features were all simple ones that were mentioned on the forums here at some point: the extracted product weights and piece counts, factor frequencies for Agencia_ID, Ruta_SAK, Cliente_ID, and Producto_ID, the client count per town (i.e., the number of clients in the client's town), and the mean (under log(1+p)) of Demanda_uni_equil over all previous weeks (except the 'skipped' week)." }, "source": "meta" }