{ "id": 24961, "title": "Shake-up???", "url": "/competitions/bosch-production-line-performance/discussion/24961", "author_name": "", "post_date": "2016-10-31T17:57:44.283Z", "votes": 8, "comment_count": 44, "views": 4852, "content": "

Too early to discuss shake-up? This would be my thoughts:

\n\n

• The dataset has a large number of rows. This would seem to indicate a small shake-up.

\n\n

• The train/test split is random, and the public/private split is random. This would seem to indicate to a small shake-up.

\n\n

• The ratio of positives to negatives is very skewed. This would seem to indicate a non-small shake-up because although the overall number of rows is large, the number of positives is fairly small, and it would be easy to overfit to those few positives.

\n\n

• The scores on the public leaderboard are moderately close. Currently 0.028 separates #1 from #20.

\n\n

• The MCC evaluation function seems to me to lend itself to overfit. It is very sensitive to a small number of “borderline” rows. Many rows in the dataset are clearly negatives. Some rows in the dataset are clearly positives. Then there are the “borderline” rows. So although we started with a dataset that has a large number of rows, we end up with only a few rows that “matter”. My estimate is that there are fewer than 1000 rows that matter. So that’s actually a pretty small dataset. Let me clarify what I mean by “matter”. Lots and lots of rows matter for training, but really only a few matter when calculating MCC on the test set (for all practical purposes).

\n\n

So we start with a large number of rows, but we end up with only 1000 or so rows that matter. Public LB is based on 30% of the data, therefore public LB is based on about 300 rows that matter.

\n\n

So all of that to say that I don’t expect a huge, global shake-up, but it is very possible that we could see some people moving up by 0.01 to 0.02 while others move down by 0.01 to 0.02. So it is possible that there might be a significant reshuffling of the Top 20.

\n\n

What do you think? How many test rows "matter"?

", "messages": [ { "id": "142204", "postDate": "10/31/2016 17:57:44", "content": "

Too early to discuss shake-up? This would be my thoughts:

\n\n

• The dataset has a large number of rows. This would seem to indicate a small shake-up.

\n\n

• The train/test split is random, and the public/private split is random. This would seem to indicate to a small shake-up.

\n\n

• The ratio of positives to negatives is very skewed. This would seem to indicate a non-small shake-up because although the overall number of rows is large, the number of positives is fairly small, and it would be easy to overfit to those few positives.

\n\n

• The scores on the public leaderboard are moderately close. Currently 0.028 separates #1 from #20.

\n\n

• The MCC evaluation function seems to me to lend itself to overfit. It is very sensitive to a small number of “borderline” rows. Many rows in the dataset are clearly negatives. Some rows in the dataset are clearly positives. Then there are the “borderline” rows. So although we started with a dataset that has a large number of rows, we end up with only a few rows that “matter”. My estimate is that there are fewer than 1000 rows that matter. So that’s actually a pretty small dataset. Let me clarify what I mean by “matter”. Lots and lots of rows matter for training, but really only a few matter when calculating MCC on the test set (for all practical purposes).

\n\n

So we start with a large number of rows, but we end up with only 1000 or so rows that matter. Public LB is based on 30% of the data, therefore public LB is based on about 300 rows that matter.

\n\n

So all of that to say that I don’t expect a huge, global shake-up, but it is very possible that we could see some people moving up by 0.01 to 0.02 while others move down by 0.01 to 0.02. So it is possible that there might be a significant reshuffling of the Top 20.

\n\n

What do you think? How many test rows "matter"?

", "rawMarkdown": "Too early to discuss shake-up? This would be my thoughts:\r\n\r\n•\tThe dataset has a large number of rows. This would seem to indicate a small shake-up.\r\n\r\n•\tThe train/test split is random, and the public/private split is random. This would seem to indicate to a small shake-up.\r\n\r\n•\tThe ratio of positives to negatives is very skewed. This would seem to indicate a non-small shake-up because although the overall number of rows is large, the number of positives is fairly small, and it would be easy to overfit to those few positives.\r\n\r\n•\tThe scores on the public leaderboard are moderately close. Currently 0.028 separates #1 from #20.\r\n\r\n•\tThe MCC evaluation function seems to me to lend itself to overfit. It is very sensitive to a small number of “borderline” rows. Many rows in the dataset are clearly negatives. Some rows in the dataset are clearly positives. Then there are the “borderline” rows. So although we started with a dataset that has a large number of rows, we end up with only a few rows that “matter”. My estimate is that there are fewer than 1000 rows that matter. So that’s actually a pretty small dataset. Let me clarify what I mean by “matter”. Lots and lots of rows matter for training, but really only a few matter when calculating MCC on the test set (for all practical purposes).\r\n\r\n\r\n\r\nSo we start with a large number of rows, but we end up with only 1000 or so rows that matter. Public LB is based on 30% of the data, therefore public LB is based on about 300 rows that matter.\r\n\r\n\r\n\r\nSo all of that to say that I don’t expect a huge, global shake-up, but it is very possible that we could see some people moving up by 0.01 to 0.02 while others move down by 0.01 to 0.02. So it is possible that there might be a significant reshuffling of the Top 20.\r\n\r\n\r\n\r\nWhat do you think? How many test rows \"matter\"?", "votes": null }, { "id": "142210", "postDate": "10/31/2016 18:58:05", "content": "

Hi Steve, I have the same feeling about shakeup. But maybe a +-0.01MCC is too optimistic, sure it will be much more ;-P

", "rawMarkdown": "Hi Steve, I have the same feeling about shakeup. But maybe a +-0.01MCC is too optimistic, sure it will be much more ;-P", "votes": null }, { "id": "142412", "postDate": "11/02/2016 05:16:09", "content": "

Yeah - and some people've reported a bigger CV<->LB gap than others, it's quite possible the ones with lower CV will drop. (Or they might not. That's part of what keeps the endings interesting!)

", "rawMarkdown": "Yeah - and some people've reported a bigger CV<->LB gap than others, it's quite possible the ones with lower CV will drop. (Or they might not. That's part of what keeps the endings interesting!)", "votes": null }, { "id": "142572", "postDate": "11/03/2016 03:25:47", "content": "

I might be wrong but sniffing around the data, I found that there are over 1500+ records in the training data with label (1) that have identical values in the test data for all numeric and categorical variables....I repeat "IDENTICAL" values for every single numeric and categorical variable. I engineered some features based on this info and got 0.62+ MCC local cv!!! I was ecstatic "eat my dust", upon submission only ~0.40 on LB. What happened? I think there will be huge shake up in this competition. I see many Grand-masters at the top and they have a knack for hanging on....but I think there will be huge shake up "Santander style!"

", "rawMarkdown": "I might be wrong but sniffing around the data, I found that there are over 1500+ records in the training data with label (1) that have identical values in the test data for all numeric and categorical variables....I repeat \"IDENTICAL\" values for every single numeric and categorical variable. I engineered some features based on this info and got 0.62+ MCC local cv!!! I was ecstatic \"eat my dust\", upon submission only ~0.40 on LB. What happened? I think there will be huge shake up in this competition. I see many Grand-masters at the top and they have a knack for hanging on....but I think there will be huge shake up \"Santander style!\"", "votes": null }, { "id": "142836", "postDate": "11/04/2016 19:59:53", "content": "

[quote=DavidGbodiOdaibo;142572]

\n\n

got 0.62+ MCC local cv!!! I was ecstatic "eat my dust", upon submission only ~0.40 on LB. What happened?

\n\n

[/quote]

\n\n

Have you checked whether information leaked from your training folds to your test folds in CV? A 0.62 CV seems almost too good to be true. If a person were creating features such as "What is the failure % for rows with exactly these feature values?" it would be easy for info to leak from training folds to test folds. Specifically, the "failure %" for such features would need to be calculated within the CV loop on just the training folds. Calculating such "failure %" features based on the whole set of training data before CV can lead to leakage.

", "rawMarkdown": "[quote=DavidGbodiOdaibo;142572]\r\n\r\ngot 0.62+ MCC local cv!!! I was ecstatic \"eat my dust\", upon submission only ~0.40 on LB. What happened? \r\n\r\n[/quote]\r\n\r\nHave you checked whether information leaked from your training folds to your test folds in CV? A 0.62 CV seems almost too good to be true. If a person were creating features such as \"What is the failure % for rows with exactly these feature values?\" it would be easy for info to leak from training folds to test folds. Specifically, the \"failure %\" for such features would need to be calculated within the CV loop on just the training folds. Calculating such \"failure %\" features based on the whole set of training data before CV can lead to leakage.", "votes": null }, { "id": "142837", "postDate": "11/04/2016 20:28:59", "content": "

@breakfastpirate, I agree there probably was leakage....

", "rawMarkdown": "breakfastpirate, I agree there probably was leakage....", "votes": null }, { "id": "142838", "postDate": "11/04/2016 20:33:50", "content": "

@breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :)

", "rawMarkdown": "breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :)", "votes": null }, { "id": "142840", "postDate": "11/04/2016 20:47:01", "content": "

[quote=DavidGbodiOdaibo;142838]

\n\n

@breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :)

\n\n

[/quote]

\n\n

There are other threads in the forum that discuss more details about these duplicates and how they might be useful.

", "rawMarkdown": "[quote=DavidGbodiOdaibo;142838]\r\n\r\n@breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :) \r\n\r\n[/quote]\r\n\r\nThere are other threads in the forum that discuss more details about these duplicates and how they might be useful.", "votes": null }, { "id": "143009", "postDate": "11/06/2016 04:13:14", "content": "

There may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.

", "rawMarkdown": "There may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.", "votes": null }, { "id": "143015", "postDate": "11/06/2016 05:59:15", "content": "

[quote=CPMP;143009]

\n\n

There may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.

\n\n

[/quote]

\n\n

The problem with Santander was that people were adding a bunch of post-algorithm adjustments, and those were overfit to the the public LB. "The Script" seemed to feed this overfitting frenzy by crowdsourcing the addition of overfit post-algorithm adjustments. I agree that there is potential for overfit to the 300 or so "rows that matter" in the public LB for Bosch, but so far we have not seen a feeding frenzy of people suggesting post-algorithm adjustments in the forum/scripts.

\n\n

So I think there is certainly the possibility of significant local reshuffling, but I don't think we will see quite the global bloodbath we saw on Santander.

\n\n

I would say use both CV and LB feedback, but give a bigger weight to CV.

", "rawMarkdown": "[quote=CPMP;143009]\r\n\r\nThere may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.\r\n\r\n[/quote]\r\n\r\nThe problem with Santander was that people were adding a bunch of post-algorithm adjustments, and those were overfit to the the public LB. \"The Script\" seemed to feed this overfitting frenzy by crowdsourcing the addition of overfit post-algorithm adjustments. I agree that there is potential for overfit to the 300 or so \"rows that matter\" in the public LB for Bosch, but so far we have not seen a feeding frenzy of people suggesting post-algorithm adjustments in the forum/scripts.\r\n\r\nSo I think there is certainly the possibility of significant local reshuffling, but I don't think we will see quite the global bloodbath we saw on Santander.\r\n\r\nI would say use both CV and LB feedback, but give a bigger weight to CV.", "votes": null }, { "id": "143045", "postDate": "11/06/2016 13:09:48", "content": "

Yep, agree ~900 rows matter. With MCC my feedback tends to be LBPalm. Moe is the LB and I am Curly: https://www.youtube.com/watch?v=u4ZgVRJ-H8U

", "rawMarkdown": "Yep, agree ~900 rows matter. With MCC my feedback tends to be LBPalm. Moe is the LB and I am Curly: https://www.youtube.com/watch?v=u4ZgVRJ-H8U", "votes": null }, { "id": "143071", "postDate": "11/06/2016 18:46:59", "content": "

there are 1183747 rows in train and 1183748 rows in test, ~6000 failures in train, yet how many 1's in submission shows some at the top of the leaderboard report 1800-2000 failures in their submissions. We have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.

", "rawMarkdown": "there are 1183747 rows in train and 1183748 rows in test, ~6000 failures in train, yet [how many 1's in submission][1] shows some at the top of the leaderboard report 1800-2000 failures in their submissions. We have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24083/how-many-1-s-in-your-submission", "votes": null }, { "id": "143073", "postDate": "11/06/2016 18:55:43", "content": "

mcc is such that it is better to have less true positive if you also drop the number of false positive drastically.

\n\n

For instance, our best submission (about 0.485 LB) has only 2672 1s in it.

\n\n

People with better LB reported similar rates.

\n\n

It also means that MCC may not be the best metric for getting good prediction when the data is so imbalanced. But that's not for us to judge...

", "rawMarkdown": "mcc is such that it is better to have less true positive if you also drop the number of false positive drastically.\r\n\r\nFor instance, our best submission (about 0.485 LB) has only 2672 1s in it.\r\n\r\nPeople with better LB reported similar rates.\r\n\r\nIt also means that MCC may not be the best metric for getting good prediction when the data is so imbalanced. But that's not for us to judge...", "votes": null }, { "id": "143093", "postDate": "11/06/2016 20:27:35", "content": "

[quote=DavidGbodiOdaibo;143071]

\n\n

(...)

\n\n

We have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.

\n\n

[/quote]

\n\n

Because it is how MCC works: as we have many true negatives, the only objectives you have to optimize are the diagonals of the confusion matrix. If FP or FN is very small, then the other is larger. But the multiplication is smaller than if you balanced FP and FN (think of 0 x 1000 < 500 x 500). For most of us (if not predicting a lot of positives), it depends on how many true positives you have vs the amount of positives you predict. It's a chase for maximum true positive accuracy in your positive predictions, while being able to maximize also the amount of positive predictions (the two objectives to chase and to balance appropriately).

\n\n

If your features are tuned for a large amount of positives, you may be able to do as well as underpredicting the positive class. For instance, I have one submission with 5500+ positives, and yet it scores 0.4+ on LB (for MCC, it is about the same as just swapping FP and FN, which mathematically changes nearly nothing - just bigger numbers making improvement more difficult). You can do as well by predicting only 2000 positives, and get 0.4+ LB too. Obviously, if you predict 5500+, you lower the risk you take - but getting such amount of positives with a high score is not trivial as you need to catch most of them.

\n\n

The difference post-bias (= after substracting the bias between public and private LB) should be around 0.005-0.006, as it seems to be the standard deviation for 20% of data at around 0.5 MCC (so for 30% it's even smaller).

\n\n

This one will never be like the Santander one, unless the private LB is really much different (still a possibility, but a very low one). But then, this competition does not seem like the Seizure one with 3 patients only.

\n\n

Remember also that you may see MCC linear in the brain, but the improvement you need is quadratic (because the true negative count is so large). Going from 0.5 to 0.6 is significantly harder than from 0.2 to 0.3 (approximately 2.2x more difficult). Therefore, as you go higher in MCC, it becomes more difficult to see linear improvements (in the MCC value) than if you were at a low MCC. Add into the factor how hard it becomes to improve once so many good features were engineered, and you hit a wall for improvement which makes it look like a real sigmoid (from a human perspective of performance improvement against time):

\n\n

\"enter

\n\n

Therefore, if you have 75% true positives in your 2000 positives, it is clearly not trivial to go to 7500 positives and 93% true positives (about the same score). Let alone improving 93% true positives, it should be easier to increase from 75% than from 93%.

", "rawMarkdown": "[quote=DavidGbodiOdaibo;143071]\r\n\r\n(...)\r\n\r\nWe have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.\r\n\r\n[/quote]\r\n\r\nBecause it is how MCC works: as we have many true negatives, the only objectives you have to optimize are the [diagonals of the confusion matrix][1]. If FP or FN is very small, then the other is larger. But the multiplication is smaller than if you balanced FP and FN (think of 0 x 1000 < 500 x 500). For most of us (if not predicting a lot of positives), it depends on how many true positives you have vs the amount of positives you predict. It's a chase for maximum true positive accuracy in your positive predictions, while being able to maximize also the amount of positive predictions (the two objectives to chase and to balance appropriately).\r\n\r\nIf your features are tuned for a large amount of positives, you may be able to do as well as underpredicting the positive class. For instance, I have one submission with 5500+ positives, and yet it scores 0.4+ on LB (for MCC, it is *about the same as just swapping FP and FN*, which mathematically changes nearly nothing - just bigger numbers making improvement more difficult). You can do as well by predicting only 2000 positives, and get 0.4+ LB too. Obviously, if you predict 5500+, you lower the risk you take - but getting such amount of positives with a high score is not trivial as you need to catch most of them.\r\n\r\nThe difference post-bias (= after substracting the bias between public and private LB) should be around 0.005-0.006, as it seems to be the standard deviation for 20% of data at around 0.5 MCC (so for 30% it's even smaller).\r\n\r\nThis one will never be like the Santander one, unless the private LB is really much different (still a possibility, but a very low one). But then, this competition does not seem like [the Seizure one][2] with 3 patients only.\r\n\r\nRemember also that you may see MCC linear in the brain, but the improvement you need is quadratic (because the true negative count is so large). Going from 0.5 to 0.6 is significantly harder than from 0.2 to 0.3 (approximately 2.2x more difficult). Therefore, as you go higher in MCC, it becomes more difficult to see linear improvements (in the MCC value) than if you were at a low MCC. Add into the factor how hard it becomes to improve once so many good features were engineered, and you hit a wall for improvement which makes it look like a real sigmoid (from a human perspective of performance improvement against time):\r\n\r\n![enter image description here][3]\r\n\r\nTherefore, if you have 75% true positives in your 2000 positives, it is clearly not trivial to go to 7500 positives and 93% true positives (about the same score). Let alone improving 93% true positives, it should be easier to increase from 75% than from 93%.\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24022/train-test-event-rate-imbalance/137512#post137512\r\n [2]: https://www.kaggle.com/c/melbourne-university-seizure-prediction/leaderboard\r\n [3]: http://i.imgur.com/ROYk0Zt.png", "votes": null }, { "id": "143381", "postDate": "11/08/2016 13:11:01", "content": "

Just waiting for the Shake-Up...\n\"enter\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code

", "rawMarkdown": "Just waiting for the Shake-Up...\r\n![enter image description here][1]\r\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code\r\n\r\n\r\n [1]: https://www.kaggle.io/svf/434942/82a01d76067d3b0bd2fc8fda31fe16c8/last_week_xkcd.png", "votes": null }, { "id": "143384", "postDate": "11/08/2016 13:16:52", "content": "

Shake-Up Is Coming...

", "rawMarkdown": "Shake-Up Is Coming...", "votes": null }, { "id": "143385", "postDate": "11/08/2016 13:27:16", "content": "

[quote=beluga;143381]

\n\n

Just waiting for the Shake-Up...\n\"enter\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code

\n\n

[/quote]

\n\n

This never fails to make me laugh :D

", "rawMarkdown": "[quote=beluga;143381]\r\n\r\nJust waiting for the Shake-Up...\r\n![enter image description here][1]\r\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code\r\n\r\n\r\n [1]: https://www.kaggle.io/svf/434942/82a01d76067d3b0bd2fc8fda31fe16c8/last_week_xkcd.png\r\n\r\n[/quote]\r\n\r\nThis never fails to make me laugh :D", "votes": null }, { "id": "143551", "postDate": "11/09/2016 14:21:50", "content": "

Hopefully, LB shake-up on this competition will not be as big as the LB shake-up was in U.S. elections last night. I think the election was a Santander-level shake-up.

", "rawMarkdown": "Hopefully, LB shake-up on this competition will not be as big as the LB shake-up was in U.S. elections last night. I think the election was a Santander-level shake-up.", "votes": null }, { "id": "143673", "postDate": "11/10/2016 02:58:26", "content": "

SHAKE IT UP!\n44 hours of catnip watching LB Shake-up

\n\n

PS: Good one @BreakfastPirate :) and looking forward to hearing the stories of Road-to-0.5

", "rawMarkdown": "SHAKE IT UP!\r\n44 hours of catnip watching LB Shake-up\r\n\r\nPS: Good one @BreakfastPirate :) and looking forward to hearing the stories of Road-to-0.5", "votes": null }, { "id": "143914", "postDate": "11/11/2016 11:35:09", "content": "

Mini-shake up probably incoming specifically there:

\n\n

\"enter

\n\n

This will be fun for the last 12 hours!

", "rawMarkdown": "Mini-shake up probably incoming specifically there:\r\n\r\n![enter image description here][1]\r\n\r\nThis will be fun for the last 12 hours!\r\n\r\n [1]: http://i.imgur.com/ZXKIlfi.png", "votes": null }, { "id": "143916", "postDate": "11/11/2016 11:40:55", "content": "

Best of luck all ! May the gods of overfitting be with you

", "rawMarkdown": "Best of luck all ! May the gods of overfitting be with you", "votes": null }, { "id": "143917", "postDate": "11/11/2016 11:55:32", "content": "

Good luck everyone. Let's the over-fitting be revealed.

", "rawMarkdown": "Good luck everyone. Let's the over-fitting be revealed.", "votes": null }, { "id": "143928", "postDate": "11/11/2016 12:55:50", "content": "

Here is the shake-up historical spectrum (with some recent competitions added). Any predictions of where this competition will fall along the spectrum?

\n\n

\"enter

", "rawMarkdown": "Here is the shake-up historical spectrum (with some recent competitions added). Any predictions of where this competition will fall along the spectrum?\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/143928/5336/Shakeup20161111.jpg?sv=2012-02-12&se=2016-11-14T12%3A56%3A00Z&sr=b&sp=r&sig=TE4SBSiPFyQgQO6b2EQRA1g4AZFzV%2FRSH7bBsplXpJo%3D", "votes": null }, { "id": "143959", "postDate": "11/11/2016 15:20:03", "content": "

[quote=Laurae;143914]

\n\n

Mini-shake up probably incoming specifically there:

\n\n

\"enter

\n\n

This will be fun for the last 12 hours!

\n\n

[/quote]

\n\n

Are you saying overfitting increases with team size? ;)

", "rawMarkdown": "[quote=Laurae;143914]\r\n\r\nMini-shake up probably incoming specifically there:\r\n\r\n![enter image description here][1]\r\n\r\nThis will be fun for the last 12 hours!\r\n\r\n [1]: http://i.imgur.com/ZXKIlfi.png\r\n\r\n[/quote]\r\n\r\n\r\nAre you saying overfitting increases with team size? ;)", "votes": null }, { "id": "143983", "postDate": "11/11/2016 18:47:15", "content": "

[quote=CPMP;143959]

\n\n

Are you saying overfitting increases with team size? ;)

\n\n

[/quote]

\n\n

It's more like there are 4 there competing for 2 gold medal spots at that range. The private LB will "shuffle" and maybe none, some, or all of these 4 will be in top 12.

", "rawMarkdown": "[quote=CPMP;143959]\r\n\r\nAre you saying overfitting increases with team size? ;)\r\n\r\n[/quote]\r\n\r\nIt's more like there are 4 there competing for 2 gold medal spots at that range. The private LB will \"shuffle\" and maybe none, some, or all of these 4 will be in top 12.", "votes": null }, { "id": "143993", "postDate": "11/11/2016 20:36:04", "content": "

... I'd just like to keep a bronze medal - I ran out of ideas a while ago and I've been busy the last few weeks. :)

", "rawMarkdown": "... I'd just like to keep a bronze medal - I ran out of ideas a while ago and I've been busy the last few weeks. :)", "votes": null }, { "id": "143997", "postDate": "11/11/2016 21:11:29", "content": "

Damn, my team might get that dreaded top Silver spot. I should've gone to church last Sunday...

", "rawMarkdown": "Damn, my team might get that dreaded top Silver spot. I should've gone to church last Sunday...", "votes": null }, { "id": "144002", "postDate": "11/11/2016 21:29:39", "content": "

I think a shake-up within 0.02 is expected.

", "rawMarkdown": "I think a shake-up within 0.02 is expected.", "votes": null }, { "id": "144003", "postDate": "11/11/2016 21:34:08", "content": "

Team "Make dishwashers great again" ... are you hoping your slogan will lead to a positive shake-up surprise?

", "rawMarkdown": "Team \"Make dishwashers great again\" ... are you hoping your slogan will lead to a positive shake-up surprise?", "votes": null }, { "id": "144007", "postDate": "11/11/2016 22:12:58", "content": "

Country girl shake it for me ...\nhttps://www.youtube.com/watch?v=pEUCuX5cFRA

\n\n

Enjoy and good luck to everybody!

", "rawMarkdown": "Country girl shake it for me ...\r\nhttps://www.youtube.com/watch?v=pEUCuX5cFRA\r\n\r\nEnjoy and good luck to everybody!", "votes": null }, { "id": "144008", "postDate": "11/11/2016 22:17:04", "content": "

When you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s

", "rawMarkdown": "When you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s", "votes": null }, { "id": "144011", "postDate": "11/11/2016 22:42:30", "content": "

Any teams excluding their best public LB submission from their final subs?

", "rawMarkdown": "Any teams excluding their best public LB submission from their final subs?", "votes": null }, { "id": "144012", "postDate": "11/11/2016 22:44:57", "content": "

[quote=BreakfastPirate;144003]

\n\n

Team "Make dishwashers great again" ... are you hoping your slogan will lead to a positive shake-up surprise?

\n\n

[/quote]

\n\n

We know dishwashers, we have the best dishwashers. People always tell me, "anokas, you have the most beautiful dishwashers."

", "rawMarkdown": "[quote=BreakfastPirate;144003]\r\n\r\nTeam \"Make dishwashers great again\" ... are you hoping your slogan will lead to a positive shake-up surprise?\r\n\r\n[/quote]\r\n\r\nWe know dishwashers, we have the best dishwashers. People always tell me, \"anokas, you have the most beautiful dishwashers.\"", "votes": null }, { "id": "144013", "postDate": "11/11/2016 22:48:40", "content": "

we are selecting our best LB model, but i am quite certain it will fail (we just dont have anything else to select, lol); second is more safe pick ~0.513. anyway, shakeup will only be in ~0.005 range, so top3 are safe with their position, I think :)

", "rawMarkdown": "we are selecting our best LB model, but i am quite certain it will fail (we just dont have anything else to select, lol); second is more safe pick ~0.513. anyway, shakeup will only be in ~0.005 range, so top3 are safe with their position, I think :)", "votes": null }, { "id": "144023", "postDate": "11/12/2016 00:04:57", "content": "

Shake-up is fairly low: 0.015 overall and 0.011 for the Top 10%. So in the neighborhood of Caterpillar Tube Pricing.

", "rawMarkdown": "Shake-up is fairly low: 0.015 overall and 0.011 for the Top 10%. So in the neighborhood of Caterpillar Tube Pricing.", "votes": null }, { "id": "144024", "postDate": "11/12/2016 00:06:09", "content": "

our score ended up identical to what we got on CV. so fairly stable competition :) gratz to the winners

", "rawMarkdown": "our score ended up identical to what we got on CV. so fairly stable competition :) gratz to the winners", "votes": null }, { "id": "144026", "postDate": "11/12/2016 00:13:33", "content": "

We did not select our best public model we chose 0.52429 over 0.52448.

\n\n

EDIT: Our best public model would have been even better (0.526 private mcc). I was the one who suggested that the second best would be more stable. Fortunately it did not matter :)

", "rawMarkdown": "We did not select our best public model we chose 0.52429 over 0.52448.\r\n\r\n**EDIT:** Our best public model would have been even better (0.526 private mcc). I was the one who suggested that the second best would be more stable. Fortunately it did not matter :)", "votes": null }, { "id": "144027", "postDate": "11/12/2016 00:13:47", "content": "

Of our two solutions, the one with the lower public LB score (and fewer 1s and higher CV score) got better on the private LB.

", "rawMarkdown": "Of our two solutions, the one with the lower public LB score (and fewer 1s and higher CV score) got better on the private LB.", "votes": null }, { "id": "144050", "postDate": "11/12/2016 00:55:45", "content": "

Will you shake us up with your Road-to-0.52 bedtime story?

\n\n

How many ones did your have?

", "rawMarkdown": "Will you shake us up with your Road-to-0.52 bedtime story?\r\n\r\nHow many ones did your have?", "votes": null }, { "id": "144060", "postDate": "11/12/2016 02:21:10", "content": "

Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.

", "rawMarkdown": "Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.", "votes": null }, { "id": "144070", "postDate": "11/12/2016 03:39:35", "content": "

Thanks BreakfastPirate :)

", "rawMarkdown": "Thanks BreakfastPirate :)", "votes": null }, { "id": "144091", "postDate": "11/12/2016 09:30:05", "content": "

[quote=BreakfastPirate;144060]

\n\n

Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.

\n\n

[/quote]

\n\n

We followed the same strategy, and our submission with fewer 1s (2629) also scored better on the PB than the one with more 1s (3495) even though the sub with more 1s scored significantly better on the LB. This was surprising.

", "rawMarkdown": "[quote=BreakfastPirate;144060]\r\n\r\nOur submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.\r\n\r\n[/quote]\r\n\r\nWe followed the same strategy, and our submission with fewer 1s (2629) also scored better on the PB than the one with more 1s (3495) even though the sub with more 1s scored significantly better on the LB. This was surprising.", "votes": null }, { "id": "144418", "postDate": "11/14/2016 07:15:12", "content": "

[quote=BreakfastPirate;144060]

\n\n

Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.

\n\n

[/quote]

\n\n

Just curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..

", "rawMarkdown": "[quote=BreakfastPirate;144060]\r\n\r\nOur submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.\r\n\r\n[/quote]\r\n\r\nJust curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..", "votes": null }, { "id": "144507", "postDate": "11/14/2016 12:50:00", "content": "

[quote=Wei Tang;144418]

\n\n

Just curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..

\n\n

[/quote]

\n\n

One way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the "My Submissions" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.

", "rawMarkdown": "[quote=Wei Tang;144418]\r\n\r\nJust curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..\r\n\r\n[/quote]\r\n\r\nOne way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the \"My Submissions\" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.", "votes": null }, { "id": "144605", "postDate": "11/14/2016 19:46:30", "content": "

[quote=BreakfastPirate;144507]

\n\n

[quote=Wei Tang;144418]

\n\n

Just curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..

\n\n

[/quote]

\n\n

One way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the "My Submissions" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.

\n\n

[/quote]

\n\n

Thanks a lot BreakfastPirate! I did not know that there is this nice feature of post-competition submissions.

", "rawMarkdown": "[quote=BreakfastPirate;144507]\r\n\r\n[quote=Wei Tang;144418]\r\n\r\nJust curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..\r\n\r\n[/quote]\r\n\r\nOne way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the \"My Submissions\" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.\r\n\r\n[/quote]\r\n\r\n\r\nThanks a lot BreakfastPirate! I did not know that there is this nice feature of post-competition submissions.", "votes": null } ], "comments": [ { "id": 142210, "author_name": "titericz", "author_url": "", "post_date": "10/31/2016 18:58:05", "content": "

Hi Steve, I have the same feeling about shakeup. But maybe a +-0.01MCC is too optimistic, sure it will be much more ;-P

", "votes": null, "replies": [] }, { "id": 142412, "author_name": "happycube", "author_url": "", "post_date": "11/02/2016 05:16:09", "content": "

Yeah - and some people've reported a bigger CV<->LB gap than others, it's quite possible the ones with lower CV will drop. (Or they might not. That's part of what keeps the endings interesting!)

", "votes": null, "replies": [] }, { "id": 142572, "author_name": "godaibo", "author_url": "", "post_date": "11/03/2016 03:25:47", "content": "

I might be wrong but sniffing around the data, I found that there are over 1500+ records in the training data with label (1) that have identical values in the test data for all numeric and categorical variables....I repeat "IDENTICAL" values for every single numeric and categorical variable. I engineered some features based on this info and got 0.62+ MCC local cv!!! I was ecstatic "eat my dust", upon submission only ~0.40 on LB. What happened? I think there will be huge shake up in this competition. I see many Grand-masters at the top and they have a knack for hanging on....but I think there will be huge shake up "Santander style!"

", "votes": null, "replies": [] }, { "id": 142836, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/04/2016 19:59:53", "content": "

[quote=DavidGbodiOdaibo;142572]

\n\n

got 0.62+ MCC local cv!!! I was ecstatic "eat my dust", upon submission only ~0.40 on LB. What happened?

\n\n

[/quote]

\n\n

Have you checked whether information leaked from your training folds to your test folds in CV? A 0.62 CV seems almost too good to be true. If a person were creating features such as "What is the failure % for rows with exactly these feature values?" it would be easy for info to leak from training folds to test folds. Specifically, the "failure %" for such features would need to be calculated within the CV loop on just the training folds. Calculating such "failure %" features based on the whole set of training data before CV can lead to leakage.

", "votes": null, "replies": [] }, { "id": 142837, "author_name": "godaibo", "author_url": "", "post_date": "11/04/2016 20:28:59", "content": "

@breakfastpirate, I agree there probably was leakage....

", "votes": null, "replies": [] }, { "id": 142838, "author_name": "godaibo", "author_url": "", "post_date": "11/04/2016 20:33:50", "content": "

@breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :)

", "votes": null, "replies": [] }, { "id": 142840, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/04/2016 20:47:01", "content": "

[quote=DavidGbodiOdaibo;142838]

\n\n

@breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :)

\n\n

[/quote]

\n\n

There are other threads in the forum that discuss more details about these duplicates and how they might be useful.

", "votes": null, "replies": [] }, { "id": 143009, "author_name": "cpmpml", "author_url": "", "post_date": "11/06/2016 04:13:14", "content": "

There may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.

", "votes": null, "replies": [] }, { "id": 143015, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/06/2016 05:59:15", "content": "

[quote=CPMP;143009]

\n\n

There may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.

\n\n

[/quote]

\n\n

The problem with Santander was that people were adding a bunch of post-algorithm adjustments, and those were overfit to the the public LB. "The Script" seemed to feed this overfitting frenzy by crowdsourcing the addition of overfit post-algorithm adjustments. I agree that there is potential for overfit to the 300 or so "rows that matter" in the public LB for Bosch, but so far we have not seen a feeding frenzy of people suggesting post-algorithm adjustments in the forum/scripts.

\n\n

So I think there is certainly the possibility of significant local reshuffling, but I don't think we will see quite the global bloodbath we saw on Santander.

\n\n

I would say use both CV and LB feedback, but give a bigger weight to CV.

", "votes": null, "replies": [] }, { "id": 143045, "author_name": "sasrdw", "author_url": "", "post_date": "11/06/2016 13:09:48", "content": "

Yep, agree ~900 rows matter. With MCC my feedback tends to be LBPalm. Moe is the LB and I am Curly: https://www.youtube.com/watch?v=u4ZgVRJ-H8U

", "votes": null, "replies": [] }, { "id": 143071, "author_name": "godaibo", "author_url": "", "post_date": "11/06/2016 18:46:59", "content": "

there are 1183747 rows in train and 1183748 rows in test, ~6000 failures in train, yet how many 1's in submission shows some at the top of the leaderboard report 1800-2000 failures in their submissions. We have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.

", "votes": null, "replies": [] }, { "id": 143073, "author_name": "cpmpml", "author_url": "", "post_date": "11/06/2016 18:55:43", "content": "

mcc is such that it is better to have less true positive if you also drop the number of false positive drastically.

\n\n

For instance, our best submission (about 0.485 LB) has only 2672 1s in it.

\n\n

People with better LB reported similar rates.

\n\n

It also means that MCC may not be the best metric for getting good prediction when the data is so imbalanced. But that's not for us to judge...

", "votes": null, "replies": [] }, { "id": 143093, "author_name": "laurae2", "author_url": "", "post_date": "11/06/2016 20:27:35", "content": "

[quote=DavidGbodiOdaibo;143071]

\n\n

(...)

\n\n

We have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.

\n\n

[/quote]

\n\n

Because it is how MCC works: as we have many true negatives, the only objectives you have to optimize are the diagonals of the confusion matrix. If FP or FN is very small, then the other is larger. But the multiplication is smaller than if you balanced FP and FN (think of 0 x 1000 < 500 x 500). For most of us (if not predicting a lot of positives), it depends on how many true positives you have vs the amount of positives you predict. It's a chase for maximum true positive accuracy in your positive predictions, while being able to maximize also the amount of positive predictions (the two objectives to chase and to balance appropriately).

\n\n

If your features are tuned for a large amount of positives, you may be able to do as well as underpredicting the positive class. For instance, I have one submission with 5500+ positives, and yet it scores 0.4+ on LB (for MCC, it is about the same as just swapping FP and FN, which mathematically changes nearly nothing - just bigger numbers making improvement more difficult). You can do as well by predicting only 2000 positives, and get 0.4+ LB too. Obviously, if you predict 5500+, you lower the risk you take - but getting such amount of positives with a high score is not trivial as you need to catch most of them.

\n\n

The difference post-bias (= after substracting the bias between public and private LB) should be around 0.005-0.006, as it seems to be the standard deviation for 20% of data at around 0.5 MCC (so for 30% it's even smaller).

\n\n

This one will never be like the Santander one, unless the private LB is really much different (still a possibility, but a very low one). But then, this competition does not seem like the Seizure one with 3 patients only.

\n\n

Remember also that you may see MCC linear in the brain, but the improvement you need is quadratic (because the true negative count is so large). Going from 0.5 to 0.6 is significantly harder than from 0.2 to 0.3 (approximately 2.2x more difficult). Therefore, as you go higher in MCC, it becomes more difficult to see linear improvements (in the MCC value) than if you were at a low MCC. Add into the factor how hard it becomes to improve once so many good features were engineered, and you hit a wall for improvement which makes it look like a real sigmoid (from a human perspective of performance improvement against time):

\n\n

\"enter

\n\n

Therefore, if you have 75% true positives in your 2000 positives, it is clearly not trivial to go to 7500 positives and 93% true positives (about the same score). Let alone improving 93% true positives, it should be easier to increase from 75% than from 93%.

", "votes": null, "replies": [] }, { "id": 143381, "author_name": "gaborfodor", "author_url": "", "post_date": "11/08/2016 13:11:01", "content": "

Just waiting for the Shake-Up...\n\"enter\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code

", "votes": null, "replies": [] }, { "id": 143384, "author_name": "titericz", "author_url": "", "post_date": "11/08/2016 13:16:52", "content": "

Shake-Up Is Coming...

", "votes": null, "replies": [] }, { "id": 143385, "author_name": "anokas", "author_url": "", "post_date": "11/08/2016 13:27:16", "content": "

[quote=beluga;143381]

\n\n

Just waiting for the Shake-Up...\n\"enter\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code

\n\n

[/quote]

\n\n

This never fails to make me laugh :D

", "votes": null, "replies": [] }, { "id": 143551, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/09/2016 14:21:50", "content": "

Hopefully, LB shake-up on this competition will not be as big as the LB shake-up was in U.S. elections last night. I think the election was a Santander-level shake-up.

", "votes": null, "replies": [] }, { "id": 143673, "author_name": "", "author_url": "", "post_date": "11/10/2016 02:58:26", "content": "

SHAKE IT UP!\n44 hours of catnip watching LB Shake-up

\n\n

PS: Good one @BreakfastPirate :) and looking forward to hearing the stories of Road-to-0.5

", "votes": null, "replies": [] }, { "id": 143914, "author_name": "laurae2", "author_url": "", "post_date": "11/11/2016 11:35:09", "content": "

Mini-shake up probably incoming specifically there:

\n\n

\"enter

\n\n

This will be fun for the last 12 hours!

", "votes": null, "replies": [] }, { "id": 143916, "author_name": "darraghdog", "author_url": "", "post_date": "11/11/2016 11:40:55", "content": "

Best of luck all ! May the gods of overfitting be with you

", "votes": null, "replies": [] }, { "id": 143917, "author_name": "", "author_url": "", "post_date": "11/11/2016 11:55:32", "content": "

Good luck everyone. Let's the over-fitting be revealed.

", "votes": null, "replies": [] }, { "id": 143928, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/11/2016 12:55:50", "content": "

Here is the shake-up historical spectrum (with some recent competitions added). Any predictions of where this competition will fall along the spectrum?

\n\n

\"enter

", "votes": null, "replies": [] }, { "id": 143959, "author_name": "cpmpml", "author_url": "", "post_date": "11/11/2016 15:20:03", "content": "

[quote=Laurae;143914]

\n\n

Mini-shake up probably incoming specifically there:

\n\n

\"enter

\n\n

This will be fun for the last 12 hours!

\n\n

[/quote]

\n\n

Are you saying overfitting increases with team size? ;)

", "votes": null, "replies": [] }, { "id": 143983, "author_name": "laurae2", "author_url": "", "post_date": "11/11/2016 18:47:15", "content": "

[quote=CPMP;143959]

\n\n

Are you saying overfitting increases with team size? ;)

\n\n

[/quote]

\n\n

It's more like there are 4 there competing for 2 gold medal spots at that range. The private LB will "shuffle" and maybe none, some, or all of these 4 will be in top 12.

", "votes": null, "replies": [] }, { "id": 143993, "author_name": "happycube", "author_url": "", "post_date": "11/11/2016 20:36:04", "content": "

... I'd just like to keep a bronze medal - I ran out of ideas a while ago and I've been busy the last few weeks. :)

", "votes": null, "replies": [] }, { "id": 143997, "author_name": "ben519", "author_url": "", "post_date": "11/11/2016 21:11:29", "content": "

Damn, my team might get that dreaded top Silver spot. I should've gone to church last Sunday...

", "votes": null, "replies": [] }, { "id": 144002, "author_name": "alanwuxm", "author_url": "", "post_date": "11/11/2016 21:29:39", "content": "

I think a shake-up within 0.02 is expected.

", "votes": null, "replies": [] }, { "id": 144003, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/11/2016 21:34:08", "content": "

Team "Make dishwashers great again" ... are you hoping your slogan will lead to a positive shake-up surprise?

", "votes": null, "replies": [] }, { "id": 144007, "author_name": "girmak", "author_url": "", "post_date": "11/11/2016 22:12:58", "content": "

Country girl shake it for me ...\nhttps://www.youtube.com/watch?v=pEUCuX5cFRA

\n\n

Enjoy and good luck to everybody!

", "votes": null, "replies": [] }, { "id": 144008, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/11/2016 22:17:04", "content": "

When you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s

", "votes": null, "replies": [] }, { "id": 144011, "author_name": "ben519", "author_url": "", "post_date": "11/11/2016 22:42:30", "content": "

Any teams excluding their best public LB submission from their final subs?

", "votes": null, "replies": [] }, { "id": 144012, "author_name": "anokas", "author_url": "", "post_date": "11/11/2016 22:44:57", "content": "

[quote=BreakfastPirate;144003]

\n\n

Team "Make dishwashers great again" ... are you hoping your slogan will lead to a positive shake-up surprise?

\n\n

[/quote]

\n\n

We know dishwashers, we have the best dishwashers. People always tell me, "anokas, you have the most beautiful dishwashers."

", "votes": null, "replies": [] }, { "id": 144013, "author_name": "raddar", "author_url": "", "post_date": "11/11/2016 22:48:40", "content": "

we are selecting our best LB model, but i am quite certain it will fail (we just dont have anything else to select, lol); second is more safe pick ~0.513. anyway, shakeup will only be in ~0.005 range, so top3 are safe with their position, I think :)

", "votes": null, "replies": [] }, { "id": 144023, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/12/2016 00:04:57", "content": "

Shake-up is fairly low: 0.015 overall and 0.011 for the Top 10%. So in the neighborhood of Caterpillar Tube Pricing.

", "votes": null, "replies": [] }, { "id": 144024, "author_name": "raddar", "author_url": "", "post_date": "11/12/2016 00:06:09", "content": "

our score ended up identical to what we got on CV. so fairly stable competition :) gratz to the winners

", "votes": null, "replies": [] }, { "id": 144026, "author_name": "gaborfodor", "author_url": "", "post_date": "11/12/2016 00:13:33", "content": "

We did not select our best public model we chose 0.52429 over 0.52448.

\n\n

EDIT: Our best public model would have been even better (0.526 private mcc). I was the one who suggested that the second best would be more stable. Fortunately it did not matter :)

", "votes": null, "replies": [] }, { "id": 144027, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/12/2016 00:13:47", "content": "

Of our two solutions, the one with the lower public LB score (and fewer 1s and higher CV score) got better on the private LB.

", "votes": null, "replies": [] }, { "id": 144050, "author_name": "", "author_url": "", "post_date": "11/12/2016 00:55:45", "content": "

Will you shake us up with your Road-to-0.52 bedtime story?

\n\n

How many ones did your have?

", "votes": null, "replies": [] }, { "id": 144060, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/12/2016 02:21:10", "content": "

Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.

", "votes": null, "replies": [] }, { "id": 144070, "author_name": "", "author_url": "", "post_date": "11/12/2016 03:39:35", "content": "

Thanks BreakfastPirate :)

", "votes": null, "replies": [] }, { "id": 144091, "author_name": "psilogram", "author_url": "", "post_date": "11/12/2016 09:30:05", "content": "

[quote=BreakfastPirate;144060]

\n\n

Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.

\n\n

[/quote]

\n\n

We followed the same strategy, and our submission with fewer 1s (2629) also scored better on the PB than the one with more 1s (3495) even though the sub with more 1s scored significantly better on the LB. This was surprising.

", "votes": null, "replies": [] }, { "id": 144418, "author_name": "tangweipku", "author_url": "", "post_date": "11/14/2016 07:15:12", "content": "

[quote=BreakfastPirate;144060]

\n\n

Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.

\n\n

[/quote]

\n\n

Just curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..

", "votes": null, "replies": [] }, { "id": 144507, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/14/2016 12:50:00", "content": "

[quote=Wei Tang;144418]

\n\n

Just curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..

\n\n

[/quote]

\n\n

One way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the "My Submissions" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.

", "votes": null, "replies": [] }, { "id": 144605, "author_name": "tangweipku", "author_url": "", "post_date": "11/14/2016 19:46:30", "content": "

[quote=BreakfastPirate;144507]

\n\n

[quote=Wei Tang;144418]

\n\n

Just curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..

\n\n

[/quote]

\n\n

One way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the "My Submissions" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.

\n\n

[/quote]

\n\n

Thanks a lot BreakfastPirate! I did not know that there is this nice feature of post-competition submissions.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "142204": "Too early to discuss shake-up? This would be my thoughts:\r\n\r\n•\tThe dataset has a large number of rows. This would seem to indicate a small shake-up.\r\n\r\n•\tThe train/test split is random, and the public/private split is random. This would seem to indicate to a small shake-up.\r\n\r\n•\tThe ratio of positives to negatives is very skewed. This would seem to indicate a non-small shake-up because although the overall number of rows is large, the number of positives is fairly small, and it would be easy to overfit to those few positives.\r\n\r\n•\tThe scores on the public leaderboard are moderately close. Currently 0.028 separates #1 from #20.\r\n\r\n•\tThe MCC evaluation function seems to me to lend itself to overfit. It is very sensitive to a small number of “borderline” rows. Many rows in the dataset are clearly negatives. Some rows in the dataset are clearly positives. Then there are the “borderline” rows. So although we started with a dataset that has a large number of rows, we end up with only a few rows that “matter”. My estimate is that there are fewer than 1000 rows that matter. So that’s actually a pretty small dataset. Let me clarify what I mean by “matter”. Lots and lots of rows matter for training, but really only a few matter when calculating MCC on the test set (for all practical purposes).\r\n\r\n\r\n\r\nSo we start with a large number of rows, but we end up with only 1000 or so rows that matter. Public LB is based on 30% of the data, therefore public LB is based on about 300 rows that matter.\r\n\r\n\r\n\r\nSo all of that to say that I don’t expect a huge, global shake-up, but it is very possible that we could see some people moving up by 0.01 to 0.02 while others move down by 0.01 to 0.02. So it is possible that there might be a significant reshuffling of the Top 20.\r\n\r\n\r\n\r\nWhat do you think? How many test rows \"matter\"?", "142210": "Hi Steve, I have the same feeling about shakeup. But maybe a +-0.01MCC is too optimistic, sure it will be much more ;-P", "142412": "Yeah - and some people've reported a bigger CV<->LB gap than others, it's quite possible the ones with lower CV will drop. (Or they might not. That's part of what keeps the endings interesting!)", "142572": "I might be wrong but sniffing around the data, I found that there are over 1500+ records in the training data with label (1) that have identical values in the test data for all numeric and categorical variables....I repeat \"IDENTICAL\" values for every single numeric and categorical variable. I engineered some features based on this info and got 0.62+ MCC local cv!!! I was ecstatic \"eat my dust\", upon submission only ~0.40 on LB. What happened? I think there will be huge shake up in this competition. I see many Grand-masters at the top and they have a knack for hanging on....but I think there will be huge shake up \"Santander style!\"", "142836": "[quote=DavidGbodiOdaibo;142572]\r\n\r\ngot 0.62+ MCC local cv!!! I was ecstatic \"eat my dust\", upon submission only ~0.40 on LB. What happened? \r\n\r\n[/quote]\r\n\r\nHave you checked whether information leaked from your training folds to your test folds in CV? A 0.62 CV seems almost too good to be true. If a person were creating features such as \"What is the failure % for rows with exactly these feature values?\" it would be easy for info to leak from training folds to test folds. Specifically, the \"failure %\" for such features would need to be calculated within the CV loop on just the training folds. Calculating such \"failure %\" features based on the whole set of training data before CV can lead to leakage.", "142837": "breakfastpirate, I agree there probably was leakage....", "142838": "breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :)", "142840": "[quote=DavidGbodiOdaibo;142838]\r\n\r\n@breafastpirate I still believe these duplicates are the reason for the high scores at the top of the leaderboard, I have not been able to exploit it, obviously you have :) \r\n\r\n[/quote]\r\n\r\nThere are other threads in the forum that discuss more details about these duplicates and how they might be useful.", "143009": "There may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.", "143015": "[quote=CPMP;143009]\r\n\r\nThere may be some bad surprise for those who use LB score to select settings. I have been burned on Santander by that, only using local cv for now.\r\n\r\n[/quote]\r\n\r\nThe problem with Santander was that people were adding a bunch of post-algorithm adjustments, and those were overfit to the the public LB. \"The Script\" seemed to feed this overfitting frenzy by crowdsourcing the addition of overfit post-algorithm adjustments. I agree that there is potential for overfit to the 300 or so \"rows that matter\" in the public LB for Bosch, but so far we have not seen a feeding frenzy of people suggesting post-algorithm adjustments in the forum/scripts.\r\n\r\nSo I think there is certainly the possibility of significant local reshuffling, but I don't think we will see quite the global bloodbath we saw on Santander.\r\n\r\nI would say use both CV and LB feedback, but give a bigger weight to CV.", "143045": "Yep, agree ~900 rows matter. With MCC my feedback tends to be LBPalm. Moe is the LB and I am Curly: https://www.youtube.com/watch?v=u4ZgVRJ-H8U", "143071": "there are 1183747 rows in train and 1183748 rows in test, ~6000 failures in train, yet [how many 1's in submission][1] shows some at the top of the leaderboard report 1800-2000 failures in their submissions. We have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24083/how-many-1-s-in-your-submission", "143073": "mcc is such that it is better to have less true positive if you also drop the number of false positive drastically.\r\n\r\nFor instance, our best submission (about 0.485 LB) has only 2672 1s in it.\r\n\r\nPeople with better LB reported similar rates.\r\n\r\nIt also means that MCC may not be the best metric for getting good prediction when the data is so imbalanced. But that's not for us to judge...", "143093": "[quote=DavidGbodiOdaibo;143071]\r\n\r\n(...)\r\n\r\nWe have almost identical number of train and test records, so one can assume if the test train split was random your models should predict ~6000 failures in test as well and not 1800 that give optimal LB rank. Why isn't anyone concerned about this. This is just more evidence for massive shakeup.\r\n\r\n[/quote]\r\n\r\nBecause it is how MCC works: as we have many true negatives, the only objectives you have to optimize are the [diagonals of the confusion matrix][1]. If FP or FN is very small, then the other is larger. But the multiplication is smaller than if you balanced FP and FN (think of 0 x 1000 < 500 x 500). For most of us (if not predicting a lot of positives), it depends on how many true positives you have vs the amount of positives you predict. It's a chase for maximum true positive accuracy in your positive predictions, while being able to maximize also the amount of positive predictions (the two objectives to chase and to balance appropriately).\r\n\r\nIf your features are tuned for a large amount of positives, you may be able to do as well as underpredicting the positive class. For instance, I have one submission with 5500+ positives, and yet it scores 0.4+ on LB (for MCC, it is *about the same as just swapping FP and FN*, which mathematically changes nearly nothing - just bigger numbers making improvement more difficult). You can do as well by predicting only 2000 positives, and get 0.4+ LB too. Obviously, if you predict 5500+, you lower the risk you take - but getting such amount of positives with a high score is not trivial as you need to catch most of them.\r\n\r\nThe difference post-bias (= after substracting the bias between public and private LB) should be around 0.005-0.006, as it seems to be the standard deviation for 20% of data at around 0.5 MCC (so for 30% it's even smaller).\r\n\r\nThis one will never be like the Santander one, unless the private LB is really much different (still a possibility, but a very low one). But then, this competition does not seem like [the Seizure one][2] with 3 patients only.\r\n\r\nRemember also that you may see MCC linear in the brain, but the improvement you need is quadratic (because the true negative count is so large). Going from 0.5 to 0.6 is significantly harder than from 0.2 to 0.3 (approximately 2.2x more difficult). Therefore, as you go higher in MCC, it becomes more difficult to see linear improvements (in the MCC value) than if you were at a low MCC. Add into the factor how hard it becomes to improve once so many good features were engineered, and you hit a wall for improvement which makes it look like a real sigmoid (from a human perspective of performance improvement against time):\r\n\r\n![enter image description here][3]\r\n\r\nTherefore, if you have 75% true positives in your 2000 positives, it is clearly not trivial to go to 7500 positives and 93% true positives (about the same score). Let alone improving 93% true positives, it should be easier to increase from 75% than from 93%.\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24022/train-test-event-rate-imbalance/137512#post137512\r\n [2]: https://www.kaggle.com/c/melbourne-university-seizure-prediction/leaderboard\r\n [3]: http://i.imgur.com/ROYk0Zt.png", "143381": "Just waiting for the Shake-Up...\r\n![enter image description here][1]\r\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code\r\n\r\n\r\n [1]: https://www.kaggle.io/svf/434942/82a01d76067d3b0bd2fc8fda31fe16c8/last_week_xkcd.png", "143384": "Shake-Up Is Coming...", "143385": "[quote=beluga;143381]\r\n\r\nJust waiting for the Shake-Up...\r\n![enter image description here][1]\r\nhttps://www.kaggle.com/gaborfodor/bosch-production-line-performance/last-week/code\r\n\r\n\r\n [1]: https://www.kaggle.io/svf/434942/82a01d76067d3b0bd2fc8fda31fe16c8/last_week_xkcd.png\r\n\r\n[/quote]\r\n\r\nThis never fails to make me laugh :D", "143551": "Hopefully, LB shake-up on this competition will not be as big as the LB shake-up was in U.S. elections last night. I think the election was a Santander-level shake-up.", "143673": "SHAKE IT UP!\r\n44 hours of catnip watching LB Shake-up\r\n\r\nPS: Good one @BreakfastPirate :) and looking forward to hearing the stories of Road-to-0.5", "143914": "Mini-shake up probably incoming specifically there:\r\n\r\n![enter image description here][1]\r\n\r\nThis will be fun for the last 12 hours!\r\n\r\n [1]: http://i.imgur.com/ZXKIlfi.png", "143916": "Best of luck all ! May the gods of overfitting be with you", "143917": "Good luck everyone. Let's the over-fitting be revealed.", "143928": "Here is the shake-up historical spectrum (with some recent competitions added). Any predictions of where this competition will fall along the spectrum?\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/143928/5336/Shakeup20161111.jpg?sv=2012-02-12&se=2016-11-14T12%3A56%3A00Z&sr=b&sp=r&sig=TE4SBSiPFyQgQO6b2EQRA1g4AZFzV%2FRSH7bBsplXpJo%3D", "143959": "[quote=Laurae;143914]\r\n\r\nMini-shake up probably incoming specifically there:\r\n\r\n![enter image description here][1]\r\n\r\nThis will be fun for the last 12 hours!\r\n\r\n [1]: http://i.imgur.com/ZXKIlfi.png\r\n\r\n[/quote]\r\n\r\n\r\nAre you saying overfitting increases with team size? ;)", "143983": "[quote=CPMP;143959]\r\n\r\nAre you saying overfitting increases with team size? ;)\r\n\r\n[/quote]\r\n\r\nIt's more like there are 4 there competing for 2 gold medal spots at that range. The private LB will \"shuffle\" and maybe none, some, or all of these 4 will be in top 12.", "143993": "... I'd just like to keep a bronze medal - I ran out of ideas a while ago and I've been busy the last few weeks. :)", "143997": "Damn, my team might get that dreaded top Silver spot. I should've gone to church last Sunday...", "144002": "I think a shake-up within 0.02 is expected.", "144003": "Team \"Make dishwashers great again\" ... are you hoping your slogan will lead to a positive shake-up surprise?", "144007": "Country girl shake it for me ...\r\nhttps://www.youtube.com/watch?v=pEUCuX5cFRA\r\n\r\nEnjoy and good luck to everybody!", "144008": "When you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s", "144011": "Any teams excluding their best public LB submission from their final subs?", "144012": "[quote=BreakfastPirate;144003]\r\n\r\nTeam \"Make dishwashers great again\" ... are you hoping your slogan will lead to a positive shake-up surprise?\r\n\r\n[/quote]\r\n\r\nWe know dishwashers, we have the best dishwashers. People always tell me, \"anokas, you have the most beautiful dishwashers.\"", "144013": "we are selecting our best LB model, but i am quite certain it will fail (we just dont have anything else to select, lol); second is more safe pick ~0.513. anyway, shakeup will only be in ~0.005 range, so top3 are safe with their position, I think :)", "144023": "Shake-up is fairly low: 0.015 overall and 0.011 for the Top 10%. So in the neighborhood of Caterpillar Tube Pricing.", "144024": "our score ended up identical to what we got on CV. so fairly stable competition :) gratz to the winners", "144026": "We did not select our best public model we chose 0.52429 over 0.52448.\r\n\r\n**EDIT:** Our best public model would have been even better (0.526 private mcc). I was the one who suggested that the second best would be more stable. Fortunately it did not matter :)", "144027": "Of our two solutions, the one with the lower public LB score (and fewer 1s and higher CV score) got better on the private LB.", "144050": "Will you shake us up with your Road-to-0.52 bedtime story?\r\n\r\nHow many ones did your have?", "144060": "Our submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.", "144070": "Thanks BreakfastPirate :)", "144091": "[quote=BreakfastPirate;144060]\r\n\r\nOur submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.\r\n\r\n[/quote]\r\n\r\nWe followed the same strategy, and our submission with fewer 1s (2629) also scored better on the PB than the one with more 1s (3495) even though the sub with more 1s scored significantly better on the LB. This was surprising.", "144418": "[quote=BreakfastPirate;144060]\r\n\r\nOur submission with 2848 1s got 0.51116 on private LB. Our second submission with 3465 1s got about 0.507XX. We tried one with more 1s, and the other with fewer 1s to make sure we had all of our bases covered.\r\n\r\n[/quote]\r\n\r\nJust curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..", "144507": "[quote=Wei Tang;144418]\r\n\r\nJust curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..\r\n\r\n[/quote]\r\n\r\nOne way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the \"My Submissions\" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.", "144605": "[quote=BreakfastPirate;144507]\r\n\r\n[quote=Wei Tang;144418]\r\n\r\nJust curious how do you find out the private LB score for each of the final two submission? I have looked hard in the dashboard menu but could not find such option..\r\n\r\n[/quote]\r\n\r\nOne way is to resubmit your two submissions as post-competition submissions. It will show you their private LB score. Eventually, the \"My Submissions\" page will show both public and private scores for all of your submissions, but that usually takes a few days before that is available.\r\n\r\n[/quote]\r\n\r\n\r\nThanks a lot BreakfastPirate! I did not know that there is this nice feature of post-competition submissions." }, "source": "meta" }