{ "id": 80944, "title": "Trust CV or LB in ELO?", "url": "/competitions/elo-merchant-category-recommendation/discussion/80944", "author_name": "Neo Zhao", "post_date": "2019-02-18T05:09:31.267000", "votes": 9, "comment_count": 19, "views": 0, "content": "

sometimes i hit 3.688 lb based on 3.64x cv,\nbut sometimes i hit 3.681 lb based on 3.651 cv....

", "messages": [ { "id": 473497, "postDate": "2019-02-18T05:09:31.267Z", "content": "

sometimes i hit 3.688 lb based on 3.64x cv,\nbut sometimes i hit 3.681 lb based on 3.651 cv....

", "rawMarkdown": "sometimes i hit 3.688 lb based on 3.64x cv,\nbut sometimes i hit 3.681 lb based on 3.651 cv....\n ", "votes": 9 }, { "id": 475462, "postDate": "2019-02-20T19:24:02.050Z", "content": "

I think LB too small correlate with local CV.\nWe hane 2200 -33 value - its ~1% of data and if you correct mark this row you can get 1.7 score !! Only 1% and so much influence on score. For LB we have ~ 700 outlier row with huge power on score.

", "rawMarkdown": "I think LB too small correlate with local CV.\nWe hane 2200 -33 value - its ~1% of data and if you correct mark this row you can get 1.7 score !! Only 1% and so much influence on score. For LB we have ~ 700 outlier row with huge power on score.", "votes": 1, "replies": [ { "id": 475604, "postDate": "2019-02-21T01:21:44.927Z", "content": "

agree with u

", "rawMarkdown": "agree with u" } ] }, { "id": 473908, "postDate": "2019-02-18T17:05:15.857Z", "content": "

@NeoZhao have a look at similar discussion topic by a grandmaster in a previous competition https://www.kaggle.com/c/homesite-quote-conversion/discussion/18287#103892 . You should consider the CV strategy used as well not just the CV score. Since you have not mentioned your CV strategy I would prefer model with higher LB over CV scores. Hope it helps.

\n\n

Happy Kaggling :)

", "rawMarkdown": "@NeoZhao have a look at similar discussion topic by a grandmaster in a previous competition https://www.kaggle.com/c/homesite-quote-conversion/discussion/18287#103892 . You should consider the CV strategy used as well not just the CV score. Since you have not mentioned your CV strategy I would prefer model with higher LB over CV scores. Hope it helps.\n\nHappy Kaggling :)", "votes": 1, "replies": [ { "id": 474136, "postDate": "2019-02-19T01:10:33.173Z", "content": "

@Aman.cyberia Thanks a lot

", "rawMarkdown": "@Aman.cyberia Thanks a lot" } ] }, { "id": 473664, "postDate": "2019-02-18T10:39:38.913Z", "content": "

you have two submissions.

", "rawMarkdown": "you have two submissions.", "votes": 1, "replies": [ { "id": 473698, "postDate": "2019-02-18T11:51:54.893Z", "content": "

haha

", "rawMarkdown": "haha" } ] }, { "id": 474048, "postDate": "2019-02-18T21:32:17.900Z", "content": "

This competition is driving me crazy. \nCV is improving but not LB, eventhough stacking madness. \nI'm looking at correlation between out of fold stacker predictions, and it seems linear. \nI guess it'll all come down to how many outliers there are in private leaderboard.

\n\n

Correctly predicting one outlier bu manual processing has a huge effect on private LB, which is only 30% of 130k rows.

", "rawMarkdown": "This competition is driving me crazy. \nCV is improving but not LB, eventhough stacking madness. \nI'm looking at correlation between out of fold stacker predictions, and it seems linear. \nI guess it'll all come down to how many outliers there are in private leaderboard. \n\nCorrectly predicting one outlier bu manual processing has a huge effect on private LB, which is only 30% of 130k rows. ", "votes": 2, "replies": [ { "id": 474137, "postDate": "2019-02-19T01:12:17.420Z", "content": "

agree with u, i think the influence of outlier over than the gap between cv and lb

", "rawMarkdown": "agree with u, i think the influence of outlier over than the gap between cv and lb", "votes": 1 } ] }, { "id": 473549, "postDate": "2019-02-18T07:08:34.220Z", "content": "

Trust yourself.

", "rawMarkdown": "Trust yourself.", "votes": 1, "replies": [ { "id": 473557, "postDate": "2019-02-18T07:17:25.800Z", "content": "

haha thanks

", "rawMarkdown": "haha thanks" } ] }, { "id": 474962, "postDate": "2019-02-20T03:54:29.867Z", "content": "

I spent a few days doing feature engineering, which improved my cv score, but dropped the LB score. Maybe some features are too complicated for me to do.

", "rawMarkdown": "I spent a few days doing feature engineering, which improved my cv score, but dropped the LB score. Maybe some features are too complicated for me to do." }, { "id": 473809, "postDate": "2019-02-18T14:51:12.823Z", "content": "

the LB based to 30% in test data , so trust at your CV

", "rawMarkdown": "the LB based to 30% in test data , so trust at your CV" }, { "id": 473612, "postDate": "2019-02-18T09:16:47.507Z", "content": "

considering the type of problem here, i am bit confused. We are probably all looking for the outliers at this moment.

", "rawMarkdown": "considering the type of problem here, i am bit confused. We are probably all looking for the outliers at this moment. ", "replies": [ { "id": 473628, "postDate": "2019-02-18T09:43:05.453Z", "content": "

We are all looking for the outliers and we will never overfitting alone haha

", "rawMarkdown": "We are all looking for the outliers and we will never overfitting alone haha" }, { "id": 473832, "postDate": "2019-02-18T15:19:22.807Z", "content": "

I am not sure about the tops.but seems like they are also exploring the outliers..

", "rawMarkdown": "I am not sure about the tops.but seems like they are also exploring the outliers.." } ] }, { "id": 473535, "postDate": "2019-02-18T06:38:29.287Z", "content": "

mostly competition my lb will improved if my cv improved a lot , but in ELO, my cv improved 0.004 but lb decrease 0.008......have anyone can explain for me?

", "rawMarkdown": "mostly competition my lb will improved if my cv improved a lot , but in ELO, my cv improved 0.004 but lb decrease 0.008......have anyone can explain for me?", "replies": [ { "id": 473897, "postDate": "2019-02-18T16:52:15.287Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 473508, "postDate": "2019-02-18T05:36:26.433Z", "content": "

me too, but My CV 3.652x,LB 3.691.... so painful

", "rawMarkdown": "me too, but My CV 3.652x,LB 3.691.... so painful" }, { "id": 473506, "postDate": "2019-02-18T05:29:48.533Z", "content": "

me too, but if your cv get much improve then lb will get improve if target wasn't leaked. our team currenty cv 3.62X , lb 3.66X。

", "rawMarkdown": "me too, but if your cv get much improve then lb will get improve if target wasn't leaked. our team currenty cv 3.62X , lb 3.66X。" } ], "comments": [ { "id": 475462, "author_name": "Leigh", "author_url": "", "post_date": "2019-02-20T19:24:02.050000", "content": "

I think LB too small correlate with local CV.\nWe hane 2200 -33 value - its ~1% of data and if you correct mark this row you can get 1.7 score !! Only 1% and so much influence on score. For LB we have ~ 700 outlier row with huge power on score.

", "votes": 1, "replies": [ { "id": 475604, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-21T01:21:44.927000", "content": "

agree with u

", "votes": 0, "replies": [] } ] }, { "id": 473908, "author_name": "cyberia", "author_url": "", "post_date": "2019-02-18T17:05:15.857000", "content": "

@NeoZhao have a look at similar discussion topic by a grandmaster in a previous competition https://www.kaggle.com/c/homesite-quote-conversion/discussion/18287#103892 . You should consider the CV strategy used as well not just the CV score. Since you have not mentioned your CV strategy I would prefer model with higher LB over CV scores. Hope it helps.

\n\n

Happy Kaggling :)

", "votes": 1, "replies": [ { "id": 474136, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-19T01:10:33.173000", "content": "

@Aman.cyberia Thanks a lot

", "votes": 0, "replies": [] } ] }, { "id": 473664, "author_name": "30CrMnSiA", "author_url": "", "post_date": "2019-02-18T10:39:38.913000", "content": "

you have two submissions.

", "votes": 1, "replies": [ { "id": 473698, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-18T11:51:54.893000", "content": "

haha

", "votes": 0, "replies": [] } ] }, { "id": 474048, "author_name": "Jacques Peeters", "author_url": "", "post_date": "2019-02-18T21:32:17.900000", "content": "

This competition is driving me crazy. \nCV is improving but not LB, eventhough stacking madness. \nI'm looking at correlation between out of fold stacker predictions, and it seems linear. \nI guess it'll all come down to how many outliers there are in private leaderboard.

\n\n

Correctly predicting one outlier bu manual processing has a huge effect on private LB, which is only 30% of 130k rows.

", "votes": 2, "replies": [ { "id": 474137, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-19T01:12:17.420000", "content": "

agree with u, i think the influence of outlier over than the gap between cv and lb

", "votes": 1, "replies": [] } ] }, { "id": 473549, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2019-02-18T07:08:34.220000", "content": "

Trust yourself.

", "votes": 1, "replies": [ { "id": 473557, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-18T07:17:25.800000", "content": "

haha thanks

", "votes": 0, "replies": [] } ] }, { "id": 474962, "author_name": "Guang Yang", "author_url": "", "post_date": "2019-02-20T03:54:29.867000", "content": "

I spent a few days doing feature engineering, which improved my cv score, but dropped the LB score. Maybe some features are too complicated for me to do.

", "votes": 0, "replies": [] }, { "id": 473809, "author_name": "Abdellatif", "author_url": "", "post_date": "2019-02-18T14:51:12.823000", "content": "

the LB based to 30% in test data , so trust at your CV

", "votes": 0, "replies": [] }, { "id": 473612, "author_name": "SubikashPal", "author_url": "", "post_date": "2019-02-18T09:16:47.507000", "content": "

considering the type of problem here, i am bit confused. We are probably all looking for the outliers at this moment.

", "votes": 0, "replies": [ { "id": 473628, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-18T09:43:05.453000", "content": "

We are all looking for the outliers and we will never overfitting alone haha

", "votes": 0, "replies": [] }, { "id": 473832, "author_name": "SubikashPal", "author_url": "", "post_date": "2019-02-18T15:19:22.807000", "content": "

I am not sure about the tops.but seems like they are also exploring the outliers..

", "votes": 0, "replies": [] } ] }, { "id": 473535, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-18T06:38:29.287000", "content": "

mostly competition my lb will improved if my cv improved a lot , but in ELO, my cv improved 0.004 but lb decrease 0.008......have anyone can explain for me?

", "votes": 0, "replies": [ { "id": 473897, "author_name": "", "author_url": "", "post_date": "2019-02-18T16:52:15.287000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 473508, "author_name": "Kang0709", "author_url": "", "post_date": "2019-02-18T05:36:26.433000", "content": "

me too, but My CV 3.652x,LB 3.691.... so painful

", "votes": 0, "replies": [] }, { "id": 473506, "author_name": "AIFIRST", "author_url": "", "post_date": "2019-02-18T05:29:48.533000", "content": "

me too, but if your cv get much improve then lb will get improve if target wasn't leaked. our team currenty cv 3.62X , lb 3.66X。

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "473497": "sometimes i hit 3.688 lb based on 3.64x cv,\nbut sometimes i hit 3.681 lb based on 3.651 cv....\n ", "475462": "I think LB too small correlate with local CV.\nWe hane 2200 -33 value - its ~1% of data and if you correct mark this row you can get 1.7 score !! Only 1% and so much influence on score. For LB we have ~ 700 outlier row with huge power on score.", "473908": "@NeoZhao have a look at similar discussion topic by a grandmaster in a previous competition https://www.kaggle.com/c/homesite-quote-conversion/discussion/18287#103892 . You should consider the CV strategy used as well not just the CV score. Since you have not mentioned your CV strategy I would prefer model with higher LB over CV scores. Hope it helps.\n\nHappy Kaggling :)", "473664": "you have two submissions.", "474048": "This competition is driving me crazy. \nCV is improving but not LB, eventhough stacking madness. \nI'm looking at correlation between out of fold stacker predictions, and it seems linear. \nI guess it'll all come down to how many outliers there are in private leaderboard. \n\nCorrectly predicting one outlier bu manual processing has a huge effect on private LB, which is only 30% of 130k rows. ", "473549": "Trust yourself.", "474962": "I spent a few days doing feature engineering, which improved my cv score, but dropped the LB score. Maybe some features are too complicated for me to do.", "473809": "the LB based to 30% in test data , so trust at your CV", "473612": "considering the type of problem here, i am bit confused. We are probably all looking for the outliers at this moment. ", "473535": "mostly competition my lb will improved if my cv improved a lot , but in ELO, my cv improved 0.004 but lb decrease 0.008......have anyone can explain for me?", "473508": "me too, but My CV 3.652x,LB 3.691.... so painful", "473506": "me too, but if your cv get much improve then lb will get improve if target wasn't leaked. our team currenty cv 3.62X , lb 3.66X。" } }