Here are two of my results, and I am using the same train-valid split：\n1. CV:3.65*7*04289 LB:3.69*6*\n2. CV:3.65*5*87777 LB:3.69*8* (adding more feature on 1)
\n\nI have observed inconsistency of CV and LB (i.e. lower CV but get higher LB), which makes it hard to evaluate models.\nDo you have similar issue? If so, how do you handle this?
", "messages": [ { "id": 456561, "postDate": "2019-01-16T03:56:17.147Z", "content": "Here are two of my results, and I am using the same train-valid split：\n1. CV:3.65*7*04289 LB:3.69*6*\n2. CV:3.65*5*87777 LB:3.69*8* (adding more feature on 1)
\n\nI have observed inconsistency of CV and LB (i.e. lower CV but get higher LB), which makes it hard to evaluate models.\nDo you have similar issue? If so, how do you handle this?
", "rawMarkdown": "Here are two of my results, and I am using the same train-valid split：\n1. CV:3.65**7**04289\t LB:3.69**6**\n2. CV:3.65**5**87777\t LB:3.69**8** (adding more feature on 1)\n\nI have observed inconsistency of CV and LB (i.e. lower CV but get higher LB), which makes it hard to evaluate models.\nDo you have similar issue? If so, how do you handle this? ", "votes": 3 }, { "id": 456631, "postDate": "2019-01-16T08:13:29.583Z", "content": "I think you meet \"null importance\" feature, because any feature could have prob to lower local CV.\nYou can look at this kernel https://www.kaggle.com/ogrellier/feature-selection-with-null-importances
", "rawMarkdown": "I think you meet \"null importance\" feature, because any feature could have prob to lower local CV.\nYou can look at this kernel https://www.kaggle.com/ogrellier/feature-selection-with-null-importances", "votes": 1, "replies": [ { "id": 456659, "postDate": "2019-01-16T08:54:00.070Z", "content": "thanks.
", "rawMarkdown": "thanks." } ] }, { "id": 456668, "postDate": "2019-01-16T09:06:53.887Z", "content": "I think it the case due to huge variance in CV score. If your improvement is larger than given variance than you are more likely get higher score. Increase number of number of folds , and get your own test set. Try to check on test set first before submission.
", "rawMarkdown": "I think it the case due to huge variance in CV score. If your improvement is larger than given variance than you are more likely get higher score. Increase number of number of folds , and get your own test set. Try to check on test set first before submission.", "votes": 2 }, { "id": 456766, "postDate": "2019-01-16T13:48:45.480Z", "content": "Thanks for bringing this out. i had come across same issue
\n\nCV 3.64936 (lgb 5 fold) -->> 3.689 LB\nCV 3.64753 (lgb 5 fold) -->> 3.691 LB
\n\nI am confused should i improve the model further
", "rawMarkdown": "Thanks for bringing this out. i had come across same issue\n\nCV 3.64936 (lgb 5 fold) -->> 3.689 LB\nCV 3.64753 (lgb 5 fold) -->> 3.691 LB\n\nI am confused should i improve the model further \n", "replies": [ { "id": 457131, "postDate": "2019-01-17T01:48:47.233Z", "content": "I think single model for 3.689 is great.
", "rawMarkdown": "I think single model for 3.689 is great." }, { "id": 457180, "postDate": "2019-01-17T03:46:23.720Z", "content": "Well, i am not quite sure that stacking and blending will help how much
", "rawMarkdown": "Well, i am not quite sure that stacking and blending will help how much" } ] }, { "id": 456652, "postDate": "2019-01-16T08:40:50.683Z", "content": "Actually, this competition has many incosistencies on data explanation. I believe that nobody has understood what we are exactly trying to predict.
\n\nActually the 0.002 difference on LB is non significant, concerning that it is evaluated by the 30% of the total submission file
", "rawMarkdown": "Actually, this competition has many incosistencies on data explanation. I believe that nobody has understood what we are exactly trying to predict.\n\nActually the 0.002 difference on LB is non significant, concerning that it is evaluated by the 30% of the total submission file", "replies": [ { "id": 456660, "postDate": "2019-01-16T08:56:59.433Z", "content": "emm, the key thing is that: we do not know how to evaluate models if we do not care this 0.002 difference
", "rawMarkdown": "emm, the key thing is that: we do not know how to evaluate models if we do not care this 0.002 difference" }, { "id": 456670, "postDate": "2019-01-16T09:08:59.397Z", "content": "the history of kaggle competitions has shown that you must only care about your CV and non for the LB as soon as the LB is calculated on a small portion. So, in your case, the 2nd model is an improvement of your 1st, even for this slight rmse decrease
", "rawMarkdown": "the history of kaggle competitions has shown that you must only care about your CV and non for the LB as soon as the LB is calculated on a small portion. So, in your case, the 2nd model is an improvement of your 1st, even for this slight rmse decrease", "votes": 1 }, { "id": 456683, "postDate": "2019-01-16T09:48:12.533Z", "content": "Thanks,\nI think I know what you mean: trust local cv, not public LB.\nLet's see the final private LB on this two models.
", "rawMarkdown": "Thanks,\nI think I know what you mean: trust local cv, not public LB.\nLet's see the final private LB on this two models.", "votes": 1 }, { "id": 457359, "postDate": "2019-01-17T09:21:48.010Z", "content": "sometimes it can be really confusing what to trust
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77295
", "rawMarkdown": "sometimes it can be really confusing what to trust\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77295" }, { "id": 457933, "postDate": "2019-01-18T11:10:09.317Z", "content": "yes, it is really hard to say ...
", "rawMarkdown": "yes, it is really hard to say ... " } ] }, { "id": 456564, "postDate": "2019-01-16T04:04:50.073Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 456631, "author_name": "kwang", "author_url": "", "post_date": "2019-01-16T08:13:29.583000", "content": "I think you meet \"null importance\" feature, because any feature could have prob to lower local CV.\nYou can look at this kernel https://www.kaggle.com/ogrellier/feature-selection-with-null-importances
", "votes": 1, "replies": [ { "id": 456659, "author_name": "good good study", "author_url": "", "post_date": "2019-01-16T08:54:00.070000", "content": "thanks.
", "votes": 0, "replies": [] } ] }, { "id": 456668, "author_name": "Salamat", "author_url": "", "post_date": "2019-01-16T09:06:53.887000", "content": "I think it the case due to huge variance in CV score. If your improvement is larger than given variance than you are more likely get higher score. Increase number of number of folds , and get your own test set. Try to check on test set first before submission.
", "votes": 2, "replies": [] }, { "id": 456766, "author_name": "Manoj", "author_url": "", "post_date": "2019-01-16T13:48:45.480000", "content": "Thanks for bringing this out. i had come across same issue
\n\nCV 3.64936 (lgb 5 fold) -->> 3.689 LB\nCV 3.64753 (lgb 5 fold) -->> 3.691 LB
\n\nI am confused should i improve the model further
", "votes": 0, "replies": [ { "id": 457131, "author_name": "good good study", "author_url": "", "post_date": "2019-01-17T01:48:47.233000", "content": "I think single model for 3.689 is great.
", "votes": 0, "replies": [] }, { "id": 457180, "author_name": "Manoj", "author_url": "", "post_date": "2019-01-17T03:46:23.720000", "content": "Well, i am not quite sure that stacking and blending will help how much
", "votes": 0, "replies": [] } ] }, { "id": 456652, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2019-01-16T08:40:50.683000", "content": "Actually, this competition has many incosistencies on data explanation. I believe that nobody has understood what we are exactly trying to predict.
\n\nActually the 0.002 difference on LB is non significant, concerning that it is evaluated by the 30% of the total submission file
", "votes": 0, "replies": [ { "id": 456660, "author_name": "good good study", "author_url": "", "post_date": "2019-01-16T08:56:59.433000", "content": "emm, the key thing is that: we do not know how to evaluate models if we do not care this 0.002 difference
", "votes": 0, "replies": [] }, { "id": 456670, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2019-01-16T09:08:59.397000", "content": "the history of kaggle competitions has shown that you must only care about your CV and non for the LB as soon as the LB is calculated on a small portion. So, in your case, the 2nd model is an improvement of your 1st, even for this slight rmse decrease
", "votes": 1, "replies": [] }, { "id": 456683, "author_name": "good good study", "author_url": "", "post_date": "2019-01-16T09:48:12.533000", "content": "Thanks,\nI think I know what you mean: trust local cv, not public LB.\nLet's see the final private LB on this two models.
", "votes": 1, "replies": [] }, { "id": 457359, "author_name": "Manoj", "author_url": "", "post_date": "2019-01-17T09:21:48.010000", "content": "sometimes it can be really confusing what to trust
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77295
", "votes": 0, "replies": [] }, { "id": 457933, "author_name": "good good study", "author_url": "", "post_date": "2019-01-18T11:10:09.317000", "content": "yes, it is really hard to say ...
", "votes": 0, "replies": [] } ] }, { "id": 456564, "author_name": "", "author_url": "", "post_date": "2019-01-16T04:04:50.073000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "456561": "Here are two of my results, and I am using the same train-valid split：\n1. CV:3.65**7**04289\t LB:3.69**6**\n2. CV:3.65**5**87777\t LB:3.69**8** (adding more feature on 1)\n\nI have observed inconsistency of CV and LB (i.e. lower CV but get higher LB), which makes it hard to evaluate models.\nDo you have similar issue? If so, how do you handle this? ", "456631": "I think you meet \"null importance\" feature, because any feature could have prob to lower local CV.\nYou can look at this kernel https://www.kaggle.com/ogrellier/feature-selection-with-null-importances", "456668": "I think it the case due to huge variance in CV score. If your improvement is larger than given variance than you are more likely get higher score. Increase number of number of folds , and get your own test set. Try to check on test set first before submission.", "456766": "Thanks for bringing this out. i had come across same issue\n\nCV 3.64936 (lgb 5 fold) -->> 3.689 LB\nCV 3.64753 (lgb 5 fold) -->> 3.691 LB\n\nI am confused should i improve the model further \n", "456652": "Actually, this competition has many incosistencies on data explanation. I believe that nobody has understood what we are exactly trying to predict.\n\nActually the 0.002 difference on LB is non significant, concerning that it is evaluated by the 30% of the total submission file", "456564": "" } }