I have observed that CV error is not correlated to Test error (CV error is close to ~3.66-3.64(all train data) with #folds=5; test_size=20% for experiment). If you remove outliers, and do similar test you will observe that CV error is linear with test error meaning that they are correlated. This indicates that outlier makes evaluation metric very unstable and not reliable. Although model without outliers is reliable, it won't improve score since it has little effect in overall score (mixing with the model with outliers).
\n\nI was not able to use this information to get better score. Please, share your ideas about it.
", "messages": [ { "id": 464630, "postDate": "2019-02-01T07:10:11.313Z", "content": "I have observed that CV error is not correlated to Test error (CV error is close to ~3.66-3.64(all train data) with #folds=5; test_size=20% for experiment). If you remove outliers, and do similar test you will observe that CV error is linear with test error meaning that they are correlated. This indicates that outlier makes evaluation metric very unstable and not reliable. Although model without outliers is reliable, it won't improve score since it has little effect in overall score (mixing with the model with outliers).
\n\nI was not able to use this information to get better score. Please, share your ideas about it.
", "rawMarkdown": "I have observed that CV error is not correlated to Test error (CV error is close to ~3.66-3.64(all train data) with #folds=5; test_size=20% for experiment). If you remove outliers, and do similar test you will observe that CV error is linear with test error meaning that they are correlated. This indicates that outlier makes evaluation metric very unstable and not reliable. Although model without outliers is reliable, it won't improve score since it has little effect in overall score (mixing with the model with outliers).\n\nI was not able to use this information to get better score. Please, share your ideas about it.\n", "votes": 10 }, { "id": 464643, "postDate": "2019-02-01T07:46:48.670Z", "content": "Single model 9 fold CV score 3.648; LB score 3.680.\nBreak up of above score based on outliers\nScore only for Non-Outlier Targets: 1.826 \nScore only for Outlier Targets: 30.261
\n\n@Salamat What was your breakup score for the best single model.
", "rawMarkdown": "Single model 9 fold CV score 3.648; LB score 3.680.\nBreak up of above score based on outliers\nScore only for Non-Outlier Targets: 1.826 \nScore only for Outlier Targets: 30.261\n\n@Salamat What was your breakup score for the best single model.", "votes": 1, "replies": [ { "id": 464691, "postDate": "2019-02-01T10:01:23.220Z", "content": "cv score for Non outlier target is about 1.5x.
", "rawMarkdown": "cv score for Non outlier target is about 1.5x." }, { "id": 464715, "postDate": "2019-02-01T10:57:37.650Z", "content": "Thanks.. Is CV of 1.5x a score for all train data single model.. or for a model that is trained on the train data without the outlier..
", "rawMarkdown": "Thanks.. Is CV of 1.5x a score for all train data single model.. or for a model that is trained on the train data without the outlier.." }, { "id": 464895, "postDate": "2019-02-01T18:19:55.593Z", "content": "It is CV error for 5 fold. My personal best score for single model is 3.673 in LB and CV is about 3.648(5fold). As I mentioned, I can get much better CV scores but it doesn't reflect in LB score.
", "rawMarkdown": "It is CV error for 5 fold. My personal best score for single model is 3.673 in LB and CV is about 3.648(5fold). As I mentioned, I can get much better CV scores but it doesn't reflect in LB score." } ] }, { "id": 464912, "postDate": "2019-02-01T19:28:58.100Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 464643, "author_name": "mrxnew", "author_url": "", "post_date": "2019-02-01T07:46:48.670000", "content": "Single model 9 fold CV score 3.648; LB score 3.680.\nBreak up of above score based on outliers\nScore only for Non-Outlier Targets: 1.826 \nScore only for Outlier Targets: 30.261
\n\n@Salamat What was your breakup score for the best single model.
", "votes": 1, "replies": [ { "id": 464691, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2019-02-01T10:01:23.220000", "content": "cv score for Non outlier target is about 1.5x.
", "votes": 0, "replies": [] }, { "id": 464715, "author_name": "mrxnew", "author_url": "", "post_date": "2019-02-01T10:57:37.650000", "content": "Thanks.. Is CV of 1.5x a score for all train data single model.. or for a model that is trained on the train data without the outlier..
", "votes": 0, "replies": [] }, { "id": 464895, "author_name": "Salamat", "author_url": "", "post_date": "2019-02-01T18:19:55.593000", "content": "It is CV error for 5 fold. My personal best score for single model is 3.673 in LB and CV is about 3.648(5fold). As I mentioned, I can get much better CV scores but it doesn't reflect in LB score.
", "votes": 0, "replies": [] } ] }, { "id": 464912, "author_name": "", "author_url": "", "post_date": "2019-02-01T19:28:58.100000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "464630": "I have observed that CV error is not correlated to Test error (CV error is close to ~3.66-3.64(all train data) with #folds=5; test_size=20% for experiment). If you remove outliers, and do similar test you will observe that CV error is linear with test error meaning that they are correlated. This indicates that outlier makes evaluation metric very unstable and not reliable. Although model without outliers is reliable, it won't improve score since it has little effect in overall score (mixing with the model with outliers).\n\nI was not able to use this information to get better score. Please, share your ideas about it.\n", "464643": "Single model 9 fold CV score 3.648; LB score 3.680.\nBreak up of above score based on outliers\nScore only for Non-Outlier Targets: 1.826 \nScore only for Outlier Targets: 30.261\n\n@Salamat What was your breakup score for the best single model.", "464912": "" } }