I've been having trouble with seeing my CV improvements materialize to improvements in LB.
\n\ni.e. My best score: LightGBM 5fold CV 3.6598 -> LB 3.692.\nbut I've had CV scores as good as 3.6525 but only translate to LB of 3.694 to 3.698.
\n\nI've read a recent post that the test set has many outliers and am wondering if this is contributing to this inconsistency. For those that do have scores in the top 5% that still have CV improvements that are consistent with LB improvements, any tips on how to do this?
\n\nUpdate: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.
\n\nPublic LB is the new CV?
", "messages": [ { "id": "438101", "postDate": "12/13/2018 05:16:52", "content": "I've been having trouble with seeing my CV improvements materialize to improvements in LB.
\n\ni.e. My best score: LightGBM 5fold CV 3.6598 -> LB 3.692.\nbut I've had CV scores as good as 3.6525 but only translate to LB of 3.694 to 3.698.
\n\nI've read a recent post that the test set has many outliers and am wondering if this is contributing to this inconsistency. For those that do have scores in the top 5% that still have CV improvements that are consistent with LB improvements, any tips on how to do this?
\n\nUpdate: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.
\n\nPublic LB is the new CV?
", "rawMarkdown": "I've been having trouble with seeing my CV improvements materialize to improvements in LB.\n\ni.e. My best score: LightGBM 5fold CV 3.6598 -> LB 3.692.\nbut I've had CV scores as good as 3.6525 but only translate to LB of 3.694 to 3.698.\n\nI've read a recent post that the test set has many outliers and am wondering if this is contributing to this inconsistency. For those that do have scores in the top 5% that still have CV improvements that are consistent with LB improvements, any tips on how to do this?\n\nUpdate: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.\n\nPublic LB is the new CV?", "votes": null }, { "id": "438140", "postDate": "12/13/2018 06:57:54", "content": "Maybe your model variance is too high, do some regularizations? just my two cents.
", "rawMarkdown": "Maybe your model variance is too high, do some regularizations? just my two cents.", "votes": null }, { "id": "438174", "postDate": "12/13/2018 08:01:01", "content": "Thanks will try it out :)
", "rawMarkdown": "Thanks will try it out :)", "votes": null }, { "id": "438424", "postDate": "12/13/2018 16:26:04", "content": "Same here, my 5fold CV 3.65648 -> LB 3.699. It looks like seed has quite an influence on the scores.
", "rawMarkdown": "Same here, my 5fold CV 3.65648 -> LB 3.699. It looks like seed has quite an influence on the scores.", "votes": null }, { "id": "438680", "postDate": "12/14/2018 02:41:57", "content": "my 5fold CV 3.63716 on LB 3.692. But I set the seed \"0\".So sad
", "rawMarkdown": "my 5fold CV 3.63716 on LB 3.692. But I set the seed \"0\".So sad", "votes": null }, { "id": "440562", "postDate": "12/17/2018 17:46:34", "content": "I came across a situation where for essentually the same model (CV about 3.652), by swapping the order I append feature files - I group my feature into different files - I get two very different LB scores: one is 3.695, the other one is 3.690.
\n\nMy observation is that the LB is extremely sensitive to small change in the model - I have experienced similar jump up/down the LB by fairly moderate change of modelling parameter.
\n\nPersonally, I would definitely place more confident on my CV score, but at the sametime be very careful about modelling approach that might lead to overfit.
", "rawMarkdown": "I came across a situation where for essentually the same model (CV about 3.652), by swapping the order I append feature files - I group my feature into different files - I get two very different LB scores: one is 3.695, the other one is 3.690.\n\nMy observation is that the LB is extremely sensitive to small change in the model - I have experienced similar jump up/down the LB by fairly moderate change of modelling parameter. \n\nPersonally, I would definitely place more confident on my CV score, but at the sametime be very careful about modelling approach that might lead to overfit.", "votes": null }, { "id": "440800", "postDate": "12/18/2018 01:27:15", "content": "Same feeling!
", "rawMarkdown": "Same feeling!", "votes": null }, { "id": "440811", "postDate": "12/18/2018 01:48:04", "content": "Update: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.
", "rawMarkdown": "Update: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.", "votes": null }, { "id": "440812", "postDate": "12/18/2018 01:48:55", "content": "And now you are #1 in LB. Congrats!
", "rawMarkdown": "And now you are #1 in LB. Congrats!", "votes": null }, { "id": "440988", "postDate": "12/18/2018 06:38:26", "content": "in all honesty, I have a lot more trust to my CV and the LB :) - although I would push for improvement on both, as well as alternative CV fold splits
", "rawMarkdown": "in all honesty, I have a lot more trust to my CV and the LB :) - although I would push for improvement on both, as well as alternative CV fold splits", "votes": null }, { "id": "441002", "postDate": "12/18/2018 07:09:37", "content": "What will be your basis in choosing your final submission?
", "rawMarkdown": "What will be your basis in choosing your final submission?", "votes": null }, { "id": "441145", "postDate": "12/18/2018 10:22:17", "content": "we are still so far away - so I don't have an exact idea yet. but you can for instance do a weighed score between LB & CV.
", "rawMarkdown": "we are still so far away - so I don't have an exact idea yet. but you can for instance do a weighed score between LB & CV.", "votes": null } ], "comments": [ { "id": 438140, "author_name": "marcuslin", "author_url": "", "post_date": "12/13/2018 06:57:54", "content": "Maybe your model variance is too high, do some regularizations? just my two cents.
", "votes": null, "replies": [ { "id": 438174, "author_name": "enzoamp", "author_url": "", "post_date": "12/13/2018 08:01:01", "content": "Thanks will try it out :)
", "votes": null, "replies": [] } ] }, { "id": 438424, "author_name": "frjo0083", "author_url": "", "post_date": "12/13/2018 16:26:04", "content": "Same here, my 5fold CV 3.65648 -> LB 3.699. It looks like seed has quite an influence on the scores.
", "votes": null, "replies": [] }, { "id": 438680, "author_name": "lblhandsome", "author_url": "", "post_date": "12/14/2018 02:41:57", "content": "my 5fold CV 3.63716 on LB 3.692. But I set the seed \"0\".So sad
", "votes": null, "replies": [ { "id": 440812, "author_name": "enzoamp", "author_url": "", "post_date": "12/18/2018 01:48:55", "content": "And now you are #1 in LB. Congrats!
", "votes": null, "replies": [] } ] }, { "id": 440562, "author_name": "yifanxie", "author_url": "", "post_date": "12/17/2018 17:46:34", "content": "I came across a situation where for essentually the same model (CV about 3.652), by swapping the order I append feature files - I group my feature into different files - I get two very different LB scores: one is 3.695, the other one is 3.690.
\n\nMy observation is that the LB is extremely sensitive to small change in the model - I have experienced similar jump up/down the LB by fairly moderate change of modelling parameter.
\n\nPersonally, I would definitely place more confident on my CV score, but at the sametime be very careful about modelling approach that might lead to overfit.
", "votes": null, "replies": [ { "id": 440800, "author_name": "codlife", "author_url": "", "post_date": "12/18/2018 01:27:15", "content": "Same feeling!
", "votes": null, "replies": [] }, { "id": 440811, "author_name": "enzoamp", "author_url": "", "post_date": "12/18/2018 01:48:04", "content": "Update: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.
", "votes": null, "replies": [] }, { "id": 440988, "author_name": "yifanxie", "author_url": "", "post_date": "12/18/2018 06:38:26", "content": "in all honesty, I have a lot more trust to my CV and the LB :) - although I would push for improvement on both, as well as alternative CV fold splits
", "votes": null, "replies": [] }, { "id": 441002, "author_name": "enzoamp", "author_url": "", "post_date": "12/18/2018 07:09:37", "content": "What will be your basis in choosing your final submission?
", "votes": null, "replies": [] }, { "id": 441145, "author_name": "yifanxie", "author_url": "", "post_date": "12/18/2018 10:22:17", "content": "we are still so far away - so I don't have an exact idea yet. but you can for instance do a weighed score between LB & CV.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "438101": "I've been having trouble with seeing my CV improvements materialize to improvements in LB.\n\ni.e. My best score: LightGBM 5fold CV 3.6598 -> LB 3.692.\nbut I've had CV scores as good as 3.6525 but only translate to LB of 3.694 to 3.698.\n\nI've read a recent post that the test set has many outliers and am wondering if this is contributing to this inconsistency. For those that do have scores in the top 5% that still have CV improvements that are consistent with LB improvements, any tips on how to do this?\n\nUpdate: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.\n\nPublic LB is the new CV?", "438140": "Maybe your model variance is too high, do some regularizations? just my two cents.", "438174": "Thanks will try it out :)", "438424": "Same here, my 5fold CV 3.65648 -> LB 3.699. It looks like seed has quite an influence on the scores.", "438680": "my 5fold CV 3.63716 on LB 3.692. But I set the seed \"0\".So sad", "440562": "I came across a situation where for essentually the same model (CV about 3.652), by swapping the order I append feature files - I group my feature into different files - I get two very different LB scores: one is 3.695, the other one is 3.690.\n\nMy observation is that the LB is extremely sensitive to small change in the model - I have experienced similar jump up/down the LB by fairly moderate change of modelling parameter. \n\nPersonally, I would definitely place more confident on my CV score, but at the sametime be very careful about modelling approach that might lead to overfit.", "440800": "Same feeling!", "440811": "Update: My new best score: 6.87 had a CV score of 3.656. Really can't trust my CV for this competition. I now retrain my model whenever I have a new idea and automatically submit it to check if it worked.", "440812": "And now you are #1 in LB. Congrats!", "440988": "in all honesty, I have a lot more trust to my CV and the LB :) - although I would push for improvement on both, as well as alternative CV fold splits", "441002": "What will be your basis in choosing your final submission?", "441145": "we are still so far away - so I don't have an exact idea yet. but you can for instance do a weighed score between LB & CV." }, "source": "meta" }