![\"\"](\"https://i.imgur.com/CIMuSS6.png\")
Correct validation scheme is one of the most important topics in each competition.\nLet's discuss which approaches work the best.
\n\nI wrote this kernel: https://www.kaggle.com/artgor/validation-feature-selection-interpretation-etc where I compare various validation schemes for three types:\n
It seems that group kfold by molecules works best. What do you think?
", "messages": [ { "id": "554260", "postDate": "06/17/2019 07:44:48", "content": "Correct validation scheme is one of the most important topics in each competition.\nLet's discuss which approaches work the best.
\n\nI wrote this kernel: https://www.kaggle.com/artgor/validation-feature-selection-interpretation-etc where I compare various validation schemes for three types:\n
It seems that group kfold by molecules works best. What do you think?
", "rawMarkdown": "Correct validation scheme is one of the most important topics in each competition.\nLet's discuss which approaches work the best.\n\nI wrote this kernel: https://www.kaggle.com/artgor/validation-feature-selection-interpretation-etc where I compare various validation schemes for three types:\n\n\nIt seems that group kfold by molecules works best. What do you think?", "votes": null }, { "id": "554641", "postDate": "06/17/2019 19:59:35", "content": "Group KFold by molecule has worked best for me and I have almost no difference between CV and LB scores.
", "rawMarkdown": "Group KFold by molecule has worked best for me and I have almost no difference between CV and LB scores.", "votes": null }, { "id": "554653", "postDate": "06/17/2019 20:50:23", "content": "I'll give that a try for my next submission. I've been using Kfold shuffle and my LB scores have been consistently ~0.2 better than my CV scores.
", "rawMarkdown": "I'll give that a try for my next submission. I've been using Kfold shuffle and my LB scores have been consistently ~0.2 better than my CV scores.", "votes": null }, { "id": "554699", "postDate": "06/17/2019 23:19:30", "content": "Same here. My CV and LB are very, very close using group kfold by molecule.
", "rawMarkdown": "Same here. My CV and LB are very, very close using group kfold by molecule.", "votes": null }, { "id": "555022", "postDate": "06/18/2019 10:37:19", "content": "IMHO, it's a good idea to use GroupKFold by molecule since it can avoid overfitting caused by symmetry leakage. Notice that some molecules have rotational symmetry. For example dsgdb9nsd_000001 (CH4) has 10 observations, but essentially there are only 2 rotationally-invariant observations. In practice we usually use symmetry operations to reduce computational cost.
", "rawMarkdown": "IMHO, it's a good idea to use GroupKFold by molecule since it can avoid overfitting caused by symmetry leakage. Notice that some molecules have rotational symmetry. For example dsgdb9nsd_000001 (CH4) has 10 observations, but essentially there are only 2 rotationally-invariant observations. In practice we usually use symmetry operations to reduce computational cost.", "votes": null }, { "id": "556906", "postDate": "06/20/2019 21:20:19", "content": "GroupKFold lowered my CV-LB gap but returned the same LB score. Time for some feature reduction!
", "rawMarkdown": "GroupKFold lowered my CV-LB gap but returned the same LB score. Time for some feature reduction!", "votes": null }, { "id": "556910", "postDate": "06/20/2019 21:30:49", "content": "May I ask roughly how many features you are currently using? I have nearly 200 and am now looking to decrease this number after poor adversarial validation results (AUC>0.95).
\n\nDespite having features which make rows easy to distinguish between train and test, my CV and LB scores are similar with LB being slightly better.
", "rawMarkdown": "May I ask roughly how many features you are currently using? I have nearly 200 and am now looking to decrease this number after poor adversarial validation results (AUC>0.95). \n\nDespite having features which make rows easy to distinguish between train and test, my CV and LB scores are similar with LB being slightly better.", "votes": null }, { "id": "557486", "postDate": "06/21/2019 08:00:21", "content": "468, but I'll probably pare that down soon
", "rawMarkdown": "468, but I'll probably pare that down soon", "votes": null }, { "id": "557754", "postDate": "06/21/2019 12:49:13", "content": "I have a question regaring adversarial validation.\nIf I uderstood the basics right, AUC should be ~0,5 - this means my features a stable (please correct me if I'm wrong).
\n\nI'm using 89 features, my mean AV AUC is 0.97+. In theory, I should drop or replace top features (~15-18 in my case, which have importance a lot higher than the other features). If I just drop them, AUC is ~0,7 and my LB decreases drastically. If I try to replace them (using stats and math to construct new features from them) - AUC remains the same or becomes even higher.\nSo, the question is: how to get good AV AUC and keep my LB score? \nAny hints and tips will be highly appreciated.
", "rawMarkdown": "I have a question regaring adversarial validation.\nIf I uderstood the basics right, AUC should be ~0,5 - this means my features a stable (please correct me if I'm wrong).\n\nI'm using 89 features, my mean AV AUC is 0.97+. In theory, I should drop or replace top features (~15-18 in my case, which have importance a lot higher than the other features). If I just drop them, AUC is ~0,7 and my LB decreases drastically. If I try to replace them (using stats and math to construct new features from them) - AUC remains the same or becomes even higher.\nSo, the question is: how to get good AV AUC and keep my LB score? \nAny hints and tips will be highly appreciated.", "votes": null }, { "id": "557868", "postDate": "06/21/2019 14:41:58", "content": "I'm very impressed you've got such a low CV-LB gap with so many features! Mine vary with bond type but I've got between 80-125ish. Then again, I've only been using my own structural features so far and haven't included Coulomb interaction or any of the additional train data files.
", "rawMarkdown": "I'm very impressed you've got such a low CV-LB gap with so many features! Mine vary with bond type but I've got between 80-125ish. Then again, I've only been using my own structural features so far and haven't included Coulomb interaction or any of the additional train data files.", "votes": null }, { "id": "557911", "postDate": "06/21/2019 15:15:59", "content": "I don't believe a high AV-AUC is always an indication that a feature is bad. It can simply mean that you've identified a feature that is broadly different between train and test and which can improve the performance of your final model on the test set. But if the aim of your model is to generalise as much as possible then removing these features is probably a good idea. If you have a large gap between CV and LB scores, or a low correlation between CV and LB scores, then features with a high AV-AUC are probably to blame and should be pruned.
", "rawMarkdown": "I don't believe a high AV-AUC is *always* an indication that a feature is bad. It can simply mean that you've identified a feature that is broadly different between train and test and which can improve the performance of your final model on the test set. But if the aim of your model is to generalise as much as possible then removing these features is probably a good idea. If you have a large gap between CV and LB scores, or a low correlation between CV and LB scores, then features with a high AV-AUC are probably to blame and should be pruned.", "votes": null }, { "id": "557968", "postDate": "06/21/2019 16:11:36", "content": "CV-LB corr is okay, but there is a gap. It's not large, but Branden writes below that his CV-LB is almost in line. So I've tried to do smth to reduce gap, which led to questions above 🕵️ \nAnd thanks for the insights!
", "rawMarkdown": "CV-LB corr is okay, but there is a gap. It's not large, but Branden writes below that his CV-LB is almost in line. So I've tried to do smth to reduce gap, which led to questions above 🕵️ \nAnd thanks for the insights!", "votes": null }, { "id": "561124", "postDate": "06/26/2019 06:50:38", "content": "@brandenkmurray , may I ask if you use kaggle kernels or your local PC? \nMy kernels can't bear ~400 features.
", "rawMarkdown": "brandenkmurray , may I ask if you use kaggle kernels or your local PC? \nMy kernels can't bear ~400 features.", "votes": null }, { "id": "561439", "postDate": "06/26/2019 13:00:41", "content": "KFold/GroupKFold by molecule is what has typically been done in the chemistry machine learning field so far. Not that this implies that it's the best strategy, but others have employed this approach before.
", "rawMarkdown": "KFold/GroupKFold by molecule is what has typically been done in the chemistry machine learning field so far. Not that this implies that it's the best strategy, but others have employed this approach before.", "votes": null }, { "id": "563273", "postDate": "06/28/2019 04:55:09", "content": "local PC
", "rawMarkdown": "local PC", "votes": null }, { "id": "588123", "postDate": "07/30/2019 07:28:26", "content": "Group holdout is working good for me with better LB than validation score (~validation score - 0.01). I know it is not robust. But then, I don't have a robust PC ;) . Using kernels with ~70-100 features per group.
", "rawMarkdown": "Group holdout is working good for me with better LB than validation score (~validation score - 0.01). I know it is not robust. But then, I don't have a robust PC ;) . Using kernels with ~70-100 features per group.", "votes": null } ], "comments": [ { "id": 554641, "author_name": "brandenkmurray", "author_url": "", "post_date": "06/17/2019 19:59:35", "content": "Group KFold by molecule has worked best for me and I have almost no difference between CV and LB scores.
", "votes": null, "replies": [ { "id": 554653, "author_name": "joshxsarah", "author_url": "", "post_date": "06/17/2019 20:50:23", "content": "I'll give that a try for my next submission. I've been using Kfold shuffle and my LB scores have been consistently ~0.2 better than my CV scores.
", "votes": null, "replies": [] }, { "id": 554699, "author_name": "abendorf", "author_url": "", "post_date": "06/17/2019 23:19:30", "content": "Same here. My CV and LB are very, very close using group kfold by molecule.
", "votes": null, "replies": [] }, { "id": 556906, "author_name": "bigironsphere", "author_url": "", "post_date": "06/20/2019 21:20:19", "content": "GroupKFold lowered my CV-LB gap but returned the same LB score. Time for some feature reduction!
", "votes": null, "replies": [] }, { "id": 556910, "author_name": "joshxsarah", "author_url": "", "post_date": "06/20/2019 21:30:49", "content": "May I ask roughly how many features you are currently using? I have nearly 200 and am now looking to decrease this number after poor adversarial validation results (AUC>0.95).
\n\nDespite having features which make rows easy to distinguish between train and test, my CV and LB scores are similar with LB being slightly better.
", "votes": null, "replies": [] }, { "id": 557486, "author_name": "brandenkmurray", "author_url": "", "post_date": "06/21/2019 08:00:21", "content": "468, but I'll probably pare that down soon
", "votes": null, "replies": [] }, { "id": 557868, "author_name": "bigironsphere", "author_url": "", "post_date": "06/21/2019 14:41:58", "content": "I'm very impressed you've got such a low CV-LB gap with so many features! Mine vary with bond type but I've got between 80-125ish. Then again, I've only been using my own structural features so far and haven't included Coulomb interaction or any of the additional train data files.
", "votes": null, "replies": [] }, { "id": 561124, "author_name": "stanislavblinov", "author_url": "", "post_date": "06/26/2019 06:50:38", "content": "@brandenkmurray , may I ask if you use kaggle kernels or your local PC? \nMy kernels can't bear ~400 features.
", "votes": null, "replies": [] }, { "id": 563273, "author_name": "brandenkmurray", "author_url": "", "post_date": "06/28/2019 04:55:09", "content": "local PC
", "votes": null, "replies": [] } ] }, { "id": 555022, "author_name": "taufikadi", "author_url": "", "post_date": "06/18/2019 10:37:19", "content": "IMHO, it's a good idea to use GroupKFold by molecule since it can avoid overfitting caused by symmetry leakage. Notice that some molecules have rotational symmetry. For example dsgdb9nsd_000001 (CH4) has 10 observations, but essentially there are only 2 rotationally-invariant observations. In practice we usually use symmetry operations to reduce computational cost.
", "votes": null, "replies": [] }, { "id": 557754, "author_name": "stanislavblinov", "author_url": "", "post_date": "06/21/2019 12:49:13", "content": "I have a question regaring adversarial validation.\nIf I uderstood the basics right, AUC should be ~0,5 - this means my features a stable (please correct me if I'm wrong).
\n\nI'm using 89 features, my mean AV AUC is 0.97+. In theory, I should drop or replace top features (~15-18 in my case, which have importance a lot higher than the other features). If I just drop them, AUC is ~0,7 and my LB decreases drastically. If I try to replace them (using stats and math to construct new features from them) - AUC remains the same or becomes even higher.\nSo, the question is: how to get good AV AUC and keep my LB score? \nAny hints and tips will be highly appreciated.
", "votes": null, "replies": [ { "id": 557911, "author_name": "bigironsphere", "author_url": "", "post_date": "06/21/2019 15:15:59", "content": "I don't believe a high AV-AUC is always an indication that a feature is bad. It can simply mean that you've identified a feature that is broadly different between train and test and which can improve the performance of your final model on the test set. But if the aim of your model is to generalise as much as possible then removing these features is probably a good idea. If you have a large gap between CV and LB scores, or a low correlation between CV and LB scores, then features with a high AV-AUC are probably to blame and should be pruned.
", "votes": null, "replies": [] }, { "id": 557968, "author_name": "stanislavblinov", "author_url": "", "post_date": "06/21/2019 16:11:36", "content": "CV-LB corr is okay, but there is a gap. It's not large, but Branden writes below that his CV-LB is almost in line. So I've tried to do smth to reduce gap, which led to questions above 🕵️ \nAnd thanks for the insights!
", "votes": null, "replies": [] } ] }, { "id": 561439, "author_name": "larsbratholm", "author_url": "", "post_date": "06/26/2019 13:00:41", "content": "KFold/GroupKFold by molecule is what has typically been done in the chemistry machine learning field so far. Not that this implies that it's the best strategy, but others have employed this approach before.
", "votes": null, "replies": [] }, { "id": 588123, "author_name": "silence2", "author_url": "", "post_date": "07/30/2019 07:28:26", "content": "Group holdout is working good for me with better LB than validation score (~validation score - 0.01). I know it is not robust. But then, I don't have a robust PC ;) . Using kernels with ~70-100 features per group.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "554260": "Correct validation scheme is one of the most important topics in each competition.\nLet's discuss which approaches work the best.\n\nI wrote this kernel: https://www.kaggle.com/artgor/validation-feature-selection-interpretation-etc where I compare various validation schemes for three types:\n\n\nIt seems that group kfold by molecules works best. What do you think?", "554641": "Group KFold by molecule has worked best for me and I have almost no difference between CV and LB scores.", "554653": "I'll give that a try for my next submission. I've been using Kfold shuffle and my LB scores have been consistently ~0.2 better than my CV scores.", "554699": "Same here. My CV and LB are very, very close using group kfold by molecule.", "555022": "IMHO, it's a good idea to use GroupKFold by molecule since it can avoid overfitting caused by symmetry leakage. Notice that some molecules have rotational symmetry. For example dsgdb9nsd_000001 (CH4) has 10 observations, but essentially there are only 2 rotationally-invariant observations. In practice we usually use symmetry operations to reduce computational cost.", "556906": "GroupKFold lowered my CV-LB gap but returned the same LB score. Time for some feature reduction!", "556910": "May I ask roughly how many features you are currently using? I have nearly 200 and am now looking to decrease this number after poor adversarial validation results (AUC>0.95). \n\nDespite having features which make rows easy to distinguish between train and test, my CV and LB scores are similar with LB being slightly better.", "557486": "468, but I'll probably pare that down soon", "557754": "I have a question regaring adversarial validation.\nIf I uderstood the basics right, AUC should be ~0,5 - this means my features a stable (please correct me if I'm wrong).\n\nI'm using 89 features, my mean AV AUC is 0.97+. In theory, I should drop or replace top features (~15-18 in my case, which have importance a lot higher than the other features). If I just drop them, AUC is ~0,7 and my LB decreases drastically. If I try to replace them (using stats and math to construct new features from them) - AUC remains the same or becomes even higher.\nSo, the question is: how to get good AV AUC and keep my LB score? \nAny hints and tips will be highly appreciated.", "557868": "I'm very impressed you've got such a low CV-LB gap with so many features! Mine vary with bond type but I've got between 80-125ish. Then again, I've only been using my own structural features so far and haven't included Coulomb interaction or any of the additional train data files.", "557911": "I don't believe a high AV-AUC is *always* an indication that a feature is bad. It can simply mean that you've identified a feature that is broadly different between train and test and which can improve the performance of your final model on the test set. But if the aim of your model is to generalise as much as possible then removing these features is probably a good idea. If you have a large gap between CV and LB scores, or a low correlation between CV and LB scores, then features with a high AV-AUC are probably to blame and should be pruned.", "557968": "CV-LB corr is okay, but there is a gap. It's not large, but Branden writes below that his CV-LB is almost in line. So I've tried to do smth to reduce gap, which led to questions above 🕵️ \nAnd thanks for the insights!", "561124": "brandenkmurray , may I ask if you use kaggle kernels or your local PC? \nMy kernels can't bear ~400 features.", "561439": "KFold/GroupKFold by molecule is what has typically been done in the chemistry machine learning field so far. Not that this implies that it's the best strategy, but others have employed this approach before.", "563273": "local PC", "588123": "Group holdout is working good for me with better LB than validation score (~validation score - 0.01). I know it is not robust. But then, I don't have a robust PC ;) . Using kernels with ~70-100 features per group." }, "source": "meta" }