In the Overview the host says \"no molecule in the training data is found in the test data.\"
\n\nI'm relatively new to kaggle, but I'm guessing that with a training set of this size and molecules that are relatively small this may not be a big source of leakage. But, has anyone improved their LB score relative to their CV by holding out entire molecules when they train?
\n\n(Sorry if already addressed, just joining competition)
", "messages": [ { "id": "574284", "postDate": "07/13/2019 15:04:06", "content": "In the Overview the host says \"no molecule in the training data is found in the test data.\"
\n\nI'm relatively new to kaggle, but I'm guessing that with a training set of this size and molecules that are relatively small this may not be a big source of leakage. But, has anyone improved their LB score relative to their CV by holding out entire molecules when they train?
\n\n(Sorry if already addressed, just joining competition)
", "rawMarkdown": "In the Overview the host says \"no molecule in the training data is found in the test data.\"\n\nI'm relatively new to kaggle, but I'm guessing that with a training set of this size and molecules that are relatively small this may not be a big source of leakage. But, has anyone improved their LB score relative to their CV by holding out entire molecules when they train?\n\n(Sorry if already addressed, just joining competition)", "votes": null }, { "id": "574388", "postDate": "07/13/2019 18:36:04", "content": "WHy would there be a leakage, can you explain?
\n\nI am not sure I understand what you mean by holding out entire molecules. But, according to previous discussions in this forum, using group kfold with groups defined by molecules is very usual in machine learning for chemistry.
", "rawMarkdown": "WHy would there be a leakage, can you explain?\n\nI am not sure I understand what you mean by holding out entire molecules. But, according to previous discussions in this forum, using group kfold with groups defined by molecules is very usual in machine learning for chemistry.", "votes": null }, { "id": "574402", "postDate": "07/13/2019 19:12:00", "content": "There would be a leakage if you train on molecule A and validate on molecule A at the same time. Many molecules are symmetrical (consider CH4) resulting in identical scalar coupling constants for different atom indices
", "rawMarkdown": "There would be a leakage if you train on molecule A and validate on molecule A at the same time. Many molecules are symmetrical (consider CH4) resulting in identical scalar coupling constants for different atom indices", "votes": null }, { "id": "574414", "postDate": "07/13/2019 19:33:01", "content": "Ah ok, yes this is what I was getting at. Making sure that one molecule doesn't end up split between two different folds. I missed this in the forums, it makes sense that this is the accepted method to me. Now I just looked over Giba's kernel and I see he's defining his folds with a groupKfold by molecule_name. So far I've just been doing random CV with this competition in my testing, and was thinking that it's probably not a good idea to train on a part of a given molecule then testing on the other part of that same molecule. In principle, it's probably not good because the test set wasn't generated in this manner, they're all brand new molecules. I'm thinking there is some risk (not sure how big of a risk) that the random CV approach might cause you to build a model that wouldn't generalize well to a brand new molecule that the model hasn't seen before. Though, this risk is probably reduced by the large volume of training data and the small size of the molecules.\nThat's what I meant by leakage, that there may be something specific (maybe characteristics particular to the exact moment in time when the molecule was measured, or maybe molecules are like snowflakes??? haha but I have noooo idea) to a particular molecule that would cause some interaction with predictions on the same molecule, maybe making the CV look good and the LB look bad. I wasn't aware of this groupKfold concept explicitly, though I do recall folks doing a version of this in the LANL earthquake competition. In that competition it also made sense to me because there was possibly some autocorrelation with samples that were recorded in the same earthquake or recorded at roughly the same time (though I'm not really sure about it.) Please know I have zero domain knowledge here, and I'm only speculating. Would be interested to know why the groupKfold method is important here, though I have little doubt that it's the right way to go!
\n\nEDIT: CoreyLevinson's explanation above makes a lot of sense to me. Makes more sense than what I am jabbering about above.
", "rawMarkdown": "Ah ok, yes this is what I was getting at. Making sure that one molecule doesn't end up split between two different folds. I missed this in the forums, it makes sense that this is the accepted method to me. Now I just looked over Giba's kernel and I see he's defining his folds with a groupKfold by molecule_name. So far I've just been doing random CV with this competition in my testing, and was thinking that it's probably not a good idea to train on a part of a given molecule then testing on the other part of that same molecule. In principle, it's probably not good because the test set wasn't generated in this manner, they're all brand new molecules. I'm thinking there is some risk (not sure how big of a risk) that the random CV approach might cause you to build a model that wouldn't generalize well to a brand new molecule that the model hasn't seen before. Though, this risk is probably reduced by the large volume of training data and the small size of the molecules.\nThat's what I meant by leakage, that there may be something specific (maybe characteristics particular to the exact moment in time when the molecule was measured, or maybe molecules are like snowflakes??? haha but I have noooo idea) to a particular molecule that would cause some interaction with predictions on the same molecule, maybe making the CV look good and the LB look bad. I wasn't aware of this groupKfold concept explicitly, though I do recall folks doing a version of this in the LANL earthquake competition. In that competition it also made sense to me because there was possibly some autocorrelation with samples that were recorded in the same earthquake or recorded at roughly the same time (though I'm not really sure about it.) Please know I have zero domain knowledge here, and I'm only speculating. Would be interested to know why the groupKfold method is important here, though I have little doubt that it's the right way to go!\n\nEDIT: CoreyLevinson's explanation above makes a lot of sense to me. Makes more sense than what I am jabbering about above.", "votes": null }, { "id": "574432", "postDate": "07/13/2019 19:59:32", "content": "I never thought one would have parts of the same molecule in train and parts in validation data. But this is not what the post discusses. The post says that not having molecules in both train and test is a source of leakage. That is why I asked a question.
", "rawMarkdown": "I never thought one would have parts of the same molecule in train and parts in validation data. But this is not what the post discusses. The post says that not having molecules in both train and test is a source of leakage. That is why I asked a question.", "votes": null } ], "comments": [ { "id": 574388, "author_name": "cpmpml", "author_url": "", "post_date": "07/13/2019 18:36:04", "content": "WHy would there be a leakage, can you explain?
\n\nI am not sure I understand what you mean by holding out entire molecules. But, according to previous discussions in this forum, using group kfold with groups defined by molecules is very usual in machine learning for chemistry.
", "votes": null, "replies": [ { "id": 574402, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/13/2019 19:12:00", "content": "There would be a leakage if you train on molecule A and validate on molecule A at the same time. Many molecules are symmetrical (consider CH4) resulting in identical scalar coupling constants for different atom indices
", "votes": null, "replies": [] }, { "id": 574414, "author_name": "andyatkinson", "author_url": "", "post_date": "07/13/2019 19:33:01", "content": "Ah ok, yes this is what I was getting at. Making sure that one molecule doesn't end up split between two different folds. I missed this in the forums, it makes sense that this is the accepted method to me. Now I just looked over Giba's kernel and I see he's defining his folds with a groupKfold by molecule_name. So far I've just been doing random CV with this competition in my testing, and was thinking that it's probably not a good idea to train on a part of a given molecule then testing on the other part of that same molecule. In principle, it's probably not good because the test set wasn't generated in this manner, they're all brand new molecules. I'm thinking there is some risk (not sure how big of a risk) that the random CV approach might cause you to build a model that wouldn't generalize well to a brand new molecule that the model hasn't seen before. Though, this risk is probably reduced by the large volume of training data and the small size of the molecules.\nThat's what I meant by leakage, that there may be something specific (maybe characteristics particular to the exact moment in time when the molecule was measured, or maybe molecules are like snowflakes??? haha but I have noooo idea) to a particular molecule that would cause some interaction with predictions on the same molecule, maybe making the CV look good and the LB look bad. I wasn't aware of this groupKfold concept explicitly, though I do recall folks doing a version of this in the LANL earthquake competition. In that competition it also made sense to me because there was possibly some autocorrelation with samples that were recorded in the same earthquake or recorded at roughly the same time (though I'm not really sure about it.) Please know I have zero domain knowledge here, and I'm only speculating. Would be interested to know why the groupKfold method is important here, though I have little doubt that it's the right way to go!
\n\nEDIT: CoreyLevinson's explanation above makes a lot of sense to me. Makes more sense than what I am jabbering about above.
", "votes": null, "replies": [] }, { "id": 574432, "author_name": "cpmpml", "author_url": "", "post_date": "07/13/2019 19:59:32", "content": "I never thought one would have parts of the same molecule in train and parts in validation data. But this is not what the post discusses. The post says that not having molecules in both train and test is a source of leakage. That is why I asked a question.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "574284": "In the Overview the host says \"no molecule in the training data is found in the test data.\"\n\nI'm relatively new to kaggle, but I'm guessing that with a training set of this size and molecules that are relatively small this may not be a big source of leakage. But, has anyone improved their LB score relative to their CV by holding out entire molecules when they train?\n\n(Sorry if already addressed, just joining competition)", "574388": "WHy would there be a leakage, can you explain?\n\nI am not sure I understand what you mean by holding out entire molecules. But, according to previous discussions in this forum, using group kfold with groups defined by molecules is very usual in machine learning for chemistry.", "574402": "There would be a leakage if you train on molecule A and validate on molecule A at the same time. Many molecules are symmetrical (consider CH4) resulting in identical scalar coupling constants for different atom indices", "574414": "Ah ok, yes this is what I was getting at. Making sure that one molecule doesn't end up split between two different folds. I missed this in the forums, it makes sense that this is the accepted method to me. Now I just looked over Giba's kernel and I see he's defining his folds with a groupKfold by molecule_name. So far I've just been doing random CV with this competition in my testing, and was thinking that it's probably not a good idea to train on a part of a given molecule then testing on the other part of that same molecule. In principle, it's probably not good because the test set wasn't generated in this manner, they're all brand new molecules. I'm thinking there is some risk (not sure how big of a risk) that the random CV approach might cause you to build a model that wouldn't generalize well to a brand new molecule that the model hasn't seen before. Though, this risk is probably reduced by the large volume of training data and the small size of the molecules.\nThat's what I meant by leakage, that there may be something specific (maybe characteristics particular to the exact moment in time when the molecule was measured, or maybe molecules are like snowflakes??? haha but I have noooo idea) to a particular molecule that would cause some interaction with predictions on the same molecule, maybe making the CV look good and the LB look bad. I wasn't aware of this groupKfold concept explicitly, though I do recall folks doing a version of this in the LANL earthquake competition. In that competition it also made sense to me because there was possibly some autocorrelation with samples that were recorded in the same earthquake or recorded at roughly the same time (though I'm not really sure about it.) Please know I have zero domain knowledge here, and I'm only speculating. Would be interested to know why the groupKfold method is important here, though I have little doubt that it's the right way to go!\n\nEDIT: CoreyLevinson's explanation above makes a lot of sense to me. Makes more sense than what I am jabbering about above.", "574432": "I never thought one would have parts of the same molecule in train and parts in validation data. But this is not what the post discusses. The post says that not having molecules in both train and test is a source of leakage. That is why I asked a question." }, "source": "meta" }