To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting. Adding more feature may not help but at least wont harm current performance.
\n\nBut when I add a set of generated features (mean/std/min/max of origin features), the mae of each round becomes worse than that on origin set. Also at convergence the CV is much worse. I've tunned a bunch of hyper params and almost sure it's not about hyper param tunning.
Could someone help me out, why is this happenning and what to do to avoid this...
", "messages": [ { "id": "570228", "postDate": "07/08/2019 03:04:43", "content": "To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting. Adding more feature may not help but at least wont harm current performance.
\n\nBut when I add a set of generated features (mean/std/min/max of origin features), the mae of each round becomes worse than that on origin set. Also at convergence the CV is much worse. I've tunned a bunch of hyper params and almost sure it's not about hyper param tunning.
Could someone help me out, why is this happenning and what to do to avoid this...
", "rawMarkdown": "To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting. Adding more feature may not help but at least wont harm current performance.\n\nBut when I add a set of generated features (mean/std/min/max of origin features), the `mae` of each round becomes worse than that on origin set. Also at convergence the CV is much worse. I've tunned a bunch of hyper params and almost sure it's not about hyper param tunning.\n\nCould someone help me out, why is this happenning and what to do to avoid this...", "votes": null }, { "id": "570421", "postDate": "07/08/2019 09:08:05", "content": "Did you check the features were correlated with scalar coupling - have a look at Giba's excellent script or the python version.
\n\nAlso I am trying to predict fc,pso,sdo and sd which are used to calculate scalar coupling.\nTypically the error should be less if you include predictions for these 4 variables.
", "rawMarkdown": "Did you check the features were correlated with scalar coupling - have a look at Giba's excellent script or the python version.\n\nAlso I am trying to predict fc,pso,sdo and sd which are used to calculate scalar coupling.\nTypically the error should be less if you include predictions for these 4 variables.", "votes": null }, { "id": "570439", "postDate": "07/08/2019 09:46:32", "content": "\"To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.\"
\n\nOf course not. Feature selection is very important.
", "rawMarkdown": "\"To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.\"\n\nOf course not. Feature selection is very important.", "votes": null }, { "id": "570447", "postDate": "07/08/2019 09:58:04", "content": "Thanks for your advice! Do you mean the R kernel? I'll take a look.
", "rawMarkdown": "Thanks for your advice! Do you mean the R kernel? I'll take a look.", "votes": null }, { "id": "570449", "postDate": "07/08/2019 09:58:28", "content": "Dude I didnt get it why of course not.....
", "rawMarkdown": "Dude I didnt get it why of course not.....", "votes": null }, { "id": "570458", "postDate": "07/08/2019 10:15:40", "content": "@scirpus doesn't your model just remember fc feature? In theory it's almost 1.0 correlation with target, so for exaple LGB would just remember this feature and other features just wont have any influence
", "rawMarkdown": "scirpus doesn't your model just remember fc feature? In theory it's almost 1.0 correlation with target, so for exaple LGB would just remember this feature and other features just wont have any influence", "votes": null }, { "id": "570493", "postDate": "07/08/2019 11:24:15", "content": "@stanislavblinov - it seems the error associated with scalar coupling is bigger than predicting fc,sd,pso,dso so I am hoping that incorporating all four make predicting scalar coupling easier.
\n\nIt is a hypothesis at the minute but that is what research is all about!
", "rawMarkdown": "stanislavblinov - it seems the error associated with scalar coupling is bigger than predicting fc,sd,pso,dso so I am hoping that incorporating all four make predicting scalar coupling easier.\n\nIt is a hypothesis at the minute but that is what research is all about!", "votes": null }, { "id": "570496", "postDate": "07/08/2019 11:33:03", "content": "\n\n\nthat is what research is all about!
\n
Agreed. Predicted features have a very interesting influence.\nI used them since Andrew released his great kernel, and all seemed good, but after -1.5 LB I decided to update them, and faced the problem I described above. Tho correlation was almost 1.0 both with old features and new ones.\nOr there's just a simplier explanation - I'm just doing smth wrong ¯_(ツ)_/¯
", "rawMarkdown": "> that is what research is all about!\n\nAgreed. Predicted features have a very interesting influence.\nI used them since Andrew released his great kernel, and all seemed good, but after -1.5 LB I decided to update them, and faced the problem I described above. Tho correlation was almost 1.0 both with old features and new ones.\nOr there's just a simplier explanation - I'm just doing smth wrong ¯\\_(ツ)_/¯", "votes": null }, { "id": "571270", "postDate": "07/09/2019 12:24:19", "content": "@littlebeandog I have the same reaction as Jacek. Where do you get this from?
\n\n\n", "rawMarkdown": "littlebeandog I have the same reaction as Jacek. Where do you get this from?\n\n> To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.", "votes": null }, { "id": "571277", "postDate": "07/09/2019 12:39:56", "content": "To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.
\n
If you add useless features, those features could be noise for your model's input, so definitely it will lead to worse performance. So it's important to select useful features.
", "rawMarkdown": "If you add useless features, those features could be noise for your model's input, so definitely it will lead to worse performance. So it's important to select useful features.", "votes": null }, { "id": "571292", "postDate": "07/09/2019 13:13:02", "content": "Emm....I think GBT family will iterate over all splits of all feature when spliting a node.\nIf the feature is useless to gain, it wont be chosen anyway. So just to my understanding adding useless feature wont harm the model.\nEarly stop can avoid overfitting anyway.\nDid I misunderstand something here?
", "rawMarkdown": "Emm....I think GBT family will iterate over all splits of all feature when spliting a node.\nIf the feature is useless to gain, it wont be chosen anyway. So just to my understanding adding useless feature wont harm the model.\nEarly stop can avoid overfitting anyway.\nDid I misunderstand something here?", "votes": null }, { "id": "571300", "postDate": "07/09/2019 13:26:37", "content": "You are right for this: the more feature, the more fit to training data. Adding features will improve the score on training data. But it can lead to worse score on test, precisely because it fits training data too well.
", "rawMarkdown": "You are right for this: the more feature, the more fit to training data. Adding features will improve the score on training data. But it can lead to worse score on test, precisely because it fits training data too well.", "votes": null }, { "id": "571387", "postDate": "07/09/2019 14:51:05", "content": "OH I think I know what's wrong.....thanks for pointing that out
", "rawMarkdown": "OH I think I know what's wrong.....thanks for pointing that out", "votes": null } ], "comments": [ { "id": 570421, "author_name": "scirpus", "author_url": "", "post_date": "07/08/2019 09:08:05", "content": "Did you check the features were correlated with scalar coupling - have a look at Giba's excellent script or the python version.
\n\nAlso I am trying to predict fc,pso,sdo and sd which are used to calculate scalar coupling.\nTypically the error should be less if you include predictions for these 4 variables.
", "votes": null, "replies": [ { "id": 570447, "author_name": "littlebeandog", "author_url": "", "post_date": "07/08/2019 09:58:04", "content": "Thanks for your advice! Do you mean the R kernel? I'll take a look.
", "votes": null, "replies": [] }, { "id": 570458, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/08/2019 10:15:40", "content": "@scirpus doesn't your model just remember fc feature? In theory it's almost 1.0 correlation with target, so for exaple LGB would just remember this feature and other features just wont have any influence
", "votes": null, "replies": [] }, { "id": 570493, "author_name": "scirpus", "author_url": "", "post_date": "07/08/2019 11:24:15", "content": "@stanislavblinov - it seems the error associated with scalar coupling is bigger than predicting fc,sd,pso,dso so I am hoping that incorporating all four make predicting scalar coupling easier.
\n\nIt is a hypothesis at the minute but that is what research is all about!
", "votes": null, "replies": [] }, { "id": 570496, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/08/2019 11:33:03", "content": "\n\n\nthat is what research is all about!
\n
Agreed. Predicted features have a very interesting influence.\nI used them since Andrew released his great kernel, and all seemed good, but after -1.5 LB I decided to update them, and faced the problem I described above. Tho correlation was almost 1.0 both with old features and new ones.\nOr there's just a simplier explanation - I'm just doing smth wrong ¯_(ツ)_/¯
", "votes": null, "replies": [] } ] }, { "id": 570439, "author_name": "jacekpoplawski", "author_url": "", "post_date": "07/08/2019 09:46:32", "content": "\"To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.\"
\n\nOf course not. Feature selection is very important.
", "votes": null, "replies": [ { "id": 570449, "author_name": "littlebeandog", "author_url": "", "post_date": "07/08/2019 09:58:28", "content": "Dude I didnt get it why of course not.....
", "votes": null, "replies": [] }, { "id": 571270, "author_name": "cpmpml", "author_url": "", "post_date": "07/09/2019 12:24:19", "content": "@littlebeandog I have the same reaction as Jacek. Where do you get this from?
\n\n\n", "votes": null, "replies": [] }, { "id": 571292, "author_name": "littlebeandog", "author_url": "", "post_date": "07/09/2019 13:13:02", "content": "To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.
\n
Emm....I think GBT family will iterate over all splits of all feature when spliting a node.\nIf the feature is useless to gain, it wont be chosen anyway. So just to my understanding adding useless feature wont harm the model.\nEarly stop can avoid overfitting anyway.\nDid I misunderstand something here?
", "votes": null, "replies": [] }, { "id": 571300, "author_name": "cpmpml", "author_url": "", "post_date": "07/09/2019 13:26:37", "content": "You are right for this: the more feature, the more fit to training data. Adding features will improve the score on training data. But it can lead to worse score on test, precisely because it fits training data too well.
", "votes": null, "replies": [] }, { "id": 571387, "author_name": "littlebeandog", "author_url": "", "post_date": "07/09/2019 14:51:05", "content": "OH I think I know what's wrong.....thanks for pointing that out
", "votes": null, "replies": [] } ] }, { "id": 571277, "author_name": "leonshangguan", "author_url": "", "post_date": "07/09/2019 12:39:56", "content": "If you add useless features, those features could be noise for your model's input, so definitely it will lead to worse performance. So it's important to select useful features.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "570228": "To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting. Adding more feature may not help but at least wont harm current performance.\n\nBut when I add a set of generated features (mean/std/min/max of origin features), the `mae` of each round becomes worse than that on origin set. Also at convergence the CV is much worse. I've tunned a bunch of hyper params and almost sure it's not about hyper param tunning.\n\nCould someone help me out, why is this happenning and what to do to avoid this...", "570421": "Did you check the features were correlated with scalar coupling - have a look at Giba's excellent script or the python version.\n\nAlso I am trying to predict fc,pso,sdo and sd which are used to calculate scalar coupling.\nTypically the error should be less if you include predictions for these 4 variables.", "570439": "\"To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.\"\n\nOf course not. Feature selection is very important.", "570447": "Thanks for your advice! Do you mean the R kernel? I'll take a look.", "570449": "Dude I didnt get it why of course not.....", "570458": "scirpus doesn't your model just remember fc feature? In theory it's almost 1.0 correlation with target, so for exaple LGB would just remember this feature and other features just wont have any influence", "570493": "stanislavblinov - it seems the error associated with scalar coupling is bigger than predicting fc,sd,pso,dso so I am hoping that incorporating all four make predicting scalar coupling easier.\n\nIt is a hypothesis at the minute but that is what research is all about!", "570496": "> that is what research is all about!\n\nAgreed. Predicted features have a very interesting influence.\nI used them since Andrew released his great kernel, and all seemed good, but after -1.5 LB I decided to update them, and faced the problem I described above. Tho correlation was almost 1.0 both with old features and new ones.\nOr there's just a simplier explanation - I'm just doing smth wrong ¯\\_(ツ)_/¯", "571270": "littlebeandog I have the same reaction as Jacek. Where do you get this from?\n\n> To my understanding of GBM, it can theoretically figure out important feature and avoid overfitting.", "571277": "If you add useless features, those features could be noise for your model's input, so definitely it will lead to worse performance. So it's important to select useful features.", "571292": "Emm....I think GBT family will iterate over all splits of all feature when spliting a node.\nIf the feature is useless to gain, it wont be chosen anyway. So just to my understanding adding useless feature wont harm the model.\nEarly stop can avoid overfitting anyway.\nDid I misunderstand something here?", "571300": "You are right for this: the more feature, the more fit to training data. Adding features will improve the score on training data. But it can lead to worse score on test, precisely because it fits training data too well.", "571387": "OH I think I know what's wrong.....thanks for pointing that out" }, "source": "meta" }