I am a bit surprised by my own findings. Maybe experienced Kagglers can explain what is going on.
\n\nI train one xgboost model on a dataset of all engineered features. I did not contain any of the actual data. It was roughly 60 features, all created by me. I got a LB score of 0.140661.
I train another xgboost model on part of the train_numeric data. I got a LB score of about 0.18731.
Finally I said, I'll make a big dataset: my engineered features + the subset of the numeric data that I used before. LB score was about 0.19x.
I was not expecting my score to be the sum of the scores I got in the two previous attempts but I was not expecting it to be improved so little either. Can anyone explain a bit how does this work? Is there any suggested reading on the topic?
", "messages": [ { "id": "138107", "postDate": "10/07/2016 08:15:06", "content": "I am a bit surprised by my own findings. Maybe experienced Kagglers can explain what is going on.
\n\nI train one xgboost model on a dataset of all engineered features. I did not contain any of the actual data. It was roughly 60 features, all created by me. I got a LB score of 0.140661.
I train another xgboost model on part of the train_numeric data. I got a LB score of about 0.18731.
Finally I said, I'll make a big dataset: my engineered features + the subset of the numeric data that I used before. LB score was about 0.19x.
I was not expecting my score to be the sum of the scores I got in the two previous attempts but I was not expecting it to be improved so little either. Can anyone explain a bit how does this work? Is there any suggested reading on the topic?
", "rawMarkdown": "I am a bit surprised by my own findings. Maybe experienced Kagglers can explain what is going on.\r\n\r\n1. I train one xgboost model on a dataset of all engineered features. I did not contain any of the actual data. It was roughly 60 features, all created by me. I got a LB score of 0.140661.\r\n\r\n2. I train another xgboost model on part of the train_numeric data. I got a LB score of about 0.18731.\r\n\r\n3. Finally I said, I'll make a big dataset: my engineered features + the subset of the numeric data that I used before. LB score was about 0.19x.\r\n\r\nI was not expecting my score to be the sum of the scores I got in the two previous attempts but I was not expecting it to be improved so little either. Can anyone explain a bit how does this work? Is there any suggested reading on the topic?", "votes": null }, { "id": "138112", "postDate": "10/07/2016 08:39:26", "content": "The key is how similar the features are in explaining the target variable. From what you describe your engineered features help to classify a similar set of y-values than the original train_numeric data, meaning that there is a huge information overlap. For example some observations can be easily classified from the numerical S24 variables. If you go ahead and create new features, such as max(S24), mean(S24) you might not add much new information to the model as this observations were already classified correct before. In general the more different the kind of features are and the more different the aspects of the target they explain the more gain you can expect from merging them. For example some well engineered time features would probably give a bigger boost in LB score when merged with the numeric.\nThis is not only true for XGBoost but for all machine learning models. In this regard XGBoost is actually quite forgiving, as you dont have to care about colinearity between variables and overfitting so much.
", "rawMarkdown": "The key is how similar the features are in explaining the target variable. From what you describe your engineered features help to classify a similar set of y-values than the original train_numeric data, meaning that there is a huge information overlap. For example some observations can be easily classified from the numerical S24 variables. If you go ahead and create new features, such as max(S24), mean(S24) you might not add much new information to the model as this observations were already classified correct before. In general the more different the kind of features are and the more different the aspects of the target they explain the more gain you can expect from merging them. For example some well engineered time features would probably give a bigger boost in LB score when merged with the numeric.\r\nThis is not only true for XGBoost but for all machine learning models. In this regard XGBoost is actually quite forgiving, as you dont have to care about colinearity between variables and overfitting so much.", "votes": null }, { "id": "138114", "postDate": "10/07/2016 08:56:06", "content": "Thank you for your reply, Malte! I don't think there is much overlap between my engineered features and the numeric ones, meaning I did not create the engineered features from the numeric dataset. In fact when I plot feature importance, 4 of my engineered features rank on top. My take on this is that they are actually quite useful.
\n\nI get that probably they two sets of data are helping classify correctly the same datapoints, you are right there. I wonder what am I missing...
", "rawMarkdown": "Thank you for your reply, Malte! I don't think there is much overlap between my engineered features and the numeric ones, meaning I did not create the engineered features from the numeric dataset. In fact when I plot feature importance, 4 of my engineered features rank on top. My take on this is that they are actually quite useful.\r\n\r\nI get that probably they two sets of data are helping classify correctly the same datapoints, you are right there. I wonder what am I missing...", "votes": null }, { "id": "138391", "postDate": "10/08/2016 14:32:40", "content": "I'm not sure, but i think if you measure the correlation between your engineered features and the original ones you can get an idea about the overlap
", "rawMarkdown": "I'm not sure, but i think if you measure the correlation between your engineered features and the original ones you can get an idea about the overlap", "votes": null } ], "comments": [ { "id": 138112, "author_name": "malten", "author_url": "", "post_date": "10/07/2016 08:39:26", "content": "The key is how similar the features are in explaining the target variable. From what you describe your engineered features help to classify a similar set of y-values than the original train_numeric data, meaning that there is a huge information overlap. For example some observations can be easily classified from the numerical S24 variables. If you go ahead and create new features, such as max(S24), mean(S24) you might not add much new information to the model as this observations were already classified correct before. In general the more different the kind of features are and the more different the aspects of the target they explain the more gain you can expect from merging them. For example some well engineered time features would probably give a bigger boost in LB score when merged with the numeric.\nThis is not only true for XGBoost but for all machine learning models. In this regard XGBoost is actually quite forgiving, as you dont have to care about colinearity between variables and overfitting so much.
", "votes": null, "replies": [] }, { "id": 138114, "author_name": "yamiomar88", "author_url": "", "post_date": "10/07/2016 08:56:06", "content": "Thank you for your reply, Malte! I don't think there is much overlap between my engineered features and the numeric ones, meaning I did not create the engineered features from the numeric dataset. In fact when I plot feature importance, 4 of my engineered features rank on top. My take on this is that they are actually quite useful.
\n\nI get that probably they two sets of data are helping classify correctly the same datapoints, you are right there. I wonder what am I missing...
", "votes": null, "replies": [] }, { "id": 138391, "author_name": "abriosi", "author_url": "", "post_date": "10/08/2016 14:32:40", "content": "I'm not sure, but i think if you measure the correlation between your engineered features and the original ones you can get an idea about the overlap
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "138107": "I am a bit surprised by my own findings. Maybe experienced Kagglers can explain what is going on.\r\n\r\n1. I train one xgboost model on a dataset of all engineered features. I did not contain any of the actual data. It was roughly 60 features, all created by me. I got a LB score of 0.140661.\r\n\r\n2. I train another xgboost model on part of the train_numeric data. I got a LB score of about 0.18731.\r\n\r\n3. Finally I said, I'll make a big dataset: my engineered features + the subset of the numeric data that I used before. LB score was about 0.19x.\r\n\r\nI was not expecting my score to be the sum of the scores I got in the two previous attempts but I was not expecting it to be improved so little either. Can anyone explain a bit how does this work? Is there any suggested reading on the topic?", "138112": "The key is how similar the features are in explaining the target variable. From what you describe your engineered features help to classify a similar set of y-values than the original train_numeric data, meaning that there is a huge information overlap. For example some observations can be easily classified from the numerical S24 variables. If you go ahead and create new features, such as max(S24), mean(S24) you might not add much new information to the model as this observations were already classified correct before. In general the more different the kind of features are and the more different the aspects of the target they explain the more gain you can expect from merging them. For example some well engineered time features would probably give a bigger boost in LB score when merged with the numeric.\r\nThis is not only true for XGBoost but for all machine learning models. In this regard XGBoost is actually quite forgiving, as you dont have to care about colinearity between variables and overfitting so much.", "138114": "Thank you for your reply, Malte! I don't think there is much overlap between my engineered features and the numeric ones, meaning I did not create the engineered features from the numeric dataset. In fact when I plot feature importance, 4 of my engineered features rank on top. My take on this is that they are actually quite useful.\r\n\r\nI get that probably they two sets of data are helping classify correctly the same datapoints, you are right there. I wonder what am I missing...", "138391": "I'm not sure, but i think if you measure the correlation between your engineered features and the original ones you can get an idea about the overlap" }, "source": "meta" }