I have some question about EDA and Feature Engineering.\nI have seen so many kernels about EDA and the top solution.I can use their methods to constrution many features(like use count,agg etc.)，and use these features in my model，but it hardly help my model to perform better.\nI think that's because I separate EDA from the idea of feature engineering.I don't konw how to combine EDA with feature engineering.I just can find some abnormal values and fix them.I realy realy want to know how can I verify a feature is useful.\nCan anybody tell me how can I do？
", "messages": [ { "id": "652765", "postDate": "10/19/2019 11:42:24", "content": "I have some question about EDA and Feature Engineering.\nI have seen so many kernels about EDA and the top solution.I can use their methods to constrution many features(like use count,agg etc.)，and use these features in my model，but it hardly help my model to perform better.\nI think that's because I separate EDA from the idea of feature engineering.I don't konw how to combine EDA with feature engineering.I just can find some abnormal values and fix them.I realy realy want to know how can I verify a feature is useful.\nCan anybody tell me how can I do？
", "rawMarkdown": "I have some question about EDA and Feature Engineering.\nI have seen so many kernels about EDA and the top solution.I can use their methods to constrution many features(like use count,agg etc.)，and use these features in my model，but it hardly help my model to perform better.\nI think that's because I separate EDA from the idea of feature engineering.I don't konw how to combine EDA with feature engineering.I just can find some abnormal values and fix them.I realy realy want to know how can I verify a feature is useful.\nCan anybody tell me how can I do？", "votes": null }, { "id": "652815", "postDate": "10/19/2019 12:59:36", "content": "Hello, that's why we calculate feature importance . For example you may check this kernel in this competition. As long as the feature importance in high we may deduce that feature in useful . Now if you want to calculate feature importance, LightGBM, XGBoost already provide that attribute. You can always check for the feature importance after model train and see where the new features stand.
", "rawMarkdown": "Hello, that's why we calculate feature importance . For example you may check this [kernel](https://www.kaggle.com/divrikwicky/ashrae-lofo-feature-importance-on-best-pub-kernel) in this competition. As long as the feature importance in high we may deduce that feature in useful . Now if you want to calculate feature importance, LightGBM, XGBoost already provide that attribute. You can always check for the feature importance after model train and see where the new features stand.", "votes": null }, { "id": "652910", "postDate": "10/19/2019 16:00:09", "content": "I can advice you three decision:\n 1. You can try used any linear model(logistic regression, decision tree and i. e.) and based on trained model, contruct feature importance graph. \n 2. You can used correltion coefficients based on wich you can make conclusion about feature importance. \n 3. Trust ml algorithms. You can used simple sklearn feature selector, which based on principces described before, or used more complex like GA algorithms.
", "rawMarkdown": "I can advice you three decision:\n 1. You can try used any linear model(logistic regression, decision tree and i. e.) and based on trained model, contruct feature importance graph. \n 2. You can used correltion coefficients based on wich you can make conclusion about feature importance. \n 3. Trust ml algorithms. You can used simple sklearn feature selector, which based on principces described before, or used more complex like GA algorithms.", "votes": null }, { "id": "664171", "postDate": "11/03/2019 09:22:48", "content": "when you know the answer ,can you tell me?
", "rawMarkdown": "when you know the answer ,can you tell me?", "votes": null }, { "id": "665297", "postDate": "11/04/2019 21:23:25", "content": "The answer (the features selection stage) is at my kernels:\n* ASHRAE - Automatic FE: Featuretools & Selection FE\n* ASHRAE - Automatic FE&FC
", "rawMarkdown": "The answer (the features selection stage) is at my kernels:\n* [ASHRAE - Automatic FE: Featuretools & Selection FE](https://www.kaggle.com/vbmokin/ashrae-automatic-fe-featuretools-selection-fe)\n* [ASHRAE - Automatic FE&FC](https://www.kaggle.com/vbmokin/ashrae-automatic-fe-fc)", "votes": null }, { "id": "704968", "postDate": "12/28/2019 08:17:42", "content": "thank you very much!
", "rawMarkdown": "thank you very much!", "votes": null }, { "id": "704970", "postDate": "12/28/2019 08:18:34", "content": "thank you!
", "rawMarkdown": "thank you!", "votes": null }, { "id": "704971", "postDate": "12/28/2019 08:19:14", "content": "thank you!
", "rawMarkdown": "thank you!", "votes": null } ], "comments": [ { "id": 652815, "author_name": "khairulislam", "author_url": "", "post_date": "10/19/2019 12:59:36", "content": "Hello, that's why we calculate feature importance . For example you may check this kernel in this competition. As long as the feature importance in high we may deduce that feature in useful . Now if you want to calculate feature importance, LightGBM, XGBoost already provide that attribute. You can always check for the feature importance after model train and see where the new features stand.
", "votes": null, "replies": [ { "id": 704971, "author_name": "laqwei", "author_url": "", "post_date": "12/28/2019 08:19:14", "content": "thank you!
", "votes": null, "replies": [] } ] }, { "id": 652910, "author_name": "miklgr500", "author_url": "", "post_date": "10/19/2019 16:00:09", "content": "I can advice you three decision:\n 1. You can try used any linear model(logistic regression, decision tree and i. e.) and based on trained model, contruct feature importance graph. \n 2. You can used correltion coefficients based on wich you can make conclusion about feature importance. \n 3. Trust ml algorithms. You can used simple sklearn feature selector, which based on principces described before, or used more complex like GA algorithms.
", "votes": null, "replies": [ { "id": 704970, "author_name": "laqwei", "author_url": "", "post_date": "12/28/2019 08:18:34", "content": "thank you!
", "votes": null, "replies": [] } ] }, { "id": 664171, "author_name": "juezhanangle", "author_url": "", "post_date": "11/03/2019 09:22:48", "content": "when you know the answer ,can you tell me?
", "votes": null, "replies": [] }, { "id": 665297, "author_name": "vbmokin", "author_url": "", "post_date": "11/04/2019 21:23:25", "content": "The answer (the features selection stage) is at my kernels:\n* ASHRAE - Automatic FE: Featuretools & Selection FE\n* ASHRAE - Automatic FE&FC
", "votes": null, "replies": [ { "id": 704968, "author_name": "laqwei", "author_url": "", "post_date": "12/28/2019 08:17:42", "content": "thank you very much!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "652765": "I have some question about EDA and Feature Engineering.\nI have seen so many kernels about EDA and the top solution.I can use their methods to constrution many features(like use count,agg etc.)，and use these features in my model，but it hardly help my model to perform better.\nI think that's because I separate EDA from the idea of feature engineering.I don't konw how to combine EDA with feature engineering.I just can find some abnormal values and fix them.I realy realy want to know how can I verify a feature is useful.\nCan anybody tell me how can I do？", "652815": "Hello, that's why we calculate feature importance . For example you may check this [kernel](https://www.kaggle.com/divrikwicky/ashrae-lofo-feature-importance-on-best-pub-kernel) in this competition. As long as the feature importance in high we may deduce that feature in useful . Now if you want to calculate feature importance, LightGBM, XGBoost already provide that attribute. You can always check for the feature importance after model train and see where the new features stand.", "652910": "I can advice you three decision:\n 1. You can try used any linear model(logistic regression, decision tree and i. e.) and based on trained model, contruct feature importance graph. \n 2. You can used correltion coefficients based on wich you can make conclusion about feature importance. \n 3. Trust ml algorithms. You can used simple sklearn feature selector, which based on principces described before, or used more complex like GA algorithms.", "664171": "when you know the answer ,can you tell me?", "665297": "The answer (the features selection stage) is at my kernels:\n* [ASHRAE - Automatic FE: Featuretools & Selection FE](https://www.kaggle.com/vbmokin/ashrae-automatic-fe-featuretools-selection-fe)\n* [ASHRAE - Automatic FE&FC](https://www.kaggle.com/vbmokin/ashrae-automatic-fe-fc)", "704968": "thank you very much!", "704970": "thank you!", "704971": "thank you!" }, "source": "meta" }