Perhaps someone can help me here. So 2 weeks ago I used XGB and pulled feature importance for each type, top 60 for each. Being lazy and since the same function on LightGBM is slow, I simply used the same on my LightGBM model, and interestingly got a similar CV score as with XGB. By close I mean 2 decimals close.
\n\nA week ago I finally got around to doing same from scratch with LightGBM and as I expected got different feature importance. Call this model A with feature importance A - top 60 features.
\n\nToday I build a new model, and run feature importance on each type again, yielding model B with feature importance B - top 60 features.
\n\nI would expect that feature importance B would give the best CV with model B, right? Wrong.\nUsing the top 60 from A, on model B, consistently gives me a better score. On totally different model parameters. How is this possible?
\n\nIt appears to me XGB's feature importance is not only faster, but overall results in better scores, even when used with LightGBM.
\n\nCan anyone explain to me what I am doing wrong?
", "messages": [ { "id": "569314", "postDate": "07/06/2019 12:40:38", "content": "Perhaps someone can help me here. So 2 weeks ago I used XGB and pulled feature importance for each type, top 60 for each. Being lazy and since the same function on LightGBM is slow, I simply used the same on my LightGBM model, and interestingly got a similar CV score as with XGB. By close I mean 2 decimals close.
\n\nA week ago I finally got around to doing same from scratch with LightGBM and as I expected got different feature importance. Call this model A with feature importance A - top 60 features.
\n\nToday I build a new model, and run feature importance on each type again, yielding model B with feature importance B - top 60 features.
\n\nI would expect that feature importance B would give the best CV with model B, right? Wrong.\nUsing the top 60 from A, on model B, consistently gives me a better score. On totally different model parameters. How is this possible?
\n\nIt appears to me XGB's feature importance is not only faster, but overall results in better scores, even when used with LightGBM.
\n\nCan anyone explain to me what I am doing wrong?
", "rawMarkdown": "Perhaps someone can help me here. So 2 weeks ago I used XGB and pulled feature importance for each type, top 60 for each. Being lazy and since the same function on LightGBM is slow, I simply used the same on my LightGBM model, and interestingly got a similar CV score as with XGB. By close I mean 2 decimals close.\n\nA week ago I finally got around to doing same from scratch with LightGBM and as I expected got different feature importance. Call this model A with feature importance A - top 60 features.\n\nToday I build a new model, and run feature importance on each type again, yielding model B with feature importance B - top 60 features.\n\nI would expect that feature importance B would give the best CV with model B, right? Wrong.\nUsing the top 60 from A, on model B, consistently gives me a better score. On totally different model parameters. How is this possible?\n\nIt appears to me XGB's feature importance is not only faster, but overall results in better scores, even when used with LightGBM.\n\nCan anyone explain to me what I am doing wrong?", "votes": null }, { "id": "569338", "postDate": "07/06/2019 13:36:26", "content": "Feature importance is based on how features are used to build trees. Not how the overall score evolves if you remove the feature. It can happen that removing the most important feature improves model quality.
", "rawMarkdown": "Feature importance is based on how features are used to build trees. Not how the overall score evolves if you remove the feature. It can happen that removing the most important feature improves model quality.", "votes": null }, { "id": "569348", "postDate": "07/06/2019 13:54:31", "content": "As CPMP explained, the importances you were looking at are not calculated based on model performance with/without them. I would suggest you to look at LOFO (Leave-One-Feature-Out) Feature Importance.
\n\nhttps://github.com/aerdem4/lofo-importance
", "rawMarkdown": "As CPMP explained, the importances you were looking at are not calculated based on model performance with/without them. I would suggest you to look at LOFO (Leave-One-Feature-Out) Feature Importance.\n\nhttps://github.com/aerdem4/lofo-importance", "votes": null }, { "id": "569355", "postDate": "07/06/2019 14:02:46", "content": "Is LOFO better then permutation importance (when model isn't retrained each time)?
", "rawMarkdown": "Is LOFO better then permutation importance (when model isn't retrained each time)?", "votes": null }, { "id": "569362", "postDate": "07/06/2019 14:19:50", "content": "One of the main advantage of LOFO is that it depends on the validation scheme. I would suggest you to check OS Version example from Microsoft Malware Competition on README.
\n\nThere is also FLOFO (Fast LOFO) which is like permutation importance but a bit different.
\n\nhttps://www.youtube.com/watch?v=9Vv9UcK-vag\nhttps://www.youtube.com/watch?v=zqsQ2ojj7sE
", "rawMarkdown": "One of the main advantage of LOFO is that it depends on the validation scheme. I would suggest you to check OS Version example from Microsoft Malware Competition on README.\n\nThere is also FLOFO (Fast LOFO) which is like permutation importance but a bit different.\n\nhttps://www.youtube.com/watch?v=9Vv9UcK-vag\nhttps://www.youtube.com/watch?v=zqsQ2ojj7sE", "votes": null }, { "id": "569404", "postDate": "07/06/2019 15:27:06", "content": "Wouldn't it take too much time to train model each time?
", "rawMarkdown": "Wouldn't it take too much time to train model each time?", "votes": null }, { "id": "569777", "postDate": "07/07/2019 10:01:13", "content": "I would say that permutations are more consistent as the dataset in terms of feature order remains the same which is relevant for any columns sampling algos. On the other hand the permutation add an additional noise which though random might trigger model to react to it. I do not have any personal bias but tent to use LOFO more than permutations. I believe that it is more consistent as it reduces dimentionality of the model and this is also a desirable feature on its own :)
", "rawMarkdown": "I would say that permutations are more consistent as the dataset in terms of feature order remains the same which is relevant for any columns sampling algos. On the other hand the permutation add an additional noise which though random might trigger model to react to it. I do not have any personal bias but tent to use LOFO more than permutations. I believe that it is more consistent as it reduces dimentionality of the model and this is also a desirable feature on its own :)", "votes": null }, { "id": "569785", "postDate": "07/07/2019 10:42:04", "content": "Thanks for the great feedback everyone, much appreciated.
", "rawMarkdown": "Thanks for the great feedback everyone, much appreciated.", "votes": null } ], "comments": [ { "id": 569338, "author_name": "cpmpml", "author_url": "", "post_date": "07/06/2019 13:36:26", "content": "Feature importance is based on how features are used to build trees. Not how the overall score evolves if you remove the feature. It can happen that removing the most important feature improves model quality.
", "votes": null, "replies": [ { "id": 569348, "author_name": "aerdem4", "author_url": "", "post_date": "07/06/2019 13:54:31", "content": "As CPMP explained, the importances you were looking at are not calculated based on model performance with/without them. I would suggest you to look at LOFO (Leave-One-Feature-Out) Feature Importance.
\n\nhttps://github.com/aerdem4/lofo-importance
", "votes": null, "replies": [] }, { "id": 569355, "author_name": "artgor", "author_url": "", "post_date": "07/06/2019 14:02:46", "content": "Is LOFO better then permutation importance (when model isn't retrained each time)?
", "votes": null, "replies": [] }, { "id": 569362, "author_name": "aerdem4", "author_url": "", "post_date": "07/06/2019 14:19:50", "content": "One of the main advantage of LOFO is that it depends on the validation scheme. I would suggest you to check OS Version example from Microsoft Malware Competition on README.
\n\nThere is also FLOFO (Fast LOFO) which is like permutation importance but a bit different.
\n\nhttps://www.youtube.com/watch?v=9Vv9UcK-vag\nhttps://www.youtube.com/watch?v=zqsQ2ojj7sE
", "votes": null, "replies": [] }, { "id": 569404, "author_name": "artgor", "author_url": "", "post_date": "07/06/2019 15:27:06", "content": "Wouldn't it take too much time to train model each time?
", "votes": null, "replies": [] }, { "id": 569777, "author_name": "snovik1975", "author_url": "", "post_date": "07/07/2019 10:01:13", "content": "I would say that permutations are more consistent as the dataset in terms of feature order remains the same which is relevant for any columns sampling algos. On the other hand the permutation add an additional noise which though random might trigger model to react to it. I do not have any personal bias but tent to use LOFO more than permutations. I believe that it is more consistent as it reduces dimentionality of the model and this is also a desirable feature on its own :)
", "votes": null, "replies": [] }, { "id": 569785, "author_name": "gideonvos", "author_url": "", "post_date": "07/07/2019 10:42:04", "content": "Thanks for the great feedback everyone, much appreciated.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "569314": "Perhaps someone can help me here. So 2 weeks ago I used XGB and pulled feature importance for each type, top 60 for each. Being lazy and since the same function on LightGBM is slow, I simply used the same on my LightGBM model, and interestingly got a similar CV score as with XGB. By close I mean 2 decimals close.\n\nA week ago I finally got around to doing same from scratch with LightGBM and as I expected got different feature importance. Call this model A with feature importance A - top 60 features.\n\nToday I build a new model, and run feature importance on each type again, yielding model B with feature importance B - top 60 features.\n\nI would expect that feature importance B would give the best CV with model B, right? Wrong.\nUsing the top 60 from A, on model B, consistently gives me a better score. On totally different model parameters. How is this possible?\n\nIt appears to me XGB's feature importance is not only faster, but overall results in better scores, even when used with LightGBM.\n\nCan anyone explain to me what I am doing wrong?", "569338": "Feature importance is based on how features are used to build trees. Not how the overall score evolves if you remove the feature. It can happen that removing the most important feature improves model quality.", "569348": "As CPMP explained, the importances you were looking at are not calculated based on model performance with/without them. I would suggest you to look at LOFO (Leave-One-Feature-Out) Feature Importance.\n\nhttps://github.com/aerdem4/lofo-importance", "569355": "Is LOFO better then permutation importance (when model isn't retrained each time)?", "569362": "One of the main advantage of LOFO is that it depends on the validation scheme. I would suggest you to check OS Version example from Microsoft Malware Competition on README.\n\nThere is also FLOFO (Fast LOFO) which is like permutation importance but a bit different.\n\nhttps://www.youtube.com/watch?v=9Vv9UcK-vag\nhttps://www.youtube.com/watch?v=zqsQ2ojj7sE", "569404": "Wouldn't it take too much time to train model each time?", "569777": "I would say that permutations are more consistent as the dataset in terms of feature order remains the same which is relevant for any columns sampling algos. On the other hand the permutation add an additional noise which though random might trigger model to react to it. I do not have any personal bias but tent to use LOFO more than permutations. I believe that it is more consistent as it reduces dimentionality of the model and this is also a desirable feature on its own :)", "569785": "Thanks for the great feedback everyone, much appreciated." }, "source": "meta" }