![\"feature](\"http://2.bp.blogspot.com/-bjRCSHtwAI0/Uv30_ZlxojI/AAAAAAAABmo/dX2Z3nwmKMY/s1600/Screen+Shot+2014-02-14+at+12.49.50+PM.png\")
I am struggling to understand why my vw important features are significantly different that from say XGboost or a Random forest.
\n\nI noticed this on a previous competition--since our data is normalized (numeric data) the coefficient should be the feature weight (annd i did the vw-varinfo--just to be sure)? Now I don't expect the same numbers, but I do expect similar orders (close would be nice, im not even close)
\n\nAnd yet in vw, I get an entirely different set of features (My random forest I am guessing is accurate as it agrees with what others found)! Can anyone help me along to the path of enlightenment?
", "messages": [ { "id": "136565", "postDate": "09/22/2016 23:31:42", "content": "I am struggling to understand why my vw important features are significantly different that from say XGboost or a Random forest.
\n\nI noticed this on a previous competition--since our data is normalized (numeric data) the coefficient should be the feature weight (annd i did the vw-varinfo--just to be sure)? Now I don't expect the same numbers, but I do expect similar orders (close would be nice, im not even close)
\n\nAnd yet in vw, I get an entirely different set of features (My random forest I am guessing is accurate as it agrees with what others found)! Can anyone help me along to the path of enlightenment?
", "rawMarkdown": "I am struggling to understand why my vw important features are significantly different that from say XGboost or a Random forest.\r\n\r\nI noticed this on a previous competition--since our data is normalized (numeric data) the coefficient should be the feature weight (annd i did the vw-varinfo--just to be sure)? Now I don't expect the same numbers, but I do expect similar orders (close would be nice, im not even close)\r\n\r\nAnd yet in vw, I get an entirely different set of features (My random forest I am guessing is accurate as it agrees with what others found)! Can anyone help me along to the path of enlightenment?", "votes": null }, { "id": "136568", "postDate": "09/23/2016 01:16:04", "content": "To get technical, you need to read their papers or understand how the algorithm works, then, it becomes (sort of) obvious why they differ.
\n\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.
\n\nVW: Speaks Spanish and some English
\n\nXGB and RF, both speak English and some Spanish.
\n\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.
", "rawMarkdown": "To get technical, you need to read their papers or understand how the algorithm works, then, it becomes (sort of) obvious why they differ.\r\n\r\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.\r\n\r\nVW: Speaks Spanish and some English\r\n\r\nXGB and RF, both speak English and some Spanish.\r\n\r\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.", "votes": null }, { "id": "136572", "postDate": "09/23/2016 02:16:31", "content": "THey actually doesn't make a lot of sense. Information is information--its not the values--its the actual features are different--3 algorithms say column X, Y , + Z are important--a fourth says A B +C--i would expect something more like X Y Z and say X Y and A. Not two different sets of features.
", "rawMarkdown": "THey actually doesn't make a lot of sense. Information is information--its not the values--its the actual features are different--3 algorithms say column X, Y , + Z are important--a fourth says A B +C--i would expect something more like X Y Z and say X Y and A. Not two different sets of features.", "votes": null }, { "id": "136585", "postDate": "09/23/2016 06:19:12", "content": "Because xgb and RF are tree-based and they look at regions of a particular feature. Other algorithms like linear/logistic regression look at the general trend of that feature. The importance of xgb/rf is measured by how many times a feature pops up in a tree, for logreg/linreg it's the individual predictive power of the trend of that feature towards the target.
\n\nSo features that have clear trends and good predictive power pop up as important in vw; features that have many subregions that have strong predictive power towards the target eventually pop up in xgb/rf, because the number of trees using that feature somehow (selecting a region and boost) define how important that feature is.
\n\nAt least that's my intuitive understanding of it.
", "rawMarkdown": "Because xgb and RF are tree-based and they look at regions of a particular feature. Other algorithms like linear/logistic regression look at the general trend of that feature. The importance of xgb/rf is measured by how many times a feature pops up in a tree, for logreg/linreg it's the individual predictive power of the trend of that feature towards the target.\r\n\r\nSo features that have clear trends and good predictive power pop up as important in vw; features that have many subregions that have strong predictive power towards the target eventually pop up in xgb/rf, because the number of trees using that feature somehow (selecting a region and boost) define how important that feature is.\r\n\r\nAt least that's my intuitive understanding of it.", "votes": null }, { "id": "136612", "postDate": "09/23/2016 11:15:41", "content": "[quote=Bob, Lord of Cats;136572]
\n\nTHey actually doesn't make a lot of sense. Information is information
\n\n[/quote]
\n\nFeature importance is not computed the same way between xgboost and VW, but also between a LR and xgboost/VW.
\n\nGetting (non)-technical (with simple questions) in a generalized way, you would ask:
\n\nThen, you can ask yourself these additional questions when picking a specific quantification metric:
\n\nIf you pick the gain ratio, answer yes to question 1 & 2, answer no to question 3. But this is true for all quantification metrics.
\n\nAlso, it is up to the developer to choose which feature importance metric they want to use (and how it is calculated). Actually, the best features in xgboost might not be the ones providing the highest gain (due to how biased that quantity is in a tree-based model).
\n\nA bit off-topic, but relevant to feature importance algorithms: some businesses are adding "super duper cool" feature importance metrics to sell their products. Not calling a name, but one of these is using a "super duper cool" name by hiding the fact it is just the rank of features issued from a feature importance algorithm. Hence, if you have 10 features and only 1 is relevant, it would report junk features as "important" (who knows if "hates pizzas" is not relevant to the prediction "likes pizzas" because it was shown 10th in the features of the "super duper cool" algorithm... yes it is 10th, so what? You just didn't rescale your data beforehand, its weight is 1e-20 unscaled when it is 1 scaled properly.).
\n\nThe non-technical explanation was provided by NxGTR:
\n\n[quote=NxGTR;136568]
\n\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.
\n\nVW: Speaks Spanish and some English
\n\nXGB and RF, both speak English and some Spanish.
\n\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.
\n\n[/quote]
\n\nAnother (and metaphorical) example from a sentence: "I eventually did it!"
\n\nBecause "eventually" is not interpreted the same way in French ("English: if appropriate" - "French: le cas échéant" <=? similiarity ?=> "éventuellement") than in German.
\n\nHence, between French people they would agree for the feature importance (even if it is wrong), while with an German businessman speaking English they would disagree (comparing Apple with Oranges even if the main source of comparison is exactly the same thing). Add a third language, and you may even have a 3rd source of disagreement.
\n\nObviously, this does not generalize to all sentences. But there will be always disagreements somewhere due to the language barrier.
", "rawMarkdown": "[quote=Bob, Lord of Cats;136572]\r\n\r\nTHey actually doesn't make a lot of sense. Information is information\r\n\r\n[/quote]\r\n\r\nFeature importance is not computed the same way between xgboost and VW, but also between a LR and xgboost/VW.\r\n\r\nGetting (non)-technical (with simple questions) in a generalized way, you would ask:\r\n\r\n* Can you quantify a feature importance? (answer: by using a metric) \r\n* Are there existing unbiased quantifiers? (answer: no)\r\n* How to assess sample (tree) vs population (linear) feature importance? (should data be rescaled beforehand? etc.)\r\n* Is there a quantifier that works on any type of model? (answer: yes - just ignore the model and go univariate)\r\n\r\nThen, you can ask yourself these additional questions when picking a specific quantification metric:\r\n\r\n* Is that quantifier biased? (answer: yes)\r\n* Can you make that quantifier unbiased (for its specific usage) towards the population (answer: yes, by cross-validating)\r\n* Can you make that quantifier unbiased in a general case? (answer: no, never - this is a still an area being actively researched)\r\n\r\nIf you pick the gain ratio, answer yes to question 1 & 2, answer no to question 3. But this is true for all quantification metrics.\r\n\r\nAlso, it is up to the developer to choose which feature importance metric they want to use (and how it is calculated). Actually, the best features in xgboost might not be the ones providing the highest gain (due to how biased that quantity is in a tree-based model).\r\n\r\nA bit off-topic, but relevant to feature importance algorithms: some businesses are adding \"super duper cool\" feature importance metrics to sell their products. Not calling a name, but one of these is using a \"super duper cool\" name by hiding the fact it is just the rank of features issued from a feature importance algorithm. Hence, if you have 10 features and only 1 is relevant, it would report junk features as \"important\" (*who knows if \"hates pizzas\" is not relevant to the prediction \"likes pizzas\" because it was shown 10th in the features of the \"super duper cool\" algorithm... yes it is 10th, so what? You just didn't rescale your data beforehand, its weight is 1e-20 unscaled when it is 1 scaled properly.*).\r\n\r\nThe non-technical explanation was provided by NxGTR:\r\n\r\n[quote=NxGTR;136568]\r\n\r\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.\r\n\r\nVW: Speaks Spanish and some English\r\n\r\nXGB and RF, both speak English and some Spanish.\r\n\r\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.\r\n\r\n[/quote]\r\n\r\nAnother (and metaphorical) example from a sentence: \"I eventually did it!\"\r\n\r\n* German (perfect) interpretation: \"In the end I did it!\" (feature importance of words correct for speaking English)\r\n* French approximate (wrong word + wrong correction) interpretation: \"If appropriate I'll do it!\" (feature importance of words incorrect for speaking English)\r\n\r\nBecause \"eventually\" is not interpreted the same way in French (\"English: if appropriate\" - \"French: le cas échéant\" <=? similiarity ?=> \"éventuellement\") than in German.\r\n\r\nHence, between French people they would agree for the feature importance (even if it is wrong), while with an German businessman speaking English they would disagree (comparing Apple with Oranges even if the main source of comparison is exactly the same thing). Add a third language, and you may even have a 3rd source of disagreement.\r\n\r\nObviously, this does not generalize to all sentences. But there will be always disagreements somewhere due to the language barrier.", "votes": null }, { "id": "136618", "postDate": "09/23/2016 13:07:45", "content": "That is a little better, but all my research (link for example) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm--you get a superset of features of which the true "important features" are a subset--that subset (or a portion thereof) should appear regardless. Going back to my original analogy and expanding:
\n\nMethod 1: X, Y, Z, R, S, T, U are your top 7 features
\n\nMethod 2: A, X, Z, Q, S, V, Y are your top 7 features
\n\nX, ,Y, and Z are common to both--there is a good chance those are truly the important features--they are not the same rank in both, or same weight, but all are listed as "important"
\n\nIt just doesn't make sense that the "information" about your model would be hidden in different columns depending on your method. The information doesn't change--how we extract it may be--but what surprised me was getting 2 totally independent sets of features--going back to the BNP competition--everyone pulled out that one column (V55 I think) as strongly important--independent of method.
\n\nWhere I struggle with is the thinking "Data is Data" If I did quantum mechanics--it doesn't matter if I use the Schroedinger or Heisenberg approach--the answers are the same. Certain things are easier in one formulation than the other--but the underlying information doesn't matter. Like in computer coding--information is an object.
", "rawMarkdown": "That is a little better, but all my research (link for [example][1]) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm--you get a superset of features of which the true \"important features\" are a subset--that subset (or a portion thereof) should appear regardless. Going back to my original analogy and expanding:\r\n\r\nMethod 1: X, Y, Z, R, S, T, U are your top 7 features\r\n\r\nMethod 2: A, X, Z, Q, S, V, Y are your top 7 features\r\n\r\nX, ,Y, and Z are common to both--there is a good chance those are truly the important features--they are not the same rank in both, or same weight, but all are listed as \"important\"\r\n\r\nIt just doesn't make sense that the \"information\" about your model would be hidden in different columns depending on your method. The information doesn't change--how we extract it may be--but what surprised me was getting 2 totally independent sets of features--going back to the BNP competition--everyone pulled out that one column (V55 I think) as strongly important--independent of method.\r\n\r\nWhere I struggle with is the thinking \"Data is Data\" If I did quantum mechanics--it doesn't matter if I use the Schroedinger or Heisenberg approach--the answers are the same. Certain things are easier in one formulation than the other--but the underlying information doesn't matter. Like in computer coding--information is an object.\r\n\r\n\r\n [1]: https://www.aaai.org/Papers/Workshops/2007/WS-07-05/WS07-05-007.pdf", "votes": null }, { "id": "136620", "postDate": "09/23/2016 13:32:05", "content": "@Bob, if particular features have a strong correlation to the labels, they will clearly be found in various models and there will be agreement between the models.
\n\nHowever, if lots of features have a little bit of correlation or if their non-linear combinations have some correlation, then the information is widely distributed and different models will pick up on different components of the correlation. The fact that different features are shown to be important in different models means that there are no simple/clear correlations between any one feature and the label. The information is widely distributed amongst the features in complicated ways.
", "rawMarkdown": "Bob, if particular features have a strong correlation to the labels, they will clearly be found in various models and there will be agreement between the models.\r\n\r\nHowever, if lots of features have a little bit of correlation or if their non-linear combinations have some correlation, then the information is widely distributed and different models will pick up on different components of the correlation. The fact that different features are shown to be important in different models means that there are no simple/clear correlations between any one feature and the label. The information is widely distributed amongst the features in complicated ways.", "votes": null }, { "id": "136621", "postDate": "09/23/2016 13:36:53", "content": "Bob, have you seen this sort of illustration before? A linear model would say both features are irrelevant, random forest / gradient boosted tree would say both are critical.
\n\n
The rest of the article is pretty good\nhttp://www.denizyuret.com/2014/02/machine-learning-in-5-pictures.html
", "rawMarkdown": "Bob, have you seen this sort of illustration before? A linear model would say both features are irrelevant, random forest / gradient boosted tree would say both are critical.\r\n\r\n![feature combinations][1]\r\n\r\n\r\nThe rest of the article is pretty good\r\nhttp://www.denizyuret.com/2014/02/machine-learning-in-5-pictures.html\r\n\r\n\r\n [1]: http://2.bp.blogspot.com/-bjRCSHtwAI0/Uv30_ZlxojI/AAAAAAAABmo/dX2Z3nwmKMY/s1600/Screen+Shot+2014-02-14+at+12.49.50+PM.png", "votes": null }, { "id": "136622", "postDate": "09/23/2016 13:55:47", "content": "\n\n\nThat is a little better, but all my research (link for example) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm
\n
That is wrong, and we can make a quite simple example proving that. Lets compare a Tree based algorithm with Logistic Regression for a classification problem. Let there be the following data set:
\n\nX_age: [18, 20, 22, 24, 26, 28, 30, 32, 34, 36]\nX_sex: [ 1, 1, 1, 1, 1, 1, 0, 0, 1, 1]\ny: [ 0, 0, 1, 1, 1, 1, 0, 0, 1, 1]\nNow lets investigate which feature is important to which algorithm. The Tree model is able to perfectly separate the two classes on the the age feature. Therefore it would give age the highest importance (however measured) and sex the lowest (while not needed).\nA logistic regression cant make much use out of the age feature because of its non linearity and will assign a low weight. But since the sex feature is strongly linear correlated to y, it will give that feature a high weight.
\n\nTo sum this up, the feature selection quite depends on the algorithm used.
", "rawMarkdown": "> That is a little better, but all my research (link for example) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm\r\n\r\nThat is wrong, and we can make a quite simple example proving that. Lets compare a Tree based algorithm with Logistic Regression for a classification problem. Let there be the following data set:\r\n\r\n X_age: [18, 20, 22, 24, 26, 28, 30, 32, 34, 36]\r\n X_sex: [ 1, 1, 1, 1, 1, 1, 0, 0, 1, 1]\r\n y: [ 0, 0, 1, 1, 1, 1, 0, 0, 1, 1]\r\n\r\nNow lets investigate which feature is important to which algorithm. The Tree model is able to perfectly separate the two classes on the the age feature. Therefore it would give age the highest importance (however measured) and sex the lowest (while not needed).\r\nA logistic regression cant make much use out of the age feature because of its non linearity and will assign a low weight. But since the sex feature is strongly linear correlated to y, it will give that feature a high weight.\r\n\r\nTo sum this up, the feature selection quite depends on the algorithm used.", "votes": null }, { "id": "136625", "postDate": "09/23/2016 14:46:53", "content": "@lewis and Tony--that makes more sense plus additional learning materials are always welcome. The picture wasn't showing up--work proxies FTW! (ok I hit it on my cell--that does make perfect sense), but as you say--the link contains much good detail and further info. Just when I think I am starting to gain a grasp of things, life pulls out the rug again.
", "rawMarkdown": "lewis and Tony--that makes more sense plus additional learning materials are always welcome. The picture wasn't showing up--work proxies FTW! (ok I hit it on my cell--that does make perfect sense), but as you say--the link contains much good detail and further info. Just when I think I am starting to gain a grasp of things, life pulls out the rug again.", "votes": null }, { "id": "136651", "postDate": "09/23/2016 18:12:16", "content": "Here's a totally non-data-related analogy if any further clarification is needed.
\n\nOur eyes are designed to only focus on a small area within our field of vision, even though we are aware of a much larger image (just try reading this and keeping the rest of your field of vision, or even just the rest of this paragraph, in focus at the same time). Now imagine you and a friend are looking at a large painting at your local art museum. You, being fond of patterns, see something interesting towards the upper left corner and focus on that. Your friend, being fond of colors, sees something interesting near the bottom center and focuses on that. Neither one of you are focusing on the same thing even though both of you are aware of the larger picture, neither one of you are emphasizing the same thing as "interesting", yet both of you are looking at the exact same painting.
\n\nAnd a point seemingly totally missed in this entire discussion: If your two models are independently behaving well in terms of CV scores, yet are emphasizing different features, that is a good thing, because you now have two excellent candidates for ensembling!
", "rawMarkdown": "Here's a totally non-data-related analogy if any further clarification is needed.\r\n\r\nOur eyes are designed to only focus on a small area within our field of vision, even though we are aware of a much larger image (just try reading this and keeping the rest of your field of vision, or even just the rest of this paragraph, in focus at the same time). Now imagine you and a friend are looking at a large painting at your local art museum. You, being fond of patterns, see something interesting towards the upper left corner and focus on that. Your friend, being fond of colors, sees something interesting near the bottom center and focuses on that. Neither one of you are focusing on the same thing even though both of you are aware of the larger picture, neither one of you are emphasizing the same thing as \"interesting\", yet both of you are looking at the exact same painting.\r\n\r\nAnd a point seemingly totally missed in this entire discussion: If your two models are independently behaving well in terms of CV scores, yet are emphasizing different features, that is a good thing, because you now have two excellent candidates for ensembling!", "votes": null } ], "comments": [ { "id": 136568, "author_name": "carloshuertas", "author_url": "", "post_date": "09/23/2016 01:16:04", "content": "To get technical, you need to read their papers or understand how the algorithm works, then, it becomes (sort of) obvious why they differ.
\n\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.
\n\nVW: Speaks Spanish and some English
\n\nXGB and RF, both speak English and some Spanish.
\n\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.
", "votes": null, "replies": [] }, { "id": 136572, "author_name": "rdslater", "author_url": "", "post_date": "09/23/2016 02:16:31", "content": "THey actually doesn't make a lot of sense. Information is information--its not the values--its the actual features are different--3 algorithms say column X, Y , + Z are important--a fourth says A B +C--i would expect something more like X Y Z and say X Y and A. Not two different sets of features.
", "votes": null, "replies": [] }, { "id": 136585, "author_name": "remap1", "author_url": "", "post_date": "09/23/2016 06:19:12", "content": "Because xgb and RF are tree-based and they look at regions of a particular feature. Other algorithms like linear/logistic regression look at the general trend of that feature. The importance of xgb/rf is measured by how many times a feature pops up in a tree, for logreg/linreg it's the individual predictive power of the trend of that feature towards the target.
\n\nSo features that have clear trends and good predictive power pop up as important in vw; features that have many subregions that have strong predictive power towards the target eventually pop up in xgb/rf, because the number of trees using that feature somehow (selecting a region and boost) define how important that feature is.
\n\nAt least that's my intuitive understanding of it.
", "votes": null, "replies": [] }, { "id": 136612, "author_name": "laurae2", "author_url": "", "post_date": "09/23/2016 11:15:41", "content": "[quote=Bob, Lord of Cats;136572]
\n\nTHey actually doesn't make a lot of sense. Information is information
\n\n[/quote]
\n\nFeature importance is not computed the same way between xgboost and VW, but also between a LR and xgboost/VW.
\n\nGetting (non)-technical (with simple questions) in a generalized way, you would ask:
\n\nThen, you can ask yourself these additional questions when picking a specific quantification metric:
\n\nIf you pick the gain ratio, answer yes to question 1 & 2, answer no to question 3. But this is true for all quantification metrics.
\n\nAlso, it is up to the developer to choose which feature importance metric they want to use (and how it is calculated). Actually, the best features in xgboost might not be the ones providing the highest gain (due to how biased that quantity is in a tree-based model).
\n\nA bit off-topic, but relevant to feature importance algorithms: some businesses are adding "super duper cool" feature importance metrics to sell their products. Not calling a name, but one of these is using a "super duper cool" name by hiding the fact it is just the rank of features issued from a feature importance algorithm. Hence, if you have 10 features and only 1 is relevant, it would report junk features as "important" (who knows if "hates pizzas" is not relevant to the prediction "likes pizzas" because it was shown 10th in the features of the "super duper cool" algorithm... yes it is 10th, so what? You just didn't rescale your data beforehand, its weight is 1e-20 unscaled when it is 1 scaled properly.).
\n\nThe non-technical explanation was provided by NxGTR:
\n\n[quote=NxGTR;136568]
\n\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.
\n\nVW: Speaks Spanish and some English
\n\nXGB and RF, both speak English and some Spanish.
\n\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.
\n\n[/quote]
\n\nAnother (and metaphorical) example from a sentence: "I eventually did it!"
\n\nBecause "eventually" is not interpreted the same way in French ("English: if appropriate" - "French: le cas échéant" <=? similiarity ?=> "éventuellement") than in German.
\n\nHence, between French people they would agree for the feature importance (even if it is wrong), while with an German businessman speaking English they would disagree (comparing Apple with Oranges even if the main source of comparison is exactly the same thing). Add a third language, and you may even have a 3rd source of disagreement.
\n\nObviously, this does not generalize to all sentences. But there will be always disagreements somewhere due to the language barrier.
", "votes": null, "replies": [] }, { "id": 136618, "author_name": "rdslater", "author_url": "", "post_date": "09/23/2016 13:07:45", "content": "That is a little better, but all my research (link for example) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm--you get a superset of features of which the true "important features" are a subset--that subset (or a portion thereof) should appear regardless. Going back to my original analogy and expanding:
\n\nMethod 1: X, Y, Z, R, S, T, U are your top 7 features
\n\nMethod 2: A, X, Z, Q, S, V, Y are your top 7 features
\n\nX, ,Y, and Z are common to both--there is a good chance those are truly the important features--they are not the same rank in both, or same weight, but all are listed as "important"
\n\nIt just doesn't make sense that the "information" about your model would be hidden in different columns depending on your method. The information doesn't change--how we extract it may be--but what surprised me was getting 2 totally independent sets of features--going back to the BNP competition--everyone pulled out that one column (V55 I think) as strongly important--independent of method.
\n\nWhere I struggle with is the thinking "Data is Data" If I did quantum mechanics--it doesn't matter if I use the Schroedinger or Heisenberg approach--the answers are the same. Certain things are easier in one formulation than the other--but the underlying information doesn't matter. Like in computer coding--information is an object.
", "votes": null, "replies": [] }, { "id": 136620, "author_name": "fredtony", "author_url": "", "post_date": "09/23/2016 13:32:05", "content": "@Bob, if particular features have a strong correlation to the labels, they will clearly be found in various models and there will be agreement between the models.
\n\nHowever, if lots of features have a little bit of correlation or if their non-linear combinations have some correlation, then the information is widely distributed and different models will pick up on different components of the correlation. The fact that different features are shown to be important in different models means that there are no simple/clear correlations between any one feature and the label. The information is widely distributed amongst the features in complicated ways.
", "votes": null, "replies": [] }, { "id": 136621, "author_name": "cartographic", "author_url": "", "post_date": "09/23/2016 13:36:53", "content": "Bob, have you seen this sort of illustration before? A linear model would say both features are irrelevant, random forest / gradient boosted tree would say both are critical.
\n\n
The rest of the article is pretty good\nhttp://www.denizyuret.com/2014/02/machine-learning-in-5-pictures.html
", "votes": null, "replies": [] }, { "id": 136622, "author_name": "rageagainstml", "author_url": "", "post_date": "09/23/2016 13:55:47", "content": "\n\n\nThat is a little better, but all my research (link for example) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm
\n
That is wrong, and we can make a quite simple example proving that. Lets compare a Tree based algorithm with Logistic Regression for a classification problem. Let there be the following data set:
\n\nX_age: [18, 20, 22, 24, 26, 28, 30, 32, 34, 36]\nX_sex: [ 1, 1, 1, 1, 1, 1, 0, 0, 1, 1]\ny: [ 0, 0, 1, 1, 1, 1, 0, 0, 1, 1]\nNow lets investigate which feature is important to which algorithm. The Tree model is able to perfectly separate the two classes on the the age feature. Therefore it would give age the highest importance (however measured) and sex the lowest (while not needed).\nA logistic regression cant make much use out of the age feature because of its non linearity and will assign a low weight. But since the sex feature is strongly linear correlated to y, it will give that feature a high weight.
\n\nTo sum this up, the feature selection quite depends on the algorithm used.
", "votes": null, "replies": [] }, { "id": 136625, "author_name": "rdslater", "author_url": "", "post_date": "09/23/2016 14:46:53", "content": "@lewis and Tony--that makes more sense plus additional learning materials are always welcome. The picture wasn't showing up--work proxies FTW! (ok I hit it on my cell--that does make perfect sense), but as you say--the link contains much good detail and further info. Just when I think I am starting to gain a grasp of things, life pulls out the rug again.
", "votes": null, "replies": [] }, { "id": 136651, "author_name": "taffeylewis", "author_url": "", "post_date": "09/23/2016 18:12:16", "content": "Here's a totally non-data-related analogy if any further clarification is needed.
\n\nOur eyes are designed to only focus on a small area within our field of vision, even though we are aware of a much larger image (just try reading this and keeping the rest of your field of vision, or even just the rest of this paragraph, in focus at the same time). Now imagine you and a friend are looking at a large painting at your local art museum. You, being fond of patterns, see something interesting towards the upper left corner and focus on that. Your friend, being fond of colors, sees something interesting near the bottom center and focuses on that. Neither one of you are focusing on the same thing even though both of you are aware of the larger picture, neither one of you are emphasizing the same thing as "interesting", yet both of you are looking at the exact same painting.
\n\nAnd a point seemingly totally missed in this entire discussion: If your two models are independently behaving well in terms of CV scores, yet are emphasizing different features, that is a good thing, because you now have two excellent candidates for ensembling!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "136565": "I am struggling to understand why my vw important features are significantly different that from say XGboost or a Random forest.\r\n\r\nI noticed this on a previous competition--since our data is normalized (numeric data) the coefficient should be the feature weight (annd i did the vw-varinfo--just to be sure)? Now I don't expect the same numbers, but I do expect similar orders (close would be nice, im not even close)\r\n\r\nAnd yet in vw, I get an entirely different set of features (My random forest I am guessing is accurate as it agrees with what others found)! Can anyone help me along to the path of enlightenment?", "136568": "To get technical, you need to read their papers or understand how the algorithm works, then, it becomes (sort of) obvious why they differ.\r\n\r\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.\r\n\r\nVW: Speaks Spanish and some English\r\n\r\nXGB and RF, both speak English and some Spanish.\r\n\r\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.", "136572": "THey actually doesn't make a lot of sense. Information is information--its not the values--its the actual features are different--3 algorithms say column X, Y , + Z are important--a fourth says A B +C--i would expect something more like X Y Z and say X Y and A. Not two different sets of features.", "136585": "Because xgb and RF are tree-based and they look at regions of a particular feature. Other algorithms like linear/logistic regression look at the general trend of that feature. The importance of xgb/rf is measured by how many times a feature pops up in a tree, for logreg/linreg it's the individual predictive power of the trend of that feature towards the target.\r\n\r\nSo features that have clear trends and good predictive power pop up as important in vw; features that have many subregions that have strong predictive power towards the target eventually pop up in xgb/rf, because the number of trees using that feature somehow (selecting a region and boost) define how important that feature is.\r\n\r\nAt least that's my intuitive understanding of it.", "136612": "[quote=Bob, Lord of Cats;136572]\r\n\r\nTHey actually doesn't make a lot of sense. Information is information\r\n\r\n[/quote]\r\n\r\nFeature importance is not computed the same way between xgboost and VW, but also between a LR and xgboost/VW.\r\n\r\nGetting (non)-technical (with simple questions) in a generalized way, you would ask:\r\n\r\n* Can you quantify a feature importance? (answer: by using a metric) \r\n* Are there existing unbiased quantifiers? (answer: no)\r\n* How to assess sample (tree) vs population (linear) feature importance? (should data be rescaled beforehand? etc.)\r\n* Is there a quantifier that works on any type of model? (answer: yes - just ignore the model and go univariate)\r\n\r\nThen, you can ask yourself these additional questions when picking a specific quantification metric:\r\n\r\n* Is that quantifier biased? (answer: yes)\r\n* Can you make that quantifier unbiased (for its specific usage) towards the population (answer: yes, by cross-validating)\r\n* Can you make that quantifier unbiased in a general case? (answer: no, never - this is a still an area being actively researched)\r\n\r\nIf you pick the gain ratio, answer yes to question 1 & 2, answer no to question 3. But this is true for all quantification metrics.\r\n\r\nAlso, it is up to the developer to choose which feature importance metric they want to use (and how it is calculated). Actually, the best features in xgboost might not be the ones providing the highest gain (due to how biased that quantity is in a tree-based model).\r\n\r\nA bit off-topic, but relevant to feature importance algorithms: some businesses are adding \"super duper cool\" feature importance metrics to sell their products. Not calling a name, but one of these is using a \"super duper cool\" name by hiding the fact it is just the rank of features issued from a feature importance algorithm. Hence, if you have 10 features and only 1 is relevant, it would report junk features as \"important\" (*who knows if \"hates pizzas\" is not relevant to the prediction \"likes pizzas\" because it was shown 10th in the features of the \"super duper cool\" algorithm... yes it is 10th, so what? You just didn't rescale your data beforehand, its weight is 1e-20 unscaled when it is 1 scaled properly.*).\r\n\r\nThe non-technical explanation was provided by NxGTR:\r\n\r\n[quote=NxGTR;136568]\r\n\r\nNow, for a non technical analysis, lets put it this way, your data comes in two languages, English and Spanish.\r\n\r\nVW: Speaks Spanish and some English\r\n\r\nXGB and RF, both speak English and some Spanish.\r\n\r\nIt becomes obvious that one would prefer some features over the others, depending on how the algorithm works.\r\n\r\n[/quote]\r\n\r\nAnother (and metaphorical) example from a sentence: \"I eventually did it!\"\r\n\r\n* German (perfect) interpretation: \"In the end I did it!\" (feature importance of words correct for speaking English)\r\n* French approximate (wrong word + wrong correction) interpretation: \"If appropriate I'll do it!\" (feature importance of words incorrect for speaking English)\r\n\r\nBecause \"eventually\" is not interpreted the same way in French (\"English: if appropriate\" - \"French: le cas échéant\" <=? similiarity ?=> \"éventuellement\") than in German.\r\n\r\nHence, between French people they would agree for the feature importance (even if it is wrong), while with an German businessman speaking English they would disagree (comparing Apple with Oranges even if the main source of comparison is exactly the same thing). Add a third language, and you may even have a 3rd source of disagreement.\r\n\r\nObviously, this does not generalize to all sentences. But there will be always disagreements somewhere due to the language barrier.", "136618": "That is a little better, but all my research (link for [example][1]) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm--you get a superset of features of which the true \"important features\" are a subset--that subset (or a portion thereof) should appear regardless. Going back to my original analogy and expanding:\r\n\r\nMethod 1: X, Y, Z, R, S, T, U are your top 7 features\r\n\r\nMethod 2: A, X, Z, Q, S, V, Y are your top 7 features\r\n\r\nX, ,Y, and Z are common to both--there is a good chance those are truly the important features--they are not the same rank in both, or same weight, but all are listed as \"important\"\r\n\r\nIt just doesn't make sense that the \"information\" about your model would be hidden in different columns depending on your method. The information doesn't change--how we extract it may be--but what surprised me was getting 2 totally independent sets of features--going back to the BNP competition--everyone pulled out that one column (V55 I think) as strongly important--independent of method.\r\n\r\nWhere I struggle with is the thinking \"Data is Data\" If I did quantum mechanics--it doesn't matter if I use the Schroedinger or Heisenberg approach--the answers are the same. Certain things are easier in one formulation than the other--but the underlying information doesn't matter. Like in computer coding--information is an object.\r\n\r\n\r\n [1]: https://www.aaai.org/Papers/Workshops/2007/WS-07-05/WS07-05-007.pdf", "136620": "Bob, if particular features have a strong correlation to the labels, they will clearly be found in various models and there will be agreement between the models.\r\n\r\nHowever, if lots of features have a little bit of correlation or if their non-linear combinations have some correlation, then the information is widely distributed and different models will pick up on different components of the correlation. The fact that different features are shown to be important in different models means that there are no simple/clear correlations between any one feature and the label. The information is widely distributed amongst the features in complicated ways.", "136621": "Bob, have you seen this sort of illustration before? A linear model would say both features are irrelevant, random forest / gradient boosted tree would say both are critical.\r\n\r\n![feature combinations][1]\r\n\r\n\r\nThe rest of the article is pretty good\r\nhttp://www.denizyuret.com/2014/02/machine-learning-in-5-pictures.html\r\n\r\n\r\n [1]: http://2.bp.blogspot.com/-bjRCSHtwAI0/Uv30_ZlxojI/AAAAAAAABmo/dX2Z3nwmKMY/s1600/Screen+Shot+2014-02-14+at+12.49.50+PM.png", "136622": "> That is a little better, but all my research (link for example) seems to indicate (if done properly) that when doing feature selection, it is largely independent of algorithm\r\n\r\nThat is wrong, and we can make a quite simple example proving that. Lets compare a Tree based algorithm with Logistic Regression for a classification problem. Let there be the following data set:\r\n\r\n X_age: [18, 20, 22, 24, 26, 28, 30, 32, 34, 36]\r\n X_sex: [ 1, 1, 1, 1, 1, 1, 0, 0, 1, 1]\r\n y: [ 0, 0, 1, 1, 1, 1, 0, 0, 1, 1]\r\n\r\nNow lets investigate which feature is important to which algorithm. The Tree model is able to perfectly separate the two classes on the the age feature. Therefore it would give age the highest importance (however measured) and sex the lowest (while not needed).\r\nA logistic regression cant make much use out of the age feature because of its non linearity and will assign a low weight. But since the sex feature is strongly linear correlated to y, it will give that feature a high weight.\r\n\r\nTo sum this up, the feature selection quite depends on the algorithm used.", "136625": "lewis and Tony--that makes more sense plus additional learning materials are always welcome. The picture wasn't showing up--work proxies FTW! (ok I hit it on my cell--that does make perfect sense), but as you say--the link contains much good detail and further info. Just when I think I am starting to gain a grasp of things, life pulls out the rug again.", "136651": "Here's a totally non-data-related analogy if any further clarification is needed.\r\n\r\nOur eyes are designed to only focus on a small area within our field of vision, even though we are aware of a much larger image (just try reading this and keeping the rest of your field of vision, or even just the rest of this paragraph, in focus at the same time). Now imagine you and a friend are looking at a large painting at your local art museum. You, being fond of patterns, see something interesting towards the upper left corner and focus on that. Your friend, being fond of colors, sees something interesting near the bottom center and focuses on that. Neither one of you are focusing on the same thing even though both of you are aware of the larger picture, neither one of you are emphasizing the same thing as \"interesting\", yet both of you are looking at the exact same painting.\r\n\r\nAnd a point seemingly totally missed in this entire discussion: If your two models are independently behaving well in terms of CV scores, yet are emphasizing different features, that is a good thing, because you now have two excellent candidates for ensembling!" }, "source": "meta" }