{
  "id": 24418,
  "title": "XGboost interaction features",
  "url": "/competitions/bosch-production-line-performance/discussion/24418",
  "author_name": "",
  "post_date": "2016-10-14T23:05:22.997Z",
  "votes": 6,
  "comment_count": 15,
  "views": 2457,
  "content": "<p>If features X and Y are informative only when X * Y, can xgboost discover such interactions ? </p>\n\n<p>Or should we create a new feature with X*Y to help the model. </p>",
  "messages": [
    {
      "id": "139573",
      "postDate": "10/14/2016 23:05:22",
      "content": "<p>If features X and Y are informative only when X * Y, can xgboost discover such interactions ? </p>\n\n<p>Or should we create a new feature with X*Y to help the model. </p>",
      "rawMarkdown": "If features X and Y are informative only when X * Y, can xgboost discover such interactions ? \r\n\r\nOr should we create a new feature with X*Y to help the model.",
      "votes": null
    },
    {
      "id": "139609",
      "postDate": "10/15/2016 06:48:16",
      "content": "<p>Faron has made a specific tool for this question and checking interactions.</p>\n\n<p><a href=\"https://github.com/Far0n/xgbfi\">https://github.com/Far0n/xgbfi</a></p>\n\n<p>&quot;Xgbfi is a XGBoost model dump parser, which ranks features as well as feature interactions by different metrics.&quot;</p>\n\n<p>Another option would be to use the new feature with X*Y and have a look at the feature importance afterwards.</p>",
      "rawMarkdown": "Faron has made a specific tool for this question and checking interactions.\r\n\r\nhttps://github.com/Far0n/xgbfi\r\n\r\n\"Xgbfi is a XGBoost model dump parser, which ranks features as well as feature interactions by different metrics.\"\r\n\r\nAnother option would be to use the new feature with X*Y and have a look at the feature importance afterwards.",
      "votes": null
    },
    {
      "id": "139620",
      "postDate": "10/15/2016 09:36:41",
      "content": "<p>In my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!</p>",
      "rawMarkdown": "In my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!",
      "votes": null
    },
    {
      "id": "140024",
      "postDate": "10/18/2016 00:47:53",
      "content": "<p>[quote=Malte Nalenz;139620]</p>\n\n<p>In my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!</p>\n\n<p>[/quote]</p>\n\n<p>Hi Malte,</p>\n\n<p>Just wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.</p>\n\n<p>I just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. </p>\n\n<p>XA</p>",
      "rawMarkdown": "[quote=Malte Nalenz;139620]\r\n\r\nIn my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!\r\n\r\n[/quote]\r\n\r\nHi Malte,\r\n\r\nJust wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.\r\n\r\nI just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. \r\n\r\nXA",
      "votes": null
    },
    {
      "id": "140027",
      "postDate": "10/18/2016 00:59:06",
      "content": "<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>",
      "rawMarkdown": "Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code.",
      "votes": null
    },
    {
      "id": "140032",
      "postDate": "10/18/2016 01:28:32",
      "content": "<p>[quote=leexa;140024]</p>\n\n<p>Just wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.</p>\n\n<p>[/quote]</p>\n\n<p>Hi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. </p>\n\n<p>Take a look at this <a href=\"http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/\">post on visualizing boosted decision trees</a>. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.</p>\n\n<p>Now, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.</p>\n\n<p>Good luck,\nJeff</p>",
      "rawMarkdown": "[quote=leexa;140024]\r\n\r\nJust wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.\r\n\r\n[/quote]\r\n\r\nHi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. \r\n\r\nTake a look at this [post on visualizing boosted decision trees][1]. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.\r\n\r\nNow, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.\r\n\r\nGood luck,\r\nJeff\r\n\r\n  [1]: http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/",
      "votes": null
    },
    {
      "id": "140033",
      "postDate": "10/18/2016 01:37:10",
      "content": "<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>@neuron, the methods used by xgbfi are explained on the <a href=\"https://github.com/Far0n/xgbfi\">xgbfi github site</a>. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\">gain</a> for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.</p>",
      "rawMarkdown": "[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\n@neuron, the methods used by xgbfi are explained on the [xgbfi github site][1]. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at [gain][2] for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi\r\n  [2]: https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain",
      "votes": null
    },
    {
      "id": "140039",
      "postDate": "10/18/2016 02:12:22",
      "content": "<p>[quote=JeffH;140033]</p>\n\n<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>@neuron, the methods used by xgbfi are explained on the <a href=\"https://github.com/Far0n/xgbfi\">xgbfi github site</a>. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\">gain</a> for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.</p>\n\n<p>[/quote]</p>\n\n<p>This doesn't explain interactions identification, you answer say how metric calculation for feature importance. </p>",
      "rawMarkdown": "[quote=JeffH;140033]\r\n\r\n[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\n@neuron, the methods used by xgbfi are explained on the [xgbfi github site][1]. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at [gain][2] for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi\r\n  [2]: https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\r\n\r\n[/quote]\r\n\r\nThis doesn't explain interactions identification, you answer say how metric calculation for feature importance.",
      "votes": null
    },
    {
      "id": "140076",
      "postDate": "10/18/2016 07:02:12",
      "content": "<p>[quote=JeffH;140032]</p>\n\n<p>[quote=leexa;140024]</p>\n\n<p>Just wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.</p>\n\n<p>[/quote]</p>\n\n<p>Hi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. </p>\n\n<p>Take a look at this <a href=\"http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/\">post on visualizing boosted decision trees</a>. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.</p>\n\n<p>Now, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.</p>\n\n<p>Good luck,\nJeff</p>\n\n<p>[/quote]\nEven though i agree with that post id like to add, that multiplikative interactions are quite common in many problem domains. As Jeff writes XGboost identifies very complicated relationships and in this case probably overly complicated relationships, as this kind of effect takes a lot of splits to model. Including such effect (doesnt have to be a perfect fit) directly helps the model a great deal in finding the interaction and avoiding overfitting. In X1*X2 the relationship to y can still be non-linear but maybe easier to model.\n I useually draw scatterplots of variables i suspect interactions in, but then it is time to get creative.\nFor the interactions: The basic idea is to look at the trees and count the number of times both X1 and X2 are used in the same branch. Xgboost uses then different weighting schemes, such as metrics of the together produced splits for the different branches, to better reflect the importance of the interactions.</p>",
      "rawMarkdown": "[quote=JeffH;140032]\r\n\r\n[quote=leexa;140024]\r\n\r\nJust wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.\r\n\r\n[/quote]\r\n\r\nHi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. \r\n\r\nTake a look at this [post on visualizing boosted decision trees][1]. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.\r\n\r\nNow, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.\r\n\r\nGood luck,\r\nJeff\r\n\r\n  [1]: http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/\r\n\r\n[/quote]\r\nEven though i agree with that post id like to add, that multiplikative interactions are quite common in many problem domains. As Jeff writes XGboost identifies very complicated relationships and in this case probably overly complicated relationships, as this kind of effect takes a lot of splits to model. Including such effect (doesnt have to be a perfect fit) directly helps the model a great deal in finding the interaction and avoiding overfitting. In X1*X2 the relationship to y can still be non-linear but maybe easier to model.\r\n I useually draw scatterplots of variables i suspect interactions in, but then it is time to get creative.\r\nFor the interactions: The basic idea is to look at the trees and count the number of times both X1 and X2 are used in the same branch. Xgboost uses then different weighting schemes, such as metrics of the together produced splits for the different branches, to better reflect the importance of the interactions.",
      "votes": null
    },
    {
      "id": "140083",
      "postDate": "10/18/2016 07:43:43",
      "content": "<p>Thanks JeffH and Malte for your inputs. </p>\n\n<p>Things have become clearer. Currently, the dominating feature i have is &quot;magic4&quot;, which is extremely non-linear. It is pretty difficult to find an interaction term. </p>\n\n<p>Here is a comment i found on the xgbfi <a href=\"https://github.com/Far0n/xgbfi/issues/2;140027\">github</a>, which i think is very useful regarding feature engineering.  It talks about xgbfi outputs interation terms with itself, eg f10|f10 and f10|f10|f10.\n[quote= walterreade commented on 14 Feb ,]</p>\n\n<p>I'll throw in my (less knowledgeable than Mathias') view on this.</p>\n\n<p>Same feature &quot;interactions&quot; are just algorithm discretizing the feature. So, in other words, if x &gt; 10, and x &gt; 25, and x &lt; 100, then y = 7. That would be a 3x interaction. It's doing this because the response has a strong dependence on this feature and/or there is a lot of non-linearity.</p>\n\n<p>So the question - what should you do with this information? Well, in situations like this, my leans towards what might be getting missed because of the dominance of this feature. So, The first thing I'd do is remove that feature and run the model again to see what else floats to the surface in terms of importance. Then I might stack the output (predictions) of that model as a new feature.</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "Thanks JeffH and Malte for your inputs. \r\n\r\nThings have become clearer. Currently, the dominating feature i have is \"magic4\", which is extremely non-linear. It is pretty difficult to find an interaction term. \r\n\r\n\r\nHere is a comment i found on the xgbfi [github][1], which i think is very useful regarding feature engineering.  It talks about xgbfi outputs interation terms with itself, eg f10|f10 and f10|f10|f10.\r\n[quote= walterreade commented on 14 Feb ,]\r\n\r\n\r\n\r\nI'll throw in my (less knowledgeable than Mathias') view on this.\r\n\r\nSame feature \"interactions\" are just algorithm discretizing the feature. So, in other words, if x > 10, and x > 25, and x < 100, then y = 7. That would be a 3x interaction. It's doing this because the response has a strong dependence on this feature and/or there is a lot of non-linearity.\r\n\r\nSo the question - what should you do with this information? Well, in situations like this, my leans towards what might be getting missed because of the dominance of this feature. So, The first thing I'd do is remove that feature and run the model again to see what else floats to the surface in terms of importance. Then I might stack the output (predictions) of that model as a new feature.\r\n\r\n\r\n[/quote]\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi/issues/2;140027",
      "votes": null
    },
    {
      "id": "140109",
      "postDate": "10/18/2016 11:02:39",
      "content": "<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>For xgbfi an <em>&quot;interaction&quot;</em> of length <em>n</em> is defined as a subpath of length <em>n</em> within a tree.</p>",
      "rawMarkdown": "[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\nFor xgbfi an *\"interaction\"* of length *n* is defined as a subpath of length *n* within a tree.",
      "votes": null
    },
    {
      "id": "140111",
      "postDate": "10/18/2016 11:11:52",
      "content": "<p>[quote=leexa;140024]</p>\n\n<p>I just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. </p>\n\n<p>[/quote]</p>\n\n<p>Note, that this it not the way to go, if you want to improve your xgb-model. What you get out of xgb dumps is what xgb already captures. It's basically the same as staying on a sailboat while blowing into the canvas: it won't move. We need to stay at the landing stage instead. </p>",
      "rawMarkdown": "[quote=leexa;140024]\r\n\r\nI just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. \r\n\r\n[/quote]\r\n\r\nNote, that this it not the way to go, if you want to improve your xgb-model. What you get out of xgb dumps is what xgb already captures. It's basically the same as staying on a sailboat while blowing into the canvas: it won't move. We need to stay at the landing stage instead.",
      "votes": null
    },
    {
      "id": "140208",
      "postDate": "10/18/2016 22:09:26",
      "content": "<p>[quote=neuron;140039]</p>\n\n<p>[quote=JeffH;140033]</p>\n\n<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>@neuron, the methods used by xgbfi are explained on the <a href=\"https://github.com/Far0n/xgbfi\">xgbfi github site</a>. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\">gain</a> for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.</p>\n\n<p>[/quote]</p>\n\n<p>This doesn't explain interactions identification, you answer say how metric calculation for feature importance. </p>\n\n<p>[/quote]</p>\n\n<p>I'm shamelessly stealing the image from Far0n's site to explain this better.</p>\n\n<p>This tree shows interactions of three variables: F1, F2, and F3. To calculate the importance of interactions between F1 and F2, the algorithm first finds ALL places where F1 and F2 show up in sequence.  Then, the algorithm calculates the change in entropy (or your favorite metric) by that interaction. In this tree, there are two-way interactions between F1 and F2, between F1 and F3, and between F2 and F3. There is also a three-way interaction between F1, F2, and F3. The interaction importance is related to the change in entropy between the top node and the bottom nodes of each of those interactions.</p>\n\n<p>Ooh, I just realized that you can think of feature importance as the importance of each node (split), and interaction importance as the importance of each edge (path).</p>\n\n<p>Keep in mind that this is just one of many trees. There may be hundreds or thousands of branches where F1 and F2 determine the outcome. </p>\n\n<p><img src=\"https://raw.githubusercontent.com/Far0n/xgbfi/master/doc/ScoresExample_small.png\" alt=\"from github.com/Far0n/xgbfi\" title></p>",
      "rawMarkdown": "[quote=neuron;140039]\r\n\r\n[quote=JeffH;140033]\r\n\r\n[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\n@neuron, the methods used by xgbfi are explained on the [xgbfi github site][1]. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at [gain][2] for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.\r\n\r\n\r\n[/quote]\r\n\r\nThis doesn't explain interactions identification, you answer say how metric calculation for feature importance. \r\n\r\n[/quote]\r\n\r\n\r\nI'm shamelessly stealing the image from Far0n's site to explain this better.\r\n\r\nThis tree shows interactions of three variables: F1, F2, and F3. To calculate the importance of interactions between F1 and F2, the algorithm first finds ALL places where F1 and F2 show up in sequence.  Then, the algorithm calculates the change in entropy (or your favorite metric) by that interaction. In this tree, there are two-way interactions between F1 and F2, between F1 and F3, and between F2 and F3. There is also a three-way interaction between F1, F2, and F3. The interaction importance is related to the change in entropy between the top node and the bottom nodes of each of those interactions.\r\n\r\nOoh, I just realized that you can think of feature importance as the importance of each node (split), and interaction importance as the importance of each edge (path).\r\n\r\nKeep in mind that this is just one of many trees. There may be hundreds or thousands of branches where F1 and F2 determine the outcome. \r\n\r\n![from github.com/Far0n/xgbfi][3]\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi\r\n  [2]: https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\r\n  [3]: https://raw.githubusercontent.com/Far0n/xgbfi/master/doc/ScoresExample_small.png",
      "votes": null
    },
    {
      "id": "140218",
      "postDate": "10/18/2016 23:46:22",
      "content": "<p>@JeffH, Thanks for the nice explanation. </p>",
      "rawMarkdown": "JeffH, Thanks for the nice explanation.",
      "votes": null
    },
    {
      "id": "163255",
      "postDate": "02/23/2017 06:50:31",
      "content": "<p>I got an overview of the feature interactions, that XGBfi produces but my question is how do I use this information, in feature selection or modeling? Should I just select the features, which exist in high ranking interactions? Coming to interactions, How do I use that information?</p>",
      "rawMarkdown": "I got an overview of the feature interactions, that XGBfi produces but my question is how do I use this information, in feature selection or modeling? Should I just select the features, which exist in high ranking interactions? Coming to interactions, How do I use that information?",
      "votes": null
    },
    {
      "id": "953094",
      "postDate": "07/31/2020 14:27:16",
      "content": "<p>I tried this library and the biggest gain pair interaction is between a feature and itself. What can I make of this?</p>",
      "rawMarkdown": "I tried this library and the biggest gain pair interaction is between a feature and itself. What can I make of this?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 139609,
      "author_name": "justfor",
      "author_url": "",
      "post_date": "10/15/2016 06:48:16",
      "content": "<p>Faron has made a specific tool for this question and checking interactions.</p>\n\n<p><a href=\"https://github.com/Far0n/xgbfi\">https://github.com/Far0n/xgbfi</a></p>\n\n<p>&quot;Xgbfi is a XGBoost model dump parser, which ranks features as well as feature interactions by different metrics.&quot;</p>\n\n<p>Another option would be to use the new feature with X*Y and have a look at the feature importance afterwards.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139620,
      "author_name": "malten",
      "author_url": "",
      "post_date": "10/15/2016 09:36:41",
      "content": "<p>In my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140024,
      "author_name": "leexiongan",
      "author_url": "",
      "post_date": "10/18/2016 00:47:53",
      "content": "<p>[quote=Malte Nalenz;139620]</p>\n\n<p>In my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!</p>\n\n<p>[/quote]</p>\n\n<p>Hi Malte,</p>\n\n<p>Just wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.</p>\n\n<p>I just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. </p>\n\n<p>XA</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140027,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "10/18/2016 00:59:06",
      "content": "<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140032,
      "author_name": "jeffhebert",
      "author_url": "",
      "post_date": "10/18/2016 01:28:32",
      "content": "<p>[quote=leexa;140024]</p>\n\n<p>Just wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.</p>\n\n<p>[/quote]</p>\n\n<p>Hi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. </p>\n\n<p>Take a look at this <a href=\"http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/\">post on visualizing boosted decision trees</a>. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.</p>\n\n<p>Now, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.</p>\n\n<p>Good luck,\nJeff</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140033,
      "author_name": "jeffhebert",
      "author_url": "",
      "post_date": "10/18/2016 01:37:10",
      "content": "<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>@neuron, the methods used by xgbfi are explained on the <a href=\"https://github.com/Far0n/xgbfi\">xgbfi github site</a>. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\">gain</a> for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140039,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "10/18/2016 02:12:22",
      "content": "<p>[quote=JeffH;140033]</p>\n\n<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>@neuron, the methods used by xgbfi are explained on the <a href=\"https://github.com/Far0n/xgbfi\">xgbfi github site</a>. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\">gain</a> for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.</p>\n\n<p>[/quote]</p>\n\n<p>This doesn't explain interactions identification, you answer say how metric calculation for feature importance. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140076,
      "author_name": "malten",
      "author_url": "",
      "post_date": "10/18/2016 07:02:12",
      "content": "<p>[quote=JeffH;140032]</p>\n\n<p>[quote=leexa;140024]</p>\n\n<p>Just wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.</p>\n\n<p>[/quote]</p>\n\n<p>Hi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. </p>\n\n<p>Take a look at this <a href=\"http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/\">post on visualizing boosted decision trees</a>. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.</p>\n\n<p>Now, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.</p>\n\n<p>Good luck,\nJeff</p>\n\n<p>[/quote]\nEven though i agree with that post id like to add, that multiplikative interactions are quite common in many problem domains. As Jeff writes XGboost identifies very complicated relationships and in this case probably overly complicated relationships, as this kind of effect takes a lot of splits to model. Including such effect (doesnt have to be a perfect fit) directly helps the model a great deal in finding the interaction and avoiding overfitting. In X1*X2 the relationship to y can still be non-linear but maybe easier to model.\n I useually draw scatterplots of variables i suspect interactions in, but then it is time to get creative.\nFor the interactions: The basic idea is to look at the trees and count the number of times both X1 and X2 are used in the same branch. Xgboost uses then different weighting schemes, such as metrics of the together produced splits for the different branches, to better reflect the importance of the interactions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140083,
      "author_name": "leexiongan",
      "author_url": "",
      "post_date": "10/18/2016 07:43:43",
      "content": "<p>Thanks JeffH and Malte for your inputs. </p>\n\n<p>Things have become clearer. Currently, the dominating feature i have is &quot;magic4&quot;, which is extremely non-linear. It is pretty difficult to find an interaction term. </p>\n\n<p>Here is a comment i found on the xgbfi <a href=\"https://github.com/Far0n/xgbfi/issues/2;140027\">github</a>, which i think is very useful regarding feature engineering.  It talks about xgbfi outputs interation terms with itself, eg f10|f10 and f10|f10|f10.\n[quote= walterreade commented on 14 Feb ,]</p>\n\n<p>I'll throw in my (less knowledgeable than Mathias') view on this.</p>\n\n<p>Same feature &quot;interactions&quot; are just algorithm discretizing the feature. So, in other words, if x &gt; 10, and x &gt; 25, and x &lt; 100, then y = 7. That would be a 3x interaction. It's doing this because the response has a strong dependence on this feature and/or there is a lot of non-linearity.</p>\n\n<p>So the question - what should you do with this information? Well, in situations like this, my leans towards what might be getting missed because of the dominance of this feature. So, The first thing I'd do is remove that feature and run the model again to see what else floats to the surface in terms of importance. Then I might stack the output (predictions) of that model as a new feature.</p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140109,
      "author_name": "mmueller",
      "author_url": "",
      "post_date": "10/18/2016 11:02:39",
      "content": "<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>For xgbfi an <em>&quot;interaction&quot;</em> of length <em>n</em> is defined as a subpath of length <em>n</em> within a tree.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140111,
      "author_name": "mmueller",
      "author_url": "",
      "post_date": "10/18/2016 11:11:52",
      "content": "<p>[quote=leexa;140024]</p>\n\n<p>I just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. </p>\n\n<p>[/quote]</p>\n\n<p>Note, that this it not the way to go, if you want to improve your xgb-model. What you get out of xgb dumps is what xgb already captures. It's basically the same as staying on a sailboat while blowing into the canvas: it won't move. We need to stay at the landing stage instead. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140208,
      "author_name": "jeffhebert",
      "author_url": "",
      "post_date": "10/18/2016 22:09:26",
      "content": "<p>[quote=neuron;140039]</p>\n\n<p>[quote=JeffH;140033]</p>\n\n<p>[quote=neuron;140027]</p>\n\n<p>Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. </p>\n\n<p>[/quote]</p>\n\n<p>@neuron, the methods used by xgbfi are explained on the <a href=\"https://github.com/Far0n/xgbfi\">xgbfi github site</a>. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\">gain</a> for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.</p>\n\n<p>[/quote]</p>\n\n<p>This doesn't explain interactions identification, you answer say how metric calculation for feature importance. </p>\n\n<p>[/quote]</p>\n\n<p>I'm shamelessly stealing the image from Far0n's site to explain this better.</p>\n\n<p>This tree shows interactions of three variables: F1, F2, and F3. To calculate the importance of interactions between F1 and F2, the algorithm first finds ALL places where F1 and F2 show up in sequence.  Then, the algorithm calculates the change in entropy (or your favorite metric) by that interaction. In this tree, there are two-way interactions between F1 and F2, between F1 and F3, and between F2 and F3. There is also a three-way interaction between F1, F2, and F3. The interaction importance is related to the change in entropy between the top node and the bottom nodes of each of those interactions.</p>\n\n<p>Ooh, I just realized that you can think of feature importance as the importance of each node (split), and interaction importance as the importance of each edge (path).</p>\n\n<p>Keep in mind that this is just one of many trees. There may be hundreds or thousands of branches where F1 and F2 determine the outcome. </p>\n\n<p><img src=\"https://raw.githubusercontent.com/Far0n/xgbfi/master/doc/ScoresExample_small.png\" alt=\"from github.com/Far0n/xgbfi\" title></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140218,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "10/18/2016 23:46:22",
      "content": "<p>@JeffH, Thanks for the nice explanation. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 163255,
      "author_name": "remidi",
      "author_url": "",
      "post_date": "02/23/2017 06:50:31",
      "content": "<p>I got an overview of the feature interactions, that XGBfi produces but my question is how do I use this information, in feature selection or modeling? Should I just select the features, which exist in high ranking interactions? Coming to interactions, How do I use that information?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 953094,
      "author_name": "barney12",
      "author_url": "",
      "post_date": "07/31/2020 14:27:16",
      "content": "<p>I tried this library and the biggest gain pair interaction is between a feature and itself. What can I make of this?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "139573": "If features X and Y are informative only when X * Y, can xgboost discover such interactions ? \r\n\r\nOr should we create a new feature with X*Y to help the model.",
    "139609": "Faron has made a specific tool for this question and checking interactions.\r\n\r\nhttps://github.com/Far0n/xgbfi\r\n\r\n\"Xgbfi is a XGBoost model dump parser, which ranks features as well as feature interactions by different metrics.\"\r\n\r\nAnother option would be to use the new feature with X*Y and have a look at the feature importance afterwards.",
    "139620": "In my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!",
    "140024": "[quote=Malte Nalenz;139620]\r\n\r\nIn my experience decision trees can have problems in approximating this kind of smooth interactions. Reason is, that decision trees sometimes need a lot of splits before it is able to capture such effects. Being a greedy procedure that choses the next best split, the algorithm might not go this way but use splits on other variables instead. So if you think there might be a interaction of this type its definately better to include it as a new variable directly!\r\n\r\n[/quote]\r\n\r\nHi Malte,\r\n\r\nJust wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.\r\n\r\nI just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. \r\n\r\nXA",
    "140027": "Can anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code.",
    "140032": "[quote=leexa;140024]\r\n\r\nJust wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.\r\n\r\n[/quote]\r\n\r\nHi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. \r\n\r\nTake a look at this [post on visualizing boosted decision trees][1]. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.\r\n\r\nNow, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.\r\n\r\nGood luck,\r\nJeff\r\n\r\n  [1]: http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/",
    "140033": "[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\n@neuron, the methods used by xgbfi are explained on the [xgbfi github site][1]. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at [gain][2] for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi\r\n  [2]: https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain",
    "140039": "[quote=JeffH;140033]\r\n\r\n[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\n@neuron, the methods used by xgbfi are explained on the [xgbfi github site][1]. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at [gain][2] for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi\r\n  [2]: https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\r\n\r\n[/quote]\r\n\r\nThis doesn't explain interactions identification, you answer say how metric calculation for feature importance.",
    "140076": "[quote=JeffH;140032]\r\n\r\n[quote=leexa;140024]\r\n\r\nJust wondering how to you find out the type of interaction between two factors. Xgbfi only states two features are interacting, but not the type of interaction eg. X1/X2, X1-X2, X1*X2, X1*cos(X2) etc.\r\n\r\n[/quote]\r\n\r\nHi leexa, decision tree algorithms (like XGBoost) often identify very complicated and non-linear interactions. These interactions may only show up when other conditions exist. These are NOT continuous functions so they probably won't translate into anything like X1/X2. \r\n\r\nTake a look at this [post on visualizing boosted decision trees][1]. The first decision tree graph shows interactions between f1 and f5 in the right side of the tree. Try plotting this in an f1 vs f5 grid to see what I mean about non-linear functions.\r\n\r\nNow, given that most of these interactions are not linear, it may still be worthwhile to add 'engineered features' to your model. In the best case, an engineered feature will make it easier for the model to discover relationships in the data. There is an art and a science to feature engineering. I think the art starts with dreaming up possible formulas like you already constructed. The science is in using a measure of feature importance to screen out the new features that don't help your model.\r\n\r\nGood luck,\r\nJeff\r\n\r\n  [1]: http://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/\r\n\r\n[/quote]\r\nEven though i agree with that post id like to add, that multiplikative interactions are quite common in many problem domains. As Jeff writes XGboost identifies very complicated relationships and in this case probably overly complicated relationships, as this kind of effect takes a lot of splits to model. Including such effect (doesnt have to be a perfect fit) directly helps the model a great deal in finding the interaction and avoiding overfitting. In X1*X2 the relationship to y can still be non-linear but maybe easier to model.\r\n I useually draw scatterplots of variables i suspect interactions in, but then it is time to get creative.\r\nFor the interactions: The basic idea is to look at the trees and count the number of times both X1 and X2 are used in the same branch. Xgboost uses then different weighting schemes, such as metrics of the together produced splits for the different branches, to better reflect the importance of the interactions.",
    "140083": "Thanks JeffH and Malte for your inputs. \r\n\r\nThings have become clearer. Currently, the dominating feature i have is \"magic4\", which is extremely non-linear. It is pretty difficult to find an interaction term. \r\n\r\n\r\nHere is a comment i found on the xgbfi [github][1], which i think is very useful regarding feature engineering.  It talks about xgbfi outputs interation terms with itself, eg f10|f10 and f10|f10|f10.\r\n[quote= walterreade commented on 14 Feb ,]\r\n\r\n\r\n\r\nI'll throw in my (less knowledgeable than Mathias') view on this.\r\n\r\nSame feature \"interactions\" are just algorithm discretizing the feature. So, in other words, if x > 10, and x > 25, and x < 100, then y = 7. That would be a 3x interaction. It's doing this because the response has a strong dependence on this feature and/or there is a lot of non-linearity.\r\n\r\nSo the question - what should you do with this information? Well, in situations like this, my leans towards what might be getting missed because of the dominance of this feature. So, The first thing I'd do is remove that feature and run the model again to see what else floats to the surface in terms of importance. Then I might stack the output (predictions) of that model as a new feature.\r\n\r\n\r\n[/quote]\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi/issues/2;140027",
    "140109": "[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\nFor xgbfi an *\"interaction\"* of length *n* is defined as a subpath of length *n* within a tree.",
    "140111": "[quote=leexa;140024]\r\n\r\nI just tried xgbfi, it is really great. But i was wondering how to formulate feature interaction terms, especially if they a mix of categorical and numeric. \r\n\r\n[/quote]\r\n\r\nNote, that this it not the way to go, if you want to improve your xgb-model. What you get out of xgb dumps is what xgb already captures. It's basically the same as staying on a sailboat while blowing into the canvas: it won't move. We need to stay at the landing stage instead.",
    "140208": "[quote=neuron;140039]\r\n\r\n[quote=JeffH;140033]\r\n\r\n[quote=neuron;140027]\r\n\r\nCan anyone tell me how xgbfi identifies interactions ? I just want the basic idea, rest I can find from source code. \r\n\r\n[/quote]\r\n\r\n@neuron, the methods used by xgbfi are explained on the [xgbfi github site][1]. The program parses all trees in the model and calculates various statistics related to feature importance. For example, if you are looking at [gain][2] for variable F1, the parser will look through all the trees and add up the gain at each split. Features with the higher total gain explain more of the variation than features with lower total gain.\r\n\r\n\r\n[/quote]\r\n\r\nThis doesn't explain interactions identification, you answer say how metric calculation for feature importance. \r\n\r\n[/quote]\r\n\r\n\r\nI'm shamelessly stealing the image from Far0n's site to explain this better.\r\n\r\nThis tree shows interactions of three variables: F1, F2, and F3. To calculate the importance of interactions between F1 and F2, the algorithm first finds ALL places where F1 and F2 show up in sequence.  Then, the algorithm calculates the change in entropy (or your favorite metric) by that interaction. In this tree, there are two-way interactions between F1 and F2, between F1 and F3, and between F2 and F3. There is also a three-way interaction between F1, F2, and F3. The interaction importance is related to the change in entropy between the top node and the bottom nodes of each of those interactions.\r\n\r\nOoh, I just realized that you can think of feature importance as the importance of each node (split), and interaction importance as the importance of each edge (path).\r\n\r\nKeep in mind that this is just one of many trees. There may be hundreds or thousands of branches where F1 and F2 determine the outcome. \r\n\r\n![from github.com/Far0n/xgbfi][3]\r\n\r\n\r\n  [1]: https://github.com/Far0n/xgbfi\r\n  [2]: https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain\r\n  [3]: https://raw.githubusercontent.com/Far0n/xgbfi/master/doc/ScoresExample_small.png",
    "140218": "JeffH, Thanks for the nice explanation.",
    "163255": "I got an overview of the feature interactions, that XGBfi produces but my question is how do I use this information, in feature selection or modeling? Should I just select the features, which exist in high ranking interactions? Coming to interactions, How do I use that information?",
    "953094": "I tried this library and the biggest gain pair interaction is between a feature and itself. What can I make of this?"
  },
  "source": "meta"
}