{
  "id": 115454,
  "title": "Feature Order in Light GBM",
  "url": "/competitions/ashrae-energy-prediction/discussion/115454",
  "author_name": "",
  "post_date": "2019-11-03T01:13:03.143913200Z",
  "votes": 5,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Consider myself a novice even though I have participated in quiet a few competitions. Does feature order matter in Light GBM? Will using the training data with a different feature order change the results? For example my featureSet initially was set to numeric + categorical. That yielded one set of results. However when I changed it to categorical + numeric the results were totally different.</p>",
  "messages": [
    {
      "id": "663965",
      "postDate": "11/03/2019 01:13:03",
      "content": "<p>Consider myself a novice even though I have participated in quiet a few competitions. Does feature order matter in Light GBM? Will using the training data with a different feature order change the results? For example my featureSet initially was set to numeric + categorical. That yielded one set of results. However when I changed it to categorical + numeric the results were totally different.</p>",
      "rawMarkdown": "Consider myself a novice even though I have participated in quiet a few competitions. Does feature order matter in Light GBM? Will using the training data with a different feature order change the results? For example my featureSet initially was set to numeric + categorical. That yielded one set of results. However when I changed it to categorical + numeric the results were totally different.",
      "votes": null
    },
    {
      "id": "663988",
      "postDate": "11/03/2019 02:23:18",
      "content": "<p>I guess you are using feature subsampling ?</p>",
      "rawMarkdown": "I guess you are using feature subsampling ?",
      "votes": null
    },
    {
      "id": "664018",
      "postDate": "11/03/2019 03:37:04",
      "content": "<p>Did you use the same <code>random_state</code> for both?</p>",
      "rawMarkdown": "Did you use the same `random_state` for both?",
      "votes": null
    },
    {
      "id": "664066",
      "postDate": "11/03/2019 05:41:28",
      "content": "<p>Yes. 'subsample': 0.25</p>",
      "rawMarkdown": "Yes. 'subsample': 0.25",
      "votes": null
    },
    {
      "id": "664071",
      "postDate": "11/03/2019 05:44:10",
      "content": "<p>and how is feature subsampling related to feature order?</p>",
      "rawMarkdown": "and how is feature subsampling related to feature order?",
      "votes": null
    },
    {
      "id": "664075",
      "postDate": "11/03/2019 05:47:39",
      "content": "<p>Yes the order of features matter . You can read about it in the <a href=\"https://github.com/Microsoft/LightGBM/issues/1294\">Here</a></p>\n\n<p>In gist it says : </p>\n\n<blockquote>\n  <p>This is by design. feature order will affect the accuracy.\n  The reason is, when choose a feature to split tree node, if two features have the same split gain, the feature with smaller index(id) will be chosen.</p>\n</blockquote>\n\n<p>I would suggest you to use <a href=\"https://github.com/slundberg/shap\">SHAP</a> for understanding the feature importance</p>",
      "rawMarkdown": "Yes the order of features matter . You can read about it in the [Here](https://github.com/Microsoft/LightGBM/issues/1294)\n\nIn gist it says : \n&gt; This is by design. feature order will affect the accuracy.\nThe reason is, when choose a feature to split tree node, if two features have the same split gain, the feature with smaller index(id) will be chosen.\n\nI would suggest you to use [SHAP](https://github.com/slundberg/shap) for understanding the feature importance",
      "votes": null
    },
    {
      "id": "664106",
      "postDate": "11/03/2019 06:48:53",
      "content": "<p>In fact I believe that feature subsampling will still take the same indices, but the problem is that you change your feature order from numeric + categorical to categorical + numeric, and by the same indices the model just gets different subsets of features, that's why you encountered a different result I think</p>",
      "rawMarkdown": "In fact I believe that feature subsampling will still take the same indices, but the problem is that you change your feature order from numeric + categorical to categorical + numeric, and by the same indices the model just gets different subsets of features, that's why you encountered a different result I think",
      "votes": null
    },
    {
      "id": "664221",
      "postDate": "11/03/2019 11:23:22",
      "content": "<p><a href=\"/avikrams\">@avikrams</a> nice share. Thanks.</p>",
      "rawMarkdown": "avikrams nice share. Thanks.",
      "votes": null
    },
    {
      "id": "664257",
      "postDate": "11/03/2019 12:12:48",
      "content": "<p>Then how can we decide order ? Thanks</p>",
      "rawMarkdown": "Then how can we decide order ? Thanks",
      "votes": null
    },
    {
      "id": "664395",
      "postDate": "11/03/2019 16:04:38",
      "content": "<p>I also suggest to try the same random_state and let us know if the problem/behavior persists or not.</p>",
      "rawMarkdown": "I also suggest to try the same random_state and let us know if the problem/behavior persists or not.",
      "votes": null
    },
    {
      "id": "664551",
      "postDate": "11/03/2019 21:19:47",
      "content": "<p>Thanks a lot to everyone who replied. In this particular case I do not think it was a problem of numeric + categorical or vice versa. I think I was making mistakes in NaN imputations. After cleaning up the code now I am getting the same results in either order!</p>",
      "rawMarkdown": "Thanks a lot to everyone who replied. In this particular case I do not think it was a problem of numeric + categorical or vice versa. I think I was making mistakes in NaN imputations. After cleaning up the code now I am getting the same results in either order!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 663988,
      "author_name": "jielu0728",
      "author_url": "",
      "post_date": "11/03/2019 02:23:18",
      "content": "<p>I guess you are using feature subsampling ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 664066,
          "author_name": "viswajithkn",
          "author_url": "",
          "post_date": "11/03/2019 05:41:28",
          "content": "<p>Yes. 'subsample': 0.25</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 664071,
          "author_name": "viswajithkn",
          "author_url": "",
          "post_date": "11/03/2019 05:44:10",
          "content": "<p>and how is feature subsampling related to feature order?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 664106,
          "author_name": "jielu0728",
          "author_url": "",
          "post_date": "11/03/2019 06:48:53",
          "content": "<p>In fact I believe that feature subsampling will still take the same indices, but the problem is that you change your feature order from numeric + categorical to categorical + numeric, and by the same indices the model just gets different subsets of features, that's why you encountered a different result I think</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 664018,
      "author_name": "juanmah",
      "author_url": "",
      "post_date": "11/03/2019 03:37:04",
      "content": "<p>Did you use the same <code>random_state</code> for both?</p>",
      "votes": null,
      "replies": [
        {
          "id": 664395,
          "author_name": "arnabbiswas1",
          "author_url": "",
          "post_date": "11/03/2019 16:04:38",
          "content": "<p>I also suggest to try the same random_state and let us know if the problem/behavior persists or not.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 664075,
      "author_name": "avikrams",
      "author_url": "",
      "post_date": "11/03/2019 05:47:39",
      "content": "<p>Yes the order of features matter . You can read about it in the <a href=\"https://github.com/Microsoft/LightGBM/issues/1294\">Here</a></p>\n\n<p>In gist it says : </p>\n\n<blockquote>\n  <p>This is by design. feature order will affect the accuracy.\n  The reason is, when choose a feature to split tree node, if two features have the same split gain, the feature with smaller index(id) will be chosen.</p>\n</blockquote>\n\n<p>I would suggest you to use <a href=\"https://github.com/slundberg/shap\">SHAP</a> for understanding the feature importance</p>",
      "votes": null,
      "replies": [
        {
          "id": 664221,
          "author_name": "themonologue",
          "author_url": "",
          "post_date": "11/03/2019 11:23:22",
          "content": "<p><a href=\"/avikrams\">@avikrams</a> nice share. Thanks.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 664257,
          "author_name": "negi009",
          "author_url": "",
          "post_date": "11/03/2019 12:12:48",
          "content": "<p>Then how can we decide order ? Thanks</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 664551,
      "author_name": "viswajithkn",
      "author_url": "",
      "post_date": "11/03/2019 21:19:47",
      "content": "<p>Thanks a lot to everyone who replied. In this particular case I do not think it was a problem of numeric + categorical or vice versa. I think I was making mistakes in NaN imputations. After cleaning up the code now I am getting the same results in either order!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "663965": "Consider myself a novice even though I have participated in quiet a few competitions. Does feature order matter in Light GBM? Will using the training data with a different feature order change the results? For example my featureSet initially was set to numeric + categorical. That yielded one set of results. However when I changed it to categorical + numeric the results were totally different.",
    "663988": "I guess you are using feature subsampling ?",
    "664018": "Did you use the same `random_state` for both?",
    "664066": "Yes. 'subsample': 0.25",
    "664071": "and how is feature subsampling related to feature order?",
    "664075": "Yes the order of features matter . You can read about it in the [Here](https://github.com/Microsoft/LightGBM/issues/1294)\n\nIn gist it says : \n&gt; This is by design. feature order will affect the accuracy.\nThe reason is, when choose a feature to split tree node, if two features have the same split gain, the feature with smaller index(id) will be chosen.\n\nI would suggest you to use [SHAP](https://github.com/slundberg/shap) for understanding the feature importance",
    "664106": "In fact I believe that feature subsampling will still take the same indices, but the problem is that you change your feature order from numeric + categorical to categorical + numeric, and by the same indices the model just gets different subsets of features, that's why you encountered a different result I think",
    "664221": "avikrams nice share. Thanks.",
    "664257": "Then how can we decide order ? Thanks",
    "664395": "I also suggest to try the same random_state and let us know if the problem/behavior persists or not.",
    "664551": "Thanks a lot to everyone who replied. In this particular case I do not think it was a problem of numeric + categorical or vice versa. I think I was making mistakes in NaN imputations. After cleaning up the code now I am getting the same results in either order!"
  },
  "source": "meta"
}