{
  "id": 82187,
  "title": "Feature Selection Issues",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/82187",
  "author_name": "",
  "post_date": "2019-02-27T19:27:27.180211400Z",
  "votes": 7,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi Everyone,</p>\n\n<p>I wanted to share what procedures i set up for feature selection and what my drawbacks are,\nIt'll be great if you can tell me what you think I can do to make it better</p>\n\n<hr>\n\n<p><strong>Forward Selection:</strong>\nI take 3-5 features from top importance and then add features 1 by 1, running a model and adding the feature to the model if the avg CV is better and at least 3 of the folds CV is better.\nEach feature that is added, is used in next selection, and each feature that does not improve this way is discarded.</p>\n\n<p><strong>Backward Selection:</strong>\nI run a model with all features and then going 1 by 1 from lower importance to high importance.\nFor each feature i discard it and run a model,  if the avg CV is better and at least 3 of the folds CV is better, the feature is completely discarded, if not, it is kept.</p>\n\n<p><strong>My drawbacks here:</strong>\n- Given a set of 5 features that will work well together, adding them 1 by 1 and not all together, may get them discarded.</p>\n\n<ul>\n<li>the hyperparameters used in the beginning of the selection with 10-20 features, won't work the same when 50 features are selected, and the CV might get to a cap of improvement and not select further features because of this reason</li>\n</ul>\n\n<hr>\n\n<p>These issues did allow me to find sets of 20-30 feature that alone produce strong single models, but not amazing ones.</p>\n\n<p>I have a procedure for permutation importance but it has to run 3 times with different samples at least for each feature to avoid random improvement\\worsening of the model, this takes a long time in R to do in a sparse matrix that is needed for LGB prediction.</p>\n\n<p>Boruta also takes a HUGE amount of time in R, sometimes even more than 24 hours if dataset is large and many features.</p>\n\n<p>Any tips will be greatly appreciated\nThanks!!</p>",
  "messages": [
    {
      "id": "480103",
      "postDate": "02/27/2019 19:27:27",
      "content": "<p>Hi Everyone,</p>\n\n<p>I wanted to share what procedures i set up for feature selection and what my drawbacks are,\nIt'll be great if you can tell me what you think I can do to make it better</p>\n\n<hr>\n\n<p><strong>Forward Selection:</strong>\nI take 3-5 features from top importance and then add features 1 by 1, running a model and adding the feature to the model if the avg CV is better and at least 3 of the folds CV is better.\nEach feature that is added, is used in next selection, and each feature that does not improve this way is discarded.</p>\n\n<p><strong>Backward Selection:</strong>\nI run a model with all features and then going 1 by 1 from lower importance to high importance.\nFor each feature i discard it and run a model,  if the avg CV is better and at least 3 of the folds CV is better, the feature is completely discarded, if not, it is kept.</p>\n\n<p><strong>My drawbacks here:</strong>\n- Given a set of 5 features that will work well together, adding them 1 by 1 and not all together, may get them discarded.</p>\n\n<ul>\n<li>the hyperparameters used in the beginning of the selection with 10-20 features, won't work the same when 50 features are selected, and the CV might get to a cap of improvement and not select further features because of this reason</li>\n</ul>\n\n<hr>\n\n<p>These issues did allow me to find sets of 20-30 feature that alone produce strong single models, but not amazing ones.</p>\n\n<p>I have a procedure for permutation importance but it has to run 3 times with different samples at least for each feature to avoid random improvement\\worsening of the model, this takes a long time in R to do in a sparse matrix that is needed for LGB prediction.</p>\n\n<p>Boruta also takes a HUGE amount of time in R, sometimes even more than 24 hours if dataset is large and many features.</p>\n\n<p>Any tips will be greatly appreciated\nThanks!!</p>",
      "rawMarkdown": "Hi Everyone,\n\nI wanted to share what procedures i set up for feature selection and what my drawbacks are,\nIt'll be great if you can tell me what you think I can do to make it better\n\n\n----------\n\n\n**Forward Selection:**\nI take 3-5 features from top importance and then add features 1 by 1, running a model and adding the feature to the model if the avg CV is better and at least 3 of the folds CV is better.\nEach feature that is added, is used in next selection, and each feature that does not improve this way is discarded.\n\n\n**Backward Selection:**\nI run a model with all features and then going 1 by 1 from lower importance to high importance.\nFor each feature i discard it and run a model,  if the avg CV is better and at least 3 of the folds CV is better, the feature is completely discarded, if not, it is kept.\n\n\n\n**My drawbacks here:**\n- Given a set of 5 features that will work well together, adding them 1 by 1 and not all together, may get them discarded.\n\n- the hyperparameters used in the beginning of the selection with 10-20 features, won't work the same when 50 features are selected, and the CV might get to a cap of improvement and not select further features because of this reason\n\n\n----------\n\n\nThese issues did allow me to find sets of 20-30 feature that alone produce strong single models, but not amazing ones.\n\n\nI have a procedure for permutation importance but it has to run 3 times with different samples at least for each feature to avoid random improvement\\worsening of the model, this takes a long time in R to do in a sparse matrix that is needed for LGB prediction.\n\nBoruta also takes a HUGE amount of time in R, sometimes even more than 24 hours if dataset is large and many features.\n\n\nAny tips will be greatly appreciated\nThanks!!",
      "votes": null
    },
    {
      "id": "480850",
      "postDate": "02/28/2019 19:08:11",
      "content": "<p>I basically uses same methodology which you mentioned here for feature selection. I would be glad if you can throw some more light on permutation selection. BTW congrats for a solid finish:)</p>",
      "rawMarkdown": "I basically uses same methodology which you mentioned here for feature selection. I would be glad if you can throw some more light on permutation selection. BTW congrats for a solid finish:)",
      "votes": null
    },
    {
      "id": "480888",
      "postDate": "02/28/2019 20:23:57",
      "content": "<p>Sure <a href=\"/aashish\">@aashish</a> malik,\nIn permutation importance I basically train a model, make a prediction on an OOF set and save the CV score. let's call the CV score \"Base Score\", then for each feature I:\n1) Shuffle its values randomly in the OOF table\n2)  Predict the target on the OOF table again,after the shuffle of the feature , and save aside the CV score, let's call that \"Score After Shuffle\"\n3) If \"Score After Shuffle\" is same or better then \"Base Score\", this means that the feature is not good and can be discarded.</p>\n\n<p>In order to avoid random good\\bad results, do this 3-5 times  for each feature (shuffle, predict, compare) and act accordingly.</p>",
      "rawMarkdown": "Sure @aashish malik,\nIn permutation importance I basically train a model, make a prediction on an OOF set and save the CV score. let's call the CV score \"Base Score\", then for each feature I:\n1) Shuffle its values randomly in the OOF table\n2)  Predict the target on the OOF table again,after the shuffle of the feature , and save aside the CV score, let's call that \"Score After Shuffle\"\n3) If \"Score After Shuffle\" is same or better then \"Base Score\", this means that the feature is not good and can be discarded.\n\nIn order to avoid random good\\bad results, do this 3-5 times  for each feature (shuffle, predict, compare) and act accordingly.",
      "votes": null
    },
    {
      "id": "480894",
      "postDate": "02/28/2019 20:32:25",
      "content": "<p>Thanks a ton AmirH, I will surely try this in future competitions. I guess in statistical terms we call it \"hypothesis testing\".</p>",
      "rawMarkdown": "Thanks a ton AmirH, I will surely try this in future competitions. I guess in statistical terms we call it \"hypothesis testing\".",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 480850,
      "author_name": "aashish7936",
      "author_url": "",
      "post_date": "02/28/2019 19:08:11",
      "content": "<p>I basically uses same methodology which you mentioned here for feature selection. I would be glad if you can throw some more light on permutation selection. BTW congrats for a solid finish:)</p>",
      "votes": null,
      "replies": [
        {
          "id": 480888,
          "author_name": "tpthegreat",
          "author_url": "",
          "post_date": "02/28/2019 20:23:57",
          "content": "<p>Sure <a href=\"/aashish\">@aashish</a> malik,\nIn permutation importance I basically train a model, make a prediction on an OOF set and save the CV score. let's call the CV score \"Base Score\", then for each feature I:\n1) Shuffle its values randomly in the OOF table\n2)  Predict the target on the OOF table again,after the shuffle of the feature , and save aside the CV score, let's call that \"Score After Shuffle\"\n3) If \"Score After Shuffle\" is same or better then \"Base Score\", this means that the feature is not good and can be discarded.</p>\n\n<p>In order to avoid random good\\bad results, do this 3-5 times  for each feature (shuffle, predict, compare) and act accordingly.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480894,
          "author_name": "aashish7936",
          "author_url": "",
          "post_date": "02/28/2019 20:32:25",
          "content": "<p>Thanks a ton AmirH, I will surely try this in future competitions. I guess in statistical terms we call it \"hypothesis testing\".</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "480103": "Hi Everyone,\n\nI wanted to share what procedures i set up for feature selection and what my drawbacks are,\nIt'll be great if you can tell me what you think I can do to make it better\n\n\n----------\n\n\n**Forward Selection:**\nI take 3-5 features from top importance and then add features 1 by 1, running a model and adding the feature to the model if the avg CV is better and at least 3 of the folds CV is better.\nEach feature that is added, is used in next selection, and each feature that does not improve this way is discarded.\n\n\n**Backward Selection:**\nI run a model with all features and then going 1 by 1 from lower importance to high importance.\nFor each feature i discard it and run a model,  if the avg CV is better and at least 3 of the folds CV is better, the feature is completely discarded, if not, it is kept.\n\n\n\n**My drawbacks here:**\n- Given a set of 5 features that will work well together, adding them 1 by 1 and not all together, may get them discarded.\n\n- the hyperparameters used in the beginning of the selection with 10-20 features, won't work the same when 50 features are selected, and the CV might get to a cap of improvement and not select further features because of this reason\n\n\n----------\n\n\nThese issues did allow me to find sets of 20-30 feature that alone produce strong single models, but not amazing ones.\n\n\nI have a procedure for permutation importance but it has to run 3 times with different samples at least for each feature to avoid random improvement\\worsening of the model, this takes a long time in R to do in a sparse matrix that is needed for LGB prediction.\n\nBoruta also takes a HUGE amount of time in R, sometimes even more than 24 hours if dataset is large and many features.\n\n\nAny tips will be greatly appreciated\nThanks!!",
    "480850": "I basically uses same methodology which you mentioned here for feature selection. I would be glad if you can throw some more light on permutation selection. BTW congrats for a solid finish:)",
    "480888": "Sure @aashish malik,\nIn permutation importance I basically train a model, make a prediction on an OOF set and save the CV score. let's call the CV score \"Base Score\", then for each feature I:\n1) Shuffle its values randomly in the OOF table\n2)  Predict the target on the OOF table again,after the shuffle of the feature , and save aside the CV score, let's call that \"Score After Shuffle\"\n3) If \"Score After Shuffle\" is same or better then \"Base Score\", this means that the feature is not good and can be discarded.\n\nIn order to avoid random good\\bad results, do this 3-5 times  for each feature (shuffle, predict, compare) and act accordingly.",
    "480894": "Thanks a ton AmirH, I will surely try this in future competitions. I guess in statistical terms we call it \"hypothesis testing\"."
  },
  "source": "meta"
}