{
  "id": 131221,
  "title": "How to train an ML model with lots of various features generated from FE? ",
  "url": "/competitions/cat-in-the-dat-ii/discussion/131221",
  "author_name": "",
  "post_date": "2020-02-18T22:11:40.621917700Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi Kagglers\nThere are many ways to do feature engineering on categorical variables(I'm talking about classical FE like dummy encoding target encoding, ...) and those features have different impacts on an ML model. Besides, those features might be correlated or not. In some cases, we might get more than 1000 features. </p>\n\n<p>When I am training the ML model, what is your recommendation? should I keep all features and add regularization? Feature selection? How to avoid overfitting? and so on.</p>\n\n<p>Your thoughts will be really helpful,\nThanks,</p>",
  "messages": [
    {
      "id": "749744",
      "postDate": "02/18/2020 22:11:40",
      "content": "<p>Hi Kagglers\nThere are many ways to do feature engineering on categorical variables(I'm talking about classical FE like dummy encoding target encoding, ...) and those features have different impacts on an ML model. Besides, those features might be correlated or not. In some cases, we might get more than 1000 features. </p>\n\n<p>When I am training the ML model, what is your recommendation? should I keep all features and add regularization? Feature selection? How to avoid overfitting? and so on.</p>\n\n<p>Your thoughts will be really helpful,\nThanks,</p>",
      "rawMarkdown": "Hi Kagglers\nThere are many ways to do feature engineering on categorical variables(I'm talking about classical FE like dummy encoding target encoding, ...) and those features have different impacts on an ML model. Besides, those features might be correlated or not. In some cases, we might get more than 1000 features. \n\nWhen I am training the ML model, what is your recommendation? should I keep all features and add regularization? Feature selection? How to avoid overfitting? and so on.\n\nYour thoughts will be really helpful,\nThanks,",
      "votes": null
    },
    {
      "id": "749825",
      "postDate": "02/18/2020 23:24:03",
      "content": "<p>1000 features can really slow a model training down. If you want to limit the features and speed up training, look into feature selection i.e. stepwise regression (forward or backward depending on your situation), PCA (Prinicipal component analysis), or svd (singular value decomposition). It depends on your resources and how important the variables can be. </p>",
      "rawMarkdown": "1000 features can really slow a model training down. If you want to limit the features and speed up training, look into feature selection i.e. stepwise regression (forward or backward depending on your situation), PCA (Prinicipal component analysis), or svd (singular value decomposition). It depends on your resources and how important the variables can be.",
      "votes": null
    },
    {
      "id": "749856",
      "postDate": "02/19/2020 00:05:59",
      "content": "<p>Thank you so much for your comment. The computation power is not the main issue for this conceptual question. Besides, I agree that linear models work properly when the number of features grows up. Moreover, imagine, the data is big enough to avoid curse dimensionality, and 1000 features are not still big for many cases. \nImagine again, lots of different features are created through many various ways of FE, then your suggestion is that doing PCA and SVD on top of them?  </p>\n\n<p>What PCA can add to this extensive FE? </p>",
      "rawMarkdown": "Thank you so much for your comment. The computation power is not the main issue for this conceptual question. Besides, I agree that linear models work properly when the number of features grows up. Moreover, imagine, the data is big enough to avoid curse dimensionality, and 1000 features are not still big for many cases. \nImagine again, lots of different features are created through many various ways of FE, then your suggestion is that doing PCA and SVD on top of them?  \n\nWhat PCA can add to this extensive FE?",
      "votes": null
    },
    {
      "id": "749947",
      "postDate": "02/19/2020 03:14:01",
      "content": "<p>PCA and SVD are built to capture the important variance and \"read\" through the white noise of the data. Keep in mind it won't capture all the information but it will make it much easier to run computations. For overfitting, I would be more concerned with the observation count given to the data rather than the number of variables that are given. </p>\n\n<p>PCA is useful for linear combinations but I wouldn't rely on it to help replace FE. Here is an article with some mathematical explination behind it: <a href=\"https://www.dezyre.com/data-science-in-python-tutorial/principal-component-analysis-tutorial\">https://www.dezyre.com/data-science-in-python-tutorial/principal-component-analysis-tutorial</a></p>",
      "rawMarkdown": "PCA and SVD are built to capture the important variance and \"read\" through the white noise of the data. Keep in mind it won't capture all the information but it will make it much easier to run computations. For overfitting, I would be more concerned with the observation count given to the data rather than the number of variables that are given. \n\nPCA is useful for linear combinations but I wouldn't rely on it to help replace FE. Here is an article with some mathematical explination behind it: https://www.dezyre.com/data-science-in-python-tutorial/principal-component-analysis-tutorial",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 749825,
      "author_name": "proselotis",
      "author_url": "",
      "post_date": "02/18/2020 23:24:03",
      "content": "<p>1000 features can really slow a model training down. If you want to limit the features and speed up training, look into feature selection i.e. stepwise regression (forward or backward depending on your situation), PCA (Prinicipal component analysis), or svd (singular value decomposition). It depends on your resources and how important the variables can be. </p>",
      "votes": null,
      "replies": [
        {
          "id": 749856,
          "author_name": "p333ir",
          "author_url": "",
          "post_date": "02/19/2020 00:05:59",
          "content": "<p>Thank you so much for your comment. The computation power is not the main issue for this conceptual question. Besides, I agree that linear models work properly when the number of features grows up. Moreover, imagine, the data is big enough to avoid curse dimensionality, and 1000 features are not still big for many cases. \nImagine again, lots of different features are created through many various ways of FE, then your suggestion is that doing PCA and SVD on top of them?  </p>\n\n<p>What PCA can add to this extensive FE? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 749947,
          "author_name": "proselotis",
          "author_url": "",
          "post_date": "02/19/2020 03:14:01",
          "content": "<p>PCA and SVD are built to capture the important variance and \"read\" through the white noise of the data. Keep in mind it won't capture all the information but it will make it much easier to run computations. For overfitting, I would be more concerned with the observation count given to the data rather than the number of variables that are given. </p>\n\n<p>PCA is useful for linear combinations but I wouldn't rely on it to help replace FE. Here is an article with some mathematical explination behind it: <a href=\"https://www.dezyre.com/data-science-in-python-tutorial/principal-component-analysis-tutorial\">https://www.dezyre.com/data-science-in-python-tutorial/principal-component-analysis-tutorial</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "749744": "Hi Kagglers\nThere are many ways to do feature engineering on categorical variables(I'm talking about classical FE like dummy encoding target encoding, ...) and those features have different impacts on an ML model. Besides, those features might be correlated or not. In some cases, we might get more than 1000 features. \n\nWhen I am training the ML model, what is your recommendation? should I keep all features and add regularization? Feature selection? How to avoid overfitting? and so on.\n\nYour thoughts will be really helpful,\nThanks,",
    "749825": "1000 features can really slow a model training down. If you want to limit the features and speed up training, look into feature selection i.e. stepwise regression (forward or backward depending on your situation), PCA (Prinicipal component analysis), or svd (singular value decomposition). It depends on your resources and how important the variables can be.",
    "749856": "Thank you so much for your comment. The computation power is not the main issue for this conceptual question. Besides, I agree that linear models work properly when the number of features grows up. Moreover, imagine, the data is big enough to avoid curse dimensionality, and 1000 features are not still big for many cases. \nImagine again, lots of different features are created through many various ways of FE, then your suggestion is that doing PCA and SVD on top of them?  \n\nWhat PCA can add to this extensive FE?",
    "749947": "PCA and SVD are built to capture the important variance and \"read\" through the white noise of the data. Keep in mind it won't capture all the information but it will make it much easier to run computations. For overfitting, I would be more concerned with the observation count given to the data rather than the number of variables that are given. \n\nPCA is useful for linear combinations but I wouldn't rely on it to help replace FE. Here is an article with some mathematical explination behind it: https://www.dezyre.com/data-science-in-python-tutorial/principal-component-analysis-tutorial"
  },
  "source": "meta"
}