{
  "id": 82153,
  "title": "ELO Million dollar question",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/82153",
  "author_name": "",
  "post_date": "2019-02-27T14:27:20.602332300Z",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Ok it's not a million dollar question but when I see some of the top50 competitors used a lot of models with different features, anyone can shed some light why this is beneficial? Assuming you have a total of 300 features, why need to have 10 models with 30 features each? The moment I start to remove some features from my single model, the rmse CV score goes up.</p>",
  "messages": [
    {
      "id": "479878",
      "postDate": "02/27/2019 14:27:20",
      "content": "<p>Ok it's not a million dollar question but when I see some of the top50 competitors used a lot of models with different features, anyone can shed some light why this is beneficial? Assuming you have a total of 300 features, why need to have 10 models with 30 features each? The moment I start to remove some features from my single model, the rmse CV score goes up.</p>",
      "rawMarkdown": "Ok it's not a million dollar question but when I see some of the top50 competitors used a lot of models with different features, anyone can shed some light why this is beneficial? Assuming you have a total of 300 features, why need to have 10 models with 30 features each? The moment I start to remove some features from my single model, the rmse CV score goes up.",
      "votes": null
    },
    {
      "id": "479896",
      "postDate": "02/27/2019 14:43:54",
      "content": "<p>One reason is that most machine learning models are greedy. They will add more weights to an important feature that give immediate improvement. If there are some highly correlated features, it is very common that a model will just pick one of them and ignore the rest. For example, the month_lags statistics for all transactions vs the month_lags statistics for all historical transactions vs the month_lags statistics for all authorized transactions etc...</p>\n\n<p>Blending models with different feature sets can take information from those \"not so important features\".</p>",
      "rawMarkdown": "One reason is that most machine learning models are greedy. They will add more weights to an important feature that give immediate improvement. If there are some highly correlated features, it is very common that a model will just pick one of them and ignore the rest. For example, the month_lags statistics for all transactions vs the month_lags statistics for all historical transactions vs the month_lags statistics for all authorized transactions etc...\n\nBlending models with different feature sets can take information from those \"not so important features\".",
      "votes": null
    },
    {
      "id": "479901",
      "postDate": "02/27/2019 14:47:34",
      "content": "<p>I would like to know how the top people push public LB rmse to 3.690 or lower without ensemble, stacking and any other post-processing technique. This is my number 1 question other than parameter tuning. </p>",
      "rawMarkdown": "I would like to know how the top people push public LB rmse to 3.690 or lower without ensemble, stacking and any other post-processing technique. This is my number 1 question other than parameter tuning.",
      "votes": null
    },
    {
      "id": "479908",
      "postDate": "02/27/2019 14:56:16",
      "content": "<blockquote>\n  <p>why need to have 10 models with 30 features each? </p>\n</blockquote>\n\n<p>You'd have 10 models with 300 features each rather.  </p>",
      "rawMarkdown": "&gt; why need to have 10 models with 30 features each? \n\nYou'd have 10 models with 300 features each rather.",
      "votes": null
    },
    {
      "id": "479919",
      "postDate": "02/27/2019 15:15:03",
      "content": "<p>But if you split to multiple models with less features, each of this multiple models have an average rmse CV score that is higher than a single model with less rmse CV. So this is what confuses me. Blending with less important features will introduce more rmse error in the prediction right? </p>",
      "rawMarkdown": "But if you split to multiple models with less features, each of this multiple models have an average rmse CV score that is higher than a single model with less rmse CV. So this is what confuses me. Blending with less important features will introduce more rmse error in the prediction right?",
      "votes": null
    },
    {
      "id": "479927",
      "postDate": "02/27/2019 15:24:37",
      "content": "<p>General idea, you can try to blend models with reasonable performance with not highly correlated predictions. Models with super high error won't be helpful, but models with slightly higher error will be useful.</p>",
      "rawMarkdown": "General idea, you can try to blend models with reasonable performance with not highly correlated predictions. Models with super high error won't be helpful, but models with slightly higher error will be useful.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 479896,
      "author_name": "kingychiu",
      "author_url": "",
      "post_date": "02/27/2019 14:43:54",
      "content": "<p>One reason is that most machine learning models are greedy. They will add more weights to an important feature that give immediate improvement. If there are some highly correlated features, it is very common that a model will just pick one of them and ignore the rest. For example, the month_lags statistics for all transactions vs the month_lags statistics for all historical transactions vs the month_lags statistics for all authorized transactions etc...</p>\n\n<p>Blending models with different feature sets can take information from those \"not so important features\".</p>",
      "votes": null,
      "replies": [
        {
          "id": 479919,
          "author_name": "klloke",
          "author_url": "",
          "post_date": "02/27/2019 15:15:03",
          "content": "<p>But if you split to multiple models with less features, each of this multiple models have an average rmse CV score that is higher than a single model with less rmse CV. So this is what confuses me. Blending with less important features will introduce more rmse error in the prediction right? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479927,
          "author_name": "kingychiu",
          "author_url": "",
          "post_date": "02/27/2019 15:24:37",
          "content": "<p>General idea, you can try to blend models with reasonable performance with not highly correlated predictions. Models with super high error won't be helpful, but models with slightly higher error will be useful.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479901,
      "author_name": "pukkinming",
      "author_url": "",
      "post_date": "02/27/2019 14:47:34",
      "content": "<p>I would like to know how the top people push public LB rmse to 3.690 or lower without ensemble, stacking and any other post-processing technique. This is my number 1 question other than parameter tuning. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479908,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "02/27/2019 14:56:16",
      "content": "<blockquote>\n  <p>why need to have 10 models with 30 features each? </p>\n</blockquote>\n\n<p>You'd have 10 models with 300 features each rather.  </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "479878": "Ok it's not a million dollar question but when I see some of the top50 competitors used a lot of models with different features, anyone can shed some light why this is beneficial? Assuming you have a total of 300 features, why need to have 10 models with 30 features each? The moment I start to remove some features from my single model, the rmse CV score goes up.",
    "479896": "One reason is that most machine learning models are greedy. They will add more weights to an important feature that give immediate improvement. If there are some highly correlated features, it is very common that a model will just pick one of them and ignore the rest. For example, the month_lags statistics for all transactions vs the month_lags statistics for all historical transactions vs the month_lags statistics for all authorized transactions etc...\n\nBlending models with different feature sets can take information from those \"not so important features\".",
    "479901": "I would like to know how the top people push public LB rmse to 3.690 or lower without ensemble, stacking and any other post-processing technique. This is my number 1 question other than parameter tuning.",
    "479908": "&gt; why need to have 10 models with 30 features each? \n\nYou'd have 10 models with 300 features each rather.",
    "479919": "But if you split to multiple models with less features, each of this multiple models have an average rmse CV score that is higher than a single model with less rmse CV. So this is what confuses me. Blending with less important features will introduce more rmse error in the prediction right?",
    "479927": "General idea, you can try to blend models with reasonable performance with not highly correlated predictions. Models with super high error won't be helpful, but models with slightly higher error will be useful."
  },
  "source": "meta"
}