{
  "id": 548471,
  "title": "There are systematic errors, so ensemble accordingly",
  "url": "/competitions/brist1d/discussion/548471",
  "author_name": "Tilii",
  "post_date": "2024-11-27T02:04:50.780000",
  "votes": 15,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Below is a scatterplot of target values vs out-of-fold predictions by CatBoost. It is hopefully obvious that for small target values (up to ~7 on x-axis) most of data points are above the diagonal. That means we are systematically over-predicting small targets. Conversely, most of high targets are below the  diagonal, which means they are under-predicted.</p>\n<p><img src=\"https://i.ibb.co/WVRDvzX/Catboost-errors.png\" alt=\"CatBoost errors\"></p>\n<p>When ensembling, we need to combine models that complement each other, rather than models that are similar to each other.</p>\n<p>Below are cumulative distribution functions of the two neural network models. They are very similar to each other, and make similar errors with regard to target values (over-predict small and under-predict large values).</p>\n<p><img src=\"https://i.ibb.co/VVbHBbH/distributions-NN1-NN2.png\" alt=\"CDF similar comparison\"></p>\n<p>Below are cumulative distribution functions of a neural network model and a CatBoost model. They are more different than in the above plot: the NN model is closer to the target for small values (target values are now on the y-axis), while the CatBoost model is closer to the target for large values. These two models should complement each other well.</p>\n<p><img src=\"https://i.ibb.co/m576nxY/distributions-NN-Cat-B.png\" alt=\"CDF diverse comparison\"></p>",
  "messages": [
    {
      "id": 3056464,
      "postDate": "2024-11-27T02:04:50.780Z",
      "content": "<p>Below is a scatterplot of target values vs out-of-fold predictions by CatBoost. It is hopefully obvious that for small target values (up to ~7 on x-axis) most of data points are above the diagonal. That means we are systematically over-predicting small targets. Conversely, most of high targets are below the  diagonal, which means they are under-predicted.</p>\n<p><img src=\"https://i.ibb.co/WVRDvzX/Catboost-errors.png\" alt=\"CatBoost errors\"></p>\n<p>When ensembling, we need to combine models that complement each other, rather than models that are similar to each other.</p>\n<p>Below are cumulative distribution functions of the two neural network models. They are very similar to each other, and make similar errors with regard to target values (over-predict small and under-predict large values).</p>\n<p><img src=\"https://i.ibb.co/VVbHBbH/distributions-NN1-NN2.png\" alt=\"CDF similar comparison\"></p>\n<p>Below are cumulative distribution functions of a neural network model and a CatBoost model. They are more different than in the above plot: the NN model is closer to the target for small values (target values are now on the y-axis), while the CatBoost model is closer to the target for large values. These two models should complement each other well.</p>\n<p><img src=\"https://i.ibb.co/m576nxY/distributions-NN-Cat-B.png\" alt=\"CDF diverse comparison\"></p>",
      "rawMarkdown": "Below is a scatterplot of target values vs out-of-fold predictions by CatBoost. It is hopefully obvious that for small target values (up to ~7 on x-axis) most of data points are above the diagonal. That means we are systematically over-predicting small targets. Conversely, most of high targets are below the  diagonal, which means they are under-predicted.\n\n![CatBoost errors](https://i.ibb.co/WVRDvzX/Catboost-errors.png)\n\nWhen ensembling, we need to combine models that complement each other, rather than models that are similar to each other.\n\nBelow are cumulative distribution functions of the two neural network models. They are very similar to each other, and make similar errors with regard to target values (over-predict small and under-predict large values).\n\n![CDF similar comparison](https://i.ibb.co/VVbHBbH/distributions-NN1-NN2.png)\n\nBelow are cumulative distribution functions of a neural network model and a CatBoost model. They are more different than in the above plot: the NN model is closer to the target for small values (target values are now on the y-axis), while the CatBoost model is closer to the target for large values. These two models should complement each other well.\n\n![CDF diverse comparison](https://i.ibb.co/m576nxY/distributions-NN-Cat-B.png)",
      "votes": 14
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3056464": "Below is a scatterplot of target values vs out-of-fold predictions by CatBoost. It is hopefully obvious that for small target values (up to ~7 on x-axis) most of data points are above the diagonal. That means we are systematically over-predicting small targets. Conversely, most of high targets are below the  diagonal, which means they are under-predicted.\n\n![CatBoost errors](https://i.ibb.co/WVRDvzX/Catboost-errors.png)\n\nWhen ensembling, we need to combine models that complement each other, rather than models that are similar to each other.\n\nBelow are cumulative distribution functions of the two neural network models. They are very similar to each other, and make similar errors with regard to target values (over-predict small and under-predict large values).\n\n![CDF similar comparison](https://i.ibb.co/VVbHBbH/distributions-NN1-NN2.png)\n\nBelow are cumulative distribution functions of a neural network model and a CatBoost model. They are more different than in the above plot: the NN model is closer to the target for small values (target values are now on the y-axis), while the CatBoost model is closer to the target for large values. These two models should complement each other well.\n\n![CDF diverse comparison](https://i.ibb.co/m576nxY/distributions-NN-Cat-B.png)"
  }
}