{
  "id": 16692,
  "title": "How come I always have submissions KS score 0.0906, 0.0907...",
  "url": "/competitions/flavours-of-physics/discussion/16692",
  "author_name": "",
  "post_date": "2015-09-27T17:36:21.897Z",
  "votes": null,
  "comment_count": 4,
  "views": 609,
  "content": "<p>Not sure what's going on, I had this kind of submission on several different models.</p>\n\n<p>Is there any quick fix for it ?  Or the only thing I can do is to reduce the number of iterations of my model.</p>",
  "messages": [
    {
      "id": "93539",
      "postDate": "09/27/2015 17:36:21",
      "content": "<p>Not sure what's going on, I had this kind of submission on several different models.</p>\n\n<p>Is there any quick fix for it ?  Or the only thing I can do is to reduce the number of iterations of my model.</p>",
      "rawMarkdown": "Not sure what's going on, I had this kind of submission on several different models.\r\n\r\nIs there any quick fix for it ?  Or the only thing I can do is to reduce the number of iterations of my model.",
      "votes": null
    },
    {
      "id": "93581",
      "postDate": "09/28/2015 13:36:47",
      "content": "<p>You can try to exclude SPDhits from the features you use while training your model.</p>",
      "rawMarkdown": "You can try to exclude SPDhits from the features you use while training your model.",
      "votes": null
    },
    {
      "id": "93591",
      "postDate": "09/28/2015 15:37:56",
      "content": "<p>I already did that, like what they did in the script.</p>",
      "rawMarkdown": "I already did that, like what they did in the script.",
      "votes": null
    },
    {
      "id": "93593",
      "postDate": "09/28/2015 15:55:31",
      "content": "<p>Then there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other &quot;bad&quot; features or to change your model in a different way.</p>",
      "rawMarkdown": "Then there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other \"bad\" features or to change your model in a different way.",
      "votes": null
    },
    {
      "id": "93609",
      "postDate": "09/28/2015 21:45:25",
      "content": "<p>[quote=Alexander Gramolin;93593]</p>\n\n<p>Then there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other &quot;bad&quot; features or to change your model in a different way.</p>\n\n<p>[/quote]\nYes, it is true. In this post <a href=\"https://www.kaggle.com/benhamner/flavours-of-physics/exploring-the-cern-lhcb-data\">https://www.kaggle.com/benhamner/flavours-of-physics/exploring-the-cern-lhcb-data</a> the admin shows the feature importance from Random Forest where some of these top features are not nicely simulated in the Monte Carlo, e.g. SPDhits and others (IP, IPsig etc), so if the classifier is too powerful, e.g. xgboost which can pickup very complicated features, it may 'overfit' on these features, and have problem on KS score. The solution is about, either finding 'bad' features, or make these 'good' features a little bit 'bad'.</p>",
      "rawMarkdown": "[quote=Alexander Gramolin;93593]\r\n\r\nThen there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other \"bad\" features or to change your model in a different way.\r\n\r\n[/quote]\r\nYes, it is true. In this post https://www.kaggle.com/benhamner/flavours-of-physics/exploring-the-cern-lhcb-data the admin shows the feature importance from Random Forest where some of these top features are not nicely simulated in the Monte Carlo, e.g. SPDhits and others (IP, IPsig etc), so if the classifier is too powerful, e.g. xgboost which can pickup very complicated features, it may 'overfit' on these features, and have problem on KS score. The solution is about, either finding 'bad' features, or make these 'good' features a little bit 'bad'.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 93581,
      "author_name": "gramolin",
      "author_url": "",
      "post_date": "09/28/2015 13:36:47",
      "content": "<p>You can try to exclude SPDhits from the features you use while training your model.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 93591,
      "author_name": "skylibrary",
      "author_url": "",
      "post_date": "09/28/2015 15:37:56",
      "content": "<p>I already did that, like what they did in the script.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 93593,
      "author_name": "gramolin",
      "author_url": "",
      "post_date": "09/28/2015 15:55:31",
      "content": "<p>Then there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other &quot;bad&quot; features or to change your model in a different way.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 93609,
      "author_name": "phunter",
      "author_url": "",
      "post_date": "09/28/2015 21:45:25",
      "content": "<p>[quote=Alexander Gramolin;93593]</p>\n\n<p>Then there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other &quot;bad&quot; features or to change your model in a different way.</p>\n\n<p>[/quote]\nYes, it is true. In this post <a href=\"https://www.kaggle.com/benhamner/flavours-of-physics/exploring-the-cern-lhcb-data\">https://www.kaggle.com/benhamner/flavours-of-physics/exploring-the-cern-lhcb-data</a> the admin shows the feature importance from Random Forest where some of these top features are not nicely simulated in the Monte Carlo, e.g. SPDhits and others (IP, IPsig etc), so if the classifier is too powerful, e.g. xgboost which can pickup very complicated features, it may 'overfit' on these features, and have problem on KS score. The solution is about, either finding 'bad' features, or make these 'good' features a little bit 'bad'.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "93539": "Not sure what's going on, I had this kind of submission on several different models.\r\n\r\nIs there any quick fix for it ?  Or the only thing I can do is to reduce the number of iterations of my model.",
    "93581": "You can try to exclude SPDhits from the features you use while training your model.",
    "93591": "I already did that, like what they did in the script.",
    "93593": "Then there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other \"bad\" features or to change your model in a different way.",
    "93609": "[quote=Alexander Gramolin;93593]\r\n\r\nThen there is no a quick fix, I'm afraid. The general idea is that some features (SPDhits, first of all) are poorly reproduced in the Monte Carlo simulation they provide for the signal channel. The model you use takes advantages of those features, while the organizers want to have a physically meaningful classifier. You can try to find other \"bad\" features or to change your model in a different way.\r\n\r\n[/quote]\r\nYes, it is true. In this post https://www.kaggle.com/benhamner/flavours-of-physics/exploring-the-cern-lhcb-data the admin shows the feature importance from Random Forest where some of these top features are not nicely simulated in the Monte Carlo, e.g. SPDhits and others (IP, IPsig etc), so if the classifier is too powerful, e.g. xgboost which can pickup very complicated features, it may 'overfit' on these features, and have problem on KS score. The solution is about, either finding 'bad' features, or make these 'good' features a little bit 'bad'."
  },
  "source": "meta"
}