{
  "id": 16768,
  "title": "How to pass KS test?",
  "url": "/competitions/flavours-of-physics/discussion/16768",
  "author_name": "",
  "post_date": "2015-10-02T03:02:44.987Z",
  "votes": null,
  "comment_count": 5,
  "views": 736,
  "content": "<p>I just joined this competition and logic that helped me in all previous challenges is not useful here for me so far.</p>\n\n<p>As a first step I am trying to do cross validation using xgboost on the default features. </p>\n\n<p>I have understanding of how xgboost works, what parameters will make model more conservative/more complex  and this usually allows me control gap between train/validation set.</p>\n\n<p>Right now I have a problem. For any set of the hyperparameters that I checked so far for at leas on of the folds KS test give &gt; 0.09.</p>\n\n<p>But I have no clue, what parameters should I change/tune to be able to get less than 0.09 on the Kolmogorov-Smirnov test.</p>\n\n<p>Can anyone give me a hint where should I look to be able to pass KS test? How does value of the KS test related to the algorithm/data/physics of the model?  </p>",
  "messages": [
    {
      "id": "93849",
      "postDate": "10/02/2015 03:02:44",
      "content": "<p>I just joined this competition and logic that helped me in all previous challenges is not useful here for me so far.</p>\n\n<p>As a first step I am trying to do cross validation using xgboost on the default features. </p>\n\n<p>I have understanding of how xgboost works, what parameters will make model more conservative/more complex  and this usually allows me control gap between train/validation set.</p>\n\n<p>Right now I have a problem. For any set of the hyperparameters that I checked so far for at leas on of the folds KS test give &gt; 0.09.</p>\n\n<p>But I have no clue, what parameters should I change/tune to be able to get less than 0.09 on the Kolmogorov-Smirnov test.</p>\n\n<p>Can anyone give me a hint where should I look to be able to pass KS test? How does value of the KS test related to the algorithm/data/physics of the model?  </p>",
      "rawMarkdown": "I just joined this competition and logic that helped me in all previous challenges is not useful here for me so far.\r\n\r\nAs a first step I am trying to do cross validation using xgboost on the default features. \r\n\r\nI have understanding of how xgboost works, what parameters will make model more conservative/more complex  and this usually allows me control gap between train/validation set.\r\n\r\nRight now I have a problem. For any set of the hyperparameters that I checked so far for at leas on of the folds KS test give > 0.09.\r\n \r\nBut I have no clue, what parameters should I change/tune to be able to get less than 0.09 on the Kolmogorov-Smirnov test.\r\n\r\nCan anyone give me a hint where should I look to be able to pass KS test? How does value of the KS test related to the algorithm/data/physics of the model?",
      "votes": null
    },
    {
      "id": "94953",
      "postDate": "10/03/2015 14:11:55",
      "content": "<p>One solution is not to use SPDhits. </p>",
      "rawMarkdown": "One solution is not to use SPDhits.",
      "votes": null
    },
    {
      "id": "94956",
      "postDate": "10/03/2015 15:39:04",
      "content": "<p>Usually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.</p>",
      "rawMarkdown": "Usually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.",
      "votes": null
    },
    {
      "id": "94957",
      "postDate": "10/03/2015 15:46:22",
      "content": "<p>[quote=rakhlin;94956]</p>\n\n<p>Usually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.</p>\n\n<p>[/quote]\nI got similar feeling, just playing around with parameters. But there is something conceptually wrong with this approach...</p>\n\n<p>RF does not have a problem passing KS test, but I was not able to get more than 0.981 with it.</p>",
      "rawMarkdown": "[quote=rakhlin;94956]\r\n\r\nUsually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.\r\n\r\n[/quote]\r\nI got similar feeling, just playing around with parameters. But there is something conceptually wrong with this approach...\r\n\r\nRF does not have a problem passing KS test, but I was not able to get more than 0.981 with it.",
      "votes": null
    },
    {
      "id": "94960",
      "postDate": "10/03/2015 15:54:58",
      "content": "<p><strong>phunter</strong> mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.</p>",
      "rawMarkdown": "**phunter** mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.",
      "votes": null
    },
    {
      "id": "94992",
      "postDate": "10/04/2015 01:34:57",
      "content": "<p>[quote=rakhlin;94960]</p>\n\n<p><strong>phunter</strong> mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.</p>\n\n<p>[/quote]\nthere are two ways to pass KS test: either building a classifier where both tau-&gt;3 mu and Ds-&gt;phi pi (phi-&gt;2 mu) are positive, or a classifier of distinguishing tau-&gt;3 mu vs Ds-&gt;phi pi using check_agreement.csv since Ds-&gt;phi pi events are not considered for the final score.  The second method is very efficient, and rather simple if you have found the secret, but it is not allowed and I will be disqualified if I tell the secret here.. The first method needs some tricks/experiments in both features and models for having more tolerance in your classifier. In short, don't let your classifier overfit on tau-&gt;3mu vs background, but be general for (tau-&gt;3mu and Ds-&gt;phi pi) vs background.</p>",
      "rawMarkdown": "[quote=rakhlin;94960]\r\n\r\n**phunter** mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.\r\n\r\n[/quote]\r\nthere are two ways to pass KS test: either building a classifier where both tau->3 mu and Ds->phi pi (phi->2 mu) are positive, or a classifier of distinguishing tau->3 mu vs Ds->phi pi using check_agreement.csv since Ds->phi pi events are not considered for the final score.  The second method is very efficient, and rather simple if you have found the secret, but it is not allowed and I will be disqualified if I tell the secret here.. The first method needs some tricks/experiments in both features and models for having more tolerance in your classifier. In short, don't let your classifier overfit on tau->3mu vs background, but be general for (tau->3mu and Ds->phi pi) vs background.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 94953,
      "author_name": "pinocchio",
      "author_url": "",
      "post_date": "10/03/2015 14:11:55",
      "content": "<p>One solution is not to use SPDhits. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 94956,
      "author_name": "rakhlin",
      "author_url": "",
      "post_date": "10/03/2015 15:39:04",
      "content": "<p>Usually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 94957,
      "author_name": "iglovikov",
      "author_url": "",
      "post_date": "10/03/2015 15:46:22",
      "content": "<p>[quote=rakhlin;94956]</p>\n\n<p>Usually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.</p>\n\n<p>[/quote]\nI got similar feeling, just playing around with parameters. But there is something conceptually wrong with this approach...</p>\n\n<p>RF does not have a problem passing KS test, but I was not able to get more than 0.981 with it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 94960,
      "author_name": "rakhlin",
      "author_url": "",
      "post_date": "10/03/2015 15:54:58",
      "content": "<p><strong>phunter</strong> mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 94992,
      "author_name": "phunter",
      "author_url": "",
      "post_date": "10/04/2015 01:34:57",
      "content": "<p>[quote=rakhlin;94960]</p>\n\n<p><strong>phunter</strong> mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.</p>\n\n<p>[/quote]\nthere are two ways to pass KS test: either building a classifier where both tau-&gt;3 mu and Ds-&gt;phi pi (phi-&gt;2 mu) are positive, or a classifier of distinguishing tau-&gt;3 mu vs Ds-&gt;phi pi using check_agreement.csv since Ds-&gt;phi pi events are not considered for the final score.  The second method is very efficient, and rather simple if you have found the secret, but it is not allowed and I will be disqualified if I tell the secret here.. The first method needs some tricks/experiments in both features and models for having more tolerance in your classifier. In short, don't let your classifier overfit on tau-&gt;3mu vs background, but be general for (tau-&gt;3mu and Ds-&gt;phi pi) vs background.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "93849": "I just joined this competition and logic that helped me in all previous challenges is not useful here for me so far.\r\n\r\nAs a first step I am trying to do cross validation using xgboost on the default features. \r\n\r\nI have understanding of how xgboost works, what parameters will make model more conservative/more complex  and this usually allows me control gap between train/validation set.\r\n\r\nRight now I have a problem. For any set of the hyperparameters that I checked so far for at leas on of the folds KS test give > 0.09.\r\n \r\nBut I have no clue, what parameters should I change/tune to be able to get less than 0.09 on the Kolmogorov-Smirnov test.\r\n\r\nCan anyone give me a hint where should I look to be able to pass KS test? How does value of the KS test related to the algorithm/data/physics of the model?",
    "94953": "One solution is not to use SPDhits.",
    "94956": "Usually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.",
    "94957": "[quote=rakhlin;94956]\r\n\r\nUsually it is not enough if you train xgboost for maximum performance. Undertraining would 'help'.\r\n\r\n[/quote]\r\nI got similar feeling, just playing around with parameters. But there is something conceptually wrong with this approach...\r\n\r\nRF does not have a problem passing KS test, but I was not able to get more than 0.981 with it.",
    "94960": "**phunter** mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.",
    "94992": "[quote=rakhlin;94960]\r\n\r\n**phunter** mentioned another solution, to make SPDhits and some other top predictors a little 'bad', but hasn't provided details. I tried adding normally distributed noise, it didn't help.\r\n\r\n[/quote]\r\nthere are two ways to pass KS test: either building a classifier where both tau->3 mu and Ds->phi pi (phi->2 mu) are positive, or a classifier of distinguishing tau->3 mu vs Ds->phi pi using check_agreement.csv since Ds->phi pi events are not considered for the final score.  The second method is very efficient, and rather simple if you have found the secret, but it is not allowed and I will be disqualified if I tell the secret here.. The first method needs some tricks/experiments in both features and models for having more tolerance in your classifier. In short, don't let your classifier overfit on tau->3mu vs background, but be general for (tau->3mu and Ds->phi pi) vs background."
  },
  "source": "meta"
}