{
  "id": 16619,
  "title": "agreement check",
  "url": "/competitions/flavours-of-physics/discussion/16619",
  "author_name": "",
  "post_date": "2015-09-22T15:39:39.193Z",
  "votes": null,
  "comment_count": 3,
  "views": 501,
  "content": "<p>Hi All, </p>\n\n<p>I submitted a couple predictions, but I got error messages saying that my agreement check is higher than 0.09. I did ks.test and I got below 0.09. Do you know what happened? </p>",
  "messages": [
    {
      "id": "93182",
      "postDate": "09/22/2015 15:39:39",
      "content": "<p>Hi All, </p>\n\n<p>I submitted a couple predictions, but I got error messages saying that my agreement check is higher than 0.09. I did ks.test and I got below 0.09. Do you know what happened? </p>",
      "rawMarkdown": "Hi All, \r\n\r\nI submitted a couple predictions, but I got error messages saying that my agreement check is higher than 0.09. I did ks.test and I got below 0.09. Do you know what happened?",
      "votes": null
    },
    {
      "id": "93217",
      "postDate": "09/22/2015 22:15:04",
      "content": "<p>Even if your algorithm passes the KS test locally, it may fail to pass the KS test when you submit your result to Kaggle. Local and Kaggle agreement checks base on different data set.</p>",
      "rawMarkdown": "Even if your algorithm passes the KS test locally, it may fail to pass the KS test when you submit your result to Kaggle. Local and Kaggle agreement checks base on different data set.",
      "votes": null
    },
    {
      "id": "93218",
      "postDate": "09/22/2015 22:21:56",
      "content": "<p>So what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?</p>",
      "rawMarkdown": "So what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?",
      "votes": null
    },
    {
      "id": "93233",
      "postDate": "09/23/2015 00:43:40",
      "content": "<p>[quote=Anton Wiranata;93218]</p>\n\n<p>So what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?</p>\n\n<p>[/quote]\nFailure from KS test mostly comes from 'over-power' classifier which can pickup complicated rules from data. Collision data can have some noise, so if a classifier is too sensitive to the data, it might 'overfit' to the data and use 'noise' as features for classification. </p>",
      "rawMarkdown": "[quote=Anton Wiranata;93218]\r\n\r\nSo what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?\r\n\r\n[/quote]\r\nFailure from KS test mostly comes from 'over-power' classifier which can pickup complicated rules from data. Collision data can have some noise, so if a classifier is too sensitive to the data, it might 'overfit' to the data and use 'noise' as features for classification.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 93217,
      "author_name": "liuchengwang",
      "author_url": "",
      "post_date": "09/22/2015 22:15:04",
      "content": "<p>Even if your algorithm passes the KS test locally, it may fail to pass the KS test when you submit your result to Kaggle. Local and Kaggle agreement checks base on different data set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 93218,
      "author_name": "awipalembang",
      "author_url": "",
      "post_date": "09/22/2015 22:21:56",
      "content": "<p>So what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 93233,
      "author_name": "phunter",
      "author_url": "",
      "post_date": "09/23/2015 00:43:40",
      "content": "<p>[quote=Anton Wiranata;93218]</p>\n\n<p>So what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?</p>\n\n<p>[/quote]\nFailure from KS test mostly comes from 'over-power' classifier which can pickup complicated rules from data. Collision data can have some noise, so if a classifier is too sensitive to the data, it might 'overfit' to the data and use 'noise' as features for classification. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "93182": "Hi All, \r\n\r\nI submitted a couple predictions, but I got error messages saying that my agreement check is higher than 0.09. I did ks.test and I got below 0.09. Do you know what happened?",
    "93217": "Even if your algorithm passes the KS test locally, it may fail to pass the KS test when you submit your result to Kaggle. Local and Kaggle agreement checks base on different data set.",
    "93218": "So what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?",
    "93233": "[quote=Anton Wiranata;93218]\r\n\r\nSo what to do to deal with this problems? Does it mean that the classifier is wrong or there is a target leak?\r\n\r\n[/quote]\r\nFailure from KS test mostly comes from 'over-power' classifier which can pickup complicated rules from data. Collision data can have some noise, so if a classifier is too sensitive to the data, it might 'overfit' to the data and use 'noise' as features for classification."
  },
  "source": "meta"
}