{
  "id": 15557,
  "title": "Why is the ROC only calculated on the min_ANNmuon > 0.4 data?",
  "url": "/competitions/flavours-of-physics/discussion/15557",
  "author_name": "",
  "post_date": "2015-07-26T16:56:41.470Z",
  "votes": 3,
  "comment_count": 4,
  "views": 625,
  "content": "<p>I don't think it's explained anywhere, but why is the ROC/AUC score only computed on the data where min_ANNmuon &gt; 0.4? See here for an example: <a href=\"https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/flatness_boosting.ipynb\">https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/flatness_boosting.ipynb</a></p>\n\n<p>Furthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the <em>training</em> data... surely this will give high scores to models that overfit?</p>",
  "messages": [
    {
      "id": "87052",
      "postDate": "07/26/2015 16:56:41",
      "content": "<p>I don't think it's explained anywhere, but why is the ROC/AUC score only computed on the data where min_ANNmuon &gt; 0.4? See here for an example: <a href=\"https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/flatness_boosting.ipynb\">https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/flatness_boosting.ipynb</a></p>\n\n<p>Furthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the <em>training</em> data... surely this will give high scores to models that overfit?</p>",
      "rawMarkdown": "I don't think it's explained anywhere, but why is the ROC/AUC score only computed on the data where min_ANNmuon > 0.4? See here for an example: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/flatness_boosting.ipynb\r\n\r\nFurthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the *training* data... surely this will give high scores to models that overfit?",
      "votes": null
    },
    {
      "id": "87053",
      "postDate": "07/26/2015 17:03:48",
      "content": "<p>Hey,</p>\n\n<p>Let me answer the physics part of the question. In our data sample you will have some peaking backgrounds like D_(s)-&gt; 3 hadrons. If all 3 hadrons get miss identified as muons you will have a something that looks almost like signal in your background sample. The cut min_ANNmuon &gt; 0.4 removes this peaking background. We described this in Section 2.2 in our documentation.</p>\n\n<p>Cheers,\nMarcin</p>",
      "rawMarkdown": "Hey,\r\n\r\nLet me answer the physics part of the question. In our data sample you will have some peaking backgrounds like D_(s)-> 3 hadrons. If all 3 hadrons get miss identified as muons you will have a something that looks almost like signal in your background sample. The cut min_ANNmuon > 0.4 removes this peaking background. We described this in Section 2.2 in our documentation.\r\n\r\nCheers,\r\nMarcin",
      "votes": null
    },
    {
      "id": "87055",
      "postDate": "07/26/2015 17:16:44",
      "content": "<p>Cool - thanks.</p>",
      "rawMarkdown": "Cool - thanks.",
      "votes": null
    },
    {
      "id": "87077",
      "postDate": "07/26/2015 20:50:39",
      "content": "<p>Hi, Erlend!</p>\n\n<blockquote>\n  <p>Furthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the training data... surely this will give high scores to models that overfit?</p>\n</blockquote>\n\n<p>Example demonstrates usage of all metric functions (together with usage of some classifier).\nSurely, we expect participants to use any way of cross-validation. (should this be explained on kaggle?)</p>",
      "rawMarkdown": "Hi, Erlend!\r\n\r\n>Furthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the training data... surely this will give high scores to models that overfit?\r\n\r\nExample demonstrates usage of all metric functions (together with usage of some classifier).\r\nSurely, we expect participants to use any way of cross-validation. (should this be explained on kaggle?)",
      "votes": null
    },
    {
      "id": "87107",
      "postDate": "07/27/2015 00:07:36",
      "content": "<p>HI Tatiana,</p>\n\n<p>I had assumed that the example was just a demonstration of the usage of the provided metric functions, but I wanted to check. Thanks for clarifying. It wouldn't harm to explain it, but it's not strictly necessary.</p>",
      "rawMarkdown": "HI Tatiana,\r\n\r\nI had assumed that the example was just a demonstration of the usage of the provided metric functions, but I wanted to check. Thanks for clarifying. It wouldn't harm to explain it, but it's not strictly necessary.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 87053,
      "author_name": "albercik1905",
      "author_url": "",
      "post_date": "07/26/2015 17:03:48",
      "content": "<p>Hey,</p>\n\n<p>Let me answer the physics part of the question. In our data sample you will have some peaking backgrounds like D_(s)-&gt; 3 hadrons. If all 3 hadrons get miss identified as muons you will have a something that looks almost like signal in your background sample. The cut min_ANNmuon &gt; 0.4 removes this peaking background. We described this in Section 2.2 in our documentation.</p>\n\n<p>Cheers,\nMarcin</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 87055,
      "author_name": "erlend",
      "author_url": "",
      "post_date": "07/26/2015 17:16:44",
      "content": "<p>Cool - thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 87077,
      "author_name": "tataantares",
      "author_url": "",
      "post_date": "07/26/2015 20:50:39",
      "content": "<p>Hi, Erlend!</p>\n\n<blockquote>\n  <p>Furthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the training data... surely this will give high scores to models that overfit?</p>\n</blockquote>\n\n<p>Example demonstrates usage of all metric functions (together with usage of some classifier).\nSurely, we expect participants to use any way of cross-validation. (should this be explained on kaggle?)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 87107,
      "author_name": "erlend",
      "author_url": "",
      "post_date": "07/27/2015 00:07:36",
      "content": "<p>HI Tatiana,</p>\n\n<p>I had assumed that the example was just a demonstration of the usage of the provided metric functions, but I wanted to check. Thanks for clarifying. It wouldn't harm to explain it, but it's not strictly necessary.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "87052": "I don't think it's explained anywhere, but why is the ROC/AUC score only computed on the data where min_ANNmuon > 0.4? See here for an example: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/flatness_boosting.ipynb\r\n\r\nFurthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the *training* data... surely this will give high scores to models that overfit?",
    "87053": "Hey,\r\n\r\nLet me answer the physics part of the question. In our data sample you will have some peaking backgrounds like D_(s)-> 3 hadrons. If all 3 hadrons get miss identified as muons you will have a something that looks almost like signal in your background sample. The cut min_ANNmuon > 0.4 removes this peaking background. We described this in Section 2.2 in our documentation.\r\n\r\nCheers,\r\nMarcin",
    "87055": "Cool - thanks.",
    "87077": "Hi, Erlend!\r\n\r\n>Furthermore why, in the example scripts (starter-kit), is the ROC/AUC calculated on the training data... surely this will give high scores to models that overfit?\r\n\r\nExample demonstrates usage of all metric functions (together with usage of some classifier).\r\nSurely, we expect participants to use any way of cross-validation. (should this be explained on kaggle?)",
    "87107": "HI Tatiana,\r\n\r\nI had assumed that the example was just a demonstration of the usage of the provided metric functions, but I wanted to check. Thanks for clarifying. It wouldn't harm to explain it, but it's not strictly necessary."
  },
  "source": "meta"
}