{
  "id": 331301,
  "title": "Metric on Judging AUC/Score",
  "url": "/competitions/energy-anomaly-detection/discussion/331301",
  "author_name": "",
  "post_date": "2022-06-16T14:05:54.455297300Z",
  "votes": 3,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Hi,</p>\n<p>We are wondering if it is possible to know which metric is being used to infer the scores on the public test set? Is it AUC or Accuracy? We've seen it says AUC, but we are a bit confused because it is not possible to obtain an AUC/ROC from binary labels (0 and 1), which are the ones that we submit.<br>\nAlso, is this metric going to be the same for judging the final submission?</p>\n<p>Thank you for your help!<br>\nBest,</p>\n<p>Paola</p>",
  "messages": [
    {
      "id": "1822608",
      "postDate": "06/16/2022 14:05:54",
      "content": "<p>Hi,</p>\n<p>We are wondering if it is possible to know which metric is being used to infer the scores on the public test set? Is it AUC or Accuracy? We've seen it says AUC, but we are a bit confused because it is not possible to obtain an AUC/ROC from binary labels (0 and 1), which are the ones that we submit.<br>\nAlso, is this metric going to be the same for judging the final submission?</p>\n<p>Thank you for your help!<br>\nBest,</p>\n<p>Paola</p>",
      "rawMarkdown": "Hi,\n\nWe are wondering if it is possible to know which metric is being used to infer the scores on the public test set? Is it AUC or Accuracy? We've seen it says AUC, but we are a bit confused because it is not possible to obtain an AUC/ROC from binary labels (0 and 1), which are the ones that we submit.\nAlso, is this metric going to be the same for judging the final submission?\n\nThank you for your help!\nBest,\n\nPaola",
      "votes": null
    },
    {
      "id": "1823209",
      "postDate": "06/17/2022 06:54:24",
      "content": "<p>Hello, Paola.</p>\n<p>That was my confusion a few days ago&nbsp;as well, until I realized that ROC AUC could be calculated using classification probability.</p>\n<p>Although our goal is to provide binary predictions of faults, most classification models can also&nbsp;output classification probability.</p>\n<p>Because the probability threshold could be changed, the ROC AUC score could be therefore computed.</p>",
      "rawMarkdown": "Hello, Paola.\n\nThat was my confusion a few days ago as well, until I realized that ROC AUC could be calculated using classification probability.\n\nAlthough our goal is to provide binary predictions of faults, most classification models can also output classification probability.\n\nBecause the probability threshold could be changed, the ROC AUC score could be therefore computed.",
      "votes": null
    },
    {
      "id": "1840780",
      "postDate": "07/02/2022 14:49:43",
      "content": "<p><a href=\"https://www.kaggle.com/patrick0302\" target=\"_blank\">@patrick0302</a> Though in this case, we're only submitting the classes themselves, yes? We're not submitting the probabilities of the classes. How will the ROC AUC be computed here?</p>",
      "rawMarkdown": "patrick0302 Though in this case, we're only submitting the classes themselves, yes? We're not submitting the probabilities of the classes. How will the ROC AUC be computed here?",
      "votes": null
    },
    {
      "id": "1841305",
      "postDate": "07/03/2022 01:24:30",
      "content": "<p>Hi Rahul Sangole,<br>\nNo, I submit predicted probablilities from classification model.<br>\nAlthough you could submit predicted class results and get an AUC based on them, in most cases the AUC for binary classes would be worse than the one for probabilities.</p>",
      "rawMarkdown": "Hi Rahul Sangole,\nNo, I submit predicted probablilities from classification model.\nAlthough you could submit predicted class results and get an AUC based on them, in most cases the AUC for binary classes would be worse than the one for probabilities.",
      "votes": null
    },
    {
      "id": "1841947",
      "postDate": "07/03/2022 15:08:32",
      "content": "<p>Interesting. The sample data response states 0 or 1. How’d you figure out probabilities will work?</p>",
      "rawMarkdown": "Interesting. The sample data response states 0 or 1. How’d you figure out probabilities will work?",
      "votes": null
    },
    {
      "id": "1841970",
      "postDate": "07/03/2022 15:32:13",
      "content": "<p>I realized it after reading one of the shared notebooks and googled it : D</p>",
      "rawMarkdown": "I realized it after reading one of the shared notebooks and googled it : D",
      "votes": null
    },
    {
      "id": "1842140",
      "postDate": "07/03/2022 18:28:15",
      "content": "<p>I wonder if my score would improve any for the same model, but submitted as prediction scores. </p>",
      "rawMarkdown": "I wonder if my score would improve any for the same model, but submitted as prediction scores.",
      "votes": null
    },
    {
      "id": "1842395",
      "postDate": "07/04/2022 00:54:25",
      "content": "<p>Try it and see how it goes : )</p>",
      "rawMarkdown": "Try it and see how it goes : )",
      "votes": null
    },
    {
      "id": "1844204",
      "postDate": "07/05/2022 12:16:20",
      "content": "<p>Hah, score moved from 0.75 -&gt; 0.85 for the same model.</p>",
      "rawMarkdown": "Hah, score moved from 0.75 -> 0.85 for the same model.",
      "votes": null
    },
    {
      "id": "1844247",
      "postDate": "07/05/2022 12:40:12",
      "content": "<p>0.57 -&gt; 0.92</p>",
      "rawMarkdown": "0.57 -> 0.92",
      "votes": null
    },
    {
      "id": "1845076",
      "postDate": "07/06/2022 03:40:25",
      "content": "<p>Congrat!!! That's a nice improvement👍</p>",
      "rawMarkdown": "Congrat!!! That's a nice improvement👍",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1823209,
      "author_name": "patrick0302",
      "author_url": "",
      "post_date": "06/17/2022 06:54:24",
      "content": "<p>Hello, Paola.</p>\n<p>That was my confusion a few days ago&nbsp;as well, until I realized that ROC AUC could be calculated using classification probability.</p>\n<p>Although our goal is to provide binary predictions of faults, most classification models can also&nbsp;output classification probability.</p>\n<p>Because the probability threshold could be changed, the ROC AUC score could be therefore computed.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1840780,
      "author_name": "rsangole",
      "author_url": "",
      "post_date": "07/02/2022 14:49:43",
      "content": "<p><a href=\"https://www.kaggle.com/patrick0302\" target=\"_blank\">@patrick0302</a> Though in this case, we're only submitting the classes themselves, yes? We're not submitting the probabilities of the classes. How will the ROC AUC be computed here?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1841305,
          "author_name": "patrick0302",
          "author_url": "",
          "post_date": "07/03/2022 01:24:30",
          "content": "<p>Hi Rahul Sangole,<br>\nNo, I submit predicted probablilities from classification model.<br>\nAlthough you could submit predicted class results and get an AUC based on them, in most cases the AUC for binary classes would be worse than the one for probabilities.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1841947,
          "author_name": "rsangole",
          "author_url": "",
          "post_date": "07/03/2022 15:08:32",
          "content": "<p>Interesting. The sample data response states 0 or 1. How’d you figure out probabilities will work?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1841970,
          "author_name": "patrick0302",
          "author_url": "",
          "post_date": "07/03/2022 15:32:13",
          "content": "<p>I realized it after reading one of the shared notebooks and googled it : D</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1842140,
          "author_name": "rsangole",
          "author_url": "",
          "post_date": "07/03/2022 18:28:15",
          "content": "<p>I wonder if my score would improve any for the same model, but submitted as prediction scores. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1842395,
          "author_name": "patrick0302",
          "author_url": "",
          "post_date": "07/04/2022 00:54:25",
          "content": "<p>Try it and see how it goes : )</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1844204,
          "author_name": "rsangole",
          "author_url": "",
          "post_date": "07/05/2022 12:16:20",
          "content": "<p>Hah, score moved from 0.75 -&gt; 0.85 for the same model.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1844247,
          "author_name": "rsangole",
          "author_url": "",
          "post_date": "07/05/2022 12:40:12",
          "content": "<p>0.57 -&gt; 0.92</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1845076,
          "author_name": "patrick0302",
          "author_url": "",
          "post_date": "07/06/2022 03:40:25",
          "content": "<p>Congrat!!! That's a nice improvement👍</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1822608": "Hi,\n\nWe are wondering if it is possible to know which metric is being used to infer the scores on the public test set? Is it AUC or Accuracy? We've seen it says AUC, but we are a bit confused because it is not possible to obtain an AUC/ROC from binary labels (0 and 1), which are the ones that we submit.\nAlso, is this metric going to be the same for judging the final submission?\n\nThank you for your help!\nBest,\n\nPaola",
    "1823209": "Hello, Paola.\n\nThat was my confusion a few days ago as well, until I realized that ROC AUC could be calculated using classification probability.\n\nAlthough our goal is to provide binary predictions of faults, most classification models can also output classification probability.\n\nBecause the probability threshold could be changed, the ROC AUC score could be therefore computed.",
    "1840780": "patrick0302 Though in this case, we're only submitting the classes themselves, yes? We're not submitting the probabilities of the classes. How will the ROC AUC be computed here?",
    "1841305": "Hi Rahul Sangole,\nNo, I submit predicted probablilities from classification model.\nAlthough you could submit predicted class results and get an AUC based on them, in most cases the AUC for binary classes would be worse than the one for probabilities.",
    "1841947": "Interesting. The sample data response states 0 or 1. How’d you figure out probabilities will work?",
    "1841970": "I realized it after reading one of the shared notebooks and googled it : D",
    "1842140": "I wonder if my score would improve any for the same model, but submitted as prediction scores.",
    "1842395": "Try it and see how it goes : )",
    "1844204": "Hah, score moved from 0.75 -> 0.85 for the same model.",
    "1844247": "0.57 -> 0.92",
    "1845076": "Congrat!!! That's a nice improvement👍"
  },
  "source": "meta"
}