{
  "id": 14460,
  "title": "Clarification on AUC",
  "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14460",
  "author_name": "",
  "post_date": "2015-05-31T09:51:33.607Z",
  "votes": 1,
  "comment_count": 2,
  "views": 769,
  "content": "<p>Hello everyone,</p>\n<p>I am confused by the use of AUC to assess the accuracy of the prediction.</p>\n<p>I take the AUC to be the area under the ROC curve. This curve is a plot of the relationship between <span style=\"text-decoration: underline\">True Positive Rate (y-axis) and False Positive Rate (x-axis) when a parameter of the classifier (e.g. the threshold) is being sweeped over its entire range</span>.</p>\n<p>If anything, a submission corresponds to one point in the TPR-FPR plane, not to a curve. The curve would arise as multiple sets of predictions are considered, each set&nbsp; with its True Positive rate and False Positive rate.<br>So, what do we mean by AUC in this context?</p>\n<p>Is it the area of the trapezoid with just one (FPR,TPR) point?</p>\n<p>Also, it appears that it is possible (and beneficial, from what I read in some of the posts) to submit a probability rather than a hard classification. What meaning does the AUC have in this context? How are TPR and FPR computed?</p>\n<p>It seems an unusual choice of loss function, at least in my experience, compared to more conventional choices for this sort of problems, such as square-loss, log-loss, etc.</p>\n<p>Thank you</p>\n<p>Paolo</p>",
  "messages": [
    {
      "id": "80514",
      "postDate": "05/31/2015 09:51:33",
      "content": "<p>Hello everyone,</p>\n<p>I am confused by the use of AUC to assess the accuracy of the prediction.</p>\n<p>I take the AUC to be the area under the ROC curve. This curve is a plot of the relationship between <span style=\"text-decoration: underline\">True Positive Rate (y-axis) and False Positive Rate (x-axis) when a parameter of the classifier (e.g. the threshold) is being sweeped over its entire range</span>.</p>\n<p>If anything, a submission corresponds to one point in the TPR-FPR plane, not to a curve. The curve would arise as multiple sets of predictions are considered, each set&nbsp; with its True Positive rate and False Positive rate.<br>So, what do we mean by AUC in this context?</p>\n<p>Is it the area of the trapezoid with just one (FPR,TPR) point?</p>\n<p>Also, it appears that it is possible (and beneficial, from what I read in some of the posts) to submit a probability rather than a hard classification. What meaning does the AUC have in this context? How are TPR and FPR computed?</p>\n<p>It seems an unusual choice of loss function, at least in my experience, compared to more conventional choices for this sort of problems, such as square-loss, log-loss, etc.</p>\n<p>Thank you</p>\n<p>Paolo</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80525",
      "postDate": "05/31/2015 14:43:49",
      "content": "<p>Paolo, here the threshold t is a real number such that for predicted probability p &gt;= t you will call it a positive, and for p &lt; t a negative. Hence (for a fixed submission) every choice of threshold determines a tpr / fpr. The curve arises as the threshold varies (i.e., the curve is parameterized by the threshold). </p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80533",
      "postDate": "05/31/2015 17:10:34",
      "content": "<p>Thanks, I get it now.</p>\n\n<p>Paolo</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 80525,
      "author_name": "cdjbee",
      "author_url": "",
      "post_date": "05/31/2015 14:43:49",
      "content": "<p>Paolo, here the threshold t is a real number such that for predicted probability p &gt;= t you will call it a positive, and for p &lt; t a negative. Hence (for a fixed submission) every choice of threshold determines a tpr / fpr. The curve arises as the threshold varies (i.e., the curve is parameterized by the threshold). </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80533,
      "author_name": "astolfo",
      "author_url": "",
      "post_date": "05/31/2015 17:10:34",
      "content": "<p>Thanks, I get it now.</p>\n\n<p>Paolo</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "80514": "",
    "80525": "",
    "80533": ""
  },
  "source": "meta"
}