{
  "id": 1233,
  "title": "R function for capped binomial deviance",
  "url": "/competitions/WhatDoYouKnow/discussion/1233",
  "author_name": "",
  "post_date": "2012-01-06T22:21:14.890Z",
  "votes": null,
  "comment_count": 2,
  "views": 2650,
  "content": "<p>I dunno if this has been posted here yet, but it's very useful:</p>\r\n<div>\r\n<p><a href=\"http://www.kaggle.com/c/PhotoQualityPrediction/forums/t/1013/r-function-for-binomial-deviance/6392#post6392\">http://www.kaggle.com/c/PhotoQualityPrediction/forums/t/1013/r-function-for-binomial-deviance/6392#post6392</a></p>\r\n</div>",
  "messages": [
    {
      "id": "7733",
      "postDate": "01/06/2012 22:21:14",
      "content": "<p>I dunno if this has been posted here yet, but it's very useful:</p>\r\n<div>\r\n<p><a href=\"http://www.kaggle.com/c/PhotoQualityPrediction/forums/t/1013/r-function-for-binomial-deviance/6392#post6392\">http://www.kaggle.com/c/PhotoQualityPrediction/forums/t/1013/r-function-for-binomial-deviance/6392#post6392</a></p>\r\n</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8060",
      "postDate": "01/11/2012 17:16:08",
      "content": "<p>Is capped BD the best measure for evaluation? In a real word scenario, I would have used AUC or a confusion matrix instead..any thoughts? Thanks</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8064",
      "postDate": "01/11/2012 18:50:53",
      "content": "<p>A prediction can (should?) be thought of as a probability of student answering correctly on a given question rather than a measure of group membership (i.e. this is not really a classification problem). &nbsp;From that perspective I don't think a either a confusion\r\n matrix or AUC would be an appropriate evaluation method. &nbsp;I could argue that binomial deviance (i.e. log liklihood) isn't the best measure, either. &nbsp;By definition it rewards risk-averse models, which means there will be fewer really bad predictions (a good\r\n thing) but also fewer really good predictions (not such a good thing). &nbsp;In reality, though, binomial deviance seems like as good a method as any with which to measure probabilistic models.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 8060,
      "author_name": "shaz24",
      "author_url": "",
      "post_date": "01/11/2012 17:16:08",
      "content": "<p>Is capped BD the best measure for evaluation? In a real word scenario, I would have used AUC or a confusion matrix instead..any thoughts? Thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8064,
      "author_name": "yetiman",
      "author_url": "",
      "post_date": "01/11/2012 18:50:53",
      "content": "<p>A prediction can (should?) be thought of as a probability of student answering correctly on a given question rather than a measure of group membership (i.e. this is not really a classification problem). &nbsp;From that perspective I don't think a either a confusion\r\n matrix or AUC would be an appropriate evaluation method. &nbsp;I could argue that binomial deviance (i.e. log liklihood) isn't the best measure, either. &nbsp;By definition it rewards risk-averse models, which means there will be fewer really bad predictions (a good\r\n thing) but also fewer really good predictions (not such a good thing). &nbsp;In reality, though, binomial deviance seems like as good a method as any with which to measure probabilistic models.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "7733": "",
    "8060": "",
    "8064": ""
  },
  "source": "meta"
}