{
  "id": 5699,
  "title": "ROC, AUC and submission format",
  "url": "/competitions/accelerometer-biometric-competition/discussion/5699",
  "author_name": "",
  "post_date": "2013-09-09T02:33:47.710Z",
  "votes": null,
  "comment_count": 3,
  "views": 1560,
  "content": "<p>Hello,</p>\n<p>I'm a newbie and I'm having trouble understanding the submission format. Here's how I've proceeded so far:</p>\n<p>- imported train and test data</p>\n<p>- extracted some features</p>\n<p>- trained a classifier in scikit-learn</p>\n<p>- used the classifier to predict the device associated with each test sequence</p>\n<p>- translated this in a binary answer if my prediction matches the questions file or not</p>\n<p>I am afraid the last step is wrong, but I'm not entirely sure how to go from the trained classifier to an answer that is not just 0-1 but is a probability of truth.</p>\n<p>Can anyone explain?</p>\n<p>&nbsp;</p>\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "30514",
      "postDate": "09/09/2013 02:33:47",
      "content": "<p>Hello,</p>\n<p>I'm a newbie and I'm having trouble understanding the submission format. Here's how I've proceeded so far:</p>\n<p>- imported train and test data</p>\n<p>- extracted some features</p>\n<p>- trained a classifier in scikit-learn</p>\n<p>- used the classifier to predict the device associated with each test sequence</p>\n<p>- translated this in a binary answer if my prediction matches the questions file or not</p>\n<p>I am afraid the last step is wrong, but I'm not entirely sure how to go from the trained classifier to an answer that is not just 0-1 but is a probability of truth.</p>\n<p>Can anyone explain?</p>\n<p>&nbsp;</p>\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30516",
      "postDate": "09/09/2013 04:06:51",
      "content": "<p>many of sklearn's classifiers have a method named&nbsp;predict_proba, that returns the probability of each class (in your case for true and false, so two values that will sum to one)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30521",
      "postDate": "09/09/2013 07:02:25",
      "content": "<p>Thanks!</p>\n<p>Let me understand this thoroughly: currently my classifier does not spit out a True or False prediction, but a label prediction. So, if I use predict_proba on the classifier, I end up with probabilities for 387 different classes, not just 2.</p>\n<p>How do I train the classifier so that it just gives a true or false prediction?</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30522",
      "postDate": "09/09/2013 07:19:12",
      "content": "<p>one thing you can do is to take the probability computed for the class (device id) proposed in the question file.</p>\n<p>You can also train your classifier to predict whether the proposed device id is the actual device id (which will change your accuracy, maybe for the worse, maybe for the better), and than you'll only have two classes.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 30516,
      "author_name": "roulinir",
      "author_url": "",
      "post_date": "09/09/2013 04:06:51",
      "content": "<p>many of sklearn's classifiers have a method named&nbsp;predict_proba, that returns the probability of each class (in your case for true and false, so two values that will sum to one)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30521,
      "author_name": "ghegoo",
      "author_url": "",
      "post_date": "09/09/2013 07:02:25",
      "content": "<p>Thanks!</p>\n<p>Let me understand this thoroughly: currently my classifier does not spit out a True or False prediction, but a label prediction. So, if I use predict_proba on the classifier, I end up with probabilities for 387 different classes, not just 2.</p>\n<p>How do I train the classifier so that it just gives a true or false prediction?</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30522,
      "author_name": "roulinir",
      "author_url": "",
      "post_date": "09/09/2013 07:19:12",
      "content": "<p>one thing you can do is to take the probability computed for the class (device id) proposed in the question file.</p>\n<p>You can also train your classifier to predict whether the proposed device id is the actual device id (which will change your accuracy, maybe for the worse, maybe for the better), and than you'll only have two classes.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "30514": "",
    "30516": "",
    "30521": "",
    "30522": ""
  },
  "source": "meta"
}