{
  "id": 15051,
  "title": "Clarifications On Rules and Scoring Needed",
  "url": "/competitions/grasp-and-lift-eeg-detection/discussion/15051",
  "author_name": "",
  "post_date": "2015-07-05T00:42:35.757Z",
  "votes": null,
  "comment_count": 4,
  "views": 712,
  "content": "<p>Greetings -</p>\n<p>I have several questions.</p>\n<p>1) The instructions say that all of the data from the other series can be used to predict results from a given series. Is this true even if the series is from the same subject?</p>\n<p>2) Can we use subject-specific models on the test data?</p>\n<p>3) How, exactly, is area under the ROC curve calculated? Specifically, what threshold values are used?</p>\n<p>Thanks.</p>\n<p>- Ken D</p>",
  "messages": [
    {
      "id": "83450",
      "postDate": "07/05/2015 00:42:35",
      "content": "<p>Greetings -</p>\n<p>I have several questions.</p>\n<p>1) The instructions say that all of the data from the other series can be used to predict results from a given series. Is this true even if the series is from the same subject?</p>\n<p>2) Can we use subject-specific models on the test data?</p>\n<p>3) How, exactly, is area under the ROC curve calculated? Specifically, what threshold values are used?</p>\n<p>Thanks.</p>\n<p>- Ken D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "83451",
      "postDate": "07/05/2015 01:13:03",
      "content": "<p>Ken,</p>\n<p>1. Yes.&nbsp; See this <a href=\"https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/14935/use-of-future-data\">thread </a>for an official statement from the competition admin.</p>\n<p>2. Yes. See previous link. For an example see this <a href=\"https://www.kaggle.com/alexandrebarachant/grasp-and-lift-eeg-detection/beat-the-benchmark-0-67\">benchmark.</a></p>\n<p>3. Here is a link to the Kaggle wiki on <a href=\"https://www.kaggle.com/wiki/AUC\">AUC</a>. There are some videos and further resources at the bottom. I also attached an often posted paper on the topic as well. </p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "83479",
      "postDate": "07/05/2015 17:53:47",
      "content": "<p>Devin -</p>\n<p>Thank you for clarifying items (1) and (2), and for the references.</p>\n<p>Let me restate item (3). I know about ROC curves. (I will, however, read the paper as it looks worthwhile.) I was asking for a detailed description of how Kaggle will calculate AUC.</p>\n<p>- Ken D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "83480",
      "postDate": "07/05/2015 18:02:47",
      "content": "<p>Ken,</p>\n<p>My understanding is that all relevant thresholds are used, this depends on your predictions. Because you asked which thresholds were used I assumed there may have been a misunderstanding about how AUC is calculated. There is pseudocode for one implementation of AUC calculation in the paper I attached and the wiki page discusses the topic. If you want specific algorithmic details of how Kaggle computes AUC you will have to wait for an admin to respond. I use the&nbsp;<a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html\">sklearn implementation</a> for local validation. </p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "83512",
      "postDate": "07/06/2015 13:11:19",
      "content": "<p>We use the trapezoidal method of AUC calculation. It matches the popular ML metrics packages and does not depend on picking threshold values.</p>",
      "rawMarkdown": "We use the trapezoidal method of AUC calculation. It matches the popular ML metrics packages and does not depend on picking threshold values.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 83451,
      "author_name": "devinanzelmo",
      "author_url": "",
      "post_date": "07/05/2015 01:13:03",
      "content": "<p>Ken,</p>\n<p>1. Yes.&nbsp; See this <a href=\"https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/14935/use-of-future-data\">thread </a>for an official statement from the competition admin.</p>\n<p>2. Yes. See previous link. For an example see this <a href=\"https://www.kaggle.com/alexandrebarachant/grasp-and-lift-eeg-detection/beat-the-benchmark-0-67\">benchmark.</a></p>\n<p>3. Here is a link to the Kaggle wiki on <a href=\"https://www.kaggle.com/wiki/AUC\">AUC</a>. There are some videos and further resources at the bottom. I also attached an often posted paper on the topic as well. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 83479,
      "author_name": "kdoniger",
      "author_url": "",
      "post_date": "07/05/2015 17:53:47",
      "content": "<p>Devin -</p>\n<p>Thank you for clarifying items (1) and (2), and for the references.</p>\n<p>Let me restate item (3). I know about ROC curves. (I will, however, read the paper as it looks worthwhile.) I was asking for a detailed description of how Kaggle will calculate AUC.</p>\n<p>- Ken D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 83480,
      "author_name": "devinanzelmo",
      "author_url": "",
      "post_date": "07/05/2015 18:02:47",
      "content": "<p>Ken,</p>\n<p>My understanding is that all relevant thresholds are used, this depends on your predictions. Because you asked which thresholds were used I assumed there may have been a misunderstanding about how AUC is calculated. There is pseudocode for one implementation of AUC calculation in the paper I attached and the wiki page discusses the topic. If you want specific algorithmic details of how Kaggle computes AUC you will have to wait for an admin to respond. I use the&nbsp;<a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html\">sklearn implementation</a> for local validation. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 83512,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "07/06/2015 13:11:19",
      "content": "<p>We use the trapezoidal method of AUC calculation. It matches the popular ML metrics packages and does not depend on picking threshold values.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "83450": "",
    "83451": "",
    "83479": "",
    "83480": "",
    "83512": "We use the trapezoidal method of AUC calculation. It matches the popular ML metrics packages and does not depend on picking threshold values."
  },
  "source": "meta"
}