{
  "id": 16011,
  "title": "One more question about scoring.",
  "url": "/competitions/grasp-and-lift-eeg-detection/discussion/16011",
  "author_name": "",
  "post_date": "2015-08-18T21:52:29.623Z",
  "votes": null,
  "comment_count": 2,
  "views": 428,
  "content": "<p>I just joined the competition. I looked through the the forum to find a solution, but I do still have a question.</p>\n\n<p>In the train set we have 12 people(4 men, 8 women) that do lifting 10 times each.\nduring this process 7 events can happen:</p>\n\n<ol>\n<li>Handstart </li>\n<li>FirstDigitTouch</li>\n<li>BothStartLoadPhase</li>\n<li>LiftOff</li>\n<li>Replace</li>\n<li>BothReleased</li>\n<li>Something in else</li>\n</ol>\n\n<p>&quot;The six label columns are either zero or one, depending on whether the corresponding event has occurred within &#177;150ms (&#177;75frames).&quot;</p>\n\n<p>we need to predict probability of one of the events in 1-6 happening and final score - mean of the ROC of the 1-6 labels.</p>\n\n<p>Looking at the train set I can see that all columns in some row set can be 0, or two cells in this row can have value 1.</p>\n\n<p>It is said that &quot;Since the columns span multiple subjects and series, you should submit calibrated probabilities that fall on the same scale.&quot; </p>\n\n<p>And this last part I do not understand. What exactly should be normalized? </p>",
  "messages": [
    {
      "id": "89726",
      "postDate": "08/18/2015 21:52:29",
      "content": "<p>I just joined the competition. I looked through the the forum to find a solution, but I do still have a question.</p>\n\n<p>In the train set we have 12 people(4 men, 8 women) that do lifting 10 times each.\nduring this process 7 events can happen:</p>\n\n<ol>\n<li>Handstart </li>\n<li>FirstDigitTouch</li>\n<li>BothStartLoadPhase</li>\n<li>LiftOff</li>\n<li>Replace</li>\n<li>BothReleased</li>\n<li>Something in else</li>\n</ol>\n\n<p>&quot;The six label columns are either zero or one, depending on whether the corresponding event has occurred within &#177;150ms (&#177;75frames).&quot;</p>\n\n<p>we need to predict probability of one of the events in 1-6 happening and final score - mean of the ROC of the 1-6 labels.</p>\n\n<p>Looking at the train set I can see that all columns in some row set can be 0, or two cells in this row can have value 1.</p>\n\n<p>It is said that &quot;Since the columns span multiple subjects and series, you should submit calibrated probabilities that fall on the same scale.&quot; </p>\n\n<p>And this last part I do not understand. What exactly should be normalized? </p>",
      "rawMarkdown": "I just joined the competition. I looked through the the forum to find a solution, but I do still have a question.\r\n\r\nIn the train set we have 12 people(4 men, 8 women) that do lifting 10 times each.\r\nduring this process 7 events can happen:\r\n\r\n 1. Handstart \r\n 2. FirstDigitTouch\r\n 3. BothStartLoadPhase\r\n 4. LiftOff\r\n 5. Replace\r\n 6. BothReleased\r\n 7. Something in else\r\n\r\n\"The six label columns are either zero or one, depending on whether the corresponding event has occurred within ±150ms (±75frames).\"\r\n\r\nwe need to predict probability of one of the events in 1-6 happening and final score - mean of the ROC of the 1-6 labels.\r\n\r\nLooking at the train set I can see that all columns in some row set can be 0, or two cells in this row can have value 1.\r\n\r\nIt is said that \"Since the columns span multiple subjects and series, you should submit calibrated probabilities that fall on the same scale.\" \r\n\r\nAnd this last part I do not understand. What exactly should be normalized?",
      "votes": null
    },
    {
      "id": "89751",
      "postDate": "08/19/2015 05:14:35",
      "content": "<p>@Vladimir, that means different models could have different scales for probabilites depending on which target you are predicting and which subject you are dealing with. So, its beneficial to put probabilites on the same scale.</p>",
      "rawMarkdown": "Vladimir, that means different models could have different scales for probabilites depending on which target you are predicting and which subject you are dealing with. So, its beneficial to put probabilites on the same scale.",
      "votes": null
    },
    {
      "id": "89762",
      "postDate": "08/19/2015 08:04:05",
      "content": "<p>Yes this is a comment about the AUC scoring. Because for calculating an ROC the scoring function implicitly tries all (relevant) thresholds to decide between the two classes. So usually the scaling doesn't matter. But since you likely need to train several models for the different subjects and they are all evaluated at once, they have to operate on the same scale. </p>",
      "rawMarkdown": "Yes this is a comment about the AUC scoring. Because for calculating an ROC the scoring function implicitly tries all (relevant) thresholds to decide between the two classes. So usually the scaling doesn't matter. But since you likely need to train several models for the different subjects and they are all evaluated at once, they have to operate on the same scale.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 89751,
      "author_name": "deepcnn",
      "author_url": "",
      "post_date": "08/19/2015 05:14:35",
      "content": "<p>@Vladimir, that means different models could have different scales for probabilites depending on which target you are predicting and which subject you are dealing with. So, its beneficial to put probabilites on the same scale.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89762,
      "author_name": "qwlouse",
      "author_url": "",
      "post_date": "08/19/2015 08:04:05",
      "content": "<p>Yes this is a comment about the AUC scoring. Because for calculating an ROC the scoring function implicitly tries all (relevant) thresholds to decide between the two classes. So usually the scaling doesn't matter. But since you likely need to train several models for the different subjects and they are all evaluated at once, they have to operate on the same scale. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "89726": "I just joined the competition. I looked through the the forum to find a solution, but I do still have a question.\r\n\r\nIn the train set we have 12 people(4 men, 8 women) that do lifting 10 times each.\r\nduring this process 7 events can happen:\r\n\r\n 1. Handstart \r\n 2. FirstDigitTouch\r\n 3. BothStartLoadPhase\r\n 4. LiftOff\r\n 5. Replace\r\n 6. BothReleased\r\n 7. Something in else\r\n\r\n\"The six label columns are either zero or one, depending on whether the corresponding event has occurred within ±150ms (±75frames).\"\r\n\r\nwe need to predict probability of one of the events in 1-6 happening and final score - mean of the ROC of the 1-6 labels.\r\n\r\nLooking at the train set I can see that all columns in some row set can be 0, or two cells in this row can have value 1.\r\n\r\nIt is said that \"Since the columns span multiple subjects and series, you should submit calibrated probabilities that fall on the same scale.\" \r\n\r\nAnd this last part I do not understand. What exactly should be normalized?",
    "89751": "Vladimir, that means different models could have different scales for probabilites depending on which target you are predicting and which subject you are dealing with. So, its beneficial to put probabilites on the same scale.",
    "89762": "Yes this is a comment about the AUC scoring. Because for calculating an ROC the scoring function implicitly tries all (relevant) thresholds to decide between the two classes. So usually the scaling doesn't matter. But since you likely need to train several models for the different subjects and they are all evaluated at once, they have to operate on the same scale."
  },
  "source": "meta"
}