{
  "id": 437740,
  "title": "Questions about the metric",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437740",
  "author_name": "JVH",
  "post_date": "2023-09-08T01:20:55.574000",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi team, </p>\n<p>I am slightly confused by the metric in this competition. We are trying to predict the transition events (wakeup, transition) at different time thresholds instead of being asleep vs awake right? </p>\n<p>So for example, say someone fell asleep ('onset') at 9:00PM, with the single threshold at 15minutes, the evaluation metric basically takes the highest 'onset' label during 8:45PM - 9:15PM and assigns that as the prediction for that event.</p>\n<p>If this is the case, during the Assignment phase of the metric, why does it say \"Some ground-truths may not be matched to a prediction and <strong>some predictions may not be matched to a ground-truth. They will still be accounted for in the scoring</strong>\"? I understand the first half, but for the second half, if a prediction does not get mapped with an event, how does it get accounted for scoring? Say there is only one threshold at 15 like in the example above and our only prediction was 0.8 at 10:00PM, would this still be accounted for scoring?</p>\n<p>Thanks in advance and best of luck to everybody!</p>",
  "messages": [
    {
      "id": 2428530,
      "postDate": "2023-09-08T01:20:55.573Z",
      "content": "<p>Hi team, </p>\n<p>I am slightly confused by the metric in this competition. We are trying to predict the transition events (wakeup, transition) at different time thresholds instead of being asleep vs awake right? </p>\n<p>So for example, say someone fell asleep ('onset') at 9:00PM, with the single threshold at 15minutes, the evaluation metric basically takes the highest 'onset' label during 8:45PM - 9:15PM and assigns that as the prediction for that event.</p>\n<p>If this is the case, during the Assignment phase of the metric, why does it say \"Some ground-truths may not be matched to a prediction and <strong>some predictions may not be matched to a ground-truth. They will still be accounted for in the scoring</strong>\"? I understand the first half, but for the second half, if a prediction does not get mapped with an event, how does it get accounted for scoring? Say there is only one threshold at 15 like in the example above and our only prediction was 0.8 at 10:00PM, would this still be accounted for scoring?</p>\n<p>Thanks in advance and best of luck to everybody!</p>",
      "rawMarkdown": "Hi team, \n\nI am slightly confused by the metric in this competition. We are trying to predict the transition events (wakeup, transition) at different time thresholds instead of being asleep vs awake right? \n\nSo for example, say someone fell asleep ('onset') at 9:00PM, with the single threshold at 15minutes, the evaluation metric basically takes the highest 'onset' label during 8:45PM - 9:15PM and assigns that as the prediction for that event.\n\nIf this is the case, during the Assignment phase of the metric, why does it say \"Some ground-truths may not be matched to a prediction and **some predictions may not be matched to a ground-truth. They will still be accounted for in the scoring**\"? I understand the first half, but for the second half, if a prediction does not get mapped with an event, how does it get accounted for scoring? Say there is only one threshold at 15 like in the example above and our only prediction was 0.8 at 10:00PM, would this still be accounted for scoring?\n\nThanks in advance and best of luck to everybody!",
      "votes": 7
    },
    {
      "id": 2429302,
      "postDate": "2023-09-08T14:19:48.320Z",
      "content": "<p>Unmatched ground-truth events are false negatives and unmatched predictions are false positives. They are accounted for in the calculations of precision and recall in the average precision score.</p>",
      "rawMarkdown": "Unmatched ground-truth events are false negatives and unmatched predictions are false positives. They are accounted for in the calculations of precision and recall in the average precision score.",
      "votes": 3
    },
    {
      "id": 2428547,
      "postDate": "2023-09-08T02:05:08.070Z",
      "content": "<p>Ref: <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation\" target=\"_blank\">https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation</a></p>\n<blockquote>\n  <p>Unmatched ground-truths are scored as FN.</p>\n</blockquote>\n<p>Let me know if this answers your question. Not sure if I understood your question properly.</p>",
      "rawMarkdown": "Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation\n\n> Unmatched ground-truths are scored as FN.\n\nLet me know if this answers your question. Not sure if I understood your question properly.",
      "votes": 2,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2429302,
      "author_name": "Ryan Holbrook",
      "author_url": "",
      "post_date": "2023-09-08T14:19:48.320000",
      "content": "<p>Unmatched ground-truth events are false negatives and unmatched predictions are false positives. They are accounted for in the calculations of precision and recall in the average precision score.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 2428547,
      "author_name": "Bhargav Kowshik",
      "author_url": "",
      "post_date": "2023-09-08T02:05:08.070000",
      "content": "<p>Ref: <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation\" target=\"_blank\">https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation</a></p>\n<blockquote>\n  <p>Unmatched ground-truths are scored as FN.</p>\n</blockquote>\n<p>Let me know if this answers your question. Not sure if I understood your question properly.</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2428530": "Hi team, \n\nI am slightly confused by the metric in this competition. We are trying to predict the transition events (wakeup, transition) at different time thresholds instead of being asleep vs awake right? \n\nSo for example, say someone fell asleep ('onset') at 9:00PM, with the single threshold at 15minutes, the evaluation metric basically takes the highest 'onset' label during 8:45PM - 9:15PM and assigns that as the prediction for that event.\n\nIf this is the case, during the Assignment phase of the metric, why does it say \"Some ground-truths may not be matched to a prediction and **some predictions may not be matched to a ground-truth. They will still be accounted for in the scoring**\"? I understand the first half, but for the second half, if a prediction does not get mapped with an event, how does it get accounted for scoring? Say there is only one threshold at 15 like in the example above and our only prediction was 0.8 at 10:00PM, would this still be accounted for scoring?\n\nThanks in advance and best of luck to everybody!",
    "2429302": "Unmatched ground-truth events are false negatives and unmatched predictions are false positives. They are accounted for in the calculations of precision and recall in the average precision score.",
    "2428547": "Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation\n\n> Unmatched ground-truths are scored as FN.\n\nLet me know if this answers your question. Not sure if I understood your question properly."
  }
}