{
  "id": 460516,
  "title": "Thoughts on metric",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/460516",
  "author_name": "cosmosaa",
  "post_date": "2023-12-09T16:14:55.414000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I believe the metric choice could have been improved to better match how the sleep predictions would be used in a real-world setting for sleep analysis. The metric stayed the same or improved if multiple predictions were made for the same event. I believe this is not useful in a real-world setting since for sleep analysis to be useful, we would want a single onset/wakeup event per true event. I believe the predictions would need to be thresholded and only those predictions greater than the threshold would be included. But the metric does not take this into account. People were able to exploit this flaw in the metric in order to boost scores. </p>\n<p><a href=\"https://github.com/aamster/detect_sleep_states/blob/master/scripts/notebooks/metric.ipynb\" target=\"_blank\">Here</a> is a short notebook I made to show this.</p>",
  "messages": [
    {
      "id": 2555044,
      "postDate": "2023-12-09T16:14:55.413Z",
      "content": "<p>I believe the metric choice could have been improved to better match how the sleep predictions would be used in a real-world setting for sleep analysis. The metric stayed the same or improved if multiple predictions were made for the same event. I believe this is not useful in a real-world setting since for sleep analysis to be useful, we would want a single onset/wakeup event per true event. I believe the predictions would need to be thresholded and only those predictions greater than the threshold would be included. But the metric does not take this into account. People were able to exploit this flaw in the metric in order to boost scores. </p>\n<p><a href=\"https://github.com/aamster/detect_sleep_states/blob/master/scripts/notebooks/metric.ipynb\" target=\"_blank\">Here</a> is a short notebook I made to show this.</p>",
      "rawMarkdown": "I believe the metric choice could have been improved to better match how the sleep predictions would be used in a real-world setting for sleep analysis. The metric stayed the same or improved if multiple predictions were made for the same event. I believe this is not useful in a real-world setting since for sleep analysis to be useful, we would want a single onset/wakeup event per true event. I believe the predictions would need to be thresholded and only those predictions greater than the threshold would be included. But the metric does not take this into account. People were able to exploit this flaw in the metric in order to boost scores. \n\n[Here](https://github.com/aamster/detect_sleep_states/blob/master/scripts/notebooks/metric.ipynb) is a short notebook I made to show this.",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2555044": "I believe the metric choice could have been improved to better match how the sleep predictions would be used in a real-world setting for sleep analysis. The metric stayed the same or improved if multiple predictions were made for the same event. I believe this is not useful in a real-world setting since for sleep analysis to be useful, we would want a single onset/wakeup event per true event. I believe the predictions would need to be thresholded and only those predictions greater than the threshold would be included. But the metric does not take this into account. People were able to exploit this flaw in the metric in order to boost scores. \n\n[Here](https://github.com/aamster/detect_sleep_states/blob/master/scripts/notebooks/metric.ipynb) is a short notebook I made to show this."
  }
}