@carlmcbrideellis made a great notebook exploring the competition metric. From that notebook, it seems like the score column doesn't affect the score, as long as it is changed globally for all predictions. However, when changing the global score value in my submissions, I got different scores. I am still not super sure how the score column works, my understanding was that if there were multiple different predictions in the same window, it would take the one with the highest score. But in that case, a global change of the score column shouldn't affect the LB score.
\nDoes anyone have the score column figured out and uses it to increase the score or do you generally leave it at some constant? And if so, how to choose the constant? So far, I just left it at 0.5.
@carlmcbrideellis made a great notebook exploring the competition metric. From that notebook, it seems like the score column doesn't affect the score, as long as it is changed globally for all predictions. However, when changing the global score value in my submissions, I got different scores. I am still not super sure how the score column works, my understanding was that if there were multiple different predictions in the same window, it would take the one with the highest score. But in that case, a global change of the score column shouldn't affect the LB score.
\nDoes anyone have the score column figured out and uses it to increase the score or do you generally leave it at some constant? And if so, how to choose the constant? So far, I just left it at 0.5.
Dear @fritzcremer
\nIn general on the Overview page it is stated:
\n\n\n\"…and the step where the event occurred as well as a confidence
\nscorefor that event.\"
Say one had the perfect classifier for this data, there will be a step at time \\(t\\) where the participant is awake having a classifier probability of 1 for the awake class, and at step \\(t+1\\) the participant is asleep, having a classifier probability of 0 for the awake class.
\nNow if the \"confidence\" were our classifier probability, if the event onset timestamp was recorded as being at time \\(t+1\\) we would assign a confidence score of 0, whereas only 5 seconds earlier it was 1. This is clearly an unstable situation and in such a situation best way would be to hedge ones bets and simply assign a score of 0.5.
All the best,
\ncarl
I finally figured out that the score is the value that is used (thresholded) to make a Precision-Recall curve of the matched events within a given event-type X tolerance set. When the score is a constant this P-R curve has only a single point (plus the starting p=1 r=0 point.) In this case the Average Precision for the \"curve\" is just given by:
\n AP for this event X tolerance = * Recall =\n (TPs / of actual events) * (TPs / predicted events)\n
\nAn example comparing this simple calculation with the competition AP metric code is given in the notebook CMI 2023 - AP metric and score .
The score is useful when detecting events whose number is unknow and that may or may not happen at any time, for example detecting when a person stood up from sitting and started walking. Rather than giving a list of definite events, the ML can provide a score for candidate events and the P-R process essentially averages over the score thresholds. So the ML score doesn't have to be calibrated (like >0.5 is an event), it just has to give an accurate ranking to the events.
\nIn our case of finding onset and wakeup for the daily sleep period, we know there will be at most one of each per night, that they come in pairs, and we have information to decide if a night is valid or not (i.e., the watch is not worn). I'm thinking this means it is better to either predict or not predict a night rather than assigning different score values. It would be interesting, though, to try assigning scores to less-certain nights and see if that help or hurts.
\nPS I had an ah-ha moment: instead of thinking of score as a measure of how sure we are there is an onset/wakeup event, we can assign score based on how accurate we think the time (step) of the event is determined. I added an example of this to the notebook mentioned above. The example showed that:
\nFlagging some onsets with a score of 0.7 (because looking at graphs they had some more activity after the predicted onset) improved the onset AP because the rater's onset was delay by the activity.
\nFlagging a wakeup that seemed to have activity before its location decreased the wakeup AP - this made sense because the rater's wakeup did not think the activity mattered and the prediction was good.
Hi @fritzcremer based on my experience and what I know, to calculate the area of precision curve (which is calculated under the hood in event detection AP), it requires the prediction should be in terms of score 0 to 1. It is indeed affect the LB score.
\n[OOT] Regarding experimenting with the score, I have not done any. But, I have an experiment of rebinarizing the score with certain threshold.
\nMost of the public notebook rebinarizing the score by something like df[\"event\"] = df.smooth.round() which is actually thresholding by 0.5. I have done experimenting by changing the threshold to 0.4 and 0.7 but it's much worse, so I think 0.5 the best threshold to binarize your score (or basically rounding it to the nearest integer).
\n\nthe prediction should be in terms of score 0 to 1.
\n
Fwiw, I had a valid submission where none of my predictions were within that range. Its LB score was about the same as my other submissions.
\nI'm pretty sure that this competition essentially uses the score column to rank the confidence of each prediction -- which would be consistent with sklearn's average precision implementation
I think that why score outside range of [0, 1] still can be accepted, because to calculate precision recall curve area, one only care about whether a score below or above the threshold (if the threshold is 0.5, score of -1.7 is indeed below the threshold).
\nWhat am I trying to say is, the score may or may not be indirectly affect the event detection AP. I need to experiment with it. Again, I am prone to mistake so CMIIW.
", "rawMarkdown": "I think that why score outside range of [0, 1] still can be accepted, because to calculate precision recall curve area, one only care about whether a score below or above the threshold (if the threshold is 0.5, score of -1.7 is indeed below the threshold).\n\nWhat am I trying to say is, the score may or may not be indirectly affect the event detection AP. I need to experiment with it. Again, I am prone to mistake so CMIIW.", "votes": 1 }, { "id": 2449865, "postDate": "2023-09-21T13:33:49.567Z", "content": "Yes, the average precision is calculated just like the sklearn average precision with some modifications to account for the ground-truth / prediction matching step. In particular, the confidence score isn't treated like a probability (in [0, 1]), but rather a ranking. It's similar to how the AUCROC is calculated. (Using probabilities doesn't hurt, but it's not necessary.)
", "rawMarkdown": "Yes, the average precision is calculated just like the sklearn average precision with some modifications to account for the ground-truth / prediction matching step. In particular, the confidence score isn't treated like a probability (in [0, 1]), but rather a ranking. It's similar to how the AUCROC is calculated. (Using probabilities doesn't hurt, but it's not necessary.)", "votes": 3 }, { "id": 2463439, "postDate": "2023-10-01T10:29:10.423Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2463442, "postDate": "2023-10-01T10:29:38.777Z", "content": "I am still very confused about this \"score\" or \"confidence\". Is there a paper or website that we can read about?
", "rawMarkdown": "I am still very confused about this \"score\" or \"confidence\". Is there a paper or website that we can read about?" } ] } ] } ], "comments": [ { "id": 2446922, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-19T17:52:33", "content": "Dear @fritzcremer
\nIn general on the Overview page it is stated:
\n\n\n\"…and the step where the event occurred as well as a confidence
\nscorefor that event.\"
Say one had the perfect classifier for this data, there will be a step at time \\(t\\) where the participant is awake having a classifier probability of 1 for the awake class, and at step \\(t+1\\) the participant is asleep, having a classifier probability of 0 for the awake class.
\nNow if the \"confidence\" were our classifier probability, if the event onset timestamp was recorded as being at time \\(t+1\\) we would assign a confidence score of 0, whereas only 5 seconds earlier it was 1. This is clearly an unstable situation and in such a situation best way would be to hedge ones bets and simply assign a score of 0.5.
All the best,
\ncarl
I finally figured out that the score is the value that is used (thresholded) to make a Precision-Recall curve of the matched events within a given event-type X tolerance set. When the score is a constant this P-R curve has only a single point (plus the starting p=1 r=0 point.) In this case the Average Precision for the \"curve\" is just given by:
\n AP for this event X tolerance = * Recall =\n (TPs / of actual events) * (TPs / predicted events)\n
\nAn example comparing this simple calculation with the competition AP metric code is given in the notebook CMI 2023 - AP metric and score .
The score is useful when detecting events whose number is unknow and that may or may not happen at any time, for example detecting when a person stood up from sitting and started walking. Rather than giving a list of definite events, the ML can provide a score for candidate events and the P-R process essentially averages over the score thresholds. So the ML score doesn't have to be calibrated (like >0.5 is an event), it just has to give an accurate ranking to the events.
\nIn our case of finding onset and wakeup for the daily sleep period, we know there will be at most one of each per night, that they come in pairs, and we have information to decide if a night is valid or not (i.e., the watch is not worn). I'm thinking this means it is better to either predict or not predict a night rather than assigning different score values. It would be interesting, though, to try assigning scores to less-certain nights and see if that help or hurts.
\nPS I had an ah-ha moment: instead of thinking of score as a measure of how sure we are there is an onset/wakeup event, we can assign score based on how accurate we think the time (step) of the event is determined. I added an example of this to the notebook mentioned above. The example showed that:
\nFlagging some onsets with a score of 0.7 (because looking at graphs they had some more activity after the predicted onset) improved the onset AP because the rater's onset was delay by the activity.
\nFlagging a wakeup that seemed to have activity before its location decreased the wakeup AP - this made sense because the rater's wakeup did not think the activity mattered and the prediction was good.
Hi @fritzcremer based on my experience and what I know, to calculate the area of precision curve (which is calculated under the hood in event detection AP), it requires the prediction should be in terms of score 0 to 1. It is indeed affect the LB score.
\n[OOT] Regarding experimenting with the score, I have not done any. But, I have an experiment of rebinarizing the score with certain threshold.
\nMost of the public notebook rebinarizing the score by something like df[\"event\"] = df.smooth.round() which is actually thresholding by 0.5. I have done experimenting by changing the threshold to 0.4 and 0.7 but it's much worse, so I think 0.5 the best threshold to binarize your score (or basically rounding it to the nearest integer).
\n\nthe prediction should be in terms of score 0 to 1.
\n
Fwiw, I had a valid submission where none of my predictions were within that range. Its LB score was about the same as my other submissions.
\nI'm pretty sure that this competition essentially uses the score column to rank the confidence of each prediction -- which would be consistent with sklearn's average precision implementation
I think that why score outside range of [0, 1] still can be accepted, because to calculate precision recall curve area, one only care about whether a score below or above the threshold (if the threshold is 0.5, score of -1.7 is indeed below the threshold).
\nWhat am I trying to say is, the score may or may not be indirectly affect the event detection AP. I need to experiment with it. Again, I am prone to mistake so CMIIW.
", "votes": 1, "replies": [] }, { "id": 2449865, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-21T13:33:49.567000", "content": "Yes, the average precision is calculated just like the sklearn average precision with some modifications to account for the ground-truth / prediction matching step. In particular, the confidence score isn't treated like a probability (in [0, 1]), but rather a ranking. It's similar to how the AUCROC is calculated. (Using probabilities doesn't hurt, but it's not necessary.)
", "votes": 3, "replies": [] }, { "id": 2463439, "author_name": "", "author_url": "", "post_date": "2023-10-01T10:29:10.423000", "content": "", "votes": 0, "replies": [] }, { "id": 2463442, "author_name": "yhkao", "author_url": "", "post_date": "2023-10-01T10:29:38.777000", "content": "I am still very confused about this \"score\" or \"confidence\". Is there a paper or website that we can read about?
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2446324": "@carlmcbrideellis made a [great notebook](https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric) exploring the competition metric. From that notebook, it seems like the score column doesn't affect the score, as long as it is changed globally for all predictions. However, when changing the global score value in my submissions, I got different scores. I am still not super sure how the score column works, my understanding was that if there were multiple different predictions in the same window, it would take the one with the highest score. But in that case, a global change of the score column shouldn't affect the LB score.\nDoes anyone have the score column figured out and uses it to increase the score or do you generally leave it at some constant? And if so, how to choose the constant? So far, I just left it at 0.5.", "2446922": "Dear @fritzcremer \n\n* When I change my global score in my off-line tests I see no effect on the result\n* As @quillen mentions, it is more akin to a ranking\n\nIn general on the Overview page it is stated: \n\n> \"...*and the step where the event occurred as well as a confidence `score` for that event.*\"\n\nSay one had the perfect classifier for this data, there will be a step at time \\\\(t\\\\) where the participant is awake having a classifier probability of 1 for the `awake` class, and at step \\\\(t+1\\\\) the participant is asleep, having a classifier probability of 0 for the `awake` class.\nNow if the \"confidence\" were our classifier probability, if the event `onset` timestamp was recorded as being at time \\\\(t+1\\\\) we would assign a confidence score of 0, whereas only 5 seconds earlier it was 1. This is clearly an unstable situation and in such a situation best way would be to hedge ones bets and simply assign a score of 0.5.\n\nAll the best,\ncarl", "2451455": "I finally figured out that the score is the value that is used (thresholded) to make a Precision-Recall curve of the matched events within a given event-type X tolerance set. When the score is a constant this P-R curve has only a single point (plus the starting p=1 r=0 point.) In this case the Average Precision for the \"curve\" is just given by:\n```\n AP for this event X tolerance = Precision * Recall =\n (TPs / Number of actual events) * (TPs / Number predicted events)\n``` \nAn example comparing this simple calculation with the competition AP metric code is given in the notebook [CMI 2023 - AP metric and score](https://www.kaggle.com/code/dan3dewey/cmi-2023-ap-metric-and-score) .\n\nThe score is useful when detecting events whose number is unknow and that may or may not happen at any time, for example detecting when a person stood up from sitting and started walking. Rather than giving a list of definite events, the ML can provide a score for candidate events and the P-R process essentially averages over the score thresholds. So the ML score doesn't have to be calibrated (like >0.5 is an event), it just has to give an accurate ranking to the events.\n\nIn our case of finding onset and wakeup for the daily sleep period, we know there will be at most one of each per night, that they come in pairs, and we have information to decide if a night is valid or not (i.e., the watch is not worn). I'm thinking this means it is better to either predict or not predict a night rather than assigning different score values. It would be interesting, though, to try assigning scores to less-certain nights and see if that help or hurts.\n\n_ _ _\nPS I had an ah-ha moment: instead of thinking of score as a measure of how sure we are there is an onset/wakeup event, we can assign score based on how accurate we think the time (step) of the event is determined. I added an example of this to the notebook mentioned above. The example showed that:\nFlagging some onsets with a score of 0.7 (because looking at graphs they had some more activity after the predicted onset) improved the onset AP because the rater's onset was delay by the activity.\nFlagging a wakeup that seemed to have activity before its location decreased the wakeup AP - this made sense because the rater's wakeup did not think the activity mattered and the prediction was good.", "2446364": "Hi @fritzcremer based on my experience and what I know, to calculate the area of precision curve (which is calculated under the hood in event detection AP), it requires the prediction should be in terms of score 0 to 1. It is indeed affect the LB score.\n\n[OOT] Regarding experimenting with the score, I have not done any. But, I have an experiment of rebinarizing the score with certain threshold.\n\nMost of the public notebook rebinarizing the score by something like `df[\"event\"] = df.smooth.round()` which is actually thresholding by 0.5. I have done experimenting by changing the threshold to 0.4 and 0.7 but it's much worse, so I think 0.5 the best threshold to binarize your score (or basically rounding it to the nearest integer)." } }