Submissions are evaluated on the [average precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) of detected events, averaged over timestamp error tolerance thresholds, averaged over event classes.

Detections are matched to ground-truth events within error tolerances, with ambiguities resolved in order of decreasing confidence. For both event classes, we use error tolerance thresholds of `1, 3, 5, 7.5, 10, 12.5, 15, 20, 25, 30` in minutes, or `12,  36,  60,  90, 120, 150, 180, 240, 300, 360` in steps.

You may find Python code for this metric here: [**Event Detection AP**](https://www.kaggle.com/code/metric/event-detection-ap/notebook).

## Detailed Description ##

Evaluation proceeds in three steps:
1. **Assignment** - Predicted events are matched with ground-truth events.
2. **Scoring** - Each group of predictions is scored against its corresponding group of ground-truth events via Average Precision. 
3. **Reduction** - The multiple AP scores are averaged to produce a single overall score.

### Assignment ###

For each set of predictions and ground-truths within the same `event x tolerance x series_id` group, we **match** each ground-truth to the highest-confidence unmatched prediction occurring within the allowed `tolerance`.

Some ground-truths may not be matched to a prediction and some predictions may not be matched to a ground-truth. They will still be accounted for in the scoring, however.

### Scoring ###

Collecting the events within each `series_id`, we compute an **Average Precision** score for each `event x tolerance` group. The average precision score is the area under the precision-recall curve generated by decreasing confidence `score` thresholds over the predictions. In this calculation, matched predictions over the threshold are scored as TP and unmatched predictions as FP. Unmatched ground-truths are scored as FN.

### Reduction ###

The final score is the average of the above AP scores, first averaged over `tolerance`, then over `event`.

## Submission File ##

For each series indicated by `series_id`, predict each event occurring in that series by giving the `event` type and the `step` where the event occurred as well as a confidence `score` for that event. The evaluation metric additionally requires a `row_id` with an enumeration of events.

The file should contain a header and have the following format:

```
row_id,series_id,step,event,score
0,038441c925bb,100,onset,0.0
1,038441c925bb,105,wakeup,0.0
2,03d92c9f6f8a,80,onset,0.5
3,03d92c9f6f8a,110,wakeup,0.5
4,0402a003dae9,90,onset,1.0
5,0402a003dae9,120,wakeup,1.0
...
```

Note that while the ground-truth annotations were made following certain conventions (as described on the [Data](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data) page), there are no such restrictions on your submission file.