Submissions are evaluated on the [average precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) of detected events, averaged over timestamp error thresholds, averaged over event classes.

Detections are matched to ground-truth events by class-specific error tolerances, with ambiguities resolved in order of decreasing confidence. The timestamp error tolerances, in seconds, for each class are:
- **Challenge**: `[ 0.30, 0.40, 0.50, 0.60, 0.70 ]`
- **Play**: `[ 0.15, 0.20, 0.25, 0.30, 0.35 ]`
- **Throw-In**: `[ 0.15, 0.20, 0.25, 0.30, 0.35 ]`

You may find a Python implementation of the metric in this notebook: [**Competition Metric - DFL Event Detection AP**](https://www.kaggle.com/code/ryanholbrook/competition-metric-dfl-event-detection-ap).

## Detailed Description ##

Evaluation proceeds in four steps:
1. **Selection** - Predictions not within a video's scoring intervals are dropped.
2. **Assignment** - Predicted events are matched with ground-truth events.
3. **Scoring** - Each group of predictions is scored against its corresponding group of ground-truth events via Average Precision. 
4. **Reduction** - The multiple AP scores are averaged to produce a single overall score.

### Selection ###

With each video there is a defined set of **scoring intervals** giving the intervals of time over which zero or more ground-truth events might be annotated in that video. A prediction is only evaluated if it falls within a scoring interval. These scoring intervals were chosen to improve the fairness of evaluation by, for instance, ignoring edge-cases or ambiguous events.

For reference, we provide the scoring intervals for the training data. The scoring intervals for the test data will not be available to your model during evaluation, however.

### Assignment ###

For each set of predictions and ground-truths within the same `event x tolerance x video_id` group, we **match** each ground-truth to the highest-confidence unmatched prediction occurring within the allowed `tolerance`.

Some ground-truths may not be matched to a prediction and some predictions may not be matched to a ground-truth. They will still be accounted for in the scoring, however.

### Scoring ###

Collecting the events within each `video_id`, we compute an **Average Precision** score for each `event x tolerance` group. The average precision score is the area under the precision-recall curve generated by decreasing confidence `score` thresholds over the predictions. In this calculation, matched predictions over the threshold are scored as TP and unmatched predictions as FP. Unmatched ground-truths are scored as FN.

### Reduction ###

The final score is the average of the above AP scores, first averaged over `tolerance`, then over `event`.

## Submission File ##

For each video indicated by `video_id`, predict each event occurring in that video by giving the `event` type and the `time` of occurrence (in seconds) as well as a confidence `score` for that event.

The file should contain a header and have the following format:

```
video_id,time,event,score
13bfe65e_0,100.0,play,0.0
13bfe65e_0,110.5,challenge,0.07
13bfe65e_1,90.25,throwin,0.13
13bfe65e_1,105.0,play,0.2
40f283bc_0,100.0,challenge,0.27
40f283bc_0,110.5,throwin,0.33
...
```

Only predictions occurring within a video's defined **scoring intervals** will be scored; the metric ignores any predictions occurring outside of these scoring intervals. You will not be given the scoring intervals for videos in the test set.