Hi, I made a notebook that computes AP events much faster. Kaggle's method compares predicted events to ground truth events pairwise, which probably takes O(nk) complexity, while a method using searchsorted probably takes O(n log k) complexity. Testing shows that it is faster.
\nThis helps a lot when some form of early stopping is needed in the training loop, since the AP metrics are computed every epoch. Moreover, it is possible to plot the precision-recall curves per tolerance using matplotlib, which may help diagnosis.
\nNote that this code assumes the intervals of the ground truth events with the maximum tolerance width are disjoint, which should be ok as there's at most one wake/sleep event per day. The AP values of both methods will slightly be different if the series_id with no events are not discarded (see post)
", "messages": [ { "id": 2531674, "postDate": "2023-11-20T12:17:19.363Z", "content": "Hi, I made a notebook that computes AP events much faster. Kaggle's method compares predicted events to ground truth events pairwise, which probably takes O(nk) complexity, while a method using searchsorted probably takes O(n log k) complexity. Testing shows that it is faster.
\nThis helps a lot when some form of early stopping is needed in the training loop, since the AP metrics are computed every epoch. Moreover, it is possible to plot the precision-recall curves per tolerance using matplotlib, which may help diagnosis.
\nNote that this code assumes the intervals of the ground truth events with the maximum tolerance width are disjoint, which should be ok as there's at most one wake/sleep event per day. The AP values of both methods will slightly be different if the series_id with no events are not discarded (see post)
", "rawMarkdown": "Hi, I made a [notebook](https://www.kaggle.com/code/chauyh/kagglechildsleep-fast-ap-metric-computation/notebook) that computes AP events much faster. Kaggle's method compares predicted events to ground truth events pairwise, which probably takes O(nk) complexity, while a method using searchsorted probably takes O(n log k) complexity. Testing shows that it is faster.\n\nThis helps a lot when some form of early stopping is needed in the training loop, since the AP metrics are computed every epoch. Moreover, it is possible to plot the precision-recall curves per tolerance using matplotlib, which may help diagnosis.\n\nNote that this code assumes the intervals of the ground truth events with the maximum tolerance width are disjoint, which should be ok as there's at most one wake/sleep event per day. The AP values of both methods will slightly be different if the series_id with no events are not discarded ([see post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/454071))", "votes": 8 } ], "comments": [], "raw_markdown_by_id": { "2531674": "Hi, I made a [notebook](https://www.kaggle.com/code/chauyh/kagglechildsleep-fast-ap-metric-computation/notebook) that computes AP events much faster. Kaggle's method compares predicted events to ground truth events pairwise, which probably takes O(nk) complexity, while a method using searchsorted probably takes O(n log k) complexity. Testing shows that it is faster.\n\nThis helps a lot when some form of early stopping is needed in the training loop, since the AP metrics are computed every epoch. Moreover, it is possible to plot the precision-recall curves per tolerance using matplotlib, which may help diagnosis.\n\nNote that this code assumes the intervals of the ground truth events with the maximum tolerance width are disjoint, which should be ok as there's at most one wake/sleep event per day. The AP values of both methods will slightly be different if the series_id with no events are not discarded ([see post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/454071))" } }