When using score function from https://www.kaggle.com/code/metric/event-detection-ap/notebook and train_events.csv as solution, does removing rows where the step column is nan actually affect the score.I wonder if the test set solution contains nan steps and counts towards the scoring?
", "messages": [ { "id": "2446797", "postDate": "09/19/2023 16:08:29", "content": "When using score function from https://www.kaggle.com/code/metric/event-detection-ap/notebook and train_events.csv as solution, does removing rows where the step column is nan actually affect the score.I wonder if the test set solution contains nan steps and counts towards the scoring?
", "rawMarkdown": "When using score function from https://www.kaggle.com/code/metric/event-detection-ap/notebook and train_events.csv as solution, does removing rows where the step column is nan actually affect the score.I wonder if the test set solution contains nan steps and counts towards the scoring?", "votes": null }, { "id": "2446852", "postDate": "09/19/2023 16:50:39", "content": "Excellent question. When I calculate the metric myself, I always remove the NaN steps, but the score function provided doesn't do this---which leads to an artificially low recall.
\nEven if one uses the train data as the submission data, running the scoring code provided will not give a score of 1., because it is not possible to correctly predict the NaNs and so the recall is less than 1.
\nIt would be great to know if they remove the NaNs when scoring a submission though!
", "rawMarkdown": "Excellent question. When I calculate the metric myself, I always remove the NaN steps, but the score function provided doesn't do this---which leads to an artificially low recall.\n\nEven if one uses the train data as the submission data, running the scoring code provided will not give a score of 1., because it is not possible to correctly predict the NaNs and so the recall is less than 1.\n\nIt would be great to know if they remove the NaNs when scoring a submission though!", "votes": null }, { "id": "2447228", "postDate": "09/19/2023 23:45:21", "content": "I believe that having NaN values in step in submission.csv will generate a Submission Error.
\nMy current understanding of the metric is in the discussion: CMI Average Precision metric example . However, I do need to compare my understanding with the event-detection-ap notebook 😏
", "rawMarkdown": "I believe that having NaN values in step in submission.csv will generate a Submission Error.\n\nMy current understanding of the metric is in the discussion: [CMI Average Precision metric example](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440594) . However, I do need to compare my understanding with the event-detection-ap notebook 😏", "votes": null } ], "comments": [ { "id": 2446852, "author_name": "adamstones", "author_url": "", "post_date": "09/19/2023 16:50:39", "content": "Excellent question. When I calculate the metric myself, I always remove the NaN steps, but the score function provided doesn't do this---which leads to an artificially low recall.
\nEven if one uses the train data as the submission data, running the scoring code provided will not give a score of 1., because it is not possible to correctly predict the NaNs and so the recall is less than 1.
\nIt would be great to know if they remove the NaNs when scoring a submission though!
", "votes": null, "replies": [] }, { "id": 2447228, "author_name": "dan3dewey", "author_url": "", "post_date": "09/19/2023 23:45:21", "content": "I believe that having NaN values in step in submission.csv will generate a Submission Error.
\nMy current understanding of the metric is in the discussion: CMI Average Precision metric example . However, I do need to compare my understanding with the event-detection-ap notebook 😏
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2446797": "When using score function from https://www.kaggle.com/code/metric/event-detection-ap/notebook and train_events.csv as solution, does removing rows where the step column is nan actually affect the score.I wonder if the test set solution contains nan steps and counts towards the scoring?", "2446852": "Excellent question. When I calculate the metric myself, I always remove the NaN steps, but the score function provided doesn't do this---which leads to an artificially low recall.\n\nEven if one uses the train data as the submission data, running the scoring code provided will not give a score of 1., because it is not possible to correctly predict the NaNs and so the recall is less than 1.\n\nIt would be great to know if they remove the NaNs when scoring a submission though!", "2447228": "I believe that having NaN values in step in submission.csv will generate a Submission Error.\n\nMy current understanding of the metric is in the discussion: [CMI Average Precision metric example](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440594) . However, I do need to compare my understanding with the event-detection-ap notebook 😏" }, "source": "meta" }