{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## **An example to show how to use a metric notebook (Event Detection AP)**  \nIs there any documentation on this?","metadata":{}},{"cell_type":"markdown","source":"# Playing with the [Child Mind Institute - Detect Sleep States](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states) competition metric\n\nThe source of the meric is in the notebook [\"Event Detection AP\"](https://www.kaggle.com/code/metric/event-detection-ap/notebook).\n\nThe aim of this notebook is to perform experiments so as to help to become more familiar with the scoring process.\n\nThe undelying metric for each individual series is the **area under the precision-recall curve**.\n\n# Notes:\n\n* In this competition I think **all** of the data is scored, so the `filter_detections` function has been excluded.\n\n* There is no reference to `row_id` so I am not sure if this column is actually ever used?\n\n* 1 step = 5 seconds\n* 1 minute  = 12 steps\n","metadata":{}},{"cell_type":"markdown","source":"# The source code","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport pandas.api.types\nfrom typing import Dict, List, Tuple\n\n# tolerances in steps\ntolerances = {\n    \"onset\":  [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],\n    \"wakeup\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],\n}","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-09-24T02:00:33.521646Z","iopub.execute_input":"2023-09-24T02:00:33.522358Z","iopub.status.idle":"2023-09-24T02:00:33.960232Z","shell.execute_reply.started":"2023-09-24T02:00:33.522318Z","shell.execute_reply":"2023-09-24T02:00:33.959204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Using event-detection-ap. Note that it has globals that need to be set up.\nimport metric\nfrom metric import event_detection_ap\n\n# Globals\nmetric.series_id_column_name = \"series_id\"\nmetric.time_column_name = \"step\"\nmetric.event_column_name = \"event\"\nmetric.score_column_name = \"score\"\nmetric.use_scoring_intervals = None","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:33.962634Z","iopub.execute_input":"2023-09-24T02:00:33.963198Z","iopub.status.idle":"2023-09-24T02:00:33.979099Z","shell.execute_reply.started":"2023-09-24T02:00:33.963158Z","shell.execute_reply":"2023-09-24T02:00:33.978138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test: Calculate the **Average Precision** (\"AP\") for one series\nEach series AP score is averaged to produce the final LB score.","metadata":{}},{"cell_type":"code","source":"train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\n\n# extract just one series to be the \"ground truth\"\nsolution =  train_events.loc[(train_events[\"series_id\"] == \"d515236bdeec\")].copy()\n\n# create a perfect submission\nperfect_submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\n\n# create a perfect score column\nperfect_submission[\"score\"] = 1","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:33.980609Z","iopub.execute_input":"2023-09-24T02:00:33.981107Z","iopub.status.idle":"2023-09-24T02:00:34.044528Z","shell.execute_reply.started":"2023-09-24T02:00:33.981068Z","shell.execute_reply":"2023-09-24T02:00:34.043371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# take a look\nperfect_submission","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:34.049665Z","iopub.execute_input":"2023-09-24T02:00:34.051850Z","iopub.status.idle":"2023-09-24T02:00:34.078590Z","shell.execute_reply.started":"2023-09-24T02:00:34.051815Z","shell.execute_reply":"2023-09-24T02:00:34.077110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"calculate the score","metadata":{}},{"cell_type":"code","source":"event_detection_ap(solution, perfect_submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:34.080137Z","iopub.execute_input":"2023-09-24T02:00:34.080496Z","iopub.status.idle":"2023-09-24T02:00:34.376480Z","shell.execute_reply.started":"2023-09-24T02:00:34.080466Z","shell.execute_reply":"2023-09-24T02:00:34.375333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed a perfect score\n\n# Setting a different global `score` value","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 0.5\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:34.378351Z","iopub.execute_input":"2023-09-24T02:00:34.379105Z","iopub.status.idle":"2023-09-24T02:00:34.660500Z","shell.execute_reply.started":"2023-09-24T02:00:34.379063Z","shell.execute_reply":"2023-09-24T02:00:34.659298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 0.1\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:34.661852Z","iopub.execute_input":"2023-09-24T02:00:34.662189Z","iopub.status.idle":"2023-09-24T02:00:34.939561Z","shell.execute_reply.started":"2023-09-24T02:00:34.662160Z","shell.execute_reply":"2023-09-24T02:00:34.938360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If all of the `score` values are the same,  the `score` value seems to make no difference","metadata":{}},{"cell_type":"markdown","source":"# Zero  score (non-event?)","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 0\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:34.940830Z","iopub.execute_input":"2023-09-24T02:00:34.941149Z","iopub.status.idle":"2023-09-24T02:00:35.223967Z","shell.execute_reply.started":"2023-09-24T02:00:34.941121Z","shell.execute_reply":"2023-09-24T02:00:35.223059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We still have a result of `1`\n\n# Shifting the `step`\n\nLess than 1 minute off (less than 12 steps)","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission[\"step\"] = submission[\"step\"]+10\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:35.225529Z","iopub.execute_input":"2023-09-24T02:00:35.225878Z","iopub.status.idle":"2023-09-24T02:00:35.566010Z","shell.execute_reply.started":"2023-09-24T02:00:35.225849Z","shell.execute_reply":"2023-09-24T02:00:35.564519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"just over 1 minute off:","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission[\"step\"] = submission[\"step\"]+15\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:35.573454Z","iopub.execute_input":"2023-09-24T02:00:35.573907Z","iopub.status.idle":"2023-09-24T02:00:35.854126Z","shell.execute_reply.started":"2023-09-24T02:00:35.573867Z","shell.execute_reply":"2023-09-24T02:00:35.853056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission[\"step\"] = submission[\"step\"]+350\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:35.855479Z","iopub.execute_input":"2023-09-24T02:00:35.855837Z","iopub.status.idle":"2023-09-24T02:00:36.250434Z","shell.execute_reply.started":"2023-09-24T02:00:35.855807Z","shell.execute_reply":"2023-09-24T02:00:36.249381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If our `event` is more than 30 minutes from the `ground truth`  event the score is zero","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission[\"step\"] = submission[\"step\"]+370\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:36.252081Z","iopub.execute_input":"2023-09-24T02:00:36.253041Z","iopub.status.idle":"2023-09-24T02:00:36.588143Z","shell.execute_reply.started":"2023-09-24T02:00:36.253002Z","shell.execute_reply":"2023-09-24T02:00:36.587118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# change both `step` and `score`","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 0.1\nsubmission[\"step\"] = submission[\"step\"]+350\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:36.589549Z","iopub.execute_input":"2023-09-24T02:00:36.590461Z","iopub.status.idle":"2023-09-24T02:00:36.959117Z","shell.execute_reply.started":"2023-09-24T02:00:36.590419Z","shell.execute_reply":"2023-09-24T02:00:36.958099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Random `score` with no time shift","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = np.random.uniform(size=len(submission))\nsubmission[\"step\"] = submission[\"step\"]\n\n# take a look\nsubmission.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:36.960661Z","iopub.execute_input":"2023-09-24T02:00:36.961281Z","iopub.status.idle":"2023-09-24T02:00:36.981746Z","shell.execute_reply.started":"2023-09-24T02:00:36.961242Z","shell.execute_reply":"2023-09-24T02:00:36.980760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"event_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:36.983562Z","iopub.execute_input":"2023-09-24T02:00:36.984323Z","iopub.status.idle":"2023-09-24T02:00:37.432106Z","shell.execute_reply.started":"2023-09-24T02:00:36.984267Z","shell.execute_reply":"2023-09-24T02:00:37.431354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Only `wakeup` events","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission[\"event\"] = \"wakeup\"\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:37.433655Z","iopub.execute_input":"2023-09-24T02:00:37.434706Z","iopub.status.idle":"2023-09-24T02:00:37.712540Z","shell.execute_reply.started":"2023-09-24T02:00:37.434654Z","shell.execute_reply":"2023-09-24T02:00:37.711333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Only `onset` events","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission[\"event\"] = \"onset\"\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:37.714372Z","iopub.execute_input":"2023-09-24T02:00:37.715064Z","iopub.status.idle":"2023-09-24T02:00:37.994085Z","shell.execute_reply.started":"2023-09-24T02:00:37.715023Z","shell.execute_reply":"2023-09-24T02:00:37.992868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inverting `wakeup` and `onset` events","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"event\"] = submission[\"event\"].replace({\"onset\":\"wakeup\",\"wakeup\":\"onset\"})\nsubmission[\"score\"] = 1\nevent_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:37.995451Z","iopub.execute_input":"2023-09-24T02:00:37.995842Z","iopub.status.idle":"2023-09-24T02:00:38.279910Z","shell.execute_reply.started":"2023-09-24T02:00:37.995810Z","shell.execute_reply":"2023-09-24T02:00:38.278791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Randomly shuffle the rows","metadata":{}},{"cell_type":"code","source":"submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\nsubmission[\"score\"] = 1\nsubmission = submission.sample(frac=1)\n\n# take a look\nsubmission.head(15)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:38.281789Z","iopub.execute_input":"2023-09-24T02:00:38.282102Z","iopub.status.idle":"2023-09-24T02:00:38.300043Z","shell.execute_reply.started":"2023-09-24T02:00:38.282075Z","shell.execute_reply":"2023-09-24T02:00:38.298737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"event_detection_ap(solution, submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:38.301680Z","iopub.execute_input":"2023-09-24T02:00:38.302033Z","iopub.status.idle":"2023-09-24T02:00:38.587898Z","shell.execute_reply.started":"2023-09-24T02:00:38.302004Z","shell.execute_reply":"2023-09-24T02:00:38.586739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Using two `series_id`","metadata":{}},{"cell_type":"code","source":"\n# extract two series to be the \"ground truth\"\nsolution =  train_events.loc[(train_events[\"series_id\"] == \"d515236bdeec\") | (train_events[\"series_id\"] == \"d5e47b94477e\")].copy()\n\n# create a perfect submission\nperfect_submission = solution[[\"series_id\",\"step\",\"event\"]].copy()\n\n# create a perfect score column\nperfect_submission[\"score\"] = 1\n\nevent_detection_ap(solution, perfect_submission, tolerances)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T02:00:38.589765Z","iopub.execute_input":"2023-09-24T02:00:38.590476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Randomly shuffle the two series ","metadata":{}},{"cell_type":"code","source":"# create a perfect submission\nsubmission = solution[[\"series_id\",\"step\",\"event\"]].copy()\n\n# create a perfect score column\nsubmission[\"score\"] = 1\n\nsubmission = submission.sample(frac=1)\n\nevent_detection_ap(solution, submission, tolerances)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Two series, different `score` values","metadata":{}},{"cell_type":"code","source":"series_1 =  train_events.loc[(train_events[\"series_id\"] == \"d515236bdeec\")].copy()\nseries_1[\"score\"] = 1\n\nseries_2 =  train_events.loc[(train_events[\"series_id\"] == \"d5e47b94477e\")].copy()\nseries_2[\"score\"] = 0.1\n\nsubmission = pd.concat([series_1, series_2], axis=0)\n\nprint(event_detection_ap(solution, submission, tolerances))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Two series, same `score` (`1`), one with shifted `step`","metadata":{}},{"cell_type":"code","source":"solution_1 =  train_events.loc[(train_events[\"series_id\"] == \"d515236bdeec\")].copy()\nseries_1_submission = solution_1[[\"series_id\",\"step\",\"event\"]].copy()\nseries_1_submission[\"score\"] = 1\n\n\nsolution_2 =  train_events.loc[(train_events[\"series_id\"] == \"d5e47b94477e\")].copy()\nseries_2_submission = solution_2[[\"series_id\",\"step\",\"event\"]].copy()\nseries_2_submission[\"score\"] = 1\nseries_2_submission[\"step\"] = series_2_submission[\"step\"]+350\n\n\nsubmission = pd.concat([series_1_submission, series_2_submission], axis=0)\n\nprint(event_detection_ap(solution, submission, tolerances))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Two series, one with shifted `step` and lower `score` (`0.1`)","metadata":{}},{"cell_type":"code","source":"solution_1 =  train_events.loc[(train_events[\"series_id\"] == \"d515236bdeec\")].copy()\nseries_1_submission = solution_1[[\"series_id\",\"step\",\"event\"]].copy()\nseries_1_submission[\"score\"] = 1\n\n\nsolution_2 =  train_events.loc[(train_events[\"series_id\"] == \"d5e47b94477e\")].copy()\nseries_2_submission = solution_2[[\"series_id\",\"step\",\"event\"]].copy()\nseries_2_submission[\"score\"] = 0.1\nseries_2_submission[\"step\"] = series_2_submission[\"step\"]+350\n\n\nsubmission = pd.concat([series_1_submission, series_2_submission], axis=0)\n\nprint(event_detection_ap(solution, submission, tolerances))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Related notebooks\n* [\"Competition Metric - DFL Event Detection AP](https://www.kaggle.com/code/ryanholbrook/competition-metric-dfl-event-detection-ap)","metadata":{}}]}