![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Fb50436c9118b0c9117e6c9bc6ebc404e%2Fassigned_scores_with_ML.png?generation=1698860103746733&alt=media\")
I took a simple transition-fitting approach to finding the onset and wakeup event locations in the CMI 2023 - Sleepy Times notebook; see some images below and some details in the \"Summary of the current algorithm\" at the notebook's beginning. This method got me an LB of 0.434 with the event score values all the same (1.0) because the fitting doesn't provide an appropriate measure of \"confidence\".
By adding a score value to each event, the LB for the same event predictions increased to 0.503. To set the scores I used a simple random forest model and trained it to predict the scaled error (0 to 1) of an event based on some feature values from the event's fitting; then setting score = 1 - level-of-error. This simple ML is done in the CMI 2023 - Sleepy Times - phat ML notebook using data in \"phat_frame.csv\" -- a submission-like file with many extra columns, including the true step values and fit feature values. The ML models are dill'ed and can be used by the main notebook to assign scores to the test events when run on test data.
Assigned scores vs the errors:
\n
_ _ _ Example of score helping AP
\nIncluding a score that is higher for more precise predictions can improve the AP value. For example, my recall for wakeups in the +/-12 step tolerance range is 0.172 ( = # of predicted wakeups within +/-12 steps / # of all ground-truth wakeups.) Without a score, the precision is also about 0.170 ( = # of predicted wakeups within +/-12 steps / # of all predicted wakeups) since there are similar numbers of predicted and ground-truth wakeups. So the recall-averaged AP will be 0.172 * 0.170 ~ 0.0292 .
\nHowever when scores are assigned, a score threshold is used and only predicted events with scores above the threshold are considered in calculating the recall and precision. The precision with the score selection can be larger than overall, and the recall will be smaller. As the score threshold goes from high to low values the recall increases. Each time the threshold is lowered, the precision in the new selection is multiplied by the increase in the recall and this product is added onto the Total AP.
\nAs an example this table shows how the AP for wakeups with tol=12 steps is calculated using a few score thresholds. In this case the Total AP comes out to 0.0461 which is larger than the without-score value of 0.0292. This happens because the precision of high score events, ~ 0.332, is higher than the precision overall, 0.172.
\n #Gnd #Above\n Truth Thresh #TP Prec. * dRecall = dAP Total AP\n .: . * . = . .\n .: . * . = . .\n .: . * . = . .\n .: . * . = . .\n_ _ _ Some images from the fitting method:
\nA sleep/awake feature is fit with a cosine to find approx middle-of-sleep locations for each night of a series:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Fbc4a4bdf14756f952cf00448f5a87d22%2Ffit_to_find_events.png?generation=1698860035760187&alt=media\")
Zooming in on an expected onset location, a step-down ramp is fit to get the time of onset. The plots here show that the predicted time for Night 9 (blue vertical line) is very close to the ground-truth time (green vertical line.) On the other hand, for Night 12 the predicted time is too early by about 50 minutes.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F778d10aa034cc71fd887a572760e1c1f%2Fgood_fit_example_onset9.png?generation=1698860055664561&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Fe220866b0f06a42d0c78fd1ee5e2ee22%2Fpoor_fit_example_onset12.png?generation=1698860071004311&alt=media\")
I took a simple transition-fitting approach to finding the onset and wakeup event locations in the CMI 2023 - Sleepy Times notebook; see some images below and some details in the \"Summary of the current algorithm\" at the notebook's beginning. This method got me an LB of 0.434 with the event score values all the same (1.0) because the fitting doesn't provide an appropriate measure of \"confidence\".
By adding a score value to each event, the LB for the same event predictions increased to 0.503. To set the scores I used a simple random forest model and trained it to predict the scaled error (0 to 1) of an event based on some feature values from the event's fitting; then setting score = 1 - level-of-error. This simple ML is done in the CMI 2023 - Sleepy Times - phat ML notebook using data in \"phat_frame.csv\" -- a submission-like file with many extra columns, including the true step values and fit feature values. The ML models are dill'ed and can be used by the main notebook to assign scores to the test events when run on test data.
Assigned scores vs the errors:
\n
_ _ _ Example of score helping AP
\nIncluding a score that is higher for more precise predictions can improve the AP value. For example, my recall for wakeups in the +/-12 step tolerance range is 0.172 ( = # of predicted wakeups within +/-12 steps / # of all ground-truth wakeups.) Without a score, the precision is also about 0.170 ( = # of predicted wakeups within +/-12 steps / # of all predicted wakeups) since there are similar numbers of predicted and ground-truth wakeups. So the recall-averaged AP will be 0.172 * 0.170 ~ 0.0292 .
\nHowever when scores are assigned, a score threshold is used and only predicted events with scores above the threshold are considered in calculating the recall and precision. The precision with the score selection can be larger than overall, and the recall will be smaller. As the score threshold goes from high to low values the recall increases. Each time the threshold is lowered, the precision in the new selection is multiplied by the increase in the recall and this product is added onto the Total AP.
\nAs an example this table shows how the AP for wakeups with tol=12 steps is calculated using a few score thresholds. In this case the Total AP comes out to 0.0461 which is larger than the without-score value of 0.0292. This happens because the precision of high score events, ~ 0.332, is higher than the precision overall, 0.172.
\n #Gnd #Above\n Truth Thresh #TP Prec. * dRecall = dAP Total AP\n .: . * . = . .\n .: . * . = . .\n .: . * . = . .\n .: . * . = . .\n_ _ _ Some images from the fitting method:
\nA sleep/awake feature is fit with a cosine to find approx middle-of-sleep locations for each night of a series:
\n
Zooming in on an expected onset location, a step-down ramp is fit to get the time of onset. The plots here show that the predicted time for Night 9 (blue vertical line) is very close to the ground-truth time (green vertical line.) On the other hand, for Night 12 the predicted time is too early by about 50 minutes.
\n