![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F0abbce9e3bf61a0b52dd188b920b9b8b%2Fddv32_onset_APs.png?generation=1695849993068430&alt=media\")
As others have alluded to, it looks like some series are easier to predict than other ones. For examples, using a simple non-ML edge-fitting model which has LB ~ 0.343, the series[163] 9ee455e4770d is predicted well with AP onsets = 0.647 and AP wakeups = 0.613. On the other hand, series[276] fe90110788d2 has AP onsets = 0.290 and AP wakeups = 0.236.
\nTo look at the range of AP values (my model gives) I processed all the training series and made plots of the AP values for the onsets and for the wakeups. Here are plots and histograms for them:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F6d3eb8df260907cb6268fb464739a964%2Fddv32_wakeup_APs.png?generation=1695850009478752&alt=media\")
Not surprisingly there are large variations between APs for the series (i.e., between subjects, their behaviors, environments, etc..) What surprised me is that the APs for onsets and the APs for wakeups for a given series are not very correlated, so it's possible to have a high onset AP but a lower wakeup AP and the other way around. This plot shows each series plotted by its onset and wakeup APs.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F46775e8bf560502074fd6f9b5caec9b6%2Fddv32_wakeup_vs_onset.png?generation=1695850025019714&alt=media\")
These plots are for my model, I'm wondering what these plots would look like for different models 🤔
", "messages": [ { "id": "2458897", "postDate": "09/27/2023 22:00:32", "content": "As others have alluded to, it looks like some series are easier to predict than other ones. For examples, using a simple non-ML edge-fitting model which has LB ~ 0.343, the series[163] 9ee455e4770d is predicted well with AP onsets = 0.647 and AP wakeups = 0.613. On the other hand, series[276] fe90110788d2 has AP onsets = 0.290 and AP wakeups = 0.236.
\nTo look at the range of AP values (my model gives) I processed all the training series and made plots of the AP values for the onsets and for the wakeups. Here are plots and histograms for them:
\n
Not surprisingly there are large variations between APs for the series (i.e., between subjects, their behaviors, environments, etc..) What surprised me is that the APs for onsets and the APs for wakeups for a given series are not very correlated, so it's possible to have a high onset AP but a lower wakeup AP and the other way around. This plot shows each series plotted by its onset and wakeup APs.
\n
These plots are for my model, I'm wondering what these plots would look like for different models 🤔
", "rawMarkdown": "As others have alluded to, it looks like some series are easier to predict than other ones. For examples, using a simple non-ML edge-fitting model which has LB ~ 0.343, the series[163] 9ee455e4770d is predicted well with AP onsets = 0.647 and AP wakeups = 0.613. On the other hand, series[276] fe90110788d2 has AP onsets = 0.290 and AP wakeups = 0.236.\n\nTo look at the range of AP values (my model gives) I processed all the training series and made plots of the AP values for the onsets and for the wakeups. Here are plots and histograms for them:\n\n\n\n\n\nNot surprisingly there are large variations between APs for the series (i.e., between subjects, their behaviors, environments, etc..) What surprised me is that the APs for onsets and the APs for wakeups for a given series are not very correlated, so it's possible to have a high onset AP but a lower wakeup AP and the other way around. This plot shows each series plotted by its onset and wakeup APs.\n\n\n\nThese plots are for my model, I'm wondering what these plots would look like for different models 🤔", "votes": null }, { "id": "2459594", "postDate": "09/28/2023 09:43:11", "content": "I'm wondering is AP correlated with number of NaN nights in series.
\nPs: How do you deal with NaNs in train events, drop it?
Hi @dan3dewey. Please tell me the CV of the results shown this time.
", "rawMarkdown": "Hi @dan3dewey. Please tell me the CV of the results shown this time.", "votes": null }, { "id": "2461419", "postDate": "09/29/2023 13:49:39", "content": "The average of the APs over all the series, 0.393, is a good value for my \"CV\" (added to plot above) since I'm not using an ML model that is trained on, or can 'memorize', the series true values. This 0.393 is a little better than the LB value of 0.343.
", "rawMarkdown": "The average of the APs over all the series, 0.393, is a good value for my \"CV\" (added to plot above) since I'm not using an ML model that is trained on, or can 'memorize', the series true values. This 0.393 is a little better than the LB value of 0.343.", "votes": null }, { "id": "2461424", "postDate": "09/29/2023 13:52:33", "content": "@dan3dewey Thank you because I was concerned about the correlation between CV and LB.
", "rawMarkdown": "dan3dewey Thank you because I was concerned about the correlation between CV and LB.", "votes": null }, { "id": "2461476", "postDate": "09/29/2023 14:19:41", "content": "Here's a plot of AP (onset,wakeup combined) vs the ratio of number of valid (non-NaN) nights to the number of labeled nights (i.e., the max of the night value) in each series. You can draw your own conclusions :)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F34bbb3a26efc1910577ba2a316d8319c%2Fddv34_AP_vs_ratio.png?generation=1695996595549304&alt=media\")
NaNs in the train events tell us that the ground truth (if we believe the raters) for that night is that there was not a valid sleeping period. I think the scoring only cares about predicted events and ground truth events, so there is no use to submit an NaN event. But, the AP is reduced if we provide events which are not matched to ground truth, so giving onset,wakeup values for a night that is NaN in train events will just be counted as False Positives.
", "rawMarkdown": "Here's a plot of AP (onset,wakeup combined) vs the ratio of number of valid (non-NaN) nights to the number of labeled nights (i.e., the max of the `night` value) in each series. You can draw your own conclusions :)\n\n\n\nNaNs in the train events tell us that the ground truth (if we believe the raters) for that night is that there was not a valid sleeping period. I think the scoring only cares about predicted events and ground truth events, so there is no use to submit an NaN event. But, the AP is reduced if we provide events which are not matched to ground truth, so giving onset,wakeup values for a night that is NaN in train events will just be counted as False Positives.", "votes": null }, { "id": "2461514", "postDate": "09/29/2023 14:48:59", "content": "Thanks for the plot. What am I really asking is, when we use this line metric.score(solution, submission, ...) to evaluate, should we drop nans from solution? We can't predict nans anyway, so excluding nans makes sense.
Yes, I don't think there is any reason to submit an NaN -- in fact, I was getting scoring errors from my competition submissions and decided that the reason was that I was including NaNs in submission.csv .
", "rawMarkdown": "Yes, I don't think there is any reason to submit an NaN -- in fact, I was getting scoring errors from my competition submissions and decided that the reason was that I was including NaNs in submission.csv .", "votes": null }, { "id": "2461873", "postDate": "09/29/2023 21:17:19", "content": "P.S. Another useful visualization, besides looking at the AP values for each series, is to look at the median onset and wakeup errors for each series. Specifically something like:
\n = np.nanmedian(predict_onsets - actual_onsets)\n = np.nanmedian(predict_wakeups - actual_wakeups)\nwhere the actuals are from the events file for that series and the predicted ones match (NaNs if needed) and are made the same length as the actuals. Here's the initial plot for my model:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Ff32d7089370cb3de727e7bf03995530c%2Fmedian_errors_before.png?generation=1696021188410468&alt=media\")
Finding the medians of these medians (around 30 and -30 steps), I applied correction constants to the model step values to get 0 for the median of medians, see below. My LB went from 0.343 to 0.348, so for my model it's probably more valuable for the visuals than for an LB improvement 😆
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F6743abdfff78dce6c54eed7ead63621c%2Fmedian_errors_zeroed.png?generation=1696021211676457&alt=media\")
I'm wondering is AP correlated with number of NaN nights in series.
\nPs: How do you deal with NaNs in train events, drop it?
Here's a plot of AP (onset,wakeup combined) vs the ratio of number of valid (non-NaN) nights to the number of labeled nights (i.e., the max of the night value) in each series. You can draw your own conclusions :)
NaNs in the train events tell us that the ground truth (if we believe the raters) for that night is that there was not a valid sleeping period. I think the scoring only cares about predicted events and ground truth events, so there is no use to submit an NaN event. But, the AP is reduced if we provide events which are not matched to ground truth, so giving onset,wakeup values for a night that is NaN in train events will just be counted as False Positives.
", "votes": null, "replies": [] }, { "id": 2461514, "author_name": "aphysict", "author_url": "", "post_date": "09/29/2023 14:48:59", "content": "Thanks for the plot. What am I really asking is, when we use this line metric.score(solution, submission, ...) to evaluate, should we drop nans from solution? We can't predict nans anyway, so excluding nans makes sense.
Yes, I don't think there is any reason to submit an NaN -- in fact, I was getting scoring errors from my competition submissions and decided that the reason was that I was including NaNs in submission.csv .
", "votes": null, "replies": [] } ] } ] }, { "id": 2459914, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/28/2023 13:16:09", "content": "Hi @dan3dewey. Please tell me the CV of the results shown this time.
", "votes": null, "replies": [ { "id": 2461419, "author_name": "dan3dewey", "author_url": "", "post_date": "09/29/2023 13:49:39", "content": "The average of the APs over all the series, 0.393, is a good value for my \"CV\" (added to plot above) since I'm not using an ML model that is trained on, or can 'memorize', the series true values. This 0.393 is a little better than the LB value of 0.343.
", "votes": null, "replies": [ { "id": 2461424, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/29/2023 13:52:33", "content": "@dan3dewey Thank you because I was concerned about the correlation between CV and LB.
", "votes": null, "replies": [] } ] } ] }, { "id": 2461873, "author_name": "dan3dewey", "author_url": "", "post_date": "09/29/2023 21:17:19", "content": "P.S. Another useful visualization, besides looking at the AP values for each series, is to look at the median onset and wakeup errors for each series. Specifically something like:
\n = np.nanmedian(predict_onsets - actual_onsets)\n = np.nanmedian(predict_wakeups - actual_wakeups)\nwhere the actuals are from the events file for that series and the predicted ones match (NaNs if needed) and are made the same length as the actuals. Here's the initial plot for my model:
\n
Finding the medians of these medians (around 30 and -30 steps), I applied correction constants to the model step values to get 0 for the median of medians, see below. My LB went from 0.343 to 0.348, so for my model it's probably more valuable for the visuals than for an LB improvement 😆
\n