If we look at true events, they:
\nSee code and histogram below for the 15 minute pattern.
\nI wonder if this pattern can be exploited in post-processing of predictions.
\nimport pandas as pd
\nimport matplotlib.pyplot as plt
\ntrain_events = pd.read_csv('train_events.csv')
\ntrain_events = train_events.loc[~train_events['timestamp'].isna()] # drop NA
\nmin15 = train_events['timestamp'].str[-10:-8].astype('int32')%15 # minute of event, in 15 min blocks
\nh=plt.hist(min15, bins=50)
\nplt.show()
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1753973%2Fdf4a40576379f0d234d60042d5cec4f2%2FFigure_1.png?generation=1696172816055059&alt=media\")
If we look at true events, they:
\nSee code and histogram below for the 15 minute pattern.
\nI wonder if this pattern can be exploited in post-processing of predictions.
\nimport pandas as pd
\nimport matplotlib.pyplot as plt
\ntrain_events = pd.read_csv('train_events.csv')
\ntrain_events = train_events.loc[~train_events['timestamp'].isna()] # drop NA
\nmin15 = train_events['timestamp'].str[-10:-8].astype('int32')%15 # minute of event, in 15 min blocks
\nh=plt.hist(min15, bins=50)
\nplt.show()
\n
Hmm that is interesting. But I suspect that this alone won't have a lot of impact since the metric has a threshold of at least 12 minutes.
", "rawMarkdown": "Hmm that is interesting. But I suspect that this alone won't have a lot of impact since the metric has a threshold of at least 12 minutes.", "votes": null }, { "id": "2463883", "postDate": "10/01/2023 16:44:13", "content": "Seems the distribution and period of minute also differ by event type.
", "rawMarkdown": "Seems the distribution and period of minute also differ by event type.", "votes": null }, { "id": "2463920", "postDate": "10/01/2023 17:27:22", "content": "Smallest tolerance is 12 steps which is 1 minute 😉
", "rawMarkdown": "Smallest tolerance is 12 **steps** which is 1 minute 😉", "votes": null }, { "id": "2464670", "postDate": "10/02/2023 12:21:45", "content": "This is a very interesting result @ymatioun. However, it seems difficult to apply this tendency to post-processing.
", "rawMarkdown": "This is a very interesting result @ymatioun. However, it seems difficult to apply this tendency to post-processing.", "votes": null }, { "id": "2464901", "postDate": "10/02/2023 15:21:39", "content": "Thanks for sharing, very cool discovery!
\nBuilding on what @w5833946 said, here are the distributions grouped by event type:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2673089%2F29a172f71a29e2ff7b35625d5f95f068%2Fonset%20(2).png?generation=1696260016614106&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2673089%2F7cdcbc9e4387863c3b221c1ccce897eb%2Fwakeup.png?generation=1696259037774905&alt=media\")
Constructive content, and nice to see your work @ymatioun
", "rawMarkdown": "Constructive content, and nice to see your work @ymatioun", "votes": null }, { "id": "2467244", "postDate": "10/04/2023 11:35:35", "content": "@ymatioun : interesting findings, thanks for sharing!
\nLooking at the phenomenon you spotted deeper, I can share additional note for wakeups. Namely, we can see almost regularly equal number of events in every minute category within the 4 event frequency buckets there
\nOh right, that's what I meant!
", "rawMarkdown": "Oh right, that's what I meant!", "votes": null }, { "id": "2468151", "postDate": "10/05/2023 08:30:06", "content": "Have anyone tried post processing with success? I tried rounding and my CV and LB decreased, maybe I'm doing something wrong
", "rawMarkdown": "Have anyone tried post processing with success? I tried rounding and my CV and LB decreased, maybe I'm doing something wrong", "votes": null }, { "id": "2473859", "postDate": "10/08/2023 16:12:18", "content": "A very cool observation! This suggests that the \"ground truth\" in the Events file is not exactly correct within 2 minutes or so. So, a model that correctly identifies the real onsets/wakeups scored against this would not get an AP of 1.0, maybe a max of 0.95?
\nI wonder if this pattern over 15 min -- or over an hour as in @bkowshik 's Event annotations distributed by second, minute and hour of the day notebook -- varies from series to series and is a function of the human rater and the tools they used? Is it possible to identify clusters of series with similar raters-tools? There aren't many events per series so comparing the number in 0, 3, 7, 11 vs the number of others could be a measure of the effect with some amount of signal to noise.
", "rawMarkdown": "A very cool observation! This suggests that the \"ground truth\" in the Events file is not exactly correct within 2 minutes or so. So, a model that correctly identifies the real onsets/wakeups scored against this would not get an AP of 1.0, maybe a max of 0.95?\n \nI wonder if this pattern over 15 min -- or over an hour as in @bkowshik 's [Event annotations distributed by second, minute and hour of the day](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/445200) notebook -- varies from series to series and is a function of the human rater and the tools they used? Is it possible to identify clusters of series with similar raters-tools? There aren't many events per series so comparing the number in 0, 3, 7, 11 vs the number of others could be a measure of the effect with some amount of signal to noise.", "votes": null }, { "id": "2475187", "postDate": "10/09/2023 17:27:35", "content": "(cont.) It's actually a classic Binomial distribution if we take n = number of events in the series and x = sum of events in 0,3,7,11 bins. Based on about 9600 events in all and 4 bins of 1450 in @ymatioun 's plot, the probability of an event being in the summed-bins is p ~ 41450/9600 ~ 0.60 (0,3,7,11-to-total ratio 0.60). \nThe output below is from Matt Bognar's nice Binomial applet. It shows that if all the series follow this pattern then the probability of a series with 18 nights, n=36 events, having 15 or fewer (ratio 0.42 or less) in the combined 0,3,7,11 bins is unlikely, only 2 %. The evenly-distributed case would peak at about (4/15)36 ~ 9 or 10 events (ratio 0.26). So, if some series are \"0,3,7,11-quantized\" and some are evenly-distributed, the 0,3,7,11-to-total ratio histogram might show two peaks.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Ff4da5839ab07df59fcc846ce22bd87ab%2Fbinomial_timestamps_cmi.png?generation=1696871508071293&alt=media\")
@dan3dewey : thanks for sharing the insights on the possible impact of the human raters/tools used on the event labelling as well as spotting the possible binomial distribution of the labelled events across the 15-min intervals.
\nI prepared the respective visualizations (see the respective notebook, 'Event Minute Patterns' subsection) that confirm your findings (for wakeup and onset events, separately).
", "rawMarkdown": "dan3dewey : thanks for sharing the insights on the possible impact of the human raters/tools used on the event labelling as well as spotting the possible binomial distribution of the labelled events across the 15-min intervals.\n\nI prepared the respective visualizations (see the [respective notebook](https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive#Deep-Dive-into-Training-Events), *'Event Minute Patterns'* subsection) that confirm your findings (for wakeup and onset events, separately).", "votes": null }, { "id": "2477686", "postDate": "10/11/2023 13:49:52", "content": "@w5833946 : yes, exactly. I prepared the visualizations (see the respective notebook, 'Event Minute Patterns' subsection) that confirm it.
", "rawMarkdown": "w5833946 : yes, exactly. I prepared the visualizations (see the [respective notebook](https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive#Deep-Dive-into-Training-Events), *'Event Minute Patterns'* subsection) that confirm it.", "votes": null }, { "id": "2478418", "postDate": "10/12/2023 02:07:28", "content": "Nice graphics in the notebook! It does seem that most series have the 0,3,7,11 minutes enhanced, with ratios of 0.5 to 0.8. These are much higher than the ~0.3 expected if the minutes were evenly distributed.
", "rawMarkdown": "Nice graphics in the notebook! It does seem that most series have the 0,3,7,11 minutes enhanced, with ratios of 0.5 to 0.8. These are much higher than the ~0.3 expected if the minutes were evenly distributed.", "votes": null }, { "id": "2479116", "postDate": "10/12/2023 12:20:58", "content": "i noticed one more pattern: all series in training data begin exactly on 15 minute boundaries - at 0/15/30/45 minutes.
", "rawMarkdown": "i noticed one more pattern: all series in training data begin exactly on 15 minute boundaries - at 0/15/30/45 minutes.", "votes": null }, { "id": "2479204", "postDate": "10/12/2023 13:46:17", "content": "In fact, this tendency may be useful as a feature in ML.
", "rawMarkdown": "In fact, this tendency may be useful as a feature in ML.", "votes": null } ], "comments": [ { "id": 2463830, "author_name": "fritzcremer", "author_url": "", "post_date": "10/01/2023 15:43:52", "content": "Hmm that is interesting. But I suspect that this alone won't have a lot of impact since the metric has a threshold of at least 12 minutes.
", "votes": null, "replies": [ { "id": 2463920, "author_name": "dan3dewey", "author_url": "", "post_date": "10/01/2023 17:27:22", "content": "Smallest tolerance is 12 steps which is 1 minute 😉
", "votes": null, "replies": [ { "id": 2467840, "author_name": "fritzcremer", "author_url": "", "post_date": "10/04/2023 22:31:45", "content": "Oh right, that's what I meant!
", "votes": null, "replies": [] } ] } ] }, { "id": 2463883, "author_name": "w5833946", "author_url": "", "post_date": "10/01/2023 16:44:13", "content": "Seems the distribution and period of minute also differ by event type.
", "votes": null, "replies": [ { "id": 2477686, "author_name": "gvyshnya", "author_url": "", "post_date": "10/11/2023 13:49:52", "content": "@w5833946 : yes, exactly. I prepared the visualizations (see the respective notebook, 'Event Minute Patterns' subsection) that confirm it.
", "votes": null, "replies": [] } ] }, { "id": 2464670, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "10/02/2023 12:21:45", "content": "This is a very interesting result @ymatioun. However, it seems difficult to apply this tendency to post-processing.
", "votes": null, "replies": [ { "id": 2479204, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "10/12/2023 13:46:17", "content": "In fact, this tendency may be useful as a feature in ML.
", "votes": null, "replies": [] } ] }, { "id": 2464901, "author_name": "erijoel", "author_url": "", "post_date": "10/02/2023 15:21:39", "content": "Thanks for sharing, very cool discovery!
\nBuilding on what @w5833946 said, here are the distributions grouped by event type:
\n
Constructive content, and nice to see your work @ymatioun
", "votes": null, "replies": [] }, { "id": 2467244, "author_name": "gvyshnya", "author_url": "", "post_date": "10/04/2023 11:35:35", "content": "@ymatioun : interesting findings, thanks for sharing!
\nLooking at the phenomenon you spotted deeper, I can share additional note for wakeups. Namely, we can see almost regularly equal number of events in every minute category within the 4 event frequency buckets there
\nHave anyone tried post processing with success? I tried rounding and my CV and LB decreased, maybe I'm doing something wrong
", "votes": null, "replies": [] }, { "id": 2473859, "author_name": "dan3dewey", "author_url": "", "post_date": "10/08/2023 16:12:18", "content": "A very cool observation! This suggests that the \"ground truth\" in the Events file is not exactly correct within 2 minutes or so. So, a model that correctly identifies the real onsets/wakeups scored against this would not get an AP of 1.0, maybe a max of 0.95?
\nI wonder if this pattern over 15 min -- or over an hour as in @bkowshik 's Event annotations distributed by second, minute and hour of the day notebook -- varies from series to series and is a function of the human rater and the tools they used? Is it possible to identify clusters of series with similar raters-tools? There aren't many events per series so comparing the number in 0, 3, 7, 11 vs the number of others could be a measure of the effect with some amount of signal to noise.
", "votes": null, "replies": [ { "id": 2475187, "author_name": "dan3dewey", "author_url": "", "post_date": "10/09/2023 17:27:35", "content": "(cont.) It's actually a classic Binomial distribution if we take n = number of events in the series and x = sum of events in 0,3,7,11 bins. Based on about 9600 events in all and 4 bins of 1450 in @ymatioun 's plot, the probability of an event being in the summed-bins is p ~ 41450/9600 ~ 0.60 (0,3,7,11-to-total ratio 0.60). \nThe output below is from Matt Bognar's nice Binomial applet. It shows that if all the series follow this pattern then the probability of a series with 18 nights, n=36 events, having 15 or fewer (ratio 0.42 or less) in the combined 0,3,7,11 bins is unlikely, only 2 %. The evenly-distributed case would peak at about (4/15)36 ~ 9 or 10 events (ratio 0.26). So, if some series are \"0,3,7,11-quantized\" and some are evenly-distributed, the 0,3,7,11-to-total ratio histogram might show two peaks.
\n
@dan3dewey : thanks for sharing the insights on the possible impact of the human raters/tools used on the event labelling as well as spotting the possible binomial distribution of the labelled events across the 15-min intervals.
\nI prepared the respective visualizations (see the respective notebook, 'Event Minute Patterns' subsection) that confirm your findings (for wakeup and onset events, separately).
", "votes": null, "replies": [ { "id": 2478418, "author_name": "dan3dewey", "author_url": "", "post_date": "10/12/2023 02:07:28", "content": "Nice graphics in the notebook! It does seem that most series have the 0,3,7,11 minutes enhanced, with ratios of 0.5 to 0.8. These are much higher than the ~0.3 expected if the minutes were evenly distributed.
", "votes": null, "replies": [ { "id": 2479116, "author_name": "ymatioun", "author_url": "", "post_date": "10/12/2023 12:20:58", "content": "i noticed one more pattern: all series in training data begin exactly on 15 minute boundaries - at 0/15/30/45 minutes.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2463797": "If we look at true events, they:\n1. are always on exact minutes - probably due to manual classification leading to limited precision\n2. show pattern repeating in 15 minutes cycle, with pronounces peaks at 0, 3, 7 and 11 minutes. I have no idea why this happens.\n\nSee code and histogram below for the 15 minute pattern.\n\nI wonder if this pattern can be exploited in post-processing of predictions.\n*****************************************************************\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\ntrain_events = pd.read_csv('train_events.csv')\ntrain_events = train_events.loc[~train_events['timestamp'].isna()] # drop NA\nmin15 = train_events['timestamp'].str[-10:-8].astype('int32')%15 # minute of event, in 15 min blocks\nh=plt.hist(min15, bins=50)\nplt.show()\n", "2463830": "Hmm that is interesting. But I suspect that this alone won't have a lot of impact since the metric has a threshold of at least 12 minutes.", "2463883": "Seems the distribution and period of minute also differ by event type.", "2463920": "Smallest tolerance is 12 **steps** which is 1 minute 😉", "2464670": "This is a very interesting result @ymatioun. However, it seems difficult to apply this tendency to post-processing.", "2464901": "Thanks for sharing, very cool discovery!\n\nBuilding on what @w5833946 said, here are the distributions grouped by event type:\n\n.png?generation=1696260016614106&alt=media)\n\n", "2465202": "Constructive content, and nice to see your work @ymatioun", "2467244": "ymatioun : interesting findings, thanks for sharing!\n\nLooking at the phenomenon you spotted deeper, I can share additional note for wakeups. Namely, we can see almost regularly equal number of events in every minute category within the 4 event frequency buckets there\n\n- the most frequent event minutes (0, 3, 7, 11 minutes)\n- second the most frequent event minutes (2, 6, 10, 14 minutes) - they stay 1 min behind the minutes in the most frequent bucket, btw\n- modereate events bucket minutes (4, 8, 12 minutes)\n- least frequent event bucket (1, 5, 9, 13 minutes)", "2467840": "Oh right, that's what I meant!", "2468151": "Have anyone tried post processing with success? I tried rounding and my CV and LB decreased, maybe I'm doing something wrong", "2473859": "A very cool observation! This suggests that the \"ground truth\" in the Events file is not exactly correct within 2 minutes or so. So, a model that correctly identifies the real onsets/wakeups scored against this would not get an AP of 1.0, maybe a max of 0.95?\n \nI wonder if this pattern over 15 min -- or over an hour as in @bkowshik 's [Event annotations distributed by second, minute and hour of the day](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/445200) notebook -- varies from series to series and is a function of the human rater and the tools they used? Is it possible to identify clusters of series with similar raters-tools? There aren't many events per series so comparing the number in 0, 3, 7, 11 vs the number of others could be a measure of the effect with some amount of signal to noise.", "2475187": "(cont.) It's actually a classic Binomial distribution if we take n = number of events in the series and x = sum of events in 0,3,7,11 bins. Based on about 9600 events in all and 4 bins of 1450 in @ymatioun 's plot, the probability of an event being in the summed-bins is p ~ 4*1450/9600 ~ 0.60 (0,3,7,11-to-total ratio 0.60). \nThe output below is from [Matt Bognar's nice Binomial applet](https://homepage.divms.uiowa.edu/~mbognar/applets/bin.html). It shows that if all the series follow this pattern then the probability of a series with 18 nights, n=36 events, having 15 or fewer (ratio 0.42 or less) in the combined 0,3,7,11 bins is unlikely, only 2 %. The evenly-distributed case would peak at about (4/15)*36 ~ 9 or 10 events (ratio 0.26). So, if some series are \"0,3,7,11-quantized\" and some are evenly-distributed, the 0,3,7,11-to-total ratio histogram might show two peaks. \n\n", "2477667": "dan3dewey : thanks for sharing the insights on the possible impact of the human raters/tools used on the event labelling as well as spotting the possible binomial distribution of the labelled events across the 15-min intervals.\n\nI prepared the respective visualizations (see the [respective notebook](https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive#Deep-Dive-into-Training-Events), *'Event Minute Patterns'* subsection) that confirm your findings (for wakeup and onset events, separately).", "2477686": "w5833946 : yes, exactly. I prepared the visualizations (see the [respective notebook](https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive#Deep-Dive-into-Training-Events), *'Event Minute Patterns'* subsection) that confirm it.", "2478418": "Nice graphics in the notebook! It does seem that most series have the 0,3,7,11 minutes enhanced, with ratios of 0.5 to 0.8. These are much higher than the ~0.3 expected if the minutes were evenly distributed.", "2479116": "i noticed one more pattern: all series in training data begin exactly on 15 minute boundaries - at 0/15/30/45 minutes.", "2479204": "In fact, this tendency may be useful as a feature in ML." }, "source": "meta" }