![\"web](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F40584%2F1a82b6d9f78178ebcde1559db5b87e7b%2FScreenshot%202023-12-05%20at%208.25.42%20PM.png?generation=1701826083840049&alt=media\")
This was a neat competition and taught me several new about time series data; thanks Kaggle, the Child Mind Institute, and my team! Every team member contributed something vital to the final outcome.
\nI will talk just a bit about my part, and how you could get a pretty good result with a fairly simple setup:
\n
Predict many more than just 1 prediction per night: The metric was set up more like an AUC curve than just a straight accuracy prediction, so recognizing that was vital. In the end I selected up to (35 * # of days) predictions for both onset and wakeup. It definitely pays to really understand the metric!
Simple features: I noticed that enmo (and any feature based on it) made my training somewhat unstable, so I just ignored it (for my part; the team final solution did include it). In the end, my main model only had 3 features: anglez, hour, and anglez set to 0 if it was a \"repeat\" (The data repeated a lot when it was actually bad data, so I threw all that away in the 3rd feature)
Double GRU + CNN: My model was two GRU backbones - one took the full sequence and used a CNN to reduce its size by 12x before the GRU, and the other took the sequence binned into minutes (average,std,min,max per 12 steps). I added the outputs of the two GRUs and put that though a final head layer. The predictions were gaussians at the \"minute\" level (/ 12).
Post processing: In the end our team had quite a bit of post-processing (I'll link to that when it's posted), but my model mostly focused on selecting the main peaks and isolating them, but then including any \"sub\" peaks >= 5 minutes from the main peak. This ended up with up to 35 peaks per day (though often fewer, with a threshold cutoff). I used CV to figure out those params.
Finally, I still think there's a lot of room for a secondary model to take peaks like the below image, and do a secondary prediction… but I never got that working :)
\n![\"peaks\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F40584%2Fa1e54122df83bb14abb40f8abdd8f6b6%2FScreenshot%202023-12-05%20at%208.48.52%20PM.png?generation=1701827415174271&alt=media\")
Thanks again team! I'll post links to their writeups once they are live.
", "messages": [ { "id": "2550352", "postDate": "12/06/2023 01:54:31", "content": "This was a neat competition and taught me several new about time series data; thanks Kaggle, the Child Mind Institute, and my team! Every team member contributed something vital to the final outcome.
\nI will talk just a bit about my part, and how you could get a pretty good result with a fairly simple setup:
\n
Predict many more than just 1 prediction per night: The metric was set up more like an AUC curve than just a straight accuracy prediction, so recognizing that was vital. In the end I selected up to (35 * # of days) predictions for both onset and wakeup. It definitely pays to really understand the metric!
Simple features: I noticed that enmo (and any feature based on it) made my training somewhat unstable, so I just ignored it (for my part; the team final solution did include it). In the end, my main model only had 3 features: anglez, hour, and anglez set to 0 if it was a \"repeat\" (The data repeated a lot when it was actually bad data, so I threw all that away in the 3rd feature)
Double GRU + CNN: My model was two GRU backbones - one took the full sequence and used a CNN to reduce its size by 12x before the GRU, and the other took the sequence binned into minutes (average,std,min,max per 12 steps). I added the outputs of the two GRUs and put that though a final head layer. The predictions were gaussians at the \"minute\" level (/ 12).
Post processing: In the end our team had quite a bit of post-processing (I'll link to that when it's posted), but my model mostly focused on selecting the main peaks and isolating them, but then including any \"sub\" peaks >= 5 minutes from the main peak. This ended up with up to 35 peaks per day (though often fewer, with a threshold cutoff). I used CV to figure out those params.
Finally, I still think there's a lot of room for a secondary model to take peaks like the below image, and do a secondary prediction… but I never got that working :)
\n
Thanks again team! I'll post links to their writeups once they are live.
", "rawMarkdown": "This was a neat competition and taught me several new about time series data; thanks Kaggle, the Child Mind Institute, and my team! Every team member contributed something vital to the final outcome.\n\nI will talk just a bit about my part, and how you could get a pretty good result with a fairly simple setup:\n\n1. **EDA:** I built a mini web app so I could look at all of the series, and this turned out to be really vital for my understanding of the data. It also allowed me to see when I thought the data was labelled incorrectly (which happened quite often), or tag what I thought would be difficult series to predict.\n\n\n\n2. **Predict many more than just 1 prediction per night:** The metric was set up more like an AUC curve than just a straight accuracy prediction, so recognizing that was vital. In the end I selected _up to_ (35 * # of days) predictions for both onset and wakeup. It definitely pays to really understand the metric!\n\n3. **Simple features:** I noticed that enmo (and any feature based on it) made my training somewhat unstable, so I just ignored it (for my part; the team final solution did include it). In the end, my main model only had 3 features: anglez, hour, and anglez set to 0 if it was a \"repeat\" (The data repeated a lot when it was actually bad data, so I threw all that away in the 3rd feature)\n\n4. **Double GRU + CNN:** My model was two GRU backbones - one took the full sequence and used a CNN to reduce its size by 12x before the GRU, and the other took the sequence binned into minutes (average,std,min,max per 12 steps). I added the outputs of the two GRUs and put that though a final head layer. The predictions were gaussians at the \"minute\" level (/ 12).\n\n5. **Post processing:** In the end our team had quite a bit of post-processing (I'll link to that when it's posted), but my model mostly focused on selecting the main peaks and isolating them, but then including any \"sub\" peaks >= 5 minutes from the main peak. This ended up with up to 35 peaks per day (though often fewer, with a threshold cutoff). I used CV to figure out those params.\n\nFinally, I _still_ think there's a lot of room for a secondary model to take peaks like the below image, and do a secondary prediction... but I never got that working :)\n\n\n\n\nThanks again team! I'll post links to their writeups once they are live.", "votes": null }, { "id": "2555054", "postDate": "12/09/2023 16:20:42", "content": "Hi @chris62 did you correct labels on the entire dataset? If so did this improve performance? I also found that the data was incorrectly labeled in a significant number of cases. But I didn't continue fixing labels because
\na) it was very time consuming
\nb) I found there was a lot of inconsistency in terms of how the data was labeled (how close to the event the event was marked), most likely due to differences in how different labelers labeled the data. So I became less sure in my ability to label the data or capture how it would have been labeled.
I didn't do the entire dataset, no. I started with just the very worst ones (the ones that were very obviously wrong or missing; that was maybe 20 or so series), and that did help, yes.
\nI then did about 20 more series, where I adjusted events that weren't quite as obvious but I thought looked wrong. That had much less of an effect (in fact, it sometimes had a negative effect depending on training), so I didn't do any more.
\nIn particular, yes: there was a lot of labelling inconsistency. I tried \"fixing\" some of that - but since the test set was labeled like the train set (presumably), that's when it started to hurt performance. I do think there is a lot of room to be more consistent though (better rules/examples for the labelers perhaps).
\nI saw that other teams simply removed the 10-20 sets with bad data though, and that would have probably worked about as well :) so that was the easy way to do it.
", "rawMarkdown": "I didn't do the entire dataset, no. I started with just the very worst ones (the ones that were very obviously wrong or missing; that was maybe 20 or so series), and that did help, yes. \n\nI then did about 20 more series, where I adjusted events that weren't quite as obvious but I thought looked wrong. That had much less of an effect (in fact, it sometimes had a negative effect depending on training), so I didn't do any more.\n\nIn particular, yes: there was a lot of labelling inconsistency. I tried \"fixing\" some of that - but since the test set was labeled like the train set (presumably), that's when it started to hurt performance. I do think there is a lot of room to be more consistent though (better rules/examples for the labelers perhaps).\n\nI saw that other teams simply removed the 10-20 sets with bad data though, and that would have probably worked about as well :) so that was the easy way to do it.", "votes": null } ], "comments": [ { "id": 2555054, "author_name": "aamster", "author_url": "", "post_date": "12/09/2023 16:20:42", "content": "Hi @chris62 did you correct labels on the entire dataset? If so did this improve performance? I also found that the data was incorrectly labeled in a significant number of cases. But I didn't continue fixing labels because
\na) it was very time consuming
\nb) I found there was a lot of inconsistency in terms of how the data was labeled (how close to the event the event was marked), most likely due to differences in how different labelers labeled the data. So I became less sure in my ability to label the data or capture how it would have been labeled.
I didn't do the entire dataset, no. I started with just the very worst ones (the ones that were very obviously wrong or missing; that was maybe 20 or so series), and that did help, yes.
\nI then did about 20 more series, where I adjusted events that weren't quite as obvious but I thought looked wrong. That had much less of an effect (in fact, it sometimes had a negative effect depending on training), so I didn't do any more.
\nIn particular, yes: there was a lot of labelling inconsistency. I tried \"fixing\" some of that - but since the test set was labeled like the train set (presumably), that's when it started to hurt performance. I do think there is a lot of room to be more consistent though (better rules/examples for the labelers perhaps).
\nI saw that other teams simply removed the 10-20 sets with bad data though, and that would have probably worked about as well :) so that was the easy way to do it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2550352": "This was a neat competition and taught me several new about time series data; thanks Kaggle, the Child Mind Institute, and my team! Every team member contributed something vital to the final outcome.\n\nI will talk just a bit about my part, and how you could get a pretty good result with a fairly simple setup:\n\n1. **EDA:** I built a mini web app so I could look at all of the series, and this turned out to be really vital for my understanding of the data. It also allowed me to see when I thought the data was labelled incorrectly (which happened quite often), or tag what I thought would be difficult series to predict.\n\n\n\n2. **Predict many more than just 1 prediction per night:** The metric was set up more like an AUC curve than just a straight accuracy prediction, so recognizing that was vital. In the end I selected _up to_ (35 * # of days) predictions for both onset and wakeup. It definitely pays to really understand the metric!\n\n3. **Simple features:** I noticed that enmo (and any feature based on it) made my training somewhat unstable, so I just ignored it (for my part; the team final solution did include it). In the end, my main model only had 3 features: anglez, hour, and anglez set to 0 if it was a \"repeat\" (The data repeated a lot when it was actually bad data, so I threw all that away in the 3rd feature)\n\n4. **Double GRU + CNN:** My model was two GRU backbones - one took the full sequence and used a CNN to reduce its size by 12x before the GRU, and the other took the sequence binned into minutes (average,std,min,max per 12 steps). I added the outputs of the two GRUs and put that though a final head layer. The predictions were gaussians at the \"minute\" level (/ 12).\n\n5. **Post processing:** In the end our team had quite a bit of post-processing (I'll link to that when it's posted), but my model mostly focused on selecting the main peaks and isolating them, but then including any \"sub\" peaks >= 5 minutes from the main peak. This ended up with up to 35 peaks per day (though often fewer, with a threshold cutoff). I used CV to figure out those params.\n\nFinally, I _still_ think there's a lot of room for a secondary model to take peaks like the below image, and do a secondary prediction... but I never got that working :)\n\n\n\n\nThanks again team! I'll post links to their writeups once they are live.", "2555054": "Hi @chris62 did you correct labels on the entire dataset? If so did this improve performance? I also found that the data was incorrectly labeled in a significant number of cases. But I didn't continue fixing labels because\na) it was very time consuming\nb) I found there was a lot of inconsistency in terms of how the data was labeled (how close to the event the event was marked), most likely due to differences in how different labelers labeled the data. So I became less sure in my ability to label the data or capture how it would have been labeled.", "2555081": "I didn't do the entire dataset, no. I started with just the very worst ones (the ones that were very obviously wrong or missing; that was maybe 20 or so series), and that did help, yes. \n\nI then did about 20 more series, where I adjusted events that weren't quite as obvious but I thought looked wrong. That had much less of an effect (in fact, it sometimes had a negative effect depending on training), so I didn't do any more.\n\nIn particular, yes: there was a lot of labelling inconsistency. I tried \"fixing\" some of that - but since the test set was labeled like the train set (presumably), that's when it started to hurt performance. I do think there is a lot of room to be more consistent though (better rules/examples for the labelers perhaps).\n\nI saw that other teams simply removed the 10-20 sets with bad data though, and that would have probably worked about as well :) so that was the easy way to do it." }, "source": "meta" }