First of all, thanks my teammate @robikscube . It was a fun ride. We have noticed that Public LB is not very trustworthy and focused on our CV. We had around 0.826 CV at the end.
\nOur main model is a wavenet with some modifications. It works on a sequence of minutely aggregated 3 days.
\n (nn.Module):\n ():\n (SleepModel, self).__init__()\n emb_size = \n\n self.minute_emb = nn.Embedding(, embedding_dim=emb_size)\n\n self.bn = nn.BatchNorm1d(inch-)\n self.wave_block1 = Wave_Block(inch-+emb_size, , , kernel_size)\n self.wave_block2 = Wave_Block(, , , kernel_size, base=)\n self.wave_block3 = Wave_Block(, , , kernel_size)\n self.wave_block4 = nn.Sequential(nn.Conv1d(, , kernel_size=, dilation=DAY_LEN//), \n nn.BatchNorm1d(), nn.LeakyReLU(),\n nn.Conv1d(, , kernel_size=, dilation=DAY_LEN//), \n nn.BatchNorm1d(), nn.LeakyReLU()\n )\n self.top = nn.Conv1d(, , kernel_size=)\n self.top2 = nn.Conv1d(, , kernel_size=)\n\n\n self.gn1 = nn.GroupNorm(, )\n self.gn2 = nn.GroupNorm(, )\n\n self.avgpool = nn.AvgPool1d(kernel_size=DAY_LEN+, padding=DAY_LEN//, stride=)\n self.maxpool = nn.MaxPool1d(kernel_size=DAY_LEN+, padding=DAY_LEN//, stride=)\n\n\n ():\n x = torch.cat([x[:, -DAY_LEN:]*, x, x[:, :DAY_LEN]*], axis=)\n x = x.permute(, , )\n minute, x = x[:, -, :], x[:, :-, :]\n\n x[:, -, :] += minute / \n x = self.bn(x)\n\n minute_emb = self.minute_emb(torch.fmod(minute, ).long()) \n x = torch.cat([x, minute_emb.permute(, , )], axis=)\n\n x = self.wave_block1(x)\n x = self.gn1(x)\n x = self.wave_block2(x)\n x = self.gn2(x)\n x = self.wave_block3(x)\n\n x = torch.cat([x, self.avgpool(x[:, :]), self.maxpool(x[:, :])], axis=)\n\n x = self.wave_block4(x)\n x, x2 = self.top(x), self.top2(x)\n x, x2\nIts features are: \n[\"target\", \"idx\", \"anglez_mean\", \"anglez_std\", \n \"enmo_mean\", \"enmo_std\", \n \"same_anglez_prev_min\", \"same_anglez_next_min\"] +\n volatility_cols + [\"hour\", \"minute\"]\n
\nVolatility columns are median over anglez absolute difference on 3 different time windows. (5, 30, 480 steps)
\nSame anglez prev and next features are difference of anglez with the last day same minute.
Training details:
\nPrediction details:
\nFirst of all, thanks my teammate @robikscube . It was a fun ride. We have noticed that Public LB is not very trustworthy and focused on our CV. We had around 0.826 CV at the end.
\nOur main model is a wavenet with some modifications. It works on a sequence of minutely aggregated 3 days.
\n (nn.Module):\n ():\n (SleepModel, self).__init__()\n emb_size = \n\n self.minute_emb = nn.Embedding(, embedding_dim=emb_size)\n\n self.bn = nn.BatchNorm1d(inch-)\n self.wave_block1 = Wave_Block(inch-+emb_size, , , kernel_size)\n self.wave_block2 = Wave_Block(, , , kernel_size, base=)\n self.wave_block3 = Wave_Block(, , , kernel_size)\n self.wave_block4 = nn.Sequential(nn.Conv1d(, , kernel_size=, dilation=DAY_LEN//), \n nn.BatchNorm1d(), nn.LeakyReLU(),\n nn.Conv1d(, , kernel_size=, dilation=DAY_LEN//), \n nn.BatchNorm1d(), nn.LeakyReLU()\n )\n self.top = nn.Conv1d(, , kernel_size=)\n self.top2 = nn.Conv1d(, , kernel_size=)\n\n\n self.gn1 = nn.GroupNorm(, )\n self.gn2 = nn.GroupNorm(, )\n\n self.avgpool = nn.AvgPool1d(kernel_size=DAY_LEN+, padding=DAY_LEN//, stride=)\n self.maxpool = nn.MaxPool1d(kernel_size=DAY_LEN+, padding=DAY_LEN//, stride=)\n\n\n ():\n x = torch.cat([x[:, -DAY_LEN:]*, x, x[:, :DAY_LEN]*], axis=)\n x = x.permute(, , )\n minute, x = x[:, -, :], x[:, :-, :]\n\n x[:, -, :] += minute / \n x = self.bn(x)\n\n minute_emb = self.minute_emb(torch.fmod(minute, ).long()) \n x = torch.cat([x, minute_emb.permute(, , )], axis=)\n\n x = self.wave_block1(x)\n x = self.gn1(x)\n x = self.wave_block2(x)\n x = self.gn2(x)\n x = self.wave_block3(x)\n\n x = torch.cat([x, self.avgpool(x[:, :]), self.maxpool(x[:, :])], axis=)\n\n x = self.wave_block4(x)\n x, x2 = self.top(x), self.top2(x)\n x, x2\nIts features are: \n[\"target\", \"idx\", \"anglez_mean\", \"anglez_std\", \n \"enmo_mean\", \"enmo_std\", \n \"same_anglez_prev_min\", \"same_anglez_next_min\"] +\n volatility_cols + [\"hour\", \"minute\"]\n
\nVolatility columns are median over anglez absolute difference on 3 different time windows. (5, 30, 480 steps)
\nSame anglez prev and next features are difference of anglez with the last day same minute.
Training details:
\nPrediction details:
\nHuge congrats @aerdem4 and @robikscube !!! Great ending. Looking forward to read your solution in detail later on.
", "rawMarkdown": "Huge congrats @aerdem4 and @robikscube !!! Great ending. Looking forward to read your solution in detail later on.", "votes": null }, { "id": "2550348", "postDate": "12/06/2023 01:47:47", "content": "\n\nWavenet and Some Tricks
\n
and
\n\n\nStacking with LightGBM
\n
haha, sounds like Ion Comp Rob and Ahmet. I also tried these right away 😀 Congratulations on great gold finish and jump upward on private LB!
\nWhat made you conclude We have noticed that Public LB is not very trustworthy? (I think I focused on public LB too much and I only used 1 fold of 10 fold CV so my model didn't generalize to private test very well).
Thank you @cdeotte , congrats to you too!
\nWe were using 4 folds locally and we noticed how sensitive the folds are even though each fold was larger than Public LB size. Public LB score was also much lower than our worst fold, which indicates that labels were probably very noisy. Simple improvements like averaging more models etc was harming Public LB time to time. So we decided that Public LB was very noisy and mostly ignored it.
\nWhy did you use only 1 fold? Were your models running slow?
", "rawMarkdown": "Thank you @cdeotte , congrats to you too!\n\nWe were using 4 folds locally and we noticed how sensitive the folds are even though each fold was larger than Public LB size. Public LB score was also much lower than our worst fold, which indicates that labels were probably very noisy. Simple improvements like averaging more models etc was harming Public LB time to time. So we decided that Public LB was very noisy and mostly ignored it.\n\nWhy did you use only 1 fold? Were your models running slow?", "votes": null }, { "id": "2550602", "postDate": "12/06/2023 06:15:00", "content": "Very nice! Used a WaveNet-Grunet based approach myself but training was significantly slower on kaggle-only resources and a non resampled inout. In addition, was limited to a much smaller feature depth and the use of only the raw data 2 features (my own fault for not resampling dataset). Glad to see feature engineering worked, especially the volatility!
", "rawMarkdown": "Very nice! Used a WaveNet-Grunet based approach myself but training was significantly slower on kaggle-only resources and a non resampled inout. In addition, was limited to a much smaller feature depth and the use of only the raw data 2 features (my own fault for not resampling dataset). Glad to see feature engineering worked, especially the volatility!", "votes": null }, { "id": "2551115", "postDate": "12/06/2023 14:08:37", "content": "Really happy to have teamed with @aerdem4 in this competition - I learned a lot from him and it was a lot of fun! Thanks to the hosts and congrats to everyone who participated. In this competition I was reminded of how important CV is, and how elegant processes can often outperform complicated ones.
\nI streamed on my youtube/twitch channel a debrief for this competition you can watch here: https://www.youtube.com/watch?v=cuwNhIn4nco
\nPrior to our team merge I had a pipeline that had kept me in the top 10 of the public LB for the first few months of the competition. I spent most of the first month in the top 2 spots but then was distracted by the DefCon competition for a bit in the middle 😁 . Later after our team merged my pipeline was underperforming Ahmet's wavenet model in CV so I focused more on post processing and modifying my model to match his performance. Our final solution did ensemble both models, but with the majority of the weight given to Wavenet (90/10) and very small gain in CV and LB (0.0001). On its own this pipeline would've ended up in ~24th place on the final LB (0.814 private LB). I'll briefly describe it here:
\nvalue_counts per series were really helpful in identifying these offline cycles.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F644036%2Fb2e8b10fcffc5c443c99b4879136e1bf%2Ftarget-process.png?generation=1701870372076768&alt=media\")
find_peaks with optimization process for finding the optimal width and height parameters. 2) Code to identify when two sleep events occurred in a day and suppressing the smaller one like this:![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F644036%2F3dd0620fbf4ba9c5eeb99aa434337065%2Fpostprocess.png?generation=1701870571658891&alt=media\")
We tried a lot of ideas to post-process our solution to account for the strange pattern in the label minutes, but didn't find the magic. Excited to read if/how other teams were able to do this. My best guess was that this pattern was due to the labeling software and would depend on how zoomed in the labeler was when marking the event.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F644036%2F5048cb8e16c088afcab921ecc0f525cf%2Fminutes-offset.png?generation=1701870985004424&alt=media\")
Couldn't find relationship between this and series ids:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F644036%2F2cce3c8d0802105ce0caeec57e9c9d79%2Fonset-heatmap.png?generation=1701870918914631&alt=media\")
Great Advice
", "rawMarkdown": "Great Advice", "votes": null }, { "id": "2551399", "postDate": "12/06/2023 17:16:42", "content": "Those are smart observations. Using full CV was a good idea.
\n\n\nWhy did you use only 1 fold? Were your models running slow?
\n
I used 1 fold to accelerate experiments because I was enjoying creating and trying different DL model architectures. In two months, I trained 1000+ different architectures! haha 😀 It was a good opportunity to learn how well CNN, RNN, and Transformer can generate features on their own, and work together.
\nI tried all sorts of crazy stuff like two tower models that each focused on either onset or wakeup, then combined their embeddings before the final layers. Or models that zoomed in on areas of interest and increased resolution there etc etc. I also designed architectures and features to try to exploit weird patterns in data. In the end simple architectures with fewer features worked best.
\nEach model was trained 15 epochs and took between 15-30 minutes to train, so they were fast. I was only using 2xV100 GPU and continuously tried new models each day.
", "rawMarkdown": "Those are smart observations. Using full CV was a good idea.\n\n>Why did you use only 1 fold? Were your models running slow?\n\nI used 1 fold to accelerate experiments because I was enjoying creating and trying different DL model architectures. In two months, I trained 1000+ different architectures! haha 😀 It was a good opportunity to learn how well CNN, RNN, and Transformer can generate features on their own, and work together.\n\nI tried all sorts of crazy stuff like two tower models that each focused on either onset or wakeup, then combined their embeddings before the final layers. Or models that zoomed in on areas of interest and increased resolution there etc etc. I also designed architectures and features to try to exploit weird patterns in data. In the end simple architectures with fewer features worked best.\n\nEach model was trained 15 epochs and took between 15-30 minutes to train, so they were fast. I was only using 2xV100 GPU and continuously tried new models each day.", "votes": null }, { "id": "2551421", "postDate": "12/06/2023 17:27:36", "content": "Congrats Rob, great job! I like your analysis of different target shapes. Predicting shapes instead of predicting a single target was an interesting new idea in this comp.
\nYou joined early and jumped to top of LB. Were you using \"predict shape\" for those models too? Where did you learn this idea? I didn't know this idea until I started reading public discussion and notebook in this comp.
\n\n\nWe tried a lot of ideas to post-process our solution to account for the strange pattern in the label minutes, but didn't find the magic.
\n
I found a few ways to exploit this, perhaps I will make a post. My model had a multiclass head (i.e. predict which minute of the day was onset) where the probabilities added up to 1 for the 1440 minutes in a single day. I used RELU activations before softmax so the logits are all non-negative. We can infer the model and extract the logits before softmax. Then we can Bayesian update the logits by multiplying the four rare values of minute%15 that did not occur for target by 50%. Then we can apply softmax after that. This boost CV LB a bunch for my model that only used anglez and enmo (without hour nor minute) as features.
Alternatively, when i added the feature minute%15 as input during training, the model learned it on its own, but the Bayesian update above worked better. Maybe because the model was trained on the harder task of not knowing the hour nor minute.
Thanks Chris! Congrats to you also on the solo gold.
\nThe multiclass head is a really clever idea. We tried some manual post processing and also we used the minute%15 as a feature in both of our models. That feature alone let the model learn the pattern, but I don't think it optimally selected for the AP metric. I've only skimmed the 1st and 2nd place writeups but it seems like they got a big boost from modeling this pattern. Definately not what I expected this competition to depend on, but it was an interesting problem!
Hi, everybody. It was a nice job from a nice team. I encountered a problem on sleep detection and would love to hear your solutions. I am attempting to do almost the same thing with the same data, but there are some differences. My data is much noisier due to 1-minute intervals and low accuracy.
\nI am working on a web-based solution, so it needs to be very CPU-friendly. I came up with the idea of implementing a score-based system. First, I normalize the heart rate, step, and activity data, then multiply each by a related weight, and finally, add up scores for each point. I then define a threshold score to classify as sleep or wake. However, all the weight parameters and the threshold are defined manually.
\nDoes anyone have an idea on how to improve this or how to use ML to find the best parameters for this score-based system? I am generating my own data and labeling it, so I have a hard time accessing large data as well.
", "rawMarkdown": "Hi, everybody. It was a nice job from a nice team. I encountered a problem on sleep detection and would love to hear your solutions. I am attempting to do almost the same thing with the same data, but there are some differences. My data is much noisier due to 1-minute intervals and low accuracy.\n\nI am working on a web-based solution, so it needs to be very CPU-friendly. I came up with the idea of implementing a score-based system. First, I normalize the heart rate, step, and activity data, then multiply each by a related weight, and finally, add up scores for each point. I then define a threshold score to classify as sleep or wake. However, all the weight parameters and the threshold are defined manually.\n\nDoes anyone have an idea on how to improve this or how to use ML to find the best parameters for this score-based system? I am generating my own data and labeling it, so I have a hard time accessing large data as well.", "votes": null } ], "comments": [ { "id": 2550287, "author_name": "snnclsr", "author_url": "", "post_date": "12/06/2023 00:35:50", "content": "Huge congrats @aerdem4 and @robikscube !!! Great ending. Looking forward to read your solution in detail later on.
", "votes": null, "replies": [] }, { "id": 2550348, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 01:47:47", "content": "\n\nWavenet and Some Tricks
\n
and
\n\n\nStacking with LightGBM
\n
haha, sounds like Ion Comp Rob and Ahmet. I also tried these right away 😀 Congratulations on great gold finish and jump upward on private LB!
\nWhat made you conclude We have noticed that Public LB is not very trustworthy? (I think I focused on public LB too much and I only used 1 fold of 10 fold CV so my model didn't generalize to private test very well).
Thank you @cdeotte , congrats to you too!
\nWe were using 4 folds locally and we noticed how sensitive the folds are even though each fold was larger than Public LB size. Public LB score was also much lower than our worst fold, which indicates that labels were probably very noisy. Simple improvements like averaging more models etc was harming Public LB time to time. So we decided that Public LB was very noisy and mostly ignored it.
\nWhy did you use only 1 fold? Were your models running slow?
", "votes": null, "replies": [ { "id": 2551399, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 17:16:42", "content": "Those are smart observations. Using full CV was a good idea.
\n\n\nWhy did you use only 1 fold? Were your models running slow?
\n
I used 1 fold to accelerate experiments because I was enjoying creating and trying different DL model architectures. In two months, I trained 1000+ different architectures! haha 😀 It was a good opportunity to learn how well CNN, RNN, and Transformer can generate features on their own, and work together.
\nI tried all sorts of crazy stuff like two tower models that each focused on either onset or wakeup, then combined their embeddings before the final layers. Or models that zoomed in on areas of interest and increased resolution there etc etc. I also designed architectures and features to try to exploit weird patterns in data. In the end simple architectures with fewer features worked best.
\nEach model was trained 15 epochs and took between 15-30 minutes to train, so they were fast. I was only using 2xV100 GPU and continuously tried new models each day.
", "votes": null, "replies": [] } ] } ] }, { "id": 2550602, "author_name": "exjustice", "author_url": "", "post_date": "12/06/2023 06:15:00", "content": "Very nice! Used a WaveNet-Grunet based approach myself but training was significantly slower on kaggle-only resources and a non resampled inout. In addition, was limited to a much smaller feature depth and the use of only the raw data 2 features (my own fault for not resampling dataset). Glad to see feature engineering worked, especially the volatility!
", "votes": null, "replies": [] }, { "id": 2551115, "author_name": "robikscube", "author_url": "", "post_date": "12/06/2023 14:08:37", "content": "Really happy to have teamed with @aerdem4 in this competition - I learned a lot from him and it was a lot of fun! Thanks to the hosts and congrats to everyone who participated. In this competition I was reminded of how important CV is, and how elegant processes can often outperform complicated ones.
\nI streamed on my youtube/twitch channel a debrief for this competition you can watch here: https://www.youtube.com/watch?v=cuwNhIn4nco
\nPrior to our team merge I had a pipeline that had kept me in the top 10 of the public LB for the first few months of the competition. I spent most of the first month in the top 2 spots but then was distracted by the DefCon competition for a bit in the middle 😁 . Later after our team merged my pipeline was underperforming Ahmet's wavenet model in CV so I focused more on post processing and modifying my model to match his performance. Our final solution did ensemble both models, but with the majority of the weight given to Wavenet (90/10) and very small gain in CV and LB (0.0001). On its own this pipeline would've ended up in ~24th place on the final LB (0.814 private LB). I'll briefly describe it here:
\nvalue_counts per series were really helpful in identifying these offline cycles.find_peaks with optimization process for finding the optimal width and height parameters. 2) Code to identify when two sleep events occurred in a day and suppressing the smaller one like this:We tried a lot of ideas to post-process our solution to account for the strange pattern in the label minutes, but didn't find the magic. Excited to read if/how other teams were able to do this. My best guess was that this pattern was due to the labeling software and would depend on how zoomed in the labeler was when marking the event.
\n
Couldn't find relationship between this and series ids:
\n
Congrats Rob, great job! I like your analysis of different target shapes. Predicting shapes instead of predicting a single target was an interesting new idea in this comp.
\nYou joined early and jumped to top of LB. Were you using \"predict shape\" for those models too? Where did you learn this idea? I didn't know this idea until I started reading public discussion and notebook in this comp.
\n\n\nWe tried a lot of ideas to post-process our solution to account for the strange pattern in the label minutes, but didn't find the magic.
\n
I found a few ways to exploit this, perhaps I will make a post. My model had a multiclass head (i.e. predict which minute of the day was onset) where the probabilities added up to 1 for the 1440 minutes in a single day. I used RELU activations before softmax so the logits are all non-negative. We can infer the model and extract the logits before softmax. Then we can Bayesian update the logits by multiplying the four rare values of minute%15 that did not occur for target by 50%. Then we can apply softmax after that. This boost CV LB a bunch for my model that only used anglez and enmo (without hour nor minute) as features.
Alternatively, when i added the feature minute%15 as input during training, the model learned it on its own, but the Bayesian update above worked better. Maybe because the model was trained on the harder task of not knowing the hour nor minute.
Thanks Chris! Congrats to you also on the solo gold.
\nThe multiclass head is a really clever idea. We tried some manual post processing and also we used the minute%15 as a feature in both of our models. That feature alone let the model learn the pattern, but I don't think it optimally selected for the AP metric. I've only skimmed the 1st and 2nd place writeups but it seems like they got a big boost from modeling this pattern. Definately not what I expected this competition to depend on, but it was an interesting problem!
Great Advice
", "votes": null, "replies": [] }, { "id": 2552306, "author_name": "rezamm79", "author_url": "", "post_date": "12/07/2023 11:22:56", "content": "Hi, everybody. It was a nice job from a nice team. I encountered a problem on sleep detection and would love to hear your solutions. I am attempting to do almost the same thing with the same data, but there are some differences. My data is much noisier due to 1-minute intervals and low accuracy.
\nI am working on a web-based solution, so it needs to be very CPU-friendly. I came up with the idea of implementing a score-based system. First, I normalize the heart rate, step, and activity data, then multiply each by a related weight, and finally, add up scores for each point. I then define a threshold score to classify as sleep or wake. However, all the weight parameters and the threshold are defined manually.
\nDoes anyone have an idea on how to improve this or how to use ML to find the best parameters for this score-based system? I am generating my own data and labeling it, so I have a hard time accessing large data as well.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2550283": "First of all, thanks my teammate @robikscube . It was a fun ride. We have noticed that Public LB is not very trustworthy and focused on our CV. We had around 0.826 CV at the end.\n\n**Our main model is a wavenet with some modifications.** It works on a sequence of minutely aggregated 3 days.\n\n```python\nclass SleepModel(nn.Module):\n def __init__(self, inch, kernel_size):\n super(SleepModel, self).__init__()\n emb_size = 4\n \n self.minute_emb = nn.Embedding(15, embedding_dim=emb_size)\n \n self.bn = nn.BatchNorm1d(inch-1)\n self.wave_block1 = Wave_Block(inch-1+emb_size, 32, 8, kernel_size)\n self.wave_block2 = Wave_Block(32, 32, 7, kernel_size, base=2.25)\n self.wave_block3 = Wave_Block(32, 64, 4, kernel_size)\n self.wave_block4 = nn.Sequential(nn.Conv1d(128, 64, kernel_size=5, dilation=DAY_LEN//6), \n nn.BatchNorm1d(64), nn.LeakyReLU(),\n nn.Conv1d(64, 64, kernel_size=5, dilation=DAY_LEN//3), \n nn.BatchNorm1d(64), nn.LeakyReLU()\n )\n self.top = nn.Conv1d(64, 3, kernel_size=1)\n self.top2 = nn.Conv1d(64, 2, kernel_size=1)\n \n \n self.gn1 = nn.GroupNorm(4, 32)\n self.gn2 = nn.GroupNorm(4, 32)\n \n self.avgpool = nn.AvgPool1d(kernel_size=DAY_LEN+1, padding=DAY_LEN//2, stride=1)\n self.maxpool = nn.MaxPool1d(kernel_size=DAY_LEN+1, padding=DAY_LEN//2, stride=1)\n \n\n def forward(self, x):\n x = torch.cat([x[:, -DAY_LEN:]*0, x, x[:, :DAY_LEN]*0], axis=1)\n x = x.permute(0, 2, 1)\n minute, x = x[:, -1, :], x[:, :-1, :]\n \n x[:, -1, :] += minute / 60 # hour + minute/60\n x = self.bn(x)\n \n minute_emb = self.minute_emb(torch.fmod(minute, 15).long()) # minute % 15 is an important feature\n x = torch.cat([x, minute_emb.permute(0, 2, 1)], axis=1)\n\n x = self.wave_block1(x)\n x = self.gn1(x)\n x = self.wave_block2(x)\n x = self.gn2(x)\n x = self.wave_block3(x)\n \n x = torch.cat([x, self.avgpool(x[:, :32]), self.maxpool(x[:, 32:])], axis=1)\n\n x = self.wave_block4(x)\n x, x2 = self.top(x), self.top2(x)\n return x, x2\n```\n\nIts **features** are: ```\n[\"target\", \"idx\", \"anglez_mean\", \"anglez_std\", \n \"enmo_mean\", \"enmo_std\", \n \"same_anglez_prev_min\", \"same_anglez_next_min\"] +\n volatility_cols + [\"hour\", \"minute\"]\n```\nVolatility columns are median over anglez absolute difference on 3 different time windows. (5, 30, 480 steps)\nSame anglez prev and next features are difference of anglez with the last day same minute.\n\n**Training details:**\n* Target is also set to 1 for the adjacent minute because we predict with +3 step offset to cover the next minute too.\n* Ignoring near misses in loss: 2nd and 3rd minutes around the target are set to -1. SO that we dont penalize positive predictions there.\n* 2 heads (1 for actual target -> categorical crossentropy, 1 for 15min windows -> BCE)\n* Online Hard Example Mining with 50%\n* 6 epochs with diminishing LR. Each epoch takes 18 seconds on RTX3090. (and submission was taking 30 minutes in total)\n* 2 iterations with different seed for diversity. 1st iteration excludes some bad series and 2nd iteration has very little augmentation.\n\n**Prediction details:**\n* 1 day sliding predictions with 3 days span. Only the middle day is used.\n* Postprocessing: Starting from the max prediction, we add their location as prediction. Score is their probability and 2nd highest probability within +-4 minute window. We set these windows to zero. We multiply surrounding +-20 minutes windows by 0.5 and continue. Later, we sample very low probability predictions for +-18 steps span for predictions with high score.\n* Stacking with LightGBM: Lightgm with a lot of median absolute anglez difference features are used. NN probabilities and time to/till onset/wakeup features using this probabilities are added. AUC got significantly improved but the metric is improved very little. (0.001)\n* Ensembling: We had another model with LSTM+Transformer. We added it to get 0.001 improvement.", "2550287": "Huge congrats @aerdem4 and @robikscube !!! Great ending. Looking forward to read your solution in detail later on.", "2550348": ">Wavenet and Some Tricks\n\nand\n\n>Stacking with LightGBM\n\nhaha, sounds like [Ion Comp][1] Rob and Ahmet. I also tried these right away 😀 Congratulations on great gold finish and jump upward on private LB! \n\nWhat made you conclude `We have noticed that Public LB is not very trustworthy`? (I think I focused on public LB too much and I only used 1 fold of 10 fold CV so my model didn't generalize to private test very well).\n\n[1]: https://www.kaggle.com/competitions/liverpool-ion-switching/leaderboard", "2550584": "Thank you @cdeotte , congrats to you too!\n\nWe were using 4 folds locally and we noticed how sensitive the folds are even though each fold was larger than Public LB size. Public LB score was also much lower than our worst fold, which indicates that labels were probably very noisy. Simple improvements like averaging more models etc was harming Public LB time to time. So we decided that Public LB was very noisy and mostly ignored it.\n\nWhy did you use only 1 fold? Were your models running slow?", "2550602": "Very nice! Used a WaveNet-Grunet based approach myself but training was significantly slower on kaggle-only resources and a non resampled inout. In addition, was limited to a much smaller feature depth and the use of only the raw data 2 features (my own fault for not resampling dataset). Glad to see feature engineering worked, especially the volatility!", "2551115": "Really happy to have teamed with @aerdem4 in this competition - I learned a lot from him and it was a lot of fun! Thanks to the hosts and congrats to everyone who participated. In this competition I was reminded of how important CV is, and how elegant processes can often outperform complicated ones.\n\nI streamed on my youtube/twitch channel a debrief for this competition you can watch here: https://www.youtube.com/watch?v=cuwNhIn4nco\n\nPrior to our team merge I had a pipeline that had kept me in the top 10 of the public LB for the first few months of the competition. I spent most of the first month in the top 2 spots but then was distracted by the DefCon competition for a bit in the middle 😁 . Later after our team merged my pipeline was underperforming Ahmet's wavenet model in CV so I focused more on post processing and modifying my model to match his performance. Our final solution did ensemble both models, but with the majority of the weight given to Wavenet (90/10) and very small gain in CV and LB (0.0001). On its own this pipeline would've ended up in ~24th place on the final LB (0.814 private LB). I'll briefly describe it here:\n\n- **Preprocessing and cycle filtering** - I split the data into 24 hour cycles optimized to time periods that would capture full sleep periods. I then trained an XGB model to predict \"offline\" cycles for prefiltering. This model had a very high recall (0.99-1.0) depending on the threshold selected. Features like `value_counts` per series were really helpful in identifying these offline cycles.\n- **Target processing** - Tried many different ramps applied to the target (linear, stepwise, quadratic). Quadratic seemed to work best with a width of ~24 steps:\n\n- **Main model** : 1DCNN-TranformerEncoder-LSTM model based on the winning solution of the Parkinson's Gait competition -> https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416026 with addition of some CNN layers at the beginning and tuned for this dataset.\n- **Post processing** - 1) Scypi's `find_peaks` with optimization process for finding the optimal `width` and `height` parameters. 2) Code to identify when two sleep events occurred in a day and suppressing the smaller one like this:\n\n\nWe tried a lot of ideas to post-process our solution to account for the strange pattern in the label minutes, but didn't find the magic. Excited to read if/how other teams were able to do this. My best guess was that this pattern was due to the labeling software and would depend on how zoomed in the labeler was when marking the event.\n\n\n\nCouldn't find relationship between this and series ids:\n", "2551369": "Great Advice", "2551399": "Those are smart observations. Using full CV was a good idea.\n\n>Why did you use only 1 fold? Were your models running slow?\n\nI used 1 fold to accelerate experiments because I was enjoying creating and trying different DL model architectures. In two months, I trained 1000+ different architectures! haha 😀 It was a good opportunity to learn how well CNN, RNN, and Transformer can generate features on their own, and work together.\n\nI tried all sorts of crazy stuff like two tower models that each focused on either onset or wakeup, then combined their embeddings before the final layers. Or models that zoomed in on areas of interest and increased resolution there etc etc. I also designed architectures and features to try to exploit weird patterns in data. In the end simple architectures with fewer features worked best.\n\nEach model was trained 15 epochs and took between 15-30 minutes to train, so they were fast. I was only using 2xV100 GPU and continuously tried new models each day.", "2551421": "Congrats Rob, great job! I like your analysis of different target shapes. Predicting shapes instead of predicting a single target was an interesting new idea in this comp. \n\nYou joined early and jumped to top of LB. Were you using \"predict shape\" for those models too? Where did you learn this idea? I didn't know this idea until I started reading public discussion and notebook in this comp.\n\n>We tried a lot of ideas to post-process our solution to account for the strange pattern in the label minutes, but didn't find the magic.\n\nI found a few ways to exploit this, perhaps I will make a post. My model had a multiclass head (i.e. predict which minute of the day was onset) where the probabilities added up to 1 for the 1440 minutes in a single day. I used RELU activations before softmax so the logits are all non-negative. We can infer the model and extract the logits before softmax. Then we can Bayesian update the logits by multiplying the four rare values of `minute%15` that did not occur for target by 50%. Then we can apply softmax after that. This boost CV LB a bunch for my model that only used anglez and enmo (without hour nor minute) as features.\n\nAlternatively, when i added the feature `minute%15` as input during training, the model learned it on its own, but the Bayesian update above worked better. Maybe because the model was trained on the harder task of not knowing the hour nor minute.", "2551590": "Thanks Chris! Congrats to you also on the solo gold.\n\nThe multiclass head is a really clever idea. We tried some manual post processing and also we used the `minute%15` as a feature in both of our models. That feature alone let the model learn the pattern, but I don't think it optimally selected for the AP metric. I've only skimmed the 1st and 2nd place writeups but it seems like they got a big boost from modeling this pattern. Definately not what I expected this competition to depend on, but it was an interesting problem!", "2552306": "Hi, everybody. It was a nice job from a nice team. I encountered a problem on sleep detection and would love to hear your solutions. I am attempting to do almost the same thing with the same data, but there are some differences. My data is much noisier due to 1-minute intervals and low accuracy.\n\nI am working on a web-based solution, so it needs to be very CPU-friendly. I came up with the idea of implementing a score-based system. First, I normalize the heart rate, step, and activity data, then multiply each by a related weight, and finally, add up scores for each point. I then define a threshold score to classify as sleep or wake. However, all the weight parameters and the threshold are defined manually.\n\nDoes anyone have an idea on how to improve this or how to use ML to find the best parameters for this score-based system? I am generating my own data and labeling it, so I have a hard time accessing large data as well." }, "source": "meta" }