\nSince there were large repeated periods in the data, I used an algorithm that utilized hashing and hashmaps which looked for such patterns in the data:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F5afb15a36f6816d1a90766c8983d30fb%2Frepeat1.PNG?generation=1701822960376424&alt=media\")
Thanks a lot to the competition host and kaggle for hosting this fun competition!
\nI split the data into different separate days with a padding of 60 minutes. The days are shifted by 500 minutes (so that one day starts and ends at 15:40). I did this to avoid wakeup/onset events near the end or start of a day, which I hypothesized to be benefical to the model (since there is only limited padding in both directions). I discarded 8 series ids which were mostly filled with nan values, and hand picked a first-n-day cutoff for a few series (removing the end of the series whenever there were no events but still regular activity). Since events only ever happened on full minutes, I constructed features for every one-minute bucket of data and generated an array of size (1440, n_feats). The features were mostly basic: anglez, enmo, anglez*enmo, hdcza were selectively paired with min, max, mean, std aggregations to generate 10 features, where hdcza was inspired by this discussion post:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/453267
Since it was discovered by @ymatioun made the great discovery that events had different frequencies at different steps in the 15 minute cycle. Therefore, one additional feature is the (step // 12) % 15.
\nSince there were large repeated periods in the data, I used an algorithm that utilized hashing and hashmaps which looked for such patterns in the data:
\n
Such a feature proved incredibly useful since there were many, non-obvious such repeating patterns in the data.
\nHere is a plot of enmo (blue), awake (red), and this new features, which indicates whether the data occurs somewhere else in the series or not (black):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F83b03dd0cfbd6bb590934c9a3ad8a71b%2Frepeat2.PNG?generation=1701823230374898&alt=media\")
As you can see, there were many parts of the data which were copy pasted from somewhere else. I am sure some models will have learned this anyway from the sometimes apparent cuts in the signal, but this of course works much better. The important thing about this feature is, that the patient is almost always sleeping when it is 1 (>99%). I didn't do too much in depth testing on how this affected the score, but local cv indicated that it improved the score by roughly ~0.015. Since the data was repeated and likely not valid anyway, I null out all anglez and enmo values where this features is 1 to prevent overfitting.
\nI used 4 model architectures which were all very similar and mostly based on Bidirectional LSTM layers. I used a 6 fold split.
\nMy main model used a UNet LSTM/Transformer architecture with a fully connected dimension of 1200 and an LSTM/Transformer dim of 300. In contrast to the typical UNet implementation, I added the layers in the skip connection rather than concatenating.
\nThis is the model pipeline used by the best single models. The other 3 architectures I used are very similar (with varying layer counts and dimensions), except 1 architecture which doesn't use the UNet.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2Fdf7fde6fec78f361a99cad3a934031b6%2Fmodel1.PNG?generation=1702143119651974&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F40cbb13011a969d0f9f1a6969af1bb07%2Fmodel2.PNG?generation=1702142987749483&alt=media\")
1 day of data in 1 minute intervals (60 minute padded in both directions) shape: (batch_size, 1440 + 60*2, n_feats) -> fully connected layers -> BiLSTM -> max pooling (pool size 15) -> 2 lstms layers -> max pooling (pool size 2) -> transformer block -> upsampling (factor 2) -> skip -> 2 lstm layers -> upsampling (factor 15) -> skip -> lstm layer -> fully connected layers -> output layer
\nThe 'skip' layers indicated the the output from the same stage at the downsampling stage is added onto the current sequence in the upsampling stage.
\nThe minute-mod-15 feature was used to calculate an embedding which was added to the sequence before, and after the UNet. Also, I used the mish activation most of the time.
\nThe other models had slight variations, e.g. leaving out the UNet, changing dimensions, increasing layer count.
\nSome variations also include a sub-lstm layer, which applies an LSTM layer on all 180 minutes intervals within a day, with 120 minute overlaps between the intervals.
The model predicted if an onset or wakeup event occurred within 0, 1, 3, 5, 7 minutes of the current step. I use a weighted BCE loss which weights the loss of the different minute thresholds by 1/(threshold+1). The prediction is the sum of all the probabilities that an event is in the 0, 1, 3, 5, 7 threshold. This is done for wakeup and onset separately so the model had 10 outputs per step.
\nThe model output itself had dimension (minutes_in_series, 10). This is the process of retrieving predictions from the model output:
\nThanks to @maruichi01 for pointing that out! The discussion related to this is here:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/446919#2487136
I used the AdamW optimizer with a weight decay of 1e-7 and lr of 1-e3 that exponentially decayed. It helped training stability to clip the norm of the gradient to 1e-2. Regular dropout was used and a kind of dropout that randomly masked a 15 minute segments with a probability of 15-25%. I trained on 4 seeds and picked the best seed of each architecture within each fold. I also refitted the best 2 architectures using the entire data.
\nLike Chris mentioned in his write-up, the discovery that we can make a lot of predictions per event was also very important. I first had a decoder model, which used the previous predictions the generate the next one using a probability distribution over all future steps. This worked quite well but only if the maximum prediction would have been counted. So I discarded this idea after discovering that we can make a lot of predictions per event without getting punished too much.
\nMy local cv is ~0.833.
Like I said earlier, I first worked on a decoder model to predict the next change event, which didn't work so well since it was beneficial to make many predictions per night. I also worked on stacking the model with an LGBM, but that only gave tiny improvements so I avoided the idea to not add too much complexity, though if I had more time I would have probably reconsidered it.
\nSince the ranking of the predictions was important I tried using other loss like e.g. a pearson correlation loss but that did not work better.
I hope I covered the most important parts and didn't forget anything. I tried a lot of things I may add later. I am very happy I managed to get my second solo Gold today, thus becoming a Kaggle master 😃
\nEdit:
\nLink to git repo:
\nhttps://gitlab.com/fritz_cremer/cmidss_final
Thanks a lot to the competition host and kaggle for hosting this fun competition!
\nI split the data into different separate days with a padding of 60 minutes. The days are shifted by 500 minutes (so that one day starts and ends at 15:40). I did this to avoid wakeup/onset events near the end or start of a day, which I hypothesized to be benefical to the model (since there is only limited padding in both directions). I discarded 8 series ids which were mostly filled with nan values, and hand picked a first-n-day cutoff for a few series (removing the end of the series whenever there were no events but still regular activity). Since events only ever happened on full minutes, I constructed features for every one-minute bucket of data and generated an array of size (1440, n_feats). The features were mostly basic: anglez, enmo, anglez*enmo, hdcza were selectively paired with min, max, mean, std aggregations to generate 10 features, where hdcza was inspired by this discussion post:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/453267
Since it was discovered by @ymatioun made the great discovery that events had different frequencies at different steps in the 15 minute cycle. Therefore, one additional feature is the (step // 12) % 15.
\nSince there were large repeated periods in the data, I used an algorithm that utilized hashing and hashmaps which looked for such patterns in the data:
\n
Such a feature proved incredibly useful since there were many, non-obvious such repeating patterns in the data.
\nHere is a plot of enmo (blue), awake (red), and this new features, which indicates whether the data occurs somewhere else in the series or not (black):
\n
As you can see, there were many parts of the data which were copy pasted from somewhere else. I am sure some models will have learned this anyway from the sometimes apparent cuts in the signal, but this of course works much better. The important thing about this feature is, that the patient is almost always sleeping when it is 1 (>99%). I didn't do too much in depth testing on how this affected the score, but local cv indicated that it improved the score by roughly ~0.015. Since the data was repeated and likely not valid anyway, I null out all anglez and enmo values where this features is 1 to prevent overfitting.
\nI used 4 model architectures which were all very similar and mostly based on Bidirectional LSTM layers. I used a 6 fold split.
\nMy main model used a UNet LSTM/Transformer architecture with a fully connected dimension of 1200 and an LSTM/Transformer dim of 300. In contrast to the typical UNet implementation, I added the layers in the skip connection rather than concatenating.
\nThis is the model pipeline used by the best single models. The other 3 architectures I used are very similar (with varying layer counts and dimensions), except 1 architecture which doesn't use the UNet.
\n
1 day of data in 1 minute intervals (60 minute padded in both directions) shape: (batch_size, 1440 + 60*2, n_feats) -> fully connected layers -> BiLSTM -> max pooling (pool size 15) -> 2 lstms layers -> max pooling (pool size 2) -> transformer block -> upsampling (factor 2) -> skip -> 2 lstm layers -> upsampling (factor 15) -> skip -> lstm layer -> fully connected layers -> output layer
\nThe 'skip' layers indicated the the output from the same stage at the downsampling stage is added onto the current sequence in the upsampling stage.
\nThe minute-mod-15 feature was used to calculate an embedding which was added to the sequence before, and after the UNet. Also, I used the mish activation most of the time.
\nThe other models had slight variations, e.g. leaving out the UNet, changing dimensions, increasing layer count.
\nSome variations also include a sub-lstm layer, which applies an LSTM layer on all 180 minutes intervals within a day, with 120 minute overlaps between the intervals.
The model predicted if an onset or wakeup event occurred within 0, 1, 3, 5, 7 minutes of the current step. I use a weighted BCE loss which weights the loss of the different minute thresholds by 1/(threshold+1). The prediction is the sum of all the probabilities that an event is in the 0, 1, 3, 5, 7 threshold. This is done for wakeup and onset separately so the model had 10 outputs per step.
\nThe model output itself had dimension (minutes_in_series, 10). This is the process of retrieving predictions from the model output:
\nThanks to @maruichi01 for pointing that out! The discussion related to this is here:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/446919#2487136
I used the AdamW optimizer with a weight decay of 1e-7 and lr of 1-e3 that exponentially decayed. It helped training stability to clip the norm of the gradient to 1e-2. Regular dropout was used and a kind of dropout that randomly masked a 15 minute segments with a probability of 15-25%. I trained on 4 seeds and picked the best seed of each architecture within each fold. I also refitted the best 2 architectures using the entire data.
\nLike Chris mentioned in his write-up, the discovery that we can make a lot of predictions per event was also very important. I first had a decoder model, which used the previous predictions the generate the next one using a probability distribution over all future steps. This worked quite well but only if the maximum prediction would have been counted. So I discarded this idea after discovering that we can make a lot of predictions per event without getting punished too much.
\nMy local cv is ~0.833.
Like I said earlier, I first worked on a decoder model to predict the next change event, which didn't work so well since it was beneficial to make many predictions per night. I also worked on stacking the model with an LGBM, but that only gave tiny improvements so I avoided the idea to not add too much complexity, though if I had more time I would have probably reconsidered it.
\nSince the ranking of the predictions was important I tried using other loss like e.g. a pearson correlation loss but that did not work better.
I hope I covered the most important parts and didn't forget anything. I tried a lot of things I may add later. I am very happy I managed to get my second solo Gold today, thus becoming a Kaggle master 😃
\nEdit:
\nLink to git repo:
\nhttps://gitlab.com/fritz_cremer/cmidss_final
Congratulations, and thank you for your sharing
", "rawMarkdown": "Congratulations, and thank you for your sharing", "votes": null }, { "id": "2550404", "postDate": "12/06/2023 02:40:36", "content": "Very clever pre-processing (hashing);
\nAlso great architecture (reminds of MobileVIT--i.e. only bringing out the more powerful components at the highest levels of aggregation)
", "rawMarkdown": "Very clever pre-processing (hashing); \n\nAlso great architecture (reminds of MobileVIT--i.e. only bringing out the more powerful components at the highest levels of aggregation)", "votes": null }, { "id": "2550580", "postDate": "12/06/2023 05:53:15", "content": "Congratulations！
", "rawMarkdown": "Congratulations！", "votes": null }, { "id": "2550609", "postDate": "12/06/2023 06:21:54", "content": "Congrats @fritzcremer for 6th place finish! I guess the shakeup did help us get solo gold after all 🤣
\nI pruned away repeating data too, but I used a large window (6hrs) which probably is too much. BTW, you can have a look at this library which can catch repetitions easily: https://stumpy.readthedocs.io/en/latest/Tutorial_STUMPY_Basics.html
\nMay I ask what kinds of postprocessing steps you've done to enforce the one event per day rule? I originally predicted probabilities of events per step, but it didn't work too well so I switched to a method similar to @cdeotte's softmax over a given window.
", "rawMarkdown": "Congrats @fritzcremer for 6th place finish! I guess the shakeup did help us get solo gold after all 🤣\n\nI pruned away repeating data too, but I used a large window (6hrs) which probably is too much. BTW, you can have a look at this library which can catch repetitions easily: https://stumpy.readthedocs.io/en/latest/Tutorial_STUMPY_Basics.html\n\nMay I ask what kinds of postprocessing steps you've done to enforce the one event per day rule? I originally predicted probabilities of events per step, but it didn't work too well so I switched to a method similar to @cdeotte's softmax over a given window.", "votes": null }, { "id": "2550827", "postDate": "12/06/2023 10:02:04", "content": "Yes thats true :) Ah I don't know why I didn't think about using some library for that.
\nI am not sure what you mean, as I didn't predict only one event per day. I enforced a one event per 21 minutes rule by repeatedly selecting the maximum, and then nulling out the predictions 10 minutes before- and after that. This way there can'e be another prediction in this range.
Thanks, yes I figured it was important to capture long- and short-term dependencies. Pure Transformer didn't work out for me. Possibly because there wasn't enough data to accurately learn the position embeddings.
", "rawMarkdown": "Thanks, yes I figured it was important to capture long- and short-term dependencies. Pure Transformer didn't work out for me. Possibly because there wasn't enough data to accurately learn the position embeddings.", "votes": null }, { "id": "2550956", "postDate": "12/06/2023 12:37:04", "content": "Yes I did exactly that too, but with a one event per 6 mins rule instead.
\nWhat I meant by \"one event per day\" is to train the neural network to predict probabilities for each (group of) step(s) so that it sums up to 1 in a day, which \"softly\" restricts the amount of onset/wakeup to one per day.
\nI initially experimented with a method similar to yours - that is to predict whether an event occurs in a given step within some range. This didn't work well and I got only ~0.79 cv and ~0.75 public LB. Changing to the aforementioned one event per day rule gave me +0.02 in both cv and lb. I was kind of surprised that most top solutions didn't use this method.
\nI guess postprocessing is really important in this competition.
", "rawMarkdown": "Yes I did exactly that too, but with a one event per 6 mins rule instead.\n\nWhat I meant by \"one event per day\" is to train the neural network to predict probabilities for each (group of) step(s) so that it sums up to 1 in a day, which \"softly\" restricts the amount of onset/wakeup to one per day. \n\nI initially experimented with a method similar to yours - that is to predict whether an event occurs in a given step within some range. This didn't work well and I got only ~0.79 cv and ~0.75 public LB. Changing to the aforementioned one event per day rule gave me +0.02 in both cv and lb. I was kind of surprised that most top solutions didn't use this method.\n\nI guess postprocessing is really important in this competition.", "votes": null }, { "id": "2551104", "postDate": "12/06/2023 14:02:23", "content": "I thought about a few things like that but in the end I just let the model predict probabilities between 0 and 1 for each step without constraint. And yes postprocessing was one of the most important things I believe.
", "rawMarkdown": "I thought about a few things like that but in the end I just let the model predict probabilities between 0 and 1 for each step without constraint. And yes postprocessing was one of the most important things I believe.", "votes": null }, { "id": "2552197", "postDate": "12/07/2023 09:31:57", "content": "Congratulations on winning the 6th position in this competition. Appreciate your sharing of the solution details.
", "rawMarkdown": "Congratulations on winning the 6th position in this competition. Appreciate your sharing of the solution details.", "votes": null }, { "id": "2552537", "postDate": "12/07/2023 14:37:41", "content": "Congratulations for the solo gold medal you've got this time! Thank you for sharing your insights during the journey, including what didn't work.
", "rawMarkdown": "Congratulations for the solo gold medal you've got this time! Thank you for sharing your insights during the journey, including what didn't work.", "votes": null }, { "id": "2552573", "postDate": "12/07/2023 15:24:42", "content": "Congratulations for becoming a master.
", "rawMarkdown": "Congratulations for becoming a master.", "votes": null }, { "id": "2552574", "postDate": "12/07/2023 15:25:45", "content": "It seems like you didn't use convolutional block, why did you pad?
", "rawMarkdown": "It seems like you didn't use convolutional block, why did you pad?", "votes": null }, { "id": "2555098", "postDate": "12/09/2023 17:16:42", "content": "Thank you! I tried using convolutional layers but it didn't improve my model, though I am sure if I had further tried it could have improved my ensemble. I padded, since otherwise, the model has no context about the minutes right before and after the day, which might be helpful to make predictions at these steps.
", "rawMarkdown": "Thank you! I tried using convolutional layers but it didn't improve my model, though I am sure if I had further tried it could have improved my ensemble. I padded, since otherwise, the model has no context about the minutes right before and after the day, which might be helpful to make predictions at these steps.", "votes": null }, { "id": "2555099", "postDate": "12/09/2023 17:16:53", "content": "Thanks a lot :)
", "rawMarkdown": "Thanks a lot :)", "votes": null }, { "id": "2558954", "postDate": "12/12/2023 13:31:38", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null } ], "comments": [ { "id": 2550325, "author_name": "kerrysun", "author_url": "", "post_date": "12/06/2023 01:22:56", "content": "Congratulations, and thank you for your sharing
", "votes": null, "replies": [] }, { "id": 2550404, "author_name": "stochoshi", "author_url": "", "post_date": "12/06/2023 02:40:36", "content": "Very clever pre-processing (hashing);
\nAlso great architecture (reminds of MobileVIT--i.e. only bringing out the more powerful components at the highest levels of aggregation)
", "votes": null, "replies": [ { "id": 2550943, "author_name": "fritzcremer", "author_url": "", "post_date": "12/06/2023 12:21:00", "content": "Thanks, yes I figured it was important to capture long- and short-term dependencies. Pure Transformer didn't work out for me. Possibly because there wasn't enough data to accurately learn the position embeddings.
", "votes": null, "replies": [] } ] }, { "id": 2550580, "author_name": "xianghuang666", "author_url": "", "post_date": "12/06/2023 05:53:15", "content": "Congratulations！
", "votes": null, "replies": [] }, { "id": 2550609, "author_name": "chauyh", "author_url": "", "post_date": "12/06/2023 06:21:54", "content": "Congrats @fritzcremer for 6th place finish! I guess the shakeup did help us get solo gold after all 🤣
\nI pruned away repeating data too, but I used a large window (6hrs) which probably is too much. BTW, you can have a look at this library which can catch repetitions easily: https://stumpy.readthedocs.io/en/latest/Tutorial_STUMPY_Basics.html
\nMay I ask what kinds of postprocessing steps you've done to enforce the one event per day rule? I originally predicted probabilities of events per step, but it didn't work too well so I switched to a method similar to @cdeotte's softmax over a given window.
", "votes": null, "replies": [ { "id": 2550827, "author_name": "fritzcremer", "author_url": "", "post_date": "12/06/2023 10:02:04", "content": "Yes thats true :) Ah I don't know why I didn't think about using some library for that.
\nI am not sure what you mean, as I didn't predict only one event per day. I enforced a one event per 21 minutes rule by repeatedly selecting the maximum, and then nulling out the predictions 10 minutes before- and after that. This way there can'e be another prediction in this range.
Yes I did exactly that too, but with a one event per 6 mins rule instead.
\nWhat I meant by \"one event per day\" is to train the neural network to predict probabilities for each (group of) step(s) so that it sums up to 1 in a day, which \"softly\" restricts the amount of onset/wakeup to one per day.
\nI initially experimented with a method similar to yours - that is to predict whether an event occurs in a given step within some range. This didn't work well and I got only ~0.79 cv and ~0.75 public LB. Changing to the aforementioned one event per day rule gave me +0.02 in both cv and lb. I was kind of surprised that most top solutions didn't use this method.
\nI guess postprocessing is really important in this competition.
", "votes": null, "replies": [ { "id": 2551104, "author_name": "fritzcremer", "author_url": "", "post_date": "12/06/2023 14:02:23", "content": "I thought about a few things like that but in the end I just let the model predict probabilities between 0 and 1 for each step without constraint. And yes postprocessing was one of the most important things I believe.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2552197, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "12/07/2023 09:31:57", "content": "Congratulations on winning the 6th position in this competition. Appreciate your sharing of the solution details.
", "votes": null, "replies": [] }, { "id": 2552537, "author_name": "coyamax", "author_url": "", "post_date": "12/07/2023 14:37:41", "content": "Congratulations for the solo gold medal you've got this time! Thank you for sharing your insights during the journey, including what didn't work.
", "votes": null, "replies": [ { "id": 2555099, "author_name": "fritzcremer", "author_url": "", "post_date": "12/09/2023 17:16:53", "content": "Thanks a lot :)
", "votes": null, "replies": [] } ] }, { "id": 2552573, "author_name": "dongyk", "author_url": "", "post_date": "12/07/2023 15:24:42", "content": "Congratulations for becoming a master.
", "votes": null, "replies": [ { "id": 2552574, "author_name": "dongyk", "author_url": "", "post_date": "12/07/2023 15:25:45", "content": "It seems like you didn't use convolutional block, why did you pad?
", "votes": null, "replies": [ { "id": 2555098, "author_name": "fritzcremer", "author_url": "", "post_date": "12/09/2023 17:16:42", "content": "Thank you! I tried using convolutional layers but it didn't improve my model, though I am sure if I had further tried it could have improved my ensemble. I padded, since otherwise, the model has no context about the minutes right before and after the day, which might be helpful to make predictions at these steps.
", "votes": null, "replies": [ { "id": 2558954, "author_name": "dongyk", "author_url": "", "post_date": "12/12/2023 13:31:38", "content": "Thank you.
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2550320": "Thanks a lot to the competition host and kaggle for hosting this fun competition!\n\n# Data Processing\n\nI split the data into different separate days with a padding of 60 minutes. The days are shifted by 500 minutes (so that one day starts and ends at 15:40). I did this to avoid wakeup/onset events near the end or start of a day, which I hypothesized to be benefical to the model (since there is only limited padding in both directions). I discarded 8 series ids which were mostly filled with nan values, and hand picked a first-n-day cutoff for a few series (removing the end of the series whenever there were no events but still regular activity). Since events only ever happened on full minutes, I constructed features for every one-minute bucket of data and generated an array of size (1440, n_feats). The features were mostly basic: anglez, enmo, anglez*enmo, hdcza were selectively paired with min, max, mean, std aggregations to generate 10 features, where hdcza was inspired by this discussion post:\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/453267\n\nSince it was discovered by @ymatioun made the great discovery that events had different frequencies at different steps in the 15 minute cycle. Therefore, one additional feature is the (step // 12) % 15.\nSince there were large repeated periods in the data, I used an algorithm that utilized hashing and hashmaps which looked for such patterns in the data:\n\n\nSuch a feature proved incredibly useful since there were many, non-obvious such repeating patterns in the data.\nHere is a plot of enmo (blue), awake (red), and this new features, which indicates whether the data occurs somewhere else in the series or not (black):\n\n\nAs you can see, there were many parts of the data which were copy pasted from somewhere else. I am sure some models will have learned this anyway from the sometimes apparent cuts in the signal, but this of course works much better. The important thing about this feature is, that the patient is almost always sleeping when it is 1 (>99%). I didn't do too much in depth testing on how this affected the score, but local cv indicated that it improved the score by roughly ~0.015. Since the data was repeated and likely not valid anyway, I null out all anglez and enmo values where this features is 1 to prevent overfitting.\n\n# Model Architecture\nI used 4 model architectures which were all very similar and mostly based on Bidirectional LSTM layers. I used a 6 fold split. \n\nMy main model used a UNet LSTM/Transformer architecture with a fully connected dimension of 1200 and an LSTM/Transformer dim of 300. In contrast to the typical UNet implementation, I added the layers in the skip connection rather than concatenating.\n\n## Main model: \n\nThis is the model pipeline used by the best single models. The other 3 architectures I used are very similar (with varying layer counts and dimensions), except 1 architecture which doesn't use the UNet.\n\n\n\n\n\n1 day of data in 1 minute intervals (60 minute padded in both directions) shape: (batch_size, 1440 + 60*2, n_feats) -> fully connected layers -> BiLSTM -> max pooling (pool size 15) -> 2 lstms layers -> max pooling (pool size 2) -> transformer block -> upsampling (factor 2) -> skip -> 2 lstm layers -> upsampling (factor 15) -> skip -> lstm layer -> fully connected layers -> output layer\n\nThe 'skip' layers indicated the the output from the same stage at the downsampling stage is added onto the current sequence in the upsampling stage.\n\nThe minute-mod-15 feature was used to calculate an embedding which was added to the sequence before, and after the UNet. Also, I used the mish activation most of the time.\n\nThe other models had slight variations, e.g. leaving out the UNet, changing dimensions, increasing layer count.\nSome variations also include a sub-lstm layer, which applies an LSTM layer on all 180 minutes intervals within a day, with 120 minute overlaps between the intervals.\n\n## Loss function and output\nThe model predicted if an onset or wakeup event occurred within 0, 1, 3, 5, 7 minutes of the current step. I use a weighted BCE loss which weights the loss of the different minute thresholds by 1/(threshold+1). The prediction is the sum of all the probabilities that an event is in the 0, 1, 3, 5, 7 threshold. This is done for wakeup and onset separately so the model had 10 outputs per step.\n\n# Postprocessing\nThe model output itself had dimension (minutes_in_series, 10). This is the process of retrieving predictions from the model output:\n1.\tConcatenate the model predictions from each day in a series.\n2.\tTake the sum of the prediction that an event is in 0, 1, 3, 5, 7 minutes range at each step, for onset and wakeup respectively.\n3.\tUntil there are no predictions larger than 5e-5 left:\na.\tSelect the highest prediction.\nb.\tNull out all predictions within a range of 10 minutes of the prediction.\n4.\tSince we are still in minute space, multiply by 12 to obtain the correct steps.\n5.\tAs was pointed out in the discussion forums, predictions for full minutes are punished since the metric doesn’t count a prediction as within a threshold, if it is directly on a threshold. Therefore, to maximize the number of possible thresholds we may be in, it is beneficial to add or subtract 1. This is decided by looking at the neighboring prediction.\n6.\tUse the summed predictions from step 2. As scores for the retrieved steps.\n\nThanks to @maruichi01 for pointing that out! The discussion related to this is here:\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/446919#2487136\n\n# Training\nI used the AdamW optimizer with a weight decay of 1e-7 and lr of 1-e3 that exponentially decayed. It helped training stability to clip the norm of the gradient to 1e-2. Regular dropout was used and a kind of dropout that randomly masked a 15 minute segments with a probability of 15-25%. I trained on 4 seeds and picked the best seed of each architecture within each fold. I also refitted the best 2 architectures using the entire data.\n\n# Other stuff\nLike Chris mentioned in his write-up, the discovery that we can make a lot of predictions per event was also very important. I first had a decoder model, which used the previous predictions the generate the next one using a probability distribution over all future steps. This worked quite well but only if the maximum prediction would have been counted. So I discarded this idea after discovering that we can make a lot of predictions per event without getting punished too much.\nMy local cv is ~0.833.\n\n# Things that didn't work\nLike I said earlier, I first worked on a decoder model to predict the next change event, which didn't work so well since it was beneficial to make many predictions per night. I also worked on stacking the model with an LGBM, but that only gave tiny improvements so I avoided the idea to not add too much complexity, though if I had more time I would have probably reconsidered it.\nSince the ranking of the predictions was important I tried using other loss like e.g. a pearson correlation loss but that did not work better.\n\nI hope I covered the most important parts and didn't forget anything. I tried a lot of things I may add later. I am very happy I managed to get my second solo Gold today, thus becoming a Kaggle master 😃\n\nEdit:\nLink to git repo:\nhttps://gitlab.com/fritz_cremer/cmidss_final", "2550325": "Congratulations, and thank you for your sharing", "2550404": "Very clever pre-processing (hashing); \n\nAlso great architecture (reminds of MobileVIT--i.e. only bringing out the more powerful components at the highest levels of aggregation)", "2550580": "Congratulations！", "2550609": "Congrats @fritzcremer for 6th place finish! I guess the shakeup did help us get solo gold after all 🤣\n\nI pruned away repeating data too, but I used a large window (6hrs) which probably is too much. BTW, you can have a look at this library which can catch repetitions easily: https://stumpy.readthedocs.io/en/latest/Tutorial_STUMPY_Basics.html\n\nMay I ask what kinds of postprocessing steps you've done to enforce the one event per day rule? I originally predicted probabilities of events per step, but it didn't work too well so I switched to a method similar to @cdeotte's softmax over a given window.", "2550827": "Yes thats true :) Ah I don't know why I didn't think about using some library for that.\nI am not sure what you mean, as I didn't predict only one event per day. I enforced a one event per 21 minutes rule by repeatedly selecting the maximum, and then nulling out the predictions 10 minutes before- and after that. This way there can'e be another prediction in this range.", "2550943": "Thanks, yes I figured it was important to capture long- and short-term dependencies. Pure Transformer didn't work out for me. Possibly because there wasn't enough data to accurately learn the position embeddings.", "2550956": "Yes I did exactly that too, but with a one event per 6 mins rule instead.\n\nWhat I meant by \"one event per day\" is to train the neural network to predict probabilities for each (group of) step(s) so that it sums up to 1 in a day, which \"softly\" restricts the amount of onset/wakeup to one per day. \n\nI initially experimented with a method similar to yours - that is to predict whether an event occurs in a given step within some range. This didn't work well and I got only ~0.79 cv and ~0.75 public LB. Changing to the aforementioned one event per day rule gave me +0.02 in both cv and lb. I was kind of surprised that most top solutions didn't use this method.\n\nI guess postprocessing is really important in this competition.", "2551104": "I thought about a few things like that but in the end I just let the model predict probabilities between 0 and 1 for each step without constraint. And yes postprocessing was one of the most important things I believe.", "2552197": "Congratulations on winning the 6th position in this competition. Appreciate your sharing of the solution details.", "2552537": "Congratulations for the solo gold medal you've got this time! Thank you for sharing your insights during the journey, including what didn't work.", "2552573": "Congratulations for becoming a master.", "2552574": "It seems like you didn't use convolutional block, why did you pad?", "2555098": "Thank you! I tried using convolutional layers but it didn't improve my model, though I am sure if I had further tried it could have improved my ensemble. I padded, since otherwise, the model has no context about the minutes right before and after the day, which might be helpful to make predictions at these steps.", "2555099": "Thanks a lot :)", "2558954": "Thank you." }, "source": "existing_discussion", "complete": true, "competition_id": 53666, "competition_slug": "child-mind-institute-detect-sleep-states", "team_id": 10875109, "solution_url": "/competitions/child-mind-institute-detect-sleep-states/writeups/fritz-cremer-6th-place-solution-bilstm-unet" }