First, big thanks to the organizers for preparing such an interesting competition. I really enjoyed working on it and learned a lot. I was quite surprised how well the NN models worked in this competition and how easily they surpassed the handcrafted features and decision tree models.
\nAlso big thanks to all the people who shared their knowledge in discussions and code. Especially to:
\nEven though my best solution was based on code published by 213tubo in Detecting sleep using UNet(2D) post, I decided to publish the write up to show some simple ways how to improve on it and that it's possible to get good results using only limited resources. All training was done on Google Colab using T4 GPU instance.
\nBusiness Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview
\nData Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data
My best scoring pipeline is based on code published by 213tubo in Detecting sleep using UNet(2D) post.
\nAs described by in the original post the model consist of feature extractor, encoder and decoder.
\nModel input: 8 hour window (5760 samples) with 4 features: normalized anglez, normalized enmo, hour sin and hour cos
\nUsed around 20% of data for validation and trained on remaining 80%.
\n1) Train more models
\n2) Ensemble using average output of multiple models
\n3) Use timeframes with overlap during inference, average out predictions from overlapping segments
\n4) Tune threshold and peak gap - lower threshold with 5 minute gap between events produced the best results
Model published in CMI-submit
\n| Base model | \nCV Score | \nPublic Score | \nPrivate Score | \n
|---|---|---|---|
| Base model | \n0.7305 | \n0.707 | \n0.728 | \n
I trained around 30 models with different encoders, feature extractors, downsampling rate, losses, architectures and sizes. Most of the models were trained with downsample rate 4 which sped up the training and did not decrease performance. From those models I selected best 6 for the final ensemble based on their score.
\n| Feature extractor base filters | \nEncoder architecture | \nDecoder | \ndownsample rate | \nCV Score | \n
|---|---|---|---|---|
| 256 | \nUnet | \nUNet1DDecoder | \n4 | \n0.7545 | \n
| 64 | \nUnet | \nTransformerCNNDecoder | \n4 | \n0.7526 | \n
| 128 | \nUnet | \nUNet1DDecoder | \n4 | \n0.7525 | \n
| 128 | \nUnet | \nUNet1DDecoder | \n2 | \n0.7519 | \n
| 128 | \nUnet | \nUNet1DDecoder | \n3 | \n0.7510 | \n
| 128 | \nFPN | \nUNet1DDecoder | \n4 | \n0.7486 | \n
Average output of models was used as prediction. I tested mean, max, min and some simple weighted average based on CV score but simple average gave best results.
\n| Ensemble | \nEnsemble method | \nCV Score | \n
|---|---|---|
| 6 model ensemble | \nmean | \n0.7717 | \n
Inference was made on 8 hour windows with 4 hour step, overlapping segments were avereged out. Bigger overlap gave better results but unfortunatelly finished few minutes after competition ended. Taking average from overlapping segments was better than taking minimum prediction, using maximum was giving the worst results.
\n| Overlap during inference | \nCV Score | \n
|---|---|
| 4 hours | \n0.7818 | \n
Lower threshold with bigger gap between events produced better score. Missing on event seems to be punished more than extra prediction but too many predicted events led to submission scoring error. Best score I was able to get was using 0.001 threshold and 60 sample distance (5 minutes) between predicted events.
\n| Threshold | \nMin event distance | \nCV Score | \nPublic Score | \nPrivate Score | \n
|---|---|---|---|---|
| 0.001 | \n60 | \n0.7871 | \n0.740 | \n0.782 | \n
I started with my own pipeline, using handcrafted features and decision tree models and later moved to NNs. Even though the models were getting really good CV score (over 0.8) the leaderboard score was always about 0.15 lower. I tried multiple CV strategies, random split, split based on length, number of nights without labels etc. and could not get a good CV strategy. I checked my code multiple times for leaks and was not able to find out what could be the cause. Getting discouraged by the low leaderboard score I abandoned my pipeline and switched to 213tubo's amazing solution about a week before the competition end. After the private leaderboard was revealed I found out that my pipeline was working just as well and only the public score was low. Lesson learned to trust your CV even though it's hard sometimes.
", "messages": [ { "id": "2566523", "postDate": "12/18/2023 23:00:48", "content": "First, big thanks to the organizers for preparing such an interesting competition. I really enjoyed working on it and learned a lot. I was quite surprised how well the NN models worked in this competition and how easily they surpassed the handcrafted features and decision tree models.
\nAlso big thanks to all the people who shared their knowledge in discussions and code. Especially to:
\nEven though my best solution was based on code published by 213tubo in Detecting sleep using UNet(2D) post, I decided to publish the write up to show some simple ways how to improve on it and that it's possible to get good results using only limited resources. All training was done on Google Colab using T4 GPU instance.
\nBusiness Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview
\nData Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data
My best scoring pipeline is based on code published by 213tubo in Detecting sleep using UNet(2D) post.
\nAs described by in the original post the model consist of feature extractor, encoder and decoder.
\nModel input: 8 hour window (5760 samples) with 4 features: normalized anglez, normalized enmo, hour sin and hour cos
\nUsed around 20% of data for validation and trained on remaining 80%.
\n1) Train more models
\n2) Ensemble using average output of multiple models
\n3) Use timeframes with overlap during inference, average out predictions from overlapping segments
\n4) Tune threshold and peak gap - lower threshold with 5 minute gap between events produced the best results
Model published in CMI-submit
\n| Base model | \nCV Score | \nPublic Score | \nPrivate Score | \n
|---|---|---|---|
| Base model | \n0.7305 | \n0.707 | \n0.728 | \n
I trained around 30 models with different encoders, feature extractors, downsampling rate, losses, architectures and sizes. Most of the models were trained with downsample rate 4 which sped up the training and did not decrease performance. From those models I selected best 6 for the final ensemble based on their score.
\n| Feature extractor base filters | \nEncoder architecture | \nDecoder | \ndownsample rate | \nCV Score | \n
|---|---|---|---|---|
| 256 | \nUnet | \nUNet1DDecoder | \n4 | \n0.7545 | \n
| 64 | \nUnet | \nTransformerCNNDecoder | \n4 | \n0.7526 | \n
| 128 | \nUnet | \nUNet1DDecoder | \n4 | \n0.7525 | \n
| 128 | \nUnet | \nUNet1DDecoder | \n2 | \n0.7519 | \n
| 128 | \nUnet | \nUNet1DDecoder | \n3 | \n0.7510 | \n
| 128 | \nFPN | \nUNet1DDecoder | \n4 | \n0.7486 | \n
Average output of models was used as prediction. I tested mean, max, min and some simple weighted average based on CV score but simple average gave best results.
\n| Ensemble | \nEnsemble method | \nCV Score | \n
|---|---|---|
| 6 model ensemble | \nmean | \n0.7717 | \n
Inference was made on 8 hour windows with 4 hour step, overlapping segments were avereged out. Bigger overlap gave better results but unfortunatelly finished few minutes after competition ended. Taking average from overlapping segments was better than taking minimum prediction, using maximum was giving the worst results.
\n| Overlap during inference | \nCV Score | \n
|---|---|
| 4 hours | \n0.7818 | \n
Lower threshold with bigger gap between events produced better score. Missing on event seems to be punished more than extra prediction but too many predicted events led to submission scoring error. Best score I was able to get was using 0.001 threshold and 60 sample distance (5 minutes) between predicted events.
\n| Threshold | \nMin event distance | \nCV Score | \nPublic Score | \nPrivate Score | \n
|---|---|---|---|---|
| 0.001 | \n60 | \n0.7871 | \n0.740 | \n0.782 | \n
I started with my own pipeline, using handcrafted features and decision tree models and later moved to NNs. Even though the models were getting really good CV score (over 0.8) the leaderboard score was always about 0.15 lower. I tried multiple CV strategies, random split, split based on length, number of nights without labels etc. and could not get a good CV strategy. I checked my code multiple times for leaks and was not able to find out what could be the cause. Getting discouraged by the low leaderboard score I abandoned my pipeline and switched to 213tubo's amazing solution about a week before the competition end. After the private leaderboard was revealed I found out that my pipeline was working just as well and only the public score was low. Lesson learned to trust your CV even though it's hard sometimes.
", "rawMarkdown": "First, big thanks to the organizers for preparing such an interesting competition. I really enjoyed working on it and learned a lot. I was quite surprised how well the NN models worked in this competition and how easily they surpassed the handcrafted features and decision tree models.\n\nAlso big thanks to all the people who shared their knowledge in discussions and code. Especially to:\n- [213tubo](https://www.kaggle.com/tubotubo) for [CMI-submit](https://www.kaggle.com/code/tubotubo/cmi-submit)\n- [Daniel Phalen](https://www.kaggle.com/danielphalen) for [CMSS-GRUNET](https://www.kaggle.com/code/danielphalen/cmss-grunet-infer)\n- [werus23](https://www.kaggle.com/werus23) for [Sleep - Critical point](https://www.kaggle.com/code/werus23/sleep-critical-point-infer)\n- [Lucas Burke](https://www.kaggle.com/lccburk) for [Feature Engineering and Random Forest Prediction](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction)\n\nEven though my best solution was based on code published by [213tubo](https://www.kaggle.com/tubotubo) in [Detecting sleep using UNet(2D) post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940), I decided to publish the write up to show some simple ways how to improve on it and that it's possible to get good results using only limited resources. All training was done on Google Colab using T4 GPU instance.\n\n#Context\nBusiness Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview\nData Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\n\n#Overview of the Approach\nMy best scoring pipeline is based on code published by [213tubo](https://www.kaggle.com/tubotubo) in [Detecting sleep using UNet(2D) post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940).\n\n## Model\nAs described by in the original post the model consist of feature extractor, encoder and decoder.\n- **Feature extractor** takes raw input and transforms it into wavegram of image like shape. 3 Conv1d layers with different kernel sizes were used as feature extractors.\n- **Encoder** takes the wavegram and creates a two dimensional heatmap. Unet and FPN models with resnet34 as backbone from [PyTorch Segmentation models library](https://github.com/qubvel/segmentation_models.pytorch) were used as encoders.\n- **Decoder** creates probabilities of sleep, onset and wakeup from the heatmap generated by encoder. 1d Unet and TransformerEncoder were used as decoders. Only onset and wakeup predictions are used during inference.\n\nModel input: 8 hour window (5760 samples) with 4 features: normalized anglez, normalized enmo, hour sin and hour cos\n\nUsed around 20% of data for validation and trained on remaining 80%.\n\n## Additional improvements that boosted the score:\n1) Train more models\n2) Ensemble using average output of multiple models\n3) Use timeframes with overlap during inference, average out predictions from overlapping segments\n4) Tune threshold and peak gap - lower threshold with 5 minute gap between events produced the best results\n\n#Details of the submission\nModel published in [CMI-submit](https://www.kaggle.com/code/tubotubo/cmi-submit)\n|Base model|CV Score|Public Score|Private Score|\n|---|---|---|---|\n|Base model|0.7305|0.707|0.728|\n\n## Experiment with different models, architectures, settings\nI trained around 30 models with different encoders, feature extractors, downsampling rate, losses, architectures and sizes. Most of the models were trained with downsample rate 4 which sped up the training and did not decrease performance. From those models I selected best 6 for the final ensemble based on their score.\n\n|Feature extractor base filters|Encoder architecture|Decoder|downsample rate|CV Score|\n|---|---|---|---|---|\n|256|Unet|UNet1DDecoder|4|0.7545|\n|64|Unet|TransformerCNNDecoder|4|0.7526|\n|128|Unet|UNet1DDecoder|4|0.7525|\n|128|Unet|UNet1DDecoder|2|0.7519|\n|128|Unet|UNet1DDecoder|3|0.7510|\n|128| FPN|UNet1DDecoder|4|0.7486|\n\n## Ensemble\nAverage output of models was used as prediction. I tested mean, max, min and some simple weighted average based on CV score but simple average gave best results.\n|Ensemble|Ensemble method|CV Score|\n|---|---|---|\n|6 model ensemble|mean|0.7717|\n\n## Timeframe with overlaps during inference\nInference was made on 8 hour windows with 4 hour step, overlapping segments were avereged out. Bigger overlap gave better results but unfortunatelly finished few minutes after competition ended. Taking average from overlapping segments was better than taking minimum prediction, using maximum was giving the worst results.\n|Overlap during inference|CV Score|\n|---|---|\n|4 hours|0.7818|\n\n## Threshold and peak gap tuning\nLower threshold with bigger gap between events produced better score. Missing on event seems to be punished more than extra prediction but too many predicted events led to submission scoring error. Best score I was able to get was using 0.001 threshold and 60 sample distance (5 minutes) between predicted events.\n|Threshold|Min event distance|CV Score|Public Score|Private Score|\n|---|---|---|\n|0.001|60|0.7871|0.740|0.782|\n\n# Sources\n- https://www.kaggle.com/code/tubotubo/cmi-submit\n- https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940\n- https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/tree/main\n\n# Ending note - trust your CV or not\nI started with my own pipeline, using handcrafted features and decision tree models and later moved to NNs. Even though the models were getting really good CV score (over 0.8) the leaderboard score was always about 0.15 lower. I tried multiple CV strategies, random split, split based on length, number of nights without labels etc. and could not get a good CV strategy. I checked my code multiple times for leaks and was not able to find out what could be the cause. Getting discouraged by the low leaderboard score I abandoned my pipeline and switched to 213tubo's amazing solution about a week before the competition end. After the private leaderboard was revealed I found out that my pipeline was working just as well and only the public score was low. Lesson learned to trust your CV even though it's hard sometimes.", "votes": null }, { "id": "2566951", "postDate": "12/19/2023 09:28:41", "content": "Congrats! I guess the one thing to takeaway from this competition, is that the train dataset is very robust (probably one of the best I have seen) that you can trust the CV score easily.
\nFor my team, when the private LB was released, we realised our CV scores are always around 0.005-0.008 higher than the private LB, with little deviation. For once, the right submissions were chosen.
", "rawMarkdown": "Congrats! I guess the one thing to takeaway from this competition, is that the train dataset is very robust (probably one of the best I have seen) that you can trust the CV score easily.\n\nFor my team, when the private LB was released, we realised our CV scores are always around 0.005-0.008 higher than the private LB, with little deviation. For once, the right submissions were chosen.", "votes": null } ], "comments": [ { "id": 2566951, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "12/19/2023 09:28:41", "content": "Congrats! I guess the one thing to takeaway from this competition, is that the train dataset is very robust (probably one of the best I have seen) that you can trust the CV score easily.
\nFor my team, when the private LB was released, we realised our CV scores are always around 0.005-0.008 higher than the private LB, with little deviation. For once, the right submissions were chosen.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2566523": "First, big thanks to the organizers for preparing such an interesting competition. I really enjoyed working on it and learned a lot. I was quite surprised how well the NN models worked in this competition and how easily they surpassed the handcrafted features and decision tree models.\n\nAlso big thanks to all the people who shared their knowledge in discussions and code. Especially to:\n- [213tubo](https://www.kaggle.com/tubotubo) for [CMI-submit](https://www.kaggle.com/code/tubotubo/cmi-submit)\n- [Daniel Phalen](https://www.kaggle.com/danielphalen) for [CMSS-GRUNET](https://www.kaggle.com/code/danielphalen/cmss-grunet-infer)\n- [werus23](https://www.kaggle.com/werus23) for [Sleep - Critical point](https://www.kaggle.com/code/werus23/sleep-critical-point-infer)\n- [Lucas Burke](https://www.kaggle.com/lccburk) for [Feature Engineering and Random Forest Prediction](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction)\n\nEven though my best solution was based on code published by [213tubo](https://www.kaggle.com/tubotubo) in [Detecting sleep using UNet(2D) post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940), I decided to publish the write up to show some simple ways how to improve on it and that it's possible to get good results using only limited resources. All training was done on Google Colab using T4 GPU instance.\n\n#Context\nBusiness Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview\nData Context - https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\n\n#Overview of the Approach\nMy best scoring pipeline is based on code published by [213tubo](https://www.kaggle.com/tubotubo) in [Detecting sleep using UNet(2D) post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940).\n\n## Model\nAs described by in the original post the model consist of feature extractor, encoder and decoder.\n- **Feature extractor** takes raw input and transforms it into wavegram of image like shape. 3 Conv1d layers with different kernel sizes were used as feature extractors.\n- **Encoder** takes the wavegram and creates a two dimensional heatmap. Unet and FPN models with resnet34 as backbone from [PyTorch Segmentation models library](https://github.com/qubvel/segmentation_models.pytorch) were used as encoders.\n- **Decoder** creates probabilities of sleep, onset and wakeup from the heatmap generated by encoder. 1d Unet and TransformerEncoder were used as decoders. Only onset and wakeup predictions are used during inference.\n\nModel input: 8 hour window (5760 samples) with 4 features: normalized anglez, normalized enmo, hour sin and hour cos\n\nUsed around 20% of data for validation and trained on remaining 80%.\n\n## Additional improvements that boosted the score:\n1) Train more models\n2) Ensemble using average output of multiple models\n3) Use timeframes with overlap during inference, average out predictions from overlapping segments\n4) Tune threshold and peak gap - lower threshold with 5 minute gap between events produced the best results\n\n#Details of the submission\nModel published in [CMI-submit](https://www.kaggle.com/code/tubotubo/cmi-submit)\n|Base model|CV Score|Public Score|Private Score|\n|---|---|---|---|\n|Base model|0.7305|0.707|0.728|\n\n## Experiment with different models, architectures, settings\nI trained around 30 models with different encoders, feature extractors, downsampling rate, losses, architectures and sizes. Most of the models were trained with downsample rate 4 which sped up the training and did not decrease performance. From those models I selected best 6 for the final ensemble based on their score.\n\n|Feature extractor base filters|Encoder architecture|Decoder|downsample rate|CV Score|\n|---|---|---|---|---|\n|256|Unet|UNet1DDecoder|4|0.7545|\n|64|Unet|TransformerCNNDecoder|4|0.7526|\n|128|Unet|UNet1DDecoder|4|0.7525|\n|128|Unet|UNet1DDecoder|2|0.7519|\n|128|Unet|UNet1DDecoder|3|0.7510|\n|128| FPN|UNet1DDecoder|4|0.7486|\n\n## Ensemble\nAverage output of models was used as prediction. I tested mean, max, min and some simple weighted average based on CV score but simple average gave best results.\n|Ensemble|Ensemble method|CV Score|\n|---|---|---|\n|6 model ensemble|mean|0.7717|\n\n## Timeframe with overlaps during inference\nInference was made on 8 hour windows with 4 hour step, overlapping segments were avereged out. Bigger overlap gave better results but unfortunatelly finished few minutes after competition ended. Taking average from overlapping segments was better than taking minimum prediction, using maximum was giving the worst results.\n|Overlap during inference|CV Score|\n|---|---|\n|4 hours|0.7818|\n\n## Threshold and peak gap tuning\nLower threshold with bigger gap between events produced better score. Missing on event seems to be punished more than extra prediction but too many predicted events led to submission scoring error. Best score I was able to get was using 0.001 threshold and 60 sample distance (5 minutes) between predicted events.\n|Threshold|Min event distance|CV Score|Public Score|Private Score|\n|---|---|---|\n|0.001|60|0.7871|0.740|0.782|\n\n# Sources\n- https://www.kaggle.com/code/tubotubo/cmi-submit\n- https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940\n- https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/tree/main\n\n# Ending note - trust your CV or not\nI started with my own pipeline, using handcrafted features and decision tree models and later moved to NNs. Even though the models were getting really good CV score (over 0.8) the leaderboard score was always about 0.15 lower. I tried multiple CV strategies, random split, split based on length, number of nights without labels etc. and could not get a good CV strategy. I checked my code multiple times for leaks and was not able to find out what could be the cause. Getting discouraged by the low leaderboard score I abandoned my pipeline and switched to 213tubo's amazing solution about a week before the competition end. After the private leaderboard was revealed I found out that my pipeline was working just as well and only the public score was low. Lesson learned to trust your CV even though it's hard sometimes.", "2566951": "Congrats! I guess the one thing to takeaway from this competition, is that the train dataset is very robust (probably one of the best I have seen) that you can trust the CV score easily.\n\nFor my team, when the private LB was released, we realised our CV scores are always around 0.005-0.008 higher than the private LB, with little deviation. For once, the right submissions were chosen." }, "source": "existing_discussion", "complete": true, "competition_id": 53666, "competition_slug": "child-mind-institute-detect-sleep-states", "team_id": 11031115, "solution_url": "/competitions/child-mind-institute-detect-sleep-states/writeups/michal-86th-place-solution-for-the-child-mind-inst" }