# 86th Place Solution for the Child Mind Institute - Detect Sleep States Competition First, big thanks to the organizers for preparing such an interesting competition. I really enjoyed working on it and learned a lot. I was quite surprised how well the NN models worked in this competition and how easily they surpassed the handcrafted features and decision tree models. Also big thanks to all the people who shared their knowledge in discussions and code. Especially to: [213tubo](https://www.kaggle.com/tubotubo) for [CMI-submit](https://www.kaggle.com/code/tubotubo/cmi-submit) [Daniel Phalen](https://www.kaggle.com/danielphalen) for [CMSS-GRUNET](https://www.kaggle.com/code/danielphalen/cmss-grunet-infer) [werus23](https://www.kaggle.com/werus23) for [Sleep - Critical point](https://www.kaggle.com/code/werus23/sleep-critical-point-infer) [Lucas Burke](https://www.kaggle.com/lccburk) for [Feature Engineering and Random Forest Prediction](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction) Even though my best solution was based on code published by [213tubo](https://www.kaggle.com/tubotubo) in [Detecting sleep using UNet(2D) post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) , I decided to publish the write up to show some simple ways how to improve on it and that it's possible to get good results using only limited resources. All training was done on Google Colab using T4 GPU instance. Context Business Context - [https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview) Data Context - [https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data) Overview of the Approach My best scoring pipeline is based on code published by [213tubo](https://www.kaggle.com/tubotubo) in [Detecting sleep using UNet(2D) post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) . Model As described by in the original post the model consist of feature extractor, encoder and decoder. Feature extractor takes raw input and transforms it into wavegram of image like shape. 3 Conv1d layers with different kernel sizes were used as feature extractors. Encoder takes the wavegram and creates a two dimensional heatmap. Unet and FPN models with resnet34 as backbone from [PyTorch Segmentation models library](https://github.com/qubvel/segmentation_models.pytorch) were used as encoders. Decoder creates probabilities of sleep, onset and wakeup from the heatmap generated by encoder. 1d Unet and TransformerEncoder were used as decoders. Only onset and wakeup predictions are used during inference. Model input: 8 hour window (5760 samples) with 4 features: normalized anglez, normalized enmo, hour sin and hour cos Used around 20% of data for validation and trained on remaining 80%. Additional improvements that boosted the score: 1) Train more models 2) Ensemble using average output of multiple models 3) Use timeframes with overlap during inference, average out predictions from overlapping segments 4) Tune threshold and peak gap - lower threshold with 5 minute gap between events produced the best results Details of the submission Model published in [CMI-submit](https://www.kaggle.com/code/tubotubo/cmi-submit) Base model CV Score Public Score Private Score Base model 0.7305 0.707 0.728 Experiment with different models, architectures, settings I trained around 30 models with different encoders, feature extractors, downsampling rate, losses, architectures and sizes. Most of the models were trained with downsample rate 4 which sped up the training and did not decrease performance. From those models I selected best 6 for the final ensemble based on their score. Feature extractor base filters Encoder architecture Decoder downsample rate CV Score 256 Unet UNet1DDecoder 4 0.7545 64 Unet TransformerCNNDecoder 4 0.7526 128 Unet UNet1DDecoder 4 0.7525 128 Unet UNet1DDecoder 2 0.7519 128 Unet UNet1DDecoder 3 0.7510 128 FPN UNet1DDecoder 4 0.7486 Ensemble Average output of models was used as prediction. I tested mean, max, min and some simple weighted average based on CV score but simple average gave best results. Ensemble Ensemble method CV Score 6 model ensemble mean 0.7717 Timeframe with overlaps during inference Inference was made on 8 hour windows with 4 hour step, overlapping segments were avereged out. Bigger overlap gave better results but unfortunatelly finished few minutes after competition ended. Taking average from overlapping segments was better than taking minimum prediction, using maximum was giving the worst results. Overlap during inference CV Score 4 hours 0.7818 Threshold and peak gap tuning Lower threshold with bigger gap between events produced better score. Missing on event seems to be punished more than extra prediction but too many predicted events led to submission scoring error. Best score I was able to get was using 0.001 threshold and 60 sample distance (5 minutes) between predicted events. Threshold Min event distance CV Score Public Score Private Score 0.001 60 0.7871 0.740 0.782 Sources [https://www.kaggle.com/code/tubotubo/cmi-submit](https://www.kaggle.com/code/tubotubo/cmi-submit) [https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) [https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/tree/main](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/tree/main) Ending note - trust your CV or not I started with my own pipeline, using handcrafted features and decision tree models and later moved to NNs. Even though the models were getting really good CV score (over 0.8) the leaderboard score was always about 0.15 lower. I tried multiple CV strategies, random split, split based on length, number of nights without labels etc. and could not get a good CV strategy. I checked my code multiple times for leaks and was not able to find out what could be the cause. Getting discouraged by the low leaderboard score I abandoned my pipeline and switched to 213tubo's amazing solution about a week before the competition end. After the private leaderboard was revealed I found out that my pipeline was working just as well and only the public score was low. Lesson learned to trust your CV even though it's hard sometimes.