# 858th Place Solution: MLP-BiGRU

First, I'd like to thank the organizers and Kaggle team for this interesting competition. Second, a huge congratulation to all the winners and everyone who participated in this competition. It was a very great competition for me to learn more about time-series approaches. Finally, many thanks to
[werus23](https://www.kaggle.com/werus23)
,
[Tolga](https://www.kaggle.com/tolgadincer)
,
[213tubo](https://www.kaggle.com/tubotubo)
, and
[Chris Deotte](https://www.kaggle.com/cdeotte)
for sharing their valuable knowledge and discussions throughout the competition. It was super helpful for me to understand and develop the final solution for this competition.
1. Data preparation:
Data
: I truncated the original data at the end based on the valid events. Then, I added some engineered features based on the public notebooks and codes from
[[1]](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data)
[[2]](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py)
. The 18 final features I selected for my training/inference phase were:
num_features = ['anglez', 'enmo', 
                'anglez_lag_diff', 'anglez_lead_diff','anglez_mean_60','anglez_max_60','anglez_std_60',
                'enmo_lag_diff','enmo_lead_diff','enmo_mean_60', 'enmo_max_60','enmo_std_60', 
                'hour_sin','hour_cos','month_sin','month_cos','minute_sin','minute_cos'
               ]
Label
: There were 2 approaches for preparing the labels in this problem: hard labelling vs soft labelling. Hard labelling is created by pre-selecting the number of offset from the events and considering it as a part of the training labels. In contrast, soft labeling is created using Gaussian distribution around the onset/awake positions with different sigma and offset values
[[3]](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py)
. This approach has demonstrated much better results compared to hard labelling in this competition. Since I didn't have a good understanding of the soft labelling approach, I shifted my focus on training hard labelled data. A visualization of these two approaches is as follows (with an offset of 180 on each side):
![](assets/6c05d480a4c23428a27a56ce.png)
2. Cross-validation:
I used 269 series ids with group 10-fold. I initially used K-Fold but after reading
[Chris's discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882#2491744)
, I switched to group K-Fold and it boosted my LB score by ~0.05. However, I only selected fold 4 and 7 for training purposes.
Valid series ids:
fold_4:
['062dbd4c95e6', '0ef7d94fde99', '154fe824ed87', '280e08693c6d',
   '31011ade7c0a', '3aceb17ef7bd', '4743bdde25df', '51fdcc8d9fe7',
   '5f40907ec171', '653622ac8363', '703b5efa9bc1', '7822ee8fe3ec',
   '87a6cbb7c4ed', '8877a6586606', '89bd631d1769', '91127c2b0e60',
   '971207c6a525', '9b9cd7b7af8c', 'aed3850f65f0', 'ba8083a2c3b8',
   'c38707ef76df', 'cca14d1966c1', 'd2d6b9af0553', 'db5e0ee1c0ab',
   'e69aff66e0cb', 'eef041dd50aa', 'f981a0805fd0']
fold_7:
['04f547b8017d', '0dee4fda51c3', '137b99e936ab', '25e2b3dd9c3b',
   '2e9ced2c7976', '3664fe9233f9', '3c336d6ba566', '449766346eb1',
   '5c088d7e916c', '601559e1777d', '6d6b9d22d48a', '752900afe3a6',
   '83fa182bec3a', '8f6f15b9f598', '99b829cbad2d', 'a2b0a64ec9cf',
   'a88088855de5', 'a9e5f5314bcb', 'b750c8c1556c', 'c107b5789660',
   'd0f613c700f7', 'd93b0c7de16b', 'dff367373725', 'e0686434d029',
   'e34b496b84ce', 'ece2561f07e9', 'f7eb179216c2']
3. Model architecture:
The model architecture was adapted and modified based on this MLP-Residual BiGRU model
[[4]](https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook)
[[5]](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410)
.
![](assets/8fa73a567203483819c61cfb.png)
4. Training:
Optimizer: Adam
LR=1.0e-3
Scheduler: CosineAnnealing
T_max=10
Epochs=10, batch_size=3
Loss function:
BCEWithLogitsLoss()
Evaluation metrics:
Event Detection AP
,
precision_score(average='macro')
![](assets/4bbb69ab1642c315089b6996.png)
5. Inference:
Used the same preprocessing steps as training phase, except there was no truncating in the test series.
Inferred with batch size = 1, and input_size=(17280,18)
Filtered out insignificant predictions
Added as many onset/awake positions as possible within the input time frame of 17280 (24 hours)
Selected only middle point of onset/awake transition periods
![](assets/e09419e8d12a303296ca9d5e.png)
6. Results:
Fold
Model
Hidden_size
CV
Public LB
Private LB
4
model_0
128
0.743
0.681
0.706
4
model_1
224
0.737
0.675
0.716
7
model_2
128
0.748
0.680
0.71
7
model_3
224
0.745
0.677
0.711
Ensemble
public LB
:  0.25*(model_0+model_1+mode_2+model_3) =
0.687
Ensemble
private LB
: 0.25*(model_0+model_1+mode_2+model_3) =
0.716
7. Conclusion:
Things I tried but unsuccessful 🫠:
WaveNet
UNet
Spectrogram-UNet
Transformer
Soft-labelling
References:
[[1] Sleep critical point prepare data-werus23](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data)
[[2] Prepare data-tubo213](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py)
[[3] Common utils funtions-tubo213](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py)
[[4] Sleep critical point train-werus23](https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook)
[[5] 4th Place Solution: a MultiLayer Bidirectional GRU with Residual Connections-zinxira](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410)
Thanks for reading! 😀