# 1st place solution

First of all, I would like to express gratitude to all participants and the competition host. It was a challenging competition, but I am pleased with the positive outcome and feel relieved.
Here is a brief summary of our solution.
You can check our code
[here](https://github.com/sakami0000/child-mind-institute-detect-sleep-states-1st-place)
.
Single model
Here's a log on how to improve the CV score after the summary. The final scores were: CV: 0.8206, public LB: 0.768, private LB: 0.829 (equivalent to 9th place).
Summary
Model structure
The model structure is primarily based on
[this amazing notebook](https://www.kaggle.com/code/danielphalen/cmss-grunet-train)
, with a structure comprising:
CNN (down sample) → Residual GRU → CNN (up sample)
![](assets/e96819f20b0c38bc50118efb.jpeg)
SEScale
For input scaling, SEModule was utilized. (
[https://arxiv.org/abs/1709.01507](https://arxiv.org/abs/1709.01507)
)
(nn.Module):
    () -> :
       ().__init__()
       self.fc1 = nn.Linear(ch, r)
       self.fc2 = nn.Linear(r, ch)

    () -> torch.FloatTensor:
       h = self.fc1(x)
       h = F.relu(h)
       h = self.fc2(h).sigmoid()
        h * x
Minute connection
As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred. To account for this, features related to minutes were concatenated separately in the final layer.
() -> torch.FloatTensor:
       cat_embeddings = [embedding(cat_x[:, :, i])  i, embedding  (self.category_embeddings)]
       num_x = self.numerical_linear(num_x)

       x = torch.cat([num_x] + cat_embeddings, dim=)
       x = self.input_linear(x)
       x = self.conv(x.transpose(-, -)).transpose(-, -)

        gru  self.gru_layers:
           x, _ = gru(x)

       x = self.dconv(x.transpose(-, -)).transpose(-, -)
       minute_embedding = self.minute_embedding(cat_x[:, :, ])
       x = self.output_linear(torch.cat([x, minute_embedding], dim=))
        x
Data Preparation
Each series of data is divided into daily chunks, offset by 0.35 days.
train_df = train_df.with_columns(pl.arange(, pl.count()).alias())
       series_row_ids = (train_df.group_by().agg().rows())

       series_chunk_ids = []  
       series_chunk_row_ids = []  
        series_id, row_ids  tqdm(series_row_ids.items(), desc=):
            start_idx  (, (row_ids), (config.stride_size / config.epoch_sample_rate)):
                start_idx + config.chunk_size <= (row_ids):
                   chunk_row_ids = row_ids[start_idx : start_idx + config.chunk_size]
                   series_chunk_ids.append(series_id)
                   series_chunk_row_ids.append(np.array(chunk_row_ids))
               :
                   chunk_row_ids = row_ids[-config.chunk_size :]
                   series_chunk_ids.append(series_id)
                   series_chunk_row_ids.append(np.array(chunk_row_ids))
During training, half of each chunk is used in every epoch.
sampled_train_idx = train_idx[epoch % config.epoch_sample_rate :: config.epoch_sample_rate]
For evaluation, overlapping sections are averaged, and the ends of each chunk are trimmed by 30 minutes.
Target
A decaying target is created based on the distance from the ground truth event, with diminishing values as the distance increases.
![](assets/7dfea36b4eb2dcdfe8de16d5.png)
tolerance_steps = [, , , , , , , , , ]
target_columns = [, ]

train_df = (
   train_df.join(train_events_df.select([, , ]), on=[, ], how=)
   .to_dummies(columns=[])
   .with_columns(
       pl.max_horizontal(
           pl.col(target_columns)
           .rolling_max(window_size *  - , min_periods=, center=)
           .over()
           * ( - i / (tolerance_steps))
            i, window_size  (tolerance_steps)
       )
   )
)
The target is updated each epoch to decay further.
targets = np.where(targets == , , (targets - ( / config.n_epochs)).clip(=))
By attenuating the target, the range of predicted values narrows, allowing for the detection of finer peaks.
![](assets/48697cb2e1251be9638edd43.png)
Periodicity Filter
As discussed in
[here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/456177)
, there is a daily periodicity in the data when the measuring device is removed. This is leveraged to predict these periods rule-based and used as a filter for inputs and predictions.
![](assets/b010b930e369c640d7661da9.png)
Input Features
categorical features
hour
minute
weekday
periodicity flag
numerical features
anglez / 45
enmo.log1p().clip_max(1.0) / 0.1
anglez, enmo 12 steps rolling_mean, rolling_std, rolling_max
anglez_diff_abs 5 min rolling median
Change logs
baseline model (cv: 0.7510) - public: 0.728
Add a process to decay the target every epoch (cv: 0.7699, +19pt)
Add a periodicity filter to the output (cv: 0.7807, +11pt)
Add a periodicity flag to the input as well (cv: 0.7870, +6pt) - public: 0.739
batch_size: 16 → 4, hidden_size: 128 → 64, num_layers: 2 → 8 (cv: 0.7985, +11pt) - public: 0.755
Normalize the score in the submission file by the daily score sum (cv: 0.8044, +6pt)
Remove month and day from the input (cv: 0.8117, +7pt)
Trim the edges of the chunk by 30 minutes on both sides (cv: 0.8142, +4pt) - public: 0.765
Modify to concatenate the minute features to the final layer (cv: 0.8206, +6pt) - public: 0.768
Post Processing
This post-processing creates a submission DataFrame to optimize the evaluation metrics. With this post-processing method, we significantly improved our scores (public: 0.768 →
0.790
, private: 0.829 →
0.852
!!!).
This was a complex procedure, which I will explain step by step.
Data Characteristics
First, let's discuss the characteristics of the data. As noted in several discussions and notebooks, the second of the target events was always set to zero.
The competition's evaluation metric doesn't differentiate predictions within a 30-second range from the ground truth event. So, whether the submission timestamp's seconds are 5, 10, 15, 20, … 25, the same score is returned.
![](assets/835c493e86f41c5e717b09c0.png)
Creation of the 2nd Level Model
The 1st level model's predictions were trained to recognize events within a certain range from the ground truth as positive. However, the 2nd level model transforms these into probabilities of a ground truth event existing for each minute.
The output of the 1st level model was at seconds 0, 5, 10, …, but the 2nd level model aggregates these to always be at second 0. Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0. Details of the 2nd level model will be described later.
![](assets/f0d5367f69d1ef106f6dccab.png)
Score Calculation for Each Point
As explained earlier, submitting any second within the same minute yields the same score. Therefore, we estimate the score at the 15 and 45 second points of each minute, and submit the one with the highest value, effectively submitting the highest score for all points. The method of score estimation is as follows:
For instance, let's estimate the score at 10:00:15.
![](assets/cacbdaa466de736defbee80a.png)
First, we create a window of 12 steps from the point of interest and sum the predictions of the 2nd level model within this window to calculate the
tolerance_12_score
.
![](assets/b3ecb3cfbdb538c9931bdc9b.png)
Similarly, we calculate
tolerance_36_score
,
tolerance_60_score
, …, for the respective tolerances used in the evaluation, and the sum of these scores is considered the score for the point of interest.
![](assets/8a0ed4bd8e6659b222b0a48f.png)
We perform this calculation for all points, and for each series, we adopt the point with the highest score and add it to the submission DataFrame.
![](assets/a8d1f9062c4826c9b8b9d70f.png)
Score Recalculation
Next, we recalculate the score to determine the next point to be adopted. For example, suppose the point 09:59:15 was chosen.
First, consider updating the
tolerance_12_score
. Events within tolerance 12 of the adopted point cannot match overlappingly with the next point to be submitted.
![](assets/39caba7a06262e4c1eb19625.png)
Therefore, when calculating the
tolerance_12_score
for the next point to be adopted, it's necessary to discount the prediction values within 12 steps of the currently adopted point.
![](assets/cf949b96191a8469a8b56357.png)
Likewise, for
tolerance_36_score
,
tolerance_60_score
, …, we recalculate by discounting the prediction values within 36, 60, …, steps of the adopted point.
![](assets/4ffc5d6b5ccd58ce05471cbc.png)
With the updated scores calculated, we again adopt the highest scoring point for each series and add it to the submission dataframe.
![](assets/4aeacdea8dfd8de3c5d1cf88.png)
Creating Submissions
We repeat the above step 4 to extract a sufficient number of submission points, then compile these into a DataFrame to create the submission file.
Additional Techniques
Several other techniques were employed to make the post-processing work effectively:
Normalize the predictions of the 2nd level model daily.
When recalculating the score, calculate the difference from the previous score to reduce the computation.
Speed up the above calculations using JIT compilation.
Details of the 2nd Level Model
The 2nd level model starts by averaging the 1st level model's predictions on a per-minute basis and then detecting peaks in these averages using
find_peaks
with a height of 0.001 and a distance of 8.
Based on the detected peaks, chunks are created from the original time series, capturing 8 minutes before and after each peak. (Recall: 0.9845)
This step_size was crucial because the ratio of positive to negative examples changes depending on how many steps are included, affecting the accuracy of subsequent stages. Therefore, we tuned the number of steps for optimal performance.
If chunks are connected, they are treated as a single chunk.
![](assets/f51fad46e99603813766f058.png)
For each chunk, we aggregated features from the 1st model's predictions and other features like anglez and enmo. These aggregated features were then used to train models such as LightGBM and CatBoost.
Additionally, we treated each chunk as a sequence for training CNN-RNN, CNN, and Transformer models. As a result, we developed a model that could account for minute biases not fully addressed by the 1st level model.
![](assets/25f3b26af891896a78c3c852.png)
The predictions of the 2nd level model were sufficiently calibrated, so there was no need for further transformation.
![](assets/11dbdc78e4c71a10f9fabec8.png)