# Ablation Study on Publicly-Shared Solutions through Late Submission

Overview
This report presents the results of an ablation study that I conducted locally and on the LB by incorporating ideas from solutions shared in the public discussion after the closure of the Kaggle competition
[Child Mind Institute - Detect Sleep States](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states)
.
Pipeline
The pipeline is similar to that described in [3]. The model, based on a 1D-CNN at the 1st stage, predicts the probability of onset/wakeup events for each steps, which is then post-processed into table data and further regressed with GBDT (Gradient Boosting Decision Tree) model by adding aggregated features of series/night.
It is important to note that in a multi-stage model, the pipeline sometimes become slightly different between training and inference phases. Therefore, eliminating domain shift in both the input features and the output results of the 2nd stage model during training and inference is crucial.
For example, in this ablation study, the average of the event probability predictions from 3 seeds per fold was calculated during training, and the results of the post-processing were used as input features. In contrast, during inference, the average of event probability predictions from 15 models (3 seeds x 5 folds) was used as input features (similar settings were employed in [2, 3]).
On the other hand, in my initial pipeline before starting this ablation study, I used the WBF (Weighted Box Fusion) ensemble results for each fold based on the average event probability predictions from 3 seeds as input features for the 2nd stage model. This configuration significantly worsened the correlation between local validation results and LB (Figure 2). The possible reason for this could be the domain shift in input features; the
step
optimized for AP used during training was shifted from its optimal point due to the ensemble with WBF during inference.
Based on such a hypothesis and switching to a pipeline similar to [2, 3], I was able to restore the correlation between local validation results and LB to the same level as before applying the 2nd stage model.
Thus, I leaned that in a multi-stage model, it is crucial to design the sequence of inputs and outputs to minimize domain shift between training and inference phases.
![](assets/2d2f8b86844dc36343eb9488.svg)
Figure 1. Overall Configuration of the 2-stage Pipeline
![](assets/54d1a0d2915a41b74f0eea4c.svg)
Figure 2. CB-LB Correlation over Different Pipeline Settings
1st Stage
Input Features
In addition to the features used in [1], I added the
minute % 15
feature which was claimed to be effective in [2]. Thus I utilized 6 features listed below.
numerical
log_anglez_std_2min
: The logarithm of the rolling standard deviation of angle-z over a 2-minute interval.
log_enmo
: The logarithm of ENMO.
hour_sin
: Positional encoding of the hour within one night.
hour_cos
: Positional encoding of the hour within one night.
categorical
is_valid
: A flag indicating whether the data is imputed or not. (1: normal data, 0: imputed data)
min_mod_15_plus_1
: minute % 15 + 1 (1-15: normal data, 0: padded data)
Model Architecture
The design is based on [2]. To reduce computation, I used the strides larger than in the original solution. Additionally, by adding the average-pooled feature maps from the raw series to the U-Net encoder, which is used in [1], significantly improved learning convergence and stabilization.
Note that this model is very lightweight, with the checkpoint size being only 3.5MB per model, totaling 54MB for 15 models. The training time is about 6 minutes per model for 20 epochs, and it takes about 1.5 hours for all 15 models using Nvidia RTX4090. Furthermore, the scoring time on Kaggle Notebook for the entire pipeline was approximately 20 minutes.
![](assets/918aa5613c3f9f2ab5b99fe5.svg)
Figure 3. The architecture of the 1st stage model
AP-Optimized Postprocessing
I made some improvements to the AP evaluation metric-optimized postprocessing that was shared in [2].
In [2], they computed the rolling sum of scores within a window of
k=2 * (tolerance_quarter_minute - 1) + 1
, whereas in this ablation study, I used rolling mean to aggregate scores of quarter points in each tolerance range. Since the total score aggregated by the rolling sum is proportional to the window size, it gives more weight to windows with greater tolerance. By using the rolling mean as the aggregation function, it's possible to mitigate the bias in weights for each tolerance.
2nd Stage
Model
I created a regression model using XGBoost[6].
Input Features
I utilized the following 14 features. All are extracted solely from the 1st stage model's prediction results (
series_id
,
event
,
night_id
,
step
,
score
), without using the original series features. The importance of these features are visualized in Figure 4.
event
rel_night_id
valid_rate_per_night
valid_rate_per_series
confidence
rank_per_night
mean_confidence_per_night
mean_confidence_per_series
norm_confidence_per_night
norm_confidence_per_series
abs_min_diff_from_HDCZA
[4]
abs_min_diff_from_top_0
sum_confidence_per_night
confidence_normalized_by_mean_per_night
![](assets/6bf39c235e2f031a8e83a5ad.png)
Figure 4. Importance of 2nd Stage Features
Target
The design of the label was inspired by [3]. The difference is that while [3] used top-k accuracy as the target in training phase, then applies diff of top-k accuracy as predicted score in inference phase, this ablation study regressed diff of top-k accuracy directly. Furthermore, the original implementation supplied binary labels by fixed threshold, but in this study, I used soft labels with logistic regression. The local validation results showed a slightly higher AP score with this configuration.
Ablation Study
The results of the Ablation Study are shown in Table 1.
The largest impact was observed from ablating the flag indicating whether the data was imputed or not (AP=-0.0236), presumably because the model generated many False Positives due to features indicative of spurious onset/wakeup. Note that the ablation of the flag for imputed data was achieved by providing
is_valid=1
for all series during inference, using a model that is trained
including
the
is_valid
feature. This is because I wanted to save time for training another models. Note that this settings can be slightly different to the result of the model
without
is_valid
feature both training and inference phases.
The contributions of the AP-optimized postprocessing and the ablation of the 2nd stage model were roughly equivalent (each reducing the local AP by -0.0075 and -0.0071, respectively).
The algorithm used in the original [2] (AP Optimization with rolling summation) alone decreased AP (local AP by -0.0144), but could be somewhat restored in performance by the 2nd stage model (local AP by -0.0016). However, I couldn't observe positive gain over the baseline using
find_peak
. Thus, this ablation study failed to achieve the huge gain as reported in [2] with the 2nd stage model. This is possibly due to not feeding the original input series features to the 2nd stage models. Feeding with richer input features could improve 2nd stage models higher beyond.
Importantly, as shown in this case, the impact of postprocessing which is optimized for the specific evaluation metrics
can be significant
. In tasks where AP is used as the evaluation metric for models (e.g., object detection tasks), substantial gains might be achieved through such kind of postprocessing, as used in [2] and this ablation study. Conversely, this also means evaluating solely on the AP metric might make it difficult to distinguish whether the results are evaluating the model's performance itself or an (over-)optimized outcome for the specific evaluation metrics. In the case of object detection tasks, fortunately, most literature applies only simple postprocessing, such as NMS (Non-Maximum Suppression), so the impact might not be as severe as I concerned. However, there are some examples, like DETR[5], which employ tricks such as allocating empty object query slots for predicting a second probable candidate bounding boxes to boost AP scores. When evaluating such models, careful judgment is required to determine whether the results are attributable to the model's performance itself or the results of postprocessing optimized for specific metrics.
Table 1. Results of the ablation study:
w
is_valid
feature
w AP Optimization (Sum)
w AP Optimization (Avg)
w 2nd Stage (Weight=0.2)
w 2nd Stage (Weight=1.0)
AP Local (OOF)
LB Public
LB Private
Scoring Time (min)
0.79
x
0.8136
x
x
0.7992
x
x
x
0.8009
x
x
x
0.812
x
x
0.8211
0.772
0.833
22
x
x
x
0.8263
0.77
0.838
20
x
x
x
0.8282
0.777
0.838
20
What Didn't Work
For the input features of the 2nd stage, I tried adding target encoding features based on the overall distribution of Ground truth, such as event timing, inspired by the idea in [1]. However, this did not improve the local validation score.
Regarding the features of the 2nd stage model, I also tested patterns using features similar to those used in [3], but this slightly worsened the local AP. Additionally, while some features improved the local validation score, they worsened the LB score. This stage's model is prone to overfitting and domain shift effects, so careful design of features is necessary.
Acknowledgements
I would like to express my gratitude to the hosts who organized the competition and to everyone at Kaggle management. I also extend my thanks to all the participants who shared their solutions and ideas.
Reference
[1]
[14th place solution by Jack](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/460274)
[2]
[1st place solution by shimacos, sakami, and kami](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459715)
[3]
[2nd place solution by K-mat](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459627)
[4]
[https://www.kaggle.com/code/tatamikenn/sleep-hdcza-a-pure-heuristic-approach-lb-0-447](https://www.kaggle.com/code/tatamikenn/sleep-hdcza-a-pure-heuristic-approach-lb-0-447)
[5]
[End-to-end object detection with Transformers](https://ai.meta.com/research/publications/end-to-end-object-detection-with-transformers/)
[6]
[https://xgboost.readthedocs.io/en/stable](https://xgboost.readthedocs.io/en/stable)