![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fb5d384215833d8ef12b51cb9b1f4e84e%2Fcmi_pipeline.svg?generation=1704803887493821&alt=media\")
This report presents the results of an ablation study that I conducted locally and on the LB by incorporating ideas from solutions shared in the public discussion after the closure of the Kaggle competition Child Mind Institute - Detect Sleep States.
\nThe pipeline is similar to that described in [3]. The model, based on a 1D-CNN at the 1st stage, predicts the probability of onset/wakeup events for each steps, which is then post-processed into table data and further regressed with GBDT (Gradient Boosting Decision Tree) model by adding aggregated features of series/night.
\nIt is important to note that in a multi-stage model, the pipeline sometimes become slightly different between training and inference phases. Therefore, eliminating domain shift in both the input features and the output results of the 2nd stage model during training and inference is crucial.
\nFor example, in this ablation study, the average of the event probability predictions from 3 seeds per fold was calculated during training, and the results of the post-processing were used as input features. In contrast, during inference, the average of event probability predictions from 15 models (3 seeds x 5 folds) was used as input features (similar settings were employed in [2, 3]).
\nOn the other hand, in my initial pipeline before starting this ablation study, I used the WBF (Weighted Box Fusion) ensemble results for each fold based on the average event probability predictions from 3 seeds as input features for the 2nd stage model. This configuration significantly worsened the correlation between local validation results and LB (Figure 2). The possible reason for this could be the domain shift in input features; the step optimized for AP used during training was shifted from its optimal point due to the ensemble with WBF during inference.
Based on such a hypothesis and switching to a pipeline similar to [2, 3], I was able to restore the correlation between local validation results and LB to the same level as before applying the 2nd stage model.
\nThus, I leaned that in a multi-stage model, it is crucial to design the sequence of inputs and outputs to minimize domain shift between training and inference phases.
\n
Figure 1. Overall Configuration of the 2-stage Pipeline
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F4227916f1645584f265809cbc676f222%2Fcmi_cv_lb_correlation.svg?generation=1704803960433173&alt=media\")
Figure 2. CB-LB Correlation over Different Pipeline Settings
\nIn addition to the features used in [1], I added the minute % 15 feature which was claimed to be effective in [2]. Thus I utilized 6 features listed below.
numerical
\nlog_anglez_std_2min: The logarithm of the rolling standard deviation of angle-z over a 2-minute interval.log_enmo: The logarithm of ENMO.hour_sin: Positional encoding of the hour within one night.hour_cos: Positional encoding of the hour within one night.categorical
\nis_valid: A flag indicating whether the data is imputed or not. (1: normal data, 0: imputed data)min_mod_15_plus_1: minute % 15 + 1 (1-15: normal data, 0: padded data)The design is based on [2]. To reduce computation, I used the strides larger than in the original solution. Additionally, by adding the average-pooled feature maps from the raw series to the U-Net encoder, which is used in [1], significantly improved learning convergence and stabilization.
\nNote that this model is very lightweight, with the checkpoint size being only 3.5MB per model, totaling 54MB for 15 models. The training time is about 6 minutes per model for 20 epochs, and it takes about 1.5 hours for all 15 models using Nvidia RTX4090. Furthermore, the scoring time on Kaggle Notebook for the entire pipeline was approximately 20 minutes.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F145f459b4db889e7a678d4417f0b43cd%2Fcmi_1st_stage_architecture.svg?generation=1704803853869284&alt=media\")
Figure 3. The architecture of the 1st stage model
\nI made some improvements to the AP evaluation metric-optimized postprocessing that was shared in [2].
\nIn [2], they computed the rolling sum of scores within a window of k=2 * (tolerance_quarter_minute - 1) + 1, whereas in this ablation study, I used rolling mean to aggregate scores of quarter points in each tolerance range. Since the total score aggregated by the rolling sum is proportional to the window size, it gives more weight to windows with greater tolerance. By using the rolling mean as the aggregation function, it's possible to mitigate the bias in weights for each tolerance.
I created a regression model using XGBoost[6].
\nI utilized the following 14 features. All are extracted solely from the 1st stage model's prediction results (series_id, event, night_id, step, score), without using the original series features. The importance of these features are visualized in Figure 4.
eventrel_night_idvalid_rate_per_nightvalid_rate_per_seriesconfidencerank_per_nightmean_confidence_per_nightmean_confidence_per_seriesnorm_confidence_per_nightnorm_confidence_per_seriesabs_min_diff_from_HDCZA [4]abs_min_diff_from_top_0sum_confidence_per_nightconfidence_normalized_by_mean_per_night![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F485962d7022b3611992cf7d254dcfd43%2Fcmi_feature_importance_of_2nd_stage_features.png?generation=1704806006943498&alt=media\")
Figure 4. Importance of 2nd Stage Features
\nThe design of the label was inspired by [3]. The difference is that while [3] used top-k accuracy as the target in training phase, then applies diff of top-k accuracy as predicted score in inference phase, this ablation study regressed diff of top-k accuracy directly. Furthermore, the original implementation supplied binary labels by fixed threshold, but in this study, I used soft labels with logistic regression. The local validation results showed a slightly higher AP score with this configuration.
\nThe results of the Ablation Study are shown in Table 1.
\nThe largest impact was observed from ablating the flag indicating whether the data was imputed or not (AP=-0.0236), presumably because the model generated many False Positives due to features indicative of spurious onset/wakeup. Note that the ablation of the flag for imputed data was achieved by providing is_valid=1 for all series during inference, using a model that is trained including the is_valid feature. This is because I wanted to save time for training another models. Note that this settings can be slightly different to the result of the model without is_valid feature both training and inference phases.
The contributions of the AP-optimized postprocessing and the ablation of the 2nd stage model were roughly equivalent (each reducing the local AP by -0.0075 and -0.0071, respectively).
\nThe algorithm used in the original [2] (AP Optimization with rolling summation) alone decreased AP (local AP by -0.0144), but could be somewhat restored in performance by the 2nd stage model (local AP by -0.0016). However, I couldn't observe positive gain over the baseline using find_peak. Thus, this ablation study failed to achieve the huge gain as reported in [2] with the 2nd stage model. This is possibly due to not feeding the original input series features to the 2nd stage models. Feeding with richer input features could improve 2nd stage models higher beyond.
Importantly, as shown in this case, the impact of postprocessing which is optimized for the specific evaluation metrics can be significant. In tasks where AP is used as the evaluation metric for models (e.g., object detection tasks), substantial gains might be achieved through such kind of postprocessing, as used in [2] and this ablation study. Conversely, this also means evaluating solely on the AP metric might make it difficult to distinguish whether the results are evaluating the model's performance itself or an (over-)optimized outcome for the specific evaluation metrics. In the case of object detection tasks, fortunately, most literature applies only simple postprocessing, such as NMS (Non-Maximum Suppression), so the impact might not be as severe as I concerned. However, there are some examples, like DETR[5], which employ tricks such as allocating empty object query slots for predicting a second probable candidate bounding boxes to boost AP scores. When evaluating such models, careful judgment is required to determine whether the results are attributable to the model's performance itself or the results of postprocessing optimized for specific metrics.
\nTable 1. Results of the ablation study:
\nw is_valid feature | \nw AP Optimization (Sum) | \nw AP Optimization (Avg) | \nw 2nd Stage (Weight=0.2) | \nw 2nd Stage (Weight=1.0) | \nAP Local (OOF) | \nLB Public | \nLB Private | \nScoring Time (min) | \n
|---|---|---|---|---|---|---|---|---|
| \n | \n | \n | \n | \n | 0.79 | \n\n | \n | \n |
| x | \n\n | \n | \n | \n | 0.8136 | \n\n | \n | \n |
| x | \nx | \n\n | \n | \n | 0.7992 | \n\n | \n | \n |
| x | \nx | \n\n | x | \n\n | 0.8009 | \n\n | \n | \n |
| x | \nx | \n\n | \n | x | \n0.812 | \n\n | \n | \n |
| x | \n\n | x | \n\n | \n | 0.8211 | \n0.772 | \n0.833 | \n22 | \n
| x | \n\n | x | \n\n | x | \n0.8263 | \n0.77 | \n0.838 | \n20 | \n
| x | \n\n | x | \nx | \n\n | 0.8282 | \n0.777 | \n0.838 | \n20 | \n
For the input features of the 2nd stage, I tried adding target encoding features based on the overall distribution of Ground truth, such as event timing, inspired by the idea in [1]. However, this did not improve the local validation score.
\nRegarding the features of the 2nd stage model, I also tested patterns using features similar to those used in [3], but this slightly worsened the local AP. Additionally, while some features improved the local validation score, they worsened the LB score. This stage's model is prone to overfitting and domain shift effects, so careful design of features is necessary.
\nI would like to express my gratitude to the hosts who organized the competition and to everyone at Kaggle management. I also extend my thanks to all the participants who shared their solutions and ideas.
\nThis report presents the results of an ablation study that I conducted locally and on the LB by incorporating ideas from solutions shared in the public discussion after the closure of the Kaggle competition Child Mind Institute - Detect Sleep States.
\nThe pipeline is similar to that described in [3]. The model, based on a 1D-CNN at the 1st stage, predicts the probability of onset/wakeup events for each steps, which is then post-processed into table data and further regressed with GBDT (Gradient Boosting Decision Tree) model by adding aggregated features of series/night.
\nIt is important to note that in a multi-stage model, the pipeline sometimes become slightly different between training and inference phases. Therefore, eliminating domain shift in both the input features and the output results of the 2nd stage model during training and inference is crucial.
\nFor example, in this ablation study, the average of the event probability predictions from 3 seeds per fold was calculated during training, and the results of the post-processing were used as input features. In contrast, during inference, the average of event probability predictions from 15 models (3 seeds x 5 folds) was used as input features (similar settings were employed in [2, 3]).
\nOn the other hand, in my initial pipeline before starting this ablation study, I used the WBF (Weighted Box Fusion) ensemble results for each fold based on the average event probability predictions from 3 seeds as input features for the 2nd stage model. This configuration significantly worsened the correlation between local validation results and LB (Figure 2). The possible reason for this could be the domain shift in input features; the step optimized for AP used during training was shifted from its optimal point due to the ensemble with WBF during inference.
Based on such a hypothesis and switching to a pipeline similar to [2, 3], I was able to restore the correlation between local validation results and LB to the same level as before applying the 2nd stage model.
\nThus, I leaned that in a multi-stage model, it is crucial to design the sequence of inputs and outputs to minimize domain shift between training and inference phases.
\n
Figure 1. Overall Configuration of the 2-stage Pipeline
\n
Figure 2. CB-LB Correlation over Different Pipeline Settings
\nIn addition to the features used in [1], I added the minute % 15 feature which was claimed to be effective in [2]. Thus I utilized 6 features listed below.
numerical
\nlog_anglez_std_2min: The logarithm of the rolling standard deviation of angle-z over a 2-minute interval.log_enmo: The logarithm of ENMO.hour_sin: Positional encoding of the hour within one night.hour_cos: Positional encoding of the hour within one night.categorical
\nis_valid: A flag indicating whether the data is imputed or not. (1: normal data, 0: imputed data)min_mod_15_plus_1: minute % 15 + 1 (1-15: normal data, 0: padded data)The design is based on [2]. To reduce computation, I used the strides larger than in the original solution. Additionally, by adding the average-pooled feature maps from the raw series to the U-Net encoder, which is used in [1], significantly improved learning convergence and stabilization.
\nNote that this model is very lightweight, with the checkpoint size being only 3.5MB per model, totaling 54MB for 15 models. The training time is about 6 minutes per model for 20 epochs, and it takes about 1.5 hours for all 15 models using Nvidia RTX4090. Furthermore, the scoring time on Kaggle Notebook for the entire pipeline was approximately 20 minutes.
\n
Figure 3. The architecture of the 1st stage model
\nI made some improvements to the AP evaluation metric-optimized postprocessing that was shared in [2].
\nIn [2], they computed the rolling sum of scores within a window of k=2 * (tolerance_quarter_minute - 1) + 1, whereas in this ablation study, I used rolling mean to aggregate scores of quarter points in each tolerance range. Since the total score aggregated by the rolling sum is proportional to the window size, it gives more weight to windows with greater tolerance. By using the rolling mean as the aggregation function, it's possible to mitigate the bias in weights for each tolerance.
I created a regression model using XGBoost[6].
\nI utilized the following 14 features. All are extracted solely from the 1st stage model's prediction results (series_id, event, night_id, step, score), without using the original series features. The importance of these features are visualized in Figure 4.
eventrel_night_idvalid_rate_per_nightvalid_rate_per_seriesconfidencerank_per_nightmean_confidence_per_nightmean_confidence_per_seriesnorm_confidence_per_nightnorm_confidence_per_seriesabs_min_diff_from_HDCZA [4]abs_min_diff_from_top_0sum_confidence_per_nightconfidence_normalized_by_mean_per_night
Figure 4. Importance of 2nd Stage Features
\nThe design of the label was inspired by [3]. The difference is that while [3] used top-k accuracy as the target in training phase, then applies diff of top-k accuracy as predicted score in inference phase, this ablation study regressed diff of top-k accuracy directly. Furthermore, the original implementation supplied binary labels by fixed threshold, but in this study, I used soft labels with logistic regression. The local validation results showed a slightly higher AP score with this configuration.
\nThe results of the Ablation Study are shown in Table 1.
\nThe largest impact was observed from ablating the flag indicating whether the data was imputed or not (AP=-0.0236), presumably because the model generated many False Positives due to features indicative of spurious onset/wakeup. Note that the ablation of the flag for imputed data was achieved by providing is_valid=1 for all series during inference, using a model that is trained including the is_valid feature. This is because I wanted to save time for training another models. Note that this settings can be slightly different to the result of the model without is_valid feature both training and inference phases.
The contributions of the AP-optimized postprocessing and the ablation of the 2nd stage model were roughly equivalent (each reducing the local AP by -0.0075 and -0.0071, respectively).
\nThe algorithm used in the original [2] (AP Optimization with rolling summation) alone decreased AP (local AP by -0.0144), but could be somewhat restored in performance by the 2nd stage model (local AP by -0.0016). However, I couldn't observe positive gain over the baseline using find_peak. Thus, this ablation study failed to achieve the huge gain as reported in [2] with the 2nd stage model. This is possibly due to not feeding the original input series features to the 2nd stage models. Feeding with richer input features could improve 2nd stage models higher beyond.
Importantly, as shown in this case, the impact of postprocessing which is optimized for the specific evaluation metrics can be significant. In tasks where AP is used as the evaluation metric for models (e.g., object detection tasks), substantial gains might be achieved through such kind of postprocessing, as used in [2] and this ablation study. Conversely, this also means evaluating solely on the AP metric might make it difficult to distinguish whether the results are evaluating the model's performance itself or an (over-)optimized outcome for the specific evaluation metrics. In the case of object detection tasks, fortunately, most literature applies only simple postprocessing, such as NMS (Non-Maximum Suppression), so the impact might not be as severe as I concerned. However, there are some examples, like DETR[5], which employ tricks such as allocating empty object query slots for predicting a second probable candidate bounding boxes to boost AP scores. When evaluating such models, careful judgment is required to determine whether the results are attributable to the model's performance itself or the results of postprocessing optimized for specific metrics.
\nTable 1. Results of the ablation study:
\nw is_valid feature | \nw AP Optimization (Sum) | \nw AP Optimization (Avg) | \nw 2nd Stage (Weight=0.2) | \nw 2nd Stage (Weight=1.0) | \nAP Local (OOF) | \nLB Public | \nLB Private | \nScoring Time (min) | \n
|---|---|---|---|---|---|---|---|---|
| \n | \n | \n | \n | \n | 0.79 | \n\n | \n | \n |
| x | \n\n | \n | \n | \n | 0.8136 | \n\n | \n | \n |
| x | \nx | \n\n | \n | \n | 0.7992 | \n\n | \n | \n |
| x | \nx | \n\n | x | \n\n | 0.8009 | \n\n | \n | \n |
| x | \nx | \n\n | \n | x | \n0.812 | \n\n | \n | \n |
| x | \n\n | x | \n\n | \n | 0.8211 | \n0.772 | \n0.833 | \n22 | \n
| x | \n\n | x | \n\n | x | \n0.8263 | \n0.77 | \n0.838 | \n20 | \n
| x | \n\n | x | \nx | \n\n | 0.8282 | \n0.777 | \n0.838 | \n20 | \n
For the input features of the 2nd stage, I tried adding target encoding features based on the overall distribution of Ground truth, such as event timing, inspired by the idea in [1]. However, this did not improve the local validation score.
\nRegarding the features of the 2nd stage model, I also tested patterns using features similar to those used in [3], but this slightly worsened the local AP. Additionally, while some features improved the local validation score, they worsened the LB score. This stage's model is prone to overfitting and domain shift effects, so careful design of features is necessary.
\nI would like to express my gratitude to the hosts who organized the competition and to everyone at Kaggle management. I also extend my thanks to all the participants who shared their solutions and ideas.
\n\n\n**Figure 1. Overall Configuration of the 2-stage Pipeline**\n\n\n\n**Figure 2. CB-LB Correlation over Different Pipeline Settings**\n\n## 1st Stage\n\n### Input Features\n\nIn addition to the features used in [1], I added the `minute % 15` feature which was claimed to be effective in [2]. Thus I utilized 6 features listed below.\n\n**numerical**\n\n- `log_anglez_std_2min`: The logarithm of the rolling standard deviation of angle-z over a 2-minute interval.\n- `log_enmo`: The logarithm of ENMO.\n- `hour_sin`: Positional encoding of the hour within one night.\n- `hour_cos`: Positional encoding of the hour within one night.\n\n**categorical**\n\n- `is_valid`: A flag indicating whether the data is imputed or not. (1: normal data, 0: imputed data)\n- `min_mod_15_plus_1`: minute % 15 + 1 (1-15: normal data, 0: padded data)\n\n### Model Architecture\n\nThe design is based on [2]. To reduce computation, I used the strides larger than in the original solution. Additionally, by adding the average-pooled feature maps from the raw series to the U-Net encoder, which is used in [1], significantly improved learning convergence and stabilization.\n\nNote that this model is very lightweight, with the checkpoint size being only 3.5MB per model, totaling 54MB for 15 models. The training time is about 6 minutes per model for 20 epochs, and it takes about 1.5 hours for all 15 models using Nvidia RTX4090. Furthermore, the scoring time on Kaggle Notebook for the entire pipeline was approximately 20 minutes.\n\n\n\n**Figure 3. The architecture of the 1st stage model**\n\n### AP-Optimized Postprocessing\n\nI made some improvements to the AP evaluation metric-optimized postprocessing that was shared in [2].\n\nIn [2], they computed the rolling sum of scores within a window of `k=2 * (tolerance_quarter_minute - 1) + 1`, whereas in this ablation study, I used rolling mean to aggregate scores of quarter points in each tolerance range. Since the total score aggregated by the rolling sum is proportional to the window size, it gives more weight to windows with greater tolerance. By using the rolling mean as the aggregation function, it's possible to mitigate the bias in weights for each tolerance.\n\n## 2nd Stage\n\n### Model\n\nI created a regression model using XGBoost[6].\n\n### Input Features\n\nI utilized the following 14 features. All are extracted solely from the 1st stage model's prediction results (`series_id`, `event`, `night_id`, `step`, `score`), without using the original series features. The importance of these features are visualized in Figure 4.\n\n- `event`\n- `rel_night_id`\n- `valid_rate_per_night`\n- `valid_rate_per_series`\n- `confidence`\n- `rank_per_night`\n- `mean_confidence_per_night`\n- `mean_confidence_per_series`\n- `norm_confidence_per_night`\n- `norm_confidence_per_series`\n- `abs_min_diff_from_HDCZA` [4]\n- `abs_min_diff_from_top_0`\n- `sum_confidence_per_night`\n- `confidence_normalized_by_mean_per_night`\n\n\n\n**Figure 4. Importance of 2nd Stage Features**\n\n### Target\n\nThe design of the label was inspired by [3]. The difference is that while [3] used top-k accuracy as the target in training phase, then applies diff of top-k accuracy as predicted score in inference phase, this ablation study regressed diff of top-k accuracy directly. Furthermore, the original implementation supplied binary labels by fixed threshold, but in this study, I used soft labels with logistic regression. The local validation results showed a slightly higher AP score with this configuration.\n\n## Ablation Study\n\nThe results of the Ablation Study are shown in Table 1.\n\nThe largest impact was observed from ablating the flag indicating whether the data was imputed or not (AP=-0.0236), presumably because the model generated many False Positives due to features indicative of spurious onset/wakeup. Note that the ablation of the flag for imputed data was achieved by providing `is_valid=1` for all series during inference, using a model that is trained *including* the `is_valid` feature. This is because I wanted to save time for training another models. Note that this settings can be slightly different to the result of the model *without* `is_valid` feature both training and inference phases.\n\nThe contributions of the AP-optimized postprocessing and the ablation of the 2nd stage model were roughly equivalent (each reducing the local AP by -0.0075 and -0.0071, respectively).\n\nThe algorithm used in the original [2] (AP Optimization with rolling summation) alone decreased AP (local AP by -0.0144), but could be somewhat restored in performance by the 2nd stage model (local AP by -0.0016). However, I couldn't observe positive gain over the baseline using `find_peak`. Thus, this ablation study failed to achieve the huge gain as reported in [2] with the 2nd stage model. This is possibly due to not feeding the original input series features to the 2nd stage models. Feeding with richer input features could improve 2nd stage models higher beyond.\n\nImportantly, as shown in this case, the impact of postprocessing which is optimized for the specific evaluation metrics *can be significant*. In tasks where AP is used as the evaluation metric for models (e.g., object detection tasks), substantial gains might be achieved through such kind of postprocessing, as used in [2] and this ablation study. Conversely, this also means evaluating solely on the AP metric might make it difficult to distinguish whether the results are evaluating the model's performance itself or an (over-)optimized outcome for the specific evaluation metrics. In the case of object detection tasks, fortunately, most literature applies only simple postprocessing, such as NMS (Non-Maximum Suppression), so the impact might not be as severe as I concerned. However, there are some examples, like DETR[5], which employ tricks such as allocating empty object query slots for predicting a second probable candidate bounding boxes to boost AP scores. When evaluating such models, careful judgment is required to determine whether the results are attributable to the model's performance itself or the results of postprocessing optimized for specific metrics.\n\n**Table 1. Results of the ablation study:**\n\n| w `is_valid` feature | w AP Optimization (Sum) | w AP Optimization (Avg) | w 2nd Stage (Weight=0.2) | w 2nd Stage (Weight=1.0) | AP Local (OOF) | LB Public | LB Private | Scoring Time (min) |\n| -------------------- | ----------------------- | ----------------------- | ------------------------ | ------------------------ | -------------- | --------- | ---------- | --------------------- |\n| | | | | | 0.79 | | | |\n| x | | | | | 0.8136 | | | |\n| x | x | | | | 0.7992 | | | |\n| x | x | | x | | 0.8009 | | | |\n| x | x | | | x | 0.812 | | | |\n| x | | x | | | 0.8211 | 0.772 | 0.833 | 22 |\n| x | | x | | x | 0.8263 | 0.77 | 0.838 | 20 |\n| x | | x | x | | 0.8282 | 0.777 | 0.838 | 20 |\n\n## What Didn't Work\n\nFor the input features of the 2nd stage, I tried adding target encoding features based on the overall distribution of Ground truth, such as event timing, inspired by the idea in [1]. However, this did not improve the local validation score.\n\nRegarding the features of the 2nd stage model, I also tested patterns using features similar to those used in [3], but this slightly worsened the local AP. Additionally, while some features improved the local validation score, they worsened the LB score. This stage's model is prone to overfitting and domain shift effects, so careful design of features is necessary.\n\n## Acknowledgements\n\nI would like to express my gratitude to the hosts who organized the competition and to everyone at Kaggle management. I also extend my thanks to all the participants who shared their solutions and ideas.\n\n## Reference\n\n- [1] [14th place solution by Jack](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/460274)\n- [2] [1st place solution by shimacos, sakami, and kami](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459715)\n- [3] [2nd place solution by K-mat](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459627)\n- [4] https://www.kaggle.com/code/tatamikenn/sleep-hdcza-a-pure-heuristic-approach-lb-0-447\n- [5] [End-to-end object detection with Transformers](https://ai.meta.com/research/publications/end-to-end-object-detection-with-transformers/)\n- [6] https://xgboost.readthedocs.io/en/stable",
"votes": null
},
{
"id": "2593809",
"postDate": "01/09/2024 13:28:18",
"content": "The below are the visualization of 1st stage scores and 2nd scores sampled from night chunks. In the general tendency, the model fails to increase confidence of lower-ranked predictions where the GT events are placed nearby. However, it seems to succeed in increasing the confidence of 1st models for higher-ranked predictions. This can be because of with the information gained from aggregated features over series_id or night_id, they can successfully decide whether and how much confidence the 2nd model should increase over these predictions.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Feb754f608959c110eb45a1395de40f38%2Fsample_01.png?generation=1704806148833178&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F82320fe1c18d896e2817f5496fd6e745%2Fsample_02.png?generation=1704806160612952&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F644ff610a10ee6039e4b8de00b7746e7%2Fsample_03.png?generation=1704806469987441&alt=media\")
The below are the visualization of 1st stage scores and 2nd scores sampled from night chunks. In the general tendency, the model fails to increase confidence of lower-ranked predictions where the GT events are placed nearby. However, it seems to succeed in increasing the confidence of 1st models for higher-ranked predictions. This can be because of with the information gained from aggregated features over series_id or night_id, they can successfully decide whether and how much confidence the 2nd model should increase over these predictions.
\n\n**Figure 1. Overall Configuration of the 2-stage Pipeline**\n\n\n\n**Figure 2. CB-LB Correlation over Different Pipeline Settings**\n\n## 1st Stage\n\n### Input Features\n\nIn addition to the features used in [1], I added the `minute % 15` feature which was claimed to be effective in [2]. Thus I utilized 6 features listed below.\n\n**numerical**\n\n- `log_anglez_std_2min`: The logarithm of the rolling standard deviation of angle-z over a 2-minute interval.\n- `log_enmo`: The logarithm of ENMO.\n- `hour_sin`: Positional encoding of the hour within one night.\n- `hour_cos`: Positional encoding of the hour within one night.\n\n**categorical**\n\n- `is_valid`: A flag indicating whether the data is imputed or not. (1: normal data, 0: imputed data)\n- `min_mod_15_plus_1`: minute % 15 + 1 (1-15: normal data, 0: padded data)\n\n### Model Architecture\n\nThe design is based on [2]. To reduce computation, I used the strides larger than in the original solution. Additionally, by adding the average-pooled feature maps from the raw series to the U-Net encoder, which is used in [1], significantly improved learning convergence and stabilization.\n\nNote that this model is very lightweight, with the checkpoint size being only 3.5MB per model, totaling 54MB for 15 models. The training time is about 6 minutes per model for 20 epochs, and it takes about 1.5 hours for all 15 models using Nvidia RTX4090. Furthermore, the scoring time on Kaggle Notebook for the entire pipeline was approximately 20 minutes.\n\n\n\n**Figure 3. The architecture of the 1st stage model**\n\n### AP-Optimized Postprocessing\n\nI made some improvements to the AP evaluation metric-optimized postprocessing that was shared in [2].\n\nIn [2], they computed the rolling sum of scores within a window of `k=2 * (tolerance_quarter_minute - 1) + 1`, whereas in this ablation study, I used rolling mean to aggregate scores of quarter points in each tolerance range. Since the total score aggregated by the rolling sum is proportional to the window size, it gives more weight to windows with greater tolerance. By using the rolling mean as the aggregation function, it's possible to mitigate the bias in weights for each tolerance.\n\n## 2nd Stage\n\n### Model\n\nI created a regression model using XGBoost[6].\n\n### Input Features\n\nI utilized the following 14 features. All are extracted solely from the 1st stage model's prediction results (`series_id`, `event`, `night_id`, `step`, `score`), without using the original series features. The importance of these features are visualized in Figure 4.\n\n- `event`\n- `rel_night_id`\n- `valid_rate_per_night`\n- `valid_rate_per_series`\n- `confidence`\n- `rank_per_night`\n- `mean_confidence_per_night`\n- `mean_confidence_per_series`\n- `norm_confidence_per_night`\n- `norm_confidence_per_series`\n- `abs_min_diff_from_HDCZA` [4]\n- `abs_min_diff_from_top_0`\n- `sum_confidence_per_night`\n- `confidence_normalized_by_mean_per_night`\n\n\n\n**Figure 4. Importance of 2nd Stage Features**\n\n### Target\n\nThe design of the label was inspired by [3]. The difference is that while [3] used top-k accuracy as the target in training phase, then applies diff of top-k accuracy as predicted score in inference phase, this ablation study regressed diff of top-k accuracy directly. Furthermore, the original implementation supplied binary labels by fixed threshold, but in this study, I used soft labels with logistic regression. The local validation results showed a slightly higher AP score with this configuration.\n\n## Ablation Study\n\nThe results of the Ablation Study are shown in Table 1.\n\nThe largest impact was observed from ablating the flag indicating whether the data was imputed or not (AP=-0.0236), presumably because the model generated many False Positives due to features indicative of spurious onset/wakeup. Note that the ablation of the flag for imputed data was achieved by providing `is_valid=1` for all series during inference, using a model that is trained *including* the `is_valid` feature. This is because I wanted to save time for training another models. Note that this settings can be slightly different to the result of the model *without* `is_valid` feature both training and inference phases.\n\nThe contributions of the AP-optimized postprocessing and the ablation of the 2nd stage model were roughly equivalent (each reducing the local AP by -0.0075 and -0.0071, respectively).\n\nThe algorithm used in the original [2] (AP Optimization with rolling summation) alone decreased AP (local AP by -0.0144), but could be somewhat restored in performance by the 2nd stage model (local AP by -0.0016). However, I couldn't observe positive gain over the baseline using `find_peak`. Thus, this ablation study failed to achieve the huge gain as reported in [2] with the 2nd stage model. This is possibly due to not feeding the original input series features to the 2nd stage models. Feeding with richer input features could improve 2nd stage models higher beyond.\n\nImportantly, as shown in this case, the impact of postprocessing which is optimized for the specific evaluation metrics *can be significant*. In tasks where AP is used as the evaluation metric for models (e.g., object detection tasks), substantial gains might be achieved through such kind of postprocessing, as used in [2] and this ablation study. Conversely, this also means evaluating solely on the AP metric might make it difficult to distinguish whether the results are evaluating the model's performance itself or an (over-)optimized outcome for the specific evaluation metrics. In the case of object detection tasks, fortunately, most literature applies only simple postprocessing, such as NMS (Non-Maximum Suppression), so the impact might not be as severe as I concerned. However, there are some examples, like DETR[5], which employ tricks such as allocating empty object query slots for predicting a second probable candidate bounding boxes to boost AP scores. When evaluating such models, careful judgment is required to determine whether the results are attributable to the model's performance itself or the results of postprocessing optimized for specific metrics.\n\n**Table 1. Results of the ablation study:**\n\n| w `is_valid` feature | w AP Optimization (Sum) | w AP Optimization (Avg) | w 2nd Stage (Weight=0.2) | w 2nd Stage (Weight=1.0) | AP Local (OOF) | LB Public | LB Private | Scoring Time (min) |\n| -------------------- | ----------------------- | ----------------------- | ------------------------ | ------------------------ | -------------- | --------- | ---------- | --------------------- |\n| | | | | | 0.79 | | | |\n| x | | | | | 0.8136 | | | |\n| x | x | | | | 0.7992 | | | |\n| x | x | | x | | 0.8009 | | | |\n| x | x | | | x | 0.812 | | | |\n| x | | x | | | 0.8211 | 0.772 | 0.833 | 22 |\n| x | | x | | x | 0.8263 | 0.77 | 0.838 | 20 |\n| x | | x | x | | 0.8282 | 0.777 | 0.838 | 20 |\n\n## What Didn't Work\n\nFor the input features of the 2nd stage, I tried adding target encoding features based on the overall distribution of Ground truth, such as event timing, inspired by the idea in [1]. However, this did not improve the local validation score.\n\nRegarding the features of the 2nd stage model, I also tested patterns using features similar to those used in [3], but this slightly worsened the local AP. Additionally, while some features improved the local validation score, they worsened the LB score. This stage's model is prone to overfitting and domain shift effects, so careful design of features is necessary.\n\n## Acknowledgements\n\nI would like to express my gratitude to the hosts who organized the competition and to everyone at Kaggle management. I also extend my thanks to all the participants who shared their solutions and ideas.\n\n## Reference\n\n- [1] [14th place solution by Jack](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/460274)\n- [2] [1st place solution by shimacos, sakami, and kami](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459715)\n- [3] [2nd place solution by K-mat](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459627)\n- [4] https://www.kaggle.com/code/tatamikenn/sleep-hdcza-a-pure-heuristic-approach-lb-0-447\n- [5] [End-to-end object detection with Transformers](https://ai.meta.com/research/publications/end-to-end-object-detection-with-transformers/)\n- [6] https://xgboost.readthedocs.io/en/stable",
"2593809": "## Appendix: The visualization of 2nd stage model's predictions\n\nThe below are the visualization of 1st stage scores and 2nd scores sampled from night chunks. In the general tendency, the model fails to increase confidence of lower-ranked predictions where the GT events are placed nearby. However, it seems to succeed in increasing the confidence of 1st models for higher-ranked predictions. This can be because of with the information gained from aggregated features over `series_id` or `night_id`, they can successfully decide whether and how much confidence the 2nd model should increase over these predictions.\n\n\n\n\n\n"
},
"source": "existing_discussion",
"complete": true,
"competition_id": 53666,
"competition_slug": "child-mind-institute-detect-sleep-states",
"team_id": 11134590,
"solution_url": "/competitions/child-mind-institute-detect-sleep-states/writeups/bilzard-ablation-study-on-publicly-shared-solution"
}