{
  "id": 461154,
  "title": "41st Place Solution for the Child Mind Institute - Detect Sleep States Competition",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/461154",
  "author_name": "YuYagi",
  "post_date": "2023-12-12T22:09:26.774000",
  "votes": 7,
  "comment_count": 0,
  "views": 0,
  "content": "<p>First, I would like to express my sincere thanks to the organizers of the Child Mind Institute - Detect Sleep States Competition. This competition provided an invaluable opportunity to engage with long one-dimensional sequences, offering a distinct and educational experience in an important area of data science.</p>\n<h2><strong>Overview of the Inference Pipeline</strong></h2>\n<p>Our training pipeline and models are based on the excellent code by <a href=\"https://www.kaggle.com/tubotubo\" target=\"_blank\">@213tubo</a>, available at <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940?rvi=1\" target=\"_blank\">this page</a>. </p>\n<h4><strong>Our solution is as follows</strong>:</h4>\n<ul>\n<li><p><strong>Data Segmentation:</strong></p>\n<ul>\n<li>Segmented input features using overlapping time windows for inference</li></ul></li>\n<li><p><strong>Models:</strong></p>\n<ul>\n<li>Utilized 10 Unet-based models, each trained on two distinct training-validation splits across five unique models.</li>\n<li>Each of the 5 models has a different backbone.</li></ul></li>\n<li><p><strong>PostProcessing(Single Model):</strong></p>\n<ul>\n<li>Applied Non-Maximum Suppression (NMS) to retain only the highest-scoring prediction when detected peaks were within a specified proximity.</li></ul></li>\n<li><p><strong>Ensemble Predictions:</strong></p>\n<ul>\n<li>Adopted an original ensemble method inspired by Weighted Boxes Fusion (WBF)</li>\n<li>In the first stage, models within the same fold were assembled.</li>\n<li>In the second stage, these results from different folds were combined.</li></ul></li>\n<li><p><strong>Post Processing(Ensembled Results):</strong></p>\n<ul>\n<li>Removal of predictions in periodic padding intervals.</li></ul></li>\n</ul>\n<p>Below is an overview of our solution, as illustrated in the following diagram.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7135891%2F6087b564bf8572d8309d45d1586fdd6e%2Foverview_zzz.svg?generation=1702429424591245&amp;alt=media\" alt=\"\"></p>\n<h2>Details of the Submission</h2>\n<h3>Models Used:</h3>\n<p>We trained Unet-based models featuring five different CNN backbones: ResNet101, MiT-B3, MiT-B5, ResNeXt101, and ResNet152.<br>\nOur models were trained using two different combinations of training-validation data, with approximately 20% of the training set allocated as validation data in each combination.</p>\n<p>For each training session of the models, we varied the seed values to increase their diversity. This approach was employed to enhance the reliability of cross-validation (CV) results and to enrich the ensemble for submission.</p>\n<h3>Data Segmentation for Inference:</h3>\n<p>For the input data segmentation, we used a specific window size and set the hop size to half of this window size, creating overlapping segments for inference. This method allowed us to effectively utilize the central part of each window in the final analysis, thereby reducing false detections at the edges of the segments.</p>\n<h3>Ensemble Method</h3>\n<p>We used an original ensemble method inspired by Weighted Boxes Fusion (WBF). The algorithm steps include:</p>\n<ul>\n<li>Grouping the input dataframe by series_id and event.<br>\nSorting each event by its score in descending order.</li>\n<li>Extracting other events within a certain range from different dataframes.</li>\n<li>Calculating the weighted step as sum(step*score) and calculating the new score as score/(number of dataframes)</li>\n<li>Removing events within the specified range but with lower scores.</li>\n</ul>\n<h4>Our Code:</h4>\n<pre><code> () -&gt; pd.DataFrame:\n\n    grouped_list = [df.groupby([, ])  df  df_list]\n    events = [, ]\n    ensemble_predictions = []\n     series_id  series_ids:\n         event  events:\n            sorted_group_list = []\n             grouped  grouped_list:\n                 (event, series_id)  grouped.groups:\n                    sorted_group_list.append(grouped.get_group((event, series_id)).sort_values(by=, ascending=))\n\n\n             ([ sorted_group.empty  sorted_group  sorted_group_list]):\n                specific_index = ((index  index, sorted_group  (sorted_group_list)   sorted_group.empty), )\n                highest = sorted_group_list[specific_index].iloc[]\n                ensemble_steps = [highest[]]\n                ensemble_probas = [highest[]]\n                probas_sum = highest[]\n                highest_step = highest[]\n\n                 i, sorted_group  (sorted_group_list):\n                    compared_prediction = sorted_group[sorted_group[].sub(highest_step).() &lt;= step_threshold]\n                    condition = sorted_group[].sub(highest_step).() &gt; step_threshold\n                    filtered_group = sorted_group[condition]\n                    sorted_group_list[i] = filtered_group\n                     i != specific_index:\n                          compared_prediction[].empty:\n                            first_score = compared_prediction[].iloc[]\n                            first_step = compared_prediction[].iloc[]\n                        :\n                            first_score =  \n                            first_step = \n                        ensemble_steps.append(first_step)\n                        ensemble_probas.append(first_score)\n                        probas_sum += first_score\n\n                ensemble_step = \n                 probas_sum!=:\n                     s, p  (ensemble_steps, ensemble_probas):\n                        ensemble_step += s*(p/probas_sum)\n\n                ensemble_predictions.append({: series_id, : (ensemble_step), : event, : probas_sum/(grouped_list)})\n\n    ensemble_df = pd.DataFrame(ensemble_predictions)\n    ensemble_df[] = ensemble_df.index\n     ensemble_df\n</code></pre>\n<h3>Post Processing</h3>\n<p>Post-processing steps were implemented for both single models and ensembled results.</p>\n<ul>\n<li><p><strong>For single models:</strong></p>\n<ul>\n<li>Applied Non-Maximum Suppression (NMS) using the excellent code from <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/458822\" target=\"_blank\">this discussion</a> to retain only the highest-scoring prediction when detected peaks were within a specified proximity.</li></ul></li>\n<li><p><strong>For ensembled models:</strong></p>\n<ul>\n<li>Removal of needless predictions in periodic padding intervals was inspired by <a href=\"https://www.kaggle.com/code/takanashihumbert/a-simple-way-trying-to-find-the-dark-zones\" target=\"_blank\">this notebook</a>.</li></ul></li>\n</ul>\n<h2>Ablation Study</h2>\n<p>The details of the scores are as follows.</p>\n<table>\n<thead>\n<tr>\n<th>Number</th>\n<th>Description</th>\n<th>CV (Fold1)</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>(1)</td>\n<td>ResNet101 (No overlap)</td>\n<td>0.756</td>\n<td>0.722</td>\n<td>0.748</td>\n</tr>\n<tr>\n<td>(2)</td>\n<td>ResNet101 (overlap)</td>\n<td>0.77</td>\n<td>0.732</td>\n<td>0.778</td>\n</tr>\n<tr>\n<td>(3)</td>\n<td>ResNet101 (overlap) + NMS</td>\n<td>0.773</td>\n<td>Not Submitted</td>\n<td>Not Submitted</td>\n</tr>\n<tr>\n<td>(4)</td>\n<td>Ensemble of 5 models (overlap, same fold) + NMS</td>\n<td>0.7844</td>\n<td>0.753</td>\n<td>0.790</td>\n</tr>\n<tr>\n<td>(5)</td>\n<td>Ensemble of 5 models (overlap, same fold) + NMS + Remove Predictions in periodic padding</td>\n<td><strong>0.7847</strong></td>\n<td><strong>0.756</strong></td>\n<td>0.801</td>\n</tr>\n<tr>\n<td>(6)</td>\n<td>Ensemble of 5 models(overlap) across 2 folds + NMS</td>\n<td>-</td>\n<td>0.749</td>\n<td>0.792</td>\n</tr>\n<tr>\n<td>(7)</td>\n<td>Ensemble of 5 models(overlap) across 2 folds + NMS + Remove Predictions in periodic padding</td>\n<td>-</td>\n<td>0.753</td>\n<td><strong>0.802</strong></td>\n</tr>\n</tbody>\n</table>",
  "messages": [
    {
      "id": 2559513,
      "postDate": "2023-12-12T22:09:26.773Z",
      "content": "<p>First, I would like to express my sincere thanks to the organizers of the Child Mind Institute - Detect Sleep States Competition. This competition provided an invaluable opportunity to engage with long one-dimensional sequences, offering a distinct and educational experience in an important area of data science.</p>\n<h2><strong>Overview of the Inference Pipeline</strong></h2>\n<p>Our training pipeline and models are based on the excellent code by <a href=\"https://www.kaggle.com/tubotubo\" target=\"_blank\">@213tubo</a>, available at <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940?rvi=1\" target=\"_blank\">this page</a>. </p>\n<h4><strong>Our solution is as follows</strong>:</h4>\n<ul>\n<li><p><strong>Data Segmentation:</strong></p>\n<ul>\n<li>Segmented input features using overlapping time windows for inference</li></ul></li>\n<li><p><strong>Models:</strong></p>\n<ul>\n<li>Utilized 10 Unet-based models, each trained on two distinct training-validation splits across five unique models.</li>\n<li>Each of the 5 models has a different backbone.</li></ul></li>\n<li><p><strong>PostProcessing(Single Model):</strong></p>\n<ul>\n<li>Applied Non-Maximum Suppression (NMS) to retain only the highest-scoring prediction when detected peaks were within a specified proximity.</li></ul></li>\n<li><p><strong>Ensemble Predictions:</strong></p>\n<ul>\n<li>Adopted an original ensemble method inspired by Weighted Boxes Fusion (WBF)</li>\n<li>In the first stage, models within the same fold were assembled.</li>\n<li>In the second stage, these results from different folds were combined.</li></ul></li>\n<li><p><strong>Post Processing(Ensembled Results):</strong></p>\n<ul>\n<li>Removal of predictions in periodic padding intervals.</li></ul></li>\n</ul>\n<p>Below is an overview of our solution, as illustrated in the following diagram.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7135891%2F6087b564bf8572d8309d45d1586fdd6e%2Foverview_zzz.svg?generation=1702429424591245&amp;alt=media\" alt=\"\"></p>\n<h2>Details of the Submission</h2>\n<h3>Models Used:</h3>\n<p>We trained Unet-based models featuring five different CNN backbones: ResNet101, MiT-B3, MiT-B5, ResNeXt101, and ResNet152.<br>\nOur models were trained using two different combinations of training-validation data, with approximately 20% of the training set allocated as validation data in each combination.</p>\n<p>For each training session of the models, we varied the seed values to increase their diversity. This approach was employed to enhance the reliability of cross-validation (CV) results and to enrich the ensemble for submission.</p>\n<h3>Data Segmentation for Inference:</h3>\n<p>For the input data segmentation, we used a specific window size and set the hop size to half of this window size, creating overlapping segments for inference. This method allowed us to effectively utilize the central part of each window in the final analysis, thereby reducing false detections at the edges of the segments.</p>\n<h3>Ensemble Method</h3>\n<p>We used an original ensemble method inspired by Weighted Boxes Fusion (WBF). The algorithm steps include:</p>\n<ul>\n<li>Grouping the input dataframe by series_id and event.<br>\nSorting each event by its score in descending order.</li>\n<li>Extracting other events within a certain range from different dataframes.</li>\n<li>Calculating the weighted step as sum(step*score) and calculating the new score as score/(number of dataframes)</li>\n<li>Removing events within the specified range but with lower scores.</li>\n</ul>\n<h4>Our Code:</h4>\n<pre><code> () -&gt; pd.DataFrame:\n\n    grouped_list = [df.groupby([, ])  df  df_list]\n    events = [, ]\n    ensemble_predictions = []\n     series_id  series_ids:\n         event  events:\n            sorted_group_list = []\n             grouped  grouped_list:\n                 (event, series_id)  grouped.groups:\n                    sorted_group_list.append(grouped.get_group((event, series_id)).sort_values(by=, ascending=))\n\n\n             ([ sorted_group.empty  sorted_group  sorted_group_list]):\n                specific_index = ((index  index, sorted_group  (sorted_group_list)   sorted_group.empty), )\n                highest = sorted_group_list[specific_index].iloc[]\n                ensemble_steps = [highest[]]\n                ensemble_probas = [highest[]]\n                probas_sum = highest[]\n                highest_step = highest[]\n\n                 i, sorted_group  (sorted_group_list):\n                    compared_prediction = sorted_group[sorted_group[].sub(highest_step).() &lt;= step_threshold]\n                    condition = sorted_group[].sub(highest_step).() &gt; step_threshold\n                    filtered_group = sorted_group[condition]\n                    sorted_group_list[i] = filtered_group\n                     i != specific_index:\n                          compared_prediction[].empty:\n                            first_score = compared_prediction[].iloc[]\n                            first_step = compared_prediction[].iloc[]\n                        :\n                            first_score =  \n                            first_step = \n                        ensemble_steps.append(first_step)\n                        ensemble_probas.append(first_score)\n                        probas_sum += first_score\n\n                ensemble_step = \n                 probas_sum!=:\n                     s, p  (ensemble_steps, ensemble_probas):\n                        ensemble_step += s*(p/probas_sum)\n\n                ensemble_predictions.append({: series_id, : (ensemble_step), : event, : probas_sum/(grouped_list)})\n\n    ensemble_df = pd.DataFrame(ensemble_predictions)\n    ensemble_df[] = ensemble_df.index\n     ensemble_df\n</code></pre>\n<h3>Post Processing</h3>\n<p>Post-processing steps were implemented for both single models and ensembled results.</p>\n<ul>\n<li><p><strong>For single models:</strong></p>\n<ul>\n<li>Applied Non-Maximum Suppression (NMS) using the excellent code from <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/458822\" target=\"_blank\">this discussion</a> to retain only the highest-scoring prediction when detected peaks were within a specified proximity.</li></ul></li>\n<li><p><strong>For ensembled models:</strong></p>\n<ul>\n<li>Removal of needless predictions in periodic padding intervals was inspired by <a href=\"https://www.kaggle.com/code/takanashihumbert/a-simple-way-trying-to-find-the-dark-zones\" target=\"_blank\">this notebook</a>.</li></ul></li>\n</ul>\n<h2>Ablation Study</h2>\n<p>The details of the scores are as follows.</p>\n<table>\n<thead>\n<tr>\n<th>Number</th>\n<th>Description</th>\n<th>CV (Fold1)</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>(1)</td>\n<td>ResNet101 (No overlap)</td>\n<td>0.756</td>\n<td>0.722</td>\n<td>0.748</td>\n</tr>\n<tr>\n<td>(2)</td>\n<td>ResNet101 (overlap)</td>\n<td>0.77</td>\n<td>0.732</td>\n<td>0.778</td>\n</tr>\n<tr>\n<td>(3)</td>\n<td>ResNet101 (overlap) + NMS</td>\n<td>0.773</td>\n<td>Not Submitted</td>\n<td>Not Submitted</td>\n</tr>\n<tr>\n<td>(4)</td>\n<td>Ensemble of 5 models (overlap, same fold) + NMS</td>\n<td>0.7844</td>\n<td>0.753</td>\n<td>0.790</td>\n</tr>\n<tr>\n<td>(5)</td>\n<td>Ensemble of 5 models (overlap, same fold) + NMS + Remove Predictions in periodic padding</td>\n<td><strong>0.7847</strong></td>\n<td><strong>0.756</strong></td>\n<td>0.801</td>\n</tr>\n<tr>\n<td>(6)</td>\n<td>Ensemble of 5 models(overlap) across 2 folds + NMS</td>\n<td>-</td>\n<td>0.749</td>\n<td>0.792</td>\n</tr>\n<tr>\n<td>(7)</td>\n<td>Ensemble of 5 models(overlap) across 2 folds + NMS + Remove Predictions in periodic padding</td>\n<td>-</td>\n<td>0.753</td>\n<td><strong>0.802</strong></td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "First, I would like to express my sincere thanks to the organizers of the Child Mind Institute - Detect Sleep States Competition. This competition provided an invaluable opportunity to engage with long one-dimensional sequences, offering a distinct and educational experience in an important area of data science.\n\n\n## **Overview of the Inference Pipeline**\nOur training pipeline and models are based on the excellent code by [@213tubo](https://www.kaggle.com/tubotubo), available at [this page](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940?rvi=1). \n#### **Our solution is as follows**:\n\n- **Data Segmentation:**\n  - Segmented input features using overlapping time windows for inference\n- **Models:**\n  - Utilized 10 Unet-based models, each trained on two distinct training-validation splits across five unique models.\n  - Each of the 5 models has a different backbone.\n\n- **PostProcessing(Single Model):**\n  - Applied Non-Maximum Suppression (NMS) to retain only the highest-scoring prediction when detected peaks were within a specified proximity.\n\n- **Ensemble Predictions:**\n  - Adopted an original ensemble method inspired by Weighted Boxes Fusion (WBF)\n  - In the first stage, models within the same fold were assembled.\n  - In the second stage, these results from different folds were combined.\n\n- **Post Processing(Ensembled Results):**\n  - Removal of predictions in periodic padding intervals.\n  \nBelow is an overview of our solution, as illustrated in the following diagram.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7135891%2F6087b564bf8572d8309d45d1586fdd6e%2Foverview_zzz.svg?generation=1702429424591245&alt=media)\n\n\n## Details of the Submission\n\n### Models Used:\nWe trained Unet-based models featuring five different CNN backbones: ResNet101, MiT-B3, MiT-B5, ResNeXt101, and ResNet152.\nOur models were trained using two different combinations of training-validation data, with approximately 20% of the training set allocated as validation data in each combination.\n\nFor each training session of the models, we varied the seed values to increase their diversity. This approach was employed to enhance the reliability of cross-validation (CV) results and to enrich the ensemble for submission.\n\n### Data Segmentation for Inference:\nFor the input data segmentation, we used a specific window size and set the hop size to half of this window size, creating overlapping segments for inference. This method allowed us to effectively utilize the central part of each window in the final analysis, thereby reducing false detections at the edges of the segments.\n\n### Ensemble Method\nWe used an original ensemble method inspired by Weighted Boxes Fusion (WBF). The algorithm steps include:\n\n- Grouping the input dataframe by series_id and event.\nSorting each event by its score in descending order.\n- Extracting other events within a certain range from different dataframes.\n- Calculating the weighted step as sum(step*score) and calculating the new score as score/(number of dataframes)\n- Removing events within the specified range but with lower scores.\n#### Our Code:\n```python\ndef ensemble_predictions(df_list: list, series_ids: list,step_threshold: int=50) -> pd.DataFrame:\n\n    grouped_list = [df.groupby(['event', 'series_id']) for df in df_list]\n    events = [\"onset\", \"wakeup\"]\n    ensemble_predictions = []\n    for series_id in series_ids:\n        for event in events:\n            sorted_group_list = []\n            for grouped in grouped_list:\n                if (event, series_id) in grouped.groups:\n                    sorted_group_list.append(grouped.get_group((event, series_id)).sort_values(by='score', ascending=False))\n            \n\n            while any([not sorted_group.empty for sorted_group in sorted_group_list]):\n                specific_index = next((index for index, sorted_group in enumerate(sorted_group_list) if not sorted_group.empty), None)\n                highest = sorted_group_list[specific_index].iloc[0]\n                ensemble_steps = [highest[\"step\"]]\n                ensemble_probas = [highest[\"score\"]]\n                probas_sum = highest[\"score\"]\n                highest_step = highest['step']\n\n                for i, sorted_group in enumerate(sorted_group_list):\n                    compared_prediction = sorted_group[sorted_group['step'].sub(highest_step).abs() <= step_threshold]\n                    condition = sorted_group['step'].sub(highest_step).abs() > step_threshold\n                    filtered_group = sorted_group[condition]\n                    sorted_group_list[i] = filtered_group\n                    if i != specific_index:\n                        if not compared_prediction[\"score\"].empty:\n                            first_score = compared_prediction[\"score\"].iloc[0]\n                            first_step = compared_prediction[\"step\"].iloc[0]\n                        else:\n                            first_score = 0 \n                            first_step = 0\n                        ensemble_steps.append(first_step)\n                        ensemble_probas.append(first_score)\n                        probas_sum += first_score\n                \n                ensemble_step = 0\n                if probas_sum!=0:\n                    for s, p in zip(ensemble_steps, ensemble_probas):\n                        ensemble_step += s*(p/probas_sum)\n\n                ensemble_predictions.append({\"series_id\": series_id, \"step\": int(ensemble_step), \"event\": event, \"score\": probas_sum/len(grouped_list)})\n\n    ensemble_df = pd.DataFrame(ensemble_predictions)\n    ensemble_df['row_id'] = ensemble_df.index\n    return ensemble_df\n\n```\n### Post Processing\nPost-processing steps were implemented for both single models and ensembled results.\n\n- **For single models:**\n  - Applied Non-Maximum Suppression (NMS) using the excellent code from [this discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/458822) to retain only the highest-scoring prediction when detected peaks were within a specified proximity.\n\n- **For ensembled models:**\n  - Removal of needless predictions in periodic padding intervals was inspired by [this notebook](https://www.kaggle.com/code/takanashihumbert/a-simple-way-trying-to-find-the-dark-zones).\n\n\n## Ablation Study\nThe details of the scores are as follows.\n| Number | Description                                                               | CV (Fold1) | Public LB | Private LB | \n|--------|---------------------------------------------------------------------------|------------|-----------|------------|\n| (1)    | ResNet101 (No overlap)                                                    | 0.756      | 0.722     | 0.748      |\n| (2)    | ResNet101 (overlap)                                                       | 0.77       | 0.732     | 0.778      |\n| (3)    | ResNet101 (overlap) + NMS                                                 | 0.773      | Not Submitted | Not Submitted |\n| (4)    | Ensemble of 5 models (overlap, same fold) + NMS                           | 0.7844     | 0.753     | 0.790      |\n| (5)    | Ensemble of 5 models (overlap, same fold) + NMS + Remove Predictions in periodic padding       | **0.7847**     | **0.756** | 0.801      |\n| (6)    | Ensemble of 5 models(overlap) across 2 folds + NMS                                      | -          | 0.749     | 0.792      |\n| (7)    | Ensemble of 5 models(overlap) across 2 folds + NMS + Remove Predictions in periodic padding            | -          | 0.753     | **0.802**  |",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2559513": "First, I would like to express my sincere thanks to the organizers of the Child Mind Institute - Detect Sleep States Competition. This competition provided an invaluable opportunity to engage with long one-dimensional sequences, offering a distinct and educational experience in an important area of data science.\n\n\n## **Overview of the Inference Pipeline**\nOur training pipeline and models are based on the excellent code by [@213tubo](https://www.kaggle.com/tubotubo), available at [this page](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940?rvi=1). \n#### **Our solution is as follows**:\n\n- **Data Segmentation:**\n  - Segmented input features using overlapping time windows for inference\n- **Models:**\n  - Utilized 10 Unet-based models, each trained on two distinct training-validation splits across five unique models.\n  - Each of the 5 models has a different backbone.\n\n- **PostProcessing(Single Model):**\n  - Applied Non-Maximum Suppression (NMS) to retain only the highest-scoring prediction when detected peaks were within a specified proximity.\n\n- **Ensemble Predictions:**\n  - Adopted an original ensemble method inspired by Weighted Boxes Fusion (WBF)\n  - In the first stage, models within the same fold were assembled.\n  - In the second stage, these results from different folds were combined.\n\n- **Post Processing(Ensembled Results):**\n  - Removal of predictions in periodic padding intervals.\n  \nBelow is an overview of our solution, as illustrated in the following diagram.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7135891%2F6087b564bf8572d8309d45d1586fdd6e%2Foverview_zzz.svg?generation=1702429424591245&alt=media)\n\n\n## Details of the Submission\n\n### Models Used:\nWe trained Unet-based models featuring five different CNN backbones: ResNet101, MiT-B3, MiT-B5, ResNeXt101, and ResNet152.\nOur models were trained using two different combinations of training-validation data, with approximately 20% of the training set allocated as validation data in each combination.\n\nFor each training session of the models, we varied the seed values to increase their diversity. This approach was employed to enhance the reliability of cross-validation (CV) results and to enrich the ensemble for submission.\n\n### Data Segmentation for Inference:\nFor the input data segmentation, we used a specific window size and set the hop size to half of this window size, creating overlapping segments for inference. This method allowed us to effectively utilize the central part of each window in the final analysis, thereby reducing false detections at the edges of the segments.\n\n### Ensemble Method\nWe used an original ensemble method inspired by Weighted Boxes Fusion (WBF). The algorithm steps include:\n\n- Grouping the input dataframe by series_id and event.\nSorting each event by its score in descending order.\n- Extracting other events within a certain range from different dataframes.\n- Calculating the weighted step as sum(step*score) and calculating the new score as score/(number of dataframes)\n- Removing events within the specified range but with lower scores.\n#### Our Code:\n```python\ndef ensemble_predictions(df_list: list, series_ids: list,step_threshold: int=50) -> pd.DataFrame:\n\n    grouped_list = [df.groupby(['event', 'series_id']) for df in df_list]\n    events = [\"onset\", \"wakeup\"]\n    ensemble_predictions = []\n    for series_id in series_ids:\n        for event in events:\n            sorted_group_list = []\n            for grouped in grouped_list:\n                if (event, series_id) in grouped.groups:\n                    sorted_group_list.append(grouped.get_group((event, series_id)).sort_values(by='score', ascending=False))\n            \n\n            while any([not sorted_group.empty for sorted_group in sorted_group_list]):\n                specific_index = next((index for index, sorted_group in enumerate(sorted_group_list) if not sorted_group.empty), None)\n                highest = sorted_group_list[specific_index].iloc[0]\n                ensemble_steps = [highest[\"step\"]]\n                ensemble_probas = [highest[\"score\"]]\n                probas_sum = highest[\"score\"]\n                highest_step = highest['step']\n\n                for i, sorted_group in enumerate(sorted_group_list):\n                    compared_prediction = sorted_group[sorted_group['step'].sub(highest_step).abs() <= step_threshold]\n                    condition = sorted_group['step'].sub(highest_step).abs() > step_threshold\n                    filtered_group = sorted_group[condition]\n                    sorted_group_list[i] = filtered_group\n                    if i != specific_index:\n                        if not compared_prediction[\"score\"].empty:\n                            first_score = compared_prediction[\"score\"].iloc[0]\n                            first_step = compared_prediction[\"step\"].iloc[0]\n                        else:\n                            first_score = 0 \n                            first_step = 0\n                        ensemble_steps.append(first_step)\n                        ensemble_probas.append(first_score)\n                        probas_sum += first_score\n                \n                ensemble_step = 0\n                if probas_sum!=0:\n                    for s, p in zip(ensemble_steps, ensemble_probas):\n                        ensemble_step += s*(p/probas_sum)\n\n                ensemble_predictions.append({\"series_id\": series_id, \"step\": int(ensemble_step), \"event\": event, \"score\": probas_sum/len(grouped_list)})\n\n    ensemble_df = pd.DataFrame(ensemble_predictions)\n    ensemble_df['row_id'] = ensemble_df.index\n    return ensemble_df\n\n```\n### Post Processing\nPost-processing steps were implemented for both single models and ensembled results.\n\n- **For single models:**\n  - Applied Non-Maximum Suppression (NMS) using the excellent code from [this discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/458822) to retain only the highest-scoring prediction when detected peaks were within a specified proximity.\n\n- **For ensembled models:**\n  - Removal of needless predictions in periodic padding intervals was inspired by [this notebook](https://www.kaggle.com/code/takanashihumbert/a-simple-way-trying-to-find-the-dark-zones).\n\n\n## Ablation Study\nThe details of the scores are as follows.\n| Number | Description                                                               | CV (Fold1) | Public LB | Private LB | \n|--------|---------------------------------------------------------------------------|------------|-----------|------------|\n| (1)    | ResNet101 (No overlap)                                                    | 0.756      | 0.722     | 0.748      |\n| (2)    | ResNet101 (overlap)                                                       | 0.77       | 0.732     | 0.778      |\n| (3)    | ResNet101 (overlap) + NMS                                                 | 0.773      | Not Submitted | Not Submitted |\n| (4)    | Ensemble of 5 models (overlap, same fold) + NMS                           | 0.7844     | 0.753     | 0.790      |\n| (5)    | Ensemble of 5 models (overlap, same fold) + NMS + Remove Predictions in periodic padding       | **0.7847**     | **0.756** | 0.801      |\n| (6)    | Ensemble of 5 models(overlap) across 2 folds + NMS                                      | -          | 0.749     | 0.792      |\n| (7)    | Ensemble of 5 models(overlap) across 2 folds + NMS + Remove Predictions in periodic padding            | -          | 0.753     | **0.802**  |"
  },
  "complete": true,
  "source": "existing_discussion",
  "competition_id": 53666,
  "competition_slug": "child-mind-institute-detect-sleep-states",
  "team_id": 10885834,
  "solution_url": "/competitions/child-mind-institute-detect-sleep-states/writeups/223-41st-place-solution-for-the-child-mind-institu"
}