{
  "id": 459894,
  "title": "10th Place Solution",
  "url": "/competitions/child-mind-institute-detect-sleep-states/writeups/ohkawa3-10th-place-solution",
  "author_name": "",
  "post_date": "2023-12-10T23:34:07.857Z",
  "votes": 32,
  "comment_count": 4,
  "views": 0,
  "content": "<p>First and foremost, I would like to express my gratitude to all participants and organizers.<br>\nIt was a very tough competition as I did not have many good scores on public LB, but I was satisfied with the final results.</p>\n<h2>Features engineering</h2>\n<ul>\n<li>enmo<ul>\n<li>Utilized as is</li></ul></li>\n<li>anglez<ul>\n<li>Used <code>anglez.diff().abs()</code></li></ul></li>\n<li>Time Encoding<ul>\n<li>Normalized the hours into t=[0~1) and employed <code>sin(2*pi*t)</code> and <code>cos(2*pi*t)</code> as features</li>\n<li>Experimented with finer periodicities (e.g., <code>sin(4*pi*t)</code>, <code>sin(8*pi*t)</code>, <code>cos(4*pi*t)</code>, <code>cos(8*pi*t)</code>), but performance worsened</li>\n<li>Considered features like week and month, but they were rejected due to performance degradation.</li></ul></li>\n<li>Duplicate Data<ul>\n<li>Identified instances of identical sensor data occurring in the same series at the same time.</li>\n<li>Assigned a flag of 1 for times with duplicates and 0 for times without duplicates</li></ul></li>\n<li>Step<ul>\n<li>Flags set for within 1 hour and within 8 hours from the start of measurement</li>\n<li>Because, included data points where onset occurred within 1 hour or wakeup within 8 hours since there were none</li></ul></li>\n</ul>\n<h2>Model</h2>\n<p>Adopted a 1D-UNet GRU model as illustrated. Features, excluding enmo and anglez, are added just before the GRU.  Trained using Binary Cross Entropy.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2F5e93b7c1e25dd2fbcc18d346c320ec93%2Fmodel.png?generation=1701906351601543&amp;alt=media\" alt=\"\"></p>\n<h2>Target Design</h2>\n<p>Utilized a heatmap based on a Gaussian distribution. For stability in training and improved AP at a large tolerance, a Gaussian distribution with a large sigma was advantageous. Contrastly, a Gaussian distribution with a small sigma led to instability in training but improved AP at a small tolerance. As a compromise, a weighted sum of Gaussian distributions with large and small sigmas was used.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2Ff2d28c6cc637e3f55d3ca2383a845d25%2Ftarget.png?generation=1701906225296881&amp;alt=media\" alt=\"\"></p>\n<pre><code>gauss_small_sigma_minute = \ngauss_large_sigma_size = \ngauss_mix_ratio = \ngauss_small_sigma_size = minute2step(gauss_small_sigma_minute)\ngauss_large_sigma_size = minute2step(gauss_large_sigma_minute)\n\ngauss_large = np.exp(-((x - gauss_center)**) / ( * (gauss_large_sigma_size)**))\ngauss_small = np.exp(-((x - gauss_center)**) / ( * (gauss_small_sigma_size)**))\n\ngauss = (gauss_large + gauss_mix_ratio * gauss_small)\ngauss /= gauss.()\n</code></pre>\n<p>`</p>\n<h2>Training</h2>\n<ul>\n<li>Randomly sample 12 hours</li>\n<li>150 epoch, AdamW</li>\n<li>Apply SWA after 50 epoch</li>\n</ul>\n<h2>Inference</h2>\n<ul>\n<li>In contrast to training, infer a entire series at a time</li>\n</ul>\n<h2>Ensemble</h2>\n<p>The processing time for the entire test dataset, including data loading, pre-processing, and post-processing, was approximately 18 minutes.<br>\nOf that time, model inference took 45 seconds.</p>\n<p>Public LB is calculated by 25% of the test data, so some shake was expected.<br>\nTherefore, the decision was made to perform ensemble learning with large number of models. The final ensemble consists of 120 models, and the processing is completed within 120 minutes.</p>\n<h2>Post-processing</h2>\n<p>It is crucial to detect multiple candidates in one night.</p>\n<ul>\n<li>Smooth the output values with a width of 11</li>\n<li>Detect all local maxima with a threshold of 0.01 and add them as candidate points</li>\n<li>For all remaining timestamps, perform the following in descending order of output values<ul>\n<li>If the timestamp is more than 21 minutes away from any previously detected timestamp, add the product of 0.1 and the output value as a candidate point</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": "2551756",
      "postDate": "12/06/2023 23:58:07",
      "content": "<p>First and foremost, I would like to express my gratitude to all participants and organizers.<br>\nIt was a very tough competition as I did not have many good scores on public LB, but I was satisfied with the final results.</p>\n<h2>Features engineering</h2>\n<ul>\n<li>enmo<ul>\n<li>Utilized as is</li></ul></li>\n<li>anglez<ul>\n<li>Used <code>anglez.diff().abs()</code></li></ul></li>\n<li>Time Encoding<ul>\n<li>Normalized the hours into t=[0~1) and employed <code>sin(2*pi*t)</code> and <code>cos(2*pi*t)</code> as features</li>\n<li>Experimented with finer periodicities (e.g., <code>sin(4*pi*t)</code>, <code>sin(8*pi*t)</code>, <code>cos(4*pi*t)</code>, <code>cos(8*pi*t)</code>), but performance worsened</li>\n<li>Considered features like week and month, but they were rejected due to performance degradation.</li></ul></li>\n<li>Duplicate Data<ul>\n<li>Identified instances of identical sensor data occurring in the same series at the same time.</li>\n<li>Assigned a flag of 1 for times with duplicates and 0 for times without duplicates</li></ul></li>\n<li>Step<ul>\n<li>Flags set for within 1 hour and within 8 hours from the start of measurement</li>\n<li>Because, included data points where onset occurred within 1 hour or wakeup within 8 hours since there were none</li></ul></li>\n</ul>\n<h2>Model</h2>\n<p>Adopted a 1D-UNet GRU model as illustrated. Features, excluding enmo and anglez, are added just before the GRU.  Trained using Binary Cross Entropy.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2F5e93b7c1e25dd2fbcc18d346c320ec93%2Fmodel.png?generation=1701906351601543&amp;alt=media\" alt=\"\"></p>\n<h2>Target Design</h2>\n<p>Utilized a heatmap based on a Gaussian distribution. For stability in training and improved AP at a large tolerance, a Gaussian distribution with a large sigma was advantageous. Contrastly, a Gaussian distribution with a small sigma led to instability in training but improved AP at a small tolerance. As a compromise, a weighted sum of Gaussian distributions with large and small sigmas was used.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2Ff2d28c6cc637e3f55d3ca2383a845d25%2Ftarget.png?generation=1701906225296881&amp;alt=media\" alt=\"\"></p>\n<pre><code>gauss_small_sigma_minute = \ngauss_large_sigma_size = \ngauss_mix_ratio = \ngauss_small_sigma_size = minute2step(gauss_small_sigma_minute)\ngauss_large_sigma_size = minute2step(gauss_large_sigma_minute)\n\ngauss_large = np.exp(-((x - gauss_center)**) / ( * (gauss_large_sigma_size)**))\ngauss_small = np.exp(-((x - gauss_center)**) / ( * (gauss_small_sigma_size)**))\n\ngauss = (gauss_large + gauss_mix_ratio * gauss_small)\ngauss /= gauss.()\n</code></pre>\n<p>`</p>\n<h2>Training</h2>\n<ul>\n<li>Randomly sample 12 hours</li>\n<li>150 epoch, AdamW</li>\n<li>Apply SWA after 50 epoch</li>\n</ul>\n<h2>Inference</h2>\n<ul>\n<li>In contrast to training, infer a entire series at a time</li>\n</ul>\n<h2>Ensemble</h2>\n<p>The processing time for the entire test dataset, including data loading, pre-processing, and post-processing, was approximately 18 minutes.<br>\nOf that time, model inference took 45 seconds.</p>\n<p>Public LB is calculated by 25% of the test data, so some shake was expected.<br>\nTherefore, the decision was made to perform ensemble learning with large number of models. The final ensemble consists of 120 models, and the processing is completed within 120 minutes.</p>\n<h2>Post-processing</h2>\n<p>It is crucial to detect multiple candidates in one night.</p>\n<ul>\n<li>Smooth the output values with a width of 11</li>\n<li>Detect all local maxima with a threshold of 0.01 and add them as candidate points</li>\n<li>For all remaining timestamps, perform the following in descending order of output values<ul>\n<li>If the timestamp is more than 21 minutes away from any previously detected timestamp, add the product of 0.1 and the output value as a candidate point</li></ul></li>\n</ul>",
      "rawMarkdown": "First and foremost, I would like to express my gratitude to all participants and organizers.\nIt was a very tough competition as I did not have many good scores on public LB, but I was satisfied with the final results.\n\n## Features engineering\n+ enmo\n  + Utilized as is\n+ anglez\n  + Used `anglez.diff().abs()`\n+ Time Encoding\n  + Normalized the hours into t=[0~1) and employed `sin(2*pi*t)` and `cos(2*pi*t)` as features\n  + Experimented with finer periodicities (e.g., `sin(4*pi*t)`, `sin(8*pi*t)`, `cos(4*pi*t)`, `cos(8*pi*t)`), but performance worsened\n  + Considered features like week and month, but they were rejected due to performance degradation.\n+ Duplicate Data\n  + Identified instances of identical sensor data occurring in the same series at the same time.\n  + Assigned a flag of 1 for times with duplicates and 0 for times without duplicates\n+ Step\n  + Flags set for within 1 hour and within 8 hours from the start of measurement\n  + Because, included data points where onset occurred within 1 hour or wakeup within 8 hours since there were none\n\n## Model\nAdopted a 1D-UNet GRU model as illustrated. Features, excluding enmo and anglez, are added just before the GRU.  Trained using Binary Cross Entropy.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2F5e93b7c1e25dd2fbcc18d346c320ec93%2Fmodel.png?generation=1701906351601543&alt=media)\n \n\n\n## Target Design\nUtilized a heatmap based on a Gaussian distribution. For stability in training and improved AP at a large tolerance, a Gaussian distribution with a large sigma was advantageous. Contrastly, a Gaussian distribution with a small sigma led to instability in training but improved AP at a small tolerance. As a compromise, a weighted sum of Gaussian distributions with large and small sigmas was used.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2Ff2d28c6cc637e3f55d3ca2383a845d25%2Ftarget.png?generation=1701906225296881&alt=media)\n\n```python\ngauss_small_sigma_minute = 2.5\ngauss_large_sigma_size = 10\ngauss_mix_ratio = 8\ngauss_small_sigma_size = minute2step(gauss_small_sigma_minute)\ngauss_large_sigma_size = minute2step(gauss_large_sigma_minute)\n\ngauss_large = np.exp(-((x - gauss_center)**2) / (2 * (gauss_large_sigma_size)**2))\ngauss_small = np.exp(-((x - gauss_center)**2) / (2 * (gauss_small_sigma_size)**2))\n\ngauss = (gauss_large + gauss_mix_ratio * gauss_small)\ngauss /= gauss.max()\n````\n\n\n## Training\n+ Randomly sample 12 hours\n+ 150 epoch, AdamW\n+ Apply SWA after 50 epoch\n\n## Inference\n+ In contrast to training, infer a entire series at a time\n\n## Ensemble\nThe processing time for the entire test dataset, including data loading, pre-processing, and post-processing, was approximately 18 minutes.\nOf that time, model inference took 45 seconds.\n\nPublic LB is calculated by 25% of the test data, so some shake was expected.\nTherefore, the decision was made to perform ensemble learning with large number of models. The final ensemble consists of 120 models, and the processing is completed within 120 minutes.\n\n## Post-processing\nIt is crucial to detect multiple candidates in one night.\n\n+ Smooth the output values with a width of 11\n+ Detect all local maxima with a threshold of 0.01 and add them as candidate points\n+ For all remaining timestamps, perform the following in descending order of output values\n   + If the timestamp is more than 21 minutes away from any previously detected timestamp, add the product of 0.1 and the output value as a candidate point",
      "votes": null
    },
    {
      "id": "2551764",
      "postDate": "12/07/2023 00:22:12",
      "content": "<p>Why were the hour features not initially utilized as input but introduced from the compressed portion? Can you provide insight into the reason behind this?</p>",
      "rawMarkdown": "Why were the hour features not initially utilized as input but introduced from the compressed portion? Can you provide insight into the reason behind this?",
      "votes": null
    },
    {
      "id": "2552189",
      "postDate": "12/07/2023 09:22:31",
      "content": "<p>This is because hour features are highly redundant and sufficient information has been extracted, so we thought that they should not be treated the same as enmo and anglez.</p>",
      "rawMarkdown": "This is because hour features are highly redundant and sufficient information has been extracted, so we thought that they should not be treated the same as enmo and anglez.",
      "votes": null
    },
    {
      "id": "2552216",
      "postDate": "12/07/2023 09:46:47",
      "content": "<p>Can support this, at least in my approach the addition of sinusoidal temporal features did not improve performance.</p>",
      "rawMarkdown": "Can support this, at least in my approach the addition of sinusoidal temporal features did not improve performance.",
      "votes": null
    },
    {
      "id": "2552780",
      "postDate": "12/07/2023 18:10:21",
      "content": "<p>Thank you 👍</p>",
      "rawMarkdown": "Thank you 👍",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2551764,
      "author_name": "gunparksom",
      "author_url": "",
      "post_date": "12/07/2023 00:22:12",
      "content": "<p>Why were the hour features not initially utilized as input but introduced from the compressed portion? Can you provide insight into the reason behind this?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2552189,
          "author_name": "chack3",
          "author_url": "",
          "post_date": "12/07/2023 09:22:31",
          "content": "<p>This is because hour features are highly redundant and sufficient information has been extracted, so we thought that they should not be treated the same as enmo and anglez.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2552216,
              "author_name": "exjustice",
              "author_url": "",
              "post_date": "12/07/2023 09:46:47",
              "content": "<p>Can support this, at least in my approach the addition of sinusoidal temporal features did not improve performance.</p>",
              "votes": null,
              "replies": []
            },
            {
              "id": 2552780,
              "author_name": "gunparksom",
              "author_url": "",
              "post_date": "12/07/2023 18:10:21",
              "content": "<p>Thank you 👍</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2551756": "First and foremost, I would like to express my gratitude to all participants and organizers.\nIt was a very tough competition as I did not have many good scores on public LB, but I was satisfied with the final results.\n\n## Features engineering\n+ enmo\n  + Utilized as is\n+ anglez\n  + Used `anglez.diff().abs()`\n+ Time Encoding\n  + Normalized the hours into t=[0~1) and employed `sin(2*pi*t)` and `cos(2*pi*t)` as features\n  + Experimented with finer periodicities (e.g., `sin(4*pi*t)`, `sin(8*pi*t)`, `cos(4*pi*t)`, `cos(8*pi*t)`), but performance worsened\n  + Considered features like week and month, but they were rejected due to performance degradation.\n+ Duplicate Data\n  + Identified instances of identical sensor data occurring in the same series at the same time.\n  + Assigned a flag of 1 for times with duplicates and 0 for times without duplicates\n+ Step\n  + Flags set for within 1 hour and within 8 hours from the start of measurement\n  + Because, included data points where onset occurred within 1 hour or wakeup within 8 hours since there were none\n\n## Model\nAdopted a 1D-UNet GRU model as illustrated. Features, excluding enmo and anglez, are added just before the GRU.  Trained using Binary Cross Entropy.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2F5e93b7c1e25dd2fbcc18d346c320ec93%2Fmodel.png?generation=1701906351601543&alt=media)\n \n\n\n## Target Design\nUtilized a heatmap based on a Gaussian distribution. For stability in training and improved AP at a large tolerance, a Gaussian distribution with a large sigma was advantageous. Contrastly, a Gaussian distribution with a small sigma led to instability in training but improved AP at a small tolerance. As a compromise, a weighted sum of Gaussian distributions with large and small sigmas was used.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1630583%2Ff2d28c6cc637e3f55d3ca2383a845d25%2Ftarget.png?generation=1701906225296881&alt=media)\n\n```python\ngauss_small_sigma_minute = 2.5\ngauss_large_sigma_size = 10\ngauss_mix_ratio = 8\ngauss_small_sigma_size = minute2step(gauss_small_sigma_minute)\ngauss_large_sigma_size = minute2step(gauss_large_sigma_minute)\n\ngauss_large = np.exp(-((x - gauss_center)**2) / (2 * (gauss_large_sigma_size)**2))\ngauss_small = np.exp(-((x - gauss_center)**2) / (2 * (gauss_small_sigma_size)**2))\n\ngauss = (gauss_large + gauss_mix_ratio * gauss_small)\ngauss /= gauss.max()\n````\n\n\n## Training\n+ Randomly sample 12 hours\n+ 150 epoch, AdamW\n+ Apply SWA after 50 epoch\n\n## Inference\n+ In contrast to training, infer a entire series at a time\n\n## Ensemble\nThe processing time for the entire test dataset, including data loading, pre-processing, and post-processing, was approximately 18 minutes.\nOf that time, model inference took 45 seconds.\n\nPublic LB is calculated by 25% of the test data, so some shake was expected.\nTherefore, the decision was made to perform ensemble learning with large number of models. The final ensemble consists of 120 models, and the processing is completed within 120 minutes.\n\n## Post-processing\nIt is crucial to detect multiple candidates in one night.\n\n+ Smooth the output values with a width of 11\n+ Detect all local maxima with a threshold of 0.01 and add them as candidate points\n+ For all remaining timestamps, perform the following in descending order of output values\n   + If the timestamp is more than 21 minutes away from any previously detected timestamp, add the product of 0.1 and the output value as a candidate point",
    "2551764": "Why were the hour features not initially utilized as input but introduced from the compressed portion? Can you provide insight into the reason behind this?",
    "2552189": "This is because hour features are highly redundant and sufficient information has been extracted, so we thought that they should not be treated the same as enmo and anglez.",
    "2552216": "Can support this, at least in my approach the addition of sinusoidal temporal features did not improve performance.",
    "2552780": "Thank you 👍"
  },
  "source": "meta"
}