{
  "id": 460112,
  "title": "77th solution - UNet1D -> WaveNet, Bi-GRU",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/460112",
  "author_name": "Ryota",
  "post_date": "2023-12-07T23:41:11.392000",
  "votes": 13,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Thank you to the organizer and Kaggle staff for this excited competition. And congratulations to winners. I only participated in the last three weeks, but I learned a lot and enjoy this competition.</p>\n<h1>PreProcess</h1>\n<ul>\n<li>standard scaler (for each subject)</li>\n<li>sequence_size = 17,280 (12 * 60 * 24)</li>\n<li>sequence overlap = 11,520 (12 * 60 * 16)</li>\n<li>prepare features to be used for NN inputs<ul>\n<li>anglez</li>\n<li>enmo</li>\n<li>anglez_diff</li>\n<li>enmo_diff</li>\n<li>time_unit_sin</li>\n<li>time_unit_cos</li></ul></li>\n</ul>\n<h1>Validation</h1>\n<ul>\n<li>GroupKFold (k=5)</li>\n</ul>\n<h1>Model</h1>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV(=5fold)</th>\n<th>Public</th>\n<th>Private</th>\n<th>Weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>UNet1D - WaveNet</td>\n<td>0.746</td>\n<td>0.735</td>\n<td>0.772</td>\n<td>0.3375</td>\n</tr>\n<tr>\n<td>UNet1D - BiGRU(num_block=2)</td>\n<td>0.731</td>\n<td>not submit</td>\n<td>not submit</td>\n<td>0.14875</td>\n</tr>\n<tr>\n<td>UNet1D - BiGRU(num_block=3)</td>\n<td>0.721</td>\n<td>0.708</td>\n<td>0.754</td>\n<td>0.06375</td>\n</tr>\n<tr>\n<td>UNet1D - ViT - BiGRU(num_block=2)</td>\n<td>0.728</td>\n<td>not submit</td>\n<td>not submit</td>\n<td>0.0375</td>\n</tr>\n<tr>\n<td>UNet1D - WaveNet - BiGRU(num_block=1)</td>\n<td>0.751</td>\n<td>0.730</td>\n<td>0.770</td>\n<td>0.4125</td>\n</tr>\n<tr>\n<td>Ensemble</td>\n<td>0.775</td>\n<td>0.756</td>\n<td>0.786</td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h3>Common in all models</h3>\n<ul>\n<li>Linear Layer at beginning</li>\n<li>Downsampling before calculation of loss (negative = positive * 10)</li>\n<li>Use hard label (window=12)</li>\n<li>Multi Task Learning<ul>\n<li>segmentation: “asleep”, “awake”, “not_worn”</li>\n<li>event detection: “onset”, “wakeup”</li></ul></li>\n<li>I add the output of segmentation to the input of the event detection head. After that, I add multiple conv1d layer with various kernel_size(3, 7, 15) before the head.</li>\n<li>Skip Connection is very important.</li>\n</ul>\n<h3>Each Component</h3>\n<ul>\n<li><strong>UNet1D</strong><ul>\n<li>used skip connection instead concatenate when merge encoder and decoder</li></ul></li>\n<li><strong>WaveNet</strong><ul>\n<li>block = 2</li>\n<li>dilation = 4 ** N (N = 6, 4)</li></ul></li>\n<li><strong>Bi-GRU</strong><ul>\n<li>Transformer-like architecture ( (Bi-GRU → FeedForward) * num_block, with skip connection and layer normalization)</li></ul></li>\n<li><strong>ViT</strong><ul>\n<li>patch_size=4</li>\n<li>I use ViT for diversity, but attention-based model isn't well in my case.</li></ul></li>\n</ul>\n<h1>Ensemble</h1>\n<ul>\n<li>Weighted average of the model's output probabilities</li>\n</ul>\n<h1>PostProcess</h1>\n<ul>\n<li>Moving Average (window=3)</li>\n<li>Find Peak (using scipy.signal.find_peaks, score_th=0.01, distance=100))</li>\n<li>Eliminate time periods when events are unlikely to occur. (Removed predictions outside the following duration)<ul>\n<li>onset : 20:00 ~ 12:00</li>\n<li>wakeup : 3:00 ~ 20:00</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 2552973,
      "postDate": "2023-12-07T23:41:11.393Z",
      "content": "<p>Thank you to the organizer and Kaggle staff for this excited competition. And congratulations to winners. I only participated in the last three weeks, but I learned a lot and enjoy this competition.</p>\n<h1>PreProcess</h1>\n<ul>\n<li>standard scaler (for each subject)</li>\n<li>sequence_size = 17,280 (12 * 60 * 24)</li>\n<li>sequence overlap = 11,520 (12 * 60 * 16)</li>\n<li>prepare features to be used for NN inputs<ul>\n<li>anglez</li>\n<li>enmo</li>\n<li>anglez_diff</li>\n<li>enmo_diff</li>\n<li>time_unit_sin</li>\n<li>time_unit_cos</li></ul></li>\n</ul>\n<h1>Validation</h1>\n<ul>\n<li>GroupKFold (k=5)</li>\n</ul>\n<h1>Model</h1>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV(=5fold)</th>\n<th>Public</th>\n<th>Private</th>\n<th>Weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>UNet1D - WaveNet</td>\n<td>0.746</td>\n<td>0.735</td>\n<td>0.772</td>\n<td>0.3375</td>\n</tr>\n<tr>\n<td>UNet1D - BiGRU(num_block=2)</td>\n<td>0.731</td>\n<td>not submit</td>\n<td>not submit</td>\n<td>0.14875</td>\n</tr>\n<tr>\n<td>UNet1D - BiGRU(num_block=3)</td>\n<td>0.721</td>\n<td>0.708</td>\n<td>0.754</td>\n<td>0.06375</td>\n</tr>\n<tr>\n<td>UNet1D - ViT - BiGRU(num_block=2)</td>\n<td>0.728</td>\n<td>not submit</td>\n<td>not submit</td>\n<td>0.0375</td>\n</tr>\n<tr>\n<td>UNet1D - WaveNet - BiGRU(num_block=1)</td>\n<td>0.751</td>\n<td>0.730</td>\n<td>0.770</td>\n<td>0.4125</td>\n</tr>\n<tr>\n<td>Ensemble</td>\n<td>0.775</td>\n<td>0.756</td>\n<td>0.786</td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h3>Common in all models</h3>\n<ul>\n<li>Linear Layer at beginning</li>\n<li>Downsampling before calculation of loss (negative = positive * 10)</li>\n<li>Use hard label (window=12)</li>\n<li>Multi Task Learning<ul>\n<li>segmentation: “asleep”, “awake”, “not_worn”</li>\n<li>event detection: “onset”, “wakeup”</li></ul></li>\n<li>I add the output of segmentation to the input of the event detection head. After that, I add multiple conv1d layer with various kernel_size(3, 7, 15) before the head.</li>\n<li>Skip Connection is very important.</li>\n</ul>\n<h3>Each Component</h3>\n<ul>\n<li><strong>UNet1D</strong><ul>\n<li>used skip connection instead concatenate when merge encoder and decoder</li></ul></li>\n<li><strong>WaveNet</strong><ul>\n<li>block = 2</li>\n<li>dilation = 4 ** N (N = 6, 4)</li></ul></li>\n<li><strong>Bi-GRU</strong><ul>\n<li>Transformer-like architecture ( (Bi-GRU → FeedForward) * num_block, with skip connection and layer normalization)</li></ul></li>\n<li><strong>ViT</strong><ul>\n<li>patch_size=4</li>\n<li>I use ViT for diversity, but attention-based model isn't well in my case.</li></ul></li>\n</ul>\n<h1>Ensemble</h1>\n<ul>\n<li>Weighted average of the model's output probabilities</li>\n</ul>\n<h1>PostProcess</h1>\n<ul>\n<li>Moving Average (window=3)</li>\n<li>Find Peak (using scipy.signal.find_peaks, score_th=0.01, distance=100))</li>\n<li>Eliminate time periods when events are unlikely to occur. (Removed predictions outside the following duration)<ul>\n<li>onset : 20:00 ~ 12:00</li>\n<li>wakeup : 3:00 ~ 20:00</li></ul></li>\n</ul>",
      "rawMarkdown": "Thank you to the organizer and Kaggle staff for this excited competition. And congratulations to winners. I only participated in the last three weeks, but I learned a lot and enjoy this competition.\n\n# PreProcess\n- standard scaler (for each subject)\n- sequence_size = 17,280 (12 * 60 * 24)\n- sequence overlap = 11,520 (12 * 60 * 16)\n- prepare features to be used for NN inputs\n    - anglez\n    - enmo\n    - anglez_diff\n    - enmo_diff\n    - time_unit_sin\n    - time_unit_cos\n\n# Validation\n* GroupKFold (k=5)\n\n# Model\n|  | CV(=5fold) | Public | Private | Weight |\n| :---: | :---: | :---: |\n| UNet1D - WaveNet | 0.746 | 0.735 | 0.772 | 0.3375 |\n| UNet1D - BiGRU(num_block=2) | 0.731 | not submit | not submit | 0.14875 |\n| UNet1D - BiGRU(num_block=3) | 0.721 | 0.708 | 0.754 | 0.06375 |\n| UNet1D - ViT - BiGRU(num_block=2) | 0.728 | not submit | not submit | 0.0375 |\n| UNet1D - WaveNet - BiGRU(num_block=1) | 0.751 | 0.730 | 0.770 | 0.4125 |\n| Ensemble | 0.775 | 0.756 | 0.786 |\n\n### Common in all models\n- Linear Layer at beginning\n- Downsampling before calculation of loss (negative = positive * 10)\n- Use hard label (window=12)\n- Multi Task Learning\n    - segmentation: “asleep”, “awake”, “not_worn”\n    - event detection: “onset”, “wakeup”\n- I add the output of segmentation to the input of the event detection head. After that, I add multiple conv1d layer with various kernel_size(3, 7, 15) before the head.\n- Skip Connection is very important.\n\n### Each Component\n- **UNet1D**\n    - used skip connection instead concatenate when merge encoder and decoder\n- **WaveNet**\n    - block = 2\n    - dilation = 4 ** N (N = 6, 4)\n- **Bi-GRU**\n    - Transformer-like architecture ( (Bi-GRU → FeedForward) * num_block, with skip connection and layer normalization)\n- **ViT**\n    - patch_size=4\n    - I use ViT for diversity, but attention-based model isn't well in my case.\n\n# Ensemble\n* Weighted average of the model's output probabilities\n\n# PostProcess\n* Moving Average (window=3)\n* Find Peak (using scipy.signal.find_peaks, score_th=0.01, distance=100))\n* Eliminate time periods when events are unlikely to occur. (Removed predictions outside the following duration)\n    - onset : 20:00 ~ 12:00\n    - wakeup : 3:00 ~ 20:00",
      "votes": 13
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2552973": "Thank you to the organizer and Kaggle staff for this excited competition. And congratulations to winners. I only participated in the last three weeks, but I learned a lot and enjoy this competition.\n\n# PreProcess\n- standard scaler (for each subject)\n- sequence_size = 17,280 (12 * 60 * 24)\n- sequence overlap = 11,520 (12 * 60 * 16)\n- prepare features to be used for NN inputs\n    - anglez\n    - enmo\n    - anglez_diff\n    - enmo_diff\n    - time_unit_sin\n    - time_unit_cos\n\n# Validation\n* GroupKFold (k=5)\n\n# Model\n|  | CV(=5fold) | Public | Private | Weight |\n| :---: | :---: | :---: |\n| UNet1D - WaveNet | 0.746 | 0.735 | 0.772 | 0.3375 |\n| UNet1D - BiGRU(num_block=2) | 0.731 | not submit | not submit | 0.14875 |\n| UNet1D - BiGRU(num_block=3) | 0.721 | 0.708 | 0.754 | 0.06375 |\n| UNet1D - ViT - BiGRU(num_block=2) | 0.728 | not submit | not submit | 0.0375 |\n| UNet1D - WaveNet - BiGRU(num_block=1) | 0.751 | 0.730 | 0.770 | 0.4125 |\n| Ensemble | 0.775 | 0.756 | 0.786 |\n\n### Common in all models\n- Linear Layer at beginning\n- Downsampling before calculation of loss (negative = positive * 10)\n- Use hard label (window=12)\n- Multi Task Learning\n    - segmentation: “asleep”, “awake”, “not_worn”\n    - event detection: “onset”, “wakeup”\n- I add the output of segmentation to the input of the event detection head. After that, I add multiple conv1d layer with various kernel_size(3, 7, 15) before the head.\n- Skip Connection is very important.\n\n### Each Component\n- **UNet1D**\n    - used skip connection instead concatenate when merge encoder and decoder\n- **WaveNet**\n    - block = 2\n    - dilation = 4 ** N (N = 6, 4)\n- **Bi-GRU**\n    - Transformer-like architecture ( (Bi-GRU → FeedForward) * num_block, with skip connection and layer normalization)\n- **ViT**\n    - patch_size=4\n    - I use ViT for diversity, but attention-based model isn't well in my case.\n\n# Ensemble\n* Weighted average of the model's output probabilities\n\n# PostProcess\n* Moving Average (window=3)\n* Find Peak (using scipy.signal.find_peaks, score_th=0.01, distance=100))\n* Eliminate time periods when events are unlikely to occur. (Removed predictions outside the following duration)\n    - onset : 20:00 ~ 12:00\n    - wakeup : 3:00 ~ 20:00"
  }
}