{
  "id": 459671,
  "title": "26th solution - 1D Unet with 1D CNN, Hybrid CNN-Transformer and WBF",
  "url": "/competitions/child-mind-institute-detect-sleep-states/writeups/robusta-26th-solution-1d-unet-with-1d-cnn-hybrid-c",
  "author_name": "",
  "post_date": "2023-12-16T14:25:31.730Z",
  "votes": 20,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Many thanks competition host and Kaggle for another interesting challenge. A big congrats to all the winners !<br>\nAs usual, I have a good time learning during this competition. In this post, I will brieftly describe my solution.<br>\nAll of my models learn to predict 3 targets:</p>\n<ul>\n<li>Lower resolution heatmap of eventness (transition/critical points)</li>\n<li>Offset to event (from coarser grid coordinates to pixel-level event step)</li>\n<li>Binary segmentation mask for sleep period</li>\n</ul>\n<p>Input is a window of 2 or 4 days, where first and last 4 hours (near border) is still included in the input sequence, but ignored in loss computation. The intuition is to prevent ambigous near border region since we need more context around to judge.<br>\nThe longer crop window, the higher change of overfit, but better context to make accurate prediction. Especially for Hybrid CNN-Transformer with much larger model capacity, model quickly overfit when using large window. I consider crop_len is also an important hyperparam.<br>\nSegmentation mask prediction was used as an auxiliary target and was not ultilized in the inference pipeline.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F6e48e25ef889bac5268817b5557210a2%2Ftarget.png?generation=1701850406918104&amp;alt=media\" alt=\"target visualization\"><br>\nThe offset prediction then be used as \"error correction\" to provide sub-pixel level accuracy despite large target strides (4, 8, 16, ..), inspired by Human Pose Estimation and Object Detection models. It looks nearly the same as CCRF ( Combined Classification and Regression Form) used in <a href=\"https://github.com/HuangJunJie2017/UDP-Pose\" target=\"_blank\">UDP-Pose</a>.</p>\n<h2>Model</h2>\n<p>1D Unet with hierarchical 1D encoder. Two type of encoders: pure 1D-CNN or Hybrid CNN-Transformer. Output prediction is from 1/4 stride from original resolution (that is, make prediction per 3 steps or 15 seconds)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F4751841db413e7df537c31b08a23d845%2Farchitecture.png?generation=1701850445592192&amp;alt=media\" alt=\"architecture\"></p>\n<ul>\n<li>Tried various stem type, Squeezeformer stem give slightly better/consistent CV</li>\n</ul>\n<h3>1D-CNN</h3>\n<p>Simply try to transform 2D-CNN image models to 1D version, replace all BatchNorm -&gt; LayerNorm</p>\n<ul>\n<li>Crop_len = 4 days</li>\n<li>Basic Block: MobilenetV3, expand_ratio = 2, constant dim = 96 across stages</li>\n<li>Depth = 8 with stage computation ratios [3, 3, 3, 3, 3, 3, 3, 3] (VGG style ?) </li>\n<li>ECA/SE attention</li>\n<li>backbone_dropout = 0.4, head_dropout = 0.4</li>\n</ul>\n<h3>Hybrid CNN-Transformer</h3>\n<ul>\n<li>Crop_len = 2 days</li>\n<li>SqueezeformerBlock with RoPE from top solutions in <a href=\"https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434485\" target=\"_blank\">previous ASL Fingerspelling competition</a></li>\n<li>Two architecture included in final submission: hierarchical/non-hierarchical Transformer</li>\n</ul>\n<h2>Loss</h2>\n<p>Simple weighted combination of:</p>\n<ul>\n<li>BCE for heatmap</li>\n<li>MSE or L1Loss for offset</li>\n<li>BCE for segmentation</li>\n</ul>\n<p>Best weights: total_loss =  1.0 * heatmap + 0.2 * offset + 0.2 * segment</p>\n<h2>Training</h2>\n<ul>\n<li>FP16, Model EMA, batch size 16</li>\n<li>Max updates: 500 * 40 for CNN, 200 * 60 for Hybrid Encoder</li>\n<li>Optimizer: Lookahead AdamW + Cosine LR</li>\n</ul>\n<h2>Post processing</h2>\n<ul>\n<li>During training, track validation score outputed from simple NMS (Point-based NMS).</li>\n<li>Submission using Point-based version of <strong>Weighted Boxes Fusion (WBF)</strong> to ensemble multiple models (after NMS).</li>\n</ul>\n<h2>Results</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fb4310b112b80db10d35a33ae37e533c3%2Fresults.png?generation=1701865443994509&amp;alt=media\" alt=\"local cv results\"></p>\n<h1>What worked</h1>\n<ul>\n<li>Tuning sigma for Gaussian-like heatmap target: very important and give significant boost. Best sigma = 5 * 12 = 5 minutes;  Gaussian width = 2 * 3 * sigma = 30 minutes.</li>\n<li>Auxiliary segmentation loss: + ~0.01 AP</li>\n<li>Tuning loss weights</li>\n<li>Try various architecture designs: depth, dim, norm, activation, stage computation ratios,..</li>\n<li>Simple downsample the sequence length by reshaping: worked better than Convolution downsampling on raw sequence. Perhaps help avoid overfiting as mentioned in <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416026\" target=\"_blank\">this solution</a></li>\n<li>Model EMA: significantly stablize training and allow training for more epochs, crucial for Hybrid encoder</li>\n<li>Weighted Boxes Fusion: with 2 models ensemble, +1.3 OOF AP from 77.0 -&gt; 78.3</li>\n</ul>\n<h1>What not worked</h1>\n<ul>\n<li>Feature engineering: tried various features (on single fold) but no boost. One make me suprised is just single feature 'anglez_sign' (+1 or -1) could archive CV=60.89</li>\n<li>Augmentation: just implement and tried some simple methods (Flip, Noise, Resampling, TimeStretch, ..) in the last day, also seem no boost and need more tuning.</li>\n<li>Decoupled head: Decoupled head for objectness &amp; offset usually boost Object Detection model, but in this case simple single linear head with output_dim=5 work best.</li>\n<li>Ultilize segmentation prediction in post processing: no clear boost</li>\n</ul>",
  "messages": [
    {
      "id": "2550707",
      "postDate": "12/06/2023 08:22:08",
      "content": "<p>Many thanks competition host and Kaggle for another interesting challenge. A big congrats to all the winners !<br>\nAs usual, I have a good time learning during this competition. In this post, I will brieftly describe my solution.<br>\nAll of my models learn to predict 3 targets:</p>\n<ul>\n<li>Lower resolution heatmap of eventness (transition/critical points)</li>\n<li>Offset to event (from coarser grid coordinates to pixel-level event step)</li>\n<li>Binary segmentation mask for sleep period</li>\n</ul>\n<p>Input is a window of 2 or 4 days, where first and last 4 hours (near border) is still included in the input sequence, but ignored in loss computation. The intuition is to prevent ambigous near border region since we need more context around to judge.<br>\nThe longer crop window, the higher change of overfit, but better context to make accurate prediction. Especially for Hybrid CNN-Transformer with much larger model capacity, model quickly overfit when using large window. I consider crop_len is also an important hyperparam.<br>\nSegmentation mask prediction was used as an auxiliary target and was not ultilized in the inference pipeline.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F6e48e25ef889bac5268817b5557210a2%2Ftarget.png?generation=1701850406918104&amp;alt=media\" alt=\"target visualization\"><br>\nThe offset prediction then be used as \"error correction\" to provide sub-pixel level accuracy despite large target strides (4, 8, 16, ..), inspired by Human Pose Estimation and Object Detection models. It looks nearly the same as CCRF ( Combined Classification and Regression Form) used in <a href=\"https://github.com/HuangJunJie2017/UDP-Pose\" target=\"_blank\">UDP-Pose</a>.</p>\n<h2>Model</h2>\n<p>1D Unet with hierarchical 1D encoder. Two type of encoders: pure 1D-CNN or Hybrid CNN-Transformer. Output prediction is from 1/4 stride from original resolution (that is, make prediction per 3 steps or 15 seconds)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F4751841db413e7df537c31b08a23d845%2Farchitecture.png?generation=1701850445592192&amp;alt=media\" alt=\"architecture\"></p>\n<ul>\n<li>Tried various stem type, Squeezeformer stem give slightly better/consistent CV</li>\n</ul>\n<h3>1D-CNN</h3>\n<p>Simply try to transform 2D-CNN image models to 1D version, replace all BatchNorm -&gt; LayerNorm</p>\n<ul>\n<li>Crop_len = 4 days</li>\n<li>Basic Block: MobilenetV3, expand_ratio = 2, constant dim = 96 across stages</li>\n<li>Depth = 8 with stage computation ratios [3, 3, 3, 3, 3, 3, 3, 3] (VGG style ?) </li>\n<li>ECA/SE attention</li>\n<li>backbone_dropout = 0.4, head_dropout = 0.4</li>\n</ul>\n<h3>Hybrid CNN-Transformer</h3>\n<ul>\n<li>Crop_len = 2 days</li>\n<li>SqueezeformerBlock with RoPE from top solutions in <a href=\"https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434485\" target=\"_blank\">previous ASL Fingerspelling competition</a></li>\n<li>Two architecture included in final submission: hierarchical/non-hierarchical Transformer</li>\n</ul>\n<h2>Loss</h2>\n<p>Simple weighted combination of:</p>\n<ul>\n<li>BCE for heatmap</li>\n<li>MSE or L1Loss for offset</li>\n<li>BCE for segmentation</li>\n</ul>\n<p>Best weights: total_loss =  1.0 * heatmap + 0.2 * offset + 0.2 * segment</p>\n<h2>Training</h2>\n<ul>\n<li>FP16, Model EMA, batch size 16</li>\n<li>Max updates: 500 * 40 for CNN, 200 * 60 for Hybrid Encoder</li>\n<li>Optimizer: Lookahead AdamW + Cosine LR</li>\n</ul>\n<h2>Post processing</h2>\n<ul>\n<li>During training, track validation score outputed from simple NMS (Point-based NMS).</li>\n<li>Submission using Point-based version of <strong>Weighted Boxes Fusion (WBF)</strong> to ensemble multiple models (after NMS).</li>\n</ul>\n<h2>Results</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fb4310b112b80db10d35a33ae37e533c3%2Fresults.png?generation=1701865443994509&amp;alt=media\" alt=\"local cv results\"></p>\n<h1>What worked</h1>\n<ul>\n<li>Tuning sigma for Gaussian-like heatmap target: very important and give significant boost. Best sigma = 5 * 12 = 5 minutes;  Gaussian width = 2 * 3 * sigma = 30 minutes.</li>\n<li>Auxiliary segmentation loss: + ~0.01 AP</li>\n<li>Tuning loss weights</li>\n<li>Try various architecture designs: depth, dim, norm, activation, stage computation ratios,..</li>\n<li>Simple downsample the sequence length by reshaping: worked better than Convolution downsampling on raw sequence. Perhaps help avoid overfiting as mentioned in <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416026\" target=\"_blank\">this solution</a></li>\n<li>Model EMA: significantly stablize training and allow training for more epochs, crucial for Hybrid encoder</li>\n<li>Weighted Boxes Fusion: with 2 models ensemble, +1.3 OOF AP from 77.0 -&gt; 78.3</li>\n</ul>\n<h1>What not worked</h1>\n<ul>\n<li>Feature engineering: tried various features (on single fold) but no boost. One make me suprised is just single feature 'anglez_sign' (+1 or -1) could archive CV=60.89</li>\n<li>Augmentation: just implement and tried some simple methods (Flip, Noise, Resampling, TimeStretch, ..) in the last day, also seem no boost and need more tuning.</li>\n<li>Decoupled head: Decoupled head for objectness &amp; offset usually boost Object Detection model, but in this case simple single linear head with output_dim=5 work best.</li>\n<li>Ultilize segmentation prediction in post processing: no clear boost</li>\n</ul>",
      "rawMarkdown": "Many thanks competition host and Kaggle for another interesting challenge. A big congrats to all the winners !\n\nAs usual, I have a good time learning during this competition. In this post, I will brieftly describe my solution.\n\nAll of my models learn to predict 3 targets:\n+ Lower resolution heatmap of eventness (transition/critical points)\n+ Offset to event (from coarser grid coordinates to pixel-level event step)\n+ Binary segmentation mask for sleep period\n\n \nInput is a window of 2 or 4 days, where first and last 4 hours (near border) is still included in the input sequence, but ignored in loss computation. The intuition is to prevent ambigous near border region since we need more context around to judge.\nThe longer crop window, the higher change of overfit, but better context to make accurate prediction. Especially for Hybrid CNN-Transformer with much larger model capacity, model quickly overfit when using large window. I consider crop_len is also an important hyperparam.\n\nSegmentation mask prediction was used as an auxiliary target and was not ultilized in the inference pipeline.\n\n![target visualization](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F6e48e25ef889bac5268817b5557210a2%2Ftarget.png?generation=1701850406918104&alt=media)\n\nThe offset prediction then be used as \"error correction\" to provide sub-pixel level accuracy despite large target strides (4, 8, 16, ..), inspired by Human Pose Estimation and Object Detection models. It looks nearly the same as CCRF ( Combined Classification and Regression Form) used in [UDP-Pose](https://github.com/HuangJunJie2017/UDP-Pose).\n\n\n\n## Model\n1D Unet with hierarchical 1D encoder. Two type of encoders: pure 1D-CNN or Hybrid CNN-Transformer. Output prediction is from 1/4 stride from original resolution (that is, make prediction per 3 steps or 15 seconds)\n\n\n![architecture](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F4751841db413e7df537c31b08a23d845%2Farchitecture.png?generation=1701850445592192&alt=media)\n\n\n- Tried various stem type, Squeezeformer stem give slightly better/consistent CV\n\n### 1D-CNN\nSimply try to transform 2D-CNN image models to 1D version, replace all BatchNorm -> LayerNorm\n- Crop_len = 4 days\n- Basic Block: MobilenetV3, expand_ratio = 2, constant dim = 96 across stages\n- Depth = 8 with stage computation ratios [3, 3, 3, 3, 3, 3, 3, 3] (VGG style ?) \n- ECA/SE attention\n- backbone_dropout = 0.4, head_dropout = 0.4\n\n### Hybrid CNN-Transformer\n- Crop_len = 2 days\n- SqueezeformerBlock with RoPE from top solutions in [previous ASL Fingerspelling competition](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434485)\n- Two architecture included in final submission: hierarchical/non-hierarchical Transformer\n\n\n## Loss\nSimple weighted combination of:\n- BCE for heatmap\n- MSE or L1Loss for offset\n- BCE for segmentation\n \nBest weights: total_loss =  1.0 * heatmap + 0.2 * offset + 0.2 * segment\n\n\n## Training\n- FP16, Model EMA, batch size 16\n- Max updates: 500 * 40 for CNN, 200 * 60 for Hybrid Encoder\n- Optimizer: Lookahead AdamW + Cosine LR\n\n\n## Post processing\n- During training, track validation score outputed from simple NMS (Point-based NMS).\n- Submission using Point-based version of **Weighted Boxes Fusion (WBF)** to ensemble multiple models (after NMS).\n\n## Results\n\n![local cv results](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fb4310b112b80db10d35a33ae37e533c3%2Fresults.png?generation=1701865443994509&alt=media)\n\n# What worked\n- Tuning sigma for Gaussian-like heatmap target: very important and give significant boost. Best sigma = 5 * 12 = 5 minutes;  Gaussian width = 2 * 3 * sigma = 30 minutes.\n- Auxiliary segmentation loss: + ~0.01 AP\n- Tuning loss weights\n- Try various architecture designs: depth, dim, norm, activation, stage computation ratios,..\n- Simple downsample the sequence length by reshaping: worked better than Convolution downsampling on raw sequence. Perhaps help avoid overfiting as mentioned in [this solution](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416026)\n- Model EMA: significantly stablize training and allow training for more epochs, crucial for Hybrid encoder\n- Weighted Boxes Fusion: with 2 models ensemble, +1.3 OOF AP from 77.0 -> 78.3\n\n\n# What not worked\n- Feature engineering: tried various features (on single fold) but no boost. One make me suprised is just single feature 'anglez_sign' (+1 or -1) could archive CV=60.89\n- Augmentation: just implement and tried some simple methods (Flip, Noise, Resampling, TimeStretch, ..) in the last day, also seem no boost and need more tuning.\n- Decoupled head: Decoupled head for objectness & offset usually boost Object Detection model, but in this case simple single linear head with output_dim=5 work best.\n- Ultilize segmentation prediction in post processing: no clear boost",
      "votes": null
    },
    {
      "id": "2550812",
      "postDate": "12/06/2023 09:41:59",
      "content": "<p>What an impressive solution! Congrats bro</p>",
      "rawMarkdown": "What an impressive solution! Congrats bro",
      "votes": null
    },
    {
      "id": "2550853",
      "postDate": "12/06/2023 10:31:18",
      "content": "<p>Thanks bro! 😀</p>",
      "rawMarkdown": "Thanks bro! 😀",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2550812,
      "author_name": "huyduong7101",
      "author_url": "",
      "post_date": "12/06/2023 09:41:59",
      "content": "<p>What an impressive solution! Congrats bro</p>",
      "votes": null,
      "replies": [
        {
          "id": 2550853,
          "author_name": "dangnh0611",
          "author_url": "",
          "post_date": "12/06/2023 10:31:18",
          "content": "<p>Thanks bro! 😀</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2550707": "Many thanks competition host and Kaggle for another interesting challenge. A big congrats to all the winners !\n\nAs usual, I have a good time learning during this competition. In this post, I will brieftly describe my solution.\n\nAll of my models learn to predict 3 targets:\n+ Lower resolution heatmap of eventness (transition/critical points)\n+ Offset to event (from coarser grid coordinates to pixel-level event step)\n+ Binary segmentation mask for sleep period\n\n \nInput is a window of 2 or 4 days, where first and last 4 hours (near border) is still included in the input sequence, but ignored in loss computation. The intuition is to prevent ambigous near border region since we need more context around to judge.\nThe longer crop window, the higher change of overfit, but better context to make accurate prediction. Especially for Hybrid CNN-Transformer with much larger model capacity, model quickly overfit when using large window. I consider crop_len is also an important hyperparam.\n\nSegmentation mask prediction was used as an auxiliary target and was not ultilized in the inference pipeline.\n\n![target visualization](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F6e48e25ef889bac5268817b5557210a2%2Ftarget.png?generation=1701850406918104&alt=media)\n\nThe offset prediction then be used as \"error correction\" to provide sub-pixel level accuracy despite large target strides (4, 8, 16, ..), inspired by Human Pose Estimation and Object Detection models. It looks nearly the same as CCRF ( Combined Classification and Regression Form) used in [UDP-Pose](https://github.com/HuangJunJie2017/UDP-Pose).\n\n\n\n## Model\n1D Unet with hierarchical 1D encoder. Two type of encoders: pure 1D-CNN or Hybrid CNN-Transformer. Output prediction is from 1/4 stride from original resolution (that is, make prediction per 3 steps or 15 seconds)\n\n\n![architecture](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F4751841db413e7df537c31b08a23d845%2Farchitecture.png?generation=1701850445592192&alt=media)\n\n\n- Tried various stem type, Squeezeformer stem give slightly better/consistent CV\n\n### 1D-CNN\nSimply try to transform 2D-CNN image models to 1D version, replace all BatchNorm -> LayerNorm\n- Crop_len = 4 days\n- Basic Block: MobilenetV3, expand_ratio = 2, constant dim = 96 across stages\n- Depth = 8 with stage computation ratios [3, 3, 3, 3, 3, 3, 3, 3] (VGG style ?) \n- ECA/SE attention\n- backbone_dropout = 0.4, head_dropout = 0.4\n\n### Hybrid CNN-Transformer\n- Crop_len = 2 days\n- SqueezeformerBlock with RoPE from top solutions in [previous ASL Fingerspelling competition](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434485)\n- Two architecture included in final submission: hierarchical/non-hierarchical Transformer\n\n\n## Loss\nSimple weighted combination of:\n- BCE for heatmap\n- MSE or L1Loss for offset\n- BCE for segmentation\n \nBest weights: total_loss =  1.0 * heatmap + 0.2 * offset + 0.2 * segment\n\n\n## Training\n- FP16, Model EMA, batch size 16\n- Max updates: 500 * 40 for CNN, 200 * 60 for Hybrid Encoder\n- Optimizer: Lookahead AdamW + Cosine LR\n\n\n## Post processing\n- During training, track validation score outputed from simple NMS (Point-based NMS).\n- Submission using Point-based version of **Weighted Boxes Fusion (WBF)** to ensemble multiple models (after NMS).\n\n## Results\n\n![local cv results](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fb4310b112b80db10d35a33ae37e533c3%2Fresults.png?generation=1701865443994509&alt=media)\n\n# What worked\n- Tuning sigma for Gaussian-like heatmap target: very important and give significant boost. Best sigma = 5 * 12 = 5 minutes;  Gaussian width = 2 * 3 * sigma = 30 minutes.\n- Auxiliary segmentation loss: + ~0.01 AP\n- Tuning loss weights\n- Try various architecture designs: depth, dim, norm, activation, stage computation ratios,..\n- Simple downsample the sequence length by reshaping: worked better than Convolution downsampling on raw sequence. Perhaps help avoid overfiting as mentioned in [this solution](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416026)\n- Model EMA: significantly stablize training and allow training for more epochs, crucial for Hybrid encoder\n- Weighted Boxes Fusion: with 2 models ensemble, +1.3 OOF AP from 77.0 -> 78.3\n\n\n# What not worked\n- Feature engineering: tried various features (on single fold) but no boost. One make me suprised is just single feature 'anglez_sign' (+1 or -1) could archive CV=60.89\n- Augmentation: just implement and tried some simple methods (Flip, Noise, Resampling, TimeStretch, ..) in the last day, also seem no boost and need more tuning.\n- Decoupled head: Decoupled head for objectness & offset usually boost Object Detection model, but in this case simple single linear head with output_dim=5 work best.\n- Ultilize segmentation prediction in post processing: no clear boost",
    "2550812": "What an impressive solution! Congrats bro",
    "2550853": "Thanks bro! 😀"
  },
  "source": "meta"
}