{
  "id": 459678,
  "title": "9th Place Solution - shu421 part (1D WaveNet + Transformer + GRU)",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/459678",
  "author_name": "",
  "post_date": "2023-12-06T08:35:15.431549100Z",
  "votes": 38,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Thank you organizer and all participants, I learned a lot from this competition!<br>\nIt was a good opportunity to learn about NN modeling for such a long sequence.</p>\n<h1>Material</h1>\n<ul>\n<li><a href=\"https://github.com/shu421/Kaggle_CMI_9thSolution\" target=\"_blank\">Training Code</a></li>\n<li><a href=\"https://boulder-swamp-0ea.notion.site/Child-Mind-Institute-Detect-Sleep-States-37affba149e04a5498cb0cd9f2bb68ec?pvs=4\" target=\"_blank\">Experiment Log</a><ul>\n<li>I used Notion to manage my experiments.</li>\n<li>Sorry, not much maintained</li></ul></li>\n</ul>\n<h1>Overview</h1>\n<ul>\n<li>1D CNN + GRU encoder and simple mlp decoder model</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fc7d2ee03be91818c7a23c655a5c7e4c3%2F2023-12-06%2017.33.16.png?generation=1701851622179529&amp;alt=media\" alt=\"exp105 model\"></p>\n<h1>Feature</h1>\n<ul>\n<li>1-4 features are scaled with RobustScaler</li>\n</ul>\n<ol>\n<li>anglez</li>\n<li>enmo</li>\n<li>anglez_diff</li>\n<li>enmo_diff</li>\n<li>hour_sin</li>\n<li>hour_cos</li>\n<li>similar_wave<ul>\n<li>0-1 variable</li>\n<li>calculate the similarity of the patched data with 17280 length</li>\n<li>if similarity == 1: similar_wave = 1 else 0</li></ul></li>\n</ol>\n<p><strong>Calculate similar_wave code</strong></p>\n<pre><code> ():\n    \n    correlation = np.correlate(signal, pattern, mode=)\n    normalized_correlation = correlation / (\n        np.linalg.norm(pattern)\n        * np.sqrt(np.convolve(signal**, np.ones((pattern)), ))\n    )\n     normalized_correlation\n\n ():\n    \n    num_bins = (signal) // window_size\n    similar_steps = ()\n\n     i  (num_bins):\n         j  (i + , num_bins):\n            bin_i = signal[i * window_size : (i + ) * window_size]\n            bin_j = signal[j * window_size : (j + ) * window_size]\n            correlation = (cross_correlation(bin_i, bin_j))\n             correlation &gt; threshold:\n                similar_steps.add(i * window_size)\n                similar_steps.add(j * window_size)\n\n     ((similar_steps))\n\n ():\n    similar_steps = find_similar_steps(_df[], window_size=, threshold=)\n\n    similar_wave = np.zeros((_df))\n     (similar_steps) &gt; :\n         start_step  similar_steps:\n            end_step = start_step + \n\n            similar_wave[start_step:end_step] = \n\n     pl.DataFrame(similar_wave.astype(np.Int8))\n</code></pre>\n<h1>Label</h1>\n<ol>\n<li>asleep<ul>\n<li>0-1 label</li>\n<li>BCEWithLogitsLoss</li></ul></li>\n<li>onset and wakeup<ul>\n<li>gaussian distribution(sigma=12)</li>\n<li>BCEWithLogitsLoss(pos_weight=100)</li></ul></li>\n</ol>\n<h1>Model</h1>\n<h2>Model105</h2>\n<p>CV: 0.778, PublicLB: 0.752, PrivateLB: 0.805</p>\n<ul>\n<li>Feature Extractor<ul>\n<li>WaveNet(dilation=1,2,…,6) × 3</li>\n<li>the outputs are concatenated and reshaped</li></ul></li>\n<li>Encoder1<ul>\n<li>Conv1d(stride=6)</li>\n<li>MultiResidualBiGRU × 4</li></ul></li>\n<li>Encoder2<ul>\n<li>Conv1d(stride=2)</li>\n<li>MultiResidualBiGRU × 4</li></ul></li>\n<li>Decoder<ul>\n<li>Upsampling with ConvTranspose1d(half overlap)<ul>\n<li>half overlap: kernel_size=strie*2, stride=stride</li></ul></li></ul></li>\n<li>mlp layer</li>\n</ul>\n<h2>Model101</h2>\n<p>CV: 0.776, PublicLB: 0.742, PrivateLB: 0.811</p>\n<p>Feature Extractor and Decoder are almost the same as Model105</p>\n<ul>\n<li>Encoder<ul>\n<li>Conv1d(stride=12)</li>\n<li>PositionalEncoding</li>\n<li>Transformer × 4</li>\n<li>MultiResidualBiGRU × 4</li></ul></li>\n</ul>\n<h1>Training</h1>\n<ul>\n<li>5 StratifiedKFold(y=sum(asleep), groups=series_id)</li>\n<li>n_epochs = 30</li>\n<li>batch size = 32</li>\n<li>CosineAnnealingLR</li>\n<li>AdamW(lr=8e-4)</li>\n<li>input feature length = 17280</li>\n<li>no overlap training</li>\n</ul>\n<h1>Inference</h1>\n<ul>\n<li>batch size = 1</li>\n<li>Feature engineering and inference are done for each series_id to save memory</li>\n<li>no overlap inference<ul>\n<li>I tried to use overlap inference, but it didn't work well.</li></ul></li>\n</ul>\n<h1>Post Processing</h1>\n<ul>\n<li>find_peaks(threshold=0.1, size=120)<ul>\n<li>It was not enough</li>\n<li>my teammate youri had more strong post processing</li></ul></li>\n</ul>\n<h1>Worked</h1>\n<ul>\n<li>use <strong>fp32</strong>: PublicLB +0.005</li>\n<li><a href=\"https://www.kaggle.com/code/junkoda/pytorch-lstm-with-tensorflow-like-initialization\" target=\"_blank\">tensorflow-like initialization</a></li>\n<li>sharply peaked gaussian label</li>\n<li>onset and wakeup pos_weight=100</li>\n<li>RobustScaler</li>\n<li>predicts a lot of steps</li>\n</ul>\n<h1>Not Worked</h1>\n<ul>\n<li>data augmentation(flip, mixup, etc…)</li>\n<li>mlm pretraining</li>\n<li>GBDT reranking(only tried binary classification)</li>\n<li>stacking(GBDT, NN, etc…)</li>\n<li>hand labeling on train data</li>\n<li>full sequence training<ul>\n<li>I tried it with UNet, but it didn't work well.</li></ul></li>\n<li>external data</li>\n<li>other model(ViT, LSTM, etc…)</li>\n<li>other loss(focal loss, dice loss, etc…)</li>\n</ul>\n<h1>Other</h1>\n<ul>\n<li>I created visualization app using Streamlit to check the predictions.</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fd44856caebcc2c2a053158e5ee31e5ea%2F2023-12-06%2017.34.11.png?generation=1701851684361984&amp;alt=media\" alt=\"viz example\"></p>",
  "messages": [
    {
      "id": "2550725",
      "postDate": "12/06/2023 08:35:15",
      "content": "<p>Thank you organizer and all participants, I learned a lot from this competition!<br>\nIt was a good opportunity to learn about NN modeling for such a long sequence.</p>\n<h1>Material</h1>\n<ul>\n<li><a href=\"https://github.com/shu421/Kaggle_CMI_9thSolution\" target=\"_blank\">Training Code</a></li>\n<li><a href=\"https://boulder-swamp-0ea.notion.site/Child-Mind-Institute-Detect-Sleep-States-37affba149e04a5498cb0cd9f2bb68ec?pvs=4\" target=\"_blank\">Experiment Log</a><ul>\n<li>I used Notion to manage my experiments.</li>\n<li>Sorry, not much maintained</li></ul></li>\n</ul>\n<h1>Overview</h1>\n<ul>\n<li>1D CNN + GRU encoder and simple mlp decoder model</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fc7d2ee03be91818c7a23c655a5c7e4c3%2F2023-12-06%2017.33.16.png?generation=1701851622179529&amp;alt=media\" alt=\"exp105 model\"></p>\n<h1>Feature</h1>\n<ul>\n<li>1-4 features are scaled with RobustScaler</li>\n</ul>\n<ol>\n<li>anglez</li>\n<li>enmo</li>\n<li>anglez_diff</li>\n<li>enmo_diff</li>\n<li>hour_sin</li>\n<li>hour_cos</li>\n<li>similar_wave<ul>\n<li>0-1 variable</li>\n<li>calculate the similarity of the patched data with 17280 length</li>\n<li>if similarity == 1: similar_wave = 1 else 0</li></ul></li>\n</ol>\n<p><strong>Calculate similar_wave code</strong></p>\n<pre><code> ():\n    \n    correlation = np.correlate(signal, pattern, mode=)\n    normalized_correlation = correlation / (\n        np.linalg.norm(pattern)\n        * np.sqrt(np.convolve(signal**, np.ones((pattern)), ))\n    )\n     normalized_correlation\n\n ():\n    \n    num_bins = (signal) // window_size\n    similar_steps = ()\n\n     i  (num_bins):\n         j  (i + , num_bins):\n            bin_i = signal[i * window_size : (i + ) * window_size]\n            bin_j = signal[j * window_size : (j + ) * window_size]\n            correlation = (cross_correlation(bin_i, bin_j))\n             correlation &gt; threshold:\n                similar_steps.add(i * window_size)\n                similar_steps.add(j * window_size)\n\n     ((similar_steps))\n\n ():\n    similar_steps = find_similar_steps(_df[], window_size=, threshold=)\n\n    similar_wave = np.zeros((_df))\n     (similar_steps) &gt; :\n         start_step  similar_steps:\n            end_step = start_step + \n\n            similar_wave[start_step:end_step] = \n\n     pl.DataFrame(similar_wave.astype(np.Int8))\n</code></pre>\n<h1>Label</h1>\n<ol>\n<li>asleep<ul>\n<li>0-1 label</li>\n<li>BCEWithLogitsLoss</li></ul></li>\n<li>onset and wakeup<ul>\n<li>gaussian distribution(sigma=12)</li>\n<li>BCEWithLogitsLoss(pos_weight=100)</li></ul></li>\n</ol>\n<h1>Model</h1>\n<h2>Model105</h2>\n<p>CV: 0.778, PublicLB: 0.752, PrivateLB: 0.805</p>\n<ul>\n<li>Feature Extractor<ul>\n<li>WaveNet(dilation=1,2,…,6) × 3</li>\n<li>the outputs are concatenated and reshaped</li></ul></li>\n<li>Encoder1<ul>\n<li>Conv1d(stride=6)</li>\n<li>MultiResidualBiGRU × 4</li></ul></li>\n<li>Encoder2<ul>\n<li>Conv1d(stride=2)</li>\n<li>MultiResidualBiGRU × 4</li></ul></li>\n<li>Decoder<ul>\n<li>Upsampling with ConvTranspose1d(half overlap)<ul>\n<li>half overlap: kernel_size=strie*2, stride=stride</li></ul></li></ul></li>\n<li>mlp layer</li>\n</ul>\n<h2>Model101</h2>\n<p>CV: 0.776, PublicLB: 0.742, PrivateLB: 0.811</p>\n<p>Feature Extractor and Decoder are almost the same as Model105</p>\n<ul>\n<li>Encoder<ul>\n<li>Conv1d(stride=12)</li>\n<li>PositionalEncoding</li>\n<li>Transformer × 4</li>\n<li>MultiResidualBiGRU × 4</li></ul></li>\n</ul>\n<h1>Training</h1>\n<ul>\n<li>5 StratifiedKFold(y=sum(asleep), groups=series_id)</li>\n<li>n_epochs = 30</li>\n<li>batch size = 32</li>\n<li>CosineAnnealingLR</li>\n<li>AdamW(lr=8e-4)</li>\n<li>input feature length = 17280</li>\n<li>no overlap training</li>\n</ul>\n<h1>Inference</h1>\n<ul>\n<li>batch size = 1</li>\n<li>Feature engineering and inference are done for each series_id to save memory</li>\n<li>no overlap inference<ul>\n<li>I tried to use overlap inference, but it didn't work well.</li></ul></li>\n</ul>\n<h1>Post Processing</h1>\n<ul>\n<li>find_peaks(threshold=0.1, size=120)<ul>\n<li>It was not enough</li>\n<li>my teammate youri had more strong post processing</li></ul></li>\n</ul>\n<h1>Worked</h1>\n<ul>\n<li>use <strong>fp32</strong>: PublicLB +0.005</li>\n<li><a href=\"https://www.kaggle.com/code/junkoda/pytorch-lstm-with-tensorflow-like-initialization\" target=\"_blank\">tensorflow-like initialization</a></li>\n<li>sharply peaked gaussian label</li>\n<li>onset and wakeup pos_weight=100</li>\n<li>RobustScaler</li>\n<li>predicts a lot of steps</li>\n</ul>\n<h1>Not Worked</h1>\n<ul>\n<li>data augmentation(flip, mixup, etc…)</li>\n<li>mlm pretraining</li>\n<li>GBDT reranking(only tried binary classification)</li>\n<li>stacking(GBDT, NN, etc…)</li>\n<li>hand labeling on train data</li>\n<li>full sequence training<ul>\n<li>I tried it with UNet, but it didn't work well.</li></ul></li>\n<li>external data</li>\n<li>other model(ViT, LSTM, etc…)</li>\n<li>other loss(focal loss, dice loss, etc…)</li>\n</ul>\n<h1>Other</h1>\n<ul>\n<li>I created visualization app using Streamlit to check the predictions.</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fd44856caebcc2c2a053158e5ee31e5ea%2F2023-12-06%2017.34.11.png?generation=1701851684361984&amp;alt=media\" alt=\"viz example\"></p>",
      "rawMarkdown": "Thank you organizer and all participants, I learned a lot from this competition!\nIt was a good opportunity to learn about NN modeling for such a long sequence.\n\n# Material\n- [Training Code](https://github.com/shu421/Kaggle_CMI_9thSolution)\n- [Experiment Log](https://boulder-swamp-0ea.notion.site/Child-Mind-Institute-Detect-Sleep-States-37affba149e04a5498cb0cd9f2bb68ec?pvs=4)\n    - I used Notion to manage my experiments.\n    - Sorry, not much maintained\n\n# Overview\n- 1D CNN + GRU encoder and simple mlp decoder model\n\n![exp105 model](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fc7d2ee03be91818c7a23c655a5c7e4c3%2F2023-12-06%2017.33.16.png?generation=1701851622179529&alt=media)\n\n# Feature\n- 1-4 features are scaled with RobustScaler\n1. anglez\n2. enmo\n3. anglez_diff\n4. enmo_diff\n5. hour_sin\n6. hour_cos\n7. similar_wave\n    - 0-1 variable\n    - calculate the similarity of the patched data with 17280 length\n    - if similarity == 1: similar_wave = 1 else 0\n\n**Calculate similar_wave code**\n```python\ndef cross_correlation(signal, pattern):\n    \"\"\"\n    Compute cross correlation between signal and pattern\n    \"\"\"\n    correlation = np.correlate(signal, pattern, mode=\"valid\")\n    normalized_correlation = correlation / (\n        np.linalg.norm(pattern)\n        * np.sqrt(np.convolve(signal**2, np.ones(len(pattern)), \"valid\"))\n    )\n    return normalized_correlation\n\ndef find_similar_steps(signal, window_size=17280, threshold=1.0):\n    \"\"\"\n    Compare waveforms of each bin and find pairs with high similarity.\n    \"\"\"\n    num_bins = len(signal) // window_size\n    similar_steps = set()\n\n    for i in range(num_bins):\n        for j in range(i + 1, num_bins):\n            bin_i = signal[i * window_size : (i + 1) * window_size]\n            bin_j = signal[j * window_size : (j + 1) * window_size]\n            correlation = max(cross_correlation(bin_i, bin_j))\n            if correlation > threshold:\n                similar_steps.add(i * window_size)\n                similar_steps.add(j * window_size)\n\n    return sorted(list(similar_steps))\n\ndef get_similar_wave(_df):\n    similar_steps = find_similar_steps(_df[\"anglez\"], window_size=17280, threshold=1.0)\n\n    similar_wave = np.zeros(len(_df))\n    if len(similar_steps) > 0:\n        for start_step in similar_steps:\n            end_step = start_step + 17280\n\n            similar_wave[start_step:end_step] = 1\n\n    return pl.DataFrame(similar_wave.astype(np.Int8))\n```\n\n# Label\n1. asleep\n    - 0-1 label\n    - BCEWithLogitsLoss\n2. onset and wakeup\n    - gaussian distribution(sigma=12)\n    - BCEWithLogitsLoss(pos_weight=100)\n\n\n# Model\n## Model105\nCV: 0.778, PublicLB: 0.752, PrivateLB: 0.805\n- Feature Extractor\n    - WaveNet(dilation=1,2,...,6) × 3\n    - the outputs are concatenated and reshaped\n- Encoder1\n    - Conv1d(stride=6)\n    - MultiResidualBiGRU × 4\n- Encoder2\n    - Conv1d(stride=2)\n    - MultiResidualBiGRU × 4\n- Decoder\n    - Upsampling with ConvTranspose1d(half overlap)\n        - half overlap: kernel_size=strie*2, stride=stride\n- mlp layer\n\n## Model101\nCV: 0.776, PublicLB: 0.742, PrivateLB: 0.811\n\nFeature Extractor and Decoder are almost the same as Model105\n- Encoder\n    - Conv1d(stride=12)\n    - PositionalEncoding\n    - Transformer × 4\n    - MultiResidualBiGRU × 4\n\n\n# Training\n- 5 StratifiedKFold(y=sum(asleep), groups=series_id)\n- n_epochs = 30\n- batch size = 32\n- CosineAnnealingLR\n- AdamW(lr=8e-4)\n- input feature length = 17280\n- no overlap training\n\n# Inference\n- batch size = 1\n- Feature engineering and inference are done for each series_id to save memory\n- no overlap inference\n    - I tried to use overlap inference, but it didn't work well.\n\n# Post Processing\n- find_peaks(threshold=0.1, size=120)\n    - It was not enough\n    - my teammate youri had more strong post processing\n\n# Worked\n- use **fp32**: PublicLB +0.005\n- [tensorflow-like initialization](https://www.kaggle.com/code/junkoda/pytorch-lstm-with-tensorflow-like-initialization)\n- sharply peaked gaussian label\n- onset and wakeup pos_weight=100\n- RobustScaler\n- predicts a lot of steps\n\n# Not Worked\n- data augmentation(flip, mixup, etc...)\n- mlm pretraining\n- GBDT reranking(only tried binary classification)\n- stacking(GBDT, NN, etc...)\n- hand labeling on train data\n- full sequence training\n    - I tried it with UNet, but it didn't work well.\n- external data\n- other model(ViT, LSTM, etc...)\n- other loss(focal loss, dice loss, etc...)\n\n# Other\n- I created visualization app using Streamlit to check the predictions.\n\n![viz example](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fd44856caebcc2c2a053158e5ee31e5ea%2F2023-12-06%2017.34.11.png?generation=1701851684361984&alt=media)",
      "votes": null
    },
    {
      "id": "2550869",
      "postDate": "12/06/2023 10:48:53",
      "content": "<p>Congratulations on achieving the 9th place, <a href=\"https://www.kaggle.com/shu42\" target=\"_blank\">@shu42</a>! What train loss have you been using  for this?</p>",
      "rawMarkdown": "Congratulations on achieving the 9th place, @shu42! What train loss have you been using  for this?",
      "votes": null
    },
    {
      "id": "2550881",
      "postDate": "12/06/2023 11:11:15",
      "content": "<p>Thank you!<br>\nI tried some of these <a href=\"https://www.kaggle.com/code/bigironsphere/loss-function-library-keras-pytorch\" target=\"_blank\">losses</a>, but torch.nn.BCEWithLogitsLoss was best for me.</p>",
      "rawMarkdown": "Thank you!\nI tried some of these [losses](https://www.kaggle.com/code/bigironsphere/loss-function-library-keras-pytorch), but torch.nn.BCEWithLogitsLoss was best for me.",
      "votes": null
    },
    {
      "id": "2551094",
      "postDate": "12/06/2023 13:57:27",
      "content": "<p>Great piece of code! <a href=\"https://www.kaggle.com/SHU421\" target=\"_blank\">@SHU421</a></p>",
      "rawMarkdown": "Great piece of code! @SHU421",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2550869,
      "author_name": "demko1",
      "author_url": "",
      "post_date": "12/06/2023 10:48:53",
      "content": "<p>Congratulations on achieving the 9th place, <a href=\"https://www.kaggle.com/shu42\" target=\"_blank\">@shu42</a>! What train loss have you been using  for this?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2550881,
          "author_name": "shu421",
          "author_url": "",
          "post_date": "12/06/2023 11:11:15",
          "content": "<p>Thank you!<br>\nI tried some of these <a href=\"https://www.kaggle.com/code/bigironsphere/loss-function-library-keras-pytorch\" target=\"_blank\">losses</a>, but torch.nn.BCEWithLogitsLoss was best for me.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2551094,
      "author_name": "abhasmalguri",
      "author_url": "",
      "post_date": "12/06/2023 13:57:27",
      "content": "<p>Great piece of code! <a href=\"https://www.kaggle.com/SHU421\" target=\"_blank\">@SHU421</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2550725": "Thank you organizer and all participants, I learned a lot from this competition!\nIt was a good opportunity to learn about NN modeling for such a long sequence.\n\n# Material\n- [Training Code](https://github.com/shu421/Kaggle_CMI_9thSolution)\n- [Experiment Log](https://boulder-swamp-0ea.notion.site/Child-Mind-Institute-Detect-Sleep-States-37affba149e04a5498cb0cd9f2bb68ec?pvs=4)\n    - I used Notion to manage my experiments.\n    - Sorry, not much maintained\n\n# Overview\n- 1D CNN + GRU encoder and simple mlp decoder model\n\n![exp105 model](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fc7d2ee03be91818c7a23c655a5c7e4c3%2F2023-12-06%2017.33.16.png?generation=1701851622179529&alt=media)\n\n# Feature\n- 1-4 features are scaled with RobustScaler\n1. anglez\n2. enmo\n3. anglez_diff\n4. enmo_diff\n5. hour_sin\n6. hour_cos\n7. similar_wave\n    - 0-1 variable\n    - calculate the similarity of the patched data with 17280 length\n    - if similarity == 1: similar_wave = 1 else 0\n\n**Calculate similar_wave code**\n```python\ndef cross_correlation(signal, pattern):\n    \"\"\"\n    Compute cross correlation between signal and pattern\n    \"\"\"\n    correlation = np.correlate(signal, pattern, mode=\"valid\")\n    normalized_correlation = correlation / (\n        np.linalg.norm(pattern)\n        * np.sqrt(np.convolve(signal**2, np.ones(len(pattern)), \"valid\"))\n    )\n    return normalized_correlation\n\ndef find_similar_steps(signal, window_size=17280, threshold=1.0):\n    \"\"\"\n    Compare waveforms of each bin and find pairs with high similarity.\n    \"\"\"\n    num_bins = len(signal) // window_size\n    similar_steps = set()\n\n    for i in range(num_bins):\n        for j in range(i + 1, num_bins):\n            bin_i = signal[i * window_size : (i + 1) * window_size]\n            bin_j = signal[j * window_size : (j + 1) * window_size]\n            correlation = max(cross_correlation(bin_i, bin_j))\n            if correlation > threshold:\n                similar_steps.add(i * window_size)\n                similar_steps.add(j * window_size)\n\n    return sorted(list(similar_steps))\n\ndef get_similar_wave(_df):\n    similar_steps = find_similar_steps(_df[\"anglez\"], window_size=17280, threshold=1.0)\n\n    similar_wave = np.zeros(len(_df))\n    if len(similar_steps) > 0:\n        for start_step in similar_steps:\n            end_step = start_step + 17280\n\n            similar_wave[start_step:end_step] = 1\n\n    return pl.DataFrame(similar_wave.astype(np.Int8))\n```\n\n# Label\n1. asleep\n    - 0-1 label\n    - BCEWithLogitsLoss\n2. onset and wakeup\n    - gaussian distribution(sigma=12)\n    - BCEWithLogitsLoss(pos_weight=100)\n\n\n# Model\n## Model105\nCV: 0.778, PublicLB: 0.752, PrivateLB: 0.805\n- Feature Extractor\n    - WaveNet(dilation=1,2,...,6) × 3\n    - the outputs are concatenated and reshaped\n- Encoder1\n    - Conv1d(stride=6)\n    - MultiResidualBiGRU × 4\n- Encoder2\n    - Conv1d(stride=2)\n    - MultiResidualBiGRU × 4\n- Decoder\n    - Upsampling with ConvTranspose1d(half overlap)\n        - half overlap: kernel_size=strie*2, stride=stride\n- mlp layer\n\n## Model101\nCV: 0.776, PublicLB: 0.742, PrivateLB: 0.811\n\nFeature Extractor and Decoder are almost the same as Model105\n- Encoder\n    - Conv1d(stride=12)\n    - PositionalEncoding\n    - Transformer × 4\n    - MultiResidualBiGRU × 4\n\n\n# Training\n- 5 StratifiedKFold(y=sum(asleep), groups=series_id)\n- n_epochs = 30\n- batch size = 32\n- CosineAnnealingLR\n- AdamW(lr=8e-4)\n- input feature length = 17280\n- no overlap training\n\n# Inference\n- batch size = 1\n- Feature engineering and inference are done for each series_id to save memory\n- no overlap inference\n    - I tried to use overlap inference, but it didn't work well.\n\n# Post Processing\n- find_peaks(threshold=0.1, size=120)\n    - It was not enough\n    - my teammate youri had more strong post processing\n\n# Worked\n- use **fp32**: PublicLB +0.005\n- [tensorflow-like initialization](https://www.kaggle.com/code/junkoda/pytorch-lstm-with-tensorflow-like-initialization)\n- sharply peaked gaussian label\n- onset and wakeup pos_weight=100\n- RobustScaler\n- predicts a lot of steps\n\n# Not Worked\n- data augmentation(flip, mixup, etc...)\n- mlm pretraining\n- GBDT reranking(only tried binary classification)\n- stacking(GBDT, NN, etc...)\n- hand labeling on train data\n- full sequence training\n    - I tried it with UNet, but it didn't work well.\n- external data\n- other model(ViT, LSTM, etc...)\n- other loss(focal loss, dice loss, etc...)\n\n# Other\n- I created visualization app using Streamlit to check the predictions.\n\n![viz example](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Fd44856caebcc2c2a053158e5ee31e5ea%2F2023-12-06%2017.34.11.png?generation=1701851684361984&alt=media)",
    "2550869": "Congratulations on achieving the 9th place, @shu42! What train loss have you been using  for this?",
    "2550881": "Thank you!\nI tried some of these [losses](https://www.kaggle.com/code/bigironsphere/loss-function-library-keras-pytorch), but torch.nn.BCEWithLogitsLoss was best for me.",
    "2551094": "Great piece of code! @SHU421"
  },
  "source": "meta"
}