{
  "id": 460143,
  "title": "858th Place Solution: MLP-BiGRU",
  "url": "/competitions/child-mind-institute-detect-sleep-states/writeups/nghi-huynh-858th-place-solution-mlp-bigru",
  "author_name": "",
  "post_date": "2023-12-08T02:07:29.033Z",
  "votes": 7,
  "comment_count": 1,
  "views": 0,
  "content": "<p>First, I'd like to thank the organizers and Kaggle team for this interesting competition. Second, a huge congratulation to all the winners and everyone who participated in this competition. It was a very great competition for me to learn more about time-series approaches. Finally, many thanks to <a href=\"https://www.kaggle.com/werus23\" target=\"_blank\">werus23</a>, <a href=\"https://www.kaggle.com/tolgadincer\" target=\"_blank\">Tolga</a>, <a href=\"https://www.kaggle.com/tubotubo\" target=\"_blank\">213tubo</a>, and <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">Chris Deotte</a> for sharing their valuable knowledge and discussions throughout the competition. It was super helpful for me to understand and develop the final solution for this competition. </p>\n<p><strong>1. Data preparation:</strong></p>\n<ul>\n<li><strong>Data</strong>: I truncated the original data at the end based on the valid events. Then, I added some engineered features based on the public notebooks and codes from <a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data\" target=\"_blank\">[1]</a> <a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py\" target=\"_blank\">[2]</a>. The 18 final features I selected for my training/inference phase were: <br>\n<code>num_features = ['anglez', 'enmo', \n                'anglez_lag_diff', 'anglez_lead_diff','anglez_mean_60','anglez_max_60','anglez_std_60',\n                'enmo_lag_diff','enmo_lead_diff','enmo_mean_60', 'enmo_max_60','enmo_std_60', \n                'hour_sin','hour_cos','month_sin','month_cos','minute_sin','minute_cos'\n               ]</code></li>\n<li><strong>Label</strong>: There were 2 approaches for preparing the labels in this problem: hard labelling vs soft labelling. Hard labelling is created by pre-selecting the number of offset from the events and considering it as a part of the training labels. In contrast, soft labeling is created using Gaussian distribution around the onset/awake positions with different sigma and offset values <a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py\" target=\"_blank\">[3]</a>. This approach has demonstrated much better results compared to hard labelling in this competition. Since I didn't have a good understanding of the soft labelling approach, I shifted my focus on training hard labelled data. A visualization of these two approaches is as follows (with an offset of 180 on each side): <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F8618580d7b03ac5fd064a28762da89f0%2Flabel_encoding.png?generation=1701984345645339&amp;alt=media\" alt=\"\"></li>\n</ul>\n<p><strong>2. Cross-validation:</strong><br>\nI used 269 series ids with group 10-fold. I initially used K-Fold but after reading <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882#2491744\" target=\"_blank\">Chris's discussion</a>, I switched to group K-Fold and it boosted my LB score by ~0.05. However, I only selected fold 4 and 7 for training purposes. <br>\nValid series ids:</p>\n<ul>\n<li><strong>fold_4:</strong> <code>['062dbd4c95e6', '0ef7d94fde99', '154fe824ed87', '280e08693c6d',\n   '31011ade7c0a', '3aceb17ef7bd', '4743bdde25df', '51fdcc8d9fe7',\n   '5f40907ec171', '653622ac8363', '703b5efa9bc1', '7822ee8fe3ec',\n   '87a6cbb7c4ed', '8877a6586606', '89bd631d1769', '91127c2b0e60',\n   '971207c6a525', '9b9cd7b7af8c', 'aed3850f65f0', 'ba8083a2c3b8',\n   'c38707ef76df', 'cca14d1966c1', 'd2d6b9af0553', 'db5e0ee1c0ab',\n   'e69aff66e0cb', 'eef041dd50aa', 'f981a0805fd0']</code></li>\n<li><strong>fold_7:</strong> <code>['04f547b8017d', '0dee4fda51c3', '137b99e936ab', '25e2b3dd9c3b',\n   '2e9ced2c7976', '3664fe9233f9', '3c336d6ba566', '449766346eb1',\n   '5c088d7e916c', '601559e1777d', '6d6b9d22d48a', '752900afe3a6',\n   '83fa182bec3a', '8f6f15b9f598', '99b829cbad2d', 'a2b0a64ec9cf',\n   'a88088855de5', 'a9e5f5314bcb', 'b750c8c1556c', 'c107b5789660',\n   'd0f613c700f7', 'd93b0c7de16b', 'dff367373725', 'e0686434d029',\n   'e34b496b84ce', 'ece2561f07e9', 'f7eb179216c2']</code></li>\n</ul>\n<p><strong>3. Model architecture:</strong><br>\nThe model architecture was adapted and modified based on this MLP-Residual BiGRU model <a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook\" target=\"_blank\">[4]</a><a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\" target=\"_blank\">[5]</a>. <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F4326451a4c802ae15d99fb35f8e039de%2FChild_mind_model_architecture.png?generation=1701992082562879&amp;alt=media\" alt=\"\"><br>\n<strong>4. Training:</strong></p>\n<ul>\n<li><p>Optimizer: Adam</p>\n<ul>\n<li>LR=1.0e-3</li></ul></li>\n<li><p>Scheduler: CosineAnnealing</p>\n<ul>\n<li>T_max=10</li></ul></li>\n<li><p>Epochs=10, batch_size=3</p></li>\n<li><p>Loss function: <code>BCEWithLogitsLoss()</code></p></li>\n<li><p>Evaluation metrics: <code>Event Detection AP</code>, <code>precision_score(average='macro')</code><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2Ff326b99e48fb43ea867cad1343ff4705%2Floss.png?generation=1702001203166923&amp;alt=media\" alt=\"\"><br>\n<strong>5. Inference:</strong></p></li>\n<li><p>Used the same preprocessing steps as training phase, except there was no truncating in the test series.</p></li>\n<li><p>Inferred with batch size = 1, and input_size=(17280,18)</p></li>\n<li><p>Filtered out insignificant predictions</p></li>\n<li><p>Added as many onset/awake positions as possible within the input time frame of 17280 (24 hours)</p></li>\n<li><p>Selected only middle point of onset/awake transition periods<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F2da3fe65d4c375da4d9282b66fa93c2c%2Fmodel_0.png?generation=1701998529729107&amp;alt=media\" alt=\"\"><br>\n<strong>6. Results:</strong></p>\n<table>\n<thead>\n<tr>\n<th>Fold</th>\n<th>Model</th>\n<th>Hidden_size</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>4</td>\n<td>model_0</td>\n<td>128</td>\n<td>0.743</td>\n<td>0.681</td>\n<td>0.706</td>\n</tr>\n<tr>\n<td>4</td>\n<td>model_1</td>\n<td>224</td>\n<td>0.737</td>\n<td>0.675</td>\n<td>0.716</td>\n</tr>\n<tr>\n<td>7</td>\n<td>model_2</td>\n<td>128</td>\n<td>0.748</td>\n<td>0.680</td>\n<td>0.71</td>\n</tr>\n<tr>\n<td>7</td>\n<td>model_3</td>\n<td>224</td>\n<td>0.745</td>\n<td>0.677</td>\n<td>0.711</td>\n</tr>\n</tbody>\n</table></li>\n<li><p>Ensemble <strong>public LB</strong>:  0.25*(model_0+model_1+mode_2+model_3) = <strong>0.687</strong></p></li>\n<li><p>Ensemble <strong>private LB</strong>: 0.25*(model_0+model_1+mode_2+model_3) = <strong>0.716</strong></p></li>\n</ul>\n<p><strong>7. Conclusion:</strong><br>\nThings I tried but unsuccessful 🫠:</p>\n<ul>\n<li>WaveNet</li>\n<li>UNet</li>\n<li>Spectrogram-UNet</li>\n<li>Transformer</li>\n<li>Soft-labelling</li>\n</ul>\n<p><strong>References:</strong><br>\n<a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data\" target=\"_blank\"><em>[1] Sleep critical point prepare data-werus23</em></a> </p>\n<p><a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py\" target=\"_blank\"><em>[2] Prepare data-tubo213</em></a></p>\n<p><a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py\" target=\"_blank\"><em>[3] Common utils funtions-tubo213</em></a></p>\n<p><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook\" target=\"_blank\"><em>[4] Sleep critical point train-werus23</em></a></p>\n<p><a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\" target=\"_blank\"><em>[5] 4th Place Solution: a MultiLayer Bidirectional GRU with Residual Connections-zinxira</em></a></p>\n<p>Thanks for reading! 😀</p>",
  "messages": [
    {
      "id": "2553084",
      "postDate": "12/08/2023 02:07:03",
      "content": "<p>First, I'd like to thank the organizers and Kaggle team for this interesting competition. Second, a huge congratulation to all the winners and everyone who participated in this competition. It was a very great competition for me to learn more about time-series approaches. Finally, many thanks to <a href=\"https://www.kaggle.com/werus23\" target=\"_blank\">werus23</a>, <a href=\"https://www.kaggle.com/tolgadincer\" target=\"_blank\">Tolga</a>, <a href=\"https://www.kaggle.com/tubotubo\" target=\"_blank\">213tubo</a>, and <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">Chris Deotte</a> for sharing their valuable knowledge and discussions throughout the competition. It was super helpful for me to understand and develop the final solution for this competition. </p>\n<p><strong>1. Data preparation:</strong></p>\n<ul>\n<li><strong>Data</strong>: I truncated the original data at the end based on the valid events. Then, I added some engineered features based on the public notebooks and codes from <a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data\" target=\"_blank\">[1]</a> <a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py\" target=\"_blank\">[2]</a>. The 18 final features I selected for my training/inference phase were: <br>\n<code>num_features = ['anglez', 'enmo', \n                'anglez_lag_diff', 'anglez_lead_diff','anglez_mean_60','anglez_max_60','anglez_std_60',\n                'enmo_lag_diff','enmo_lead_diff','enmo_mean_60', 'enmo_max_60','enmo_std_60', \n                'hour_sin','hour_cos','month_sin','month_cos','minute_sin','minute_cos'\n               ]</code></li>\n<li><strong>Label</strong>: There were 2 approaches for preparing the labels in this problem: hard labelling vs soft labelling. Hard labelling is created by pre-selecting the number of offset from the events and considering it as a part of the training labels. In contrast, soft labeling is created using Gaussian distribution around the onset/awake positions with different sigma and offset values <a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py\" target=\"_blank\">[3]</a>. This approach has demonstrated much better results compared to hard labelling in this competition. Since I didn't have a good understanding of the soft labelling approach, I shifted my focus on training hard labelled data. A visualization of these two approaches is as follows (with an offset of 180 on each side): <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F8618580d7b03ac5fd064a28762da89f0%2Flabel_encoding.png?generation=1701984345645339&amp;alt=media\" alt=\"\"></li>\n</ul>\n<p><strong>2. Cross-validation:</strong><br>\nI used 269 series ids with group 10-fold. I initially used K-Fold but after reading <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882#2491744\" target=\"_blank\">Chris's discussion</a>, I switched to group K-Fold and it boosted my LB score by ~0.05. However, I only selected fold 4 and 7 for training purposes. <br>\nValid series ids:</p>\n<ul>\n<li><strong>fold_4:</strong> <code>['062dbd4c95e6', '0ef7d94fde99', '154fe824ed87', '280e08693c6d',\n   '31011ade7c0a', '3aceb17ef7bd', '4743bdde25df', '51fdcc8d9fe7',\n   '5f40907ec171', '653622ac8363', '703b5efa9bc1', '7822ee8fe3ec',\n   '87a6cbb7c4ed', '8877a6586606', '89bd631d1769', '91127c2b0e60',\n   '971207c6a525', '9b9cd7b7af8c', 'aed3850f65f0', 'ba8083a2c3b8',\n   'c38707ef76df', 'cca14d1966c1', 'd2d6b9af0553', 'db5e0ee1c0ab',\n   'e69aff66e0cb', 'eef041dd50aa', 'f981a0805fd0']</code></li>\n<li><strong>fold_7:</strong> <code>['04f547b8017d', '0dee4fda51c3', '137b99e936ab', '25e2b3dd9c3b',\n   '2e9ced2c7976', '3664fe9233f9', '3c336d6ba566', '449766346eb1',\n   '5c088d7e916c', '601559e1777d', '6d6b9d22d48a', '752900afe3a6',\n   '83fa182bec3a', '8f6f15b9f598', '99b829cbad2d', 'a2b0a64ec9cf',\n   'a88088855de5', 'a9e5f5314bcb', 'b750c8c1556c', 'c107b5789660',\n   'd0f613c700f7', 'd93b0c7de16b', 'dff367373725', 'e0686434d029',\n   'e34b496b84ce', 'ece2561f07e9', 'f7eb179216c2']</code></li>\n</ul>\n<p><strong>3. Model architecture:</strong><br>\nThe model architecture was adapted and modified based on this MLP-Residual BiGRU model <a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook\" target=\"_blank\">[4]</a><a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\" target=\"_blank\">[5]</a>. <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F4326451a4c802ae15d99fb35f8e039de%2FChild_mind_model_architecture.png?generation=1701992082562879&amp;alt=media\" alt=\"\"><br>\n<strong>4. Training:</strong></p>\n<ul>\n<li><p>Optimizer: Adam</p>\n<ul>\n<li>LR=1.0e-3</li></ul></li>\n<li><p>Scheduler: CosineAnnealing</p>\n<ul>\n<li>T_max=10</li></ul></li>\n<li><p>Epochs=10, batch_size=3</p></li>\n<li><p>Loss function: <code>BCEWithLogitsLoss()</code></p></li>\n<li><p>Evaluation metrics: <code>Event Detection AP</code>, <code>precision_score(average='macro')</code><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2Ff326b99e48fb43ea867cad1343ff4705%2Floss.png?generation=1702001203166923&amp;alt=media\" alt=\"\"><br>\n<strong>5. Inference:</strong></p></li>\n<li><p>Used the same preprocessing steps as training phase, except there was no truncating in the test series.</p></li>\n<li><p>Inferred with batch size = 1, and input_size=(17280,18)</p></li>\n<li><p>Filtered out insignificant predictions</p></li>\n<li><p>Added as many onset/awake positions as possible within the input time frame of 17280 (24 hours)</p></li>\n<li><p>Selected only middle point of onset/awake transition periods<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F2da3fe65d4c375da4d9282b66fa93c2c%2Fmodel_0.png?generation=1701998529729107&amp;alt=media\" alt=\"\"><br>\n<strong>6. Results:</strong></p>\n<table>\n<thead>\n<tr>\n<th>Fold</th>\n<th>Model</th>\n<th>Hidden_size</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>4</td>\n<td>model_0</td>\n<td>128</td>\n<td>0.743</td>\n<td>0.681</td>\n<td>0.706</td>\n</tr>\n<tr>\n<td>4</td>\n<td>model_1</td>\n<td>224</td>\n<td>0.737</td>\n<td>0.675</td>\n<td>0.716</td>\n</tr>\n<tr>\n<td>7</td>\n<td>model_2</td>\n<td>128</td>\n<td>0.748</td>\n<td>0.680</td>\n<td>0.71</td>\n</tr>\n<tr>\n<td>7</td>\n<td>model_3</td>\n<td>224</td>\n<td>0.745</td>\n<td>0.677</td>\n<td>0.711</td>\n</tr>\n</tbody>\n</table></li>\n<li><p>Ensemble <strong>public LB</strong>:  0.25*(model_0+model_1+mode_2+model_3) = <strong>0.687</strong></p></li>\n<li><p>Ensemble <strong>private LB</strong>: 0.25*(model_0+model_1+mode_2+model_3) = <strong>0.716</strong></p></li>\n</ul>\n<p><strong>7. Conclusion:</strong><br>\nThings I tried but unsuccessful 🫠:</p>\n<ul>\n<li>WaveNet</li>\n<li>UNet</li>\n<li>Spectrogram-UNet</li>\n<li>Transformer</li>\n<li>Soft-labelling</li>\n</ul>\n<p><strong>References:</strong><br>\n<a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data\" target=\"_blank\"><em>[1] Sleep critical point prepare data-werus23</em></a> </p>\n<p><a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py\" target=\"_blank\"><em>[2] Prepare data-tubo213</em></a></p>\n<p><a href=\"https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py\" target=\"_blank\"><em>[3] Common utils funtions-tubo213</em></a></p>\n<p><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook\" target=\"_blank\"><em>[4] Sleep critical point train-werus23</em></a></p>\n<p><a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\" target=\"_blank\"><em>[5] 4th Place Solution: a MultiLayer Bidirectional GRU with Residual Connections-zinxira</em></a></p>\n<p>Thanks for reading! 😀</p>",
      "rawMarkdown": "First, I'd like to thank the organizers and Kaggle team for this interesting competition. Second, a huge congratulation to all the winners and everyone who participated in this competition. It was a very great competition for me to learn more about time-series approaches. Finally, many thanks to [werus23](https://www.kaggle.com/werus23), [Tolga](https://www.kaggle.com/tolgadincer), [213tubo](https://www.kaggle.com/tubotubo), and [Chris Deotte](https://www.kaggle.com/cdeotte) for sharing their valuable knowledge and discussions throughout the competition. It was super helpful for me to understand and develop the final solution for this competition. \n\n**1. Data preparation:**\n* **Data**: I truncated the original data at the end based on the valid events. Then, I added some engineered features based on the public notebooks and codes from [[1]](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data) [[2]](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py). The 18 final features I selected for my training/inference phase were: \n`num_features = ['anglez', 'enmo', \n                    'anglez_lag_diff', 'anglez_lead_diff','anglez_mean_60','anglez_max_60','anglez_std_60',\n                    'enmo_lag_diff','enmo_lead_diff','enmo_mean_60', 'enmo_max_60','enmo_std_60', \n                    'hour_sin','hour_cos','month_sin','month_cos','minute_sin','minute_cos'\n                   ]`\n* **Label**: There were 2 approaches for preparing the labels in this problem: hard labelling vs soft labelling. Hard labelling is created by pre-selecting the number of offset from the events and considering it as a part of the training labels. In contrast, soft labeling is created using Gaussian distribution around the onset/awake positions with different sigma and offset values [[3]](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py). This approach has demonstrated much better results compared to hard labelling in this competition. Since I didn't have a good understanding of the soft labelling approach, I shifted my focus on training hard labelled data. A visualization of these two approaches is as follows (with an offset of 180 on each side): \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F8618580d7b03ac5fd064a28762da89f0%2Flabel_encoding.png?generation=1701984345645339&alt=media)\n\n**2. Cross-validation:**\nI used 269 series ids with group 10-fold. I initially used K-Fold but after reading [Chris's discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882#2491744), I switched to group K-Fold and it boosted my LB score by ~0.05. However, I only selected fold 4 and 7 for training purposes. \nValid series ids:\n* **fold_4:** `['062dbd4c95e6', '0ef7d94fde99', '154fe824ed87', '280e08693c6d',\n       '31011ade7c0a', '3aceb17ef7bd', '4743bdde25df', '51fdcc8d9fe7',\n       '5f40907ec171', '653622ac8363', '703b5efa9bc1', '7822ee8fe3ec',\n       '87a6cbb7c4ed', '8877a6586606', '89bd631d1769', '91127c2b0e60',\n       '971207c6a525', '9b9cd7b7af8c', 'aed3850f65f0', 'ba8083a2c3b8',\n       'c38707ef76df', 'cca14d1966c1', 'd2d6b9af0553', 'db5e0ee1c0ab',\n       'e69aff66e0cb', 'eef041dd50aa', 'f981a0805fd0']`\n* **fold_7:** `['04f547b8017d', '0dee4fda51c3', '137b99e936ab', '25e2b3dd9c3b',\n       '2e9ced2c7976', '3664fe9233f9', '3c336d6ba566', '449766346eb1',\n       '5c088d7e916c', '601559e1777d', '6d6b9d22d48a', '752900afe3a6',\n       '83fa182bec3a', '8f6f15b9f598', '99b829cbad2d', 'a2b0a64ec9cf',\n       'a88088855de5', 'a9e5f5314bcb', 'b750c8c1556c', 'c107b5789660',\n       'd0f613c700f7', 'd93b0c7de16b', 'dff367373725', 'e0686434d029',\n       'e34b496b84ce', 'ece2561f07e9', 'f7eb179216c2']`\n\n**3. Model architecture:**\nThe model architecture was adapted and modified based on this MLP-Residual BiGRU model [[4]](https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook)[[5]](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410). \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F4326451a4c802ae15d99fb35f8e039de%2FChild_mind_model_architecture.png?generation=1701992082562879&alt=media)\n**4. Training:**\n* Optimizer: Adam\n    -  LR=1.0e-3\n* Scheduler: CosineAnnealing\n    - T_max=10\n* Epochs=10, batch_size=3\n* Loss function: `BCEWithLogitsLoss()`\n* Evaluation metrics: `Event Detection AP`, `precision_score(average='macro')`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2Ff326b99e48fb43ea867cad1343ff4705%2Floss.png?generation=1702001203166923&alt=media)\n**5. Inference:**\n* Used the same preprocessing steps as training phase, except there was no truncating in the test series.\n* Inferred with batch size = 1, and input_size=(17280,18)\n* Filtered out insignificant predictions\n* Added as many onset/awake positions as possible within the input time frame of 17280 (24 hours)\n* Selected only middle point of onset/awake transition periods\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F2da3fe65d4c375da4d9282b66fa93c2c%2Fmodel_0.png?generation=1701998529729107&alt=media)\n**6. Results:**\n| Fold | Model  | Hidden_size | CV | Public LB | Private LB |\n| --- | --- | --- | --- | --- | --- |\n| 4 | model_0 | 128 | 0.743 | 0.681 | 0.706 |\n| 4 | model_1 | 224| 0.737|  0.675 | 0.716|\n| 7 | model_2 | 128 | 0.748| 0.680 | 0.71|\n| 7 | model_3 | 224| 0.745| 0.677| 0.711|\n\n* Ensemble **public LB**:  0.25*(model_0+model_1+mode_2+model_3) = **0.687**\n* Ensemble **private LB**: 0.25*(model_0+model_1+mode_2+model_3) = **0.716**\n\n**7. Conclusion:**\nThings I tried but unsuccessful 🫠:\n* WaveNet\n* UNet\n* Spectrogram-UNet\n* Transformer\n* Soft-labelling\n\n**References:**\n[*[1] Sleep critical point prepare data-werus23*](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data) \n\n[*[2] Prepare data-tubo213*](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py)\n\n[*[3] Common utils funtions-tubo213*](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py)\n\n[*[4] Sleep critical point train-werus23*](https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook)\n\n[*[5] 4th Place Solution: a MultiLayer Bidirectional GRU with Residual Connections-zinxira*](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410)\n\n\nThanks for reading! 😀",
      "votes": null
    },
    {
      "id": "2554023",
      "postDate": "12/08/2023 17:54:01",
      "content": "<p>I'm so glad to read yours MLP-BiGRU Solution N. Huynh.</p>",
      "rawMarkdown": "I'm so glad to read yours MLP-BiGRU Solution N. Huynh.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2554023,
      "author_name": "mpwolke",
      "author_url": "",
      "post_date": "12/08/2023 17:54:01",
      "content": "<p>I'm so glad to read yours MLP-BiGRU Solution N. Huynh.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2553084": "First, I'd like to thank the organizers and Kaggle team for this interesting competition. Second, a huge congratulation to all the winners and everyone who participated in this competition. It was a very great competition for me to learn more about time-series approaches. Finally, many thanks to [werus23](https://www.kaggle.com/werus23), [Tolga](https://www.kaggle.com/tolgadincer), [213tubo](https://www.kaggle.com/tubotubo), and [Chris Deotte](https://www.kaggle.com/cdeotte) for sharing their valuable knowledge and discussions throughout the competition. It was super helpful for me to understand and develop the final solution for this competition. \n\n**1. Data preparation:**\n* **Data**: I truncated the original data at the end based on the valid events. Then, I added some engineered features based on the public notebooks and codes from [[1]](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data) [[2]](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py). The 18 final features I selected for my training/inference phase were: \n`num_features = ['anglez', 'enmo', \n                    'anglez_lag_diff', 'anglez_lead_diff','anglez_mean_60','anglez_max_60','anglez_std_60',\n                    'enmo_lag_diff','enmo_lead_diff','enmo_mean_60', 'enmo_max_60','enmo_std_60', \n                    'hour_sin','hour_cos','month_sin','month_cos','minute_sin','minute_cos'\n                   ]`\n* **Label**: There were 2 approaches for preparing the labels in this problem: hard labelling vs soft labelling. Hard labelling is created by pre-selecting the number of offset from the events and considering it as a part of the training labels. In contrast, soft labeling is created using Gaussian distribution around the onset/awake positions with different sigma and offset values [[3]](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py). This approach has demonstrated much better results compared to hard labelling in this competition. Since I didn't have a good understanding of the soft labelling approach, I shifted my focus on training hard labelled data. A visualization of these two approaches is as follows (with an offset of 180 on each side): \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F8618580d7b03ac5fd064a28762da89f0%2Flabel_encoding.png?generation=1701984345645339&alt=media)\n\n**2. Cross-validation:**\nI used 269 series ids with group 10-fold. I initially used K-Fold but after reading [Chris's discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882#2491744), I switched to group K-Fold and it boosted my LB score by ~0.05. However, I only selected fold 4 and 7 for training purposes. \nValid series ids:\n* **fold_4:** `['062dbd4c95e6', '0ef7d94fde99', '154fe824ed87', '280e08693c6d',\n       '31011ade7c0a', '3aceb17ef7bd', '4743bdde25df', '51fdcc8d9fe7',\n       '5f40907ec171', '653622ac8363', '703b5efa9bc1', '7822ee8fe3ec',\n       '87a6cbb7c4ed', '8877a6586606', '89bd631d1769', '91127c2b0e60',\n       '971207c6a525', '9b9cd7b7af8c', 'aed3850f65f0', 'ba8083a2c3b8',\n       'c38707ef76df', 'cca14d1966c1', 'd2d6b9af0553', 'db5e0ee1c0ab',\n       'e69aff66e0cb', 'eef041dd50aa', 'f981a0805fd0']`\n* **fold_7:** `['04f547b8017d', '0dee4fda51c3', '137b99e936ab', '25e2b3dd9c3b',\n       '2e9ced2c7976', '3664fe9233f9', '3c336d6ba566', '449766346eb1',\n       '5c088d7e916c', '601559e1777d', '6d6b9d22d48a', '752900afe3a6',\n       '83fa182bec3a', '8f6f15b9f598', '99b829cbad2d', 'a2b0a64ec9cf',\n       'a88088855de5', 'a9e5f5314bcb', 'b750c8c1556c', 'c107b5789660',\n       'd0f613c700f7', 'd93b0c7de16b', 'dff367373725', 'e0686434d029',\n       'e34b496b84ce', 'ece2561f07e9', 'f7eb179216c2']`\n\n**3. Model architecture:**\nThe model architecture was adapted and modified based on this MLP-Residual BiGRU model [[4]](https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook)[[5]](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410). \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F4326451a4c802ae15d99fb35f8e039de%2FChild_mind_model_architecture.png?generation=1701992082562879&alt=media)\n**4. Training:**\n* Optimizer: Adam\n    -  LR=1.0e-3\n* Scheduler: CosineAnnealing\n    - T_max=10\n* Epochs=10, batch_size=3\n* Loss function: `BCEWithLogitsLoss()`\n* Evaluation metrics: `Event Detection AP`, `precision_score(average='macro')`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2Ff326b99e48fb43ea867cad1343ff4705%2Floss.png?generation=1702001203166923&alt=media)\n**5. Inference:**\n* Used the same preprocessing steps as training phase, except there was no truncating in the test series.\n* Inferred with batch size = 1, and input_size=(17280,18)\n* Filtered out insignificant predictions\n* Added as many onset/awake positions as possible within the input time frame of 17280 (24 hours)\n* Selected only middle point of onset/awake transition periods\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F2da3fe65d4c375da4d9282b66fa93c2c%2Fmodel_0.png?generation=1701998529729107&alt=media)\n**6. Results:**\n| Fold | Model  | Hidden_size | CV | Public LB | Private LB |\n| --- | --- | --- | --- | --- | --- |\n| 4 | model_0 | 128 | 0.743 | 0.681 | 0.706 |\n| 4 | model_1 | 224| 0.737|  0.675 | 0.716|\n| 7 | model_2 | 128 | 0.748| 0.680 | 0.71|\n| 7 | model_3 | 224| 0.745| 0.677| 0.711|\n\n* Ensemble **public LB**:  0.25*(model_0+model_1+mode_2+model_3) = **0.687**\n* Ensemble **private LB**: 0.25*(model_0+model_1+mode_2+model_3) = **0.716**\n\n**7. Conclusion:**\nThings I tried but unsuccessful 🫠:\n* WaveNet\n* UNet\n* Spectrogram-UNet\n* Transformer\n* Soft-labelling\n\n**References:**\n[*[1] Sleep critical point prepare data-werus23*](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data) \n\n[*[2] Prepare data-tubo213*](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/run/prepare_data.py)\n\n[*[3] Common utils funtions-tubo213*](https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/main/src/utils/common.py)\n\n[*[4] Sleep critical point train-werus23*](https://www.kaggle.com/code/werus23/sleep-critical-point-train/notebook)\n\n[*[5] 4th Place Solution: a MultiLayer Bidirectional GRU with Residual Connections-zinxira*](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410)\n\n\nThanks for reading! 😀",
    "2554023": "I'm so glad to read yours MLP-BiGRU Solution N. Huynh."
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 53666,
  "competition_slug": "child-mind-institute-detect-sleep-states",
  "team_id": 10873076,
  "solution_url": "/competitions/child-mind-institute-detect-sleep-states/writeups/nghi-huynh-858th-place-solution-mlp-bigru"
}