{
  "id": 460276,
  "title": "9th Place Solution - tomo’s Part",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/460276",
  "author_name": "tomo20180402",
  "post_date": "2023-12-08T14:12:04.127000",
  "votes": 23,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Thanks to the organizers and team members!<br>\nI am sharing the part of my model that was integrated into the team's ensemble and post-processing.<br>\nResult: using only peak detection for post-processing, <strong>publicLB=0.769, privateLB=0.810</strong></p>\n<h1>Feature</h1>\n<p>I switched the features depending on the model.<br>\nThe following Pattern 1 and Pattern 2 were created with the intention of representing the changes in sensor fluctuations before and after each step.</p>\n<ul>\n<li>Common<ul>\n<li>anglez, anglez_diff_lag/lead<ul>\n<li>lag=-1, lead=1</li></ul></li>\n<li>enmo, enmo_diff_lag/lead</li>\n<li>hour_sin/cos</li>\n<li>anglez_cossim_max/mean<ul>\n<li>explain it later.</li></ul></li></ul></li>\n<li>Pattern 1<ul>\n<li>anglez_rstd_w360_b/f/diff<ul>\n<li>rstd=rolling_std, w=window_size, b=backward, f=forward, diff=b-f</li>\n<li>b: <code>pl.col('anglez').rolling_std(360, center=False, min_periods=1)</code></li></ul></li>\n<li>enmo_rmean_w360_b/f/diff<ul>\n<li>rmean=rolling_mean</li></ul></li></ul></li>\n<li>Pattern 2<ul>\n<li>anglez_diff_lag_abs_rmean_w360_b/f/diff<ul>\n<li>b: <code>pl.col('anglez_diff_lag').abs().rolling_mean(360, center=False, min_periods=1)</code></li></ul></li>\n<li>enmo_diff_lag_abs_rmean_w360_b/f/diff</li></ul></li>\n</ul>\n<h3>Cosine Similarity Feature - anglez_cossim_max/mean</h3>\n<p>In order to capture the periodicity of anglez, I calculated the cosine similarity by cutting it into 10,000steps while shifting it, and the max and mean of the similarity were used as the features.</p>\n<pre><code> () -&gt; pl.DataFrame:\n    anglez = series_pl[].to_numpy()\n\n    \n    anglez_matrix_np = np.lib.stride_tricks.sliding_window_view(anglez, width)[::shift]\n    cossims = cosine_similarity(anglez_matrix_np)\n    np.fill_diagonal(cossims, )\n    cossims = cossims.(axis=)\n\n    \n    cossims_fix_len = np.repeat(cossims, shift)\n    nan_len = anglez.shape[] - cossims_fix_len.shape[]\n    cossims_fix_len = np.hstack([cossims_fix_len, np.full((nan_len,), np.nan)])\n\n    \n    num_elements = (cossims_fix_len)\n    new_rows = [\n        np.concatenate([np.full((width * i / ), np.nan), \n        cossims_fix_len[:-(width * i / )],])  i  (, )]\n    cossims_fix_len_aug = np.vstack([cossims_fix_len, new_rows])\n    cossims_fix_len_max = np.nanmax(cossims_fix_len_aug, axis=)\n    cossims_fix_len_mean = np.nanmean(cossims_fix_len_aug, axis=)\n\n    \n    series_pl = series_pl.with_columns(\n        pl.lit(cossims_fix_len_max).alias(),\n        pl.lit(cossims_fix_len_mean).alias(),\n    )\n\n     series_pl\n</code></pre>\n<h1>Model</h1>\n<p>I trained two types of architectures for each of the two types of features.</p>\n<ul>\n<li>Target<ul>\n<li>sleep/onset/awake, gaussian label(sigma=10)</li></ul></li>\n<li>Loss<ul>\n<li>BCEWithLogitsLoss</li></ul></li>\n<li>Model type 1<ul>\n<li>1DCNN + Residual-Transformer + GRU</li>\n<li>downsample=2, 4</li>\n<li>Deeper GRU was powerful in my model. (num_layers=10)</li>\n<li>Extend the sequence length during the inference phase: train=5760, inference=17280</li>\n<li>inference w/o overlap</li></ul></li>\n<li>Model type 2<ul>\n<li>Customized <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940\" target=\"_blank\">213tubo's UNet(2D) model</a> (add Residual-Transformer)</li>\n<li>downsample=2, 4</li>\n<li>sequence length: 5760</li>\n<li>inference w/ overlap, shift=720</li></ul></li>\n</ul>\n<h1>Score</h1>\n<p>Split: StratifiedGroupKFold(k=5, y=sleep, group=series_id)<br>\nUsing only peak detection for post-processing:</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Model type 1</td>\n<td>0.795</td>\n<td>0.759</td>\n<td>0.802</td>\n</tr>\n<tr>\n<td>Model type 2</td>\n<td>0.792</td>\n<td>0.760</td>\n<td>0.801</td>\n</tr>\n<tr>\n<td>Model type 1+2</td>\n<td>0.804</td>\n<td>0.769</td>\n<td>0.810</td>\n</tr>\n</tbody>\n</table>\n<h1>Not Worked</h1>\n<ul>\n<li>augmentation<ul>\n<li>Gaussian Noise, Time-directed expansion and contraction, etc.</li></ul></li>\n<li>stacking<ul>\n<li>While utilizing the predicted values from the 1st stage, add features for each night's data and optimize using LightGBM.</li>\n<li>Simple stacking by NN did not work either.</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 2553792,
      "postDate": "2023-12-08T14:12:04.127Z",
      "content": "<p>Thanks to the organizers and team members!<br>\nI am sharing the part of my model that was integrated into the team's ensemble and post-processing.<br>\nResult: using only peak detection for post-processing, <strong>publicLB=0.769, privateLB=0.810</strong></p>\n<h1>Feature</h1>\n<p>I switched the features depending on the model.<br>\nThe following Pattern 1 and Pattern 2 were created with the intention of representing the changes in sensor fluctuations before and after each step.</p>\n<ul>\n<li>Common<ul>\n<li>anglez, anglez_diff_lag/lead<ul>\n<li>lag=-1, lead=1</li></ul></li>\n<li>enmo, enmo_diff_lag/lead</li>\n<li>hour_sin/cos</li>\n<li>anglez_cossim_max/mean<ul>\n<li>explain it later.</li></ul></li></ul></li>\n<li>Pattern 1<ul>\n<li>anglez_rstd_w360_b/f/diff<ul>\n<li>rstd=rolling_std, w=window_size, b=backward, f=forward, diff=b-f</li>\n<li>b: <code>pl.col('anglez').rolling_std(360, center=False, min_periods=1)</code></li></ul></li>\n<li>enmo_rmean_w360_b/f/diff<ul>\n<li>rmean=rolling_mean</li></ul></li></ul></li>\n<li>Pattern 2<ul>\n<li>anglez_diff_lag_abs_rmean_w360_b/f/diff<ul>\n<li>b: <code>pl.col('anglez_diff_lag').abs().rolling_mean(360, center=False, min_periods=1)</code></li></ul></li>\n<li>enmo_diff_lag_abs_rmean_w360_b/f/diff</li></ul></li>\n</ul>\n<h3>Cosine Similarity Feature - anglez_cossim_max/mean</h3>\n<p>In order to capture the periodicity of anglez, I calculated the cosine similarity by cutting it into 10,000steps while shifting it, and the max and mean of the similarity were used as the features.</p>\n<pre><code> () -&gt; pl.DataFrame:\n    anglez = series_pl[].to_numpy()\n\n    \n    anglez_matrix_np = np.lib.stride_tricks.sliding_window_view(anglez, width)[::shift]\n    cossims = cosine_similarity(anglez_matrix_np)\n    np.fill_diagonal(cossims, )\n    cossims = cossims.(axis=)\n\n    \n    cossims_fix_len = np.repeat(cossims, shift)\n    nan_len = anglez.shape[] - cossims_fix_len.shape[]\n    cossims_fix_len = np.hstack([cossims_fix_len, np.full((nan_len,), np.nan)])\n\n    \n    num_elements = (cossims_fix_len)\n    new_rows = [\n        np.concatenate([np.full((width * i / ), np.nan), \n        cossims_fix_len[:-(width * i / )],])  i  (, )]\n    cossims_fix_len_aug = np.vstack([cossims_fix_len, new_rows])\n    cossims_fix_len_max = np.nanmax(cossims_fix_len_aug, axis=)\n    cossims_fix_len_mean = np.nanmean(cossims_fix_len_aug, axis=)\n\n    \n    series_pl = series_pl.with_columns(\n        pl.lit(cossims_fix_len_max).alias(),\n        pl.lit(cossims_fix_len_mean).alias(),\n    )\n\n     series_pl\n</code></pre>\n<h1>Model</h1>\n<p>I trained two types of architectures for each of the two types of features.</p>\n<ul>\n<li>Target<ul>\n<li>sleep/onset/awake, gaussian label(sigma=10)</li></ul></li>\n<li>Loss<ul>\n<li>BCEWithLogitsLoss</li></ul></li>\n<li>Model type 1<ul>\n<li>1DCNN + Residual-Transformer + GRU</li>\n<li>downsample=2, 4</li>\n<li>Deeper GRU was powerful in my model. (num_layers=10)</li>\n<li>Extend the sequence length during the inference phase: train=5760, inference=17280</li>\n<li>inference w/o overlap</li></ul></li>\n<li>Model type 2<ul>\n<li>Customized <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940\" target=\"_blank\">213tubo's UNet(2D) model</a> (add Residual-Transformer)</li>\n<li>downsample=2, 4</li>\n<li>sequence length: 5760</li>\n<li>inference w/ overlap, shift=720</li></ul></li>\n</ul>\n<h1>Score</h1>\n<p>Split: StratifiedGroupKFold(k=5, y=sleep, group=series_id)<br>\nUsing only peak detection for post-processing:</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Model type 1</td>\n<td>0.795</td>\n<td>0.759</td>\n<td>0.802</td>\n</tr>\n<tr>\n<td>Model type 2</td>\n<td>0.792</td>\n<td>0.760</td>\n<td>0.801</td>\n</tr>\n<tr>\n<td>Model type 1+2</td>\n<td>0.804</td>\n<td>0.769</td>\n<td>0.810</td>\n</tr>\n</tbody>\n</table>\n<h1>Not Worked</h1>\n<ul>\n<li>augmentation<ul>\n<li>Gaussian Noise, Time-directed expansion and contraction, etc.</li></ul></li>\n<li>stacking<ul>\n<li>While utilizing the predicted values from the 1st stage, add features for each night's data and optimize using LightGBM.</li>\n<li>Simple stacking by NN did not work either.</li></ul></li>\n</ul>",
      "rawMarkdown": "Thanks to the organizers and team members!\nI am sharing the part of my model that was integrated into the team's ensemble and post-processing.\nResult: using only peak detection for post-processing, **publicLB=0.769, privateLB=0.810**\n\n# Feature\nI switched the features depending on the model.\nThe following Pattern 1 and Pattern 2 were created with the intention of representing the changes in sensor fluctuations before and after each step.\n* Common\n    * anglez, anglez_diff_lag/lead\n        * lag=-1, lead=1\n    * enmo, enmo_diff_lag/lead\n    * hour_sin/cos\n    * anglez_cossim_max/mean\n        * explain it later.\n* Pattern 1\n    * anglez_rstd_w360_b/f/diff\n        * rstd=rolling_std, w=window_size, b=backward, f=forward, diff=b-f\n        * b: `pl.col('anglez').rolling_std(360, center=False, min_periods=1)`\n    * enmo_rmean_w360_b/f/diff\n        * rmean=rolling_mean\n* Pattern 2\n    * anglez_diff_lag_abs_rmean_w360_b/f/diff\n        * b: `pl.col('anglez_diff_lag').abs().rolling_mean(360, center=False, min_periods=1)`\n    * enmo_diff_lag_abs_rmean_w360_b/f/diff\n\n### Cosine Similarity Feature - anglez_cossim_max/mean\nIn order to capture the periodicity of anglez, I calculated the cosine similarity by cutting it into 10,000steps while shifting it, and the max and mean of the similarity were used as the features.\n```python\ndef add_anglez_cossim_feature(series_pl: pl.DataFrame, width: int = 10_000, shift: int = 1_000) -> pl.DataFrame:\n    anglez = series_pl[\"anglez\"].to_numpy()\n    \n    # cossim\n    anglez_matrix_np = np.lib.stride_tricks.sliding_window_view(anglez, width)[::shift]\n    cossims = cosine_similarity(anglez_matrix_np)\n    np.fill_diagonal(cossims, 0)\n    cossims = cossims.max(axis=0)\n    \n    # fix length\n    cossims_fix_len = np.repeat(cossims, shift)\n    nan_len = anglez.shape[0] - cossims_fix_len.shape[0]\n    cossims_fix_len = np.hstack([cossims_fix_len, np.full((nan_len,), np.nan)])\n    \n    # cossim max/mean\n    num_elements = len(cossims_fix_len)\n    new_rows = [\n        np.concatenate([np.full(int(width * i / 100), np.nan), \n        cossims_fix_len[:-int(width * i / 100)],]) for i in range(5, 101)]\n    cossims_fix_len_aug = np.vstack([cossims_fix_len, new_rows])\n    cossims_fix_len_max = np.nanmax(cossims_fix_len_aug, axis=0)\n    cossims_fix_len_mean = np.nanmean(cossims_fix_len_aug, axis=0)\n    \n    # add features\n    series_pl = series_pl.with_columns(\n        pl.lit(cossims_fix_len_max).alias(\"anglez_cossim_max\"),\n        pl.lit(cossims_fix_len_mean).alias(\"anglez_cossim_mean\"),\n    )\n    \n    return series_pl\n```\n\n# Model\nI trained two types of architectures for each of the two types of features.\n\n* Target\n    * sleep/onset/awake, gaussian label(sigma=10)\n* Loss\n    * BCEWithLogitsLoss\n* Model type 1\n    * 1DCNN + Residual-Transformer + GRU\n    * downsample=2, 4\n    * Deeper GRU was powerful in my model. (num_layers=10)\n    * Extend the sequence length during the inference phase: train=5760, inference=17280\n    * inference w/o overlap\n* Model type 2\n    * Customized [213tubo's UNet(2D) model](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) (add Residual-Transformer)\n    * downsample=2, 4\n    * sequence length: 5760\n    * inference w/ overlap, shift=720\n\n# Score\nSplit: StratifiedGroupKFold(k=5, y=sleep, group=series_id)\nUsing only peak detection for post-processing:\n| | CV | Public LB | Private LB |\n| --- | --- | --- | --- |\n| Model type 1 | 0.795 | 0.759 | 0.802 |\n| Model type 2 | 0.792 | 0.760 | 0.801 |\n| Model type 1+2 | 0.804 | 0.769 | 0.810 |\n\n\n# Not Worked\n* augmentation\n    * Gaussian Noise, Time-directed expansion and contraction, etc.\n* stacking\n    * While utilizing the predicted values from the 1st stage, add features for each night's data and optimize using LightGBM.\n    * Simple stacking by NN did not work either.",
      "votes": 23
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2553792": "Thanks to the organizers and team members!\nI am sharing the part of my model that was integrated into the team's ensemble and post-processing.\nResult: using only peak detection for post-processing, **publicLB=0.769, privateLB=0.810**\n\n# Feature\nI switched the features depending on the model.\nThe following Pattern 1 and Pattern 2 were created with the intention of representing the changes in sensor fluctuations before and after each step.\n* Common\n    * anglez, anglez_diff_lag/lead\n        * lag=-1, lead=1\n    * enmo, enmo_diff_lag/lead\n    * hour_sin/cos\n    * anglez_cossim_max/mean\n        * explain it later.\n* Pattern 1\n    * anglez_rstd_w360_b/f/diff\n        * rstd=rolling_std, w=window_size, b=backward, f=forward, diff=b-f\n        * b: `pl.col('anglez').rolling_std(360, center=False, min_periods=1)`\n    * enmo_rmean_w360_b/f/diff\n        * rmean=rolling_mean\n* Pattern 2\n    * anglez_diff_lag_abs_rmean_w360_b/f/diff\n        * b: `pl.col('anglez_diff_lag').abs().rolling_mean(360, center=False, min_periods=1)`\n    * enmo_diff_lag_abs_rmean_w360_b/f/diff\n\n### Cosine Similarity Feature - anglez_cossim_max/mean\nIn order to capture the periodicity of anglez, I calculated the cosine similarity by cutting it into 10,000steps while shifting it, and the max and mean of the similarity were used as the features.\n```python\ndef add_anglez_cossim_feature(series_pl: pl.DataFrame, width: int = 10_000, shift: int = 1_000) -> pl.DataFrame:\n    anglez = series_pl[\"anglez\"].to_numpy()\n    \n    # cossim\n    anglez_matrix_np = np.lib.stride_tricks.sliding_window_view(anglez, width)[::shift]\n    cossims = cosine_similarity(anglez_matrix_np)\n    np.fill_diagonal(cossims, 0)\n    cossims = cossims.max(axis=0)\n    \n    # fix length\n    cossims_fix_len = np.repeat(cossims, shift)\n    nan_len = anglez.shape[0] - cossims_fix_len.shape[0]\n    cossims_fix_len = np.hstack([cossims_fix_len, np.full((nan_len,), np.nan)])\n    \n    # cossim max/mean\n    num_elements = len(cossims_fix_len)\n    new_rows = [\n        np.concatenate([np.full(int(width * i / 100), np.nan), \n        cossims_fix_len[:-int(width * i / 100)],]) for i in range(5, 101)]\n    cossims_fix_len_aug = np.vstack([cossims_fix_len, new_rows])\n    cossims_fix_len_max = np.nanmax(cossims_fix_len_aug, axis=0)\n    cossims_fix_len_mean = np.nanmean(cossims_fix_len_aug, axis=0)\n    \n    # add features\n    series_pl = series_pl.with_columns(\n        pl.lit(cossims_fix_len_max).alias(\"anglez_cossim_max\"),\n        pl.lit(cossims_fix_len_mean).alias(\"anglez_cossim_mean\"),\n    )\n    \n    return series_pl\n```\n\n# Model\nI trained two types of architectures for each of the two types of features.\n\n* Target\n    * sleep/onset/awake, gaussian label(sigma=10)\n* Loss\n    * BCEWithLogitsLoss\n* Model type 1\n    * 1DCNN + Residual-Transformer + GRU\n    * downsample=2, 4\n    * Deeper GRU was powerful in my model. (num_layers=10)\n    * Extend the sequence length during the inference phase: train=5760, inference=17280\n    * inference w/o overlap\n* Model type 2\n    * Customized [213tubo's UNet(2D) model](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) (add Residual-Transformer)\n    * downsample=2, 4\n    * sequence length: 5760\n    * inference w/ overlap, shift=720\n\n# Score\nSplit: StratifiedGroupKFold(k=5, y=sleep, group=series_id)\nUsing only peak detection for post-processing:\n| | CV | Public LB | Private LB |\n| --- | --- | --- | --- |\n| Model type 1 | 0.795 | 0.759 | 0.802 |\n| Model type 2 | 0.792 | 0.760 | 0.801 |\n| Model type 1+2 | 0.804 | 0.769 | 0.810 |\n\n\n# Not Worked\n* augmentation\n    * Gaussian Noise, Time-directed expansion and contraction, etc.\n* stacking\n    * While utilizing the predicted values from the 1st stage, add features for each night's data and optimize using LightGBM.\n    * Simple stacking by NN did not work either."
  }
}