{
  "id": 603669,
  "title": "LB 58th/PB 152th Place solution. Mainly about \"Bert replication[SingleLB.847]\"",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/603669",
  "author_name": "yukiZ",
  "post_date": "2025-09-03T15:15:03.890000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>🔶TLDR</h1>\n<ul>\n<li>The final submission will be a blend of the four models.<ul>\n<li><strong>(1)Bert replication model.</strong></li>\n<li><strong>(2)IMU+THM/TOF TF BiLSTM+GRU+Attention</strong> ※This is code I previously published, but the unpublished part is that it uses Transformer and AdditiveAttention to improve LB/CV.</li>\n<li><strong>(3)\"Gated GRU + Hybrid Ensemble_v02\"</strong> ※Improvements to the published model</li>\n<li><strong>(4)\"cmi imu-only lstm\"</strong> ※Improvements to the published model</li></ul></li>\n<li>The final ensemble strategy employed static weighted blending.</li>\n</ul>\n<pre><code>(exsample)\npred1 = predict1(sequence, demographics)\npred2 = predict2(sequence, demographics)\npred3 = predict3(sequence, demographics)\npred4 = predict4(sequence, demographics)\n\nwts=[0.1938, 0.3362, 0.39, 0.08]\navg_pred = pred1*wts[0] + pred2*wts[1] + pred3*wts[2] + pred4*wts[3]\n</code></pre>\n<ul>\n<li>In this post, I will focus on the model that performed best out of the four model blends.</li>\n</ul>\n<hr>\n<h1>🔶Best Single performed model \"Bert replication model[PublicLB.847]\"</h1>\n<ul>\n<li>The highPerformance \"LB 0.82 | 5-fold single bert model\" notebook, which was suddenly released during the competition, was very impressive!</li>\n<li>The discussions and questions to the author were very lively, and many people tried to recreate the story, including me.<ul>\n<li>refs <a href=\"https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291\" target=\"_blank\">https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291</a></li>\n<li>refs <a href=\"https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3267899\" target=\"_blank\">https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3267899</a></li></ul></li>\n<li>Model I improved ultimately achieved a high performance of <strong>PubliLB.847</strong>, so I would like to share the details here.</li>\n</ul>\n<h2>🔹 PubliLB.847 CV Result</h2>\n<ul>\n<li>10 Fold SGKF</li>\n</ul>\n<pre><code>[Fold:0][Best_HF1]0.832204179 [Best_HF1_FULL]0.855916216 [Best_HF1_IMU]0.808492142\n[Fold:1][Best_HF1]0.809271826 [Best_HF1_FULL]0.835628130 [Best_HF1_IMU]0.782915521\n[Fold:2][Best_HF1]0.824733014 [Best_HF1_FULL]0.859665940 [Best_HF1_IMU]0.789800088\n[Fold:3][Best_HF1]0.826254680 [Best_HF1_FULL]0.859683637 [Best_HF1_IMU]0.792825722\n[Fold:4][Best_HF1]0.865529764 [Best_HF1_FULL]0.902876183 [Best_HF1_IMU]0.828183345\n[Fold:5][Best_HF1]0.877647467 [Best_HF1_FULL]0.915935206 [Best_HF1_IMU]0.839359728\n[Fold:6][Best_HF1]0.833841773 [Best_HF1_FULL]0.870622879 [Best_HF1_IMU]0.797060667\n[Fold:7][Best_HF1]0.845450227 [Best_HF1_FULL]0.876962063 [Best_HF1_IMU]0.813938390\n[Fold:8][Best_HF1]0.826240766 [Best_HF1_FULL]0.861277057 [Best_HF1_IMU]0.791204475\n[Fold:9][Best_HF1]0.855453823 [Best_HF1_FULL]0.893455497 [Best_HF1_IMU]0.817452149\n----------------------------------------------------------------------------------------------\n[OOF]   [Best_HF1]0.839662752 [Best_HF1_FULL]0.873202281 [Best_HF1_IMU]0.806123223\n</code></pre>\n<ul>\n<li>Inf Notebook <a href=\"https://www.kaggle.com/code/hideyukizushi/cmi25-bert-replication-singlepublb-847?scriptVersionId=259804173\" target=\"_blank\">https://www.kaggle.com/code/hideyukizushi/cmi25-bert-replication-singlepublb-847?scriptVersionId=259804173</a></li>\n<li>Model Weight <a href=\"https://www.kaggle.com/datasets/hideyukizushi/vav-b-101-basej104bertclonev5-augx2-rejsubx2/\" target=\"_blank\">https://www.kaggle.com/datasets/hideyukizushi/vav-b-101-basej104bertclonev5-augx2-rejsubx2/</a></li>\n</ul>\n<h2>🔹Key performance improvements</h2>\n<h3>(1)TimeStretchAugmentation※IMU Only</h3>\n<ul>\n<li>The EDA results gave the impression that the waveforms of the target gesture sequences were similar, but the periods were different.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3802145%2F0f6088e53fb2fc1312268ee963befbb5%2FSolution1.jpg?generation=1756912549173218&amp;alt=media\" alt=\"\"></li>\n<li>The implementation is below</li>\n</ul>\n<pre><code>def time_stretch_IMU(imus, thms, tofs, p=0.5):\n    if random.random() &gt; p:\n      return imus, thms, tofs\n    AUG_TH = 105 # 210*0.5\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          sq_max = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if sq_max &lt; AUG_TH:\n            interpolator = interp1d(np.linspace(0, 1, sq_max), imus_clone[:,list(range(sq_max))][1][:,imu_i],\n                                          kind='linear',\n                                          bounds_error=False,\n                                          fill_value='extrapolate')\n            time_stretch_array = torch.tensor(interpolator(np.linspace(0, 1, int(sq_max*1.5))))\n            \"\"\" 補完したシーケンスの代入 \"\"\"\n            imus[imu_j][:,][imu_k][:,imu_i][:len(time_stretch_array)]=time_stretch_array\n    return imus, thms, tofs\n</code></pre>\n<h3>(2)CopyPasteTail※IMU Only</h3>\n<ul>\n<li>The sequence consists of three steps (1.Transition-&gt;2.Pause-&gt;3.Gesture) and always ends with \"3.Gesture\".</li>\n<li>The objective variable of the competition was Gesture, and we thought that this sequence region should be of interest as a feature for classifying BFRBs.</li>\n<li>Since the \"Gesture\" sequence region had many similar cycles, we aimed to improve robustness by copy-pasting the tail of the sequence so that the model could better capture these features.</li>\n<li>The implementation is below</li>\n</ul>\n<pre><code>def copy_paste_tail_IMU(imus, thms, tofs, p=0.5):\n    if random.random() &gt; p:\n      return imus, thms, tofs\n    CROP_LEN = 20\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          tail_idx = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if (tail_idx+CROP_LEN) &gt; 200:continue\n          if (tail_idx-CROP_LEN) &lt; 1:continue\n\n          crop_tail_block = torch.tensor(imus_clone[:,list(range(210))][imu_k][:,imu_i][tail_idx-CROP_LEN:tail_idx])\n          imus[imu_j][:,][imu_k][:,imu_i][tail_idx:] = torch.cat([crop_tail_block, torch.zeros(210-tail_idx-CROP_LEN)], dim=0)\n    return imus, thms, tofs\n</code></pre>\n<h3>(3)Removing data from Train that shows abnormal device wearing</h3>\n<ul>\n<li>Subjects who used unusual methods of wearing the device(SUBJ_045235, SUBJ_019262) </li>\n<li>These subjects were considered outliers and were removed to boost CV/LB.</li>\n</ul>\n<pre><code>ex)\nself.exclude_subjects = set(['SUBJ_019262', 'SUBJ_045235'])\n</code></pre>\n<h3>Regarding specific Train sources.</h3>\n<ul>\n<li>This can be reproduced by incorporating the above three points into the Train source code that was published during the competition period.</li>\n<li>PublicNB: <a href=\"https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3270306\" target=\"_blank\">https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3270306</a></li>\n</ul>",
  "messages": [
    {
      "id": 3281075,
      "postDate": "2025-09-03T15:15:03.890Z",
      "content": "<h1>🔶TLDR</h1>\n<ul>\n<li>The final submission will be a blend of the four models.<ul>\n<li><strong>(1)Bert replication model.</strong></li>\n<li><strong>(2)IMU+THM/TOF TF BiLSTM+GRU+Attention</strong> ※This is code I previously published, but the unpublished part is that it uses Transformer and AdditiveAttention to improve LB/CV.</li>\n<li><strong>(3)\"Gated GRU + Hybrid Ensemble_v02\"</strong> ※Improvements to the published model</li>\n<li><strong>(4)\"cmi imu-only lstm\"</strong> ※Improvements to the published model</li></ul></li>\n<li>The final ensemble strategy employed static weighted blending.</li>\n</ul>\n<pre><code>(exsample)\npred1 = predict1(sequence, demographics)\npred2 = predict2(sequence, demographics)\npred3 = predict3(sequence, demographics)\npred4 = predict4(sequence, demographics)\n\nwts=[0.1938, 0.3362, 0.39, 0.08]\navg_pred = pred1*wts[0] + pred2*wts[1] + pred3*wts[2] + pred4*wts[3]\n</code></pre>\n<ul>\n<li>In this post, I will focus on the model that performed best out of the four model blends.</li>\n</ul>\n<hr>\n<h1>🔶Best Single performed model \"Bert replication model[PublicLB.847]\"</h1>\n<ul>\n<li>The highPerformance \"LB 0.82 | 5-fold single bert model\" notebook, which was suddenly released during the competition, was very impressive!</li>\n<li>The discussions and questions to the author were very lively, and many people tried to recreate the story, including me.<ul>\n<li>refs <a href=\"https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291\" target=\"_blank\">https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291</a></li>\n<li>refs <a href=\"https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3267899\" target=\"_blank\">https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3267899</a></li></ul></li>\n<li>Model I improved ultimately achieved a high performance of <strong>PubliLB.847</strong>, so I would like to share the details here.</li>\n</ul>\n<h2>🔹 PubliLB.847 CV Result</h2>\n<ul>\n<li>10 Fold SGKF</li>\n</ul>\n<pre><code>[Fold:0][Best_HF1]0.832204179 [Best_HF1_FULL]0.855916216 [Best_HF1_IMU]0.808492142\n[Fold:1][Best_HF1]0.809271826 [Best_HF1_FULL]0.835628130 [Best_HF1_IMU]0.782915521\n[Fold:2][Best_HF1]0.824733014 [Best_HF1_FULL]0.859665940 [Best_HF1_IMU]0.789800088\n[Fold:3][Best_HF1]0.826254680 [Best_HF1_FULL]0.859683637 [Best_HF1_IMU]0.792825722\n[Fold:4][Best_HF1]0.865529764 [Best_HF1_FULL]0.902876183 [Best_HF1_IMU]0.828183345\n[Fold:5][Best_HF1]0.877647467 [Best_HF1_FULL]0.915935206 [Best_HF1_IMU]0.839359728\n[Fold:6][Best_HF1]0.833841773 [Best_HF1_FULL]0.870622879 [Best_HF1_IMU]0.797060667\n[Fold:7][Best_HF1]0.845450227 [Best_HF1_FULL]0.876962063 [Best_HF1_IMU]0.813938390\n[Fold:8][Best_HF1]0.826240766 [Best_HF1_FULL]0.861277057 [Best_HF1_IMU]0.791204475\n[Fold:9][Best_HF1]0.855453823 [Best_HF1_FULL]0.893455497 [Best_HF1_IMU]0.817452149\n----------------------------------------------------------------------------------------------\n[OOF]   [Best_HF1]0.839662752 [Best_HF1_FULL]0.873202281 [Best_HF1_IMU]0.806123223\n</code></pre>\n<ul>\n<li>Inf Notebook <a href=\"https://www.kaggle.com/code/hideyukizushi/cmi25-bert-replication-singlepublb-847?scriptVersionId=259804173\" target=\"_blank\">https://www.kaggle.com/code/hideyukizushi/cmi25-bert-replication-singlepublb-847?scriptVersionId=259804173</a></li>\n<li>Model Weight <a href=\"https://www.kaggle.com/datasets/hideyukizushi/vav-b-101-basej104bertclonev5-augx2-rejsubx2/\" target=\"_blank\">https://www.kaggle.com/datasets/hideyukizushi/vav-b-101-basej104bertclonev5-augx2-rejsubx2/</a></li>\n</ul>\n<h2>🔹Key performance improvements</h2>\n<h3>(1)TimeStretchAugmentation※IMU Only</h3>\n<ul>\n<li>The EDA results gave the impression that the waveforms of the target gesture sequences were similar, but the periods were different.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3802145%2F0f6088e53fb2fc1312268ee963befbb5%2FSolution1.jpg?generation=1756912549173218&amp;alt=media\" alt=\"\"></li>\n<li>The implementation is below</li>\n</ul>\n<pre><code>def time_stretch_IMU(imus, thms, tofs, p=0.5):\n    if random.random() &gt; p:\n      return imus, thms, tofs\n    AUG_TH = 105 # 210*0.5\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          sq_max = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if sq_max &lt; AUG_TH:\n            interpolator = interp1d(np.linspace(0, 1, sq_max), imus_clone[:,list(range(sq_max))][1][:,imu_i],\n                                          kind='linear',\n                                          bounds_error=False,\n                                          fill_value='extrapolate')\n            time_stretch_array = torch.tensor(interpolator(np.linspace(0, 1, int(sq_max*1.5))))\n            \"\"\" 補完したシーケンスの代入 \"\"\"\n            imus[imu_j][:,][imu_k][:,imu_i][:len(time_stretch_array)]=time_stretch_array\n    return imus, thms, tofs\n</code></pre>\n<h3>(2)CopyPasteTail※IMU Only</h3>\n<ul>\n<li>The sequence consists of three steps (1.Transition-&gt;2.Pause-&gt;3.Gesture) and always ends with \"3.Gesture\".</li>\n<li>The objective variable of the competition was Gesture, and we thought that this sequence region should be of interest as a feature for classifying BFRBs.</li>\n<li>Since the \"Gesture\" sequence region had many similar cycles, we aimed to improve robustness by copy-pasting the tail of the sequence so that the model could better capture these features.</li>\n<li>The implementation is below</li>\n</ul>\n<pre><code>def copy_paste_tail_IMU(imus, thms, tofs, p=0.5):\n    if random.random() &gt; p:\n      return imus, thms, tofs\n    CROP_LEN = 20\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          tail_idx = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if (tail_idx+CROP_LEN) &gt; 200:continue\n          if (tail_idx-CROP_LEN) &lt; 1:continue\n\n          crop_tail_block = torch.tensor(imus_clone[:,list(range(210))][imu_k][:,imu_i][tail_idx-CROP_LEN:tail_idx])\n          imus[imu_j][:,][imu_k][:,imu_i][tail_idx:] = torch.cat([crop_tail_block, torch.zeros(210-tail_idx-CROP_LEN)], dim=0)\n    return imus, thms, tofs\n</code></pre>\n<h3>(3)Removing data from Train that shows abnormal device wearing</h3>\n<ul>\n<li>Subjects who used unusual methods of wearing the device(SUBJ_045235, SUBJ_019262) </li>\n<li>These subjects were considered outliers and were removed to boost CV/LB.</li>\n</ul>\n<pre><code>ex)\nself.exclude_subjects = set(['SUBJ_019262', 'SUBJ_045235'])\n</code></pre>\n<h3>Regarding specific Train sources.</h3>\n<ul>\n<li>This can be reproduced by incorporating the above three points into the Train source code that was published during the competition period.</li>\n<li>PublicNB: <a href=\"https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3270306\" target=\"_blank\">https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3270306</a></li>\n</ul>",
      "rawMarkdown": "# 🔶TLDR\n* The final submission will be a blend of the four models.\n     * **(1)Bert replication model.**\n     * **(2)IMU+THM/TOF TF BiLSTM+GRU+Attention** ※This is code I previously published, but the unpublished part is that it uses Transformer and AdditiveAttention to improve LB/CV.\n     * **(3)\"Gated GRU + Hybrid Ensemble_v02\"** ※Improvements to the published model\n     * **(4)\"cmi imu-only lstm\"** ※Improvements to the published model\n* The final ensemble strategy employed static weighted blending.\n```\n(exsample)\npred1 = predict1(sequence, demographics)\npred2 = predict2(sequence, demographics)\npred3 = predict3(sequence, demographics)\npred4 = predict4(sequence, demographics)\n\nwts=[0.1938, 0.3362, 0.39, 0.08]\navg_pred = pred1*wts[0] + pred2*wts[1] + pred3*wts[2] + pred4*wts[3]\n```\n\n* In this post, I will focus on the model that performed best out of the four model blends.\n\n---\n\n# 🔶Best Single performed model \"Bert replication model[PublicLB.847]\"\n* The highPerformance \"LB 0.82 | 5-fold single bert model\" notebook, which was suddenly released during the competition, was very impressive!\n* The discussions and questions to the author were very lively, and many people tried to recreate the story, including me.\n    * refs https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291\n    * refs https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3267899\n* Model I improved ultimately achieved a high performance of **PubliLB.847**, so I would like to share the details here.\n\n## 🔹 PubliLB.847 CV Result\n* 10 Fold SGKF\n```\n[Fold:0][Best_HF1]0.832204179 [Best_HF1_FULL]0.855916216 [Best_HF1_IMU]0.808492142\n[Fold:1][Best_HF1]0.809271826 [Best_HF1_FULL]0.835628130 [Best_HF1_IMU]0.782915521\n[Fold:2][Best_HF1]0.824733014 [Best_HF1_FULL]0.859665940 [Best_HF1_IMU]0.789800088\n[Fold:3][Best_HF1]0.826254680 [Best_HF1_FULL]0.859683637 [Best_HF1_IMU]0.792825722\n[Fold:4][Best_HF1]0.865529764 [Best_HF1_FULL]0.902876183 [Best_HF1_IMU]0.828183345\n[Fold:5][Best_HF1]0.877647467 [Best_HF1_FULL]0.915935206 [Best_HF1_IMU]0.839359728\n[Fold:6][Best_HF1]0.833841773 [Best_HF1_FULL]0.870622879 [Best_HF1_IMU]0.797060667\n[Fold:7][Best_HF1]0.845450227 [Best_HF1_FULL]0.876962063 [Best_HF1_IMU]0.813938390\n[Fold:8][Best_HF1]0.826240766 [Best_HF1_FULL]0.861277057 [Best_HF1_IMU]0.791204475\n[Fold:9][Best_HF1]0.855453823 [Best_HF1_FULL]0.893455497 [Best_HF1_IMU]0.817452149\n----------------------------------------------------------------------------------------------\n[OOF]   [Best_HF1]0.839662752 [Best_HF1_FULL]0.873202281 [Best_HF1_IMU]0.806123223\n\n```\n\n* Inf Notebook https://www.kaggle.com/code/hideyukizushi/cmi25-bert-replication-singlepublb-847?scriptVersionId=259804173\n* Model Weight https://www.kaggle.com/datasets/hideyukizushi/vav-b-101-basej104bertclonev5-augx2-rejsubx2/\n\n## 🔹Key performance improvements\n### (1)TimeStretchAugmentation※IMU Only\n* The EDA results gave the impression that the waveforms of the target gesture sequences were similar, but the periods were different.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3802145%2F0f6088e53fb2fc1312268ee963befbb5%2FSolution1.jpg?generation=1756912549173218&alt=media)\n* The implementation is below\n```\ndef time_stretch_IMU(imus, thms, tofs, p=0.5):\n    if random.random() > p:\n      return imus, thms, tofs\n    AUG_TH = 105 # 210*0.5\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          sq_max = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if sq_max < AUG_TH:\n            interpolator = interp1d(np.linspace(0, 1, sq_max), imus_clone[:,list(range(sq_max))][1][:,imu_i],\n                                          kind='linear',\n                                          bounds_error=False,\n                                          fill_value='extrapolate')\n            time_stretch_array = torch.tensor(interpolator(np.linspace(0, 1, int(sq_max*1.5))))\n            \"\"\" 補完したシーケンスの代入 \"\"\"\n            imus[imu_j][:,][imu_k][:,imu_i][:len(time_stretch_array)]=time_stretch_array\n    return imus, thms, tofs\n```\n\n### (2)CopyPasteTail※IMU Only\n* The sequence consists of three steps (1.Transition->2.Pause->3.Gesture) and always ends with \"3.Gesture\".\n* The objective variable of the competition was Gesture, and we thought that this sequence region should be of interest as a feature for classifying BFRBs.\n* Since the \"Gesture\" sequence region had many similar cycles, we aimed to improve robustness by copy-pasting the tail of the sequence so that the model could better capture these features.\n* The implementation is below\n```\ndef copy_paste_tail_IMU(imus, thms, tofs, p=0.5):\n    if random.random() > p:\n      return imus, thms, tofs\n    CROP_LEN = 20\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          tail_idx = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if (tail_idx+CROP_LEN) > 200:continue\n          if (tail_idx-CROP_LEN) < 1:continue\n\n          crop_tail_block = torch.tensor(imus_clone[:,list(range(210))][imu_k][:,imu_i][tail_idx-CROP_LEN:tail_idx])\n          imus[imu_j][:,][imu_k][:,imu_i][tail_idx:] = torch.cat([crop_tail_block, torch.zeros(210-tail_idx-CROP_LEN)], dim=0)\n    return imus, thms, tofs\n```\n\n### (3)Removing data from Train that shows abnormal device wearing\n*  Subjects who used unusual methods of wearing the device(SUBJ_045235, SUBJ_019262) \n* These subjects were considered outliers and were removed to boost CV/LB.\n```\nex)\nself.exclude_subjects = set(['SUBJ_019262', 'SUBJ_045235'])\n```\n\n### Regarding specific Train sources.\n* This can be reproduced by incorporating the above three points into the Train source code that was published during the competition period.\n* PublicNB: https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3270306",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3281075": "# 🔶TLDR\n* The final submission will be a blend of the four models.\n     * **(1)Bert replication model.**\n     * **(2)IMU+THM/TOF TF BiLSTM+GRU+Attention** ※This is code I previously published, but the unpublished part is that it uses Transformer and AdditiveAttention to improve LB/CV.\n     * **(3)\"Gated GRU + Hybrid Ensemble_v02\"** ※Improvements to the published model\n     * **(4)\"cmi imu-only lstm\"** ※Improvements to the published model\n* The final ensemble strategy employed static weighted blending.\n```\n(exsample)\npred1 = predict1(sequence, demographics)\npred2 = predict2(sequence, demographics)\npred3 = predict3(sequence, demographics)\npred4 = predict4(sequence, demographics)\n\nwts=[0.1938, 0.3362, 0.39, 0.08]\navg_pred = pred1*wts[0] + pred2*wts[1] + pred3*wts[2] + pred4*wts[3]\n```\n\n* In this post, I will focus on the model that performed best out of the four model blends.\n\n---\n\n# 🔶Best Single performed model \"Bert replication model[PublicLB.847]\"\n* The highPerformance \"LB 0.82 | 5-fold single bert model\" notebook, which was suddenly released during the competition, was very impressive!\n* The discussions and questions to the author were very lively, and many people tried to recreate the story, including me.\n    * refs https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291\n    * refs https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3267899\n* Model I improved ultimately achieved a high performance of **PubliLB.847**, so I would like to share the details here.\n\n## 🔹 PubliLB.847 CV Result\n* 10 Fold SGKF\n```\n[Fold:0][Best_HF1]0.832204179 [Best_HF1_FULL]0.855916216 [Best_HF1_IMU]0.808492142\n[Fold:1][Best_HF1]0.809271826 [Best_HF1_FULL]0.835628130 [Best_HF1_IMU]0.782915521\n[Fold:2][Best_HF1]0.824733014 [Best_HF1_FULL]0.859665940 [Best_HF1_IMU]0.789800088\n[Fold:3][Best_HF1]0.826254680 [Best_HF1_FULL]0.859683637 [Best_HF1_IMU]0.792825722\n[Fold:4][Best_HF1]0.865529764 [Best_HF1_FULL]0.902876183 [Best_HF1_IMU]0.828183345\n[Fold:5][Best_HF1]0.877647467 [Best_HF1_FULL]0.915935206 [Best_HF1_IMU]0.839359728\n[Fold:6][Best_HF1]0.833841773 [Best_HF1_FULL]0.870622879 [Best_HF1_IMU]0.797060667\n[Fold:7][Best_HF1]0.845450227 [Best_HF1_FULL]0.876962063 [Best_HF1_IMU]0.813938390\n[Fold:8][Best_HF1]0.826240766 [Best_HF1_FULL]0.861277057 [Best_HF1_IMU]0.791204475\n[Fold:9][Best_HF1]0.855453823 [Best_HF1_FULL]0.893455497 [Best_HF1_IMU]0.817452149\n----------------------------------------------------------------------------------------------\n[OOF]   [Best_HF1]0.839662752 [Best_HF1_FULL]0.873202281 [Best_HF1_IMU]0.806123223\n\n```\n\n* Inf Notebook https://www.kaggle.com/code/hideyukizushi/cmi25-bert-replication-singlepublb-847?scriptVersionId=259804173\n* Model Weight https://www.kaggle.com/datasets/hideyukizushi/vav-b-101-basej104bertclonev5-augx2-rejsubx2/\n\n## 🔹Key performance improvements\n### (1)TimeStretchAugmentation※IMU Only\n* The EDA results gave the impression that the waveforms of the target gesture sequences were similar, but the periods were different.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3802145%2F0f6088e53fb2fc1312268ee963befbb5%2FSolution1.jpg?generation=1756912549173218&alt=media)\n* The implementation is below\n```\ndef time_stretch_IMU(imus, thms, tofs, p=0.5):\n    if random.random() > p:\n      return imus, thms, tofs\n    AUG_TH = 105 # 210*0.5\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          sq_max = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if sq_max < AUG_TH:\n            interpolator = interp1d(np.linspace(0, 1, sq_max), imus_clone[:,list(range(sq_max))][1][:,imu_i],\n                                          kind='linear',\n                                          bounds_error=False,\n                                          fill_value='extrapolate')\n            time_stretch_array = torch.tensor(interpolator(np.linspace(0, 1, int(sq_max*1.5))))\n            \"\"\" 補完したシーケンスの代入 \"\"\"\n            imus[imu_j][:,][imu_k][:,imu_i][:len(time_stretch_array)]=time_stretch_array\n    return imus, thms, tofs\n```\n\n### (2)CopyPasteTail※IMU Only\n* The sequence consists of three steps (1.Transition->2.Pause->3.Gesture) and always ends with \"3.Gesture\".\n* The objective variable of the competition was Gesture, and we thought that this sequence region should be of interest as a feature for classifying BFRBs.\n* Since the \"Gesture\" sequence region had many similar cycles, we aimed to improve robustness by copy-pasting the tail of the sequence so that the model could better capture these features.\n* The implementation is below\n```\ndef copy_paste_tail_IMU(imus, thms, tofs, p=0.5):\n    if random.random() > p:\n      return imus, thms, tofs\n    CROP_LEN = 20\n    for imu_j in range(len(imus)):\n      imus_clone = imus[imu_j].to('cpu').detach().numpy().copy()\n      for imu_i in range(imus_clone.shape[2]):\n        for imu_k in range(imus_clone.shape[0]):\n          if np.all(imus_clone[:,list(range(210))][imu_k][:,imu_i]):continue\n          tail_idx = int(np.where(imus_clone[:,list(range(210))][imu_k][:,imu_i]==0)[0][0])\n          if (tail_idx+CROP_LEN) > 200:continue\n          if (tail_idx-CROP_LEN) < 1:continue\n\n          crop_tail_block = torch.tensor(imus_clone[:,list(range(210))][imu_k][:,imu_i][tail_idx-CROP_LEN:tail_idx])\n          imus[imu_j][:,][imu_k][:,imu_i][tail_idx:] = torch.cat([crop_tail_block, torch.zeros(210-tail_idx-CROP_LEN)], dim=0)\n    return imus, thms, tofs\n```\n\n### (3)Removing data from Train that shows abnormal device wearing\n*  Subjects who used unusual methods of wearing the device(SUBJ_045235, SUBJ_019262) \n* These subjects were considered outliers and were removed to boost CV/LB.\n```\nex)\nself.exclude_subjects = set(['SUBJ_019262', 'SUBJ_045235'])\n```\n\n### Regarding specific Train sources.\n* This can be reproduced by incorporating the above three points into the Train source code that was published during the competition period.\n* PublicNB: https://www.kaggle.com/code/wasupandceacar/lb-0-841-5fold-single-model-with-split-sensors/comments#3270306"
  }
}