{
  "id": 603594,
  "title": "2nd Place Solution",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/603594",
  "author_name": "daiwakun",
  "post_date": "2025-09-03T08:32:25.637000",
  "votes": 113,
  "comment_count": 20,
  "views": 0,
  "content": "<p>First, thanks to the organizers and hosts for an interesting competition, and to the community for insightful discussions and notebooks.</p>\n<h1>Overview</h1>\n<ul>\n<li>If you watched the leaderboard, you may have noticed a late jump: I discovered a dataset trick four days before the deadline and boosted the score substantially via post-processing.</li>\n<li>Without post-processing, my model scores were: Public <strong>0.865</strong> / Private <strong>0.858</strong>.</li>\n<li>Below I first describe the model (data handling, features, architecture, training), then the post-processing that made the big difference.</li>\n</ul>\n<h1>Model</h1>\n<p><strong>Handling Missingness</strong></p>\n<ul>\n<li>Training separate models gave better accuracy when certain sensors were missing.</li>\n<li>I trained four variants depending on whether IMU rotation was missing and whether THM/TOF were missing (rot present/absent × THM and TOF present/absent).</li>\n</ul>\n<p><strong>Feature Engineering</strong></p>\n<ul>\n<li><p>IMU</p>\n<ul>\n<li>acc(x/y/z)</li>\n<li>quaternion 6D representation (to avoid discontinuities)</li>\n<li>angular velocity(x/y/z)</li>\n<li>linear acceleration(x/y/z)</li></ul></li>\n<li><p>THM/TOF</p>\n<ul>\n<li>Filled NaN and -1 with zero.</li>\n<li>I did not use THM.</li></ul></li>\n<li><p>Left-handed subjects</p>\n<ul>\n<li>Multiply by −1: acc_x, linear_acc_x, angular velocity y and z, and components 0 and 4 of the quaternion 6D representation.</li>\n<li>Swap tof_3 and tof_5, and horizontally flip the left–right direction of the 2D TOF grids.</li></ul></li>\n<li><p>Subject-specific correction</p>\n<ul>\n<li>In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.</li>\n<li>I multiplied all channels by −1 except acc_z and linear_acc_z.</li>\n<li>I could not find a reliable correction for their TOF, so I set TOF to NaN for these subjects.</li></ul></li>\n</ul>\n<p><strong>Architecture</strong></p>\n<ul>\n<li>Based on public notebooks: a Residual SE-CNN Block + Attention model.</li>\n<li>Modality-specific stems:<ul>\n<li>Independent 1D CNN branches for each of: acc, quaternion 6D, angular velocity, linear_acc, and each tof1~5; then concatenate.</li></ul></li>\n<li>TOF processing:<ul>\n<li>Before 1D CNN branches, I use a 2D CNN on the TOF grids per frame, followed by mean pooling to get a per-step feature.</li></ul></li>\n<li>Phase-aware Attention<ul>\n<li>Plain temporal attention can over-focus on long phases (e.g., long transitions to the target).</li>\n<li>I add an auxiliary 3-class phase predictor at each time step:</li></ul><ol>\n<li>(Relaxes and) Moves hand to target location</li>\n<li>Hand at target location</li>\n<li>Performs gesture</li></ol><ul>\n<li>I construct three attentions, one per phase, and weight each attention score by the corresponding phase probability. This extracts features specific to each phase instead of letting long phases dominate.</li></ul></li>\n</ul>\n<p><strong>Prediction Target</strong></p>\n<ul>\n<li>I predict a composite label:(Initial behavior ∈ {Relaxes&amp;Moves / Moves}, orientation, gesture).</li>\n<li>This choice is important for the post-processing step.</li>\n</ul>\n<p><strong>Mixup</strong></p>\n<ul>\n<li>Simple linear mixup, but phase-aligned to avoid mixing behaviors:<ul>\n<li>Split each sequence into the three phases above and mix within the same phase only.</li>\n<li>Especially for “moves to target,” I align the end of the phase so the arrival at the target is synchronized before mixing.</li></ul></li>\n</ul>\n<p><strong>Training Details</strong></p>\n<ul>\n<li>Optimizer: Adam (lr=1e-3, wd=1e-4)</li>\n<li>Scheduler: Cosine Annealing</li>\n<li>Batch size: 32</li>\n<li>Epochs: 50</li>\n<li>Folds: 10</li>\n<li>Final model: ensemble of 3 variants (different depths/layers of 1DCNN)</li>\n</ul>\n<p><strong>Online Pseudo-Labeling</strong></p>\n<ul>\n<li>With ~8k training sequences and ~3.5k test sequences, pseudo labels helped.</li>\n<li>The submission API predicts one sequence at a time, so I performed test-time updates: whenever a batch-sized group of test sequences accumulated, I fine-tuned one step with a small LR (5e-5) using pseudo labels.</li>\n</ul>\n<h1>Post-Processing</h1>\n<p><strong>Key observation from train.csv:</strong></p>\n<ul>\n<li>There are 4 orientations × 18 gestures = 72 possible (orientation, gesture) pairs, but only 51 were actually present.</li>\n<li>For each subject, the dataset contains exactly those 51 pairs, each recorded with two possible initial behaviors (Relaxes&amp;Moves / Moves), totaling 51 × 2 = 102 sequences per subject.</li>\n<li>Based on this, I trained the model to predict 102 classes (the composite label described above).</li>\n</ul>\n<p><strong>Maximizing Conditional Joint Probability (with no-repeat constraint)</strong></p>\n<ul>\n<li>For each subject, I keep all predicted class probabilities across their sequences.</li>\n<li>When predicting the N-th sequence for that subject, instead of taking the argmax for the N-th item independently, I choose the class sequence of length N that maximizes the joint probability (sum of log-probabilities) subject to “no label is used twice.”</li>\n<li>Because labels are assigned jointly per subject under a no-repeat constraint, early high-confidence picks lock in their labels; when a later sample is only moderately confident in a label that’s already taken, an alternative is chosen to yield a higher joint log-likelihood.</li>\n<li>Computationally, this is an assignment problem solvable efficiently (e.g., Hungarian algorithm) using negative log-probabilities as costs.</li>\n</ul>\n<h1>Scores</h1>\n<table>\n<thead>\n<tr>\n<th>Method</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>No pseudo / No post-proc</td>\n<td>0.862</td>\n<td>0.854</td>\n</tr>\n<tr>\n<td>With pseudo / No post-proc</td>\n<td>0.865</td>\n<td>0.858</td>\n</tr>\n<tr>\n<td>No pseudo / With post-proc</td>\n<td>0.891</td>\n<td>0.875</td>\n</tr>\n<tr>\n<td>With pseudo / With post-proc</td>\n<td>0.900</td>\n<td>0.878</td>\n</tr>\n</tbody>\n</table>\n<h1>Codes</h1>\n<p><a href=\"https://github.com/Yamato-Arai/kaggle-cmi-detect-behavior-2nd-place-solution\" target=\"_blank\">github</a></p>",
  "messages": [
    {
      "id": 3280846,
      "postDate": "2025-09-03T08:32:25.637Z",
      "content": "<p>First, thanks to the organizers and hosts for an interesting competition, and to the community for insightful discussions and notebooks.</p>\n<h1>Overview</h1>\n<ul>\n<li>If you watched the leaderboard, you may have noticed a late jump: I discovered a dataset trick four days before the deadline and boosted the score substantially via post-processing.</li>\n<li>Without post-processing, my model scores were: Public <strong>0.865</strong> / Private <strong>0.858</strong>.</li>\n<li>Below I first describe the model (data handling, features, architecture, training), then the post-processing that made the big difference.</li>\n</ul>\n<h1>Model</h1>\n<p><strong>Handling Missingness</strong></p>\n<ul>\n<li>Training separate models gave better accuracy when certain sensors were missing.</li>\n<li>I trained four variants depending on whether IMU rotation was missing and whether THM/TOF were missing (rot present/absent × THM and TOF present/absent).</li>\n</ul>\n<p><strong>Feature Engineering</strong></p>\n<ul>\n<li><p>IMU</p>\n<ul>\n<li>acc(x/y/z)</li>\n<li>quaternion 6D representation (to avoid discontinuities)</li>\n<li>angular velocity(x/y/z)</li>\n<li>linear acceleration(x/y/z)</li></ul></li>\n<li><p>THM/TOF</p>\n<ul>\n<li>Filled NaN and -1 with zero.</li>\n<li>I did not use THM.</li></ul></li>\n<li><p>Left-handed subjects</p>\n<ul>\n<li>Multiply by −1: acc_x, linear_acc_x, angular velocity y and z, and components 0 and 4 of the quaternion 6D representation.</li>\n<li>Swap tof_3 and tof_5, and horizontally flip the left–right direction of the 2D TOF grids.</li></ul></li>\n<li><p>Subject-specific correction</p>\n<ul>\n<li>In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.</li>\n<li>I multiplied all channels by −1 except acc_z and linear_acc_z.</li>\n<li>I could not find a reliable correction for their TOF, so I set TOF to NaN for these subjects.</li></ul></li>\n</ul>\n<p><strong>Architecture</strong></p>\n<ul>\n<li>Based on public notebooks: a Residual SE-CNN Block + Attention model.</li>\n<li>Modality-specific stems:<ul>\n<li>Independent 1D CNN branches for each of: acc, quaternion 6D, angular velocity, linear_acc, and each tof1~5; then concatenate.</li></ul></li>\n<li>TOF processing:<ul>\n<li>Before 1D CNN branches, I use a 2D CNN on the TOF grids per frame, followed by mean pooling to get a per-step feature.</li></ul></li>\n<li>Phase-aware Attention<ul>\n<li>Plain temporal attention can over-focus on long phases (e.g., long transitions to the target).</li>\n<li>I add an auxiliary 3-class phase predictor at each time step:</li></ul><ol>\n<li>(Relaxes and) Moves hand to target location</li>\n<li>Hand at target location</li>\n<li>Performs gesture</li></ol><ul>\n<li>I construct three attentions, one per phase, and weight each attention score by the corresponding phase probability. This extracts features specific to each phase instead of letting long phases dominate.</li></ul></li>\n</ul>\n<p><strong>Prediction Target</strong></p>\n<ul>\n<li>I predict a composite label:(Initial behavior ∈ {Relaxes&amp;Moves / Moves}, orientation, gesture).</li>\n<li>This choice is important for the post-processing step.</li>\n</ul>\n<p><strong>Mixup</strong></p>\n<ul>\n<li>Simple linear mixup, but phase-aligned to avoid mixing behaviors:<ul>\n<li>Split each sequence into the three phases above and mix within the same phase only.</li>\n<li>Especially for “moves to target,” I align the end of the phase so the arrival at the target is synchronized before mixing.</li></ul></li>\n</ul>\n<p><strong>Training Details</strong></p>\n<ul>\n<li>Optimizer: Adam (lr=1e-3, wd=1e-4)</li>\n<li>Scheduler: Cosine Annealing</li>\n<li>Batch size: 32</li>\n<li>Epochs: 50</li>\n<li>Folds: 10</li>\n<li>Final model: ensemble of 3 variants (different depths/layers of 1DCNN)</li>\n</ul>\n<p><strong>Online Pseudo-Labeling</strong></p>\n<ul>\n<li>With ~8k training sequences and ~3.5k test sequences, pseudo labels helped.</li>\n<li>The submission API predicts one sequence at a time, so I performed test-time updates: whenever a batch-sized group of test sequences accumulated, I fine-tuned one step with a small LR (5e-5) using pseudo labels.</li>\n</ul>\n<h1>Post-Processing</h1>\n<p><strong>Key observation from train.csv:</strong></p>\n<ul>\n<li>There are 4 orientations × 18 gestures = 72 possible (orientation, gesture) pairs, but only 51 were actually present.</li>\n<li>For each subject, the dataset contains exactly those 51 pairs, each recorded with two possible initial behaviors (Relaxes&amp;Moves / Moves), totaling 51 × 2 = 102 sequences per subject.</li>\n<li>Based on this, I trained the model to predict 102 classes (the composite label described above).</li>\n</ul>\n<p><strong>Maximizing Conditional Joint Probability (with no-repeat constraint)</strong></p>\n<ul>\n<li>For each subject, I keep all predicted class probabilities across their sequences.</li>\n<li>When predicting the N-th sequence for that subject, instead of taking the argmax for the N-th item independently, I choose the class sequence of length N that maximizes the joint probability (sum of log-probabilities) subject to “no label is used twice.”</li>\n<li>Because labels are assigned jointly per subject under a no-repeat constraint, early high-confidence picks lock in their labels; when a later sample is only moderately confident in a label that’s already taken, an alternative is chosen to yield a higher joint log-likelihood.</li>\n<li>Computationally, this is an assignment problem solvable efficiently (e.g., Hungarian algorithm) using negative log-probabilities as costs.</li>\n</ul>\n<h1>Scores</h1>\n<table>\n<thead>\n<tr>\n<th>Method</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>No pseudo / No post-proc</td>\n<td>0.862</td>\n<td>0.854</td>\n</tr>\n<tr>\n<td>With pseudo / No post-proc</td>\n<td>0.865</td>\n<td>0.858</td>\n</tr>\n<tr>\n<td>No pseudo / With post-proc</td>\n<td>0.891</td>\n<td>0.875</td>\n</tr>\n<tr>\n<td>With pseudo / With post-proc</td>\n<td>0.900</td>\n<td>0.878</td>\n</tr>\n</tbody>\n</table>\n<h1>Codes</h1>\n<p><a href=\"https://github.com/Yamato-Arai/kaggle-cmi-detect-behavior-2nd-place-solution\" target=\"_blank\">github</a></p>",
      "rawMarkdown": "First, thanks to the organizers and hosts for an interesting competition, and to the community for insightful discussions and notebooks.\n\n# Overview\n- If you watched the leaderboard, you may have noticed a late jump: I discovered a dataset trick four days before the deadline and boosted the score substantially via post-processing.\n- Without post-processing, my model scores were: Public **0.865** / Private **0.858**.\n- Below I first describe the model (data handling, features, architecture, training), then the post-processing that made the big difference.\n\n# Model\n\n**Handling Missingness**\n- Training separate models gave better accuracy when certain sensors were missing.\n- I trained four variants depending on whether IMU rotation was missing and whether THM/TOF were missing (rot present/absent × THM and TOF present/absent).\n\n**Feature Engineering**\n\n- IMU\n - acc(x/y/z)\n - quaternion 6D representation (to avoid discontinuities)\n - angular velocity(x/y/z)\n - linear acceleration(x/y/z)\n\n- THM/TOF\n - Filled NaN and -1 with zero.\n - I did not use THM.\n\n- Left-handed subjects\n - Multiply by −1: acc_x, linear_acc_x, angular velocity y and z, and components 0 and 4 of the quaternion 6D representation.\n - Swap tof_3 and tof_5, and horizontally flip the left–right direction of the 2D TOF grids.\n\n- Subject-specific correction\n - In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.\n - I multiplied all channels by −1 except acc_z and linear_acc_z.\n - I could not find a reliable correction for their TOF, so I set TOF to NaN for these subjects.\n\n**Architecture**\n\n- Based on public notebooks: a Residual SE-CNN Block + Attention model.\n- Modality-specific stems:\n - Independent 1D CNN branches for each of: acc, quaternion 6D, angular velocity, linear_acc, and each tof1~5; then concatenate.\n- TOF processing:\n - Before 1D CNN branches, I use a 2D CNN on the TOF grids per frame, followed by mean pooling to get a per-step feature.\n- Phase-aware Attention\n - Plain temporal attention can over-focus on long phases (e.g., long transitions to the target).\n - I add an auxiliary 3-class phase predictor at each time step:\n  1. (Relaxes and) Moves hand to target location\n  2. Hand at target location\n  3. Performs gesture\n - I construct three attentions, one per phase, and weight each attention score by the corresponding phase probability. This extracts features specific to each phase instead of letting long phases dominate.\n\n**Prediction Target**\n- I predict a composite label:(Initial behavior ∈ {Relaxes&Moves / Moves}, orientation, gesture).\n- This choice is important for the post-processing step.\n\n**Mixup**\n- Simple linear mixup, but phase-aligned to avoid mixing behaviors:\n - Split each sequence into the three phases above and mix within the same phase only.\n - Especially for “moves to target,” I align the end of the phase so the arrival at the target is synchronized before mixing.\n\n**Training Details**\n- Optimizer: Adam (lr=1e-3, wd=1e-4)\n- Scheduler: Cosine Annealing\n- Batch size: 32\n- Epochs: 50\n- Folds: 10\n- Final model: ensemble of 3 variants (different depths/layers of 1DCNN)\n\n**Online Pseudo-Labeling**\n- With ~8k training sequences and ~3.5k test sequences, pseudo labels helped.\n- The submission API predicts one sequence at a time, so I performed test-time updates: whenever a batch-sized group of test sequences accumulated, I fine-tuned one step with a small LR (5e-5) using pseudo labels.\n\n# Post-Processing\n\n**Key observation from train.csv:**\n- There are 4 orientations × 18 gestures = 72 possible (orientation, gesture) pairs, but only 51 were actually present.\n- For each subject, the dataset contains exactly those 51 pairs, each recorded with two possible initial behaviors (Relaxes&Moves / Moves), totaling 51 × 2 = 102 sequences per subject.\n- Based on this, I trained the model to predict 102 classes (the composite label described above).\n\n**Maximizing Conditional Joint Probability (with no-repeat constraint)**\n- For each subject, I keep all predicted class probabilities across their sequences.\n- When predicting the N-th sequence for that subject, instead of taking the argmax for the N-th item independently, I choose the class sequence of length N that maximizes the joint probability (sum of log-probabilities) subject to “no label is used twice.”\n- Because labels are assigned jointly per subject under a no-repeat constraint, early high-confidence picks lock in their labels; when a later sample is only moderately confident in a label that’s already taken, an alternative is chosen to yield a higher joint log-likelihood.\n- Computationally, this is an assignment problem solvable efficiently (e.g., Hungarian algorithm) using negative log-probabilities as costs.\n\n# Scores\n| Method | Public LB | Private LB |\n| --- | ---: | ---: |\n| No pseudo / No post-proc | 0.862 | 0.854 |\n| With pseudo / No post-proc | 0.865 | 0.858 |\n| No pseudo / With post-proc | 0.891 | 0.875 |\n| With pseudo / With post-proc | 0.900 | 0.878 |\n\n# Codes\n[github](https://github.com/Yamato-Arai/kaggle-cmi-detect-behavior-2nd-place-solution)\n\n",
      "votes": 113
    },
    {
      "id": 3280879,
      "postDate": "2025-09-03T09:38:06.183Z",
      "content": "<p>I used a similar approach with the targets but excluded Initial behavior ∈ {Relaxes&amp;Moves / Moves}, resulting in 51 targets total. I'm curious about how much performance difference there would be using 102 targets instead of 51 - it would be interesting to see that comparison!<br>\nCongratulations on the incredible performance and innovative approach!</p>",
      "rawMarkdown": "I used a similar approach with the targets but excluded Initial behavior ∈ {Relaxes&Moves / Moves}, resulting in 51 targets total. I'm curious about how much performance difference there would be using 102 targets instead of 51 - it would be interesting to see that comparison!\nCongratulations on the incredible performance and innovative approach!",
      "votes": 1
    },
    {
      "id": 3280862,
      "postDate": "2025-09-03T08:59:31.617Z",
      "content": "<p>Congratz on 2nd place !</p>\n<p>Your sudden jump in LB was one of the clues that helped us find the trick - given your background in optimization :)</p>\n<p>We figured part of it, but your approach is definitely better. </p>",
      "rawMarkdown": "Congratz on 2nd place !\n\nYour sudden jump in LB was one of the clues that helped us find the trick - given your background in optimization :)\n\nWe figured part of it, but your approach is definitely better. ",
      "votes": 1,
      "replies": [
        {
          "id": 3281042,
          "postDate": "2025-09-03T13:54:16.677Z",
          "content": "<p>The meta of Kaggle competitions is wild lol.</p>",
          "rawMarkdown": "The meta of Kaggle competitions is wild lol."
        }
      ]
    },
    {
      "id": 3280852,
      "postDate": "2025-09-03T08:42:27.283Z",
      "content": "<p>The idea of training 102 classes is wild!! Well deserved</p>",
      "rawMarkdown": "The idea of training 102 classes is wild!! Well deserved",
      "votes": 1,
      "replies": [
        {
          "id": 3282776,
          "postDate": "2025-09-06T19:30:53.520Z",
          "content": "<p>102 😲, damn!</p>",
          "rawMarkdown": "102 😲, damn!"
        }
      ]
    },
    {
      "id": 3284806,
      "postDate": "2025-09-08T16:46:25.063Z",
      "content": "<p>Congrats on the 2nd place! Really impressed by the subject-specific corrections; especially the 180° rotation fix; and the way you structured separate models for missing sensor cases. The phase-aware attention with the auxiliary phase predictor is a clever way to balance long vs. short phases, and predicting the composite label set up the post-processing beautifully. The Hungarian assignment trick on top of that was a brilliant use of dataset structure. Thanks for the detailed breakdown; lots of takeaways here!</p>",
      "rawMarkdown": "Congrats on the 2nd place! Really impressed by the subject-specific corrections; especially the 180° rotation fix; and the way you structured separate models for missing sensor cases. The phase-aware attention with the auxiliary phase predictor is a clever way to balance long vs. short phases, and predicting the composite label set up the post-processing beautifully. The Hungarian assignment trick on top of that was a brilliant use of dataset structure. Thanks for the detailed breakdown; lots of takeaways here!"
    },
    {
      "id": 3282363,
      "postDate": "2025-09-06T07:31:02.993Z",
      "content": "<h3>Congrats Bro!!!</h3>\n<p><strong>Issue:</strong> My code submission is failing with an error. I am unable to resolve this despite multiple attempts.</p>\n<p>This is my <strong>Code :</strong> <a href=\"https://github.com/AshwinAshok3/cmi-detect-behavior-with-sensor-data/blob/main/cmi-wrist-watch-sensors_final.ipynb\" target=\"_blank\">cmi-watch-sensors_final</a></p>\n<p>I was only able to catch <code>64</code><strong>%</strong> accuracy with <code>XGBoost</code> and <code>CatBoost</code> &amp; <code>62</code><strong>%</strong> with <code>Pytorch</code></p>\n<h3>A Journey of Discovery</h3>\n<p>This project served as a profound learning experience, challenging my initial assumptions about feature engineering. The multi-class, grouped nature of the dataset forced a re-evaluation of my approach and opened my eyes to the vast, intricate landscape of data science. This competition was an invaluable \"first\" for me, highlighting the need to constantly evolve and think creatively to solve complex problems.</p>",
      "rawMarkdown": "### Congrats Bro!!!\n**Issue:** My code submission is failing with an error. I am unable to resolve this despite multiple attempts.\n\nThis is my **Code :** [cmi-watch-sensors_final](https://github.com/AshwinAshok3/cmi-detect-behavior-with-sensor-data/blob/main/cmi-wrist-watch-sensors_final.ipynb)\n\nI was only able to catch `64`**%** accuracy with `XGBoost` and `CatBoost` & `62`**%** with `Pytorch`\n\n### A Journey of Discovery\n\nThis project served as a profound learning experience, challenging my initial assumptions about feature engineering. The multi-class, grouped nature of the dataset forced a re-evaluation of my approach and opened my eyes to the vast, intricate landscape of data science. This competition was an invaluable \"first\" for me, highlighting the need to constantly evolve and think creatively to solve complex problems."
    },
    {
      "id": 3282151,
      "postDate": "2025-09-05T19:32:57Z",
      "content": "<p>Hi, in your write up:  In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.<br>\nHow do you find this, can you elaborate. Thanks.</p>",
      "rawMarkdown": "Hi, in your write up:  In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.\nHow do you find this, can you elaborate. Thanks.\n"
    },
    {
      "id": 3281848,
      "postDate": "2025-09-05T04:41:19.940Z",
      "content": "<p>Congrats on 2nd place!!!</p>",
      "rawMarkdown": "Congrats on 2nd place!!!"
    },
    {
      "id": 3281301,
      "postDate": "2025-09-04T03:03:27.450Z",
      "content": "<blockquote>\n  <p>Without post-processing, my model scores were: Public 0.865 / Private 0.858.</p>\n</blockquote>\n<p>Congrats. I think your models are really strong even without the post-processing!</p>\n<p>Which step do you think has the highest impact for your modelling approach? (arch, feature-engineering, handling nan, augmentation)</p>",
      "rawMarkdown": "> Without post-processing, my model scores were: Public 0.865 / Private 0.858.\n\nCongrats. I think your models are really strong even without the post-processing!\n\nWhich step do you think has the highest impact for your modelling approach? (arch, feature-engineering, handling nan, augmentation)"
    },
    {
      "id": 3281178,
      "postDate": "2025-09-03T18:45:11.830Z",
      "content": "<p>If each prediction is final and cannot be changed afterwards wouldn't that make early mistakes propagate? For example, if:</p>\n<ul>\n<li>In the first prediction the model incorrectly predicts sequence 1 as of class 1.</li>\n<li>Now, imagine that later (e.g. sequence 50) the model correctly predicts sequence 50 as of class 1. Would the algorithm not be able to assign it to class 1 anymore, because that label is already used?</li>\n</ul>\n<p>This would cause a double error.</p>\n<p>Perhaps I didn't understand exactly your post-processing approach.</p>\n<p>Congratz!</p>",
      "rawMarkdown": "If each prediction is final and cannot be changed afterwards wouldn't that make early mistakes propagate? For example, if:\n- In the first prediction the model incorrectly predicts sequence 1 as of class 1.\n- Now, imagine that later (e.g. sequence 50) the model correctly predicts sequence 50 as of class 1. Would the algorithm not be able to assign it to class 1 anymore, because that label is already used?\n\nThis would cause a double error.\n\nPerhaps I didn't understand exactly your post-processing approach.\n\nCongratz!\n",
      "replies": [
        {
          "id": 3281298,
          "postDate": "2025-09-04T02:58:57.547Z",
          "content": "<p>my interpretation is that there is always a chance of making double error, but \"maximizes the joint probability\" was superior than just taking \"argmax independently\"</p>",
          "rawMarkdown": "my interpretation is that there is always a chance of making double error, but \"maximizes the joint probability\" was superior than just taking \"argmax independently\""
        },
        {
          "id": 3281327,
          "postDate": "2025-09-04T04:03:17.680Z",
          "content": "<p>Thanks! You’re right that if a predicted probability is extremely close to 1, it’s effectively fixed—but my method doesn’t hard-lock past predictions.</p>\n<p>In your example, even if sequence 1 was initially assigned class 1, when I get to sequence 50 I recompute the best joint assignment for sequences 1–50 from scratch, under the no-repeat constraint, by maximizing the sum of log-probabilities. That means sequence 1 can be reassigned if it improves the overall joint likelihood, which can free up class 1 for sequence 50 if that’s the better global choice.</p>\n<p>So early mistakes don’t necessarily propagate: at each step, I re-optimize the joint assignment for all sequences seen so far. This outperforms per-sample argmax on the set {1…50} as a whole. Ultimately, I only predict sequence 50 at that step, but I can still improve its accuracy.</p>",
          "rawMarkdown": "Thanks! You’re right that if a predicted probability is extremely close to 1, it’s effectively fixed—but my method doesn’t hard-lock past predictions.\n\nIn your example, even if sequence 1 was initially assigned class 1, when I get to sequence 50 I recompute the best joint assignment for sequences 1–50 from scratch, under the no-repeat constraint, by maximizing the sum of log-probabilities. That means sequence 1 can be reassigned if it improves the overall joint likelihood, which can free up class 1 for sequence 50 if that’s the better global choice.\n\nSo early mistakes don’t necessarily propagate: at each step, I re-optimize the joint assignment for all sequences seen so far. This outperforms per-sample argmax on the set {1…50} as a whole. Ultimately, I only predict sequence 50 at that step, but I can still improve its accuracy.",
          "votes": 5
        }
      ]
    },
    {
      "id": 3280911,
      "postDate": "2025-09-03T10:56:53.757Z",
      "content": "<p>Perfect work! congrats!</p>",
      "rawMarkdown": "Perfect work! congrats!"
    },
    {
      "id": 3280873,
      "postDate": "2025-09-03T09:19:52.877Z",
      "content": "<p>Awsome! Congratulations!</p>",
      "rawMarkdown": "Awsome! Congratulations!"
    },
    {
      "id": 3280870,
      "postDate": "2025-09-03T09:17:52.323Z",
      "content": "<p>Great work! I tried to use LSTM for this task but the result was really poor.</p>",
      "rawMarkdown": "Great work! I tried to use LSTM for this task but the result was really poor."
    },
    {
      "id": 3280864,
      "postDate": "2025-09-03T09:04:25.220Z",
      "content": "<p>The post-processing method is really smart, it's just like(maybe slightly better than) add a transformer model to learn the sequence information!</p>",
      "rawMarkdown": "The post-processing method is really smart, it's just like(maybe slightly better than) add a transformer model to learn the sequence information!"
    },
    {
      "id": 3280863,
      "postDate": "2025-09-03T08:59:33.363Z",
      "content": "<p>Elegant - super cool!</p>",
      "rawMarkdown": "Elegant - super cool!"
    },
    {
      "id": 3281846,
      "postDate": "2025-09-05T04:40:43.710Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 3281189,
      "postDate": "2025-09-03T19:07:49.170Z",
      "content": "<p>I learnt a lot from this, thank you!</p>",
      "rawMarkdown": "I learnt a lot from this, thank you!"
    }
  ],
  "comments": [
    {
      "id": 3280879,
      "author_name": "Fnoa",
      "author_url": "",
      "post_date": "2025-09-03T09:38:06.183000",
      "content": "<p>I used a similar approach with the targets but excluded Initial behavior ∈ {Relaxes&amp;Moves / Moves}, resulting in 51 targets total. I'm curious about how much performance difference there would be using 102 targets instead of 51 - it would be interesting to see that comparison!<br>\nCongratulations on the incredible performance and innovative approach!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3280862,
      "author_name": "Theo Viel",
      "author_url": "",
      "post_date": "2025-09-03T08:59:31.617000",
      "content": "<p>Congratz on 2nd place !</p>\n<p>Your sudden jump in LB was one of the clues that helped us find the trick - given your background in optimization :)</p>\n<p>We figured part of it, but your approach is definitely better. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 3281042,
          "author_name": "Mauro Abidal Carrer",
          "author_url": "",
          "post_date": "2025-09-03T13:54:16.677000",
          "content": "<p>The meta of Kaggle competitions is wild lol.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3280852,
      "author_name": "EliKal",
      "author_url": "",
      "post_date": "2025-09-03T08:42:27.283000",
      "content": "<p>The idea of training 102 classes is wild!! Well deserved</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3282776,
          "author_name": "Ashwin Ashok",
          "author_url": "",
          "post_date": "2025-09-06T19:30:53.520000",
          "content": "<p>102 😲, damn!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3284806,
      "author_name": "Jason Wood",
      "author_url": "",
      "post_date": "2025-09-08T16:46:25.063000",
      "content": "<p>Congrats on the 2nd place! Really impressed by the subject-specific corrections; especially the 180° rotation fix; and the way you structured separate models for missing sensor cases. The phase-aware attention with the auxiliary phase predictor is a clever way to balance long vs. short phases, and predicting the composite label set up the post-processing beautifully. The Hungarian assignment trick on top of that was a brilliant use of dataset structure. Thanks for the detailed breakdown; lots of takeaways here!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3282363,
      "author_name": "Ashwin Ashok",
      "author_url": "",
      "post_date": "2025-09-06T07:31:02.993000",
      "content": "<h3>Congrats Bro!!!</h3>\n<p><strong>Issue:</strong> My code submission is failing with an error. I am unable to resolve this despite multiple attempts.</p>\n<p>This is my <strong>Code :</strong> <a href=\"https://github.com/AshwinAshok3/cmi-detect-behavior-with-sensor-data/blob/main/cmi-wrist-watch-sensors_final.ipynb\" target=\"_blank\">cmi-watch-sensors_final</a></p>\n<p>I was only able to catch <code>64</code><strong>%</strong> accuracy with <code>XGBoost</code> and <code>CatBoost</code> &amp; <code>62</code><strong>%</strong> with <code>Pytorch</code></p>\n<h3>A Journey of Discovery</h3>\n<p>This project served as a profound learning experience, challenging my initial assumptions about feature engineering. The multi-class, grouped nature of the dataset forced a re-evaluation of my approach and opened my eyes to the vast, intricate landscape of data science. This competition was an invaluable \"first\" for me, highlighting the need to constantly evolve and think creatively to solve complex problems.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3282151,
      "author_name": "Y. Xiao",
      "author_url": "",
      "post_date": "2025-09-05T19:32:57",
      "content": "<p>Hi, in your write up:  In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.<br>\nHow do you find this, can you elaborate. Thanks.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3281848,
      "author_name": "josiah",
      "author_url": "",
      "post_date": "2025-09-05T04:41:19.940000",
      "content": "<p>Congrats on 2nd place!!!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3281301,
      "author_name": "Gerwyn",
      "author_url": "",
      "post_date": "2025-09-04T03:03:27.450000",
      "content": "<blockquote>\n  <p>Without post-processing, my model scores were: Public 0.865 / Private 0.858.</p>\n</blockquote>\n<p>Congrats. I think your models are really strong even without the post-processing!</p>\n<p>Which step do you think has the highest impact for your modelling approach? (arch, feature-engineering, handling nan, augmentation)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3281178,
      "author_name": "moth",
      "author_url": "",
      "post_date": "2025-09-03T18:45:11.830000",
      "content": "<p>If each prediction is final and cannot be changed afterwards wouldn't that make early mistakes propagate? For example, if:</p>\n<ul>\n<li>In the first prediction the model incorrectly predicts sequence 1 as of class 1.</li>\n<li>Now, imagine that later (e.g. sequence 50) the model correctly predicts sequence 50 as of class 1. Would the algorithm not be able to assign it to class 1 anymore, because that label is already used?</li>\n</ul>\n<p>This would cause a double error.</p>\n<p>Perhaps I didn't understand exactly your post-processing approach.</p>\n<p>Congratz!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3281298,
          "author_name": "Gerwyn",
          "author_url": "",
          "post_date": "2025-09-04T02:58:57.547000",
          "content": "<p>my interpretation is that there is always a chance of making double error, but \"maximizes the joint probability\" was superior than just taking \"argmax independently\"</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 3281327,
          "author_name": "daiwakun",
          "author_url": "",
          "post_date": "2025-09-04T04:03:17.680000",
          "content": "<p>Thanks! You’re right that if a predicted probability is extremely close to 1, it’s effectively fixed—but my method doesn’t hard-lock past predictions.</p>\n<p>In your example, even if sequence 1 was initially assigned class 1, when I get to sequence 50 I recompute the best joint assignment for sequences 1–50 from scratch, under the no-repeat constraint, by maximizing the sum of log-probabilities. That means sequence 1 can be reassigned if it improves the overall joint likelihood, which can free up class 1 for sequence 50 if that’s the better global choice.</p>\n<p>So early mistakes don’t necessarily propagate: at each step, I re-optimize the joint assignment for all sequences seen so far. This outperforms per-sample argmax on the set {1…50} as a whole. Ultimately, I only predict sequence 50 at that step, but I can still improve its accuracy.</p>",
          "votes": 5,
          "replies": []
        }
      ]
    },
    {
      "id": 3280911,
      "author_name": "Kris Xu",
      "author_url": "",
      "post_date": "2025-09-03T10:56:53.757000",
      "content": "<p>Perfect work! congrats!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3280873,
      "author_name": "Artem Bakalinskiy",
      "author_url": "",
      "post_date": "2025-09-03T09:19:52.877000",
      "content": "<p>Awsome! Congratulations!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3280870,
      "author_name": "yuanzhe zhou",
      "author_url": "",
      "post_date": "2025-09-03T09:17:52.323000",
      "content": "<p>Great work! I tried to use LSTM for this task but the result was really poor.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3280864,
      "author_name": "hyd",
      "author_url": "",
      "post_date": "2025-09-03T09:04:25.220000",
      "content": "<p>The post-processing method is really smart, it's just like(maybe slightly better than) add a transformer model to learn the sequence information!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3280863,
      "author_name": "Konstantin Yakovlev",
      "author_url": "",
      "post_date": "2025-09-03T08:59:33.363000",
      "content": "<p>Elegant - super cool!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3281846,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-09-05T04:40:43.710000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3281189,
      "author_name": "Irakoze Ntawigenga Kelly",
      "author_url": "",
      "post_date": "2025-09-03T19:07:49.170000",
      "content": "<p>I learnt a lot from this, thank you!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3280846": "First, thanks to the organizers and hosts for an interesting competition, and to the community for insightful discussions and notebooks.\n\n# Overview\n- If you watched the leaderboard, you may have noticed a late jump: I discovered a dataset trick four days before the deadline and boosted the score substantially via post-processing.\n- Without post-processing, my model scores were: Public **0.865** / Private **0.858**.\n- Below I first describe the model (data handling, features, architecture, training), then the post-processing that made the big difference.\n\n# Model\n\n**Handling Missingness**\n- Training separate models gave better accuracy when certain sensors were missing.\n- I trained four variants depending on whether IMU rotation was missing and whether THM/TOF were missing (rot present/absent × THM and TOF present/absent).\n\n**Feature Engineering**\n\n- IMU\n - acc(x/y/z)\n - quaternion 6D representation (to avoid discontinuities)\n - angular velocity(x/y/z)\n - linear acceleration(x/y/z)\n\n- THM/TOF\n - Filled NaN and -1 with zero.\n - I did not use THM.\n\n- Left-handed subjects\n - Multiply by −1: acc_x, linear_acc_x, angular velocity y and z, and components 0 and 4 of the quaternion 6D representation.\n - Swap tof_3 and tof_5, and horizontally flip the left–right direction of the 2D TOF grids.\n\n- Subject-specific correction\n - In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.\n - I multiplied all channels by −1 except acc_z and linear_acc_z.\n - I could not find a reliable correction for their TOF, so I set TOF to NaN for these subjects.\n\n**Architecture**\n\n- Based on public notebooks: a Residual SE-CNN Block + Attention model.\n- Modality-specific stems:\n - Independent 1D CNN branches for each of: acc, quaternion 6D, angular velocity, linear_acc, and each tof1~5; then concatenate.\n- TOF processing:\n - Before 1D CNN branches, I use a 2D CNN on the TOF grids per frame, followed by mean pooling to get a per-step feature.\n- Phase-aware Attention\n - Plain temporal attention can over-focus on long phases (e.g., long transitions to the target).\n - I add an auxiliary 3-class phase predictor at each time step:\n  1. (Relaxes and) Moves hand to target location\n  2. Hand at target location\n  3. Performs gesture\n - I construct three attentions, one per phase, and weight each attention score by the corresponding phase probability. This extracts features specific to each phase instead of letting long phases dominate.\n\n**Prediction Target**\n- I predict a composite label:(Initial behavior ∈ {Relaxes&Moves / Moves}, orientation, gesture).\n- This choice is important for the post-processing step.\n\n**Mixup**\n- Simple linear mixup, but phase-aligned to avoid mixing behaviors:\n - Split each sequence into the three phases above and mix within the same phase only.\n - Especially for “moves to target,” I align the end of the phase so the arrival at the target is synchronized before mixing.\n\n**Training Details**\n- Optimizer: Adam (lr=1e-3, wd=1e-4)\n- Scheduler: Cosine Annealing\n- Batch size: 32\n- Epochs: 50\n- Folds: 10\n- Final model: ensemble of 3 variants (different depths/layers of 1DCNN)\n\n**Online Pseudo-Labeling**\n- With ~8k training sequences and ~3.5k test sequences, pseudo labels helped.\n- The submission API predicts one sequence at a time, so I performed test-time updates: whenever a batch-sized group of test sequences accumulated, I fine-tuned one step with a small LR (5e-5) using pseudo labels.\n\n# Post-Processing\n\n**Key observation from train.csv:**\n- There are 4 orientations × 18 gestures = 72 possible (orientation, gesture) pairs, but only 51 were actually present.\n- For each subject, the dataset contains exactly those 51 pairs, each recorded with two possible initial behaviors (Relaxes&Moves / Moves), totaling 51 × 2 = 102 sequences per subject.\n- Based on this, I trained the model to predict 102 classes (the composite label described above).\n\n**Maximizing Conditional Joint Probability (with no-repeat constraint)**\n- For each subject, I keep all predicted class probabilities across their sequences.\n- When predicting the N-th sequence for that subject, instead of taking the argmax for the N-th item independently, I choose the class sequence of length N that maximizes the joint probability (sum of log-probabilities) subject to “no label is used twice.”\n- Because labels are assigned jointly per subject under a no-repeat constraint, early high-confidence picks lock in their labels; when a later sample is only moderately confident in a label that’s already taken, an alternative is chosen to yield a higher joint log-likelihood.\n- Computationally, this is an assignment problem solvable efficiently (e.g., Hungarian algorithm) using negative log-probabilities as costs.\n\n# Scores\n| Method | Public LB | Private LB |\n| --- | ---: | ---: |\n| No pseudo / No post-proc | 0.862 | 0.854 |\n| With pseudo / No post-proc | 0.865 | 0.858 |\n| No pseudo / With post-proc | 0.891 | 0.875 |\n| With pseudo / With post-proc | 0.900 | 0.878 |\n\n# Codes\n[github](https://github.com/Yamato-Arai/kaggle-cmi-detect-behavior-2nd-place-solution)\n\n",
    "3280879": "I used a similar approach with the targets but excluded Initial behavior ∈ {Relaxes&Moves / Moves}, resulting in 51 targets total. I'm curious about how much performance difference there would be using 102 targets instead of 51 - it would be interesting to see that comparison!\nCongratulations on the incredible performance and innovative approach!",
    "3280862": "Congratz on 2nd place !\n\nYour sudden jump in LB was one of the clues that helped us find the trick - given your background in optimization :)\n\nWe figured part of it, but your approach is definitely better. ",
    "3280852": "The idea of training 102 classes is wild!! Well deserved",
    "3284806": "Congrats on the 2nd place! Really impressed by the subject-specific corrections; especially the 180° rotation fix; and the way you structured separate models for missing sensor cases. The phase-aware attention with the auxiliary phase predictor is a clever way to balance long vs. short phases, and predicting the composite label set up the post-processing beautifully. The Hungarian assignment trick on top of that was a brilliant use of dataset structure. Thanks for the detailed breakdown; lots of takeaways here!",
    "3282363": "### Congrats Bro!!!\n**Issue:** My code submission is failing with an error. I am unable to resolve this despite multiple attempts.\n\nThis is my **Code :** [cmi-watch-sensors_final](https://github.com/AshwinAshok3/cmi-detect-behavior-with-sensor-data/blob/main/cmi-wrist-watch-sensors_final.ipynb)\n\nI was only able to catch `64`**%** accuracy with `XGBoost` and `CatBoost` & `62`**%** with `Pytorch`\n\n### A Journey of Discovery\n\nThis project served as a profound learning experience, challenging my initial assumptions about feature engineering. The multi-class, grouped nature of the dataset forced a re-evaluation of my approach and opened my eyes to the vast, intricate landscape of data science. This competition was an invaluable \"first\" for me, highlighting the need to constantly evolve and think creatively to solve complex problems.",
    "3282151": "Hi, in your write up:  In the training data, data from SUBJ_019262 and SUBJ_045235 were rotated by 180° around the z-axis.\nHow do you find this, can you elaborate. Thanks.\n",
    "3281848": "Congrats on 2nd place!!!",
    "3281301": "> Without post-processing, my model scores were: Public 0.865 / Private 0.858.\n\nCongrats. I think your models are really strong even without the post-processing!\n\nWhich step do you think has the highest impact for your modelling approach? (arch, feature-engineering, handling nan, augmentation)",
    "3281178": "If each prediction is final and cannot be changed afterwards wouldn't that make early mistakes propagate? For example, if:\n- In the first prediction the model incorrectly predicts sequence 1 as of class 1.\n- Now, imagine that later (e.g. sequence 50) the model correctly predicts sequence 50 as of class 1. Would the algorithm not be able to assign it to class 1 anymore, because that label is already used?\n\nThis would cause a double error.\n\nPerhaps I didn't understand exactly your post-processing approach.\n\nCongratz!\n",
    "3280911": "Perfect work! congrats!",
    "3280873": "Awsome! Congratulations!",
    "3280870": "Great work! I tried to use LSTM for this task but the result was really poor.",
    "3280864": "The post-processing method is really smart, it's just like(maybe slightly better than) add a transformer model to learn the sequence information!",
    "3280863": "Elegant - super cool!",
    "3281846": "",
    "3281189": "I learnt a lot from this, thank you!"
  },
  "complete": true,
  "source": "existing_discussion",
  "competition_id": 102335,
  "competition_slug": "cmi-detect-behavior-with-sensor-data",
  "team_id": 14077688,
  "solution_url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/2nd-place-solution"
}