{ "id": 603544, "title": "Two useful oberservations about the data", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544", "author_name": "", "post_date": "2025-09-03T00:22:27.758939700Z", "votes": 32, "comment_count": 13, "views": 0, "content": "

I'm sure people will post more detail in their writeups but one of the simple things we noticed in the data was that each subject only did a given gesture twice per orientation. Using this knowledge we were able to draft up some postprocessing that would keep this constraint in mind. If we had global knowledge of all predictions we could very easily just sort based on most confident 2 predictions for each class in each orientation and get a massive boost in performance. Because of how the submissions were structured though we could not exploit the global information. Instead, we had to do it in streaming fashion. Determining one at a time and not able to reverse previous predictions. In this way we got a significantly smaller gain, but still measurable and consistent.

\n

The other piece to this was that 21 of the gesture orientation combos were never observed in the data. Certain gestures were never done in certain orientations. With this knowledge we could simply turn off certain combos, forcing redirection to valid options. Again, this was not common, but it gave a measurable and consistent boost.

\n

Looking forward to hearing what everyone else did

", "messages": [ { "id": "3280616", "postDate": "09/03/2025 00:22:27", "content": "

I'm sure people will post more detail in their writeups but one of the simple things we noticed in the data was that each subject only did a given gesture twice per orientation. Using this knowledge we were able to draft up some postprocessing that would keep this constraint in mind. If we had global knowledge of all predictions we could very easily just sort based on most confident 2 predictions for each class in each orientation and get a massive boost in performance. Because of how the submissions were structured though we could not exploit the global information. Instead, we had to do it in streaming fashion. Determining one at a time and not able to reverse previous predictions. In this way we got a significantly smaller gain, but still measurable and consistent.

\n

The other piece to this was that 21 of the gesture orientation combos were never observed in the data. Certain gestures were never done in certain orientations. With this knowledge we could simply turn off certain combos, forcing redirection to valid options. Again, this was not common, but it gave a measurable and consistent boost.

\n

Looking forward to hearing what everyone else did

", "rawMarkdown": "I'm sure people will post more detail in their writeups but one of the simple things we noticed in the data was that each subject only did a given gesture twice per orientation. Using this knowledge we were able to draft up some postprocessing that would keep this constraint in mind. If we had global knowledge of all predictions we could very easily just sort based on most confident 2 predictions for each class in each orientation and get a massive boost in performance. Because of how the submissions were structured though we could not exploit the global information. Instead, we had to do it in streaming fashion. Determining one at a time and not able to reverse previous predictions. In this way we got a significantly smaller gain, but still measurable and consistent. \n\nThe other piece to this was that 21 of the gesture orientation combos were never observed in the data. Certain gestures were never done in certain orientations. With this knowledge we could simply turn off certain combos, forcing redirection to valid options. Again, this was not common, but it gave a measurable and consistent boost. \n\nLooking forward to hearing what everyone else did", "votes": null }, { "id": "3280629", "postDate": "09/03/2025 00:47:10", "content": "

Organizers should have randomized the subject ids to avoid exploit… it seemed a reasonable and simple thing to do if you decide to make an api for sequence level submission…

", "rawMarkdown": "Organizers should have randomized the subject ids to avoid exploit… it seemed a reasonable and simple thing to do if you decide to make an api for sequence level submission…", "votes": null }, { "id": "3280631", "postDate": "09/03/2025 00:50:36", "content": "

Yeah, it seems like that was the intention. I dont know why else they would feed us the sequences one at a time. I only really got tipped off to this idea because some comments from @devinanzelmo made it sound like there was something that could be done to probe information out.

", "rawMarkdown": "Yeah, it seems like that was the intention. I dont know why else they would feed us the sequences one at a time. I only really got tipped off to this idea because some comments from @devinanzelmo made it sound like there was something that could be done to probe information out.", "votes": null }, { "id": "3280636", "postDate": "09/03/2025 00:58:41", "content": "

Yes I remember that comments :)

\n

How much of an improvement are we talking about ? for public and private ?

\n

This seemed a bit risky too since the api order is random.

", "rawMarkdown": "Yes I remember that comments :)\n\nHow much of an improvement are we talking about ? for public and private ?\n\nThis seemed a bit risky too since the api order is random.", "votes": null }, { "id": "3280637", "postDate": "09/03/2025 01:01:52", "content": "

For us it was like this

\n

IMU: 0.8436
\nIMU Postprocessed: 0.8548

\n

Full:0.9023
\nFull Postprocessed:0.9140

", "rawMarkdown": "For us it was like this\n\nIMU: 0.8436\nIMU Postprocessed: 0.8548\n\nFull:0.9023\nFull Postprocessed:0.9140", "votes": null }, { "id": "3280638", "postDate": "09/03/2025 01:02:31", "content": "

We trusted it because validation and leaderboard were nearly perfectly aligned and we tried out random shuffling and the variation was quite low

", "rawMarkdown": "We trusted it because validation and leaderboard were nearly perfectly aligned and we tried out random shuffling and the variation was quite low", "votes": null }, { "id": "3280640", "postDate": "09/03/2025 01:04:52", "content": "

Wow big boost! Congrats!

", "rawMarkdown": "Wow big boost! Congrats!", "votes": null }, { "id": "3280654", "postDate": "09/03/2025 01:57:01", "content": "

How did you use orientation data in your model implementation when test set lacked them? If possible, could you share more about your approach in this challenge?

", "rawMarkdown": "How did you use orientation data in your model implementation when test set lacked them? If possible, could you share more about your approach in this challenge?", "votes": null }, { "id": "3280663", "postDate": "09/03/2025 02:22:26", "content": "

Our models predicted simultaneously the gesture and orientation off of different heads. The models were extremely good at predicting the orientation so it was nearly as good as just being directly given it

", "rawMarkdown": "Our models predicted simultaneously the gesture and orientation off of different heads. The models were extremely good at predicting the orientation so it was nearly as good as just being directly given it", "votes": null }, { "id": "3280670", "postDate": "09/03/2025 02:38:36", "content": "

Ohhh. That makes sense.

", "rawMarkdown": "Ohhh. That makes sense.", "votes": null }, { "id": "3280754", "postDate": "09/03/2025 05:31:52", "content": "

Seems orientation made it more stable then. For me the range was 10+ pts and I selected unlucky 😅

", "rawMarkdown": "Seems orientation made it more stable then. For me the range was 10+ pts and I selected unlucky 😅", "votes": null }, { "id": "3280755", "postDate": "09/03/2025 05:33:59", "content": "

I found that the best models for gesture prediction were not the same for orientation. For gesture it seemed to be mainly local patterns but with orientation the global was what mattered so transformers seemed to smoke cnns in that regard.

", "rawMarkdown": "I found that the best models for gesture prediction were not the same for orientation. For gesture it seemed to be mainly local patterns but with orientation the global was what mattered so transformers seemed to smoke cnns in that regard.", "votes": null }, { "id": "3280806", "postDate": "09/03/2025 07:04:18", "content": "
\n

Organizers should have randomized the subject ids to avoid exploit

\n
\n

Useless since we can use dmographic data to match subjects. They will have to get rid of most dmo data (imo can get rid of all except handedness).

", "rawMarkdown": ">Organizers should have randomized the subject ids to avoid exploit \n\nUseless since we can use dmographic data to match subjects. They will have to get rid of most dmo data (imo can get rid of all except handedness).", "votes": null }, { "id": "3280826", "postDate": "09/03/2025 07:42:44", "content": "

You are right, I forgot about arm length, which no one used I guess.

", "rawMarkdown": "You are right, I forgot about arm length, which no one used I guess.", "votes": null } ], "comments": [ { "id": 3280629, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 00:47:10", "content": "

Organizers should have randomized the subject ids to avoid exploit… it seemed a reasonable and simple thing to do if you decide to make an api for sequence level submission…

", "votes": null, "replies": [ { "id": 3280631, "author_name": "ryches", "author_url": "", "post_date": "09/03/2025 00:50:36", "content": "

Yeah, it seems like that was the intention. I dont know why else they would feed us the sequences one at a time. I only really got tipped off to this idea because some comments from @devinanzelmo made it sound like there was something that could be done to probe information out.

", "votes": null, "replies": [ { "id": 3280636, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 00:58:41", "content": "

Yes I remember that comments :)

\n

How much of an improvement are we talking about ? for public and private ?

\n

This seemed a bit risky too since the api order is random.

", "votes": null, "replies": [ { "id": 3280637, "author_name": "ryches", "author_url": "", "post_date": "09/03/2025 01:01:52", "content": "

For us it was like this

\n

IMU: 0.8436
\nIMU Postprocessed: 0.8548

\n

Full:0.9023
\nFull Postprocessed:0.9140

", "votes": null, "replies": [] }, { "id": 3280638, "author_name": "ryches", "author_url": "", "post_date": "09/03/2025 01:02:31", "content": "

We trusted it because validation and leaderboard were nearly perfectly aligned and we tried out random shuffling and the variation was quite low

", "votes": null, "replies": [ { "id": 3280640, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 01:04:52", "content": "

Wow big boost! Congrats!

", "votes": null, "replies": [] }, { "id": 3280754, "author_name": "philippsinger", "author_url": "", "post_date": "09/03/2025 05:31:52", "content": "

Seems orientation made it more stable then. For me the range was 10+ pts and I selected unlucky 😅

", "votes": null, "replies": [ { "id": 3280755, "author_name": "ryches", "author_url": "", "post_date": "09/03/2025 05:33:59", "content": "

I found that the best models for gesture prediction were not the same for orientation. For gesture it seemed to be mainly local patterns but with orientation the global was what mattered so transformers seemed to smoke cnns in that regard.

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3280806, "author_name": "shlomoron", "author_url": "", "post_date": "09/03/2025 07:04:18", "content": "
\n

Organizers should have randomized the subject ids to avoid exploit

\n
\n

Useless since we can use dmographic data to match subjects. They will have to get rid of most dmo data (imo can get rid of all except handedness).

", "votes": null, "replies": [ { "id": 3280826, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 07:42:44", "content": "

You are right, I forgot about arm length, which no one used I guess.

", "votes": null, "replies": [] } ] } ] }, { "id": 3280654, "author_name": "ojassapkota", "author_url": "", "post_date": "09/03/2025 01:57:01", "content": "

How did you use orientation data in your model implementation when test set lacked them? If possible, could you share more about your approach in this challenge?

", "votes": null, "replies": [ { "id": 3280663, "author_name": "ryches", "author_url": "", "post_date": "09/03/2025 02:22:26", "content": "

Our models predicted simultaneously the gesture and orientation off of different heads. The models were extremely good at predicting the orientation so it was nearly as good as just being directly given it

", "votes": null, "replies": [ { "id": 3280670, "author_name": "ojassapkota", "author_url": "", "post_date": "09/03/2025 02:38:36", "content": "

Ohhh. That makes sense.

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3280616": "I'm sure people will post more detail in their writeups but one of the simple things we noticed in the data was that each subject only did a given gesture twice per orientation. Using this knowledge we were able to draft up some postprocessing that would keep this constraint in mind. If we had global knowledge of all predictions we could very easily just sort based on most confident 2 predictions for each class in each orientation and get a massive boost in performance. Because of how the submissions were structured though we could not exploit the global information. Instead, we had to do it in streaming fashion. Determining one at a time and not able to reverse previous predictions. In this way we got a significantly smaller gain, but still measurable and consistent. \n\nThe other piece to this was that 21 of the gesture orientation combos were never observed in the data. Certain gestures were never done in certain orientations. With this knowledge we could simply turn off certain combos, forcing redirection to valid options. Again, this was not common, but it gave a measurable and consistent boost. \n\nLooking forward to hearing what everyone else did", "3280629": "Organizers should have randomized the subject ids to avoid exploit… it seemed a reasonable and simple thing to do if you decide to make an api for sequence level submission…", "3280631": "Yeah, it seems like that was the intention. I dont know why else they would feed us the sequences one at a time. I only really got tipped off to this idea because some comments from @devinanzelmo made it sound like there was something that could be done to probe information out.", "3280636": "Yes I remember that comments :)\n\nHow much of an improvement are we talking about ? for public and private ?\n\nThis seemed a bit risky too since the api order is random.", "3280637": "For us it was like this\n\nIMU: 0.8436\nIMU Postprocessed: 0.8548\n\nFull:0.9023\nFull Postprocessed:0.9140", "3280638": "We trusted it because validation and leaderboard were nearly perfectly aligned and we tried out random shuffling and the variation was quite low", "3280640": "Wow big boost! Congrats!", "3280654": "How did you use orientation data in your model implementation when test set lacked them? If possible, could you share more about your approach in this challenge?", "3280663": "Our models predicted simultaneously the gesture and orientation off of different heads. The models were extremely good at predicting the orientation so it was nearly as good as just being directly given it", "3280670": "Ohhh. That makes sense.", "3280754": "Seems orientation made it more stable then. For me the range was 10+ pts and I selected unlucky 😅", "3280755": "I found that the best models for gesture prediction were not the same for orientation. For gesture it seemed to be mainly local patterns but with orientation the global was what mattered so transformers seemed to smoke cnns in that regard.", "3280806": ">Organizers should have randomized the subject ids to avoid exploit \n\nUseless since we can use dmographic data to match subjects. They will have to get rid of most dmo data (imo can get rid of all except handedness).", "3280826": "You are right, I forgot about arm length, which no one used I guess." }, "source": "meta" }