As for this competition, the non-target behaviors are all collapsed into 1 class into the final evaluation. So I think it's non-relevant for the model to predict the other 10 non-target behaviors. I tried
\nThe result is predicting 18 classes are slightly better.
\nOriginally I thought reducing classes help the model to focus more on the relevant targets, but it seems the other way around. What do you think?
As for this competition, the non-target behaviors are all collapsed into 1 class into the final evaluation. So I think it's non-relevant for the model to predict the other 10 non-target behaviors. I tried
\nThe result is predicting 18 classes are slightly better.
\nOriginally I thought reducing classes help the model to focus more on the relevant targets, but it seems the other way around. What do you think?
I also got similar results. My hypothesis is that grouping unrelated non-target classes together \"confuses\" the model.
\nFor example, Wave hello and Write name on leg have very different sensor data, so grouping them into one label makes the task unnecessarily harder, lowering the score.
Collapsing all non-targets into a single class can create issues when selecting the final prediction.
\nFor example, consider the following classes: [target1, target2, non-target] with associated probabilities: [0.3, 0.3, 0.4]
\nIf we simply choose the class with the highest probability as our prediction (as is commonly done in public notebooks), we would select non-target. This would be correct choice in terms of the macro-F1 component of evaluation metric, but not for the binary-F1 as P(target1 or target2) > P(non-target)
\n(This is not to say using all 18 classes + picking the highest probability class is the optimal way either, it's just less wrong)
", "rawMarkdown": "Collapsing all non-targets into a single class can create issues when selecting the final prediction.\n\nFor example, consider the following classes: [target1, target2, non-target] with associated probabilities: [0.3, 0.3, 0.4]\n\nIf we simply choose the class with the highest probability as our prediction (as is commonly done in public notebooks), we would select non-target. This would be correct choice in terms of the macro-F1 component of evaluation metric, but not for the binary-F1 as P(target1 or target2) > P(non-target)\n\n(This is not to say using all 18 classes + picking the highest probability class is the optimal way either, it's just less wrong)", "votes": null }, { "id": "3224798", "postDate": "06/15/2025 13:58:19", "content": "In my opinion, the 18 classes allow better supervision of the model during training, helping it to learn features that allow non-targets to be discriminated between. Although this ability is not the goal of this competition, these features can still help discriminate targets.
", "rawMarkdown": "In my opinion, the 18 classes allow better supervision of the model during training, helping it to learn features that allow non-targets to be discriminated between. Although this ability is not the goal of this competition, these features can still help discriminate targets.", "votes": null }, { "id": "3224840", "postDate": "06/15/2025 14:54:20", "content": "There must exists a way to handle those classes, I still think 18 classes is too fined-grained. Maybe predict the classes hierarchically?
", "rawMarkdown": "There must exists a way to handle those classes, I still think 18 classes is too fined-grained. Maybe predict the classes hierarchically?", "votes": null }, { "id": "3225188", "postDate": "06/16/2025 04:46:30", "content": "I agree, this makes the model harder to distinguish the non-target
", "rawMarkdown": "I agree, this makes the model harder to distinguish the non-target", "votes": null }, { "id": "3225189", "postDate": "06/16/2025 04:47:37", "content": "I think maybe some of the non-target behaviours can be grouped into one to make it less granular?
", "rawMarkdown": "I think maybe some of the non-target behaviours can be grouped into one to make it less granular?", "votes": null }, { "id": "3225203", "postDate": "06/16/2025 05:20:59", "content": "Using gbm multiclass:
\nMore on (2.) I observed that my model usually mis-classify \"similar\" non-target gesture, like Wave hello <-> Write name in air and Write name on leg <-> Scratch knee/leg skin <-> Pinch knee/leg skin. But i had no luck trying to group these gestures in one way or another.
Does anyone know how to fix this problem? I have created the model to predict non-target as one class but it doesn't allow me the submission due to this error:
\nGatewayRuntimeError: (, 'All gestures must match those found in the train data. Got non_target')
", "rawMarkdown": "Does anyone know how to fix this problem? I have created the model to predict non-target as one class but it doesn't allow me the submission due to this error:\n\nGatewayRuntimeError: (Hi @tahaheidari,
\nYour predict function needs to return one of the gesture labels specified by the competition host. It seems you're returning 'non_target' as a class, but this exact string isn't in the approved list. Try replacing it with an actual non-target gesture name.
I'm not sure which objective is best, so I use a weighted sum of three losses: binary classification (1 class), 9-class classification, and 18-class classification. Although the competition metric only uses binary and 9-class outputs, there's historical evidence that including auxiliary losses—like the 18-class task—can help guide the model toward a better local minimum in the loss landscape, improving generalization.
", "rawMarkdown": "I'm not sure which objective is best, so I use a weighted sum of three losses: binary classification (1 class), 9-class classification, and 18-class classification. Although the competition metric only uses binary and 9-class outputs, there's historical evidence that including auxiliary losses—like the 18-class task—can help guide the model toward a better local minimum in the loss landscape, improving generalization.", "votes": null } ], "comments": [ { "id": 3224635, "author_name": "howarudo", "author_url": "", "post_date": "06/15/2025 08:14:32", "content": "I also got similar results. My hypothesis is that grouping unrelated non-target classes together \"confuses\" the model.
\nFor example, Wave hello and Write name on leg have very different sensor data, so grouping them into one label makes the task unnecessarily harder, lowering the score.
I agree, this makes the model harder to distinguish the non-target
", "votes": null, "replies": [] } ] }, { "id": 3224707, "author_name": "herrahuu", "author_url": "", "post_date": "06/15/2025 11:01:30", "content": "Collapsing all non-targets into a single class can create issues when selecting the final prediction.
\nFor example, consider the following classes: [target1, target2, non-target] with associated probabilities: [0.3, 0.3, 0.4]
\nIf we simply choose the class with the highest probability as our prediction (as is commonly done in public notebooks), we would select non-target. This would be correct choice in terms of the macro-F1 component of evaluation metric, but not for the binary-F1 as P(target1 or target2) > P(non-target)
\n(This is not to say using all 18 classes + picking the highest probability class is the optimal way either, it's just less wrong)
", "votes": null, "replies": [] }, { "id": 3224798, "author_name": "thomasdubail", "author_url": "", "post_date": "06/15/2025 13:58:19", "content": "In my opinion, the 18 classes allow better supervision of the model during training, helping it to learn features that allow non-targets to be discriminated between. Although this ability is not the goal of this competition, these features can still help discriminate targets.
", "votes": null, "replies": [] }, { "id": 3224840, "author_name": "tom99763", "author_url": "", "post_date": "06/15/2025 14:54:20", "content": "There must exists a way to handle those classes, I still think 18 classes is too fined-grained. Maybe predict the classes hierarchically?
", "votes": null, "replies": [ { "id": 3225189, "author_name": "zhangyue199", "author_url": "", "post_date": "06/16/2025 04:47:37", "content": "I think maybe some of the non-target behaviours can be grouped into one to make it less granular?
", "votes": null, "replies": [] } ] }, { "id": 3225203, "author_name": "gerwynng", "author_url": "", "post_date": "06/16/2025 05:20:59", "content": "Using gbm multiclass:
\nMore on (2.) I observed that my model usually mis-classify \"similar\" non-target gesture, like Wave hello <-> Write name in air and Write name on leg <-> Scratch knee/leg skin <-> Pinch knee/leg skin. But i had no luck trying to group these gestures in one way or another.
Does anyone know how to fix this problem? I have created the model to predict non-target as one class but it doesn't allow me the submission due to this error:
\nGatewayRuntimeError: (, 'All gestures must match those found in the train data. Got non_target')
", "votes": null, "replies": [ { "id": 3225768, "author_name": "thomasdubail", "author_url": "", "post_date": "06/16/2025 20:20:32", "content": "Hi @tahaheidari,
\nYour predict function needs to return one of the gesture labels specified by the competition host. It seems you're returning 'non_target' as a class, but this exact string isn't in the approved list. Try replacing it with an actual non-target gesture name.
I'm not sure which objective is best, so I use a weighted sum of three losses: binary classification (1 class), 9-class classification, and 18-class classification. Although the competition metric only uses binary and 9-class outputs, there's historical evidence that including auxiliary losses—like the 18-class task—can help guide the model toward a better local minimum in the loss landscape, improving generalization.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3224628": "As for this competition, the non-target behaviors are all collapsed into 1 class into the final evaluation. So I think it's non-relevant for the model to predict the other 10 non-target behaviors. I tried\n\n- Build a baseline model to predict all 18 classes\n- Build a varnat model to predict 9 classes, where non-target behavior are encoded to -1.\n\nThe result is predicting 18 classes are slightly better. \nOriginally I thought reducing classes help the model to focus more on the relevant targets, but it seems the other way around. What do you think?", "3224635": "I also got similar results. My hypothesis is that grouping unrelated non-target classes together \"confuses\" the model. \nFor example, `Wave hello` and `Write name on leg` have very different sensor data, so grouping them into one label makes the task unnecessarily harder, lowering the score.", "3224707": "Collapsing all non-targets into a single class can create issues when selecting the final prediction.\n\nFor example, consider the following classes: [target1, target2, non-target] with associated probabilities: [0.3, 0.3, 0.4]\n\nIf we simply choose the class with the highest probability as our prediction (as is commonly done in public notebooks), we would select non-target. This would be correct choice in terms of the macro-F1 component of evaluation metric, but not for the binary-F1 as P(target1 or target2) > P(non-target)\n\n(This is not to say using all 18 classes + picking the highest probability class is the optimal way either, it's just less wrong)", "3224798": "In my opinion, the 18 classes allow better supervision of the model during training, helping it to learn features that allow non-targets to be discriminated between. Although this ability is not the goal of this competition, these features can still help discriminate targets.", "3224840": "There must exists a way to handle those classes, I still think 18 classes is too fined-grained. Maybe predict the classes hierarchically?", "3225188": "I agree, this makes the model harder to distinguish the non-target", "3225189": "I think maybe some of the non-target behaviours can be grouped into one to make it less granular?", "3225203": "Using gbm multiclass:\n1. Collapse all \"non-target\" into one class does not work for me\n2. Manually merging some \"similar\" non-target gesture also does not work for me.\n\nMore on (2.) I observed that my model usually mis-classify \"similar\" non-target gesture, like `Wave hello` <-> `Write name in air` and `Write name on leg` <-> `Scratch knee/leg skin` <-> `Pinch knee/leg skin`. But i had no luck trying to group these gestures in one way or another.", "3225668": "Does anyone know how to fix this problem? I have created the model to predict non-target as one class but it doesn't allow me the submission due to this error:\n\nGatewayRuntimeError: (