I’d like to ask a practical question: Why do my attempts at using more complex models often result in worse performance than simple models in competitions?
\nFor example, in this competition, I tried a 2-layer CNN + 1-layer GRU. Whenever I increased the depth or switched to a Transformer, the performance either degraded or remained the same compared to a simpler baseline. I’ve encountered similar situations in past competitions as well—it feels like my features can't adapt to the complexity of these advanced models.
\nWhat could be the main reasons behind this? Has anyone else experienced this, and how did you address it?
", "messages": [ { "id": "3377289", "postDate": "12/16/2025 03:01:45", "content": "I’d like to ask a practical question: Why do my attempts at using more complex models often result in worse performance than simple models in competitions?
\nFor example, in this competition, I tried a 2-layer CNN + 1-layer GRU. Whenever I increased the depth or switched to a Transformer, the performance either degraded or remained the same compared to a simpler baseline. I’ve encountered similar situations in past competitions as well—it feels like my features can't adapt to the complexity of these advanced models.
\nWhat could be the main reasons behind this? Has anyone else experienced this, and how did you address it?
", "rawMarkdown": "I’d like to ask a practical question: Why do my attempts at using more complex models often result in worse performance than simple models in competitions?\n\nFor example, in this competition, I tried a 2-layer CNN + 1-layer GRU. Whenever I increased the depth or switched to a Transformer, the performance either degraded or remained the same compared to a simpler baseline. I’ve encountered similar situations in past competitions as well—it feels like my features can't adapt to the complexity of these advanced models.\n\nWhat could be the main reasons behind this? Has anyone else experienced this, and how did you address it?", "votes": null }, { "id": "3377301", "postDate": "12/16/2025 03:38:34", "content": "A complex model is smarter, so to successfully train a complex model and prevent overfitting, we need to make it harder for the model to learn the train data. The first thing we can do is data augmentation. This makes all the data different each epoch, so the model cannot memorize it. Second, if the data is a bunch of small groups like this comp had small data for each lab, then we transform the data so it all looks similar. Then instead of having lots of small data which the model can overfit, it has 1 group of large data which is more difficult to memorize. A third trick is auxiliary\n learning. We add an additional head which predicts an additional target. Then during inference, we just ignore this additional target. Requiring the model to multi-task during training makes it harder for the model.
", "rawMarkdown": "A complex model is smarter, so to successfully train a complex model and prevent overfitting, we need to make it harder for the model to learn the train data. The first thing we can do is data augmentation. This makes all the data different each epoch, so the model cannot memorize it. Second, if the data is a bunch of small groups like this comp had small data for each lab, then we transform the data so it all looks similar. Then instead of having lots of small data which the model can overfit, it has 1 group of large data which is more difficult to memorize. A third trick is auxiliary\n learning. We add an additional head which predicts an additional target. Then during inference, we just ignore this additional target. Requiring the model to multi-task during training makes it harder for the model.", "votes": null }, { "id": "3377309", "postDate": "12/16/2025 04:00:20", "content": "I found model architecture not worth spending too much effort tweaking from recent competitions. The goal is to learn y given X, any sufficiently good model architecture (there are many) can get the job done. From reading the shared top solutions, I can see what I missed are things like preprocessing (invariant agent-centric coordinate system, augmentation did not work for me probably because I simply used standard scaling without adjusting for arenas), and overall design of the training (combining both self and pair actions), etc.
", "rawMarkdown": "I found model architecture not worth spending too much effort tweaking from recent competitions. The goal is to learn y given X, any sufficiently good model architecture (there are many) can get the job done. From reading the shared top solutions, I can see what I missed are things like preprocessing (invariant agent-centric coordinate system, augmentation did not work for me probably because I simply used standard scaling without adjusting for arenas), and overall design of the training (combining both self and pair actions), etc.", "votes": null }, { "id": "3377313", "postDate": "12/16/2025 04:10:10", "content": "Hi Xz, congrats winning bronze medal. Regarding data augmentation it works better on the invariant features. If we scale the original x and y coordinates then the mouse will move to a new position, but if we scale agent centric coordinates then the mouse will simply change size (and still stay centered at 0,0). Similarly, if we do horizontal flip, then the mouse will move to the other side of the arena where it may normally do different actions. But if we do horizontal flip on agent centric mouse, then is just flips the mouse (and mouse stays at 0,0). Same applies for lots of other data augmentaions and same applies for lots of other invariant position feature transformations besides agent centric coordinates.
", "rawMarkdown": "Hi Xz, congrats winning bronze medal. Regarding data augmentation it works better on the invariant features. If we scale the original x and y coordinates then the mouse will move to a new position, but if we scale agent centric coordinates then the mouse will simply change size (and still stay centered at 0,0). Similarly, if we do horizontal flip, then the mouse will move to the other side of the arena where it may normally do different actions. But if we do horizontal flip on agent centric mouse, then is just flips the mouse (and mouse stays at 0,0). Same applies for lots of other data augmentaions and same applies for lots of other invariant position feature transformations besides agent centric coordinates.", "votes": null }, { "id": "3377336", "postDate": "12/16/2025 05:38:19", "content": "Hi Chris, thanks for sharing your insights. I really like your point about making it harder for the model to memorize.
", "rawMarkdown": "Hi Chris, thanks for sharing your insights. I really like your point about making it harder for the model to memorize.", "votes": null } ], "comments": [ { "id": 3377301, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 03:38:34", "content": "A complex model is smarter, so to successfully train a complex model and prevent overfitting, we need to make it harder for the model to learn the train data. The first thing we can do is data augmentation. This makes all the data different each epoch, so the model cannot memorize it. Second, if the data is a bunch of small groups like this comp had small data for each lab, then we transform the data so it all looks similar. Then instead of having lots of small data which the model can overfit, it has 1 group of large data which is more difficult to memorize. A third trick is auxiliary\n learning. We add an additional head which predicts an additional target. Then during inference, we just ignore this additional target. Requiring the model to multi-task during training makes it harder for the model.
", "votes": null, "replies": [] }, { "id": 3377309, "author_name": "xiaoz259", "author_url": "", "post_date": "12/16/2025 04:00:20", "content": "I found model architecture not worth spending too much effort tweaking from recent competitions. The goal is to learn y given X, any sufficiently good model architecture (there are many) can get the job done. From reading the shared top solutions, I can see what I missed are things like preprocessing (invariant agent-centric coordinate system, augmentation did not work for me probably because I simply used standard scaling without adjusting for arenas), and overall design of the training (combining both self and pair actions), etc.
", "votes": null, "replies": [ { "id": 3377313, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 04:10:10", "content": "Hi Xz, congrats winning bronze medal. Regarding data augmentation it works better on the invariant features. If we scale the original x and y coordinates then the mouse will move to a new position, but if we scale agent centric coordinates then the mouse will simply change size (and still stay centered at 0,0). Similarly, if we do horizontal flip, then the mouse will move to the other side of the arena where it may normally do different actions. But if we do horizontal flip on agent centric mouse, then is just flips the mouse (and mouse stays at 0,0). Same applies for lots of other data augmentaions and same applies for lots of other invariant position feature transformations besides agent centric coordinates.
", "votes": null, "replies": [ { "id": 3377336, "author_name": "xiaoz259", "author_url": "", "post_date": "12/16/2025 05:38:19", "content": "Hi Chris, thanks for sharing your insights. I really like your point about making it harder for the model to memorize.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3377289": "I’d like to ask a practical question: Why do my attempts at using more complex models often result in worse performance than simple models in competitions?\n\nFor example, in this competition, I tried a 2-layer CNN + 1-layer GRU. Whenever I increased the depth or switched to a Transformer, the performance either degraded or remained the same compared to a simpler baseline. I’ve encountered similar situations in past competitions as well—it feels like my features can't adapt to the complexity of these advanced models.\n\nWhat could be the main reasons behind this? Has anyone else experienced this, and how did you address it?", "3377301": "A complex model is smarter, so to successfully train a complex model and prevent overfitting, we need to make it harder for the model to learn the train data. The first thing we can do is data augmentation. This makes all the data different each epoch, so the model cannot memorize it. Second, if the data is a bunch of small groups like this comp had small data for each lab, then we transform the data so it all looks similar. Then instead of having lots of small data which the model can overfit, it has 1 group of large data which is more difficult to memorize. A third trick is auxiliary\n learning. We add an additional head which predicts an additional target. Then during inference, we just ignore this additional target. Requiring the model to multi-task during training makes it harder for the model.", "3377309": "I found model architecture not worth spending too much effort tweaking from recent competitions. The goal is to learn y given X, any sufficiently good model architecture (there are many) can get the job done. From reading the shared top solutions, I can see what I missed are things like preprocessing (invariant agent-centric coordinate system, augmentation did not work for me probably because I simply used standard scaling without adjusting for arenas), and overall design of the training (combining both self and pair actions), etc.", "3377313": "Hi Xz, congrats winning bronze medal. Regarding data augmentation it works better on the invariant features. If we scale the original x and y coordinates then the mouse will move to a new position, but if we scale agent centric coordinates then the mouse will simply change size (and still stay centered at 0,0). Similarly, if we do horizontal flip, then the mouse will move to the other side of the arena where it may normally do different actions. But if we do horizontal flip on agent centric mouse, then is just flips the mouse (and mouse stays at 0,0). Same applies for lots of other data augmentaions and same applies for lots of other invariant position feature transformations besides agent centric coordinates.", "3377336": "Hi Chris, thanks for sharing your insights. I really like your point about making it harder for the model to memorize." }, "source": "meta" }