Hi everyone, I'm currently a bit stuck with architecture iteration and finding it difficult to effectively incorporate classic temporal models like GRU/LSTM. It seems that adding any additional module to my current 1D-CNN architecture actually reduces performance on both CV and LB. Has anyone else experienced similar results?
", "messages": [ { "id": 3235922, "postDate": "2025-06-29T18:58:53.460Z", "content": "Same here, whenever I add anything rnn or attention related to my cnn architecture, performance decreases. 🥸
", "rawMarkdown": "Same here, whenever I add anything rnn or attention related to my cnn architecture, performance decreases. 🥸", "votes": 3 }, { "id": 3235384, "postDate": "2025-06-29T06:44:59.113Z", "content": "Hi everyone, I'm currently a bit stuck with architecture iteration and finding it difficult to effectively incorporate classic temporal models like GRU/LSTM. It seems that adding any additional module to my current 1D-CNN architecture actually reduces performance on both CV and LB. Has anyone else experienced similar results?
", "rawMarkdown": "Hi everyone, I'm currently a bit stuck with architecture iteration and finding it difficult to effectively incorporate classic temporal models like GRU/LSTM. It seems that adding any additional module to my current 1D-CNN architecture actually reduces performance on both CV and LB. Has anyone else experienced similar results?", "votes": 3 }, { "id": 3235495, "postDate": "2025-06-29T10:05:15.330Z", "content": "I use LSTM-based models and will try to beat it with attention-based ones and 1D CNN.
", "rawMarkdown": "I use LSTM-based models and will try to beat it with attention-based ones and 1D CNN.", "votes": 4, "replies": [ { "id": 3246430, "postDate": "2025-07-10T23:37:55.697Z", "content": "Hey hey, any results to share from your attention and CNN experiments?
", "rawMarkdown": "Hey hey, any results to share from your attention and CNN experiments? ", "replies": [ { "id": 3246531, "postDate": "2025-07-11T05:23:07.213Z", "content": "Attention is on par with LSTM!
", "rawMarkdown": "Attention is on par with LSTM!", "votes": 1 }, { "id": 3246726, "postDate": "2025-07-11T12:06:08.883Z", "content": "Thanks for the response. Are you using an implementation that's different from public notebooks? I'll revisit my testing with attention.
", "rawMarkdown": "Thanks for the response. Are you using an implementation that's different from public notebooks? I'll revisit my testing with attention. " }, { "id": 3246743, "postDate": "2025-07-11T12:44:51.137Z", "content": "I don't even remember any public notebook with pure attention-based architecture.
", "rawMarkdown": "I don't even remember any public notebook with pure attention-based architecture." }, { "id": 3246792, "postDate": "2025-07-11T14:41:30.640Z", "content": "Are you using attention pooling? I'm pretty sure the biggest improvements I can make to my models relate to pooling before class predictions.
", "rawMarkdown": "Are you using attention pooling? I'm pretty sure the biggest improvements I can make to my models relate to pooling before class predictions. " } ] } ] }, { "id": 3236191, "postDate": "2025-06-30T04:03:32.913Z", "content": "i tried using attention and my score dropped too!
", "rawMarkdown": "i tried using attention and my score dropped too!", "votes": 1 }, { "id": 3235459, "postDate": "2025-06-29T09:26:31.620Z", "content": "Hello, I am in the process of doing some ablation tests to my current IMU only architecture over the next week, but I remember some random attempts of incorporating LSTM also not working. Same thing for any sort of attention usage, but I will revisit both.
", "rawMarkdown": "Hello, I am in the process of doing some ablation tests to my current IMU only architecture over the next week, but I remember some random attempts of incorporating LSTM also not working. Same thing for any sort of attention usage, but I will revisit both.", "votes": 2 }, { "id": 3246734, "postDate": "2025-07-11T12:30:52.803Z", "content": "Long-term dependency causes overfit on the imu data.
", "rawMarkdown": "Long-term dependency causes overfit on the imu data." }, { "id": 3236979, "postDate": "2025-06-30T19:38:07.923Z", "content": "I think using gru after cnn does help me in my model
", "rawMarkdown": "I think using gru after cnn does help me in my model", "replies": [ { "id": 3236989, "postDate": "2025-06-30T19:48:18.343Z", "content": "BTW, did you reach 0.84 with imu only model?
", "rawMarkdown": "BTW, did you reach 0.84 with imu only model?", "replies": [ { "id": 3237581, "postDate": "2025-07-01T07:16:57.063Z", "content": "nope, IMU only 0.82 for now
", "rawMarkdown": "nope, IMU only 0.82 for now", "votes": 3 }, { "id": 3237645, "postDate": "2025-07-01T07:58:49.323Z", "content": "Unbelievable, that still so high.😂
", "rawMarkdown": "Unbelievable, that still so high.😂" }, { "id": 3237686, "postDate": "2025-07-01T08:32:44.030Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3237687, "postDate": "2025-07-01T08:33:00.570Z", "content": "if it's not a secret, is it single model or ensemble?
", "rawMarkdown": "if it's not a secret, is it single model or ensemble?" }, { "id": 3237688, "postDate": "2025-07-01T08:33:34.537Z", "content": "single model
", "rawMarkdown": "single model", "votes": 1 }, { "id": 3237759, "postDate": "2025-07-01T09:36:35.733Z", "content": "so, features other than imu are of no use at all?
", "rawMarkdown": "so, features other than imu are of no use at all?" } ] } ] } ], "comments": [ { "id": 3235922, "author_name": "Jan Brederecke", "author_url": "", "post_date": "2025-06-29T18:58:53.460000", "content": "Same here, whenever I add anything rnn or attention related to my cnn architecture, performance decreases. 🥸
", "votes": 3, "replies": [] }, { "id": 3235495, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-06-29T10:05:15.330000", "content": "I use LSTM-based models and will try to beat it with attention-based ones and 1D CNN.
", "votes": 4, "replies": [ { "id": 3246430, "author_name": "Jack", "author_url": "", "post_date": "2025-07-10T23:37:55.697000", "content": "Hey hey, any results to share from your attention and CNN experiments?
", "votes": 0, "replies": [ { "id": 3246531, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-07-11T05:23:07.213000", "content": "Attention is on par with LSTM!
", "votes": 1, "replies": [] }, { "id": 3246726, "author_name": "Jack", "author_url": "", "post_date": "2025-07-11T12:06:08.883000", "content": "Thanks for the response. Are you using an implementation that's different from public notebooks? I'll revisit my testing with attention.
", "votes": 0, "replies": [] }, { "id": 3246743, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-07-11T12:44:51.137000", "content": "I don't even remember any public notebook with pure attention-based architecture.
", "votes": 0, "replies": [] }, { "id": 3246792, "author_name": "Jack", "author_url": "", "post_date": "2025-07-11T14:41:30.640000", "content": "Are you using attention pooling? I'm pretty sure the biggest improvements I can make to my models relate to pooling before class predictions.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3236191, "author_name": "Vaishnavi Mudaliar", "author_url": "", "post_date": "2025-06-30T04:03:32.913000", "content": "i tried using attention and my score dropped too!
", "votes": 1, "replies": [] }, { "id": 3235459, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-06-29T09:26:31.620000", "content": "Hello, I am in the process of doing some ablation tests to my current IMU only architecture over the next week, but I remember some random attempts of incorporating LSTM also not working. Same thing for any sort of attention usage, but I will revisit both.
", "votes": 2, "replies": [] }, { "id": 3246734, "author_name": "Tom", "author_url": "", "post_date": "2025-07-11T12:30:52.803000", "content": "Long-term dependency causes overfit on the imu data.
", "votes": 0, "replies": [] }, { "id": 3236979, "author_name": "menglei", "author_url": "", "post_date": "2025-06-30T19:38:07.923000", "content": "I think using gru after cnn does help me in my model
", "votes": 0, "replies": [ { "id": 3236989, "author_name": "menglei", "author_url": "", "post_date": "2025-06-30T19:48:18.343000", "content": "BTW, did you reach 0.84 with imu only model?
", "votes": 0, "replies": [ { "id": 3237581, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-07-01T07:16:57.063000", "content": "nope, IMU only 0.82 for now
", "votes": 3, "replies": [] }, { "id": 3237645, "author_name": "menglei", "author_url": "", "post_date": "2025-07-01T07:58:49.323000", "content": "Unbelievable, that still so high.😂
", "votes": 0, "replies": [] }, { "id": 3237686, "author_name": "", "author_url": "", "post_date": "2025-07-01T08:32:44.030000", "content": "", "votes": 0, "replies": [] }, { "id": 3237687, "author_name": "lightsource<3", "author_url": "", "post_date": "2025-07-01T08:33:00.570000", "content": "if it's not a secret, is it single model or ensemble?
", "votes": 0, "replies": [] }, { "id": 3237688, "author_name": "Ryenhails", "author_url": "", "post_date": "2025-07-01T08:33:34.537000", "content": "single model
", "votes": 1, "replies": [] }, { "id": 3237759, "author_name": "Vaishnavi Mudaliar", "author_url": "", "post_date": "2025-07-01T09:36:35.733000", "content": "so, features other than imu are of no use at all?
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3235922": "Same here, whenever I add anything rnn or attention related to my cnn architecture, performance decreases. 🥸", "3235384": "Hi everyone, I'm currently a bit stuck with architecture iteration and finding it difficult to effectively incorporate classic temporal models like GRU/LSTM. It seems that adding any additional module to my current 1D-CNN architecture actually reduces performance on both CV and LB. Has anyone else experienced similar results?", "3235495": "I use LSTM-based models and will try to beat it with attention-based ones and 1D CNN.", "3236191": "i tried using attention and my score dropped too!", "3235459": "Hello, I am in the process of doing some ablation tests to my current IMU only architecture over the next week, but I remember some random attempts of incorporating LSTM also not working. Same thing for any sort of attention usage, but I will revisit both.", "3246734": "Long-term dependency causes overfit on the imu data.", "3236979": "I think using gru after cnn does help me in my model" } }