I saw lots of notebooks using the conv + lstm framework, but given our sequences are of variable length, some short sequences are padded with lots of zeros at the end, would conv layer work as expected?
Suppose I have a sequence with length 50, and it is padded to 200, so the last 150 timesteps are all zeros. With the conv + lstm frame work, in the last 150 timesteps the conv result will all be zeros, thus meaningless, so when sending to LSTM, we would want the LSTM to ignore those steps, however I didn't see any notebook track the valid timesteps after multiple convolutional layers.
\nI wonder is it actually legit to use this framework here?
The ideal way is to use
\n x_packed = pack_padded_sequence(x, seq_len, =, =)\n out_packed, (h_n, c_n) = self.lstm(x_packed)\n out, _ = pad_packed_sequence(out_packed, =) \nbut after multiple layers of conv, it is hard to track the valid length of seq_len.
I saw lots of notebooks using the conv + lstm framework, but given our sequences are of variable length, some short sequences are padded with lots of zeros at the end, would conv layer work as expected?
Suppose I have a sequence with length 50, and it is padded to 200, so the last 150 timesteps are all zeros. With the conv + lstm frame work, in the last 150 timesteps the conv result will all be zeros, thus meaningless, so when sending to LSTM, we would want the LSTM to ignore those steps, however I didn't see any notebook track the valid timesteps after multiple convolutional layers.
\nI wonder is it actually legit to use this framework here?
The ideal way is to use
\n x_packed = pack_padded_sequence(x, seq_len, =, =)\n out_packed, (h_n, c_n) = self.lstm(x_packed)\n out, _ = pad_packed_sequence(out_packed, =) \nbut after multiple layers of conv, it is hard to track the valid length of seq_len.
Use mask to indicate which position is padding.
", "rawMarkdown": "Use mask to indicate which position is padding.", "votes": null }, { "id": "3247639", "postDate": "07/13/2025 06:38:50", "content": "so the direct way of using conv and then send to LSTM is wrong, is it?
", "rawMarkdown": "so the direct way of using conv and then send to LSTM is wrong, is it?", "votes": null }, { "id": "3247648", "postDate": "07/13/2025 06:57:06", "content": "I also recently tried to handle padding by using pad_packed_sequence and assigning -inf attention scores to padding positions, but it seems to have little effect.
", "rawMarkdown": "I also recently tried to handle padding by using pad_packed_sequence and assigning -inf attention scores to padding positions, but it seems to have little effect.", "votes": null }, { "id": "3247662", "postDate": "07/13/2025 07:17:22", "content": "yeah, but do you understand why that not handling padding works on conv+lstm?
", "rawMarkdown": "yeah, but do you understand why that not handling padding works on conv+lstm?", "votes": null }, { "id": "3247674", "postDate": "07/13/2025 07:39:58", "content": "I used attention to merge all the output from every step like those hot public notebooks , so I guess padding with 0 or -1 is enough for the model to focus on the important steps.
", "rawMarkdown": "I used attention to merge all the output from every step like those hot public notebooks , so I guess padding with 0 or -1 is enough for the model to focus on the important steps.", "votes": null }, { "id": "3272283", "postDate": "08/20/2025 17:00:10", "content": "Hi @zhangyue199, sorry for reaching out here, but I noticed your Kaggle messages are disabled.
\nI wanted to ask if you might be open to teaming up for this competition. If you’re interested, I’d be happy to share more details.
Can I join?
", "rawMarkdown": "Can I join?", "votes": null }, { "id": "3277155", "postDate": "08/27/2025 12:33:23", "content": "Using padding without mask in cnn+lstm means incorporating sequence length as a feature. In fact, my EDA shows significant differences between classes even when excluding the influence of subjects. Also, when excluding the influence of sequence length and using LayerNorm, my IMU-only Conv1d model (without lstm) achieves a score of 0.805 on the LB. I dropped out of this competition, but I believe that effectively incorporating sequence length and tof features would improve the score.
", "rawMarkdown": "Using padding without mask in cnn+lstm means incorporating sequence length as a feature. In fact, my EDA shows significant differences between classes even when excluding the influence of subjects. Also, when excluding the influence of sequence length and using LayerNorm, my IMU-only Conv1d model (without lstm) achieves a score of 0.805 on the LB. I dropped out of this competition, but I believe that effectively incorporating sequence length and tof features would improve the score.", "votes": null } ], "comments": [ { "id": 3247637, "author_name": "tom99763", "author_url": "", "post_date": "07/13/2025 06:30:12", "content": "Use mask to indicate which position is padding.
", "votes": null, "replies": [ { "id": 3247639, "author_name": "zhangyue199", "author_url": "", "post_date": "07/13/2025 06:38:50", "content": "so the direct way of using conv and then send to LSTM is wrong, is it?
", "votes": null, "replies": [] } ] }, { "id": 3247648, "author_name": "minerppdy", "author_url": "", "post_date": "07/13/2025 06:57:06", "content": "I also recently tried to handle padding by using pad_packed_sequence and assigning -inf attention scores to padding positions, but it seems to have little effect.
", "votes": null, "replies": [ { "id": 3247662, "author_name": "zhangyue199", "author_url": "", "post_date": "07/13/2025 07:17:22", "content": "yeah, but do you understand why that not handling padding works on conv+lstm?
", "votes": null, "replies": [ { "id": 3247674, "author_name": "minerppdy", "author_url": "", "post_date": "07/13/2025 07:39:58", "content": "I used attention to merge all the output from every step like those hot public notebooks , so I guess padding with 0 or -1 is enough for the model to focus on the important steps.
", "votes": null, "replies": [] }, { "id": 3272283, "author_name": "junhanzangai", "author_url": "", "post_date": "08/20/2025 17:00:10", "content": "Hi @zhangyue199, sorry for reaching out here, but I noticed your Kaggle messages are disabled.
\nI wanted to ask if you might be open to teaming up for this competition. If you’re interested, I’d be happy to share more details.
Can I join?
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3277155, "author_name": "kyoumonemui", "author_url": "", "post_date": "08/27/2025 12:33:23", "content": "Using padding without mask in cnn+lstm means incorporating sequence length as a feature. In fact, my EDA shows significant differences between classes even when excluding the influence of subjects. Also, when excluding the influence of sequence length and using LayerNorm, my IMU-only Conv1d model (without lstm) achieves a score of 0.805 on the LB. I dropped out of this competition, but I believe that effectively incorporating sequence length and tof features would improve the score.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3247630": "I saw lots of notebooks using the `conv + lstm` framework, but given our sequences are of variable length, some short sequences are padded with lots of zeros at the end, would conv layer work as expected?\n\nSuppose I have a sequence with length 50, and it is padded to 200, so the last 150 timesteps are all zeros. With the `conv + lstm` frame work, in the last 150 timesteps the conv result will all be zeros, thus meaningless, so when sending to LSTM, we would want the LSTM to ignore those steps, however I didn't see any notebook track the valid timesteps after multiple convolutional layers.\nI wonder is it actually legit to use this framework here?\n\nThe ideal way is to use\n\n```\n x_packed = pack_padded_sequence(x, seq_len, batch_first=True, enforce_sorted=False)\n out_packed, (h_n, c_n) = self.lstm(x_packed)\n out, _ = pad_packed_sequence(out_packed, batch_first=True) \n```\nbut after multiple layers of conv, it is hard to track the valid length of `seq_len`.", "3247637": "Use mask to indicate which position is padding.", "3247639": "so the direct way of using conv and then send to LSTM is wrong, is it?", "3247648": "I also recently tried to handle padding by using pad_packed_sequence and assigning -inf attention scores to padding positions, but it seems to have little effect.", "3247662": "yeah, but do you understand why that not handling padding works on conv+lstm?", "3247674": "I used attention to merge all the output from every step like those hot public notebooks , so I guess padding with 0 or -1 is enough for the model to focus on the important steps.", "3272283": "Hi @zhangyue199, sorry for reaching out here, but I noticed your Kaggle messages are disabled.\nI wanted to ask if you might be open to teaming up for this competition. If you’re interested, I’d be happy to share more details.", "3272729": "Can I join?", "3277155": "Using padding without mask in cnn+lstm means incorporating sequence length as a feature. In fact, my EDA shows significant differences between classes even when excluding the influence of subjects. Also, when excluding the influence of sequence length and using LayerNorm, my IMU-only Conv1d model (without lstm) achieves a score of 0.805 on the LB. I dropped out of this competition, but I believe that effectively incorporating sequence length and tof features would improve the score." }, "source": "meta" }