![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16438831%2F4e6d5e8edfa6c2798f54d583f416c017%2Fmodel1.JPG?generation=1756867480286012&alt=media\")
\nWaveNet is superior at finding patterns in waveforms. Using dilated convolutions, it decomposes a signal into tis different frequency since waves just like the Fourier Transfomer.
First of all, I would like to thank my teemmate @jaewook704 and the Child Mind Institute for hosting this competition. It has been a great learning experience for me. I also want to acknowledge and appreciate everyone who dedicated their time and effort over the past three months to participate in this challenge.
\nOne of my main motivations for joining was to study WaveNet in greater depth. However, during the experiments, I found that temporal 1D CNNs performed better than WaveNet, and as a result, my score was not as high as I had hoped.
\nThat said, I would like to take this opportunity to share the model architectures I explored throughout the competition.
\nWaveNet is superior at finding patterns in waveforms. Using dilated convolutions, it decomposes a signal into tis different frequency since waves just like the Fourier Transfomer.
And then I modiy the original dilated convolution to dilated inception convolution. Concretely speaking, instead of using only one kernel size per layer in WaveBlock , this model tries to capture enriched temporal patterns by considering different kernel sizes(2,4,6,8) at the same time. Futhermore, to restrain model parameters from growing too much, the model narrows down the output channels of convolutions with different kernel sizes. Let out_channels = 64, each dilated convolution with a specific kernel size only outputs 16 channels, then the concatenation along channel dimension is applied to match out_channels = 64
\n
class (nn.Module):\n def (self, in_channels, out_channels, dilation_rates, kernel_size, conv_module):\n (Wave_Block, self).()\n self.num_rates = dilation_rates\n self.convs = nn.()\n self.filter_convs = nn.()\n self.gate_convs = nn.()\n self.convs.(nn.(in_channels, out_channels, kernel_size=, bias=True))\n dilation_rates = [ ** i for iin (dilation_rates)]<pre><code> for dilation_ratein dilation_rates:\n self.filter_convs.(\n (out_channels, out_channels, kernel_size, dilation_rate)\n )\n self.gate_convs.(\n (out_channels, out_channels, kernel_size, dilation_rate)\n )\n self.convs.(\n nn.(out_channels, out_channels, kernel_size=, bias=True)\n )\n\n for iin ((self.convs)):\n nn.init.(self.convs[i].weight, gain=nn.init.())\n nn.init.(self.convs[i].bias)\n\n for min self.filter_convs:\n for convin m.convs:\n nn.init.(conv.weight, gain=nn.init.())\n nn.init.(conv.bias)\n\n for min self.gate_convs:\n for convin m.convs:\n nn.init.(conv.weight, gain=nn.init.())\n nn.init.(conv.bias)\n\ndef (self,x):\n x = self.convs[](x)\n res = x# skip_connectionfor iin (self.num_rates):\n tanh_out = torch.(self.filter_convs[i](x))\n sigmoid_out = torch.(self.gate_convs[i](x))\n x = tanh_out * sigmoid_out\n x = self.convs[i+](x)\n res = res + x\n\n return res\n</code></pre>\n
If you want to learn more about dilated inception wavenet,
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16438831%2Fa89dd579f3818cbf2d1d4127f0383f47%2Fmodel2.JPG?generation=1756867550984042&alt=media\")
we propose a hybrid architecture that integrates WaveNet with a Transformer encoder to effecitively capture both local temporal patterns and long-range contextual information.
\nSince the Tranformer architecture does not inherently encode sequential order, it is necessary to inject positional information into the input features. We consider two widely adopted startgies for positional encoding
\nMethod 1 | Fixed Sinusoidal Encoding
\nA deterministic embedding function based on sine and cosine with varying frequencies.
\ndef (seq_len, feature_dim):\n\n depth = feature_dim // \n positions = torch.(seq_len, dtype=torch.float32).()\n\n depths = torch.(depth, dtype=torch.float32).() / depth\n\n angle_rates = / (_000 ** depths)\n angle_rads = positions @ angle_rates\n\n pos_encoding = torch.([torch.(angle_rads), torch.(angle_rads)], dim=-) \n\n return pos_encoding\nMethod 2 | Learnable Positional Embedding
\nA trainable embedding vector assigned to each position index.
\nFeature Extraction via WaveNet
\nWaveNet is employed as a feature extractor to model fine-grained temporal dependencies in the input sequence. Its dilated causal convolutional structure enables the network to capture multi-scale temporal patterns without loss of resolution. The output of WaveNet serves as a high-level representation that is subsequently fed into the Transformer encoder.
\nAnd then Finally, yon can also try [WaveNet + Transformer + GRU]
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16438831%2F29c1ec9a3e630d63f1dc1cb343b709ba%2Fmodel3.JPG?generation=1756867571658122&alt=media\")
xLSTM
\n⇒ xLSTM GitHub: https://github.com/NX-AI/xlstm
\n⇒ xLSTM Discussion: https://www.kaggle.com/discussions/general/501522
\nxLSTM is a new Recurrent Neural Network architecture based on ideas of the original LSTM. Through Exponential Gating with appropriate normalization and stabilization techniques and a new Matrix Memory it overcomes the limitations of the original LSTM and shows promising performance on Language Modeling when compared to Transformers or State Space Models.
\nTemporal Attention Mechanism
\nOn top of the xLSTM outputs, an Attention Pooling is applied
\nThis hybridization address both local structure(WaveNet), long-term dependencies(xLSTM), and global context(Attention).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16438831%2F9c5f2eadf06c9ffd19dc8b4a2efc3d23%2Fmodel4.JPG?generation=1756867592334566&alt=media\")
1. Data Characteristics and Motivation
\nExploratory Data Analysis(EDA) revealed that each input sequence consists of two distinct phases: a transition phase and a gesture phase. On average, both phases account for approximately 50% of the total sequence length. The transition phase primarily reflects temporal variations occuring before the execution of a gesture, while the gesture phase encodes discriminative motion patterns corresponding to the performed action.
\nThis observation motivated us to design a phase-aware feature extraction strategy, where the model separately processes the two phase rather than treating the sequence as a homogeneous temporal signal.
\n2. Phase-Specific Feature Extraction
\nTo explicitly leverage the different temporal characteristics of each phase, we divided the sequence into two equal segments.
\nFirst 50%(Transition Part):
\nThis portion contains gradual temporal variations leading to gestures We employed a WaveNet-based module due to its ability to capture long-range dependencies and temporal dynamics with dilated causal convolutions. The WaveNet architecture is particularly effective at modeling non-stationary temporal transitions, making it well-suited for this phase.
Last 50%(Gesture Part):
\nThis segment contains the core gesture information with relatively stable temporal structures. We applied a Temporal 1D Convolutional Neural Network(1D - CNN) to capture localied and discriminative motion patterns. The convolutiona filters effectively extract fine-grained temporal features, which are crucial for gesture recognition.
First of all, I would like to thank my teemmate @jaewook704 and the Child Mind Institute for hosting this competition. It has been a great learning experience for me. I also want to acknowledge and appreciate everyone who dedicated their time and effort over the past three months to participate in this challenge.
\nOne of my main motivations for joining was to study WaveNet in greater depth. However, during the experiments, I found that temporal 1D CNNs performed better than WaveNet, and as a result, my score was not as high as I had hoped.
\nThat said, I would like to take this opportunity to share the model architectures I explored throughout the competition.
\nWaveNet is superior at finding patterns in waveforms. Using dilated convolutions, it decomposes a signal into tis different frequency since waves just like the Fourier Transfomer.
And then I modiy the original dilated convolution to dilated inception convolution. Concretely speaking, instead of using only one kernel size per layer in WaveBlock , this model tries to capture enriched temporal patterns by considering different kernel sizes(2,4,6,8) at the same time. Futhermore, to restrain model parameters from growing too much, the model narrows down the output channels of convolutions with different kernel sizes. Let out_channels = 64, each dilated convolution with a specific kernel size only outputs 16 channels, then the concatenation along channel dimension is applied to match out_channels = 64
\n
class (nn.Module):\n def (self, in_channels, out_channels, dilation_rates, kernel_size, conv_module):\n (Wave_Block, self).()\n self.num_rates = dilation_rates\n self.convs = nn.()\n self.filter_convs = nn.()\n self.gate_convs = nn.()\n self.convs.(nn.(in_channels, out_channels, kernel_size=, bias=True))\n dilation_rates = [ ** i for iin (dilation_rates)]<pre><code> for dilation_ratein dilation_rates:\n self.filter_convs.(\n (out_channels, out_channels, kernel_size, dilation_rate)\n )\n self.gate_convs.(\n (out_channels, out_channels, kernel_size, dilation_rate)\n )\n self.convs.(\n nn.(out_channels, out_channels, kernel_size=, bias=True)\n )\n\n for iin ((self.convs)):\n nn.init.(self.convs[i].weight, gain=nn.init.())\n nn.init.(self.convs[i].bias)\n\n for min self.filter_convs:\n for convin m.convs:\n nn.init.(conv.weight, gain=nn.init.())\n nn.init.(conv.bias)\n\n for min self.gate_convs:\n for convin m.convs:\n nn.init.(conv.weight, gain=nn.init.())\n nn.init.(conv.bias)\n\ndef (self,x):\n x = self.convs[](x)\n res = x# skip_connectionfor iin (self.num_rates):\n tanh_out = torch.(self.filter_convs[i](x))\n sigmoid_out = torch.(self.gate_convs[i](x))\n x = tanh_out * sigmoid_out\n x = self.convs[i+](x)\n res = res + x\n\n return res\n</code></pre>\n
If you want to learn more about dilated inception wavenet,
\n\n
we propose a hybrid architecture that integrates WaveNet with a Transformer encoder to effecitively capture both local temporal patterns and long-range contextual information.
\nSince the Tranformer architecture does not inherently encode sequential order, it is necessary to inject positional information into the input features. We consider two widely adopted startgies for positional encoding
\nMethod 1 | Fixed Sinusoidal Encoding
\nA deterministic embedding function based on sine and cosine with varying frequencies.
\ndef (seq_len, feature_dim):\n\n depth = feature_dim // \n positions = torch.(seq_len, dtype=torch.float32).()\n\n depths = torch.(depth, dtype=torch.float32).() / depth\n\n angle_rates = / (_000 ** depths)\n angle_rads = positions @ angle_rates\n\n pos_encoding = torch.([torch.(angle_rads), torch.(angle_rads)], dim=-) \n\n return pos_encoding\nMethod 2 | Learnable Positional Embedding
\nA trainable embedding vector assigned to each position index.
\nFeature Extraction via WaveNet
\nWaveNet is employed as a feature extractor to model fine-grained temporal dependencies in the input sequence. Its dilated causal convolutional structure enables the network to capture multi-scale temporal patterns without loss of resolution. The output of WaveNet serves as a high-level representation that is subsequently fed into the Transformer encoder.
\nAnd then Finally, yon can also try [WaveNet + Transformer + GRU]
\n
xLSTM
\n⇒ xLSTM GitHub: https://github.com/NX-AI/xlstm
\n⇒ xLSTM Discussion: https://www.kaggle.com/discussions/general/501522
\nxLSTM is a new Recurrent Neural Network architecture based on ideas of the original LSTM. Through Exponential Gating with appropriate normalization and stabilization techniques and a new Matrix Memory it overcomes the limitations of the original LSTM and shows promising performance on Language Modeling when compared to Transformers or State Space Models.
\nTemporal Attention Mechanism
\nOn top of the xLSTM outputs, an Attention Pooling is applied
\nThis hybridization address both local structure(WaveNet), long-term dependencies(xLSTM), and global context(Attention).
\n
1. Data Characteristics and Motivation
\nExploratory Data Analysis(EDA) revealed that each input sequence consists of two distinct phases: a transition phase and a gesture phase. On average, both phases account for approximately 50% of the total sequence length. The transition phase primarily reflects temporal variations occuring before the execution of a gesture, while the gesture phase encodes discriminative motion patterns corresponding to the performed action.
\nThis observation motivated us to design a phase-aware feature extraction strategy, where the model separately processes the two phase rather than treating the sequence as a homogeneous temporal signal.
\n2. Phase-Specific Feature Extraction
\nTo explicitly leverage the different temporal characteristics of each phase, we divided the sequence into two equal segments.
\nFirst 50%(Transition Part):
\nThis portion contains gradual temporal variations leading to gestures We employed a WaveNet-based module due to its ability to capture long-range dependencies and temporal dynamics with dilated causal convolutions. The WaveNet architecture is particularly effective at modeling non-stationary temporal transitions, making it well-suited for this phase.
Last 50%(Gesture Part):
\nThis segment contains the core gesture information with relatively stable temporal structures. We applied a Temporal 1D Convolutional Neural Network(1D - CNN) to capture localied and discriminative motion patterns. The convolutiona filters effectively extract fine-grained temporal features, which are crucial for gesture recognition.