# 119th Place Solution: Transformer is much much better than GRU!

First, I want to thank
Kaggle
,
PopSign
,
Google
, and
Partners
for organizing this interesting competition! It provided me a great opportunity to learn and compete with other participants. Second, I want to congratulate to all the winners and other Kagglers for participating in this competition.
To document my learning journey in this competition, I want to share my approach towards this competition. The full inference code is publicly shared as
[GISLR: Inference Transformer](https://www.kaggle.com/code/nghihuynh/gislr-inference-transformer)
.
Data Preprocessing:
![](assets/896dc5774e7e90b118a4becf.jpg)
I applied a simple data preprocessing pipeline with a small modification (without data normalization) from my shared notebook:
[GISLR: EDA + Feature Processing](https://www.kaggle.com/code/nghihuynh/gislr-eda-feature-processing)
. Then, I saved it as featured_preprocessed data for further training.
Model Architecture:
![](assets/5e7c4eb906cd0c8de5c74196.jpg)
Model
Architecture
Original Size
Post Quantization (TFLite FP16 )
GRUNet
Landmark embedding:128; Combined embedding:396; GRU units:368; Conv1D: filter: 256, kernel_size: 5
8.78MB
4.4MB
BiGRUNet
Landmark embedding: 128; GRU units: 768; BiGRU units: 396; Conv1D: filter:396, kernel_size:5
30.84MB
15.4MB
Transformer
Landmark embedding: 256; Combined embedding: 396; GRU units: 368; Transformer: MHA units: 368, n_head:8, num_block:1; Conv1D: filters:396, kernel_size:5
13.87MB
6.9MB
Cross-Validation
: Participant K-Fold=4
2 sets of validation from fold 3= [62590, 36257, 22343, 37779]
GRU: val_set= [22343, 37779]
BiGRU & Transformer: val_set=[37779]
Training:
Reduce LR on plateau: factor=0.30, patient=3
Adam optimizer: lr=4.0e-4
Sparse Categorical Loss Entropy
Epochs:100
Results:
Model (Ensemble)
Public LB
Private LB
GRU-BiGRU-Transformer (selected as final submission)
0.7452
0.82339
Transformer (not-selected 🙁)
0.73918
0.82484
Transformer (x2)-GRU (not-selected 🙁)
0.74217
0.82457
Conclusion:
Special thanks to the following public notebooks, which helped me a lot during the competition.
[GISLR TF Data Processing & Transformer Training by Mark Wijkhuizen](https://www.kaggle.com/code/markwijkhuizen/gislr-tf-data-processing-transformer-training)
[GISLR TF: On the Shoulders ENSAMBLE V2 0.69 by Andrij](https://www.kaggle.com/code/aikhmelnytskyy/gislr-tf-on-the-shoulders-ensamble-v2-0-69)
[LSTM Baseline for Starters - Sign Language by JvThunder](https://www.kaggle.com/code/jvthunder/lstm-baseline-for-starters-sign-language)
[[LB 0.67] one pytorch transformer solution by hengck23](https://www.kaggle.com/code/hengck23/lb-0-67-one-pytorch-transformer-solution)
Things I tried but didn't work:
Different preprocessing (increase number of frames, include more pose, and eyes data, normalize data)
Positional embedding(overfitting)
Attention mask in GRU, BiGRU(overfitting)